Daily Blog #673: Working AWS EBS Blocks

Hello Reader,
     In the yesterdays blog we tested out the AWS Direct EBS Block API. This allowed us to extract any block we choose from a AWS EBS Snapshot. In the test kitchen below we go deeper, looking into how many 512 sectors make up a Snapshot block (1024) and parse out the MBR to show how to work with the data thats retrieved.

We then ended the recording with some questions we need to answer:
1. The snapshot block index had gaps, did I not make a full snapshot? We will have to experiment and find out
2. Looking quickly at my cloudtrail logs I didn't see my API actions, so tomorrow we will use AWS Glue to crawl my cloudtrail logs and search for the activity we've created.

See you tomorrow and until then you can watch the video below:

Also Read: Daily Blog #672 -  AWS EBS Snapshot Block Access

Read More

Daily Blog #640: Regipy - A New Python Windows Registry Forensics Library

Hello Reader,
        As I was talking about in #638 I believe automation in DFIR is a big part of our future. With the idea of automating the extraction and basic correlation of data so that a human can use their brain. As we work towards that I'm always looking for new libraries that can support that effort, especially lately if they are written in Python.

So I was happy to see that Martin Korman (who writes the DFIR Dudes blog with Hada Yudovich the winner of the 2018 Defcon DFIR CTF) put out a new python library for parsing Windows Registries.

Read the blog here:
https://medium.com/dfir-dudes/regipy-automating-registry-forensics-with-python-b170a1e2b474

See the code here:
https://github.com/mkorman90/regipy

What's interesting is that Martin has taken alot of the registry parsing and transaction log handling we've seen in YARP and added on the ability for creating simple plugins to automatically parse the data it extracts.

I haven't had a chance to compare the library or its output to any other, but I'm always happy to see more options out there. If nothing else take a look at the code to get an idea of how to handle these kinds of data structures in Python. 

Also Read: Daily Blog #639

Read More

Daily Blog #575: Solution Saturday 12/22/18 - DFVFS Challenge Solution

Hello Reader,
I always love introducing new winners to the community and this week I get my wish. Please congratulate Bastien Lardy with his winning Python DFVFS submission!

The Challenge:

Write a python script using DFVFS that uses the source scanner function to enumerate partitions and shadow copies. It should then provide the ability to extract a file and provide its hash. What additional functionality you decide to add in from there will determine which answer is the most complete. If you need test images to code against consider the Defcon CTF images. 

The winning answer:

This is a python2 (I had issue with python3 and mediator...) script that reads an input disk image and searches (based on full path or regex filters), extracts or computes hash. If shadow copies exist, it will prompt a message whether to process those or not.

https://github.com/blardy/dfvfs_extractor

Also Read: Daily Blog #574

Read More

Daily Blog #572: Forensic Lunch Test Kitchen 12/19/18 Syscache and Python

Hello Reader,
      Tonight we wrote some python code to recover the full path of the files referenced in the Syscache hive, added in the ProgramID and then viewed the data in Timeline Explorer to see the relation between the executables. We learned:

• That pytsk does not have an attribute for parent reference number, so we had to extract it from the file name attribute
• That analyzemft has a great set of example code to pull your unpack's from if you are looking to write your own attribute parser
• That when I grouped my syscache entries by programID I only had 60+ entries which seems more like just what has been executed on this lightly used VM
• That there is no entry of any program run directly from my Desktop

You can get the code here: https://github.com/dlcowen/TestKitchen/blob/master/PrintFileNamesByEntry.py

You can watch the video here:

Also Read: Daily Blog #571 

Read More

Daily Blog #570: Forensic Lunch Test Kitchen 12/17/18 Syscache.hve

Hello Reader,
       Tonight in the Test Kitchen we expanded our testing of the Syscache hive by adding more data from our python script that is matching MFT entries to the Syscache entries. Here is what we learned:

• The syscache hive seems to record atleast exe, dll, bat and cmd files executed
• The syscache hive like the Amcache hive will store which program by sha1 hash the executable is associated with
• If there is no associated executable (no MSI installer) it will use the sha1 hash 'da39a3ee5e6b4b0d3255bfef95601890afd80709' which is the empty hash. Meaning like the Amcache it is storing information about executables not associated with any MSI installed program
• It appears, this needs more testing, that it may record more related executables than the Amcache does when a program is installed/executed

Still much more testing to be done to really make sure we understand what we are really seeing. To  be tested:

1. Installing a program but not executing it, will it appear
2. Adding full path support to the python script to understand where the associated files are being pulled from
3. Checking to see if powershell scripts get logged
4. Looking to see if deletions of programs/executables change the hive
5. Testing again to see if programs run from the desktop are include

You can watch the video here:

Also Read: Daily Blog #569 

Read More

Daily Blog #566: Forensic Lunch Test Kitchen 12/13/18

Hello Reader,
         This was another test kitchen were we mainly got some python code to work and in the end were able to print all of the file name's out of the file name attributes for every file referenced in the Syscache hive Object key. This isn't done though as next week I need to add in the sequence numbers to the checks to make sure I'm looking at the right file.

So next week we will be able to start making some observations about what exactly Syscache is actually tracking.

You can watch me use Eric Zimmerman's new Syscache plugin and write python code to parse the filename attribute here:

Also Read: Seeing Double (Access Dates)

Read More

Daily Blog #525: Office 2016 Backstage Artifact Parser

Hello Reader,
            One of the things I love the most is collaboration within the DFIR world. Today I'm happy to link to Brian Gerdon's (of Arsenal Recon) implementation of the Office 2016 backstage artifact into a python parser so you don't have to just stare at a bunch of text files or json files. You can find it here:

https://github.com/ArsenalRecon/BackstageParser

There is no better way to learn the details of an artifact that code to a parser for it and learn all the structures and nuances. So if you see something you think is interesting don't feel that you shouldn't try to write a parser for it just because one already exists, the learning experience alone will be worth your effort.

Also Read: Daily Blog #524

Read More

Daily Blog #473: Sunday Funday 9/9/18 - Python Program Challenge

Hello Reader,
           Another week passes and I'm full of ideas of things I want to test, program and try. After Phill Moore's program to recursively call fsutil to determine ObjectIDs I thought maybe a smaller scale challenge would help us all move forward in this path. So this week we are doing another Python programming challenge but with possibly a smaller scope.


The Prize:

$100 Amazon Giftcard

The Rules:

1. You must post your answer before Friday 9/14/18 7PM CST (GMT -5)
2. The most complete answer wins
3. You are allowed to edit your answer after posting
4. If two answers are too similar for one to win, the one with the earlier posting time wins
5. Be specific and be thoughtful
6. Anonymous entries are allowed, please email them to dcowen@g-cpartners.com. Please state in your email if you would like to be anonymous or not if you win.
7. In order for an anonymous winner to receive a prize they must give their name to me, but i will not release it in a blog post

The Challenge:

Writing a program in Python to parse the the $O ADS stream found at \$Extend\$ObjId:$O in any NTFS drive with Windows running. 

Good luck!

Also Read: Daily Blog #472 

Read More

Daily Blog #471: Gearing up for more dfvfs programming

Hello Reader,
        In my attempt to get a Windows VM up and running as a test development environment for doing some tutorials on 64 bit Python 3.6 and DFVFS I ran into an interesting challenge I thought I would document here to help you and myself in the future when I forget I figured this out.

If you are installing a library like PYTSK that requires some Visual Studio runtime DLLs to be installed then you'll get an interesting error like

"python can't find installed module"

when attempting to import the module. I hit this when I was using the windows python 3.6 install from python.org. I attempted different versions of Python 3.6 only in the end to discover that the real issue was the missing DLL. So instead just install the activestate version of Python 3.6 and it will install the needed DLLs to get the libraries working.

Looking forward to documenting more as the week goes on!

Also Read: Daily Blog #470

Read More

Daily Blog #469: Book Highlight Learning Python for Forensics

Hello Reader,
            If you've read some of the older blog series you know that I'm a big proponent of getting new and old examiners programming. In my Automating DFIR series I focused on people who already knew Python and wanted to learn how to interact with forensic images in their scripting, but it didn't help those examiners who didn't have a good place to start with programming in the first place.

I've started reading Learning Python for Forensics and the Python Digital Forensics Cookbook to see what Preston Miller and Chapin Bryce had to say. So far I think they've done an excellent job getting new programmers up to speed with useful scripts that will help to automate their workflow.

You should go check it out here:

https://www.amazon.com/Learning-Python-Forensics-Preston-Miller/dp/1783285230

Also Read: Daily Blog #468

Read More

FSEventsParser 3.1 Released

By Nicole Ibrahim

G-C Partners' FSEventsParser python script 3.1 has been released. Version 3.1 now supports parsing macOS High Sierra FSEvents.

You can get the updated script here: https://github.com/dlcowen/FSEventsParser 

Prior versions of the script do not support High Sierra parsing, so it's important to upgrade to the current version of FSEventsParser.

Other recent updates include:

• Better handling of carved gzip files has been added. Invalid record entries in corrupted gzips are now being excluded from the output reports.

• Even more dates are being found using the names of system and application logs within each fsevent file. The dates are stored in the column 'approx_dates(plus_minus_one_day)' and indicates the approximate date or date range that the event occurred, plus or minus one day.

• Script now reads a json file that contains custom SQLite queries to filter and export targeted reports from the database during parsing.

macOS High Sierra 10.13 and FSEvents

With the release of High Sierra, updates to the FSEvents API resulted in the following changes:

• Magic Header: In macOS versions prior to 10.13, the magic header within a decompressed FSEvents log was '1SLD'. Beginning with 10.13, the magic header is now '2SLD'.

• ItemCloned Flag: The ItemCloned flag was introduced with macOS 10.13.  When set, it indicates that the file system object at the specific path supplied in the event is a clone or was cloned. 

•  File System Node ID: Beginning with 10.13, FSEvents records now contain a File System Node ID. 

• e.g. If FSEvents were from an HFS+ formatted volume, this value would represent the Catalog Node ID.

FSEventsParser Database Report Views

Within the SQLite database, report views have been added for common artifacts. The report views are defined in the 'report_queries.json' file. They include:

• Downloads Activity

• Mount Activity

• Browser Activity

• User Profile Activity

• Dropbox Activity

• Email Attachments Activity

• and more..

To access the report views, open the SQLite database generated by running the script using your SQLite viewer of choice. Expand "Views".

FSEventsParser Custom Reports

The FSEventsParser script now exports custom report views from the database during processing to individual TSV files.


The custom report views are defined in the file 'report_queries.json' which is also available on GitHub.

Users can modify the queries or add new ones to the json file using a text editor. Two examples are shown below: TrashActivity and MountActivity.

To add new queries to the json processing list, follow the json syntax shown below. Define the report views within the 'processing_list' array. To add a new item to the array, define:
1) 'report_name': The report/view name.
2) 'query': The SQLite query to be run.

Notes:

• The report name must be unique and must match the view name in the SQLite query. e.g.
• 'report_name': 'TrashActivity'
• 'query':'CREATE VIEW TrashActivity AS ....'

• The query follows standard SQLite syntax, must be valid, and is stored in the json file as a single-line string value.

FSEventsParser Usage

All options are required when running the script. 

==========================================================================
FSEParser v 3.1  -- provided by G-C Partners, LLC
==========================================================================

Usage: FSEParser_V3.1.py -c CASENAME -q REPORT_QUERIES -s SOURCEDIR -o OUTDIR

Options:
  -h, --help        show this help message and exit
  -c CASENAME       The name of the current session, used for naming standards
  -q REPORTQUERIES  The location of the report_queries.json file containing custom report
                    queries to generate targeted reports
  -s SOURCEDIR      The source directory containing fsevent files to be parsed
  -o OUTDIR         The destination directory used to store parsed reports

 Below is an example of running the script.


For more information about FSEvents and how you can use them in your investigation visit http://nicoleibrahim.com/apple-fsevents-forensics/.

If you have any comments or questions, please feel free to leave them below.

Also Read: Daily Blog #525: Office 2016 Backstage Artifact Parser

Read More

Forensic Lunch 4/3/15 - Devon Kerr - WMI and DFIR and Automating DFIR

Hello Reader,

We had another great Forensic Lunch!

Guests this week:
Devon Kerr talking about his work at Mandiant/Fireeye and his research into WMI for both IR and attacker usage.

• You can email Devon here: devon.kerr@mandiant.com 
• You can follow him on twitter here: https://twitter.com/_devonkerr_
• Get cool tools from the Mandiant github here: https://github.com/mandiant
• Watch Devon talk more about WMI and IR at the SANS DFIR Summit: http://dfir.to/1BvOw7G

Matthew and I going into the Automating DFIR series and our upcoming talk at CEIC

• We are on the CEIC agenda here:
• https://www.guidancesoftware.com/ceic/Pages/ceic-agenda-table.aspx
• and here is the latest post in the Automating DFIR Series:
•  https://www.hecfblog.com/2015/03/automating-dfir-how-to-series-on_22.html

You can watch the show on Youtube:  https://www.youtube.com/watch?v=y-xtRkwaP2g

or below!

Also Read: Automating DFIR - How to series on programming libtsk with python Part 11

Read More

Automating DFIR - How to series on programming libtsk with python Part 10

Hello Reader,
If you just found this series I have good news! There is way more of it to read and you should start at Part 1. See the links to all the parts below:

Part 1 - Accessing an image and printing the partition table
Part 2 - Extracting a file from an image
Part 3  - Extracting a file from a live system
Part 4 - Turning a python script into a windows executable
Part 5 - Auto escalating your python script to administrator
Part 6 - Accessing an E01 image and extracting files
Part 7 - Taking in command line options with argparse to specify an image
Part 8 - Hashing a file stored in a forensic image
Part 9 - Recursively hashing all the files in an image

Following this post the series continues:

Part 11 - Recursively searching for files and extracting them from a live system 
Part 12 - Accessing different file systems  
Part 13 - Accessing Volume Shadow Copies 

For those of you who are up to date let's get going! In this part of the series we are going to take our recursive hashing script and make it even more useful. We are going to allow the user to search for the kind of files they want to hash with a regular expression enabled search and give them the option to extract those files as they are found back to their original path under your output directory. Ready? Let's go!

First we will need to import two more libraries, so many libraries!. The good news is these are still standard python system libraries, so there is nothing new to install. The first library we will bring is called 'os' which gives us os related functions and will map the proper function based on the os you are running on. The second library called 're' will provide us with regular expression support when evaluating our search criteria. We bring in those libraries as before with an import command:

import os
import re

Now we need to add two more command line arguments to let our user take advantage of the new code we are about to write:

argparser.add_argument(
'-s', '--search',
dest='search',
action="store",
type=str,
default='.*',
required=False,
help='Specify search parameter e.g. *.lnk'
)
argparser.add_argument(
'-e', '--extract',
dest='extract',
action="store_true",
default=False,
required=False,
help='Pass this option to extract files found'
    )

Our first new argument is letting the user provide a search parameter. We are storing the search parameter in the variable args.search and if the user does not provide one we default to .* which will match anything.

The second argument is using a different variable then the rest of our options. We are setting the variable args.extract as a True or False value with the store_true option under action. If the user provides this argument then the variable will be true and the files matched will be extracted, if the user does not then the value will be false and the program will only hash the files it finds.

It's always good to show the user that the argument we received from them is doing something so let's add two lines to see if we have a search term and print it:

if not args.search == '.*':
  print "Search Term Provided",args.search

 Remember that .* is our default search term, so if the value stored in args.search is anything other than .* our program will print out the search term provided, otherwise it will just move on.

Our next changes all happen within our directoryRecurse function. First we need to capture the full path where the file we are looking at exists. We will do this by combining the partition number and the full path that lead to this file in order to make sure its unique between partitions.

outputPath ='./%s/%s/' % (str(partition.addr),'/'.join(parentPath))

Next we will go into the else if statement we wrote in the prior post to handle regular files with non zero lengths. We will add a new line of code to the beginning of the code block that gets executed to do our regular expression search as follows:

elif f_type == pytsk3.TSK_FS_META_TYPE_REG and entryObject.info.meta.size != 0: searchResult = re.match(args.search,entryObject.info.name.name)

You can see we are use the re library here and calling the match function it provides. We are providing two arguments to the match function. The first is the search term the user provided us and the second is the file name of the regular, non zero sized file we are inspecting. If the regular expression provided by the user is a match then a match object will be returned and stored in the searchResult variable, if there is not a match then the variable will contain no data. We write a conditional to test this result next:

if not searchResult:
continue

This allows us to skip any file that did not match the search result provided. If the user did not specify a search term our default value of .* will kick in and everything will be a match.

Our last modification revolves around extracting out the files if our user selected the option to. The code looks like this

if args.extract == True:
if not os.path.exists(outputPath):
os.makedirs(outputPath)
extractFile = open(outputPath+entryObject.info.name.name,'w')
extractFile.write(filedata)
extractFile.close

The first thing we are doing is checking to see if the user has set the extract variable and caused it to be set to True. If they do then we will extract the file, if not we skip it. If it is true the first thing we do is make use of the os library's path.exists function. This will allow us to look at the output directory we want to create (and set in the outputPath variable above) already exists. If it does we can move on, if it does not than we call another os library provided function named makedirs. makedirs is nice because it will recursively create the path you specify so you don't have to loop through all the directories in between if they don't exist.

Now that our output path exists its time to extract it. We are modifying our old extractFile variable and now we are appending on our outputPath to the filename we want to create. This will place the file we are extracting in to the directory we have created. Next we write the data out to it as before and then close the handle since we will be reusing it.

If I was to run this program against the Level5 image we've been working with and specify both the extraction flag and provide a search term of .*jpg it would look like this

C:\Users\dave\Desktop>python dfirwizard-v9.py -i SSFCC-Level5.E01 -e -s .*jpg
Search Term Provided .*jpg
0 Primary Table (#0) 0s(0) 1
1 Unallocated 0s(0) 8064
2 NTFS (0x07) 8064s(4128768) 61759616
Directory: /
Directory: /$Extend/$RmMetadata/$Txf
Directory: /$Extend/$RmMetadata/$TxfLog
Directory: /$Extend/$RmMetadata
Directory: //$Extend
match  BeardsBeardsBeards.jpg
match  ILoveBeards.jpg
match  ItsRob.jpg
match  NiceShirtRob.jpg
match  OhRob.jpg
match  OnlyBeardsForMe.jpg
match  RobGoneWild.jpg
match  RobInRed.jpg
match  RobRepresenting.jpg
match  RobToGo.jpg
match  WhatchaWantRob.jpg
Directory: //$OrphanFiles

On my desktop there would be a folder named 2 and underneath that the full path to the file that matched the search term would exist.

That's it! Part 9 had a lot going on but now that we've built the base for our recursion it gets easier from here. The code for this part is located on the series Github here: https://github.com/dlcowen/dfirwizard/blob/master/dfirwizard-v9.py

Next in part 11 we will do the same thing from live systems but allow our code to enumerate all the physical disks present rather than hardcoding an option.

Also Read: Automating DFIR - How to series on programming libtsk with python Part 9

Read More

Automating DFIR - How to series on programming libtsk with python Part 8

Hello Reader,
            Welcome to part 8 of the Automating DFIR series, if this is the post your starting with... Stop! You need to read all the prior parts or you will be really, really lost. There is a lot going on here and the better you understand it the easier it will all make sense to you, allowing you to done your wizard robe and hat of DFIR Wizardry! Catch up on the part you left on below:

Part 1 - Accessing an image and printing the partition table
Part 2 - Extracting a file from an image
Part 3  - Extracting a file from a live system
Part 4 - Turning a python script into a windows executable
Part 5 - Auto escalating your python script to administrator
Part 6 - Accessing an E01 image and extracting files
Part 7 - Taking in command line options with argparse to specify an image

Following this post the series continues:

Part 9 - Recursively hashing all the files in an image
Part 10 - Recursively searching for files and extracting them from an image
Part 11 - Recursively searching for files and extracting them from a live system 
Part 12 - Accessing different file systems
Part 13 - Accessing Volume Shadow Copies  

Now that we have that out of the way for the new people, let's get back into the good stuff. One of the most common things we need to do, other than pulling a file out of an image, is to hash a file or files contained within an image. Let's start simply by just adding the hashing library that Python provides and generating some hashes for our $MFTs. Then we'll work on making our script a bit more useful with some more command line options and much better intelligence.

So to start with, as you may be expecting by now, we need to import a library. In Python 2.7 the library to import for hashing, that comes with Python, is called hashlib. So the first thing we need to do is import is as follows:

import hashlib

You can read the documentation on hashlib here: https://docs.python.org/2/library/hashlib.html#module-hashlib

Next we need to create a hashlib object to generate, store and print our hashes. Hashlib supports many different types of hashing and as of this post it supports md5, sha1, sha224, sha256, sha384, and sha512. When we create our hashlib option we do so while also choosing which of the hashing algorithms our object will use. For each hash algorithm you want to use you'll need a separate object. Let's start with MD5 by making the object and storing it in a variable called md5hash:

md5hash = hashlib.md5()

If we wanted to make a sha1 hashlib object we would just change the function at the end we are calling to sha1() and store it in a new variable, as follows:

sha1hash = hashlib.sha1()

If want to then generate a hash we would call the update function built into our hashlib object. When you call the update function you need to give it something to hash. You can give it a string or a variable that contains data to be hashed. In our program I am giving the object the contents of the $MFT we read from the image whose data is being stored in the variable filedata. So the whole call looks like this:

md5hash.update(filedata)

Similarly the call to generate the sha1 hash looks like this:

sha1hash.update(filedata)

We've now generated md5 and sha1 hashes for the $MFT we've extracted from the image, now we need to get the hashes printed so our user can see them.To do this we use the hexdigist() method built into our hashlib object. The hexdigest function takes no arguments, it just prints in hex whatever hash value we last set with update. In this version of DFIR Wizard! we are just going to print out the hash value to our command prompt window. It looks like this:

print "MD5 Hash",md5hash.hexdigest()
print "SHA1 Hash",sha1hash.hexdigest()

Taken all together the program looks like this

#!/usr/bin/python
# Sample program or step 6 in becoming a DFIR Wizard!
# No license as this code is simple and free!
import sys
import pytsk3
import datetime
import pyewf
import argparse
import hashlib     
class ewf_Img_Info(pytsk3.Img_Info):
  def __init__(self, ewf_handle):
    self._ewf_handle = ewf_handle
    super(ewf_Img_Info, self).__init__(
        url="", type=pytsk3.TSK_IMG_TYPE_EXTERNAL)
  def close(self):
    self._ewf_handle.close()
  def read(self, offset, size):
    self._ewf_handle.seek(offset)
    return self._ewf_handle.read(size)
  def get_size(self):
    return self._ewf_handle.get_media_size()
argparser = argparse.ArgumentParser(description='Extract the $MFT from all of the NTFS partitions of an E01')
argparser.add_argument(
        '-i', '--image',
        dest='imagefile',
        action="store",
        type=str,
        default=None,
        required=True,
        help='E01 to extract from'
    )
args = argparser.parse_args()
filenames = pyewf.glob(args.imagefile)
ewf_handle = pyewf.handle()
ewf_handle.open(filenames)
imagehandle = ewf_Img_Info(ewf_handle)
partitionTable = pytsk3.Volume_Info(imagehandle)
for partition in partitionTable:
  print partition.addr, partition.desc, "%ss(%s)" % (partition.start, partition.start * 512), partition.len
  if 'NTFS' in partition.desc:
    filesystemObject = pytsk3.FS_Info(imagehandle, offset=(partition.start*512))
    fileobject = filesystemObject.open("/$MFT")
    print "File Inode:",fileobject.info.meta.addr
    print "File Name:",fileobject.info.name.name
    print "File Creation Time:",datetime.datetime.fromtimestamp(fileobject.info.meta.crtime).strftime('%Y-%m-%d %H:%M:%S')
    outFileName = str(partition.addr)+fileobject.info.name.name
    print outFileName
    outfile = open(outFileName, 'w')
    filedata = fileobject.read_random(0,fileobject.info.meta.size)
    md5hash = hashlib.md5()
    md5hash.update(filedata)    print "MD5 Hash",md5hash.hexdigest()    sha1hash = hashlib.sha1()    sha1hash.update(filedata)    print "SHA1 Hash",sha1hash.hexdigest()    outfile.write(filedata)
    outfile.close

Pretty easy right? When I run it I get the following:

C:\Users\dave\Desktop>python dfirwizard-v7.py -i SSFCC-Level5.E01
0 Primary Table (#0) 0s(0) 1
1 Unallocated 0s(0) 8064
2 NTFS (0x07) 8064s(4128768) 61759616
File Inode: 0
File Name: $MFT
File Creation Time: 2014-09-12 12:20:52
2$MFT
MD5 Hash d91df0fb48c36f77a7a9c65870761beb
SHA1 Hash 26213c341902111a68464020965e5fb50108730c

Now the neat thing about the update method build into hashlib is that it will just keep adding to the hash the data you pass in. So if you wanted to buffer your reads, which we will do in a future part in the series, to prevent too much memory usage you can just pass the chunks to update as you read them and still get a hash for the complete file. 

You can get the source of this version of DFIR Wizard on the series github here: https://github.com/dlcowen/dfirwizard/blob/master/dfirwizard-v7.py

In the next part we are going to go into how to recurse through an entire image and hash all the files within it. Building up and bigger as our DFIR Wizard program continues to grow.

Continue Reading: Automating DFIR - How to series on programming libtsk with python Part 7

Read More

Automating DFIR - How to series on programming libtsk with python Part 5

Hello Reader,

              This is part 5 of a planned 24 part series. If you haven't read the prior parts I would highly recommend you do to understand how we got to this point!

Part 1 - Accessing an image and printing the partition table
Part 2 - Extracting a file from an image
Part 3  - Extracting a file from a live system
Part 4 - Turning a python script into a windows executable

Following this post the series continues:

Part 6 - Accessing an E01 image and extracting files
Part 7 - Taking in command line options with argparse to specify an image
Part 8 - Hashing a file stored in a forensic image
Part 9 - Recursively hashing all the files in an image
Part 10 - Recursively searching for files and extracting them from an image
Part 11 - Recursively searching for files and extracting them from a live system 
Part 12 - Accessing different file systems
Part 13 - Accessing Volume Shadow Copies  

In this post, before continuing on to accessing an E01 image which is a bit more complicated, let's make our lives a little bit easier. It's always a pain when you forget to open an administrative command prompt to run your script and in future posts when we get to GUIs its easy to forget to right click and run as administrator/sudo your script. So instead let's have our code do it for us. Now I can't take credit for this code like most good programmers I turn to Google for answers which most frequently will lead you to stackoverflow.com for answers. On stackoverflow I found a series of threads which offered solutions to the problem of elevating a python script and in testing I found the following thread to offer the best solution: http://stackoverflow.com/questions/19672352/how-to-run-python-script-with-elevated-privilage-on-windows

So let's look at what changes we need to make our DFIR Wizard program to do this. By this I mean check to see if our script is running as administrator/root and if it's not then to try to do so (if the account has permissions to do so).

#!/usr/bin/python
# Sample program or step 3 in becoming a DFIR Wizard!
# No license as this code is simple and free!
import sys
import pytsk3
import datetime
import admin
if not admin.isUserAdmin():
   admin.runAsAdmin()
   sys.exit()
imagefile = "\\\\.\\PhysicalDrive0"
imagehandle = pytsk3.Img_Info(imagefile)
partitionTable = pytsk3.Volume_Info(imagehandle)
for partition in partitionTable:
  print partition.addr, partition.desc, "%ss(%s)" % (partition.start, partition.start * 512), partition.len
  if 'NTFS' in partition.desc:
    filesystemObject = pytsk3.FS_Info(imagehandle, offset=(partition.start*512))
    fileobject = filesystemObject.open("/$MFT")
    print "File Inode:",fileobject.info.meta.addr
    print "File Name:",fileobject.info.name.name
    print "File Creation Time:",datetime.datetime.fromtimestamp(fileobject.info.meta.crtime).strftime('%Y-%m-%d %H:%M:%S')
    outFileName = str(partition.addr)+fileobject.info.name.name
    print outFileName
    outfile = open(outFileName, 'w')
    filedata = fileobject.read_random(0,fileobject.info.meta.size)
    outfile.write(filedata)
    outfile.close

I have bolded the parts of the code that I have changed. You can see the first thing we are doing is importing in a new class. In this case that class is called 'admin'. Now before you try to install admin with pip you should know that admin is actually an new python script we are going to create. Rather than embed the admin functions, which are quite lengthy, in our main script we are going to import the functions it provides and use them.

After importing the admin class we are doing a test using the 'if' conditional statement. We are testing the result of calling the admin class function 'isUserAdmin' for the negative. In other words our 'if' statement here returns 'true' that we are running as administrator then the script will not execute the next two lines of code and just continue on executing. However if the 'isUserAdmin' function comes back that the process is not currently running as the administrator then the 'not' applies before the function will make it return true and thus the two lines of code indented after the 'if' statement will execute.

So let's talk about those two files after the 'if' statement. The first line after the 'if' statement is calling the 'runAsAdmin' function provided from the admin class we imported. This function will start a new process of our python program as administrator for us and when it executes again as administrator our program will skip over this if statement and run the rest of our DFIR Wizard program. The next line tells our program that is not running as administrator to stop running as the rest of the script requires us to run as administrator to execute correctly. The sys library provides this 'exit' function that tells our program to quit and it will only quit after spawning off the new administrative version of our python script. That's all the modifications we are going to make to our main script, dfirwizard-v4.py.

Now, let's take a look at the new python script we are making called admin.py.

#!/usr/bin/env python
# -*- coding: utf-8; mode: python; py-indent-offset: 4; indent-tabs-mode: nil -*-
# vim: fileencoding=utf-8 tabstop=4 expandtab shiftwidth=4
# (C) COPYRIGHT © Preston Landers 2010
# Released under the same license as Python 2.6.5

import sys, os, traceback, types
def isUserAdmin():
    if os.name == 'nt':
        import ctypes
        # WARNING: requires Windows XP SP2 or higher!
        try:
            return ctypes.windll.shell32.IsUserAnAdmin()
        except:
            traceback.print_exc()
            print "Admin check failed, assuming not an admin."
            return False
    elif os.name == 'posix':
        # Check for root on Posix
        return os.getuid() == 0
    else:
        raise RuntimeError, "Unsupported operating system for this module: %s" % (os.name,)
def runAsAdmin(cmdLine=None, wait=True):
    if os.name != 'nt':
        raise RuntimeError, "This function is only implemented on Windows."
    import win32api, win32con, win32event, win32process
    from win32com.shell.shell import ShellExecuteEx
    from win32com.shell import shellcon
    python_exe = sys.executable
    if cmdLine is None:
        cmdLine = [python_exe] + sys.argv
    elif type(cmdLine) not in (types.TupleType,types.ListType):
        raise ValueError, "cmdLine is not a sequence."
    cmd = '"%s"' % (cmdLine[0],)
    # XXX TODO: isn't there a function or something we can call to massage command line params?
    params = " ".join(['"%s"' % (x,) for x in cmdLine[1:]])
    cmdDir = ''
    showCmd = win32con.SW_SHOWNORMAL
    #showCmd = win32con.SW_HIDE
    lpVerb = 'runas'  # causes UAC elevation prompt.
    # print "Running", cmd, params
    # ShellExecute() doesn't seem to allow us to fetch the PID or handle
    # of the process, so we can't get anything useful from it. Therefore
    # the more complex ShellExecuteEx() must be used.
    # procHandle = win32api.ShellExecute(0, lpVerb, cmd, params, cmdDir, showCmd)
    procInfo = ShellExecuteEx(nShow=showCmd,
                              fMask=shellcon.SEE_MASK_NOCLOSEPROCESS,
                              lpVerb=lpVerb,
                              lpFile=cmd,
                              lpParameters=params)
    if wait:
        procHandle = procInfo['hProcess']  
        obj = win32event.WaitForSingleObject(procHandle, win32event.INFINITE)
        rc = win32process.GetExitCodeProcess(procHandle)
        #print "Process handle %s returned code %s" % (procHandle, rc)
    else:
        rc = None
    return rc

if __name__ == "__main__":
    sys.exit(test())

Now there is a lot of code here to get through to understand whats going on here. Let's break it down in chunks:

import sys, os, traceback, types

Here we importing 4 libraries needed for the admin class to work; sys, os, traceback and types. 

Next we are gong to define the first function the admin class provides, the 'isUserAdmin' function.

def isUserAdmin():
    if os.name == 'nt':
        import ctypes
        # WARNING: requires Windows XP SP2 or higher!
        try:
            return ctypes.windll.shell32.IsUserAnAdmin()
        except:
            traceback.print_exc()
            print "Admin check failed, assuming not an admin."
            return False
    elif os.name == 'posix':
        # Check for root on Posix
        return os.getuid() == 0
    else:
        raise RuntimeError, "Unsupported operating system for this module: %s" % (os.name,)

The first thing we are doing is determining what operating system our program is running by checking the contents of the variable 'name' from the os library we imported in the first line. We are testing to see if the operating system name is defined as 'nt' which means windows. To see the full list of operating system name returned go here: https://docs.python.org/2/library/os.html#os.name

If we are running under windows we are going to import another library for called 'ctypes' via the import ctypes line. The ctypes library is going to give us access to several windows internal functions, but it can do way more than that. These functions that we will call are made available from the windows api also called the win32 api its running to let programmers request information about the windows session they are currently running under. To learn more about the ctypes library go here: https://docs.python.org/2/library/ctypes.html?highlight=ctypes#module-ctypes

Next we see a new python conditional called 'try' and 'except'. This will allow us to to attempt to execute a function and in the event that it returns a failure we can define how to handle the error. Our try line is calling the following function ctypes.windll.shell32.IsUserAnAdmin(), for more information about this win32 api function go here: https://msdn.microsoft.com/en-us/library/windows/desktop/bb776463%28v=vs.85%29.aspx. Look at what we are doing with the ctypes library to call this function. We are using ctypes to call the win32 api through windll which is then calling shell32 to call the IsUserAnAdmin function. Using this syntax we can call any other shell32 function of which their are many! For a much longer read on the shell32 library go here: https://msdn.microsoft.com/en-us/library/windows/desktop/bb773177(v=vs.85).aspx

Now if we are running as administrator our code will return true. If we are not then the except clause will be called and we will print to the console that we are not running as administrator and then return false. The next elif or 'else if' will check to see if we are running on a 'posix' operating system. Posix should return for most unix operating systems such as BSD, OSX and Linux. If we are running under a Posix operating system we will check to see if we are running as root or uid 0 and then return truse or false. Otherwise we have an 'else' operator at the end stating we don't know how to handle any non windows non posix operating system.

We are now done with this function! Let's move on to the next function 

def runAsAdmin(cmdLine=None, wait=True):
    if os.name != 'nt':
        raise RuntimeError, "This function is only implemented on Windows."
    import win32api, win32con, win32event, win32process
    from win32com.shell.shell import ShellExecuteEx
    from win32com.shell import shellcon
    python_exe = sys.executable
    if cmdLine is None:
        cmdLine = [python_exe] + sys.argv
    elif type(cmdLine) not in (types.TupleType,types.ListType):
        raise ValueError, "cmdLine is not a sequence."
    cmd = '"%s"' % (cmdLine[0],)
    # XXX TODO: isn't there a function or something we can call to massage command line params?
    params = " ".join(['"%s"' % (x,) for x in cmdLine[1:]])
    cmdDir = ''
    showCmd = win32con.SW_SHOWNORMAL
    #showCmd = win32con.SW_HIDE
    lpVerb = 'runas'  # causes UAC elevation prompt.
    # print "Running", cmd, params
    # ShellExecute() doesn't seem to allow us to fetch the PID or handle
    # of the process, so we can't get anything useful from it. Therefore
    # the more complex ShellExecuteEx() must be used.
    # procHandle = win32api.ShellExecute(0, lpVerb, cmd, params, cmdDir, showCmd)
    procInfo = ShellExecuteEx(nShow=showCmd,
                              fMask=shellcon.SEE_MASK_NOCLOSEPROCESS,
                              lpVerb=lpVerb,
                              lpFile=cmd,
                              lpParameters=params)
    if wait:
        procHandle = procInfo['hProcess']  
        obj = win32event.WaitForSingleObject(procHandle, win32event.INFINITE)
        rc = win32process.GetExitCodeProcess(procHandle)
        #print "Process handle %s returned code %s" % (procHandle, rc)
    else:
        rc = None
    return rc

This is not a short function so let's break it down in chunks again. Our function is being defined here with two variables; cmdLine and wait. These two variables are being given default values in case no value was passed in. In other words if we called this function with a specific cmdLine variable then the function would accept it, otherwise if no such variable is passed in (as we are doing in our program) then it will use the default value of None.  Next we make sure that we are running under Windows as this function won't work on another operating system as its currently written. We do this with a check to os.name again with the condition != or not equals. If our operating system is not windows (returned as nt here) then the function will raise an error stating it won't work!

Next we need import more libraries! We are importing win32api, win32con, win32event, win32process libraries in order to start this up. From these libraries we are importing two functions into our local namespace. Importing into our local namespace means we don't have to call it with the full library path, instead we can call it by function name alone. We bringing in ShellExecuteEx and shellcon into our local namespace.

Now its time to figure out what python instance we are going to run as administrator with the following line, python_exe = sys.executable. We are assigning the name of the currently running python interpreter we are using from sys.executable to the variable python_exe.

Let's look at the next chunk of code:

if cmdLine is None:
        cmdLine = [python_exe] + sys.argv
    elif type(cmdLine) not in (types.TupleType,types.ListType):
        raise ValueError, "cmdLine is not a sequence."
    cmd = '"%s"' % (cmdLine[0],)

If we didn't pass in a value to cmdLine and its default value of 'None' is applied than we will update the cmdLine variable to be the name of our executable that we capture prior and the command line arguments passed into our currently running script via the sys provided variable argv.

If our cmdLine variable contains a value we passed in but it is not a list (a series of values in an array) then we throw an error. Lastly if neither applies, meaning a value was supplied and it is a list type variable then we assign it to our cmd to execute.

This is followed by:

params = " ".join(['"%s"' % (x,) for x in cmdLine[1:]])
    cmdDir = ''

Where we are joining all the command line arguments into a variable called params and defining the directory we want our program to run in to be the directory our program is currently running from.

Next we set the option of whether to show the console window of the running application. If this was a GUI program we would want to hide this, but since this is currently a command line program we want to show it. We are using the win32con library constants SW_SHOWNORMAL and SW_HIDE to set that value which will be pass into the administrative process we create.

showCmd = win32con.SW_SHOWNORMAL
    #showCmd = win32con.SW_HIDE

Next we need to make sure we set the right flag for an elevated process if we are in a UAC aware environment:  lpVerb = 'runas'  # causes UAC elevation prompt. by setting the lpVerb variable equal to runas.

Now we are ready to create our administrative process using the ShellExecuteEx command and passing in all the variables we set in the lines prior. We store the resulting process id in a variable named procInfo. We are passing one additional constant value here from shellcon, SEE_MASK_NOCLOSEPROCESS to make sure our parent process does not exit here.

  procInfo = ShellExecuteEx(nShow=showCmd,
                              fMask=shellcon.SEE_MASK_NOCLOSEPROCESS,
                              lpVerb=lpVerb,
                              lpFile=cmd,
                              lpParameters=params)

For more information on the ShellExecuteEx function go here: http://www.pinvoke.net/default.aspx/shell32/ShellExecuteEx.html

Lastly we need to check if we passed in a wait flag (true or false)

 if wait:
        procHandle = procInfo['hProcess']  
        obj = win32event.WaitForSingleObject(procHandle, win32event.INFINITE)
        rc = win32process.GetExitCodeProcess(procHandle)
        #print "Process handle %s returned code %s" % (procHandle, rc)
    else:
        rc = None
    return rc

If wait is true, which is by default, then we will wait for our administrative execution to finish and then we set the variable rc to exit code of the administrative process. If we are not waiting to get the return state then we set rc to the value None. Lastly we return the value of the variable rc to the program that called this function to begin with.

Lastly we have a default constructor

if __name__ == "__main__":
    sys.exit(test())

That was a lot to get through! But now that we have it we can reuse it every time we want to make sure our executable runs as administrator rather than having to restart it manually ourselves. If you notice in my code I put an exit after the administrative process returns, this is because if we let our script continue to run in the original non administrative process it will throw an error and likely just confuse the user.

If you want to grab these two files do it from the series github:
dfirwizard-v4.py: https://github.com/dlcowen/dfirwizard/blob/master/dfirwizard-v4.py
admin.py: https://github.com/dlcowen/dfirwizard/blob/master/admin.py

In the next part we will access an E01 image!

Continue Reading: Automating DFIR - How to series on programming libtsk with python Part 4

Read More