Hello Reader,
I had a bit of a break thanks to a long overdue vacation but I'm back and the code I'll be talking about today has been up on the github repository for almost 3 weeks, so if you ever want to get ahead go there as I write the code before I try to explain it! Github repository is here: https://github.com/dlcowen/dfirwizard
Now before we continue a reminder, don't start on this post! We've come a long way to get to this point and you should start at part 1 if you haven't already!
Part 1 - Accessing an image and printing the partition table
Part 2 - Extracting a file from an image
Part 3 - Extracting a file from a live system
Part 4 - Turning a python script into a windows executable
Part 5 - Auto escalating your python script to administrator
Part 6 - Accessing an E01 image and extracting files
Part 7 - Taking in command line options with argparse to specify an image
Part 8 - Hashing a file stored in a forensic image
Part 9 - Recursively hashing all the files in an image
Part 10 - Recursively searching for files and extracting them from an image
Following this post the series continues:
Part 12 - Accessing different file systems
Part 13 - Accessing Volume Shadow Copies
In this post we are going to augment the script on part 10 which went through an image and search/extracted files from all the NTFS partition in an image, and now we are going to do the same against all the NTFS partitions on a live system. You can obviously tweak this for any other file system but we will get to that in later posts in this series.
The first thing we need is a way to figure out what partitions exist on a live system in cross platform way so our future code can be tweaked to run anywhere. For this I choose the python library psutil which can provide a wealth of information about a system its running on, included information about available disks and partitions, you can read all about it here: https://pypi.python.org/pypi/psutil
To bring it into our program we need to call the import function again:
and then because we are going to work against a live running system again we need our old buddy admin
which if you remember from part 5 will auto escalate our script to administrator just in case we forgot to run it as such.
We are going to strip out the functions we need to find all the parts of a forensic image and replace it with out code to test for administrative access:
partitionList = psutil.disk_partitions()
for partition in partitionList:
imagehandle = pytsk3.Img_Info('\\\\.\\'+partition.device.strip("\\"))
if 'NTFS' in partition.fstype:
So here instead of calling pytsk for a partition table we are calling psutil.disk_partitions which will return a list of partitions that are available to the local system. I much prefer this method than trying to iterate through all volume letters as we will get back just those partitions available as well as what file system they are seen as running as. Our list of active partitions will be stored in the varaible partitionList. Next we will iterate through the partitions using the for operator storing each partition returned into the partition variable. Next we are creating a pytsk3 Img_Info object for each partition returned but only continuing if psutil recognized the partition is NTFS.
try:
f_type = entryObject.info.name.type
size = entryObject.info.meta.size
except Exception as error:
print "Cannot retrieve type or size of",entryObject.info.name.name
print error.message
continue
So in the above code I'm getting the type of file (lnk, regular, etc..) and it's size and if I can't I'm handling the error and printing out the error before continuing on. You will see errors, live systems are an interesting place to do forensics.
I am now going to make a change I alluded to earlier on in the series. We are going to buffer out reads and writes so we don't crash our of our program because we are trying to read a massive file into memory. This wasn't a problem in our first examples as we were working from small test images I made before, but now they we are dealing with real systems and real data we need to handle our data with care.
Our code looks as follows:
BUFF_SIZE = 1024 * 1024
offset=0
md5hash = hashlib.md5()
sha1hash = hashlib.sha1()
if args.extract == True:
if not os.path.exists(outputPath):
os.makedirs(outputPath)
extractFile = open(outputPath+entryObject.info.name.name,'w')
while offset < entryObject.info.meta.size:
available_to_read = min(BUFF_SIZE, entryObject.info.meta.size - offset)
filedata = entryObject.read_random(offset,available_to_read)
md5hash.update(filedata)
sha1hash.update(filedata)
offset += len(filedata)
if args.extract == True:
extractFile.write(filedata)
if args.extract == True:
extractFile.close
First we need to determine how much data we want to read or write at one time from a file. I've copied several other examples I've found and I'm setting that amount to 1 meg of data at a time by setting the variable BUFF_SIZE equal to 1024*1024 or one megabyte. Next we need to keep track of where we are in the file we are dealing with, we do that by creating a new variable called offset and setting the offset to 0 to start with.
You'll notice that we are creating our hash objects, directories and file handles before we read in any data. That is because we want to do all of these things one time prior to iterating through the contents of a file. If a file is a gigabyte in size then our function will be called 1,024 times and we just want one hash and one output file to be created.
Next we starting a while loop which will continue to execute until our offset is greater or equal to the size of our file, meaning we've read all the data within it. Now files are not guaranteed to be allocated in 1 meg chunks, so to deal with that we are going to take advantage of a python function called min. Min returns the smaller of to values presented which in our code is the size of the buffer compared to the remaining data left to read (the size of the file - our current offset). Whichever value is smaller will be stored in the variable available_to_read.
After we know how much data we want to read in this execution of our while loop we are going to read it as before from our entryObject passing in the offset to start from and how much data to read, storing the data read into the variable filedata. We are then calling the update function provided by our hashing objects. One of the nice things our the hashlibs provided by python is that if you provide additional data to an already instantiated object it will just continue to build the hash rather than having to read it all in at once.
Next we are incrementing our offset by adding to itself the length of data we just read so we will skip past it on the next while loop execution. Finally we write the data out to our output file if we elected to extract the files we are searching for.
I've added one last bit of code to help me catch any other weirdness that may seep through.
else:
print "This went wrong",entryObject.info.name.name,f_type
An else to look for any condition that does not match one of existing if statements.
That's it! You now have a super DFIR Wizard program that will go through all the active NTFS partitions on a running system and pull out and hash whatever files you want!
You can find the complete code here: https://github.com/dlcowen/dfirwizard/blob/master/dfirwizard-v10.py
In the next post we will talk about parsing partitions types other than NTFS and then go into volume shadow copy access!
I had a bit of a break thanks to a long overdue vacation but I'm back and the code I'll be talking about today has been up on the github repository for almost 3 weeks, so if you ever want to get ahead go there as I write the code before I try to explain it! Github repository is here: https://github.com/dlcowen/dfirwizard
Now before we continue a reminder, don't start on this post! We've come a long way to get to this point and you should start at part 1 if you haven't already!
Part 1 - Accessing an image and printing the partition table
Part 2 - Extracting a file from an image
Part 3 - Extracting a file from a live system
Part 4 - Turning a python script into a windows executable
Part 5 - Auto escalating your python script to administrator
Part 6 - Accessing an E01 image and extracting files
Part 7 - Taking in command line options with argparse to specify an image
Part 8 - Hashing a file stored in a forensic image
Part 9 - Recursively hashing all the files in an image
Part 10 - Recursively searching for files and extracting them from an image
Following this post the series continues:
Part 12 - Accessing different file systems
Part 13 - Accessing Volume Shadow Copies
In this post we are going to augment the script on part 10 which went through an image and search/extracted files from all the NTFS partition in an image, and now we are going to do the same against all the NTFS partitions on a live system. You can obviously tweak this for any other file system but we will get to that in later posts in this series.
The first thing we need is a way to figure out what partitions exist on a live system in cross platform way so our future code can be tweaked to run anywhere. For this I choose the python library psutil which can provide a wealth of information about a system its running on, included information about available disks and partitions, you can read all about it here: https://pypi.python.org/pypi/psutil
To bring it into our program we need to call the import function again:
import psutil
and then because we are going to work against a live running system again we need our old buddy admin
import admin
which if you remember from part 5 will auto escalate our script to administrator just in case we forgot to run it as such.
We are going to strip out the functions we need to find all the parts of a forensic image and replace it with out code to test for administrative access:
if not admin.isUserAdmin():Next we replace the functions we called to get a partition table from a forensic image with a call to psutil to return a listing of paritions and iterate through them. The code looks like the following which I will explain:
admin.runAsAdmin()
sys.exit()
partitionList = psutil.disk_partitions()
for partition in partitionList:
imagehandle = pytsk3.Img_Info('\\\\.\\'+partition.device.strip("\\"))
if 'NTFS' in partition.fstype:
So here instead of calling pytsk for a partition table we are calling psutil.disk_partitions which will return a list of partitions that are available to the local system. I much prefer this method than trying to iterate through all volume letters as we will get back just those partitions available as well as what file system they are seen as running as. Our list of active partitions will be stored in the varaible partitionList. Next we will iterate through the partitions using the for operator storing each partition returned into the partition variable. Next we are creating a pytsk3 Img_Info object for each partition returned but only continuing if psutil recognized the partition is NTFS.
The next thing we are changing is our try catch blog in our recursive directory function. Why? I found in my testing that live systems react much differently than forensic images in setting certain values in libtsk. So rather than using entryObject.info.meta.type to determine if I'm dealing with a regular file I am using entryObject.info.name.type which seem to always be set regardless if its a live system or a forensic image. I'm testing to see if I can capture the type of the file and it's size here as there are a lot of interesting special files that only appear at run time that will throw an error if you try to get their size.
f_type = entryObject.info.name.type
size = entryObject.info.meta.size
except Exception as error:
print "Cannot retrieve type or size of",entryObject.info.name.name
print error.message
continue
So in the above code I'm getting the type of file (lnk, regular, etc..) and it's size and if I can't I'm handling the error and printing out the error before continuing on. You will see errors, live systems are an interesting place to do forensics.
I am now going to make a change I alluded to earlier on in the series. We are going to buffer out reads and writes so we don't crash our of our program because we are trying to read a massive file into memory. This wasn't a problem in our first examples as we were working from small test images I made before, but now they we are dealing with real systems and real data we need to handle our data with care.
Our code looks as follows:
BUFF_SIZE = 1024 * 1024
offset=0
md5hash = hashlib.md5()
sha1hash = hashlib.sha1()
if args.extract == True:
if not os.path.exists(outputPath):
os.makedirs(outputPath)
extractFile = open(outputPath+entryObject.info.name.name,'w')
while offset < entryObject.info.meta.size:
available_to_read = min(BUFF_SIZE, entryObject.info.meta.size - offset)
filedata = entryObject.read_random(offset,available_to_read)
md5hash.update(filedata)
sha1hash.update(filedata)
offset += len(filedata)
if args.extract == True:
extractFile.write(filedata)
if args.extract == True:
extractFile.close
First we need to determine how much data we want to read or write at one time from a file. I've copied several other examples I've found and I'm setting that amount to 1 meg of data at a time by setting the variable BUFF_SIZE equal to 1024*1024 or one megabyte. Next we need to keep track of where we are in the file we are dealing with, we do that by creating a new variable called offset and setting the offset to 0 to start with.
You'll notice that we are creating our hash objects, directories and file handles before we read in any data. That is because we want to do all of these things one time prior to iterating through the contents of a file. If a file is a gigabyte in size then our function will be called 1,024 times and we just want one hash and one output file to be created.
Next we starting a while loop which will continue to execute until our offset is greater or equal to the size of our file, meaning we've read all the data within it. Now files are not guaranteed to be allocated in 1 meg chunks, so to deal with that we are going to take advantage of a python function called min. Min returns the smaller of to values presented which in our code is the size of the buffer compared to the remaining data left to read (the size of the file - our current offset). Whichever value is smaller will be stored in the variable available_to_read.
After we know how much data we want to read in this execution of our while loop we are going to read it as before from our entryObject passing in the offset to start from and how much data to read, storing the data read into the variable filedata. We are then calling the update function provided by our hashing objects. One of the nice things our the hashlibs provided by python is that if you provide additional data to an already instantiated object it will just continue to build the hash rather than having to read it all in at once.
Next we are incrementing our offset by adding to itself the length of data we just read so we will skip past it on the next while loop execution. Finally we write the data out to our output file if we elected to extract the files we are searching for.
I've added one last bit of code to help me catch any other weirdness that may seep through.
else:
print "This went wrong",entryObject.info.name.name,f_type
An else to look for any condition that does not match one of existing if statements.
That's it! You now have a super DFIR Wizard program that will go through all the active NTFS partitions on a running system and pull out and hash whatever files you want!
You can find the complete code here: https://github.com/dlcowen/dfirwizard/blob/master/dfirwizard-v10.py
In the next post we will talk about parsing partitions types other than NTFS and then go into volume shadow copy access!
