Home Archives for January 2014

Daily Blog #223: Saturday Reading 2/1/14

Get even better at DFIR with David Cowen

Hello Reader,
It's Saturday and after another week of hard work you deserve a break. Use that break time to get even better at DFIR with this weeks Saturday Reading!

1. We had another great forensic lunch this week! You can watch it here: http://www.youtube.com/watch?feature=player_embedded&v=2P5Sv6yyd5Y This weeks guests:

Ian Duffy, +Ian Duffy , talking about his research into the Microsoft Office compound file format.
You can read Ian's blogs on this topic here: http://forensecurity.blogspot.com/2014/01/microsoft-office-compound-document.html

Andrew Case, +Andrew Case , discussing his work in the memory forensics and Volatility

The Volatility project page is here: http://code.google.com/p/volatility/
You can pre-order the memory forensics book here: http://www.amazon.com/gp/product/1118825098/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1118825098&linkCode=as2&tag=malwacookb-20
You can find out more about Volatility training here: http://volatility-labs.blogspot.com/2013/10/2014-malware-and-memory-forensics.html
Volatility Community Documentation can be found here: http://code.google.com/p/volatility/wiki/VolatilityDocumentationProject
You can find out more about Bsides NOLA here: http://www.securitybsides.com/w/page/71231585/BsidesNola2014
Read the blog analyzing ADD that Andrew talked about here: http://blog.handlerdiaries.com/?p=363

Matthew and I showing the latest changes for this months Beta release of ANJP.
2. Jason Hale has a neat blog up this week relating to a new feature in Microsoft Word 2013. The feature tracks where a user left off in their reading of a document, something that can be very useful in showing more than just opening of a word document. Read more about it here: http://dfstream.blogspot.com/2014/01/ms-word-2013-reading-locations.html

3. Lenny Zeltser has a good blog up on the SANS Forensics blog this week talking about all the different specialties of DFIR that are forming, http://digital-forensics.sans.org/blog/2014/01/30/many-fields-of-dfir. I think this type of knowledge needs a wider audience to understand just how wide and deep our field is.

4. Jamie Levy has put a link to her slides for OMFW talk about profiling normal system memory http://gleeda.blogspot.com/2014/01/omfw-2013-slides.html. This is something we've been talking about for the last two Forensic Lunches so I'm very interested in learning more.

5. The Bsides NOLA CFP ends today! Quick get your submission in! http://www.securitybsides.com/w/page/71231585/BsidesNola2014

6. Jack Crook has a great analysis of the ADD affected memory image up on his blog, http://blog.handlerdiaries.com/?p=363. This is a great post for understanding how to spot whats abnormal and track it down.

7. Here is a good post by Brian Moran showing how open source tools fare against the Target POS Malware, http://brimorlabs.blogspot.com/2014/01/target-pos-malware-vs-open-source-tools.html.

8. Julie Desautels has put up an interesting blog using her Google Glass forensic research to make the case that a driver was or was not operating the Glass device at the time she was pulled over. These types of devices are only going to grow in the future so get a head start and read this here, http://desautelsja.blogspot.com/2014/01/proving-case-of-cecilia-abadie-using.html.

Also Read: Daily Blog #221

RHEL Forensics Part 3 by David Cowen HECF Blog

Hello Reader,
Today we talk about recovering deleted mlocate databases. This was actually harder than I expected as not only did ext3 set the size of the file equal to 0 but the direct block that istat -B came back with was not the first block in our database. So instead I followed the instructions here: http://wiki.sleuthkit.org/index.php?title=FS_Analysis to do a manual recovery of deleted databases. There is still some work to be done here in order to clean up whats been recovered back into parsable databases but I'll leave that bit for next week.

Today let's go through the steps necessary to recover deleted mlocate databases on a RHEL v5 system using Ext3 as the file system. Remember this is necessary as the updatedb command runs daily and deletes the mlocate database before creating a new one.

Step 1. We need to figure out which group of inodes our parent belongs to. You can see in the screenshot below that the parent directory /var/lib/mlocate has inode number 1077298 so that is the group we need to find.

Step 2. Run fsstat to find out which group contains our inode, in this case Group 33 contains our inode as shown below. We can use them to determine which blocks to recover for deleted databases.

Step 3. Use blkls to recover those unallocated blocks within Group 33 as shown below to a new file:

Step 4. Use xxd to parse the recovered blocks and find the mlocate database signature of 'mlocate'.

This looks like an mlocate database but right now its stuck in the middle of the rest of the unallocated data. So the next thing we need to do next week as we continue this series is to write some code to carve out the mlocate database from this unallocated block chunk.

Make sure to come back tomorrow for the Forensic Lunch with guests Ian Duffy and Andrew Case!

Also Read: Daily Blog #220 - RHEL Forensics Part 2

Hello Reader,
Yesterday we talked about extending this weeks Sunday Funday answer using the mlocate database on RHEL. Today let's look at what we can determine from the mlocate database using Hal Pomeran'z mlocate-time script and setup tomorrow's entry regarding the recovery of deleted mlocate databases.

mlocate, default on RHEL since v4, queries a database of known files and directories called /var/lib/mlocate/mlocate.db. The database stores the full path to every file and directory it encounters as well as the timestamps of the directories. The timestamp according to the man page will be either the change time or modification time of the directory, whichever is more recent. The timestamp is being kept to determine if during the update process if mlocate should re-index the contents of a directory. This leads to the question, will timestamp manipulation get around mlocate indexing of a file's existence, which is something we can test in this series.

For today's example I have created a file in my home directory called 'secret_file' and then deleted it.

Since this filesystem is ext3 the name of the file 'secret_file' is now zero'd out within the directory inode. The only way to know it existed is to hope that there is another recent reference to the file within the ext3 journal to re-associate it or to search the mlocate database. There may be other artifacts but we will focus on those two for the moment.

Searching the mlocate database confirms the file entry still exists:

Looking into the parsed database records shows the last time the directory was modified when the file still existed within it:

So that's great we can establish a timeframe when the file did exist and we could compare the contents of the current filesystem against the mlocate database to determine which files had been deleted since the last daily cron run. This can be helpful for determining what has changed in the last day in a live response scenario. This does not help though when we want to know what is occurring on a longer term basis.

The mlocate database is updated by default once daily when /etc/cron.daily/mlocate.cron runs and execute updatedb. What Hal pointed out from his tests though is that when that updatedb command runs that it does not overwrite the database but instead unlinks (deletes) it and then creates a new one. We can see that in the following screenshots showing the inode numbers of the mlocate database.

Before updatedb:

After updatedb:

Notice that the inode number of mlocate.db has changed from 1077692 to 1077693 meaning a new inode has been created and the old inode is still recoverable. As Hal also pointed out the mlocate database has a unique signature that can make determining which deleted inodes contain mlocate databases so tomorow let's do that. Let's see if we can make a quick script that will find and recovery deleted mlocate databases for longer historical comparisons of which files existed on our system.

Also when I'm done with this series I'll be uploading my test image for download so you'll be able to recover the same data! Come back tomorrow and through the rest of this series as we determine:

1. How to identify and recover inodes containing mlocate databases.

2. Examining the possibility of carving mlocate database entries from freespace.

This is a 6-part series. Also Read:

RHEL Forensics Part 1

RHEL Forensics Part 3

RHEL Forensics Part 4

RHEL Forensics Part 5

RHEL Forensics Part 6

RHEL Forensics Part 1 by David Cowen - HECF Blog

Hello Reader,
If you read this weekends Sunday Funday winning answer you learned a lot about how to do forensics on a redhat enterprise linux server. As with anything we do in digital forensics though, there is always more to learn. Today we start a series on what we can do to go beyond this weeks winning answer, which was very good. Let's start by looking into a tool that Hal Pomeranz introduced us to on last weeks Forensic Lunch.

Hal's tool, mlocate-time, allows us to parse the mlocate database for all of the files, directories and timestamps for all files that existed on the system as of the last mlocate database update. By default there is a chron job that is set to run daily to update the mlocate database so the live data will only contain those files that existed since the last daily cron job run. Comparing the files known to exist in the mlocate database to the files live on the current system can reveal files that have been deleted, but what we want to look at is the recovery of past mlocate databases. There are two sources for these:

1. Hope there are backups of the system stored somewhere. From the backups we can extract out all copies of the mlocate database and then parse them with Hal's tool.

2. Recover the deleted inode or carve them out of unallocated space. While on the lunch Hal tested and confirmed that each time the database is updated it deletes the old database and creates a new one. That means that all the old locate databases and their timestamps are either recoverable inodes or carvable data blocks allowing you to bring back the proof of the prior existence of files and timestamps on your system. This is not just true for RHEL but other linux distributions making use of mlocate as well.

Tomorrow let's get into what the data contained with mlocate can do for our investigation and end with some mlocate database reovery. I'm downloading a RHEL v5 evaluation iso tonight to start my tests!

Continue Reading:

RHEL Forensics Part 2

RHEL Forensics Part 3

RHEL Forensics Part 4

RHEL Forensics Part 5

RHEL Forensics Part 6

Redhat Enterprise Linux v5 sever challenge - HECF Blog

Hello Reader,
One of the great things about Sunday Funday's is that we get to find those individuals out there whose experience shines through in their answers. This weeks challenge had a few great answers,but this weeks winning answer was not only received before the other contenders but shown through as a winner. Take the time to read this one, you'll find some great ideas for your future Linux server investigations.

The Challenge:

You have a Redhat Enterprise Linux v5 sever running an eCommerce site. The server was breached as the attacker logged in as the root user two weeks ago and linked the shell history file to /dev/null. What other artifacts can you rely on to determine what the attacker did over the past two weeks?

The Winning Answer:

Anonmyous

TL;DR: /var/log/secure, SSH log, syslog, wtmp & btmp, Apache logs, firewall logs, acct files, memory image, file system metadata & journal & deleted content.

RHEL 5 first released 2007, uses kernel 2.6.18, even in the latest update (update 10, October 2013).

My strategy for approaching this investigation would consist of two phases: first, identify the periods of potential attacker activity; second, drill into these suspicious time ranges collect attacker commands and actions. Generally speaking, I would use multiple log sources to draft an initial list of suspicious time ranges. Then, I would use more specific tools to recover evidence of commands and actions within those ranges.

Due to the specific wording used in the scenario, I don’t have to worry about reviewing the system for evidence of a remote exploit such as SQL injection. Of course, the best place to start running that down is by reviewing application and server logs.

To begin, I would review the file /var/log/secure to identify how the attacker logged in for the first time. This is a log file that records entries associated with authentication requests, including timestamps, usernames, source processes, and error messages. According to the scenario, the system was compromised for the first time via the root login. So, I’d need to cross reference all legitimate administrator activity with root logins since approximately two weeks ago. The outstanding entries should be associated with the attacker (or, poorly configured services).

If I saw a single authentication attempt leading to a successful login, I would suspect that the attacker acquired legitimate credentials (account password or SSH certificate) elsewhere, perhaps by compromising another system, phishing the administrators, etc. I’d have to track this down by expanding the scope of the investigation. It is also possible that the password fell to a brute force attack, in which case I’d expect to see many, many unsuccessful attempts before a single successful authentication. The answer to this question may give me some insight into type of attack I was dealing with, and how I might expect to see the remainder of the system configured. For instance, a properly secured environment should not fall to a brute force attack targetting the entire internet.

I’d review and cross reference wtmp and btmp files for additional session information. wtmp tracks a history of logins and logouts by user, and btmp tracks failed authentication attempts. utmp could be helpful, but it typically tracks the current state of the system. All these files can be found in the /var/log directory. These are binary files, but the format is well known, and similar version of Linux (such as the Fedora release using kernel 2.6.18) can be used as effective analysis machines.

Once I had identified the first relevant login session, I would confirm the means of access: was it SSH, VNC, or some other remote access protocol. In each of these cases, I’d review the network architecture to determine from which network segments this protocol was allowed. Ideally, these administrative interfaces would not be exposed to the greater internet, but we’ve all see that too often. If the administrative ports were not accessible to the internet, then it again means that the scope of the investigation should be relaxed to include additional systems on the local network segment.

From the scenario description, the server is running an eCommerce site. An eCommerce site is typically composed of front end web services (serving static media like HTML, images, CSS, and Javascript, as well as dynamic pages generated by languages like PHP, Perl, or Python) and databases (MySQL or Postgres are popular). It is probably running at least some of the frontend services, and therefore that it is probably accessible to the internet.

This internet connection might be direct through a firewall, or through a load balancer/reverse proxy and firewall. I would review logs from the firewalls and load balancers to identify requests related activity from the same source IP address. This would help define additional periods of activity associated with the attacker.

I would timeline all application logs (usually, /var/log/*/*), syslog entries (/var/log/messages, etc.), and file system activity. Log2timeline or Plaso are good tools for organizing all this information. Some types of interesting application log entries could be yum package manager entries (/var/log/yum.log) indicating that the attacker installed additional software, or Apache web server entries (usually, /var/log/httpd/*) showing that the attacker test access to web directories via a web browser.

I would pay particular attention to the file system activity, reviewing file system metadata for newly created, modified, or deleted files. The Sleuthkit and loopback devices are my favorite tools for working with Linux images. I’d hope to find attacker tools and/or attacker archived data using the file system metadata. To recover further deleted files, I might try Foremost and extundelete. Foremost carves chunks from a binary stream using known file signatures. extundelete processes the journal on ext3/ext4 and attempts to recover old copies of inodes, and subsequently the file data.

Of course, I would also acquire a memory image of the server, and subsequently use Volatility to extract artifacts. I would first attempt to use the “linux_bash” plugin, which extracts Bash shell history entries from memory. These entries may still be in memory despite the /dev/null link. However, due to the duration of the compromise (two weeks), I would not consider this source authoritative of all activity. A number of the other plugins (for instance, linux_check_*) are also appropriate to use to identify the presence of rootkits and other suspicious processes.

Finally, I would review the process accounting information tracked by the “acct” service. This service typically stores its data within the directory /var/account/pacct that includes processes run and resources consumed. I would start by reviewing the data using “lastcomm” and “sar” programs to identify process names I don’t recognize. I could also correlate processes run before two weeks ago with those run after. Though the process accounting logs do not always contain verbose information, they could be effective in identifying Bitcoin miners or other rogue processes.