Monday, July 31, 2017

National Collegiate Cyber Defense Competition Red Team Debrief 2017

Hello Reader,
        I've been busy lately, so busy I didn't get around to posting this years red team debrief from the National CCDC. After just leaving Blackhat/ Bsides LV/ Defcon and running our first Defcon DFIR CTF I thought it was important to get these up and talk about the lessons learned.

The Debrief

First of all for those of you coming just to get the presentation its below:

For those of you who have no idea what any of these means, let me take a step back.

What is CCDC?

The National Collegiate Cyber Defense Competition ( CCDC ) is a now 12 year old competition where colleges around the United States form student teams to defend networks. CCDC is different from other competition involving network security as it focuses strictly on defense. Students who play are put in charge of a working network that they must defend, the only offensive activity in the competition comes from a centralized red team.

The kind of enterprise network students take charge of changes each year. Past years business scenarios have included:

  • Private Prison Operator
  • Electric Utility
  • Web hosting
  • Game Developer
  • Pharma
  • Defense Contractor
  • and more!
The idea is that the last IT team has been fired and the student team is coming in to keep it running and defend it. While the students are working on making sure their systems are functioning they also have to watch for, respond to and defend against the competition red team. 

Scoring happens a couple ways. 

Students get points for:
  • Keeping scored services running (websites, ecommerce sites, ssh access, email, etc..)
  • Completing business requests such as policy creation, password audits and disaster recovery plans
  • Presenting their work to the CEO of the fake company
  • Responding to customers 

Students lose points for:

  • Red team access to user or administrative credentials
  • Red team access to PII data
  • Services not responding to scoring checks aka services being down
  • SLA violations kick in if the service stays down for a period of time
There are now 160 universities competing in 10 regions across the united states. If a student team wins their region they make it to nationals where the top 10 teams in the country compete for some pretty amazing prizes, including on the spot job offers from raytheon. 

If you are a student or a professor who would like to know more about competing you can go here:

What is the National CCDC Red Team?

The National CCDC Red Team is a group of volunteers who works to build custom malware, c2 and exfiltration and persistence strategies to bear each year to give the students the best real world threat experience. I'm the captain of the red team and have been for the last 10 years.

How do I get on it? 

When the call for volunteers goes out send a resume to

Be advised our threshold for acceptance is very high and we look for the following:
- Active projects on github or otherwise to show your experience
- Real experience in developing, maintaining and layering persistence
- Custom malware kits that are unpublished to bring to bear

We don't care about certs, years of experience or who you work for. We need people who can not only get in (the easy part) but to stay in over a two day period of competition while an aggressive group of defenders seeks to keep you out. 

Thursday, May 4, 2017

Contents in sparse mirror may be smaller than they appear

By Matthew Seyer

As many of you know, David Cowen and I are huge fans of file system journals! This love also includes all change journals designed by operating systems such as FSEvents and the $UsnJrnl:$J. We have spent much of our Dev time writing tools to parse the journals. Needless to say, we have lots of experience with file system and change journals. Here is some USN Journal logic for you.

USN Journal Logic

First off it is important to know that the USN Journal file is a sparse file. MSDN explains what a sparse file is: When the USN Journal (referred to as $J from here on out) is created it is created with a max size (the area of allocated data) and an allocation delta (the size in memory that stores records before it is committed to the $J on disk). This is seen here:

The issue is that many forensics tools cannot export a file out as a sparse file. Even if they could only a few file systems support them and I don’t even know if a sparse file on NTFS is the same as a sparse file on OSX. But this leads to a common problem. The forensic tool sees the $J as larger than it really is:

While this file is 20,380,061,592 bytes in size, the allocated portion of records is much smaller. Most forensic tools will export out the entire file with the unallocated data as 0x00. Which makes sense when you look at the MSDN Sparse File section (link above). When we extract this file out with FTK Imager we can verify with the windows command `fsutil sparse` to see that the exported file is not a sparse file (

 Trimming the $J

Once its exported out what’s a good way to find the start of the records? I like to use 010 Editor. I scroll towards the end of the file where there are still empty blocks (all 0x00s) then I search for 0x02 as I know I am looking for USN Record version 2:

Now if I want to export out just the record area I can start at the beginning of this found record and select to the end of the file and save the selection as a new file: 

The resulting file is 37,687,192 bytes in size and contains just the record portion of the file.

This is significantly smaller in size! Now, how do we go about this programmatically?


While other sparse files can have interspersed data, the $J sparse file keeps all of its data at the end of the file. This is because you can associate the Update Sequence Number in the record to the offset of the file! If you want to look at the structure of the USN record here it is: Now I will note that I would go about this two different ways. One method for a file that has been extracted out from one tool and a different method for if it would be extracted out using the TSK lib. But for now, we will just look at the first scenario.

Because the records are located in the lasts blocks of the file, I would start from the end of the file and work our way backwards to find the first record, then write out just the records portion of the file. This saves a lot of time because you are not searching through potentially many gigs of zeros. You can find the code at I have commented the code so that it is easy to understand what is happening.

Now lets use the tool:

We see that the usn.trim file is the same as the one we did manually but lets check the hash to make sure we have the same results as the manual extract:

So far I have verified this on SANS408 image system $J extract and some local $J files. But of course, make sure you use multiple techniques to verify. This was a quick proof of concept code.

Questions? Ask them in the comments below or send me a tweet at @forensic_matt

Monday, May 1, 2017

Forensic Lunch with Paul Shomo, Matt Bromiley, Phil Hagen, Lee Whitfield and David Cowen

Hello Reader,
     It's been awhile since I've cross posted that the videocast/podcast went up. We had a pretty great forensic lunch with lots of details about programs that are relevant frome everyone from academic students in forensics to serious artifact hunters. Here are the show notes:

Paul Shomo comes on to talk about Guidance Software's new Forensic Artifact Research Program where you can get $5,000 USD just for research you are already doing! Find out more here:

Phil Hagen introduced the new SANS Network Forensics poster to be released later this month

Matt Bromiley is talking about the Ken Johnson Scholarship setup by SANS and KPMG you can learn more and apply here

Phil, Matt, Lee and I talked about the DFIR Summit

Lee Whitfield and I talked about the 4Cast Awards, Voting is open here:

You can watch the lunch here:
You can listen to the podcast here:
Or you can subscribe to it on iTunes, Stitcher, Tunein and any other quality podcast provider. 

Wednesday, April 19, 2017

Windows, Now with built in anti forensics!

Hello Reader,

Update: TZWorks updated USP in November of 2016 and it now not only shows these removed devices but if you pass in the -show_other_times flag it will tell you when the devices were removed. Very cool!

             If you've been using a tool to parse external storage device storage devices that relies on USB, USBStor, WPDBUSENUM or STORAGE as its primary key for fining all external devices you might be being tricked by Windows. Windows has been doing something new (to me at least) that I first observed in the Suncoast v Peter Scoppa et al case (Case No. 4:13-cv-03125) back in 2015 where Windows on its own and without user request is removing unused device entries from the registry on a regular basis driven by Task Scheduler.

This behavior that I've observed in my case work started in Windows 8.1 and I've confirmed it in Windows 10. A PDF I found that references this found here states he has seen it in Windows 7 but I can't confirm this behavior. The behavior is initiated from the Plug and Play scheduled task named 'Plug and Play Cleanup' as seen in the following screenshot:

I've found very few people talking about this online and even fewer DFIR people who seem to be aware, I know we are going to add it to the SANS windows forensics course. According to this post on MSDN the scheduled task will remove from the most common device storage registry keys all devices that haven't been plugged in for 30 days. When this removal happens like all other PnP installs and uninstalls it will be logged in and here is an example of such an entry:

">>>  [Device and Driver Disk Cleanup Handler]
>>>  Section start 2017/04/08 18:54:37.650
      cmd: taskhostw.exe
     set: Searching for not-recently detected devices that may be removed from the system.
     set: Devices will be removed during this pass.
     set: Device STORAGE\VOLUME\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#07075B8D9E409826&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} will be removed.
     set: Device STORAGE\VOLUME\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#07075B8D9E409826&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} was removed.
     set: Device SWD\WPDBUSENUM\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#07075B8D9E409826&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} will be removed.
     set: Device SWD\WPDBUSENUM\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#07075B8D9E409826&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} was removed.
     set: Device USB\VID_13FE&PID_5200\07075B8D9E409826 will be removed.
     set: Device USB\VID_13FE&PID_5200\07075B8D9E409826 was removed.
     set: Device STORAGE\VOLUME\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#070A6C62772BB880&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} will be removed.
     set: Device STORAGE\VOLUME\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#070A6C62772BB880&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} was removed.
     set: Device USBSTOR\DISK&VEN_&PROD_&REV_PMAP\070A6C62772BB880&0 will be removed.
     set: Device USBSTOR\DISK&VEN_&PROD_&REV_PMAP\070A6C62772BB880&0 was removed.
     set: Device USB\VID_13FE&PID_5500\070A6C62772BB880 will be removed.
     set: Device USB\VID_13FE&PID_5500\070A6C62772BB880 was removed.
     set: Device SWD\WPDBUSENUM\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#070A6C62772BB880&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} will be removed.
     set: Device SWD\WPDBUSENUM\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#070A6C62772BB880&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B} was removed.
     set: Device USBSTOR\DISK&VEN_&PROD_&REV_PMAP\07075B8D9E409826&0 will be removed.
     set: Device USBSTOR\DISK&VEN_&PROD_&REV_PMAP\07075B8D9E409826&0 was removed.
     set: Devices removed: 23
     set: Searching for unused drivers that may be removed from the system.
     set: Drivers will be removed during this pass.
     set: Recovery Timestamp: 11/11/2016 19:51:27:0391.
     set: Driver packages removed: 0
     set: Total size on disk: 0
<<<  Section end 2017/04/08 18:54:41.415
<<<  [Exit status: SUCCESS]"

Followed one of these for each device identified:
>>>  [Delete Device - STORAGE\VOLUME\_??_USBSTOR#DISK&VEN_&PROD_&REV_PMAP#07075B8D9E409826&0#{53F56307-B6BF-11D0-94F2-00A0C91EFB8B}]
>>>  Section start 2017/04/08 18:54:37.666
      cmd: taskhostw.exe
<<<  Section end 2017/04/08 18:54:37.704
<<<  [Exit status: SUCCESS]

Now the isn't the only place these devices will remain. You will also find them in the following registry keys in my testing:

Note that these removals do not effect the shell items artifacts (Lnk, Jumplist, Shellbags) that would be pointing to files accessed from these devices, just the common registry entries that record their existence. 

So why is this important? If you are being asked to review external devices accessed in a Windows 8.1 or newer system you will have to take additional steps to ensure that you account for any device that hasn't been plugged in for 30 days. In my testing the Woanware USBDeviceForensics tool will miss these devices in their reports.

So make sure to check! It's on by default and there could be a lot of devices you miss. 

Monday, March 20, 2017

DFIR Exposed #1: The crime of silence

Hello Reader,
          I've often been told I should commit to writing some of the stories of the cases we've worked as to not forget them. I've been told that I should write a book of them, and maybe some day I will. Until then I wanted to share some of cases we've worked where things went outside the norm to help you be aware of not what usually happens, but what happens with humans get involved.

Our story begins...
It's early January, the Christmas rush has just ended and my client reaches out to me stating.

"Hey Dave, Our customer has suffered a breach and credit cards have been sent to an email address in Russia"

No problem, this is unfortunately fairly common so I respond we can meet the client as soon as they are ready. After contracts are signed we are informed there are two locations we need to visit.

1. The datacenter where the servers affected are hosted which have not been preserved yet
2. The offices where the developers worked who noticed the breach

Now at this point you are saying, Dave... you do IR work? You don't talk about that much. No, we don't talk about it much, for a reason. We do IR work through attorneys mainly to preserve the privilege and I've always been worried that making that a public part of our offering would effect the DF part of our services as my people would be flying around the country.

BTW Did you know that IR investigations lead by an attorney are considered work product by case law on a case I'm happy to say I worked on? Read more here:

So we send out one examiner to the datacenter while we gather information from the developers. Now you may be wondering why we were talking to the developers and not the security staff. It was the developers who found the intrusion after trying to track down an error in their code and comparing the production code to their checked in repository. Once they compared it they found a change in their shopping cart that showed the form submitted with the payment instructions was being processed while also being emailed to a russian hosted email address.

The developers claimed this was their first knowledge of any change and company management was quite upset with the datacenter as they were supposed to provide security in their view of the hosting contracts signed. Ideas of liability and litigation against the hosting provider were floating around and I was put on notice to see what existed to support that.

It was then that I got a call from my examiner who went to the datacenter, he let me know that one of the employees of the hosting company handed him a thumbdrive while he was imaging the systems saying only:

 'You'll want to read this'

You know what? He was right!

On the thumbdrive was a transcript of a ticket that was opened by the hosting companies SOC. In the transcript it was revealed that a month earlier the SOC staff was informing the same developers who claimed to have no prior knowledge of an intrusion that a foreign ip had logged into their VPS as root ... and that probably wasn't a good thing.

I called the attorney right away and let her know she likely needed to switch her focus from possible litigation against the hosting provider and to an internal investigation to find out what actually happened. Of course we still needed to finish our investigation of the compromise itself to make sure the damage was understood from a notification perspective.

Step 1. Analyzing the compromised server

Luckily for us the SOC ticket showed us when the attacker had first logged in as the root account which we were able to verify through the carved syslog files. We then went through the servers and located the effected files, established the mechanism used and helped them define the time frame of compromise so they could through their account records to find all the affected customers.

Unfortunately for our client, it was the Christmas season and one their busiest time of year. Luckily for the client it happened after Black Friday which IS their busiest time of the year. After identifying the access, modifications and exfil methods we turned our focus to the developers.

We talked to the attorney and came up with a game plan. First we would inform them that we needed to examine each of their workstations to make sure they were not compromised and open for re-exploitation, which was true. Then we would go back through their emails, chat logs and forensic artifacts to understand what they knew and or did when they were first notified of the breach. Lastly we would bring them in to be interviewed to see who would admit to what.

Imaging the computers was uneventful as you always hope it was, but the examination turned out to be very interesting. The developers used Skype to talk to each other and if you've ever analyzed Skype before you know that by default it keeps history Forever. There in the Skype chats was the developers talking to each other about the breach when it happened, asking each other questions about the attackers ip address, passing links and answers back to each other.

And then.... Nothing

Step 2. Investigating the developers

You see investigations are not strictly about the technical analysis in many cases, some are though, there is always the human element which is why I've stayed so enthralled by this field for so long. In this case the developers were under the belief that they were going to be laid off after Christmas so rather than take action they decided it wasn't their problem and went on with their lives. They did ask the hosting provider for recommendations of what to do next, but never followed up on them.

A month later they got informed they were not being laid off, and instead were going to be transferred to a different department. With the knowledge that this was suddenly their problem again they decided to actually look at the hosted system and found the modified code.

Step 3. Wrapping it up

So knowing this and comparing notes with the attorney we brought them in for an interview.

The first round we simply asked questions to see what they would say, who would admit what and possibly who could keep their jobs. When we finished talking to all the developers, all of which pretended to know nothing of the earlier date we documented their 'facts' and thanked them.

Then we asked them back in and one fact at a time showed them what we knew. Suddenly memories returned, apologies were given and the chronology of events was established. As it turns out the developers never notified management of the issue until the knew they were going to remain employed and just sat on the issue.

Needless to say, they no longer had that transfer option open as they were summarily terminated.

So in this case a breach that should have only lasted 4 hours at most (time of login notice by SOC to time of remediation) lasted 30 days of Christmas shopping because the developers of the eCommerce site committed the crime of silence for purely human reasons.