Monday, January 13, 2014

Daily Blog #204: Sunday Funday 1/12/14 Winner!

Hello Reader,
          I thought this weeks challenge would have gotten more of you to write in with your best stories. Instead I received just one submission that was willing to tell a story, in return he just won a $1495 ticket to the SANS DFIR Summit. Take this as a lesson, if your answer or response isn't in your mind the greatest thing ever ... it may be enough to win! I'll take care of filling in the details you missed in later blog entries.

The Challenge:
 Write your most challenging DFIR case and how you overcame the obstacles and the outcome. I'll take the top best cases based on our opinion and open it up for voting to all of you to pick the winner. Any kind of DFIR case is valid here, there are no boundaries on what makes something a good case. We will be judging your case with the following criteria to determine those cases to vote for:
1. Technical Challenges faced
2. Novel solutions
3. Result of your work
4. Interesting scenario
The Winning Answer:

 
This was one of my first DFIR experiences so I learned a lot from this and got me my first taste and want to get into IR and forensics.

I was working as a NT Server administrator for a dot com back in the late 90s/00s.  The security team for the company was contacted by a three letter agency that our IP addresses had been seen in part of another case they were working on led them to believe we had a compromised host.  The IP address given to us was of course the one and only email server for the entire company.  We had a dedicated security team but they were all linux guys and we were using NT4 and Exchange 5.5 for email.  Being the lone windows admin meant the investigation fell upon me.  I was told that to make sure the investigation was in-depth as if it wasn’t then there was a chance the three letter government agency may come in and seize the equipment.

First the technical challenges faced:
This was a long time ago and DFIR is not what it is today so the tools, documentation, etc was not what it is today so one of the first challenges was having to make it up as I went along.  Since it was a mail/web server it obviously had several paths of entry.  First I went and logged directly into the server from the console and began looking at running processes and other active sessions on the system.  I ended up finding Serv-U FTP had been installed and had a user list with accounts that were all using leetspeak.  Luckily the ports being used for Serv-U FTP was blocked at the firewall so it had only been installed but wasn’t able to be accessed.  As part of the investigation I also ran into some a folder that were flagged as hidden and no matter what I did to change permissions I could not access it.  

Novel solution:
After trying several methods to access it I dug out an old copy of dos based file/folder viewer.  It somehow was able to ignore permissions, flags, etc and allowed access to it.  Within the folder I was able to find a clear text log file from where msgina.dll had been replaced and any accounts that had logged directly into the console had been logged with their password in clear text! This was both good and bad as it was the first time I had seen my own account and the domain admin account in a clear text keylog file.  This of course led to more efforts and had to force password changes on every account, service, etc.

Results of the work:
Since the server was obviously compromised the end result was the decision to wipe and rebuild the server.  Of course this was all decided on a Friday afternoon and it was my task to now figure out how to wipe the OS, rebuild it, and retain all the MS Exchange databases and have it all back clean and working by Monday morning to minimize the impact to the company.  Given this scenario I made my first and only call to Microsoft to get exact detailed directions from them on the process of rebuilding an Exchange server but retaining all the mail databases.  So just to ensure I could successfully do this I took the process and verified I could successfully complete it using other hardware before I completely wiped the lone corporate mail server.  After a very late night on Friday I had success and came back in the next day and had to repeat the process on the real server.  In the end the agency didn’t come take away our mail server, I got to learn more about IR and Exchange, and I ended up finding an interest in the security side of IT.  So to whoever it was all those many years ago thanks for helping me find the desire and interest in having a career in DFIR.

TLDR; Bad guys own company mail server, server admin thrown into DIFR and decides to make a career of it.