NCCDC 2013 Lessons Learned
Hello readers,
For those who are not familiar, the National Collegiate Cyber Defense Competition (NCCDC) is held once a year in San Antonio, Texas. The 10 winning teams from regionals held across the united states come to San Antonio to prove their abilities against an active and aggressive attacker while at the same time completing business objectives and dealing with simulated customers/users. It's only open to teams of college students and they train through the year and begin the qualification process in late winter.
I am the captain of the NCCDC red team and have been for 7 years now and in those 7 years we've expanded and grown as a red team to deliver what I would like to think is a unique and compelling experience to the student teams.
On of my redteamers, Alex Levinson, posted a blog with his takeaways from being a CCDC bad guy. You can read it here: https://alexlevinson.wordpress.com/2013/04/24/ccdc-2013-red-team-feedback/
Raphael Mudge gives a more thorough accounting of this years tricks in his write up: http://blog.strategiccyber.com/2013/04/24/national-ccdc-red-team-fair-and-balanced/
I thought I would follow up as promised and give a break down that goes beyond the slide show I posted in the previous post. When I give my redteam debrief I try to collect the screenshots from around the redteam that best illustrate the mistakes that we see made during the competition. As I write this post I've realized its important that I detail more about what we do at national ccdc as a red team for those of you that have never experienced our welcoming redteam hospitality. As the red team captain I have several duties:
1. Set the strategy for this years attacks
2. Make sure the best possible volunteers are located and brought in to bring the best possible redteam to bear against teams shown to be capable enough to survive the other redteams on their journey.
3. Restrict access to the redteam room to prevent distractions
4. The assignment of redteam members to teams
5. Make a room full of usually solo penetration testers work together and follow my plan
6. Talk to tour groups as they are escorted into the redteam room to explain how we operate, our plan and to prevent them from distracting the redteam.
7. To force communication and cooperation across the different redteam members so all student teams receive the same love.
8. To assist in cultivating custom toolkits, backdoors and tools and get communication started before the event so that everyone knows what is possible.
9. To enforce the rules of the competition to make sure no redteam member voids the provisions put in place to insure a fair contest.
10. To make sure the students meet their redteam members and that they hopefully learn something from the experience.
11. Facilitate requests between the redteam and the gold team
12. Take care of my own assigned student team in making sure I leave behind my own presents.
13. Make sure all the red team members fill out their incident submissions
In short, I have lots to do over the weekend and over the years I've learned alot from the process. I even did a talk at derbycon lat year about what I've learned: http://www.irongeek.com/i.php?page=videos/derbycon2/david-cowen-running-a-successful-red-team If you are running a red team I recommend you watch it.
So after explaining all of that, here is what I think student teams and the industry in general needs to consider when defending.
1. Reinstall is not the solution for remediation
This idea that reinstalling is the best way to recover from an intrusion is something that is not isolated to CCDC students, its a common trend in the industry. However as a CCDC competitor you are under a microscope with an attacker who knows you have to put that system back up as soon as possible to stop the bleeding.
This in the short term is true, however in the long term (as in the rest of the competition) its a faulty perception. The SLA violation you take for having your services down is the largest continual point disruption we can generate as a redteam, all other actions we take against you are one time/one point deduction activities. We bring a bag of tricks to NCCDC but we don't have the time a real world attacker does to continually generate new tools/new techniques in the span of two days. Once you detect and block our years sneaky du jour you will have effectively blocked us for the rest of the competition in a form that will scale to the rest of your systems. This means that in the long term you can keep us out, keep your systems up and your SLA violations to a minimum.
Failing to do this an reacting to the short term access will just cause the same pain to reoccur. This creates a race between you and the redteam to see who can get back into your system faster once you've restored and then take the system down again .. hopefully before the scoring engine checks for an update and you are just continually seen as being down. The SLA violation grows for each period your down, keeping you down is a tactic not just a funny thing to do.
2. Logfiles are important and part of a bigger picture of data
Past teams were quite observant of the logs to their external services, current teams have seem to lost the art. Watching logs for the services you are providing externally and for errors and login events will help you go a long way in proactively detecting our accesses, probes and intrusions. Beyond the default logs being created for you, learn how to configure them to add additional information to capture more of what we are doing.
3. You have to understand normal to identify abnormal
This may be the most important bit of advice, and the hardest to understand. You need to have worked with an operating system long enough to know what processes, behaviors, files, activity is part of the actual operating system. The only way to do this is with practice, installing and working with different parts of the operating system and seeing what changes, gets added, gets deleted, gets executed.
Once you've learned what is normal, what accounts should own processes, what ports should be open, what ips defined services should be communicating with, etc... Our activities and especially our persistence will stand out much more. Watching a team launch TCP View while you watch them and they don't notice your connection means they are hoping for some giant flag saying 'HACKER FOUND' rather than understanding what traffic is abnormal/bad.
4. Knowing your operating system outside of a google search is important
I understand we all use google, I use google and other search engines every day. However, if to quickly manage and configure your operating system and its services you have to turn to google then you've created a problem. You should know the system and the commands well enough before competition to be able to secure your system as quickly as possible and bring up new services without us watching you google instructions for the next hour as you stumble.
5. Knowing your applications/services capabilities is the only way to secure them
If you encounter a new application either that you have to install or that you find already running the first things you need to understand are:
1. How to administer it locally
2. Does it have default credentials
3. Does it have remote administration capability
4. Search the documentation for security configuration
5. Find out where the application creates logs and error logs
6. Does it connect to another service/database
Then go back through 1-6 and lock it down. We are doing the same thing on the red team side, after all these things fail to get us in will we then start a code review/known exploit search .
6. Learn some basic Incident Response tools and techniques
Alex Levinson said something particularly insightful one evening over margaritas, "When I was a student I saw CCDC as a system administration contest, but really its an Incident Response contest". I think there is alot of truth to this, most students focus on how to install, configure and setup the operating system. Some students get interested in the red teaming aspect of it, but very few get interested into the forensics and incident response aspect of CCDC.
Forensics is what I do for a living so maybe I'm a bit bias, but the amount of grief you could save your team and the number of points you could recoup from our attacks is enough to make atleast one person of your team the incident responder. They should focus on the following:
1. Learn how to capture live memory
2. Learn how to use volatility to find possible malware
3. Learn how to scan for alternate data streams
4. Learn how to work with forensic artifacts such as prefetch and the application compatibility cache
5. Learn how to make and scan timelines for malicious activies
6. Capture network traffic and look for us
Doing this does not take as long as you think once you get good at it and in doing so you will be able to identify, detect, respond and eliminate us and our persistence.
Next year will be harder, I warn you now. I have plans blue teams and you are are the center of them. Take this blog post as a warning and be ready.
I'll be doing an AMA on reddit Monday April 29th 2013 at 1:30pm if you want ask questions or you can leave comments below.
