Posted October 21st, 2010 by Jason
Things have been really busy lately. First off, my Mentor session for SANS Security 504 started on September 21st. We are at the halfway point right now and leading this has been incredible. It seems whenever I need to present or teach something I learn more than anyone else. Plus teaching is just fun! Particularly when it is about stuff that I really enjoy. The student reviews have been great so far, so I must be doing something right.
Next, the Utah Open Source Conference (UTOSC) was two weeks ago, from October 7th to the 9th. There were a lot of great presentations and I had an absolute blast hanging out with all the technology loving folks who came. I was somewhat surprised on how far some people came to get there. I met people from Idaho, Wyoming, and California. It was really fun to sit in presentations and workshops on things that aren’t necessarily security related. It gave me some time to listen to what’s going on in other areas of technology and that’s can be really refreshing.
If that wasn’t enough, I also was able to speak at UTOSC again this year. Last year I spoke on building a toolkit of open source security tools. This year I did a presentation on Metasploit. I picked Metasploit because I started learning how to write modules for the framework this year. So doing a presentation on it seemed like a great way to learn even more about it. If nothing else, it would get me to spend more time using it and that’s where things really take off. My recording of the presentation didn’t go so well, so I’m waiting for the folks at UTOSC to release their recording. My slides are online though and you can download them on this site or view them at SlideShare.net/Tadaka. I put the presentation slides up on SlideShare after some folks expressed their reservations about downloading a PDF file from me after I mentioned backdooring PDF files. Go figure!
One offshoot of the UTOSC presentation was that one of the audience members, Joshua Williams, participates in the Linux Basix Podcast. It turns out that Joshua does this podcast with Infolookup, who is someone I know from IRC. They were chatting about my presentation and Infolookup realized that I was the guy he knew in IRC. One thing lead to another and I was invited to participate on last week’s Linux Basix Podcast! We spent about an hour chatting it up about Metasploit and just covering some of the basics about it. I really want to thank they guys for inviting me on. I had a ton of fun and they were all extremely friendly. You can download the podcast at http://www.linuxbasix.com/026-LB.
So that’s what I’ve been doing for the last month and a half or so. Now to something still to come. Paraben’s forensics conference (PFIC) will be on November 7th to 10th. I went last year and had a great time. Being very new to the forensics world, it was quite an interesting event. It is in Park City, UT at The Canyons Resort. I’m really looking forward to this year. If you are in the Utah area and want to attend a great, inexpensive conference then check it out. The cost is $299 and has some excellent speakers scheduled up. If you are there, let me know and we can meet somewhere. Meeting new people is one of the really cool things about conferences like this.
That’s all the events at this point. I’m hoping to get something else scheduled up in the next few months, but we will have to see how that goes. I’m also planning on doing another SANS Mentor session next year. I take what I learned this year and apply it to the next class.
Tags: JW Network Consulting, Linux Basix Podcast, Metasploit, PFIC 2010, Utah Technology Events, UTOSC
Posted July 1st, 2010 by Jason
Having a solid incident response capability isn’t an accident. It’s the result of focused preparation, training and culture. Incidents come at unexpected times, frequently with little warning, and can have a severe impact on an organization. It’s during these times that inadequate planning, documentation and missing tools become painfully apparent. That high level incident response plan that made the auditor happy suddenly doesn’t seem to be very useful. Perhaps we think that the plan will give us the overall guidance and we’ll be able to figure it out as we go from there. Think this will work? Think again. Let’s look at one possibility outside of information security.
What if a Fire Department Wasn’t Prepared?
The dispatch alarm goes off in the middle of the night, waking up the firefighters on duty. They go scrambling down the stairs to their truck and equipment. Two of them start running around trying to find their turnouts.
“Hey Sam, where are my boots?!?”
Another has his gear, but he’s looking around the cab of the fire engine for the keys. The last member of the crew is hurriedly going through the equipment on the engine, attempting to make sure they have everything needed for the current call.
“Was that a house fire or a traffic accident? Where did we put the first aid kit last time?”
After a lot of fumbling around, everyone thinks they have the proper gear and the engine rolls out of the station with the lights and sirens wailing. One problem though, the gas tank is nearly empty. After partially filling up, they are back on the road heading to the call.
On arrival they start pulling out tangled hoses, hook the fire hydrant up to the engine and disaster strikes again…
“Does anyone know where the wrench for the fire hydrant is?”
Thankfully, scenes like this are not played out in our fire departments. The men and women who become firefighters are trained, professional and prepared. Their turnouts are always ready that they can grab them, climb into the engine and race to save someone’s life. They don’t have to rummage around the fire engine checking for equipment, because when they got back from the last call they made sure they were ready for the next call. And they did so in a way that the next call can be a fire, an accident or the proverbial cat stuck in the tree.
Fire fighters are the consummate “incident responders”. They are able to drop what they are doing or wake up from a dead sleep, grab their gear and start saving lives. Their preparation is such that they are ready to respond to a variety of issues.
As computer incident responders, there are some things that we can learn from these professionals. First, they are selected carefully and trained in the proper tools and techniques. Next, they assemble the equipment and store it on a mobile “jump kit” known as a fire engine. Last, after they use their fire engine, they restock it and prepare it for the next emergency. So let’s look at the nearest equivalents in computer incident response.
Selection of Incident Handlers
Incident handlers need to be selected with serious consideration. When an incident occurs the incident response team is called in to deal with the situation. The business is willing to make this call because the incident handlers are the experts in this stuff. They trust them. The team is made up of people with solid technical backgrounds, they know what needs to be done, how it needs to be done and can guide everyone else through the process.
An incident is stressful on everyone around it. People may fear for their jobs or careers. They aren’t sure what to do or how to even begin. It may be pure panic. Incident response team members need to be able to handle their own stress and the stress of those around them. They need to be calm and help calm down everyone else around them. They must make good decisions under pressure. Communication is a key skill that will be relied on heavily. They will need a good understanding of the business and what’s important to it. Plus they must know the procedures to handle evidence gathered in response properly so that the business is protected or has legal recourse. Last, they must to have the technical knowledge and analytic capabilities to get the business out of the incident and back into regular operations.
This kind of expertise becomes a reality because people have made a deliberate choice to pursue and gain these skills. They are not individuals who can be picked at random. Choose carefully and deliberately. Then support them in fulfilling the mission they’ve been assigned.
Incident response requires some equipment on hand and ready to go. It’s not a kit filled with terribly unusual or expensive items. It involves deciding what needs to be in the kit and putting it together into something frequently referred to as the “Jump Kit”. It’s an organized bag with all the basics needed to respond to a variety of incidents. As you assemble it an iron clad rule must be followed.
NO BORROWING FROM THE JUMP KIT! NO EXCEPTIONS!
With that rule out of the way, what should be in a jump kit? NIST has created Special Publication 800-61 – Computer Security Incident Handling Guide to help government agencies prepare for their incident response capabilities. It contains a number of recommendations for a jump kit. SANS also has assembled a similar list of information, which is distributed to students of Security 504, Hacker Techniques, Exploits and Incident Handling. Both are excellent sources of information.
The list below is a combination of ingredients from both sources. I have some of these items but not all yet. I’m still working on building my kit.
- Contact information for team members and others within and outside the organization (primary and backup contacts). If you’re depending on the online address list, but that server is compromised you may be in trouble.
- Pagers or cell phones. Don’t forget spare batteries and chargers
- Laptop with lots of RAM, a large hard drive and pre-loaded with forensics applications
- Virtual machine software such as Xen, VMware, or Virtual PC so you can run multiple operating systems
- Live CDs of various types. Some options are Helix, CAIN or DEFT for forensics. Backtrack is another possiblity, though it is focused on penetration testing
- DVDs or CDs with trusted, statically linked versions of programs to be used to gather evidence from systems
- Packet sniffers and protocol analyzers to capture and analyze network traffic
- Encryption software. If the network is not trustworthy, you need encryption to keep your communication confidential
- Blank media, such as floppy disks, CD-Rs, and DVD-Rs
- USB thumb drives. 8 and 16 GB drives aren’t too expensive
- A network hub or a tap. A tap would be better, but they cost more
- Write blocking device(s) to use when imaging hard drives or other media
- A large external hard drive. Maybe more than one.
- Tools such as screwdrivers, flashlight, tweezers, telescoping magnet or hands. I tend to bring a Gerber utility knife with me where ever I go. Watch out when flying some where though. Be prepared to check your kit if you carry a pocket knife or screwdrivers
- USB to serial port adapter
- Network cables. Straight and crossover. I carry a crossover adapter
- Hard drive jumpers
- Cisco rollover cable and a serial cable
- Female to female RJ45 connector
- Hard-bound notebooks with numbered pages
- Digital camera and audio recorder
- Chain of custody and other incident forms
- Evidence storage bags and tags, and evidence tape. Amazon.com has these items
- Desiccants for protecting against moisture in the bags
- Port lists, including commonly used ports and Trojan horse ports
- Cryptographic hashes of critical files to speed the analysis, verification, and eradication of incidents
- Media, including OS boot disks and CD-ROMs, OS media, and application media
- Security patches from OS and application vendors
Edit — 7/21/2010
One quick note to add to this list. Consider what items are practical to have a couple of. In my case the power switch on my write blocker failed when I was imaging a number of systems. The vendor, Digital Intelligence, responded in short order to replace my write blocker and was awesome. But in the mean time I was stuck. The lesson of redundancy was re-taught to me at that point. If duplicate equipment is not possible consider what your backup plan is if something dies on you.
– Continuing the original post…
If you are doing incident response internally…
- Documentation for operating systems, applications, protocols, and intrusion detection and anti-virus systems
- Network diagrams and lists of critical assets, such as Web, email, and database servers
- Baselines of expected network, system and application activity
- Backup images of OS, applications, and data stored on secondary media
You may decide that this list isn’t complete. It’s not. It’s just a starting place. There may be that one time when you really needed something and didn’t have it. So now you always make sure to have it with you. After each incident, review how things went and ask the team if there was some piece of equipment that wasn’t there. Adjust, improve, and then restock.
Jump Kit Restocking
After the incident is over several items from the kit may have been used and need to be replaced. Compare your inventory of what the bag should have to what is left in it. Note any missing or nearly depleted items. The team may have noted some items that they wish that they had had or that needed to be purchased during the incident. If appropriate, add them to the list of jump bag items and purchase it for next time. Make sure that everything is stored properly, so that it can be used without needing to untangle it from something else first. Once the bag is ready, put it in its normal place and have it ready for next time. Your team is now ready to respond to the next incident with confidence that when they grab the bag that it is ready to go.
How Are We Doing Now?
The team has been selected, vetted and trained. Everyone knows where the jump kit is and that it has the gear necessary. When the phones start ringing and an incident is declared, the team starts gathering their gear and congregating as planned. Instead of scrambling for equipment and wondering what to do next, the team is ready, confident and it shows. The incident has a much better chance of being resolved quicker, more completely and according to accepted practices.
Tags: computer incident, incident response, preparation, security incident
Posted June 15th, 2010 by Jason
Last year I was able to speak at the Utah Open Source Conference on building a security toolkit with open source software. I just finished submitting my proposal for this year entitled “Metasploit: Free, Powerful, Flexible”. Being able to present at UTOSC 2009 was an absolute blast and I hope that my presentation is accepted this year as well. The conference is well run and attended. There are a ton of great topics and speakers to listen to. Even if I don’t get accepted as a speaker, I will be there again this year. It’s just to great to pass up on and for $35 it can’t be beat.
Hope to see you there!
Tags: Metasploit, Security Presentation, Utah Open Source Conference, UTOSC
Posted June 15th, 2010 by Jason
Just a quick note today. I finished working on a Metasploit module to create usernames the same way that the other two scripts in Reconnoiter does. However, this module is able to search Yahoo or Google and does not require separate scripts to do so. It also provides the option to use msfweb to get a web interface to run it from and lets you chose whether to dump the output to the console or text files in a directory of your choosing. It has been tested on version 3.4 and 3.3 of Metasploit, so it should be good to go. You can download it from https://sourceforge.net/projects/reconnoiter/files/.
So far it looks to be working very well. A big thanks to Robin Wood (digininja), Larry Pesce (haxorthematrix) and Carlos Perez (Carlos_Perez or darkoperator) for their help. All three were extremely willing to help me add functions, correct bugs and provide advice.
Tags: Metasploit module, reconnoiter project, username generation
Posted May 31st, 2010 by Jason
One of the most disheartening things about the Gulf of Mexico disaster is to watch BP, the government and other involved parties appear to make up their response as they go along. Aren’t oil companies required to plan for failures and how to recover from them? As it turns out yes, they are. Tonight I found the official “Regional Oil Spill Response Plan – Gulf of Mexico” that BP filed with the US government. Some sections I read completely, but at a minimum I looked at every single page of it. (All 583 of them.) If you want to read through it you can download it here. In fact, you probably ought to grab your Incident Response Plan and compare it to BP’s response plan. Here’s some of my thoughts on their plan and what lessons may be for those of us in InfoSec to learn from.
The first thing I saw that this document was definitely written to meet regulatory requirements. It has lots of flow charts, organizational charts, and contact information. It details the contracts maintained with ships to perform oil skimming and position containment booms. Planes to drop dispersants? Check. Detailed info about the dispersants themselves? Yup, got that. Do they have procedures to use when deciding how to perform surface burns? Of course! There is information on how to assist animal life of all kinds, including some that have probably never been in the Gulf of Mexico. It even has a section on the “Worst Case Discharge”. And on it goes. So why are we still spilling 500,000 – 700,000 gallons (12,000 – 19,000 barrels) of oil per day in spite of this lovely plan?
The most glaring omission is that the plan has not a single mention of how to stop an active oil spill. Not a peep about it, whether its on land or on the ocean floor. There isn’t even a reference to other documents on how a blowout would be halted. It does mention practice exercises and ongoing training, but even that appears to be focused on mopping up oil on beaches and the ocean surface. I suspect that this plays into why containment chambers, top kills, junk shots and other ideas are getting thrown around. Question one. Does your incident response plan include or at least reference other docs on how to stop the bleeding?
Next issue. The “Worst Case Discharge” appendix contains an estimate of 250,000 barrels leaking per day using an official formula required by the US government. It has a containment and clean up strategy using oil skimmers and dispersants. Last it has a detailed inventory of ships and planes that will be used in the attempt. The problem that I see with this is that it has little relation to reality. Fine, its the worst possible amount of oil that can be discharged and fortunately, the current spill is not that big. But what happens if the entire drilling rig blows up? What do you do when the blowout preventer fails? How will you handle it when hurricane season hits? What if the failure is a mile underwater? What happens if multiple wells blow out? No mention of it. Question two. Does your incident response plan take into account business requirements? Like end of the month billing and an incident? Is your worst case scenario a dry formula or does it mean something to the business?
Last issue. The BP response plan looks like it was written to keep auditors happy. It looks like it has some really important information that I’d bet has been used during this process. Contact numbers for service providers and contracts with critical damage control resources is important. But I think this thing was written to meet a checklist. It met the legal requirements currently in force and passed muster. A document designed to keep an auditor happy is trouble in the making. Instead how about writing one that tries to meet the need of the potential crisis? It will probably have what the auditor wants and if there are a couple of hoops to add, so be it. But write the thing to deal with the leak of 500,000 gallons of oil coming out of the ground. Or is it credit cards leaking? Compliance is great… as long as it comes after the main issue is addressed.
Tags: incident response, lessons learned, planning for disaster