Preparing for Incident Response

Having a solid incident response capability isn’t an accident.  It’s the result of focused preparation, training and culture.  Incidents come at unexpected times, frequently with little warning, and can have a severe impact on an organization.  It’s during these times that inadequate planning, documentation and missing tools become painfully apparent.  That high level incident response plan that made the auditor happy suddenly doesn’t seem to be very useful.  Perhaps we think that the plan will give us the overall guidance and we’ll be able to figure it out as we go from there.  Think this will work?  Think again.  Let’s look at one possibility outside of information security.

Thankfully, scenes like this are not played out in our fire departments.  The men and women who become firefighters are trained, professional and prepared.  Their turnouts are always ready that they can grab them, climb into the engine and race to save someone’s life.  They don’t have to rummage around the fire engine checking for equipment, because when they got back from the last call they made sure they were ready for the next call.  And they did so in a way that the next call can be a fire, an accident or the proverbial cat stuck in the tree.

Fire fighters are the consummate “incident responders”.  They are able to drop what they are doing or wake up from a dead sleep, grab their gear and start saving lives.  Their preparation is such that they are ready to respond to a variety of issues. 

As computer incident responders, there are some things that we can learn from these professionals.  First, they are selected carefully and trained in the proper tools and techniques.  Next, they assemble the equipment and store it on a mobile “jump kit” known as a fire engine.  Last, after they use their fire engine, they restock it and prepare it for the next emergency.  So let’s look at the nearest equivalents in computer incident response.

Selection of Incident Handlers

Incident handlers need to be selected with serious consideration.  When an incident occurs the incident response team is called in to deal with the situation.  The business is willing to make this call because the incident handlers are the experts in this stuff.  They trust them.  The team is made up of people with solid technical backgrounds, they know what needs to be done, how it needs to be done and can guide everyone else through the process.

An incident is stressful on everyone around it.  People may fear for their jobs or careers.  They aren’t sure what to do or how to even begin.  It may be pure panic.  Incident response team members need to be able to handle their own stress and the stress of those around them.  They need to be calm and help calm down everyone else around them.  They must make good decisions under pressure.  Communication is a key skill that will be relied on heavily.  They will need a good understanding of the business and what’s important to it.  Plus they must know the procedures to handle evidence gathered in response properly so that the business is protected or has legal recourse.  Last, they must to have the technical knowledge and analytic capabilities to get the business out of the incident and back into regular operations.

This kind of expertise becomes a reality because people have made a deliberate choice to pursue and gain these skills.  They are not individuals who can be picked at random.  Choose carefully and deliberately.  Then support them in fulfilling the mission they’ve been assigned.

Equipment Preparation

Incident response requires some equipment on hand and ready to go.  It’s not a kit filled with terribly unusual or expensive items.  It involves deciding what needs to be in the kit and putting it together into something frequently referred to as the “Jump Kit”.  It’s an organized bag with all the basics needed to respond to a variety of incidents.  As you assemble it an iron clad rule must be followed.

With that rule out of the way, what should be in a jump kit?  NIST has created Special Publication 800-61 – Computer Security Incident Handling Guide to help government agencies prepare for their incident response capabilities.  It contains a number of recommendations for a jump kit.  SANS also has assembled a similar list of information, which is distributed to students of Security 504, Hacker Techniques, Exploits and Incident Handling.  Both are excellent sources of information.

The list below is a combination of ingredients from both sources.  I have some of these items but not all yet.  I’m still working on building my kit.

• Contact information for team members and others within and outside the organization (primary and backup contacts). If you’re depending on the online address list, but that server is compromised you may be in trouble.
• Pagers or cell phones. Don’t forget spare batteries and chargers
• Laptop with lots of RAM, a large hard drive and pre-loaded with forensics applications
• Virtual machine software such as Xen, VMware, or Virtual PC so you can run multiple operating systems
• Live CDs of various types. Some options are Helix, CAIN or DEFT for forensics. Backtrack is another possiblity, though it is focused on penetration testing
• DVDs or CDs with trusted, statically linked versions of programs to be used to gather evidence from systems
• Packet sniffers and protocol analyzers to capture and analyze network traffic
• Encryption software. If the network is not trustworthy, you need encryption to keep your communication confidential
• Blank media, such as floppy disks, CD-Rs, and DVD-Rs
• USB thumb drives. 8 and 16 GB drives aren’t too expensive
• A network hub or a tap. A tap would be better, but they cost more
• Write blocking device(s) to use when imaging hard drives or other media
• A large external hard drive. Maybe more than one.
• Tools such as screwdrivers, flashlight, tweezers, telescoping magnet or hands. I tend to bring a Gerber utility knife with me where ever I go. Watch out when flying some where though. Be prepared to check your kit if you carry a pocket knife or screwdrivers
• USB to serial port adapter
• Network cables. Straight and crossover. I carry a crossover adapter
• Hard drive jumpers
• Cisco rollover cable and a serial cable
• Female to female RJ45 connector
• Hard-bound notebooks with numbered pages
• Digital camera and audio recorder
• Chain of custody and other incident forms
• Evidence storage bags and tags, and evidence tape. Amazon.com has these items
• Desiccants for protecting against moisture in the bags
• Port lists, including commonly used ports and Trojan horse ports
• Cryptographic hashes of critical files to speed the analysis, verification, and eradication of incidents
• Media, including OS boot disks and CD-ROMs, OS media, and application media
• Security patches from OS and application vendors

Edit — 7/21/2010
One quick note to add to this list. Consider what items are practical to have a couple of. In my case the power switch on my write blocker failed when I was imaging a number of systems. The vendor, Digital Intelligence, responded in short order to replace my write blocker and was awesome. But in the mean time I was stuck. The lesson of redundancy was re-taught to me at that point. If duplicate equipment is not possible consider what your backup plan is if something dies on you.

– Continuing the original post…

If you are doing incident response internally…

• Documentation for operating systems, applications, protocols, and intrusion detection and anti-virus systems
• Network diagrams and lists of critical assets, such as Web, email, and database servers
• Baselines of expected network, system and application activity
• Backup images of OS, applications, and data stored on secondary media

You may decide that this list isn’t complete. It’s not. It’s just a starting place. There may be that one time when you really needed something and didn’t have it. So now you always make sure to have it with you. After each incident, review how things went and ask the team if there was some piece of equipment that wasn’t there. Adjust, improve, and then restock.

Jump Kit Restocking

After the incident is over several items from the kit may have been used and need to be replaced.  Compare your inventory of what the bag should have to what is left in it.  Note any missing or nearly depleted items.  The team may have noted some items that they wish that they had had or that needed to be purchased during the incident.  If appropriate, add them to the list of jump bag items and purchase it for next time.  Make sure that everything is stored properly, so that it can be used without needing to untangle it from something else first.  Once the bag is ready, put it in its normal place and have it ready for next time.  Your team is now ready to respond to the next incident with confidence that when they grab the bag that it is ready to go.

How Are We Doing Now?
The team has been selected, vetted and trained. Everyone knows where the jump kit is and that it has the gear necessary. When the phones start ringing and an incident is declared, the team starts gathering their gear and congregating as planned. Instead of scrambling for equipment and wondering what to do next, the team is ready, confident and it shows. The incident has a much better chance of being resolved quicker, more completely and according to accepted practices.

Technorati Tags: computer incident, incident response, preparation, security incident

Tags: computer incident, incident response, preparation, security incident

Submitted Speaking Proposal to the UTOS Conference 2010

Last year I was able to speak at the Utah Open Source Conference on building a security toolkit with open source software.  I just finished submitting my proposal for this year entitled “Metasploit: Free, Powerful, Flexible”.  Being able to present at UTOSC 2009 was an absolute blast and I hope that my presentation is accepted this year as well.  The conference is well run and attended.  There are a ton of great topics and speakers to listen to.  Even if I don’t get accepted as a speaker, I will be there again this year.  It’s just to great to pass up on and for $35 it can’t be beat.

http://2010.utosc.com/presentation/199/

Hope to see you there!

Technorati Tags: Metasploit, Security Presentation, Utah Open Source Conference, UTOSC

Tags: Metasploit, Security Presentation, Utah Open Source Conference, UTOSC

Reconnoiter Updated with Metasploit Module

Just a quick note today. I finished working on a Metasploit module to create usernames the same way that the other two scripts in Reconnoiter does. However, this module is able to search Yahoo or Google and does not require separate scripts to do so. It also provides the option to use msfweb to get a web interface to run it from and lets you chose whether to dump the output to the console or text files in a directory of your choosing. It has been tested on version 3.4 and 3.3 of Metasploit, so it should be good to go.  You can download it from https://sourceforge.net/projects/reconnoiter/files/.

So far it looks to be working very well.  A big thanks to Robin Wood (digininja), Larry Pesce (haxorthematrix) and Carlos Perez (Carlos_Perez or darkoperator) for their help.  All three were extremely willing to help me add functions, correct bugs and provide advice.

Technorati Tags: Metasploit module, reconnoiter project, username generation

Tags: Metasploit module, reconnoiter project, username generation

Learning from BP’s Response to the Gulf Disaster

One of the most disheartening things about the Gulf of Mexico disaster is to watch BP, the government and other involved parties appear to make up their response as they go along.  Aren’t oil companies required to plan for failures and how to recover from them?  As it turns out yes, they are.  Tonight I found the official “Regional Oil Spill Response Plan – Gulf of Mexico” that BP filed with the US government.  Some sections I read completely, but at a minimum I looked at every single page of it.  (All 583 of them.)  If you want to read through it you can download it here. In fact, you probably ought to grab your Incident Response Plan and compare it to BP’s response plan.  Here’s some of my thoughts on their plan and what lessons may be for those of us in InfoSec to learn from.

The first thing I saw that this document was definitely written to meet regulatory requirements.  It has lots of flow charts, organizational charts, and contact information.  It details the contracts maintained with ships to perform oil skimming and position containment booms.  Planes to drop dispersants?  Check.  Detailed info about the dispersants themselves?  Yup, got that.  Do they have procedures to use when deciding how to perform surface burns?  Of course!  There is information on how to assist animal life of all kinds, including some that have probably never been in the Gulf of Mexico.  It even has a section on the “Worst Case Discharge”.  And on it goes.  So why are we still spilling 500,000 – 700,000 gallons (12,000 – 19,000 barrels) of oil per day in spite of this lovely plan?

The most glaring omission is that the plan has not a single mention of how to stop an active oil spill.  Not a peep about it, whether its on land or on the ocean floor.  There isn’t even a reference to other documents on how a blowout would be halted. It does mention practice exercises and ongoing training, but even that appears to be focused on mopping up oil on beaches and the ocean surface.  I suspect that this plays into why containment chambers, top kills, junk shots and other ideas are getting thrown around.  Question one.  Does your incident response plan include or at least reference other docs on how to stop the bleeding?

Next issue.  The “Worst Case Discharge” appendix contains an estimate of 250,000 barrels leaking per day using an official formula required by the US government.  It has a containment and clean up strategy using oil skimmers and dispersants.  Last it has a detailed inventory of ships and planes that will be used in the attempt.  The problem that I see with this is that it has little relation to reality.  Fine, its the worst possible amount of oil that can be discharged and fortunately, the current spill is not that big.  But what happens if the entire drilling rig blows up?  What do you do when the blowout preventer fails?  How will you handle it when hurricane season hits?  What if the failure is a mile underwater?  What happens if multiple wells blow out?  No mention of it.  Question two.  Does your incident response plan take into account business requirements?  Like end of the month billing and an incident?  Is your worst case scenario a dry formula or does it mean something to the business?

Last issue.  The BP response plan looks like it was written to keep auditors happy.  It looks like it has some really important information that I’d bet has been used during this process.  Contact numbers for service providers and contracts with critical damage control resources is important.  But I think this thing was written to meet a checklist.  It met the legal requirements currently in force and passed muster.  A document designed to keep an auditor happy is trouble in the making.  Instead how about writing one that tries to meet the need of the potential crisis?  It will probably have what the auditor wants and if there are a couple of hoops to add, so be it.  But write the thing to deal with the leak of 500,000 gallons of oil coming out of the ground.  Or is it credit cards leaking?  Compliance is great…  as long as it comes after the main issue is addressed.

Technorati Tags: incident response, lessons learned, planning for disaster

Tags: incident response, lessons learned, planning for disaster

Upcoming SANS Mentor Session – Security 504: Hacker Techniques, Exploits & Incident Handling

This is something that I’ve really been looking forward to announcing for a while now.  I will be running a Mentor session for SANS starting on Sept 21 and running until November 23, 2010.  We will be meeting once per week for two hours to cover course material, discuss what we’ve studied and do some of the labs.  We will be meeting in South Jordan, so its fairly central to the Salt Lake and Provo area.

The course will cover how to handle security incidents, demonstrate what tools attackers use and how they exploit systems.  I took this course in 2009 and I can’t say enough how good it is.  It’s one thing to read about how an exploit works.  It brings a whole new level of awareness when you actually run an exploit and start pulling data from the target system.

What was most valuable to me was the background on how to prepare, respond and recover from a security incident.  There is quite a bit of preparation that you need to take so that you are ready to conduct incident response in a way which will stand up in civil or criminal proceedings.  There are lots of pitfalls that you want to avoid so that you protect your employer and don’t get in trouble yourself.

Here are two key things you can use to “sell” this course to management.

1. You will learn how to handle an incident in a way that best protects your company, is thorough and preserves the company’s legal options.
2. You will see what exploits can do to a system, how they are used and what they look like.  Every exploit covered includes how to defend against them.

If you are in the Salt Lake City area and would like to read more about this course or sign up, use the link below.

Security 504: Hacker Techniques, Exploits & Incident Handling

Technorati Tags: incident handling, salt lake city, SANS, sec504, security training

Tags: incident handling, salt lake city, SANS, sec504, security training

NetApp Shell Commands

One of the guys I work with sent me a link to an article on Kings Pride and I decided to post them here.  Mostly, so I can find them again later when need them later. Any how, from Kings Pride here’s the details on how to invoke a Java based shell on a NetApp.

Here’s the link to the post:

http://kingspride.org/2009/11/17/netapp-cmds-jsh/

Technorati Tags: netapp, ontap, shell commands

Tags: netapp, ontap, shell commands

Facebook and Privacy

So Facebook has made some changes to privacy that I didn’t like much.  Thought I’d pass it on.  The new change is that if one of your friends uses a Facebook application (any application) and it requests personal information, Facebook will share that information to them without your knowledge.  So even if you don’t want your birthday published to anyone but your friends, if they visit an app that asks for it, the app gets it.  This takes place unless you specify otherwise!

So what exactly does Facebook share.  I took a look at my Privacy Settings for Applications and Websites and here’s what they tell you that they share and what they were set to.

• Personal info – enabled
• Status Updates – enabled
• Online Presence – enabled
• Website – enabled
• Family and relationship status – enabled
• Relationship details – disabled
• Education and work – enabled
• My Videos – enabled
• My Links – enabled
• My Notes – enabled
• My Photos – enabled
• About Me – enabled
• My Birthday – enabled
• My hometown – enabled
• My religious and political views – disabled

So out of 15 settings, only 2 are disabled by default.  Facebook has unilaterally decided what information you are sharing.

So what’s the big deal?  Well, in most cases it may not bother you too much.  Unless you find ads for a singles website using pictures of your wife, daughter, sister or niece on it troublesome.  (It’s happened to people.)  But probably  (hopefully?), most of these applications and ads are going to use it strictly for business purposes and do not have evil purposes in mind.

However…  Not all apps, ads, or sites have pure intentions.  And no, Facebook doesn’t have the reputation of checking that hard to see if its legit.  They make money off these things so its in their best interests NOT to check too closely.  Some ads and apps are not nice and they will provide information to the pushers of spam, spyware and other fun stuff.  If a spammer wanted legitimate email addresses to spam, what if they create an app and pull every email address from every person to use the app and all of their friends email addresses?  It would work.  That info is up for grabs under the default settings.

Ok, so I’ve complained a bit now.  So how about fixing it.  In the upper right corner of your Facebook home page, click the arrow next to Account Settings.  A menu drops down and on it, click Privacy Settings.  On the page that comes up, click Applications and Websites.  Find the link What your friends can share about you and click it.  This finally gets you to the page where you can set your privacy preferences.  Set it to taste and click Save Changes.  That’s it, you have now gained a bit more control over what information is being shared about you.

Keep an eye on the Privacy Settings in Facebook.  Things change over time and sometimes there are some big changes.

Technorati Tags: Facebook, Information Disclosure, Privacy

Tags: Facebook, Information Disclosure, Privacy

A Quick Update on the Latest Activity

It has been a busy couple of months, but my posts have been fairly quiet on the blog. Between attending the SANS Security 504 Incident Handling class, traveling for work, moving my family and the holidays things have been moving at a rapid pace.  I’m going to be trying to comment more here, but for now a brief update.

First off, I took the examination for the SANS Incident Handler certification on Friday the 12th.  All the time put into preparation paid off and I passed with 96%!  It was extremely satisfying to pass this exam, since I have been spending the last several weeks studying for it.  On top of just earning the certification, my score was high enough that I can apply to become a SANS Mentor now too.  This is something that I think would be a lot of fun and I really want to do.  Time to start writing up my application and hoping for the best.

Last, I’ve started working on rewriting Reconnoiter to run as a Metasploit module.  I started on this late last week and have made some headway in the process.  Scrapping HTML and using the data in a script or program isn’t much fun.  The main problem I’ve run into is that Google really doesn’t want anyone doing this type of stuff.  While Yahoo! has provided a nice XML web service to aid in accessing data, Google appears to be going out of their way to make this difficult.  I’m actually a bit irritated by this since Google has taken great pains to convince everyone (with good reason on our part to do so) that we need to make it easy for Google to crawl our sites.  Just don’t expect them to return the favor.  Ah well.

Anyhow, I hope to have a rough module up and running by the end of this week.  Current plans are to have the ability to pull results from Google and Yahoo both.  You will be able specify an output directory to save username lists.  Plus, since this is in Metasploit, you can choose between the command line or a web interface to run the module.  That alone may be a real kicker for folks.

The Google query is going to be pretty buggy and a pain to maintain.  The Yahoo! query should be very solid since the script pulls from their XML web service.  The down side is that you will need an AppID to use the Yahoo! query.  So the decision is whether you want to be (relatively) anonymous but have iffy results or if you don’t mind losing some of that anominity for more accurate results.

Technorati Tags: GCIH, Incident Handler, Metasploit, Reconnoiter, SANS

Tags: GCIH, Incident Handler, Metasploit, Reconnoiter, SANS

Banking via Twitter??

When I first saw this fly by on Twitter (ironic) I thought it was Rsnake joking around.  I followed his comment about it over to Vantage Credit Union‘s web site and saw that sure enough, their customers can do limited banking via Twitter.  Before I go further, let me state this openly.  I like and use Twitter quite a bit.  The ability to find people with similar interests and hear about events on the fly is awesome.  All that said, here’s some of my thoughts.

I took a look through the documentation on how to use their service.  Customers can perform simple functions such as view their balances, deposits, withdrawals, holds, and cleared checks.  They can also transfer money between accounts inside of VCU.  Nothing particularly earth shattering about this.  The commands are pretty basic and easy (at least to me) to remember.  To pull the last 5 deposits customers send a direct message to the credit union using the following command.

The credit union’s twitter account receives the message, then sends back the results.

Ok, that’s kinda neat and the actions available in the first release aren’t too worrisome. But I’m still concerned about the whole idea of this. If I’m a customer of VCU (I’m not), do I want Twitter to be the middle man for access to my account? The whole security model hangs on the idea that direct messages are completely secure and only visible to the two parties taking part of the exchange. While I suspect Twitter has gone to some lengths to protect these messages, do we think that they designed its security with banking in mind? I doubt it ever entered their mind. Keep in mind that all of this is being done without encryption too.

However, for the sake of argument, lets say that direct messages never become exposed due to a vulnerability. Do people only use Twitter with their browser directly on Twitter’s website? Not by a long shot. I personally use two other applications to keep tabs on my feeds. These apps also have their own issues. I don’t know of any Twitter apps that do this, but I have heard of some apps sending data back to the developers. They also are subject to vulnerabilities. So even if Twitter itself is perfectly safe (it’s not), the mediums with which we use it aren’t.

And last, the whole notion just strikes me as asking for trouble. VCU plans on offering functionality via Facebook and text messaging as well. They have in essence decided to teach their users to trust third parties with access to their bank accounts. These aren’t third parties that we know either. I don’t know the people who are, will or used to work at Twitter. VCU is training their customers that it’s ok to put access to their financials in the hands of others. This is a bad idea and one that may come back to bite folks who believe that Twitter has their back because otherwise their bank would have never offered it.

Technorati Tags: bad idea, banking online, twitter, web security

Tags: bad idea, banking online, twitter, web security

The Importance of Backups and the SMB

Today we had a really bad thing that happened, but it ended up not mattering at all.  One of my clients was busy doing some work and accidentally deleted a directory with about 1 GB of data in it. I got a very worried email from them and if it could be restored.  Fortunately, we had recently implemented a new backup system and it was running very well.  I logged into the system, found the needed directory in the last backup to Mozy and kicked off the restore.  40 minutes later, they were back to using the files that had gone MIA.

Why did this not matter?  Because we had good backups in place.  The system syncs nightly to Mozy’s storage systems and I get daily updates on how things went.  When I got the email telling me about the issue I knew right away where the data would be and how to get it back.  No running around for tapes.  No hoping that last job had actually worked.

In the past, backups could cost quite a bit.  Tape drives, backup software and tapes don’t come cheap when you are a small or medium sized business (SMB).  You could easily drop $5,000 or more on a single tape drive, some software and enough tapes to keep it going for a while.  Then you have to switch tapes daily and all that other fun stuff.  As a result a lot of SMBs don’t have the backups they need and we all need them.  Accidents happen.

The good news is that backups don’t have to cost you a ton of capital.  How much did this wonderful peace of mind cost my client?  For this server, the grand total is less than $10 per month.  How long did it take to setup?  About 15 minutes.  How long and expensive could it have been if the data was completely gone?

Technorati Tags: backups, business continuity, restore data

Tags: backups, business continuity, restore data