"eval(gzinflate(base64_decode("BLOB OF TEXT")));

to decode it and execute it on the web server. While it obscures what the code is doing (briefly), it fairly screams that something is not right with this file.

First to find any PHP files which use this on your server use:

egrep -r "eval\(gzinflate\(base64_de" . --include=*.php

That will find the offending files. To see what they did, I copied everything inside the “eval()” statement. Then I dumped into a text file on my laptop in something that looked like this.

<?php
$X = gzinflate(base64_decode("BLOB OF TEXT"));

print $x;
?>

Instead of displaying it in a browser, I called it from the command line.

> php foobar.txt

The code gets base64 decoded, decompressed and displayed in my console window. Because I removed “eval()” from the code I could see what the attacker was doing without worrying about executing bad code on my system. Or viewing it in my browser.

Technorati Tags: incident response, malicious php, obscured malicious code

Thankfully, scenes like this are not played out in our fire departments.  The men and women who become firefighters are trained, professional and prepared.  Their turnouts are always ready that they can grab them, climb into the engine and race to save someone’s life.  They don’t have to rummage around the fire engine checking for equipment, because when they got back from the last call they made sure they were ready for the next call.  And they did so in a way that the next call can be a fire, an accident or the proverbial cat stuck in the tree.

Fire fighters are the consummate “incident responders”.  They are able to drop what they are doing or wake up from a dead sleep, grab their gear and start saving lives.  Their preparation is such that they are ready to respond to a variety of issues. 

As computer incident responders, there are some things that we can learn from these professionals.  First, they are selected carefully and trained in the proper tools and techniques.  Next, they assemble the equipment and store it on a mobile “jump kit” known as a fire engine.  Last, after they use their fire engine, they restock it and prepare it for the next emergency.  So let’s look at the nearest equivalents in computer incident response.

Selection of Incident Handlers

Incident handlers need to be selected with serious consideration.  When an incident occurs the incident response team is called in to deal with the situation.  The business is willing to make this call because the incident handlers are the experts in this stuff.  They trust them.  The team is made up of people with solid technical backgrounds, they know what needs to be done, how it needs to be done and can guide everyone else through the process.

An incident is stressful on everyone around it.  People may fear for their jobs or careers.  They aren’t sure what to do or how to even begin.  It may be pure panic.  Incident response team members need to be able to handle their own stress and the stress of those around them.  They need to be calm and help calm down everyone else around them.  They must make good decisions under pressure.  Communication is a key skill that will be relied on heavily.  They will need a good understanding of the business and what’s important to it.  Plus they must know the procedures to handle evidence gathered in response properly so that the business is protected or has legal recourse.  Last, they must to have the technical knowledge and analytic capabilities to get the business out of the incident and back into regular operations.

This kind of expertise becomes a reality because people have made a deliberate choice to pursue and gain these skills.  They are not individuals who can be picked at random.  Choose carefully and deliberately.  Then support them in fulfilling the mission they’ve been assigned.

Equipment Preparation

Incident response requires some equipment on hand and ready to go.  It’s not a kit filled with terribly unusual or expensive items.  It involves deciding what needs to be in the kit and putting it together into something frequently referred to as the “Jump Kit”.  It’s an organized bag with all the basics needed to respond to a variety of incidents.  As you assemble it an iron clad rule must be followed.

With that rule out of the way, what should be in a jump kit?  NIST has created Special Publication 800-61 – Computer Security Incident Handling Guide to help government agencies prepare for their incident response capabilities.  It contains a number of recommendations for a jump kit.  SANS also has assembled a similar list of information, which is distributed to students of Security 504, Hacker Techniques, Exploits and Incident Handling.  Both are excellent sources of information.

The list below is a combination of ingredients from both sources.  I have some of these items but not all yet.  I’m still working on building my kit.

• Contact information for team members and others within and outside the organization (primary and backup contacts). If you’re depending on the online address list, but that server is compromised you may be in trouble.
• Pagers or cell phones. Don’t forget spare batteries and chargers
• Laptop with lots of RAM, a large hard drive and pre-loaded with forensics applications
• Virtual machine software such as Xen, VMware, or Virtual PC so you can run multiple operating systems
• Live CDs of various types. Some options are Helix, CAIN or DEFT for forensics. Backtrack is another possiblity, though it is focused on penetration testing
• DVDs or CDs with trusted, statically linked versions of programs to be used to gather evidence from systems
• Packet sniffers and protocol analyzers to capture and analyze network traffic
• Encryption software. If the network is not trustworthy, you need encryption to keep your communication confidential
• Blank media, such as floppy disks, CD-Rs, and DVD-Rs
• USB thumb drives. 8 and 16 GB drives aren’t too expensive
• A network hub or a tap. A tap would be better, but they cost more
• Write blocking device(s) to use when imaging hard drives or other media
• A large external hard drive. Maybe more than one.
• Tools such as screwdrivers, flashlight, tweezers, telescoping magnet or hands. I tend to bring a Gerber utility knife with me where ever I go. Watch out when flying some where though. Be prepared to check your kit if you carry a pocket knife or screwdrivers
• USB to serial port adapter
• Network cables. Straight and crossover. I carry a crossover adapter
• Hard drive jumpers
• Cisco rollover cable and a serial cable
• Female to female RJ45 connector
• Hard-bound notebooks with numbered pages
• Digital camera and audio recorder
• Chain of custody and other incident forms
• Evidence storage bags and tags, and evidence tape. Amazon.com has these items
• Desiccants for protecting against moisture in the bags
• Port lists, including commonly used ports and Trojan horse ports
• Cryptographic hashes of critical files to speed the analysis, verification, and eradication of incidents
• Media, including OS boot disks and CD-ROMs, OS media, and application media
• Security patches from OS and application vendors

Edit — 7/21/2010
One quick note to add to this list. Consider what items are practical to have a couple of. In my case the power switch on my write blocker failed when I was imaging a number of systems. The vendor, Digital Intelligence, responded in short order to replace my write blocker and was awesome. But in the mean time I was stuck. The lesson of redundancy was re-taught to me at that point. If duplicate equipment is not possible consider what your backup plan is if something dies on you.

– Continuing the original post…

If you are doing incident response internally…

• Documentation for operating systems, applications, protocols, and intrusion detection and anti-virus systems
• Network diagrams and lists of critical assets, such as Web, email, and database servers
• Baselines of expected network, system and application activity
• Backup images of OS, applications, and data stored on secondary media

You may decide that this list isn’t complete. It’s not. It’s just a starting place. There may be that one time when you really needed something and didn’t have it. So now you always make sure to have it with you. After each incident, review how things went and ask the team if there was some piece of equipment that wasn’t there. Adjust, improve, and then restock.

Jump Kit Restocking

After the incident is over several items from the kit may have been used and need to be replaced.  Compare your inventory of what the bag should have to what is left in it.  Note any missing or nearly depleted items.  The team may have noted some items that they wish that they had had or that needed to be purchased during the incident.  If appropriate, add them to the list of jump bag items and purchase it for next time.  Make sure that everything is stored properly, so that it can be used without needing to untangle it from something else first.  Once the bag is ready, put it in its normal place and have it ready for next time.  Your team is now ready to respond to the next incident with confidence that when they grab the bag that it is ready to go.

How Are We Doing Now?
The team has been selected, vetted and trained. Everyone knows where the jump kit is and that it has the gear necessary. When the phones start ringing and an incident is declared, the team starts gathering their gear and congregating as planned. Instead of scrambling for equipment and wondering what to do next, the team is ready, confident and it shows. The incident has a much better chance of being resolved quicker, more completely and according to accepted practices.

Technorati Tags: computer incident, incident response, preparation, security incident

The first thing I saw that this document was definitely written to meet regulatory requirements.  It has lots of flow charts, organizational charts, and contact information.  It details the contracts maintained with ships to perform oil skimming and position containment booms.  Planes to drop dispersants?  Check.  Detailed info about the dispersants themselves?  Yup, got that.  Do they have procedures to use when deciding how to perform surface burns?  Of course!  There is information on how to assist animal life of all kinds, including some that have probably never been in the Gulf of Mexico.  It even has a section on the “Worst Case Discharge”.  And on it goes.  So why are we still spilling 500,000 – 700,000 gallons (12,000 – 19,000 barrels) of oil per day in spite of this lovely plan?

The most glaring omission is that the plan has not a single mention of how to stop an active oil spill.  Not a peep about it, whether its on land or on the ocean floor.  There isn’t even a reference to other documents on how a blowout would be halted. It does mention practice exercises and ongoing training, but even that appears to be focused on mopping up oil on beaches and the ocean surface.  I suspect that this plays into why containment chambers, top kills, junk shots and other ideas are getting thrown around.  Question one.  Does your incident response plan include or at least reference other docs on how to stop the bleeding?

Next issue.  The “Worst Case Discharge” appendix contains an estimate of 250,000 barrels leaking per day using an official formula required by the US government.  It has a containment and clean up strategy using oil skimmers and dispersants.  Last it has a detailed inventory of ships and planes that will be used in the attempt.  The problem that I see with this is that it has little relation to reality.  Fine, its the worst possible amount of oil that can be discharged and fortunately, the current spill is not that big.  But what happens if the entire drilling rig blows up?  What do you do when the blowout preventer fails?  How will you handle it when hurricane season hits?  What if the failure is a mile underwater?  What happens if multiple wells blow out?  No mention of it.  Question two.  Does your incident response plan take into account business requirements?  Like end of the month billing and an incident?  Is your worst case scenario a dry formula or does it mean something to the business?

Last issue.  The BP response plan looks like it was written to keep auditors happy.  It looks like it has some really important information that I’d bet has been used during this process.  Contact numbers for service providers and contracts with critical damage control resources is important.  But I think this thing was written to meet a checklist.  It met the legal requirements currently in force and passed muster.  A document designed to keep an auditor happy is trouble in the making.  Instead how about writing one that tries to meet the need of the potential crisis?  It will probably have what the auditor wants and if there are a couple of hoops to add, so be it.  But write the thing to deal with the leak of 500,000 gallons of oil coming out of the ground.  Or is it credit cards leaking?  Compliance is great…  as long as it comes after the main issue is addressed.

Technorati Tags: incident response, lessons learned, planning for disaster