Iowa and Oregon Put Defenses in Place to Respond to Security Breaches

In September 2014, Stefan Richards began the job of chief information security officer for Oregon. The honeymoon was short: One month later, hackers breached a state Employment Department web application and made off with nearly 900,000 Social Security numbers.

IT is decentralized in Oregon, but due to the size of the breach, Richards stepped in to lead the response. “When we got there, we realized we needed a lot more people than we had, and we needed them quickly,” he says. Because it would have taken too long to hire new em­ployees or bring in contractors, he tapped experts throughout state agencies to assist with incident response. “It was exciting to see it actually work on the fly, because if it hadn’t, we would have really been in trouble.”

There’s simply no way to guarantee immunity from today’s sophisticated cyberthreats, so security professionals spend as much time monitoring and responding to incidents as they do guarding against them. Those who have thwarted attacks have the gift of hindsight as they formulate strategies to avoid or minimize damage from future breaches.

“Hackers are always at least on par with, if not ahead of, the best prevention out there,” says Richards. “I’d better be looking closely at what’s going on in my networks and systems, because all the protections in the world aren’t going to stop determined hackers. The best chance I’ve got in targeted acts is to watch carefully and act quickly to shut them down.”

For many, the first step after an attack is to take the infected system offline, but the Employment Department website provided a critical service for Oregonians. “The cost of losing the service had to be balanced with the cost of potentially losing data due to additional vulnerabilities,” Richards explains. After mitigating the problem, his agency brought systems back up while continuing the investigation.

After the event, Oregon created a security incident response team of employees with a range of expertise, such as database or mainframe skills. Staff will receive training and put those skills to use at their state agencies while waiting to respond to incidents. “I don’t have to staff a large department that sits waiting for something to happen,” says Richards, “but when something does happen, I can pull them back together.”

Receiving a Malicious Message

One Monday last January, a red dialog box popped up on the computer of the executive assistant to the mayor of Hillsboro, Ohio. CryptoLocker ransomware had encrypted all of her files, and its operators were demanding $250 to release them.

She told Systems Administrator Eric Daniels about the message, and he immediately took her computer offline.

Daniels determined that the virus had likely come from an infected banner ad. Based on the volume of data encrypted (130 gigabytes), it had breached the network a few days earlier. “It could have done its deed over the weekend without making any impact on the network to the point where we’d notice it,” he says.

The consequences were high. “It encrypted the entire brain trust of our city administration — everything,” says Daniels. “I cannot tell you how bad this would have been.”

But it wasn’t bad at all. The city recovered all the data, and the cost was less than $100 to replace the hard drive on the compromised computer. “I’m kind of pounding my chest on this one because my system worked,” Daniels says.

CryptoLocker crushed two and a half of Hillsboro’s backup tiers, but the remainder saved Daniels. His third tier was offsite backup, which had two copies of the affected files: one encrypted and one unencrypted. Daniels overwrites his entire onsite disc-to-disc backup daily, but doesn’t replicate deletions on his offsite backup.

“I’m a paranoid type. Contentious terminations can lead to data deletions,” Daniels explains. “With my replication scheme, they can delete all their data; I’ve still got a copy. I call it my sponge.”

He could have reverted to the offsite backup and had the user delete the encrypted files, but with 130 GB of infected data, that would have been a big productivity loss.

Then Daniels remembered the fourth tier of his backup — Windows Shadow Copy, a service included with Microsoft Windows Server. It makes shadow copies of data so users can go back to versions at different points in time. He used it to restore to the Friday version of the data before the virus had hit.

“I was skeptical, given the large amount of data,” Daniels says. “That’s a large task for a service to restore without error, but it never erred once.”

Daniels always enables Shadow Copy when setting up a new server. If every organization followed suit, he says, “no ransoms would have to be paid. It’s as simple as that.”

Reducing Vulnerabilities in Iowa

Jeff Franklin spends a fair amount of time figuring out where to focus his attention. “On a daily basis, our intrusion detection system alerts on roughly 200,000 different events and conditions,” says Franklin, CISO for the state of Iowa. “Our job is to filter through that noise and determine what the significant events are that we need to follow up on.”

Fine-tuning those filters and prevention efforts have paid off. One Iowa agency has reduced vulnerabilities by 80 percent within a year, and the state as a whole has seen a 50 percent drop in vulnerabilities, according to Franklin.

In addition to monitoring, Franklin makes sure he’s prepared in case malware or ransomware permeates the network. Like Richards, one of his top priorities is to have experts available to handle a crisis. The state has a cyber­security incident response team at the ready that includes public safety officers, FBI agents, governor’s aides, public information officers, attorneys and other critical personnel.

“Just knowing the players in other organizations is huge,” says Franklin. “I have no authority over law enforcement personnel, but what I do have is a relationship with somebody who does, so they can get the ball rolling versus us starting the conversation with, ‘Who do I talk to, and what do I need them to do?’ ”

Another step states and localities can take ahead of time to minimize the damage from an attack is to secure the authority to disconnect from the Internet or pull an agency’s connection.

“You might need to isolate systems to do recovery, forensics and figure out what’s happened and what the impact is, or to simply protect your data from further loss,” Franklin says.