Savvy Storage Strategies

WHEN IS THE CONTENT of an e-mail document important to government business? And when is it merely a routine communication? Only the sender and recipient know for sure. But that gray area has created an enormous dumping ground for unnecessary data in state and local records management systems.

In Texas, retention periods for records have been in place since the late 1970s, says Tim Nolan, program planning and research specialist for the Texas State Library and Archives Commission, State and Local Records Management Division, in Austin. The state requires storage of all state agency administrative correspondence, including e-mail, for three years. Routine e-mail correspondence transmitted via state computers must be kept for one year. Short-term temporary correspondence, such as an e-mail to “Meet me for lunch at 11:30,” needs to be kept only until the document has served its purpose—“which is 11:30 that day,” Nolan says.

During the past 10 years, e-mail and other electronic files, such as scanned documents, have propagated and clogged records management systems, Nolan says. In most cases, users don’t know how to classify e-mails or can’t spare the time to weed out the spam, he explains. As a result, information technology staff members have to back up and store thousands of e-mails for long periods of time.

Erring on the side of caution when determining which records should be retained helps prevent accidental deletion of valuable data. But it can wreak havoc with state and local agencies’ IT budgets and staff.

There are alternatives, however, as three IT managers have learned. The three are in different departments and different cities; they’re storing different kinds of data; and the solutions they chose were different. Only their underlying problem was the same.

Drowning in Data

College Station, an east Texas city located about 100 miles northwest of Houston, probably doubles its storage requirements every year, according to Neil Black, network analyst. “We just can’t get rid of a lot of the stuff we have,” he says.

Black continually encourages College Station’s 800 government employees to delete files that are not required by public record regulations.

“In our job, we cannot determine e-mails to be not important, so we have to back up everything—e-mails, Word documents, group directories, sales spreadsheets, even scanned documents,” Black explains. The tally came to half a terabyte (TB) of data each night, he says.

The turning point occurred last August. The piles of data that needed backing up had grown into mountains, and the department was running 10 relatively low-capacity backup tapes on 10-megabytes-per-second (M”Bps) drives in a nightly marathon.

As backup time escalated, so did the risk of errors, Black says. “The possibility of tape failure is much higher when you have 10 tapes as opposed to fewer tapes that have greater capacity,” he points out.

Rather than incrementally adding more storage space—which is little better than slapping a bandage on a fractured leg—Black took a proactive approach to meet the city’s long-term data storage needs. He chose Sony super advanced intelligent tape (SAIT) drives with 500 gigabytes (GB) of native storage space and 1.3TB of compressed storage space, and supported the drives with a tape library.

With the combined technology, the new system could support College Station’s storage needs for up to four years.

By late November 2004, Black had installed two SAIT tape drives that share backup duties for 13 Novell servers each night. Eventually, the library will expand to six drives as College Station’s storage needs grow.

Backup time remains an issue because the city’s entire IT infrastructure isn’t capable of supporting gigabit speeds. But upgrades are currently under way. SAIT drives can back up at 30MBps, well beyond the speed of Black’s old tape drives. Once College Station upgrades its infrastructure, Black can triple the backup speed.

One of the challenges of deploying the new technology was software incompatibility. As of February 2005, some of the automated backup features in its software didn’t work with the tape library because drivers hadn’t been written yet.

“Right now, it’s not affecting us too much,” Black says. “We’re only running the two tapes a night without having to switch tapes, so we don’t have to use the automated feature in the library just yet. But it will be coming.”

Locking Up Police Video

In nearby Bryan, Texas, the IT staff faces a different storage challenge: police video. Video taken from police vehicles to chronicle traffic stops is stored on VHS tape. The IT department stores thousands of tapes in a room rimmed by shelves that stretch from floor to ceiling; two full-time employees manage tape storage.

But all that is about to change. The department is finalizing plans to move to digital video that will be stored electronically. Although the solution relieves the physical storage issues, it creates huge IT storage challenges.

“We’ll need about a terabyte of storage for every five police vehicles,” says Cory Bluhm, Bryan’s network specialist. “We run 15 cars a day—so that’s 2.5 terabytes to do all of it.

“We’re going to try to put in a terabyte of serial ATA [Advanced Technology Attachment] storage in our storage area network [SAN] from EMC and keep the video there. Each year, we’ll plan [to add funds to] the budget to increase the disk space based on the usage of the year before.”

Adding low-cost ATA drivers to the SAN can significantly reduce backup. It also allows an organization to keep mission-critical information online and near at hand, which is especially important when dealing with police video.

Estimating storage space needs can be difficult, Bluhm points out. Although most police video can be erased after 90 days, video pertaining to a pending court case, such as a charge of driving under the influence, is kept indefinitely. “Even after the case has been closed,” he says, “our city’s stance is that we don’t get rid of it [because] they might appeal.”

But such caution has consequences. “That’s an ever-growing number of files,” Bluhm says. “You couldn’t throw enough disk space at that.”

Instead, he approached the police chief with a suggestion for a life-cycle management approach: Store files for five years, then transfer them to tape for offsite storage. “It’s cheaper storage without eating up readily available disk space,” he explains.

If the plan is approved, Bluhm expects to add the first five digital video systems and accompanying storage in the next budget cycle, which begins Oct. 1, 2005.

Disk-to-Disk Backup

Randy Nielsen, public safety IS analyst in Fort Smith, Ark., faced similar storage challenges. Nielsen opted for a slightly different solution, selecting disk-to-disk backup and storage. He was won over by the technology’s speed, capacity and compatibility with current software.

The Fort Smith Police Department has about 225 officers and civilians who compile e-mail, case records and database information on nine servers, each of which holds 146GB of storage. Backups take between four and six hours on each local internal DAT (digital audio tape).

At the Fort Smith Information and Technology Systems (ITS) department, “we keep all our records indefinitely,” Nielsen says. “We have to expunge juvenile records when the [juveniles] reach 18, if they committed a misdemeanor. If they [committed] felonies, we keep records for 10 years. Other than that, we’ve got every record we put into storage since Jan. 1, 2000.”

Storing data at that rate, the department was quickly outgrowing its tape drives, so Nielsen began his search for a disk storage system that would appear as a tape drive to the backup software. He chose a disk-to-disk storage system.

“The neat thing about the system is that it actually emulates a tape drive,” Nielsen says. The disk storage system works with existing backup applications that were designed to support tape devices, providing a single backup catalog and a single media management system.

“We can take our current backup software and make it look like a tape drive,” Nielsen says. “It’s more efficient because it writes all [the data] serially, and you’re not constantly having to manage your network-attached storage.”

The system also has built-in replication. “We bought two of these units separated by a T-1 line,” he says. The second system is going in the U.S. Attorney’s office, a block away, in a secure facility. “We’re going to do our backups here and replicate throughout the day to this other unit,” Nielsen adds.

The arrangement also lets the ITS department fulfill its offsite storage requirements. “Our policy requires us to have offsite storage,” he says. “We had been pulling tapes out once a week and carrying them across the street, which wasn’t efficient.”

Fort Smith’s new system will likely go live this spring, Nielsen predicts. With a capacity of up to 2TB, the disk-to-disk system will position the department to meet its storage management needs for many years.

Realistically, no storage solution is final. “I’m hoping it’s going to last us maybe 10 years,” Nielsen says.

Ten years of breathing space can be a lot. Just ask your data storage manager.