In 2008, Security Benefit enhanced their business resiliency capability by establishing a hot site for immediate recovery of critical systems in the event of a disaster.  The IT department had, over the course of the prior 3 years, successfully established new technologies in the primary data center including enterprise-class storage and backup solutions, leading-edge server virtualization solutions and IP telephony.  These key technologies, along with the newest data replication solutions and affordable, high-bandwidth MPLS connectivity combined to enable a new remote solution that provides nearly instant access to updated critical systems in the event of a failure at the primary site.  The prior shared services disaster recovery arrangement was far less flexible at a higher cost, more difficult to document and maintain and featured a recovery time window of 48 hours.  

The primary goals of the project were to shrink the recovery-time window to less than 1 hour for our most critical systems while lowering the long-term cost of providing an enterprise-wide business continuity solution.  

Information Technology at Security Benefit is positioned as a shared service, centralized to serve all three of the company's lines of business - Asset Management, Retail Retirement Plans, and Business Process Outsourcing.  The former two lines serve direct customers of Security Benefit.  The later serves the customers of outside life and annuity providers by providing the technical platform and customer service operations to service their business as an outsourcer - Goldman Sacks (Commonwealth Annuities), Prudential, Aviva and Mutual of Omaha are a few of our major BPO  customers. 

The project has benefited the roughly 1 million account holders whose life, annuity and mutual fund contracts are stored and administered on our systems by further ensuring that these critical systems will not be impacted by disasters or outages that impact the availability of our primary data center.  A large percentage of these contracts feature equity-based investment vehicles that depend heavily on availability of trading systems leading up to and during daily trading deadlines.  By further ensuring against lengthy outages, the disaster recovery solution protects the companies being served from costly gain/loss exposure which can quickly add up to tens of millions of dollars per day should the market move adversely when trading is hindered by system outages.  It also provides a ready data source to utilize immediately in the event of more common outages due to database corruption or other system errors.  

The project has also saved Security Benefit money by providing a more cost-effective solution that requires far fewer resources to maintain and test.

Technology and our strong vendor partnerships were absolutely foundational and key to the success of our project.  We blended together "best-of-class" offerings in the following major categories to create our total solution:

1) Enterprise-class Storage (EMC) -
Critical to the success of the project due to the very large volumes of data housed for the critical systems including Image/Workflow, annuity administration, our client-centric database, our general ledger system.  Our Storage Area Network (SAN) and Network Attached Storage (NAS) solutions also enable the capacity to virtualize servers and exponentially cut costs at both the primary and recovery data center sites. 

2) Server virtualization (VmWare) - 
Critical to the success of the project due to the enablement of highly efficient use of hardware that is enabled by this key, ground-breaking technology. 

3)Data replication tools (EMC's RecoverPoint)
Critical to the success of the project due to the need to efficiently keep very large databases current to the second at the recovery site 170 miles away.  

4) IP(Internet Protocol) telephony (Cisco IPCC, Call Manager, Interactive Voice Response) -
Critical to the success of the project by allowing a highly recoverable, server-based telephony platform to ensure availability of our call centers that support tens of thousands of calls per day.  

5) High bandwidth, low cost WAN (wide area network) connectivity solutions (MPLS circuitry)
Critical to the success of the project due to the need to constantly and reliably move large amounts of data from the primary data center to the recovery site.

6) Collocation vendor (CoSentry) - We wanted and successfully found a regional provider that offered an enterprise-class, secure data center and that would allow us flexibility in the contract, both for hosted services and workgroup recovery capability.

Yes

Yes

1) Cost Savings - Over the first five years, the new solution will save Security Benefit nearly $1 million when compared to the shared service model of disaster recovery previously employed. 

2) Smaller recovery windows for most critical systems - The recovery window that was 48 hours as provided by the prior solution is now 1 hour or less.  

3) The new solution provides an additional backup source available immediately for non-disaster type outages like database corruptions.  

4) Limited gain/loss exposure - the smaller the recovery time window, the less exposure we have for our critical systems being down and missing daily trading deadlines. 

5) More flexibility in DR testing - we have access to the systems and data at our recovery site at all times.  
 
6) Less and easier to maintain disaster recovery documentation - Not having to create recovery systems from backup tapes and to unlike hardware greatly reduces the amount of documentation and expertise required to establish a fully usable set of recovered systems.  

The project has benefited the entire organization by exponentially lowering our risks due to varying degrees of system outages.  Also, analysis of our disaster recovery capability is a key component of due diligence conducted by prospective clients.  By enhancing our solutions, we have greatly enhanced the perception of these prospects of our business continuity capabilities, and increased our chances for new sales for both our BPO business and our institutional asset management business.

The most exceptional aspect of the project was the capacity and expertise displayed by our own IT Infrastructure team to become experts in all of the new, best-in-class technologies that were combined to create the total solution.

The project had to be proposed and approved for funding through our corporate Capital Expenditure committee.  Our corporate risk committee also weighed in on project approval, in addition to our CEO.  Clear cost/benefit had to be shown in order to gain approval for the project. 

The most challenging technical hurdles were getting all of the various data replication tools to work flawlessly and reliably.

At Security Benefit, a corporate pool of dollars for large capital projects is established each year.  Lines of business and corporate shared services vie for these dollars by proposing projects to our CapX (Capital Expenditure) committee.  The committee approves (or declines) projects and the use of capital based on cost and benefit.  We presented a project requiring $900,000 in funding and projected a payback period of 2.3 years.  This is a very favorable payback, but beyond the monetary benefits, what really sold the project was the exponential improvement that it creates in our business continuity/disaster recovery capabilities.

Exceeded

See Benefits section above.

One month after fully implementing our hot site capabilities, we did a full recovery test at the vendor site.  The results were outstanding, and as soon as these results were shared with our business leaders, they were 100% convinced of the value of our investment.  The project was delivered on time, on budget and met every expectation that had been established over the course of the project.