The Computerworld Honors Program
Honoring those who use Information Technology to benefit society
LOCATION:
Cincinnati, OH, US

YEAR:
2007

STATUS:
Laureate

CATEGORY:
Finance, Insurance and Real Estate

NOMINATING COMPANY:
EMC

ORGANIZATION:
Fifth Third Bank

PROJECT NAME:
Business continuity, backup, recovery, and archiving

Short Summary
As a leading financial institution in the United States, Fifth Third Bank has a significant impact on the nation’s economy, requiring a keen focus on protecting the financial interests of its customers. Faced with rapidly growing volumes of data and a tape-based solution that could no longer satisfy the bank’s objectives for information protection and recovery, Fifth Third Bank implemented a comprehensive business continuity, backup, recovery, and archiving solution built on EMC storage and software.

The solution is designed to ensure that all of the bank’s critical customer-facing applications can be recovered within two to four hours with zero data loss and perfect data consistency. As a result, Fifth Third’s customers can rest assured that their financial records are safe and secure even in the most extreme disaster scenarios.

To achieve this level of protection, Fifth Third implemented a multi-site business continuity strategy based on EMC Symmetrix DMX storage and EMC Symmetrix Remote Data Facility/Star (SRDF/Star) software. This solution provides continuous remote replication of Fifth Third’s production data, and also provides the ability to shift full production to one of its two disaster recovery sites while still maintaining continuous remote replication to a third site. In this way, the bank is able to not only protect critical applications and data during normal operation, but also continue to protect its assets when operating in disaster recovery mode.


Introductory Overview
Fifth Third Bank is one of the largest financial services companies in the United States, with more than 1,150 full-service banking centers across 10 states, and over $100.7 billion in assets under care. Ensuring the availability, integrity, and privacy of customer information in its growing and dynamic technology infrastructure is a priority at Fifth Third. That’s why the bank implemented a comprehensive business continuity, backup, recovery, and archiving solution.

Fifth Third processes many millions of dollars in banking transactions every day for both its consumer and business customers. Should the bank ever suffer a major data center outage or regional disaster, the impact would be far-reaching—and financially devastating. For example, in a disaster situation, consumers will have numerous issues to worry about, one of them being whether the bank still has an accurate record of their deposits. Fifth Third’s goal is to eliminate that worry from the consumer. In addition, if Fifth Third’s services are down, all of its business customers also suffer. The impact would not only be on Fifth Third’s revenue stream, but because of its size, on the economy as a whole. Finally, the banking industry is closely regulated and Fifth Third required an effective business continuity strategy to comply with government mandates for information protection and disaster recovery.

To protect the financial interests of its customers, meet regulatory requirements, and preserve its own reputation as a banking leader, Fifth Third implemented a multi-site business continuity strategy that can recover all of its critical customer-facing applications within two to four hours with zero data loss and perfect data consistency. This is a dramatic improvement from the 16 – 20 hours it previously took to recover using tape. The new solution is based on EMC Symmetrix DMX storage and EMC Symmetrix Remote Data Facility/Star (SRDF/Star) software. This cutting-edge technology provides continuous, concurrent synchronous and asynchronous data replication between Fifth Third’s production data center and two geographically separate recovery sites across a 370-mile region. Fifth Third also employs EMC Centera content-addressed storage to archive Exchange email, workflow files, reports, and statement information. This archiving solution provides a secure and easily accessible repository that protects older data from ever being changed or inadvertently deleted.

One of the first implementations of its kind, this three-way protection strategy provides continuous remote replication of Fifth Third’s production data, and also provides the ability to shift full production to one of its disaster recovery sites while still maintaining continuous remote replication to a third site. In this way, the bank is able to not only protect critical applications and data during normal operation, but also continue to protect its assets when operating in disaster recovery mode. In addition, the solution provides consolidated point-in-time recovery for both mainframe and open systems. Because of the numerous interdependencies among the bank’s applications, rapid and effective recovery is only possible if the data is consistent and synchronized across all systems. By automatically coordinating the cycling activities among its eight EMC storage systems, the replication software ensures that if Fifth Third ever had to revert to its disaster recovery site, all those storage arrays would be at the exact same point in time.

With this comprehensive business continuity solution, the impact of a disaster on Fifth Third’s customers, and the potential risk to its business, would be minimized.


Benefits
Has your project helped those it was designed to help?   Yes

What new advantage or opportunity does your project provide to people?
The principal advantage this solution provides to Fifth Third’s customers is confidence. Retail customers have increased confidence that their financial records will be preserved even in the most extreme disaster scenarios. And business customers are assured that they can keep their own businesses up and running because the financial services they rely on from Fifth Third are always available with little to no disruption. In addition, the bank’s business customers can promote the added value they bring to their own customers by virtue of the data protection Fifth Third provides for their financial information.

Fifth Third’s business continuity solution also provides the bank with enhanced opportunities to market its services. For example, many businesses will choose a bank based on the assurances it can provide that their financial interests are well protected. Fifth Third can now demonstrate a higher level of data protection.

In addition, the bank’s disaster recovery team can perform its tasks more quickly and smoothly. Recovery processes that once took as long as 20 hours and required extensive manual procedures can now be achieved in two to four hours with greater assurance of a successful, complete recovery.

Fifth Third’s application management group also gains the advantage of performing application tests using more current and relevant data. In the past, test data was based on tape backups that might be two or three weeks old.


Has your project fundamentally changed how tasks are performed?   Yes

How do you see your project's innovation benefiting other applications, organizations, or global communities?
As an early adopter of three-way replication, Fifth Third has worked through a number of challenges to optimize the solution to meet its specific business objectives. As such, the bank has contributed significant input into technology refinements that other banks and enterprises can now deploy with greater ease.

In this process, the bank had to work extensively with its technology partner, EMC, to refine architectural designs and functional specifications. For example, initially, the cache configurations to support asynchronous replication were not optimally sized. Today, those configurations have been revised, enabling other organizations to implement the solution more quickly and effectively. Another example pertains to monitoring and reporting that were not originally available. Based on Fifth Third’s implementation, new features are now available in the replication technology that enable users to better understand their environments, identify and resolve issues, and gain greater value over the long term.

As a result of this effort, Fifth Third now provides a solid model that other organizations can use as a foundation for their own solutions. In addition, because of the modifications made in the technology, organizations can realize a faster implementation and accelerated time to benefits.


The Importance of Technology
How did the technology you used contribute to this project and why was it important?
For Fifth Third, finding the right technology at the right time was critical to ensuring business continuity and complying with government regulations. The bank’s previous strategy of backing up to tape and physically transporting the tapes to its remote disaster recovery site was no longer feasible. The disaster recovery team spent 16 to 20 hours to recover core banking applications from tape, and this still left the bank without any historical data. In addition, the risk of losing tapes was simply no longer acceptable. Fifth Third needed a way to speed recovery time and minimize any risk of data loss.

After conducting a thorough evaluation, the bank determined that asynchronous replication was the right technology to provide long-distance protection, and found EMC to be at the forefront of this technology. By implementing asynchronous replication between its primary data center and a remote disaster recovery site 370 miles away, Fifth Third was able to reduce recovery time for its most critical applications and data from 20 hours to just two hours. Yet, even with this advanced technology, there could still remain a gap of 30 seconds in which the latest transactions would not be captured. Since even 30 seconds of transactions could represent millions of dollars in business value to the bank, Fifth Third elected to also implement synchronous replication between its production data center and another site 27 miles away. With the implementation of synchronous replication, the bank will further reduce the recovery point to zero between its primary and secondary locations, closing the exposure gap and preserving business transactions.

Even with two disaster recovery sites, the bank was concerned that it wouldn’t be able to continue protecting its customers’ vital information assets when operating in disaster recovery mode. Therefore, Fifth Third implemented another asynchronous link between its two recovery sites. With this three-way replication configuration, enabled by EMC SRDF/Star software, should Fifth Third ever lose its production data center, it will be able to failover operations to either the nearby site, and immediately resume asynchronous replication from there to its remote site; or failover to the remote site and replicate asynchronously back to the nearby site.

Another critical technical requirement was to synchronize data between mainframe and open systems. Since financial data resides on multiple platforms, and there are so many interdependencies among the applications, the bank had to ensure that it could recover a complete and consistent picture of that financial data. To achieve data consistence across multiple platforms and databases, Fifth Third employed the Multi-Session Consistency feature in EMC SRDF/Asynchronous software. By automatically coordinating the cycling activities among the various banking systems, this essential technology ensures that if Fifth Third had to fail over to its disaster recovery site, all data would be at the exact same point in time.

By taking advantage of the most advanced technology available at the time, Fifth Third has created a business continuity solution that provides the utmost assurance of effective protection for customer assets and the bank’s reputation. Today, if anything ever happened to Fifth Third’s main data center, the bank could have its critical customer-facing applications back up and running in two to four hours with zero data loss and perfect data consistency, while continuing to provide remote replication protection using the remaining two sites. Any impact on customers, or potential risk to its business, would be minimal, enabling Fifth Third to keep its commitment to top-notch customer service and aggressively grow its business.


Originality
What are the exceptional aspects of your project?
It is highly unusual for a financial institution to take a pioneering role in new technology, yet that is exactly what Fifth Third Bank has done. While the three-way replication is one of the first of its kind and is setting a benchmark that other banks are now beginning to follow, just as exceptional was the foresight and commitment by the bank’s senior management to fund such a project. Asynchronous replication was relatively new in 2004 when the project was initiated, and the concept of multi-site replication was still in a formative stage of development. Yet, based on the promise of absolute protection of its critical business applications and data—primarily customer-facing applications—Fifth Third was willing to make the investment.

The fact is the bank could have chosen a simpler solution. Government regulations did not specify such a high level of protection. It would have been possible to apply a “quick fix” and wait for technology to further mature. Nevertheless, Fifth Third was willing to take a long-term view and choose an approach that would provide maximum benefit to its customers and its business. This required not only a financial commitment, but also a commitment to persevere while navigating the challenges of implementing new technology. In the end, Fifth Third made a wise investment, resulting in a business continuity solution that will return value for many years to come.


How is it original?
Fifth Third Bank was one of the first banks in the United States to implement a multi-site business continuity solution involving continuous, concurrent synchronous and asynchronous replication. As an early adopter of asynchronous technology and a trailblazer in combining asynchronous and synchronous technologies into a comprehensive solution, Fifth Third has established a model that other enterprises in many other industries are following.

The unique aspects of Fifth Third’s solution is how the bank has gone beyond failover from a production facility to a disaster recovery site to also enable continued protection even after the failover has occurred. In addition, by configuring the three-way replication as it has, the bank gains tremendous flexibility to failover to either of its disaster recovery sites, providing protection against a wide range of disaster scenarios.

Finally, Fifth Third had a unique need to ensure consistency of data residing on multiple databases and storage platforms across its disaster recovery sites. Coordinating point-in-time recovery between both mainframe and open systems was a particular challenge; but by effectively employing the Multi-Session Consistency feature of EMC SRDF/Asynchronous technology, the bank was able to synchronize the cycling activities among its various storage systems. This ensures that all applications and data would be at the exact same point in time if a recovery were necessary.


Is it the first, the only, the best or the most effective application of its kind?   Most effective

Success
Has your project achieved or exceeded its goals?   Achieved

Is it fully operational?   No

How many people benefit from it?   100,000+

If possible, include an example of how the project has benefited a specific individual, enterprise or organization. Please include personal quotes from individuals who have directly benefited from your work.
This solution has dramatically simplified tasks for the bank’s disaster recovery team. When the bank still relied on tape backups, recovery could take as long as 20 hours, requiring staff to work non-stop just to recover a baseline of application data. No historical data would be available. The effort and stress of performing a recovery was intense. But today, EMC replication technology makes the process much easier, enabling recovery in as little as two hours, including full historical data. As a result, the disaster recovery team has gained valuable time to focus on additional refinements and improvements to the recovery scenario.

Fifth Third’s archiving solution also has had a significant impact on the bank’s business and financial analysts. In the past, historical reports and statements were stored on tape, so when an analyst required access, the request had to come through the IT organization. Meanwhile, the business user had to wait for the tape to be mounted and the appropriate file located—a process that could take up to one week. Today, by archiving reports and statements on EMC Centera content-addressed storage, business users can access archived information in seconds, while the bank can ensure the files are secure and protected from ever being changed or inadvertently deleted. Productivity of the business users has increased dramatically. And the IT staff can focus its time on much more important tasks.


How quickly has your targeted audience of users embraced your innovation? Or, how rapidly do you predict they will?
Fifth Third’s disaster recovery team embraced the new solution almost immediately. The value to them was direct and dramatic. The bank’s application team, however, has been slower to adopt the new protection strategy.

Each application team at Fifth Third is responsible for its own protection and recovery. Tape backup has been the traditional solution and, because it is well understood, some application managers have been resistant to change. Most tape-based recovery strategies, however, are based on best-case scenarios. As Fifth Third educates its application teams about the new solution and the potential risks of tape under worst-case scenarios, the bank expects additional application managers to embrace the replication strategy.

With continued efforts over a 12 – 18 month period, the majority of application teams are expected to adopt Fifth Third’s multi-site business continuity solution.


Difficulty
What were the most important obstacles that had to be overcome in order for your work to be successful? Technical problems? Resources? Expertise? Organizational problems?
Like most early adopters, Fifth Third had technical challenges to overcome. For example, cache configurations had to be re-sized to support asynchronous replication for Fifth Third’s particular application requirements. In addition, early on in the implementation, the replication technology was not fully utilizing available bandwidth. This situation required working with the bank’s technology partner, EMC, to modify the microcode until the software was fully optimized.

Fifth Third recognizes that getting the full potential out of highly sophisticated technology requires effort. By working closely with EMC and providing detailed feedback during implementation, the bank now has a solid business continuity solution that enables it to meet its regulatory requirements and recover data quickly and consistently to protect customer assets.


Often the most innovative projects encounter the greatest resistance when they are originally proposed. If you had to fight for approval or funding, please provide a summary of the objections you faced and how you overcame them.
Fifth Third’s IT organization did not encounter any resistance from senior management. In fact, management encouraged the IT organization to pursue a more advanced protection strategy than originally proposed, pushing beyond the basic regulatory requirements. Because availability of customer data was so important to Fifth Third, senior management was willing to invest in a more comprehensive solution that would support the bank’s needs over the long term.
Digital/Visual Materials
The Program welcomes nominees to submit digital and visual images with their Case Study. We are currently only accepting .gif, .jpg and .xls files that are 1MB or smaller. The submission of these materials is not required; however, please note that a maximum of three files will be accepted per nominee. These files will be added to the end of your Case Study and will be labeled as "Appendix 1", "Appendix 2" or "Appendix 3." Finally, feel free to reference these images in the text of your Case Study by specifically referring to them as "Appendix 1", "Appendix 2" or "Appendix 3."

Currently Uploaded Appendices:
No appendices currently uploaded.