Data center move designed to improve customer experience 

HM Revenue & Customs (HMRC) has an IT modernisation program to underpin customer service and efficiency improvements for UK tax collection, as well as supporting families and workers through benefits and credits. Some of their data centers were built over 20 years ago and are no longer cost effective to maintain so HMRC is implementing a transformation strategy that includes moving systems into more resilient state-of-the-art facilities. The program started with the rehousing and technical upgrade of tax credits systems that were particularly constrained by their environment. 
HMRC needed to ensure the move to a new data center did not create risk to the tax credit payments that over six million families and 10 million children rely on for day-to-day living and caring for children. As Tax Credits link to more than 75 percent of HMRCs systems, including National Insurance and Pay-As-You-Earn, the data centre move was one of the most complex ever in the UK, which had the potential to impact over 80,000 HMRC employees and millions of customers if the migration was unsuccessful.
Methods Improve HMRC Services 

Customers who contact HMRC expect to speak to staff who are well informed about Tax Credits and how the targeted financial support applies to their circumstances. The old data centre was an unstable environment and during periods of planned maintenance and extended downtime, staff could not see all the relevant information to manage queries, so they frequently had to ask customers to call back. If callers were providing information about a change of circumstance or new requirements, HMRC staff had to take handwritten notes and type them up later. This impacted customer service, staff morale and workload. 

The new secure and resilient infrastructure has been instrumental in transforming the service, ensuring HMRC tax specialists and operators have the available information on hand. Behind the scene, the technology and processes that made this possible were clever as well as complex, ensuring the data centre move was invisible to the general public and there was no disruption to HMRC staff.    

 
Benefits

Business critical Tax Credits services and other closely linked systems were taken out of the building they had been in since they were launched and migrated to new hardware in a new location. 

As a result:
	Tax Credits services for more than six million families now run on secure, resilient state of the art infrastructure
	Reduced carbon footprint supports HMRC sustainability targets 
	Lights out facility automates manual operations, reducing risk of human error
	HMRC has completed the first stage of their data centre transformation, providing a tried and tested blueprint for future migrations.

The technology that contributed to the success of the project included:

	Faster, cooler and less costly server products enabled server consolidation. Considerable savings in footprint, power and cooling were achieved by reducing the number of servers by 16% and deploying 37% fewer CPUs

	Automated testing tools and reusable regression packs to test volumetrics proved the design assumptions about consolidation and replacement of older servers were valid, and that desired Service Levels could be achieved for transaction response and batch run times

	The new data centre provides multiple active power and cooling distribution paths and has redundant components. The technology is concurrently maintainable, if one supply or part fails, another will take over to prevent service interruption and guarantee 99.9%+ availability 

Yes

Yes

The project was designed to help Tax Credits, one of the highest profile benefits and credits services in the UK, and has changed the way tasks are performed:
	People on low incomes
The new data center allows taxpayers with Tax Credits questions, such as new claims or verification of payments, to speak to staff in HMRC contact centers who can provide answers seven days a week 

	HMRC executive board, tax specialists and front-end staff
The systems to administer and process Tax Credits data are always available, supporting HMRCs vision to improve customer service, efficiency, and professionalism. 

	HMRCs Information Management Service (IMS) staff
The new data center allows HMRCs IT department to focus on implementing their strategy and improving services for the organisation, without the distraction of managing user expectations. Regular power downs and loss of service had a negative effect on the reputation of IT in the organisation

	HMRCs Chief Information Officer and Head of Live Services
HMRCs data centers are outdated and house complex interdependent, business critical systems, the risks of a data centre migration had been avoided until it simply could not be put off. Moving the Tax Credits systems to a new data center proved migration could be achieved without disruption to service and provides a tried and tested blueprint for the CIOs data centre transformation.

	HMRCs IT service partners
For Capgemini and their infrastructure partner, Fujitsu, implementing the new technology to replace end of life and unsupported equipment is essential to being able to provide efficient, robust and secure IT services, allowing them to drive down defects and improve service levels

Included in the above.

At first glance, relocating a set of inter-related critical business services from an aging data centre with power and cooling limitations, to a modern data centre, appears complex and challenging, but not exceptional. This migration was different though. It required an original and unique approach due to the age of the old data center, the critical nature of the application and the dependencies with other services:

	The data center move was a first for HMRC. Problems with moving the Tax Credits system could have a knock on effect to other systems and result in loss of revenue to the UK economy. As a result, the data center had been largely untouched. Over time the number of systems it housed grew, and the complexity associated with any change increased. 

Previous data center migrations had focused on specific areas, such as Data Warehousing, not systems that had a variety of processing styles. The Tax Credits migration included legacy client/server CA Gen/Tuxedo/Oracle systems, J2EE/Weblogic/Oracle systems and an EAI service based around an Enterprise Service Bus based on Software AGs web methods products. 

	Implementation period reduced from two weeks to two days. New approaches to manage the change, including new operational procedures for secure transfer of data, were planned, rehearsed and tuned so the original program was redesigned to complete the move in a single weekend.

The transition of data from the old site to a new location 400 miles away was a major logistical exercise and the culmination of a nine month project. Because of the complex interfaces with other services, there was no regression path once live processing started at the new data centre which made the two day transition even more remarkable. 

	HMRCs collaborative IT services outsourcing contract. With absolutely no room for error, agreement to the successful two day transition was, in large part, a tribute to trust between the many different stakeholders. The achievement reflected commitment to the program across HMRC and their partners. The program included HMRCs Information Systems Management department; business stakeholders, from the Tax Credits chief operating officer to contact centre staff; IT services partner Capgemini, their material subcontractor, Fujitsu and many specialist Ecosystem partners. 

There were significant difficulties that needed careful management to ensure success

	Complexity of system interfaces: Over 300 Tax Credits system interfaces needed to be reconfigured. There were another 60 interface changes between Tax Credits and other HMRC services; with other government departments such as Department of Work & Pensions; and with external bodies like the Bank of England. 

For transactional and file transfer interfaces it was simple. However, there were a number of fairly complex interfaces such as Oracle Streams, which have to be turned off and on with great precision if data corruption is to be avoided.  A
complex sub-project, specific to the cutover weekend, moved the hub that processed around half of the publish and subscribe Enterprise Service Bus messages from the old data center to the new facility. The messages had to be drained from the old hub, management databases and log files and copied across for a controlled start-up. This was a big manual exercise, open to human error, as the identification and coding of the point of many interface and file changes between HMRC systems, third party systems and the old and new servers could not be automated. To mitigate multiple subscriptions to the same message, extensive rehearsal of different scenarios was essential.

	Impact of storage considerations on service levels. Changing storage configuration, from a mixture of Storage Area Networks (SANs) of varying vintages, to a single SAN frame, required thorough performance testing to ensure service levels could be maintained. The new configuration was designed to improve throughput and reduce raw storage volumes.  

	New network security requirements were identified the day before the migration started, caused by an unexpected and intermittent software defect which had begun to affect a set of recently installed firewalls. The new firewalls and encryption equipment were essential to future network security, but at this point posed an immediate threat of unexpected outcomes. A decision was made to modify network configuration settings on the affected servers to ensure the new requirements for communication protocols and permitted ports would not cause timeouts for apparently inactive sessions.

	There was no regression path once live processing started. After the migration, data processing needed to re-commence at the same point at which it had been frozen prior to the move. There was no margin for error. The data centre move program team had already worked through different scenarios with key business stakeholders from HMRCs tax credits chief operating officer to contact centre staff. Following the migration, application experts were again needed to test and identify any issues and resolve them with the program team.

 
	Reducing complexity to a series of simple tasks to overcome obstacles
	Testing in all levels of infrastructure and against all parts of the implementation process
	Planning, from broad phases to minute-by-minute task driven implementation 
	Active engagement of all stakeholder communities, from application end users to business managers, IT partners and key Government officials 

Included in the above.

Exceeded

The project achieved all its implementation goals, and exceeded many of them. The new data center was fully operational and available for live service ahead of the planned schedule, no processes were disrupted and performance has improved. The secure and resilient infrastructure has ensured Tax Credits systems are always available and no longer subject to scheduled downtime.  

HMRC and Capgemini evaluate their projects using an index with a maximum value of 5.0. The data centre migration achieved a rating of 4.9. Having completed the first stage of their data centre transformation, HMRC now has a tried and tested blueprint for future migrations.

The move is one of the most recent examples of how the outsourcing agreement between HMRC and Capgemini focuses on shared outcomes and aligns their aims. For HMRC the state of the art technology provides a secure and resilient infrastructure for business critical systems. For Capgemini, implementing the new technology to replace end of life and unsupported equipment is essential to being able to provide efficient, robust and secure IT services. 

There are many other examples of infrastructure innovation that have been delivered through the HMRC / Capgemini partnership. Some of the most significant achievements include implementing the largest desktop upgrade in Europe to create economies of scale, and a single finance and accounting package to replace hundreds of existing systems to drive efficiency. The data centre migration continues this successful track record, strengthens the relationship between HMRC, Capgemini and Fujitsu, and provides an ever more solid base for future achievements.   

Included in the above