BTs automation project arose from a dilemma. IT departments exist in a world of rampant complexity. Customers and markets change in the blink of an eye. Innovation is more important than ever. But technology used to power business innovation has become so complex that its difficult to manage efficiently. In facing this issue BT turned the spotlight to automation. They figured that if automotive and heavy equipment manufacturers had successfully used software to smart-wire products and eliminate wearyingly repeatable tasks from engineering processes, IT could apply similar automation to workflow management performed by humans.

On this backdrop BT targeted three key objectives: 1) Increased efficiency: One hundred of BTs 150 infrastructure staff are dedicated to support and maintenance functions. Reducing the number of manual tasks and the time required to complete them was crucial. 2) Improved quality: BT wanted to reduce errors and improve IT service quality. 3) Compliance: BT wanted to ensure continuos compliance and prevent audit and compliance incidents.    
BTs project has significantly re-engineered and automated two heavily manual and error-prone processes: 1) New hire on-boarding; and 2) Ongoing development and testing of BTs investment portfolio platform, called Wrap  a web-based application supporting BTs advisors. Broadly, the impacts have released a team of BT engineers and administrators to do higher value work; additional speed has minimised bottlenecks from testing and development; systemisation has eliminated human error; and job satisfaction and work-life balanced have been significantly improved with higher value work, less repetition of mundane tasks, and weekend and overtime work. 

Prior to this project, new hire on-boarding and user account creation was an onerous process involving the security administration team in a 46-step seven-minute process. Managing bulk user migrations, typically involving the weekly creation of 50 new user accounts, was a days work, frequently necessitating weekend and evening work. 

Certain that automation was the answer to accelerating these processes and unhitching tedious task repetition, BT implemented BMC Atrium Orchestrator  a software tool that uses pre-built workflows to automate routine operational procedures and scripts across IT functions and tools. The results are startling. What once took seven minutes was completed in just 10-15 seconds. Managing previously manual workflow as a scheduled task, which can run overnight, or on demand, means a bulk user migration (n=50) is completed in roughly the same time it once took to create a just one account manually.

BMC Atrium Orchestrator has also helped BT to slash time and effort from ongoing development and testing of its investment portfolio platform, called Wrap  a web-based application supporting BTs advisors. The critical application requires pre-production testing to minimise customer impacts. However, creating and refreshing testing environments was laborious, involving a number of steps  things like uninstalling production certificates, installing testing environment security certificates and checking the appropriateness of security in acceptance environments. The process was manual, requiring someone to sit down and run scripts, connect manual processes, and constantly watch for one task to complete before initiating the next. The whole process took the best part of one week in every month, including up to 20 hours of dedicated engineering support. Over the year BT refreshes five pre-production and testing environments 10-12 times. Using BMC Orchestrator the Wrap system refresh takes just 10 minutes.  

Based on internal charge-out rates applied to time savings, in one year BT has shaved $80,000 from the Warp application refreshes, alone. In three years it will save $250,000. Combined with savings from other process automation, in one year BT is on track to save $180,000 and $588,000 over three years.

Craig Wiseman, Manager Environment and Release Management, BT Financial Group, works in BTs 600-plus IT organisation and is a key figure in the groups 150-plus infrastructure team. He says successful automation is predicated on mature change management and extracting workflow information from people. Its really about asking the right questions and getting the decision points and logic that people store in their heads. For example, some of the security admin team were asked by the secure application groups, How do you know which groups are secure? They responded, We just keep that list in our heads. Thats excellent for them, but it makes it very difficult to automate a process, Wiseman says. 

BT couldnt have undertaken the project without a process automation tool. BMC Atrium Orchestrator applies rules and workflow to pool and coordinate a number of BT scripts. Any requirement for human intervention, such as monitoring task completion before initiating downstream scripts, is replaced by machine monitoring and automated triggers. 

Further efficiencies have been gained through repeatable automation. With processes configured in the CMDB, they can be applied to multiple applications. So it becomes a configuration change rather than an exercise in manual script writing for individual servers and applications. The tool remembers rules and workflow. Once a process has been automated repeating it is a push-button exercise. For example, automated processes spanning software kit deployments can be applied to defect releases. Mapping automation to this process has shrunk the time requirements of this exercise from around 15-20 minutes to just two-three minutes  on the face of it a modest achievement, but huge when you consider the task is completed 600-700 times a year. Says Wiseman: Its critical to BT as a business because it means we can potentially meet increased demand or volume changes without being held up by capacity issues.

Over the next three-to-six months Wiseman expects to automate more of the staff on-boarding process. Windows infrastructure maintenance automation is also on the horizon. Wiseman explains: When performing monthly infrastructure maintenance tasks, such as patch deployment and server restarts, we want to know on the weekend if weve got a service issue, rather than have our business report an incident Monday morning. In fact, we want to take immediate action during the maintenance window. We see Atrium Orchestrator as crucial to delivering that, Wiseman says. Longer-term, virtual workstations and server provisioning and deployment will be explored. We will look at automating some of our more complex software deployments, possibly even in a production environment. We want to share the success weve had across our other technology business plans, he says.

Yes

Yes

BT people have been advantaged in two ways. Firstly, BTs team of engineers and administrators enjoy new job satisfaction and work-life balance. The team of four administrators managing on-boarding regularly undertook overtime and weekend work to satisfy last minute demands for new user account set-up and activation. The automation project has minimised these requirements and rebalanced work-life commitments simply by accelerating the process and rescheduling activity overnight, on demand, or as a bulk migration. It was a very labour intensive holistic process, Wiseman says. Flexible hiring practices heightened demands on BT security administrators, who often had to scramble to deliver new user accounts within tight timeframes, he says. People worked over the weekend or in the evening to make it happen. 

Automation enables the on-boarding process to run in different modes  as a scheduled task overnight, whereby requests arriving during the day are queued up and run overnight; or run on-demand, for urgent requests; or as a simple CSV file (for bulk user migrations) which are loaded into a location for automatic deployment. The added flexibility gives BTs team more options and more time. 

Streamlining Wrap application testing environment creation and refreshes has introduced similar timesaving-related benefits. However, in this area, Wiseman also identifies impacts on motivation and retention of skilled engineers. Its de-motivating to come into work and do the same task over and over again. Reducing repetitive manual tasks is paramount to retaining skilled workers. A skilled engineer running manual tasks repeatedly for three-to-four days is not the most effective use of that skill set or knowledge, Wiseman says. 

In both cases, BT workers newly released from manual processing now work on higher value tasks. Engineers once occupied for a week at a time managing the Wrap refresh are instead using that time to assist testing and development teams troubleshoot defects. The net effect means more work is done with the same number of people. 

The other central benefit of automation is the up-skilling of BT engineers, who over the course of the project have mastered a new toolset and skills. Wiseman says the IT organisation has a keener eye for process weaknesses and task eligibility for automation.  

BT security administrator Paul Miller has worked closely with BMC Atrium Orchestrator. He says the toolset has been a great addition and specifically highlights the quantum time-savings. It has saved myself and my team from working out-of-hours. Setting up bulk users (50 at a time) prior to Atrium Orchestrator took approximately seven hours. Now we can setup 50 users in 10 minutes, he says. Another function is name changes. Previously, it required a good 30 - 45 minutes of manual work to change a users name and associated Wintel components. BMC Atrium Orchestrator allows us to complete this function in just seven minutes  another great saving for my team.

Steve Isles, automation team lead, has been responsible for implementing BMC Atrium Orchestrator and analysing and developing automated processes. He says the opportunity to enhance his technical support skills with business analysis knowledge has changed his career outlook. I now feel as though I have enhanced my skill set and offer the business more value. I can see opportunities for process improvement, deliver the desired outcome, and I can go home at night knowing I have made a difference to my colleagues day-to-day activities, he says. 

Wiseman says BT has before undertaken process automation, but on a piecemeal basis automating individual components rather than a process from beginning to end. He says speed at which previously manual tasks are completed is exceptional. My original expectation of time-savings for the Wrap testing environment refresh was reducing the 18-20 hours down to two. I targeted that as an outstanding achievement for my team. To get that down to just 10 minutes is phenomenal and far exceeded my expectations, Wiseman says. 

He is similarly impressed with the gains made by automating the on-boarding process (account creation time down from seven minutes to just 10-15 seconds). And as recently appointed finance manager of his business unit, first year savings of $80,000 for the Wrap application and projected three-year savings of $250,000 are exceptional. Overall, in one year BT is on track to save $180,000 and $588,000 over three years.

Three major obstacles were present: 1) Accessing resources to drive the project; 2) Overcoming scepticism of BT people impacted by automation and the naturally occurring questions about job insecurity; and 3) Extracting from peoples heads information vital to architecting process automation. Once you identify the decision points and logic people generally have documented about 20 percent of the actual process  everything else is stored in their heads, Wiseman says. Getting the sceptics onboard was a case of show, dont tell, he says. 

Wiseman says one of the key project learnings is the importance of extracting detail required to automate manual processes. We question people who show up with something simple requiring automation. We question them more thoroughly to extract all the detail we need. Without the information you cant scale the process.

Wiseman says people not directly involved in the project offered the greatest resistance. These people fixated on the idea that any automation should be developed using native technology. Why are we using a third-party product? they asked. They went on to question ROI cost calculations and the wisdom of ignoring the accepted practice of using technology that wasnt anchored to the platform it was addressing. If youre going to do something with a SQL database then you need to be doing it in SQL, they proffered. Wiseman says there was pervasive weariness of new technology and the likelihood of a return.

In placating the doubters Wiseman stuck to a simple ROI calculation based on four elements: the length of time the process took to run; the length of time required to automate it; process time completion post automation; and process repetition. We kept it simple because we needed to show immediate impact, Wiseman says. The other side of the pitch was evaluating process automation from a risk impact assessment. If the dollars didnt add up, but it was a high impact activity, that is the risk of someone failing to remember to do it, or doing it incorrectly, is high, then it made sense to do it.

Exceeded

Wiseman says process automation will be ongoing and over the next three-to-six months will address other manual tasks in staff on-boarding. Windows infrastructure maintenance automation is also on the horizon. Longer-term, virtual workstations and server provisioning and deployment will be explored.

When they see something happening in 10 minutes that used to take 3-4 days, its irresistible. Its hard to argue with facts, Wiseman says. He says the sceptics soon changed their position when they witnessed the improvements. We left it with them a few days and they loved it. Its one thing seeing a demo and quite another to see firsthand the benefits to a particular role. The floodgates have opened and its difficult keeping up with demand. The possibilities are endless. To tackle the emerging demand for process automation, BT has increased the team from two to four people to drive new initiatives and leverage the toolset.