Overview. Indiana University has established a new paradigm for IT infrastructure, which it has dubbed the Intelligent Infrastructure. In 2003, IU began virtualizing servers and providing infrastructure as a service. To date, the University has virtualized some 450 servers on just 34 HP ProLiant DL585 physical servers (a consolidation ratio of more than 13 virtualized servers per physical server). The number of virtualized servers is expected to nearly double in the next year as new clients take advantage of the flexible service model. 

The project transforms infrastructure management from the complex, historical approach of designing custom infrastructure for each and every application, to creating a standardized, simplified cloud computing environment that provides infrastructure as a service.

Goals. Simply put, the projects basic goals were cost savings, improved sustainability, agility, and simplification of infrastructure management.

Initially, IT managers hoped for cost savings of up to two-thirds. But in practice, they achieved cost reductions approaching 90 percent. In converting the central ERP infrastructure, for example, simply replacing the old UNIX systems would have cost approximately $1.2 million over three years for hardware, O/S acquisition, maintenance, and support. The actual cost of the new virtualized environment, however, is roughly $150,000 over that same three-year period  or 12.5 percent of the cost of traditional infrastructure.

In terms of sustainability, the project has substantially reduced electricity consumption and space requirements. Electricity needs have been reduced both in terms of direct electrical needs to support the new infrastructure, and in reduced cooling needs. Overall, electricity is estimated to cost 80-85% less than in previously installed solutions. Because the new environment supports higher equipment density as well as fewer physical servers, it has also enabled the university to reduce space requirements by roughly 90 percent. 

Simplifying the environment allows the university to manage more systems with fewer staff, or more realistically, to take responsibility for more and more systems without increasing staff. This has allowed a significant number of IT professionals to focus on activities more closely related to the core university mission, rather than working to simply ensure uptime in the hardware.

The project is transforming the culture at Indiana University. Increasingly, program managers, researchers, and academicians are realizing that cloud computing will provide the flexible IT infrastructure they need, allowing them to refocus on their core missions. In that sense, this project goes beyond improving the IT function; it makes IU more efficient at fulfilling its academic, research, and public outreach missions.

The technology behind this project is twofold: VMware ESX virtualization software technology, and the hardware that runs it: HP ProLiant DL585 servers.

Virtualization Technology: At a high level, virtualization is the abstraction of physical resources: virtualization technologies create logical views of server and storage resources that are distinct from their physical components. Resources are pooled, or aggregated, to encapsulate their resources and capabilities in useful and meaningful ways. This provides a foundation for deploying, managing, allocating, and delivering capabilities as services. 

A key goal of virtualization is to remove the complexities inherent when scores of individual components are gathered within an infrastructure, and to provide efficiencies through standardization. Standardization reduces the visible differences between entities to create efficiencies within an infrastructure, as well as greater utilization efficiency and flexibility. 

Virtualization isnt new. It was first used more than a generation ago on mainframe computers. But as the client-server model and inexpensive x86 (industry standard) servers established the model of distributed computing, virtualization was largely abandoned. Distributed computing created new challenges, though. Among them:
 Low infrastructure utilization (often only 10-15% of capacity, according to International Data Corporation)
 Increased infrastructure and IT management costs
 Inability to deal effectively with downtime of mission-critical applications

Virtualization as implemented at Indiana University with VMware ESX addresses all these challenges. First, it improves utilization by pooling common resources to create a unified structure, so that resources are utilized by applications as needed, with much less waste. Literally hundreds of individual servers are consolidated onto a much smaller number of physical servers; and in the virtualized environment, processing capability is treated as a single resource. In the virtualized environment, utilization is often between 85 and 90 percent of capacity.

Virtualization cuts infrastructure costs by reducing the number of servers and other hardware (including storage systems) in the data center. Virtualization also enables administrators to take advantage of new ways to manage the infrastructure so they spend less time on provisioning, configuring, monitoring, and maintenance. Higher-level functionality includes automation of many system administration tasks, and real-time load balancing.

Virtualization increases application availability to deliver more reliable services. With a pool of resources available, IUs virtualized environment essentially eliminates the need for planned downtime. When equipment maintenance or upgrades are necessary, applications can simply be moved to alternate processors or storage disks. In the same way, virtualization enables rapid recovery from unplanned outages through secure backup and migration to another virtual infrastructure.

Hardware Technology: HP ProLiant DL585 G5 servers are helping Indiana University maximize the utility of its Intelligent Infrastructure. The HP ProLiant DL585 G5 is a highly manageable, rack optimized, four-socket server designed for maximum performance in an industry-standard architecture. With up to four Quad-Core AMD Opteron processors and a large memory footprint, the DL585 G5 is well matched to virtualization and consolidation applications. Its remote management functions help reduce costs and improve the ability to respond quickly to business changes.

The ProLiant DL585 is the highest performing server on the VMmark benchmark among four-socket, 16-core servers, offering up to a 20% performance advantage. (VMmark is the first benchmark that was designed specifically to quantify and measure the performance of virtualized environments. The VMmark benchmark is intended to measure the performance of virtualized servers on a system under test [SUT] so that customers can compare the capabilities of different platforms for virtualization. VMmark represents the performance of virtual machines within a server running VMware ESX and a set combination of operating systems and specially tuned applications reflecting a typical datacenter environment.)

Yes

Yes

The Intelligent Infrastructure project provides several advantages to the IU user community: 

 High availability. As stated earlier, IUs virtualized environment enables rapid recovery from unplanned outages. Even if some components of the virtual environment fail, applications can be configured to quickly failover to backup infrastructure on other physical components in the environment. Planned downtime is also unnecessary, since applications can simply be moved to alternate processors or storage disks when equipment maintenance or upgrades are necessary, with zero downtime for the customer application being served.

 Faster speed to market. In the past, when a user requested a new server, it might take the IT organization 10 business days to acquire, configure, test, and deploy the equipment. Now the organization can respond to multiple requests on a same-day basis. A new server image can be cloned from the system in a few minutes and at any time, and quickly deployed for use. Researchers can be equipped with new computing resources almost as soon as they acquire funding or are awarded research grants.

 More forgiving capacity planning. With traditional IT infrastructure, inaccurate capacity planning and/or growth analysis caused real headaches: downtime, poor performance, and increased costs. Today, capacity planning is far less critical. If a particular application or function requires more resources, it can take advantage of real-time load balancing capabilities in the short run; and the system administrator can quickly deploy a new server as a more permanent solution. This is an especially appealing feature for peak load periods. The historical approach at IU was to purchase capital assets for peak loads that might represent only a small portion of the year, while the remaining periods of low usage would cost the organization substantially more. 

 Reduced carbon footprint. As stated earlier, the project has substantially reduced electricity consumption and space requirements. Electricity needs have been reduced both in terms of direct electrical needs to support the new infrastructure, and in reduced cooling needs. Overall, electricity savings are estimated to be 80-85% less than previously installed solutions. 

 Reduced physical footprint: the virtual infrastructure has also enabled the university to reduce space requirements by over 1,000 square feet, or over 90 percent in the case of the ERP system hardware.

 Outreach enhancements: Freeing up data center space and reducing incremental costs for new application servers has enabled IU to provide a low-cost Disaster Recovery site for the Indiana Office of Technology. It has also enabled IU to provide the IT infrastructure for Kuali Foundation, a non-profit organization responsible for sustaining and evolving a comprehensive suite of administrative software for colleges, universities, commercial agencies, and interested organizations that share a common vision of open, modular, and distributed systems for their software requirements. 

The most visible example is the migration of the universitys ERP system to the Intelligent Infrastructure. Indiana University previously implemented a large ERP system serving over 100,000 students and 20,000 faculty and staff, for a wide variety of administrative and self-service functions related to Student Administration and Human Resources.
 
The initial server architecture comprised a cluster of physical web servers, a cluster of application servers, and a large database server. All told, the system included 34 UNIX systems spread across 30 physical servers. As the UNIX physical servers approached end-of-life status, the IT function planned conversion of the ERP application (including web and application servers) to run under RH Linux, using VMware ESX, on the standard set of HP ProLiant DL585 hardware. 

Replacing the UNIX-based hardware with current UNIX hardware technology would have cost $1.2 million (at discounted rates) over three years, for the hardware and O/S acquisition, maintenance, and support. In comparison, the costs of HP ProLiant servers, VMware ESX, support, and maintenance for a comparable three-year period is about $150K (at discounted rates)  an 8:1 reduction. The reduction in annual UNIX hardware and software maintenance alone was enough to cover the initial acquisition of new servers to begin the initiative.

IU still receives full ERP application software vendor support, and its ability to respond quickly to changing business needs has significantly improved. System administrators supporting the ERP system can now perform maintenance during the standard work week, when the highest level of vendor support is available, without interrupting system availability. Downtime has also been substantially reduced. Overall, system availability and service levels have been improved.

The financial savings on this project alone were so compelling that even if we did not have a virtualization strategy in place, this project would singlehandedly have been a catalyst for such a decision, says Rob Lowden, Director of Enterprise Infrastructure. Making such a substantial infrastructure change required detailed communication, planning, and customer buy-in with stakeholders and application owners. With the Intelligent Infrastructure we were able to utilize the ERP applications existing load balancing software and gradually introduce new virtual machines to the server farm and demonstrate with little to no risk the effectiveness of the new solution. This also had the added benefit of ensuring current servers met the load, and the new virtual machine could achieve real user exposure for capacity planning purposes. After the customers were completely satisfied with the approach, we were then able to add more virtual machines to the farm and remove the old servers piece by piece. We repeated this process for the entire web and application layer of the ERP system without a single minute of downtown and no adverse impact to the service or customer.   

Elizabeth A. Van Gordon, CIO for two IU regional campuses, Indiana University Northwest and Indiana University Kokomo, notes that Indiana University Northwests IT computer machine room is located in a 40-year-old building. To upgrade this facility to meet modern standards would severely strain campus financial resources. By utilizing IUs Intelligent Infrastructure virtual environment, IU Northwest will save tens of thousands of dollars while offering the campus superior services. Not only have we eliminated the need for local hardware purchases, we have eliminated hardware service contracts, backup software licenses, and some software service contracts. What a relief it is to have the Indiana University Intelligent Infrastructure service that has allowed IU Northwest to migrate essential services to virtual servers in an environment that is climate-controlled, power-regulated, and secure.

At its inception in 2003, IUs Intelligent Infrastructure project was a pioneer in terms of utilizing VMWare's virtualization on x86 production systems. It was arguably the first and the most effective example of widespread virtualization in higher education. Other institutions have taken the IU business model and followed it almost to the letter.

The change in technology has set a new direction for the provisioning of IT infrastructure at IU; but what truly distinguishes the IU experience is the cultural change the Intelligent Infrastructure project has engendered.

The biggest challenge was changing the mindset of the classic server-hugger.  IUs infrastructure team had to confront the user mentality that it was standard practice to dedicate one or more physical servers to an application and its needs. Such users wanted the IT organization to be able to point out a particular physical asset that ran their application. Such a focus on distinct, physical hardware infrastructure distracted users from how the IT group really provides core value. It: 1. expertly provides IT service by knowing the applications, the requirements of those applications, and the customer needs, and 2. ensures the infrastructure -- whatever form it takes -- can meet those requirements.

That cultural barrier was quite prevalent during the early phases of the Intelligent Infrastructure project. The Director of Infrastructure spent a significant amount of his time selling the virtualization strategy and changing the culture of this antiquated approach to providing client-server services. Today, as virtualization and cloud computing become increasingly ubiquitous, that barrier is raised only occasionally and usually falls relatively easily. 

Another exceptional aspect of the project is innovation. The Intelligent Infrastructure is built on what was, at the time, very cutting-edge technology for full-production systems, and is so, even today. As early as January 2003, IU had full enterprise-wide production systems running under the virtualized environment. That exemplified the IT organizations confidence in virtualization, which in turn helped to break down the cultural barriers and obstacles.   

Early in the project, even some software vendors were unaware of VMware and the benefits of virtualization (including improved uptime). If the university wanted 24 x 7 support from them, the vendors said, it couldnt run the applications on a virtual machine, or it would require IU to replicate any support issues on a dedicated physical asset. In fact, the approach was so well designed and implemented that IU rarely needed to do that. If vendor support was needed, IU could quickly demonstrate the details of the situation. The virtualization technology was almost never at fault. This went a long way toward building the confidence and trust of IUs vendor partners. 

An additional and unintended positive outcome of this approach manifested itself in 2008. IU began a project to design and build a new data center on the Bloomington campus. As the building was completed and preparations made to move existing assets to the new facility, the Intelligent Infrastructure became a very viable option for application and system owners who wanted to minimize downtime and consolidate aging servers, while improving current business continuity postures. The net result was that IU moved 15,000 square feet of equipment into a 5,000-square-foot space, providing for even greater growth and scalability of its assets in the future, in the new facility. 

Obstacles came in multiple forms. 

Technical challenges. Initially, there were some minor compatibility issues with some software products and applications. Some smaller software vendors were not especially interested in tackling those issues. With virtualized environments a small market share, the vendors could not justify dedicating resources.  Physical security keys provide an example. Many legacy software products require a physical security key that attaches to a physical server. In a virtual environment, where the application might be shifted from one physical asset to another multiple times a day, the physical security key will not work. To overcome that issue, the IU team came up with the innovative strategy of using a network-attached USB hub, which enables the physical dongle to be addressed by any part of the virtual net. This allowed the virtual machine to migrate from physical asset to physical asset and never lose connection with the physical security key. 

Another technical challenge was incompatibility between revisions of different processors and chipsets. That prevented VMware from moving a running virtual machine from one physical server to another if the chipsets and processor were not of the same revision and manufacturer. This required all x86 servers to be either Intel- or AMD-based, and further, that they be of the same chipset and processor revision.

Resources. Resource issues rarely arose, because virtualization is inherently a more efficient, money-saving technology. Whenever the infrastructure for a given application had to be replaced, it was always cheaper to do so through virtualization. Moreover, there was commitment at the highest level of the organization to make the transition to a virtualized environment.

Organizational challenges. To overcome organizational reluctance or skepticism, the IT infrastructure team set out to demonstrate success and confidence with 100% of its own internal assets before promoting the service to other departments. This eat your own cooking approach proved very successful in instilling confidence and providing reference systems. 

As with any organization, there were some natural IT silos.  Many of these had dedicated physical assets whose users would not consider virtualization as an option. To win over such users, the IT team had to demonstrate to them with a Proof of Concept that virtualization was not only a viable alternative, but also made great business sense. 

Today that battle is largely won. It is now ingrained in the culture so that when the infrastructure team coordinates with the service or application owner to launch a new application or service, the question immediately arises, Does this run on VMware? Will you support this on VMware? When IU recently acquired a trouble ticket and tracking system, vendors that did not support virtual environments were left out of the RFP. 

As users approach end of lifecycle on existing equipment, they face important questions. Why extend the life of existing independent infrastructure? Why not put the application in the virtualized environment? Every quarter, it becomes increasingly challenging to rationalize anything other than the Intelligent Infrastructure.

In this case, the greatest resistance did not come when the IT infrastructure team initially proposed the virtualized environment. The teams first order of business was to convert its own applications and servers. The commitment to move from a mainframe environment to a client/server architecture was in place. Few people outside the IT team were aware that it was a virtual environment.

It was the wholesale consolidation and mass conversion of user groups throughout the university that prompted some resistance. But as noted earlier, the issues that arose were typically more cultural than technical (IU estimates 90 percent cultural and 10 percent technical). Even today, there is some resistance, often related to concerns over job security. People simply want to stick with what they know. 

In addition, virtualization is not a viable solution for every IT infrastructure need. It still does not work well for up to 20 percent of user needs. Applications running under the VMS operating system should not be virtualized. Nor should large-scale databases, due to VMwares limitation of four CPUs per virtual machine. For the time being, applications that need to leverage more than four CPUs need to retain their own, dedicated physical infrastructure. In addition, applications with high I/O requirements, such as a centralized backup system, encounter difficulties in the VMware environment. Finally, applications that require processor affinity can run under VMware, but will not realize the full benefits of virtualization. VMWare continues to make strides in these areas and the environments that are not a good fit are becoming fewer and fewer.  

Exceeded

At the start, the infrastructure team had targeted only its own needs (approximately 100 virtual machines). Success with those applications led to converting customer applications. As of this writing, the team has converted close to 500 servers. As customer acceptance grows, it expects to reach 1000 servers in the next 12 months.

Already, the benefits of virtualization extend to hundreds of user groups throughout IUs main campuses in Bloomington and Indianapolis. More recently, the infrastructure team brought in entire regional campuses into the Intelligent Infrastructure.

Many other entities are also benefiting. These include, as mentioned earlier, the Indiana Office of Technology, the Kuali Foundation, and numerous researchers at IU. The university supports more than $500 million in local, state, national, and international research grants. For these and many other uses, leveraging virtualization and cloud computing has been the key to making the most of scarce resources.

Looking ahead, IU can also leverage the infrastructure to support community outreach through its healthcare system, social service agencies, and other organizations with limited IT expertise. For such organizations, offering IT infrastructure as a turnkey service represents immeasurable value.

The answer here must necessarily look both forward and back. The Intelligent Infrastructure project began in 2003. Initially, it was not embraced by many users. But over time, the infrastructure team demonstrated its own belief and commitment to the technology, and by carefully presenting the benefits for outside users, won over hundreds of user groups. Looking ahead, the IT group expects future conversions to be easier  both in terms of overcoming cultural issues, and resolving technical ones. 

Given the swell of energy around virtualization technology, and the growing installed base throughout the industry, IU expects its Intelligent Infrastructure will grow faster and faster. In the early years, the infrastructure team worked hard to convert a single department or application. Now, the same effort is likely to bring an entire regional campus into the fold. That is the sign of an audience truly embracing the technology. 

IU is opening a new data center in spring 2009. With that transition, the university expects interest in leveraging the latest in infrastructure will grow  and with it, acceptance of conversion to the Intelligent Infrastructure.