Supercomputer Opening Caps Years of Effort
October 12, 2012 — It began with laying hundreds of miles of fiber-optic cable, much of it buried under the ground along stretches of interstate that traverse the mountains and plains of Wyoming. Next week, the state’s evolution from primarily pulling minerals out of the ground to a sky’s-the-limit outlook for supercomputing will be complete.
The $74 million NCAR-Wyoming Supercomputing Center (NWSC), with a primary focus of atmospheric research, is slated to open Monday, Oct. 15, with a ceremonial dedication. Located at the North Range Business Park in Cheyenne, the $30 million supercomputer, dubbed “Yellowstone,” will be used by multiple University of Wyoming researchers and their students to model detailed simulations of hydrology, carbon sequestration, planet formation, efficiency of wind turbines, and much more.
“This is a direct result of careful planning over 15 years. We were prepared for this opportunity,” says Bill Gern, UW’s vice president for research and economic development. “We have the ability, in time, to be the model for the world in measuring the subsurface flow of water. That’s what NCAR can bring to us. This is fundamentally transformative for the university.”
“This will be an attractive place for researchers. And, if you’re doing computational research, it will be very attractive,” says Bryan Shader, UW’s special assistant to the vice president of research and economic development, and a mathematics professor. “Even if you’re not in computational research and you want to be at the forefront of research, it’s good to be at a university where there is significant computational infrastructure. One need not be in the sciences or computer sciences to benefit from the NWSC.”
But a lot has transpired in 15 years for technology in southeastern Wyoming to go from subpar to the verge of super.
For years, UW, located in a city with a population of just over 30,000, struggled to obtain economical, basic telecommunications services for delivering advanced networking to campus researchers.
In the 1980s, UW began leasing circuits from US West (now Century Link), AT&T, Global Crossing and other telecommunications companies. This hodge-podge of various circuits allowed for a cobbled network from Laramie to Denver, recalls Robert Morrison, UW’s director, telecommunications & system services, and chief technologist.
“Because Wyoming didn’t have a huge population and economy, we didn’t have access to high-speed Internet. We had to lease private lines and even built a private microwave between Laramie and Fort Collins,” Morrison says.
But such a basic system was susceptible to interruption and cost prohibitive to develop to the necessary level UW, as a research university, needed.
“Our Internet would go down and the telephone companies were all pointing fingers at each other,” Morrison says. “All we know is it didn’t work.”
In 1997, UW, with support from its Office of Research and Economic Development, joined Internet2, which was a consortium of universities committed to the development of high-speed research networks. In order to connect to Internet2, UW led the charge to create the Front Range GigaPOP (FRGP), which is a peering point in Denver to connect to the commodity Internet, as well as high-speed research networks such as Internet2 and the National Lambda Rail (NLR).
The FRGP was established using funds UW received from a National Science Foundation grant to be used for networking, Morrison says.
The original partners in the FRGP included UW, Colorado State University and Colorado University-Denver. The group was later joined by the National Center for Atmospheric Research (NCAR), National Oceanographic and Atmospheric Administration (NOAA), most universities located in the Front Range, and government agencies.
The establishment of the FRGP was a great step forward, Morrison says. However, connectivity between Laramie and Denver was problematic. It was an incident in 1999 that broke the bronco’s back, so to speak. At that time, Laramie suffered its own mini-Y2K, when Internet access was down more than 24 hours -- the longest it had ever been out of service.
“After that, we were highly motivated to create a better, faster connection that we could manage,” Morrison says.
In order to address the problem of connectivity and reliability to the FRGP, UW spearheaded another initiative, creating a high-speed, high-bandwidth, flexible, fiber-optic network loop -- the BiSON ring (Bi-State Optical Network), which was completed in 2004. The BiSON ring connects UW and the new NWSC through Cheyenne, to Denver, to Boulder, Fort Collins and then back to UW. This high-bandwidth, flexible, self-managed network is a critical component for any research university, but especially a rural university like UW.
“With the BiSON ring, there is no single point of failure, no single fiber cut that can bring us down, and it provides very high performance. This gives us the connectivity that every major research university in the region had,” Morrison says. “It put us on a par with well-known universities such as Carnegie Mellon and Stanford (in terms of Internet capabilities).”
As a result, UW now benefits from the same level of access to advanced networks, such as Abilene (Internet2) and NLR, as do its collaborating institutions located in the Denver/Boulder metropolitan area, as well as major research universities in the U.S., Morrison says.
“My contention is this laid the groundwork and thinking of people at NCAR and elsewhere. You need high-speed Internet access to get to a supercomputer,” Morrison says. “Without the FRGP and the BiSON ring, you couldn’t have the supercomputer.”
“Because of the university’s leadership and foresight, they’ve put us in a position to use this resource,” Shader says. “For the last decade, we’ve strived for national distinction in computational sciences. We have hired 25 faculty members who heavily use computational sciences in their research. We have faculty who can use the NWSC. Our graduate students are plugged in. It’s an exciting time.”
Bouncing off Boulder
Right after Christmas 2006, UW received its first email on the subject of a supercomputer. It came from Randy Bruns, president and CEO of Cheyenne LEADS. As a result, then-governor Dave Freudenthal met with UW President Tom Buchanan and urged the university to become involved in efforts to lure such a facility to Wyoming.
“We were in a bidding war with Colorado on this,” Gern recalls. “NCAR was looking for property near their Mesa campus in Boulder.”
The drawback in Boulder was the city’s inability to guarantee the price of power as the Mesa Laboratory was built out. Once built out, the facility was expected to need 25 megawatts of power, about one-fifth the power the city of Cheyenne needs every day to run. The Mesa facility only had capabilities for 16 megawatts, Gern says.
“That’s why power was such an element. The Mesa lab didn’t have it,” Gern says.
“The need for this facility was pretty great,” agreed Chet Lockard, UW’s associate director of facilities planning. “They (NCAR) outgrew their space at the Mesa facility. The design (of the NWSC facility in Cheyenne) was based on the frustration at the Mesa facility.”
To secure the supercomputing center, the state of Wyoming made a $20 million investment. The Wyoming Business Council became involved, providing $4.5 million to Cheyenne LEADS to create infrastructure for the site. Cheyenne Fuel, Light & Power became another important business partner.
And UW stepped up, agreeing to provide $1 million annually for 20 years to the University Corporation for Atmospheric Research (UCAR) to help pay for the next supercomputer at the site. In exchange, UW received access to 20 percent of the NSF-base-funded computer’s computational system annually.
The 75 million core hours UW faculty members will have at their disposal represents one of the largest pools overseen by any university in the United States, Shader says.
This past summer, NSF approved seven UW research projects – ranging from hydrology of the Colorado River Basin to planet formation from star debris to fluid dynamics of wind turbines -- that will use the NWSC this fall. These initial computational science research projects will use approximately 27 million core hours and involve close to 40 researchers, faculty, post-docs, graduate students and undergraduate students.
“It’s an impressive building. But even more impressive is the science that will take place there,” Shader says. “We can use it as a carrot to foster and support collaborations with other universities. We already have collaborations with three universities in Utah, and are developing collaborations with the University of Vermont and Idaho National Laboratory. We’re forging connections with scientists at NCAR.”
The NWSC is the result of a partnership among UCAR; UW; the state of Wyoming; Cheyenne LEADS; the Wyoming Business Council; Cheyenne Light, Fuel & Power; and NCAR. The NWSC is operated by NCAR under sponsorship of the National Science Foundation (NSF).
Look, up in Cheyenne! It’s a supercomputer!
How fast is the supercomputer? Well, certainly faster than a speeding bullet and maybe more powerful than a locomotive. Just in a different way.
Imagine 7 billion people conducting a calculation 200,000 times a second, and you have the capability of the machine in Cheyenne, Gern says.
“It can provide very accurate models of the real world and visualize it. That’s the power this thing is going to give us,” he says.
The 153,000-square-foot NWSC will contain some of the world's most powerful supercomputers (1.5 petaflops, which is equal to 1.5 quadrillion computer operations per second) dedicated to improving scientific understanding of climate, severe weather, air quality and other vital atmospheric science and geoscience topics. The center also will house a premier data storage (11 petabytes) and archival facility that holds irreplaceable historical climate records and other information.
Dimitri Mavriplis, a UW professor of mechanical engineering, works with computational fluid dynamics, a branch of fluid mechanics that uses numerical methods and algorithms to analyze and solve fluid flow challenges. He will use the supercomputer to simulate the aerodynamics of aircraft wings, helicopter propellers and wind turbines.
“One thing more powerful computers can do is to resolve more details of the turbulent flow that occurs over aircraft configurations, thus, providing more realistic simulations of the aerodynamics which, in turn, lead to more effective and efficient designs,” says Mavriplis, who has grant funding from the Army, Navy, Air Force and NASA for his research. “If we do our job well, we can get at least a factor of 100 more from the NWSC than we can with currently available hardware.”
IBM, which received the contract for the supercomputer, delivered the machine in parts during June. It took 18 semitrailers to move all of the components to Cheyenne, with some of the initial assembly captured on video by local and regional media.
Build it and they will come
The NWSC opened the door for Wyoming to be considered an ideal spot for high-performance computing, something Gern mentioned during a university-sponsored field trip to the NWSC in April.
“When Cheyenne LEADS brings in another big computing company, we can declare victory,” Gern says. “Who (is there) better than NCAR to say that Wyoming has a great climate for major computing centers?”
Shortly thereafter, Microsoft publicly announced plans to build a $120 million data center in Cheyenne. The company plans to create 17 new jobs initially (with a plan to eventually hire 40 people) with median wages that are 150 percent above the county’s average wage. The facility, expected to be operational in spring 2013, will serve the Mountain West region for the company.
To encourage Microsoft to choose Wyoming, the state provided $10 million in economic incentives.
“I believe Wyoming is positioned to be a leader in the technology sector, and data centers will be a catalyst for growth,” Gov. Matt Mead said at an April 9 press conference. “Wyoming is a perfect fit for data centers. We have abundant, affordable energy. Our naturally cool climate decreases costs for data centers, and we have redundant fiber optics. Microsoft’s decision recognizes that Wyoming is a hand-and-glove fit as a leading site for data centers and technology-related companies.”
In recent months, Laramie city and economic development officials have laid the groundwork to lure data centers of their own and allow for expansion of existing technology businesses. Officials from the Laramie Economic Development Corporation have targeted land on the north edge of Laramie for the Cirrus Sky Technology Park and applied for a $5 million grant to make the site “shovel-ready.” The funding would be used to construct water, sewer and street infrastructure on approximately 162 acres between 15th and 30th streets in Laramie.
In the latest development, the Laramie City Council, on Oct. 3, passed agreements with the LEDC relative to the property and funding mechanisms. Contingent upon the city and LEDC receiving the grant, UW has signed an agreement to purchase approximately 23 acres of the technology park. The property would serve as a base for high-tech business start-ups that are ready to move from the Wyoming Technology Business Center incubator.
And, in June, UW announced that IBM has been chosen to build UW’s campus cluster, which may have as many as 100 UW faculty members using it for their computational science research starting this fall.
The Advanced Research Computing Center (ARCC), nicknamed “Mount Moran” after a mountain in western Wyoming’s Tetons, will enable atmospheric and earth sciences faculty members -- who will be able to use the NWSC -- to learn what to expect with the software. The cluster provides the opportunity for that group of faculty members to work out issues caused by scaling up parallel algorithms from tens or hundreds of processors to thousands of processors, before moving up to tens of thousands of processors on the NWSC supercomputer.
Mount Moran, which arrived on campus in late September, also will provide a research resource for UW research faculty members -- such as bioinformaticists, ecologists, social scientists, pure mathematicians and theoretical physicists -- whose research doesn’t fall within the scope of the NWSC. For more information, go here.
“This (NWSC) is a facility that is going to be used for Wyoming and universities around the world for some time to come,” Gern says.
The NCAR-Wyoming Supercomputing Center officially opens with a ceremonial dedication Oct. 15.