Support for High Performance Computing

"High performance computing (HPC) is not just hardware. It is essential that we understand that the total package consists of personnel, software, hardware, networking, physical environment and infrastructure, and University endorsement and support."

High performance computing is a pervasive, robust, evolving technology that fundamentally changes almost all facets of modern society for the better, including education and research. HPC will continue to drive these changes for the foreseeable future. In rapidly changing many aspects of our society, supercomputing continually changes even its own definition, so that many experts define supercomputing as anything within an order of magnitude of the fastest computing available for a given task. As high performance becomes ever higher, it forces the rest of computing and technology to keep pace by pulling them in its wake. What was state-of-the-art supercomputing power ten years ago is now a commodity item on every desktop. This trend, too, will continue for the foreseeable future.

Universities are also leaders that drive change in society, and it is no coincidence that universities are deeply involved in HPC. Universities take a leading role in generating personnel that develop software, hardware, and networking. Universities serve as test beds for new physical and virtual environments and infrastructure, and promote and support ever-expanding applications of computing.

U.S. universities can trace much of their success in advancing HPC (and as a result, all computing) to their treating HPC as they treat their libraries. For example, the use of books and journals is free in the library at the University of Kentucky, and so is use of the HP Superdome computer. The library is actually merging with HPC over time, as the library becomes a large distributed digital database.

Smaller libraries that generally serve the same purpose as larger ones are able to borrow and exchange resources that one library might not have available through interlibrary loans, broadening the resources available to students, faculty and staff. The linking of distributed computers for HPC on the grid is like interlibrary loan between campuses, rendering smaller computers able to perform big jobs for their users at least some of the time.
The collection of the library is large and diverse. The library has one or a few copies of many different titles. In contrast, the campus bookstore has a smaller collection, and a collection more specialized for students who want to use their books longer, write in their books, and tear out pages. The library is a database with less redundancy than the bookstore, and the library costs far less than the bookstore for each individual user. The bookstore has very many copies of a few titles. Similarly, smaller supercomputing clusters, funded by individual investigators or departments using grants, are comparable to campus bookstores in their content, cost to individual users, and usage. Users working on a cluster are able to use their own funds to make a supercomputer designed specifically for their use meet their specific needs. When the grants expire, a department or individual must leave the "bookstore." They either get more grants, or go back to the library.

It is common for campuses to permit access to HPC at no cost through allocation boards. The allocation boards are functionally similar to librarians, who allocate books and reserve materials for specific periods like the allocation boards regulate computing time. Just as the library covers a broad array of topics, no-cost HPC services large areas of science, mathematics, engineering, art and the humanities. Allocation boards attempt to serve clients in all of these areas, and as a result, each client usually receives only enough computing cycles for a project to reach a limited goal.

Institutional

The University of Kentucky's HPC allocation policies ensure that they both encourage and facilitate the utilization of HPC at UK in a manner designed to meet the institutional goals and priorities. Like the library policies that permit anyone to check out a small number of books, the allocation policies for HPC permit faculty and students to garner a small amount of HPC time (less than 1000 hours) without a proposal. This mechanism supports small efforts in new directions in uses of HPC, particularly in emerging fields that might lead to large, new research grants. There is a well-defined process in place for larger allocations that was developed through the university's Subcommittee on Research Computing (SRC) in consultation with the office of the Vice President for Information Technology and Human Services. Any Level 2 (1,000 to 10,000 hours per year) request goes to the SRC and one of the faculty members on the committee reviews and approves the request. The goal is to approve all worthwhile requests within 48 hours. The SRC response is based on basic research potential only. Because of the similarity between HPC and library services, however, a secondary goal is to be able to eventually approve all requests even if it means educating and working with the PI directly to improve the proposal. Finally, there are the researchers who want even larger amounts of resources. The university asks each of these PIs to go through an outside review through the national allocations boards. This review process is no different than getting an external grant. There is a learning curve on what the boards wanted in these proposals, and it is different from writing a NSF or DOE grant proposal or a journal article. Still, successful proposals can be written, and there are examples online as well as researchers in Kentucky that can help maintain the shared openness of the library model of HPC.

The University of Kentucky allocation review process is related to that employed by similar HPC centers at Caltech and Boston University, where the goal is to obtain HPC access for local researchers. Several classes have used HPC access for educational purposes as well, and each course request is discussed in the SRC. Their projects are not typical routine assignments, but instead are interesting and timely scientific problems. The undergraduates in these classes have generated a sudden, unusually high number of applications to graduate school (with good success).

If the University of Kentucky attempts to do all of its supercomputing elsewhere, national allocations will become a major issue. Not everyone in the country uses the same "library." Instead, libraries have many branches to serve their clientele. In reducing the number of supercomputers on the grid, impediments to usage by the research community will be increased instead of reduced. For example, Gaussian (a common program for molecular modeling and simulation used in biology and chemistry) runs best on machines with a large shared memory architecture. The University of Kentucky Superdome is the best computer on the grid for this purpose. Researchers arrive from all over the Internet to use Gaussian on the University of Kentucky machine. If Kentucky ceases to support HPC on its own, a number of Kentucky researchers will suddenly be vying nationally for "a smaller piece of the pie" with a larger number of other researchers also displaced by the shutdown of HPC in Kentucky. Similar problems would arise with other software packages. A strong commitment from the administration in support of HPC, using the library as a model for providing services, will ensure that Kentuckians take a leading role technologically in the 21st century.

Funding

The University of Kentucky has had a supercomputer as part of its computational resources since 1987 (J. Connolly, High Performance Research Computing at UK, 2002). In the 15 years since acquiring that first IBM 3090, the university has moved from invisibility in the field of scientific computing to top-10 status. Kentucky has joined the national computational program through its membership in the National Computational Science Alliance (NCSA). HPC has also made the computational scientists in Kentucky much more competitive for extramural funding. According to data from the Office of Sponsored Projects Accounting, the top users of the supercomputer at the University of Kentucky submitted 250 proposals and were awarded 137 grants totaling $21.2 million over the period from 1997-2002. Of course, one cannot simply evaluate the grant performance of the supercomputer users by comparing a list of the top 25 HPC users to a list of the top 25 funded PIs, regardless of how tempting and facile that might be. Funding varies among the disciplines, in large part because the cost of working in different areas varies. Many HPC users get grants of supercomputer time instead of dollars, so evaluating publication performance would be more realistic. However, publication rates for different disciplines vary as much as funding does for different disciplines.

At $21.2 million over five years, the funding received by the top users greatly exceeds the cost of the supercomputer, which is presently about $1.3 million per year. The amount of extramural funding awarded to the top users is likely to increase greatly over the next few years, and having a modern HPC facility is essential to the university achieving and maintaining top-20 status. Withdrawing from HPC would have a detrimental effect on research funding and would lead to the loss of many of UK's top researchers. A state in which 10% of the people cannot read would never consider that as a reason to close the libraries. On the contrary, illiteracy is a reason to increase educational outreach. The rate of technological and computational illiteracy in the state is much higher than 10%. Knowing that HPC is a pervasive evolving technology that fundamentally changes modern society for the better, the university cannot withdraw from it and expect to achieve top-20 status. In fact, it is easy to justify investing far more in HPC than we do now.
The University of Kentucky has decided that open-access HPC should function like the library for its users, and rightly so. The library itself is moving into digital collections in computers as individuals realize a library is simply a very large database best suited for HPC. Someday, if the university needs funds for HPC it might even consider charging for access to the physical library. Antiquated paper books, magazines, bricks, mortar, parking and people, cost more per book and more per periodical to deliver than a digital collection in a high-performance computer. Charges would then be used as an instrument of policy to encourage everyone to utilize the most efficient, complete, and effective means of delivering information.

The university does not charge individual users directly for centrally provided HPC. Cost Accounting Standards dictate much about recharge schemes for HPC. The university has often considered recharge schemes, but has determined that using the library as a model for providing services is the best way to ensure that Kentuckians take a leading role technologically in the 21st century. At the same time, the university can and should recharge (following the campus bookstore model) for departmental and individual HPC systems adapted to special purposes or in which access is not open to anyone through the standard allocation system.

The University of Kentucky should work with the state government in Frankfort to secure a national center for computational science with federal, state, and industrial funding. This HPC center would be funded sufficiently to stay in the Top 10 nationally. Such an HPC center, built around the up-to-date hardware platforms needed to do cutting-edge research, and with a yearly increase in its budget (instead of the 14-year-old line item budget number now in place), would recognize that HPC is a force that fundamentally alters modern society for the better in a way that propels Kentucky forward.

Staff

As noted earlier, HPC is about more than just hardware. The current fad in HPC, computer clusters, look cheap in part because they do not count the cost of the personnel to set them up and keep them running. Large supercomputers like the HP Superdome's are cheap in comparison. Peer review panels like to fund clusters more than bigger general purpose HPCs because small clusters are easier to justify as a special purpose item appropriate to one grant application. In fact, the HPC package comprises personnel, software, hardware, networking, a physical environment and infrastructure, and endorsement and support from the University, state, and nation.

An effective effort in HPC must provide competitive salaries for highly qualified staff. An expansion of support staff to include professionals with specific experience in use of the various software and visualization tools is necessary for HPC outreach. At its present level, the HPC effort is equipped like a new library without enough personnel to unlock the door every day of the week. These new support staff will be able to work on a one-on-one basis with faculty and students, and also provide more general instruction (e.g., seminars and courses) as might be appropriate. The HPC staff will identify areas that clearly need support, and work aggressively with those users to find solutions. The HPC staff must be able to work with users the way librarians are able to assist their clients.

HPC will continue to drive changes in modern society for the better in the near future. As high performance in computing becomes ever higher, it will bring the rest of computing and technology along by pulling them in its wake. Kentucky will reap these benefits from continued investment in HPC.