Evaluation Criteria for Large Allocation Requests
The WRAP assesses the merits of large computing requests from Wyoming researchers and collaborators for use of NWSC supercomputing resources. Computing requests are accepted in the atmospheric, earth system sciences and closely related sciences for funded projects. The panel recommends action with respect to a prospective user's request on the basis of the computational experimental design, computational effectiveness, broader impacts, past performance, and availability of computing resources, as described here.
Overall Context. As part of conducting merit review of the requests for CISL resource allocations, the WRAP and CISL allocations staff will ensure that:
- All requests for resources that exceed a threshold level determined by CISL and the WRAP shall be peer-reviewed.
- Written reviews of the resource requests shall be completed in a timely way and made available to the requestors.
- Recommendations to CISL management for the allocation of resources based on the requests, reviews and the available resources shall be documented.
- The allocations process shall be consistent with the conflict-of-interest policy given below and shall maintain confidentiality of requestors and their reviews.
Purpose and Scope of Reviews. In its review of allocation proposals, the WRAP verifies the suitability of the work for WRAP resources, considers the ability of the research team to complete the work, and most significantly, reviews the merit of the proposed computational plan.
The main criteria used by the WRAP to evaluate eligible large computing allocation requests are:
1. Computational experimental design. The steps in the research plan should explain how the scientific objectives will be achieved. For computational experiments, the proposed computations should encompass simulation parameters (grid size, time scale, ensemble parameters, etc.) that are needed to obtain accurate and meaningful results, as well as the human resources that can be devoted to the task. The amount of resources (of all types) requested should be derived from the methodology and research plan.
2. Computational effectiveness. For computational resource requests, the choice of applications, methods, algorithms and techniques to be employed to accomplish the stated scientific objectives should be reasonably described and motivated. For data storage resource requests, the data usage, access methods, algorithms and techniques to be employed to accomplish the stated research objectives should be reasonably described and motivated.
3. Efficiency of resource use. The resources requested should be used as efficiently as is reasonably possible and in accordance with the recommended use guidelines of those resources. For computational resources, performance and parallel scaling data should be provided along with a discussion of optimization or other work done to improve the applications. For storage resources, the WRAP will consider the choices made in deciding which data to store, the value of that data both within the proposing research team and among the wider community, approaches for data access and dissemination, and long-term retention plans.
4. Progress from prior allocations. Requesters submitting their second or later large allocation requests to the WRAP will be reviewed on the progress and accomplishments of the prior allocations, including the publications produced (which may be in various pre-published states due to delays in the publication cycle) and effective usage of the prior allocation award. Requesters should discuss how well past projects performed with respect to the three criteria above. In addition, requesters must submit a 3-4 slide powerpoint presentation highlighting the scientific results of past projects.
5. Broader impacts. The WRAP will consider broader impacts that support the WNA effort such as:
- Substantial involvement of both UW and NCAR researchers,
- Strengthening of UW’s research capacity, directly or through collaboration,
- Strengthening of university computational science capacity in EPSCoR states, or
- Research in a new or emerging area of Earth System science.
Completion of a project associated with a large allocation (and not followed by a subsequent request) should be accompanied by a final report describing the allocation outcomes, stated appropriately in terms of the original objectives. The report should be submitted to email@example.com.