CGD's CCSM Software Engineering Group

CCSM software infrastructure

CSEG infrastructure improvement efforts have continued to focus on producing a robust, scalable, efficient and easily extensible and portable model system. Efforts to improve CCSM robustness were concentrated in the areas of continued expansion of the CCSM scripts and testing framework. CSEG has created a new experimental run database, containing a web interface for both the input and query of model information, in order to provide both science and software developers the ability to easily duplicate and archive details of CCSM experiments. In the past, meta-data detailing various model runs were stored in multiple places, thereby making it difficult to easily determine details of past runs. A central, searchable database now provides the necessary information to easily reproduce a particular run and serve as a single placeholder for pointers to diagnostics and other model output. This database has provided CCSM developers and scientists with the capability to easily track experiments done both at NCAR and at remote sites. It has proven valuable in the creation of CCSM3.5 and in tracking model changes that are occurring as part of the creation of CCSM4.

Porting both the release and development versions of CCSM to new platforms continues to be an important priority for the CSEG group. Both the release and development CCSM code base were successfully ported to the Cray XT3 at ORNL. The CCSM Software Engineering Group has worked closely with our partners at the DOE Oak Ridge National Lab and Cray to carry out this effort. We are currently porting CCSM3.5 to the Cray XT3 at NERSC. CCSM portability efforts have been greatly streamlined by the creation of a single executable version of the development code base. Substantial work was done to optimize performance on the ORNL XT3, leading to throughput rates of 40 model-years/day at a 1.9x2.5_gx1v5 resolution. CSEG also helped carry out numerous multi-century simulations, both at ORNL and at NCAR, as part of the creation of CCSM3.5 in June 2007.

CAM Development

CAM development efforts have included new modifications to the deep convection and cloud fraction parameterizations and the incorporation of new chemistry modules for prognostic aerosols, greenhouse gases and prescribed aerosols.

Significant infrastructure improvements have also been made to the CAM code over the last year. The CAM namelist generation capability was extended to permit the generation of CCSM "use-cases" for CCSM3.5 simulations. The new CAM comprehensive testing framework was extended to facilitate the ability for atmospheric modelers to quickly and thoroughly test new code modifications prior to having these changes incorporated into the CAM source code base. This testing framework has proven to be an invaluable part of the CAM development effort by identifying countless problems before they entered the model.

POP Development

POP development efforts focused on the creation of a code base with new increased vertical resolution (60 levels) and with topographic modifications. Additional modifications to the code base involved the incorporation of modified anisotropic horizontal viscosity, near-surface eddy flux parameterization and tidal-driven mixing parameterization. Significant improvements to the POP infrastructure were also made. The coupling interface was reorganized such that it can now easily support various coupling architectures. This will permit POP to be incorporated into the new sequential CCSM system and easily maintain backwards compatibility with the concurrent CCSM model with minimal changes to the underlying POP code. Improvements were also made to the CCSM POP scripts.

CLM Development

Until recently, CLM has been constrained to run on the same grid as CAM. CLM software development efforts have continued in providing CLM with the functionality to run on its own high resolution, fine-mesh grid, independent of the CAM grid and couple to CAM via physically relevant downscaling and upscaling parameterizations. Major improvements to CLM memory usage and memory scaling have been achieved as part of this work. As part of improving CLM scaling, the CLM river routing code base (RTM) was parallelized. A new CLM test suite was also written that will permit a much larger number of tests to be run automatically for each new version of the CLM code base. This test suite will help ensure the robustness of the code base as major new modifications are added as part of the creation of CLM4.

CICE Development

CICE software engineering efforts have continued to focus on the incorporation of the Los Alamos National Laboratory sea ice model (CICE) as the standard CCSM sea ice component. CICE4 is now the official CCSM sea ice component. This code base adds new improved physics options and numerical and computational enhancements. A new improved treatment of snow and ice has also been incorporated such that snow and ice albedos are now based on observations. Other software engineering efforts had lead to the successful incorporation of CICE into the new sequential CCSM and the investigation of running CICE at 0.1-degree globally on peta-scale architectures.

Creation of Single-Executable Sequential CCSM

Projected peta-scale architectures will require the ability for each CCSM component to run on much higher resolution configurations utilizing processor counts that are one to two orders of magnitude greater than present settings. A sequential CCSM, where all components run on the same processor set, might provide a better architecture in which to ensure the type of quasi-local communication necessary to scale the system up to these projected processor utilizations.

A prototype sequential CCSM system has now been created that contains backwards compatibility with the current concurrent CCSM system (where each component runs on a unique set or processors, not utilized by any other component) and provides "plug and play" capability of data and active components. Work is currently underway to verify that this system produces the same climate as the current concurrent system. The resulting new system permits extensive code reuse and removes the existence of stand-alone component configurations that have a different code base than those used in coupled simulations. This will result in decreased code maintenance, consistent science across components and increased system robustness. Utilizing this system, we have recently created a new high-resolution configuration (1/2 degree atmosphere and .1 degree ocean) that we have ported to the Blue Gene platform. Finally, significant progress has also been made in creating a hybrid sequential/concurrent CCSM system where the atmospheric, land and ice components run sequentially with each other and concurrently with the ocean component.

The stand-alone CAM model (already a sequential system) was chosen as the starting point for the creation of a sequential CCSM system. To accomplish the first stage of this project, a significant advance was implemented in CAM to permit the transition from a CAM standalone model, where all the surface components were implemented as sub-components of CAM that made explicit use of CAM's data structures, to a version of CAM implemented as a sequential CCSM where the surface components are independent of CAM and communicate with CAM only utilizing framework dependent data structures. The design of these framework-dependent coupling interfaces has made it straightforward to add other CCSM components (e.g., CICE4 and POP2) in the next stage of this project.

Both ESMF and MCT (the current CCSM coupling framework) are being examined as coupling frameworks for the sequential system. The ESMF Stage 1 Evaluation Plan was completed successfully in June 2007. CSEG will continue to examination ESMF as a coupling framework via the Stage 2 Evaluation project.

Finally, the incorporation of parallel I/O throughout the entire CCSM model system has become a very high priority for the CSEG group. Parallel I/O has already been incorporated to some degree in CAM, CLM and POP. We hope to have it fully incorporated in each CCSM component within the next year.