WRF-CMake: integrating CMake support into the Advanced Research WRF (ARW) modelling system

The Weather Research and Forecasting model (WRF 1 ) model (

WRF-CMake aims at helping model developers and end-users by adding CMake support to the latest version of WRF and the WRF Processing System (WPS), while coexisting with the existing build set-up.The main goals of WRF-CMake are to simplify the build process involved in developing and building WRF and WPS, add support for automated testing using continuous integration (CI), and the generation of pre-built binary releases for Linux, macOS, and Windows thus allowing non-expert users to get started with their simulations in a few minutes, or integrating WRF and WPS into other software (see, for example, the GIS4WRF project (Meyer & Riechert, 2019)).The WRF-CMake project provides model developers, code maintainers, and end-users wishing to build WRF and WPS on their system several advantages such as robust incremental rebuilds, dependency analysis of Fortran code, flexible library dependency discovery, automatic construction of compiler command-lines based on the detected compiler, and integrated support for MPI and OpenMP.Furthermore, by using a single language to control the build, CMake removes the need to write and support several hand-written Makefiles, and Perl and Shell scripts.The current WRF-CMake set-up on GitHub offers model developers and code maintainers an automated testing infrastructure (see Testing) for Linux, macOS, and Windows, and allows end-users to directly download pre-built binaries for common configurations and architectures from the project's website (experimental).WRF-CMake is available as a free and open-source project on GitHub at https://github.com/WRF-CMake and currently includes CMake support for the main Advanced Research WRF (ARW) core and WPS.

Testing
A fundamental aspect of software development is testing.Ideally, model components should be tested individually and under several testing methodologies (Feathers, 2004).Here, however, as the WRF framework does not offer a way to unit test its components, we instead run separate build and regression tests to evaluate the effects of our changes.While build tests are used to check the absence of compilation errors, regression tests are used to estimate the size of simulation errors resulting from our change.
Build tests are performed for all supported build variants (Table 1) using CI services at every commit.As noted by Hodyss andMajumdar (2007), andGeer (2016), the high sensitivity to initial conditions of dynamical systems, such as the ones used in weather models, can lead to large differences in skill between any two forecasts.It is this high sensitivity to initial conditions that can obscure the source of model error, whether this originates from a change in compiler or architecture, an actual coding error, or indeed, the intrinsic nature of the dynamical system employed.As a result, the impact of our changes are evaluated using the range-normalized relative percentage error (δ x ) and range-normalized root-mean-square percentage error (NRMSPE; Appendix A).These are computed per domain for all grid points, and for all vertical levels.The errors are assessed by (a) comparing the outputs of prognostic-variable outputs (Table 2) from WRF (Make) against those from WRF-CMake and (b) comparing the outputs for all build variants (for both Make and CMake) against a reference build variant defined as Linux/Make/Debug/serial.
These tests are then run for all supported build variants (Table 1) using the WRF-CMake Automated Testing Suite (WATS), and a subset of namelists2 from the official WRF Testing Framework, using CI services at major code changes (e.g.before merging pull requests), and for 1 hour of simulation time, to constrain computing resources.
Here, we report summary results for the domain showing the greatest error (i.e.innermost; domain 2) after simulating 60 minutes.Values of δ x are aggregated for all quantities reported in Table 2 and referred to as δ.At the start of the simulation, the NRMSPE between WRF (Make) and WRF-CMake is zero (Appendix B, Figure 5), but small, when comparing WRF build variants (both Make and CMake) against a reference variant (Linux/Make/Debug/serial; Appendix B, Figure 6), thus suggesting an expected variability of outputs when running WRF across different platforms.
After 60 minutes (simulation time), WRF-CMake produces, on average, small values of δ, with mean close to zero, and most of the error (99.8 %) between -0.05 and 0.05 % (Figure 1).On Linux, the only build variants showing no error are for Debug/serial and Debug/dmpar (Figure 1 and 2).For NRMSPE (Figure 2), values of w show to be the most sensitive, however, the largest errors are shown for all components of wind velocity, on both, Linux and macOS.
Differences, in particular for Release build variants, most likely arise from an inconsistent use of compiler optimization options in WRF (Make) across its C and Fortran files, whereas in WRF-CMake, such options are centrally and consistently applied.Given that Debug/serial and Debug/dmpar show no error, we would expect the same to be true for the OpenMP variants Debug/smpar and Debug/dm_sm.Further investigation is required to establish the source of these differences.
When comparing both Make and CMake versions against the reference build variant (i.e.Linux/Make/Debug/serial; Figure 3 and 4), the errors appear to be of equal, or greater, magnitude than those shown when comparing WRF (Make) against WRF-CMake for both δ (Figure 3) and NRMSPE (Figure 4), thus indicating that the variability across build variants may be more important and may also be an inherent feature of WRF.
The choice of operating system has the greatest impact on both δ and NRMSPE (Figure 3 and 4) over compiler optimization strategies and build tool used.A change in build tool to CMake appears to produce values of δ and NRMSPE consistent with those obtained from versions of WRF built with the original build scripts3 .The largest errors are shown for wind velocity and, specifically for u and w.Larger values of δ and NRMSPE between operating systems appear to be a general property of WRF (i.e.with/without CMake support) and should be investigated further.

Concluding remarks
We introduce WRF-CMake as a modern replacement for the existing WRF build system.Its main goals are to simplify the build process involved in developing and building WRF and WPS, add support for automated testing using CI, and automate the generation of pre-built binary releases for Linux, macOS, and Windows.Results from regression tests indicate that, when evaluating outputs of prognostic variables, errors between WRF and WRF-CMake are generally small, or smaller, then errors originating from a change in optimization strategy (e.g.Debug, Release) or a change in platform (e.g.Linux to macOS).These larger errors appear to be a general property of WRF (i.e.with/without CMake support) and should be investigated further.Depending on feedback and general uptake by the community, future work may involve adding support for WRF-DA, WRFPLUS, WRF-Chem, and WRF-Hydro.

Figure 4 :
Figure 4: WRF (Make and CMake) vs reference build variant: range-normalized root mean-square percentage errors (NRMSPE) against the reference build variant (Linux/Make/Debug/serial) for the domain with highest errors only (domain 2) after 60 minutes (simulation time).

Table 1 :
Build variants used in build and regression tests.Make: original WRF build system files, CMake: this paper; Debug: compiler optimizations disabled, Release: enabled; serial: single processor, dmpar: multiple with distributed memory (MPI), smpar: multiple with shared memory (OpenMP), dm_sm: multiple with MPI and OpenMP.

Table 2 :
WRF prognostic variables evaluated during regression tests.
v Meridional component of wind velocity m s −1 w Vertical component of wind velocity m s −1