When SHiELD meets Container Technology

SHiELD (System for High-resolution prediction on Earth-to-Local Domains) is a unified atmospheric model supporting various configurations, including severe weather nowcasting, hurricane forecasting, and subseasonal-to-seasonal prediction.

SHiELD has become easy to use, thanks to container technology. Better yet, SHiELD can runs on all major operating systems (Windows, macOS, and Linux), as well as on the cloud.

Container technology leverages the flexibility of SHiELD and opens many possibilities for research and education.

SHiELD

  • “One code, one executable, one workflow” for weather-to-seasonal prediction. SHiELD can be a global, a global-nest, or a regional model.

  • FV3 Dynamical Core provides a powerful foundation for unified prediction modeling.

Learn more

Container

  • A container is a stand-alone, software artifact that packages applications (e.g., an atmospheric model) for portability.

  • Model installation made easy. No need to worry about environmental variables and library dependencies.

  • Model becomes portable, from laptop to cloud.

  • Ensure bitwise identical results, regardless of different computing systems.

Learn more about container on Wikipedia.

Containerization of SHiELD

  • SHiELD and its dependencies are packaged using Docker.

  • Designed for collaboration. No proprietary software is used.

  • Parallelism in mind. Support Message Passing Interface (MPI) and OpenMP.

  • High performance computer (HPC) ready. Seamless deployment on HPCs through Singularity.

Learn more in Cheng et al. (2022).

Your path to unified modeling systems starts here

Try out running SHiELD on your local computer or HPC cluster.

Conduct Hurricane Ida global-nest simulation in just three steps.

Choose your preferred container platform:

  • Most popular container platform

    1. Install

    docker pull gfdlfv3/shield

    2. Download

    Hurricane Ida case
    GFS fix files

    3. Run

    docker run --cap-add=SYS_PTRACE --ulimit='stack=-1:-1' \
          -v "/PATH_TO_/global_nest_Ida/:/rundir" \
          -v "/PATH_TO_/GFS_fix/:/GFS_fix" \
          -it gfdlfv3/shield mpirun -np 8 -oversubscribe SHiELD_nh.prod.32bit.x

  • Designed for supercomputer

    1. Install

    apptainer pull docker://gfdlfv3/shield

    2. Download

    Hurricane Ida case
    GFS fix files

    3. Run

    apptainer exec --bind
          /PATH_TO_/global_nest_Ida/:/rundir,
          /PATH_TO_/GFS_fix/:/GFS_fix
          /PATH_TO_/shield_latest.sif cd /rundir && mpirun -np 8 -oversubscribe SHiELD_nh.prod.32bit.x

All documentation details can be found in this user-friendly GitHub repository.

  • PATH_TO_: absolute path to the target directory/file.
  • Our images were created via multi-arch build. We support both ARM64 and AMD64 architectures.
  • For reference, our Hurricane Ida example (a three-hour global-nest simulation) should finish in a few minutes with Intel Core i7-12700.
  • Guide for FV3 and configuration (namelist) can be found here.
  • This workflow can be applied to any simulations. Simply modify the content in above case folder with your desired case. See Pre-processing to learn more.

Pre-processing

An easy way to create grids and initial/boundary conditions.

Simplified and modified based on UFS UTILS

Choose your preferred container platform:

  • Most popular container platform

    1. Install pre-processing tool

    docker pull gfdlfv3/preprocessing

    2. Set up

    Set up mounts (fix, workdir, GFSvOPER) with these instructions.

    3. Run

    docker run -it \
          -v "/PATH_TO_/fix:/UFS_UTILS/fix" \
          -v "/PATH_TO_/workdir:/workdir" \
          -v "/PATH_TO_/GFSvOPER:/GFSvOPER" \
          gfdlfv3/preprocessing

  • Designed for supercomputers

    1. Install pre-processing tool

    apptainer pull docker://gfdlfv3/preprocessing

    2. Set up

    Set up mounts (fix, workdir, GFSvOPER) with these instructions.

    3. Run

    apptainer exec --bind
          /PATH_TO_/fix:/UFS_UTILS/fix,
          /PATH_TO_/workdir:/workdir,
          /PATH_TO_/GFSvOPER:/GFSvOPER
          /PATH_TO_/preprocessing_latest.sif /PATH_TO_/preprocessing_entrypoint.sh

All documentation details can be found in this user-friendly GitHub repository.

  • PATH_TO_: absolute path to the target directory/file.
  • Guide for UFS_UTIL configuration can be found here.

Post-processing

A convenient way to post-process GFDL model output.

Containerization of FRE-NCtools

Choose your preferred container platform:

  • Most popular container platform

    1. Install post-processing tool

    docker pull gfdlfv3/fre-nctools

    2. Run

    Regrid a C96 output onto 1- by 1-degree latitude-longitude grid

    docker run -v /PATH_TO_/global_nest_Ida/:/workdir \
		gfdlfv3/fre-nctools \
		fregrid --input_mosaic ./INPUT/C96_coarse_mosaic.nc \
		--nlon 360 --nlat 180 --interp_method conserve_order1 \
		--input_file atmos_sos \
		--output_file atmos_sos_ltd.nc \
		--scalar_field PRMSL,TMP2m,DPT2m,PRATEsfc,ULWRFtoa

  • Designed for supercomputers

    1. Install post-processing tool

    apptainer pull docker://gfdlfv3/fre-nctools

    2. Run

    Regrid a C96 output onto 1- by 1-degree latitude-longitude grid

    apptainer exec --bind /PATH_TO_/global_nest_Ida/:/workdir \
		/PATH_TO_/fre-nctools_latest.sif \
		fregrid --input_mosaic ./INPUT/C96_coarse_mosaic.nc \
		--nlon 360 --nlat 180 --interp_method conserve_order1 \
		--input_file atmos_sos \
		--output_file atmos_sos_ltd.nc \
		--scalar_field PRMSL,TMP2m,DPT2m,PRATEsfc,ULWRFtoa

All documentation details can be found in this user-friendly GitHub repository.

  • PATH_TO_: absolute path to the target directory/file.