Ctrl+K

Running a recipe locally#

pangeo-forge-runner supports baking your recipes locally, primarily so you can test the exact setup that will be used to bake your recipe on the cloud. This allows for fast iteration on your recipe, while guaranteeing that the behavior you see on your local system is what you will get when running scaled out on the cloud.

Clone a sample recipe repo to work on#

This tutorial will work with any recipe, but to simplify things we will use this pruned GPCP Recipe that pulls a subset of GPCPC netcdf files from Google Cloud storage and writes it out as Zarr. The config we have setup for pangeo-forge-runner will fetch the files from remote storage only once on your system, caching it so future runs will be faster.

This same setup would work for any recipe!

  1. Clone a copy of the recipe to work on:

    git clone https://github.com/pforgetest/gpcp-from-gcs-feedstock
cd gpcp-from-gcs-feedstock

    You can make edits to this if you would like.

  2. Setup a virtual environment that will contain pangeo-forge-runner and any other dependencies this recipe will need. We use a venv here, but you may also use conda or other python package management setup you are familiar with.

    python -m venv venv
source venv/bin/activate

  3. Install pangeo-forge-runner into this environment.

    pip install pangeo-forge-runner

Now you’re ready to go!

Setting up config file#

Construct a local_config.py file that describes where the output data should go, and what should be used for caching the input files. Since we just want to test locally, these can point to the local filesystem!

# Let's put all our data on the same dir as this config file
from pathlib import Path
import os
HERE = Path(__file__).parent

DATA_PREFIX = HERE / 'data'
os.makedirs(DATA_PREFIX, exist_ok=True)

# Target output should be partitioned by job id
c.TargetStorage.root_path = f"{DATA_PREFIX}/{{job}}"

c.InputCacheStorage.fsspec_class = c.TargetStorage.fsspec_class
c.InputCacheStorage.fsspec_args = c.TargetStorage.fsspec_args

# Input data cache should *not* be partitioned by job id, as we want to get the datafile
# from the source only once
c.InputCacheStorage.root_path = f"{DATA_PREFIX}/cache/input"

c.MetadataCacheStorage.fsspec_class = c.TargetStorage.fsspec_class
c.MetadataCacheStorage.fsspec_args = c.TargetStorage.fsspec_args
# Metadata cache should be per job, as kwargs changing can change metadata
c.MetadataCacheStorage.root_path = f"{DATA_PREFIX}/{{job}}/cache/metadata"

This will create a directory called data in the same directory this config file is located in, and put all outputs and caches in there. To speed up multiple runs, input files will be cached under the data/cache directory.

Run a pruned version of your recipe#

You’re all set to run your recipe now!

pangeo-forge-runner bake \
    --config local_config.py \
    --repo . \
    --Bake.job_name=test1 \
    --prune

This should run for a few seconds, and your output Zarr should now be in output/tests1! Let’s explore the various parameters passed.

  1. --config local_config.py specifies the config file we want pangeo-forge-runner to read. If we were to try to run this on GCP or AWS, we can have additional aws_config.py or gcp_config.py files, and just pass those instead - everything else can remain the same. By putting most config into files, this also eases collaboration - multiple people can know they’re running the same config.

  2. --repo . specifies that we want the current directory to be treated as a recipe and run. This can instead point to a git repo, zenodo URI, etc as needed.

  3. --Bake.job_name=test1 specifies a unique job name for this particular run. In our local_config.py, we use this name to create the output directory. If not specified, this would be autogenerated.

  4. --prune specifies we only want to run the recipe on about 2 input files, rather than on everything. This makes for fast turnaround time and easy testing.

You can test the created Zarr store by opening it with xarray

>>> import xarray as xr
>>> ds = xr.open_zarr("data/test1/gpcp")
>>> ds
<xarray.Dataset>
Dimensions:      (latitude: 180, nv: 2, longitude: 360, time: 2)
Coordinates:
  * latitude     (latitude) float32 -90.0 -89.0 -88.0 -87.0 ... 87.0 88.0 89.0
  * longitude    (longitude) float32 0.0 1.0 2.0 3.0 ... 356.0 357.0 358.0 359.0
  * time         (time) datetime64[ns] 1996-10-01 1996-10-02
Dimensions without coordinates: nv
Data variables:
    lat_bounds   (latitude, nv) float32 dask.array<chunksize=(180, 2), meta=np.ndarray>
    lon_bounds   (longitude, nv) float32 dask.array<chunksize=(360, 2), meta=np.ndarray>
    precip       (time, latitude, longitude) float32 dask.array<chunksize=(1, 180, 360), meta=np.ndarray>
    time_bounds  (time, nv) datetime64[ns] dask.array<chunksize=(1, 2), meta=np.ndarray>
Attributes: (12/41)
    Conventions:                CF-1.6, ACDD 1.3
    Metadata_Conventions:       CF-1.6, Unidata Dataset Discovery v1.0, NOAA ...
    acknowledgment:             This project was supported in part by a grant...
    cdm_data_type:              Grid
    cdr_program:                NOAA Climate Data Record Program for satellit...
    cdr_variable:               precipitation
    ...                         ...
    sensor:                     Imager, TOVS > TIROS Operational Vertical Sou...
    spatial_resolution:         1 degree
    standard_name_vocabulary:   CF Standard Name Table (v41, 22 February 2017)
    summary:                    Global Precipitation Climatology Project (GPC...
    time_coverage_duration:     P1D
    title:                      Global Precipitation Climatatology Project (G...
>>>
On this page
Show Source