Commands#
pangeo-forge-runner supports various commands that
can be configured via traitlets.
expand-meta#
- class pangeo_forge_runner.commands.expand_meta.ExpandMeta(**kwargs: Any)#
Application to expand meta.yaml to be fully formed.
Will execute arbitrary code if necessary to resolve dict_object recipes.
- config_file c.ExpandMeta.config_file = Unicode('pangeo_forge_runner_config.py')#
Load traitlet config from this file if it exists
- content_providers c.ExpandMeta.content_providers = List()#
List of ContentProviders to use to fetch repo.
Uses ContentProviders from repo2docker for doing most of the work. The ordering matters, and Git is used as the default for any URL that we can not otherwise determine.
If we want to support additional contentproviders, ideally we can contribute them upstream to repo2docker.
- feedstock_subdir c.ExpandMeta.feedstock_subdir = Unicode('feedstock')#
Subdirectory inside the repository containing the meta.yaml file
- json_logs c.ExpandMeta.json_logs = Bool(False)#
Provide JSON formatted logging output to stdout.
If set to True, all output will be emitted as one JSON object per line.
Each line will have at least a ‘status’ field and a ‘message’ field. Various other keys will also be present based on the command being called and the value of ‘status’.
TODO: This must get a JSON schema.
- log_datefmt c.ExpandMeta.log_datefmt = Unicode('%Y-%m-%d %H:%M:%S')#
The date format used by logging formatters for %(asctime)s
- log_format c.ExpandMeta.log_format = Unicode('[%(name)s]%(highlevel)s %(message)s')#
The Logging format template
- logging_config c.ExpandMeta.logging_config = Dict()#
Logging configuration for this python application.
When set, this value is passed to logging.config.dictConfig, and can be used to configure how logs throughout the application are handled, not just for logs from this application alone.
See https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig for more details.
- ref c.ExpandMeta.ref = Unicode(None)#
Ref of feedstock repo to fetch.
Optional, only used for some methods of fetching (such as git or mercurial)
- repo c.ExpandMeta.repo = Unicode('')#
URL of feedstock repo to operate on.
Can be anything that is interpretable by self.content_providers, using Repo2Docker ContentProviders. By default, this includes Git repos, Mercurial Repos, Zenodo, Figshare, Dataverse, Hydroshare, Swhid and local file paths.
- show_config c.ExpandMeta.show_config = Bool(False)#
Instead of starting the Application, dump configuration to stdout
- show_config_json c.ExpandMeta.show_config_json = Bool(False)#
Instead of starting the Application, dump configuration to stdout (as JSON)
bake#
- class pangeo_forge_runner.commands.bake.Bake(**kwargs: Any)#
Command to bake a pangeo forge recipe in a given bakery
- bakery_class c.Bake.bakery_class = Type(<class 'pangeo_forge_runner.bakery.local.LocalDirectBakery'>)#
The Bakery to bake this recipe in.
The Bakery (and its configuration) determine which Apache Beam Runner is used, and how options for it are specified. Defaults to LocalDirectBakery, which bakes the recipe using Apache Beam’s “DirectRunner”. It doesn’t use Docker or the cloud, and runs everything locally. Useful only for testing!
- config_file c.Bake.config_file = Unicode('pangeo_forge_runner_config.py')#
Load traitlet config from this file if it exists
- container_image c.Bake.container_image = Unicode('')#
Container image to use for this job.
For GCP DataFlow leaving it blank defaults to letting beam automatically figure out the image to use for the workers based on version of beam and python in use.
For Flink it’s required that you pass an beam image for the version of python and beam you are targeting for example: apache/beam_python3.10_sdk:2.51.0 more info: https://hub.docker.com/layers/apache/
Note that some runners (like the local one) may not support this!
- content_providers c.Bake.content_providers = List()#
List of ContentProviders to use to fetch repo.
Uses ContentProviders from repo2docker for doing most of the work. The ordering matters, and Git is used as the default for any URL that we can not otherwise determine.
If we want to support additional contentproviders, ideally we can contribute them upstream to repo2docker.
- feedstock_subdir c.Bake.feedstock_subdir = Unicode('feedstock')#
Subdirectory inside the repository containing the meta.yaml file
- job_name c.Bake.job_name = Unicode(None)#
Optionally pass a custom job name for the job run.
If None (the default), a unique name will be generated for the job.
- json_logs c.Bake.json_logs = Bool(False)#
Provide JSON formatted logging output to stdout.
If set to True, all output will be emitted as one JSON object per line.
Each line will have at least a ‘status’ field and a ‘message’ field. Various other keys will also be present based on the command being called and the value of ‘status’.
TODO: This must get a JSON schema.
- log_datefmt c.Bake.log_datefmt = Unicode('%Y-%m-%d %H:%M:%S')#
The date format used by logging formatters for %(asctime)s
- log_format c.Bake.log_format = Unicode('[%(name)s]%(highlevel)s %(message)s')#
The Logging format template
- logging_config c.Bake.logging_config = Dict()#
Logging configuration for this python application.
When set, this value is passed to logging.config.dictConfig, and can be used to configure how logs throughout the application are handled, not just for logs from this application alone.
See https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig for more details.
- prune c.Bake.prune = Bool(False)#
Prune the recipe to only run for 2 time steps.
Makes it much easier to test recipes!
- recipe_id c.Bake.recipe_id = Unicode(None)#
Optionally pass this value to run only this recipe_id from the feedstock.
If empty, all recipes from the feedstock will be run.
- ref c.Bake.ref = Unicode(None)#
Ref of feedstock repo to fetch.
Optional, only used for some methods of fetching (such as git or mercurial)
- repo c.Bake.repo = Unicode('')#
URL of feedstock repo to operate on.
Can be anything that is interpretable by self.content_providers, using Repo2Docker ContentProviders. By default, this includes Git repos, Mercurial Repos, Zenodo, Figshare, Dataverse, Hydroshare, Swhid and local file paths.
- show_config c.Bake.show_config = Bool(False)#
Instead of starting the Application, dump configuration to stdout
- show_config_json c.Bake.show_config_json = Bool(False)#
Instead of starting the Application, dump configuration to stdout (as JSON)