Template filling capabilities

Template filling capabilities#

About#

This example demonstrates how to use the fill feature in Woom to automatically generate files from templates using workflow context variables.

The fill feature is useful when you need to create multiple files with similar structure but different values based on the current cycle, member, or other workflow parameters.

What it demonstrates:

  • Defining multiple templates to fill in the task configuration

  • Using Jinja2 template syntax to insert workflow variables

  • Automatic rendering of templates with cycle-specific values

  • Generating output files with dynamic names based on context

Path: examples/academic/fill.

Configuring#

workflow.cfg#
[app]
name=HYDRO
conf=BASIN1
exp=forecast

[cycles]
begin_date=2020-01-01
end_date=2020-01-03

[stages]
    [[cycles]]
    run=hydro_model
tasks.cfg#
[hydro_model]
    [[content]]
    commandline=cat output/config_{{ cycle }}.nml

    [[fill]]
        [[[model_config]]]
        template=model.nml.j2
        destination=output/config_{{ cycle }}.nml

        [[[param_file]]]
        template=parameters.txt.j2
        destination=output/params_{{ cycle }}.txt

Each subsection of the fill section (e.g., model_config, param_file) defines which templates to fill:

  • template: The Jinja2 template file to use (located in the templates directory)

  • destination: Where to write the filled file (supports template variables)

The filled files will be generated in the output/ directory with cycle-specific names like config_2020-01-01.nml.

Using the fill CLI command#

You can also fill templates manually using the woom fill command:

woom fill model.nml.j2 output/config.nml --task-name hydro_model --cycle 2020-01-01

This is useful for testing templates or generating one-off configuration files.

See also: woom fill.

Running#

Overview#

Let’s have an overview of stages before running the workflow.

$ woom show overview
##################################### APP ######################################
name: HYDRO
conf: BASIN1
exp: forecast
################################## TASK TREE ###################################
cycles:
    - run: hydro_model
#################################### CYCLES ####################################
2020-01-01T00:00:00+00:00 -> 2020-01-03T00:00:00+00:00 (2 days 00:00:00)
################################### ENSEMBLE ###################################
no member

Dry run#

Now let’s run the workflow in test (dry) and debug modes.

$ woom run --log-no-color --log-level debug --dry-run
woom.log    : DEBUG    *** STARTED LOG SESSION ***
woom.cli    : DEBUG    Run the workflow
woom.cli    : DEBUG    Loading extensions
woom.cli    : INFO     No extension to load
woom.cli    : DEBUG    Load workflow config: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/workflow.cfg
woom.cli    : INFO     Loaded workflow config
woom.cli    : INFO     App name: HYDRO
woom.cli    : INFO     App conf: BASIN1
woom.cli    : INFO     App exp: forecast
woom.cli    : DEBUG    Initialize the host manager
woom.cli    : INFO     Initialized the host manager
woom.cli    : DEBUG    Load hosts config file: hosts.cfg
woom.cli    : INFO     Loaded hosts config file: hosts.cfg
woom.cli    : DEBUG    Infer host
woom.cli    : INFO     Infered host: local
woom.cli    : DEBUG    Initialize the task manager
woom.cli    : INFO     Initialized the task manager
woom.cli    : DEBUG    Load the task config file: tasks.cfg
woom.cli    : INFO     Loaded the task config file: tasks.cfg
woom.cli    : DEBUG    Initialize the workflow
woom.workflow: DEBUG    Task tree:
cycles:
    - run: hydro_model
woom.cli    : INFO     Initialized the workflow
woom.cli    : INFO     Successfully setup the workflow!
woom.cli    : DEBUG    Run the workflow
woom.workflow: DEBUG    Running the workflow in fake mode
woom.workflow: DEBUG    Entering stage: prolog
woom.workflow: DEBUG    No sequence of task. Skipping...
woom.workflow: DEBUG    Entering stage: cycles
woom.workflow: INFO     Single cycle with unique date: 2020-01-01 00:00:00+00:00
woom.workflow: DEBUG    Running cycle: 2020-01-01T00:00:00+00:00 -> 2020-01-03T00:00:00+00:00 (2 days 00:00:00)
woom.workflow: DEBUG    Entering sequence: run
woom.workflow: DEBUG    Running task: cycles/run/hydro_model
woom.workflow: DEBUG    Task path: HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model
woom.workflow: DEBUG    Cleaning task: cycles/run/hydro_model
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00
woom.job    : INFO     Started job manager: BackgroundJobManager()
woom.workflow: DEBUG    Submitting task: cycles/run/hydro_model
woom.workflow: DEBUG      Dependencies: 
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model
woom.tasks  : DEBUG    Fill template 'model_config': model.nml.j2 → output/config_2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00.nml
woom.cli    : ERROR    Workflow failed: Templates model.nml.j2 not found
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/cli.py", line 425, in main_run
    workflow.run(dry=args.dry_run, force=args.force, skip=args.skip)
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/workflow.py", line 793, in run
    job = self.submit_task_fake(task_depend, blocking)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/workflow.py", line 559, in submit_task_fake
    self.context.task.fill_templates(dry=True)
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/tasks.py", line 463, in fill_templates
    template = wrender.JINJA_ENV.get_template(template_file)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/jinja2/environment.py", line 1016, in get_template
    return self._load_template(name, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/jinja2/environment.py", line 975, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/jinja2/loaders.py", line 126, in load
    source, filename, uptodate = self.get_source(environment, name)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/render.py", line 107, in get_source
    raise TemplateNotFound(f"Templates {template} not found")
jinja2.exceptions.TemplateNotFound: Templates model.nml.j2 not found

Normal run#

And finally in run it.

$ woom run --log-no-color
woom.cli    : INFO     No extension to load
woom.cli    : INFO     Loaded workflow config
woom.cli    : INFO     App name: HYDRO
woom.cli    : INFO     App conf: BASIN1
woom.cli    : INFO     App exp: forecast
woom.cli    : INFO     Initialized the host manager
woom.cli    : INFO     Loaded hosts config file: hosts.cfg
woom.cli    : INFO     Infered host: local
woom.cli    : INFO     Initialized the task manager
woom.cli    : INFO     Loaded the task config file: tasks.cfg
woom.cli    : INFO     Initialized the workflow
woom.cli    : INFO     Successfully setup the workflow!
woom.workflow: INFO     Single cycle with unique date: 2020-01-01 00:00:00+00:00
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00
woom.job    : INFO     Started job manager: BackgroundJobManager()
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model
woom.workflow: INFO     Created batch script: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/fill/jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model/job.sh
woom.cli    : ERROR    Workflow failed: Templates model.nml.j2 not found
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/cli.py", line 425, in main_run
    workflow.run(dry=args.dry_run, force=args.force, skip=args.skip)
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/workflow.py", line 796, in run
    job = self.submit_task(task_depend, blocking)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/workflow.py", line 533, in submit_task
    self.context.task.fill_templates(dry=False)
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/tasks.py", line 463, in fill_templates
    template = wrender.JINJA_ENV.get_template(template_file)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/jinja2/environment.py", line 1016, in get_template
    return self._load_template(name, globals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/jinja2/environment.py", line 975, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/jinja2/loaders.py", line 126, in load
    source, filename, uptodate = self.get_source(environment, name)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/render.py", line 107, in get_source
    raise TemplateNotFound(f"Templates {template} not found")
jinja2.exceptions.TemplateNotFound: Templates model.nml.j2 not found

Check status#

Check what is running or finished.

$ woom show status
╭──────────────┬─────────┬─────────────┬─────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────╮
│ STATUS       │ JOBID   │ TASK        │ CYCLE                                               │ SUBMISSION DIR                                                                             │
├──────────────┼─────────┼─────────────┼─────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────┤
│ NOTSUBMITTED │         │ hydro_model │ 2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00 │ jobs/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model │
╰──────────────┴─────────┴─────────────┴─────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────╯

Show run directories#

Show where tasks were executed.

$ woom show run_dirs
╭─────────────┬─────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ TASK        │ CYCLE                                               │ RUN DIR                                                                                                            │
├─────────────┼─────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ hydro_model │ 2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00 │ /home/docs/woom/scratch/woom/HYDRO/BASIN1/forecast/2020-01-01T00:00:00+00:00-2020-01-03T00:00:00+00:00/hydro_model │
╰─────────────┴─────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯