A single task with command line formatting#

About#

This is the simplest woom example, demonstrating the fundamental concepts of workflow configuration.

The example runs a single task in the prolog stage that showcases:

Basic workflow structure: Application configuration, parameters, and stages
Command line templating: Using Jinja2 syntax to insert parameters into task commands
Jinja2 filters: Applying transformations like upper and strftime to format values
Custom filters: Using the replicate_option filter to generate repeated command-line options
Parameter override: Task-specific parameters that override global values
Environment variables: Setting custom environment variables for tasks

This example is ideal for getting started with woom and understanding how configuration files, parameters, and templating work together.

Path: examples/academic/single_task.

Configuring#

workflow.cfg#

[app]
name=my_app_name
conf=my_app_conf
exp=my_app_exp

[params]
param1=foo
param2=bar
param3=toto
names=uo,vo
mydate=06/04/2010

    [[tasks]]
        [[[mytask]]]
        param3=TOTO

[env_vars]
MY_ENV_VAR=value

[stages]

    [[prolog]]
    prolog0=mytask

tasks.cfg#

[mytask]

    [[content]]
    env=myenv
    commandline="echo running mytask {{ params.names|replicate_option('--name', format='{opt_name} {value}') }} {{ params.param1 }} ; echo mytask {{ params.param2|upper }} {{ params.mydate|strftime('%A %d %B') }}; sleep 20"

    [[submit]]
    queue=seq

hosts.cfg#

[local]
#patterns=*
    [[envs]]
        [[[myenv]]]
            [[[[vars]]]]
                [[[[[set]]]]]
                MY_ENV_VAR=my_env_var_value_background

[datarmor]
patterns=datavisu*,*.ice.ifremer.fr,datarmor*
scheduler=pbspro
module_setup=. /etc/profile.d/modules.sh

    [[queues]]
    seq=sequentiel

    [[dirs]]
    scratch=$SCRATCH

    [[envs]]
        [[[myenv]]]
            [[[[vars]]]]
                [[[[[set]]]]]
                MY_ENV_VAR=my_env_var_value_datarmor

Running#

Overview#

Let’s have an overview of stages before running the workflow.

$ woom show overview
##################################### APP ######################################
name: my_app_name
conf: my_app_conf
exp: my_app_exp
################################## TASK TREE ###################################
prolog:
    - prolog0: mytask
#################################### CYCLES ####################################
No cycle
################################### ENSEMBLE ###################################
no member

Dry run#

Now let’s run the workflow in test (dry) and debug modes.

$ woom run --log-no-color --log-level debug --dry-run
woom.log    : DEBUG    *** STARTED LOG SESSION ***
woom.cli    : DEBUG    Run the workflow
woom.cli    : DEBUG    Loading extensions
woom.cli    : INFO     No extension to load
woom.cli    : DEBUG    Load workflow config: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/workflow.cfg
woom.cli    : INFO     Loaded workflow config
woom.cli    : INFO     App name: my_app_name
woom.cli    : INFO     App conf: my_app_conf
woom.cli    : INFO     App exp: my_app_exp
woom.cli    : DEBUG    Initialize the host manager
woom.cli    : INFO     Initialized the host manager
woom.cli    : DEBUG    Load hosts config file: hosts.cfg
woom.cli    : INFO     Loaded hosts config file: hosts.cfg
woom.cli    : DEBUG    Infer host
woom.cli    : INFO     Infered host: local
woom.cli    : DEBUG    Initialize the task manager
woom.cli    : INFO     Initialized the task manager
woom.cli    : DEBUG    Load the task config file: tasks.cfg
woom.cli    : INFO     Loaded the task config file: tasks.cfg
woom.cli    : DEBUG    Initialize the workflow
woom.workflow: DEBUG    Task tree:
prolog:
    - prolog0: mytask
woom.cli    : INFO     Initialized the workflow
woom.cli    : INFO     Successfully setup the workflow!
woom.cli    : DEBUG    Run the workflow
woom.workflow: DEBUG    Running the workflow in fake mode
woom.workflow: DEBUG    Entering stage: prolog
woom.workflow: DEBUG    Entering sequence: prolog0
woom.workflow: DEBUG    Running task: prolog/prolog0/mytask
woom.workflow: DEBUG    Task path: my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.workflow: DEBUG    Cleaning task: prolog/prolog0/mytask
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.job    : INFO     Started job manager: BackgroundJobManager()
woom.workflow: DEBUG    Submitting task: prolog/prolog0/mytask
woom.workflow: DEBUG      Dependencies: 
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.workflow: INFO     Fake submission: my_app_name/my_app_conf/my_app_exp/prolog/mytask → bash /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh
woom.workflow: DEBUG    Fake submission:
--------------- submission command ---------------
bash /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh
-------------- batch script content --------------
#!/bin/bash

# Prolog
set -eo pipefail
# - handler for graceful termination
on_sigterm() {
    echo "Received termination signal, cleaning up..."
    # Just exit cleanly, let on_exit handle status
    exit 0
}
# - handler for killing termination
on_sigkill() {
    echo "Received kill signal, cleaning up..." >&2
    # Just exit cleanly, let on_exit handle status
    exit 1
}
# - handler for exit (always called)
on_exit() {
    status=$?
    if [ ! -f "/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.terminating" ]; then
        echo $status > "/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.status"
    fi
    exit $status
}
trap on_sigterm SIGKILL
trap on_sigkill SIGTERM SIGINT
trap on_exit EXIT


# Environment variables
export WOOM_SCRATCH_DIR="/home/docs/woom/scratch"
export WOOM_WORK_DIR="/home/docs/woom/work"
export MY_ENV_VAR="my_env_var_value_background"
export WOOM_TASK_NAME="mytask"
export WOOM_RUN_DIR="/home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_WORKFLOW="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/workflow.cfg"
export WOOM_HOST="local"
export WOOM_TASKMANAGER="/home/docs/checkouts/readthedocs.org/user_builds/woom/envs/latest/lib/python3.11/site-packages/woom/tasks.cfg:/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/tasks.cfg"
export WOOM_JOBMANAGER="BackgroundJobManager"
export WOOM_TASK_TREE_PROLOG_PROLOG0="['mytask']"
export WOOM_CYCLES=""
export WOOM_NMEMBERS="0"
export WOOM_MEMBERS=""
export WOOM_PATHS_PATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/bin"
export WOOM_PATHS_PYTHONPATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/lib/python"
export WOOM_PATHS_LIBRARY_PATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/lib"
export WOOM_PATHS_INCLUDE_PATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/include"
export WOOM_APP_PATH="my_app_name/my_app_conf/my_app_exp"
export WOOM_APP_NAME="my_app_name"
export WOOM_APP_CONF="my_app_conf"
export WOOM_APP_EXP="my_app_exp"
export WOOM_CYCLES_BEGIN_DATE=""
export WOOM_CYCLES_END_DATE=""
export WOOM_CYCLES_ROUND=""
export WOOM_CYCLES_FREQ=""
export WOOM_CYCLES_NCYCLES="0"
export WOOM_CYCLES_INDEP="0"
export WOOM_CYCLES_AS_INTERVALS="1"
export WOOM_CYCLES_HORIZON=""
export WOOM_WORKFLOW_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task"
export WOOM_LOG_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/log"
export WOOM_CYCLE="prolog"
export WOOM_MEMBER=""
export WOOM_TASK="mytask"
export WOOM_TASK_PATH="my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_TASK_RUN_DIR="/home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_TASK_SUBMISSION_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_TASK_SCRIPT_PATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh"
export WOOM_SUBMISSION_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_SCRIPT_PATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh"
export WOOM_TASK_ENV="myenv"
export WOOM_ENV="myenv"
export WOOM_TASK_CONTEXT_JSON="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/context.json"
export WOOM_CONTEXT_JSON="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/context.json"
export WOOM_PARAMS_PARAM1="foo"
export WOOM_PARAMS_PARAM2="bar"
export WOOM_PARAMS_PARAM3="TOTO"
export WOOM_PARAMS_NAMES="uo:vo"
export WOOM_PARAMS_MYDATE="06/04/2010"
export PATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/bin:$PATH
export PYTHONPATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/lib/python:$PYTHONPATH
export LIBRARY_PATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/lib:$LIBRARY_PATH
export INCLUDE_PATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/include:$INCLUDE_PATH


# Go to run dir
{ mkdir -p /home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask; cd /home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask; } || exit 1

# Run the commandline(s)
echo running mytask --name uo --name vo foo ; echo mytask BAR Friday 04 June; sleep 20
--------------------------------------------------
woom.workflow: INFO     Submitted task: prolog/prolog0/mytask with job id 330116
woom.workflow: INFO     Successfully submitted stage: prolog
woom.workflow: DEBUG    Entering stage: cycles
woom.workflow: DEBUG    No sequence of task. Skipping...
woom.workflow: DEBUG    Entering stage: epilog
woom.workflow: DEBUG    No sequence of task. Skipping...
woom.workflow: DEBUG    Terminating non-blocking jobs
woom.cli    : INFO     Successfully ran the workflow!

Normal run#

And finally in run it.

$ woom run --log-no-color
woom.cli    : INFO     No extension to load
woom.cli    : INFO     Loaded workflow config
woom.cli    : INFO     App name: my_app_name
woom.cli    : INFO     App conf: my_app_conf
woom.cli    : INFO     App exp: my_app_exp
woom.cli    : INFO     Initialized the host manager
woom.cli    : INFO     Loaded hosts config file: hosts.cfg
woom.cli    : INFO     Infered host: local
woom.cli    : INFO     Initialized the task manager
woom.cli    : INFO     Loaded the task config file: tasks.cfg
woom.cli    : INFO     Initialized the workflow
woom.cli    : INFO     Successfully setup the workflow!
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.job    : INFO     Started job manager: BackgroundJobManager()
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.workflow: INFO     Created batch script: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/latest/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh
woom.workflow: INFO     Submitted task: prolog/prolog0/mytask with job id 937
woom.workflow: INFO     Successfully submitted stage: prolog
woom.cli    : INFO     Successfully ran the workflow!

Check status#

Check what is running or finished.

$ woom show status
╭──────────┬─────────┬────────┬─────────┬───────────────────────────────────────────────────────╮
│ STATUS   │   JOBID │ TASK   │ CYCLE   │ SUBMISSION DIR                                        │
├──────────┼─────────┼────────┼─────────┼───────────────────────────────────────────────────────┤
│ RUNNING  │     937 │ mytask │ prolog  │ jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask │
╰──────────┴─────────┴────────┴─────────┴───────────────────────────────────────────────────────╯

Show run directories#

Show where tasks were executed.

$ woom show run_dirs
╭────────┬─────────┬───────────────────────────────────────────────────────────────────────────────╮
│ TASK   │ CYCLE   │ RUN DIR                                                                       │
├────────┼─────────┼───────────────────────────────────────────────────────────────────────────────┤
│ mytask │ prolog  │ /home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask │
╰────────┴─────────┴───────────────────────────────────────────────────────────────────────────────╯

A single task with command line formatting

Contents