A single task with command line formatting

A single task with command line formatting#

About#

Path: examples/academic/single_task.

Configuring#

workflow.cfg#
[app]
name=my_app_name
conf=my_app_conf
exp=my_app_exp

[params]
param1=foo
param2=bar
param3=toto
names=uo,vo
mydate=06/04/2010

    [[tasks]]
        [[[mytask]]]
        param3=TOTO

[env_vars]
MY_ENV_VAR=value

[stages]

    [[prolog]]
    prolog0=mytask
tasks.cfg#
[mytask]

    [[content]]
    env=myenv
    commandline="echo running mytask {{ names|replicate_option('--name', format='{opt_name} {value}') }} {{ param1 }} ; echo mytask {{ param2|upper }} {{ mydate|strftime('%A %d %B') }}; sleep 20"

    [[submit]]
    queue=seq
hosts.cfg#
[local]
#patterns=*
    [[envs]]
        [[[myenv]]]
            [[[[vars]]]]
                [[[[[set]]]]]
                MY_ENV_VAR=my_env_var_value_background

[datarmor]
patterns=datavisu*,*.ice.ifremer.fr,datarmor*
scheduler=pbspro
module_setup=. /etc/profile.d/modules.sh

    [[queues]]
    seq=sequentiel

    [[dirs]]
    scratch=$SCRATCH

    [[envs]]
        [[[myenv]]]
            [[[[vars]]]]
                [[[[[set]]]]]
                MY_ENV_VAR=my_env_var_value_datarmor

Running#

Overview#

Let’s have an overview of stages before running the workflow.

$ woom show overview
##################################### APP ######################################
name: my_app_name
conf: my_app_conf
exp: my_app_exp
################################## TASK TREE ###################################
prolog:
    - prolog0: mytask
#################################### CYCLES ####################################
No cycle
################################### ENSEMBLE ###################################
no member

Dry run#

Now let’s run the workflow in test (dry) and debug modes.

$ woom run --log-no-color --log-level debug --dry-run
woom.log    : DEBUG    *** STARTED LOG SESSION ***
woom.cli    : DEBUG    Run the workflow
woom.cli    : DEBUG    Loading extensions
woom.cli    : INFO     No extension to load
woom.cli    : DEBUG    Load workflow config: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/workflow.cfg
woom.cli    : INFO     Loaded workflow config
woom.cli    : INFO     App name: my_app_name
woom.cli    : INFO     App conf: my_app_conf
woom.cli    : INFO     App exp: my_app_exp
woom.cli    : DEBUG    Initialize the host manager
woom.cli    : INFO     Initialized the host manager
woom.cli    : DEBUG    Load hosts config file: hosts.cfg
woom.cli    : INFO     Loaded hosts config file: hosts.cfg
woom.cli    : DEBUG    Infer host
woom.cli    : INFO     Infered host: local
woom.cli    : DEBUG    Initialize the task manager
woom.cli    : INFO     Initialized the task manager
woom.cli    : DEBUG    Load the task config file: tasks.cfg
woom.cli    : INFO     Loaded the task config file: tasks.cfg
woom.cli    : DEBUG    Initialize the workflow
woom.workflow: DEBUG    Task tree:
prolog:
    - prolog0: mytask
woom.cli    : INFO     Initialized the workflow
woom.cli    : INFO     Successfully setup the workflow!
woom.cli    : DEBUG    Run the workflow
woom.workflow: DEBUG    Running the workflow in fake mode
woom.workflow: DEBUG    Entering stage: prolog
woom.workflow: DEBUG    Entering sequence: prolog0
woom.workflow: DEBUG    Running task: prolog/prolog0/mytask
woom.workflow: DEBUG    Cleaning task: prolog/prolog0/mytask
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: DEBUG    Submitting task: prolog/prolog0/mytask
woom.workflow: DEBUG      Dependencies: 
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: DEBUG    Creating directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.job    : INFO     Started job manager: BackgroundJobManager()
woom.workflow: DEBUG    Fake submission:
--------------- submission command ---------------
bash /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh
-------------- batch script content --------------
#!/bin/bash

# Prolog
set -eo pipefail
on_exit() {
    status=$?
    echo $status > "$WOOM_SUBMISSION_DIR/job.status"
    exit $status
}
trap on_exit EXIT

# Environment variables
export WOOM_SCRATCH_DIR="/home/docs/woom/scratch"
export WOOM_WORK_DIR="/home/docs/woom/work"
export MY_ENV_VAR="value"
export WOOM_TASK_NAME="mytask"
export WOOM_RUN_DIR="/home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_PARAM1="foo"
export WOOM_PARAM2="bar"
export WOOM_PARAM3="TOTO"
export WOOM_NAMES="uo:vo"
export WOOM_MYDATE="06/04/2010"
export WOOM_APP_NAME="my_app_name"
export WOOM_APP_CONF="my_app_conf"
export WOOM_APP_EXP="my_app_exp"
export WOOM_CYCLES_BEGIN_DATE=""
export WOOM_CYCLES_END_DATE=""
export WOOM_CYCLES_ROUND=""
export WOOM_CYCLES_FREQ=""
export WOOM_CYCLES_NCYCLES="0"
export WOOM_CYCLES_INDEP="0"
export WOOM_CYCLES_AS_INTERVALS="1"
export WOOM_APP_PATH="my_app_name/my_app_conf/my_app_exp"
export WOOM_TASK_PATH="my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_NMEMBERS="0"
export WOOM_WORKFLOW_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task"
export WOOM_SUBMISSION_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask"
export WOOM_LOG_DIR="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/log"
export WOOM_SCRIPT_PATH="/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh"
export PATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/bin:$PATH
export PYTHONPATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/lib/python:$PYTHONPATH
export LIBRARY_PATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/lib:$LIBRARY_PATH
export INCLUDE_PATH=/home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/include:$INCLUDE_PATH

# Go to run dir
mkdir -p /home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask && cd /home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask

# Run the commandline(s)
"echo running mytask --name uo --name vo foo ; echo mytask BAR Friday 04 June; sleep 20"
--------------------------------------------------
woom.workflow: INFO     Submitted task: prolog/prolog0/mytask with job id 690700
woom.workflow: INFO     Successfully submitted stage: prolog
woom.workflow: DEBUG    Entering stage: cycles
woom.workflow: DEBUG    No sequence of task. Skipping...
woom.workflow: DEBUG    Entering stage: epilog
woom.workflow: DEBUG    No sequence of task. Skipping...
woom.cli    : INFO     Successfully ran the workflow!

Normal run#

And finally in run it.

$ woom run --log-no-color
woom.cli    : INFO     No extension to load
woom.cli    : INFO     Loaded workflow config
woom.cli    : INFO     App name: my_app_name
woom.cli    : INFO     App conf: my_app_conf
woom.cli    : INFO     App exp: my_app_exp
woom.cli    : INFO     Initialized the host manager
woom.cli    : INFO     Loaded hosts config file: hosts.cfg
woom.cli    : INFO     Infered host: local
woom.cli    : INFO     Initialized the task manager
woom.cli    : INFO     Loaded the task config file: tasks.cfg
woom.cli    : INFO     Initialized the workflow
woom.cli    : INFO     Successfully setup the workflow!
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog
woom.workflow: INFO     Created directory: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask
woom.workflow: INFO     Created batch script: /home/docs/checkouts/readthedocs.org/user_builds/woom/checkouts/stable/examples/academic/single_task/jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask/job.sh
woom.job    : INFO     Started job manager: BackgroundJobManager()
woom.workflow: INFO     Submitted task: prolog/prolog0/mytask with job id 884
woom.workflow: INFO     Successfully submitted stage: prolog
woom.cli    : INFO     Successfully ran the workflow!

Check status#

Check what is running or finished.

$ woom show status
╭──────────┬─────────┬────────┬─────────┬───────────────────────────────────────────────────────╮
│ STATUS   │   JOBID │ TASK   │ CYCLE   │ SUBMISSION DIR                                        │
├──────────┼─────────┼────────┼─────────┼───────────────────────────────────────────────────────┤
│ ERROR    │     884 │ mytask │ prolog  │ jobs/my_app_name/my_app_conf/my_app_exp/prolog/mytask │
╰──────────┴─────────┴────────┴─────────┴───────────────────────────────────────────────────────╯

Show run directories#

Show where tasks were executed.

$ woom show run_dirs
╭────────┬─────────┬───────────────────────────────────────────────────────────────────────────────╮
│ TASK   │ CYCLE   │ RUN DIR                                                                       │
├────────┼─────────┼───────────────────────────────────────────────────────────────────────────────┤
│ mytask │ prolog  │ /home/docs/woom/scratch/woom/my_app_name/my_app_conf/my_app_exp/prolog/mytask │
╰────────┴─────────┴───────────────────────────────────────────────────────────────────────────────╯
Contents