Extending woom

One can extend the capabilities of woom by supplying specific files.

Jinja filters

Jina is used to perform string substitutions to generate the task scripts to be submitted as jobs. This process supports filtering that transforms you data after converting it to string. Jinja builtin filters are available here. Woom provides a few filters in the woom.render module.

You can add you own filter functions with the following procedure:

1. Create a python file named jinja_filters.py located in the ext/ folder of your workflow directory.

2. Declare in this file all the jinja filter functions you want.

3. Declare a dictionary named JINJA_FILTERS to register your functions.

Example:

examples/academic/ensemble/ext/jinja_filters.py

import string


def filter_member2letter(member):
    """Convert a int to an uppercase letter"""
    return string.ascii_uppercase[member.id - 1]


JINJA_FILTERS = {"member2letter": filter_member2letter}

If present, this file is loaded at workflow setup by the woom.ext.load_jinja_filters() function.

Jinja templates

You can extend the default jinja templates by providing yours in the templates directory of the workflow directory.

Examples:

examples/academic/templates/templates/env.sh

{% extends "!env.sh" %}

{% block env_vars -%}
{{ super() }}
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-4}
{% endblock %}

{% block custom %}
# Utilities
function log_message() {
    echo "[$(date +%Y-%m-%d_%H:%M:%S)] $*" | \
        tee -a ${WOOM_LOG_DIR}/custom.log
}

# Initialization
log_message "Environment is initialized for ${WOOM_TASK_NAME}"
{% endblock %}

examples/academic/templates/templates/job.sh

{% extends "!job.sh" %}

{% block header %}
{{ super() }}
# Workflow: {{ app_name|default('N/A') }}
# Configuration: {{ app_conf|default('N/A') }}
# Experiment: {{ app_exp|default('N/A') }}
# Task: {{ task.name }}
{% endblock %}

See Use the templating system.

Configobj specifications

Parameters that are passed to the workflow configuration in the [params] are interpreted as strings by default. You can change this behavior by changing adding your own specification file, and optionally by providing new validator functions that convert text to the desired type.

Workflow configuration specifications

Just pass the commandline option –workflow-ini to give a configuration specification file that will be merged with the default one. For info about how to write such file, refer to the doc. An specification option can refer to an existing validator function like boolean or a new one (see next section).

Example:

examples/academic/ensemble/workflow.ini

[ensemble]
    [[iters]]
    ks=random_lognormal(default=None)

Validator functions

You can add new validator functions by following the procedure:

1. Create a python file named validator_functions.py located in the ext/ folder of your workflow directory.

2. Declare in this file all the validator functions you want.

3. Declare a dictionary named VALIDATOR_FUNCTIONS to register your functions.

Example:

examples/academic/ensemble/ext/validator_functions.py

import random


def random_lognormal(specs):
    if str(specs) == "None":
        return
    mu = float(specs[0])
    sigma = float(specs[1])
    size = int(specs[2])
    return [random.lognormvariate(mu, sigma) for i in range(size)]


VALIDATOR_FUNCTIONS = {"random_lognormal": random_lognormal}

If present, this file is loaded at workflow setup by the woom.ext.load_validator_functions() function.

Artifact paths generators

You can provide function names instead of paths in your task artifacts section to dynamically generate a list of paths in a given context.

Such a function must accept and use all objects of the current context, so its signature should typically end with a **kwargs.

1. Create a python file named artifacts_generators.py located in the ext/ folder of your workflow directory.

2. Declare in this file all the jinja functions you want to generate paths.

3. Declare a dictionary named ARTIFACTS_GENERATORS to register your functions.

4. Declare your function use in the :file`tasks.cfg` file.

In the following example, the function takes the cycle argument as input to generate daily files for the associated month:

ext/artifacts_generators.py

import pandas as pd

def gen_daily_files(**kwargs):
    date = kwargs["cycle"].date # provided by the context
    path_format = kwargs["path_format"] # provided by the tasks.cfg file
    day0 = date.to_period("M").to_timestamp()
    day1 = day0 + pd.offsets.MonthBegin()
    days = pd.date_range(day0, day1, inclusive="left")
    paths = []
    for day in days:
        paths.append(path_format.format(day))
    return paths

ARTIFACTS_GENERATORS = {"gen_daily_files": gen_daily_files}

We declare the function use in the tasks.cfg file and specify the path_format argument:

tasks.cfg

[run_croco]
...
    [[artifacts]]
        [[[outputs]]]
        paths=gen_daily_files
            [[[[[kwargs]]]]]
            path_format={run_dir}/croco.{day:%Y-%m-%d}.nc