Other Workflow Tools¶

QDO¶

(Note that qdo is not actively supported or maintained by NERSC, but is available on our systems.)

QDO (kew-doo) is a toolkit for managing many many small tasks within a larger batch framework. QDO separates the queue of tasks to perform from the batch jobs that actually perform the tasks. This simplifies managing tasks as a group, and provides greater flexibility for scaling batch worker jobs up and down or adding additional tasks to the queue even after workers have started processing them.

QDO was designed by the astrophysics community to manage queues of high throughput jobs. It supports task dependencies, priorities, management of tasks in aggregate, and flexibility such as adding additional tasks to a queue even after the batch worker jobs have started.

The qdo module provides an API for interacting with task queues. The qdo script uses this same API to provide a command line interface that can be used interchangeably with the python API. Run "qdo -h" to see the task line options.

module use /cfs/cdirs/cosmo/software/modules/$NERSC_HOST
module load qdo/0.7

qdo was developed to support workflows in processing images for cosmological surveys. For more information please contact Stephen Bailey in the Cosmology Group at Lawrence Berkeley Lab.

Tigres¶

Tigres provides a C and Python programming library to compose and execute large-scale data-intensive scientific workflows from desktops to supercomputers. We offer a community supported module for the Tigres libraries. For more information, refer to the Tigres documentation.

Spark¶

Apache Spark is a fast and general engine for large-scale data processing.

Swift¶

Swift Overview¶

The Swift scripting language provides a simple, compact way to write parallel scripts that run many copies of ordinary programs concurrently in various workflow patterns, reducing the need for complex parallel programming or scripting. Swift is very general, and is in use in domains ranging from earth systems to bioinformatics to molecular modeling.

Running Swift at NERSC¶

Type the following commands to run a simple Swift script:

% module load swift
% swift -config swift.conf myscript.swift

Swift Tutorial on NERSC Systems¶

This site has a set of excellent introductory tutorials for Swift that run on NERSC systems. On Cori, just remember to type "module load swift" before starting the tutorials.

An introductory talk by Michael Wilde was given at NERSC in December 2015. The slides from this seminar can be found here.