Skip to main content
Version: 3.2

2.8 Workflows

Workflows automate DCH's data processing. Instead of running a Build by hand each time, you assemble a sequence of steps on a canvas and let DCH run it, either on demand or on a schedule.

A Workflow is built from three kinds of step:

The colors match the step types on the canvas.

You place steps on the canvas and connect them into an order. DCH turns the finished Workflow into an Apache Airflow pipeline, which carries out the run.

Manage workflows from the Workflows section in the sidebar. The list is split into Published and Draft tabs, and each entry shows the Workflow's name, its number of steps, and its current version.

The Workflows list with its Published and Draft tabs, and the workflow canvas on the right

Create a Workflow​

Choose Add New, give the Workflow a Title, and save. DCH derives a Slug from the title (a unique identifier made of lower-case letters, digits and hyphens) that you can edit. A new Workflow starts as a Draft.

Open a Workflow to edit it. The left panel has a Setup tab and a Versions tab; the canvas on the right is where you build the step sequence.

The Workflow editor: the Setup panel on the left and the step canvas on the right showing two connected steps, with the color-coded step palette along the bottom

Build the step sequence​

Add steps from the palette at the bottom of the canvas (Build, Memory, Rule transformation), then draw connections between them to set the order they run in. Open a step to configure it; each step has its own page.

Every step shares two controls:

  • Enabled: whether the step runs. Turn it off to skip a step without removing it.
  • Run if: when the step runs, based on the steps before it:
    • All succeeded: run only if every preceding step succeeded.
    • All done: run once every preceding step has finished, whether they succeeded or not.

Settings​

The Setup tab holds the Workflow's settings:

  • Active: whether the Workflow runs on its schedule. An inactive Workflow is paused and does not run automatically, though you can still run it by hand.
  • Workflow slug: the unique identifier. It may contain only lower-case letters, digits and hyphens, and may not start or end with a hyphen.
  • Notification email: an address that is emailed when a run fails or retries.
  • Number of retries: how many times a failed run is retried.
  • Interval: how often the Workflow runs, set as a cron expression (for example 0 2 * * * runs every day at 02:00). The interval picker offers common presets (hourly, daily, weekly, monthly, yearly), and you can enter any cron expression for a custom schedule. Choose Set to apply it.

Versions​

A Workflow keeps a history of versions. Editing always changes the current draft; earlier versions stay frozen as snapshots.

  • Draft: the working copy you edit. A draft is never scheduled.
  • Published: the live version. Only one version is published at a time.
  • Archived: a previously published version, kept for reference.

The Versions tab lists them, each with its actions:

  • Publish version: make a draft the published version. The previously published version is archived, and DCH updates the schedule.
  • Execute: run a version immediately as a queued job, which you can follow in Queue Jobs. You can execute a draft as well as the published version.
  • Sync: regenerate the published Workflow's Airflow pipeline. Use it after a change that affects how the Workflow is scheduled.
  • Delete version: remove a version. At least one version must remain.

The Versions tab: the draft, published and archived versions, with the draft's Execute, Publish version and Delete version actions

note

A Workflow runs on its schedule only when a version is published and the Workflow is Active. Publishing decides which version is live; Active decides whether it runs automatically.

Duplicate or delete a Workflow​

  • Duplicate workflow: create a copy as a new draft, a convenient starting point for a variation.
  • Delete workflow: remove the Workflow. You are asked to confirm first.