Skip to main content
Version: 3.1

2.6 Workflows

Workflows enable the automation of data processing tasks in the Data Context Hub. They allow the user to define sequences of operations that can be executed manually or scheduled to run at regular intervals.

A Workflow consists of multiple steps that are executed in order:

Workflows support full versioning, allowing the user to create draft versions for testing before publishing stable versions for production. Once published, workflows can be scheduled with configurable intervals, retries, and email notifications. The system integrates with Apache Airflow for workflow orchestration.

Create a Workflow​

  1. Click the + Add new button to create a new workflow.

  2. Click the New workflow field and enter a title for the workflow.

  3. Click the Enter an alias field and enter an alias for the workflow.

  4. Click Save to save the workflow.

Execute Workflows​

After creating a workflow, you can execute it manually, upload it to Airflow, or duplicate it for reuse.

Note: Navigate to the Workflows section from the sidebar and open a Workflow to access these options.

Upload DAG​

  1. Click Upload DAG to upload a new DAG (Directed Acyclic Graph) to Airflow for orchestration.

Execute Workflow​

  1. Click Execute to execute the workflow. The workflow will be queued in the Queue Jobs and processed accordingly.

Duplicate Workflow​

  1. Click Duplicate workflow to create a copy of the workflow. This is useful for creating variations or testing changes.

Configure Workflow Settings​

Configure how the workflow behaves, including activation status, notifications, and retry logic.

Activation Status​

  1. Click the Active toggle to activate or deactivate the workflow. Inactive workflows will not execute on their schedule.

Notifications​

  1. Click the Enter notification email field and enter an email address to receive notifications about workflow execution status.

Retries​

Configure how many times the workflow should retry on failure.

  1. Click the Enter number of retries field and enter the number of retries.

  2. Alternatively, use the arrows to set the number of retries.

Scheduling​

Set up automated execution schedules for the workflow.

Set Schedule​

  1. Click Set to set up a schedule for the workflow. The workflow will be repeatedly executed according to the specified interval.

  2. Change the schedule type (daily, weekly, monthly, etc.).

  3. Enter the timestamps for running the workflow.

  4. Click Set to confirm the schedule.

Version Management​

Workflows support full versioning, allowing you to test draft versions before publishing stable versions for production.

View Versions​

  1. Click Versions to switch to the Versions tab and view all versions of the workflow.

Publish Version​

  1. Click Publish version to publish a version.

    Important: When a version of a workflow is published, the schedule for the workflow is activated and it moves to the Published tab of workflows.

  2. The published version is displayed in green after it is published.

Delete Version​

  1. Click Delete version to delete a version.

  2. Confirm deleting the version.

Delete Workflow​

  1. Click Delete workflow to delete a workflow.

  2. Click Delete to confirm the deletion.