Create a Workflow

To create a workflow do as follows:

Go to Workflow option in your Project and click on Create New Workflow button.

You will be redirected to Workflow Definitionpage.

A Workflow tile has the following fields:

Define a new Workflow here.

This page has nodes to create and execute the workflow. They are explained below:

Nodes #

To define a workflow, four nodes are available:

• Control Nodes

• Pipelines

• Actions

• Workflows

Add Control node with one or multiple pipelines, with actions applied on it. Save your workflow. Once the Workflows are saved, you can also concatenate Workflows from the Workflow tab.

Control Nodes #

Two types of control nodes are available:

Pipelines #

Batch Pipelines created in the workspace are reflected here.

Actions #

Action nodes are available to provide functionality of the following actions to a workflow.

Following are the Actions available under Action node:

• Assignment Operator

• Bash Operator

• Email

• HDFS Sensor

• HTTP Operator

• Kafka Alert Operator

• MSSQL Operator

• MySQL Operator

• Pause Operator

• Pipeline Operator

• POSTGRES Operator

• Custom Python Operator

• SFTP Operator

• SFTP Sensor

• SQL Sensor

• SSH Operator

• Time Delta Sensor

• Time Window Operator

• Vertica Operator

  

Each action node is explained below:

Assignment Operator #

This operator is used to assign workflow level variable and its value.

You can define multiple variables and their values by clicking on + ADD VARIABLE button. Make sure the values of Variable are of python data type.

BASH Operator #

This node is used to execute bash script, command or set of commands. Following are the configuration properties of Bash Operator.

Email #

This node is used for sending email. The details provided are redirected to the mail server details given in the Airflow configuration during installation.

Set the following configurations for this node.

HDFS Sensor #

HDFS Sensor is used to check whether the given location (of the file or folder path) lands on HDFS (HDFS Sensor). If the sensor finds a given location on HDFS in the given time interval, then it will be considered successful otherwise failed.

Following are the configurations for this node.

In the above configuration, once HDFSSensor is triggered, it will keep checking location(/sample/file/path) in every 20 seconds for duration of 300 seconds.

HTTP Operator #

This operator is used to hit an end point over HTTP system.

Following are the properties under the configuration:

Kafka Alert Operator #

Kafka alert operator is use to send alert/message to a Kafka topic.

MSSQL Operator #

This operator use to execute SQL statements on a Microsoft SQL database.

MySQL Operator #

This operator is used to execute SQL statement on a MySQL database.

Pause Operator #

The Pause Operator is used to Pause the current workflow. Provide the below configuration details to configure Pause Operator:

Pipeline Operator #

Pipeline operator is used to run selected pipeline. You can select pipelines that needs to run. Function of this operator is same as that of a Pipeline Node.

You have to set the following configurations for this operator.

POSTGRES Operator #

This operator is used to execute SQL statement on a PostgreSQL database.

Custom Python Operator #

This operator allows you to execute custom logic/code in a workflow. You can write custom code in python language and it will be execute by workflow.

Write custom code in a python method and provide method name that should be invoked by workflow. In addition, you can also get, set or update workflow variable in custom logic.

Get variable value:

get_dag_variable(variable_name, variable_type=None)

This method is used to get workflow variable. Arguments are:

variable_name: variable name

variable_type (optional): provide variable type

Set variable value:

set_dag_variable(variable_name, variable_value)

This method is used to set/update workflow variable. Arguments are:

variable_name: provide variable name

variable_value: provide variable value

SFTP Operator #

This operator is used for transferring files from remote host to local or vice a versa

SFTP Sensor #

SFTP Sensor is used to check if a given location (file or folder path) is landed on SFTP or not. If sensor finds given location on SFTP in the given interval, then it will considered successful, otherwise failed.

SQL Sensor #

SQLSensor runs the SQL statement until first cell is in (0,’0’,’’).

It runs the SQL statement after each poke interval until Time-Out interval.

SSH Operator #

SSH operator is use to run commands over remote machine. For that user need to provide following.

Time Delta Sensor #

Waits for a given amount of time before succeeding. User needs to provide configurations.

Time Window Operator #

Time window operator is use to check if current execution is in given time window or not. It also checks if current execution date is in calendar holiday or not. If current execution time is not in given time window or current execution date is in calendar holiday then operator returns False and consider as failed.

Vertica Operator #

This operator is used to execute SQL statement on a Vertica database.

Workflow #

All the Workflows that are created in Gathr for the logged in user are listed on the Workflow home page.

You can add workflow as an operator inside a workflow (similar to pipeline). This workflow will be act as sub-workflow and a separate instance of workflow and will be executed as sub-workflow.

Workflow Configuration #

Create Workflow #

Once a workflow is defined, provide a workflow name and click on Save Workflow button to create a workflow using another workflow. (Workflow Tile).

Once you have defined and created a workflow, following operations user can perform on workflow:

Edit Workflow #

You can edit a workflow, however not when it is active.

Subworkflow #

A workflow can have another workflow in it. This enables the Parent Child marker in the workflows.

A Parent Marker will be shown with a Parent Workflow marker icon on the workflow, similarly for a Child Workflow marker.

On every tile, a count of no. of pipelines and subworkflow is shown upto level 1.

Which means, that if a Workflow has a 2 subworkflows, you will be able to see their names on the tiles. However, if there are more subworkflows in it or pipelines, then you will have to keep clicking and expand them on the workflow definition page. For example, you can view the three subworkflows under t2, but to check the t3, you need to click on it and view the edit mode of the subworkflow.

When a user clicks on the pipeline/workflow count, the pipeline names and creation date appear on the popup. On clicking on an individual pipeline/workflow you will be redirected to its edit view.

In the edit view, you will be able to drill down every workflow till its last level. Double click on any workflow to view the window shown below:

Schedule Workflow #

Once workflow is created, deploy the workflow by clicking on SCHEDULEbutton. Set your workflow scheduling configuration as defined below:

When the user selects Schedule Interval as None then the Workflow can be Triggered as shown above.

Un-schedule Workflow #

To remove a workflow from a schedule, click on the SCHEDULING option. This option is available on the Workflow tile under the menu button.

A new window will pop up and you can un-schedule the workflow.

You can always Reschedule the workflow, using the same steps.

Start Workflow #

After scheduling workflow, status of the workflow will change to SCHEDULED. Now, you can start the workflow by clicking on START button. If the workflow starts successfully, the status will change to ACTIVE.

View Summary #

Workflow monitoring feature allows you to:

1. View the workflow run history.

2. View the component-level status and its logs details.

3. Re-trigger a failed task.

4. Re-trigger all the failed tasks with a single click.

The view summary page will take you to the dashboard of workflow monitoring where you can see the last 5 runs and their details.

Upon clicking the Run ID, user can view the workflow summary. If the workflow fails, monitor window will help you resume and run the workflow from the failed stage. Shown below is a Monitor window with a failed task. Right click on the pipeline to view the logs and re-trigger in case of a failed task, as shown in the below images respectively.

As shown above within the Workflow summary DAG window you can perform the below functions:

Pause Workflow #

Once the workflow is in ACTIVE state, you can PAUSE the workflow.

Resume Workflow #

You can RESUME a paused workflow.

Delete Workflow #

Click on Delete button, to delete the workflow.

Test Workflow #

You can test the workflow before it is scheduled.

Once you click on the Test button as shown below, a pop-up with testing configuration will appear. Details of each field is explained below:

Here workflow will consider current date/time (of given time zone) as start date. Once you click on Test button, workflow will be deployed and is instantly scheduled to start.

During testing, Workflow runs only once. You will not be able to perform other operations like Start/Stop/Schedule/Resume. Once workflow testing is complete, the workflow will be ready to be Scheduled.

Once testing is done, test status will be visible on Workflow Tile. It shows whether workflow is succeed or failed.

Connections #

You can create connections in Airflow from Gathr. After clicking on + sign in workflow creation page (Create Connection Icon), a pop-up will appear (Create Connection). Provide information required to create a connection in this tile.

Following types of connection can be created:

• HDFS

• HTTP

• MSSQL

• MySQL

• Postgres

• SFTP

• SSH

• Vertica