Pipeline Listing Page

If you are a new user and haven’t yet created a pipeline, then the Pipeline page displays an introductory screen. Click the Create a Pipeline button at top right side of the screen.

If you have already created pipelines, then the Pipeline listing page displays all the existing pipelines.

The Pipeline listing page displays Node Managers, Cores and Memory utilized.

Here, Node manager is the count of node through which the cluster is formed, cores reflect the number of cores used in workspaces/project and memory used by project/across workspace.

Next to these are the buttons to Upload Pipeline, Download Sample Project, Refresh Data, Create New Pipeline, Bulk Actions (enabled once the pipeline(s) are selected by checking the checkbox on the particular pipeline tile), Select all for bulk actions (Click to select all pipelines to execute bulk actions), List View and Grid View.

You can create a new pipeline, download sample project and upload a pipeline from your local system.

The listing page has the option to easily search and navigate to the pipeline by the below search options:

• You can search the pipeline directly by providing the name of the pipeline in the search bar and click enter.

• Alternatively, you can search the pipelines by using the Status search bar. Click the drop-down and select one of the available options: All, Active, Error, Starting, Stopped.

• You can also search the pipelines using the Sort Type search bar. click the drop-down and select one of the available options: Last Modified, Name.

• You can use the Sort search bar and click the drop-down to select one of the options: ASC or DESC.

The pipeline records can be search upto 500 pages from the Records drop-down option. Arrow button options are available to navigate to previous page and next page respectively.

Bulk Actions

Multiple pipelines can be started, stopped/deleted using bulk operations functionality in Gathr.

You can bulk Start the selected pipeline. Click on the pipeline(s) tile checkbox (top left). Select the Bulk Actions ellipses option available on the pipeline listing page (top-right). Options available are:

Bulk Start

Option to start all the selected pipelines. You can Bulk Start the selected pipeline by clicking on the pipeline(s) tile checkbox (top left).

Bulk Stop

Option to stop all the selected pipelines. You can Bulk Stop the selected pipeline by clicking on the pipeline(s) tile checkbox (top left).

Bulk Delete

Option to delete all the selected pipelines. You can bulk Delete the selected pipeline by clicking on the pipeline(s) tile checkbox (top left). The delete option will not work if pipeline is in active state and used in a Workflow.

You also have an option to select/de-select all for bulk actions next to ellipses option for Bulk Actions.

All the pipeline are listed.

You can navigate to Templates listing page by clicking at the Template tab.

Template are the pipelines in which user can add variables and parameters. Instance(s) are created within the template once these value(s)/parameters are added. To know more, see Templates Introduction →

The Pipeline listing page will display all the batch and streaming data pipeline tiles. User can create a pipeline, download sample project and upload pipeline from options available at the top right of the listing page.

For AWS, the pipelines can be deployed on Databricks and EMR.

The below image shows actions that can be performed on the pipeline upon clicking the pipeline tile ellipses.

Within the Data Pipeline, user can perform the following actions:

• Design, Import and Export real-time data pipeline.

• Deploy pipelines on AWS Cloud platform.

• Deploy pipelines on either Cloud vendor-Databricks or Amazon EMR.

• Drag, drop and connect operators to create applications.

• Create Datasets and reuse them in different pipelines.

• Create Models from the analytics operators.

• Monitor detailed metrics of each task and instance.

• Run PMML-based scripts in real-time on every incoming message.

• Explore real-time data using interactive charts and dashboards.

There could be a business scenario where the pipeline design remains the same for multiple pipelines, yet the configurations differ.

For an e.g. A user has 3 different Kafka topics containing similar data that needs to go through same kind of processing (Kafka –> Filter –> ExpressionEvaluator –> HDFS). Normally, a user would create two clones of the same pipeline and edit the configuration.

The easier way of doing this is now user can create a pipeline template that comprises of the pipeline structure and multiple instances which only differ in configuration and inherit the pipeline structure of the pipeline template.

Download Configuration Template

The user can use the Download Configuration Template option to download a json file that contains all the component configurations used in a data pipeline. This will be used to create an instance.

Create Instance #

The user will modify the downloaded json file using ‘Download Configuration Template’ option to apply configuration changes for his first instance. For eg. The user changes the topic name in the Kafka channel and saves the modified json file.

The user can create the first instance by clicking the ellipses of the pipeline tile and click View Instance.

On clicking ‘View Instance’ a page opens where user can create an instance by providing instance name and provide configuration used for pipeline instance creation. Provide a name and paste the content of the modified json file that has the amended Kafka topic name.

Click on ‘CREATE’ button to create instance or optionally ‘CREATE & LAUNCH’ to create and launch the instance.

The user can create a template in the data pipeline listing page. Thus, a template can have multiple pipeline instances running using the same set of components and different configurations.

The user can create further instances and view existing instances as well:

Start/Stop Template #

The user can Start/Stop the template by clicking at the Start/Stop button available at the template. All the instances must be deleted to delete the template.

Once the user selects Start, the user will be required to select the instance(s):

Click Start.

Likewise, to stop the instances, click Stop.

Configure Job #

Upon clicking the ellipses of pipeline tile, you can configure job by clicking on the Configure Job option. To configure a job for IBM Spectrum Conductor, below options are provided:

SPARK CONDUCTOR EGO PROPERTIES

IBM Conductor EGO Configuration

To add EGO Environment Variable, click the ADD ENVIRONMENT VARIABLE button.

Option to ADD CONFIGURATIONS is also available.

Example, any extra environment and spark job configuration attributes could be provided in these ( examples spark.ego.master.ui.retained.executor.num=500 or spark.ego.submit.file.replication=10 , Default value is the value of dfs.replication in Hadoop core-site.xml ) and Gathr manages these values while submitting job on IBM conductor Instance Group of selected cluster.

Click CONFIRM once details are provided in the Configure Job window.

Configure GCP cluster in data pipeline #

You can attach the GCP cluster in your data pipeline for its smooth execution. You can choose either existing long running cluster or a new job cluster for executing pipeline.

Click the Ellipses of the pipeline tile on the pipeline listing page and select Configure Job option.

There are two tabs on the Configure Job window. ie., Cluster and Deployment.

The below options are available under the Cluster tab:

Software Configuration

Labels

Master Nodes

Worker Nodes

Secondary Worker Nodes

Initialization Actions

Click CONFIRM button.

Under the Deployment tab, the below options are available:

Click CONFIRM button.