Manage Google Cloud Dataproc Clusters

GCP Dataproc services are used to manage Dataproc cluster(s) from the Gathr application.

The Cluster List View is a feature in Gathr where superuser and workspace users can manage Google Dataproc clusters.

Create Cluster #

From the main menu navigate to the Cluster List View page.

To create a cluster click the Create New Cluster option.

On the Create Cluster window, the below options are available:

Other configuration options available are explained below:

Software Configuration

Labels

Master Nodes

Worker Nodes

Secondary Worker Nodes

Initialization Actions

The below fields are stored with their default values in Gathr metastore and will be auto-populated while creating the cluster in Gathr.

• Region
• Primary Network
• Sub Network
• Internal IP Only
• Enter Configuration
• Initialization Actions
• Labels

These values can be updated as per your requirements either from Gathr UI manually or by update query as mentioned below.

You can modify the below query as per your requirement to update default fields:

UPDATE gcp_cluster_default_config set default_config_json = '{"internalIpOnly":"","subnetworkUri":"","region":"","executableFile":"","properties":[{"classification":"yarn","properties":{"yarn.scheduler.capacity.root.default.maximum-am-resource-percent":"0.50","yarn.log-aggregation.enabled":"true"}},{"classification":"dataproc","properties":{"dataproc.scheduler.max-concurrent-jobs":"5","dataproc.logging.stackdriver.enable":"true","dataproc.logging.stackdriver.job.driver.enable":"true","dataproc.logging.stackdriver.job.yarn.container.enable":"true","dataproc.conscrypt.provider.enable":"false"}},{"classification":"spark","properties":{"spark.yarn.preserve.staging.files":"false","spark.eventLog.enabled":"false"}}],"networkUri":"","labels":{}}'

Upon clicking SAVE button the cluster will be saved on database but it will not be launched.

You can click SAVE AND CREATE button to save the cluster on database and create the cluster on dataproc.

Cluster(s) Listing page #

On the listing page, under the Long Running Clusters tab, the below details are shown for all the long running clusters:

You can view the details of the created cluster by clicking on the expand button available on the listing page.

Details of the cluster will be available on the listing page as mentioned below:

Other cluster details include:

Start or Stop Cluster #

You can start/stop a cluster that is created by clicking at the Start/Stop option available under the Action tab of listing page.

Clone Cluster #

You can clone a cluster, by clicking on the Clone option under Action tab.

Edit Cluster #

You can edit a cluster, by clicking at the Edit option under Action tab.

Upon clicking Edit option, the Edit Cluster window opens with the following options:

Click SAVE to update the changes of the cluster.

A cluster can be edited/updated only when it is in following state:

• RUNNING
• SAVED
• DELETED

The below fields can be updated while updating the cluster when it is in RUNNING state:

• graceful-decommission-timeout
• num-secondary-workers
• num-workers
• labels
• autoscaling-policy

Delete Cluster #

You can Delete a cluster, by clicking at the Delete option under Action tab.

On a running cluster, if no pipelines are configured and you want to delete the cluster, then you will have two options to delete:

• Delete from GCP, where the cluster will be deleted from GCP and continue to remain in the Gathr database. So, later the same cluster can be started.

• Delete cluster from both GCP and Gathr and the cluster will be removed from both.

Options to Reload GCP Cluster List→, Create New Cluster →, Fetch Cluster from GCP →, Sync Cluster with GCP →,Reconfigure the Existing Pipelines→ are available on the Long Running Clusters page. These are explained below.

Reload GCP Cluster List #

You can reload/refresh the GCP cluster listing by clicking at the Reload GCP Cluster List option available on the Cluster List View page.

Create New Cluster #

You can create a GCP Dataproc cluster by clicking at the Create New Cluster option to run the ETL jobs. For details on creating a cluster click Create Cluster

Fetch Cluster from GCP #

If you create cluster from GCP console, then you have an option to fetch the cluster at Gathr UI using the Fetch Cluster from GCP option.

If you have created a cluster in GCP console and you want to use that cluster in Gathr for running the pipelines, then click the Fetch Cluster from GCP option.

Upon clicking this option, you will be able to view the cluster created in GCP, on Gathr UI and you will be able to register the same cluster in Gathr.

Upon clicking this option, provide the GCP Cluster ID and Click FETCH on the Fetch Cluster window.

Sync Cluster with GCP #

There could be scenarios when you need to sync your gathr cluster with GCP.

When you have a cluster with Deleted or Saved status in Gathr but that cluster is launched in GCP console.

That cluster will not be automatically synced with Gathr as it will be synced only with user request.

Click Sync cluster from GCP button on Gathr to sync your cluster.

Reconfigure the Existing Pipelines #

This option enables you to reconfigure the pipelines from one cluster to another.

In certain scenario this functionality is required if you have to delete a cluster and use another cluster for your pipelines. Or if you want to shift a few pipelines from one or more clusters to another cluster.

To reconfigure pipeline(s), click on the Reconfigure the Existing Pipeline(s) option.

The Migrate Pipelines to other Cluster window opens with below fields:

Job Clusters #

Under the Job Clusters tab, the list of all job clusters are available. You can filter the job clusters based on the options available in the Filters drop-down list:

• ALL
• SAVED
• CREATING
• STARTING
• STOPPING
• STOPPED
• DELETING
• ERROR
• UNKNOWN
• ERROR_DUE_TO_UPDATE
• RUNNING
• UPDATING
• DELETED

Auto Scaling Policy #

Auto scaling option lets you to automatically add or delete virtual machine instances based on increase or decrease in load, thus letting you handle increase in traffic and reduce costs when the need for resources is lower.

Option to Create and manage Auto Scaling Policy is available on the Cluster List View landing page.

Upon clicking the Auto Scaling Policy tab, below options are available:

Create Policy #

Click the + CREATE POLICY option to create a new policy under the Auto Scaling Policy page.

After cluster creation you can Configure GCP cluster in data pipeline → on Gathr.