Databricks Clusters

User has options to create interactive clusters, perform actions like start, refresh, edit and delete clusters, view logs and redirect to spark UI.

Given below is an illustration of the Databricks Interactive Clusters page followed by the steps to create a cluster.

Databricks_Interactive

On the listing page under Action tab, user has options to start the cluster. Upon clicking the ellipses, under Action tab the available options are: Restart, Edit, User Permissions and Delete.

User has option to manage permissions of the existing Databricks clusters. Upon clicking User Permissions, the below screen will open:

Cluster_list_view_UserPermission

There are four permission levels for a cluster: No Permissions, Can Attach To, Can Restart, and Can Manage thereby enabling the cluster to be accessed by a set of users with specific permissions.

Note: User must comply with the below mentioned prerequisites for User Permissions in Databricks account:

  • Admin to enable below mentioned toggles for the workspace in its settings:

  • Under Access Control, ‘Control Jobs, and pools Access Control’ toggle must be enabled.

  • Under Access Control, ‘Cluster Visibility Control’ option must be enabled to allow listing of only those clusters to the user for which it has any permission.

  • Access control is available only in the Premium plan. To see the plans, refer to AWS Pricing.

On the listing page, click CREATE CLUSTER to create Databricks cluster. Provide the below fields:

FieldDescription
Cluster PolicyOption to select the cluster policy created in Databricks to configure cluster that defines limits on the attributes available during the cluster creation. Default value is unrestricted. User have an option to download the cluster policy in the local system.
Cluster NameProvide a unique name for the cluster.
Cluster Mode

Select Cluster node from the below options available:

-Standard

-Single Node

Databricks Runtime VersionProvide the Databricks runtime version for the core components that run on the clusters managed by Databricks.

The below option will be available upon selecting Single Node as Cluster Mode

Node TypeProvide the node type, if Single Node option is selected as Cluster Mode.
Driver Type

Select the driver type from the drop down list. The available options are:

- Same as worker

- Pool

- Memory Optimized

Worker TypeSelect the worker type from the drop down list.
Terminate AfterProvide value in minutes to terminate the cluster due to inativity.

The below options will be visible upon selecting Standard as Cluster Mode

Enable auto-scalingThis option is unchecked by default. Check the checkbox to enable auto-scaling the cluster between minimum and maximum number of nodes.

Provide value for below fields if auto scaling is enabled:

Minimum WorkersProvide value for minimum workers.
Maximum WorkersProvide value for maximum workers.
WorkersIf auto scaling is unchecked, then provide value for workers.
On-demand/Spot CompositionProvide the value for on-demand number of nodes.
Spot fall back to On-demandCheck the option to use the Spot fall back to On-demand option. If the EC2 spot price exceeds the bid, use On-demand instances instead of spot instances.

Databricks Interactive Clusters

Given below is an illustration of the Databricks Interactive Clusters page followed by the steps to create a cluster.

Databricks_Interactive

Click CREATE CLUSTER to create Databricks cluster. Provide the below fields:

FieldDescription
Cluster NameProvide a unique name for the cluster.
Cluster Mode

Select Cluster node from the below options available:

-Standard

-Single Node

Databricks Runtime VersionProvide the Databricks runtime version for the core components that run on the clusters managed by Databricks.

The below option will be available upon selecting Single Node as Cluster Mode.

Node TypeProvide the node type, if Single Node option is selected as Cluster Mode.
Driver TypeSelect the driver type from the drop down list.
Worker TypeSelect the worker type from the drop down list.
Terminate AfterProvide value in minutes to terminate the cluster due to inativity.

The below options will be visible upon selecting Standard as Cluster Mode.

Enable auto-scalingThis option is unchecked by default. Check the checkbox to enable auto-scaling the cluster between minimum and maximum number of nodes.

Provide value for below fields if auto scaling is enabled.

Minimum WorkersProvide value for minimum workers.
Maximum WorkersProvide value for maximum workers.
WorkersIf auto scaling is unchecked, then provide value for workers.
On-demand/Spot CompositionProvide the value for on-demand number of nodes.
Spot fall back to On-demandCheck the option to use the Spot fall back to On-demand option. If the EC2 spot price exceeds the bid, use On-demand instances instead of spot instances.

Databricks Job Clusters

Given below is an illustration of the Databricks Job Clusters page.

Databricks_Jobs

Top