Workspace Name

Enter a unique name for your workspace.

Version Control

User can select option for Version Control from the drop-down. Currently the below mentioned options are available:

- gathr MetaStore

- Bitbucket

- GitHub

-GitLab

User can opt for gathr MetaStore option (default) for Version Control and proceed further to fill other required details mentioned below.

If user selects Bitbucket option as Version Control, fill the below fields:

Git Login via.

User can log in via. Credentials/SSH Keys.

Upon selecting Credentials option, fill the below fields:

Bitbucket User Name

Provide the Bitbucket user name.

Bitbucket Password

Provide the Bitbucket password. Click the Fetch Workspace button to fetch the Bitbucket Workspace.

Upon selecting SSH Keys option upload the Git SSH Key file.

If user selects GitHub option as Version Control, fill the below fields:

Git Login via.

User can log in via. Credentials/SSH Keys.

Upon selecting Credentials option, fill the below fields:

Git User Name

Provide the GitHub user name.

Git Password

Provide the GitHub access token or password.

If user selects GitLab option as Version Control, fill the below fields:

Git Login via.

User can log in via. Credentials/SSH Keys.

Upon selecting Credentials option, fill the below fields:

Git User Name

Provide the Git user name.

Git Password

Provide the Git password. C

User

A user can be of following types:

Create New User: Enter a user name.

If the user selects Create New User option, fill the below fields:

Username

Enter a username that will be used to log in the Application.

Email Id

Enter an email id that will be used for any communication with the user.

Password

Enter a Password.

Confirm Password

Re-type the password.

Language

Choose the language, from English (US) and German (DE).

Click Create.

If the user selects Existing User option, in that case Assign User by selecting users who can access the workspace.

Token Associated

If the Databricks user token is associated with gathr user or not.

Associate New Token

Associate a new Databricks user token.

Use Existing: Select this if you already have a databricks token_value.

Generate New: Select this option if you want to generate new databricks.

Username

Provide Databricks username with which token_value is generated.

Password

Enter the password for Databricks username.

Token Lifetime (days)

Provide Databricks token_value.

Connect Using

Select AWS Keys from the drop-down if you want to configure EMR cluster using AWS Key ID and Secret Access Key.

AWS Key ID

AWS Key that will be used for all the communication with AWS.

Secret Access Key

Secret key corresponding to AWS key.

User Role

Only a Developer user can be created from within the Workspace.

User Name

Enter a username that will be used to log in the Application.

Email Id

Enter an email id that will be used for any communication with the user.

Password

Enter a Password.

Confirm Password

Re-type the password.

Language

Choose the language, from English (US) and German (DE).

Cluster Policy

Option to select the cluster policy created in Databricks to configure cluster that defines limits on the attributes available during the cluster creation. Default value is unrestricted. User have an option to download the cluster policy in the local system.

Cluster Name

Provide a unique name for the cluster.

Cluster Mode

Select Cluster node from the below options available:

-Standard

-Single Node

Note: The Cluster Mode option is not editable on Databricks interactive cluster.

Databricks Runtime Version

Provide the Databricks runtime version for the core components that run on the clusters managed by Databricks.

The below option will be available upon selecting Single Node as Cluster Mode

Node Type

Provide the node type, if Single Node option is selected as Cluster Mode.

Driver Type

Select the driver type from the drop down list. The available options are:

- Same as worker

- Pool

- Memory Optimized

Note: If the Driver Type option is selected as Pool, then the Worker Type must also be Pool. Once the created cluster is associated with the pool and the cluster is created successfully, the pool can be utilized in configuring a job.

Worker Type

Select the worker type from the drop down list.

Terminate After

Provide value in minutes to terminate the cluster due to inativity.

The below options will be visible upon selecting Standard as Cluster Mode

Enable auto-scaling

This option is unchecked by default. Check the checkbox to enable auto-scaling the cluster between minimum and maximum number of nodes.

Provide value for below fields if auto scaling is enabled

Minimum Workers

Provide value for minimum workers.

Maximum Workers

Provide value for maximum workers.

Workers

If auto scaling is unchecked, then provide value for workers.

On-demand/Spot Composition

Provide the value for on-demand number of nodes.

Spot fall back to On-demand

Check the option to use the Spot fall back to On-demand option. If the EC2 spot price exceeds the bid, use On-demand instances instead of spot instances.

Cluster Name

Provide a unique name for the cluster.

Cluster Mode

Select Cluster node from the below options available:

-Standard

-Single Node

Databricks Runtime Version

Provide the Databricks runtime version for the core components that run on the clusters managed by Databricks.

The below option will be available upon selecting Single Node as Cluster Mode

Node Type

Provide the node type, if Single Node option is selected as Cluster Mode.

Driver Type

Select the driver type from the drop down list.

Worker Type

Select the worker type from the drop down list.

Terminate After

Provide value in minutes to terminate the cluster due to inativity.

The below options will be visible upon selecting Standard as Cluster Mode

Enable auto-scaling

This option is unchecked by default. Check the checkbox to enable auto-scaling the cluster between minimum and maximum number of nodes.

Provide value for below fields if auto scaling is enabled

Minimum Workers

Provide value for minimum workers.

Maximum Workers

Provide value for maximum workers.

Workers

If auto scaling is unchecked, then provide value for workers.

On-demand/Spot Composition

Provide the value for on-demand number of nodes.

Spot fall back to On-demand

Check the option to use the Spot fall back to On-demand option. If the EC2 spot price exceeds the bid, use On-demand instances instead of spot instances.

Cluster Name

Provide a unique name for the cluster.

VPC

Select VPC for the cluster to be launched.

Subnet ID

Select subnet with the cluster to be launched.

Security Group

Select security group for the cluster to be launched.

Security Configuration

Select security configuration from the drop down list of available options. Select None if security configuration is not required.

Service Role

Select the emrIAMRole from the drop-down list of service roles.

Job Flow Role

Select the emrIAMRole from the drop-down list of job flow roles.

Auto Scaling Role

Select the emrIAMRole from the drop-down list of auto scaling roles.

Root EBS Volume (GB)

Provide the root value for the cluster. Default value is 10.

EMR Managed Scaling

Upon checking the option, the EMR will automatically adjust the number of EC2 instances required in core and task nodes based on workload. This option is unchecked by default.

Upon checking the EMR Managed Scaling option, provide values for the below fields:

Minimum Units

Provide the minimum number of core or task units allowed in a cluster. Minimum value is 1.

Maximum Units

Provide the maximum number of core or task units allowed in a cluster. Minimum value is 1.

Maximum On-Demand Limit

Provide the maximum allowed core or task units for On-Demand market type in a cluster. If this parameter is not specified, it defaults to maximum units value. Minimum value is 0.

Maximum Core Units

Provide the maximum allowed core nodes in a cluster. If this parameter is not specified, it defaults to maximum units value. Minimum value is 1.

Auto Termination

This option is unchecked by default. Check this option for auto termination of cluster. Once the cluster becomes idle, it will terminate after the duration specified. Choose a minimum of one minute or a maximum of 24 hours value.

Steps Concurrency

This option is unchecked by default. Check this option to enable running multiple steps concurrently. Once the last step completes, the cluster will enter a waiting state.

Upon checking the Steps Concurrency check-box, the option appears:

Maximum Steps

Provide value for the maximum steps that can run at a time. Enter a value between 2 and 256.

Name

Name of the registered cluster.

Up Since

Timestamp information about the cluster since the time it is up.

CPU/Memory

The consumed CPU/Memory will be listed here.

Actions

The user can Edit/Unregister the registered cluster(s) information.

Type

Select type of cluster to be registered, i.e. kubernetes.

Name

Provide a unique name of cluster to be registered.

Ingress URL

Provide ingress URl with accurate server host and port details.

Upload Certificate as

The user has an option to upload a valid cluster configuration file having certificate data (like certificate-authority-data, server, client-certificate-data, client-key-data) definition.

Else, upload configuration and certificates separately.

These are further explained below:

Config with embedded certificate: The user can upload K8 config files with all the certificates embedded in a single zip file.

Upload Config (zip): Kubernetes configuration zip file with all the certificates embedded.

Upload certificate files: The user can upload configuration and certificates separately.

Provide the API Server URL.

Upload the below files:

- Authority Certificate: Upload the API server client authority certificate.

- Client Certificate: Upload the client certificate for API server authentication.

- Client Key: Upload the client key for server API authentication.

Image Name

Name of the container image registered.

Container Image URI

URI registered on container registry and accessible to the cluster.

Tags

Custom tags added during container image registration.

Actions

The user can Edit/Unregister the registered container image(s).

Image Name

Provide a unique name of container to be registered.

Description

Provide container image description.

Tags

Custom tags for the container image can be added.

Image URI

URI registered on container registry and accessible to the cluster must be provided.

IDE

The IDE options that are selected here will be available during Sandbox configuration.

Upload Docker File

An option to upload the docker file for user’s reference.

Upload YAML [Zip]

A zip file containing supported YAML’s for the image can be uploaded.

Once the YAML file is uploaded, the user can view details of the uploaded file by clicking on the View YAML File icon.

Note: To know more about the YAML files upload, see Upload YAML Example given after the table.

Version Control System

User has an option to choose Git or gathr Metastore as version control system.

In case of Git, the pipeline artifacts will get pushed to Git after a version is created.

In case of gathr Metastore, the pipeline artifacts will be stored in the file system.

If user selects Git option as Version Control System, fill the below fields:

Repositories Local Path

Provide the repositories local path where the Git clone will be placed in the repos of the file system.

auth.type

Choose Credentials/SSH option from the drop-down.

Note:

- Upon selecting Credentials, provide the User name and Password.

- Upon selecting SSH, upload the SSH key path.

HTTP URL

Provide the HTTP URL of the remote GIT repository.

Branch

Provide the branch name where the push operation will be performed.

Note:

Click SAVE/CLONE.

Note: In gathr, under Setup > Version Control tab, in case of Bitbucket, make sure to create App password in Bitbucket account. Once the app password for Bitbucket is created, provide that password along with username in the gathr UI to access private repos.

If user selects gathr Metastore as Version Control System, fill the below fields:

auth.type

Choose Credentials/SSH option from the drop-down.

Note:

Upon selecting Credentials, provide the User name and Password. Upon selecting SSH, upload the SSH key path.

Click SAVE.

StreamAnalytix UI Host

The IP address of gathr.

StreamAnalytix Installation Directory

The installation directory of gathr.

StreamAnalytix Web URL

The URL of gathr web studio.

StreamAnalytix UI Port

The UI port of gathr.

LogMonitoring UI Host **

The host address of LogMonitoring.

LogMonitoring UI Port **

The port of LogMonitoring.

Messaging Type

Specifies the Message Queuing System that application uses internally for messaging. Possible value is RABBITMQ (for RabbitMQ.

StreamAnalytix Monitoring Reporters Supported

The monitoring reporter type and the possible values should be comma separated graphite, console and logger.

Metric Server **

Monitoring Metric Server (Graphite or Ambari).

Password

The database password.

Driver Class

The database driver class name.

Connection URL

The database URL for the database.

User

The database username.

Database Dialect

The type of database on which gathr database is created. Possible values are MySQL, PostgreSQL, Oracle.

Host List

The comma separated list of <IP>:<PORT> of all nodes in zookeeper cluster where configuration will be stored.

Indexer Type

The default indexer type. For e.g. - Solr or ElasticSearch.

Index Default Replication Factor

Number of additional copies of data to be saved.

Enable Index Default is Batch

Default value for the Batch parameter of indexing.

Index Default Batch Size

Default batch size for the indexing store.

Enable Index Default Across Field Search

Search without specifying column names, takes extra space and time.

Index Default Number of Shards

Number of shards to be created in index store.

Index Default Routing Required

The default value for the Routing parameter of indexing.

Indexer Default Source

The default value for the Source parameter of indexing.

Index Retries

The number of retries for indexing.

Index Retries Interval(in ms)

The retries interval for the indexing when ingestion fails.

Indexer time to live in seconds

Indexed data older than mentioned time in seconds from current time will not be fetched.

Persistence Store **

The default persistence type. For e.g. - Hbase, Cassandra.

Persistence Default Is batch Enable **

Defines if by default batching should be enabled in persistence.

Persistence Default Batch Size **

The batch size for the persistence store.

Persistence Default Compression **

The default compression type for the persistence store.

User Authentication Source

Options available for login into gathr application:

LDAP

Okta

Gathr Metastore

To authenticate user’s credentials while login gathr application, user can opt to authenticate login credentials using LDAP server, Okta, or Gathr metastore.

Note: If configured with LDAP, user who is trying to login into the application should exist in LDAP server.

Similarly, for enabling SSO (Single Sign On) in gathr, choose Okta as (Service provider)/User Authentication Source to verify the login credentials for application.

Note:

Upon logging into the environment, you will be redirected to Okta sign in page.

Provide the username and password that is configured in LDAP.

The prerequisites for Okta authentication are:

In the LDAP server Group user mapping must be done accurately. For example, emailID, givenName.

For Okta account, Application configuration, Application’s access management and LDAP directory integration must be done.

Initially, the Gathr Role and LDAP role mapping should be done by login in the application in the embedded mode with superuser credentials and make the required changes.

All the parameters including cn, sn, givenName, mail, uid, userPassword must be configured:

So, mapping in LDAP is done by configuring LDAP Active Directory into Okta via. Okta agent that fetches the details from LDAP and the details are mapped in Okta.

SSO in Okta

Note:

- Single Sign-On authentication allows users to login with a single ID within the software application without managing multiple accounts and passwords thus making it easier for admins to manage all users and privileges with one centralized admin dashboard.

- Configure gathr application in Okta by providing Client credentials including details such as Client ID and secret key. User will be required to provide the details for gathr application Access Management in Okta.

- User will be required to install Okta Agent to be able to establish interaction between Okta and LDAP server. (The Okta agent can be installed in gathr/client end application, from where it can access the details of gathr application).

- Once Okta Agent is configured, provide LDAP server details.

User Authorization Source

Specify user's authorization mechanism, accordingly user will be assigned appropriate role in the gathr webstudio. Possible values are LDAP and gathr Metastore. Default value is gathr Metastore.

Choose LDAP as User Authorization Source (defining the assigned roles within application. Example: superuser, custom user, workspace admin) if Okta is used for authentication.

Superuser(Seed User) Authentication Source

Superuser authentication source needs to be selected. Currently, only gathr Metastore is supported as authentication source for Superuser.

SuperAdmin Password

The super admin password (Required to access the Dashboard UI).

ReportClient path

The path of ReportClient.properties required to connect with Report Engine.

Connection Name

The connection name created for gathr in Dashboard.

Organization ID

The name of organization for gathr in Intellicus.

SuperAdmin User ID

The Dashboard super user Username to access the Intellicus via UI.

SuperAdmin Organization

The Dashboard superuser organization name, required to access Intellicus via UI.

StreamAnalytix URL

The dashboard web admin URL, used for showing Dashboard UI from within gathr admin.

Databricks Enabled

To enable Databricks on this environment.

Databricks Instance URL

Databricks Instance URL to connect databricks account and access it over REST calls.

Databricks Authentication Token

Databricks Access token provided here will be associated with superuser account for gathr. It will be saved as encrypted text.

Databricks DBFS Upload Jar Path

DBFS Path for the gathr specific jars and files.

Maximum Polling Time (in minutes)

Maximum Polling Time.

Polling Interval

Polling Interval (in seconds).

Databricks Mediator Service URL

This is gathr web service URL for Databricks.

EMR Enabled

To enable EMR on this environment.

Jar Upload Path

S3 Path for the gathr specific jars and files.

Log URI

S3 Path for creating Logs for EMR Cluster launched by gathr.

EMR Mediator Service URL

This is gathr webservice URL for EMR.

Connect Using

Connect to the EMR cluster by using either AWS Keys or with the Instance Profile.

AWS Key

AWS Access key to the associated gathr superuser account.

AWS Secret Key

AWS Access key to the associated superuser account for gathr. It will be saved as encrypted text.

Instance Profile

AWS Instance Profile to the associated superuser account for gathr.

AWS Region

The region that the AWS EMR is to be launched in.

Spark Livy URL **

Livy web server URL through which gathr submit pipelines on Spark.

Spark Home **

The spark installation directory.

Spark Master URL **

It is the Spark Master URL for e.g. spark://host1:7077

Spark cluster manager **

Defines spark cluster manager i.e. standalone or yarn.

Spark Job Server Log Directory

Directory path where pipeline logs will be generated when using Spark Job server.

Spark UI Port

It is the port on which the spark master UI is running.

spark.history.server

The history server URL.

Spark Hadoop is HDP

If your environment is HDP, set it to True, otherwise set it to false and use for setting proxy user

Resource Manager Host

The resource manager hostname used for spark yarn deployment.

Resource Manager Webapp Port

Yarn Resource Manager UI Port.

Resource Manager Port

The resource manager port used for storm-yarn deployment.

ResourceManager High Availability

Enables Resource Manager’s High Availability.

ResourceManager HA Logical Names

ResourceManager High Availability Logical IDs defined at HA configuration.

ResourceManager HA Hosts

ResourceManager High Availability host names defined at HA configuration.

ResourceManager HA ZK Address

ResourceManager High Availability ZooKeeper-Quorum's address which is defined for HA configuration.

Spark Job Submit Mode

Submit mode of Spark pipeline using Job-Server.

Spark UI Host

Host name of the Spark Master.

Job Server Spark Home

The spark installation directory with which Job Server is configured.

Job Server URL

The host URL of Job Server.

Spark REST Host and Port

Spark REST Host name and port for e.g Host1:6066

Spark Python Path

This environment variable is used to augment the default search path for Python module files. Directories and individual zip files containing pure Python modules can be added to this path. gathr uses this variable to find PySpark modules usually located at $SPARK_HOME/python/lib.

Password

RabbitMQ Password to create connection.

Port

Port number of RabbitMQ.

RabbitMQ STOMP URL

RabbitMQ stomp URL.

Host List

IP address of the machine where RabbitMQ is running.

RabbitMQ Virtual Host

The RabbitMQ virtual hosts.

User

Username of RabbitMQ to create connection.

RabbitMQ Web URL

Web URL of RabbitMQ.

Kafka Metadata Broker List **

The list of comma separated IP:port of Kafka brokers.

Kafka Zookeeper Server List **

The list of comma separated IP:port of zookeeper for creating Kafka topic from gathr UI.

Kafka Topic Administration **

When set to true it specifies that with in the application a Kafka connection has permission to create topics in Kafka.

HBase Zookeeper Host **

The zookeeper host names used for HBase cluster.

HBase Zookeeper Port **

The zookeeper port for HBase cluster.

HBase Client Retry Number **

The number of retries for the HBase client.

HBase Zookeeper Parent Node **

Parent node in zookeeper for HBase service metadata.

HBase Zookeeper Recovery Retry **

The no. of times to retry the connection to HBase zookeeper.

system-config.hbase.table.administration **

When set to true it specifies that with in the application a HBase Default connection has permission to create tables and name spaces in HBase.

Cassandra Host List

Addresses of servers where Cassandra is running.

Cassandra User

Username for Cassandra data store authentication.

Cassandra Password

Password for Cassandra data store authentication.

Cassandra Thrift Client Retry Count

The number of retries the Cassandra client will make to make a connection with server.

Cassandra Thrift Client Delay Between Retries (in ms)

The time(in ms) after which the Cassandra client retries to make a connection to server.

Cassandra Keyspace Replicaton Factor

Defines how many copies of the data will be present in the cluster.

Cassandra Keyspace Replicaton Strategy

Strategy determines the nodes where replicas are placed. Simple Strategy places the first replica on a node determined by the partitioner. Additional replicas are placed on the next nodes clockwise in the ring without considering topology.

Cassandra Connection Retry Count

Cassandra connection retry count.

Enable Authentication

Select the check box, if Elasticsearch authentication is enabled.

Elasticsearch Cluster Name

Name of the Elasticsearch cluster.

Elasticsearch Connection URL

The http connection URL for Elasticsearch.

Connection Timeout in secs

ElasticSearch connection timeout in seconds.

Elasticsearch Embedded Data Directory

The data directory for running embedded Elasticsearch

Elasticsearch Embedded Enable data

Defines either to store data into disk or memory (true for disk, false for memory).

Elasticsearch Embedded Enable HTTP

Defines either the http connection is enabled or not for embedded Elasticsearch.

Elasticsearch Embedded Enable local

The value of this field should be true.

Elasticsearch Embedded Node Name

The node name of embedded as node.

Elasticsearch HTTP Connection URL

The http connection URL for Elasticsearch.

Elasticsearch HTTP Port

The port on which Elasticsearch REST URI is hosted.

Keystore Password

Elasticsearch keystore password.

Keystore Path

Elasticsearch keystore file (.p12) path.

Request Timeout in secs

Request Retry Timeout for ElasticSearch connection in seconds.

Enable Security

If security is enabled on Elasticsearch, set this to true.

Socket Timeout in secs

Socket Timeout for ElasticSearch connection in seconds

Enable SSL

Select the checkbox if SSL is enabled on Elasticsearch

Username

Elasticsearch authentication username.

Solr Zookeeper Hosts

The Zookeeper hosts for the Solr server.

Solr Configuration Version

Solr version number to create the zookeeper config node path for solr data.

Port

Port number of Graphite.

Host

IP address of the machine where Graphite is running.

UI Port

UI port number of Graphite.

Metric Collector Port

Ambari Metric Collector port.

Metric Collector Host

Hostname where Ambari Metric Collector is running.

Hive Meta Store URI **

Defines the hive metastore URI.

Hive Server2 JDBC URL **

Password for HiveServer2 JDBC connection. In case no password is required pass it as empty("").

Hive Server2 Password **

Defines the Hive server-2 password.

Hive Warehouse Dir **

Defines the warehouse directory path of Hive server.

Hadoop Enable HA

Hadoop cluster is HA enabled or not.

File System URI

The file system URI. For e.g. - hdfs://hostname:port, hdfs://nameservice, file://, maprfs://clustername

Hadoop User

The name of user through which the hadoop service is running.

Hadoop DFS Name Services

The name service id of Hadoop HA cluster.

Hadoop Namenode 1 Details

The RPC Address of namenode1.

Hadoop Namenode 2 Details

The RPC Address of namenode2.

Password

Password against which the user will be authenticated in LDAP Server.

Group Search Base

Defines the part of the directory tree under which group searches will be performed.

User Search Base

Defines the part of the directory tree under which DN searches will be performed.

User Search Filter

The filter which will be used to search DN within the User Search Base defined above.

Group Search Filter

The filter which is used to search for group membership. The default is member={0 corresponding to the groupOfMembers LDAP class. In this case, the substituted parameter is the full distinguished name of the user. The parameter {1} can be used if you want to filter on the login name.

Admin Group Name

LDAP group name which maps to application's Admin role.

Developer Group Name

LDAP group name which maps to application's Developer role.

Devops Group Name

LDAP group name which maps to application's Devops role.

Tier-II Group Name

LDAP group name which maps to application's Tier-II role.

LDAP Connection URL

URL of the LDAP server is a string that can be used to encapsulate the address and port of a directory server. For e.g. - ldap://host:port.

User Distinguished Name

A unique name which is used to find the user in LDAP Server.

Alert Email Character Set

The character set used for sending emails.

Alert Sender Email

The email address from which mails must be sent.

JDBC Driver Class

The database driver used for activity setup.

JDBC URL

The database URL for activity database.

JDBC User

The database user name.

JDBC Password

JDBC Password.

Host

The email server host from which emails will be sent

Port

The email server port.

User

The email id from which emails will be sent.

Password

The Password of the email account from which emails will be sent.

Default Sender Email

The default email address from which mails will be sent if you do not provide one in the UI.

Enable SSL

If SSL (Secure Sockets Layer) is enabled for establishing an encrypted link between server and client.

Enable TSL

If Transport Layer Security (TLS) enables the encrypted communication of messages between hosts that support TLS and can also allow one host to verify the identity of another.

History

Activiti history is needed or not.

Database

The database used for activity setup.

Max Pool Size **

The Couchbase Max Pool Size.

Default Bucket Memory Size **

The memory size of default bucket in Couchbase.

Password **

The Couchbase password.

Default Bucket Replica No **

The Couchbase default bucket replication number.

Host Port **

The port no. of Couchbase.

Host Name **

The host on which the Couchbase is running.

HTTP URL **

The Couchbase http URL.

Bucket List **

The Couchbase bucket list.

Polling timeout **

The polling timeout of Couchbase.

Polling sleeptime **

The sleep time between each polling.

User Name **

The username of the Couchbase user.

Hadoop NameNode Kerberos Principal

Service principal of name node.

Kerberos Configuration File Override

Set to true if you want the keytab_login.conf file to be (re)created for every running pipeline when Kerberos security is enabled.

Hadoop Core Site Location

The property should be used when trying to connect HDFS from two different realms. This property signifies the path of Hadoop core-site.xml containing roles for cross-realm communications.

Hbase Master Kerberos Principal

Service principal of HBase master.

ResourceManager Kerberos Principal

Service principal of resource manager

Hbase Regionserver Kerberos Principal

Service principal of region server.

Hive Metastore Kerberos principal

Service principal of Hive metastore.

HiveServer2 Kerberos Principal

Service principal of hive server 2

Key Tab Select Option

A Keytab is a file containing pair of Kerberos principals and encrypted keys. You can use Keytab to authenticate various remote systems. It has two options:

Specify Keytab File Path: Path where Keytab file is stored

Upload Keytab File: Upload Keytab file from your local file system.

Specify Keytab File Path

If the option selected is Specify Keytab File Path, system will display the field KeyTab File Path where you will specify the keytab file location.

Upload Keytab File

If the option selected is Upload Keytab File, system will display the field Upload Keytab File that will enable you to upload the Keytab file.

jupyter.hdfs.port

HDFS Http port.

jupyter.hdfs.dir

HDFS location where uploaded data will be saved.

jupyter.dir

Location where notebooks will be created.

jupyter.notebook.service.port

Port on which Auto create Notebook service is running.

jupyter.hdfs.connection.name

HDFS connection name use to connect HDFS (from gathr connection tab).

jupyter.url

URL contains IP address and port where Jupyter services are running.

Application Logging Level

The logging level to be used for gathr logs.

StreamAnalytix HTTPs Enabled

Whether gathr application support HTTPs protocol or not.

Spark HTTPs Enabled

Whether Spark server support HTTPs protocol or not.

Test Connection Time Out

Timeout for test connection (in ms).

Java Temp Directory

The temp directory location.

StreamAnalytix Reporting Period

Whether to enable View Data link in application or not.

View Data Enabled

Whether to enable View Data link in application or not.

TraceMessage Compression

The type of compression used on emitted TraceMessage from any component.

Message Compression

The type of compression used on emitted object from any component.

Enable StreamAnalytix Monitoring Flag

Flag to tell if monitoring is enabled or not.

CEP Type

Defines the name of the cep used. Possible value is esper as of now.

Enable Esper HA Global

To enable or disable HA.

CepHA Wait Interval

The wait interval of primary CEP task node.

StreamAnalytix Scheduler Interval

The topology stopped alert scheduler's time interval in seconds.

Enable StreamAnalytix Scheduler

Flag to enable or disable the topology stopped alert.

StreamAnalytix Session Timeout

The timeout for a login session in gathr.

Enable dashboard

Defines whether dashboard is enable or disable.

Enable Log Agent

Defines if Agent Configuration option should be visible on gathr GUI or not.

Enable Storm Error Search

Enable showing pipeline Application Errors tab using LogMonitoring search page.

StreamAnalytix Pipeline Error Search Tenant Token

Tenant token for Pipeline Error Search.

StreamAnalytix Storm Error Search Index Expression

Pipeline application error index expression (time based is expression to create indexes in ES or Solr, that is used during retrieval also).

Kafka Spout Connection Retry Sleep Time

Time between consecutive Kafka spout connection retry.

Cluster Manager Home URL

The URL of gathr Cluster Manager

StreamAnalytix Pipeline Log Location

gathr Pipeline Log Location.

HDFS Location for Pipeline Jars

HDFS Location for Pipeline Jars.

Scheduler Table Prefix

Tables name starting with a prefix which are related to storing scheduler's state.

Scheduler Thread Pool Class

Class used to implement thread pool for the scheduler.

Scheduler Thread Pool Thread Count

This count can be any positive integer, although only numbers between 1 and 100 are practical.

This is the number of threads that are available for concurrent execution of jobs.

If only a few jobs run a few times a day, then 1 thread is plenty. However if multiple jobs, with most of them running every minute, then you probably want a thread count like 50 or 100 (this is dependent on the nature of the jobs performed and available resources).

Scheduler Datasource Max Connections

The maximum number of connections that the scheduler datasource can create in its pool of connections.

Scheduler Misfire Threshold Time

Milliseconds the scheduler will tolerate a trigger to pass its next-fire-time by, before being considered misfired.

HDP Version

Version of HDP ecosystem.

CDH Version

Version of CDH ecosystem.

Audit Targets

Defines the Audit Logging Implementation to be use in the application, Default is file.

Enable Audit

Defines the value (true/false) for enabling audit in application.

Persistence Encryption Key

Specifies the encryption key used to encrypt data in persistence.

Ambari HTTPs Enabled

Whether Ambari server support HTTPs protocol or not.

Graphite HTTPs Enabled

Whether Graphite server support HTTPs protocol or not.

Elastic Search HTTPs Enabled

Whether Elasticsearch engine support HTTPs protocol or not.

SQL Query Execution Log File Path

File location for logging gathr SQL query execution statistics.

SQL Query Execution Threshold Time (in ms)

Defines the max limit of execution time for sql queries after which event will be logged (in ms).

Lineage Persistence Store

The data store that will be used by data lineage feature.

Aspectjweaver jar location

The absolute path of aspectweaver jar required for inspect pipeline or data lineage.

Is Apache Environment

Default value is false. For all apache environment set it to "true".

Zookeeper Retry Count

Zookeeper connection retry count.

Zookeeper Retry Delay Interval

Defines the retry interval for the zookeeper connection.

Zookeeper Session Timeout

Zookeeper's session timeout time.

Model Registration Validation Timeout(in seconds)

The time, in seconds, after which the MLlib, ML or H2O model registration and validation process will be failed if the process not complete.

Spark Fetch Schema Timeout(in seconds)

The time, in seconds, after which the fetch schema process of register table will be failed if the process not complete.

Spark Failover Scheduler Period(in ms)

Regular intervals to run scheduler tasks. Only applicable for testing connection of Data Sources in running pipeline.

Spark Failover Scheduler Delay(in ms)

Delay after which a scheduler task can run once it is ready. Only applicable for testing connection of Data Sources in running pipeline.

Refresh Superuser Pipelines and Connections

Whether to refresh Superuser Pipelines and Default Connections in database while web studio restart.

Gathr SparkErrorSearchPipeline Index Expression **

Pipeline application error index expression (time based js expression to create indexes in ES or Solr, that is used during retrieval).

Enable Spark Error Search **

Enabled to index and search spark pipeline error in LogMonitoring.

Register Model Minimum Memory

Minimum memory required for web studio to register tables, MLlib, ML or H2O models. Example -Xms512m.

Register Model Maximum Memory

Maximum memory required for web studio to register tables, MLlib, ML or H2O models. Example -Xmx2048m.

H2O Jar Location

Local file system's directory location at which H2O model jar will be placed after model registration.

H2O Model HDFS Jar Location

HDFS path location at which H2O model jar will be placed after model registration.

Spark Monitoring Scheduler Delay(in ms) **

Specifies the Spark monitoring scheduler delay in milliseconds.

Spark Monitoring Scheduler Period(in ms) **

Specifies the Spark monitoring scheduler period in milliseconds.

Spark Monitoring Enable **

Specifies the flag to enable the spark monitoring.

Spark Executor Java Agent Config

Spark Executor Java Agent configuration to monitor executor process, the command includes jar path, configuration file path and Name of the process.

Spark JVM Monitoring Enable **

Specifies the flag to enable the spark monitoring.

ES query monitoring index name

Provide the ES query monitoring index name which is required for indexing the data of query streaming.

Scheduler period for es monitoring purging

Scheduler period for es monitoring purging in seconds.

Rotation policy for of ES monitoring graph

Specify the rotation policy for index creation for ES monitoring graph (daily for a period of one day and weekly for 7 days).

Purging duration of ES monitoring index

Purge duration for ES in seconds for es monitoring graph index. Index created before this duration will be deleted.

Enable purging scheduler for ES Graph monitoring

Check the checkbox to enable purging scheduler for ES Graph monitoring.

Spark Version **

By default the version is set to 2.3.

Note: Set spark version to 2.2 for HDP 2.6.3”

Livy Supported JARs Location **

HDFS location where livy related jar file and application streaming jar file have been kept.

Livy Session Driver Memory **

Minimum memory that will be allocated to driver while creating livy session.

Livy Session Driver Vcores **

Minimum virtual cores that will be allocated to driver while creating Livy session.

Livy Session Executor Memory **

Minimum executor instances that will be allocated while executing while creating Livy seconds where sample data has been kept while schema auto detection.

Livy Session Executor Vcores **

Minimum virtual cores that will be allocated to executor while creating Livy session.

Livy Session Executor Instances **

Minimum executor instances that will be allocated while executing while creating Livy session.HDFS where sample data has been kept while schema auto detection.

Livy Custom Jar HDFS Path **

The full qualified path of HDFS where uploaded custom jar has been kept while creating pipeline.

Livy Data Fetch Timeout **

The query time interval in seconds for fetching data while data inspection.

isMonitoringGraphsEnabled

Whether monitoring graph is enabled or not.

ES query monitoring index name

this property stores the data of monitoring in this given index of default ES connection.

Scheduler period for ES monitoring purging

in this time interval purging scheduler will invoke and check whether the above index is eligible for purging (in sec.) (tomcat restart require).

Rotation policy of ES monitoring graph

“It can have two values daily or weekly”

If daily index will be rotated daily else weekly means only a single day data will be stored in single index otherwise a data of a week will be stored in an index.

Purging duration of ES monitoring index

It’s a duration after which index will be deleted default is 604800 sec. Means index will be deleted after 1 week.” (tomcat restart requires)

Enable purging scheduler for ES Graph monitoring

If we need purging of index or not depend on this flag. Purging will not take place if flag is disable. It requires restart of Tomcat Server.

RabbitMQ Max Retries

Defines maximum number of retries for the RabbitMQ connection.

RabbitMQ Retry Delay Interval

Defines the retry delay intervals for RabbitMQ connection.

RabbitMQ Session Timeout

Defines session timeout for the RabbitMQ connection.

Real-time Alerts Exchange Name

Defines the RabbitMQ exchange name for real time alert data.

Kafka Message Fetch Size Bytes

The number of byes of messages to attempt to fetch for each topic-partition in each fetch request.

Kafka Producer Type

Defines whether Kafka producing data in async or sync mode.

Kafka Zookeeper Session Timeout(in ms)

The Kafka Zookeeper Connection timeout.

Kafka Producer Serializer Class

The class name of the Kafka producer key serializer used.

Kafka Producer Partitioner Class

The class name of the Kafka producer partitioner used.

Kafka Key Serializer Class

The class name of the Kafka producer serializer used.

Kafka 0.9 Producer Serializer Class

The class name of the Kafka 0.9 producer key serializer used.

Kafka 0.9 Producer Partitioner Class

The class name of the Kafka 0.9 producer partitioner used.

Kafka 0.9 Key Serializer Class

The class name of the Kafka 0.9 producer serializer used.

Kafka Producer Batch Size

The batch size of data produced at Kafka from log agent.

Kafka Producer Topic Metadata Refresh Interval(in ms)

The metadata refresh time taken by Kafka when there is a failure.

Kafka Producer Retry Backoff(in ms)

The amount of time that the Kafka producer waits before refreshing the metadata.

Kafka Producer Message Send Max Retry Count

The number of times the producer will automatically retry a failed send request.

Kafka Producer Request Required Acks

The acknowledgment of when a produce request is considered completed.

Kerberos Sections

Section names in keytab_login.conf for which keytabs must be extracted from pipeline if krb.config.override is set to true.

Hadoop Security Enabled

Set to true if Hadoop in use is secured with Kerberos Authentication.

Kafka Security Enabled

Set to true if Kafka in use is secured with Kerberos Authentication.

Solr Security Enabled

Set to true if Solr in use is secured with Kerberos Authentication.

Keytab login conf file Path

Specify path for keytab_login.conf file.

Cloud Trial

The flag for Cloud Trial. Possible values are True/False.

Cloud Trial Max Datausage Monitoring Size (in bytes)

The maximum data usage limit for cloud trial.

Cloud Trial Day Data Usage Monitoring Size (in bytes)

The maximum data usage for FTP User.

Cloud Trial Data Usage Monitoring From Time

The time from where to enable the data usage monitoring.

Cloud Trial Workers Limit

The maximum number of workers for FTP user.

FTP Service URL

The URL of FTP service to create the FTP directory for logged in user (required only for cloud trial).

FTP Disk Usage Limit

The disk usage limit for FTP users.

FTP Base Path

The base path for the FTP location.

Enable Monitoring Graphs

Set to True to enable Monitoring and to view monitoring graphs.

QueryServer Monitoring Flag

Defines the flag value (true/false) for enabling the query monitoring.

QueryServer Moniting Reporters Supported

Defines the comma-separated list of appenders where metrics will be published. Valid values are graphite, console, logger.

QueryServer Metrics Conversion Rate Unit

Specifies the unit of rates for calculating the queryserver metrics.

QueryServer Metrics Duration Rate Unit

Specifies the unit of duration for the queryserver metrics.

QueryServer Metrics Report Duration

Time period after which query server metrics should be published.

Query Retries

Specifies the number of retries to make a query in indexing.

Query Retry Interval (in ms)

Defines query retry interval in milliseconds.

Error Search Scroll Size

Number of records to fetch in each page scroll. Default value is 10.

Error Search Scroll Expiry Time (in secs)

Time after which search results will expire. Default value is 300 seconds.

Index Name Prefix

Prefix to use for error search system index creation. The prefix will be used to evaluate exact index name with partitioning. Default value is sax_error_.

Index number of shards

Number of shards to create in the error search index. Default value is 5.

Index Replication Factor

Number of replica copies to maintain for each index shard. Default value is 0.

Index Scheduler Frequency (in secs)

Interval (in secs) after which scheduler will collect error data and index in index store.

Index Partitioning Duration (in hours)

Time duration after which a new index will be created using partitioning. Default value is 24 hours.

Data Retention Time (in days)

Time duration for retaining old data. Data above this threshold will be deleted by scheduler. Default value is 60 days.

Enable Event Auditing

Defines the value for enabling events auditing in the application.

true

Events Collection Frequency

(in secs)

Time interval (in seconds) in which batch of captured events will be processed for indexing.

10

Events Search Scroll size

Number of records to fetch in each page scroll on result table.

100

Events Search Scroll Expiry

(in secs)

Time duration (in seconds) for search scroll window to expire.

300

Events Index Name Prefix

Prefix string for events index name. The prefix will be used to evaluate exact target index name while data partitioning process.

sax_audit_

Events Index Number of Shards

Number of shards to create for events index.

5

Events Index Replication Factor

Number of replica copies to maintain for each index shard.

0

Index Partitioning Duration

(in hours)

Time duration (in hours) after which a new index will be created for events data. A partition number will be calculated based on this property. This calculated partition number prefixed with Events Index Name Prefix value will make target index name.

24

Events Retention Time (in days)

Retention time (in days) of data after which it will be auto deleted.

60

Events Indexing Retries

Number of retries to index events data before sending it to a WAL file.

5

Events Indexing Retries Interval

(in milliseconds)

It defines the retries interval (in milliseconds) to perform subsequent retries.

3000

QueryServer Monitoring Flag

The flag value (true/false) for enabling the query monitoring.

QueryServer Monitoring Reporters Supported

The comma-separated list of appenders where metrics will be published. Valid values are graphite, console, logger.

QueryServer Metrics Conversion Rate Unit

Specifies the unit of rates for calculating the queryserver metrics.

QueryServer Metrics Duration Rate Unit

Specifies the unit of duration for the queryserver metrics.

QueryServer Metrics Report Duration

Time after which query server metrics should be published.

QueryServer Metrics Report Duration Unit

The units for reporting query server metrics.

Query Retries

The number of retries to make a query in indexing.

Query Retry Interval (in ms)

Defines query retry interval in milliseconds.

Audit Targets

Defines the audit logging implementation to be used in the application, Default is fine.

ActiveMQ Connection Timeout(in ms)

Defines the active MQTT connection timeout interval in ms.

MQTT Max Retries

Max retries of MQTT server.

MQTT Retry Delay Interval

Retry interval, in milliseconds, for MQTT retry mechanism.

JMS Max Retries

Max retries of JMS server.

JMS Retry Delay Interval

Retry interval, in milliseconds, for JMS retry mechanism.

Metrics Conversion Rate Unit

Specifies the unit of rates for calculating the queryserver metrics.

Metrics Duration Rate Unit

Specifies the unit of duration for the metrics.

Metrics Report Duration

Specifies the duration at interval of which reporting of metrics will be done.

Metrics Report Duration Unit

Specifies the unit of the duration at which queryserver metrics will be reported.

Gathr Default Tenant Token

Token of user for HTTP calls to LogMonitoring for adding/modifying system info.

LogMonitoring Dashboard Interval(in min)

Log monitoring application refresh interval.

Logmonitoring Supervisors Servers

Servers dedicated to run LogMonitoring pipeline.

Export Search Raw Field

Comma separated fields to export LogMonitoring search result.

Elasticsearch Keystore download path prefix

Elasticsearch keystore download path prefix in case of uploading keystore.

Tail Logs Server Port

Listening port number where tail command will listen incoming streams of logs, default is 9001.

Tail Logs Max Buffer Size

Maximum number of lines, that can be stored on browser, default is 1000.

sax.datasets.profile.frequency.distribution.count.limit

Defines the number of distinct values to be shown in the frequency distribution graph of a column in a Dataset.

sax.datasets.profile.generator.json.template

common/templates/DatasetProfileGenerator.json

Template of the spark job used to generate profile of a Dataset.

Pipeline Error Notification Email IDs

Provide comma separated email IDs for pipeline error notification.

Pipeline Test Connection Enabled

Check mark the checkbox to enable the email notification when a pipeline component is down.

Maintenance mode enabled

Provide true or false value for enabling the email notification in case pipeline component stops working.

Contextual Logs

A detailed contextual information (e.g. userName, roles, projectName) will be appended in the logs once this option is enabled.

Enable Event Notifier

Check this option to enable event notification based on the provided event notifier type. For example: SNS.

Event Notifier Type

Provide the event notifier type i.e., SNS

SNS Authentication Type

Select the AWS Authentication Type from the available options:

- AWS Keys

- Instance Profile

- Role ARN

AWS Key ID

Provide the AWS account access key.

AWS Secret Key

Provide the AWS account secret key.

SNS Topic Region

Provide the AWS SNS topic region.

SNS Topic Type

Select the SNS topic type from the below available options:

- Standard

- FIFO

SNS Topic ARN

Provide the SNS topic ARN where you want to publish alert data i.e., arn:aws:sns:us-east-1:123456789012:Test.

Role ARN

Provide the AWS account Role ARN.

Message Group ID

Provide message group ID for the FIFO SNS topic.

Component Type

Shows all the available connections. Select ADLS from the drop-down list.

Connection Name

Provide a connection name to create the connection.

Account Name

Provide the valid storage account name.

Authentication Type

Select authentication type. The available options are mentioned below:

- Azure AD

- Access Key

- Credential Passthrough

Upon selecting Azure AD, provide the below fields:

Client ID

Provide a valid client ID.

Secret Client

Provide a valid client secret password.

Directory ID

Provide a valid directory ID.

Upon selecting Access Key, provide the below fields:

Account Key

Provide a valid storage account key.

The Credential Passthrough option allows you to authenticate automatically to Azure Data Lake Storage from Azure Databricks clusters using the identity that you use to log in to Azure Databricks.

Component Type

Shows all the available connections. Select AWS IoT Component type from the list.

Connection Name

Name of the connection. For example, AWSIoT.

AWS KeyId

This is the AWS Key i.e. the credential to connect to AWS console.

Secret Access Key

This is AWS secret access key, to access AWS services.

Client EndPoint

AWS IoT Client End Point, which is unique for IoT.

Role ARN

User role ARN. It is used to create rules.

Region

AWS Region.

Connection ClientId

Any stream name.

Component Type

Shows available connections.

Connection Name

Name of the connection. For example, Cassandra.

Azure Blob Connection String

Azure Blob connection String.

Component Type

Shows available connections.

Connection Name

Name of the connection. For example, Cassandra

Hosts

Hosts name and ports of Cassandra machine (comma separated).

Connection Retries

Number of retries for component connection.

Authentication Enabled

Enable if authentication is enabled in Cassandra.

Username

Username for Cassandra data store authentication

Password

Password for Cassandra data store authentication

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections.

Connection Name

Name of the connection.

Cosmos Endpoint URI

End point URI of Azure Cosmos DB Account.

Key

Provide the Azure Cosmos DB Key.

Consistency Level

Consistency levek in Azure Cosmos DB.

If the user selects Default option then the Cosmos DB account level consistency will be used as configured on the azure portal of Cosmos DB. User can alternately select the Eventual consistency option for read operations instead of using the default account level consistency.

TEST CONNECTION

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows available connections.

Connection Name

Name of the connection.

Hosts

Hosts name and ports of Couchbase machine (comma separated).

Username

Username for Couchbase datastore authentication.

Password

Password for Couchbase datastore authentication.

Note: If Spark SSL is enabled, in case of standalone click the + ADD CONFIGURATION button and provide the key-value pair for com.couchbase.sslEnabled as false.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows available connections. Select EFS Component type from the list.

Connection Name

Connection name to create.

Directory Path

Enter mounted root directory EFS.

Component Type

Shows available connections. Select Elasticsearch Component type from the list.

Connection Name

Name of the connection. For example, Elasticsearch

Hosts

Hosts name and ports of Elasticsearch machine.

http Port

Port number where elastic search is running.

Zone

Defines the zone of Elastic Search cluster.

Cluster Name

The name of the cluster to which elastic search will connect.

Connection Timeout secs

Maximum time taken by the client to connect to the Elasticsearch server (unit seconds).

Socket Timeout secs

If the continuous incoming data flow did not occur for a specified period of time, socket timeout connection occurs (unit seconds).

Request Retry Timeout secs

Sets the maximum timeout in case of multiple retries of same request.

Enable Security

Enable X-Pack security plugin for Elasticsearch authentication.

Enable SSL

If security is enabled on Elasticsearch, set this to true.

Keystore select option

Specify keystore path: Specify the Elasticsearch keystore file (.p12) path.

Upload keystore path: Upload the Elasticsearch keystore file (.p12) path.

Keystore file path

Mention the Elasticsearch keystore file (.p12) path.

Keystore password

Keystore password.

Enable Authentication

Select the checkbox if Authentication is enabled on Elasticsearch.

Username

Elasticsearch authentication username.

Password

Elasticsearch authentication password.

Component Type

Shows available connections. Select Elasticsearch Component type from the list.

Connection Name

Name of the connection. For example, EventBridge.

Connect Using

Options to connect using AWS Key or Instance Profile.

Upon selecting AWS, provide the below details:

AWS KeyID

Provide the AWS account access key.

Secret Access Key

Provide the AWS account secret key.

AWS Region

Specify region for AWS EventBridge service.

Upon selecting the Instance Profile option provide the below detail:

AWS Region

Specify region for AWS EventBridge service.

Component Type

The selected component type is reflected here which is GCP in this case.

Connection Name

Mention the connection name.

Service Account Key

Upload the service account key.

Component Type

Select the component type from the drop-down list. i.e., Google Spreadsheet in this case.

Connection Name

Unique name of the connection to be created.

Service Account ID

Provide the google service account ID.

Service Account Key Type

Preferred service account key type should be selected for authentication.

Upload File

Upload file with service account credentials as per the service account key type selected above.

Component Type

Shows all different types of available connections. Select HBase Component type from the list.

Connection Name

Name of the connection. For example, HBase

HDFS User

HDFS user name.

zK Host

Zookeeper host name for Hbase cluster.

zK Port

Zookeeper port for Hbase cluster.

Client Retries Number

Number of retries for the Hbase Client. For example, 2.

zk Recovery Retry

Number of retries to reconnect to HBase Zookeeper.

zk Parent Node

Parent node in Zookeeper for Hbase service metadata.

Table Administration

Enable this if you want to create table in Hbase.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button, if credentials provided are correct, services are up, and running, you will get the message Connection is available.

If you enter wrong credentials or server is down and you click on Test Connection, you will get the message Connection unavailable.

Component Type

Shows all different types of available connections. Select HDFS Component type from the list.

Connection Name

Name of the connection. For example, HDFS.

File System URI

File System URI of the machine where HDFS installed.

Username

The name of the user through which Hadoop user is running.

HA Enabled

Hadoop cluster is HA Enabled or not.

Name Node1 Name

NameNode1 identifier/label.

Name Node1 RPC Address

RPC Address of the Name Node1.

Name Node2 Name

NameNode2 identifier/label.

Name Services

Name service id of Hadoop cluster.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button, if credentials provided are correct, services are up, and running, you will get the message Connection is available.

If you enter wrong credentials or server is down and you click on Test Connection, you will get the message Connection unavailable.

Component Type

Shows all the available connections. Select HIVE Emitter Component type from the list.

Connection Name

Name of the connection.

MetaStore URI

Thrift URI to connect to HIVE Metastore service.

Hive Server2 URL

Defines the Hive server 2 JDBC URL. i.e. jdbc:hive2://<host>:<port>/default;principal=<principal>; ssl=<true/false>;sslTrustStore=<truststorepath>;trustStorePassword=<pass>;sslKeyStore=<keystorepath>; keyStorePassword=<pass>; transportMode=<http/binary>;httpPath=cliservice

Please view the note below the table.

HiveServer2 Password

Password for HiveServer2 JDBC connection. If there is no password, leave it blank.

FileSystem URI

HDFS File System URI

UserName

HDFS User name authorized to access the services.

HA Enabled

Check this option, if Name Node of HDFS is HA enabled.

KeyTab Select Option

You can add extra Java options for any Spark Superuser pipeline in following way:

1. Login as Superuser and click on Data Pipeline and edit any pipeline.

Upload key tab file: Enables to upload Key tab file for authentication.

Depending on the option selected. 

Specify keytab file path

If the option selected is Specify Keytab File Path, system will display the field KeyTab File Path where you will specify the keytab file location.

Upload key tab file

If the option selected is Upload Keytab File, system will display the field Upload Keytab File that will enable you to upload the keytab file.

Hive Zookeeper Quorum**

Zookeeper hosts used by LLAP. For example: host1:2181;host2:2181;host3:2181

Hive Daemon Service Hosts**

Application name for LLAP service. For example @llap0

Hive Warehouse Load Staging Directory**

Temp directory for batch writes to Hive. For e.g /tmp

Table Administration

Enable this if you want to create table in Hive Emitter.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all different types of available connections. Select JDBC Component type from the list. Then select database type as “IBM DB2”.

Connection Name

Name of the connection.

Database Type

Type of database in which data needs to be dumped. The databases available are: MySQL, POSTGRESQL, ORACLE, MSSQL, Custom, and IBM DB2.

Database Name

The name of the database to be used.

Host

The host of the machine where database is deployed.

Port

The port of the database where database is deployed.

UserName

The username of the database.

Password

The password of the database.

Enable SSL

Check the option to enable SSL.

SQL SSL Properties

SQL SSL JDBC driver url additional properties.

Keystore Select Option

Option to provide keystore file path either by selecting the Specify keystore File Path option or Upload keystore File.

ADD CONFIGURATION

In case of Databricks cluster, set the below configuration on the JDBC connection config for the respective Database type.

Connection config property with key as connectionProvider and value as GathrJdbcConnectionProvider

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Kafka Component type from the list.

Connection Name

Name of the connection to be created.

zK Hosts

Defines the list of comma separated IP port of Zookeeper for creating Kafka topic from gathr UI.

Kafka Brokers

List of Kafka nodes against which connections need to be created.

Enable Topic Administration

Enabling topic administration will allow creation, updation and deletion of a topic.

Enable SSL

Select the check box if connection is to be created with SSL enabled Kafka.

Enable Truststore

Specify Truststore file path or upload Truststore file (in JKS Format).

If option selected is Specify Truststore Path, you will view two additional fields: Truststore Path and Truststore Password.

If option selected is Upload Truststore Path, you will view two additional fields: Upload Truststore File and Truststore Password.

Upload Truststore File

Upload Truststore file by clicking on UPLOAD FILE button.

Truststore Path

Location of the Truststore file.

Truststore Password

Password of the Truststore file.

Enable Authentication

Select the checkbox if client authentication is enabled.

if selected, you will view four additional fields: Keystore Select Option, Upload keystore File, Keystore Password and Password.

Keystore Select Option

Either specify keystore file path or upload keystore file (in JKS Format)

Upload Keystore File

Enables to upload keystore file by clicking on UPLOAD FILE button.

Keystore File Path

Location of the keystore file.

Keystore Password

Keystore password for the Keystore file.

Password

Password of the private key in the keystore file.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Kinesis component type from the list. 

Connection Name

Name of the connection to be created.

Connect Using

User has the option to authenticate the Kinesis connection with either AWS Keys or Instance Profile.

Access Key Id

   AWS account access key ID.

Secret Key

AWS account secret key.

Instance Profile

AWS Instance will be used to authenticate the connection, if user selects this option to create connection.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select KUDU component type from the list. 

Access Key Id

Name of the connection to be created.   

HOSTS

IP address and port of the machine where KUDU is running.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows different types of available connections. Select Mongo DB from the drop down.

Connection Name

Name of the connection to be created.

URI Format

Select URI Format from the drop down for defining connection between StreamAnalytix and Mongo DB instance. Use DNS seed to connect with any Mongo DB instance seed list member or else select the Standard option.

Connection URL

Provide the driver URL to be used.

Authentication Options

Option to choose the authentication method. Select either None or Basic AUTH option from the drop-down. If you select Basic AUTH, then provide the below fields:

User Name

Provide the database user name with available access.

Password

Provide the database password.

Add Configuration

Option to add further configuration is available.

Component Type

Shows all different types of available connections.

Connection Name

Name of the connection to be created.

Host

IP address of the machine where Socket is running.

Port

Port of the machine where Socket is running.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections.

Connection Name

Name of the connection to be created.

Connection Factory

A connection factory is an object that a JMS client uses to create a connection with OpenJMS.

Host

IP address of the machine where Socket is running.

Port

Port of the machine where Socket is running.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successfull connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all different types of available connections.Select RedShift Component type from the list. 

Connection Name

Name of the connection to be created.

Driver Type

Define Driver version for Redshift.

Database Name

Name of the Database to be used.

Host

Hostname of the machine where Redshift cluster is running.

Port

Port number on which redshift cluster is listening.

UserName

Redshift Username.

Password

Redshift Password.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections.Select RMQ Component type from the list.

Connection Name

Name of the connection to be created. For example, RabbitMQ,

Host

IP address and port of the machine where RabbitMQ is running.

UserName

Username of RabbitMQ to create connection.

Password

Password of RabbitMQ to create connection.

Create

Click on the create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button, if credentials provided are correct, services are up, and running, you will get the message Connection is available.

If you enter wrong credentials or server is down and you click on Test Connection, you will get the message Connection unavailable.

Component Type

Shows all the available connections. Select S3 Component type from the list.

Connection Name

Name of the connection to be created.

Directory Path

Enter DBFS Parent Path for checkpointing.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select S3 Component type from the list.

Connection Name

Name of the connection to be created. For example, RabbitMQ.

Database Type

Database Type (mysql, postgresql, oracle, mssql, Amazon Aurora-Mysql, Amazon Aurora-PSSQL)

Enable SSL

Select the checkbox if SSL is enabled on RDS.

Keystore Select Option

Specify keystore path: Specify the Elasticsearch keystore file (.p12) path.

Upload keystore path: Upload the Elasticsearch keystore file (.p12) path.

Upload Keystore File

Enables to upload keystore file by clicking on UPLOAD FILE button.

Database Name

Name of the database.

Host

Hostname of RDS.

Port

Portname of RDS.

Username

Username of the account.

Password

Password of the account.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select S3 Component type from the list.

Connection Name

Name of the connection to be created. For example, RabbitMQ.

Connect Using

User has the option to authenticate the S3 connection with either AWS Keys or Instance Profile.

Access Key Id

   AWS account access key ID.

Secret Key

AWS account secret key.

Instance Profile

AWS Instance will be used to authenticate the connection, if user selects this option to create connection.

Create Connection

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Component type from the list.

Connection Name

Name of the connection to be created.

AWS Key id

S3 account access key.

Secret Access Key

S3 account secret key.

Region

AWS IOT Region.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Salesforce Component type from the list.

Connection Name

Name of the connection to be created.

Username

Username of Salesforce to create connection.

Password

Password of Salesforce to create connection.

securityToken

Security token is a case-sensitive alphanumeric key that is used in combination with a password to access Salesforce via API.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Snowflake component type from the list.

Connection Name

Name of the connection to be created.

Account Name

Option to enter the Snowflake account name. After entering the account name, the connection URL will get formed on its own.

Connection URL

Provide the driver URL to be used.

Database Name

Database name should be provided to access the data.

Username

Username of the Snowflake account should be provided.

Password

Password of the Snowflake account should be provided.

Keystore Select Option

Option to upload the keystore file is selected by default.

Upload Keystore File

Option to upload the keystore file.

Keystore Password

Option to provide the password to access the uploaded keystore file.

Component Type

Shows all the available connections. Select Socket component type from the list.

Connection Name

Name of the connection to be created.

Host

IP address of the machine where Socket is running.

Port

Port of the machine where Socket is running.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all different types of available connections. Select SOLR component type from the list.

Connection Name

Name of the connection to be created.

zkHost

The Zookeeper host for Solr sever.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Tibco Component type from the list.

Connection Name

Name of the connection to be created.

Connection URL

For Tibco connection the driver URL should be provided.

User Name

Tibco account username should be provided.

Password

Tibco account password should be provided.

JNDI Naming

Unchecked by default, the user can check-mark this option to provide the JNDI naming context. Upon selecting this option, provide the below details:

Initial Context Factory

Provide the initial context factory to be used for JNDI lookup.

Factory Name

Provide the Factory name which is the name of the connection factory.

Enable SSL

Unchecked by default, the user has an option to enable the SSL. Upon check-marking the Enable SSL option, provide the below details:

Identity Certificate

Option to choose the Identity Certificate. Choose one of the below options:

- Specify Identity Certificate Path

- Upload Identity Certificate

Note: Provide the Identity Certificate Path or Upload the Identity Certificate based on the selected Identity Certificate type.

Private Key File

Option to choose the Private Key File type. Choose one of the below options:

- Specify the Private Key File path

- Upload Private Key File

Note: Provide the Private Key File either by specifying the Private Key File path or by uploading the Private Key File based on the selected Private Key File options available in the drop down list.

Password for Private Key

Provide the password for Private Key.

Verify Host

Option to enable verification of Tibco server certificate. If

enabled the client will verify the server certificate.

If disabled, the client will establish secure

communication with the server but will not verify the

server's identity.

Upon checking this option, provide below details:

Server Root Certificate

Select the Server Root Certificate:

- Specify Server Root Certificate Path

- Upload Server Root Certificate

Verify Hostname

Option for client to enable verification of hostname in the

CN field of the server's certificate. If enabled the client

will verify the name specified in the "Expected

Hostname" against the server's certificate. Else, it will

take the server's hostname as default.

If disabled, the client establishes secure communication

with the server but does not verify the server's name.

Expected Hostname

Option to specify expected hostname which will be used

while hostname verification.

Auth Only

This option specifies whether SSL should be used to encrypt all server-client communications or only for client authentication. Upon check marking this option, the client request SSL will be used only for authentication.

Create

Click on the Create button to create the connection.

Test Connection

After entering all the details, click on the Test Connection button.

If credentials are correct, a successful connection message is generated.

If you enter wrong credentials or server is down, you will get the a Connection unavailable message.

Component Type

Shows all the available connections. Select Twitter Component type from the list.

Connection Name

Name of the connection to be created.

Consumer Key

Twitter User consumer key.

Consumer Secret Key

Twitter User consumer secret key. Consumer keys are the identity of an application owner.

Access Token

Application owners need to generate Authorization-access tokens that are used to call the Twitter API without sharing their passwords.

This is a twitter user access token.

Access Token Secret

This is a twitter user access token secret

Component Type

Shows all different types of available connections. Select VERTICA Component type from the list.

Connection Name

Name of the connection. For example, HDFS.

Database Name

Database name to access data.

Host

Database host name or IP address.

Port

Database port name.

Username

The username of database user.

Password

The password of database user.

Test Connection

After entering all the details, click on the Test Connection button, if credentials provided are correct, services are up, and running, you will get the message Connection is available.

If you enter wrong credentials or server is down and you click on Test Connection, you will get the message Connection unavailable.

timestamp

Epoch time, in milliseconds, when the event was performed, e.g., 1560369000

entityType

Entity type on which action was performed, e.g., pipeline, user, work-space, inspect_session, etc.

entityName

Name of the entity on which event was performed, e.g., pipeline name, workflow name, user name etc.

operation-Name

The type of action performed on entity, e.g., create, up-date, delete, list, access, share, revoke, etc.

description

Descriptive message about the event interaction.

interactionBy

User who caused the interaction event.

tenantId

The tenant id in which event occurred.

tenantName

The tenant name associated with tenant id.

Activity Time

Event time when the event was performed. To see that exact time stamp of event, expand the result row.

Description

Brief description about the event.

Operation   

Operation name which user has performed. It might be Create, Delete, Update, etc.

Entity Name

Name of entity on which event has been performed. Example pipeline created named as ‘SanityRun’ will be mapped as entity name.

Workspace

Name of the workspace under which event got performed (A single user can work on multiple workspaces)

User

User name who performed the operation

Global LDAP Groups

The global LDAP groups that are created by the superuser from main menu, cannot be deleted in the workspace listing of LDAP tab.

Add LDAP Roles

The Add LDAP Roles option is given at the top right side of the page. It can be used to add new entries for the LDAP group listings

Gathr Role

A drop-down option having list of all the global and custom roles. The desired role for the LDAP group must be selected. Example: Workspace Admin, Data Analyst I, Data Analyst II, Read Only, and so on.

LDAP Group

Search or enter the LDAP group(s) name that must be assigned with the selected Gathr Role.

Action

Option to remove the role assignment done for any specific LDAP group.

Roles

Name of the roles that are existing in Gathr.

Type

Type of role. Global if created from main menu and Custom if created from the workspace menu.

Privileges

The number of privileges that are assigned to the particular role.

The assigned privileges can be viewed by clicking on the value displayed.

Users

The number of users that are assigned with the particular role.

The list of users assigned with the particular role can be viewed by clicking on the value displayed.

Actions

Options to edit or delete the listed role(s).

Note: The out of box roles cannot be deleted.

New Role

Role Name

Unique name of the new role to be created.

Load Privilege by Users

Privilege(s) of the existing user(s) selected will instantly load in the Set Privilege section. For any privilege conflicts, load preference is given to Deny > Allow > Revoke.

Load Privilege by Roles

Privilege(s) of the existing role(s) selected will instantly load in the Set Privilege section. For any privilege conflicts, load preference is given to Deny > Allow > Revoke.

Description

Optional description can be provided.

Set Privilege

Select and set the role privilege(s), or instantly load privilege(s) of the existing role(s) and user(s) with configuration options explained above. Possible options to set privileges are:

Revoke: Set by default. User(s) assigned with this role will not be able to perform the revoked action(s).

Allow: User(s) assigned with this role will be able to perform the allowed action(s) unrestricted.

Deny: User(s) assigned with this role will not be able to perform the denied action(s).

Revoke and Allow can be changed to any other privilege as required (by providing direct privilege) when the existing roles are later mapped to user(s). But, Deny privilege(s) strictly remain unchanged.

Workspace

If the role privileges are customized in the Set Privilege section, then the privilege settings for features: Summary, Audit, Project, Cluster, Register Image and Connections can be done in the Workspace section.

Project

If the role privileges are customized in the Set Privilege section, then the privilege settings for features: Template, Models, Register Entities, Instances, Data Validation, Workflow, Version Control, Import Export Entities, Sandbox, Pipeline, Compute Environment, Processor Group, Dataset and Applications can be done in the Project section.

Name

Name of the existing user.

Workspace Name

Name of the workspace for which the user is assigned.

Email Id

The registered email id of the user.

Note: Only applicable for Gathr Metastore configuration. 

Assigned Roles

Type of role that is assigned to the user.

Actions

The actions that can be performed are:

l Enable/disable user

l Edit user details

l Edit assigned privileges

l Delete user

Note: Only applicable for Gathr Metastore configuration.

New User

The new user configuration section is divided in two parts, namely, Add Roles and Add Users. The configuration options for both parts are explained below.

Add Roles

This is an optional part.

If the new user has to be assigned with any of the existing roles, it is possible to do so with Add Roles option.

Select Roles

Privilege(s) of the role(s) selected will instantly load in the Set Privilege section. For any privilege conflicts, load preference is given to Deny > Allow > Revoke.

Deny privilege(s) strictly remain unchanged, whereas any additional privilege changes will appear as ‘Direct Privileges’.

Set Privilege

Revoke: Set by default. User(s) assigned with this role will not be able to perform the revoked action(s).

Allow: User(s) assigned with this role will be able to perform the allowed action(s) unrestricted.

Deny: User(s) assigned with this role will not be able to perform the denied action(s).

Workspace

The role privileges for features: Summary, Audit, Project, Register Image and Connections will be visible in the Workspace section.

Project

The role privileges for features: Models, Register Entities, Data Validation, Workflow, Version Control, Import Export Entities, Sandbox, Pipeline, Processor Group, Compute Environment, Dataset and Applications will be visible in the Project section.

Add User

Users selected will get assigned with the privilege(s), if configured in the Add Roles page.

User Name

Unique name for the new user must be provided.

Email Id

Email Id of the new user must be provided.

Password

Password for the new user must be provided.

Confirm Password

Re-type the password.

Language

Choose the language, from English (US) and German (DE).

Configure Artifactory

The user can configure artifactory by checkmarking the checkbox. Provide Artifactory URL, Username and Password.

Roles

Privileges assigned to the user for the role(s) selected from this list can be viewed in the Privilege section.

Privilege

Revoked: You will not be able to perform the revoked action(s). Request your System Administrator to allow any revoked privilege(s) that you may need.

Allowed: You will be able to perform the allowed action(s) unrestricted.

Denied: You will not be able to perform the denied action(s).

Workspace

The role privileges for features: Summary, Audit, Project, Register Image and Connections will be visible in the Workspace section.

Project

The role privileges for features: Models, Register Entities, Data Validation, Workflow, Version Control, Import Export Entities, Sandbox, Pipeline, Processor Group, Compute Environment, Dataset and Applications will be visible in the Project section.

Name

Name of the group created.

Members

Total count of the users in the group.

Roles

Type of roles assigned to the group.

Actions

Options to edit or delete the group.

New Group

The New Group section is divided in two parts, Users and Set Privileges.

Each of them is explained in detail below.

Users

Any number of users that exist in Gathr can be registered using the Add Users to Group option.

Either the users can be added from the given list or a CSV file having the list of users can be uploaded and validated.

Set Privileges

Select Roles

Privilege(s) of the role(s) selected will instantly load in the Set Privilege section.

For any privilege conflicts, load preference is given to Deny > Allow > Revoke.

Deny privilege(s) strictly remain unchanged, whereas any additional privilege changes appear as ‘Direct Privileges’ on the listing page.

Set Privileges as Role

For the privilege(s) that are selected and set, a new role can be created using this option.

If user clicks on SET PRIVILEGES AS ROLES option, a new window will pop up with options to provide the role name and description and thereby register the role.

Set Privilege

Select and set the role privilege(s), or instantly load privilege(s) of the existing role(s) and user(s) with configuration options explained above. Possible options to set privileges are:

Revoke: Set by default. User(s) assigned with this role will not be able to perform the revoked action(s).

Allow: User(s) assigned with this role will be able to perform the allowed action(s) unrestricted.

Deny: User(s) assigned with this role will not be able to perform the denied action(s).

Revoke and Allow can be changed to any other privilege as required (by providing direct privilege) when the existing roles are later mapped to user(s). But, Deny privilege(s) strictly remain unchanged.

Workspace

If the role privileges are customized in the Set Privilege section, then the privilege settings for features: Summary, Audit, Project, Register Image and Connections can be done in the Workspace section.

Project

If the role privileges are customized in the Set Privilege section, then the privilege settings for features: Models, Register Entities, Data Validation, Workflow, Version Control, Import Export Entities, Sandbox, Pipeline, Processor Group, Compute Environment, Dataset and Applications can be done in the Project section.

Component Name

The name of the component.

Type

Type of component. For e.g. Processor, Data Source, Emitter or Analytics.

Action

Edit the Json or Version

Connection Name

The name of the component.

Action

Edit the JSON or Version