Create Connections

A superuser can create new connections using the Connections tab. To add a new connection, follow the below steps:

• Login as a Superuser.

• Go to Connections page.

• Click on Add Connection.

Select the component from the drop-down list for which you wish to create a connection.

ADLS #

For creating an ADLS connection, select ADLS from the Component Type drop-down list and provide connection details as explained below:

Upon selecting Azure AD, provide the below fields:

Upon selecting Access Key, provide the below fields:

The Credential Passthrough option allows you to authenticate automatically to Azure Data Lake Storage from Azure Databricks clusters using the identity that you use to log in to Azure Databricks.

AWSIoT #

For creating a AWS IoT connection, select AWS IoT from the Component Type drop-down list and provide connection details as explained below.

Azure Blob #

For creating a Azure Blob connection, select Aure Blob from the Component Type drop-down list and provide connection details as explained below:

Cassandra #

For creating a Cassandra connection, select Cassandra from the Component Type drop-down list and provide connection details as explained below:

Cosmos #

For creating a Cosmos connection, select Cosmos from the Component Type drop-down list and provide connection details as explained below:

Couchbase #

For creating a Couchbase connection, select Couchbase from the Component Type drop-down list and provide connection details as explained below:

EFS #

For creating an Elasticsearch connection, select Elasticsearch from the Component Type drop-down list and provide connections details as explained below.

ElasticSearch #

For creating an Elasticsearch connection, select Elasticsearch from the Component Type drop-down list and provide connections details as explained below.

GCP #

To access GCP services, it is mandatory to Generate Service Account Key with below permission for read and write.

Big query:

For the BigQuery Data Sources and Emitters to run successfully in Gathr applications, the following permissions/policy actions should be assigned in Google Cloud:

• bigquery.datasets.get

• bigquery.jobs.create

• bigquery.jobs.delete

• bigquery.jobs.get

• bigquery.jobs.list

• bigquery.jobs.listAll

• bigquery.jobs.listExecutionMetadata

• bigquery.jobs.update

• bigquery.tables.create

• bigquery.tables.create

• bigquery.tables.getData

• bigquery.tables.list

• bigquery.tables.update

• bigquery.tables.updateData

• storage.buckets.get

• storage.buckets.create

• storage.buckets.list

• storage.objects.create

• storage.objects.delete

• storage.objects.get

• storage.objects.list

GCS:

For the GCS Data Sources and Emitters to run successfully in Gathr applications, the following permissions/policy actions should be assigned in Google Cloud:

• storage.buckets.get

• storage.buckets.create

• storage.buckets.list

• storage.objects.create

• storage.objects.delete

• storage.objects.get

• storage.objects.list

After entering all the details, click on the Test Connection button to validate if the connection is running properly.

Click on Create button to save the changes.

For using GCP Connection in gathr, generate service account key.

See the sub-topic Generate Service Account Key to know the steps required for generating Service Account ID and Service Account Key for upload.

Generate Service Account Key #

Prerequisite

You must have a Google account to access the GCP services.

Steps required to generate service account key

Follow these steps in GCP to allow Gathr the access to your GCP Component:

• Create a Service Account and key

• Enable the required APIs

Each of these steps are explained below:

Steps to generate Service Account ID and Key:

1. Use the below link to access/create Google account.

   https://console.cloud.google.com/iam-admin/serviceaccounts

2. Sign in to GCP.

3. Create a new project to proceed further. Else, use any existing project to continue.

4. Navigate to IAM & Admin section and click on the Service Accounts option.

5. On the Service Accounts page, click on the CREATE SERVICE ACCOUNT option as shown below:

   

6. Provide the service account details and click CREATE AND CONTINUE.

   

7. (Optional steps) Grant the service account access to project and users.

   

   Click Done.

8. The created service account will be listed on the Service accounts page along with the available ones (if any).

   This Service Account ID should be provided in Gathr during the GCP connection configuration.

   

   Click on the ellipses under Actions and select Manage Keys.

9. Click on ADD KEY to create a new key or download an existing key.

   Choose a key type from the available options, JSON or P12, and click CREATE.

   

   The created Private key will be downloaded and saved locally.

   

   This Private Key file should be uploaded in Gathr during the GCP connection configuration.

Enable Google Sheets API and Drive API #

1. Search for APIs & Services on Google console or follow the link given below:

   https://console.cloud.google.com/apis

2. Click on ENABLE APIS AND SERVICES option.

3. On the API Library page, search and open google sheets api.

   

4. On the Google Sheets API page, click on ENABLE.

   A confirmation will appear once the Google Sheets API is enabled.

5. On the API Library page, search and open google drive api.

6. On the Google Drive API page, click on ENABLE.

   A confirmation will appear once the Google Drive API is enabled.

Share Spreadsheet with Service Account ID

1. Navigate to Google Sheets and open a spreadsheet.

2. Share you Google Spreadsheet with the created Service Account.

   

   👉

   Make sure to share a google sheet and not any other file type.

   After completing all the required steps navigate to Gathr console and provide the necessary details to create the connection.

   Click below link to know more about how to create and manage service account keys using the Google Cloud Console.

   https://cloud.google.com/iam/docs/creating-managing-service-account-keys

Google Spreadsheet #

For creating Google Spreadsheet connection, select Google Spreadsheet option from the Component Type drop-down list and provide connection details as explained below:

See the topic Generate Service Account Key to know the steps required for generating Service Account ID and Service Account Key for upload.

Enable Google Sheets API and Drive API #

1. Search for APIs & Services on Google console or follow the link given below:

   https://console.cloud.google.com/apis

2. Click on ENABLE APIS AND SERVICES option.

3. On the API Library page, search and open google sheets api.

   

4. On the Google Sheets API page, click on ENABLE.

   A confirmation will appear once the Google Sheets API is enabled.

5. On the API Library page, search and open google drive api.

6. On the Google Drive API page, click on ENABLE.

   A confirmation will appear once the Google Drive API is enabled.

Share Spreadsheet with Service Account ID #

1. Navigate to Google Sheets and open a spreadsheet.

2. Share you Google Spreadsheet with the created Service Account.

   

   👉

   Make sure to share a google sheet and not any other file type.

   After completing all the required steps navigate to Gathr console and provide the necessary details to create the connection.

   Click below link to know more about how to create and manage service account keys using the Google Cloud Console.

   https://cloud.google.com/iam/docs/creating-managing-service-account-keys

Hbase #

For creating an Hbase connection, select Hbase from the Component Type drop-down list and provide connections details as explained below.

HDFS #

For creating a HDFS connection, select HDFS from the Component Type drop-down list and provide connections details as explained below.

HIVE Emitter #

For creating a HIVE Emitter connection, Select HIVE Emitter from the Component Type drop-down list and provide connections details as explained below.

The value of Hive Server2 URL will be the value of HiveServer2 Interactive JDBC url (given the in the screenshot). In the HDP 3.1.0 deployment, this is an additional property:

HiveServer2 Interactive JDBC URL: The value is as mentioned below:

IBM-MQ #

For creating an IBM MQ connection, select IBM MQ from the Component Type drop-down list and provide connections details as explained below:

JDBC #

For creating a JDBC connection, select JDBC from the Component Type drop-down list and provide connections details as explained below:

Jira #

For creating a Jira connection, select Jira from the Component Type drop-down list and provide connections details as explained below:

Connection Name: Name of the connection to be created.

Host: The hostname of your JIRA application should be provided.

Example: gathr.atlassian.net/

Email: The email ID registered with your JIRA account should be provided.

Api ID: Token created in JIRA account should be provided to authenticate the connection.

See the sub-topic Generate API Token for JIRA to know the steps required for generating API tokens in JIRA.

After entering all the details, click on the TEST button.

If the connection service identification and authentication details are provided correctly, a success message stating “connection available” is generated. Click on the CREATE button to save the changes.

If the details are incorrect or the server is down, you will get a message “Connection unavailable”.

Generate API Token for JIRA #

To authenticate the connection between Gathr and JIRA, you need to create an API token in your JIRA account and provide it in Gathr.

Steps to create an API token in JIRA

1. Login to your JIRA account and navigate to Account Settings:

   

2. Go to Security > Create and manage API tokens.

   

3. On the API Tokens page, create an API token.

   

4. Provide a label name and proceed to create the API token.

   

5. Copy the API token and use it to create the connection in Gathr.

   

Kafka #

For creating a Kafka connection, select Kafka from the Component Type drop-down list and provide connections details as explained below.

Kinesis #

For creating a Kinesis connection, select Kinesis from the Component Type drop-down list and provide other details required for creating the connection.

Users can also choose to authenticate Kinesis connections using Instance Profile option.

KUDU #

For creating a KUDU connection, select KUDU from the Component Type drop-down list and provide other details required for creating the connection.

Mongo DB #

For creating Mongo DB connection, select Mongo DB from the Component Type drop-down list and provide details required for creation the connection.

MQTT #

For creating an MQTT connection, select MQTT from the Component Type drop-down list and provide other details required for creating the connection.

OpenJMS #

For creating an OpenJMS connection, select OpenJMS from the Component Type drop-down list and provide other details required for creating the connection.

RedShift #

For creating a RedShift connection, select RedShift from the Component Type drop-down list and provide other details required for creating the connection.

RabbitMQ #

For creating a RabbitMQ connection, Select RabbitMQ from the Component Type drop-down list and provide connections details as explained below.

DBFS #

For creating a DBFS connection, select DBFS from the Component Type drop-down list and provide other details required for creating the connection.

RDS #

For creating a RDS connection, select RDS from the Component Type drop-down list and provide other details required for creating the connection.

S3 #

For creating a S3 connection, select S3 from the Component Type drop-down list and provide other details required for creating the connection.

SQS #

For creating an SQS connection, select SQS from the Component Type drop-down list and provide other details required for creating the connection.

Salesforce #

For creating a Salesforce connection, select Salesforce from the Component Type drop-down list and provide other details required for creating the connection.

Snowflake #

For creating a Snowflake connection, select Snowflake from the Component Type drop-down list and provide connections details as explained below.

After entering all the details, click on the TEST button.

If the connection service identification and authentication details are provided correctly, a success message stating “connection available” is generated. Click on the CREATE button to save the changes.

If the details are incorrect or the server is down, you will get a message “Connection unavailable”.

Socket #

For creating a Socket connection, select Socket from the Component Type drop-down list and provide connections details as explained below.

Solr #

For creating a Solr connection, Select Solr from the Component Type drop-down list and provide connections details as explained below.

Tibco #

For creating a Tibco connection, select Tibco from the Component Type drop-down list and provide connections details as explained below.

Twitter #

For creating a Twitter connection, select Twitter from the Component Type drop-down list and provide connections details as explained below.

Vertica #

For creating a Vertica connection, select Vertica from the Component Type drop-down list and provide connections details as explained below.