Gathr Deployment on Azure

The objective of this topic is to assist the user to install Gathr on an Azure environment.

Hardware and Software Configurations

Create Azure VM, preferably on the same VNET and Subnet where the Azure Databricks cluster is running with the below minimal specification.

Open the below ports in VM’s NSG:

Note:

CIDR range of user: This means that the customer needs to whitelist the IP ranges where the user is trying to access the Web UI of Gathr, RabbitMQ, and Elasticsearch.

CIDR range of Databricks Cluster: This means the customer needs to whitelist the IP ranges of the Databricks cluster on Azure VM.

Prerequisites

Azure Portal Access

A virtual machine with the following resources should be available to setup Gathr:

- Resource Group

- Virtual machines

- Virtual Networks

- Subnet

- Network Security Group

- Azure Databricks

Below details are required to configure Azure Databricks with Gathr

- Azure Databricks instance URL

- Azure Databricks Authentication token

- Azure Databricks DBFS Path

Gathr Installation specific prerequisites

- The SSH access should be available for Azure VM to Setup Gathr.

- The Linux SSH user used for Gathr installation should have the sudo privilege.

- Make sure the Gathr_Setup_bundle (tar.gz) and other artifacts should be downloaded from centralized location provided by the Gathr Support team and make it available on Azure VM.

- Valid Gathr license file should be available during the installation (provided by the Gathr Support team).

- Ensure the connectivity between Gathr VM, Azure Databricks cluster, and any external components (Hadoop, Kafka, Solr, etc.) if required.

The below specified software should be installed on Azure VM as part of the Gathr installation.

RabbitMQ, Elasticsearch components are optional; however, pipeline monitoring and error handling features won’t be provided in Gathr if these are not available.

Note: The Hadoop components (Kafka, HBase, Hive, etc.) are optional and can be installed and configured with Gathr depending upon the use case requirements.

Setup

To setup Gathr, you should have sufficient privileges to create and manage the resources (Resource Group, Virtual machines, Virtual Networks, Subnet, Network Security Group, Azure Databricks) in Azure.

Note: Hadoop components are optional; depending upon the requirements. You may need to install the Hadoop services (HDFS, Hive, Yarn, etc.) manually and update the configurations accordingly in Gathr.

Create Virtual Machine

Steps to Create Azure VM

- Type virtual machines in the search.

- Under Services, select Virtual machines.

- Select Create and then click Virtual machine on the Virtual machines page. The Create a virtual machine page opens.

- In the Basics tab, under Project details, make sure that the correct subscription is selected and then create a new resource group. Type myResourceGroup for the name.*.

- Under Instance details, type tomcatVM for the Virtual machine name, and choose CentOS Gen1 for your Image. Leave the other defaults. The default size and pricing are only shown as an example. Size availability and pricing are dependent on your region and subscription.

- Under Administrator account, select Password.

- Under Inbound port rules > Public inbound ports, choose to Allow selected ports and then select SSH (22) and HTTP (80) from the drop-down.

- Leave the remaining defaults and then select the Review + create button at the bottom of the page.

- On the Create a virtual machine page, you can see the details about the VM you are about to create. When you are ready, select Create.

- When the deployment is finished, select Go to resource.

- On the page for your new VM, select the public IP address and copy it to your clipboard.

Below are the ports that need to be opened in VM Security Group

Install Software

This section describes the steps required to install the prerequisite software on Virtual Machine launched on Cloud.

ssh into the Gathr VM to continue with the following steps

- Install Java 8

- Install Java 8

Set java home in. bashrc

Install RabbitMQ

This is an optional component. However, it is important as it is used for pipeline error handling in Gathr.

Install this package before installing RMQ:

Download the package

Start using this command:

Enable it with this command:

To check the status, use this command:

Enable the plugins with this command:

To create a test user:   

Install Zookeeper

Install Zookeeper 3.5.7 as follows:

Copy the zookeeper tar file either from sax_bundle.

Extract it:

Set the IP and Port in

Start the zookeeper

Install Postgres 10

Install Postgres 10 as follows:

1. Install Postgres repo as a root user into the system:

2. Install Postgresql10:

3. Initialize PGDATA:

4. Login into Postgres: su - postgres -c "psql"

5. Change the Postgres password

postgres=#\password postgres

Settings for PostgreSQL

Login as a Postgres user: su – postgres.

Add the IPs in IP4 to allow the permission:

Edit the postgresql.conf and replace listen_address from localhost to *

.

Restart the postgres: systemctl restart postgresql-10.service

Install Elasticsearch 6.4.1

This is an optional component; however, it is important as it will be used for monitoring Gathr pipelines.

Install Elasticsearch 6.4.1 as follows

- Copy Elasticsearch from sax_bundle.

- Extract the bundle.

- Open /elasticsearch-6.4.1/conf/elasticsearch.yaml and make these changes:

Setting for Elasticsearch

1. max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

2. max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

For 1. Add this line in /etc/security/limits.conf

Note: Sax is the user from which you are starting elasticsearch.

For 2. Run this command: sudo sysctl -w vm.max_map_count=262144

Start Elasticsearch in the background: nohup ./elasticsearch &

Install Gathr

Install and run in embedded mode. Copy the Gathr tar file to the Virtual Machine. Extract the tar.

Run this command to start StreamAnalytix in embedded mode:

Note: Logs are located in <GathrInstallationDir>/logs and <GathrInstallationDir>/server/tomcat/logs directories.

You can check the log files in these directories for any issues during the installation process.

To open Gathr use the below command:

Accept the End User License Agreement and hit the Next button.

The Upload License page opens.

Upload the license and confirm.

Login page is displayed.

Note: Now, the user will need to switch Gathr from embedded to cluster mode.

Follow the sections given below to switch Gathr from embedded to cluster mode:

Login to Gathr using the default username & password.

Navigate to the Setup >> Gathr and update the below details:

- Gathr Web URL

- Zookeeper Gathr Node

- Zookeeper Configuration Node

Navigate to the Setup >> Database and update the below details.

- Connection URL

- User

- Password

- Run Script

Note: Check the Run Script and click on Save. It will execute the DDL & DML in Gathr Metastore.

Navigate to the Setup >> Messaging Queue and update the below details:

- Messaging Type

- Host List

- User

- Password

Navigate to the Setup >> Elasticsearch and update the below details:

- Elasticsearch Connection URL

- Elasticsearch Cluster Name

After updating the details, restart Gathr with -config.reload=true.

Copy Cloud Vendor-specific war

Copy Cloud Vendor-specific war file into tomcat

For Azure-Databricks:

The war will get extracted in server/tomcat/webapps. Now stop tomcat and configure application files.

For Azure-Databricks:

Configure Cloud Vendor-specific details in YAML

For Azure-Databricks:

Configure Databricks details in yaml

Copy jar & init-scripts on DBFS

Restart in Cluster Mode and upload license

Steps to Restart Gathr

In case there are any updates to be done in configurations, you can restart Gathr by providing the below commands

Steps to Uninstall Gathr

1. Stop/Kill the Bootstrap Process.

2. Delete Gathr installation directory and its dependencies (like RMQ, ZK etc)

3. Delete the Gathr database.