Create Data Ingestion Application

Gathr supports a wide range of sources and targets that can be configured while creating an ingestion application.

There are prebuilt operators and functions that can help you transform your data as desired before you transfer it to the supported target data warehousing solutions.

Notes:

The steps given below captures only a basic flow on how to create an ingestion application.

You can create an ingestion application with any source and target as per requirement by following similar configuring steps as described below.

The configuration parameters for each connection, source, target, operators (transformations) and functions are explained separately in the topics which can be referred while configuring the required entity.

Refer to the Components section to check the configuration parameters for connections, sources, and targets.

The steps to create a data ingestion application are as follows:

1. Go to the Applications home page from the main menu and click on the CREATE INGESTION APPLICATION option.

   

   All the sources that are supported by Gathr will get displayed as shown below:

   

   If the source you are looking for is not there, click on the REQUEST ONE option and update the Gathr support team about the same.

   To subscribe to Premium connectors, or request available connectors request your administrator. For more details, see the Connector Marketplace topic.

   Also, if you are using any Premium Connectors in your application, you must be in a paid plan to run the application. Please see the Subscription Plans topic for more details.

2. Select the desired source and click NEXT to proceed further.

   If your selected source requires a connection to communicate with Gathr, please add a new connection or use any existing one if available in the list of Connections.

   Each connection will have different configurations based on the type of source that it needs to connect. If you need help with configuration options for any connection, see the Components section.

3. Click on TEST CONNECTION to make sure that a successful connection is established, configure the source parameters on the Configuration page and click NEXT.

   If you need help with configuration options for any source, see Ingestion Data Sources →

4. Check the populated schema details and click NEXT.

   You can choose to edit the basic details (like column alias names, column data types, drop specific columns, etc.) on this page. Similar and more advanced options to perform such operations will be available in the transformations step.

   Next to Column Alias there are two options.

   With the Update Header option you can upload a comma separated file to update all the headers in one go.

   With the other option you can remove all the unsupported characters from the column alias in a single-click.

   

   Note:

   For Advanced Mongo data source, on the Schema page there are three additional options:

   • MARK ALL AS STRING: Option to convert the data type to string for all the columns in one click.

   • DOWNLOAD SCHEMA: Option to download incoming schema as a JSON file. The same can be edited to modify the data types of columns.

   • UPLOAD SCHEMA: Option to upload schema JSON file. Make sure that only the data types are updated in the JSON file for the pipeline to give expected results.

   

5. Optional Step: You can choose to enable incremental read on the Advanced Configuration page.

   This is useful if the need is only to read the objects that have been updated in the source, since the last execution of the application for the given configuration.

6. Optional Step: Click on the plus icon placed at the top right side, and a canvas will appear containing your selected data source. Now, select transformations from the list that you may require to apply on your source data.

   

7. Add configuration details for the selected Transformations and click NEXT.

   If you need help with configuration options for any transformation, see Transformations →

   For specific transformations, if you need help with configuration options for any functions, like Array, String, Lookup, and so on, see Functions →

8. Click NEXT until you reach on the Target page.

   All the targets that are supported by Gathr will get displayed as shown below:

   

   If the target you are looking for is not there, click on the REQUEST ONE option and update the Gathr support team about the same.

9. Select the desired target and click NEXT to proceed further.

   If your selected target requires a connection to communicate with Gathr, please add a new connection or use any existing one if available in the list of Connections.

   Each connection will have different configurations based on the type of target that it needs to connect. If you need help with configuration options for any connection, see Components section.

10. Once a successful connection is established, configure the target parameters on the Configuration page and click NEXT.

    If you need help with configuration options for any target, see Ingestion Targets →

11. On the Configure page, provide the details for each field as per the topic, Configure Ingestion App.

12. Once all the necessary fields are updated, click on SAVE button.

The ingestion application once created will appear on the listing page.

To explore about the actions that can be performed on the ingestion application, see Data Ingestion Applications →