Data Ingestion Applications
Data Ingestion is one of the most crucial steps of the data handling process. It involves transferring data from a source to a target after applying any desired transformations.
Gathr offers a very user-friendly interface to create data ingestion applications on its SaaS platform and trigger them on either Gathr cluster or user preferred clusters that are registered with Gathr using Compute Setup →
Useful references for Data Ingestion Applications:
Ingestion Applications Listing Page
All the data ingestion applications you create in your project will get listed on its Ingestion page.
Listed Applications Details
Details shown for each listed application are:
Source and Target components.
Application name and status.
Time since last modified. and next sync schedule info.
Assigned cluster and scheduling information.
Application description and custom tags added by the user.
Actions on Listed Applications
The actions available on the listing page for applications are described below:
| Actions | Description |
|---|---|
| Search | You can specify the relevant keywords in the search bar to find the required application that you are looking for in the existing list of applications. |
| Sort By | You can sort the listed applications by their name, created date or next scheduled date, either in ascending or descending order. |
| Filter By Favorite | Filter applications based on favorites. Marking an application as a favorite enables you to prioritize and access frequently used applications more conveniently. |
| Refresh Applications | You can refresh all the listed applications in a single click with this action. |
| Create Application | You can create a new data ingestion application with this action. The steps to create data ingestion applications are explained in the subsequent topics. |
| Start Application | You can trigger any existing application with this action. |
| Stop Application | You can directly stop any running application with this action. |
| View Application | You can view the application details with this action. This option is further explained in the View Applications → topic. |
| Mark As Favorite | Marking an Ingestion application as a favorite allows you to easily access and prioritize it within your project. This feature ensures that your most used and important applications are readily available. To mark an application as a favorite, simply click on Mark As Favorite option from the ellipsis menu next to it. Once marked as a favorite, the option will change to Unmark as Favorite, allowing you to easily revert this action whenever needed. |
| Edit Application | You can edit an existing application with desired source, transformation, target or configuration details with this action and update to save the changes. |
| Delete Application | This option allows you to delete an Ingestion application and send it to the Gathr’s Recycle Bin. |
| Schedule Application | You can schedule an application to automatically trigger it as per the scheduling options listed as follows: - Start Time. - End Date and Time. - Frequency - minutes, hourly, daily, weekly, monthly, and yearly. - The minutes frequency can be set to 15, 20 or 30 minutes. - Every - hour(s), week(s), month(s) and so on, as per the specified interval of frequency. For Hourly frequency, an additional option Start Time will be available to provide the start time in hours and minutes. - Set the end date and time to define when the scheduled executions should stop. You can specify an end date and time after which the scheduling will no longer occur. This is useful for scheduling tasks that have a finite duration or are only needed for a certain period. - Specify the time zone in which the scheduling should occur. This ensures that the application runs at the desired time in the specified time zone. - An auto-generated Cron Expression. |
| Change Cluster | You can change the cluster on which an ingestion or ETL application gets triggered. Options are - Extra Small, Small, Medium, and Large. Also, there is an option to utilize clusters that are registered with Gathr for running Data Ingestion applications. The prerequisite for that is to setup compute environments in Gathr from User Settings > Compute Setup → tab. To trigger an application on any of the clusters other than the Small cluster, you must have a Gathr Advanced or Business plan subscription. For account upgrade information, see Subscription → |
If you have any feedback on Gathr documentation, please email us!