Batch Cosmos Data Source
On Cosmos Channel, you will be able to read from selected container of a Database in batches. You can use custom query in case of Batch Channel.
Configuring a Batch Cosmos Data Source
To add a Batch Cosmos Data Source into your pipeline, drag the Data Source to the canvas and click on it to configure.
Under the Schema Type tab, you can Upload Data File and Fetch From Source.
| Field | Description |
|---|---|
| Connection Name | Select the connection name from the available list of connections, from where you would like to read the data. |
| Override Credentials | Unchecked by default, check mark the checkbox to override credentials for user specific actions. |
| Key | Provide the Azure Cosmos DB key. Click TEST Connection to test the execution of the connection. |
| Database | Select the Cosmos Database from the drop-down list. |
| Container | Select the Cosmos Database from the drop-down list. |
| Enable Change Feed | Change feed enables to read the latest records or changed records. Effect comes only during pipeline flow. |
| Change Feed From Beginning | If set to True, data will be read from beginning. |
| CosmosDB Checkpoint Directory | It is the file path where Cosmos stores the checkpoint data for Change feed. |
| ADD CONFIGURATION | To add additional custom properties in key-value pairs. |
| Configure Pre-Action in Source → |
Click Next for Incremental Read option
| Field | Description |
|---|---|
| Enable Incremental Read | Check this checkbox to enable the incremental read support. |
| Column to Check | Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp and decimal type of values. |
| Start Value | Mention the value of reference column. Only the records whose value of the reference column is greater than this value will be read. |
| Read Control Type | Provides 3 options to control how data will be fetched: None, Limit by Count and Limit by Value. None: All the records with value of reference column greater than offset will be read. Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read. Limit By Value: All the records with value of reference column greater than offset and less than Column Value field will be read. For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order |
If you have any feedback on Gathr documentation, please email us!