ADLS Incremental Configuration

Enable Incremental Read #

Option to read the objects updated in the source since the last execution of the application for the given configuration.

Provide the details for enabling incremental read as described below:

Read By #

Choose the incremental read condition to be applied and data to be read as per File Modification Time or Column Partition.

If File Modification Time is chosen, provide the offset value.

Offset #

Specifies the last modified time of a file. All the files with the last modified time greater than the specified value will be read.

If Column Partition is chosen, provide the below details:

Read Control Type #

Control how you want to read the data from the source with one of the below options:

• None: Read the data without specifying any limit. Provide a record from the selected Column, and the data will be read starting from the provided value.

• Limit by Value: Read the data till the specified limit. Provide a record from the selected Column, and the data will be read till the provided value.

• Limit by Increment Size: Specify the number of records you want to read in a single request.

Column #

Select a column on which incremental read will work. Displays the list of columns with integer, long, date, timestamp, and decimal values. It is recommended that this column should have sequential, unique, and sorted values (in increasing order).

Start Value #

Mention a value to be set as the base for the incremental read with reference to the selected column. Only the records with values greater than the specified value will be read.

Max Value #

Mention a maximum value for the incremental read with reference to the selected column. Only the records with values lower than the specified value will be read.

Increment Size #

Specify the amount of new or changed data to read and process in each incremental step for efficient updates and synchronization.