Databricks Incremental Configuration

Enable Incremental Read #

You can choose to enable incremental read.

This is useful if the need is only to read the objects that have been updated in the source, since the last execution of the application for the given configuration.

Provide the details for enabling incremental read as described below:

Column #

Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal type of values. It is recommended that this column should have sequential, unique and sorted (in increasing order) values.

Offset #

You can specify the value from where you want to begin reading data from the source table.

Read Control Type #

You can control the reading of data from database by utilizing three different approaches.

None #

Here, you can read the data without specifying any limit. From what value of the Column field, the data needs to be read.

Limit by Count #

Here, you can specify the number of records that you want to read in a single request.

Limit by Value #

Here, you can specify the value for configured column till the point where you want to read the data.