Delta SQL Data Source

Delta SQL Channel supports HDFS and JDBC connection. To add a DeltaSQL source into your pipeline, select the source from components panel on the canvas and click on it to configure the Delta SQL source.

Under the Schema Type tab, you can Upload Data File, Fetch From Source, Use Existing Dataset.

Below are the configuration details of the DeltaSQL:

FieldDescription
Source Storage TypeSelect the source storage type from the drop-down. On Databricks cluster both JDBC and HDFS options are available. However, in GCP and Apache only HDFS option is available.
Connection Name

Select the connection name from the available list of connections, from where you would like to read the data.

Override CredentialsUnchecked by default, check the checkbox to override credentials for user specific actions.
UsernameOnce the Override Credentials option is checked, provide the user name through which the Delta SQL service is running.
PasswordProvide the password for Delta SQL override credentials.
Query

Provide delta compatible SQL query to be executed in the component.

For HDFS source table name should be like delta.hdfs://namenode:port/directory_path

Inspect Query

Provide the SQL query to be executed in the component with a set limit in records count for inspect and schema detection.

- For inspect query field you need to provide simple query with table name like “select * from tablename”

- “As of” clause is not supported for inspect query but can be used in query field.

- Also for Inspect query it should be JDBC compatible"

ADD CONFIGURATIONTo add additional custom properties in key-value pairs.
Environment ParamsUser can add further environment parameters. (Optional)

Incremental Read

FieldDescription
Enable Incremental Read

Check the radio button to enable the incremental option. The available options are:

- None

- Normal

- Version

Upon selecting None option the incremental read support will be disabled.

Upon selecting Normal, provide the inputs for the below fields:

Column to CheckSelect a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal types of values.
Start ValueMention a value of the reference column, only the records whose value of the reference column is greater than this value will be read.
Read Control Type

Provides three options to control data to be fetched -None, Limit By Count, and Maximum Value.

None: All the records with value of reference column greater than offset will be read.

Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read.

Limit by Value: All the records with value of reference column greater than offset and less than Column Value field will be read.

For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order.

Upon selecting Version, provide the inputs for the below fields:

Table NameOption to provide delta table name for querying the records.
VersionOption to provide version for querying the records. The records will be read subsequent to the selected version.
ADD CONFIGURATIONTo add additional custom properties in key-value pairs.
Environment ParamsUser can add further environment parameters. (Optional)
Top