ADLS ETL Target

Target Configuration #

Configure the target parameters that are explained below.

Access Options #

Access Option specifies the method for accessing and interacting with ADLS. Azure Storage enables integration with Azure’s cloud-based storage solution to interact with the file system of ADLS.

Connection Name #

Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for ADLS earlier. Or create one as explained in the topic - ADLS Connection →

Container #

ADLS container name in which the transformed data should be emitted.

Path #

Provide Azure Storage/Databricks Mount Point directory path based upon the chosen Access Option.

Output Type #

Output format in which results will be processed.

Delimiter #

Provide the Delimiter in case of CSV output data type.

Output Fields #

Fields that need to be part of the output data.

Partitioning Required #

If checked, data will be partitioned.

Partition Columns #

This option is to select fields on which data will be partitioned.

Save Mode #

Save Mode is used to specify the expected behavior of saving data to a data sink.

Errorifexist: When persisting data, if the data already exists, an exception is expected to be thrown.

Append: When persisting data, if data/table already exists, contents of the Schema are expected to be appended to existing data.

Overwrite: When persisting data, if data/table already exists, existing data is expected to be overwritten by the contents of the Data.

Ignore: When persisting data, if data/table already exists, the save operation is expected to not save the contents of the Data and to not change the existing data. This is similar to a CREATE TABLE IF NOT EXISTS in SQL.

Output Mode #

Output Mode specifies how to write the data.

Enable Trigger #

Trigger defines how frequently a streaming query should be executed.

If trigger is enabled, provide the processing time.

Processing Time #

The time interval or conditions set to determine when streaming data results are emitted or processed.

Compression Type #

Algorithm to use to compress the data.

Add Configuration: To add additional custom ADLS properties in a key-value pair.

Post Action #

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Post-Actions →

Notes #

Optionally, enter notes in the Notes → tab and save the configuration.