ADLS Ingestion Source

On a ADLS Batch Channel you will be able to read data from specified ADLS Container.

Data Source Configuration

Configure the data source parameters that are explained below.

Fetch From Source/Upload Data File

For designing the application, you can either fetch the sample data from the Amazon S3 source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design.

Upload Data File

If Upload Data File is selected to fetch sample data, provide the below details.

File Format

Select the sample file format (file type) depending on the data type.

Gathr-supported file formats for ADLS data source are CSV, JSON, TEXT, XML, Fixed Length, Avro, Parquet, and ORC.

For CSV file format, select its corresponding delimiter.

Header Included

Enable this option to read the first row as a header if your Amazon S3 data is in CSV format.

Upload

Please upload the sample file as per the file format selected above.


Fetch From Source

If Fetch From Source is selected, continue configuring the data source.


Connection Name

Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for ADLS earlier. Or create one as explained in the topic - ADLS Connection →


Container

ADLS container name from which the data should be read.


ADLS Directory Path

Provide directory path for ADLS file system.


File Filter

Provide a file pattern example: *csv/*json to retrieve the available files.


Recursive File Lookup

Check the option to retrieve the files from current/sub-folder(s).


Add Configuration: Additional properties can be added using this option as key-value pairs.


Schema

Check the populated schema details. For more details, see Schema Preview →


Advanced Configuration

Optionally, you can enable incremental read. For more details, see ADLS Incremental Configuration →.

Top