SFTP ETL Target

SFTP as a target is supported by Gathr.

Target Configuration #

Configure the data emitter parameters as explained below.

Connection Name #

Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for SFTP earlier. Or create one as explained in the topic - SFTP Connection →

Use the Test Connection option to ensure that the connection with the SFTP channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Maximum SFTP Channels #

Number of parallel write channels, that Gathr will open to your SFTP Server. If left blank, Gathr will evaluate this based on input data size.

Path #

Specify the SFTP target path.

Single statement MVEL expression is also supported for creating dynamic output path.

Example: /home/ec2-user/sales_@{java.time.LocalDate.now()} input will create /home/ec2-user/sales_1970-01-01 path.

Control Number of Output Files #

Control Number of Output Files: The number of output files will be generated at target location.

None: Gathr will intelligently decide the number of output files based on the input data size.

Single File: Option to write the entire input data in a single file.

Maximum Files: Option to limit the maximum number of output files that can be created while writing to SFTP server.

If Maximum Files is selected, then provide the Number of Files as described below:

Number of Files #

Option to specify the maximum number of output files.

Auto-generate Filename #

Option to auto-generate file names. It will take application name as file name and append UUID to it.

Example: <application_name_UUID>

File Name #

A custom file name should be provided.

Single statement MVEL expression is also supported for creating dynamic file name.

Example: sales_@{java.time.LocalDate.now()} input will create sales_1970-01-01. as output file.

Append Unique Identifiers #

Select unique identifiers to be appended to the filename.

General syntax for the filename is:

CustomName<MVEL_Exp>_<SequenceID>_<Timestamp>

Timestamp Format: Specify the timestamp format to be appended in the filename.

Example: yyyy-MM-dd’T’HH:mm:ss.SSS’Z’

Default is unixtime in milliseconds.

Timestamp formats contains below characters are not supported:

• Asterisk

• Slash

File Format #

Output file format choice in which the results should be processed.

Supported file formats are: Delimited, JSON, Parquet, ORC and AVRO.

Delimiter #

A message field separator should be selected for CSV (delimited) file formats.

File Content #

conventions for output file content.

Include Header #

Option to write the first row of the data file as header.

Output Fields #

Message fields that needs to be written into the file must be selected.

Write Mode #

SFTP target supports writing of the incoming data using one of the below modes:

• Never Overwrite: This option will always create a new file at the destination with UUID appended on filename.

• ErrorIfExists: An exception will be thrown if the current file already exists at the destination.

• Append: Data will be appended if the file already exists in the destination. Else, new file will be created. This option is applicable for CSV and JSON files only.

• Overwrite: Data will be overwritten if the file already exists in the destination. Else, new file will be created.

• Overwrite If Different Size: Data will be overwritten if the file already exists and the source data is of different size than the target data. If the file does not exist at the destination location, then new file(s) will be created.

Compression #

Choose the output file compression format in which the results should be processed.

File Encoding #

Choose the encoding for the output file from UTF-8 or UTF-16.

Enable Retry Mechanism #

Option to retry the job after an interruption while writing data. The job will resume from the point where it was interrupted.

Retry Count #

Specify the number of times to retry job in case of interruption.

Retry Delay #

Specify the retry delay in milliseconds. This is the time that application should wait before retrying.

Add Configuration: Additional properties can be added using this option as key-value pairs.

Post Action #

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Post-Actions →

Notes #

Optionally, enter notes in the Notes → tab and save the configuration.