File Writer Emitter

A local file emitter can save data to local file system. Local file System is the File System where Gathr is deployed.

File Writer Emitter Configuration

To add a File Writer emitter to your pipeline, drag it onto the canvas, connect it to a Data Source or processor, and right-click on it to configure.

FieldDescription
File PathPath where the data file will be read/saved. This is where you can use the Scope Variable using @. To know more about the same, read about Scope Variable.
Output FieldsFields to be included in the Output File.
Output FormatFormat in which the output file will be saved.
DelimiterSeparator to separate different fields.
Checkpoint Storage LocationSelect the check pointing storage location. Available options are HDFS, S3, and EFS.
Checkpoint ConnectionsSelect the connection. Connections are listed corresponding to the selected storage location.
Checkpoint Directory

It is the path where Spark Application stores the checkpointing data.

For HDFS and EFS, enter the relative path like /user/hadoop/, checkpointingDir system will add suitable prefix by itself.

For S3, enter an absolute path like: S3://BucketName/checkpointingDir

Time-Based Check PointSelect checkbox to enable timebased checkpoint on each pipeline run i.e. in each pipeline run above provided checkpoint location will be appended with current time in millis.
Output Mode

Mode in which File writer will run. Output mode to be used while writing the data to Streaming emitter. Select the output mode from the given three options:

Append: Output Mode in which only the new rows in the streaming data will be written to the sink.

Complete Mode: Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates.

Update Mode: Output Mode in which only the rows that were updated in the streaming data will be written to the sink every time there are some updates.

ADD CONFIGURATION: Enables to configure additional properties.

Top