File Writer Emitter
A local file emitter can save data to local file system. Local file System is the File System where Gathr is deployed.
File Writer Emitter Configuration
To add a File Writer emitter to your pipeline, drag it onto the canvas, connect it to a Data Source or processor, and right-click on it to configure.
| Field | Description |
|---|---|
| File Path | Path where the data file will be read/saved. This is where you can use the Scope Variable using @. To know more about the same, read about Scope Variable. |
| Output Fields | Fields to be included in the Output File. |
| Output Format | Format in which the output file will be saved. |
| Delimiter | Separator to separate different fields. |
| Checkpoint Storage Location | Select the check pointing storage location. Available options are HDFS, S3, and EFS. |
| Checkpoint Connections | Select the connection. Connections are listed corresponding to the selected storage location. |
| Checkpoint Directory | It is the path where Spark Application stores the checkpointing data. For HDFS and EFS, enter the relative path like /user/hadoop/, checkpointingDir system will add suitable prefix by itself. For S3, enter an absolute path like: S3://BucketName/checkpointingDir |
| Time-Based Check Point | Select checkbox to enable timebased checkpoint on each pipeline run i.e. in each pipeline run above provided checkpoint location will be appended with current time in millis. |
| Output Mode | Mode in which File writer will run. Output mode to be used while writing the data to Streaming emitter. Select the output mode from the given three options: Append: Output Mode in which only the new rows in the streaming data will be written to the sink. Complete Mode: Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates. Update Mode: Output Mode in which only the rows that were updated in the streaming data will be written to the sink every time there are some updates. |
ADD CONFIGURATION: Enables to configure additional properties.
If you have any feedback on Gathr documentation, please email us!