Turnpike Processor

Turnpike is used with streaming dataset to utilize the benefits of batch transformations in streaming pipeline.

User would also be able to perform sequential and priority-based execution of flows (Processors and Emitters).

The configuration details are as under:

Ignore Action If Empty Check mark this option that gives to the control to ignore the child component actions if current if current micro batch is not having any option.

Output Mode Output mode to be used while writing the data to Streaming emitter. Select the output mode from the given three options:

Append: Output mode in which only the new rows in the streaming data will be written to the sink.

When aggregating a streaming data source without watermark, avoid using ‘Append’ as the output mode.

Complete Mode: Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates.

Update Mode: Output Mode in which only the rows that were updated in the streaming data will be written to the sink every time there are some updates.

Enable Trigger Trigger defines how frequently a streaming query should be executed.

Trigger Type The supported trigger types are: One-time Micro Batch and Fixed interval Micro-Batch.

One-time Micro Batch: A trigger that processes only one batch of data in a streaming query, then terminates the query.

Fixed interval Micro-Batch: A trigger policy that runs the query periodically based on an interval in processing time.

Processing Time: Upon selecting the Fixed interval Micro-Batch option, provide value for the tiem interval that governs the trigger policy.

Click the +ADD CONFIGURATION button to add further configurations in key value pair.