SQS ETL Target

SQS emitter allows to emit data in streaming manner into SQS Queues.

Target Configuration #

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for SQS earlier. Or create one as explained in the topic - SQS Connection →

Queue Type: Select either type of Queue for incoming messages, Standard or SQS FIFO.

Queue Name: Name of the Queue where data will be published.

Content-Based Deduplication: Enable content based deduplication for the queue (each of your message has a unique body). The producer can omit the message deduplication ID.

Message Group ID For FIFO Queue: The tag that specifies that a message belongs to a specific message group. Messages that belong to the same message group are guaranteed to be processed in FIFO manner.

Message Deduplication ID for FIFO Queue: The token used for deduplication of messages within the deduplication interval.

Visibility Timeout (in seconds): The length of time (in seconds) that a message received from a queue will be invisible for, to other receiving components.

Message Retention Period (in seconds): The amount of time in which Amazon SQS will retain a message if it does not get deleted.

Maximum Message Size (in bytes): Maximum message size (in bytes) accepted by Amazon SQS.

Receive Message Wait Time (In seconds): The maximum amount of time that a long polling receive call will wait for a message to become available before returning an empty response.

Delivery Delay (in seconds): The amount of time to delay the first delivery of all messages added to this queue.

Output Format: Select the data format in which RabbitMQ should write the data.

Output Fields: Select the fields which should be a part of the output data.

Output Mode: Output mode to be used while writing the data to Streaming sink.

Select the output mode from below options:

Append Mode: Output Mode in which only the new rows in the streaming data will be written to the sink

Complete Mode: Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates.

Update Mode: Output Mode in which only the rows that were updated in the streaming data will be written to the sink every time there are some updates.

Enable Trigger: Trigger defines how frequently a streaming query should be executed.

Processing Time: Trigger time interval in minutes or seconds.

Add Configuration: Enables to configure additional SQS properties.

Post Action #

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Post-Actions →

Notes #

Optionally, enter notes in the Notes → tab and save the configuration.