RDS ETL Target

RDS emitter allows you to write to RDS databases. RDS is Relational Database service on Cloud.

Target Configuration #

Fetch From Target/Upload Schema File

The data source records needs to be emitted to a RDS target table.

In case if the Gathr application has access to a target table in the RDS database, choose the option Fetch From Target.

In case if Gathr application does not have access to a RDS target table, you can choose the option Upload Schema File in order to map the RDS table columns with the source columns during design-time and confirm the data type for each column. In such cases you can run the application in a registered environment, that has access to all the required resources. During run-time, the application will run on the registered cluster of your choice picking up the configuration values as provided during application design.

When you select the Upload Schema File option, a Schema Results section will get displayed at the bottom of the configuration page.

You can then download the sample schema, provide RDS - Table Column Name against mapping values and verify the data type.

Once it is updated, you can upload the saved file to see a sample of how the records from a source column will be written into the corresponding mapped RDS column.

Connection Name Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for RDS earlier. Or create one as explained in the topic - RDS Connection →

Schema Name Existing database Schema Names whose tables are fetched. (for MSSQL, DB2 and POSTGRES)

Table Name Existing table name of the specified database.

Enable Batch Enable parameter to process batch multiple messages and improve write performance.

Batch Size Batch Size determines how many rows to insert per round trip. This helps the performance on JDBC drivers. This option applies only to writing. It defaults to 1000.

Save Mode Specifies how to handle any existing data in the target. The options are:

Append: Appends data in the configured target.

Overwrite: Overwrites existing data if any, in the target.

Upsert: The existing data in the designated target table will be updated, and any new data will be inserted.

Update: Update the data rows in the target.

Ensure the target table has a primary key to perform update operation.

Complete Output Mode in which all the rows in the streaming data will be written to the sink every time there are some updates.

Enable Trigger Trigger defines how frequently a streaming query will be executed.

Processing Time It will appear only when Enable Trigger checkbox is selected. Processing Time is the trigger time interval in minutes or seconds.

Ignore Values Enable this option to exclude specified values from the data.

Values to ignore Enter comma-separated values to exclude from the data. If a record contains any of these values in columns, even partially, then matching column values will be ignored and set as null.

Priority Priority defines the execution order for the emitters.

ADD CONFIGURATION Enables to configure additional properties.

Post Action #

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Post-Actions →

Notes #

Optionally, enter notes in the Notes → tab and save the configuration.