RDS Data Source

RDS allows you to write to RDS DB Engine, which could be through SSl or without SSL. RDS is Relational Database service on Cloud.

RDS Channel can read in Batch from the RDS Databases (Postgresql, MySql, Oracle, Mssql). RDS is Relational Database service on Cloud. The properties of RDS are similar to those of a JDBC Connector with one addition of SSL Security.

SSL Security can be enabled on RDS Databases.

System should be able to connect, read and write from SSL Secured RDS.

If security is enabled, it will be configured in Connection and automatically propagated to channel.

Configuring RDS Data Source

To add an RDS Data Source to your pipeline, drag the Data Source to the canvas and right-click on it to configure.

Under the Schema Type tab, select Fetch From Source or Upload Data File:

FieldDescription
Connection Name

Connections are the Service identifiers.

Select the connection name from the available list of connections for data ingestion.

QueryHive compatible SQL query to be executed in the component.
Enable Query PartitioningTables will be partitioned and loaded RDDs if this check-box is enabled. This enables parallel reading of data from the table.
Number of PartitionsSpecifies no of parallel threads to be invoked to read from JDBC in spark.
Partition on ColumnPartitioning column can be any column of type Integer, on which spark will perform partitioning to read data in parallel.
Lower Bound/Upper BoundValue of the lower bound for partitioning column/Value of the upper bound for partitioning column.

Metadata

Enter the schema and select table. You can view the Metadata of the tables.

FieldDescription
SchemaSchema name for which the list of table will be viewed.
TableSelect table of which you want to view Metadata.

Once the Metadata is selected, Click Next and go to the Incremental Read tab.

Incremental Read

Enter the schema and select table. You can view the Metadata of the tables.

FieldDescription
Enable Incremental ReadCheck this checkbox to enable incremental read support.
Column to CheckSelect a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal types of values.
Start ValueMention a value of the reference column, only the records whose value of the reference column is greater than this value will be read.
Read Control Type

Provides three options to control data to be fetched -None, Limit By Count, and Maximum Value.

None: All the records with value of reference column greater than offset will be read.

Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read.

Maximum Value: All the records with value of reference column greater than offset and less than Column Value field will be read.

For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order.

Top