RDS Data Source
RDS allows you to write to RDS DB Engine, which could be through SSl or without SSL. RDS is Relational Database service on Cloud.
RDS Channel can read in Batch from the RDS Databases (Postgresql, MySql, Oracle, Mssql). RDS is Relational Database service on Cloud. The properties of RDS are similar to those of a JDBC Connector with one addition of SSL Security.
SSL Security can be enabled on RDS Databases.
System should be able to connect, read and write from SSL Secured RDS.
If security is enabled, it will be configured in Connection and automatically propagated to channel.
Configuring RDS Data Source
To add an RDS Data Source to your pipeline, drag the Data Source to the canvas and right-click on it to configure.
Under the Schema Type tab, select Fetch From Source or Upload Data File:
| Field | Description |
|---|---|
| Connection Name | Connections are the Service identifiers. Select the connection name from the available list of connections for data ingestion. |
| Query | Hive compatible SQL query to be executed in the component. |
| Enable Query Partitioning | Tables will be partitioned and loaded RDDs if this check-box is enabled. This enables parallel reading of data from the table. |
| Number of Partitions | Specifies no of parallel threads to be invoked to read from JDBC in spark. |
| Partition on Column | Partitioning column can be any column of type Integer, on which spark will perform partitioning to read data in parallel. |
| Lower Bound/Upper Bound | Value of the lower bound for partitioning column/Value of the upper bound for partitioning column. |
Metadata
Enter the schema and select table. You can view the Metadata of the tables.
| Field | Description |
|---|---|
| Schema | Schema name for which the list of table will be viewed. |
| Table | Select table of which you want to view Metadata. |
Once the Metadata is selected, Click Next and go to the Incremental Read tab.
Incremental Read
Enter the schema and select table. You can view the Metadata of the tables.
| Field | Description |
|---|---|
| Enable Incremental Read | Check this checkbox to enable incremental read support. |
| Column to Check | Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp, decimal types of values. |
| Start Value | Mention a value of the reference column, only the records whose value of the reference column is greater than this value will be read. |
| Read Control Type | Provides three options to control data to be fetched -None, Limit By Count, and Maximum Value. None: All the records with value of reference column greater than offset will be read. Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read. Maximum Value: All the records with value of reference column greater than offset and less than Column Value field will be read. For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order. |
If you have any feedback on Gathr documentation, please email us!