JDBC-Container Processor

JDBC Container allows you to perform- Read, Aggregation, Merge and Write operations, by using this processor over an oracle connection.

A JDBC Container allows the implementation of a retention policy as well. Retention policy lets you cleanse/delete unwanted data suing a retention period.

You can also apply checksum in case of aggregation to avoid duplicity of records.

Processor Configuration #

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for Oracle earlier. Or create one as explained in the topic - JDBC Connection →

Schema Name: Existing schema name of specified database.

Table Name: Existing table name of specified database.

Container Name: Existing container name of specified table.

Output Mode: Output mode is used to what data will be written to a streaming sink when there is new data available.

In case of Upsert the item with existing ids gets updated and if it does not exist, it gets created, that is called, Insert.

Enable Retention: When selected, each newly created item lives for the num­ber of seconds specified by retention policy. After the expi­ration time is reached item will be deleted by server.

Retention Policy: Number of days/month for which data is to be retained.

You can select a number and select either DAYS or MONTH as a unit.

Retention Column: Retention policy on a table will be applicable based on the field selected here.

Record Limit: Enable the limit to keep the maximum number of records in the container.

Maximum Number of records: Maximum Number of records to be retained for each group depending on grouping field criteria.

Grouping Field: Field on which ‘GroupBy’ is applied to limit the maximum number of records that is to be retained for specified group.

Write Data: Write the raw or aggregated data from the table.

Fields: Select the fields for aggregated data i.e., Function, Input Fields and Output fields.

Group By: Field of selected message on which group by is applied.

Data Duplication Handling: If Data Duplication Handling is enabled, already processed data will not be processed again on the basis of selected fields.

Fields: Data Duplication handling will be processed on the basis of selected fields.

Backup Period: Number of days for which the data should be backed-up.

Ignore Values: Enable this option to exclude specified values from the data.

Values to Ignore: Enter comma-separated values to exclude from the data. If a record contains any of these values in columns, even partially, then matching column values will be ignored and set as null.

Continue to configure the Mapping tab with help of the sub-topic below:

JDBC Container Mapping #

After the JDBC Container processor’s configuration is done, next is the Mapping page. Here, you can map the incoming schema in the fetched columns.

Schema Results

Table Column Name: Name of the column populated from the selected Table.

Mapping Value: Map a corresponding value to the column.

Database Data Type: Data type of the Mapped Value.

Ignore All: Select the Ignore All check box to ignore all the Schema Results or select a checkbox adjacent to the column to ignore that column from the Schema Results.

Use Ignore All or selected fields while pushing data to emitter.

This will add that field as the part of partition fields while creating the table.

Auto Fill: Auto Fill automatically populates and map all incoming schema fields with the fetched table columns. The left side shows the table columns and right side shows the incoming schema fields.

If same field, as of table column, not found in incoming schema then the first field will be selected by default.

Download Mapping: It downloads the mappings of schema fields and table columns in a file.

Upload Mapping: Uploading the mapping file automatically populates the table columns and schema fields.

Once the configuration is finalized, add notes and save the file.