Schema Flattener Processor

By using Schema Flattener user can flatten the incoming JSON as per the meta data which is provided by using a S3/RDS/Upload feature.

Meta JSON Source #

The Meta JSON files will be fetched from the source. Pipeline restart will be required once the source file is updated. User can select one of the below options and provide further required configurations:

Upon selecting S3 as Meta JSON Source, provide the below details:

S3 Connection Select the connection name for creating a connection.

S3 Bucket Name Provide the S3 bucket name.

S3 Path Provide S3 bucket path.

S3 Protocol Select the S3 protocol from the drop down list. Below protocols are supported for various versions: For HDP versions, S3a protocol is supported. For CDH versions, S3a protocol is supported.

For Apache versions, S3n protocol is supported. For GCP, S3n and S3a protocol is supported.

For EMR S3, S3n, and S3a protocol is supported. For AWS Databricks, s3a protocol is supported.

Remove Name Space User has an option to remove name space by choosing Yes/No from the drop down.

Error Column User can optionally add a new column that will contain error/null values.

Note: User can add further configurations by clicking at the ADD CONFIGURATION button.

Upon selecting RDS as Meta JSON Source, provide the below details:

RDS Connection Provide the RDS connection name for creating a connection.

Schema Name Provide the schema name.

Table Name Provide the table name.

Meta Data Name Provide the name for metadata column.

Remove Name Space User has an option to remove name space by choosing Yes/No from the drop down.

Error Column User can optionally add a new column that will contain error/null values.

Upon selecting UPLOAD META FILE as Meta JSON Source, provide the below details:

Upload File Upload the meta file.

Remove Name Space User has an option to remove name space by choosing Yes/No from the drop down.

Error Column User can optionally add a new column that will contain error/null values.