Schema Preview

The schema preview section allows you to review populated schema details before proceeding to the next step.

On this page, you can do the following:

Keep/Remove Columns #

Exclude certain columns from the schema if needed.

Column Name #

See the originating column names from your data source.

Column Alias #

Change column alias names to make them more intuitive.

Next to the Column Alias section, there are two options:

Update Header #

Use this option to upload a comma-separated file for a bulk update of all headers.

Remove Unsupported Characters #

Use this option to eliminate unsupported characters from column aliases with a single click.

Data Type #

It displays the data type of each column, allowing adjustments as needed.

Description #

Add/edit descriptions against each column.

Sample Values #

See the sample values corresponding to each column.

Exclusive Ingestion/ETL Application Conditions #

Schema Page for Advanced Mongo #

For Advanced Mongo data source, on the Schema page there are three additional options:

• MARK ALL AS STRING: Option to convert the data type to string for all the columns in one click.

• DOWNLOAD SCHEMA: Option to download incoming schema as a JSON file. The same can be edited to modify the data types of columns.

• UPLOAD SCHEMA: Option to upload schema JSON file. Make sure that only the data types are updated in the JSON file for the pipeline to give expected results.

Processing JSON Strings #

How to Process JSON String Responses?

Sources with request data type selected as JSON can give you the schema result with a very long JSON string value.

You can use the expression evaluator processor with JSON function to convert such results to JSON objects, as shown in the example below:

Example:

The value from the source HTTP request is received as a JSON string:

Next, in the data transformation step, the Expression Evaluator processor is used with the appropriate JSON function to process this HTTP source response.

Configuring the from_json function in the Expression Evaluator to fetch values from the JSON string:

The JSON string used to specify schema (used as argument for the schema_of_json function) should contain all possible keys and no null values. This string will be treated as a master JSON string and will be used to infer the schema of the parsed JSON.

In this example, the column in which data should be emitted is given as result.

The JSON objects can be seen in the updated result: