ETL Auto Inspection

Auto Inspection lets you verify the component’s data with the schema as per the configuration of the component. This happens as soon as the component is saved.

While editing the pipeline, you can choose to disable the auto inspection option. When you edit a pipeline, auto inspection will be off by default.

While editing, you can add/remove new components in the pipeline without auto inspection.

You can auto inspect the properly configured components of the pipeline if the pipeline is not entirely configured.

In case if an error is detected during the auto inspection of a component, an alert message will be displayed along with an incident ID. The incident ID can be shared with Gathr Support to get the error resolved.

Also, the auto inspection can be stopped at any point, even while schema is being detected, and pipeline can be saved even if the inspection hangs or inspect fails.

If one of the component is not configured, the auto inspect will run for the configured components only. The auto inspect window will show a notification in Red for the component that is not configured.

A single component can be inspected by clicking on the eye icon which displays on top of the respective component.

If a pipeline is saved without inspection, the pipeline tile on the data pipeline homepage will show the warning-icon of not inspected, Once you click on the exclamation mark, it shows the Pipeline Notifications.

Once the data is loaded from the source it is automatically played through the pipeline.

During auto inspect, if the schema gets changed due to any configuration changes, it will not reset the next components however it will show an exclamation mark.

This is how the corresponding configuration, logic and results from each operator is validated while the pipeline is being designed.