BigQuery ETL Target

The BigQuery emitter allows you to push data in BigQuery data warehouse.

Target Configuration #

Save as Data Asset: Select checkbox to save the schema as a data asset in Gathr.

Data Asset Name: Provide a name for the data asset to be saved.

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier. Or create one as explained in the topic Component Connections - BigQuery Connection →

Message Name: Select the name for the message configuration which will act as metadata for actual data.

BigQuery Dataset Name: Select the existing dataset name in BigQuery.

Big Query Table Name: Select the existing table name of specified BigQuery dataset.

Timestamp Column: Enter value for timestamp column that is used for tracking load time in BigQuery streaming pipeline.

Save Mode: Select save mode:

Append, Overwrite or Ignore. It specifies how to handle existing data, if present.

GCS Bucket: The GCS bucket that temporarily holds the data before it is loaded to BigQuery.

Please make sure that the permissions/policy actions required to emit data in the target GCS bucket are assigned to your BigQuery service account key. See the list of required permissions in the Prerequisites section of the topic BigQuery Connection →

Add Configuration: You can add further configurations by using this option.

Schema Results: #

• Under this section, you can view the data that is fetched and in which column does it needs to be saved. You can view: Table Column Name, Mapping Value, Database Datatype, Mode and Ignore All checkbox.

• You can also download or upload Mapping.

Post Action #

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Post-Actions →

Notes #

Optionally, enter notes in the Notes → tab and save the configuration.