BigQuery Ingestion Target

The BigQuery emitter allows you to push data in BigQuery data warehouse.

Target Configuration #

Save as Data Asset: Select checkbox to save the schema as a data asset in Gathr.

Data Asset Name: Provide a name for the data asset to be saved.

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier. Or create one as explained in the topic Component Connections - BigQuery Connection →

Use the Test Connection option to ensure that the connection with the BigQuery channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Message Name: Select the name for the message configuration which will act as metadata for actual data.

BigQuery Dataset Name: Select the existing dataset name in BigQuery.

Big Query Table Name: Select the existing table name of specified BigQuery dataset.

Save Mode

Select save mode: Append, Overwrite or Ignore. It specifies how to handle existing data, if present.

GCS Bucket: Intermediate GCS location for BigQuery table should be provided.

Please make sure that the permissions/policy actions required to emit data in the target GCS bucket are assigned to your BigQuery service account key. See the list of required permissions in the Prerequisites section of the topic BigQuery Connection →

Add configuration: Additional properties can be added using Add Configuration link.

More Configurations

Timestamp Column: Enter value for timestamp column that is used for tracking load time in BigQuery streaming pipeline.