BigQuery Data Source
The configuration for BIGQUERY is mentioned below:
| Field | Description |
|---|---|
| Connection Name | Connections are the service identifiers. Mention the connection name for creating connection. |
| Override Credentials | Unchecked by default, check the checkbox to override credentials for user specific actions. Upload the GCS service account key. |
| Load From Big Query Table/ Load From Query Results | Choose one of the options. |
Upon selecting Load From Big Query option, update the below fields:
| Dataset Name | Upon choosing, ‘Load From Big Query Table’, the user is required to provide the dataset name. |
| Table Name | Upon choosing, ‘Load From Big Query Table’, provide the table name. |
| Project ID of Dataset | Upon choosing, ‘Load From Big Query Table’, provide the Google Cloud project ID. If not specified, the project from the service account key of connection will be used. |
| Columns to Fetch | Upon choosing, ‘Load From Big Query Table’, enter vale for comma separated list of columns to select. |
| Where Condition | Upon choosing, ‘Load From Big Query Table’, enter the where condition. |
| Partition Filter Condition | Upon choosing, ‘Load From Big Query Table’, enter the partition filter condition. |
Load From Query Results
Upon selecting this option, the user will be required to mention, ‘Location of Datasets used in the query’.
| Query | Query to be executed in BigQuery. |
| Location of Dataset | Provide location of the dataset used in query. Eg: US or US-WEST1. |
| Maximum Parallelism | Mention the maximum number of partitions to split the data into. |
| Add Configuration | The user can add further configurations. |
| Schema Results | Under schema results, select the Big Query Dataset name and Big Query Table Name. |
| Details | Under details the user will be able to view the: Table of Expiration Number of Rows Last Modified Data Location Table ID Table Size Created |
| Table Schema | Table schema details can be viewed here. |
Next, in the Detect Schema window, the user can set the schema as dataset by clicking on the Save As Dataset checkbox. Click Next. The user can set the Incremental Read option.
| Field | Description |
|---|---|
| Enable Incremental Read | Check this checkbox to enable the incremental read support. |
| Column to Check | Select a column on which incremental read will work. Displays the list of columns that has integer, long, date, timestamp and decimal type of values. |
| Start Value | Mention the value of reference column. Only the records whose value of the reference column is greater than this value will be read. |
| Read Control Type | Provides 3 options to control how data will be fetched: None, Limit by Count and Limit by Value. None: All the records with value of reference column greater than offset will be read. Limit By Count: Mentioned no. of records will be read with the value of reference column greater than offset will be read. Limit By Value: All the records with value of reference column greater than offset and less than Column Value field will be read. For None and Limit by count it is recommended that table should have data in sequential and sorted (increasing) order. |
If you have any feedback on Gathr documentation, please email us!