GCS Ingestion Source

Gathr provides GCS (Google Cloud Storage) channel.

Permissions Required in GCS:

Cloud storage permissionshould be there to access and read data from Buckets of GCS data source.

Data Source Configuration #

Fetch From Source/Upload Data File

For designing the application, you can either fetch the sample data from the GCS source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design.

If Upload Data File is selected to fetch sample data, provide the below details.

File Format: Select the sample file format (file type) depending on the data type.

Gathr-supported file formats for GCS data source are CSV, JSON, TEXT, Parquet and ORC.

For CSV file format, select its corresponding delimiter.

Header Included: Enable this option to read the first row as a header if your GCS data is in CSV format.

Upload: Please upload the sample file as per the file format selected above.

If Fetch From Source is selected, continue configuring the data source.

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for GCS earlier. Or create one as explained in the topic - GCS Connection →

Use the Test Connection option to ensure that the connection with the GCS channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Bucket Name Provide path of the file for Google storage bucket name.

Path Provide value for the end path with * in case of directory. Example: outdir*

File Filter Provide a file pattern. Example: *csv/*json to retrieve the available files.

Recursive File Lookup Check the option to retrieve the files from current/sub folder(s).

Add Configuration: Additional properties can be added using this option as key-value pairs.

Schema #

Check the populated schema details. For more details, see Schema Preview →

Advanced Configuration #

Optionally, you can enable incremental read. For more details, see GCS Incremental Configuration →