MongoDB Ingestion Source

MongoDB data source allows you to read data from MongoDB.

Data Source Configuration #

Fetch From Source/Upload Data File

For designing the application, you can either fetch the sample data from the MongoDB source by providing the data source connection details or upload a sample data file in one of the supported formats to see the schema details during the application design.

If Upload Data File is selected to fetch sample data, provide the below details.

File Format: Select the sample file format (file type) depending on the data type.

Gathr-supported file formats for MongoDB data source are CSV, JSON, TEXT, Parquet and ORC.

For CSV file format, select its corresponding delimiter.

Header Included: Enable this option to read the first row as a header if your MongoDB data is in CSV format.

Upload: Please upload the sample file as per the file format selected above.

If Fetch From Source is selected, continue configuring the data source.

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for MongoDB earlier. Or create one as explained in the topic - MongoDB Connection →

Use the Test Connection option to ensure that the connection with the MongoDB channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Database Name: Name of the MongoDB database source from which data is to be fetched should be selected or provided.

Collection Name: Name of the database collection that needs to be scanned should be selected or provided. The collection name is case-sensitive and the collection name entered in this field should exactly match with the collection name in the database.

Query: Option to select the filter criteria by choosing between All Data or Match Query.

If Query option is selected as Match Query, additional field will be displayed as given below:

Filter/Query: Option to exclusively fetch schema results or read data from source as per the filter condition or query provided.

If Query option is selected as All Data, then proceed by updating the following field.

Records Per Partition: Number of records to be read per partition. Default value is 100.

Add Configuration: Additional properties can be added using this option as key-value pairs.

Schema #

Check the populated schema details. For more details, see Schema Preview →

Advanced Configuration #

Optionally, you can enable incremental read. For more details, see MongoDB Incremental Configuration →)