Milvus Vector ETL Target

Milvus ETL Target allows you to emit and manage data from your Gathr application to Milvus.

Target Configuration #

Configure the data emitter parameters as explained below.

Connection Name #

A connection name can be selected from the list if you have created and saved connection details for OpenAI earlier. Or create one as explained in the topic - Milvus Connection →

Username #

Provide the username with access to Milvus.

Password #

Provide the password to authenticate the Milvus user.

Database Name #

The name of the database to be connected. If left blank, the ‘Default’ database will be used for connection.

Collection #

Select the collection of Milvus to write the vector data into it.

Collection Info #

Additional information about the selected collection will be provided in the collection info section.

Create Collection #

Select the checkbox to create a new collection. A new Index should also be created with a new Collection.

Collection Name #

Provide collection name to be created.

Number of Shards #

Number of shards for the collection to create.

Description #

Provide a brief description about the collection to be created.

Index #

Select the index of Milvus to write the vector data into it.

Index Info #

Additional information about the selected index will be provided in the index info section.

Create Index #

Select the checkbox to create a new index of the Milvus Vector.

Index Name #

Provide index name to be created.

Index Type #

Type of Milvus Index. Currently supported index types are: FLAT and HNSW.

Metric #

Select the distance metric that will be used to measure the degree of similarity between two vectors. Supported distance metrics are L2, IP, COSINE.

Extra Param #

Provide any extra parameters for creating Milvus Vector Index.

Vector Column Name #

Provide the vector column name of Milvus Vector Index. This should be same as the vector Attribute specified in the Index Metadata section.

INDEX METADATA

Auto Generate ID #

This option is available while creating collection. When enabled, a unique identifier is assigned to each vector automatically, ensuring that every record in the database has a distinct ID when a primary key is not explicitly defined or needed.

Output Column #

Select a column name where the parsed metadata attribute should be captured.

Attribute #

Select the name of the metadata attribute to be parsed.

Data Type #

Select the data type of the metadata attribute being stored.

Additional Params #

Provide additional parameters based on the selected data type.

Is Primary Key #

This option is available while creating collection. Select the column that serves as the primary key for each record or vector in Milvus.

Auto Fill #

Automatically populate fields with relevant information, streamlining the setup of index metadata in Milvus based on existing dataset characteristics.

WRITE CONFIGURATION

Save Mode #

Save mode specifies how to handle any existing data in the target.

• Append: Input data will be appended to the existing index.

• Upsert: Contents will be inserted or updated depending on primary key values.

• Truncate and Overwrite: This option will be enabled when user selects Create Index. This will first truncate the given collection (if collection is present), recreate the collection and then insert the data in the newly created collection.

Batch Size #

Provide the batch size between 1 - 100 to determine the number of rows to insert per request.

Add Configuration: Additional properties can be added using this option as key-value pairs.

Notes #

Optionally, enter notes and save the configuration.