Pinecone Lookup Processor

The Pinecone Lookup Processor in Gathr is your tool for extracting valuable insights from your data.

It allows you to explore and analyze information by performing lookups based on vectors or IDs.

This topic helps you configure the processor effortlessly.

Processor Configuration #

Configure the processor parameters as explained below.

Connection Name #

A connection name can be selected from the list if you have created and saved connection details for OpenAI earlier. Or create one as explained in the topic - Pinecone Connection →

Index #

Select an index to read the data from.

Index Info #

Additional information about the selected Index is provided here.

For example: Name, Metric, Dimension, Pod Type, Status, Shards, Replicas, Pods, Metadata Config, and Source Collection.

QUERY CONFIGURATION #

Configure the below fields by providing query details.

Query By #

Select whether to perform the lookup by Vector or ID.

Input Column #

Choose the column based on the above selection (Vector or ID). Example: If Vector is selected, choose the column containing vectors.

Top Matches #

Specify the number of top matches to retrieve. Example: Enter ‘5’ to get the top 5 matches.

Include Metadata #

Enable this option to include the metadata in the lookup results.

Include Values #

Enable this option to include the vectors in the lookup results.

PROCESS LOOKUP RESULTS (MATCHES) #

Specify how to process the results from lookup.

Organize Matches #

Choose whether to consolidate all matches obtained from lookup results in one row or create a new row for each match.

Process JSON #

Specify how to handle JSON data retrieved from the lookup matches. You may parse and assign the JSON attributes, such as ID, Score, Values, and Metadata, to separate output columns or preserve the original JSON structure by assigning it to a designated column.

Update below fields for Parse and assign attributes to columns:

ID #

Enter or select a column name where the ID attribute obtained from the matched results should be captured.

Score #

Enter or select a column name where the Score attribute obtained from the matched results should be captured.

Values #

Enter or select a column name where the Values attribute (i.e., Vectors) obtained from the matched results should be captured.

Metadata #

Specify how to handle JSON data retrieved from the lookup matches. You may parse and assign the JSON attributes to separate output columns or preserve the original JSON structure by assigning it to a designated column.

Update below field for Parse and assign attributes to column:

Attribute #

Enter the name of the metadata attribute to be parsed.

Column #

Enter or select a column name where the parsed metadata attribute should be captured.

Data Type #

Select the data type of the metadata attribute being stored.

Update below field for Assign to a column:

Metadata Column #

Enter or select a column name where the entire metadata from the matched results should be captured as it is.

Update below field for Assign to a column:

Matches Column #

Enter or select a column name where the entire matched results should be captured as it is.

Add Configuration: Additional properties can be added using this option as key-value pairs.