Elasticsearch Ingestion Target

See the Connector Marketplace topic. Please request your administrator to start a trial or subscribe to the Premium Elasticsearch connector.

The Elasticsearch emitter allows you to store data in Elasticsearch indexes.

Target Configuration #

Connection Name: Connections are the service identifiers. A connection name can be selected from the list if you have created and saved connection details for Elasticsearch or Managed Elasticsearch. Or create one as explained in the topic - Elasticsearch Target Connection →

Use the Test Connection option to ensure that the connection with the Elasticsearch channel is established successfully.

A success message states that the connection is available. In case of any error in test connection, edit the connection to resolve the issue before proceeding further.

Output Message: Output message which is to be indexed.

Index Nested JSON: Select the checkbox if nested JSON fields are to be indexed.

If the checkbox is not selected, two additional fields are populated: Index Number of Shards and Index Replication Factor.

If selected, these three fields will be hidden and a note be displayed.

“Index will be created with Elasticsearch default 5 shards and 1 replication factor.”

Cluster Version: Please select the version of the OpenSearch/ ElasticSearch cluster. For all cluster versions less than or equal to 6.x, please select ElasticSearch 6.x.

Index Number of Shards: Number of shards to be created in Index Store.

Index Replication Factor: Number of additional copies of data.

Index Name: Index name can also be created using the document/data field like emp_{document_field}.

Here, {document_field} will be replaced during runtime by the value of document field of that particular record.

Specify the index name where data is to be indexed.

Verify dynamic index creation works only when below condition is fulfilled.

• Index Nested JSON check box should be selected on ES emitter.

• Action.auto_create_index: true should be set in Elasticsearch cluster.

• The field data should always be in lower case otherwise pipeline would fail.

Index Type: Index Type could be either Static or Dynamic.

Example of Dynamic Index Type:

Index type can also be created using the document/data field like emp_{document_field}.

Here, {document_field} will be replaced during runtime by the value of document field of that record.

Notes:

• Default Index type or blank index type will be index name +“_ type”.

• Index type should follow naming conventions of Elasticsearch.

• Make sure that in case of a static index name, index type should also be static.

ID Generator Type: Enables to generate the ID field.

Following type of ID generators are available:

UUID: Universally unique identifier.

Field Values based: In this case, ID is generated by appending the values of selected fields.

If you select this option then an additional field – “Key Fields” will be displayed, where you need to select the fields you want to combine. The fields will be appended in the same order as selected on the user interface.

Emitter Output Fields: Fields in the message that should be a part of the output data.

Connection Retries: Number of retries for component connection. Possible values are -1, 0 or positive number. -1 denotes infinite retries.

Output Mode: Output mode is used to specify what data will be written to a streaming sink when there is new data available.

Append: Output Mode in which only the new rows in the streaming data will be written to the sink.

ADD CONFIGURATION: Enables additional configuration properties of Elasticsearch.