HTTP Ingestion Target

The HTTP target can emit the extracted or transformed data to HTTP and HTTPS resources.

Target Configuration

Configure the target by providing request parameters that are explained below.

Request Method

You can choose a request method type based on the HTTP URL.

The following method types are supported:

GET: Use this method to retrieve resources from a server. The GET requests will never change the state of the resource in any way.

POST: Use this method to create a new resource in the existing resource group on a server.

PUT: Use this method to update an existing resource on the server. If the resource does not exist, PUT may create a new resource.

DELETE: Use this method to delete the resources from a server identified by the URI provided in the configuration.

PATCH: Use this method to update a portion of an existing resource.

URI

HTTP or HTTPS URI to send a request to a resource.

The URL defined can either be provided with actual values or can be parameterized. The placeholders in case of parameterized inputs will be resolved during runtime as per the value of the column name existing in the URL.

Example: http://localhost:1234/${column-name-1}/{column-name-2}

Parameters

You can specify query parameters for a request using the URI box or the Parameters tab.

Query Parameters

Query parameters are added after the ? in the HTTP URI, like this: ?id=1&type=new.

To specify a query parameter, add it directly to the URI or select the Parameters tab and enter the name and value.

In the Name field, provide a descriptive label or identifier for the parameter you need to configure. This name helps differentiate and categorize the parameters being set.

In the Value field, provide the corresponding value or content associated with the parameter specified in the Name field. This value could be a specific data value, a query string, or any other information required by the HTTP request to retrieve the desired data from the source.

The Use Input Column checkbox can be used to get a dropdown selection of available column names from the incoming data source in the value field.

Use the Add Parameters option to add multiple parameters.

Authentication

Specify the authorization type that should be used to authenticate the HTTP resource.

The supported authentication types are: None, Basic, Token, OAuth2, OAuth2 Client Credentials, and Custom Token. Each type is explained below in detail.

Auth Type - None

Choose this option to access an HTTP resource without needing any authentication.

Auth Type - Basic

If Auth Type is selected as Basic, proceed after providing the below parameters:

Username: Enter the user name for accessing the HTTP resource.

Password: Enter the password for accessing the HTTP resource.

Auth Type - Token Based

If Auth Type is selected as Token, proceed after providing the below parameters:

Token ID: The key with which the token is referred in the request.

Token: Token to access the HTTP resource.

Auth Type - Oauth2

If Auth Type is selected as OAuth2, proceed after providing the below parameters:

Auth Headers: The headers associated with Auth URL should be provided as key-value pairs, through which the authorization code is generated.

Client ID: The client identifier given during the application registration process should be provided.

Secret Key: The secret key given to the client during the application registration process should be provided.

Auth URL: The endpoint for the authorization server, which retrieves the authorization code should be provided.

Auth Type - OAuth2 (Client Credential)

If Auth Type is selected as OAuth2 (Client Credential), proceed after providing the below parameters:

Auth Headers: Header’s parameter name and value can be provided.

Auth Params: Auth parameter name and value can be provided.

ClientId: The client identifier that is given to the client during the application registration process should be provided.

Secret Key: The secret key that is given to the client during the application registration process should be provided.

Auth URL: The endpoint for the authorization server, that retrieves the authorization code.

Use Token: Use generated token in the URL parameter or header of the request.

Auth Type - Custom Token

If Auth Type is selected as Custom Token, a modal window Token Generation HTTP Configuration will appear. Proceed after providing the below parameters:

URI: HTTP or HTTPS URI to send a request to a resource.

Request Method: HTTP request method for the URI to be selected out of GET or POST.

Request Body: Request body to send a data payload to an HTTP resource in the body of the request.

Header: Header’s parameter name and value.

Example:

HTTPV2_Custom_Auth

In the below snapshot you can see that the main data API needs a token in the Header with the key as Authorization and value as BEARER ${token} and then ${token}.

HTTPV2_Custom_Auth_Header

${token} will be replaced with the custom token generated from the custom token API.

${token} can be used in the value of the header key which will be replaced by the token generated from the custom token API.

Path to Token: JSON path expression that points to tokens.

Auth Type: Used to specify the authorization type associated with the URL. The supported auth types for token generation are None, Basic, and Token.

  • None: This option specifies that the URL can be accessed without any authentication.

  • Basic: This option specifies that accessing the URL requires Basic Authorization.

    Provide a user name and password for accessing the URL.

  • Token: Token-based authentication is a security technique that authenticates the users who attempt to log in to a server, a network, or other secure systems, using a security token provided by the server.

Headers

Header’s parameter name and value.

Expressions are supported in Header Values.

Example:

Header Key: Authorization Header Value: Bearer ${Column}

Header values provided with expressions will be dynamically updated during application runtime. In the above example, Column is a column from the incoming dataset.

Accept Type

The content type that the client can understand should be selected out of JSON, CSV, or Text.

Content Type

The content-type header is used to indicate the request type of the resource. The available options are: CSV, JSON, Text, and Form-Data.

If the request type is not available in the content type options, it can be specified in the Header section as shown in the example below:

HTTP_Emitter_Content_Type

Message Name

The name of the message configuration will act as metadata for the actual data.

Body

Request body to be provided for the Request Method selected above.

Output Fields

Fields that need to be a part of the output data should be selected.

Custom Payload

Option to provide the customized payload.

Payload

If Custom Payload is enabled, provide the payload in the format that supports the Content Type field.

Example:

If the Content Type field is selected as JSON, then provide the payload as per the sample given below:

{“emp\_id” : ${id}, “emp\_name” : $name}

where id and name are output fields, the evaluated value will be as follows:

{“emp\_id” : 1, “emp\_name” : John}

Handle input as file

When enabled, the HTTP emitter considers the input dataset as a binary file, including it as a part of the HTTP request body.

Binary File

Specify key, format, and size constraints for including binary file in HTTP requests.

Key: Specify the key to identify the binary file in the HTTP request. The default value is ‘file’.

Type: Select the format of the binary file. The default is CSV.

Max File Size in MB: Limits the maximum size of the binary file to be created.

The default value is 10 MB.

Example: For an input dataset of 500 MB, if the maximum size of the file to be created is limited to 200 MB, then three HTTP requests will be placed.

Form Data

Key-value pairs to include as form data in the HTTP request.

Select the Content Type as ‘Form-Data’ when Form-Data parameters are provided.

Key: Key(s) to include as form data in the HTTP request.

Value: Corresponding value(s) for the provided key(s).

Settings

Advanced settings can be configured using this tab.

Fail on Data Loss

This option is enabled by default. The application will terminate even if a single HTTP call fails.

Disable it to allow writing successful records to the HTTP emitter, leaving out the failed HTTP calls.

Enable SSL

It is set to False by default.

Set this option to True, if the resource that is to be requested using the HTTP data source is SSL-enabled.

If set to True, choose how the SSL-enabled HTTP resource should be verified.

Either a keystore file or a certificate file needs to be uploaded based on the chosen verification method.

The Keystore Password or Certificate Alias should then be provided as per the type of file uploaded for verification.

Retry Count

Runs the URL as many number of times as mentioned, in case of failure to run the URL.

Retry Delay

A retry delay interval (in seconds) should be provided.

Read Timeout (sec)

While a URL is accessing a server and the output generation is taking time, you can provide read timeout (in seconds).

A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as undefined (system default). Default value is 5.

Connection Timeout (sec)

In certain scenarios, the connection is not established, you can specify the connection time out (in seconds).

A timeout value of zero is interpreted as an infinite timeout.

A negative value is interpreted as undefined (system default).

Default value is 5.

Handle Success Response as Error

Enable this option to customize the system’s response handling for errors, even when the HTTP status code indicates success.

Error Indicator Keywords

Enter comma-separated keywords. If any of these keywords are found in the response, it will be considered as error.

Error Definition for JSON

Specify conditions for considering the response as error.

  • Key: Enter the JSON path to identify the specific element in the response. The system will check this element to consider the response as error.

    Example: $.data[0].error

  • Value: Enter the expected value for the specified JSON path. If the specified value matches, it will consider the response as error.

  • Ignore Value: Enable this option to ignore value of the specified JSON path while deciding if the response should be treated as error. Only the JSON path will be considered as an error indicator.

Rate Limit

Enable Request Rate

Control the rate at which HTTP calls are made on each partition.

This field is disabled by default.

If enabled, as per the pre set values, the HTTP emitter is configured to allow a maximum of 50 requests to be made within a 10-second window. If the emitter exceeds this limit, it will encounter a throttle exception. Upon encountering such an exception, the emitter will wait for 60 seconds before it can resume making requests.

Example: Make 50 requests in 10 seconds, on throttle exception wait for 60 seconds.

Add Configuration: Additional properties can be added using this option as key-value pairs.

If you have any feedback on Gathr documentation, please email us!

← HTTP ETL Source
HTTP ETL Target →