HTTP Processor

HTTP Processor is used to enrich the incoming data with the response of external rest calls. The user can consume different URLs in parallel and use their response to enrich the resulting data either in a new column or existing column.

FieldDescription
HTTP URLThe user specifies a URL that will further be reflected in the output field.
Method Type

Based on the type of HTTP URL, the user chooses one of the below mentioned request method types:

1. GET

2. POST

3. PUT

4. DELETE

{‘value’.’key’}

Request BodyOption available when Method Type is selected as POST. Request Body should be provided for the selected Request Method.Request Body Format: {‘key’:‘value’}
Request Param

The user provides parameter in two ways:

e.g: param1=value1

or

param2=${column}

Enable SSLThe user can enable SSL in the HTTPS URL.
Output FieldsHere, the output field is mentioned based on the HTTP URL.
HeaderIn certain URL the user requires to provide a Header. The user can mention the header details in key value pair. Likewise, the user can add header too.

To dynamically change headers, such as Authorization headers for each new request, follow these options:

Parameterize Header ValuesIf header values are derived from a column in a previous processor, you can parameterize the header value. This way, the headers will adapt to each request based on the dynamic data from the previous connector.
Choose Auth TypeSelect the appropriate authentication type, which can be set to Custom Token. If a custom token is generated before each application run and changes for each call, ensure that its value is available in a column from the previous connector.

Pass this dynamic token as a parameter in the HTTP processor header. This approach allows you to send requests with new and updated headers for each application run, ensuring that the requests are properly authenticated without encountering errors.

FieldDescription
Auth Type

For different URLs there are different authorization types that must be provided by the user.

Types of authorization:

1. Basic

2. Token

3. Auth

4. None

Drop RecordBy selecting drop record, the record will be dropped and not shown in the output in case of an exception/error being generated in a URL.
Response Status ColumnOption available when Mehod Type is selected a GET/DELETE. Provide a column name to capture the response status codes for the HTTP request.
Retry TimeA retry delay interval (in seconds) should be provided.
Retry CountBy specifying the number of retry count, the user can run the URL as many number of times as mentioned in this field in case of failure to run the URL.
Read Timeout (sec)While a URL is accessing a server and the output generation is taking time, the user can choose read timeout (in seconds).
Connection Timeout (sec)In certain scenarios, the connection is not established, the user can specify the connection time out (in seconds).
Enable Request Rate

Control the rate at which HTTP calls are made on each partition.

Example: For a client, the API hit rate limit is set to 40 calls/sec.

Add a repartition processor before HTTP emitter and provide the partitioning value as 10.

Now, set the rate limit to 4 requests per second, which will apply to each partition.

This means 10 parallel partitions will be working with 4 requests per second on each partition allowing 40 requests per second on the HTTP emitter.

If you have any feedback on Gathr documentation, please email us!

← Hashing Processor
JDBC Container Processor →