HTTP Processor
HTTP Processor is used to enrich the incoming data with the response of external rest calls. The user can consume different URLs in parallel and use their response to enrich the resulting data either in a new column or existing column.
| Field | Description |
|---|---|
| HTTP URL | The user specifies a URL that will further be reflected in the output field. |
| Request Method | Based on the type of HTTP URL, the user chooses one of the below mentioned request method types: 1. GET User is required to fill the Request Param. 2. POST In certain posts the user will be required to mention the request body value and request param. 3. PUT Request body value needs to be specified along with request param. 4. DELETE The user is required to fill the request body value and request param fields. In POST, PUT and DELETE, when the user provides the request body value in the below format: {‘value’.’key’} |
| Request Param | The user provides parameter in two ways: e.g: param1=value1 or param2=${column} |
| Enable SSL | The user can enable SSL in the HTTPS URL. |
| Output Field | Here, the output field is mentioned based on the HTTP URL. |
| Header | In certain URL the user requires to provide a Header. The user can mention the header details in key value pair. Likewise, the user can add header too. |
| Auth Type | For different URLs there are different authorization types that must be provided by the user. Types of authorization: 1. Basic In case of basic authorization type the user will be required to fill - user name and password. 2. Token In case of token authorization type, the user will be required to specify the tokenID and the token value. 3. Auth In case of Auth type of authorization, the user will be required to specify the header(s) and also specify the clientID, secret key and end-point Auth URL. 4. None When no authorization is required, None is set as default. |
| Drop Record | By selecting drop record, the record will be dropped and not shown in the output in case of an exception/error being generated in a URL. |
| Retry Count | By specifying the number of retry count, the user can run the URL as many number of times as mentioned in this field in case of failure to run the URL. |
| Read Timeout (sec) | While a URL is accessing a server and the output generation is taking time, the user can choose read timeout (in seconds). |
| Connection Timeout (sec) | In certain scenarios, the connection is not established, the user can specify the connection time out (in seconds). |
| Enable Request Rate | Control the rate at which HTTP calls are made on each partition. The HTTP calls are made in parallel on multiple partitions. Therefore, set the number of partitions using the repartition processor before using the HTTP emitter. Example: For a client, the API hit rate limit is set to 40 calls/sec. Add a repartition processor before HTTP emitter and provide the partitioning value as 10. Now, set the rate limit to 4 requests per second, which will apply to each partition. This means 10 parallel partitions will be working with 4 requests per second on each partition allowing 40 requests per second on the HTTP emitter. |
If you have any feedback on Gathr documentation, please email us!