HTTPV2 Incremental Configuration

Enable Incremental Read #

Enable this option to read data incrementally.

Increment Using #

Choose the request component that will change for each incremental run.

URL: When the content of the URL will get updated.

Request Body: When the request body content will get updated.

Header: When the header content will get updated.

Each incremental option is further explained in detail.

Common option #

Treat as Expression #

During configuration this option can be enabled at the applicable fields to consider the entered value as an expression. Else, the value will be considered as a constant string.

Increment using URL #

Choose the relevant method to provide the variable for URL-based incremental read.

Query Parameter: When the query parameters are changing in URL.

URL Offset: When the URL will get generated from the previous response.

Custom Expression: To provide custom logic to create URL.

Query Parameter Option #

Use this option when the query parameters are changing in URL.

Parameter Name #

The query parameter name should be provided that will be used in the URL.

Example: page -> This will make the URL as http://www.<url>.com?page=<query_parameter_value>

Initial Value #

The initial value of the query parameter should be provided.

Example: 1 -> This will make the URL as http://www.<url>.com?page=1

Next Value #

The next value expression of the query parameter should be provided.

Example: $page+1 -> This will make the URL for next call as http://www.<url>.com?page=2

URL Offset Option #

Use this option when the URL will get generated from the previous response.

Initial Page URL #

Initial URL expression should be provided.

Example: “http://www.<url>.com”

Next Page URL #

The next page URL expression should be provided.

The previous response will have a URL value, that will be used as the next URL.

Example: When response is Json, and URL is present at path “data.url”, then the next page URL expression is from_json_object($last_response_last_record,’$.data.url')

Custom Expression Option #

Use this option to provide custom logic to create URL.

URI #

Custom URI expression should be provided with help of offset variables.

Example: concat($url,‘page=’,offset) where offset is the variable created in offset name section.

Offset Name #

The offset name should be provided that can be used in the Custom Expression.

Example: offset -> This can be used in the URI expression as concat($url,‘page=’,offset)

Initial Offset Value #

The initial offset value should be provided.

Example: 1 -> This will make the initial value for the URL expression as concat($url,‘page=’,<1>)

Next Offset Value #

The next offset value expression should be provided.

Example: $offset+1 -> This will make the next value for the URL expression as concat($url,‘page=’,<2>)

Increment using Request Body #

Choose the relevant method to provide the variable for body-based incremental read.

JSON Body: When the body is in JSON format.

Custom Expression: To provide custom logic to create a body for post-call requests.

JSON Body Option #

Use this option when the body is in JSON format.

Key #

The path to the JSON object that will update for each incremental run should be provided.

Example 1: $body.path_to_variable_component

Example 2: $body.a.b

Initial Value #

The initial value of the key in the JSON body should be provided.

Example: 10 -> This value 10 will be replaced for the first call at key path: $body.path_to_variable_component

Next Value #

The next value expression of the key in the JSON body should be provided.

Example: $body.path_to_variable_component+10

Custom Expression Option #

Use this option to provide custom logic to create a body for post-call requests.

Body Expression #

Provide custom expression to create body for each incremental run request.

Example: concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Offset Name #

The offset name should be provided that can be used in the Custom Expression.

Example: offset -> This can be used in the body expression as concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Initial Offset Value #

The initial offset value should be provided.

Example: 10 -> This can be used in the body expression as concat(’{“from”:’,10,’,“size”:100,“query”:{“match_all”:{}}}’)

Next Offset Value #

The next offset value expression should be provided.

Example: $offset+1 -> This can be used in the body expression as concat(’{“from”:’,<next offset value>,’,“size”:100,“query”:{“match_all”:{}}}’)

Increment using Header #

Choose t his option when the header content will get updated.

Header Expression #

All the headers with their keys and values (custom expression/constant) should be provided.

Key #

Header key should be provided.

Value #

An expression value or a constant value should be provided for each header key.

Example: $offset -> Offset value will be derived from the Offset variable fields.

Offset Name #

he offset name should be provided that can be used in the Header value expression.

Example: offset -> This can be used in the header value as $offset

Initial Offset Value #

The initial offset value should be provided.

Example: 10 -> This can be used in the header value as 10 for a key.

Next Offset Value #

The next offset value expression should be provided.

Example: $offset+10 -> This can be used in the header value as increments of 10 for each incremental read request.

Incremental Help #

The Incremental Help section offers in-app help section that can be opened by clicking INCREMANTAL HELP.