HTTPV2 Pagination

Data is collected on a driver and sent to executors in frames of 20 pages. The driver needs to store the full dataset and data type info to make the pagination more efficient.

Enable Pagination #

Enable this option to make paginated calls to fetch the desired data from the endpoint URI.

Paginate Using: Choose the request component that will change for each pagination Get/Post call.

URL: When the content of the URL will get updated.

Request Body: When the request body content will get updated. This option is only available for Post request methods.

Header: When the header content will get updated.

Each pagination option is further explained in detail.

Common options in pagination configuration #

Persist across run #

Enable the checkbox to retain the last offset value as the initial value for the subsequent run.

Treat as Expression #

During configuration, this option can be enabled in the applicable fields to consider the entered value as an expression. Else, the value will be considered a constant string.

Paginate using URL #

Choose the relevant method to provide the variable for URL-based pagination.

Query Parameter: When the query parameters are changing in URL.

URL Offset: When the URL will get generated from the previous response.

Custom Expression: To provide custom logic to create URL.

Query Parameter Option #

Use this option when the query parameters are changing in the URL.

Parameter Name #

The query parameter name should be provided that will be used in the URL.

Example: page -> This will make the URL as http://www.<url>.com?page=<query_parameter_value>

Initial Value #

The initial value of the query parameter should be provided.

Example: 1 -> This will make the URL as http://www.<url>.com?page=1

Next Value #

The next value expression of the query parameter should be provided.

Example: $page+1 -> This will make the URL for next call as http://www.<url>.com?page=2

URL Offset Option #

Use this option when the URL will get generated from the previous response.

Initial Page URL #

Initial URL expression should be provided.

Example: “http://www.<url>.com”

Next Page URL #

The next page URL expression should be provided.

The previous response will have a URL value, that will be used as the next URL.

Example: When response is Json, and URL is present at path “data.url”, then the next page URL expression is from_json_object($last_response_last_record,’$.data.url')

Custom Expression Option #

Use this option to provide custom logic to create URL.

URI #

Custom URI expression should be provided with help of offset variables.

Example: concat($url,‘page=’,offset) where offset is the variable created in offset name section.

Offset Name #

The offset name should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: offset -> This can be used in the URI expression as concat($url,‘page=’,offset)

Initial Offset Value #

The initial offset value should be provided.

Example: 1 -> This will make the initial value for the URL expression as concat($url,‘page=’,<1>)

Next Offset Value #

The next offset value expression should be provided.

Example: $offset+1 -> This will make the next value for the URL expression as concat($url,‘page=’,<2>)

Paginate using Request Body #

Choose the relevant method to provide the variable for body-based pagination.

JSON Body: When the body is in JSON format.

Custom Expression: To provide custom logic to create a body for post-call requests.

JSON Body Option #

Use this option when the body is in JSON format.

Key #

The path to the JSON object that will update for each pagination request should be provided.

Example 1: $body.path_to_variable_component

Example 2: $body.a.b

Initial Value #

The initial value of the key in the JSON body should be provided.

Example: 10 -> This value 10 will be replaced for the first call at key path: $body.path_to_variable_component

Next Value #

The next value expression of the key in the JSON body should be provided.

Example: $body.path_to_variable_component+10

Custom Expression Option #

Use this option to provide custom logic to create a body for post-call requests.

Body Expression #

Provide custom expression to create body for each pagination request.

Example: concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Offset Name #

The offset name should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: offset -> This can be used in the body expression as concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Initial Offset Value #

The initial offset value should be provided.

Example: 10 -> This can be used in the body expression as concat(’{“from”:’,10,’,“size”:100,“query”:{“match_all”:{}}}’)

Next Offset Value #

The next offset value expression should be provided.

Example: $offset+1 -> This can be used in the body expression as concat(’{“from”:’,<next offset value>,’,“size”:100,“query”:{“match_all”:{}}}’)

Paginate using Header #

Choose this option when the header content will get updated.

Header Expression #

All the headers with their keys and values (custom expression/constant) should be provided.

Key #

Header key should be provided.

Value #

An expression value or a constant value should be provided for each header key.

Example: $offset -> Offset value will be derived from the Offset variable fields.

Offset Name #

The offset name should be provided that can be used in the Header value expression and Pagination End Condition expression.

Example: offset -> This can be used in the header value as $offset

Initial Offset Value #

The initial offset value should be provided.

Example: 10 -> This can be used in the header value as 10 for a key.

Next Offset Value #

The next offset value expression should be provided.

Example: $offset+10 -> This can be used in the header value as increments of 10 for each pagination request.

Pagination End Condition #

An expression to stop the pagination should be provided.

Example 1: $offset>12

Example 2: $last_response_record_count=0

Example 3: get_json_object($last_response_last_record,’$.data.id’)>3

Example 4: get_json_object($last_response,’$.metadata.url’) is null

Timeout #

Option to set a timeout and stop fetching data from paginated calls by terminating the application after the specified time interval (in seconds).

Enable Request Rate #

Control the rate at which pagination requests are made.

Example: Make 50 requests in 10 seconds, on throttle exception wait for 60 seconds.

Pagination Help #

The Pagination section offers in-app help section that can be opened by clicking PAGINATION HELP.

The help section describes several scenarios for using pagination requests and appears as shown below:

• The code snippets will load based on the Usecase that is selected.

• You can read through the code snippets and click on USE CODE option to dynamically update the sample values in the pagination configuration.

• The Functions tab contains the list of functions with their explanation.

• The Expressions tab contains the description and sample values for several useful expressions.

Click on BACK TO CONFIGURATION option to return to the pagination configuration section.