HTTPV2 Pagination

Data is collected on a driver and sent to executors in frames of 20 pages. The driver needs to store the full dataset and data type info to make the pagination more efficient.

Enable Pagination

Enable this option to make paginated calls to fetch the desired data from the endpoint URI.

Paginate Using: Choose the request component that will change for each pagination Get/Post call.

URL: When the content of the URL will get updated.

Request Body: When the request body content will get updated. This option is only available for Post request methods.

Header: When the header content will get updated.

Each pagination option is further explained in detail.


Common options in pagination configuration

HTTPV2_Persist_Offset_Value

Persist across run

Enable the checkbox to retain the last offset value as the initial value for the subsequent run.

Treat as Expression

During configuration, this option can be enabled in the applicable fields to consider the entered value as an expression. Else, the value will be considered a constant string.


Paginate using URL

Choose the relevant method to provide the variable for URL-based pagination.

Query Parameter: When the query parameters are changing in URL.

URL Offset: When the URL will get generated from the previous response.

Custom Expression: To provide custom logic to create URL.


Query Parameter Option

Use this option when the query parameters are changing in the URL.

Parameter Name

The query parameter name should be provided that will be used in the URL.

Example: page -> This will make the URL as http://www.<url>.com?page=<query_parameter_value>

Initial Value

The initial value of the query parameter should be provided.

Example: 1 -> This will make the URL as http://www.<url>.com?page=1

Next Value

The next value expression of the query parameter should be provided.

Example: $page+1 -> This will make the URL for next call as http://www.<url>.com?page=2


URL Offset Option

Use this option when the URL will get generated from the previous response.

Initial Page URL

Initial URL expression should be provided.

Example: “http://www.<url>.com”

Next Page URL

The next page URL expression should be provided.

The previous response will have a URL value, that will be used as the next URL.

Example: When response is Json, and URL is present at path “data.url”, then the next page URL expression is from_json_object($last_response_last_record,’$.data.url')


Custom Expression Option

Use this option to provide custom logic to create URL.

URI

Custom URI expression should be provided with help of offset variables.

Example: concat($url,‘page=’,offset) where offset is the variable created in offset name section.

Offset Name

The offset name should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: offset -> This can be used in the URI expression as concat($url,‘page=’,offset)

Initial Offset Value

The initial offset value should be provided.

Example: 1 -> This will make the initial value for the URL expression as concat($url,‘page=’,<1>)

Next Offset Value

The next offset value expression should be provided.

Example: $offset+1 -> This will make the next value for the URL expression as concat($url,‘page=’,<2>)


Paginate using Request Body

Choose the relevant method to provide the variable for body-based pagination.

JSON Body: When the body is in JSON format.

Custom Expression: To provide custom logic to create a body for post-call requests.


JSON Body Option

Use this option when the body is in JSON format.

Key

The path to the JSON object that will update for each pagination request should be provided.

Example 1: $body.path_to_variable_component

Example 2: $body.a.b

Initial Value

The initial value of the key in the JSON body should be provided.

Example: 10 -> This value 10 will be replaced for the first call at key path: $body.path_to_variable_component

Next Value

The next value expression of the key in the JSON body should be provided.

Example: $body.path_to_variable_component+10


Custom Expression Option

Use this option to provide custom logic to create a body for post-call requests.

Body Expression

Provide custom expression to create body for each pagination request.

Example: concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Offset Name

The offset name should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: offset -> This can be used in the body expression as concat(’{“from”:’,$offset,’,“size”:100,“query”:{“match_all”:{}}}’)

Initial Offset Value

The initial offset value should be provided.

Example: 10 -> This can be used in the body expression as concat(’{“from”:’,10,’,“size”:100,“query”:{“match_all”:{}}}’)

Next Offset Value

The next offset value expression should be provided.

Example: $offset+1 -> This can be used in the body expression as concat(’{“from”:’,<next offset value>,’,“size”:100,“query”:{“match_all”:{}}}’)


Paginate using Header

Choose this option when the header content will get updated.

Header Expression

All the headers with their keys and values (custom expression/constant) should be provided.

Key

Header key should be provided.

Value

An expression value or a constant value should be provided for each header key.

Example: $offset -> Offset value will be derived from the Offset variable fields.

Offset Name

The offset name should be provided that can be used in the Header value expression and Pagination End Condition expression.

Example: offset -> This can be used in the header value as $offset

Initial Offset Value

The initial offset value should be provided.

Example: 10 -> This can be used in the header value as 10 for a key.

Next Offset Value

The next offset value expression should be provided.

Example: $offset+10 -> This can be used in the header value as increments of 10 for each pagination request.


Pagination End Condition

An expression to stop the pagination should be provided.

Example 1: $offset>12

Example 2: $last_response_record_count=0

Example 3: get_json_object($last_response_last_record,’$.data.id’)>3

Example 4: get_json_object($last_response,’$.metadata.url’) is null


Timeout

Option to set a timeout and stop fetching data from paginated calls by terminating the application after the specified time interval (in seconds).


Enable Request Rate

Control the rate at which pagination requests are made.

Example: Make 50 requests in 10 seconds, on throttle exception wait for 60 seconds.


Pagination Help

The Pagination section offers in-app help section that can be opened by clicking PAGINATION HELP.

Pagination_Help-1

The help section describes several scenarios for using pagination requests and appears as shown below:

Pagination_Help-2

  • The code snippets will load based on the Usecase that is selected.

  • You can read through the code snippets and click on USE CODE option to dynamically update the sample values in the pagination configuration.

  • The Functions tab contains the list of functions with their explanation.

  • The Expressions tab contains the description and sample values for several useful expressions.

Click on BACK TO CONFIGURATION option to return to the pagination configuration section.

Top