HTTP Processor Pagination

Enable Pagination

Enable this option to make paginated calls to fetch the desired data from the endpoint URI for each record from the previous component.

Response Format

The format of the response data should be selected. Currently, JSON format is supported.

Path to Data

The JSON path expression for the path to data should be provided.

Limit no. of Pages at Design Time

Please provide the maximum number of pages per row that should be used to design the pipeline. Reading all the pages during application design will add to the processing time.

Use Offset in

Please choose the best method for pagination.

Parameter: When the query parameters are changing in the URL.

URL Cursor: When the URL will get generated from the previous response.

JSON Body: When the body is in JSON format and the request body content will get updated for each paginated call.

Paginate Using Parameter

Type

Please choose the relevant offset type to provide logic for changing the parameter.

Add Constant

When the change in the parameter is constant.

Example: Page=1 and Page=2. Here, the Page parameter is changing by 1 for each paginated call.

When the parameter is a date value, then provide the frequency using which the parameter would change along with the format of date.

Example: Provide 1 as increment by value, and select day to provide frequency in days and provide the format as dd-MM-yyyy.

Fetch from last response

When the parameter’s next value is updated from the previous response.

Example: cursor=12345 can be a key in response that is available at a path in response that can be used to make another page call.

Fetch from last response last record

When the parameter’s next value is updated from the previous response’s last record.

Example: id_greater_than=2399 can be a key in response that is a column in the response’s last record that can be used to make another page call."

Increment By value fetched from last response

When the parameter’s next value is sum of previous value and value fetched the previous response. 

Example: limit=10 can be value in response which is in getting added to previous value 0 for first page call.

Increment By value fetched from last response last record

When the parameter’s next value is sum of previous value and value fetched the previous response’s last record. 

Example: limit=10 can be value in response’s last record which is in getting added to previous value 0 for first page call.

Parameter Name

The query parameter name that will be used in the URL should be provided.

Example: page

This will make the URL as http://www..com?page=<query_parameter_value>

Initial Value

The initial value of the query parameter should be selected. The parameter’s initial value must be provided in the previous component’s column. The expression evaluator component can be used in the application to add columns with a constant value.

Example: 1 -> This will make the URL as http://www..com?page=1"

Next Value

The Logic to evaluate the next value of the query parameter should be provided.

Example for Add Constant: If the parameter’s next value is incremented by some constant, then provide the constant like some numeric value by which the parameter will get an increment for each paginated call.

Example for Last Response: If the parameter’s next value is evaluated from the last response, then JSON path expression should be provided that points to the next value for the parameter from JSON response.

Example for Last Response Last Record: If the parameter’s next value is evaluated from the last response’s last record, then JSON path expression should be provided that points to the next value for the parameter from JSON response’s last record.

Example for increment by Last Response: if the parameter’s next value is sum of previous value and evaluated from the last response, then JSON path expression should be provided that points to the next value for the parameter from JSON response.

Example for increment by Last Response last record: if the parameter’s next value  is sum of previous value and evaluated from the last response’s last record, then JSON path expression should be provided that points to the next value for the parameter from JSON response’s last record.

Paginate Using URL cursor

Fetch from last response

When the next URL is fetched from the previous response. 

Example: When the response has a JSON key that provides the link for the next paginated call.

{
  "code": 0,
  "data": [
    {
      "name": "John",
      "age": "32",
      "department": "Sales"
    },
    {
      "name": "Martin",
      "age": "32",
      "department": "Marketing"
    }
  ],
  "page_info": {
    "has_more": true,
    "next_page_link": "https://<Base_URL>?cursor=1678273553000"
  }
}

Fetch from last response last record

When the parameter’s next URL is fetched from the previous response’s last record. 

Example: When the response’s last record has a JSON key that provides the link for the next paginated call.

{
  "key": "value",
  "data": [
    {
      "name": "John",
      "age": "32",
      "department": "Sales",
      "next_page_link": "https://<Base_URL>?cursor=1678273553000"
    },
    {
      "name": "Martin",
      "age": "32",
      "department": "Marketing",
      "next_page_link": "https://<Base_URL>?cursor=1678273553000"
    }
  ]
}

Paginate Using JSON Body

Type

Please choose the relevant Type to provide logic for changing the Request Body for each paginated call.

Add Constant

When the change in request JSON Body key is constant for request Json Body. 

Example: offset=0 and offset=1. Here, the JSON Body key is changing by 1 for each paginated call.

Fetch from last response

When the Request JSON Body key next value is updated from the previous response. 

Example: cursor=12345 can be a key in response that is available at a path in response that can be used to make another page call.

Fetch from last response last record

When the Request JSON Body key next value is updated from the previous response’s last record. 

Example: id_greater_than=2399 can be a key in response that is a column in the response’s last record that can be used to make another page call.

Increment By value fetched from last response

When the Request JSON Body key next value is sum of previous value and value fetched the previous response. 

Example: limit=10 can be value in response which is in getting added to previous value 0 for first page call.

Increment By value fetched from last response last record

When the Request JSON Body key next value is sum of previous value and value fetched the previous response’s last record. 

Example: limit=10 can be value in response’s last record which is in getting added to previous value 0 for first page call.

Key

The JSON Path Key in the Request Json Body that needs to be updated for each paginated call.

Example: $.offset –> Here $ denotes the request body and offset is the key present in JSON Body { “offset”: 0}

Initial Value

The initial value of the request JSON Body Key should be selected.

The parameter’s initial value must be provided in the previous component’s column.

The expression evaluator component can be used in the application to add columns with a constant value. 

Example: 0 -> This will make the intial JSON Request Body as  { “offset”: 0} for the key $.offset

Next Value

The Logic to evaluate the next value of the Request JSON Body Key should be provided.

Example for Add Constant: if the Request JSON Body key next value is incremented by some constant, then provide the constant like some numeric value by which the parameter will get an increment for each paginated call.

Example for Last Response: if the Request JSON Body key next value is evaluated from the last response, then JSON path expression should be provided that points to the next value for the parameter from JSON response.

Example for Last Response last record: if the Request JSON Body key next value is evaluated from the last response’s last record, then JSON path expression should be provided that points to the next value for the parameter from JSON response’s last record.

Example for increment by Last Response: if the Request JSON Body key next value is sum of the previous value and evaluated from the last response, then JSON path expression should be provided that points to the next value for the parameter from JSON response.

Example for increment by Last Response last record: if the Request JSON Body key next value is sum of the previous value and evaluated from the last response’s last record, then JSON path expression should be provided that points to the next value for the parameter from JSON response’s last record.

Pagination End Condition

Create a pagination end condition using And/OR operators. Pagination will stop once the end condition is true for individual records.

Cursor based last response

When the pagination stop condition depends on a key from the last response.

Example: has_more is a key with its value as false for the last page.

Cursor based last response last record

When the pagination stop condition depends on a key from the last response’s last record.

Example: id_greater_than is a key with its value as null for the last page.

Till Page number

Stop after reading the specified number of pages.

Example: To read a maximum of 2 pages for each row.

Offset Condition

Stop when the parameter/offset value is equal/less/greater than some constant value.

Example: Stop when the page parameter value becomes 10.

Till Response data

Read till the data is available in response. This option should only be selected when the last page’s data for the provided path to data field has a value as {},[], or blank.

Response based retry

When you want to retry a current paginated call based on some value present in the last response with Status: 200, then, provide the JSON path key and value for which you want to retry the call.

HTTP_Pagination_Examples_01

  • Example:

    Valid Record

    {
  "code":0,
  "data":[
          {"name":"John",
            "age":18 
          },
          {"name":"Smith",
            "age":42 
          }
      ]
}

    Invalid Record

    {
  "code":40100,
  "data":[]
}

    Due to any exception, if an error occurred with code 40100, a retry condition can be configured.

    The Retry Time and Retry Count should be specified in the HTTP processor configuration.

HTTP Pagination Examples

Specify page offset as cursor-based query parameters from response

This use case is an instance of achieving pagination by providing query parameters in the URL that are fetched from the last response.

Example:

To get results for the first page, the request URL will be https://<Base_URL>?cursor=0.

Response of the first request:

{

  "code": 0,

  "data": {

    "posts": [

      {"name": "John", "age": "32", "department": "Sales"},

      {"name": "Martin", "age": "32", "department": "Marketing"}

    ],

    "page_info": {

      "has_more": true,

      "cursor": **1678273553000  (This is the parameter cursor value for the next page call.)**

    }

  }

}

To get results for the second page, the request URL will be created from the response value of the previous request.

https://<Base_URL>?cursor=1678273553000

It will continue fetching data until the has_more key in response is true or the cursor value is greater than 1678273551000.

Following are the snippets that explain, how the pagination feature can be utilized to achieve this use case.

URL - Query parameters

  1. Enable pagination.

  2. Select Response format as JSON.

  3. Enter Path to Data Field as $.data.posts.

  4. Set Use Offset in to Parameter.

    HTTP_Pagination_Examples_01

  5. Add an entry in the parameter section by clicking Add Field.

    • Choose the Type as Fetch from last response .

    • Specify parameter name as cursor to pass the parameter cursor in the URL.

    • Select initial value from the list of columns added in previous expression evaluator. For example, consider that a cursor_start_column column that has value as 0 was added.

    • The cursor parameter will change with each request. Provide json path expression to fetch the value of next cursor from response. $ is used to access response from last request. For example, $.data.page_info.cursor

Upon configuring these values, the Parameter section will appear as shown below.

HTTP_Pagination_Examples_02

Pagination end condition using cursor-based condition

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response.

  2. Provide the json path expression for key that ends pagination as $.data.page_info.has_more

  3. Select the column that was added in the previous expression evaluator that contains the value for last page s has_more value. For example, end_condition_column with value false.

Upon configuring this, the pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_03

Pagination end condition using cursor or offset-based condition

Create Pagination end condition with AND or OR operator.

To end pagination when either the response contains has_more as false or cursor is less/greater than some value.

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response .

  2. Provide the json path expression for key that ends pagination as $.data.page_info.has_more

  3. Select the column that we added in previous expression evaluator that contains the value for last page s has_more value. For example, end_condition_column with value false.

  4. Add one more Rule and select Offset Condition .

  5. Type the offset name value cursor , select the operator as Less Than.

  6. Select the column that was added in previous expression evaluator that contains the value of cursor for stopping the pagination. For example, end cursor_date_end_column

  7. Click on the OR operator.

Upon configuring this, pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_04

Pagination end condition using till response data condition

To end pagination until there is a response present. The last response page must be an empty JSON data at path to data field.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Response data.

    HTTP_Pagination_Examples_05

Pagination end condition using till page number condition

In this use case, applying pagination condition to read 5 pages only.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Page number.

    HTTP_Pagination_Examples_06

Specify page offset as cursor-based query parameters from last record of response

This use case is an instance of achieving pagination by providing query parameters in the URL that are fetched from last response last record.

For example

To get results for the first page, the request URL will be https://<Base_URL>?cursor=0.

{

  "code": 0,

  "data": {

    "posts": [

      {"name": "John", "age": "32", "department": "Sales", "cursor": **1678273553000**},

      {"name": "Martin", "age": "32", "department": "Market", "cursor": **1678273553000 ** } **(parameter cursor value for the next page call.)**

    ]

  }

}

To get results for the second page, the request URL will be created from the value from last record in response of previous request.

https://<Base_URL>/cursor=1678273553000

It must continue fetching data until the has_more key in response is true.

Following are the snippets that explain, how to configure pagination to achieve this use case.

URL - Query parameters

  1. Enable pagination.

  2. Select Response format to JSON

  3. Enter Path to Data Field as $.data.posts.

  4. Select Use Offset in as Parameter.

    HTTP_Pagination_Examples_01

  5. Add an entry in the parameter section by clicking add field.

    • Choose the Type as Fetch from last response last record.

    • To pass parameter cursor in the URL, Specify parameter name as cursor.

    • Select initial value from the list of columns added in previous expression evaluator. For example, consider that a cursor_start_column column that has value as 0 was added.

    • The cursor parameter will change with each request. Provide json path expression to fetch the value of next cursor from last response last record. $ is used to access response from last request last response. For example, $.cursor

Upon configuring these values, the Parameter section will appear as shown below.

HTTP_Pagination_Examples_07

Pagination end condition using cursor-based last response last record condition

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response last record.

  2. Provide the json path expression for key that ends pagination as $.cursor

  3. Select the operator as is Null.

Upon configuring this, pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_08

Pagination end condition using till response data condition

To end pagination until there is response present. The last response page must be an empty JSON data at path to data field.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Response data.

    HTTP_Pagination_Examples_05

Pagination end condition using till page number condition

In this use case, applying pagination condition to read 5 pages only.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Page number.

    HTTP_Pagination_Examples_06

Specify page offset as constant incrementing query parameters

This use case is an instance of achieving pagination by providing query parameters in the URL that are incrementing by some constant value.

For example

To get results for the first page, the request URL will be https://<Base_URL>?page=1.

To get results for the Second page, the request URL will be https://<Base_URL>?page=2.

Response sample:

{

  "data": [

      {"name": "John", "age": "32", "department": "Sales"},

      {"name": "Martin", "age": "32", "department": "Marketing"}

    ],

    "page_info": {

      "has_more": true

    }

}

It must continue to fetch data until the has_more key in response is true.

Following are the snippets that explain, how to configure pagination to achieve this use case.

URL - Query parameters

  1. Enable pagination.

  2. Select Response format to JSON

  3. Enter Path to Data Field as $.data.

  4. Select Use Offset In as Parameter.

    HTTP_Pagination_Examples_09

  5. Add an entry in the parameter section by clicking add field.

    • Choose the Type as Add Constant.

    • To pass parameter page in the URL, Specify parameter name as page.

    • Select initial value from the list of columns added in previous expression evaluator. For example, suppose that a start_column column that has value as 1 was added.

    • The page parameter will be incremented with each request. Provide 1 as the next value increment by field.

HTTP_Pagination_Examples_10

Pagination end condition using cursor-based condition

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response.

  2. Provide the json path expression for key that ends pagination as $.page_info.has_more

  3. Select the column that was added in previous expression evaluator that contains the value for last page s has_more value. For example, end_condition_column with value false.

Upon configuring this, the pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_11

Pagination end condition using till response data condition

To end pagination until there is response present. The last response page must be an empty JSON data at path to data field.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Response data.

    HTTP_Pagination_Examples_05

Pagination end condition using till page number condition

In this use case, applying pagination condition to read 5 pages only.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Page number.

HTTP_Pagination_Examples_06

Specify URL cursor for pagination using last response

This use case is an instance of achieving pagination by fetching the URL from the last response.

For example

To get results for the first page, the request URL will be https://<Base_URL>?cursor=0.

Response of first request:

{

  "code": 0,

  "data":

 [

      {"name": "John", "age": "32", "department": "Sales"},

      {"name": "Martin", "age": "32", "department": "Marketing"}

    ],

    "page_info": {

      "has_more": true,

      "next\_page\_link": ** **https://&lt;Base_URL&gt;?**curso**r=1678273553000** ****  (This is URL link for the next page call.)**

    }

}

To get results for the second page, the request URL will be next_page_link from the from response of previous request.

It should continue fetching data until the has_more key in response is true or next_page_link is not null.

Following are the snippets that explain, how to configure pagination to achieve this use case.

URL - Query parameters

  1. Enable pagination.

  2. Select Response format to JSON

  3. Enter Path to Data Field as $.data.

  4. Select Use Offset in as URL cursor.

    HTTP_Pagination_Examples_12

  5. Select the type as Fetch from last response.

  6. Provide the path to json Expression for next_page_link. $ is used to signify the response and path to link would be $.page_info.next_page_link.

    HTTP_Pagination_Examples_13

Pagination end condition using cursor-based condition

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response.

  2. Provide the json path expression for key that ends pagination as $.page_info.has_more

  3. Select the column that we added in previous expression evaluator that contains the value for last page s has_more value. For example, end_condition_column with value false.

  4. Add one more rule with || (OR) condition and select Cursor based last response.

  5. Provide the json path expression for key that ends pagination as $page_info.next_page_link

  6. Select the operator as Is Null.

Upon configuring this, pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_14

Pagination end condition using till response data condition

To end pagination until there is response present. The last response page must be an empty JSON data at path to data field.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Response data.

    HTTP_Pagination_Examples_05

Pagination end condition using till page number condition

In this use case, applying pagination condition to read 5 pages only.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Page number.

    HTTP_Pagination_Examples_06

Specify URL cursor for pagination using last response last record

This use case is an instance of achieving pagination by fetching the URL from the last response.

For example

To get results for the first page, the request URL will be https://<Base_URL>?cursor=0.

Response of first request:

{

  "key":  value ,

  "data":

 [

      {"name": "John", "age": "32", "department": "Sales","next\_page\_link": ** **https://&lt;Base_URL&gt;?**curso**r=1678273553000**  ** },

      {"name": "Martin", "age": "32", "department": "Marketing","next\_page\_link": ** **https://&lt;Base_URL&gt;?**curso**r=1678273553000** **** ** }

    ]

}

To get results for the second page, the request URL will be next_page_link from the from last record of response from previous request.

It should continue to fetch data until the next_page_link is not null.

Following are the snippets that explain, how to configure pagination to achieve this use case.

URL - Query parameters

  1. Enable pagination.

  2. Select Response format to JSON

  3. Enter Path to Data Field as $.data.

  4. Select Use Offset in as URL cursor.

    HTTP_Pagination_Examples_12

  5. Select the type as Fetch from last response last record.

  6. Provide the path to json Expression for next_page_link . $ is used to signify the last record from last response and path to link would be $.next_page_link .

    HTTP_Pagination_Examples_15

Pagination end condition using cursor-based condition

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response last record .

  2. Provide the json path expression for key that ends pagination as $.next_page_link

  3. Select the operator as Is Null.

Upon configuring this, pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_16

Pagination end condition using till response data condition

To end pagination until there is response present. The last response page must be an empty JSON data at path to data field.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Response data.

HTTP_Pagination_Examples_05

Pagination end condition using till page number condition

In this use case, applying pagination condition to read 5 pages only.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Page number.

    HTTP_Pagination_Examples_06

Specify next page offset in request JSON Body

This use case is an instance of achieving pagination by providing keys to the request body of a post-call.

For example

To get results for the first-page request URL will be https://<Base_URL>/getData

And request Body for the first-page request will be

{  
    "options":{  
                     "offset":0,  
                     "limit":100  
                   }  
}

Response from first request

{

  "next_offset": 100,

  "data":

 [

      {"name": "John", "age": "32", "department": "Sales"},

      {"name": "Martin", "age": "32", "department": "Marketing"}

    ]

}

Request Body for the Second-page request will be

{  
    "options":{  
                     "offset":100,  
                     "limit":100  
                   }  
}

The offset value changes for each request call. It should continue to fetch data until next_offset is not NULL in response.

Following are the snippets that explain how we can configure pagination to achieve this use case.

URL - Query parameters

  1. Enable pagination.

  2. Select Response format to JSON.

  3. Enter Path to Data Field as $.data.

  4. Select Use Offset in as JSON Body.

    HTTP_Pagination_Examples_17

  5. Add an entry in the JSON Body section by clicking add field.

    • Choose the Type as Increment by Constant.

    • It is needed to pass change offset in request body for each call. Specify JSON Path key that needs to be updated in request body. For example, $.options.offset

    • Select initial value from the list of columns added in previous expression evaluator. For example, suppose a start_column column that has value as 1 was added.

    • The offset parameter will be incrementing with each request. Provide 100 as next value increment by field.

    HTTP_Pagination_Examples_18

Pagination end condition using cursor-based condition

  1. Add a Pagination End Condition by clicking on Add Rule and select Cursor based last response.

  2. Provide the json path expression for key that ends pagination as $.next_offset . $ denotes the last response.

  3. Select the operator as Is Null

Upon configuring this, the pagination end condition section will appear as shown below.

HTTP_Pagination_Examples_19

Pagination end condition using till response data condition

To end pagination until there is response present. The last response page must be an empty JSON data at path to data field.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Response data .

    HTTP_Pagination_Examples_05

Pagination end condition using till page number condition

In this use case, applying pagination condition to read 5 pages only.

  1. Add a Pagination End Condition by clicking on Add Rule and select Till Page number .

    HTTP_Pagination_Examples_06

If you have any feedback on Gathr documentation, please email us!

← HTTP Processor
JDBC-Container Processor →