HTTPV2 Pagination

Perform pagination using Query Parameters

This help section covers steps to perform pagination by providing Query parameters.

Use this option when the query parameters are changing in the URL.

To generate URLs for pagination condition, provide query parameters by selecting the ADD ROW option.

Follow the process to perform pagination:

Parameter Name

Provide query parameter name that will be used in the URL followed by the Initial value of the query parameter.

Example: page -> This will make the URL as http://www.<url>.com?page=<query_parameter_value>

Initial Value

The initial value of the query parameter should be provided.

Example: 1 -> This will make the URL as http://www.<url>.com?page=1

Options available to select the initial value data types are:

Integer, Double, String, Boolean, Date, Timestamp, Time in Millis.

Next Value:

Provide the options to derive the next value for the parameter using below options.

Example: $.nextpage -> This will make the URL for next call as http://www.<url>.com?page=2

Increment by constant

The initial value provided will be incremented by constant value.

Example:

The Start Parameter’s initial value is 0. This value will be incremented by 20.

increment_by_constant

Fetch from last response

Based upon the API, the value of the start parameter will be picked from the last response.

Example:

If the last response is:

fetch_from_last_response

Then, the start value will be 20 and the expression will be $.meta_data.start as shown in the JSON Path Evaluator.

json_path_evaluator

JSON Path Evaluator

Use the JSON Path Evaluator

  1. Provide a sample JSON response.

  2. Navigate to the specific key of the JSON, click on it for the path to appear on the search bar.

  3. Option to write the path in the search bar is available. Click the EVALUATE button for path evaluation. If the path and provided JSON are correct, then the evaluated data will appear in the Evaluation data section.

  4. Option to use Gathr natural language is available. Enable the toggle button to generate the path.

Fetch from last response last record

The data will be picked from the last response’s, last record.

Example:

If below json is the last response, the data will be picked from last response’s last record i.e., $.start in this case.

fetch_from_last_response_last_record

json_path_evaluator02

Increment by value fetched from last response

The current value of the parameter will be incremented from the last response.

Example:

If start initial value is 0 and the value from given path is $.meta_data.start then the next value will be 0+20 i.e., 20. And the next value incremented will be 40.

{
   meta_data: {
     start: 20,
    has_more: false
   }
 }

Increment by value fetched from last response last record

The current value of the parameter will be incremented from the last response’s last record.

Example:

If start initial value is 0 and the value from given path is $.start then the next value will be 0+20 i.e., 20. And the next value incremented will be 40.

{
   meta_data: {
     start: 20,
    has_more: false
   }
 }

Custom expression

Option to write spark custom expression.

Example:

CAST(get_json_object(last_response_last_record, “$.id”) AS int)

You can enable the Persist across run checkbox to retain the last offset value as the initial value for the subsequent run.

Using Current Date

Example:

Customize the Initial Value selected as Date.

To assign a custom date of the parameter value, select the initial value as Date by providing date format like yyyy-dd-mm and Next Value as Using Current Date option.

Assign a custom date by selecting the Edit option as shown below.

Options to add to current date or subtract to current date to derive custom value are available.

query_parameter_date

query_parameter_assign_date

Persist across run

Enable the checkbox to retain the last offset value as the initial value for the subsequent run.

Generate URL

This option is used to validate the provided configurations. If the provided configurations are provided acurately then it will generate a curl.

generate_curl_query_parameter

Pagination End Condition

Provide Pagination End Condition. To achieve so, click the ADD RULE button.

Example:

Last Page Record Count

If this option is selected, then the pagination end condition will be applied to the last record count of the page.

last_page_record_count

Cursor from last page

If this option is selected, then the pagination end condition will be like $.meta_data.start. When the $.meta_data.start value will be 60, then the pagination will stop.

cursor_from_last_page

Cursor from last record of last page

If this option is selected, then the pagination end condition will be applied to stop at last record of the last response.

Example:

If this option is selected, then the pagination end condition will be like $.meta_data.start. When the $.meta_data.start value will be 60, then the pagination will stop.

cursor_from_last_record_of_last_page

Offset Condition

The offset name which has been used in the parameter section should be provided that can be used in the Custom Expression and Pagination End Condition expression.

Example: Read till start where pagination will stop when the start value will be less than 60.

offset_condition

Total Record Count

total_record_count

The pagination end condition applied to select url Cursor from last page. Stop when attribute from last response is selected.

Based on the above selected conditions click the Generate CURL option to generate the links.

generate_curl_query_parameter

You can also provide Timeout option to set a timeout and stop fetching data from paginated calls by terminating the application after the specified time interval (in seconds).

Enable Request Rate to Control the rate at which pagination requests are made.

Example:

Make 50 requests in 10 seconds, on throttle exception wait for 60 seconds.

Perform pagination using URL Cursor

This help section covers details on how you can perform pagination using HTTPV2 by providing URL Cursor parameters.

There are 3 ways by which Url for subsequent page request can be done.

These options available in the drop-down are:

pagination_using_urlcursor

  • Assign URL. This option lets you assign a customized URL. Example: concat($url,"?page=5").

  • Fetch from last response

For Example:

fetch_from_last_response_curl_cursor

  • Fetch from last response last record

For Example:

![Fetch from last response last record_curl_cursor](./images/ETL_Data_Sources/Fetch from last response last record_curl_cursor.png)

Generate URL

This option is used to validate the provided configurations. If the provided configurations are provided acurately then it will generate a curl.

generate_curl_query_parameter

Likewise, provide the Pagination End Condition

Perform pagination using JSON Body

Pagination can be performed using JSON Body parameters.

This option is available for use when Request method is selected as POST.

  • Provide the JSON path key.

  • Provide Initial Value.

  • Based on the API, select the Next Value.

  • Assign Pagination End Condition.

  • Generate CURL value. This option is used to validate the provided configurations. If the provided configurations are provided acurately then it will generate a curl.

generate_curl_json_body

pagination_using_json_body

Perform pagination using Header

Pagination can be performed using Header parameters if the API to be used contains header.

Provide the key value and initial value.

Based on the initial value selected, choose an option from the Next value by which pagination must be performed.

The available options are as follows:

generate_curl_header

pagination_using_header

If you have any feedback on Gathr documentation, please email us!

← HTTPV2 ETL Source
HTTPV2 Incremental Configuration →