HTTPV2 Incremental Configuration

Enable Incremental Read

Enable this option to read data incrementally.

Incremental Read option using Query Parameters

This help section covers steps to perform incremental read by providing Query parameters.

Use this option when the query parameters are changing in the URL.

To generate URLs for pagination condition, provide query parameters by selecting the ADD ROW option.

Follow the process to perform pagination incrementally:

Parameter Name

Provide query parameter name that will be used in the URL followed by the Initial value of the query parameter.

Initial Value

Options available to select the initial value data types are:

Integer, Double, String, Boolean, Date, Timestamp, Time in Millis.

Next Value:

Provide the options to derive the next value for the parameter using below options.

Increment by constant

The initial value provided will be incremented by constant value.

Example:

The Start Parameter’s initial value is 0. This value will be incremented by 20.

increment_by_constant

Fetch from last response

Based upon the API, the value of the start parameter will be picked from the last response.

Example:

If the last response is:

fetch_from_last_response

Then, the start value will be 20 and the expression will be $.meta_data.start as shown in the JSON Path Evaluator.

json_path_evaluator

JSON Path Evaluator

Use the JSON Path Evaluator

  1. Provide a sample JSON response.

  2. Navigate to the specific key of the JSON, click on it for the path to appear on the search bar.

  3. Option to write the path in the search bar is available. Click the EVALUATE button for path evaluation. If the path and provided JSON are correct, then the evaluated data will appear in the Evaluation data section.

  4. Option to use Gathr natural language is available. Enable the toggle button to generate the path.

Fetch from last response last record

The data will be picked from the last response’s, last record.

Example:

If below json is the last response, the data will be picked from last response’s last record i.e., $.start in this case.

fetch_from_last_response_last_record

json_path_evaluator02

Increment by value fetched from last response

The current value of the parameter will be incremented from the last response.

Example:

If start initial value is 0 and the value from given path is $.meta_data.start then the next value will be 0+20 i.e., 20. And the next value incremented will be 40.

Increment by value fetched from last response last record

The current value of the parameter will be incremented from the last response’s last record.

Example:

If start initial value is 0 and the value from given path is $.start then the next value will be 0+20 i.e., 20. And the next value incremented will be 40.

{
   meta_data: {
     start: 20,
    has_more: false
   }
 }

Custom expression

Option to write spark custom expression.

Example:

CAST(get_json_object(last_response_last_record, “$.id”) AS int)

You can enable the Persist across run checkbox to retain the last offset value as the initial value for the subsequent run.

Using Current Date

Example:

Customize the Initial Value selected as Date.

To assign a custom date of the parameter value, select the initial value as Date by providing date format like yyyy-dd-mm and Next Value as Using Current Date option.

Assign a custom date by selecting the Edit option as shown below.

Options to add to current date or subtract to current date to derive custom value are available.

query_parameter_date

query_parameter_assign_date

Incremental Read option using URL Cursor

There are 3 ways by which Url for subsequent page request can be done.

These options available in the drop-down are:

pagination_using_urlcursor

  • Assign URL. This option lets you assign a customized URL. Example: concat($url,"?page=5").

  • Fetch from last response

For Example:

fetch_from_last_response_curl_cursor

  • Fetch from last response last record

For Example:

![Fetch from last response last record_curl_cursor](./images/ETL_Data_Sources/Fetch from last response last record_curl_cursor.PNG)

Incremental read option using JSON Body

This option is available for use when Request method is selected as POST.

pagination_using_json_body

Incremental read option using Header

Pagination can be performed using Header parameters if the API to be used contains header.

Provide the key value and initial value.

Based on the initial value selected, choose an option from the Next value by which pagination must be performed.

The available options are as follows:

pagination_using_header

Incremental Help

The Incremental Help section offers in-app help section that can be opened by clicking INCREMANTAL HELP.

If you have any feedback on Gathr documentation, please email us!

← HTTPV2 Pagination
HubSpot Source Connection →