HTTPV2 Data Source

The HTTPV2 data source can fetch data based on requests sent to HTTP and HTTPS resources via Gathr.

Information that will be useful while configuring the data source:

Type of the source data (for example, CSV, JSON, Text, or XML)
Resource URI to access the source data.
Authentication type of the resource and details required for Gathr to access the data (for example, if Auth Type is Basic, then, Username and Password will be needed to access the resource)

The HTTPV2 data source configuration is equipped to handle SSL-enabled resources.

Along with the features that HTTP Data Source supports, the HTTPV2 channel additionally supports XML Data Type, enhanced authentication methods, can fetch paginated responses from the HTTP data resource, and can read the data incrementally.

Schema Type

Fetch From Source/Upload Data File/Use Existing Dataset

For designing the application, you can either fetch the sample data from the HTTP resource by providing its URI, upload a sample data file in one of the supported formats, or, use an existing HTTP dataset created in Gathr to see the schema details during the application design.

💡

The File Format field is mandatory for both, Fetch From Source and Upload Data File options.

File Format

Select the File Format matching the format of data in the HTTP resource to be requested.

Gathr supports CSV, JSON, TEXT, and XML file formats for HTTPV2 data sources.

For CSV file format, select its corresponding delimiter.
Header Included: Specify if the first row should be considered a header row while reading the sample file.
For JSON file format, there will be an additional field called Path to Data.
For XML file format, there will be an additional field called XML Path.
XML Path is an XML path expression that points to arrays.

Upload

Please upload the sample data file as per the file format selected. The schema of data items in the sample file should be the same as the HTTP resource that is to be read.

👉

Make sure that the file size does not exceed 10 MB.

If Fetch From Source is selected, continue configuring the data source.

After providing schema type details, the next step is to configure the data source.

Data Source Configuration

Configure the data source by providing request parameters that are explained below.

1. Request Method

The HTTP request method to fetch data from a source should be selected out of GET or POST.

The method selected determines how the configured parameters in the data source will be submitted to the HTTP resource.

GET: Use the GET method if form data is included in the URI, appended as query string parameters.

POST: Use the POST method if the form data is to be given in the request body of the HTTPV2 data source.

2. URI

The HTTP or HTTPS URI should be provided to send requests to a resource via the HTTPV2 data source.

The URI should contain all the details that the HTTPV2 data source will need to find the API resource it is requesting.

Example: https://<hostName>:<portNumber>/path/_queryString

3. Request Body (Optional)

👉

The Request Body field will appear only for post requests.

Using the request body option, a data payload can be sent to an HTTP resource in the body of the request.

Example

{
  "from":1,
  "size":10,
  "query":{"match_all":{}}
}

4. Design Time Attributes (Optional)

Use this option to limit the volume of data fetched from a source during the application design time. Here, the main objective is to see the schema details of the incoming source data.

Based on the request method, the Design Time Attributes can be provided by selecting URI, Body, or a combination of both.

Design Time Request URI

💡

The request URI provided here will only be used during application design time.

Example: https://example.com?page=1&date=12-01-1999

Design Time Request Body

💡

This parameter is applicable for POST request methods. The request body provided here will only be used during application design time.

Example

{
  "from":1,
  "size":5,
  "query":{"match_all":{}}
}

5. Header (Optional)

Use the header field(s) to provide additional requests to the HTTP resource via the HTTPV2 data source.

💡

HTTP headers have a key-value pair format.

Example

6. Auth Type

Specify the authorization type that should be used to authenticate the HTTP resource.

The supported authentication types are: None, Basic, Token, OAuth2, OAuth2 Client Credentials, and Custom Token. Each type is explained below in detail.

Auth Type - None

Choose this option to access an HTTP resource without needing any authentication.

Auth Type - Basic

If Auth Type is selected as Basic, proceed after providing the below parameters:

Username: Enter the user name for accessing the HTTP resource.

Password: Enter the password for accessing the HTTP resource.

Auth Type - Token Based

💡

Token-based authentication is a security technique that authenticates the users who attempt to log in to a server, a network, or any other secure system, using a security token provided by the server.

If Auth Type is selected as Token, proceed after providing the below parameters:

Token ID: The key with which the token is referred in the request.

Token: Token to access the HTTP resource.

Auth Type - Oauth2

💡

Oauth2 is an authentication technique in which the application gets a token that authorizes access to the user's account.

If Auth Type is selected as OAuth2, proceed after providing the below parameters:

Auth Headers: The headers associated with Auth URL should be provided as key-value pairs, through which the authorization code is generated.

Client ID: The client identifier given during the application registration process should be provided.

Secret Key: The secret key given to the client during the application registration process should be provided.

Auth URL: The endpoint for the authorization server, which retrieves the authorization code should be provided.

Auth Type - OAuth2 (Client Credential)

💡

The grant_type is client credential in the case of OAuth2 (Client Credential). You can also pass parameters to the authentication URL using this authentication method.

If Auth Type is selected as OAuth2 (Client Credential), proceed after providing the below parameters:

Auth Headers: Header’s parameter name and value can be provided.

Auth Params: Auth parameter name and value can be provided.

ClientId: The client identifier that is given to the client during the application registration process should be provided.

Secret Key: The secret key that is given to the client during the application registration process should be provided.

Auth URL: The endpoint for the authorization server, that retrieves the authorization code.

Use Token: Use generated token in the URL parameter or header of the request.

Auth Type - Custom Token

💡

Use this auth method to first get a token from a custom API, that can be used in the main data API to fetch the data as an authentication key.

👉

Make sure that the response content type of the custom token request is JSON.

If Auth Type is selected as Custom Token, a modal window Token Generation HTTP Configuration will appear. Proceed after providing the below parameters:

URI: HTTP or HTTPS URI to send a request to a resource.

Request Method: HTTP request method for the URI to be selected out of GET or POST.

Request Body: Request body to send a data payload to an HTTP resource in the body of the request.

👉

The Request Body field will appear only for post requests.

Header: Header’s parameter name and value.

Example:

In the below snapshot you can see that the main data API needs a token in the Header with the key as Authorization and value as BEARER ${token} and then ${token}.

${token} will be replaced with the custom token generated from the custom token API.

${token} can be used in the value of the header key which will be replaced by the token generated from the custom token API.

Path to Token: JSON path expression that points to tokens.

Auth Type: Used to specify the authorization type associated with the URL. The supported auth types for token generation are None, Basic, and Token.

None: This option specifies that the URL can be accessed without any authentication.
Basic: This option specifies that accessing the URL requires Basic Authorization.
Provide a user name and password for accessing the URL.
Token: Token-based authentication is a security technique that authenticates the users who attempt to log in to a server, a network, or other secure systems, using a security token provided by the server.

7. Enable SSL (Optional)

It is set to False by default.

Set this option to True, if the resource that is to be requested using the HTTP data source is SSL-enabled.

If set to True, choose how the SSL-enabled HTTP resource should be verified.

Either a keystore file or a certificate file needs to be uploaded based on the chosen verification method.

The Keystore Password or Certificate Alias should then be provided as per the type of file uploaded for verification.

8. Path to Data (For JSON Data Type)

👉

Path to Data filed will appear only when the type of source data is specified as JSON in the File Format option.

Path to data is a JSON path expression that points to arrays.

Example: Consider fetching the schema details of a source for the sample JSON file illustrated below:

If you keep the Path to Data field value as $ (i.e., default value), then as per the data source configuration the entire JSON file will get fetched.

But, if you provide the Path to message field value as $.data, then only the attributes of the data element along with their values will get fetched from the source as shown in the image below:

Add Configuration

Additional properties can be added using this option as key-value pairs.

noOfFrames

To decrease the amount of data in a DataFrame, you can adjust the number of frames based on the available memory or the amount of data. By doing this, you can reduce the amount of processing required for each frame.

persistAfterPages

The configuration parameter persistAfterPages can be used to control the number of pages per frame.

👉

Use the noOfFrames configuration instead of persistAfterPages to achieve better results.

👉

Currently, the logs do not display the memory size of each frame.

Pagination

To know more about the pagination options in HTTPV2, see HTTPV2 Pagination →

Incremental Read

Optionally, you can enable incremental read. For more details, see HTTPV2 Incremental Configuration →

Pre Action

To understand how to provide SQL queries or Stored Procedures that will be executed during pipeline run, see Configure Pre-Action in Source →

If you have any feedback on Gathr documentation, please email us!

HTTPV2 Data Source

Schema Type #

Data Source Configuration #

1. Request Method #

2. URI #

3. Request Body (Optional) #

4. Design Time Attributes (Optional) #

Design Time Request URI #

Design Time Request Body #

5. Header (Optional) #

6. Auth Type #

Auth Type - None #

Auth Type - Basic #

Auth Type - Token Based #

Auth Type - Oauth2 #

Auth Type - OAuth2 (Client Credential) #

Auth Type - Custom Token #

7. Enable SSL (Optional) #

8. Path to Data (For JSON Data Type) #

Add Configuration #

noOfFrames #

persistAfterPages #

Pagination #

Incremental Read #

Pre Action #

Schema Type

Data Source Configuration

1. Request Method

2. URI

3. Request Body (Optional)

4. Design Time Attributes (Optional)

Design Time Request URI

Design Time Request Body

5. Header (Optional)

6. Auth Type

Auth Type - None

Auth Type - Basic

Auth Type - Token Based

Auth Type - Oauth2

Auth Type - OAuth2 (Client Credential)

Auth Type - Custom Token

7. Enable SSL (Optional)

8. Path to Data (For JSON Data Type)

Add Configuration

noOfFrames

persistAfterPages

Pagination

Incremental Read

Pre Action