SQL Processor
SQL Processor allows you to run SQL queries over streaming data and registered tables. It provides a common way for accessing different data sources.
For example, if you want to analyze weather data stream continuously and find the average temperature which is recorded every minute, you can use the SQL query. The output would be stream of new records showing average temperature recordings.
SQL Processor Configuration
To add an SQL processor to your pipeline, drag the processor onto the canvas and right-click on it to configure as explained below:
| Field | Description |
|---|---|
| Download entered queries JSON | User has an option to download the JSON queries. |
| ADD CONFIGURATION | User can add configuration by clicking the ADD CONFIGURATION option. |
| Table Name | Displays all the registered tables. Select the table on which query is to be performed. |
| Read queries from S3 | Check mark the checkbox if user wants to read the queries from S3. User will be required to validate the S3 queries. Note: User can download the Sample S3 queries JSON and S3 queries JSON. |
| S3 Connection | Select the S3 connection name. |
| Bucket Name | Specify the bucket name |
| Path | Specify the path. Note: - After entering values for S3 Connection, Bucket Name and Path, click Fetch S3 Queries button. - If the Read Queries from S3 checkbox option is unchecked, user can add query by following the below steps under the Query section. |
| Query Name | Provide name for each query to register query result as a dynamic table. |
| Skip While Inspect | Select this option to skip the query while inspect. If a SQl has CREATE/DROP/DELETE/UPDATE statements, it is recommended to skip it. As this will get executed over actual tables/Datasets. If select query is skipped, it will not return any dataset to be used further for pipeline creation. |
| Query | Provides option of auto suggestion that enables to write query instantaneously. Press CTRL + space in the query text box. It will display all the options like Table name, schema name, schema fields, query operations for writing the query. Examples: Click SELECT from the list and write the query. SELECT * from Kafka_657752 Click Delete from the list and write the query Delete Emp_table Accordingly, select other options from the auto suggestion list and enter the query. |
| ADD QUERY | Click this button to add multiple queries that can be executed over different tables. |
Click on the NEXT button. Enter the notes in the space provided.
Click SAVE for saving the configuration details.
If you have any feedback on Gathr documentation, please email us!