Expression Filter Processor
This processor is responsible for performing filtering operations on incoming dataset, e.g., equals, contains, not-null, ranges, matches, starts-with, ends-with, etc. It uses spark expression language for constructing filter criteria.
Expression Filter Configuration
To add an Expression Filter processor into your pipeline, drag the processor to the canvas and right click on it to configure.
| Field | Description |
|---|---|
| Expression | Provide SQL expression using SparkSQL functions, that evaluates to true/false, e.g., sqrt(ceil(TXN_ID))>2.0 will return either true or false. Rows which satisfy the criteria, e.g. return true, will remain in stream and rest will be filtered out. Note: Enclose string literal/constant values inside single quotes e.g. ‘John’, ‘Suite 130 Los Gatos, CA 95032, US’. |
| Validate | Validate the expressions applied on the column. |
Refer to the Expression Filter section in the topic Data Preparation Processors.
The user can apply any transformation and the list of functions that can be applied are all listed in the right table on the user interface.
If you have any feedback on Gathr documentation, please email us!