ExpressionFilter Processor

Perform filtering operations on incoming dataset, e.g., equals, contains, not-null, ranges, matches, starts-with, ends-with, etc. Expression Filter uses spark expression language for constructing filter criteria.

Processor Configuration #

Configure the processor by providing configuration values as detailed below.

Expression #

Provide SQL expression using SparkSQL functions, that evaluates to true/false, e.g., sqrt(ceil(TXN_ID))>2.0 will return either true or false. Rows which satisfy the criteria, e.g. return true, will remain in stream and rest will be filtered out.

Example: ‘John’, ‘Suite 130 Los Gatos, CA 95032, US’

All the columns of the schema are populated and you can apply any transformation on it. The functions that can be applied are all listed in the right table.

Download Sample #

A CSV file with a few sample expressions can be downloaded to understand how various expressions can be applied on multiple columns using the upload option.

Upload #

The sample CSV file can be modified with custom expressions and uploaded using this option.

Once it is uploaded, the intended expressions will get updated in the Expression Filter transformation.

Add Expression #

Option to add more expression entries in the Expression Filter transformation.

Validate #

Option to validate the correctness of all the expressions that have been added in the transformation.

All the columns of the schema are populated and you can apply any transformation on it. The functions that can be applied are all listed in the right table.

Once you are ready with the required expression, click on the Validate button to check its correctness.

Ask Assistant to Fix #

If your drafted expression fails the validation check, you can utilize the ‘Ask Assistant to Fix’ option to receive a more accurate response. The chat assistant will provide a corrected expression that you can use.

Alternatively, you can utilize the AI Assistant feature to generate expressions for your use cases.

Ask AI Assistant #

Use the AI assistant feature to simplify the creation of custom expressions.

It allows you to generate complex expressions effortlessly, using natural language inputs as your guide.

Describe your desired expression in plain, conversational language. The AI assistant will understand your instructions and transform them into a functional expression.

Tailor expressions to your specific requirements, whether it’s for data transformation, filtering, calculations, or any other processing task.

Note: Press Ctrl + Space to list input columns and Ctrl + Enter to submit your request.

Input Example:

Retrieve cutomers whose last_purchase_date is more than 60 days.