Custom Processor

This processor enables you to create a custom business logic. It is used for components that are not inbuilt within Gathr.

Implements the com.Gathr.framework.api.processor.JSONProcessor interface and provide custom business logic in implemented methods.

You will be able to perform operations on datasets that are not supported by other processors.

Prerequisites

To use a custom processor, you need to create a jar file containing your custom code. Then upload the jar file in a pipeline or as a registered component.

To write a custom logic for your custom Data Source, download the sample project.

Import the downloaded sample project as a maven project in Eclipse. Ensure that Apache Maven is installed on your machine and that the PATH for the same is set.

Implement your custom code. Build the project to create a jar file containing your code, using the following command:

mvn clean install –DskipTests

If the maven build is successful, you need to upload the jar on the pipeline canvas.

Custom Processor for Spark #

Custom processor for Spark supports writing code in Java programming language.

While using a Custom processor in the pipeline, you need to create a class that implements com.Gathr.framework.api.spark.processor.CustomProcessor interface. Add unimplemented methods in this class to write your business logic in it.

Shown below is a sample class structure:

There are three methods to implement.

1. Init: Enables entering any initialization calls.

2. Process: Contains actual business logic. This method is called for each tuple.

3. Cleanup: All resource cleanup occurs in this method.

Custom Processor Configuration #

To add a Custom processor to your Spark pipeline, drag the Custom processor on the canvas. Right-click to configure it as explained below:

Click on the NEXT button. Enter the notes in the space provided.

Click SAVE for saving the configuration details.