Pipeline Builder

Create and manage data pipelines with the visual drag-and-drop builder.

Overview

The Pipeline Builder uses React Flow to provide a canvas where you wire together nodes into data flows. Drag nodes from the palette, connect them, configure each node, and deploy.

Creating a Pipeline

1. Navigate to Build > Pipelines in the sidebar
2. Click Create Pipeline (or select a template)
3. Drag nodes from the palette onto the canvas
4. Connect nodes by dragging from output to input handles
5. Configure each node by clicking on it
6. Click Save Pipeline

Node Types

Source Nodes

Processor Nodes

Sink Nodes

ML Nodes

Pipeline Templates

Pre-built templates for common patterns:

Select a template from the Create Pipeline dialog to start with a pre-wired flow.

Test Mode

Before deploying, validate your pipeline with test mode:

1. Click Test in the toolbar
2. The pipeline runs with a limited data sample
3. Each node shows pass/fail status and sample output
4. Fix issues before deploying to production

Execution Logs

Pipeline execution logs stream in real-time via SSE:

• Open a running pipeline and click the Logs tab
• Logs stream live as the pipeline processes data
• Filter by log level (info, warn, error)
• Download full log history as a file

Pipeline Operations

Keyboard Shortcuts

Export / Import

Export

1. Open a pipeline
2. Click Export in the toolbar menu
3. Pipeline is downloaded as a .json file including all node configurations and connections

Import

1. Click Import Pipeline from the pipeline list
2. Select a .json file
3. Review the imported pipeline on the canvas
4. Save to persist