v0.3.0

Backend Development

Guide for developing the FacilFlow backend.

Technology Stack

Technology Purpose
Node.js 20 Runtime
Express.js HTTP framework
TypeScript Type safety
Drizzle ORM PostgreSQL queries and migrations
Socket.IO Real-time WebSocket events
Vitest Testing framework
Redis Caching, pub/sub
Docker SDK Container management
Firebase Admin JWT verification, RBAC
Winston Structured logging

Project Structure

backend/
├── src/
│   ├── server.ts              # Express app, middleware, Socket.IO setup
│   ├── routes/                # 42 route modules
│   │   ├── pipelines.ts       # Pipeline CRUD and execution
│   │   ├── agents.ts          # Agent management
│   │   ├── health.ts          # Health/readiness/liveness
│   │   ├── clusters.ts        # Compute cluster management
│   │   ├── connectors.ts      # Connector CRUD
│   │   ├── airbyte.ts         # PyAirbyte catalog and sync
│   │   ├── telegraf.ts        # Telegraf plugin management
│   │   ├── apis.ts            # API repository
│   │   ├── storage-providers.ts  # Storage provider config
│   │   ├── buckets.ts         # Bucket operations
│   │   ├── bucket-objects.ts  # Object CRUD within buckets
│   │   ├── ml.ts              # ML model registry
│   │   ├── ml-training.ts     # Training job management
│   │   ├── ml-inference.ts    # Inference endpoints
│   │   ├── ml-approvals.ts    # Model approval workflow
│   │   ├── ml-features.ts     # Feature store
│   │   ├── assistant.ts       # Claude AI chat
│   │   ├── catalog.ts         # Data catalog
│   │   ├── quality.ts         # Data quality rules
│   │   ├── lineage.ts         # Data lineage
│   │   ├── schemas.ts         # Schema registry
│   │   ├── classification.ts  # Data classification
│   │   ├── data-sources.ts    # Data source management
│   │   ├── alerts.ts          # Alert rules and notifications
│   │   ├── scheduler.ts       # Pipeline scheduling
│   │   ├── dashboards.ts      # Dashboard CRUD
│   │   ├── audit.ts           # Audit logs
│   │   ├── users.ts           # User management
│   │   ├── tenants.ts         # Multi-tenant management
│   │   ├── profile.ts         # User profile
│   │   ├── api-keys.ts        # API key management
│   │   ├── edge-config.ts     # Edge agent configuration
│   │   ├── edge-status.ts     # Edge agent status
│   │   ├── edge-registry.ts   # Edge agent registry
│   │   ├── simulator.ts       # Data simulator
│   │   ├── mcp-servers.ts     # MCP server management
│   │   └── mcp-registry.ts    # MCP tool registry
│   ├── services/              # Business logic
│   ├── middleware/             # Auth, RBAC, validation, error handling
│   ├── db/                    # Drizzle schema, migrations
│   └── utils/                 # Shared utilities
├── tests/                     # Vitest tests
└── package.json

API Routes

Core

Pipelines /api/pipelines

Method Endpoint Description
GET / List all pipelines
POST / Create pipeline
GET /:id Get pipeline details
PUT /:id Update pipeline
DELETE /:id Delete pipeline
POST /:id/start Start pipeline execution
POST /:id/stop Stop pipeline execution
GET /:id/metrics Get pipeline metrics
GET /:id/logs Get pipeline logs

Agents /api/agents

Method Endpoint Description
GET / List all agents
GET /:id Get agent details
GET /:id/status Get agent health
POST /:id/restart Restart agent
PUT /:id/config Update agent config

Health /api/health

Method Endpoint Description
GET / Basic health check
GET /ready Readiness probe
GET /live Liveness probe

Clusters /api/clusters

Method Endpoint Description
GET / List compute clusters
POST / Register cluster
GET /:id Get cluster details
PUT /:id Update cluster
DELETE /:id Remove cluster
GET /:id/metrics Cluster resource metrics

Connectors

Connectors /api/connectors

Method Endpoint Description
GET / List configured connectors
POST / Create connector
GET /:id Get connector details
PUT /:id Update connector
DELETE /:id Delete connector
POST /:id/test Test connector connection

Airbyte /api/airbyte

Method Endpoint Description
GET /catalog List available Airbyte sources
GET /catalog/:type/spec Get connector spec
POST /sources Create Airbyte source
POST /sources/:id/discover Discover source schema
POST /sources/:id/sync Trigger sync

Telegraf /api/telegraf

Method Endpoint Description
GET /plugins List available Telegraf plugins
GET /plugins/:name Get plugin details and config schema
POST /configs Generate Telegraf config

API Repository /api/repository/apis

Method Endpoint Description
GET / List registered APIs
POST / Register API endpoint
PUT /:id Update API definition
DELETE /:id Remove API

Storage

Storage Providers /api/storage/providers

Method Endpoint Description
GET / List storage providers
POST / Add storage provider
PUT /:id Update provider config
DELETE /:id Remove provider
POST /:id/test Test provider connection

Buckets /api/storage/buckets

Method Endpoint Description
GET / List buckets
POST / Create bucket
DELETE /:id Delete bucket

Bucket Objects /api/storage/buckets/:bucketId/objects

Method Endpoint Description
GET / List objects in bucket
GET /:key Get/download object
PUT /:key Upload object
DELETE /:key Delete object

ML/AI

ML Models /api/ml

Method Endpoint Description
GET /models List registered models
POST /models Register model
GET /models/:id Get model details
GET /models/:id/versions List model versions

ML Training /api/ml/training

Method Endpoint Description
POST /jobs Submit training job
GET /jobs List training jobs
GET /jobs/:id Get job status
POST /jobs/:id/cancel Cancel training job

ML Inference /api/ml/inference

Method Endpoint Description
POST /predict Run inference
GET /endpoints List inference endpoints
POST /endpoints Deploy model endpoint

ML Approvals /api/ml/approvals

Method Endpoint Description
GET / List pending approvals
POST /:id/approve Approve model version
POST /:id/reject Reject model version

ML Features /api/ml/features

Method Endpoint Description
GET / List feature sets
POST / Create feature set
GET /:id/values Get feature values

Assistant /api/assistant

Method Endpoint Description
POST /chat Send message to Claude AI
GET /conversations List conversations
GET /conversations/:id Get conversation history

Data

Catalog /api/catalog

Method Endpoint Description
GET / Get full catalog
GET /sources List data sources
GET /sources/:id/tags List source tags
POST /query Query data

Data Quality /api/quality

Method Endpoint Description
GET /rules List quality rules
POST /rules Create quality rule
GET /scores Get quality scores
POST /evaluate Run quality evaluation

Lineage /api/lineage

Method Endpoint Description
GET /graph Get lineage graph
GET /assets/:id/upstream Get upstream dependencies
GET /assets/:id/downstream Get downstream consumers

Schemas /api/schemas

Method Endpoint Description
GET / List schemas
POST / Register schema
GET /:id Get schema details
GET /:id/versions List schema versions

Classification /api/classification

Method Endpoint Description
GET /tags List classification tags
POST /classify Classify data asset
GET /assets/:id Get asset classifications

Data Sources /api/data-sources

Method Endpoint Description
GET / List data sources
POST / Register data source
GET /:id Get data source details
DELETE /:id Remove data source

Operations

Alerts /api/alerts

Method Endpoint Description
GET / List alert rules
POST / Create alert rule
PUT /:id Update alert rule
DELETE /:id Delete alert rule
GET /history Get alert history

Scheduler /api/scheduler

Method Endpoint Description
GET /jobs List scheduled jobs
POST /jobs Create scheduled job
PUT /jobs/:id Update schedule
DELETE /jobs/:id Delete scheduled job
POST /jobs/:id/trigger Manually trigger job

Dashboards /api/dashboards

Method Endpoint Description
GET / List dashboards
POST / Create dashboard
GET /:id Get dashboard with widgets
PUT /:id Update dashboard
DELETE /:id Delete dashboard

Audit /api/audit

Method Endpoint Description
GET /logs List audit logs
GET /logs/:id Get audit entry details

Admin

Users /api/users

Method Endpoint Description
GET / List users
GET /:id Get user details
PUT /:id/role Update user role

Tenants /api/tenants

Method Endpoint Description
GET / List tenants
POST / Create tenant
PUT /:id Update tenant

Profile /api/profile

Method Endpoint Description
GET / Get current user profile
PUT / Update profile

API Keys /api/api-keys

Method Endpoint Description
GET / List API keys
POST / Generate API key
DELETE /:id Revoke API key

Edge

Edge Config /api/edge/config

Method Endpoint Description
GET /:agentId Get agent configuration
PUT /:agentId Push configuration to agent

Edge Status /api/edge/status

Method Endpoint Description
GET / All edge agent statuses
GET /:agentId Single agent status
POST /:agentId/heartbeat Agent heartbeat endpoint

Edge Registry /api/edge/registry

Method Endpoint Description
POST /register Register new edge agent
DELETE /:agentId Deregister agent

Other

Simulator /api/simulator

Method Endpoint Description
POST /start Start data simulation
POST /stop Stop simulation
GET /status Simulation status

MCP Servers /api/mcp/servers

Method Endpoint Description
GET / List MCP servers
POST / Register MCP server

MCP Registry /api/mcp/registry

Method Endpoint Description
GET /tools List available MCP tools
POST /tools/:name/invoke Invoke MCP tool

Running Tests

cd backend
npm test              # Run all tests (Vitest)
npm test -- --watch   # Watch mode
npm test -- src/routes/pipelines.test.ts  # Single file

Environment Variables

Backend config lives in backend/.env (not committed):

Variable Purpose
PORT Server port (default: 3002)
DATABASE_URL PostgreSQL connection string
INFLUXDB_URL InfluxDB 3 endpoint
INFLUXDB_TOKEN InfluxDB auth token
REDIS_URL Redis connection string
FIREBASE_PROJECT_ID Firebase project for auth
ALLOWED_ORIGINS CORS allowed origins
MLFLOW_TRACKING_URI MLflow server URL
MINIO_ENDPOINT Default MinIO/S3 endpoint
esc