SPORE

Sprocket Orchestration Engine

SPORE

A totally based cluster engine for ESP8266 microcontrollers with automatic node discovery, health monitoring, and over-the-air updates. Built for real-time control of distributed embedded systems.

Core

Cluster engine firmware for ESP8266 microcontrollers

Auto Discovery

UDP-based node discovery with automatic cluster membership

Health Monitoring

Real-time node status tracking with resource monitoring and automatic failover

Event System

Local and cluster-wide event publishing/subscription with WebSocket and UDP streaming API

OTA Updates

Seamless firmware updates across the cluster via HTTP API

Service Registry

Dynamic API endpoint discovery and registration for custom services

Task Scheduler

Cooperative multitasking system for background operations

Quick Start

#include <Arduino.h>
#include "spore/Spore.h"
#include "spore/Service.h"
#include "spore/core/ApiServer.h"
#include "spore/core/TaskManager.h"

class HelloService : public Service {
public:
    HelloService() {}

    const char* getName() const override { return "HelloService"; }

    void registerEndpoints(ApiServer& api) override {
        api.registerEndpoint("/api/hello", HTTP_GET, [](AsyncWebServerRequest* req) {
            req->send(200, "application/json", "{\"message\":\"hello\"}");
        });
    }

    void registerTasks(TaskManager& taskManager) override {
        taskManager.registerTask("heartbeat", 1000, [this]() { this->heartbeat(); });
    } 

private:
    void heartbeat() {
        // e.g., blink LED, publish telemetry, etc.
    }
};

Spore spore({
    {"app", "my_app"},
    {"role", "controller"}
});

void setup() {
    spore.setup();
    spore.registerService(new HelloService());
    spore.begin();
}

void loop() {
    spore.loop();
}

Technical Specifications

  • Supported Hardware: ESP-01/ESP-01S (1MB Flash), Wemos D1 (4MB Flash)
  • Discovery Protocol: Discovery and heartbeat via UDP broadcast
  • API Interface: RESTful HTTP + WebSocket and UDP streaming
  • Dependencies: ESPAsyncWebServer, ArduinoJson
  • Framework: Arduino with PlatformIO build system

Gateway

Gateway service providing UDP-based node discovery, cluster management, and API endpoints

Node Discovery

UDP-based discovery listens for heartbeat messages from SPORE nodes on port 4210 with automatic cluster membership

Cluster Management

Real-time node status tracking, primary node selection, and automatic failover for high availability

WebSocket Server

Real-time cluster updates and event broadcasting to all connected WebSocket clients

MQTT Integration

Subscribe to all MQTT topics and forward messages to WebSocket clients for unified event streaming

Rollout System

Orchestrated firmware updates across multiple nodes with parallel processing and progress tracking

Proxy API

Generic proxy calls to SPORE node capabilities with automatic failover and load balancing

Architecture & Integration

SPORE Gateway acts as the central orchestrator for the SPORE cluster, bridging the gap between SPORE nodes and the UI. It integrates seamlessly with the SPORE ecosystem and external systems through multiple protocols.

Core Components

  • UDP Discovery: Listens on port 4210 for node heartbeats and maintains cluster membership in real-time
  • HTTP API Server: RESTful endpoints for cluster management, node control, and firmware operations
  • WebSocket Server: Broadcasts cluster updates, events, and rollout progress to all connected clients
  • MQTT Client: Optional integration to subscribe to MQTT brokers and forward messages via WebSocket
  • Registry Proxy: Integrates with spore-registry for firmware storage and retrieval
  • Failover Logic: Automatic primary node switching when nodes go offline

MQTT Integration

Enable MQTT integration to stream events from external IoT devices and systems:

# Start gateway with MQTT integration
./spore-gateway -mqtt tcp://localhost:1883

# With authentication
MQTT_USER=username MQTT_PASSWORD=password \
  ./spore-gateway -mqtt tcp://broker.example.com:1883

When enabled, the gateway subscribes to all MQTT topics (#) and forwards messages to connected WebSocket clients, allowing the UI to display both SPORE cluster events and external MQTT events in real-time.

Rollout Orchestration

The rollout system manages parallel firmware updates across multiple SPORE nodes:

  • Automatic Node Discovery: Identifies matching nodes based on firmware labels and current versions
  • Parallel Processing: Updates multiple nodes simultaneously using goroutines
  • Real-time Progress: WebSocket broadcasts show update status for each node
  • Error Recovery: Continues rollout even if individual nodes fail
  • Registry Integration: Seamless integration with spore-registry for firmware storage

Technology Stack

  • Language: Go (Golang)
  • Framework: Standard library net/http with Go 1.22 ServeMux
  • WebSocket: gorilla/websocket for real-time communication
  • MQTT: Eclipse Paho MQTT client library
  • Logging: Structured logging with logrus
  • Concurrency: Goroutines and channels for parallel operations

Registry

Storage backend and update server for managing versioned firmware binaries

Hierarchical Storage

Firmware binaries stored in hierarchical structure: registry/{name}/{version}/firmware.bin with semantic versioning support

SQLite Metadata

Metadata storage with labels for flexible querying, version management, and deployment tracking

REST API

Simple HTTP endpoints for upload, listing, download, and update operations with full CRUD support

Rollout Integration

Seamlessly integrated with Gateway for orchestrated firmware deployment across cluster nodes

Version Management

Semantic versioning support with unique name-version constraints and version history tracking

Label System

Flexible key-value labels for organizing and querying firmware by app, role, or custom attributes

Architecture & Role

SPORE Registry serves as the central firmware storage and management system for the SPORE ecosystem. It provides versioned storage for firmware binaries, enabling the Gateway to orchestrate firmware rollouts across cluster nodes. The registry integrates seamlessly with the Gateway's rollout system, providing the firmware binaries needed for over-the-air updates.

Storage Structure

Firmware is stored hierarchically in the file system:

registry/
├── base/
│   ├── 1.0.0/
│   │   └── firmware.bin
│   └── 1.1.0/
│       └── firmware.bin
└── sensor/
    └── 2.0.0/
        └── firmware.bin

This structure allows easy organization and access to firmware versions by name and version number.

Integration with Gateway

The registry is tightly integrated with the Gateway's rollout system:

  • Firmware Lookup: Gateway queries the registry to find matching firmware for deployment
  • Label Matching: Firmware can be organized with labels (app, role, target) for automated deployment
  • Version Tracking: Registry maintains version history for rollback and audit purposes
  • Binary Distribution: Gateway proxies firmware downloads to nodes during rollout
  • Deployment Automation: Full integration with Gateway's rollout orchestration for cluster-wide updates

Usage Examples

# Upload firmware
curl -X POST http://localhost:3002/firmware \
  -F "metadata={\"name\":\"base\",\"version\":\"1.0.0\",\"labels\":{\"app\":\"base\"}}" \
  -F "firmware=@firmware.bin"

# List firmware
curl http://localhost:3002/firmware

# Download firmware
curl http://localhost:3002/firmware/base/1.0.0 -o firmware.bin

# Health check
curl http://localhost:3002/health

Environment Configuration

The registry supports the following configuration via environment variables:

  • PORT: Server port (default: 3002)
  • DB_PATH: Database file path (default: ./registry.db)
  • REGISTRY_PATH: Firmware storage directory (default: registry)
  • MAX_UPLOAD_SIZE: Maximum upload size in bytes (default: 32MB)

Technology Stack

  • Language: Go (Golang)
  • Database: SQLite for metadata storage
  • Storage: File system for firmware binaries
  • API: HTTP REST API with standard library net/http

UI

Zero-configuration web interface for cluster monitoring and management

Cluster monitoring view

Real-time cluster member overview with auto-discovery

Events visualization

Real-time event flow visualization with interactive graph

Event messages

Live event message viewer for topic inspection

Network topology visualization

Network topology visualization with node relationships

Node monitoring dashboard

Detailed system metrics and task monitoring

Firmware management interface

Clusterwide over-the-air firmware updates

Features

  • Cluster monitoring with real-time status updates
  • Events visualization showing real-time event flow as an interactive force-directed graph
  • Event tracking with animated golden dots showing data flow through cluster topics
  • Node details including running tasks and available endpoints
  • Direct HTTP API access to all nodes in the cluster
  • Over-the-air firmware updates for entire cluster
  • WebSocket terminal for direct node interaction
  • UDP auto-discovery eliminates hardcoded IP addresses
  • Responsive design for all devices

Technology Stack

  • Backend: Express.js, Node.js
  • Frontend: Vanilla JavaScript, CSS3, HTML5
  • Architecture: Custom component-based framework

Apps

Application suite built on SPORE

Explore applications that extend SPORE.

  • LEDLab — Real-time LED matrix animation streaming and visual preset editor

LEDLab

Real-time LED matrix animation streaming and visual preset editor

LEDLab interface

Multi-node management with live canvas preview

Preset editor

Visual preset editor with building blocks

Firmware Requirements

LEDLab requires SPORE nodes running the PixelStreamController firmware, which handles UDP-based RGB data streaming to NeoPixel strips and matrices. The firmware subscribes to udp/raw cluster events and converts incoming hex-encoded pixel data into real-time LED animations.

Key firmware features include support for both strip and matrix configurations, serpentine (zig-zag) wiring patterns, and configurable pixel counts, brightness, and matrix dimensions. The controller automatically remaps pixel coordinates for proper matrix display.

See PixelStream documentation for more information.

Capabilities

  • Multi-node management with individual control
  • Real-time canvas preview for each node
  • 10+ built-in animation presets (rainbow, lava lamp, aurora, etc.)
  • Visual preset editor with reusable building blocks
  • Live parameter control with instant feedback
  • Import/export custom presets as JSON
  • Auto-discovery of SPORE nodes on network
  • Configurable FPS (1-60) and matrix dimensions
  • Requires SPORE nodes running PixelStreamController

Building Blocks

Shapes: Circle, Rectangle, Triangle, Blob, Point, Line
Patterns: Trail, Radial, Spiral
Colors: Solid, Gradient, Palette, Rainbow
Animations: Move, Rotate, Pulse, Oscillate, Fade

Deployment

Complete deployment configurations for running the SPORE stack

Docker Compose

Simple deployment with docker-compose for development and production with one command: docker compose up -d

Nomad Orchestration

Production-ready deployment with HashiCorp Nomad for multi-node clustering, rolling updates, and advanced resource management

Complete Stack

All SPORE services included: mosquitto MQTT broker, gateway, registry, UI, and LEDLab with proper networking and volumes

MQTT Integration

Built-in Mosquitto broker for message routing and event streaming with WebSocket support

Data Persistence

All data persisted in local directories for registry data, MQTT data, and firmware storage

Health Monitoring

Built-in service discovery, health checks, and real-time status monitoring for all services

Quick Start

# Clone the deployment repository
git clone https://git.dcentral.systems/iot/spore-deployment.git
cd spore-deployment

# Start all services with Docker Compose
docker compose up -d

# Or start with Nomad (production)
make nomad-start
make nomad-job-run

Services

  • mosquitto - MQTT broker (port 1883, WebSocket 9001)
  • spore-gateway - Node discovery and WebSocket gateway (port 3001, UDP 4210)
  • spore-registry - Firmware registry service (port 8090)
  • spore-ledlab - LED animation studio (port 8080)
  • spore-ui - Web UI for cluster management (port 3000)

Access Points

Once deployed, access the services at:

  • Web UI: http://localhost:3000
  • LED Lab: http://localhost:8080
  • Registry API: http://localhost:8090
  • Gateway API: http://localhost:3001
  • MQTT Broker: tcp://localhost:1883
  • Nomad UI: http://localhost:4646 (Nomad deployments only)

Documentation

SPORE Core

SPORE UI

SPORE Gateway

SPORE Registry

SPORE LEDLab

SPORE Deployment

Getting Started

SPORE Core

# Build and flash firmware
./ctl.sh build target esp01
./ctl.sh flash target esp01

SPORE Gateway

# Build and start
go mod tidy
go build
./spore-gateway

# With MQTT integration
./spore-gateway -mqtt tcp://localhost:1883

SPORE Registry

# Build and start
go mod download
go run main.go

# Or build and run
go build -o spore-registry
./spore-registry

SPORE UI

# Install and start
npm install
npm start

# Open browser
http://localhost:3001

SPORE LEDLab

# Install and start
npm install
npm start

# Open browser
http://localhost:3000

SPORE Deployment

# Docker Compose (dev)
docker compose up -d

# Nomad (production)
make nomad-start
make nomad-job-run