The EU Cyber Resilience Act (Regulation (EU) 2024/2847) is the “GDPR for connected products.” It entered into force on 10 December 2024, with critical milestones approaching fast:

• 11 September 2026: Mandatory reporting of actively exploited vulnerabilities and severe incidents (24-hour early warning, 72-hour full notification).
• 11 December 2027: Full compliance — Security by Design, lifecycle support (minimum 5 years), technical documentation, and CE marking.

For robotics fleets (AMRs, cobots, autonomous systems, and ROS 2-based platforms) the stakes are particularly high. These systems are “products with digital elements” (often Class II or critical), generating massive multimodal data streams (camera feeds, LiDAR, IMU, logs, ROS bags) under real production constraints: intermittent connectivity, edge hardware limits, and high physical safety risks.

Generic storage solutions force painful trade-offs: either accept data loss and compliance gaps, or accept exploding costs and slow performance. ReductStore eliminates this trade-off.

Why Common Storage Solutions Fall Short for CRA in Production Robotics​

Most robotics teams still rely on a patchwork of rosbag2, MinIO/S3, InfluxDB, or TimescaleDB. While these tools can technically meet some basic CRA requirements with significant custom development, they create substantial challenges when applied to real-world ROS production fleets:

• rosbag2 is excellent for local single-robot recording, but it is not designed for fleet-scale edge-to-cloud architectures, selective replication, or long-term auditability. It lacks native mechanisms for data minimisation, granular access control, and automated compliance reporting.
• MinIO/S3 and general-purpose object stores provide scalable blob storage, but they offer no built-in time-series indexing, label-based filtering, or efficient selective replication. Achieving CRA-compliant audit trails and data minimisation usually requires complex additional layers and custom glue code.
• InfluxDB and TimescaleDB are strong for structured metrics, but they have limited native support for large binary objects (images, LiDAR, ROS bags) and are not optimized for high-throughput multimodal ingestion or production-grade edge-to-cloud workflows under intermittent connectivity.

In practice, these solutions force teams to invest heavily in custom engineering to achieve CRA compliance — often at the expense of performance, simplicity, and cost efficiency. Most importantly, this custom-built approach increases the risk of documentation gaps during a CRA conformity assessment.

The result? Many operators risk non-compliance, higher insurance premiums, or blocked EU market access after 2027.

ReductStore's CRA-Native Architecture for ROS Fleets​

ReductStore is purpose-built as a time-series blob storage engine for multimodal robotics data. It turns compliance from a burden into a competitive advantage.

Key CRA-Aligned Features:

1. Token-Based Authentication & Granular Access Control Every device, service, or user receives scoped tokens. Access is logged at Bucket/Entry/Record level with timestamps. Perfect for proving “who accessed what data and when” — a core CRA requirement.

2. Label-Based Integrity Protection Attach cryptographic labels or hashes to records. The engine supports automatic corruption detection and reporting. Write-once-append-only semantics ensure data cannot be silently altered.

3. Selective Edge-to-Cloud Replication with Data Minimisation Replicate only records matching specific labels or conditions (e.g., “security-critical”, “anomaly detected”, or specific ROS topics). This dramatically reduces attack surface and cloud costs while meeting CRA’s data-minimisation principle.

4. Encryption at Rest and in Transit ReductStore enables strong data confidentiality through multiple proven approaches. Sensitive payloads can be encrypted client-side before ingestion and server-side (e.g., via S3 backend options), while preserving its high-performance batching architecture. All network communication is secured via HTTPS/TLS. The system is fully compatible with ROS 2 Security (SROS2). Native built-in client- and server-side encryption is on the near-term roadmap.

5. Extended Audit Logging & Grafana Integration Detailed logs of all read/write/replication events. Visualize compliance status directly in Grafana dashboards — ready for technical documentation and conformity assessment.

6. SBOM Export & Versioned Updates Generate Software Bill of Materials for ReductStore itself. Security updates are delivered via dedicated channels without disrupting production data flows.

These capabilities are already running in 100+ production deployments managing over 1 PB of multimodal robotics data.

Real-World Results from Production Fleets​

Operators using ReductStore report:

• “We have forgotten about disk overrun problems on our edge devices.” — Ingo Kaiser, PANDA GmbH
• “ReductStore handles terabytes of unstructured data in a production environment.” — Michael Welsch, Metric Space UG
• “We can stop worrying about data collection and focus entirely on building our robots and autonomous layers.” — Victor Massagué Respall, INSAION

In one fleet deployment, selective replication reduced cloud data volume by approximately 85% while maintaining full auditability for CRA preparedness.

Free CRA Checklist for Robotics Operators (Focus: Data Integrity & Storage Layer)​

Download our practical checklist covering:

• Immediate actions before September 2026 reporting deadline
• Mapping ReductStore features to specific CRA Annex I requirements
• Step-by-step migration guide for existing ROS fleets
• Risk assessment template for multimodal robotics data

Download CRA Checklist for ROS Fleets (PDF)

Ready to Make Your Data Streaming And Storage CRA-Ready Without Compromising Performance?​

The September 2026 reporting deadline is approaching fast. Don’t wait until conformity assessment bodies are fully operational in mid-2026.

Book a 20-minute production fleet audit with our team. We will:

• Analyze your current data pipeline against CRA requirements
• Run a live benchmark with your own ROS data
• Show exactly how ReductStore delivers compliance + 10x performance at lower cost

→ Schedule Your Free Fleet Audit

Next up in this series: Physical AI Training Data at Scale: Why Foundation Models Need Specialized Edge-to-Cloud Storage

Robots generate massive amounts of data, and managing it well is harder than it looks. Storage fills up fast, cloud transfer gets expensive, and real time ingestion is unforgiving when you're running cameras and sensors at high frequency.

This article covers practical strategies for handling robotic data, introduces ReductStore, and walks through a hands on example. Along the way, we cover native ROS integration, Grafana dashboards, MCAP export for Foxglove, a Zenoh API, and native S3 and Azure backends. We also compare ReductStore against Rosbag and MongoDB so you can pick the right tool for each part of your stack.

Challenges in Robotic Data Management​

Robots operate in dynamic environments and continuously produce large volumes of data. The core challenges engineers run into are:

• High frequency, real time requirements: A drone navigating a city must process camera and sensor data in milliseconds. Storage solutions need to keep up with these streams and make data accessible fast enough for real time decision making.
• Limited on device storage: Most robots can't store everything they generate. Size, weight, and power constraints limit local capacity, so good data management strategies are essential.
• High data volume: An autonomous vehicle can produce up to 5 terabytes per hour across camera feeds, LiDAR, radar, GPS, and sensors. Most databases weren't designed around that kind of throughput.
• Cloud storage costs: Pushing everything to the cloud isn't practical. Providers charge per gigabyte stored and transferred, and with robots producing terabytes, the bill adds up fast. What you send to the cloud is a decision worth making deliberately.
• Data reduction complexity: Filtering out irrelevant data without losing important information is tricky. Without a clear strategy, you either waste storage or throw away data you later need.

General Strategies for Managing Robotic Data​

A few approaches that work well in practice:

• Use a time series object store: Systems like ReductStore are designed specifically for high frequency, timestamped binary data. They offer fast writes, efficient querying, and manageable costs at scale.
• Balance edge and cloud storage: Keep recent, critical data on the robot for fast access. Move older or lower priority data to the cloud for long term storage and analysis. Splitting storage this way cuts costs without adding much latency to what matters.
• Use volume based retention policies: Rather than deleting data based on age, use a FIFO (first in, first out) quota. Data is only removed when the storage limit is reached, which avoids unnecessary deletion during downtime or low activity periods.
• Compress where it makes sense: H.265 for video, JPEG for images. The right format can cut storage by an order of magnitude without losing anything you actually need.
• Prioritize what to keep: Not all sensor data has equal value. Define what's critical upfront and discard the rest early in the pipeline.

ReductStore: Built for Robotic Data​

ReductStore is a time series database built specifically for unstructured, binary data. It's designed to handle high frequency sensor streams from autonomous vehicles, drones, industrial robots, and IoT. It stores data with timestamps and labels, supports fast real time ingestion, and comes with batching, filtering, and edge to cloud replication built in.

Here's what's available today.

ROS Integration with reduct-bridge​

reduct-bridge connects live ROS 1 and ROS 2 systems directly to ReductStore. You configure it with a simple TOML file that defines your ROS topic inputs, pipelines, and the ReductStore destination. It subscribes to topics and stores each message as an individual record.

It also writes a $ros attachment to each entry with the message schema, topic name, and encoding — that metadata is what powers MCAP export and Grafana visualization, both covered below. If your stack is already on ROS, setup takes minutes.

Cloud Storage: Native S3 and Azure​

ReductStore supports native cloud storage backends for both Amazon S3 and Azure Blob Storage, no FUSE drivers needed. It uses a local cache for hot data and the cloud bucket for long term retention.

For records around 100KB (e.g. JPEG images), ReductStore can be >10x faster than traditional time series object stores at a fraction of the cost.

Visualize ROS Data in Grafana​

ReductStore has a Grafana integration through the ReductStore data source plugin. You can query ROS 2 messages directly in Grafana thanks to the ReductROS extension. The extension decodes binary CDR messages into JSON on the fly.

You can monitor sensor streams live, compare data across multiple robots, and set up alerts when metrics drift.

Export to MCAP for Foxglove​

You can export raw ROS messages stored in ReductStore directly to MCAP and open them in Foxglove. This is powered by the ReductROS extension as well, which reconstructs valid MCAP files on demand.

When records are ingested via reduct-bridge, the $ros attachment carries the schema and topic information. The extension uses this to reconstruct valid MCAP files on demand. You can cover a full time range across multiple topics in a single query, and split by duration or file size for long recordings.

Zenoh Native API​

ReductStore now includes a Zenoh native API alongside the existing HTTP API. Zenoh is a pub/sub protocol designed for robotics and distributed systems, and it's widely used in next generation ROS 2 deployments.

When enabled, ReductStore opens a Zenoh session with a subscriber for writes and a queryable for reads, both running in parallel with HTTP and sharing the same data. Zenoh keys become entry names, encodings map to content types, and attachments map to labels. Time range queries, conditional filters, and label lookups all work the same way they do over HTTP.

It's a natural fit if your robot stack already speaks Zenoh, or if you want to skip HTTP overhead on high frequency ingestion paths.

Query Language and Batching​

ReductStore uses a JSON based query language that supports filtering, aggregation, and time range operations. You can query data from a specific robot within a time window, compare sensor streams across robots in parallel, or filter by any label.

Retrieval is optimized through batching: multiple records are grouped into a single response based on a time range, which cuts down on request overhead and improves throughput. SDKs are available for Python, C++, JavaScript, Go, and Rust. More in the querying guide.

Replication and Edge to Cloud Replication​

ReductStore replicates at the bucket level based on conditions. That means you can choose to send only high priority sensor data to the cloud based on rules, labels, or events. Replication is incremental, so only new data is transferred. Labels stored alongside records let you define fine grained rules based on content, for example, only replicate records flagged as anomalies or sampled at 1 in 10 seconds.

Retention Strategies​

Volume based retention follows the FIFO principle: data is only deleted when storage is full, making room for new records. This is different from time based retention, which deletes data after a fixed age regardless of whether storage is full. After an outage, a time based policy might delete data the system never had the chance to process. Volume based retention avoids that.

Example Applications​

Comparing ReductStore with Rosbag and MongoDB​

Three tools come up most often when robotics teams think about data storage: Rosbag and MCAP, MongoDB, and ReductStore. They're built for different things.

Rosbag and MCAP are the standard for recording ROS sessions. They're great for capturing a snapshot during a test run. But they're file formats, not databases. Querying across many recordings requires custom scripts, there's no built in content indexing, and managing thousands of bag files quickly becomes its own problem. Use them for short recordings and local playback, not long term storage or fleet wide analysis.

MongoDB is flexible and works well for structured metadata, labels, and event logs. But it wasn't built for large binary payloads. For blob data, it relies on GridFS, which adds complexity and hurts performance. It also uses time based retention, so data can be deleted during idle periods even when storage isn't full.

ReductStore is designed specifically for the data robots actually produce: large, binary, timestamped records at high frequency. It connects to ROS via reduct-bridge, exports to MCAP for Foxglove, plugs into Grafana, and replicates selectively to S3 or Azure.

In practice: use Rosbag or MCAP for short test recordings, MongoDB for structured metadata and event logs, and ReductStore for raw sensor data that needs to be stored at scale, queried efficiently, and managed over time.

For a deeper look: MongoDB vs ReductStore: Choosing the Right Database for Robotics Applications.

Hands-On Example: Storing Robotic Data in ReductStore​

Let's walk through a practical example of storing and querying robotic data with ReductStore. We'll use trajectory data (coordinates, speed, orientation) to keep things simple, but the same approach works for any sensor stream.

You'll need Python 3.8+ installed.

Setting Up ReductStore​

Create a folder and add a docker-compose.yaml file:

version: "3.8"

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    volumes:
      - data:/data
    environment:
      - RS_API_TOKEN=my-token

volumes:
  data:
    driver: local

Start it with:

docker compose up -d

ReductStore will be available at http://127.0.0.1:8383. Check the container is running with docker ps, then install the Python libraries:

pip install reduct-py numpy

Store and Query Data​

Create a Bucket​

async def create_trajectory_bucket():
 async with Client("http://localhost:8383", api_token="my-token") as client:
  settings = BucketSettings(
	 quota_type=QuotaType.FIFO,
	 quota_size=1000_000_000,
	)
  await client.create_bucket("trajectory_data", settings, exist_ok=True)

The bucket uses a FIFO quota of 1 GB. Old data is only deleted when the limit is reached.

Generate Trajectory Data​

async def generate_trajectory_data(frequency: int = 10, duration: int = 1):
 interval = 1 / frequency
 start_time = datetime.now()

  for i in range(frequency * duration):
    time_step = i * interval
    x = np.sin(2 * np.pi * time_step) + 0.2 * np.random.randn()
    y = np.cos(2 * np.pi * time_step) + 0.2 * np.random.randn()
    yaw = np.degrees(np.arctan2(y, x)) + np.random.uniform(-5, 5)
    speed = abs(np.sin(2 * np.pi * time_step)) + 0.1 * np.random.randn()
    timestamp = start_time + timedelta(seconds=time_step)

    yield {
      "timestamp": timestamp.isoformat(),
      "position": {"x": round(x, 2), "y": round(y, 2)},
      "orientation": {"yaw": round(yaw, 2)},
      "speed": round(speed, 2),
    }
    await asyncio.sleep(interval)

This simulates a robot moving in 2D at 10 Hz for 1 second. X and y are position coordinates, yaw is orientation, and speed is derived from position changes.

Calculate Metrics​

def calculate_trajectory_metrics(trajectory: list) -> tuple:
  positions = np.array([[point["position"]["x"], point["position"]["y"]] for   point in trajectory])
  speeds = np.array([point["speed"] for point in trajectory])

  deltas = np.diff(positions, axis=0)
  distances = np.sqrt(np.sum(deltas**2, axis=1))
  total_distance = np.sum(distances)

  average_speed = np.mean(speeds)

  return total_distance, average_speed

Write to ReductStore​

async def store_trajectory_data():
  trajectory_data = []
  async for data_point in generate_trajectory_data(frequency=10, duration=1):
    trajectory_data.append(data_point)

  total_distance, average_speed = calculate_trajectory_metrics(trajectory_data)

  labels = {
    "total_distance": total_distance,
    "average_speed": average_speed
  }

  packed_data = pack_trajectory_data(trajectory_data)

  timestamp = datetime.now()

  async with Client("http://localhost:8383", api_token="my-token") as client:
    bucket = await client.get_bucket("my-bucket")
    await bucket.write("trajectory", packed_data, timestamp, labels=labels)


def pack_trajectory_data(trajectory: list) -> bytes:
  """Pack trajectory data json format"""
  return json.dumps(trajectory).encode("utf-8")

The labels (total distance and average speed) are stored alongside each record. You can later filter or replicate records based on these values.

Once data is written, you'll see the bucket populate in ReductStore:

Query by Label​

async def query_by_label(bucket_name, entry_name, label_key, label_value):
  async with Client("http://localhost:8383", api_token="my-token") as client:
    bucket = await client.get_bucket(bucket_name)

    async for record in bucket.query(
      entry_name,
      when={
        label_key: {"$gt": label_value}
      },
    ):
      # Do something with the record

Remove the 'when' condition to return all records with no filtering.

Main Function​

async def main():
  await create_trajectory_bucket()
    await store_trajectory_data()
    label_query_result = await query_by_label("my-bucket", "trajectory", "&total_distance", HIGH_DISTANCE)
    if label_query_result:
      print(f"Data queried by label: {label_query_result}")

asyncio.run(main())

Conclusion​

Robotics data pipelines don't have to be a mess of bag files, overpriced cloud storage, and custom scripts duct taped together. ReductStore covers ingestion, retention, replication, and querying in one place, with direct integrations into ROS, Foxglove, Grafana, S3, and Azure.

Start with what you need, and add the rest as your system grows. Check out reduct.store or read through the documentation to get going.

If you have any questions or comments, feel free to use the ReductStore Community Forum.

ReductStore 1.19.0 is now available. This release extends the storage model for robotics and telemetry workloads and introduces new integration points for ROS and Zenoh.

To download the latest release, visit the Download Page.

What's new in 1.19.0?​

The first major change in v1.19 is licensing. ReductStore Core is now open source under Apache 2.0, which makes the core database easier to evaluate, integrate, and extend in production systems.

The second major change is the data model. ReductStore now supports hierarchical entry names, similar to ROS topics, and adds entry attachments for schemas and metadata. This makes it possible to represent structured robotics data without flattening topic hierarchies or moving context into external systems.

The release also introduces a native Zenoh API for direct ingestion and querying over Zenoh, and ReductBridge for ROS1 and ROS2 integration.

Nested Data Model with Attachments​

The new hierarchical data model lets you organize data in a path-based structure, similar to ROS topics, Zenoh key expressions, or MQTT topics. Instead of relying on a flat namespace, ReductStore can now store data in a form that matches the structure used by upstream systems.

Each entry can also include attachments for schemas and metadata. These attachments preserve the context required by downstream tooling without changing the record payload itself. For example, the ROS Extension can use them to decode serialized ROS messages and export them to MCAP files.

Native Zenoh API​

Zenoh is increasingly used in robotics and edge environments as a low-overhead protocol for distributed data exchange. With the native Zenoh API, ReductStore can participate directly in Zenoh-based systems without requiring an additional bridge or adapter.

You can start ReductStore with the Zenoh API enabled using a minimal Docker configuration:

docker pull reduct/store:v1.19.0
docker run --env "RS_ZENOH_ENABLED=ON" \
  --env "RS_ZENOH_CONFIG={}" \
  --env "RS_ZENOH_SUB_KEYEXPRS=**" \
  -p 8383:8383 -p 36597:36597 -p 7446:7446 \
  reduct/store:v1.19.0

Once enabled, the API allows you to write data to ReductStore directly over Zenoh. If a sample includes a JSON attachment, ReductStore stores it as record labels:

import json
import zenoh

KEY = "factory/line1/camera"
PAYLOAD = b"<binary payload>"
LABELS = {"robot": "alpha", "status": "ok"}

with zenoh.open(zenoh.Config()) as session:
    session.put(
        KEY,
        PAYLOAD,
        attachment=json.dumps(LABELS).encode(),
    )

You can also query data through Zenoh. For conditional queries, pass a when expression in the query attachment. If you want all matching records returned individually, use zenoh.ConsolidationMode.NONE:

import json
import zenoh

KEY = "factory/line1/when-query"
CONSOLIDATION = zenoh.ConsolidationMode.NONE
attachment = json.dumps({"when": {"&status": {"$eq": "ok"}}}).encode()

with zenoh.open(zenoh.Config()) as session:
    replies = [
        reply
        for reply in session.get(
            KEY,
            timeout=5.0,
            attachment=attachment,
            consolidation=CONSOLIDATION,
        )
        if reply.ok
    ]

    for reply in replies:
        print(reply.ok.payload.to_bytes())

Data ingested through Zenoh is stored in ReductStore like data written through the HTTP API, so it remains available for querying, replication, and downstream tools and extensions.

ReductBridge for ROS Integration​

This release also introduces ReductBridge, a new project for integrating ReductStore with ROS1 and ROS2. ReductBridge automatically labels ROS messages and stores related schemas and metadata as attachments.

This makes raw ROS payloads usable after ingestion: they can be decoded later with the ROS Extension or exported to MCAP files. ROS support is the first target, but the same pattern can be extended to operating system metrics, logs, and other telemetry to build a unified storage layer with consistent labeling and metadata.

What’s Next​

The next area of work is data compression and storage efficiency, especially for robotics workloads where payload sizes and formats vary significantly. We plan to introduce backend compression optimized for ReductStore's storage model, where many records are packed into a single block.

We also plan to store metadata in Parquet format. This should improve query efficiency, make metadata accessible without scanning the underlying data blocks, and simplify integration with analytics and data lake tooling.

Compatibility and Migration​

Starting with v1.19, ReductStore Docker images no longer run as root for security reasons. If you deploy with Docker, make sure the mounted data directory is writable by UID/GID 10001:10001 before upgrading.

If you have questions or feedback, join the ReductStore Community forum.

Thanks for using ReductStore!

Grafana is a powerful tool for visualizing time-series data, and it is widely used for monitoring and analysis. However, it does not natively understand robotics data formats, such as ROS 2 messages, since they are usually stored in binary formats (e.g., CDR). ReductStore's flexible query engine and extension system can bridge this gap. With the ReductROS extension, you can extract ROS 2 messages as JSON directly in Grafana queries. This enables you to build rich dashboards and alerts on your robotics data without preprocessing it into a different format.

In this tutorial, we will create a Grafana dashboard supported by the ReductStore data source plugin. Using the ReductROS extension, the dashboard will convert ROS 2 messages to JSON upon query, enabling us to visualize and analyze the data directly in Grafana.

Prerequisites​

This tutorial assumes that you have the following set up:

• A ReductStore instance with ROS 2 data (either your own or from the demo instance). Some robotics data is available on our demo ReductStore instance.
• A Grafana 10+ instance (local or cloud) with the ReductStore data source plugin installed. You can use our Docker image, or install Grafana separately and add the plugin manually, following the documentation.

Set up the ReductStore data source in Grafana​

Let's run Grafana locally using Docker and connect it to ReductStore:

docker run -d -p 3000:3000 reduct/grafana:latest

Open Grafana at http://localhost:3000 and log in with the default credentials, admin/admin. Then, go to the "Data Sources" panel. There, you can find the default ReductStore data source if you are using our Docker image. Otherwise, you can add it manually on the "Add new connection" page.

Once the data source has been added, configure it using the following parameters:

• Name: reductstore-datasource
• Server URL: https://play.reduct.store/replica (or your ReductStore instance URL)
• API Token: reductstore for the demo instance (or your API token)

Click "Save & Test" to verify the connection. You should see a success message confirming that Grafana can connect to ReductStore.

Query and extract ROS 2 data to JSON​

Now we can create a new dashboard and add a panel to query ROS 2 data from ReductStore. Go to Dashboards -> New Dashboard -> Add new panel. In the query editor, select the ReductStore data source and provide the following query parameters to extract a ROS 2 topic as JSON:

• Bucket: orion
• Entry: any entry, you can use wildcards (e.g., *) to query multiple entries
• Scope: LabelAndContent , which allows us to view both labels and content of the messages if it's JSON

when the parameters are set, we can read data with the following query:

{
  "$each_t": "$__interval",
  "#ext": {
    "ros": {
      "extract": {}
    }
  }
}

The magic happens in the #ext section, where we activate the ros extension and specify that we want to extract data. The extension reads the serialized ROS content for the specified bucket/entry, decodes the ROS 2 messages, and returns them as JSON records with timestamps and labels. You can read more about the query format in the ReductROS documentation.

Scale and performance​

Since your data may be large and data extraction can be computationally intensive, here are some tips to ensure good performance:

• Use Grafana $__interval with $each_t for interval-based queries
• Add label filters to narrow down the data (e.g., filter by topic name or message type)
• Start with a narrow time range and expand as needed

Optional: Visualize binary topics (images)​

If you're interested in visualizing binary data, such as images or point clouds, you might be interested in the ReductROS extension. The ReductROS extension can encode binary data as base64 strings or in JPEG format. This data can then be displayed in Grafana's "Text" panel with HTML rendering. For instance, if your topic contains raw images, you can specify the encoding in the query.

{
  "$each_t": "$__interval",
  "#ext": {
    "ros": {
      "extract": {
        "encode": { "data": "jpeg" }
      }
    }
  }
}

Conclusion​

Using ReductStore and the ReductROS extension allows you to easily extract and visualize ROS 2 data in Grafana without preprocessing it into a different format. This enables you to build rich dashboards and alerts on your robotics data, allowing for better monitoring and analysis of your robotic systems.

Importantly, this approach preserves your data's original format, enabling flexible querying without the need for additional pipelines or transformations. You can also combine it with other Grafana features, such as alerting and annotations, to gain additional insights from your robotics data.

I hope this tutorial gives you a good starting point for visualizing your ROS 2 data in Grafana using ReductStore. If you have questions or want to discuss which edition fits your use case, please reach out on the ReductStore Community forum or via our contact page.

Starting with ReductStore v1.19, we are changing how we license and package the project. From this version onward, ReductStore is split into two editions: ReductStore Core and ReductStore Pro.

Previously, ReductStore shipped as a single edition under BUSL-1.1 (Business Source License). With v1.19, the core database is open source under Apache 2.0, while Pro continues under commercial terms.

What's changing​

With this release, ReductStore Core becomes the open-source foundation of the project and is now available under the Apache License 2.0. It includes the database server and the core functionality that most users rely on for edge deployments and everyday data workflows.

Alongside Core, we will continue offering ReductStore Pro, which is distributed under a commercial license. The Pro edition provides additional capabilities and support options for teams with more advanced requirements. See Pricing for a clear comparison of what is included in each edition.

Why we're doing this​

We want to better support open source communities and make it easier to contribute to, integrate with, and build on top of ReductStore Core. Just as importantly, we want users to be able to run ReductStore on the edge without licensing restrictions for the core database use cases.

At the same time, we want to keep a sustainable commercial model for companies that build more complex cloud setups or need advanced support and functionality around robotics and IIoT data formats. That is the role of ReductStore Pro.

What this means for you​

If you use the core database functionality, you can freely adopt, integrate, and distribute ReductStore Core under the Apache 2.0 license. Existing workflows and upgrades remain straightforward.

For teams that rely on advanced functionality or require commercial support, ReductStore Pro remains available as the supported commercial edition. We will clearly document which features belong to Core and which are part of Pro in the documentation and release notes.

If you have questions or want to discuss which edition fits your use case, please reach out on the ReductStore Community forum or via our contact page.

Drones in air-gapped environments produce a lot of data (camera images, telemetry, logs, model outputs). Storing this data reliably on each drone and syncing it to a ground station later can be hard. ReductStore makes this easier: it's a lightweight, time-series object store that works offline and replicate data when a connection is available.

This guide explains a simple setup where each drone stores data locally with labels, replicates records to a ground station based on what it detects, and keeps a clear audit trail of what was captured and replicated.

What we'll cover:

• Drone-to-Ground Architecture
• Setting Up the Drone Node
• Storing Drone Data with Labels
• Setting Up Selective Replication
• Querying for Audit Reports
• Why This Setup Works Well for Drones

Drone-to-Ground Architecture​

The architecture has three main components:

• Each drone runs a small ReductStore server to save images and telemetry locally on disk (this lets the drone operate fully offline).
• A ground station runs a ReductStore instance that receives replicated data for analysis and archiving.
• ReductStore replication tasks copy data from drone to ground based on labels and conditions (e.g., only records flagged as anomalies, plus context around them).

Each drone pushes its data to the ground whenever it is connected. If the network disconnects, replication continues when the drone reconnects. This approach provides offline capability, lets you decide which data to replicate, and keeps a clear record of what happened.

Setting Up the Drone Node​

Start by running ReductStore on the drone's companion computer. Here is a minimal docker-compose.yml:

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    environment:
      RS_API_TOKEN: <DRONE_TOKEN>
      RS_BUCKET_1_NAME: mission-data
      RS_BUCKET_1_QUOTA_TYPE: FIFO
      RS_BUCKET_1_QUOTA_SIZE: 10000000000 # 10 GB
    volumes:
      user: "0:0" # Run as root with host bind mount in this example
      - ./data:/data

Run it with:

docker compose up -d

This starts a ReductStore server with a mission-data bucket that uses FIFO retention. Old data is deleted only when the 10 GB limit is reached, so the drone always keeps as much history as possible.

If you prefer Snap instead of Docker:

sudo snap install reductstore

That starts a ReductStore server on port 8383 by default. You can then create the bucket using the Reduct CLI:

reduct-cli alias add drone -L http://localhost:8383 -t "<DRONE_TOKEN>"
reduct-cli bucket create drone/mission-data --quota-type FIFO --quota-size 10GB

Storing Drone Data with Labels​

Use labels to tag every record with mission context. This is what makes selective replication and auditing possible later. Here is an example using the Python SDK:

import asyncio
import time
import hashlib
from reduct import Client


async def main():
    async with Client("http://localhost:8383", api_token="<DRONE_TOKEN>") as client:
        bucket = await client.get_bucket("mission-data")

        # Read a camera frame
        payload = open("frame.jpg", "rb").read()
        checksum = hashlib.sha256(payload).hexdigest()
        timestamp = int(time.time() * 1_000_000)  # microseconds

        # Write with mission labels
        await bucket.write(
            "camera",
            payload,
            timestamp=timestamp,
            labels={
                "mission_id": "m-2026-02-24-01",
                "platform_id": "uav-07",
                "anomaly": "false",
                "confidence": "0.95",
                "checksum": checksum,
            },
            content_type="image/jpeg",
        )

        # Write telemetry as a CSV batch
        csv_data = "ts,lat,lon,alt,speed\n"
        csv_data += "1708771200000000,47.3769,8.5417,450.2,12.5\n"
        csv_data += "1708771201000000,47.3770,8.5418,451.0,12.8\n"

        await bucket.write(
            "telemetry",
            csv_data.encode(),
            timestamp=timestamp,
            labels={
                "mission_id": "m-2026-02-24-01",
                "platform_id": "uav-07",
                "anomaly": "false",
                "checksum": hashlib.sha256(csv_data.encode()).hexdigest(),
            },
            content_type="text/csv",
        )


asyncio.run(main())

The anomaly label is important: it lets the replication task decide what to sync based on what the drone actually sees. For example, if the drone detects something unusual (an object, a warning, a low confidence score), it sets anomaly=true. The replication task can then automatically sync that record — plus the context around it.

The checksum label gives you a simple way to verify data integrity during audits.

Setting Up Selective Replication​

Once the drone connects to a trusted network, replication sends only the relevant records to the ground station. The simplest approach is to replicate based on a label, for example only records where the drone detected an anomaly:

reduct-cli alias add drone -L http://localhost:8383 -t "<DRONE_TOKEN>"

reduct-cli replica create drone/mission-to-ground \
    mission-data \
    https://<GROUND_TOKEN>@<ground-address>/drone-data \
    --when '{"&anomaly": {"$eq": "true"}}'

This creates a replication task that copies only records where anomaly=true from the drone's mission-data bucket to the ground station.

Replicating with context (before and after)​

In many cases, you don't just want the anomaly record itself — you also want to see what happened before it. ReductStore supports this with the #ctx_before and #ctx_after directives. For example, to replicate each anomaly record plus 30 seconds of data before it and 10 seconds after:

{
  "&anomaly": { "$eq": "true" },
  "#ctx_before": "30s",
  "#ctx_after": "10s"
}

This is powerful for drone operations: imagine the drone's onboard model detects an unexpected object. ReductStore will replicate that record and the 30 seconds of camera frames leading up to the detection, so the ground team can review what happened.

You can provision this directly in Docker using environment variables:

services:
  reductstore:
    image: reduct/store:latest
    ports:
      - "8383:8383"
    environment:
      RS_API_TOKEN: <DRONE_TOKEN>
      RS_BUCKET_1_NAME: mission-data
      RS_BUCKET_1_QUOTA_TYPE: FIFO
      RS_BUCKET_1_QUOTA_SIZE: 10000000000
      RS_REPLICATION_1_NAME: mission-to-ground
      RS_REPLICATION_1_SRC_BUCKET: mission-data
      RS_REPLICATION_1_DST_BUCKET: drone-data
      RS_REPLICATION_1_DST_HOST: https://<ground-address>
      RS_REPLICATION_1_DST_TOKEN: <GROUND_TOKEN>
      RS_REPLICATION_1_WHEN: |
        {
          "&anomaly": { "$$eq": "true" },
          "#ctx_before": "30s",
          "#ctx_after": "10s"
        }
    volumes:
      user: "0:0" # Run as root with host bind mount in this example
      - ./data:/data

Querying for Audit Reports​

After a mission, you can query the ground station to check what was captured and replicated. Here is a simple example that lists all records from a specific mission:

import asyncio
from reduct import Client


async def main():
    async with Client("https://<ground-address>", api_token="<GROUND_TOKEN>") as client:
        bucket = await client.get_bucket("drone-data")

        # Query all camera records from a specific mission
        async for record in bucket.query(
            "camera",
            when={"&mission_id": {"$eq": "m-2026-02-24-01"}},
        ):
            print(
                f"ts={record.timestamp}, "
                f"anomaly={record.labels.get('anomaly')}, "
                f"checksum={record.labels.get('checksum')}, "
                f"size={record.size}"
            )


asyncio.run(main())

This gives you a clear log of every record in that mission: timestamp, anomaly flag, checksum, and size. You can use this to verify that all expected data arrived on the ground side.

To go further, compare the checksums on the drone with the ground side to confirm nothing was altered during transfer. You can also check the error logs of the replication task to see if any records failed to replicate.

Why This Setup Works Well for Drones​

Drones have specific constraints that general purpose databases don't handle well. Here is what makes this setup practical:

• Full offline operation. Drones store everything locally and don't need a network connection during the mission. Data is safe on disk until sync happens.
• Automatic sync when connected. When the drone lands or connects to a trusted network, replication picks up where it left off. No manual file transfers, no rsync scripts, no USB sticks.
• Smart replication with context. You don't have to sync everything. The replication task filters by labels and can include past records around each event using #ctx_before. The ground team gets exactly what they need to understand what happened.
• Disk never fills up unexpectedly. FIFO retention removes the oldest data only when the disk is full. The drone always keeps as much history as possible without running out of space mid mission.
• Easy auditing. Every record has a timestamp, labels, and a checksum. After a mission, you can query the ground station and verify exactly what was captured and what was synced.
• Store any file type. Camera frames, telemetry CSV, logs, MCAP files, model outputs. Everything goes into the same system with the same interface.

Next Steps​

If you want to go deeper, check out these articles:

• Distributed Storage in Mobile Robotics for a similar setup with mobile robots and S3 cloud backend
• How to Store and Manage Robotics Data for a broader look at ReductStore features for robotics
• Data Replication Guide for the full documentation on replication tasks, filters, and modes
• Conditional Query Reference for all available conditional query operators you can use in replication filters and queries

I hope you found this article helpful! If you have any questions or feedback, don't hesitate to reach out on our ReductStore Community forum.

We are pleased to announce the release of the latest minor version of ReductStore, 1.18.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.18.0?​

In this release, we have added support for resilient deployments to build a more robust, fault-tolerant, and highly available ReductStore cluster. Now, you can implement hot-standby configurations, automatic failover, and seamless recovery to ensure uninterrupted service even in the face of hardware failures or network issues. You can also elastically scale read-only nodes to handle increased read workloads without impacting the performance of the primary nodes.

Additionally, we have introduced a new Multi-entry API that allows you to efficiently manage and query multiple entries in a single request. This API is designed to optimize performance and reduce latency when working with large datasets, making it easier to retrieve and manipulate data in bulk.

Resilient Deployments​

In ReductStore v1.18.0, resilient deployments are now a first-class feature. Using the RS_INSTANCE_ROLE setting, you can build topologies that keep your ingestion endpoint available during node failures and scale reads independently from writes.

Hot standby (active-passive) for write availability​

Run two nodes against the same backend (a shared filesystem or the same remote backend). Only one node is active at a time: the active node holds a lock file and refreshes it, while the standby waits and takes over when the lock becomes stale.

• Set RS_INSTANCE_ROLE=PRIMARY for the active node and RS_INSTANCE_ROLE=SECONDARY for the standby.
• Put both nodes behind a single virtual endpoint (load balancer / reverse proxy).
• Route traffic only to the node that returns 200 OK on GET /api/v1/ready (the inactive node returns 503).
• Tune failover behavior with RS_LOCK_FILE_TTL (how long the standby waits) and RS_LOCK_FILE_TIMEOUT (how long a node waits to acquire the lock).

To avoid split-brain writes, don’t run both nodes in STANDALONE mode against the same dataset.

Read-only replicas for read scaling​

Add one or more REPLICA nodes to serve queries from the same dataset. Replicas never write and periodically refresh bucket metadata and indexes from the backend, so newly written data may appear with a small delay.

• Route writes to a dedicated ingestion node (or the active node in a hot-standby pair).
• Route reads to replicas to scale query workloads horizontally.
• Tune staleness with RS_ENGINE_REPLICA_UPDATE_INTERVAL.

For an end-to-end walkthrough (including S3-based standalone, active-passive, and replicas), see the S3 Backend tutorial. For architecture options and operational notes, see the Disaster Recovery guide.

Multi-entry API​

The new Multi-entry API makes it possible to work with multiple entries in a single request. In practice, this is most useful for querying: instead of running one query per sensor/stream and merging results on the client, you can request all the entries you need at once and process a single result stream (each returned record includes its entry name).

Here is a Python example using reduct-py to query multiple entries in one call:

import asyncio

from reduct import Client


async def main() -> None:
    async with Client("http://localhost:8383", api_token="my-token") as client:
        bucket = await client.get_bucket("my-bucket")

        # Query multiple entries in a single request (since ReductStore v1.18).
        # You can mix exact names and wildcards.
        entries = ["sensor-*", "camera"]
        async for record in bucket.query(
            entries,
            start="2026-02-05T10:00:00Z",
            stop="2026-02-05T10:05:00Z",
            when={"&score": {"$gte": 10}},
        ):
            payload = await record.read_all()
            print(record.entry, record.timestamp, len(payload))


if __name__ == "__main__":
    asyncio.run(main())

What’s Next​

We’re already working on the next improvements to make ReductStore easier to integrate into real-world data pipelines:

Native Zenoh API​

Zenoh is becoming a common choice for data exchange in distributed, edge-first systems (robotics, industrial IoT, and telemetry). In upcoming releases, we plan to add a native Zenoh API so ReductStore can join Zenoh networks seamlessly.

This will make it easier to ingest and serve data directly through Zenoh—without custom bridges—so your storage layer fits naturally into existing Zenoh-based deployments.

Entry attachments (metadata)​

Today, labels work well for filtering and replication, but many projects also need structured metadata tied to an entry itself: data format, schema version, units, encoding, calibration details, and other context.

We plan to introduce attachments for entries, allowing you to store and retrieve this kind of metadata alongside your data streams, making datasets more self-describing and easier to consume across teams and tools.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

Modern robots collect a lot of data from sensors, cameras, logs, and system outputs. Managing this data well is important for debugging, performance tracking, and training machine learning models.

Over the past few years, we've been building a storage system from scratch. As part of that work, we spoke with many robotics teams across different industries to understand their challenges with data management.

Here's what we heard often:

• Only a subset of what robots generate is actually useful
• Network connections are not always stable or fast
• On-device storage is limited (hard drive swaps is not practical)
• Teams rely on manual workflows with scripts and raw files
• It's hard to find and extract the right data later
• ROS bag files get large quickly and are difficult to manage

In this article, we compare four tools built to handle robotics data: ReductStore, Foxglove, Rerun, and Heex. We look at how they work, what they're good at, and which use cases they support.

If you're working with robots and need to organize, stream, or store data more effectively, this overview should help.

Key Criteria for Comparison​

When picking a data tool for robotics, focus on these areas:

• Data Types Robotics is a large field with many sensor types. The tool should support the data you work with, such as:
  • Telemetry: Lightweight (GPS, IMU, joints), ideal for monitoring.
  • Downsampled Data: Lower-rate images or lidar for incident review without high storage cost.
  • Full-Resolution: Raw sensor outputs for deep debugging or training. This is storage-intensive but essential for some applications.
• Integration The tool should work with what you already use, like ROS, Grafana, MQTT, cloud platforms (S3, Azure, Google Cloud), and your development environment to avoid extra glue code and simplify workflows.
• Performance and Scalability Data must move quickly (both locally and to the cloud). Large files or slow queries can block robots or delay analysis.
• Ease of Use and APIs A simple UI and solid API support make it easier to automate, scale, and adapt the tool to different use cases.

Tool Overviews​

ReductStore​

ReductStore is a storage and streaming system designed for robotics data. It works both on the robot and in central storage (on-premise/self-hosted or in the cloud) with the same interface and SDKs (in Python, C++, Go, Javascript/TypeScript or Rust). That means your code stays the same whether you're reading local, remote data (or creating a browser-based dashboard).

To move data to the cloud, ReductStore uses conditional replication. You can define rules to upload only certain records: by label, rules, or event. For example, replicate all incident data, or just 1 out of 10 entries for routine monitoring.

ReductStore handles storage limits on edge devices with FIFO retention. Old data is deleted only when the device is full. Each bucket can have different rules, so you can keep more images and less telemetry, for example.

With an S3 backend, ReductStore batches small records together before uploading. This cuts down the number of requests and lowers cloud storage costs. For observability, you can connect Grafana to ReductStore to create dashboards with system metrics and sensor data. For MCAP files, ReductStore supports shareable query links that open directly in Foxglove v1/v2.

It also lets you filter or merge records server-side. For example, you can pull all temperature readings above a threshold over a time range without downloading full datasets.

Want more technical detail? Check out The Missing Database for Robotics Is Out.

Foxglove and MCAP​

Foxglove is a browser-based visualization and observability tool for robotics. It supports ROS 1, ROS 2, and MCAP logs, and handles data types like telemetry, camera feeds, lidar, and depth maps.

It uses MCAP, an open-source log format built for robotics, to store high-resolution data efficiently. You can explore MCAP files interactively in Foxglove Studio or stream them programmatically.

Foxglove provides an agent that detects new MCAP files on the robot and uploads them to the cloud automatically. This requires robots to record short rosbag segments (typically a few minutes each) which are closed and rotated continuously.

It integrates natively with ROS topics, services, and actions, and offers WebSocket and REST APIs. It also connects to major cloud providers like AWS, Azure, and Google Cloud for scalable storage.

The interface is built for time-series and sensor data, with interactive 2D/3D views, plots, and drag-and-drop panels for quick setup and review.

Rerun​

Rerun is an open-source visualization solution for time-series and multimodal data. It supports data types like images, point clouds, lidar, depth maps, tensors, and other sensor streams.

Its main strength is combining flexible logging with a fast, built-in 3D viewer designed for robotics and extended reality (XR) applications. For large datasets, Rerun provides a column-oriented API to speed up ingestion and reduce memory usage. It also uses efficient internal structures to minimize allocations and optimize performance on edge devices.

Rerun doesn't offer native ROS integration yet, but it can be used in ROS projects by adding custom logging to nodes.

You can embed Rerun in Jupyter notebooks or web pages, and use loggers for Python, Rust, and C++ to stream data into the viewer.

The UI is built for real-time 3D exploration, with overlays and live tracking that make it easy to inspect different data types in the same visual space.

Heex​

Heex is a data capture and review platform for autonomous systems that focuses on collecting only key moments—like errors or specific events instead of logging everything. This reduces bandwidth and storage needs while keeping important context.

Robots using Heex record data continuously in short ROSbag segments. A small agent on the robot watches for triggers and uploads only selected segments to the cloud based on rules.

A core feature is RDA (Resource and Data Automation) for ROS 2, which automates what to record and when. Rules can be changed remotely without restarting the robot.

Data is stored in ROSbag and can be reviewed directly in the Heex dashboard, which includes built-in open-source version of Foxglove. This setup makes it easy to manage data across fleets and locations.

Heex supports both ROS 1 and ROS 2, and integrates with other systems through SDKs, APIs, and a CLI.

The interface includes customizable dashboards to monitor sensor data, errors, and system status. Timelines and streams are easy to navigate for quick analysis.

Comparative Analysis Table​

To help visualize the differences between the tools, here is a comparison table summarizing their main characteristics:

Conclusion​

Each tool addresses a different part of the robotics data workflow. ReductStore is ideal for distributed storage across many robots, with selective replication to the cloud and flexible integration with tools like Grafana and Foxglove. Foxglove excels at visualizing MCAP logs and ROS topics. Rerun offers flexible, real-time 3D inspection for custom applications. Heex focuses on capturing just the important moments for efficient fleet analysis.

Choosing the right tool depends on what kind of data you collect, how you process it, and where you need it to go. In many cases, combining tools can give you the best of all worlds.

Thanks for reading. I hope this article helps you decide on the right storage strategy for your vibration data. If you have questions or comments, feel free to visit the ReductStore Community Forum.

Mobile robots produce a lot of data (camera images, IMU readings, logs, etc). Storing this data reliably on each robot and syncing it to the cloud can be hard. ReductStore makes this easier: it's a lightweight, time-series object store built for robotics and industrial IoT. It stores binary blobs (images, logs, CSV sensor data, MCAP, JSON) with timestamps and labels so you can quickly find and query them later.

This introduction guide explains a simple setup where each robot stores data locally and automatically syncs it to a cloud ReductStore instance backed by Amazon S3.

Edge-to-Cloud Architecture​

The architecture has three main components:

• Each robot runs a small ReductStore server in order to save images and IMU data locally on disk (this let the robot operate offline).
• A cloud ReductStore instance runs on a server (e.g., EC2) and uses S3 for long-term storage.
• ReductStore replication tasks copies data from robot to cloud based on labels, events, or rules (e.g., 1 record every minute).

Each robot pushes its data to the cloud whenever it is connected to the network. This approach provides the robots with offline capability, allows you to decide which data to replicate, and easily scales to support many robots.

How Replication Works​

ReductStore uses an append-only replication model:

• The robot stores new data locally.
• ReductStore automatically detects new records.
• It sends them to the cloud in batches (or streams large files).
• If the network disconnects, replication continues when the robot reconnects.

You can replicate:

• everything
• or only specific sensors
• or only records with certain labels
• or based on rules (e.g., 1 record every S seconds or every N records)

This can be configured per robot using environment variables (provisioning), with the web console or via the CLI (as shown in this guide).

Cloud ReductStore With S3 Backend​

ReductStore supports storing all records directly in S3. It keeps a local cache for fast access and batches many small blobs into larger blocks to save on S3 costs.

Here is an example docker-compose.yml to run a ReductStore server that uses S3 as the remote backend and provisions buckets for robots:

services:
  reductstore:
    image: reduct/store:latest
    container_name: reductstore
    ports:
      - "8383:8383"
    environment:
      # AWS credentials and S3 bucket configuration
      RS_REMOTE_BACKEND_TYPE: s3
      RS_REMOTE_BUCKET: <YOUR_S3_BUCKET_NAME>
      RS_REMOTE_REGION: <YOUR_S3_REGION>
      RS_REMOTE_ACCESS_KEY: <YOUR_AWS_ACCESS_KEY_ID>
      RS_REMOTE_SECRET_KEY: <YOUR_AWS_SECRET_ACCESS_KEY>
      RS_REMOTE_CACHE_PATH: /data/cache
      # Bucket provisioning
      RS_BUCKET_ROBOT_1_NAME: robot1-data
      RS_BUCKET_ROBOT_2_NAME: robot2-data
      # .. additional buckets as needed
    volumes:
      user: "0:0" # Run as root with host bind mount in this example
      - ./cache:/data/cache

Run it with:

docker compose up -d

This starts a ReductStore server that writes to S3 automatically. There are many more configuration options available in the configuration documentation.

Setting Up Replication​

First spin up a local ReductStore on each robot. Here with Snap:

sudo snap install reductstore

That starts a ReductStore server on port 8383 by default. Then you can use the Reduct CLI to set up replication from the robot to the cloud instance:

# Point the CLI to the robot's local ReductStore
reduct-cli alias add local -L http://localhost:8383 -t "<ROBOT_API_TOKEN>"

# Create a bucket for that robot
reduct-cli bucket create local/robot1-data

# Create a replication task to the cloud
reduct-cli replica create local/robot1-to-cloud \
    robot1-data \
    https://<CLOUD_API_TOKEN>@<cloud-address>/robot1-data

This creates a replication task called robot1-to-cloud that copies all data from the robot's local robot1-data bucket to the cloud instance. You can customize replication further by adding filters or rules. See the replication guide for more details.

Storing Sensor Data​

There are many ways to store data. When it comes to high-frequency sensor data like IMU readings, a common approach is to store them in 1-second files. Images can be stored as binary blobs (e.g., JPEG or PNG files). Here is an example of storing IMU data as CSV files and images as binary blobs using the Python SDK (this stores 10,000 samples and one camera image for a given timestamp as an example):

import asyncio
import time
import random
from reduct import Client


async def main():
    async with Client("http://localhost:8383", api_token="<ROBOT_API_TOKEN>") as client:
        bucket = await client.get_bucket("robot1-data")

        # Current timestamp to index the data by time in ReductStore
        timestamp = int(time.time() * 1_000_000)  # microseconds

        # Generate 10'000 IMU samples
        rows = []
        for i in range(10_000):
            rows.append(
                {
                    "ts": timestamp + i * 100,  # microseconds
                    "linear_acceleration_x": round(random.uniform(-2, 2), 3),
                    "linear_acceleration_y": round(random.uniform(-2, 2), 3),
                    "linear_acceleration_z": round(random.uniform(8.0, 10.0), 3),
                }
            )

        # Convert to CSV (store 1 seconds of data per file)
        csv = (
            "ts,linear_acceleration_x,linear_acceleration_y,linear_acceleration_z\n"
            + "\n".join(
                f"{r['ts']},{r['linear_acceleration_x']},"
                f"{r['linear_acceleration_y']},{r['linear_acceleration_z']}"
                for r in rows
            )
        )

        # Write the IMU batch
        await bucket.write(
            entry_name="imu_logs",
            data=csv.encode(),
            timestamp=timestamp,
            labels={"sensor": "imu", "rows": "1000"},
            content_type="text/csv",  # MIME type
        )

        # Write one camera image
        with open("camera_image.png", "rb") as img:
            await bucket.write(
                entry_name="images",
                data=img.read(),
                timestamp=timestamp,
                labels={"sensor": "camera"},
                content_type="image/png",
            )


asyncio.run(main())

Querying Sensor Data Using ReductSelect​

If your IMU data is stored as CSV, the ReductSelect extension lets you:

• extract only certain columns
• filter rows based on conditions

Example: filter CSV rows where acc_x > 10:

{
    "#ext": {
        "select": {
            "csv": {"has_headers": True},
            "columns": [
                {"name": "ts", "as_label": "ts_ns"},
                {"name": "linear_acceleration_x", "as_label": "acc_x"},
                {"name": "linear_acceleration_y"},
                {"name": "linear_acceleration_z"},
            ],
        },
        "when": {"@acc_x": {"$gt": 1.9}},
    }
}

Python example:

import asyncio
from reduct import Client

when = { ... } # the JSON condition from above

async def main():
    async with Client("https://<cloud-address>", api_token="<TOKEN>") as client:
        bucket = await client.get_bucket("robot1-data")

        async for rec in bucket.query("imu_logs", when=when):
            data = await rec.read_all()
            print(data.decode())

asyncio.run(main())

This returns only the rows where linear_acceleration_x > 1.9, along with the timestamp.

Why This Setup Works Well for Robotics​

There are several advantages to using a specialized storage solution like ReductStore for mobile robotics:

• Robots can store data locally and operate offline without network connectivity.
• Automatic replication when connected to avoid manual uploads and simplify data management.
• Selective replication lets you control what data is sent to the cloud (i.e. decide on your reduction strategy) to save bandwidth and storage.
• Labels and timestamps make it easy to organize and query sensor data later.
• Store files of any type (images, CSV, logs, MCAP) in a single system without needing separate storage solutions for each data type.

Next Steps​

ReductStore also integrates into robotics observability stacks such as the Canonical Observability Stack (COS) for robotics. You can visualize sensor data, logs, and metrics in Grafana dashboards alongside your other robot telemetry. More details in our blog post The Missing Database for Robotics Is Out.

I hope you found this article helpful! If you have any questions or feedback, don't hesitate to reach out on our ReductStore Community forum.

Robotics teams today wrestle with data that grows faster than their infrastructure. Every robot generates streams of images, sensor readings, logs, and events in different formats. These data piles are fragmented, expensive to move, and slow to analyze. Teams often rely on generic cloud tools that are not built for robotics. They charge way too much per gigabyte (when it should cost little per terabyte), hide the raw data behind proprietary APIs, and make it hard for robots (and developers) to access or use their own data.

ReductStore introduces a new category: a database purpose built for robotics data pipelines. It is open, efficient, and developer friendly. It lets teams store, query, and manage any time series of unstructured data directly from robots to the cloud.

What Makes It a New Category​

ReductStore treats robotics with the respect it deserves. It captures everything in its raw form and stores it with a time index and labels for flexible querying and management. It ingests and streams any type of data (images, sensor frames, logs, MCAP files, CSVs, JSON, etc) without forcing developers to convert or reformat it.

It works on robots and in the cloud using the same interface and SDKs (Python, C++, Rust, Javascript, Go). This means developers can build data pipelines that run the same way on robots or in the cloud without needing to change code or learn new tools.

Developers can run ReductStore on an edge device for local data capture and replicate to a cloud instance (with S3 backend) for cloud analytics or archiving.

It is the first and only database designed specifically for unstructured, time series robotics data.

Data Handling and Querying​

Developers can work directly with data using simple queries and SDKs. The focus is speed and flexibility.

1. MCAP topic filtering​

You can filter topics directly from multiple MCAP files stored in ReductStore without needing to download and reprocess everything locally.

import json
import pandas as pd
from reduct import Client

# Extract only the IMU topic from MCAP files
ext = {
    "ros": {"extract": {"topic": "/imu/data"}},
}

async with Client("https://test.reduct.store") as client:
    bucket = await client.get_bucket("my-robotics-data")
    parts = []
    async for rec in bucket.query("mcap-entry", ext=ext):
        blob = await rec.read_all()
        data = json.loads(blob.decode("utf-8"))
        rows = [
            {
                "ts": data["header"]["stamp"]["sec"] * 1_000_000_000
                + data["header"]["stamp"]["nanosec"],
                "linear_acceleration_x": data["linear_acceleration"]["x"],
                "linear_acceleration_y": data["linear_acceleration"]["y"],
                "linear_acceleration_z": data["linear_acceleration"]["z"],
            }
        ]

This allows you to extract only the relevant topics from multiple bags. In this example, we extract only the IMU topic as a stream of JSON records, which would look like this:

2. CSV/JSON field extraction and filtering​

You can extract specific JSON fields or CSV columns when querying data. This lets you select only the information you need, for example, filtering and visualizing certain fields from streams of JSON or CSV sensor readings.

import io
import pandas as pd
from reduct import Client

# Select specific CSV columns and filter rows
ext = {
    "select": {
        "csv": {"has_headers": True},
        # Use "json": {}, for JSON data
        "columns": [
            {"name": "ts"},
            {"name": "linear_acceleration_x", "as_label": "acc_x"},
            {"name": "linear_acceleration_y"},
            {"name": "linear_acceleration_z"},
        ],
    },
    "when": {"$gt": [{"$abs": ["@acc_x"]}, 10]},
}

async with Client("https://test.reduct.store") as client:
    bucket = await client.get_bucket("my-robotics-data")

    # Loop over filtered CSV entries
    async for rec in bucket.query("csv_sensor_readings", ext=ext):
        blob = await rec.read_all()
        csv_data = pd.read_csv(io.BytesIO(blob))

The tabular result will only include the selected columns and rows that match the filter abs(linear_acceleration_x) > 10:

3. Query any type of data​

ReductStore automatically batches small records and streams large ones for efficient storage and access. You can query any type of data, from lightweight telemetry to high-resolution images or point clouds, efficiently.

import io
from PIL import Image
from reduct import Client

# Every 5 seconds, limit to 5 records
when = {"$each_t": "5s", "$limit": 5}

async with Client("https://test.reduct.store") as client:
    bucket = await client.get_bucket("my-robotics-data")
    async for rec in bucket.query("camera_frames", when=when):
        blob = await rec.read_all()
        img = Image.open(io.BytesIO(blob))

The example above retrieves camera frames at 5-second intervals. You can then process or visualize these images as needed.

4. Browse petabytes of data​

ReductStore is designed to handle massive volumes of data. Its indexing and storage architecture allows you to efficiently browse data at scale without downloading everything locally.

For example, you can quickly navigate records and preview your data directly in the ReductStore web console, even when working with petabytes of robotics data.

Cloud Integration and Cost Savings​

ReductStore connects robots and the cloud in a simple and flexible way. It works with S3-compatible storage and includes a robust replication system to transfer data from robots to the cloud (even when the network is unstable or intermittent), making it perfect for field robots that often go offline.

Replication tasks can be configured to replicate only specific data based on labels or any criteria (for example, only replicate data when the confidence score is below a threshold, or replicate everything from a 10-minute window around a specific event).

In the cloud, by batching multiple records into single data blocks, ReductStore minimizes both the number of blobs and the number of API calls to S3. This design reduces storage and retrieval costs by leveraging S3's pricing model.

This approach can deliver major savings when working with large volumes of robotics data.

Observability Stack Integration​

ReductStore works with the tools robotics engineers already trust.

Foxglove Studio​

Foxglove is an amazing tool for visualizing robotics data and debugging robots for the MCAP format.

To share data from ReductStore to Foxglove, you can use the ReductStore web console (or the SDKs) to generate a query link that Foxglove can open directly.

You can then paste the query link into Foxglove Studio to visualize the data.

Grafana​

Grafana is a popular open-source tool for creating dashboards and visualising time-series data. You can connect Grafana to ReductStore using the ReductStore data source plugin, which allows you to query and visualise data stored in ReductStore.

You can query data using labels, for example, localization coordinates, object detected, confidence score, etc:

Or you can query based on content, such as JSON files with sensor readings or other structured data:

Canonical Observability Stack (COS)​

Canonical's COS (Canonical Observability Stack) for robotics is an end to end observability framework built on open source tools such as Prometheus, Loki, Grafana, and Foxglove.

The missing piece in this stack has always been a purpose built system for storing and managing robotics data efficiently from robot to cloud.

ReductStore closes that gap. It provides a data storage and streaming solution optimized for both edge and cloud environments, along with an agent that captures data directly from ROS and streams it into the observability pipeline.

Closing Thoughts​

Robotics teams no longer need to choose between control and convenience. ReductStore gives full ownership of data from robot to cloud. It removes vendor lock, cuts cost, and keeps everything observable and connected. It is the new foundation for robotics data infrastructure (the missing database for robotics).

If you are interested to compare ReductStore with other databases (like MongoDB or InfluxDB), you can read our white paper that goes deeper into the architecture and design choices.

I hope you found this article helpful! If you have any questions or feedback, don't hesitate to reach out on our ReductStore Community forum.

We are pleased to announce the release of the latest minor version of ReductStore, 1.17.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.17.0?​

This release includes several new features and enhancements. First, there are query links for simplified data access. Second, there is support for S3-compatible storage backends.

These new features enhance the usability and flexibility of ReductStore for various use cases in the cloud and on-premises environments and make it easier to share and access data stored in the database.

🔗 Query Links for Data Access​

ReductStore now supports query links, enabling users to generate temporary, public URLs for specific data records — without requiring authentication. This makes it easier to share datasets with external collaborators, embed links into dashboards, or integrate with third-party systems that need read-only access to specific data.

You can create query links directly from the Web Console (or any SDKs):

1. Open the Data Browser page and select a record you want to share.
2. Click the “Share record” icon in the action panel.
3. Configure an expiration time to automatically revoke access after a defined period.

Once generated, anyone with the link can access the selected record via a simple HTTP(S) request — no access token required. The link only has access to the specific query for which it was created, along with the creator's permissions. This provides a secure and convenient way to expose selected data for collaboration and analysis.

☁️ S3-Compatible Storage Backend​

ReductStore now supports S3-compatible storage backends, allowing you to use object storage instead of a local file system for your underlying data. This update brings greater flexibility and scalability for managing large datasets in the cloud.

Previously, ReductStore supported only local disk storage, and users had to mount S3 buckets as local disks via FUSE drivers. With this release, ReductStore can now natively integrate with S3-compatible backends — no additional software or mounting is required.

This feature is designed with performance and cost optimization in mind. ReductStore uses a local disk cache layer to speed up read and write operations, while batching multiple records into a single data block to reduce storage and retrieval costs. This approach works especially well with cost-efficient AWS S3 storage classes such as S3 Standard-IA or S3 Glacier.

To run ReductStore with an S3-compatible backend, use the following environment variables:

mkdir -p ./data
sudo chown -R 10001:10001 ./data
docker run -p 8383:8383 \
    -e RS_REMOTE_BACKEND_TYPE=s3 \
    -e RS_REMOTE_BUCKET=<YOUR_S3_BUCKET_NAME> \
    -e RS_REMOTE_REGION=<YOUR_S3_REGION> \
    -e RS_REMOTE_ACCESS_KEY=<YOUR_S3_ACCESS_KEY_ID> \
    -e RS_REMOTE_SECRET_KEY=<YOUR_S3_SECRET_ACCESS_KEY> \
    -e RS_REMOTE_CACHE_PATH=/data/cache \
    -e RS_LICENSE_PATH=<PATH_TO_YOUR_LICENSE_FILE> \
    -v ${PWD}/data:/data/cache \
    reduct/store:latest

Read more about configuring S3-compatible storage backend in the documentation

What’s Next​

We’re continuing to develop new features to make ReductStore even more powerful and user-friendly. Here’s a preview of what’s coming in the next releases:

📦 Multiple Entries in a Single Request​

Currently, each write or query request must target a single entry. This can be limiting when dealing with multiple entries or dynamic lists of entries in your applications.

In upcoming versions, ReductStore will support batch operations across multiple entries within a single API request. This improvement will simplify integrations and reduce overhead for large-scale data ingestion and querying workflows.

🔒 Read-Only Mode for ReductStore​

Like most databases, ReductStore currently requires exclusive access to its data directory while running. As a result, running multiple instances on the same dataset—for load balancing or high availability—is not yet possible.

To address this, we’re introducing a read-only mode that will allow one writer instance* and multiple reader instances to access the same dataset concurrently. This approach will enable scalable read operations and improved availability without adding the complexity of clustering or replication mechanisms.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

This tutorial walks you through the reference implementation, explains the architecture, and shows production-grade NGINX snippets you can adapt.

What We’ll Build​

We’ll set up a ReductStore cluster with NGINX as a reverse proxy, separating the ingress and egress layers. This architecture allows for independent scaling of write and read workloads, ensuring high availability and performance.

Ingress layer​

The ingress layer handles all writes and replicates data to the egress layer. Its nodes may have limited storage capacity, while they need only to handle writes and replicate data to the egress nodes. It can use high-rate storage like NVMe SSDs or even RAM disks, depending on your data volume.

Egress layer​

The egress layer handles all reads and serves data to clients. Its nodes are optimized for read performance and can use larger, slower storage like HDDs or cloud object storage. Each egress node holds a complete copy of the dataset, allowing for high availability and load balancing.

NGINX Load Balancer​

The NGINX load balancer sits in front of both layers, exposing two stable endpoints:

• http://<host>/ingress → load balances writes across ingress nodes
• http://<host>/egress → load balances reads across egress nodes

This separation allows you to scale each layer independently and ensures that writes and reads are handled optimally.

It is also important to note that NGINX must maintain session affinity (stickiness) for both ingress and egress requests to ensure that queries remain consistent and throughput is maximized.

Quick Start​

Clone the example and bring it up:

git clone https://github.com/reductstore/nginx-resilient-setup
cd nginx-resilient-setup
docker compose up -d

This will start two ingress nodes and two egress nodes with NGINX in front, all configured to replicate data between them. Check the docker compose file for details on how the nodes are set up.

Now we need to write some data and verify that we can read it back. Install the reduct-cli tool if you haven't already, then run the following commands to set up aliases for the ingress and egress endpoints:

reduct-cli alias add ingress -L http://localhost:80/ingress --token secret
reduct-cli alias add egress -L http://localhost:80/egress --token secret

Then copy some data from our Demo Server to the ingress layer and read it back from the egress layer:

# Add demo server alias to the CLI
reduct-cli alias add play -L https://play.reduct.store --token reductstore

# Copy data from the demo server to ingress
reduct-cli cp play/datasets ingress/bucket-1 --limit 1000

# Read/export via egress
reduct-cli cp egress/bucket-1 ./export_folder --limit 1000

NGINX Configuration​

Below is a distilled config you can adapt for open-source NGINX:

# Upstreams
# Separate pools for ingress (writes) and egress (reads)
upstream reduct_ingress {
    ip_hash;   # stickiness for writes
    server ingress-1:8383 max_fails=3 fail_timeout=10s;
    server ingress-2:8383 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

upstream reduct_egress {
    ip_hash;   # stickiness for queries
    server egress-1:8383 max_fails=3 fail_timeout=10s;
    server egress-2:8383 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

server {
    listen 80;
    server_name _;

    client_max_body_size 512m;
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;

    location /ingress/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://reduct_ingress/;
    }

    location /egress/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://reduct_egress/;
    }
}

In the config above, we define two upstream blocks: reduct_ingress for handling write requests and reduct_egress for handling read requests. Each block uses ip_hash to ensure session affinity, which is crucial for maintaining consistent writes and reads.

ReductStore Configuration Notes​

The configuration between nodes of each layer is identical. To reach the desired architecture, you need to provision buckets and replication tasks for ingress nodes and buckets only for egress nodes. See the configuration files in the example repo for details.

Failure Drills​

When the setup is running, you can simulate failures to see how it behaves:

1. Kill an ingress node → writes continue via other ingress nodes.
2. Kill an egress node → reads continue via other egress nodes; replication resyncs when it’s back.
3. Simulate total ingress outage → analysis continues on egress; for true ingestion continuity, pair with a pilot-light instance in another location.

Runbook​

Here’s a high-level runbook for deploying this architecture in production:

1. Provision ingress + egress ReductStore nodes
2. Create buckets and replication tasks
3. Expose /ingress and /egress via NGINX with ip_hash
4. Test with demo dataset
5. Validate reads from egress
6. Run failure drills

References​

• NGINX Resilient Setup Example
• Disaster Recovery Guide

I hope you find this article interesting and useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

We are pleased to announce the release of the latest minor version of ReductStore, 1.16.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.16.0?​

The v1.16.0 release introduces two new extensions designed to enhance data workflows for robotics and columnar data, along with support for replicating context records during queries.

Querying and Replicating Data with Context​

We’ve extended the conditional query syntax with directives that allow users to modify global query behavior. The first directives introduced are #ctx_before and #ctx_after, which enable the inclusion of context records that occur before or after each matching record in a query.

This feature is particularly useful when analyzing specific events or conditions in your data, as it helps provide a clearer picture of the surrounding context. For instance, you can use these directives to include records from a few seconds before or after an anomaly or incident, aiding in root cause analysis or pattern recognition.

Here’s an example of how to use the #ctx_before directive in a query:

{
  "#ctx_before": "5s",
  "&anomaly_score": { "$gt": 0.8 }
}

This query returns all records with an anomaly score greater than 0.8, along with the context records that occurred within 5 seconds before each matching entry.

New ReductSelect Extension​

ReductStore is fundamentally a blob storage system and does not allow direct manipulation of stored data. However, with its extension mechanism, we can introduce new capabilities while keeping the core system simple.

The new ReductSelect extension enables users to query and transform data stored in CSV or JSON formats, making it easier to build flexible and efficient data processing workflows.

For example, the following query uses ReductSelect to extract specific columns from CSV data and filter rows using the same conditional syntax available in ReductStore's native query language:

{
  "ext": {
    "select": {
      "csv": {
        "has_headers": true
      },
      "columns": [
        { "name": "temperature", "as_labels": "temp" },
        { "name": "humidity" }
      ]
    }
  },
  "when": {
    "&temperature": { "$gt": 30 }
  }
}

This query selects the temperature and humidity columns from a CSV file, renames temperature to temp, and filters rows where the temperature is greater than 30°C.

These simple transformations enable you to ingest structured data very quickly and retrieve only subsets of it for further processing and analysis.

New ReductROS Extension​

Another exciting addition is the ReductROS extension, which provides tools for extracting and transforming data stored in ReductStore into formats compatible with the Robot Operating System (ROS).

With this extension, you can extract data from MCAP files containing ROS 2 messages and convert it into JSON format, making it easier to analyze and visualize. It also supports transforming raw binary data—such as images—into more accessible formats like JPEG or base64 strings.

For example, the following query extracts data from a ROS 2 topic and encodes the image payload as a JPEG:

{
  "ext": {
    "ros": {
      "extract": {
        "topic": "/camera/image",
        "encode": { "data": "jpeg" }
      }
    }
  }
}

ReductROS is still in active development, and we plan to expand its capabilities with support for additional ROS message types and more flexible extraction options in future releases. Stay tuned for updates!

What next?​

We are constantly working on improving ReductStore and adding new features to provide the best experience for our users. In the next release we plan to add new features and improvements, including:

Shareable Query Links​

We are developing a feature that allows users to generate and share links to specific queries in ReductStore.

This will simplify collaboration by enabling team members to access query results without needing direct access to the ReductStore instance. It will also allow users to download results directly via a link and support integration with external tools and platforms such as Foxglove.

Integration with Grafana​

We are also working on a Grafana plugin that enables users to visualize and analyze data stored in ReductStore directly within Grafana dashboards.

This integration will provide a seamless experience with Grafana’s powerful visualization tools, allowing you to:

• Build custom dashboards using data from ReductStore.
• Monitor your data streams and historical records in real time.
• Visualize labels and data output in JSON or CSV formats.

Stay tuned for the first release—coming soon!

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

In robotics development, effective visualization and analysis tools are essential for monitoring, debugging, and interpreting complex sensor data. Platforms like RViz, Foxglove, and Rerun play a key role at the visualization layer of the observability stack. They help developers interact with both live and recorded data. These tools rely on timely, well-structured access to the underlying data streams. That's where ReductStore comes in. It handles the data logging, storage, and processing, with a focus on capturing high-volume time-series data efficiently. ReductStore aims to integrate with tools like RViz, Foxglove, and Rerun, supporting a complete observability pipeline: from raw data ingestion to actionable insights.

Each visualization platform has its unique role in the development workflow. RViz (ROS Visualization) is the classic 3D visualization tool built for the ROS ecosystem, widely used for real-time robot monitoring and debugging. Foxglove is a modern data visualization and inspection platform for robotics and physical AI systems, aiming to simplify how teams collect, visualize, analyze, and manage large volumes of diverse sensor data. Rerun is a lightweight, native desktop application focused on fast and efficient visualization of robotics data, enabling developers to quickly explore and debug both live and recorded sensor streams with minimal setup.

This article compares RViz, Foxglove, and Rerun across key criteria: pricing, cross-platform support, remote collaboration, user interface, extensibility, ROS integration, performance with large datasets, and visualization and analysis features. The goal is to help robotics developers choose the right tool for their specific needs.

Pricing​

RViz and RViz 2 are part of the ROS ecosystem and released under the BSD 3-Clause License. This permissive open-source license allows free use, modification, and redistribution (including for commercial purposes), as long as the original copyright and license notices are preserved.

Foxglove offers a free tier that includes core features for up to 3 users, 10 devices, and 10 GB of cloud storage. For larger teams or needs (e.g., extra users, storage, private extensions, enterprise integrations), paid subscriptions are available. Pricing is based on the number of users and storage volume, as well as usage and support level. There is also a free academic plan for qualified institutions, which includes more users and storage. Foxglove itself is proprietary software, though it is built on open protocols like MCAP and integrates with open-source ROS tools.

Rerun is fully open-source under both the MIT and Apache 2.0 licenses. There are no current paid plans for the open-source core. The project follows an open-core model: the core visualizer and SDK are free, while a commercial platform is in early access for teams needing cloud-based storage, collaboration tools, advanced analytics, and scalable CI/CD workflows. This commercial layer is designed to build on top of the open-source foundation.

Platform & Collaboration​

RViz and RViz 2 are primarily developed for Linux, where they offer the most stable and reliable performance. RViz 2 also supports Windows and macOS as part of ROS 2, but these versions are less mature and less commonly used. They often require manual setup or compilation, though support continues to improve with newer ROS 2 releases.

Both RViz versions are local desktop applications and are not designed for remote or multi-user use out of the box. Workarounds like SSH with X11 forwarding, VNC, or running RViz locally while connecting remotely to a ROS system are possible, but they are often fragile, require manual configuration, and may suffer from performance or latency issues depending on the network and hardware.

To address these limitations, early tools like ROS3D.js offered browser-based ROS 1 visualization, but they are now mostly unmaintained and incompatible with ROS 2. Modern web visualization is typically done with tools like Foxglove, Webviz, or custom WebSocket-based interfaces. Some cloud robotics platforms also offer remote ROS visualization, though they typically require extra integration work.

Foxglove runs on Windows, macOS, and Linux, available both as a native desktop app and in a web browser. This gives users the flexibility to work locally or remotely without installing software. The browser version supports multi-user collaboration, allowing teams to share layouts and stream live data securely in real time from any internet-connected device.

Rerun is a lightweight native desktop application for Windows, macOS, and Linux. It requires minimal setup and enables developers to quickly visualize and debug live or recorded sensor data without needing a browser or complex configuration. Although Rerun does not support multi-user or collaborative features, teams often share log files for offline review. This approach is usually more practical than using remote desktop tools. Rerun also integrates well into development workflows, such as Python environments, which typically require installing Rerun's SDKs and dependencies.

Note: All three tools support sharing of recorded data, such as rosbag files for RViz & Rviz 2 (.bag for ROS 1 and .db3, .mcap for ROS 2), .mcap files for Foxglove, and .rrd for Rerun. To support these workflows at scale, you can use ReductStore solutions to manage continuous recording, indexing, and long-term storage of these file types across teams and infrastructure.

User Interface​

RViz and RViz 2 have a powerful but somewhat dated interface that focuses more on functionality than modern design. The learning curve can be steep, especially for beginners, due to the complex layout and the need to manually configure displays, topics, coordinate frames, and tools. The interface is built around multiple panels and dialogs that require careful configuration. It lacks the visual polish and streamlined workflows of newer visualization tools.

Foxglove features a modern, user-friendly interface with flexible dashboards and responsive controls. It is designed to be accessible to users at all experience levels, making it easier to explore, analyze, and share robotics data. The interface relies heavily on graphical elements instead of commands or configuration files, which lowers the entry barrier for users unfamiliar with ROS or robotics tools.

Rerun offers a clean and straightforward interface focused on efficient data visualization. It balances ease of use with core functionality, providing easy-to-navigate views without overwhelming users. The interface requires minimal setup and supports intuitive exploration of data streams and logs. However, it currently has fewer customization options than RViz or Foxglove.

Extensibility​

RViz (both ROS 1 and ROS 2) supports extensibility through C++ plugins, allowing users to develop and integrate custom visualizations, tools, and panels. This plugin architecture makes RViz highly adaptable across robotics domains such as perception, navigation, and manipulation. Many ROS packages include their own RViz plugins by default. However, developing and using plugins requires tight integration with the specific ROS environment. Plugins made for RViz in ROS 1 are not directly compatible with RViz 2; they often require modification or a complete rewrite.

Foxglove offers extensibility through an Extensions SDK, which allows developers to build React-based visualizations using TypeScript. Extensions can be shared via an online registry and do not require recompilation. Foxglove also provides APIs and libraries in C++, Python, and Rust, primarily for working with the MCAP file format, enabling integration with ROS (both versions), WebSocket streams, and recorded sensor data. Foxglove's ecosystem also supports integration with popular robotics and simulation tools such as NVIDIA Isaac Sim, Velodyne LiDAR, and Jupyter Notebooks, either directly or via external bridges.

Rerun focuses on extensibility through SDKs and APIs, especially for Python and other programming environments. It does not support plugin-based customization or drag-and-drop extensions like RViz or Foxglove. Instead, it prioritizes programmatic data embedding and visualization, making it well-suited for users who prefer scripting and code-driven workflows.

Rerun offers strong Python support, but its core is built with Rust and the egui GUI framework — technologies less familiar to many robotics developers. This can introduce a learning curve and limit low-level customization unless users are comfortable with Rust.

Rerun does not offer a simple or dynamic plugin system or scripting layer similar to RViz's C++ plugins or Foxglove's TypeScript extensions. This limits rapid prototyping or quick third-party integration.

Still, its APIs offer robust integration with diverse data sources, including ROS topics, sensor streams, and machine learning frameworks like TensorFlow and PyTorch. This makes Rerun a flexible tool for logging, visualizing, and debugging complex data pipelines.

Rerun is best suited for developers who prefer programming-driven customization over GUI-based tools. It provides direct control over data ingestion and visualization, enabling highly tailored, dynamic workflows that can grow with project needs.

ROS Integration​

RViz is tightly integrated with ROS and supports direct interaction with live ROS topics. Originally developed for ROS 1, it was succeeded by RViz 2 for ROS 2, and it remains a core visualization tool in many robotics workflows. However, this deep integration limits RViz's usability outside the ROS ecosystem. Both versions depend on a fully functioning ROS environment and are not designed to run independently or handle non-ROS data without conversion.

Foxglove connects to live ROS systems using foxglove_bridge, a WebSocket-based bridge designed for this purpose. It runs on the same network as the ROS system and streams real-time ROS messages to Foxglove over WebSocket. This architecture allows remote monitoring and interaction with installing ROS locally. Unlike RViz, Foxglove can be used without a full ROS setup.

In addition to live data, Foxglove also supports opening and analyzing ROS bag files locally. This makes it easy to review recorded data, visualize topics, and troubleshoot issues offline, without needing an active ROS system.

Rerun supports integration with both ROS 1 and ROS 2, enabling live topic visualization and recorded data inspection. For ROS 2, Rerun officially maintaines basic example scripts, hosted on GitHub, that use Python (rclpy) or C++ to subscribe to ROS 2 topics and forward selected data to the Rerun viewer. This is a user-defined bridge rather than a native-plugin integration. ROS 1 integration is possible using custom nodes written in either C++ or Python (rospy), but usually requires more manual setup. Unlike Foxglove, which uses standardized communication protocols like foxglove_websocket via foxglove_bridge (and optionally rosbridge), Rerun ingests data directly through user-defined code and does not rely on ROS-specific bridge protocols. While Rerun avoids protocol-based bridging, it still requires users to write custom nodes that translate ROS messages into its API.

Rerun is especially useful for visualizing time-synchronized multimodal data, such as sensor readings, 3D geometry, camera images, transforms, and trajectories. However, it currently lacks built-in support for certain ROS-specific features like interactive TF tree exploration, occupancy/grid map overlays, and full URDF-based robot model visualization. Community-maintained examples (e.g., the urdf_loader) offer partial support for URDF rendering, but do not yet match RViz’s depth or interactivity.

Rerun also cannot currently open ROS bag files directly (.bag for ROS 1 or .db3 for ROS 2). Instead, users replay them with rosbag play or ros2 bag play and forward selected topics to Rerun using custom Python or C++ bridge nodes. This workflow offers flexibility and performance but requires additional configuration. Rerun uses its own .rrd log format, which is optimized for high-throughput, time-seekable storage and streaming.

Performance with Large Data​

RViz is not fully optimized for very large datasets, such as dense point clouds, high-frequency topics, or long message histories. When visualizing large volumes of data, users may encounter performance issues like low frame rates, rendering lag, and high CPU or GPU usage. This happens because RViz continuously renders incoming ROS messages and stores message history in memory, which can quickly overwhelm system resources.

RViz 2 improves on this with better multithreading and more efficient message transport via DDS. These changes help boost performance and scalability in ROS 2 environments. However, RViz 2 still struggles with very dense or high-rate data streams, especially when rendering complex 3D data in real time, and these improvements do not fully solve the challenges of high-density visualization. To improve performance, users often reduce message history length, filter or downsample data, and disable non-essential displays.

Foxglove, particularly its web version, can underperform RViz in high-data scenarios. Because it runs in a web browser, it's constrained by browser memory limits, single-threaded JavaScript execution, and limited access to hardware acceleration. As a result, visualizing large point clouds or streaming high-frequency topics may lead to lag, dropped frames, or browser instability. These limitations are especially evident when handling continuous 3D data or large bag files.

Performance can vary depending on the use case and browser environment. The desktop application bypasses some browser limitations and can perform better. However, since it is built on Electron, it still has overhead related to memory usage and resource management common to Electron-based apps. Though these issues are generally less severe than in the web version. For lighter workloads, such as 2D plots or moderate-frequency telemetry, Foxglove often performs well and benefits from its accessible UI and cross-platform support.

Rerun is designed with high performance in mind for large-scale, multimodal data workflows. It is a native desktop application written in Rust and uses the modern WGPU rendering backend. This gives it direct access to system resources, helping it efficiently handle dense point clouds, long message histories, and high-frequency data streams. Behind the scenes, Rerun uses techniques such as memory-mapped I/O, zero-copy data handling, and intelligent batching to reduce latency and resource use.

Although there are only few formal benchmarks comparing Rerun with RViz or Foxglove, early community feedback and its architecture suggest that Rerun scales effectively with complex datasets. Performance can be further improved by filtering or downsampling data streams according to specific needs. Rerun is currently under active development to expand its capabilities for robotics visualization and analysis.

Note: Best practices for handling large datasets include splitting data files by time or size (e.g., every 1–5 minutes), using separate files for different topic groups, and automatically deleting old files when disk space is low. Chunk compression can also save disk space more efficiently than whole-file compression, but this approach consumes more CPU and memory resources, representing a trade-off between storage and performance.

Analysis & Visualization​

RViz & RViz 2​

Key Capabilities:

• Real-Time Visualization & Bag File Support: RViz and RViz 2 support real-time visualization by subscribing to live ROS topics. They also display data from recorded bag files (.bag for ROS 1, .db3 and .mcap for ROS 2), when those files are replayed using tools like rosbag play or ros2 bag play.

• Data Format Support: RViz visualizes a wide range of robot state information, including URDF robot models, coordinate transforms (TF), and various sensor data such as LIDAR, IMU, depth, and RGB cameras. It also supports odometry, localization, occupancy grid maps (used in SLAM), navigation data (paths, goals, trajectories), and interactive markers for user interaction. RViz 2 supports the same data types with ROS 2 message compatibility.

• Interactive Markers: These 3D UI elements enable users to manipulate objects within the visualization: setting navigation goals, adjusting robot end-effector positions, or dragging points for motion planning. Using them requires writing supporting ROS nodes and configuring interaction logic.

• Configurable Interface: Users can add, remove, and arrange panels, and customize display properties such as colors, shapes, and update rates for each data type. These configurations can be saved and reloaded using .rviz files, streamlining repetitive workflows like navigation, debugging, or SLAM visualization. Multiple camera control modes (Orbit, FPS, Top-down) allow flexible 3D scene navigation.

• Plugin-Based Architecture: Developers can extend RViz by creating custom visualizations and tools through C++ plugins. RViz 2 supports plugins too, built on a more modern and modular architecture.

Limitations:

• Limited Analysis: RViz and RViz 2 primarily serve visualization purposes and lack built-in tools for detailed message inspection, conditional logging, or advanced playback controls like pause, step, or speed adjustment. These features typically require external tools such as rqt_bag, ROS CLI utilities, or third-party RViz plugins (e.g., rosbag_panel). RViz also does not consistently warn about invalid data (e.g., NaNs or infinities), which can result in missing or misleading visuals. These tools are not designed for deep offline data analysis and are best used alongside more specialized logging or analysis solutions.

• No Time-Series Analysis: RViz and RViz 2 do not support time-series plotting or statistical analysis. For these tasks, dedicated tools like rqt_plot, PlotJuggler (with ROS 2 support), or external environments like Jupyter with Python are more appropriate.

• No Conditional Filtering: RViz and RViz 2 display all incoming data without the ability to filter messages based on content or fields. Filtering must be performed upstream, often by custom ROS nodes. Some plugins or panels offer limited filtering but are not general solutions.

• No Topic Synchronization: RViz and RViz 2 subscribe to each topic independently and display messages as they arrive. They do not synchronize data streams from different topics based on timestamps, which can cause misalignment or inconsistencies in time-sensitive visualizations (e.g., camera images, LIDAR scans, TF frames). Synchronization requires external tools like message_filters or custom nodes.

• No Built-In Logging or Export: RViz and RViz 2 cannot automatically export visualized data or record screencasts. Users are limited to manual screenshots unless using custom plugins or external tools to record sessions or extract data.

• Limited Multi-Robot Support: While RViz can display data from multiple robots using namespaces, the interface is not designed for straightforward multi-robot workflows. RViz 2 includes minor improvements, but still lacks dedicated features for managing multiple robots simultaneously.

Foxglove​

Key Capabilities:

• Multi-Modal 3D Visualization: Foxglove provides comprehensive 3D visualization for a variety of robotics data, including URDF robot models, TF trees, sensor streams (LIDAR, point clouds, camera feeds), occupancy grids, and navigation elements such as paths, goals, and costmaps. Users can interact with the scene in real time: rotating the view, toggling layers, and focusing on specific frames or topics. Multi-camera views, tooltips, and overlays enhance spatial understanding. Synchronized multi-viewports and flexible camera modes (free, fixed, follow-frame, sensor-aligned) make it possible to examine several spatial data streams side by side. All streams are synchronized through a shared timeline for consistent context across modalities.

• Topic Synchronization & Playback Timeline: Foxglove offers a unified, timestamp-based timeline that synchronizes data from multiple topics. This ensures time-aligned playback of sensor streams like RGB images, depth, point clouds, IMU, and TFs, useful both in real time and with recorded data. The timeline includes playback controls such as pause, frame-by-frame stepping, variable speed, and bookmarks for quickly navigating to key events. This tight time synchronization is a major advantage over RViz, enabling clearer insights into system behavior.

• Advanced Analysis & Time-Series Tools: Foxglove offers a capable set of tools for offline analysis of recorded data. Users can inspect messages in detail, filter them by topic or namespace, and control playback through an integrated timeline with pause, step-by-step navigation, and adjustable speed. To view custom ROS 2 message types with full support, messages are best recorded in or converted to the MCAP format, although Foxglove can open other formats with some limitations.

  For analyzing numeric data over time, Foxglove provides flexible plotting tools broadly comparable to rqt_plot or PlotJuggler. Users can visualize numeric fields from any message type, overlay multiple signals, zoom in on specific time ranges, and interactively adjust what's displayed.

  These tools make it easy to explore sensor outputs, control signals, and algorithm behavior. Interactive graphs support dynamic switching between message series.

• Modular & Configurable Interface: The Foxglove UI is fully modular, allowing users to add, remove, duplicate, and rearrange panels such as 3D views, image feeds, message viewers, plots, diagnostics, and consoles. Each panel is highly configurable, with settings for color, scale, transparency, update rate, and filtering. Users can save layouts as JSON files, enabling reproducible setups, role-based dashboards, and fast task switching (e.g., from SLAM debugging to perception analysis). Layouts can be shared across teams or versioned over time.

• Custom Panels & Extensions: Foxglove allows users to build custom panels using plugins, enabling specialized interfaces tailored to specific workflows. These panels are embedded directly into the Foxglove interface, keeping everything streamlined and centralized. This is particularly valuable for teams developing internal tools or dashboards for robotics development and testing.

  Extensions are packaged as .foxe files and can be shared via the Foxglove Extensions Registry, either privately within an organization or publicly. While layouts can be shared via URLs or layout IDs to replicate interface setups, this sharing does not include the actual panel code. Users must have the required extensions installed for full functionality.

• Cloud & Collaboration: Foxglove can be run locally or in the cloud. Its cloud features include shared dashboards, timeline comments, and real-time collaboration, enabling teams to jointly review logs or live data remotely. This makes it particularly useful for distributed development, remote testing, or asynchronous data reviews.

Limitations:

• Limited Real-Time 3D Interactivity: Foxglove does not natively support interactive 3D markers like RViz. Users cannot directly manipulate objects in the 3D scene (e.g., setting goals, editing poses, or dragging elements) without building custom extensions. This limits Foxglove's out-of-the-box usability for real-time tasks such as motion planning, teleoperation, or interactive environment setup.

• Limited Advanced Features: Foxglove currently lacks certain advanced features found in tools like PlotJuggler. For example, Foxglove does not yet support strict axis ratio locking — a critical feature for accurately visualizing spatial data where maintaining proportional relationships between axes is important. Additionally, Foxglove's built-in data transformation capabilities are limited compared to PlotJuggler's comprehensive suite of statistical and signal-processing tools, such as moving averages, derivatives, filtering, and custom mathematical expressions. These advanced features make PlotJuggler especially useful for detailed signal analysis and fine-grained data manipulation, often essential when debugging sensor data or control signals.

• No Automated Anomaly Detection: Foxglove does not include built-in automated validation or anomaly detection. It does not use ML models or rule-based systems to automatically flag issues. Instead, it offers detailed message introspection and customizable visualizations that enable users to manually identify irregularities such as NaNs, infinities, or out-of-range values. This hands-on approach requires user expertise but provides flexible, in-depth analysis without automated alerts.

Rerun​

Key Capabilities:

• Real-time & Recorded Data Visualization: Rerun supports both live-streamed and recorded sensor data visualization with minimal latency. It ingests data via Rust- or Python-based logging SDKs, handling a wide range of robotics sensor modalities including 3D spatial data, camera imagery, numeric time-series, semantic segmentation maps, depth maps, annotations (bounding boxes, keypoints), and textual or categorical event data. Recorded datasets can be replayed with full timeline control for stepwise inspection or smooth playback, aiding in bug reproduction and model validation.

• Collaboration & Sharing Features: Rerun streamlines collaborative workflows through data export and session sharing via .rrd files. Teams can share recorded .rrd files for offline inspection, annotate data using Annotation Context (which supports labeling via class IDs and color mapping), and use shared Recording IDs to log streams from multiple processes or machines into a unified session, as long as the Recording ID is set consistently at the time of logging. Note: merging previously recorded .rrd files with different Recording IDs offline is currently not supported. Users can also export screenshots (for reports or dashboards) via the CLI or viewer options, depending on the version and available commands.

• Customizable & Extensible UI: The Rerun Viewer offers a modular, layout-aware interface tailored for tasks such as SLAM debugging, multi-sensor calibration, and performance profiling. Users can save and reload Blueprints — serialized UI configurations that preserve panel layouts, timelines, selected entities, and styling (e.g., color, transparency, size). A full styling hierarchy (override → store → default → fallback) makes it easy to customize visuals without modifying source data. Multiple synchronized views (3D scenes, timelines, 2D plots, raw data inspectors) support comprehensive analysis.

  Developers can extend functionality by adding custom egui-based panels, entity-aware views, or by embedding the Viewer into Rust applications via the re_viewer crate. Custom data loaders and visualization types can be created for proprietary or specialized formats. Interactive tools such as selection, pinning, and hover highlighting assist detailed exploration. The Viewer also supports interactive callbacks for events like hover, selection, and context menu actions, enabling highly tailored, real-time data interactions.

• Rich 3D Visualization with Spatial Context: Built on egui and WGPU, Rerun's 3D viewer efficiently renders large-scale scenes on consumer hardware. It uses an entity-path-based scene graph that reflects the hierarchical kinematic tree, allowing intuitive navigation and inspection of components, sensor frames, trajectories, bounding boxes, segmentation masks, dense point clouds, annotated images, 3D meshes, and time-series plots. Users can customize visual parameters (e.g., color maps, visibility, annotations, rendering modes) and navigate using orbit, zoom, and pan controls.

• Flexible Time-Series & Event Logging: Rerun supports synchronized timeline playback of multiple data streams, using both explicit (user-defined) and implicit (auto-derived) timestamps. It manages multiple time domains (logical/log time and timeline time) to accurately align heterogeneous data sources. Timeline controls include zooming, scrubbing, filtering by entity path or timeline, and detailed event inspection with metadata. Conditional filtering and selective visibility help isolate anomalies or relevant events in complex multi-agent or multi-sensor deployments.

• Programmable Data Access & Web Integration: The Rerun SDK provides semantic logging primitives (e.g., log_scalar, log_image, log_point_cloud, log_text_entry, log_tensor) that render automatically in the Viewer. Rerun uses Apache Arrow for efficient data handling, supporting advanced analysis with tools like Pandas and Jupyter. Direct export to formats like Parquet is supported via the API, making it suitable for both streaming visualization and offline batch analysis. The Viewer is also available as a React component, enabling seamless embedding within React applications and custom web dashboards, though integration with other JavaScript frameworks may require additional adaptation.

  Underneath these features, Rerun's flexible and scalable logging is powered by an Entity-Component-System (ECS) data model, which organizes time-stamped components attached to hierarchical entities to maintain performance and extensibility.

• Emerging Features: Experimental capabilities include graph-based views for visualizing system architectures, connectivity, and agent interactions, extending Rerun's utility beyond traditional sensor data visualization into system design and research workflows.

Limitations:

• No Built-In Advanced Analytics: Rerun focuses primarily on visualization and lacks integrated statistical analysis, anomaly detection, or expression-based plotting features. In contrast, Foxglove provides richer analytics, including expression plots and integration with monitoring systems like Prometheus.

• Not Optimized for Live Robot Control: Although it supports real-time data streaming, Rerun is not designed for robot teleoperation or control input interaction. RViz and Foxglove offer more mature tools for monitoring and interacting with live robots.

• No Native Support for Navigation and SLAM Maps: Unlike RViz, Rerun does not natively visualize occupancy grids, costmaps, or SLAM results, limiting its utility for path planning or localization workflows.

• Limited Real-Time Collaboration: While Rerun supports offline session sharing, it lacks live multi-user collaboration features such as synchronized remote views or cloud-hosted live sessions, which are available in Foxglove.

• Limited Visualization of Large-Scale System Architectures: Rerun's entity-based model focuses on spatial and temporal data but does not yet offer comprehensive tools for exploring complex system communication graphs or architecture diagrams interactively.

Conclusion​

This article provided a detailed comparison of RViz, Foxglove, and Rerun, evaluating them across practical dimensions: pricing, platform, and collaboration support, user interface, extensibility, ROS integration, performance with large datasets, and analysis and visualization capabilities. By outlining their strengths and limitations, we offer a clear perspective to help robotics engineers and developers choose the right tool for their specific needs.

Choosing the right tool depends on your context: use RViz for real-time ROS development and interactive debugging, Foxglove for collaborative data analysis, time-synchronized playback, and remote team workflows, and Rerun for fast, developer-centric visualization of structured data in programmatic pipelines. In practice, many robotics teams find that combining these tools enables more effective development and validation across different stages of their workflows.

We hope this comparison helps you make informed decisions and inspires you to keep exploring better tools and workflows. If you have questions, feedback, or insights to share, join the conversation on the ReductStore Community Forum.

When building computer vision systems, efficient data storage is a fundamental requirement. Whether you're capturing images for training, storing inference results for validation, or archiving sensor data for future analysis, your storage solution must be both reliable and high-performance.

Ingestion speed is especially critical. If your system can’t write data fast enough — whether it’s high-frequency frames or accompanying metadata — you risk losing valuable information or creating bottlenecks in the pipeline.

In this post, we’ll look at three common approaches to storing data in computer vision applications: a traditional file system, S3-compatible object storage, and ReductStore, a time-series-optimized blob storage. We’ll explore the strengths and limitations of each approach to help you choose the best fit for your application.

A Simple Computer Vision Application​

For demonstration purposes, we’ll use a simple computer vision (CV) application which is connected to a CV camera and runs on an edge device:

The camera driver captures images from the CV camera every second and forwards them to the model, which then detects objects and displays the results in the user interface.

Your images and results need to be stored for training and validation purposes. The customer may also wish to view images featuring anomalous objects. These requirements present the challenge of maintaining a history of blob or unstructured data.

File System​

One of the most straightforward ways to store images from a (CV) camera is to write them directly to a local file system. Each image can be saved using a timestamp as a unique identifier, and the directory structure can be organized by time intervals for easier access.

The main advantage of this approach is its simplicity — no external dependencies, no infrastructure setup, and it works out of the box. However, it comes with significant limitations:

• Data retention Disk space is finite. Eventually, you'll need to delete older data — either manually, using tools like cron, or by building this logic into your application.

• Data lookup While naming conventions and folder hierarchies can help, accessing data by time or context becomes inefficient as the dataset grows. In practice, this often requires a companion database to index and query files — adding complexity and undermining the simplicity of the original setup.

• Labeling and search If you want to tag images with labels (e.g., object type, camera location, quality), or filter by those labels later, the file system alone doesn’t support it. You’ll need to implement a separate mechanism.

• Replication Copying data to other machines (e.g., for redundancy or aggregation) is possible using tools like rsync or scp, but it requires manual setup, SSH access, and custom scripts.

• Security Providing access to stored data often requires giving users access to the entire file system or to network shares, which increases the risk of unauthorized access or accidental modification. Additionally, if you need to share data externally — for example, with a remote team or a cloud service — you may end up copying large volumes of files. This not only consumes time and bandwidth but also raises concerns about secure transfer, data leaks, and access control once the data is outside your network.

In summary, while the file system is an easy starting point, it rarely scales well. As your needs grow — especially around querying, labeling, replication, and security — you’ll often end up building database-like functionality on top of it. At that point, it may be worth reconsidering your storage architecture altogether.

S3-like Object Storage​

A more advanced approach to storing images is to use an object storage system, such as an S3-compatible service. This allows data to be organized similarly to folders and files, but accessed and managed through an HTTP-based API.

This method offers more flexibility than a local file system and comes with several advantages:

• Remote storage Data can be stored on remote servers or in the cloud, where capacity is virtually unlimited. If your organization or customer already uses an object storage–based data lake, it's possible to write data there directly.

• Security Access to the data can be controlled using fine-grained permissions. For example, the application might have write-only access, while users are granted read-only rights.

• Data replication Many object storage systems support built-in replication, allowing data to be mirrored automatically or manually to other locations.

However, this approach is not without its drawbacks:

• Data retention Although object storage typically offers more capacity, most systems don't provide real-time data expiration or fine-grained retention policies out of the box. Some platforms support time-based retention (e.g., lifecycle rules in S3), but these are often batch processes that run periodically, not in real time. This can be problematic if your data ingestion rate is unpredictable or bursty — you might exceed storage limits before the cleanup is triggered. In such cases, you’ll need to implement custom logic or external tooling to ensure timely data reduction.

• Data accessibility Despite its API, the data model still mimics a file system hierarchy, which may cause similar performance issues as the dataset grows. Moreover, access via HTTP introduces latency — each read or write requires a separate HTTP request, which can be inefficient for small objects. See our benchmark on performance limitations.

An S3-like object storage may seem like a natural upgrade from a file system, especially in terms of scalability and remote access. However, its hierarchical structure and limited native querying make it less suitable for time-series or high-ingestion workloads. Over time, you might find yourself building additional infrastructure — such as a time-series database or metadata index — to overcome these limitations. We’ve explored this alternative in detail and found that it introduces its own trade-offs.

ReductStore​

ReductStore is a high-performance storage and streaming solution for robotics, computer vision, and industrial IoT applications. It is purpose-built to ingest and store unstructured time-series data — such as images, sensor payloads, or inference results — directly at the edge. Once stored locally, the data can be selectively replicated to central storage or the cloud, enabling efficient data consolidation without sacrificing real-time performance at the edge.

As you can see, instead of a tree-like structure, it has a flat structure of entries where data is batched and partitioned into blocks. This approach is more efficient for writing, since blocks can be pre-allocated for incoming records. It also improves time-based querying: because data typically arrives in time order, records for a given time interval are physically located close to each other — making scans faster and more predictable.

Another very important feature of ReductStore is its support for data labeling. This allows you to store the image and the corresponding model output together in a single entry. Labels not only help organize and filter data during queries — they can also be used for conditional replication, so only relevant data is streamed to central storage or other nodes, or selectively retained on the edge for long-term storage.

Let’s summarize the benefits of using ReductStore:

• Data Accessibility ReductStore uses record timestamps as unique identifiers and provides an HTTP API for efficient access by exact time or time range. It also batches small objects into larger HTTP requests, which significantly improves performance on high-latency networks.

• Data Reduction Each bucket can be assigned a storage quota. When the quota is reached, the engine automatically removes the oldest data — enabling built-in, real-time retention management.

• Labeling Data can be tagged with labels and used as annotations or search criteria. Labels also enable advanced workflows such as conditional replication or selective long-term retention on the edge.

• Security ReductStore supports role-based access with token authentication, allowing you to define fine-grained read and write permissions per bucket. This makes it possible to isolate access between devices, applications, or users — ensuring that each has access only to the data it needs.

• Data Replication ReductStore supports append-only data replication, enabling automatic streaming of data from one instance to another. You can replicate all data or filter it using labels to replicate only what’s relevant.

Why Not a Relational Database?​

Relational databases are great for structured data, but they’re not designed for the demands of computer vision applications — especially when it comes to storing and managing image data at scale.

In most CV pipelines, you deal with large volumes of unstructured binary data: raw images, video frames, model outputs, and metadata. Trying to store this data in a relational database quickly becomes inefficient for several reasons:

• Inefficient for Large Binary Data Relational databases can store images as blobs, but they aren’t optimized for high-throughput write operations. Ingesting large numbers of images per second puts significant strain on the database engine.

• Poor Time-Series Access CV systems often need to query image data by time intervals — e.g., “give me all frames from the past 10 minutes.” Relational databases can support this, but performance degrades with large datasets unless you implement table partitioning and maintain complex indexes. This adds operational overhead and still may not scale well for high-ingestion workloads like real-time image capture.

Conclusion​

Choosing the right storage solution is critical for building efficient and reliable computer vision systems. While traditional file systems and object stores can serve as starting points, they quickly fall short when it comes to performance, searchability, and data lifecycle management — especially at the edge.

ReductStore offers a purpose-built alternative: a high-performance storage and streaming solution optimized for unstructured time-series data. With features like efficient time-based access, built-in data reduction, flexible labeling, and conditional replication, it bridges the gap between raw image capture and scalable, intelligent data pipelines.

If you're building applications in robotics, industrial vision, or edge-based AI, ReductStore can help you focus on what matters most — your models and insights — without getting bogged down in the complexity of data infrastructure.

I hope you find this article interesting and useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

At ReductStore, we specialize in the high-performance storage and streaming of robotics data from edge devices to the cloud. In this tutorial, we will demonstrate how to develop a robust ROS 2 data logging pipeline for practical robotics applications.

First, we will set up a Raspberry Pi with a USB camera running a lightweight YOLOv5n object detection model via ONNX Runtime. Then, a recorder node will capture selected ROS 2 topics, including images, detection results, and logs. Next, these topics will be saved as segmented MCAP files locally with ReductStore. Finally, we will configure automatic replication to stream data to another ReductStore instance.

This minimal setup shows how to efficiently capture, store, and replicate ROS 2 data from a robot to a central server or cloud instance. These techniques can be applied to any ROS 2 system, whether it's a single robot or a fleet of autonomous systems.

Let's get started!

1. Prerequisites and Architecture​

Before beginning, ensure you have the following:

• Hardware: Raspberry Pi, a USB camera, a laptop, and an internet connection.
• OS: Ubuntu or Desktop Server (22.04 LTS or later) on the Raspberry Pi.
• Laptop/PC: A separate machine on the same network, to serve as a central data store.

In this tutorial, we will set up the following architecture:

• Raspberry Pi: Running ROS 2, interfacing with a USB camera, and running a YOLOv5n object detection node with ONNX Runtime.
• Laptop/PC: Running a ReductStore instance (acting as a central data store) to receive data from the Raspberry Pi.
• Network: For simplicity, both the Raspberry Pi and the laptop are assumed to be on the same local network.

1.1. Install Ubuntu on Raspberry Pi​

If not already done, flash Ubuntu for Raspberry Pi on a microSD card and boot your Pi. You can download Ubuntu images for Raspberry Pi from the official site. Ensure you have internet access and update the system:

# On Raspberry Pi
sudo apt update && sudo apt upgrade -y

1.2. Install ROS 2 on Raspberry Pi​

Follow the steps below to set up ROS 2 on your Raspberry Pi. This guide assumes you are using a supported 64-bit Ubuntu OS (e.g., 22.04, 24.04). ROS 2 is available in multiple distributions such as Jazzy, Humble, and Noetic. If you're unsure, we recommend using the latest LTS version, which we can install via the commands below.

1. Setup Locale:

   sudo locale-gen en_US en_US.UTF-8
   sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
   export LANG=en_US.UTF-8
   

2. Add ROS 2 apt Repository:

   sudo apt install software-properties-common
   sudo add-apt-repository universe
   sudo apt update && sudo apt install curl -y
   sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
   echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null
   sudo apt update
   

3. Install ROS 2 Packages: For a baseline, install ROS 2 base packages (or ros-<distro>-desktop if you need GUI tools):

   sudo apt install ros-${ROS_DISTRO}-ros-base -y
   

   This will install core ROS 2 packages including rosbag2. You may verify the installation by sourcing the setup script:

   source /opt/ros/${ROS_DISTRO}/setup.bash
   ros2 -h    # Should show ROS 2 commands
   

4. Install Build Tools: We will create custom packages, so install development tools:

   sudo apt install python3-colcon-common-extensions python3-rosdep -y
   sudo rosdep init
   rosdep update
   

1.3. Create a ROS 2 Workspace​

Set up a workspace for our project (if you don't have one):

# On Raspberry Pi
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws
colcon build  # just to initialize, will be empty initially

Add source ~/ros2_ws/install/setup.bash to your ~/.bashrc so that the workspace is sourced on each new shell, or remember to source it in each terminal when using the workspace. We will add packages to this workspace in subsequent steps.

2. Setting up the USB Camera with v4l2_camera​

We'll use the v4l2_camera ROS 2 package to interface with the USB camera via Video4Linux2. This package publishes images from any V4L2-compatible camera (most USB webcams) as ROS 2 image topics.

2.1. Install the v4l2_camera Package​

On the Raspberry Pi, install the driver node via apt:

sudo apt install ros-${ROS_DISTRO}-v4l2-camera -y

This installs the v4l2_camera node and its dependencies. Alternatively, you could build it from source, but the binary is available for many ROS 2 distributions.

2.2. Connect and Verify the Camera​

Plug in the USB camera to the Pi. Verify that it's recognized by listing video devices:

ls /dev/video*
# You should see /dev/video0 (and possibly /dev/video1, etc. if multiple video capture interfaces are connected)

If /dev/video0 is present, the system sees the camera (at list one). You might also install v4l2-utils and run v4l2-ctl --list-devices to see the camera name and capabilities. You can also run v4l2-ctl --list-formats-ext to see supported resolutions and formats.

2.3. Run the Camera Node​

Launch the camera driver to start publishing images:

# In a sourced ROS 2 environment on the Pi
ros2 run v4l2_camera v4l2_camera_node

By default, this node will open /dev/video0 and start publishing images to the ~/image_raw topic (type sensor_msgs/Image) at a default resolution of 640x480 and pixel format YUYV converted to rgb8 (see ROS 2 camera driver for Video4Linux2 Documentation). You should see console output from the node indicating it opened the device and is streaming.

Open a new terminal (with ROS sourced) on the Pi (or from a laptop connected to the ROS 2 network) and verify images are coming through, e.g., by running rqt_image_view:

sudo apt install ros-${ROS_DISTRO}-rqt-image-view -y  # if not installed
ros2 run rqt_image_view rqt_image_view

In rqt_image_view, select /image_raw to view the camera feed. This confirms the camera setup is working.

Note: You can adjust parameters by remapping or via ROS 2 parameters, e.g., to change resolution or device:

ros2 run v4l2_camera v4l2_camera_node --ros-args -p image_size:="[1280,720]" -p video_device:="/dev/video0"

This would set the camera to 1280x720 resolution (if supported).

3. Deploying a Lightweight YOLOv5 Object Detection Node​

Next, we set up an object detection node to analyze the camera images and output metadata (object_detected and confidence_score). We'll use YOLOv5n (Nano) - the smallest YOLOv5 model ("only" 1.9 million parameters) which is ideal for resource-constrained devices (see releases at ultralytics/yolov5). We will run inference using the ONNX Runtime, which allows running the model without needing the full PyTorch framework on the Pi.

3.1. Install ONNX Runtime and Dependencies​

On the Raspberry Pi, install the ONNX Runtime Python package and OpenCV (for image processing):

pip install onnxruntime opencv-python

(If pip isn't available, use sudo apt install python3-pip to install it. You may also install numpy if not already present, as ONNX Runtime will likely need it.)

3.2. YOLOv5n ONNX Model​

We need the YOLOv5n model in ONNX format. To do that, we can clone the YOLOv5 repository on a more powerful machine than the Pi and export the model:

# On a PC or via Colab:
git clone https://github.com/ultralytics/yolov5.git
cd yolov5
pip install -r requirements.txt  # includes PyTorch
python export.py --weights yolov5n.pt --include onnx

This will create yolov5n.onnx. Transfer that file to your Raspberry Pi (e.g., via SCP).

For this tutorial, assume yolov5n.onnx is now on the Raspberry Pi (e.g., placed in ~/ros2_ws/src).

3.3. Create a ROS 2 Package for the YOLO Node (optional)​

You can integrate the YOLO inference in the same package as your data recorder, but for modularity, let's create a separate ROS 2 Python package called yolo_detector.

In the workspace src directory, run:

cd ~/ros2_ws/src
ros2 pkg create --build-type ament_python yolo_detector --dependencies rclpy sensor_msgs std_msgs

This will create a yolo_detector folder with a Python package structure.

3.4. Implement the YOLO Detection Node​

Create a file yolo_detector/yolo_detector/yolo_node.py with the following content:

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from std_msgs.msg import String, Float32
import cv2
import numpy as np
import onnxruntime as ort

class YoloDetectorNode(Node):
    def __init__(self):
        super().__init__('yolo_detector')
        # Load the YOLOv5n ONNX model
        model_path = '/path/to/yolov5n.onnx'  # TODO: update to actual path
        self.session = ort.InferenceSession(model_path, providers=['CPUExecutionProvider'])
        self.get_logger().info(f"Loaded model {model_path}")

        # Get model input details for preprocessing
        model_inputs = self.session.get_inputs()
        self.input_name = model_inputs[0].name
        self.input_shape = model_inputs[0].shape  # e.g., [1, 3, 640, 640]
        self.img_height = self.input_shape[2]
        self.img_width = self.input_shape[3]

        # Subscribers and publishers
        self.subscription = self.create_subscription(Image, '/image_raw', self.image_callback, 10)
        self.pub_object = self.create_publisher(String, 'object_detected', 10)
        self.pub_conf = self.create_publisher(Float32, 'confidence_score', 10)

        # If the model requires normalization factors or specific transformations, define them:
        self.mean = np.array([0.0, 0.0, 0.0])  # YOLOv5 models assume 0-255 input, no mean subtraction
        self.std = np.array([255.0, 255.0, 255.0])  # we'll scale 0-1 later by dividing by 255

    def image_callback(self, msg: Image):
        # Convert ROS Image message to OpenCV format (BGR array)
        # Assuming msg.encoding is 'rgb8' as provided by v4l2_camera default output
        img = np.frombuffer(msg.data, dtype=np.uint8).reshape(msg.height, msg.width, -1)
        # Convert RGB to BGR as YOLO model might expect BGR input (depending on training)
        img_bgr = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

        # Resize and pad image to model input shape (letterboxing if needed)
        input_img = cv2.resize(img_bgr, (self.img_width, self.img_height))
        # Convert to float32 and normalize 0-1
        input_img = input_img.astype('float32') / 255.0
        # transpose to [channels, height, width]
        input_blob = np.transpose(input_img, (2, 0, 1))
        input_blob = np.expand_dims(input_blob, axis=0)  # shape [1,3,H,W]

        # Run inference
        outputs = self.session.run(None, {self.input_name: input_blob})

        # Parse outputs to find the highest confidence detection (for simplicity)
        # YOLOv5 ONNX output typically includes [1, num_boxes, 85] array (for COCO: 4 box coords, 1 objness, 80 class scores)
        detections = outputs[0]
        # Filter by confidence threshold (e.g., 0.5)
        conf_threshold = 0.5
        best_label = "none"
        best_conf = 0.0
        if detections is not None:
            for det in detections[0]:
                obj_conf = det[4]
                class_conf = det[5:]  # class confidences
                score = obj_conf * np.max(class_conf)
                class_id = np.argmax(class_conf)
                if score > conf_threshold and score > best_conf:
                    best_conf = float(score)
                    best_label = str(class_id)  # or use a class id->name mapping

        # Publish results
        self.pub_object.publish(String(data=best_label))
        self.pub_conf.publish(Float32(data=best_conf))
        self.get_logger().info(f"Detected: {best_label} ({best_conf:.2f})")

def main(args=None):
    rclpy.init(args=args)
    node = YoloDetectorNode()
    rclpy.spin(node)
    node.destroy_node()
    rclpy.shutdown()