Skip to main content

Gateway Data Architecture

This page explains how Aurora’s data storage solutions work under the hood.

High-level overview​

At the core, our storage architecture comprises two components:

  1. Gateway — a virtual appliance that exposes familiar, compatible storage APIs.
  2. DSN backend — a software-and-hardware stack that leverages decentralized storage network principles to provide the underlying storage infrastructure.

High Level Overview

Physical data location​

Client applications are generally assumed to run on‑premises. Aurora can also provide compute (including confidential compute) and AI services hosted by Aurora.

The Gateway deployment strategy depends on client requirements; Aurora recommends an approach for each client. As a rule of thumb, clients with large data volumes or I/O patterns that require low latency or very high bandwidth benefit from co‑locating more components - Gateway, and for larger deployments, the DSN - with their infrastructure. When on‑premises presence is warranted but not feasible, Aurora can establish presence geographically close to the client (within the same city or jurisdiction).

Clients retain full control over data placement, access, and encryption. Aurora can facilitate data migrations between regions and jurisdictions as needed.

Storage architecture​

Aurora operates two storage engines:

  • Aurora IPLD Gateway — an IPLD‑first engine that maximizes data integrity. It uses multiple layers of integrity checks and is optimized for strongly adversarial environments.
  • Aurora Sovereign Gateway — optimized for enterprise environments, maximizing performance, efficiency, and availability while retaining strong provenance capabilities.

Aurora IPLD Gateway​

Aurora IPLD Gateway distills years of R&D into an easy‑to‑use, compatible, and scalable storage product. It provides a high level of assurance for data integrity through extensive use of content‑addressed, immutable IPLD data structures and advanced Merkle‑tree applications.

High Level Overview

Component overview​

The IPLD Gateway comprises several components:

  • Storage protocol frontends and backends
    • For S3, the frontend implements advanced S3 features, while the backend focuses on robust basic storage primitives and exposing provenance data.
    • Other storage protocols and primitives (IPLD objects, files, block namespaces) are not currently supported on this gateway.

HeadObject with Aurora IPLD Gateway metadata​

  • Clients opt in to Aurora metadata by including the query parameter fil-include-meta=1 on HEAD requests.
    • When the object is not yet offloaded, the gateway adds X-Fil-Offload-Status: staging.
    • The Aurora Gateway (AGW) HeadObject response will contain:
      • X-Fil-Cid: IPLD CID of the object represented as unixfs v1 file
      • X-Fil-Offload-Status: current DSN offload status
    • These headers are added to the S3 response sent back to the client.

Client usage tips​

  • To retrieve Aurora metadata for an object without downloading it, send:
    • HEAD s3://bucket/key?fil-include-meta=1
    • Inspect the X-Fil-Cid and X-Fil-Offload-Status headers.
  • Standard S3 tools can be used unchanged; gateway behavior is transparent unless the Aurora extension is requested.

Aurora Sovereign Gateway​

Aurora Sovereign Gateway is an evolution of the original Aurora Gateway, re‑engineered to take advantage of modern hyperscale hardware, enabling sovereign data solutions at exabyte scale.

The architecture employs carefully designed, advanced data structures that enable a much higher degree of hardware acceleration, unlocking new kinds of AI applications. It also provides stronger provenance capabilities: clients can tag and track all data assets, create proofs of data origination and proofs of existence at a point in time, and define fine‑grained access grants to data, including for attested confidential compute (TEE) enclaves.

High Level Overview

Note: The Sovereign Gateway is currently in late‑stage R&D and not yet available for production use.