Following our work on a Private Indexer for Swarmkey’d network (cf : Optimizing Private IPFS: Building a Private Indexer for Swarmkey'd Networks ), we focused on improve the performance by building a CDN-like for better upload and download time.
-> We are now using it in production if you want to test it before, please reach us here or through Github (on the Open Source repo regarding these two projects).
Technical Guide
A complete specification for the Swarm Accelerator: a CDN-like content gateway for private IPFS networks that provides indexer-guided provider lookup, hedged fetching, LRU caching with pin lifecycle, concurrent prefetch, hot-set management, and Prometheus-style operational metrics.
1. Purpose
Private IPFS networks (swarm.key v1) isolate all traffic from the public DHT, but they have no built-in CDN or edge-cache layer. Every content request traverses Bitswap peer-to-peer, paying full latency every time - even for the same block requested seconds earlier. The Swarm Accelerator addresses this by providing:
∙ CDN-like caching - An in-memory LRU cache fronts every Kubo fetch. Repeat requests for the same CID are served from RAM in sub-millisecond time, avoiding redundant Bitswap round-trips entirely.
∙ Indexer-guided provider selection - Instead of relying on Bitswap’s blind flooding, the accelerator queries the Private Indexer for pre-ranked providers (scored by freshness, latency, and success rate) before fetching, routing requests toward the best-performing nodes.
∙ Hedged fetching - A tail-latency mitigation technique: if the primary Kubo fetch does not complete within a configurable delay, a parallel backup request is launched. The first successful response wins; the slower one is discarded. This eliminates stalls caused by a single slow peer.
∙ Concurrent prefetch - Operators can proactively warm the cache with batches of CIDs or entire DAG trees (/prefetch and /prefetch/tree), so users never experience a cold-start penalty for predictable content (app bundles, static assets).
∙ Hot-set protection - Critical CIDs (e.g., application shell, config manifests) can be pinned into a hot set that is exempt from LRU eviction and TTL expiry. A background refresh loop re-warms any cold hot-set entries automatically.
∙ Kubo pin lifecycle - Fetched content is pinned on the local Kubo node; evicted content is unpinned asynchronously, keeping Kubo’s pin set aligned with the accelerator’s cache policy.
∙ Operational metrics - /health (JSON summary) and /metrics (Prometheus text format) expose cache hit rate, average warm time, eviction counts, hedged-fetch win ratios, prefetch throughput, and hot-set status for dashboards and alerting.
Relationship to Other Components
| Component | Relationship to Accelerator |
|---|---|
| Private Indexer | Accelerator calls GET /providers?cid=… to discover ranked providers before fetching. The indexer’s scoring engine returns providers sorted by quality. |
| Announcer | No direct interaction. Announcers POST /announce to the indexer so CIDs are discoverable. The accelerator benefits indirectly when it queries the indexer. |
| Kubo | Accelerator calls Kubo’s HTTP API (/api/v0/cat, /api/v0/pin/add, /api/v0/pin/rm, /api/v0/ls) for all block retrieval, pinning, unpinning, and DAG traversal. |
| Scoring / Ranking | Ranking is computed inside the indexer (package internal/scoring). The accelerator consumes already-ordered ProviderRecord lists — it does not import or execute ranking logic itself. |
2. Architecture
2.1 High-Level Architecture Diagram
Shows where the accelerator fits within the full Private Swarm stack:
In the First Comment
2.2 Gateway Request Flow (GET /ipfs/{cid})
The primary accelerator workflow - from client request to response:
In the Second Comment
2.3 Hedged Fetching Detail
Shows the internal race between primary and backup requests:
In the Third Comment
2.4 LRU Cache with Hot-Set and Pin Lifecycle
In the Fourth Comment
2.5 Prefetch and Hot-Set Refresh Flow
In the Fifth Comment
3. Environment and Prerequisites
3.1 Runtime Requirements
| Requirement | Version | Notes |
|---|---|---|
| Go | 1.22+ | Build and local development |
| Docker | 20.10+ | Container runtime for production |
| Docker Compose | v2+ | Multi-service orchestration |
| Kubo | v0.37.0 | Local IPFS node (HTTP API must be accessible) |
| Private Indexer | running | Reachable at INDEXER_URL (default :8090) |
| curl, jq | any | For integration/manual tests |
3.2 Network Topology
The accelerator must have HTTP access to:
| Service | Default Address | Purpose |
|---|---|---|
| Local Kubo HTTP API | http:// localhost:5001 | Content retrieval, pinning, DAG listing |
| Private Indexer | http:// localhost:8090 | Provider lookups |
| Clients | :8070 (listen) | Serving gateway requests |
