My understanding is that the project provides a self-hosted w3storage backend fully compatible with the w3storage CLI and SDK that isn’t reliant on any special backend services like DynamoDB. Instead it stores everything on disk
That’s correct, and all the blocks in the DAG are stored in an IPFS compatible blockstore: blockstore-fs - npm - so you could just hook a kubo up to it and take a look around if you wanted.
How does the integration with local.freeway work? Does it rely on the API exposed by local.storage?
Yes, specifically 2 APIs exposed by local.storage:
- Blob storage -
GET /blob/:multihash - Get the bytes for a given hash. It supports HTTP Range requests so if the blob you’re retrieving is a CAR file, you can use a range request to pick out specific blocks you need.
- Content claims -
GET /claims/:cid - Get content claims for a given CID. We primarily use a location claim to know where the data is stored and an inclusion claim to know byte offsets of blocks within a CAR. Secondarily, a partition claim to know which CARs a DAG can be found in (since our client library will shard big DAGs across multiple CAR files).
Really, local.freeway just needs to get hold of the content claims, since they describe where to get the data from i.e. which local.storage node. When it knows where to ask, it can make requests to read blocks and “export” the file from the unixfs data.
Have you thought about how this could realistically be deployed in a way where the blocks are available to the IPFS network, i.e blocks (perhaps even only root blocks) are announced to the DHT/network indexer
Yes, this is literally what we’re aiming to achieve next with web3.storage - decentralize hot storage for filecoin.
We need to figure out exactly how to decentralize storing and querying for content claims. We’re currently making progress on a proposal for integrating with IPNI.
FYI local.freeway is very similar to how w3s.link and nftstorage.link work under the hood. The difference is just that instead of a blob storage API, they talk to an R2 bucket (cloudflare’s version of S3). We’re transitioning the gateways to not assume R2 bucket and to rely on fetching from claimed locations.
Note that you can use the same system to implement a bitswap server (this is what we will do with hoverboard - the evolution of Elastic IPFS).
What’s your vision for this? You said that one of the goals is to identify deps on centralised infra services, i.e. Dynamo, but I’m curious what you have in mind in terms of use-cases.
Not so much my vision as the collective vision for DAG House (team behind web3.storage and nft.storage) and Saturn. We’re building a decentralized hot storage/retrieval network, using web3.storage and it’s underlying tech as a base.
I wrote many more details here on the 2 projects if you’re interested: Spike: Self-Hosted Web3.Storage · Issue #1 · w3s-project/project-tracking · GitHub
