Developing an open source remote pinning middleware with quota

Ecosystem and Usage
1

Hi

I hope it will not be treated as spam. I started writing a remote pinning service that is put in front of ipfs-cluster, this allows:

  • users have a dedicated quota they can pin
  • they can directly pin a CID from the service
  • they can put a pinned CID as “public” and it allows a public profile page containing a list of all remote pins (I know all data over IPFS is public, but the wording may be bad)
  • integrates with ipfs command
  • uses bearer tokens for authentication
  • deduplication for pinned data per user

The service is exposed on the network and exposes a pinning API, then applies some logic to see if an user is authenticated and have enough quota, then put the pin request in queue. An internal worker takes the queue and start querying a kubo API to fetch information about the CID and its graph, if all the blocks were reachable and the quota is not overpass, the pinning request is sent to ipfs-cluster.

The project is open source under MIT licence, mostly generated with AI help. There were complicated steps like not accepting blindly a CID, requiring to crawl every block from the dag and calculate the real size by small chunks, in case someone’s forged a fake CID with a really huge size. Deduplication is taken in account per user, which was easy once the crawling was done. There is a local database to store all the CID / size and graph for each pinned CID, allowing to quickly pin a already pinned by a previous user or from a subset of something pinned in the service.

Here are some screenshots, tell me what you think, I’m testing a bit more in the wild now and I didn’t try much the code path with multiple users. And authentication is done on a reverse proxy because I trust it more than my own code to handle authentication (the bearer token is a bit easier to handle). The source code will be available after I reworked a bit the history, I’m ashamed by all those “updates” commit :sweat_smile:

As it is a middleware, it should possibly be possible to chain this service. So if you have a quota on an instance, you could rehost your own quota service authenticated as your user on the remote one, and divide that quota into your own users. (not tested but in theory this should work).

I have no idea if it would be useful to someone, but I liked the idea and wanted to implement it.

Please share what you think. :+1:

Alongside this, I wrote a python nautilus extension for GNOME users so they can directly import stuff from their local disk to IPFS + pin it if a remote service is defined: See public/nautilus-ipfs: Context menu to add content to IPFS - Forgejo: Beyond coding. We Forge.

Trivia: I started all of this because I made a private IPFS swarm for fun but this wasn’t useful, so I started thinking at how to make this useful for regular users. This required an IPFS remote pinning service but I only found paid service and closed source software that would give authentication and quota… So I made this, and some helpers to make it “easy to use” ^^ The current IPFS webui does not help much seeing what is pinned on a remote, which was also an incentive…

user dashboard
user dashboard1920×1288 248 KB

public profile
public profile1198×426 34.7 KB

Public profile used in the screenshot URL is Public pins

2

The project is now published as open source :tada:

I saw on the forum someone was looking for a remote pinning service that could allow uploading files/directories from the UI, I implemented it.

add-dir-from-middleware
add-dir-from-middleware1230×465 69 KB

3

nice! and good timing: I was looking for something like this, but Protocol Labs stopped investing in IPFS and I’d given up on seeing any pinning innovation that isn’t tied to their shitcoin :sad_but_relieved_face:

quick feedback:

  1. is the crawler’s kubo the same node cluster pins to? if not, every DAG gets fetched twice?
  2. resolveCIDSizeFromCrawl walks the whole DAG before the quota check, so 10GB of quota can pull a 1TB DAG :weary_cat: consider reading the cumulative size from the root block (ipfs files stat /ipfs/cid --enc=json | jq .CumulativeSize which is sum of Tsizes on the links) and failing fast when it self-certifies as too big.

:+1:

4

It is configurable with a variable, I personally use the my gateway ipfs daemon

normally it should crawl the API to retrieve blocks by small chunks and stop if the quota / max single CID is exceeded. In practice, it seemed to work fine to me.

I don’t really want to rely at all on an associated size with a CID, but maybe this could help genuine requests to be handled faster.

I added some features recently:

  • there is a /admin route API to show the delta between what is pinned on the middleware and what is pinned on the backend (ipfs-cluster for me), which can help ensuring it’s working, there is an equivalent API call that pins what is missing on the remote
  • it’s now possible to pin a CID or upload content and make it public directly instead of a 2 step process, my idea is to use it for publishing content from CI and advertise it with an RSS feed