Ipfs-cluster sharding and recursive/partial pin

Hi there,

I know that sharding is currently not accessible, and according to issue#5133, issue#5142, the main reason goes to poor support for partial pin. However, I can’t completely get why partial pin is a must for sharding.

So far as I know, in single mode (not sharding), Finalize() will trigger peers in dests to pin DAG root node, and I suppose it’s recursive pinning here which can pin this whole DAG all the way down perfectly. While in sharding mode, every shard can Flush() to pin a cbor-wrapped map (recording cid of block belonging to this shard) node, and this whole dgs can Finalize() to pin a cbor-wrapped map (recording all shard’s cids) node.

Here’re my questions:

  1. if the cbor-wrapped map is also pinned recursively, why would it influence sharding? Is that because some child of this node is an internal node of origin data DAG, and recursive pinning here needs to pin blocks not belonging to this shard?

  2. Or doe it have something to do with the depth of shard DAG, and would recursive pin still be a problem if all those maps (shard.dagnode, dgs.shards) don’t exceed the MaxLink() limits, i.e. always trigger makeDAGSimple() to wrap?

Thanks for your patience, hope to hear from you guys soon.


Say you add some files/folders


This results in a DAG with 6 dag.protobuf nodes which are produced as you add in the following order:

[Node(1.txt), Node(2.txt), Node(a/), Node(3.txt), Node(b/), Node(root)]

Where the a/ b/ and root/ node have links to children because they are folders.

Sharding will create buckets for this nodes as they produced, adding them to the same bucket until it’s full. For example:

Node(Shard1): [Node(1.txt), Node(2.txt)]
Node(Shard2): [Node(a/), Node(3.txt)]
Node(Shard3): [Node(b/), Node(root)]
plus a Node(ClusterDAG) with links to each shard Node.

Once a shard is full it is pinned. Ideally it would be pinned in a depth-limited mode to a max-depth of 1. It is ok to pin Node(Shard1) recursively because it is pointing to leaves. But pinning Node(Shard2) recursively would end up also pinning Node(1.txt) and Node(2.txt), since those are children of the Node(a/) dag.protobuf node. But those nodes are in a different shard and should be pinned in a different place corresponding to Shard1 only. That would only happen if Node(Shard2) could be pinned in “depth+1” limited mode, as it would then avoid following the links from Node(a/) to children.

The Node(ClusterDAG) is pinned in direct mode so this one poses no problem.

While you could process a full DAG and then make decisions on how to split it in ways that you can pin subroots recursively, the current approach was the best we could come up with to shard as the data is getting streamed into the cluster, therefore not knowing the final dag structure (nor it’s size).

You could also pin everything is direct mode individually, but this is unrealistic in both number of requests to pin needed and performance of an IPFS pinset that would explode in size.

Hope I clarified?

That’s what I currently use in my setup. It’s not that pretty, but works great performance wise.

@SwJay I run a mirror for some Linux packages, where every file is individually pinned and in the MFS is the directory structure maintained in parallel.

The cluster pinset has 9845 pins and the pinset is many many thousands in commit height, since currently every single add or remove from the pinset adds one commit.

This will be changed in the future, which would reduce the pinset commits to a couple of dozends maximum per day.

Since until 0.13 pins couldn’t be added as direct, only as recursive, I haven’t pushed the directories itself to the cluster. I will change this soon.

Yes, it’s definitely helpful. Thanks a lot!

So currently, a feasible solution for sharding would be to pin every data node and shard node in direct mode, like triggering ba.Pin() every time sharding dgs calls ingestBlock()?

However, I’m still not sure how to pin those nodes in direct pin mode. I’ve noticed that PinType in Pin indicates which pin strategy should it be dealt with, but I failed to find codes dealing with these PinType differently, especially on how to connect them with direct/indirect/recursive pin?

It’s good to hear someone doing so, do you mind sharing how you setup direct pin for every node? Have you modified the source code, or just the config file?

Don’t waste time pinning everything in direct mode. If you need sharding, you are likely dealing with large amounts of data (really big pins that do not fit a single node), or with really small nodes (unable to handle really big pins because they are underspecced).

Injecting a direct pin for every block in an object gives you ~4000 direct pins per Gigabyte. How big are your pins and how big are the nodes that you want to use? Pinning in direct mode not only will be painfully slow (4000 pin API calls per GB, each of them locking the pinning system, loading the direct-pin DAG into memory, inserting then saving), but likely to completely clog IPFS: 1TB of data will need 4 M of pins and it is going to be super slow to do anything on that pinset with the current pinset implemenation in IPFS.

Thanks again for your kindly reply. Hope to see your “MaxDepth” pin mode works in practice soon!