How to auto-rebalance and duplicate data?

Help
1

Suppose I have a small cluster, and it isn’t at all uniform, so there are drives of different sizes. I have a data processing task, and it uses more than any one single drive can hold on it’s own. Across the many drives, there is plenty of space however. I would like IPFS to automatically perform some form of balancing and distribution of pinned data. Also, I would like to configure that it implicitly duplicate anything that is pinned by a parameter I give, and ensure that duplication spreads across several drives.

How can I configure IPFS to auto-rebalance and duplicate my data when I pin it?

3

See https://cluster.ipfs.io.

4

Actually, after I wrote this I did go and read about cluster ipfs. That solves part of my problem, the duplication problem. I still need to find a way to make sure that data is siphoned to the larger drives.

5

Create an LVM volume or RAID array for your disks.

6

I can see that I can manage the replication with this, but not the rebalancing.

7

That only works for the extent that I have a single machine with many disks. My data center may be very unbalanced in terms of hardware.