I have to transfer data between HPC clusters and have tried IPFS but performance was very bad. I think the issue here is that the file system used (lustre) really doesn’t like large amounts of small files, so the default block store slows everything down.
Are there ways to store the blocks that are friendlier to these kind of file systems?