I have to transfer data between HPC clusters and have tried IPFS but performance was very bad. I think the issue here is that the file system used (lustre) really doesn’t like large amounts of small files, so the default block store slows everything down. Are there ways to store the blocks that are…

If the file-system on which your IPFS repo is stored is the issue, it may be worth trying one of the other IPFS storage backends, such as pebbleds. See Kubo CLI | IPFS Docs It might also be worth trying to stress the file-system by using flatfs: if that doesn’t change the observed behaviour then it…

Block stores for cluster file systems

gkraemer December 17, 2024, 10:33am 2

I have done some reading and there used to be some general performance caveats [1]. Have these been fixed?

[1] How to tune a private IPFS swarm for large files?

Topic		Replies	Views
Any plans to increas maximum block size? Help	4	515	February 16, 2024
Private IPFS with > 200GB Files Help go-ipfs	3	449	August 16, 2023
IPFS for files over 100GB Help go-ipfs , ipfs-cluster	2	679	September 18, 2021
May I ask the choice of ipfs datastore Help files	1	427	December 17, 2021
How to tune a private IPFS swarm for large files? Help kubo	16	738	November 14, 2025

Block stores for cluster file systems

Related topics