IPFS using too much disk space

Help
1

Hey,

We have an IPF hosting our generative arts. I tried to configure it to use 300GB max, and it’s running with --enable-gc flag. We restart the server every few hours. I tried running GC manually but it always failed with Error: EOF. The instance is running with a HDD.

Currently on the webui/files what I see is “123.5GiB FILES - 1.2TiB ALL BLOCKS”. So I have few questions,

  • Why my IPFS instance might not be respecting the storagemax value? Is it just always failing to GC? But then if it’s already above storagemax, why it’s still caching files?
  • What can I make to make our IPFS instance less accepting of this random blocks? I want to contribute to IPFS network, but I really need it to stay in the usage limits I define.
  • What happens if the disk is full? Right now we just have few gigs of space remaining and I’m not sure if I should shut down the server before it hits 100%.

Here’s the IPFS config we are using: IPFS Config · GitHub

Thank you all!

2

Why my IPFS instance might not be respecting the storagemax value?

storagemax isn’t a maximum of storage, it’s a target for the GC.

Is it just always failing to GC?

Yeah maybe, if you have files pinned see ipfs pin ls or in the mfs see ipfs files ls / they can’t be removed.

But then if it’s already above storagemax, why it’s still caching files?

IPFS’s codebase is almost incapable of streaming blocks without storing them somewhere, for example if you fetch a file it is very likely you will reuse the roots / manifest blocks again to fetch an other part of that file, caching things speeds up things a lot.

What can I make to make our IPFS instance less accepting of this random blocks?

You don’t, IPFS doesn’t have such feature.
IPFS only shares:

  • Files you add to your node.
  • Files you pin to your node.
  • Files you download (for a temporary amount of time until next GC).

What happens if the disk is full?

With:

  • FlatFS: It errors.
  • Badgerds: it likely corrupt itself half writing vlogs.

What to do ?

You should check your pins and MFS, it is very likely most of the storage is exampted from GC and so can’t be removed.
Then after removing things, you can force a gc with:

ipfs repo gc
2 Likes