Garbage Collection Fails

I’m running go-ipfs v0.14.0. I have a public gateway exposed, and it receives a lot of calls that quickly fill up the hard drive. I’ve got a nginx rate limiter in front of the gateway, to limit the amount of calls, but these anonymous calls to the gateway still cause the node to fill up the hard drive after a few days.

I’m trying to combat this by running the garbage collector. However, when I run this command:

  • ipfs repo gc

It exits with this error:

  • Error: EOF

So running the garbage collector is not an effective way to deal with the hard drive filling up.

Can anyone shed light on why the garbage collector is failing? Any workarounds or potential solutions to help mitigate the filling of the hard drive?

@christroutner is the daemon crashing when that happen ? (could you check your logs please)

if it does please send me the panic logs in dms (could be a security issue, or just running out of ram idk).

Thanks for replying @Jorropo

The daemon was not crashing.

What I ended up doing was wiping the data directory and starting over from scratch. The garbage collector started operating and finishing successfully after I did that. I will continue to check on it a couple times a day, and I’ll report if it goes off the rails again. I’ll also keep an eye on the logs and paste them here.

In addition to running with the --enable-gc flag, I’ve also got a cron job that is running the ipfs repo gc command every hour.

I’ve got rate limits on my IPFS gateway set to 2 requests per second. I’m waiting to see if this is enough ‘defense’ to prevent the system hard drive from filling up. I’ll report back here over the next couple days.

After waiting a day, I was able to replicate the EOF error. As requested, I captured the the logs from IPFS. It seemed to have a core dump or some other kind of massive failure. The logs were large and filled up my terminal buffer. But what I was able to capture, I added to this gist.

In this case, the garbage collection started off well. I suspect it may have run out of memory. The VPS I’m running it on has 2GB of RAM and 8GB of swap memory.

Despite running the ipfs repo gc multiple times, the amount of data on the disk does not appear to be decreasing. I was hoping that go-ipfs would fail elegantly by removing a significant amount of data before crashing. That would allow me to achieve a sustainable disk usage eventually, but alas, that is not happening.

You are using an old version of Kubo (currently 0.16.0).

That said, 2GB of RAM is usually too little for a gateway node. If you can, lower the GC threshold and increase the GC interval in the config. The only way you stand a chance is if GC runs very often as much as possible and clears as little as possible on every run.

1 Like

I appreciate the tips. I’ll make the configuration changes you suggest.

Is there some way to operate a gateway and only serve content that the node has pinned?

That would solve my particular use case. The only reason I’m operating a gateway is to give access to the pinned content. I would prefer not to serve all the data on IPFS, especially if it comes with a big hit to performance.

Yes: kubo/config.md at master · ipfs/kubo · GitHub

1 Like

Oh, wow! That’s awesome. Exactly what I needed. Thank you!

this is a small extract.
You really need the first few 20 (ideally all of it) goroutines of the stack trace.
Because the first goroutines explain why it panics, others just list anythign that running.