IPFS daemon doesn't add file after a while

Help
1

I’m using ipfs to add files on some external events, it’s running on ubuntu linux server. After a while though I inevitably run into the problem: ipfs stops adding files altogether. It just hangs the add with no return and timeout. Nothing happenes at all.

What can I do to diagnose it?

I’m using the latest version.
There’s no ERROR level log on the event. Will switch to INFO to see.

2

Hi salux-x,
Thanks for bringing this to our attention. Have few questions to narrow down the problem:

What is the size of the file/directory you are trying to add?
What command are you running when adding?
How far does the add operation make it before stalling?
Are you able to run other commands when the add it hanging? Try running ipfs id, does that work?

You may also find the comments in this issue helpful, specifically:

For now, when adding very large sets of data, its advisable to do so without the daemon running, or by using ipfs add --local. The issue is that you will clog up the DHT when adding such huge volumes of objects, and that slows everything else your node is doing down. It then also has the impact of consuming lots of extra memory.

Edit: Issue I forgot to link - ipfs add hangs and stalls with large folders · Issue #3885 · ipfs/kubo · GitHub

3

Btw, what does --local do exactly (I don’t see description in ipfs --help, ipfs add --help)? Will the blocks get advertised after the daemon restarts? Also, I thought that you would get ‘api not available’ error if you try to run any command while the daemon is not running (or am I misremembering)?

4

--local is a global command:

$ ipfs --help
...
 -L,   --local  bool   - Run the command locally, instead of using the daemon.
...

However it probably doesn’t belong there see – https://github.com/ipfs/go-ipfs/issues/4182
If you use the --local flag when adding, your daemon won’t broadcast the content to the network.

You are remembering correctly. However after staring my daemon then killing it I am able to run the add command and not receive the api not available error message, I think that might be dependent on some state with the api. What OS are you on?

1 Like
5
  1. To answer the question about the initial problem:
  • the file size I try adding isn’t relevant, it doesn’t add any file
  • I try using bash like ‘ipfs add ./file.txt’ - it shows
    [===============================>100%]file.txt
    and nothing more. it just stops and freezes like that.
    I tried using curl command with same effect.
  • it happens after some time with no apparent reason
  1. Now as I left server running overnight it shows me logs, I can see daemon running, but when adding file it says API not running. Also, id command works, but takes like 30 seconds to process.
1 Like
6

Now I get

Error: open /proc/self/status: too many open files

on

ipfs diag sys

7

I think I may still be missing some context, but here is my best guess at the problem - if the below does not lead to a resolution, please open an issue here: https://github.com/ipfs/go-ipfs/issues

Problems such as the one described above can happen when adding large files to ipfs, this is because too many file descriptors get opened. This can sometimes be avoided by using the --offline flag when starting the daemon or the --local flag when running the add command. There is discussion on trouble shooting this issue here - https://github.com/ipfs/go-ipfs/issues/3792 and here - https://github.com/ipfs/go-ipfs/issues/3885.
Once QUIC support is added this should happen less frequently.

8

Offline option is not acceptable since I need files I add to be downloadable, and the key difference is that I don’t get error
on opened descriptors (however I start using IPFS_FD_MAX=99999 and set ulimit to 65000), it just hangs on adding file, no error returned. I’ll post DEBUG log when adding file while problem is active.

for now script like this works for me:

timeout 10 /etc/init.d/ipfs add /root/ipfs/spam.log
if [ $? = 124 ] ; then
ps ax | grep ‘/etc/init.d/ipfs daemon’ | awk -F ’ ’ ‘{print $1}’ | xargs sudo kill -9

    echo 'kill success'

    sleep 3

    nohup sudo -u root IPFS_LOGGING=DEBUG IPFS_FD_MAX=10000 /etc/init.d/ipfs daemon --enable-pubsub-experiment > /var/log/rc.local2.log  2>&1 &

end

which as you can see reacts on hanging file add and reboots ipfs

9

Hi there,

we’ve currently ran into the same problem … we don’t have any idea why this is happening, the debug logs also say nothing error related… all normal … it suddenly stops adding files.

10

Did anyone ever solve this? I’m seeing this issue as well D: And we see it in adding empty files, files with 1B of data, etc.

What’s the difference between free_space and total_space in the ipfs sys diag? It seems like these numbers always differ by 16,777,216 which has some incrementing issues if it’s a float32.

11

What’s the difference between free_space and total_space in the ipfs sys diag? It seems like these numbers always differ by 16,777,216 which has some incrementing issues if it’s a float32.

Free space is the amount fo free space. Total is the total space. These are both uint64s and are returned by syscall.Statfs.

Did anyone ever solve this? I’m seeing this issue as well D: And we see it in adding empty files, files with 1B of data, etc.

It could be anything. You could be out of disk space, you could be running into Pinning is slow when there are many pins · Issue #5221 · ipfs/kubo · GitHub.