IPFS 0.4.13 on RPI: "protocol mux failed"

IPFS version: 0.4.13
Go version: go1.9 linux/arm
Hardware: Raspberry Pi 3
OS: Raspbian
Firewalls: none

ipfs init works just fine, daemon gets an interrupt.

$ IPFS_LOGGING=warning ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
05:37:08.157 WARNI  basichost: protocol mux failed: connection reset (took 251.376623ms) basic_host.go:258
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.2.6/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
Swarm listening on /p2p-circuit/ipfs/QmUDRSooVxHeRMkkp3EjRjwbw2yx82rYv7TKrxS7W6n3Wm
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/192.168.2.6/tcp/4001
Swarm announcing /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)

Could you try to do dmesg | egrep -i 'killed process' in another terminal and see if something comes up?

I’m guessing this is because of your Raspberry running out of memory and the oom killer decided to shutdown go-ipfs.

Blank - no results. I’m not surprised, though - there’s nothing else running on that Pi. Also, IPFS used to work pretty well on that very RPI just a few releases ago (don’t remember which…), so I’m presuming something changed in how the daemon works since then.

Looks like running the daemon as root works… Any ideas why this is the case, anyone?

IPFS_PATH=/home/pi/.ipfs sudo ipfs daemon

^^ Actually, not quite true. Error persists. I have no clue as to why it worked for a short period of time… Very confusing.

Are you getting the “received interrupt signal” message when you log out of or get disconnected from your terminal session? Have you tried running it inside of screen or tmux session?

I get the message on the fly - I don’t even need to do anything (like logging out of the session) in order to see it. Also, running it from Tmux didn’t make any difference.

IPFS_LOGGING=warning ipfs daemon
Initializing daemon...
19:02:20.155 WARNI  basichost: protocol mux failed: %s (took %s)connection reset 111.743152ms basic_host.go:135
19:02:20.155 WARNI  basichost: protocol mux failed: %s (took %s)connection reset 111.438206ms basic_host.go:135
19:02:20.160 WARNI  basichost: protocol mux failed: %s (took %s)connection reset 90.597836ms basic_host.go:135
19:02:20.187 WARNI  basichost: protocol mux failed: %s (took %s)connection reset 116.858011ms basic_host.go:135
19:02:20.567 WARNI  basichost: protocol mux failed: %s (took %s)connection reset 255.588625ms basic_host.go:135
19:02:20.642 WARNI  basichost: protocol mux failed: %s (took %s)connection reset 330.34694ms basic_host.go:135

Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)
19:02:21.627 WARNI  basichost: protocol mux failed: %s (took %s)unexpected EOF 308.126789ms basic_host.go:135
19:02:21.627 WARNI        dht: checking dht client type: EOF notif.go:40
Swarm listening on /ip4/xx.xxx.xxx.xx/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
19:02:21.818 ERROR   cmd/ipfs: cannot access config, repo not open daemon.go:419
19:02:21.818 ERROR   cmd/ipfs: cannot access config, repo not open daemon.go:419
Error: cannot access config, repo not open

That’s interesting. I tried reproducing the error on a linux VM (I don’t have a Raspberry Pi 3) limited to 1GB of memory with no page file and didn’t encounter the process getting terminated.

However, I simply may not have waited long enough or tested the same commands while the daemon was running; I just let two ipfs refs -r <hash> commands run for a bit.

FWIW I do get the WARNI basichost: protocol mux failed: connection reset message on at least one of my nodes that runs non-stop, so while I see that in your daemon output I’m guessing it’s not related.

If raspbian has strace could you run the following command to try to figure out what’s killing the process?

strace -e signal -o tracedump ipfs daemon

After the daemon gets an interrupt signal, there might be a line in the tracedump file that looks like the following line which should have the pid of the process that sent the signal (25454 in the example below).

--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=25454, si_uid=1000} ---

If there is, find out what the process is using the pid (<pid>) from si_pid above

ps -o cmd <pid>

Thanks for your help, @leerspace. My bet is that the issue is related to the architecture (ARM), not RAM.

In any case, I followed the steps described in your answer but the result of ps for the given pid was empty.

Tomorrow I’ll try it on a different RPi (fresh installation) and see what happens.

Finally got it working. Looks like the problem was in the executable’s file system permissions. Weird.

1 Like

Hi, I have to re-open this issue because I have the same problem in RPI2. I am working with IPFS version 0.4.9 (my RPI is full-upgraded) and every time that I start the daemon it shows a message (without doing ctrl + c):

Daemon is ready

Received interrupt signal, shutting down…
(Hit ctrl-c again to force-shutdown the daemon.)

I don’t know what exactly you mean with file system permissions, if you can clarify to me it would be a great help.

It’s been a while, but from what I remember I had to set explicit ownership and permissions on the IPFS executable:

Hope that helps.