Ipfs-cluster pin add error

Help IPFS Cluster
1

I have just build a private network / ipfs-cluster with 3 nodes.

After adding 224GB of data, I receive an error when I want to distribute pinning.

ipfs-cluster-ctl pin add Qmam8wSeLVqGAjpKfVrRtrhHMjbBrd3jDCPRkJjfrq7iwf
Qmam8wSeLVqGAjpKfVrRtrhHMjbBrd3jDCPRkJjfrq7iwf :
> <peer.ID QmFQ47js> : CLUSTER_ERROR: Time.UnmarshalBinary: unsupported version | 2019-04-16T22:37:51.979418655+02:00
> rock64 : PINNED | 2019-04-16T22:37:50.905047818+02:00
> <peer.ID QmvtHzKV> : CLUSTER_ERROR: Time.UnmarshalBinary: unsupported version | 2019-04-16T22:37:51.979861392+02:00

ipfs-cluster-ctl peers ls show the 2 other nodes connected.
3 new pins only were transmitted to other nodes ( ipfs-cluster-ctl pin ls )

  • arm node
    /usr/local/bin/ipfs daemon --enable-gc --mount --routing=dhtclient
    “StorageMax”: “5GB”

  • amd64 node
    /usr/local/bin/ipfs-cluster-service daemon
    “StorageMax”: “700GB”

  • arm64 node
    /usr/local/bin/ipfs-cluster-service daemon
    “StorageMax”: “1.5TB”

2

hey @qo-op, can you open an issue in the repo? Can you include these errors, the logs of the clusters peers and the versions ?

3

Please
Could you tell me how to extract log? And which repo link to use…

4

Logs are what ipfs-cluster-service is printing. journalctl -u ipfs-cluster-service perhaps. Depends on your system.

5

That’s it

Apr 15 02:37:33 rock64 ipfs-cluster-service[32562]: 02:37:33.386 INFO ipfshttp: IPFS Pin request succeeded: QmNdxSBt5PXN2dpk7F48ZkP9hYUwLnxDy4MCANdciVffs9
Apr 15 14:14:14 rock64 ipfs-cluster-service[32562]: 14:14:14.913 ERROR libp2p-raf: Failed to flush response: stream reset transport.go:37
Apr 15 17:50:51 rock64 ipfs-cluster-service[32562]: 17:50:51.068 ERROR libp2p-raf: Failed to decode incoming command: stream reset transport.go:37
Apr 15 17:50:51 rock64 ipfs-cluster-service[32562]: 17:50:51.069 ERROR libp2p-raf: Failed to decode incoming command: stream reset transport.go:37
Apr 16 19:44:48 rock64 ipfs-cluster-service[32562]: 19:44:48.520 ERROR libp2p-raf: Failed to decode incoming command: stream reset transport.go:37
Apr 16 20:41:11 rock64 ipfs-cluster-service[32562]: 20:41:11.519 ERROR libp2p-raf: Failed to decode incoming command: stream reset transport.go:37
Apr 16 20:41:48 rock64 ipfs-cluster-service[32562]: 20:41:48.030 WARNI cluster: Peer <peer.ID QmW5Jnk6> received alert for freespace in <peer.ID QmvtHzKV
Apr 16 20:42:29 rock64 ipfs-cluster-service[32562]: 20:42:29.055 ERROR raft: NOTICE: Some RAFT log messages repeat and will only be logged once logging.
Apr 16 20:42:29 rock64 ipfs-cluster-service[32562]: 20:42:29.060 ERROR raft: peer {Voter QmYgYN2JYLjkGBjU17RbYvy67EJo6j6eWLsWpCi3vtHzKV QmYgYN2JYLjkGBjU
Apr 16 20:42:36 rock64 ipfs-cluster-service[32562]: 20:42:36.024 ERROR libp2p-raf: Failed to decode incoming command: stream reset transport.go:37
Apr 16 22:37:50 rock64 ipfs-cluster-service[32562]: 22:37:50.737 INFO cluster: IPFS cluster pinning Qmam8wSeLVqGAjpKfVrRtrhHMjbBrd3jDCPRkJjfrq7iwf everywh
Apr 16 22:37:51 rock64 ipfs-cluster-service[32562]: 22:37:51.978 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmFQ47js>:
Apr 16 22:37:51 rock64 ipfs-cluster-service[32562]: 22:37:51.979 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmvtHzKV>:
Apr 16 23:07:51 rock64 ipfs-cluster-service[32562]: 23:07:51.129 INFO cluster: IPFS cluster pinning QmcxnYZm3B3Df4nbut4qHUfGRRsksJyxF9capnuCj3dTyK everywh
Apr 16 23:07:52 rock64 ipfs-cluster-service[32562]: 23:07:52.448 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmFQ47js>:
Apr 16 23:07:52 rock64 ipfs-cluster-service[32562]: 23:07:52.449 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmvtHzKV>:
Apr 16 23:07:55 rock64 ipfs-cluster-service[32562]: 23:07:55.712 INFO ipfshttp: IPFS Pin request succeeded: QmcxnYZm3B3Df4nbut4qHUfGRRsksJyxF9capnuCj3dTyK
Apr 16 23:08:16 rock64 ipfs-cluster-service[32562]: 23:08:16.081 INFO cluster: IPFS cluster pinning QmQyrX2YsM2u8V1R8hqzELjWBtbumSC4XkfbwLVQbHk4Xb everywh
Apr 16 23:08:16 rock64 ipfs-cluster-service[32562]: 23:08:16.741 INFO ipfshttp: IPFS Pin request succeeded: QmQyrX2YsM2u8V1R8hqzELjWBtbumSC4XkfbwLVQbHk4Xb
Apr 16 23:08:17 rock64 ipfs-cluster-service[32562]: 23:08:17.279 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmFQ47js>:
Apr 16 23:08:17 rock64 ipfs-cluster-service[32562]: 23:08:17.279 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmvtHzKV>:

Apr 16 23:07:04 rec ipfs-cluster-service[14440]: 23:07:04.290 WARNI cluster: Peer QmUrgKrvV71EaBnNyCjPm2u3B1YGtj2YyC87q98yFQ47js received alert for freespa
Apr 16 23:07:51 rec ipfs-cluster-service[14440]: 23:07:51.326 INFO consensus: pin committed to global state: QmcxnYZm3B3Df4nbut4qHUfGRRsksJyxF9capnuCj3dTyK
Apr 16 23:08:16 rec ipfs-cluster-service[14440]: 23:08:16.226 INFO consensus: pin committed to global state: QmQyrX2YsM2u8V1R8hqzELjWBtbumSC4XkfbwLVQbHk4Xb
Apr 16 23:08:17 rec ipfs-cluster-service[14440]: 23:08:17.461 INFO ipfshttp: IPFS Pin request succeeded: QmQyrX2YsM2u8V1R8hqzELjWBtbumSC4XkfbwLVQbHk4Xb ip
Apr 16 23:08:21 rec ipfs-cluster-service[14440]: 23:08:21.338 INFO ipfshttp: IPFS Pin request succeeded: QmcxnYZm3B3Df4nbut4qHUfGRRsksJyxF9capnuCj3dTyK ip
Apr 16 23:09:04 rec ipfs-cluster-service[14440]: 23:09:04.290 WARNI cluster: Peer QmUrgKrvV71EaBnNyCjPm2u3B1YGtj2YyC87q98yFQ47js received alert for freespa

Now it is clear that the small node is creating problem…
Is it a bug? Not really…
I am trying to remove it from cluster, and just keep it in private network only. and make a clearer bug report…

In my case, plenty small node will be adding files and use Lead nodes to pin it all and maintain a base for deduplication and file availability to small nodes (old files are removed from local and found on IPFS if needed later).

I am afraid that if one small node become low space it limits cluster running for all.
Maybe, I’ll have to use subpub channel to transfer pin orders.to big leader and manage local files FIFO to IPFS in another way.

6

After stopping arm node ipfs-cluster

I receive on arm64
ipfs-cluster-ctl peers ls
An error occurred:
Code: 200
Message: input isn’t valid multihash

Apr 16 23:08:16 rock64 ipfs-cluster-service[32562]: 23:08:16.741 INFO ipfshttp: IPFS Pin request succeeded: QmQyrX2YsM2u8V1R8hqzELjWBtbumSC4XkfbwLVQbHk4Xb
Apr 16 23:08:17 rock64 ipfs-cluster-service[32562]: 23:08:17.279 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmFQ47js>:
Apr 16 23:08:17 rock64 ipfs-cluster-service[32562]: 23:08:17.279 ERROR cluster: <peer.ID QmW5Jnk6>: error in broadcast response from <peer.ID QmvtHzKV>:
Apr 17 00:23:33 rock64 ipfs-cluster-service[32562]: 00:23:33.878 ERROR p2p-gorpc: dial attempt failed: <peer.ID QmW5Jnk6> → <peer.ID QmvtHzKV> dial attem
Apr 17 00:23:44 rock64 ipfs-cluster-service[32562]: 00:23:44.650 ERROR p2p-gorpc: dial attempt failed: <peer.ID QmW5Jnk6> → <peer.ID QmvtHzKV> dial attem
Apr 17 00:23:46 rock64 ipfs-cluster-service[32562]: 00:23:46.530 ERROR p2p-gorpc: dial backoff call.go:63

  • from amd64 (leader - bootstrap)

ipfs-cluster-ctl peers ls
QmUrgKrvV71EaBnNyCjPm2u3B1YGtj2YyC87q98yFQ47js | rec | Sees 2 other peers

Addresses:
- /ip4/127.0.0.1/tcp/9096/ipfs/QmUrgKrvV71EaBnNyCjPm2u3B1YGtj2YyC87q98yFQ47js
- /ip4/51.15.166.54/tcp/9096/ipfs/QmUrgKrvV71EaBnNyCjPm2u3B1YGtj2YyC87q98yFQ47js
- /p2p-circuit/ipfs/QmUrgKrvV71EaBnNyCjPm2u3B1YGtj2YyC87q98yFQ47js
IPFS: Qmb6Gyy3KFWTxWh4v6fBeh5ANxAF6YAahWQtokmsNsqGda
- /ip4/127.0.0.1/tcp/4001/ipfs/Qmb6Gyy3KFWTxWh4v6fBeh5ANxAF6YAahWQtokmsNsqGda
- /ip4/51.15.166.54/tcp/4001/ipfs/Qmb6Gyy3KFWTxWh4v6fBeh5ANxAF6YAahWQtokmsNsqGda
- /ip6/::1/tcp/4001/ipfs/Qmb6Gyy3KFWTxWh4v6fBeh5ANxAF6YAahWQtokmsNsqGda
QmWnDnNYgP6rhWeMxUPETgjkZowoNCvja9r5PNspW5Jnk6 | rock64 | Sees 2 other peers
Addresses:
- /ip4/127.0.0.1/tcp/9096/ipfs/QmWnDnNYgP6rhWeMxUPETgjkZowoNCvja9r5PNspW5Jnk6
- /ip4/192.168.1.34/tcp/9096/ipfs/QmWnDnNYgP6rhWeMxUPETgjkZowoNCvja9r5PNspW5Jnk6
- /ip4/90.89.174.105/tcp/11828/ipfs/QmWnDnNYgP6rhWeMxUPETgjkZowoNCvja9r5PNspW5Jnk6
- /p2p-circuit/ipfs/QmWnDnNYgP6rhWeMxUPETgjkZowoNCvja9r5PNspW5Jnk6
IPFS: QmYZJoSmkPZZZmLKpq26hSo1i49tT752Z4ERx6SSXhbSjP
- /ip4/127.0.0.1/tcp/4001/ipfs/QmYZJoSmkPZZZmLKpq26hSo1i49tT752Z4ERx6SSXhbSjP
- /ip4/192.168.1.34/tcp/4001/ipfs/QmYZJoSmkPZZZmLKpq26hSo1i49tT752Z4ERx6SSXhbSjP
- /ip4/90.89.174.105/tcp/11021/ipfs/QmYZJoSmkPZZZmLKpq26hSo1i49tT752Z4ERx6SSXhbSjP
- /ip6/2a01:cb19:8344:f00:c210:4408:7a94:db90/tcp/4001/ipfs/QmYZJoSmkPZZZmLKpq26hSo1i49tT752Z4ERx6SSXhbSjP
- /ip6/::1/tcp/4001/ipfs/QmYZJoSmkPZZZmLKpq26hSo1i49tT752Z4ERx6SSXhbSjP
QmYgYN2JYLjkGBjU17RbYvy67EJo6j6eWLsWpCi3vtHzKV | ERROR: dial backoff

7

Continued in github issue

1 Like