V0.4.0 - Unable to pin large directories

Help Old Support
21

From @Kubuxu on Fri Jul 29 2016 18:02:38 GMT+0000 (UTC)

That huge mistake on our part. We didnā€™t increase version of go-ipfs in the binary to 0.4.3-rc1 from 0.4.3-dev. For now you can try updating manually from https://dist.ipfs.io

cc @whyrusleeping

22

From @sdockray on Sat Jul 30 2016 05:14:21 GMT+0000 (UTC)

fyi, i downloaded the RC manually and

% tar xvfz go-ipfs_v0.4.3-rc1_linux-amd64.tar.gz
% mv go-ipfs/ipfs /dir/work/bin/
% ipfs version
ipfs version 0.4.3-dev

will just work with dev version for now

23

From @Kubuxu on Sat Jul 30 2016 09:07:04 GMT+0000 (UTC)

It really is rc1, we just made a mistake and didnā€™t update the name of the version in the code.

24

From @sdockray on Tue Aug 02 2016 09:37:31 GMT+0000 (UTC)

Iā€™ve re-added everything and then tried to pin the root directory again and unfortunately am still having it fail

ipfs pin add QmevLmheGTxdZSQ9Vwwnb2SkY4n9SNFiznYWjRKY5z3KEK > ipfs_pin.log &
... after 30-60 minutes ...
Error: Post http://127.0.0.1:5001/api/v0/pin/add?arg=QmevLmheGTxdZSQ9Vwwnb2SkY4n9SNFiznYWjRKY5z3KEK&encoding=json&stream-channels=true: EOF
25

From @whyrusleeping on Wed Aug 03 2016 18:25:04 GMT+0000 (UTC)

@sdockray when that failure happens, does the daemon print anything?

26

From @sdockray on Fri Aug 05 2016 01:13:08 GMT+0000 (UTC)

@whyrusleeping: No, the daemon exits without outputting anything

27

From @sdockray on Fri Aug 05 2016 01:17:21 GMT+0000 (UTC)

As a general question, what is the largest directory anyone has pinned? Or would it have more to do with the number if items to be pinned (during recursion)? If you have about 1/2TB free you can test with the hash above (although it seems unlikely that youā€™ll actually pull more than a few gb before it fails)

28

From @whyrusleeping on Fri Dec 09 2016 22:30:40 GMT+0000 (UTC)

@lgierth recently pinned the CCC archives, which i think was over 2TB.

29

From @lgierth on Fri Dec 09 2016 22:38:07 GMT+0000 (UTC)

3.7 TB to be precise, but that was a plain add with the default of --pin=true, as opposed to a fetch + pin.

30

From @whyrusleeping on Fri Dec 09 2016 22:39:51 GMT+0000 (UTC)

I think the problem here is merkledag.FetchGraph, It has unbounded concurrency IIRC. Iā€™ll push a quick fix for thisā€¦

31

From @xloem on Mon Jan 09 2017 00:43:05 GMT+0000 (UTC)

Iā€™m still having this issue with 0.4.5-dev . When trying to pin a large folder, ipfs uses up over 3 gigs of ram and is then killed by oom_killer.

32

From @xloem on Mon Jan 09 2017 00:49:37 GMT+0000 (UTC)

I guess this is pretty much https://github.com/ipfs/go-ipfs/issues/3318

33

From @xloem on Mon Jan 09 2017 11:50:44 GMT+0000 (UTC)

I gave my system a ton of virtual ram and let my pin run overnight. Itā€™s still working on the pin ! And using 8.7 gigs of ram. I took the profiling stats which are at https://gateway.ipfs.io/ipfs//QmZmseT7n9MptemPcxkuWh9FeAPK7rPeTNMDiPQWTft4Qo . Writing them out was very slow. I had to forcibly kill and reboot the daemon to complete the add.

34

From @xloem on Mon Jan 09 2017 12:02:03 GMT+0000 (UTC)

I donā€™t know go, but it seems notable that there are over 4500 goroutines in ipfs.stack that are created by gx/ipfs/QmVwFjMdejJ8mGVmgyR2mKcUHrvNBDtDsKRT99soVbkFhA/go-peerstream.(*Swarm).notifyAll. Doesnā€™t quite sound like normal network app behavior.