Behavior I can't explain

Help IPFS
1

I created a standard IPFS service on a 1TB server and added thousands of videos to it. ipfs repo stat --human reports:
NumObjects: 2294713
RepoSize: 585 GB
StorageMax: 780 GB

The daemon is running and responding to API some CLI requests, however if I do an ipfs pin ls and wait an hour it just hangs.

I use a script that creates a sqlite database that includes the hash of every video I add to IPFS. After trying to copy files onto another IPFS server elsewhere on the net with “ipfs files cp /myfolder/video.mp4”, the command just hangs. I waited overnight, still hung. Yet I can do an ipfs swarm connect or ipfs ping to the source server without a problem. I did so and reran the copy op and stull hangs, and the connection goes away after a short period, but not until sometime after I start the cp.

So I went back to the source server and tried to do a simple “ipfs cat /ipfs/hash” and that hung too, despite that file being pinned locally.

Then I questioned where the sqlite database was corrupted, so I tried to list the files that were pinned and it just hung.

I’m using ipfs version 0.4.21 on the source server and 0.4.22 on the destination, not that what’s on the destination version could cause what I’m seeing on the source.

2

if I do an ipfs pin ls and wait an hour it just hangs

That’s because you have a huge number of pinned block if you count the indirect pins. The problem is that go-ipfs compute the full list before starting to output. In the next release you will be able to run “ipfs pin ls --stream” to start output value as soon as possible.

You can also run “ipfs pin ls --type=recursive” to not compute all the indirect pinned blocks, which will be way way faster.

If you want to copy data from nodes to nodes, I suggest you give a try to GitHub - INFURA/ipfs-pump: A tool to transfer data between IPFS nodes or storage

2 Likes
3

That’s exactly what I need!

Thank you Michael, greatly appreciated!!! Does it add to existing ipfs repo or replace it?

Well, it would be if it would build without errors. I tried the most recent version of go (1.13.5) and 1.13.4 on linux amd64:

ipfs@ronnyc:~/scripts/ipfs$ go get -u github.com/INFURA/ipfs-pump
# github.com/ipfs/go-ds-flatfs
../../go/src/github.com/ipfs/go-ds-flatfs/convert.go:154:7: log.Warningf    undefined (type *log.ZapEventLogger has no field or method Warningf)
../../go/src/github.com/ipfs/go-ds-flatfs/flatfs.go:927:6: log.Warningf  undefined (type *log.ZapEventLogger has no field or method Warningf)
../../go/src/github.com/ipfs/go-ds-flatfs/flatfs.go:943:6: log.Warningf undefined (type *log.ZapEventLogger has no field or method Warningf)
../../go/src/github.com/ipfs/go-ds-flatfs/flatfs.go:949:7: log.Warningf undefined (type *log.ZapEventLogger has no field or method Warningf)
../../go/src/github.com/ipfs/go-ds-flatfs/flatfs.go:955:6: log.Warningf undefined (type *log.ZapEventLogger has no field or method Warningf)
../../go/src/github.com/ipfs/go-ds-flatfs/flatfs.go:960:6: log.Warningf undefined (type *log.ZapEventLogger has no field or method Warningf)
../../go/src/github.com/ipfs/go-ds-flatfs/flatfs.go:1040:7: log.Warningf undefined (type *log.ZapEventLogger has no field or method Warningf)
# github.com/ipfs/go-ds-s3
../../go/src/github.com/ipfs/go-ds-s3/s3.go:375:5: cannot use (*S3Bucket)(nil) (type *S3Bucket) as type datastore.Batching in assignment:
*S3Bucket does not implement datastore.Batching (missing Sync method)
# github.com/ipfs/go-ds-badger
../../go/src/github.com/ipfs/go-ds-badger/datastore.go:114:13: cannot use log (type *"github.com/ipfs/go-log".ZapEventLogger) as type badger.Logger in assignment:
*"github.com/ipfs/go-log".ZapEventLogger does not implement badger.Logger (missing Warningf method)
ipfs@ronnyc:~/scripts/ipfs$ ls go/bin
total 0

Which version did you use to build it?

Never played with go src, but I removed the 2 offending lines and ran "cd ~/go/src/github.com/INFURA/ipfs-pump; “go build ./main.go” . It seemed to compile, there were no errors, were lots of lines like this last one: “go: finding GitHub - mattn/go-runewidth: wcwidth for golang v0.0.4” including downloading, extracting, finding.

However there was no ipfs-pump in ~go/bin. I finally figured out the binary was “main” so I copied it to ~/go/bin/ipfs-pump and I’m on my way, assuming the 2 now-absent lines don’t cause problems at run time.

Dec 8th:
I had some trouble “sucking” data from source A to destination B, so I am trying a push from source A to destination B.

So far I see no new objects added to destination B, and the command appears to hang. I used 50 workers.

A few minutes later I see ipfs on source A consuming lots of CPU. It’s an 8 core, 32GB RAM system with some SSD storage, and I saw one CPU peak at over 600%, however the ipfs-pump process never shows much cpu at all it stays pretty much at zero.

Not sure how to gauge progress.

netstat -tupn | grep on destination shows an entry:
tcp 0 0 <dest IP>:5001 <source IP>:41182 CLOSE_WAIT 25177/ipfs and it’s been in that state for quite awhile.

I returned to the ipfs-pump screen an hour or so later and see this:
$ ipfs-pump \

apipin --enum-api-pin-url=127.0.0.1:5001 --enum-api-pin-stream \
api --coll-api-url=127.0.0.1:5001 \
api --drain-api-url=<dest IP>:5001 \
--worker=50

1s  0 / ?  [------------------------------------------------------------------------------------------------------------------------------------------------------------------=]2019/12/08 09:57:13 <nil>
z 1s  1 / ? [---------------------------------------------------------------------------------------------------------------------------------------------------------=] 0/s 1s

$

The process seems to have died, perhaps a run time error due to the 2 lines of code I cut so it would compile.

I ran go build pump_test.go and after the downloading/extracting phase it reported: no packages to build. There was no executable produced.

4

Apart from the above, I’d suggest to upgrade to the latest version (or to master), and check iostat to see if your disk access is fully saturated. Restarting the daemon helps sometimes.

5

No worries now, I used the slow process and got the replication done, tho it took a few days. The repo has almost 10000 videos, twice that many files. It’s around 1.6TB out of 2.6TB in size now.

Have a good holiday season Hector!