Copied IPFS data directory onto new host but it's showing no pins

Help Kubo
1

Hi All!

I’m having trouble migrating IPFS to a new VM.

I copied over the IPFS data directory to the new host VM using scp. The old host was on 0.4.13 go-ipfs. I upgraded using the https://github.com/ipfs/fs-repo-migrations tool and am now running 0.6.0 go-ipfs with a docker container. Almost everything works as expected, I can quickly query all the old data via the gateway.

When I query the pins, however, I get an empty object for the keys property where as on my old machine I get about 14MB of pinned objects.

Really appreciate any advice.

Thanks in advance!

Best,
Charles

2

@charlescrain did you try ipfs pin verify? What is the log output?

3

@twdragon I get empty response:
/ # ipfs pin verify
/ #

Thanks for the help. May just write a script to pin them all since the data seems to be there

4

Does verify work for the given CID? Did you try with the known pin CID?

5

And with --verbose does it produce any output?

6

Can you actually verify that the data is still there locally with ipfs repo stat.

The pinset is just a CID that gets recorded. If ipfs pin ls returns nothing, it means that reference is lost. Are you 100% sure you copied everything in the .ipfs folder?

Did you keep a copy from the old repo? If you can get the list of pins with the old ipfs, you could just pin those in the new setup.

7

verify shows nothing even with --verbose

Does verify work for the given CID? Did you try with the known pin CID
Im not 100% sure what you mean by this. :grimacing:

8

I used scp to download the whole folder it to my new host and it didn’t fail on anything.

I didn’t get the ipfs-staging directory but I was under the impression that was not necessary.

I do have both the old and new running since I’ll need to pin anything that’s missing from the new new just before switching over.

Yes I have a script that will iterate over all the CIDs pinned in the old but if the pins should be there then I’m somewhat worried that something might be wrong.

IPFS repo stat shows:
/ # ipfs repo stat
NumObjects: 2088863
RepoSize: 298603440281
StorageMax: 10000000000
RepoPath: /data/ipfs
Version: fs-repo@10

thanks for the help!

9

Also I changed the ID of my new node as the migration docs recommended doing so. Could this cause a problem?

10

That shouldn’t be a problem…

I need to dig up where exactly the pinset CID is stored and see if there is a way it could get lost…

11

Was there a change in how the pinset CID is saved between 0.4.13 and 0.6.0? I never tried using go-ipfs@0.4.13 on the data dir to see if it had the pins before migrating.

12

I don’t think so but I am not sure, it’s super old version.

13

Okay, thanks for the help.

I think I’ll just run my pinning script and test out the node thoroughly before switching over.

14

keep a copy of the repo please (as migrated), for debugging purposes.

15

Okay will do. It’s over 300GB and will eventually need to remove it but for I can keep it for at least a year at the rate that the host is filling up.

Let me know if you’d like me to try something

16

The local pin root is stored at /local/pins https://github.com/ipfs/go-ipfs-pinner/blob/9e800d1363c2cb23b8f82b3b1f0685b83367103a/pin.go#L22

I also made a really simple datastore debugging tool https://github.com/aschmahmann/ipfs-ds. Running ./ipfs-ds get --base=base58btc /local/pins will spit out the local pin encoded as a base58btc multibase (i.e. in this case it’ll be a CIDv0 with a z tacked onto the front of it)

17

Seems like the pin verify command should also list out the CID that it tried to use to look up pins. Maybe the new installation is just pointing to the wrong “pins CID” right?

BTW, I always use ‘rsync’ instead of ‘scp’ for moving files, because you can repeat it later, and it only copies the changed data.

18

Thanks wclayf, I’ll use rsync next time :slight_smile:

Yeah that would be helpful to have the pin CID printed out like that for my situation.

Yeah I think it must be just using the wrong pin for it. Going to be investigating with the ipfs-ds tool that @adin mentioned, will report back.

1 Like
19

I found the pin CID:
When I attempt to cat it I get:

ipfs cat /ipfs/QmSmg6FMmX4PCxWYE67iybUurBccHaGry3NNRPoYds9HZ1: proto: required field "Type" not set

Could be that I’m not supposed to be able to cat the pins file :man_shrugging:

Another thing, I didn’t stop my ipfs daemon when I ran the scp command to copy over the files. Could that have prevented anything from being copied over?

I tried connecting a v0.4.13 to the unmigrated backup and I’m also getting an empty keys object so it looks like the pins just were not copied over.

20

I googled this: ipfs “proto: required field” “not set”

and the 3 links that came up indicate it’s likely something is corrupted related to that CID you tried to cat.

And yes I’d definitely stop the ipfs daemon before moving files. Good luck. I’m not really an IPFS expert, just an occasional visitor to this forum.

2 Likes