IPFS Repo Migration problems

Help
1

I just installed IPFS version 0.4.22 on my Mac.

When I try to upload/add a file in TERMINAL - using the following command:

ipfs add myLogo.jpg

I get this error:

Error: ipfs repo needs migration

My repo version is ipfs repo version fs-repo@7

Additionally, when I try to run:

ipfs daemon

I get this error:

Initializing daemon...
go-ipfs version: 0.4.22-
Repo version: 7
System version: amd64/darwin
Golang version: go1.12.7
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N]

So I tap on ā€œyā€ to run these migrations, and then I get:

=> Looking for suitable fs-repo-migrations binary.
=> None found, downloading.
=> Running: /var/folders/l_/793j9l1j7fx4n6pb0zd2tww40000gn/T/go-ipfs-migrate464045646/fs-repo-migrations -to 7 -y
Found fs-repo version 2 at /Users/johnSmith/.ipfs
===> Running migration 2 to 3...
applying 2-to-3 repo migration
locking repo at "/Users/johnSmith/.ipfs"
- verifying version is '2'
beginning pin transfer
- opening datastore at "/Users/johnSmith/.ipfs"
- created version 3 pinner
- loading recursive pins
ipfs migration: migration 2 to 3 failed: datastore: key not found
=> Failed: /var/folders/l_/793j9l1j7fx4n6pb0zd2tww40000gn/T/go-ipfs-migrate464045646/fs-repo-migrations -to 7 -y
The migrations of fs-repo failed:
migration failed: exit status 1
If you think this is a bug, please file an issue and include this whole log output.
https://github.com/ipfs/fs-repo-migrations

Iā€™ve already spent a few hours trying to troubleshoot this with no luck. So, what in the world is going on - and what do I need to do to fix this?

2

What version were you on before upgrading to 0.4.22?

3

I honestly donā€™t know. Is there a way to find out? And either way, how should I proceed to fix this?

4

I think we can get down to a range of versions based on the table in the fs-repo-migrations repo.

            1 |  0.0.0 - 0.2.3
            2 |  0.3.0 - 0.3.11
            3 |  0.4.0 - 0.4.2
            4 |  0.4.3 - 0.4.5
            5 |  0.4.6 - 0.4.10
            6 |  0.4.11 - 0.4.15
            7 |  0.4.16 - current

It looks like your repo is on version 2, which was used between versions 0.3.0 and 0.3.11. For context, 0.4.0 seems to have been released on April 7, 2016.

From the error it almost sounds like there could be some datastore corruption (datastore: key not found); but Iā€™m also not very familiar with the datastore migration process. If you were to install 0.3.11 temporarily just to access the repo, Iā€™m curious what the output of ipfs repo verify would be.

It might be worth trying to upgrade the repo with version 0.4.0 or 0.4.2 to see if an older version can migrate the repo from 2 to 3. But maybe it will just use the same migration process.

Or maybe this is a bug in the migration code and worthy of an issue. Iā€™m just not sure either way, though.

Reference:
link to all versions of go-ipfs binaries

5

ok, Iā€™m game - but how would I install version 0.3.11?

6

Hereā€™s a direct link to the darwin-flavored binary:
https://dist.ipfs.io/go-ipfs/v0.3.11/go-ipfs_v0.3.11_darwin-amd64.tar.gz

The link at the end of my previous post also has links for a bunch of other historical and current binaries.

7

Oh I see - ok cool - Iā€™ll try it and let you know what happens.

1 Like
8

I was also just thinking, this all assumes thereā€™s something in the old repo you want to salvage. If the only data in the repo is from toying around with go-ipfs 3+ years ago and thereā€™s nothing in it you care about, you could also just delete the old test repo (.ipfs directory) in your home folder and start over fresh with v0.4.22 using ipfs init.

9

well I have tried to do that. But every time I try to run ipfs init, it tells me:

Error: ipfs configuration file already exists!
Reinitializing would overwrite your keys.

And Iā€™m like ā€œfine! I donā€™t care! just do it!ā€ - but it wonā€™t let me.

I mean I would LOVE to just start fresh - but HOW?

10

Your repo is located in ~/.ipfs by default. You could rm -rf ~/.ipfs before running ipfs init again if you wanted to start fresh.

11

Iā€™ve just been trashing my repo the old fashion way: dragging and dropping it to the Trash.
I donā€™t imagine thereā€™s a difference between doing that and what youā€™re suggesting with rm -rf ~/.ipfs, is there?
If there is, how do I do that?
Meaning, I just did a cd to my usr/local/bin - which is where my ipfs repo is, and then tried your line-command from Terminal - but nothing happened. The repo is still there. So the line command basically doesnā€™t seem like its working for me.

So is this a big deal - or can I just drag nā€™ drop it to the trash?

12

There are some differences, but itā€™s similar. Dragging the .ipfs folder to the trash should be the close enough for this purpose.

The command I posted doesnā€™t depend on your current directory. ~ resolves to your home directory. And based on your first post, your repository is in /Users/johnSmith/.ipfs, which would be the default. The /usr/local/bin folder would be a strange place for a repository; but that is one of the places the ipfs binary can be installed.

The command I posted doesnā€™t return any output. So unless you got an error, it likely worked.

Iā€™d expect that if you were to run ipfs init it would work now.

Not a big deal at all. If /Users/johnSmith/.ipfs is still there for some reason, then feel free to drag and drop it into the trash.

13

Well, I deleted everything, did a fresh install - of the version you suggested, 0.3.11 - and then ran
ipfs init

hereā€™s what I got:

jonSmiths-MBP:bin jonSmith$ ipfs init
initializing ipfs node at /Users/jonSmith/.ipfs
generating 2048-bit RSA keypair...done
peer identity: QmUvF76ccLwhRcQ6DW26NtzC8YxcDpUivmZ1tkiNwbuAoM
panic: sync: inconsistent mutex state

And FYI, Iā€™ve gotten this panic error before, several times when doing this fresh install thing.

The full error message is actually much longer than that. Hereā€™s what comes right after what I just posted above:

goroutine 1 [running]:

sync.(*Mutex).Lock(0xc820437c30)

/usr/local/Cellar/go/1.5.3/libexec/src/sync/mutex.go:74 +0x16d

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).GetFiles(0xc820437c20, 0x200000002, 0x0, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:166 +0xad

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb.(*session).getFiles(0xc820411ce0, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session_util.go:44 +0x66

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal(0xc820094dc0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:454 +0x63

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb.openDB(0xc820411ce0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:111 +0x6eb

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb.Open(0x178a488, 0xc820437c20, 0xc820126f58, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:177 +0x17d

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb.OpenFile(0xc82047ad50, 0x22, 0xc820126f58, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:199 +0x97

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore/leveldb.NewDatastore(0xc82047ad50, 0x22, 0xc8201273a0, 0xc82047ad50, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/go-datastore/leveldb/datastore.go:23 +0x92

github.com/ipfs/go-ipfs/repo/fsrepo.(*FSRepo).openDatastore(0xc82046d000, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/repo/fsrepo/fsrepo.go:351 +0x171

github.com/ipfs/go-ipfs/repo/fsrepo.open(0xc8201d3a60, 0x18, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/repo/fsrepo/fsrepo.go:159 +0x488

github.com/ipfs/go-ipfs/repo/fsrepo.Open.func1(0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/repo/fsrepo/fsrepo.go:106 +0x4e

github.com/ipfs/go-ipfs/repo.(*OnlyOne).Open(0xfa7fd0, 0x79cfa0, 0xc8204788c0, 0xc820127720, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/repo/onlyone.go:35 +0x150

github.com/ipfs/go-ipfs/repo/fsrepo.Open(0xc8201d3a60, 0x18, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/repo/fsrepo/fsrepo.go:108 +0xd4

main.addDefaultAssets(0x1784198, 0xc820098008, 0xc8201d3a60, 0x18, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/cmd/ipfs/init.go:164 +0xd8

main.doInit(0x1784198, 0xc820098008, 0xc8201d3a60, 0x18, 0xc820260000, 0x800, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/cmd/ipfs/init.go:124 +0x3b3

main.glob.func2(0x1785b10, 0xc82012a480, 0x1785c78, 0xc82017bc00)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/cmd/ipfs/init.go:79 +0x405

github.com/ipfs/go-ipfs/commands.(*Command).Call(0xfaa580, 0x1785b10, 0xc82012a480, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/commands/command.go:110 +0x471

main.callCommand(0x1785be8, 0xc82024e380, 0x1785b10, 0xc82012a480, 0xfaa580, 0xf9fc80, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:343 +0xa5c

main.(*cmdInvocation).Run(0xc82024e340, 0x1785be8, 0xc82024e380, 0x0, 0x0, 0x0, 0x0)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:186 +0x1e5

main.main()

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:151 +0x65b

goroutine 17 [syscall, locked to thread]:

runtime.goexit()

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/asm_amd64.s:1721 +0x1

goroutine 27 [select, locked to thread]:

runtime.gopark(0xc1d6e8, 0xc820030f28, 0xa72a68, 0x6, 0xc1d618, 0x2)

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/proc.go:185 +0x163

runtime.selectgoImpl(0xc820030f28, 0x0, 0x18)

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/select.go:392 +0xa64

runtime.selectgo(0xc820030f28)

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/select.go:212 +0x12

runtime.ensureSigM.func1()

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/signal1_unix.go:227 +0x323

runtime.goexit()

/usr/local/Cellar/go/1.5.3/libexec/src/runtime/asm_amd64.s:1721 +0x1

goroutine 20 [chan receive]:

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics.init.1.func2()

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics/metrics.go:321 +0x73

created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics.init.1

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics/metrics.go:328 +0x7b

goroutine 21 [select]:

github.com/ipfs/go-ipfs/vendor/QmQg1J6vikuXF9oDvm4wpdeAUvvkVEKW1EYDw9HhTMnP2b/go-log.(*MirrorWriter).logRoutine(0xc820074c80)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/vendor/QmQg1J6vikuXF9oDvm4wpdeAUvvkVEKW1EYDw9HhTMnP2b/go-log/writer.go:71 +0x33c

created by github.com/ipfs/go-ipfs/vendor/QmQg1J6vikuXF9oDvm4wpdeAUvvkVEKW1EYDw9HhTMnP2b/go-log.NewMirrorWriter

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/vendor/QmQg1J6vikuXF9oDvm4wpdeAUvvkVEKW1EYDw9HhTMnP2b/go-log/writer.go:38 +0xe2

goroutine 22 [syscall]:

os/signal.loop()

/usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:22 +0x18

created by os/signal.init.1

/usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:28 +0x37

goroutine 29 [select]:

github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc820102a80)

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x271

created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool

/Users/dignifiedquire/opensource/ipfs/distributions/dists/go-ipfs/gopath/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x26b

14

crazy, isnā€™t it?

Pretty much makes me wanna give up.

Or just use Infura to access IPFS - though that doesnā€™t seem to work perfectly either.

Thoughts?

15

You shouldā€™ve tried to init with 0.4.22; it doesnā€™t make any sense to start fresh with a version so old that it probably canā€™t even connect to anything anymore. I would suggest getting rid of that old repo and then trying again with 0.4.22. The suggestion to temporarily try 0.3.11 from years ago to verify your old repo was only if you wanted to continue troubleshooting your old repo.

Partially out of curiosity, what version of macOS are you on?

16

To answer your question Iā€™m on MacOS 10.14.5 - Mojave.

In terms of the IPFS versions - I tried everything with both 0.3.11 and 0.4.22 - but Iā€™m not 100% sure if my installation process is correct?

Hereā€™s what Iā€™m doing:

  1. Downloaded go-ipfs_v0.4.22_darwin-amd64.tar.gz from ipfs.io
    -as per usual, the file automatically downloads to my ā€œDownloadsā€ folder

  2. I double-click the file there to expand it - which creates a folder called ā€œgo-ipfsā€
    -this folder contains 4 files:
    -README.md
    -LICENSE
    -install.sh
    -ipfs

  3. Double-clicking on the README.md file, it says:

    "This is the ipfs commandline tool. It contains a full ipfs node.
    To install it, move the binary somewhere in your $PATH:

       sudo mv ipfs /usr/local/bin/ipfs

    (This by the way, is why I put my ipfs in ā€œusr/local/bin/ā€, cause the instructions say so - unless Iā€™m not understanding them correctly?)

    -When I run this sudo command from Terminal, I get this:

    No such file or directory

    Iā€™m assuming thatā€™s because Iā€™m supposed to first cd into my Downloads, before running that command. So I do that. I cd into Downloads, run that command, and bingo, ipfs gets moved to my
    usr/local/bin/ directory.

  4. Next, the ā€œREADMEā€ file says:

    "First, you must initialize your local ipfs node:

     ipfs init

    OK, so I run ipfs init - and I get this:

    initializing IPFS node at /Users/johnSmith/.ipfs
Error: ipfs configuration file already exists!
Reinitializing would overwrite your keys.

Needless to say, Iā€™ve already deleted and trashed my previous installation of ipfs from my /usr/local/bin/, so I donā€™t understand why Iā€™m getting this error.

OK, so now Iā€™m curious so I type ā€˜ipfs versionā€™ and it says:

ipfs version 0.4.22

-For kicks I type in ipfs config show, and I get this nice long JSON with all this info, like

  "API": {
    "HTTPHeaders": null
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001"
    ]
  },
  "Bootstrap": [

It goes on for a bit - and importantly, it does include:

"Identity": {
    "PeerID": "Qmxxxxxxxxxxxxxxxxxxxxxx"
  },

(I put all those xā€™s in)

So part of me thinks ā€œok, that means my ipfs is already initā€™d and Iā€™m good to go!ā€ - BUT, when I then do

ipfs id

I get:

Error: ipfs repo needs migration

Soā€¦I donā€™t really know what Iā€™m supposed to do at this point.

Thoughts?

17

Yes, what you did is correct for installing ipfs on macOS. Thereā€™s a difference between the ipfs ā€œbinaryā€ and the ipfs ā€œrepositoryā€. The binary file is the ipfs executable. The repository is referring to where IPFS stores its configuration files and data structures.

Everything up through #3 sounds fine in your last post. You donā€™t need to touch the installed ipfs any more to get a fresh repo.

I think this is because after we cleaned up your old repo, you initialized a new (outdated) one with 0.3.11 instead of 0.4.22. Once you said you donā€™t care about the old repo anymore, that removed any need to use 0.3.11.

Without the $ symbols:

# remove your outdated 0.3.11 repo (leave the 0.4.22 binary as is)
$ rm -rf /Users/johnSmith/.ipfs
# initialize a new repo assuming *0.4.22* is still installed
$ ipfs init
# test it with a simple command
$ ipfs id
1 Like
18

hey you know what?

that seems to have worked!

Iā€™m gonna fiddle with it for a bit just to make sure, but you may have solved it!

1 Like
19

I went through all of this before learning about https://github.com/ipfs/ipfs-update