IPFS 11-to-12 migration duration

My IPFS filestore is tracking many video files, total size 48 million megabytes. I upgraded IPFS and it started the fs-repo-11-to-12 migration. It has been running for many days. The “generic migration worker” message shows a progress of about 5 million “objects” per day. I assume that an “object” is a chunk. If each chunk is a quarter of a megabyte, then my filestore has about 192 million objects. At 5 million objects per day, the migration will take about 38 days. Is that right?

Sounds right.

Using badger I guess? There are some env vars that control the number of workers. For a repo this size it is probably good to switch to flatfs.

Thanks, @hector. I’m not sure which datastore I’m using. Is there a way to check while the migration is running?

It should be in your .ipfs/config configuration.

It looks like I’m already using flatfs.

  "Datastore": {
     "BloomFilterSize": 0,
     "GCPeriod": "1h",
     "HashOnRead": false,
     "Spec": {
       "mounts": [
         {
           "child": {
             "path": "blocks",
             "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
             "sync": true,
             "type": "flatfs"
           },
           "mountpoint": "/blocks",
           "prefix": "flatfs.datastore",
           "type": "measure"
         },
         {
           "child": {
             "compression": "none",
             "path": "datastore",
             "type": "levelds"
           },
           "mountpoint": "/",
           "prefix": "leveldb.datastore",
           "type": "measure"
         }
       ],
       "type": "mount"
     },
     "StorageGCWatermark": 90,
     "StorageMax": "10GB"
   },

That’s weird, because then the migration should not be running with the generic worker but with the flatfs optimized one which is much faster. See:

It should have been enabled by default unless manually disabled via env-var.

I don’t have any environment variables with “IPFS” in it. Should I just let the migration finish?

if are ok waiting for a month :man_shrugging:

I will hope my computer stays up. If the process stops for some reason I will ask about recovery, but for now I’ll wait for a month.

flatfs should handle that moreless correctly.

But you have a backup anyway, don’t you ? :upside_down_face:

I guess they’re already committed to the migration but could you migrate by making the node a member of a cluster, adding a second node and allowing them to mirror the pin set and then retiring the old node? I’m sure it would take longer but it would be up the entire time so it might not matter.

That sad, but it’s sometime faster for some reason. (shouldn’t be if you use flatfs tho, mainly a thing I’ve seen with badger)

If the calculation of 192 million objects is correct, then the migration is 21% done. This pool has been here since the beginning of flatfs (2015?), so maybe there is some legacy configuration in there. Hopefully the migration will update it. This isn’t a critical production system, so I can wait. If it gets to 200 million objects and hasn’t finished (or IPFS crashes) then I’ll worry.

thank you bro im just newbie in here

Success! Upgrade finished after 35 days and 194 million objects, pretty much as anticipated. I should mention that the files are on an external USB disk array. Maybe the USB is slow and that’s why it took so long to re-scan the files. (Is that what the upgrade does? Was it converting to a new has algorithm?)

1 Like

The upgrade is just moving files around.

Basically for version 11 the files are named with an encoding the full CID.
For version 12 only the multihash is used, that means the same blocks but under different codecs is only stored once.
That enables version other optimizations too.

FYI with badgerds there is no simple rename feature, so badger is forced to copy due to badger’s limited API.