Filestore questions

Help
1

I have MFS over filestore.

The questions are:

I found with:
ipfs filestore ls | grep TotallyIrrelativeFileOutsideMFS
there are many blocks in filestore, not belong to MFS, from previous
ipfs add --nocopy
operations.
What I can do to remove these garbage?
ipfs repo gc
returns nothing.
ipfs pin ls
returns nothing.

  1. I rename a file in my hard disk.
    Now I inform MFS with
    ipfs files mv
    Must I inform somehow the filestore too?
1 Like
2

ipfs filestore verify will dump all the blocks that are bad. Currently ipfs-sync will also clean out a dirty filestore, but AFAIK go-ipfs doesn’t have this feature built-in… I’ll make a standalone program that does that.

3

Try this, I just made it, it will remove all the blocks that point to files that don’t exist. It’s not optimised, but it’s better than nothing.

4

Yes, though it’s not clear how best to do this. I’ve been wrestling with this problem for a while now. My ramblings can be found here: Filestore cleanup is too slow · Issue #38 · TheDiscordian/ipfs-sync · GitHub

1 Like
5

Whow.
That really sucks.

Thanks for program. I will inspect the code for learning things.

1 Like
6

Yes! I missed
ipfs block rm
Awesome.

For the other question:

So there are 3 filesystems, from low level to high level:

  • “Real” filesystem
  • Filestore filesystem
  • MFS filesystem

If I share part of my physical disk with IPFS, then
If I want to rename a 100GB file in my disk from “MyBestMovieEver.mkv” to “MySecondMovieEver.mkv”, then:

  • Rename file (this is the first action and the only action which I do with my hand. Following actions happen from a cron script)
  • ipfs filestore ls | grep MyBestMovieEver.mkv and then for each block
  • ipfs block rm
  • ipfs add --nocopy MySecondMovieEver.mkv
  • ipfs files cp …
  • ipfs name publish --key=… /ipfs/…

Deleting the blocks of a renamed 100GB file and then regenerate them, really sucks.
Hopefully is not the case.

7

If you want to rename an entry of filestore, you must modify
.ipfs/filestore/*.ldb
files, which are LevelDB database.
But thing start to become obscure (It must kill daemon before modification or not?)

8

Initially this is how I was going to do it, but you run into the issue where some blocks might be shared with another object. So what you actually want to do is ipfs filestore verify <CID> I believe.

I agree.

Thanks for digging into that! Yeah you’d likely want to respect whatever sort of locks it’s using … not 100% sure here. FWIW filestore code is here.

1 Like