Space Usage on Private IPFS

1

Hello
I have only 1 private ipfs node (for now) with ipfs version 0.20.0 and there are some doubts about space usage:

From WebUI “File GiB” Column i see the effective occupation, about 74 GiB
From WebUI “All Blocks GiB” Column i see about 97,1 GiB of occupation (more than 20GiB)
And from filesystem (via ssh) 102 GB is used

How can allign the space effective usage which should be correspond to “File Gib” column?
I have running several times ipfs repo gc and the service is so configured, it runs ok remove a list of hash (and i run it until it not listing anything more) but the space utilization remain the same of “All Blocks” (and it should become the same of “File GiB”)

Some times ago some upload was interrupted, i don’t know if there is some “garbage” that could be remove in another ways

Thanks in advance

1 Like
2

If you just want files that are in your MFS (Files tab), you could unpin all the files first, then run gc again:

ipfs pin ls --type recursive | cut -d' ' -f1 | xargs -n1 ipfs pin rm
ipfs repo gc

(How do I unpin and remove all IPFS content from my machine? - Stack Overflow)

This won’t remove the data in your MFS, but will purge all your extra pinned data.

3

Thanks for the anwer.
Yep, it works, now files occupation is almost the same for all blocks also at the filesystem occupation level it’s allign.

Screenshot from 2023-08-01 13-08-36

Is there a way to do this cleaning without unpin? Te re-pin all files that need to colud be long and not always possibile…
Thanks again

2 Likes
4

Well your issue is you’re ending up with pinned data you no longer want pinned. So you’d need to track your CIDs and unpin when they’re no longer needed. The command I provided will unpin everything, which is a brute-force way of accomplishing a similar thing.

If your data is in MFS anyways too, pinning provides little benefit unless you’re using external pinning services.

5

The thing is that I repinned all the files that were previously pinned. It is as if ip repo gc removed pinned things that are not visible. I don’t use external services but single node pinning, there have never been in other nodes because we are trying IPFS for sharing and transferring large files on different sites (we would like to be able to take advantage of deduplication)
How can I check if is in use MFS? During installation and configuration I don’t remember introducing it

6

The Files tab on the webUI is MFS. If a CID exists in there, it’s safe from garbage collection. MFS is useful for easily organizing your data, and protects your data from GC (like pinning). MFS exists by default in Kubo / IPFS Desktop.

If you want data removed you have to ensure it’s unpinned, and no longer in the MFS, then run garbage collection. To see if something is in MFS you’d have to traverse your tree (ipfs files ls /), to see if something is pinned you can use ipfs pin ls (this will return CIDs, not file/directory names).

7

If i well got, to see if there are CID that are not visible in MFS or WebUI i should run
ipfs files ls /
and match the output with the visibile content pinned
ipfs pin ls
The CID that exist in output ipfs files ls and not exist ipfs pin ls are unpinned and potentially garbage to remove?
In my case there were CID pinned (about 20 GB) that are not visible from WebUI. I can’t explain myself this.
Anyway thanks, you solve me the issue

1 Like