IPFS needs as space as the files shared?

In my trip to convert my FTP server to a P2P FTP server, many questions arise.
Now I ask this:

My data is 10TB.
On “ipfs add” folder “/root/.ipfs” start increase rapidly.
So, if I understand, my 10TB will be stored in “.ipfs” folder in chunks.
I believe that 10TB of unused data (duplicated) is too much.
IPFS is not Torrent, but torrent does not need this.
Is there a way for IPFS to avoid all of this duplication of data?

PS: My NAS OS is in an 2GB sdcard and I try to add 10TBs to IPFS :- )

Please see “Filestore”: go-ipfs/experimental-features.md at master · ipfs/go-ipfs · GitHub

1 Like

Whow!
This sounds like a super cool solution!

After instructions in your link (thank you!), and after I create some symbolic links inside root directory (only files inside root directory allowed), I get this:

ipfs --nocopy -r Βιβλιογραφία
Error: unknown option “nocopy”

I run linux ARM 0.7.0 distribution.
I suspect that extension is not supported yet in that version.

Any ideas?

The default blockstore will chunk your files and save the resultant chunks and metadata into the /.ipfs folder or anywhere you have configured the daemon to use. So if you are loading 10TB data into the repo, you will need 10TB in addition to the original 10TB.

ipfs add --nocopy: your are missing the add.

Read Hector’s awesome answer.

Ooops!
Consider I didn’t ask that :stuck_out_tongue:
Thank you again!

Directory added with “ipfs add --nocopy -r directory_path” MUST BE inside root directory (user who runs IPFS daemon).
If you trying to cheat IPFS with a symlink of read directory, it does not work.

If you add “/media/my_disk/real_path /root/rooted_path none bind” in “/etc/fstab” and then run “mount -a”, it works.

More info after some tests.

Filestore is another storage mechanism. It is not enabled by default and it suffers from its own limitations. The default blockstore mechanism does indeed duplicate the storage space required.

Read: https://github.com/ipfs/go-ipfs/issues/4224

1 Like