How to modify small sections of a large file stored in IPFS?

Peergos says they’re chunking in 5MB blocks, and the challenge with any chunking algorithm is that if you insert (for example) one single byte at the very front of the file, then the hash of every chunk changes (all of them shifted by a byte), and destroys the ability to reuse chunks, and it’s the same effect as duplicating all the storage of an entire file (because you get all new chunks).

I think I read somewhere that RSYNC has some kind of intelligent way of choosing their chunks, that isn’t simply at the boundary line of every chunk block size (but uses the data itself to intelligently choose block boundaries), and is based on the known need to reuse blocks.

So the key question is can IPFS do this kind of intelligent chunk selection? …because if so then theoretically it could be efficient in modifying large files and accomplishing also the equivalent of RSYNC where only small data transfers are required to sync large files/folders.

1 Like