"Abuse" of urlstore and Local http Gateway

Tutorials
1 
ipfs add --raw-leaves --nocopy --cid-version=1 http://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.localhost:8080/
added bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy

Here I used --nocopy to provide a raw leaves version of a file without storing both raw leaves and original on my computer. Is there a more elegant way of doing this?

2

I don’t think you can pass http urls to ipfs add.

Look the CIDs are different:

bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi
bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy

Today it’s the only solution, in the future I want to support reflinking, but this is a big refactor: Make IPFS reflink aware, dedup file storage between IPFS and user downloaded files · Issue #8201 · ipfs/kubo · GitHub

1 Like
3

Yes you can, even though the documentation doesn’t make that explicit. However, if --no-copy is passed, then urlstore needs to be enabled.

Look the CIDs are different:

That’s the point. In the explorer, you can see that the original CID is a dag-pb while the second CID refers to a raw block. Since the original block is proliferated on IPFS, I don’t even need to pin it. If my node ever receives the second CID as a request, it can download the first CID then re-hash it.

4

That is much more elegant. Having that deduplication going on an OS level will be a huge improvement.

5

Ok I understand what you are doing now.
It’s cleaver and terrifying at the same time.

In theory you could use the filestore pointing to the Data field of the non raw leaves blocks if you parsed the blocks and computed the protobuf offsets (so you would have filestore point to .ipfs/blocks/...).

But IMO you should just use raw leaves and don’t care about non raw leaf blocks.
The only reason raw leaves isn’t default is because we are want to bundle many hash changing features in one single big hash breaking release.

Also reflinking wont help you here, on today FSes reflinking only works on offsets multiple of the underlying blocksize, because the data isn’t stored at multiple of 4096 inside protobuf, it wont work.

The point of reflinking is more that you could do ipfs get Qmfoo and the file you downloaded and the blocks under .ipfs/blocks/... would be dedupped using CoW (this would also work with ipfs add).

1 Like