How to add a directory node without adding all its contents

Help Kubo
1

I want to add a directory node, but gradually reveal its contents.

This is for an NFT project where the metadata is linked as ipfs://CID/[1-10000].json. For example, if 5 NFTs have been minted, I would pin 1 to 5 such that ipfs://CID/[1-5] is retrievable while ipfs://CID/[6-10000] is not.

My poor attempt to solve this as follows

hash=$(ipfs add --quiet --pin=0 --wrap-with-directory --cid-version=1 *.json | tail -1)
ipfs block get $hash | ipfs add --cid-version=1
ipfs repo  gc # so that content is not retrievable until I individually add files
ipfs cat bafkr${hash##bafyb} | ipfs dag put --store-codec dag-pb --input-codec dag-pb

This works fine locally. I can add any n.json so that it is retrievable through dag node cid. However garbage collection removes the dag node, because it’s not pinned. I think ipfs-go doesn’t allow pinning it because it’s not populated. So I cannot retrieve it with other HTTP gateways either. Any help is appreciated.
Edit: I guess my poor attempt works somehow, but I assume there are better solutions.

2

Is there a way to do this for CID version 0 encodings? I can’t find a --store-codec for that…

Update (because I can only write three replies as a new user) - Thanks @Farngi and @hector .

Any ideas about the CID v0 rather than v1 issue? I can’t even find a list of the possible options for --store-codec anywhere…

1 Like
3

The other thing that I’m finding is that although the files can be retrieved through this method, the directory requests time out.

i.e. in my case the CID for the directory is bafybeibdozdo36plhkoejy4xlbdjlw6o4hfebjqil7ezlhvj6gkkoom5aq and I see:

http://127.0.0.1:8080/ipfs/bafybeibdozdo36plhkoejy4xlbdjlw6o4hfebjqil7ezlhvj6gkkoom5aq/1 retrieves a file (if file 1 has been added)
ipfs://bafybeibdozdo36plhkoejy4xlbdjlw6o4hfebjqil7ezlhvj6gkkoom5aq/2 gives a context deadline exceeded (if file 2 hasn’t been added), and
http://127.0.0.1:8080/ipfs/bafybeibdozdo36plhkoejy4xlbdjlw6o4hfebjqil7ezlhvj6gkkoom5aq just hangs without any response or error

4

And of course the interesting thing would be to work out how the directory object can be uploaded and pinned to a 3rd party IPFS provider like Pinata or Infura, with the image files subsequently uploaded later on.

1 Like
5

You may be able to take advantange of “direct” pins:

hash=$(ipfs add --quiet --pin=0 --wrap-with-directory --cid-version=1 *.json | tail -1)
ipfs pin add --recursive=false $hash # pin only the folder block
ipfs add --cid-version=1 1.json
...
1 Like
6

This flag is really useful. And my initial solution was not reliable. So I managed to do what I wanted reliably with the following. But dag node in my method is still garbage collected, so your solution seems to be the best way. Just posting this for reference:

hash=$(ipfs add --quiet --pin=0 --wrap-with-directory --cid-version=1 -- data/*.json | tail -1)
block_hash=(ipfs block get "$hash" | ipfs add --quiet --cid-version=1)
ipfs repo gc > /dev/null
ipfs cat "$block_hash" | ipfs dag put --store-codec dag-pb --input-codec dag-pb

And this allows you to restore it with only the last command if garbage collection occurs.

7

I have bumpbed your user trust level.

CidV0s do not encode a multicodec, the codec is implicit and is dag-pb, cannot be anything else.

http://127.0.0.1:8080/ipfs/bafybeibdozdo36plhkoejy4xlbdjlw6o4hfebjqil7ezlhvj6gkkoom5aq just hangs without any response or error

It may be that listing a directory on the gateway requires IPFS to access all the blocks/files that are referenced from that directory, because the directory listing gives the file sizes, and that needs retrieving them. Same with ipfs ls ..., but then you can do ipfs ls --resolve-type=false --size=false to disable that.

1 Like
8

However, ipfs cat "$block_hash" | ipfs dag put --store-codec dag-pb --input-codec dag-pb returns a CID v1. Is there an extra flag for putting a dag-pb and getting a CID v0?

cat dag.bin | ipfs dag put --store-codec dag-pb --input-codec dag-pb --cid-version=0 doesn’t work…

9

This brings up something I’ve been wondering about for a while. It’s sort of “how does IPFS handle nulls” or how does IPFS behave when portions of a DAG are unretrievable? In this case they want to make a directory available that has 10 items but only make 2 available. Anyone retrieving the directory won’t be able to verify the hash because it doesn’t have all the contents. I’m not quite sure how IPFS behaves in this situation. I’m assuming it just fails but if it doesn’t does it just say, “well, that’s the best I can do” but if that’s the case how would it communicate that it’s providing hashes that it can’t verify?

The only thing that makes me wonder if it does do best effort is no one replied with, “IPFS doesn’t work this way”. Every time the person “reveals” a new file it’s going to change the directory CID. Reading the question it seems like they want to “reveal” a new file without changing the directory CID.

11

Using the method @Farngi describes above, if you go to your localhost browser gui you do see the folder, and the list of files within it (because a folder seems to be a table of file names and CIDs), but if you click on a file that isn’t there, the browser hangs. Similarly, from the command line, if you try to export a folder where the files can’t be found, it hangs.

Revealing the files or not having them uploaded shouldn’t change the directory CID, because it’s a collection of named pointers to the files.

12

So I guess you do a recursive pin on the directory, wait for it to timeout and fail, then you’d be left with whatever you were able to retrieve, either the “revealed” files and possibly fewer if there was some other reason you weren’t able to locate the files, and then manually pin what you were able to get in the directory because the recursive pin on the directory guaranteed to fail.

Also once you’ve revealed a file to anyone you can’t be sure you haven’t revealed it to everyone. This sounds a, “I want NFT’s to be what I imagine them to be not what they are” scenario.

13

No, it’s possible to do this. I’m working on it at the moment. You need a private swarm for the initial upload, and you need to export the folder in binary form. Then import it to a public server.

14

Still not following but sounds interesting. Love to hear what you come up with.