Question: what's the most efficient way to check if the blocks of a CID exist locally?

Given a CID, I want to quickly check if it fully exists locally. In the JS implementation, the blockstore in js-ipfs-repo has a method to check if the CID exist in the store. I’m looking for a similar solution ideally using the js-http-client with go-ipfs.

The closest thing I can come up with is using files.stat with the withLocal option but I assume that this will attempt to retrieve the blocks from the network if they are not available locally.

1 Like

After digging a bit further - doesn’t seem like files.stat would work, even if the withLocal option was supported in js-http-client. I’m now thinking that going through the local refs might be my best bet.

I think you’ve figured it out ipfs.refs.local will give you a list of all CIDs with blocks in your repo.

Is there a way to check a single block only @achingbrain ?

That would be incredibly useful. In my use case, the nodes may have upwards of 4-10TBs locally and for various reasons (mainly performance), it helps to distinguish local CIDs from remote CIDs to more efficiently decide what operations to perform and in what order.

I’m going to test out pulling the local refs through js-http-client using ipfs.refs.local later today :crossed_fingers: — I’m guessing it’ll work fine but might become an issue as the list grows.

To clarify, will a CID show up in ipfs.refs.local if any or all of its blocks exist locally?

That will list ALL local blocks.

1 Like

I believe it is a reasonable feature request to extend ipfs refs local [-r] [CID] to take optional CID argument and print it if it is in local repo. If -r is passed, print all local blocks that belong to the root. This would match existing behavior of ipfs refs and ipfs refs -r. Together, they would enable people to calculate how much of data behind CID is stored locally.

Use case: IPFS Companion is bundling webui and importing it to datastore if it is not in it already. To do the check, it needs to list all local CIDs and if there is no match, perform import. Extremely not efficient check if user has a lot of blocks.

Where should I fill issue for this?
Core API is at – does it mean js-ipfs repo?

1 Like

But go-ipfs already supports this with the offline option…

That is a good point. We probably just need to improve UX:

I was not aware it is even an option.
--offline is a global flag (not listed in ipfs refs --help)

$  ipfs refs -r --offline bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq
Error: failed to fetch all nodes

Perhaps we could implement ipfs refs local [-r] as porcelain on top of ipfs refs [-r] --offline ?

1 Like

Digression: I still think we need a dedicated API for this check.
The --offline trick is not a silver bullet: it works only in go-ipfs CLI.

AFAIK it is impossible to use it for checking presence in local repo in JS:
it is missing from Core API and is not supported by js-ipfs-http-client.