Question: what's the most efficient way to check if the blocks of a CID exist locally?

Given a CID, I want to quickly check if it fully exists locally. In the JS implementation, the blockstore in js-ipfs-repo has a method to check if the CID exist in the store. I’m looking for a similar solution ideally using the js-http-client with go-ipfs.

The closest thing I can come up with is using files.stat with the withLocal option but I assume that this will attempt to retrieve the blocks from the network if they are not available locally.

1 Like

After digging a bit further - doesn’t seem like files.stat would work, even if the withLocal option was supported in js-http-client. I’m now thinking that going through the local refs might be my best bet.

I think you’ve figured it out ipfs.refs.local will give you a list of all CIDs with blocks in your repo.

Is there a way to check a single block only @achingbrain ?

That would be incredibly useful. In my use case, the nodes may have upwards of 4-10TBs locally and for various reasons (mainly performance), it helps to distinguish local CIDs from remote CIDs to more efficiently decide what operations to perform and in what order.

I’m going to test out pulling the local refs through js-http-client using ipfs.refs.local later today :crossed_fingers: — I’m guessing it’ll work fine but might become an issue as the list grows.

To clarify, will a CID show up in ipfs.refs.local if any or all of its blocks exist locally?

That will list ALL local blocks.

1 Like

I believe it is a reasonable feature request to extend ipfs refs local [-r] [CID] to take optional CID argument and print it if it is in local repo. If -r is passed, print all local blocks that belong to the root. This would match existing behavior of ipfs refs and ipfs refs -r. Together, they would enable people to calculate how much of data behind CID is stored locally.

Use case: IPFS Companion is bundling webui and importing it to datastore if it is not in it already. To do the check, it needs to list all local CIDs and if there is no match, perform import. Extremely not efficient check if user has a lot of blocks.

Where should I fill issue for this?
Core API is at https://github.com/ipfs/js-ipfs/tree/master/docs/core-api#ipfs-core-api – does it mean js-ipfs repo?

1 Like

But go-ipfs already supports this with the offline option…

That is a good point. We probably just need to improve UX:

I was not aware it is even an option.
--offline is a global flag (not listed in ipfs refs --help)

$  ipfs refs -r --offline bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq
QmPQVLHXAcDLvdf6ni24YWgGwitVTwtpiFaKkMfzZKquUB
QmPfyHeZgnf6nEoMKogLGxMeDi4YkwXDi8NbcxFdnqu8kX
bafkreigitfaavqeoyeuzloqkjfkpqw6s4fgaeheikqwb3t672ca5tzzmpe
QmVAbxVyNtTTRABp4XAJZWxfkx23enw57bdrQ6opQ8i2cE
QmXEyUJQeCyd5LuvkEgNzYJhAjQ4qQrT91dHfpExDpX9jF
bafkreiep4vnlmrj2xyjtaoegkn4xg5kmkcswzej34l36s2vo6hzfvi36li
QmcvHZnW7kY9D5a514xzxVVBofmfYvdJnt3Py5KPh6EECD
QmNYBYYjwtCYXdA2KC68rX8RqXBu9ajBM6Gi8CBrXjvk1j
QmXicEckV9gydjnQp8g87su922FaAuGzpQ76cwCJYGbYzB
bafkreib3ci43pa4icelo7d6bujipbc75jsuct3bdityz3uhfvbh4ohqvmm
bafkreidoewtgqimn2h4lgax2ltum3zwob4fwtkvjxk3ojy65vqokwwbgum
bafkreicnxzsav44ninlbnzpnzxmtxfsud3kdc3s6pnthwanxezwc4wyx7y
bafkreiaqvigkj2iurzldt7aphzsgpajp2kubblzzqiis77tia4qxyva2xy
bafkreihc7fc6ukmign4tg4uedeeahbxlvxlt6l6bmv2iogoiyr2efzq4xm
bafkreiglsoghxtoddlow7sask4w2oxk7rdru3bj3qhaixwfppps3etptju
bafkreidhagf6roig4gr6bpdbs4pwg76d45xkjujoqxvfiufpzjuxwvwu2a
bafkreibimttulkjlhxpj2zqxuh23vax22tmcvkdhezwreofuenuha7foum
bafkreifr6qifibf7gimmlksvc75qltd6xntr2dh3qvoztjurna2tumfl5u
QmdgDFvLPyLkDV8zR7r92Ceerx8u5hf968PNxjU51yRZL6
bafkreiaq5upvq6pmjus7nivdu7x7rmfkrqklcaas57lcic74nb2bbsldx4
bafkreibskrxxgvgwv7u5vdulvvcj5scd3u6tgh32it6eqarrq5v4tre2q4
Error: failed to fetch all nodes

Perhaps we could implement ipfs refs local [-r] as porcelain on top of ipfs refs [-r] --offline ?

1 Like

Digression: I still think we need a dedicated API for this check.
The --offline trick is not a silver bullet: it works only in go-ipfs CLI.

AFAIK it is impossible to use it for checking presence in local repo in JS:
it is missing from Core API and is not supported by js-ipfs-http-client.

2 Likes