Can i use bitswap for my usecase

Help
, , , , ,
1

I have custom chunker , it gets hashes of each chunk and creates json file.
and create cid out of the hash to link the data . have impleted a file sharing application over libp2p , that uses the json to find and share the content .

can i use bitswap here , if i can give cid list can it identlify which node has the content , i currently use libp2p dht.Provide to provide the CID.

i use custom replication flow to manage content availability . if the content is not available the node would fetch the missing content alone from server.

can bitswap or ipfs tell which contents are not available so that i can fetch them.

Is there any better way to achieve the same with IPFS and bitswap

2

Yes, you should be able to do this with the existing tooling (I’m guessing [boxo](GitHub - ipfs/boxo: A set of reference libraries for building IPFS applications and implementations in Go. for your use case) as it seems like a pretty good match.

If you’re using boxo with a standard setup (e.g. setup bitswap as a blockservice backed by a local blockstore) and you request data it’ll check locally before asking remotely and when asking others if no connected peers have the data it’ll check the content routing system (e.g. Amino DHT, IPNI, some delegated routing endpoint, etc.) to look for peers to ask

FYI if your chunks are all under 2MiB you can still use UnixFS (and get functionality like rendering files on gateways, tooling for range requests, etc.) and just use your custom chunker (here’s an example spec for a group that’s done this IPFS Custom File Chunking for WARC and WACZ). However, if you prefer to use a JSON manifest that’s fine too.

Also you might consider content addressing the manifest too if you want people to be able to reference the data using just a CID instead of requiring a manifest (e.g. BitTorrent clients can both deal in .torrent files and infohashes where .torrent files have more information but require the client to have all that information up front)

3

Thanks adin , for your response. Is there some mechanism that will tell me which blocks are missing even after it checking routing systems (DHT) . so that i can fetch only those blocks from central http server.

Currently we use chunk size of 2 MiB only, and since i need more availability , these individual chunks are replicated . we use private network so agents can find and initiate replication if it has less availability .

In our case we should’nt store full file in single system as much as possible. Since we store each chunk in many systems download happens using DHT content discovery.

Could you please align these points and provide a response.

Thanks.