Any crawler is available for the DHT’s in IPFS?
Thank you!
Thank you. Will try IPFS Search for crawling the DHT.
You have an idea how this works? My understanding is that there is no global view of the DHT which means this can never index the IPFS network as a whole…right?
indeed a node alone doesn’t know the whole DHT it only knows the hash it has added itself or when it has pinned. However the Kademalia protocol that does the routing uses a whispering system. So if I’m not mistaken, you have to listen to the whispers to be able to crawl the content.
1 Like
This is the first time I will be seeing the term “whispers”. mind expanding on this? or perhaps points to links where this is explained more? Thanks!
Sure ,
You check this web page to understand kademalia protocol : https://en.wikipedia.org/wiki/Kademlia
to explain “whisper” you can imagine when your IPFS node add or pin a content it will send a message all node wich know to said ‘i added this hash’. so instead to parse DHT maybe you can crawl this message sended by your node and all node into your swarm peers list.
Regards
Ha, I see…but it still not a global view. You only will get whispers from all the nodes in your swarm peers list. But I think I understand the idea. Thanks!