# Crawler for IPFS **URL:** \ **Category:** Uncategorized\ **Created:** [May 12, 2020, 1:36am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968 "2020-05-12T01:36:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sha](https://avatars.discourse-cdn.com/v4/letter/s/e8c25b/32.png) [@sha](https://discuss.ipfs.tech/u/sha)\ **Post date:** [May 12, 2020, 1:36am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/1 "2020-05-12T01:36:20Z")
Any crawler is available for the DHT’s in IPFS? Thank you! ---
**Author:** ![josselinchevalay](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/josselinchevalay/32/3078_2.png) [@josselinchevalay](https://discuss.ipfs.tech/u/josselinchevalay)\ **Post date:** [May 12, 2020, 6:10am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/2 "2020-05-12T06:10:43Z")
hello, maybe you can check that : [https://github.com/ipfs-search/ipfs-search](https://github.com/ipfs-search/ipfs-search) Regards ---
**Author:** ![sha](https://avatars.discourse-cdn.com/v4/letter/s/e8c25b/32.png) [@sha](https://discuss.ipfs.tech/u/sha)\ **Post date:** [May 15, 2020, 1:18am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/3 "2020-05-15T01:18:02Z")
Thank you. Will try IPFS Search for crawling the DHT. ---
**Author:** ![finlaybweber](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/finlaybweber/32/5868_2.png) [@finlaybweber](https://discuss.ipfs.tech/u/finlaybweber)\ **Post date:** [May 17, 2020, 8:52pm UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/4 "2020-05-17T20:52:33Z")
You have an idea how this works? My understanding is that there is no global view of the DHT which means this can never index the IPFS network as a whole…right? ---
**Author:** ![josselinchevalay](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/josselinchevalay/32/3078_2.png) [@josselinchevalay](https://discuss.ipfs.tech/u/josselinchevalay)\ **Post date:** [May 18, 2020, 6:46am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/5 "2020-05-18T06:46:30Z")
indeed a node alone doesn’t know the whole DHT it only knows the hash it has added itself or when it has pinned. However the [Kademalia protocol](https://en.wikipedia.org/wiki/Kademlia) that does the routing uses a whispering system. So if I’m not mistaken, you have to listen to the whispers to be able to crawl the content. ---
**Author:** ![finlaybweber](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/finlaybweber/32/5868_2.png) [@finlaybweber](https://discuss.ipfs.tech/u/finlaybweber)\ **Post date:** [May 19, 2020, 11:27am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/6 "2020-05-19T11:27:27Z")
This is the first time I will be seeing the term “whispers”. mind expanding on this? or perhaps points to links where this is explained more? Thanks! ---
**Author:** ![josselinchevalay](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/josselinchevalay/32/3078_2.png) [@josselinchevalay](https://discuss.ipfs.tech/u/josselinchevalay)\ **Post date:** [May 19, 2020, 11:55am UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/7 "2020-05-19T11:55:40Z")
Sure , You check this web page to understand kademalia protocol : [https://en.wikipedia.org/wiki/Kademlia](https://en.wikipedia.org/wiki/Kademlia) to explain “whisper” you can imagine when your IPFS node add or pin a content it will send a message all node wich know to said ‘i added this hash’. so instead to parse DHT maybe you can crawl this message sended by your node and all node into your swarm peers list. Regards ---
**Author:** ![finlaybweber](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/finlaybweber/32/5868_2.png) [@finlaybweber](https://discuss.ipfs.tech/u/finlaybweber)\ **Post date:** [May 19, 2020, 4:42pm UTC](https://discuss.ipfs.tech/t/crawler-for-ipfs/7968/8 "2020-05-19T16:42:12Z")
Ha, I see…but it still not a global view. You only will get whispers from all the nodes in your swarm peers list. But I think I understand the idea. Thanks!