❗️❗️DHT Hydra Peers dialling down non-bridging functionality on 2022-12-01

TL;DR

In light of recent findings regarding the performance gain provided by Hydra peers in the IPFS network, which indicate that the gain is in the area of 10% (or less) and considering the substantial cost that Hydra’s operation incurs, Protocol Labs has decided to carry out a “Hydra Dial Down” experiment. We plan to turn off certain functionality from Hydras, which we believe will allow all current critical operations that Hydras serve to continue as normal, while at the same time reducing the cost of their operation.

Pointers

Details

Hydras serve three primary purposes:

  1. Faster access to the Provider Records, i.e., reduced latency.

  2. Availability of Provider Records, given that Hydras are generally stable and online, whereas normal DHT nodes churn (i.e., if all 20 peers where the Provider Record is stored go offline, the content becomes unreachable).

  3. A “bridge” to services such as Indexers like cid.contact which advertise content from large providers like web3.storage and Filecoin nodes.

Our results show that Hydras provide at most 10% reduction in the latency to access provider records. The impact on the availability of Provider Records is expected to be minimal, given that they are published to 20 peers in total. Check this detailed report on the availability of provider records in the IPFS network over time and how Hydras affect the records’ liveness. In summary, the first two points above should not be of concern for the operation of the network in the absence of Hydra nodes.

The Hydra’s “bridging” component to other parts of IPFS and Filecoin infrastructure will not be affected. Hydras will maintain connections to cid.contact Indexer nodes and will return results from them.

When, where, what and monitoring logistics

When: The dial down will take place on the 1st of December 2022 at 3pm UTC. We will send an update to confirm, or report a date change (if the need arises) in this forum at the latest 24hrs before the above date, ideally earlier. We will post a message here once the dial down deployment has taken place.

What: We will first disable the database part of Hydras, so provider records won’t be served. Hydras will act as normal DHT Server nodes and will return the “closer peers” they know of in the DHT. The bridging to cid.contact Indexer nodes will continue to operate as normal. Further comms will be done if and when things change with respect to that part of the Hydra’s functionality.

Monitoring: We will be monitoring the performance of Hydras and plan to report updates and performance results in this thread. We will post updates to this thread and expect to have the first results report within the first hour after dialling down. We will report results for several hours on an hourly basis following the event and a summarising report for the first day 24 hours later. From then on we will be posting daily reports for up to 7 days, but will continue to monitor the situation continuously afterwards. There will be a team of engineers on-call, monitoring developments and ready to take action as needed.

Where: The best points of contact to get in touch, or report issues are: i) this thread, ii) the #hydra-dial-down channel in the Filecoin Slack Workspace [invite link], iii) through email at: hydra-dial-down@protocol.ai. Please report issues from your vantage point (node, application, infrastructure), or questions you might have before, during, or after the dial down event.

6 Likes

Update: the Hydra Dial Down event will carry on as scheduled and announced above at 3pm UTC, tomorrow 1st December 2022.

You can join #hydra-dial-down in the Filecoin Slack for more frequent updates. We’ll share here hourly or so updates after the dial-down.

Update: After some testing, which took a bit longer than expected, the DB part of the Hydras has been turned off at ~1730 UTC. Read and Write requests to Hydras have immediately gone down to 0, as expected. We’ll share first results shortly, when we have enough data to plot. Operation is continuing as expected.

Please report unexpected behaviour in this thread, at the #hydra-dial-down channel, or at: hydra-dial-down@protocol.ai.

1 Like

:mega: Update on results from the IPFS Network after the Hydra Dial Down

TL;DR: Results indicate a slight performance hit when operating without Hydras in the network, confirming our expectations and previous results from emulated Hydra-less experiments.

Measurement details: We’ve deployed nodes in several different geographic locations, who have been periodically publishing new CIDs to the network. These CIDs were subsequently requested by the rest of our controlled nodes. The experiment has been running for more than a week, while the Hydras have still been in operation (blue bars in graph) and continued after the dial down at 1730 UTC on Dec 1st (orange bar in graph).

The sample is still small, but the trend is very similar to previous results we gathered while avoiding responses from Hydras and emulating a Hydra-less network. We expect the trend to continue as more data points come in. We’ll be monitoring the situation and we’ll report again on Monday and on a daily basis for the rest of the coming week.

01-Dec-2022


02-Dec-2022

2022-12-05

An update from 2022-12-05. The measurement details that @yiannis described in the previous post still apply.


Here are the performance graphs for each of the probed regions:

1 Like

Quick add on regarding the results we’re seeing and the higher than expected performance difference: we were expecting an average performance hit of ~10% in the Hydra-less environment. Instead, we’re seeing that this is closer to ~15% is some cases. We believe that the difference here is due to the fact that Hydras are still present in the network, but do not store any provider record. The only thing they do is point to the closer peers they know of, adding one extra hop in the DHT walk than the case where they would be completely shut down. We believe this is where the extra ~5% in terms of performance hit is coming from.

2022-12-07

Another update:

An update to the Hydra Dial Down experiment done in December 2022: as was always the plan, the Hydras have been taken out of the network now. Read more in this post: Unanticipated Shutdown of Hydra Network Bridge Nodes: Analyzing Network Impact and implementing direct IPNI path for recovery. TL;DR follows.

TL;DR

Recently, it was found that Hydra peers were declining in quantity, and ultimately on 2023-04-20 the last Hydra node stopped providing network bridging to IPNI (the InterPlanetary Network Indexer). Internal studies show that Hdyras provided less a ~10% performance gain in the IPFS DHT network. Given the substantial cost of Hydra’s operation, Protocol Labs conducted a "Hydra Dial Down" experiment in December, 2022. The result of this experiment showed low risk and impact, and therefore, we have decided to leave the Hydras non functional replacing their bridging function with a direct path to IPNI on 2023-05-04 from popular gateways like ipfs.io/dweb.link.