# IPFS repair mechanism for durable storage **URL:** \ **Category:** Help\ **Created:** [April 17, 2018, 1:17pm UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673 "2018-04-17T13:17:02Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rklaehn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/rklaehn/32/244_2.png) [@rklaehn](https://discuss.ipfs.tech/u/rklaehn)\ **Post date:** [April 17, 2018, 1:17pm UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/1 "2018-04-17T13:17:02Z")
Let’s assume that you want to store some data that is very important for a long time. IPFS has some pretty good preconditions for ensuring data integrity, with content-addressed storage with secure hashes as identifiers. So how would I set up a number of IPFS nodes to actually _do_ store important data? Assuming that IPFS nodes are physically distributed in different AWS availability zones / regions as well as different physical machines. First of all, IPFS nodes should be configured in such a way that they validate data _on lookup_, so you have a guarantee that you will never get wrong data for a hash. I guess this is covered by the `HashOnRead` config option, correct? Second, there should be some kind of background process that regularly validates blocks, discards invalid blocks, and requests them again over the network. Basically a background repair mechanism similar to e.g. what is available for [cassandra](https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsRepair.html). Ideally, corrupted blocks should be stored somewhere (lost+found) in case you can’t recover the block from the network. Does such a repair mechanism exist? If so, how would I configure it? ---
**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\ **Post date:** [April 17, 2018, 2:03pm UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/2 "2018-04-17T14:03:50Z")
> So how would I set up a number of IPFS nodes to actually do store important data? Assuming that IPFS nodes are physically distributed in different AWS availability zones / regions as well as different physical machines. You’ll probably want to use something like ipfs-cluster. It’s currently alpha-quality but we are already running it ourselves and it works quite well (and uses the same CLI/HTTP interface): > **[GitHub - ipfs-cluster/ipfs-cluster: Pinset orchestration for IPFS](https://github.com/ipfs-cluster/ipfs-cluster)** > > Pinset orchestration for IPFS. Contribute to ipfs-cluster/ipfs-cluster development by creating an account on GitHub. > [@rklaehn](#): > > I guess this is covered by the HashOnRead config option, correct? All IPFS nodes will validate data received from other peers when they receive it. `HashOnRead` tells the node to re-hash when reading from the local datastore. > Does such a repair mechanism exist? If so, how would I configure it? Not at the moment. Note 1: In this case, we probably wouldn’t want repair to be fully automated as such failures usually mean that a disk is failing (and should be replaced before recovery is attempted. Note 2: If you’re willing to make the failure-domain a single IPFS node, you can use IPFS cluster (I believe it has a repair mechanism although I’m not sure how automated it is. ---
**Author:** ![rklaehn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/rklaehn/32/244_2.png) [@rklaehn](https://discuss.ipfs.tech/u/rklaehn)\ **Post date:** [April 17, 2018, 2:46pm UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/3 "2018-04-17T14:46:19Z")
> [@stebalien](#): > > You’ll probably want to use something like ipfs-cluster. It’s currently alpha-quality but we are already running it ourselves and it works quite well (and uses the same CLI/HTTP interface): Thanks. We are definitely going to use ipfs-cluster on the cloud side of things. > [@stebalien](#): > > rklaehn: > I guess this is covered by the HashOnRead config option, correct? > > All IPFS nodes will validate data received from other peers when they receive it. HashOnRead tells the node to re-hash when reading from the local datastore. Yes, this is exactly what I want, to protect against bitflips. If you store sufficiently large quantities of data, they _do_ happen, even if your storage medium is not close to end of life. The overhead should not be that big on go-ipfs, right? > [@stebalien](#): > > Note 1: In this case, we probably wouldn’t want repair to be fully automated as such failures usually mean that a disk is failing (and should be replaced before recovery is attempted. Automated + logging errors to alert of impending trouble would be the best option. > [@](#): > > Note 2: If you’re willing to make the failure-domain a single IPFS node, you can use IPFS cluster (I believe it has a repair mechanism although I’m not sure how automated it is. These will be individual devices (android devices and raspberry pi), with pretty low specs, so it would be great if this was built in to the ipfs binary. It seems to me that writing this from the outside is not possible because the ipfs repo api is not rich enough. What will `ipfs repo verify` do in case of a bad block? I could not find any docs. ---
**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\ **Post date:** [April 17, 2018, 11:31pm UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/4 "2018-04-17T23:31:28Z")
> [@rklaehn](#): > > The overhead should not be that big on go-ipfs, right? Verifying (cryptographic) hashes on every read is expensive however, we can (and probably should) randomly sweep parts of the datastore. > It seems to me that writing this from the outside is not possible because the ipfs repo api is not rich enough. What will ipfs repo verify do in case of a bad block? I could not find any docs. It returns a list of corrupted blocks. The missing pieces are: 1. A nice _machine readable_ output. One can easily parse the output of this command but it should have been machine readable from the start. 2. A way to remove the corrupted blocks (so they can be re-fetched). ---
**Author:** ![hector](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/hector/32/178_2.png) [@hector](https://discuss.ipfs.tech/u/hector)\ **Post date:** [April 18, 2018, 1:42am UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/5 "2018-04-18T01:42:28Z")
Hi @rklaehn, I’m the lead developer for ipfs-cluster. While we’re always improving, things are a bit rough to get started, so feel free to reach out if you need any help. ---
**Author:** ![rklaehn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/rklaehn/32/244_2.png) [@rklaehn](https://discuss.ipfs.tech/u/rklaehn)\ **Post date:** [April 18, 2018, 7:54am UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/6 "2018-04-18T07:54:09Z")
> [@stebalien](#): > > Verifying (cryptographic) hashes on every read is expensive however, we can (and probably should) randomly sweep parts of the datastore. Can you put numbers on it? Will it actually slow things down, or will it just increase CPU load a bit on systems without extremely good persistent storage? > [@stebalien](#): > > It returns a list of corrupted blocks. The missing pieces are: > > A nice machine readable output. One can easily parse the output of this command but it should have been machine readable from the start. > A way to remove the corrupted blocks (so they can be re-fetched). As mentioned above, I would prefer to have the block repair functionality in the IPFS binary. I mean, ensuring data integrity is a core function of ipfs for me. However, I can understand if you decide to not have this functionality built in to keep things simple. But then it would be great if at least it was possible to implement it via some kind of script using the api. > [@hector](#): > > Hi @rklaehn, I’m the lead developer for ipfs-cluster. While we’re always improving, things are a bit rough to get started, so feel free to reach out if you need any help. Thanks for the offer. You will probably hear from us once we set up a cluster on AWS. You do support “cluster of cluster”, right? That would be helpful if you want to be extra paranoid and spread the data not just over availability zones but also over regions. ---
**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\ **Post date:** [April 19, 2018, 2:09am UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/7 "2018-04-19T02:09:13Z")
> [@rklaehn](#): > > Will it actually slow things down, or will it just increase CPU load a bit on systems without extremely good persistent storage? It will significantly increase CPU load (I don’t have numbers but hashing is _already_ shows up pretty high on our CPU profiles). You can, of course, turn `HashOnRead` on if you’re willing to pay the price but most users won’t need it. > As mentioned above, I would prefer to have the block repair functionality in the IPFS binary. I would actually like this (for pinned data, at least). However, it likely won’t happen in IPFS itself in the near future (unless someone steps up to implement it). ---
**Author:** ![hector](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/hector/32/178_2.png) [@hector](https://discuss.ipfs.tech/u/hector)\ **Post date:** [April 23, 2018, 11:42am UTC](https://discuss.ipfs.tech/t/ipfs-repair-mechanism-for-durable-storage/2673/8 "2018-04-23T11:42:59Z")
> [@rklaehn](#): > > Thanks for the offer. You will probably hear from us once we set up a cluster on AWS. You do support “cluster of cluster”, right? That would be helpful if you want to be extra paranoid and spread the data not just over availability zones but also over regions. You can distribute your peers over regions while keeping them in the same cluster too. Cluster-of-cluster is not perfectly supported yet (but you can test it!)