I’m in the process of building out a production storage network using IPFS, with iSCSI based disks for the storage layer. Today while running some tests a network cable to the iSCSI machine became disconnected, which caused the filesystem that was on the iSCSI system to become read-only on my linux IPFS node. This caused some serious issues with the badger datastore and the IPFS node, and I was unable to recover from this and had to start fresh.
I believe this issue stemmed from the fact that IPFS_PATH was set to the iSCSI storage layer. This lead me to conclude that the only safe alternative was to set IPFS_PATH to a direct-attached disk, and mark the datastore path as the iSCSI based file system.
When starting the node, I got an error along the lines of "{"type": "badgerds", "path":"badgerds"} does not match what is on disk {"type": "badgerds", "path":"/ipfs-data"}".
The only alternative I can think of is symlinking $IPFS_PATH/badgerds to /ipfs-data. So far in my initial tests this appears to be satisfactory, and should resolve issues with the datastore becoming read-only leading to a total node fail-to-boot scenario.
While being satisfactory, it is undesirable and quite honestly a hack. Is there anyway around this using either:
- Badgerds
- Flatfs
- leveldb
If no, how come and is there anything I can do (PR wise) to accommodate this functionality? If the reason for this is simply due to the current code-base I would be more than happy to open a PR to fix this, as I would like to avoid the hacky symlink fix.
side note 1: Has anyone ever tried using iSCSI for the storage layer of an IPFS node and has some knowledge to share? If not I’m currently drafting up a document detailing my experience, and would be happy to share.
side note 2: It might be worth-while ading a fail-safe functionality in that if the IPFS node detects that the datastore is read-only, initiate a shutdown.