I am running an IPFS instance inside of a Proxmox VM. The datastore is on a zfs blockstore, the underlying vdev is shared with other VMs. I am using pebbleds. When ingesting large amounts of data via ipfs add, compactions of Pebble start backing up and keep the vdev so busy that it affects other VMs and there is no way of stopping them, short of killing ipfs and nuking the pebbleds/ directory.
Is pebbleds the wrong data store in my case? On a machine with NVME SSDs it did wonders speeding up ipfs add.
Is there a way to see compaction progress? It would be much more easier to justify to the other users if I could see how much time was left on compaction.
Pebble support in Kubo is experimental, so YMMV. Compactions are bursty, and shared storage is where that hurts most. Some things you can do:
To watch progress: ipfs log level pebble debug, then ipfs repo stat -s. The daemon log prints pebble’s metrics, including “estimated compaction debt” (bytes left to rewrite). Watch that number drain, to make more informed decisions.
To protect the other VMs, cap this VM’s disk bandwidth in Proxmox’s disk options.
If bulk adds are the main workload, flatfs backend may fit better stability-wise: no background compaction at all, maybe you need tuning? (see linked docs)
Longer term Pebble fix is ValueSeparationPolicy, PoC in ipfs/kubo#11385: feat(pebbleds): opt-in value separation should cut compaction IO a lot once Pebble makes this feature stable enough, but it’s not there yet. If stability is not mission-critical, you could build that PR and see if it makes any difference for your workload.