Constant writes to datastore/*.log

Help Kubo
1

Hello,

I’m running

% ipfs --version
ipfs version 0.43.0

It constantly writes datastore/*.log, about 250 KB/second of data.

What it writes is:

[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "\354\34\255\r\231\0\1C\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "\v\21\312h\231\0\1D\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "VO\225\354\231\0\1E\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "{G\262q\231\0\1F\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "\265G\24\234\231\0\1G\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "\3\327\263<\231\0\1H\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "\nG\365D\231\0\1I\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "+\256\356^\231\0\1J\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "~\264\245\243\231\0\1K\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160
[pid 2261910] write(587</home/ipfs/.ipfs/datastore/8177060.log>, "TW^\376\231\0\1L\222_\24\1\0\0\0\1\0\0\0\1\200\1/providers"..., 160) = 160

Is it expected? It hammers my HDD.

Kubo is nearly-idle, sometimes seeds some files. Not used for accessing websites or downloading.

2

IIRC those /providers/... keys are provider records other peers ask your node to store.

I assume your node is publicly reachable, so with the default Routing.Type=auto it switched into DHT Server mode and now accepts a constant stream of ADD_PROVIDER messages from the network. You are co-hosting Amino DHT records.

What strace shows is that stream hitting leveldb’s write-ahead log. Records expire after 48h and are garbage-collected, so it’s churn rather than unbounded growth, but on a HDD it’s noticeable.

If you don’t want that load, run as a DHT client only:

ipfs config Routing.Type autoclient

and restart the daemon. Your node will still fetch and seed files as before; it just stops storing routing records on behalf of other peers.

More details: kubo/docs/config.md at master · ipfs/kubo · GitHub and IPFS Kademlia DHT

3

That might be an SSD killer then. Kubo wrote 20 GB of data during these 3 hours since my post. That’s a lot!

% sudo cat /proc/`pidof ipfs`/io | grep '^write_bytes' 
write_bytes: 21821321216

Consider adding this information only to RAM.

4

I’m afraid there is no exposed config option to relax the fsync for the leveldb backend in Kubo 0.43. But there are workarounds.

What you can do today:

  1. Routing.Type=autoclient stops the stream entirely; a node that only seeds some files loses nothing.

  2. If you need to keep DHT server mode, but prefer to get your hands dirty to hold these records in RAM, you could do it without code changes. Kubo’s datastore is a set of mounts: keys with a given prefix go to a given backend (Datastore.Spec), and there is a built-in mem backend that keeps keys in memory. All these records live under the /providers prefix, so you can route just that prefix to RAM:

    • stop the daemon and add this entry to the Datastore.Spec.mounts array in ~/.ipfs/config (order does not matter, Kubo sorts mounts itself):

      { "mountpoint": "/providers", "type": "mem" }

    • start the daemon once; it will refuse with datastore configuration ... does not match what is on disk and print two JSON strings. Copy the one that contains "mountpoint":"/providers" into ~/.ipfs/datastore_spec (this file is a guard against accidental datastore changes and has to match the config), then start the daemon again.

    I did this in the past, should still work: the daemon runs fine, /providers writes go to RAM and vanish on restart, everything else persists as before. Trade-offs: RAM use grows with the number of stored records (DoS vector), and records already written stay in leveldb as dead weight.

  3. Stop the fsync storm at the OS level: keep ~/.ipfs/datastore on a filesystem where fsync is free, e.g. a ZFS dataset with sync=disabled. Trade-off: after a power loss the entire datastore may need repair.

  4. New repos can use ipfs init --profile=pebbleds; iirc Pebble syncs its WAL in the background instead of on every write; also, you could use pebble instead of mem mentioned in (2) above.

Anyway, on keeping third-party provider records in RAM by default: that could become a DoS vector if done without care, but feel free to open an upstream issue about some sort of limited in-memory provider record store (or at least dropping the per-record fsync) in Issues · ipfs/kubo · GitHub

5

I did that after your message.

Ran ipfs config Routing.Type autoclient and restarted Kubo. Right now it has:

  "Routing": {
    "AcceleratedDHTClient": false,
    "DelegatedRouters": [
      "auto"
    ],
    "Methods": null,
    "Routers": null,
    "Type": "autoclient"
  },

Nevertheless, nothing has changed:

[pid 2309636] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "\264\310&\254\217\0\1\236\303q\24\1\0\0\0\1\0\0\0\0\200\1/providers"..., 150) = 150
[pid 2309638] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "\253\6\323>\22\1\1\237\303q\24\1\0\0\0\2\0\0\0\0\200\1/providers"..., 281) = 281
[pid 2309638] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "\232y\240\242\217\0\1\241\303q\24\1\0\0\0\1\0\0\0\0\200\1/providers"..., 150) = 150
[pid 2309636] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "E\365\201O\217\0\1\242\303q\24\1\0\0\0\1\0\0\0\0\200\1/providers"..., 150) = 150
[pid 2309538] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "U\271\203u\217\0\1\243\303q\24\1\0\0\0\1\0\0\0\0\200\1/providers"..., 150) = 150
[pid 2309638] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "\37\246\36\221\217\0\1\244\303q\24\1\0\0\0\1\0\0\0\0\200\1/providers"..., 150) = 150
[pid 2309553] write(394</home/ipfs/.ipfs/datastore/8179405.log>, "S\36D\213\22\1\1\245\303q\24\1\0\0\0\2\0\0\0\0\200\1/providers"..., 281) = 281

% sudo cat /proc/`pidof ipfs`/io | awk ' /^write_bytes/ {print $2/1024/1024, "MB"}'
744.969 MB
6

Hm.. I think autoclient worked, and those writes are deletes, not new records. Note how your strace entries shrank from 160 to 150 bytes, and the type byte before \200\1/providers flipped from \1 (put) to \0 (delete).

What you see now is cleanup. Even in client mode, Kubo does not accept new DHT PUTs, but respects existing ones that are still valid. It scans the stored provider records once an hour and deletes everything older than 48h. Your datastore still holds up to 48h of records from when the node was a DHT server, so this churn should stop within ~48h of the switch.

7