IPNS hangs after many requests at same time

Help IPFS
1

Usually the IPNS works fine but after a few days or many requests at the same time(not sure) every request to publish a record starts hanging. I also suspect that this happens when trying to publish a CID that is new to the system.

Memory usage was aorund 4 GB at that time which was 50-60% of the total system memory. The service is deployed on kubernetes

would be great to get some ideas on how i can change the config to make sure this doesn’t happen.

here is my config and here are my logs - IPNS Logs 22 Oct 2023 · GitHub

  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [
      "/ip4/5.199.168.17/tcp/31401",
      "/ip4/5.199.168.17/udp/31402/quic",
      "/ip4/5.199.168.17/udp/31402/quic-v1",
      "/ip4/5.199.168.17/udp/31402/quic-v1/webtransport",
      "/ip4/127.0.0.1/tcp/4001",
      "/ip4/127.0.0.1/udp/4002/quic",
      "/ip4/127.0.0.1/udp/4002/quic-v1",
      "/ip4/127.0.0.1/udp/4002/quic-v1/webtransport",
      "/ip4/0.0.0.0/tcp/4001",
      "/ip4/0.0.0.0/udp/4002/quic",
      "/ip4/0.0.0.0/udp/4002/quic-v1",
      "/ip4/0.0.0.0/udp/4002/quic-v1/webtransport"
    ],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "child": {
        "path": "badgerds",
        "syncWrites": false,
        "truncate": true,
        "type": "badgerds"
      },
      "prefix": "badger.datastore",
      "type": "measure"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "DeserializedResponses": null,
    "HTTPHeaders": {},
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWFNF977D1rmfUdhHsWMn3AqZcXPdcLQRN4"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "120h",
    "RepublishPeriod": "3m",
    "ResolveCacheSize": 0,
    "UsePubsub": true
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "AcceleratedDHTClient": false,
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {
      "GracePeriod": "15s",
      "HighWater": 500
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}```
2

What is your Kubo version ? I fixed an IPNS bug recently.

1 Like
3

If you could show a trace instead of logs that would help a lot, the logs are extremely noisy and hard to read (it’s on us).

You can find some docs here:

Basically:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one
OTEL_TRACES_EXPORTER=jaeger ipfs daemon
1 Like
4

The version is 0.21.0

5

Here is the jaegar output

Here are some of the newer logs

2023-10-30T08:43:24.389Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.390Z	INFO	bitswap	bitswap/bitswap.go:169	Bitswap Client ReceiveError: stream reset
2023-10-30T08:43:24.405Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.407Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.414Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.415Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.430Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.477Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.488Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.491Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.494Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.507Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.518Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.549Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.556Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.556Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.576Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.587Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.592Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.613Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.617Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.618Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.622Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.632Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.647Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.656Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.664Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.689Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.694Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWN5vy1zyjQSM2UzwsekNCKzZGFpbBnHK4BXU8gySySVQq: queue full; scheduling retry
2023-10-30T08:43:24.697Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry
2023-10-30T08:43:24.707Z	INFO	pubsub	go-libp2p-pubsub@v0.9.3/pubsub.go:965	Can't send announce message to peer 12D3KooWESbdkDRduhCsvEQS1RACMPoseELdNjQXsHke8aufVHYF: queue full; scheduling retry```