Excessive data transfer for local ipfs Kubo node in docker container

Help
1

I deployed IPFS to a docker container in my Synology NAS. I added one file, of a few 100 bytes.
I added 4001 tcp/udp port forward to the container IP. Occasionally, I checked the IPFS status page and would see perhaps 1000 max peers. Seemed innocuous.
The past few days, the network was acting “funny”. The modem (fritz) was sluggish (more so than usual). The connexion was flapping down/up randomly, about 12 or more disconnects per day. Pings, just from the LAN to the external modem were losing ~10% of packets. I was chasing the connexion flapping down/up issue when I noticed that the IPFS container IP address had recently uploaded ~50GB, per some traffic snooping my firewall does. Here is my recent traffic log:

image
image1010×1210 148 KB

The usual upload is around 2 to 3 GB per day. On 2022-08-18 (the day I deployed the IPFS container) it goes up to 23GB, and keeps increasing to 51GB on 08-23.
This is an IPFS daemon running doing nothing much at all except that I wanted to start exploring IPFS.
What on earth could it be doing?

As soon as I suspend the IPFS container, the load on the modem immediately reduces: pings are not lost, and the fritz!box goes back to its normal level of sluggishness when using the WEB interface, upload traffic goes back to normal.

I did find a bunch of baf* items (raw blocks) which ipfs repo gc cleaned up. I supposed I pulled them in with my poking about, but there were only a few hundred KB total in my repo. Nothing to suggest why I have such huge upload traffic over the last few days, and increasing day-on-day, to the point where it was impacting my modem to the point it kept dropping the connexion. My ISP tech contact insists there are no line issues — apparently, the disconnects are initiated from the modem, not due to any line noise or excessive errors.

Anyway, I sort of trust docker containers to be contained. I have verified that the container does not have access to the NAS filesystem, apart for the dedicated areas I map in. I did perform a tcpdump while the madness was in progress, but natch, it is encrypted, so I can’t tell what was being sent/received.

Can anyone shed some light onto this?
For now, I have deleted the port forwards, and the traffic has reduced.

Are there any tools for seeing what is being exchanged with peers? I really can’t believe that ~200MB of data might have been in such high demand that I would have swamped my internet connexion and both downloaded and uploaded ~40GB per day.

I am running ipfs/kubo:latest from 2022-08-18. top tells me that I am running ipfs daemon --migrate=true --agent-version-suffix=docker. IPFS version is 0.14.0.

2

Here’s an additional image I was prevented from adding to the above. It shows the b/w from the period of time when I had the port forwards applied, and then subsequently removed.

image
image1920×846 84.6 KB

3

Can you show ipfs stats bw; ipfs stats bitswap pls ?

4 
/ # ipfs stats bw; ipfs bitswap stat
Bandwidth
TotalIn: 1.1 GB
TotalOut: 1.2 GB
RateIn: 70 kB/s
RateOut: 213 kB/s
bitswap status
	provides buffer: 0 / 256
	blocks received: 662
	blocks sent: 4
	data received: 1154847
	data sent: 2476
	dup blocks received: 191
	dup data received: 272187
	wantlist [0 keys]
	partners [693]
/ #
5

A few hours later:

/ # ipfs stats bw; ipfs bitswap stat
Bandwidth
TotalIn: 3.7 GB
TotalOut: 5.3 GB
RateIn: 668 kB/s
RateOut: 111 kB/s
bitswap status
	provides buffer: 0 / 256
	blocks received: 7320
	blocks sent: 41
	data received: 12699163
	data sent: 63931
	dup blocks received: 1440
	dup data received: 2435551
	wantlist [0 keys]
	partners [727]

ntopng tells me that the ipfs flow has ~19GB sent and ~9GB received.
Not sure if I trust these figures, from ntopng for the ipfs flow. Not sure what period of time they cover, but i have only been running ntopng since after first reporting this issue.

TCP: Retransmissions /     Out of Order /           Lost / KeepAlive
Sent:   1,809,410 Pkts / 1,615,916 Pkts / 2,341,510 Pkts /  7,218 Pkts 
Recv:     213,179 Pkts /   974,548 Pkts /   583,694 Pkts / 51,511 Pkts

netstat -s does not exist for the container running the ipfs daemon, which is where I am used to finding these stats.

My system cannot avoid saturating the puny outbound pipe, and would be dropping pkts from the TX queue in the Fritz!box (no stats I can get to), and for RX, with no pacing and congestion avoidance for UDP/QUIC, one would expect to drop a lot of packets. Hmm. The ISP’s bandwidth usage would only count TX pkts which successfully transit thru the puny TX pipe to the ISP, so I am probably just confusing myself more with this. I need accurate packet counters in a few places I do not have access to.

Auto-DDOS?

6

I’ve seen several ostensibly ‘idle’ nodes have this sort of behaviour.
It’s normal for an instance with 1k peers to be using serveral megabit up and down constantly.
I’ve also had that volume of connections crash out the router due to poor memory management in the state table of consumer grade routers.

I’ve dealt with it in two ways:

  • run QoS on a dedicated ip assigned to the node to limit the max bandwidth / connections
  • or use either of the ‘low power’ config profile or manualy tune the conn manager watermarks till the router stops crashing unduly
7

I tried using the limited QoS facilities in the Fritz!box, but no joy.
I’ll try pushing the ipfs install out to a hosted node (currently running peergos for no particular reason).

1 Like