Network delay when adding / reading file


I am currently trying to measure ipfs’ performances to write / read single block files (not in a folder) in an ipfs cluster of 15 nodes in raft mode. To write a file to a cluster, I use the Add function from ipfs-cluster API, to read a file, I use the cat function from ipfs API.
I am able to change the RTT between nodes, and noticed that the time needed on average to write a file is ~5-6 x maxRTT in the cluster, and reading ~3 x maxRTT in the cluster. Does it make sense ? I made experiments with maxRTT up to 2sec.

As I understand, when writing in Raft mode, the following packets are exchanged:

  1. Writing node updates the cluster leader (1/2 RTT)
  2. Leader broadcasts update to all peers and they ack (1 RTT)
  3. Leader waits for a majority of peers’ acks and broadcast the accepted update, returns to Writing node (~1/2 RTT)
    Total: 2RTT, I got: 5RTT

When reading from IPFS, the following packets are exchanged:

I guess 1RTT for step 1.
1 RTT to establish the session in 2. ?
As the file has a single block, I am guessing step 3. takes 1RTT
Total: 3RTT, I got: 3RTT

Are operations performed locally (pinning) negligible compared with network delay ? If not what is the average time it takes for the longest local operations when writing and reading?

Thank you for your help

When adding with Cluster:

  • 1 round trip to send each block to each IPFS peer on which you are adding (happens in parallel)
  • Once all the blocks are in the destinations, the pinning process starts:
    • Request to pin is forwarded to current Raft leader as needed
    • Current Raft leader broadcasts the operation and waits for acks and broadcasts accepted update.
    • Upon sucessful completion the PinAdd method returns to writing node
    • In parallel (async) once the Raft-pinning is successful, cluster peers request IPFS peers (usually on localhost) which have received the blocks to pin the root CID. I don’t know your network delay so can’t say if the local operation is negligible.
1 Like

What does exactly the PinAdd method do ? Does it require more RTTs ?

My network has a radius of roughly 300 ms.

No, it should not touch the network again.

Ok thank you for your answer!