Measured Single Peer File Transfer in IPFS (kubo -> kubo) vs scp

Help Kubo
, ,
1

Dear Community,

I was just playing around to get a feeling for IPFS and noticed that transferring a single file (roughly 220MB) between me and my server took almost 9x longer than via SCP.

I know that there is of course going to be some overhead using IPFS, but I would really like to know what is causing this amount of speed difference.

Both peers were connected via ipfs swarm connect beforehand, I’ve did multiple runs, tried default chunk size and also 1 MiB, still similar results. My Server has 8 vCPUs / 16GB Ram, and a M2. SSD, so from my understanding this shouldn’t be IO Bound.

What am I possibly doing wrong, or are my numbers realistic? Are there some tools for better benchmarking to see what is bottlenecking this?

image
image1423Ă—655 50.3 KB

Edit 1: From my first tests it seems like that my Wantlist can never exceed 10 CIDs, is there some sort of hard limit enforced?
It always does 10 —> 0 → 10 → 0 . I would’ve assumed that it could have something to do with walking the DAG, but if my DAG has only the Height Two, how is that possible?

This is my Config that I used on both Client and Server:

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/webrtc-direct",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/webrtc-direct",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoConf": {},
  "AutoNAT": {},
  "AutoTLS": {},
  "Bitswap": {},
  "Bootstrap": [
    "auto"
  ],
  "DNS": {
    "Resolvers": {
      ".": "auto"
    }
  },
  "Datastore": {
    "BlockKeyCacheSize": null,
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "mountpoint": "/blocks",
          "path": "blocks",
          "prefix": "flatfs.datastore",
          "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
          "sync": false,
          "type": "flatfs"
        },
        {
          "compression": "none",
          "mountpoint": "/",
          "path": "datastore",
          "prefix": "leveldb.datastore",
          "type": "levelds"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "DeserializedResponses": null,
    "DisableHTMLErrors": null,
    "ExposeRoutingAPI": null,
    "HTTPHeaders": {},
    "NoDNSLink": false,
    "NoFetch": false,
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "HTTPRetrieval": {},
  "Identity": {
    "PeerID": "12D3KooWASA6iKs4KPpcp3N5grb59qKZGg9G2kWMp6J4Uiv8LizV"
  },
  "Import": {
    "BatchMaxNodes": null,
    "BatchMaxSize": null,
    "CidVersion": null,
    "FastProvideRoot": null,
    "FastProvideWait": null,
    "HashFunction": null,
    "UnixFSChunker": null,
    "UnixFSDirectoryMaxLinks": null,
    "UnixFSFileMaxLinks": null,
    "UnixFSHAMTDirectoryMaxFanout": null,
    "UnixFSHAMTDirectorySizeThreshold": null,
    "UnixFSRawLeaves": null
  },
  "Internal": {
    "Bitswap": {
      "BroadcastControl": null,
      "EngineBlockstoreWorkerCount": 500,
      "EngineTaskWorkerCount": 128,
      "MaxOutstandingBytesPerPeer": 4194304,
      "ProviderSearchDelay": null,
      "ProviderSearchMaxResults": null,
      "TaskWorkerCount": 128,
      "WantHaveReplaceSize": null
    }
  },
  "Ipns": {
    "DelegatedPublishers": [
      "auto"
    ],
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {},
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns",
    "MFS": "/mfs"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provide": {
    "DHT": {}
  },
  "Provider": {},
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "DelegatedRouters": [
      "auto"
    ]
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Enabled": false
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "Version": {}
}
2

On performance in general that seems quite slow. There are a few things I’d look into:

  1. What’s the time to first byte? If say 30 of the 60 seconds are getting the first byte then it means the data is likely not advertised and then instead of kubo just giving up it effectively wanders around the network until it finds your data which is of course very slow.
  2. Depending on your available resources, and for the purposes of your test the latency on the connection, the MaxOutstandingBytesPerPeer on the server end can cause bottlenecks and need to be increased. This parameter controls some degree of “fairness” around responses, but there’s certainly room for improvement in how fairness is handled.
  3. On ipfs get and the 10 wants at a time. There is a limitation related to the ipfs get HTTP API. The API itself streams out the result, as a result the underlying API that fetches the data only pre-fetches some at a time for the purposes of streaming the results out to you. Also, not an optimal way of handling this use case, but the pre-fetching there is hard coded at 10 blocks go-ipld-format/navipld.go at 9238f65e8d039f4dee0d4717b7c0e8e39757ea3d · ipfs/go-ipld-format · GitHub. My recollection is this shouldn’t be quite so staggered because as more blocks stream back you should be requesting more, but my memory might be off here. If it really is staggered like that despite blocks streaming back to you incrementally that sounds off / like a bug to me.
    • Note: Other APIs where you effectively commit to getting more data (e.g. ipfs pin) have much higher limits since the API knows its safe to get all the data in whatever order with no need to worry about buffering data in memory, garbage collection, etc.
3

FWIW, I have found it is much faster to complete a transfer if I use ipfs pin add before ipfs get. You can run ipfs pin add in the background while you run ipfs get in the foreground; this is how I use it in my scripts. Then run ipfs pin rm if you don’t need to keep a local copy for serving to others indefinitely (adding a pin makes it exempt from garbage collection).

ipfs pin add requests all the blocks, in any order, and so keeps the wantlist full, which in my experience makes the transfer a lot faster.