# How to tune a private IPFS swarm for large files? **URL:** \ **Category:** Help\ **Tags:** kubo\ **Created:** [December 15, 2022, 2:00pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702 "2022-12-15T14:00:43Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [December 15, 2022, 2:00pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/1 "2022-12-15T14:00:43Z")
I’ve been playing with IPFS private swarms as a background activity at my work for a few months, and am really excited about how we might use it. I’d like some help understanding its performance, and if there’s anything I can do to tune it. **Use case** My use-case is transferring largish files (~6-10 GiB) built in cloud compute to our on-premise lab equipment. I estimate that a naive implementation (i.e. without IPFS) will transfer ~175 TIB/day of data. To put that in perspective that would be a sustained rate of about 17.8 Gbps. These large files are bootable disk images, such as the contents of Linux system running Debian. On a given day most of the disk images will be very similar to one another (perhaps 98% duplicate data between any two given disk images). I’m excited to use IPFS as it will allow me to have my cake and eat it: any single user of the on-premise lab equipment can treat it as private by adding their just-compiled disk image to the swarm and instructing the equipment to boot from a given CID. The IPFS swarm will deduplicate the user’s disk image and likely only transfer the unique blocks across the WAN link because the duplicate blocks are likely already available in the lab - either on the equipment itself from a previous job, or from peers in the same racks which ran a related job. I expect to have a single peer in the cloud and 100+ peers initially in the lab, one per item of equipment. I hope to grow this to all our lab equipment, so perhaps 600-1000 peers. **Performance** So far my experiments suggest IPFS cannot transfer data between two peers faster than ~250 Mbps (25MiB/s), which does not even remotely saturate our network links which are 10+ Gbps. I have not yet tried a scale test with a large number of peers. Should I expect that to be quicker? Is there anything I can tune to increase throughput? Or reduce CPU load, if that is indeed my bottleneck. I benchmarked multihash performance and settled on using blake3 as its fastest on the embedded systems we’re using (which lack HW acceleration for SHA256). The swarm is entirely private, and I find myself wondering if the default (256 kiB) and maximum (1 MiB) chunk sizes are too small to allow network flow control systems to reach full speed. Perhaps due to (e.g.) TCP window sizing? I’ve tried using 1MiB chunk sizes and it didn’t make a measurable difference. I also wonder if enabling more concurrent traffic might help, i.e. larger want lists. Any tips or ideas for experiments to try would be appreciated! ---
**Author:** ![Jorropo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/jorropo/32/3230_2.png) [@Jorropo](https://discuss.ipfs.tech/u/Jorropo)\ **Post date:** [December 15, 2022, 2:06pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/2 "2022-12-15T14:06:02Z")
The data-transfer issues are due to bugs in the go-bitswap implementation. Given that you both of the nodes are trusted and fast, a single peer protocol such as graphsync or car files over http would be much better. You could try my soft: [GitHub - Jorropo/linux2ipfs: Small pipeline and extreme-performance oriented IPFS implementation to upload files and deltas to pinning services very fast.](https://github.com/jorropo/linux2ipfs), it would be fairly easy to add blake3 support. It’s a bit buggy and doesn’t properly parallelise small files but is still much faster than Kubo, Iroh or whatever else. ---
**Author:** ![Jorropo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/jorropo/32/3230_2.png) [@Jorropo](https://discuss.ipfs.tech/u/Jorropo)\ **Post date:** [December 15, 2022, 2:08pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/3 "2022-12-15T14:08:16Z")
> [@meermanr](#): > > These large files are bootable disk images, such as the contents of Linux system running Debian. On a given day most of the disk images will be very similar to one another (perhaps 98% duplicate data between any two given disk images). I missed this, if your disk images are sparse, a software that do `SEEK_HOLE` and `SEEK_DATA` would help a lot. ---
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [December 15, 2022, 2:43pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/4 "2022-12-15T14:43:01Z")
Yes, they are. IPFS deduplicates the empty blocks, which (so far) has felt “good enough” for my purposes. ---
**Author:** ![Jorropo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/jorropo/32/3230_2.png) [@Jorropo](https://discuss.ipfs.tech/u/Jorropo)\ **Post date:** [December 15, 2022, 2:44pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/5 "2022-12-15T14:44:13Z")
Yeah … it could be much better, the default of 256KiB, will require up to 512KiB of empty blocks to see a difference. Linux2ipfs doesn’t deduplicate anything yet so it might actually be worst at this. ---
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [December 15, 2022, 2:50pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/6 "2022-12-15T14:50:09Z")
> Given that you both of the nodes are trusted and fast, a single peer protocol such as graphsync or car files over http would be much better The attraction of IPFS is that it would **autonomously** manage data locality, i.e. pulling from peers in the lab when possible, and degrading gracefully to a more-or-less straight push from the data source to destination. Which is to say that the transport layer always does the right thing, and users of the lab equipment don’t need to micro-manage data caches. > You could try my soft: [GitHub - Jorropo/linux2ipfs: Small pipeline and extreme-performance oriented IPFS implementation to upload files and deltas to pinning services very fast.](https://github.com/jorropo/linux2ipfs), it would be fairly easy to add blake3 support. Interesting, thanks! This appears to be a very fast way of chunking files into CAR format, right? It doesn’t transfer the data between two or more peers in a swarm? ---
**Author:** ![mosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/mosh/32/5637_2.png) [@mosh](https://discuss.ipfs.tech/u/mosh)\ **Post date:** [December 15, 2022, 7:44pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/7 "2022-12-15T19:44:29Z")
Welcome @meermanr! Nice to have you here. As @jorropo mentions, bitswap data transfer has been slower than many IPFS users need. The good news, though, is @bFive and team are working on a new higher-throughput data transfer protocol to be released in ~January. I’ll let him share details. ---
**Author:** ![bFive](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/bfive/32/7124_2.png) [@bFive](https://discuss.ipfs.tech/u/bFive)\ **Post date:** [December 15, 2022, 9:41pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/8 "2022-12-15T21:41:16Z")
👋 @meermanr, super interesting project! As others have mentioned, I work on [Iroh](https://iroh.computer), a bunch of us are actively working on the problem of transfer speeds, so we should have a _better_ solution to your problem. I showed your post to other iroh maintainers & this is the exact kind of thing we’re hoping to fix. However, I’d like to do a _little_ expectation management on your use case, specifically with regard to that fancy 10+Gbps connection you have, and the block de-duplication property you’re after: In practice, I think you’ll be forced to choose between de-duplication and transfer speeds. Why the tradeoff? When you cut up a file & put it into blocks, then store those blocks on a hard drive addressed by hash, reading those blocks back turns into effectively random seeks across your disk, which bottlenecks your capacity to saturate that fancy internet tube. Even with solid state drives, memory mapping, using a database, all the hopping around a merkle tree comes with a cost, and there’s a very real chance we can’t read fast enough to saturate a 10Gpbs connection. I haven’t had the chance to work out the numbers on physical feasibility, but it’s safe to say saturating a 10Gbps connection will require structural changes to the way IPFS works. So, while we’re at it, we should talk about deduplication, and check to make sure it’s actually real. In our tests the amount of _internal_ de-duplication in common IPFS data (UnixFS DAGs, to be specific) we’ve found in the wild is negligible, on the order of less than 5% of the total content. Your example is looking at de-duplication across two different graphs. If those graphs are UnixFS graphs of filesystem data, you’ll surely get massive amounts of de-duplication, whenever files or directories exactly match. But if you’re putting 2 slightly different ISO images into IPFS, that will be treated as large byte streams, and for that you’d definitly want to look into the [rabin chunker](https://docs.ipfs.tech/reference/kubo/cli/#ipfs-add) if you’re trying to maximize de-dupe. I still think you can have you cake and eat it too. A swam with faster transfer & smarter caching should make up for a lack of de-duplication, but we’re going to have to break a bunch of stuff first 😄. Ps: we too are fans of blake3, like, _big_ fans. More on that soon. ---
**Author:** ![Jorropo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/jorropo/32/3230_2.png) [@Jorropo](https://discuss.ipfs.tech/u/Jorropo)\ **Post date:** [December 16, 2022, 12:00am UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/9 "2022-12-16T00:00:03Z")
Bitswap, the only data transfer implementations that fully supports automatic data locality is not fast. I want to fix that [RAPIDE - Jorropo - YouTube](https://www.youtube.com/watch?v=376k8B9g7CE) but this doesn’t exists yet _soonTM_. We also have a datatransfer working group that has been kicked off at 2022 IPFS Camp: [https://www.youtube.com/playlist?list=PLuhRWgmPaHtQ--aQ5GlgCyKkQYXUfQpVf](https://www.youtube.com/playlist?list=PLuhRWgmPaHtQ--aQ5GlgCyKkQYXUfQpVf) Currently, you have various other solutions which are faster and have more or less level of magic happening (graphsync, …). The fastest one is to send `.car` files over HTTP to a cluster of server. ---
**Author:** ![Jorropo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/jorropo/32/3230_2.png) [@Jorropo](https://discuss.ipfs.tech/u/Jorropo)\ **Post date:** [December 16, 2022, 12:02am UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/10 "2022-12-16T00:02:02Z")
> [@meermanr](#): > > This appears to be a very fast way of chunking files into CAR format, right? It doesn’t transfer the data between two or more peers in a swarm? The only data transfer it knows to do is to stream the chunking output over HTTP POST requests. I brought this up because I guess you are using `ipfs add` and just `ipfs add`ing data is not fast (altho it’s orders of magnitude than the `go-bitswap` client & server shipping in Kubo), so I guessed that `ipfs add`ing your files would take a while. ---
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [December 16, 2022, 10:46am UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/11 "2022-12-16T10:46:39Z")
> The only data transfer it knows to do is to stream the chunking output over HTTP POST requests. I’d like to avoid any explicit network addresses - the magic of IPFS is that once the producers (cloud) and consumers (lab equipment) are members of the swarm I don’t need to micro-manage network addresses - I need only concern myself with the content (CIDs). So I am trying to avoid pushing the data directly using (e.g.) rsync or S3, because then I need to explicitly manage where data resides. For example, if I used S3 over HTTP, I _could_ use a caching proxy in the lab to accelerate data transfer, but I want to avoid that for a couple of reasons: 1. The throughput requirements of this caching proxy server would make it quite expensive - perhaps a storage cluster in its own right. 2. The threat model and security implications of caching data from multiple projects on a shared system would place burdens on my team, which I think we can avoid. As regards the performance of centrally provisioned storage: our other (older) lab has a central NFS server providing for ~400 devices, non of which have any locally attached storage. Since this is all on-premise, adding more capacity of performance takes months and a significant expense. For the new lab I want the opposite: every device has locally attached storage, and no central shared storage. If we use peer-to-peer file sharing between devices which are assigned to the same project, and I find a way to deduplicate the data over both space and time\[1\], then I think I can avoid having to manage shared storage systems. Specifically, I imagine running multiple private IPFS swarms, one per confidentiality domain (i.e. project). The IPFS peers (devices in the lab) would be moved between swarms regularly to essentially time share them between the various projects,\[2\] and avoid my team having any responsibility for data at rest. > I brought this up because I guess you are using `ipfs add` and just `ipfs add` ing data is not fast (altho it’s orders of magnitude than the `go-bitswap` client & server shipping in Kubo), so I guessed that `ipfs add` ing your files would take a while. Could I use this to quickly create a CAR file, and then import it with `ipfs dag import`? As you can see in my explanation above, I expect to _seed_ a swarm from a singe producer, but I expect a lot of fan-out and bit-swapping between the peers in the lab (who may wall want the same data if the user is running a parallel regression test, etc). * * * 1. Deduplication over space: one producer with a fan-out to many consumers, e.g. a test campaign. Deduplication of time: new content being 98% identical to previous content, e.g. edit-compile-test cycles of an individual developer. 2. We deep-clean all persistent storage on a given device when reassigning it. The host running IPFS runs from a RAM disk (i.e. no changes are persisted between reboots) and the locally-attached storage is encrypted using [LUKS](https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup) on every boot. So a reboot discards the decryption key. ---
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [December 16, 2022, 10:49am UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/12 "2022-12-16T10:49:06Z")
> [@Jorropo](#): > > I want to fix that [RAPIDE - Jorropo - YouTube](https://www.youtube.com/watch?v=376k8B9g7CE) but this doesn’t exists yet _soonTM_. That looks like exactly what I am after! Thank you for sharing it. It reminds me of bittorrent’s super seed feature, where the sole provider of some unique content will _only_ sent out chunks it has not previous sent out (and then remove that chunk from its have list). This is the sort of behaviour that would be perfect for my use-case. (And I think you touched on this in the data transfer working group - thank you for linking me to that!) ---
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [December 16, 2022, 11:27pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/13 "2022-12-16T23:27:08Z")
> [@bFive](#): > > In practice, I think you’ll be forced to choose between de-duplication and transfer speeds I was hoping to tune my workflows to maximise naive deduplication in IPFS. For example, if every disk image was derived from the same base image by mounting it, modifying a couple of files, unmounting it and then adding it to IPFS, I would expect that 98% of the disk image would be as it was before being mounted. So I expect that a fixed-size chunker, such as the default `--chunker size-262144`, would come to the same conclusion: 98% of the original and modified disk images are unaltered: same data at exactly the same offset. Therefore when the user transfers their modified disk image to the lab, I would expect that only 2% of the disk image is actually transmitted by that user since the other 98% of the data is already floating around the lab. I have been wondering if my team should write a custom chunker to exploit the structure of the disk images - GUID partition tables, ExFAT / ext4 file-systems mostly. I don’t think we can use UnixFS because it doesn’t encoded extended attributes, which we need to satisfy SELinux boot conditions. ---
**Author:** ![Dreamacro](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/dreamacro/32/8004_2.png) [@Dreamacro](https://discuss.ipfs.tech/u/Dreamacro)\ **Post date:** [December 29, 2023, 10:05am UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/14 "2023-12-29T10:05:23Z")
`The data-transfer issues are due to bugs in the go-bitswap implementation.` I’ve been using bitswap to transfer data across multiple clusters smaller than a thousand in the recent past and have been plagued by this issue as well. What specifically is being implemented incorrectly that is causing the rate to be low? I’ve looked up two possible issues and I’d like to help fix it, would you mind expanding it? > > > This is a bug I introduced in 9cb5cb54d40b57084d1221ba83b9e6bb3fcc3197 when fixi…ng CVE-2023-25568. > > Here is a screenshot of our gateway's wantlists: > !\[Screenshot 2023-12-19 at 13-36-37 View panel - Gateway - Main - IPFS Gateway - Dashboards - Grafana\](https://github.com/ipfs/boxo/assets/24391983/235a6e55-e3a6-4886-9dfd-69b1a9448858) > > You can see that many gateways instances have more than 1024 entries. > This cause issues with: > https://github.com/ipfs/boxo/blob/9cb5cb54d40b57084d1221ba83b9e6bb3fcc3197/bitswap/server/internal/decision/engine.go#L669-L677 > https://github.com/ipfs/boxo/blob/9cb5cb54d40b57084d1221ba83b9e6bb3fcc3197/bitswap/server/internal/decision/engine.go#L765 > https://github.com/ipfs/boxo/blob/9cb5cb54d40b57084d1221ba83b9e6bb3fcc3197/bitswap/server/internal/decision/engine.go#L851 > > Theses three sections of code are needed to fix CVE-2023-25568, they make the server ignore any new queries overflowing 1024. Previously the server would remember infinitely many CIDs eventually OOMing. > However the truncation code always prefer keeping existing entries over new ones. > > This means we can get in a situation where we get stuck: > \- let's say the client send a 2048 entries wantlist. > \- out of theses we only have 1/3 of the CIDs, and theses are uniformly distributed. > \- the server truncate the wantlist to 1024. > \- the server starts serving theses entries first, out of theses we don't have 683 CIDs. > \- we now only have 341 effective wantlist size. This is because the bitswap server never cleanup entries after sending \`DONT\_HAVE\`. > The point of this feature is for \`-1\` scalling, if the server is also downloading the same blocks, it might get them after having already sent \`DONT\_HAVE\`, then the server can either send the block or the a \`HAVE\` message overriding the previous \`DONT\_HAVE\`. > \- This repeat each time shrinking the usable wantlist on this connection because the part of \*\*the wantlist the server is willing to keep fills up with CIDs it does not have\*\*. > Eventually reaching 0 > > (note: the client can send CANCEL or a message with the full flag and theses 1024 "stuck" CIDs will be cleaned out properly, but the client isn't smart enough to realize this is happening) > > \--- > > I see three possible solutions: > > \### Truncate left instead of truncating right. > > This means newer entries would override older ones, instead of having older ones stick around. > My original thinking when writing the fix is that the server currently does not apply back-pressure, so we should let existing entries first so we can make some progress and clear them out when we send responses. If we always cancel what is already in flight a client that is \*too fast\* would never get any blocks because everything would be canceled before being sent. > This could be fixed by only canceling what is already queued but not entries that are actively being worked on. > > \### Completely rewrite the server and architecture it request response > > If we handled messages in a request response manner we could apply back-pressure when the client is sending \*too many queries\*. > We could still have a global \`CID → \[PeerID\]\` wantlist map however this would be limited to CIDs we don't have, for \`-1\` scalling. CIDs we already have would be completely skip this flow and the queue and be handled purely in the message handler. > We would still need a limit on how many \`-1\` tracked CIDs we have, but that means this bug would only impact blocks we don't have which SGTM. > > \### Make the client aware and handle rotation itself by truncating itself and handle canceling overwrote entries. > > This is nice because it does not require updating the servers, only the client, so fixed client can download from buggy servers. > Sounds PITA to code and I don't really want to do it tbh. Also this means we commit to having broke the protocol. > > > Experienced via Lassie's CI suite which has suddenly become flaky with some new …workloads. Using the \[testing/sws-logging\](https://github.com/ipfs/boxo/tree/testing/sws-logging) branch here (started by @hannahhoward, continued by me) to diagnose with additional debug printing. It evidences in an inability to communicate with a peer that we know has the block, and bitswap has been told has the block. > > See logs below, which come from \[here\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371) using 715365a23d36ccd0fe2c9cf95ec16afa0d50f39b on the \[testing/sws-logging\](https://github.com/ipfs/boxo/tree/testing/sws-logging) branch. Here's what I believe happens: > > \* \`session#findMorePeers\` -\> \`ProviderQueryManager#FindProvidersAsync\` > \* \`ProviderQueryManager#findProviderWorker\` does its thing, processing the queue of providers to connect to, performs a \`pqm.network.ConnectTo()\` which is goes through (bitswap/network/ipfs\_impl.go) \`bsnet#ConnectTo()\` to just do a basic \`host.Connect()\` > \* We get a \`Connected()\` notification which is received by (bitswap/network/ipfs\_impl.go) \`connectEventManager#Connected()\` and it does a \`c.setState(p, stateResponsive)\` on that peer. > \* Meanwhile, the \`ProviderQueryManager#FindProvidersAsync\` has returned, the \`Connect()\` returned, we even got a \`Connected()\`, so the \`Session\` invokes \`sessionWantSender#Update(peer, nil, have, nil)\`, i.e. the \`peer\` has the cid \`have\`, this is a "change". > \* In \`sessionWantSender\` we go from \`onChange()\` to \`processUpdates()\` to \`updateWantBlockPresence()\` to \`wantInfo#setPeerBlockPresence()\` to \`wantInfo#calculateBestPeer()\` - we see these in the logs > \* After the call to \`processUpdates()\`, \`sessionWantSender\` moves on to \`sendNextWants()\`, which compiles them and calls \`sendWants()\` which in turn calls \`PeerManager#SendWants()\`. > \* \*\*BUT\*\* it turns out that \`PeerManager\` doesn't have a \`peerQueues\` entry for this peer .. yet .. > \* Back in \`connectEventManager\`, the \`worker()\` (goroutine) loop finally picks up a change, and sees that \`p\` is now \`stateResponsive\` so it ends up calling \`Bitswap#PeerConnected()\` which calls \`Client#PeerConnected()\` which calls \`PeerManager#Connected()\` which calls \`PeerManager#getOrCreate()\` that finally sets up a \`peerQueues\` entry for that peer; so it's now ready to talk (\_"hey, what'd I miss?"\_). > > I'm a bit fuzzy on what the logs show beyond here and why it doesn't manage to rectify the situation with retries; I guess because it had one chance to take the peer from \`FindProvidersAsync\` and ask it for the block, and it failed that one chance and there isn't really another. > > The async disconnect in \`connectEventManager\` seems to be the main problem here, perhaps a call to \`Connected()\` shouldn't return until the state change has been fully propagated. > > \ > \Relevant logs\ > > \`\`\` > 2023-08-15T09:52:43.126Z DEBUG bs:sess session/session.go:361 FindMorePeers {"session": 1, "cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "pending": 1} > 2023-08-15T09:52:43.126Z DEBUG bitswap providerquerymanager/providerquerymanager.go:332 New Provider Query on cid: bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq > 2023-08-15T09:52:43.126Z DEBUG bitswap providerquerymanager/providerquerymanager.go:234 Beginning Find Provider Request for cid: bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq > 2023-08-15T09:52:43.126Z DEBUG lassie/bitswap bitswaphelpers/indexerrouting.go:86 provider records requested from bitswap, sending back indexer results {"providerCount": 1} > 2023-08-15T09:52:43.126Z DEBUG bitswap providerquerymanager/providerquerymanager.go:244 Connecting to provider 1WRsHVY1dhNKpD... > 2023-08-15T09:52:43.126Z DEBUG basichost basic/basic\_host.go:737 host QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 dialing 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.126Z DEBUG mocknet mock/mock\_peernet.go:119 QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 (newly) dialing 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.126Z DEBUG mocknet mock/mock\_peernet.go:132 QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 dialing 1WRsHVY1dhNKpD openingConn > 2023-08-15T09:52:43.126Z DEBUG mocknet mock/mock\_peernet.go:140 QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 opening connection to 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.126Z DEBUG mocknet mock/mock\_peernet.go:183 1WRsHVY1dhNKpD accepting connection from QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 > 2023-08-15T09:52:43.126Z DEBUG bitswap\_network network/connecteventmanager.go:159 connectEventManager connected to peer, setting responsive {"peer": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1"} > 2023-08-15T09:52:43.126Z DEBUG bitswap\_network network/connecteventmanager.go:140 connectEventManager detected peer connectivity {"peer": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1"} > 2023-08-15T09:52:43.126Z DEBUG mocknet mock/mock\_conn.go:148 Conn.NewStreamWithProtocol: QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 --\> 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.126Z DEBUG bitswap\_network network/connecteventmanager.go:159 connectEventManager connected to peer, setting responsive {"peer": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.126Z DEBUG mocknet mock/mock\_conn.go:148 Conn.NewStreamWithProtocol: 1WRsHVY1dhNKpD --\> QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 > 2023-08-15T09:52:43.127Z DEBUG basichost basic/basic\_host.go:433 negotiated: /ipfs/id/1.0.0 (took 25.701µs) > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:463 sending snapshot {"seq": 2, "protocols": \["/fil/datatransfer/1.2.0","/ipfs/bitswap","/ipfs/bitswap/1.0.0","/ipfs/bitswap/1.1.0","/ipfs/bitswap/1.2.0","/ipfs/graphsync/2.0.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0"\], "addrs": \["/ip6/100::2438:fce9:9d50:c714/tcp/4242"\]} > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:468 /ipfs/id/1.0.0 sending message to 1WRsHVY1dhNKpD /ip4/127.0.0.1/tcp/15381 > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:519 /ipfs/id/1.0.0 received message from QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 /ip6/100::2438:fce9:9d50:c714/tcp/4242 > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:787 1WRsHVY1dhNKpD received listen addrs for QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1: \[/ip6/100::2438:fce9:9d50:c714/tcp/4242\] > 2023-08-15T09:52:43.127Z DEBUG basichost basic/basic\_host.go:433 negotiated: /ipfs/id/1.0.0 (took 398.503µs) > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:463 sending snapshot {"seq": 2, "protocols": \["/ipfs/bitswap","/ipfs/bitswap/1.0.0","/ipfs/bitswap/1.1.0","/ipfs/bitswap/1.2.0","/ipfs/id/1.0.0","/ipfs/id/push/1.0.0"\], "addrs": \["/ip4/127.0.0.1/tcp/15381"\]} > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:468 /ipfs/id/1.0.0 sending message to QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 /ip6/100::2438:fce9:9d50:c714/tcp/4242 > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:519 /ipfs/id/1.0.0 received message from 1WRsHVY1dhNKpD /ip4/127.0.0.1/tcp/15381 > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/id.go:787 QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 received listen addrs for 1WRsHVY1dhNKpD: \[/ip4/127.0.0.1/tcp/15381\] > 2023-08-15T09:52:43.127Z WARN net/identify identify/id.go:811 QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 cannot unmarshal key from remote peer: 1WRsHVY1dhNKpD, asn1: structure error: tags don't match (16 vs {class:3 tag:27 length:52 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:\ tag:\ stringType:0 timeType:0 set:false omitEmpty:false} publicKeyInfo @2 > 2023-08-15T09:52:43.127Z DEBUG basichost basic/basic\_host.go:754 host QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 finished dialing 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.127Z DEBUG bitswap providerquerymanager/providerquerymanager.go:250 Connected to provider 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.127Z DEBUG bitswap providerquerymanager/providerquerymanager.go:264 Found 0 providers for cid: bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq > 2023-08-15T09:52:43.127Z DEBUG net/identify identify/obsaddr.go:442 added own observed listen addr {"observed": "/ip6/100::2438:fce9:9d50:c714/tcp/4242"} > 2023-08-15T09:52:43.127Z DEBUG bitswap providerquerymanager/providerquerymanager.go:332 Received provider (1WRsHVY1dhNKpD) for cid (bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq) > 2023-08-15T09:52:43.127Z DEBUG bitswap providerquerymanager/providerquerymanager.go:332 Finished Provider Query on cid: bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq > 2023-08-15T09:52:43.127Z DEBUG bs:peermgr peermanager/peermanager.go:214 pm RegisterSession {"peer": "1WRsHVY1dhNKpD", "session": 1} > 2023-08-15T09:52:43.127Z DEBUG bs:sprmgr sessionpeermanager/sessionpeermanager.go:68 Bitswap: Added peer to session {"session": 1, "peer": "1WRsHVY1dhNKpD", "peerCount": 1} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:\[65\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:67)7 sws update want block presence {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:764 sws new best peer {"peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:657 sws update want block presence {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:764 sws new best peer {"peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:535 sws send next wants {"newly available": \["1WRsHVY1dhNKpD"\]} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:558 sws send new want block {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:580 sws sending wants {"sends": 1} > 2023-08-15T09:52:43.127Z DEBUG bs:peermgr peermanager/peermanager.go:155 pm SendWants {"peer": "1WRsHVY1dhNKpD", "wantBlocks": 1, "wantHaves": 1} > 2023-08-15T09:52:43.127Z DEBUG bs:peermgr peermanager/peermanager.go:159 pm SendWants no peerQueue for {"peer": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bitswap\_network network/connecteventmanager.go:140 connectEventManager detected peer connectivity {"peer": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:peermgr peermanager/peermanager.go:91 pm Connected {"peer": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:peermgr peermanager/peermanager.go:203 pm getOrCreate new peer {"peer": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bitswap messagequeue/messagequeue.go:280 mq: add want have from broadcast {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:535 sws send next wants {"newly available": null} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:546 sws want block in process, no new want block sent {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq"} > 2023-08-15T09:52:43.127Z DEBUG bs:sess session/sessionwantsender.go:580 sws sending wants {"sends": 0} > 2023-08-15T09:52:43.129Z DEBUG mocknet mock/mock\_conn.go:148 Conn.NewStreamWithProtocol: QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 --\> 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.129Z DEBUG bitswap messagequeue/messagequeue.go:\[66\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:68)6 sent message {"type": "WANT\_HAVE", "cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "local": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1", "to": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.129Z WARN bitswap\_network network/ipfs\_impl.go:245 error setting deadline: set pipe: deadline not supported > 2023-08-15T09:52:43.130Z DEBUG mocknet mock/mock\_conn.go:148 Conn.NewStreamWithProtocol: QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 --\> 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.130Z WARN bitswap\_network network/ipfs\_impl.go:269 error resetting deadline: set pipe: deadline not supported > 2023-08-15T09:52:43.130Z DEBUG bs:sess session/session.go:4\[67\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:69) broadcastWantHaves {"session": 1, "cids": \[{"/":"bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq"}\]} > 2023-08-15T09:52:43.130Z DEBUG basichost basic/basic\_host.go:433 negotiated: /ipfs/bitswap/1.2.0 (took 800.106µs) > 2023-08-15T09:52:43.131Z DEBUG bitswap\_network network/ipfs\_impl.go:427 bitswap net handleNewStream from QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 > 2023-08-15T09:52:43.131Z DEBUG engine decision/engine.go:634 Bitswap engine \<- msg {"local": "1WRsHVY1dhNKpD", "from": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1", "entryCount": 1} > 2023-08-15T09:52:43.131Z DEBUG engine decision/engine.go:638 Bitswap engine \<- want-have {"local": "1WRsHVY1dhNKpD", "from": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1", "cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq"} > LinkSystemBlockstore.GetSize bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq > LinkSystemBlockstore.GetSize= bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq 1332 > 2023-08-15T09:52:43.132Z DEBUG engine decision/engine.go:779 Bitswap engine: block found {"local": "1WRsHVY1dhNKpD", "from": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1", "cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "isWantBlock": false} > 2023-08-15T09:52:43.132Z DEBUG engine decision/engine.go:545 Bitswap process tasks {"local": "1WRsHVY1dhNKpD", "taskCount": 1} > 2023-08-15T09:52:43.132Z DEBUG engine decision/engine.go:598 Bitswap engine -\> msg {"local": "1WRsHVY1dhNKpD", "to": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1", "blockCount": 0, "presenceCount": 1, "size": 38} > 2023-08-15T09:52:43.133Z DEBUG mocknet mock/mock\_conn.go:148 Conn.NewStreamWithProtocol: 1WRsHVY1dhNKpD --\> QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1 > 2023-08-15T09:52:43.133Z WARN bitswap\_network network/ipfs\_impl.go:245 error setting deadline: set pipe: deadline not supported > 2023-08-15T09:52:43.133Z DEBUG basichost basic/basic\_host.go:413 protocol mux failed: stream reset (took 2.485821ms) > 2023-08-15T09:52:43.133Z WARN bitswap\_network network/ipfs\_impl.go:2\[69\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:71) error resetting deadline: set pipe: deadline not supported > 2023-08-15T09:52:43.133Z DEBUG basichost basic/basic\_host.go:433 negotiated: /ipfs/bitswap/1.2.0 (took 321.\[70\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:72)3µs) > 2023-08-15T09:52:43.134Z DEBUG bitswap\_network network/ipfs\_impl.go:427 bitswap net handleNewStream from 1WRsHVY1dhNKpD > 2023-08-15T09:52:43.134Z DEBUG bs:sess session/session.go:221 Bitswap \<- HAVE {"local": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1", "from": "1WRsHVY1dhNKpD", "cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "session": 1} > 2023-08-15T09:52:43.134Z DEBUG bs:peermgr peermanager/peermanager.go:214 pm RegisterSession {"peer": "1WRsHVY1dhNKpD", "session": 1} > 2023-08-15T09:52:43.134Z DEBUG bs:sess session/sessionwantsender.go:657 sws update want block presence {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.134Z DEBUG bs:sess session/sessionwantsender.go:\[76\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:78)4 sws new best peer {"peerID": "1WRsHVY1dhNKpD"} > 2023-08-15T09:52:43.134Z DEBUG bs:sess session/sessionwantsender.go:535 sws send next wants {"newly available": null} > 2023-08-15T09:52:43.134Z DEBUG bs:sess session/sessionwantsender.go:546 sws want block in process, no new want block sent {"cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq"} > 2023-08-15T09:52:43.134Z DEBUG bs:sess session/sessionwantsender.go:580 sws sending wants {"sends": 0} > 2023-08-15T09:52:43.134Z DEBUG bitswap-server server/server.go:325 sent message {"type": "HAVE", "cid": "bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq", "local": "1WRsHVY1dhNKpD", "to": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1"} > 2023-08-15T09:52:43.135Z DEBUG bitswap-server server/server.go:3\[79\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:81) sent message {"peer": "QmRR8VvuQf4Fvr2ptGJj2qjJkmqREJXbF4jv8KkSRLsMj1"} > 2023-08-15T09:52:43.135Z DEBUG bitswap-server server/server.go:2\[82\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:84) Bitswap.TaskWorker.Loop {"ID": 0} > 2023-08-15T09:52:43.\[139\](https://github.com/filecoin-project/lassie/actions/runs/5865827197/job/15903479085?pr=371#step:63:141)Z DEBUG bs:sess session/session.go:467 broadcastWantHaves {"session": 1, "cids": \[{"/":"bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq"}\]} > 2023-08-15T09:52:43.151Z DEBUG bs:sess session/session.go:467 broadcastWantHaves {"session": 1, "cids": \[{"/":"bafybeifrrglx2issn2had5rtstn3xltla6vxmpjfwfz7o3hapvkynh4zoq"}\]} > \`\`\` > \ ---
**Author:** ![wscott](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/wscott/32/33_2.png) [@wscott](https://discuss.ipfs.tech/u/wscott)\ **Post date:** [November 4, 2025, 3:09pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/15 "2025-11-04T15:09:33Z")
I know this is an old thread and clearly a lot has changed in the last couple years so I want to ask what the answers look like now. I have fairly similar situation I am considering: - lots of machines - About a dozen clusters of machines around the world - Each cluster has thousands of machines in close proximity - each machine has good specs, a good network connection and like 1TB of local scratch space - I have relatively static bundles of files that are needed for batch jobs. - These are a handful of files totaling 20MB to 3GB in size. - The average bundle is around 100MB. - I expect little deduplication savings between bundles. - Currently this is cached using a perl script that writes to a NFS volume local to the current cluster and then the jobs read the input over NFS. - some stats - each machine fetches data like once an hour - total queries on the network is around 100k/hr - it is estimated that a local machine would get a 50-75% hit rate from a local cache using that scratch space - the per-cluster NFS cache gets a 97% hit rate. Just new files basically The hope is if I could use a more p2p cache I could avoid all the NFS traffic and decrease latency however the p2p network connection is slower than the connection to the NFS server. How would a private IPFS swarm perform in this environment? What else should I look at? ---
**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\ **Post date:** [November 13, 2025, 8:23pm UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/16 "2025-11-13T20:23:11Z")
Maybe not much to go on, but since about v0.30 I’ve been pretty happy with the performance of my private swarm: I’ve seen nodes receiving data at 90 MB/s which is good enough for my use-case, and was certainly faster then the HTTP transfers over the WAN it replaced (~30 MB/s). Latency has been somewhat unpredictable, and seems to mostly be related to how long it might take for a new node to join the swam and receive blocks from its published wantlist. My use-case involves a lot of ephemeral nodes that come online just to retrieve an update and then disappear again as they install the update. I suspect that if more of the nodes were persistent it would perform better, but I’ve no experience there to share. ---
**Author:** ![hector](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/hector/32/178_2.png) [@hector](https://discuss.ipfs.tech/u/hector)\ **Post date:** [November 14, 2025, 8:20am UTC](https://discuss.ipfs.tech/t/how-to-tune-a-private-ipfs-swarm-for-large-files/15702/17 "2025-11-14T08:20:45Z")
Starting point for faster transfer is: - To chunk files with 1MiB chunks (`"UnixFSChunker": "size-1048576",`in the config or via `ipfs add` flag). - Increase options in [kubo/docs/config.md at master · ipfs/kubo · GitHub](https://github.com/ipfs/kubo/blob/master/docs/config.md#internalbitswap) - For 128GiB RAM: ```json "Internal": { "Bitswap": { "EngineBlockstoreWorkerCount": 2500, "EngineTaskWorkerCount": 500, "MaxOutstandingBytesPerPeer": 1048576, "TaskWorkerCount": 500 } } ``` Of course depending on the topology and size etc. you may want to play with connection manager, DHT/providing settings (if everyone is mostly connected to everyone in a private network, dht is not needed). If the files will never be meant to be fetched standard clients, you could build a custom Kubo with support for larger chunk sizes. If money allows you could also touch base with [ipshipyard.com](http://ipshipyard.com) to get a closer look into the specifics of your setup.