# Distributed directory with millions of files

**URL:** <https://discuss.ipfs.tech/t/distributed-directory-with-millions-of-files/20357>\
**Category:** Help\
**Created:** [September 22, 2026, 2:21pm UTC](https://discuss.ipfs.tech/t/distributed-directory-with-millions-of-files/20357 "2026-09-22T14:21:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ki4jgt](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/ki4jgt/32/3871_2.png) [@ki4jgt](https://discuss.ipfs.tech/u/ki4jgt)\
**Post date:** [September 22, 2026, 2:21pm UTC](https://discuss.ipfs.tech/t/distributed-directory-with-millions-of-files/20357/1 "2026-09-22T14:21:17Z")

</div>

Have a directory with millions of files, and would like to host it via IPFS.

Is there a way to host it over multiple machines, as it’s way too large for one machine.

---

<div class="post-metadata">

**Author:** ![Akita](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/akita/32/2650_2.png) [@Akita](https://discuss.ipfs.tech/u/Akita)\
**Post date:** [September 25, 2026, 7:59am UTC](https://discuss.ipfs.tech/t/distributed-directory-with-millions-of-files/20357/2 "2026-09-25T07:59:33Z")

</div>

I think you should check out IPFS Clusters and see if it works for you.

> **[IPFS Cluster](https://ipfscluster.io/)**
>
> Pinset orchestration for IPFS

---

<div class="post-metadata">

**Author:** ![hector](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/hector/32/178_2.png) [@hector](https://discuss.ipfs.tech/u/hector)\
**Post date:** [October 7, 2026, 7:10am UTC](https://discuss.ipfs.tech/t/distributed-directory-with-millions-of-files/20357/3 "2026-10-07T07:10:52Z")

</div>

Hello,

> Is there a way to host it over multiple machines

Yes, roughly:

1. List the files in the directory and split it into groups that fit in a machine.
2. Create new directories or ipld nodes that reference the files in each group.
3. Pin each of them in a different machine.
4. Pin the root node (or the HAMT nodes) of the original directory (without the leaves/files) if you need a reference to all the files.

Now, if the directory is big and flat, it is probably already a HAMT structure. So you might be able to pin sub-dags of that HAMT to different places separately without having to split into groups yourself by hand.. `ipfs refs -r --max-depth=...` can help you discover how deep/branched the HAMT is before referencing leaves. The root should have children and the children should have children that are all part of the HAMT rather than file-cids.
