It becomes very slow after add millions of files

Help
1

I added millions of files, every file is about 1k, it becomes very slow to add more files（about 3 seconds）.The CPU average is very high. It is a private network with 2 peers.

1 Like
2

I’d suggest trying out the badger datastore instead of the current default.

2 Likes
3

I also have millions of pins and I’'ve tried both the default datastore and badger. Both gradually get slower at adding/pinning the more pins you have.

4

In that case I’m guessing you’re hitting this issue then: https://github.com/ipfs/go-ipfs/issues/3874.

If you don’t need the added files to be pinned, then there might be some things you can try to speed up ipfs add (for example, not pinning files on add by default).

1 Like
5

I solve it by setting pin=false , thank you very much

1 Like