Adding content to IPFS is quite slow...Any ideas on why and how to speed things up?

I am adding contents to ipfs and I cant help but notice it takes quite a while.

time rsync rsync://remote/dir ./local_dir

This took

real	1m4.086s
user	0m1.911s
sys	0m37.843s

The size of local_dir is:

du -sh ./local_dir/
205M	./local_dir/

Now when I run:

time ipfs add -r ./local_dir

This took:

real	40m48.501s
user	1m29.585s
sys	2m44.060s

That is x40 time slower!!!

Without changing the contents of ./local_dir I ran again:

time ipfs add -r ./local_dir

This took:

real	1m49.149s
user	0m18.593s
sys	0m27.180s

so some questions…
…how come ingesting into IPFS took ~40times longer than fetching the content from rsync…
…how come, with no changes in the content, re-adding into ipfs still took slightly longer than original rysnc fetch?
…most importantly, is there any way to speed things up?

Thanks!

The current default datastore is slow for small writes. I recommend trying the badger datastore. You can configure it on init by running ipfs init --profile=badgerds.

This datastore should become the default in the near future once https://github.com/ipfs/go-ds-badger/issues/86 and https://github.com/ipfs/go-ds-badger/issues/54 are fixed.

With ipfs init --profile=badgerds.

first time

real	1m43.094s
user	0m56.856s
sys	0m31.211s

second time

real	1m3.304s
user	0m17.485s
sys	0m23.332s

This is definitely much better :slight_smile:

Thanks!