Adding content to IPFS is quite slow...Any ideas on why and how to speed things up?

Help
1

I am adding contents to ipfs and I cant help but notice it takes quite a while.

time rsync rsync://remote/dir ./local_dir

This took

real	1m4.086s
user	0m1.911s
sys	0m37.843s

The size of local_dir is:

du -sh ./local_dir/
205M	./local_dir/

Now when I run:

time ipfs add -r ./local_dir

This took:

real	40m48.501s
user	1m29.585s
sys	2m44.060s

That is x40 time slower!!!

Without changing the contents of ./local_dir I ran again:

time ipfs add -r ./local_dir

This took:

real	1m49.149s
user	0m18.593s
sys	0m27.180s

so some questions…
…how come ingesting into IPFS took ~40times longer than fetching the content from rsync…
…how come, with no changes in the content, re-adding into ipfs still took slightly longer than original rysnc fetch?
…most importantly, is there any way to speed things up?

Thanks!

2

The current default datastore is slow for small writes. I recommend trying the badger datastore. You can configure it on init by running ipfs init --profile=badgerds.

This datastore should become the default in the near future once https://github.com/ipfs/go-ds-badger/issues/86 and https://github.com/ipfs/go-ds-badger/issues/54 are fixed.

3

With ipfs init --profile=badgerds.

first time

real	1m43.094s
user	0m56.856s
sys	0m31.211s

second time

real	1m3.304s
user	0m17.485s
sys	0m23.332s

This is definitely much better :slight_smile:

Thanks!