Adding folder with 49999 files and +- 500 mb stuck

Help
1

Hi guys, we face a problem that we are not able to add a folder with 49999 files in it roughly 500mb big…

via the web interface it’s always stuck at 57% and doesn’t continue…

do we try via ADD files via explorer, not much happens… .do we use the ipfs app, it’s stuck at 62% and gives us a red cross behind the upload after a while

any tips or suggestions?

1 Like
2

Hi Guys, anyone ideas?

Many small files of 10kb, everything above 1000 files and the app hangst, and basicly never finishes the transfer, is there a way i can change some config files to avoid this issue?

3

Does it work adding the folder through the CLI? It sounds like a frontend issue…

4

tried via cli add folder, same problem. i guess ipfs has issues with a lot of 10kb files i think… i guess im not the only one

5

so i can add multiple GB large files no issues, loaded in seconds…

but many over 1000’s of small 10kb files is an issue, or at least impossible… any suggestions? :slight_smile:

6

Im trying (ipfs windows app) to change from flatfs filesystem to badger, to see if we got improvement there.

Can someone put me in the right direction, what to change here to make it work? :slight_smile:

“Datastore”: {
“BloomFilterSize”: 0,
“GCPeriod”: “1h”,
“HashOnRead”: false,
“Spec”: {
“mounts”: [
{
“child”: {
“path”: “blocks”,
“shardFunc”: “/repo/flatfs/shard/v1/next-to-last/2”,
“sync”: true,
“type”: “badger”
},
“mountpoint”: “/blocks”,
“prefix”: “badger.datastore”,
“type”: “measure”
},
{
“child”: {
“compression”: “none”,
“path”: “datastore”,
“type”: “levelds”
},
“mountpoint”: “/”,
“prefix”: “leveldb.datastore”,
“type”: “measure”
}
],
“type”: “mount”
},
“StorageGCWatermark”: 90,
“StorageMax”: “10GB”
},
“Discovery”: {
“MDNS”: {
“Enabled”: true
}

7

I tried adding a folder of 50000 files of 10Kb (flatfs) and I have no issues (v0.21.0-rc3). Any errors on your logs?

8

You may try sync: false on the repo config, and/or disable the Provider just in case it is some queue getting full there.

9

Hi,

sync set to false as per config output below:

“Datastore”: {
“BloomFilterSize”: 0,
“GCPeriod”: “1h”,
“HashOnRead”: false,
“Spec”: {
“mounts”: [
{
“child”: {
“path”: “blocks”,
“shardFunc”: “/repo/flatfs/shard/v1/next-to-last/2”,
“sync”: false,
“type”: “flatfs”
},
“mountpoint”: “/blocks”,
“prefix”: “flatfs.datastore”,
“type”: “measure”
},
{
“child”: {
“compression”: “none”,
“path”: “datastore”,
“type”: “levelds”
},
“mountpoint”: “/”,
“prefix”: “leveldb.datastore”,
“type”: “measure”
}
],
“type”: “mount”
},
“StorageGCWatermark”: 90,
“StorageMax”: “500GB”
},
“Discovery”: {
“MDNS”: {
“Enabled”: true
}
},

what do you mean with “disabled the provider” ? :slight_smile:

im using windows ipfs app (gui) im trying with the sync false now… if it stops or doesn’t finish at all, ill get a log file

10

kubo/docs/config.md at master · ipfs/kubo · GitHub, or start as ipfs deamon --offline