Can anyone talk about the comparison between Minio cluster and IPFS cluster?

Help IPFS Cluster
1

Minio is said to be the fastest object storage in the world. IPFS is also object storage. Which is faster than Minio?
Minio supports erasure, does IPFS support it?
Does IPFS support Pb or ZB level storage?
Can IPFS private cluster have more than 4096 nodes?
Does IPFS cache support terabytes?
Can we manage hundreds of millions of files through IPFS cluster? And file access to millisecond?
Does IPFs cluster support S3 interface? Need to download module installation?
Does IPFs cluster have web based management?
Thanksļ¼

2

Minio is said to be the fastest object storage in the world. IPFS is also object storage. Which is faster than Minio?

Hi, I donā€™t know minio but I know IPFS, and it isnā€™t really fast.

IPFS is also object storage

yes that not false but that really a confortable way to use IPFS right now.
Currently IPFS is a distributed CDN, not object storage itā€™s a CDN structured like an object storage but itā€™s not that great if you use it as an object storage.

Minio supports erasure, does IPFS support it?

First IPFS != IPFS Cluster
IPFS is a distributed CDN, IPFS Cluster is a decentralised manager for some IPFS nodes (allowing you to manages multiples of them at once).
That said, IPFS Cluster does support earasure but nodes in your cluster should be personal (itā€™s not a public info), IPFS somehow does, basicaly when someone download a file using IPFS the node he used to download it will keep a copy of this file and share it to anyone needing it for a few times. An administrator may choose to pin some files, pinned files will never get erased from your nodes ensuring at least your nodes will provide some files on the network. So on IPFS if you unpin a file from all your nodes and run the repo gc (by default itā€™s not automatic) and every other node in the network holding a copy does so (this should be a relatively ā€œnaturalā€ process) it will be deleted.

Does IPFS support Pb or ZB level storage?

Theoricaly yes, the network very probably holds many Pb, in practice that depends, a single node might struggle with that much data.
In theory the protocol supports an infinite size of files, directory (blocks might be limited but they can keep reference a new continue block over and over and over, ā€¦), and 2^256 unique things (the sha256 and keccak256 key size (due to use of hashing fonction you are not gonna be capable to became nowhere close to this)).

Can IPFS private cluster have more than 4096 nodes?

I havnā€™t seen any in the docs, knowing how the protocol works (nodes are addressed by the hash of their public key) I think itā€™s 2^256 still.

And file access to millisecond?

:smiley: hahaha, laugthing (sorry but that really funny, IPFS is no where that fast, just for you Iā€™m gonna try 100 tiny files (1Mb (smaller than the shard block size)) over very fast internet, low latencies and optimised setup :

  • Adding : ~0.8 seconds per file (to be fair most of that time is the go binary warming up, locking stuff, ā€¦, bigger files (~50Mb maybe more depending of your CPU hashing speed) are likely not take much longer)).
  • Downloading : ~0.1 seconds per file, and that a very favorable senario (peered peers, ā€¦)

There are certainly lots of improvement possibles for the future but that not gonna be an efficient object storage soon, think it as a torrent+++ that could replace http (instead of using http to send stuff to your users, you could use ipfs).

Does IPFS cache support terabytes?

Yes, have been tried but speed doesnā€™t scale well (still in beta) so you can, it will works but you are gonna probably takes lots of time to do anythings.

Can we manage hundreds of millions of files through IPFS cluster?

Yes no problem, IPFS uses an DAG system (directory are reference to next directories, ā€¦ then reference to a file ā€œmanifestā€ by their CID (hash)).
Iā€™ve successfully tried up to 20 millions of tiny files and a bit less of directories (no more inodes) in one time, this was a bit long to add but worked fine.

Does IPFS cluster support S3 interface? Need to download module installation?

Iā€™ve sawn GitHub - ipfs/go-ds-s3: An s3 datastore implementation wich stores your blocks using s3, Iā€™m not sure if that builtin are an extra download (or build but this is very easy with go).

Does IPFS cluster have web based management?

Yes /ipns/webui.ipfs.io but that more for the average joe, not really for managing clusters (only support one node no cluster), ā€¦

You should really look /ipns/docs.ipfs.io mainly the ā€œhow toā€ and ā€œconceptā€ sections, that goes in depth in what is IPFS and what it is meant for.

PS: If you are asking why should you use IPFS after reading this (like itā€™s not fast, still beta, ā€¦) :

  • P2P that cool
  • Bandwith usage scale negatively (more user downloading a file = more bandwith available)
  • libp2p (network backend for ipfs) is a really great lib for creating your p2p app (I know Iā€™m doing it all day :slight_smile: )
  • Totaly distributed infrastructure (except for bootstrap nodes but that temporary and an easy fix later (nothing a persistant peerstore canā€™t fix))
  • DAG FS same files are stored once as they are keyed by their hash

Basicaly itā€™s torrent+++ great enough that you donā€™t use it only for downloading your favorite OS images but a webclient for your dapp or a binary or even a simple website.

PS2: By default IPFS uses one single big public network but if you want to do your stuff in your own place you can setup a private network (extra layer of encryption isolating you from the world).

1 Like
3

Thank you very much for your reply, especially for the test of file access speed.
What I really want to ask is the speed of accessing files on the private network rather than on the Internet; there is no doubt that it should be faster in the LAN.
I even think that in the private network, IPFs can be used as object storage, hahaha :slight_smile:
I hope IPFs will develop better and better
Thank you again!

4

@gogcc

You might have interest in these:

In brief, they fork many official go-ipfs libraries and claimed to have some performance improvement by basically replacing restful API with gRPC. In addition, its implementation is based on minio gateway.

https://github.com/ipfs/go-ds-s3 this project is to support S3 storage as a blockstore backend. Which means we can store blocks on the s3 cloud instead of local machine.