Who's Providing All This Storgae Space?

Trying to understand where the storage space is coming from for all the existing ipfs data . . . is it just a bunch of hobbyists setting up some servers in their basements? People with ipfs nodes on their laptops that pop into and out of existence as they power up and down? Who is providing the disk space for the photos I upload via fleek (just as an example)?

Anyone have some ideas about what the landscape looks like?

1 Like

It’s a common misconception that IPFS has some kind of magical storage space.

The fact is that you have to run a node for storage, it doesn’t have to be you but someone has to. If you use someone else storage they might give you free space but they don’t have to and most don’t.

For maximum decentralisation everyone would need to run a node.

It’s when the same data on IPFS is shared by many people that the magic happen.

2 Likes

Thanks for confirming my suspicions. With every announcement of a new IPFS application, I always wonder who’s going to be hosting all those photos/web sites/whatevers?

Does Protocol Labs (or someone else) publish any statistics on IPFS storage? One thing that the StorJ folks do well is publishing their storage node stats/details. Is there anywhere to get a picture of how many nodes there are and how much storage is out there? I guess Filecoin integration will help with getting more and persistent storage on line. Is there a timetable for that?

With every announcement of a new IPFS application, I always wonder who’s going to be hosting all those photos/web sites/whatevers?

In theory you, in a perfect world everyone runs their own IPFS node and you host your own content.

Sadly we aren’t in that world yet, so most apps use services like web3.storage, nft.storage, estuary.tech or other paid services like pinata.cloud, fleek.co or infura.io.

Secondly, why do you ask this question ?
It’s not that it’s a bad one or anything but many web2 services already found ways to make money hosting other people files for free, why couldn’t we just apply the same thing to web3.

Thanks for your insights. I’m only asking because I was just curious and didn’t understand how it was working. Like I said, I see many new apps/services in the weekly announcements, and I couldn’t figure out where it was all being stored.

Three are lots lots and lots of abandoned apps/services as well, and dead-end ipfs hosted web pages, so maybe there’s not as much out there as I thought!

I’m only asking because I was just curious and didn’t understand how it was working.

Yeah, I’ve shouldn’t have phrased things like that, what I meant is that web2 makes it work so we can too.

Three are lots lots and lots of abandoned apps/services as well, and dead-end ipfs hosted web pages, so maybe there’s not as much out there as I thought!

This is a valid concern for not IPFS. But on IPFS you can self host your files, so if you care for your files just self host them yourself (or pay someone with filecoin or a pinning service). :slight_smile:

That’s my next area for a bit of research - Filecoin and how that works with IPFS . . . . if you have any sources of info that you’ve found helpful, I’d appreciate a pointer!

I’m continually looking for better ways to explain IPFS so here’s my latest attempt. You can think of IPFS as a system that makes it very easy to ask but not tell. You can stand in the middle of a crowd and shout, “Hey does anyone have X that they can give me?” and maybe someone will throw you an X. If you hear someone else asking for X then you’re free to share you’re X with them. What you can’t do is simply walk around and say, “Take my X! You take my X and share it with people, NOW!”. That’s not very friendly to go around bossing people around like that.

So when people ask why can’t I just put something on IPFS and expect it to be replicated forever is like just showing up to your neighbors house and putting your junk in their basement because they have the room and expecting them to keep it safe for as long as you feel like putting it there.

You can’t tell people what to do but you can ask them to do things or pay them to do what you want. “If I give you $5 will you hold on to X and give it to people when they ask for it?” or you can just ask someone to do it for free but just don’t be surprised if they get distracted and wander off.

Some people noticed that people are willing to pay for a reliable service so they setup a booth and started selling the service of storing your stuff and sharing it when people ask for it.

So who’s providing the storage? Right now it’s a mix of random people just being nice and people with booths who are being paid to provide it.

Filecoin is like adding money to our fictional universe. Right now the paid providers are being paid in money that appears from a far off land. You can pay for things and take it back to that far off land and use it there but there is no local currency. Filecoin allows someone to buy and sell their storage and to prove they’re doing what they say they will.

1 Like

Your analogy is great and a helpful way to visualize the current situation. What prompted me to ask this was that I keep feeling like I’m missing something - for example, in the latest IPFS weekly newsletter (#164) there’s a link to a video with the teaser: " Discord + IPFS? [Check out this brief demo] of uploading a file to IPFS directly from a Discord channel." Great, a new tool. They talk about IPFS like it’s a place that has storage waiting for you to use. This kind of wording is used for almost all the new product announcements. “Upload to IPFS - it’s great!”, so I’ve been wondering - who’s ponying up the disk space for all these files that all these new tools generating. Are there a few large organizations proving ‘seed space’, or are there a bunch of hobbyists and casual/occasional people making some space on their ephemeral nodes available when they happen to be on line? Just a bit of a mystery to me. Paying for storage (with Filecoin) is understandable to me as it sounds similar to the STORJ model. But as far as I know, paying for storage with Filecoin is not available yet. Maybe there is much less storage space than my imagination has me believe and that all these files uploaded with these new tools don’t last long and just fade away after a while . . . .

I can see that. Unfortunately it’s a tough choice. Either you have the nice simple, “Upload to IPFS - it’s great!” or you kind of go into a long detailed explanation of how exactly that works.

It’s more like a BYOS, Bring Your Own Storage/Buy Your Own Storage else rely on the charity of others. It doesn’t make for a great sound bite but it’s better than the opaque system we have now where you’re not paying Facebook so they’re going to find some other way of making money from it.

Either you use a centralize system provided by big tech, which u don’t have to pay for it in the beginning, or you can use your own d-internet with own server
I mean you are already paying for your fiber cable

I think the technology is much broader than that. the protocols are different. instead of storing files not needed at nodes, the nodes learn what to store and what not to make things more efficient based on whats being asked for by it’s neighbors. there has to be alot of experimentation to figure things out and iron out the kinks. this is why we have the free service so that many users try it so the developers get to figure out what’s working and whats not. this is my understanding. I think this is really much needed and some funding is going to make it happen. this is my understanding, I might be wrong.

I’ve been wondering if we could use a system based on reciprocity (I’ll pin your stuff on my node if you pin my stuff on yours), and how it could be formalized into code the way filecoin has done it. Maybe it’s an opportunity to build something (I’ll give it some more thought, I’m a dev for a living, I could possibly come up with something)

Interesting idea!

Do you know if there are any IPFS storage network statistics, like how many nodes there are or how much storage space there is?

This answers some of those questions:

Measuring the Web3.0 Stack

Thank you very much for this! Honestly, I never thought that a talk titled Web3.0 would have this kind of info - I guess I need to get out more!

Thanks again

In a decentralize system, capitalism shows u provide people obligations to do it for you, some use crypto currencies but I will provide one vanilla example :You are in a network of your company, managers set up people’s ipfs so they pin everything that’s shared with you, because it benefits the entire company if resources like internal resources (binaries, pictures, etc.) are distributed and available in ipfs without paying money to store them in one server.
Or maybe if you are on your own, but app for the future d-internet are hosting a chat room, it would of course be obligated to pin all the chat messages because you want them to be available even after that person turns off devices

To add to the conversation of this topic, I am understanding the storage is provided by your computer that’s running ipfs, but all ipfs can store files in a cache and provide it to others when they run into them, for example your computer might be hosting ipfs’s documents now if you’ve browsed the document through ipfs protocol which it is available on (install the ipfs companion in your browser to automatically browse websites straight from the ipfs network if the website has dnslink enabled ). but the cache won’t stay live after garbage collection by the ipfs implementation, unless you’ve pinned it.

Just watched this presentation and found it very interesting. It seems that about 14% of the total nodes (2735) are ‘always on’ and the other 85% are on for relatively short periods of time and then go off line. 2735 is actually more nodes than I was expecting, so, for me, that’s good! My next question now is . . .how much storage space do those nodes represent? Still digging!

Thanks again Yves, for this pointer!

I do stats from time to time, I found that right now about 8~11k nodes are always on.