Audius, a large music streaming platform, uses ipfs·io to serve its audio files. I thought ipfs·io was forbidden for use under these circumstances (“Users of the ipfs·io Gateway must use resources sparingly”) and “should not be relied on for hosting” according to the IPFS gateway docs? Please clarify so that I may understand correctly, thanks.
Hi @InterPlanetary, welcome to IPFS. Intersting name you picked there
A gateway is hosted by someone. The ipfs.io gateway probably is quite beefy in terms of bandwidth that it has available, but in essence you’re burning through someone else’s money by “using” (more like abusing) a public gateway for that. And in this particular case i’d much rather see the IPFS money be spend in development of IPFS as opposed to paying for your bandwidth.
I too have a gateway publicly online. If some site pops by and sucks up my bandwidth, i’m surely not going to be happy and will take measures to prevent that.
The gateways should be used for users that can’t run a node or for whatever reason don’t want to. It’s a point of access for them to still access content on the IPFS network. A site, like audius, should host it’s own gateway and not be a cheap skate bandwidth stealer.
I tried their site just to see this. I’m guessing there is mixed IPFS usage going on behind the scenes. Music does come from the ipfs.io gateway. But everything else seems to be routed nicely through my local ipfs node.
Thanks for the reply. The music streams would definitely be the greatest bandwidth hog, and that is what they are routing through ipfs,io. So I’m interested to see what action on IPFS’ end, if any, would be taken against Audius for being a “cheap skate bandwidth stealer.” IPFS had done a case study on Audius, so perhaps both entities had worked out some deal in using excessive amounts of bandwidth through IPFS’ public gateway? I’ve also been trying to find any information on CloudFlare’s public gateway’s usage limitations, but can’t seem to find anything. I guess it would come down to whose generosity Audius would like to abuse – IPFS, or CloudFlare. I ask this because it seems unfair if Audius’ platform is allowed to utilize IPFS’ gateway for excessive amounts of audio data transfer, while all other similar platforms would have to rely on their own setup.
I run a small community for independent musicians myself, and rely on standard cloud hosting, pushing under 10GB per month in compressed MP3. I’ve been looking into switching to an IPFS solution for future scalability, but bandwidth costs are still pricy for my smaller use case. So seeing others (and in this case, possible competitors) get away with using tons of bandwidth from free public gateways such as ipfs,io while I’d be paying in full for mine is a bit discouraging. It would be nice to hear IPFS’ official thoughts on this, perhaps I’m misunderstanding something.
10 * 0.15 = $1.50 per month at Pinata.
I just don’t know the answers there.
I’m not an IPFS representative and not affiliated with them. I’m merely a developer liking this tech
But… commonsense does kick in when you talk about bandwidth there. I’m frankly surprised that there apparently is a mutual cooperation between Audius and IPFS (otherwise that case study wouldn’t be there). Regardless, commonsense does tell me to host my own gateway if i’m out to develop a site with IPFS.
That is not entirely true…
You paint a rather black and white picture. In reality you can easily host that on Vultr or digitalocean with under $50/month in terms of server and storage cost. And by that i mean you can buy a node or 2 of $5 each and some extra storage where $40 gives you 400GB. You’d then setup an ipfs node on your server under a reverse proxy. Your site should then be hosted by that reverse proxy (you can do this with DNSLink. Now every user that visits your site without them having IPFS in any way will use your bandwidth. You should obviously detect that and advice the users to install a node. Users with IPFS will get their content through IPFS (might or might not be from your gateway, the more users you have the less likely it is you actually serve the content).
Moral of that story. With IPFS it’s relatively easy to make something very scalable. As long as you don’t rely on central servers for your site to function.
Thank you, I’m aware of Pinata and other pinning services and just trying to figure out the best bandwidth/data transfer/gateway solution, as well as get an understanding of Audius’ use of ipfs,io for serving large volumes of audio content, while it is apparently not permitted by Protocol Labs.
Pinata told me there’s no pay-as-you-go solution for gateway bandwidth that they’re aware of, although they’re looking to do so in the future, and on their end there are only $99/mo gateway or $109/mo node offerings. With Backblaze B2, I currently pay $5/TB ($0.005/GB) vs. Pinata’s $150/TB, and some change in data transfer. I understand that they’re not exactly comparable (centralized vs. decentralized) and totally different technologies, however.
Thanks again to you as well @ markg85
Perhaps IPFS is not optimal for my situation when comparing storage and gateway costs to that of centralized solutions, but I posted here in Meta because I still sought to see if I’m understanding Audius’ excessive use case of ipfs,io bandwidth, and I hope that I may receive an official reply on the matter from the IPFS team
Your application requires 1TB of storage and 10GB/month of bandwidth. Is that about correct?
Bandwidth costs for 10GB per month transfer is pretty low. The base level linode provides more than enough bandwidth at something like 1TB. You could fairly easily build a decent performance scalable application combining Backblaze storage and an inexpensive IaaS.
One of the advantages of adding IPFS storage to your application would be the ability to increase bandwidth through the expanded user base. You could distribute an application which includes an IPFS node… maybe a cluster node for replication of mp3s so that a node dropping out doesn’t result in missing content. As your user base expands both the storage and your network bandwidth increase, but you are not paying for the bandwidth nor the storage since both are peer-to-peer. However, application performance will suffer if only a few users are using your cluster OR if there happens to be a few users who are spatially sparse throughout IP address space.
The reliance on a gateway to provide access to IPFS content is really applying the incorrect network model… It’s a crutch for server-client type thinking. Of course… CGNAT sometimes makes it hard to break out of that model in a practical sense…
ipfs ping <nodeID>
needs to work properly. And CGNAT doesn’t like that… from my experience.
In any case, if you are asking if you can use an IPFS gateway for free to access content… and your application consumes less than 10GB a month… I honestly don’t think the top 10 IPFS public gateways will notice that much if you randomly switch over from one to another every hour or so… but your application may not perform very well.
The public gateways are really for test purposes. It’s pretty easy to set one up. If you already have a domain and a small IaaS, you could use a reverse ssh tunnel to your LAN and easily host 10 GB/month bandwidth.
I totally forgot about that option!
One can definitely host a huge site where it’s essentially never hosted in full by a dedicated server. It is with your own local node. It’s a dangerous model as you would rely on your own node to be online all the time and be well accessible in the IPFS network. But it can work if you have a beefy connection.
One could also tunnel IPv6 connections through the IPv4 tunnel…
And that leads to all sorts of fun, especially with IPFS connections!
But be careful of security.
Under certain circumstances the gateways will rate-limit or block (content or requesters). So the notice above is correct.
Thanks ipfsme, that is all very helpful information (albeit a bit confusing for my green-thumbed understanding of IPFS gateway/infrastructure), but I’ll have to look more into it.
I’m not profiting from my platform as it’s more of a hobbyist thing, so do you have any suggestions of the most practical and cost-effective solutions for setting up one’s own pinning + gateway combined solution? Is there another service you’d recommend other than Pinata or are they one of the more affordable options? I’m a bit lost in the part where you mention that gateways should not be relied on, and neither should this ‘CGNAT’ solution? I’ve done a fair amount of my own digging but just wondering if someone with more knowledge on the topic than I has any better routes for simple, affordable pinning and delivery of content. Thanks again!
Thanks hector, I’m surprised that a large music streaming platform such as Audius had not seemed to be blocked for serving GBs upon GBs of audio files, especially when specifically violating the public IPFS gateway’s guidelines of using it for live hosting.
genuinely curious if you found more data on this.
Because indeed, if Audius would be using a “normal” pricing model for bandwidth (not using a public gateway) they would theoretically have to pay much more than what they do now, no?
e.g. for the 3.5 Tb they at least had at some point, does anyone know how much bandwidth they use?
Also, if the difference between BlackBlaze and IPFS is 30x ($5 vs $150 per Tb), does it mean that IPFS/Filecoin node operators are roughly making a 30x margin?
thanks for your help