Is there a rate limit for call? We fetch images directly from IPFS, but we are getting hit with 429s. "Too many request). We store NFTs, but need our visitors to be able to see them all when they visit our site. Please help
Hey, I think you mean you are fetching them directly from ipfs.io? {go|js|rust}-ipfs will not return 429s as far as I am aware.
The public gateway operated by Protocol Labs does return 429 when it limits requests, it’s particularly sensitive to requesting the same thing from the same IP. Do you proxy on behalf of your clients or directly embed ipfs.io/ipfs/bafywhatever in your websites and have users load from the gateway?
You have options:
- host your own gateway (you could pin the content you want to serve on it and it would always be quick, you could set your gateway to only serve pinned content),
- point a CDN at an existing gateway
Thank you so much for your response. Would Filecoin be the best approach for data retrieval from IPFS?
lots of NFT sites do embed them from ipfs.io, I still don’t know exactly what you are doing to help more though. Are you just doing embeds or are you proxying the PL gateway?
Thank you so much for your response. So we created an NFT marketplace for users to upload and sell their NFTs. Once an NFT is minted we automatically store the item in IPFS. What we are seeing is heavy lag in data retrieval. We are making a call directly to IPFS to fetch each item whenever a visitor visits the site. I started to see the 429 errors yesterday, I didn’t realize until you told me that Protocol labs had a rate limit. We are tapping them from a single IP. I think data retrieval from IPFS directly is a major pain point of mine. If we have a million images, we can’t make a million calls to IPFS. I’m wondering if a layer such as Filecoin would reduce the lag that we are experiencing?
I have to ask that you are more precise, “from IPFS directly” and “via the IPFS HTTP gateway located at ipfs.io” are two different things, only the latter will ever lead to a 429.
If you are fetching them from ipfs.io on behalf of clients, you will have to cache them as you are a lot more likely to get rate limited asking for the same things over and over.
You could run go-ipfs yourself and fetch NFTs from there (and they would then be in the repo of the ipfs node)
How are you adding the nfts to the IPFS network? You are running some nodes?
You can take a look at https://nft.storage/ too, might fit your needs, or https://www.pinata.cloud/ do a good job too
I wasnt sure it was public, but the nft.storage folk have a gateway you can use at https://nftstorage.link/. that races multiple gateways and will cache results so will probably work better for you (you can link directly to them in your webpage if you like)