Ipfs gateway timeouts

HamidullahMuslih · August 31, 2022, 6:30am

I deployed IPFS for a blockchain project, there is a need to get the NFT (metadata and picture) from its CID, but our IPFS gives timeout 504, I found to peer our IPFS with strong peers listed here Peering with content providers | IPFS Docs, still it gives timeout for some CIDs.
My question is how to implement a robust IPFS gateway? the same CID gets resolved by some of the public IPFS gateways.
Furthermore, I am running a single IPFS service n inside EKS not IPFS swarm or cluster. (edited)

4everlandorg · August 31, 2022, 1:31pm

Try our gatewat ipfs.4everland.io

danieln · August 31, 2022, 8:03pm

Do you have those CIDs that you’re getting timeouts for pinned to the node on EKS?

HamidullahMuslih · August 31, 2022, 9:07pm

@danieln thank you for replying,
I am able to get the content which is pinned in our own IPFS. but we have NFT indexer DB which is being updated by NFT metadata and data CIDs.

Below is list of the CIDs that our IPFS gives timeout.
QmeMBccZ4XwZExsPNaGANDDPq4KHEghXAQH91cKA2167pJ
QmXaHteiJkmKpqsAS1xCEet8kZqQWuZ3GncjMR5afsrWay/1108.json
QmV8UTwuFQPP4x4pRLKcmWf59uH1dssmvBZrQi3dQMVbz8

but here can be retrieved with out any issue
https://gateway.ipfs.io/ipfs/QmeMBccZ4XwZExsPNaGANDDPq4KHEghXAQH91cKA2167pJ
https://gateway.ipfs.io/ipfs/QmV8UTwuFQPP4x4pRLKcmWf59uH1dssmvBZrQi3dQMVbz8

I want to know best practices of deploying IPFS, to get the data very fast.

Some of the things I did are

I increased the UDP buffer size limit

sysctl -a | grep net.core.rmem
net.core.rmem_default = 212992
net.core.rmem_max = 2500000

peered our IPFS with Peering with content providers | IPFS Docs

Do I need to run our IPFS with server profile or badgerds ? for now it is using default profile.

ylempereur · August 31, 2022, 9:32pm

I checked those addresses you provided, and, while I was able to retrieve all of them, they each took a long time to find. Meaning, the problem isn’t on your end, it’s just that whoever is providing those blocks isn’t doing a very good job of it. Unfortunately for you, that’s what needs to improve.

HamidullahMuslih · August 31, 2022, 9:41pm

Thanks @ylempereur,
Is there anyway to increase the IPFS timeout, something that I can change in ~/.ipfs/config?
our user to IPFS traffic is via AWS ALB ===> Nginx Reverse Proxy ====> IPFS Pod.

So to concluse we should expect 40% loss of the request from our IPFS, right?

ylempereur · August 31, 2022, 9:44pm

Usually, the timeout actually comes from the web server/proxy in front of the gateway, not the gateway itself (but I’m sure the gateway has a timeout too).

I just use the ipfs command on the CLI, which doesn’t timeout, and let it run, possibly for hours, until it locates the data. That’s how I got your blocks.

HamidullahMuslih · August 31, 2022, 9:44pm

Thanks but your IPFS GW also gave timeout

ylempereur · August 31, 2022, 9:45pm

Oh, that wasn’t me, that was someone else you were talking to.

HamidullahMuslih · August 31, 2022, 9:50pm

Yes @ylempereur he is another user (@4everlandorg ) who proposed me to use his IPFS GW, which had still timeout for one CID.

@ylempereur now I can also get those CIDs which had timeout, seems after you accessed those CIDs, they got spreaded over the IPFS network.

ylempereur · August 31, 2022, 9:51pm

Actually, it just means you got them from my node, as they must still be in the cache, and my node does a really good job of providing. Which doesn’t solve your problem, but demonstrates that this can work really well, if it’s done right.

HamidullahMuslih · August 31, 2022, 9:55pm

@ylempereur yes,

Please can you share how you deployed IPFS GW:

in a single pod inside K8s
in cluster of pods inside K8s
in a single machine (EC2 or…)
or in cluster of machines.

and how much of resources you gave to the service?
and for which purpose you use it ?

NFT pinning
website hosting
other

ylempereur · August 31, 2022, 9:56pm

I’m just running IPFS Desktop on my Mac, nothing special.

The two things you want to make sure of are:

make sure your node is reachable by others
make sure you are using the accelerated DHT client

HamidullahMuslih · August 31, 2022, 9:58pm

Thanks @ylempereur,
we will be using this a core of NFT and also pinning our static websites.

I need to dig into accelerated DHT client

ylempereur · August 31, 2022, 9:59pm

Use this to turn it on:

ipfs config --json Experimental.AcceleratedDHTClient true

HamidullahMuslih · August 31, 2022, 10:01pm

@ylempereur awesome thank you,

A quick question please, does your machine have a public IP attached to it so other nodes find your local IPFS, or it is using your router’s dynamic public IP assigned by ISP?

ylempereur · August 31, 2022, 10:02pm

Sorry, I’m in a meeting, I’ll post a description after it ends.

ylempereur · August 31, 2022, 11:16pm

My situation is a little bit complex, but I’ll give you a simplified, more typical version of it:

my ISP assigns one public IP to my router, which then uses NAT to provide connectivity to the various devices in my home. I have set up port-forwarding on the router to make TCP 4001 and UDP 4001 forward to the same ports on my Mac, which the IPFS daemon uses for communication.

the daemon will announce my public IP along with the TCP and UDP ports, which is what other nodes will use to establish a connection.

on occasion, the IP address changes, but the daemon is able to notice and announce my new address when it happens.

HamidullahMuslih · September 1, 2022, 8:03am

@ylempereur is this neccessary to expose port 4001, since in the EKS/pod/svc we are not exposing it. also the AWS ALB is internal means, no one can reach https://our..internal.ipfs.com/ipfs/.
we access it via VPN.

now I have doubt if we don’t expose (port-forward) the 4001, how our IPFS see all the peers (around 600) and how the pinned a content can be accessed via other public IPFS gateways?

but I see that deamon is having public IP?
I think IPFS one purpose is also to bypass all the restrictions like firewall rules, …

danieln · September 1, 2022, 9:16am

@HamidullahMuslih,

I suggest reading through the following blog post.

When you request a CID from an IPFS node, the request can be broken down into two steps:

Content discovery/routing: asking direct peers and querying the DHT (opens new window)to find the peer IDs and network addresses (opens new window)of peers providing the CID (referred to as providers).
Content retrieval: connecting to one of the providers, fetching the CID’s content, and streaming the response to the client.

Actual debugging steps

Here’s how I debug one of the CIDs you provided: QmeMBccZ4XwZExsPNaGANDDPq4KHEghXAQH91cKA2167pJ:

Find providers in the DHT:
ipfs dht findprovs QmeMBccZ4XwZExsPNaGANDDPq4KHEghXAQH91cKA2167pJ which returns PeerIDs of providers

Result

12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF
12D3KooWKuSzGoorvsBB8zdXCYVMEfybYKa7sDgXT3AkofAzX3FV

Try to ping the PeerIDs to see if these PeerIDs are reachable

$ ipfs ping 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF
PING 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF.
Pong received: time=275.42 ms
Pong received: time=280.18 ms
Pong received: time=273.83 ms
Pong received: time=334.89 ms
Pong received: time=312.37 ms
Pong received: time=271.37 ms
Pong received: time=279.96 ms
Pong received: time=312.39 ms
Pong received: time=277.58 ms
Pong received: time=282.73 ms
Average latency: 290.07ms
----
$ ipfs ping 12D3KooWKuSzGoorvsBB8zdXCYVMEfybYKa7sDgXT3AkofAzX3FV

Looking up peer 12D3KooWKuSzGoorvsBB8zdXCYVMEfybYKa7sDgXT3AkofAzX3FV
Error: peer lookup failed: routing: not found

This means that only one of the two peers is reachable, namely 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF.

Finally, trying to fetch the blocks from that peer. First find the addresses of the peer:

ipfs dht findpeer 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF | sort
/ip4/127.0.0.1/tcp/4001
/ip4/127.0.0.1/udp/4001/quic
/ip4/172.31.80.213/tcp/4001
/ip4/172.31.80.213/udp/4001/quic
/ip4/5.161.55.227/tcp/4001/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip4/5.161.55.227/udp/4001/quic/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip4/84.146.179.248/tcp/4001/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip4/84.146.179.248/udp/4001/quic/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip6/2a01:4ff:f0:1e5a::1/tcp/4001/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip6/2a01:4ff:f0:1e5a::1/udp/4001/quic/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip6/64:ff9b::5492:b3f8/tcp/4001/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip6/64:ff9b::5492:b3f8/udp/4001/quic/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip6/64:ff9b::5a1:37e3/udp/4001/quic/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit

It looks like the node isn’t publicly reachable (only has private IPs)
All the addresses with the p2p-circuit suffix are relay nodes that can help hole punch a connection to the node to retrieve blocks.

Normally I use http://check.ipfs.network/ to debug content retrieval.

Input

@HamidullahMuslih Can you check if that is your PeerID (ipfs id -f '<id>')?

Recommendations

Enable accelerated DHT client (seems you already did)

is this neccessary to expose port 4001, since in the EKS/pod/svc we are not exposing it.

Yes! Ensure that port 4001 on your IPFS node is reachable from the IPFS node’s public IP (either port forwarding. This is critical for other nodes to be able to retrieve blocks using the Bitswap protocol.