Ipfs gateway timeouts

HamidullahMuslih · August 31, 2022, 9:44pm

Thanks but your IPFS GW also gave timeout

ylempereur · August 31, 2022, 9:45pm

Oh, that wasn’t me, that was someone else you were talking to.

HamidullahMuslih · August 31, 2022, 9:50pm

Yes @ylempereur he is another user (@4everlandorg ) who proposed me to use his IPFS GW, which had still timeout for one CID.

@ylempereur now I can also get those CIDs which had timeout, seems after you accessed those CIDs, they got spreaded over the IPFS network.

ylempereur · August 31, 2022, 9:51pm

Actually, it just means you got them from my node, as they must still be in the cache, and my node does a really good job of providing. Which doesn’t solve your problem, but demonstrates that this can work really well, if it’s done right.

HamidullahMuslih · August 31, 2022, 9:55pm

@ylempereur yes,

Please can you share how you deployed IPFS GW:

in a single pod inside K8s
in cluster of pods inside K8s
in a single machine (EC2 or…)
or in cluster of machines.

and how much of resources you gave to the service?
and for which purpose you use it ?

NFT pinning
website hosting
other

ylempereur · August 31, 2022, 9:56pm

I’m just running IPFS Desktop on my Mac, nothing special.

The two things you want to make sure of are:

make sure your node is reachable by others
make sure you are using the accelerated DHT client

HamidullahMuslih · August 31, 2022, 9:58pm

Thanks @ylempereur,
we will be using this a core of NFT and also pinning our static websites.

I need to dig into accelerated DHT client

ylempereur · August 31, 2022, 9:59pm

Use this to turn it on:

ipfs config --json Experimental.AcceleratedDHTClient true

HamidullahMuslih · August 31, 2022, 10:01pm

@ylempereur awesome thank you,

A quick question please, does your machine have a public IP attached to it so other nodes find your local IPFS, or it is using your router’s dynamic public IP assigned by ISP?

ylempereur · August 31, 2022, 10:02pm

Sorry, I’m in a meeting, I’ll post a description after it ends.

ylempereur · August 31, 2022, 11:16pm

My situation is a little bit complex, but I’ll give you a simplified, more typical version of it:

my ISP assigns one public IP to my router, which then uses NAT to provide connectivity to the various devices in my home. I have set up port-forwarding on the router to make TCP 4001 and UDP 4001 forward to the same ports on my Mac, which the IPFS daemon uses for communication.

the daemon will announce my public IP along with the TCP and UDP ports, which is what other nodes will use to establish a connection.

on occasion, the IP address changes, but the daemon is able to notice and announce my new address when it happens.

HamidullahMuslih · September 1, 2022, 8:03am

@ylempereur is this neccessary to expose port 4001, since in the EKS/pod/svc we are not exposing it. also the AWS ALB is internal means, no one can reach https://our..internal.ipfs.com/ipfs/.
we access it via VPN.

now I have doubt if we don’t expose (port-forward) the 4001, how our IPFS see all the peers (around 600) and how the pinned a content can be accessed via other public IPFS gateways?

but I see that deamon is having public IP?
I think IPFS one purpose is also to bypass all the restrictions like firewall rules, …

danieln · September 1, 2022, 9:16am

@HamidullahMuslih,

I suggest reading through the following blog post.

When you request a CID from an IPFS node, the request can be broken down into two steps:

Content discovery/routing: asking direct peers and querying the DHT (opens new window)to find the peer IDs and network addresses (opens new window)of peers providing the CID (referred to as providers).
Content retrieval: connecting to one of the providers, fetching the CID’s content, and streaming the response to the client.

Actual debugging steps

Here’s how I debug one of the CIDs you provided: QmeMBccZ4XwZExsPNaGANDDPq4KHEghXAQH91cKA2167pJ:

Find providers in the DHT:
ipfs dht findprovs QmeMBccZ4XwZExsPNaGANDDPq4KHEghXAQH91cKA2167pJ which returns PeerIDs of providers

Result

12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF
12D3KooWKuSzGoorvsBB8zdXCYVMEfybYKa7sDgXT3AkofAzX3FV

Try to ping the PeerIDs to see if these PeerIDs are reachable

$ ipfs ping 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF
PING 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF.
Pong received: time=275.42 ms
Pong received: time=280.18 ms
Pong received: time=273.83 ms
Pong received: time=334.89 ms
Pong received: time=312.37 ms
Pong received: time=271.37 ms
Pong received: time=279.96 ms
Pong received: time=312.39 ms
Pong received: time=277.58 ms
Pong received: time=282.73 ms
Average latency: 290.07ms
----
$ ipfs ping 12D3KooWKuSzGoorvsBB8zdXCYVMEfybYKa7sDgXT3AkofAzX3FV

Looking up peer 12D3KooWKuSzGoorvsBB8zdXCYVMEfybYKa7sDgXT3AkofAzX3FV
Error: peer lookup failed: routing: not found

This means that only one of the two peers is reachable, namely 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF.

Finally, trying to fetch the blocks from that peer. First find the addresses of the peer:

ipfs dht findpeer 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF | sort
/ip4/127.0.0.1/tcp/4001
/ip4/127.0.0.1/udp/4001/quic
/ip4/172.31.80.213/tcp/4001
/ip4/172.31.80.213/udp/4001/quic
/ip4/5.161.55.227/tcp/4001/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip4/5.161.55.227/udp/4001/quic/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip4/84.146.179.248/tcp/4001/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip4/84.146.179.248/udp/4001/quic/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip6/2a01:4ff:f0:1e5a::1/tcp/4001/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip6/2a01:4ff:f0:1e5a::1/udp/4001/quic/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit
/ip6/64:ff9b::5492:b3f8/tcp/4001/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip6/64:ff9b::5492:b3f8/udp/4001/quic/p2p/12D3KooWPutck3ASkfcU13Ht5jzgevBo9Fc1sHS5yYberAYupDtq/p2p-circuit
/ip6/64:ff9b::5a1:37e3/udp/4001/quic/p2p/12D3KooWSW4hoHmDXmY5rW7nCi9XmGTy3foFt72u86jNP53LTNBJ/p2p-circuit

It looks like the node isn’t publicly reachable (only has private IPs)
All the addresses with the p2p-circuit suffix are relay nodes that can help hole punch a connection to the node to retrieve blocks.

Normally I use http://check.ipfs.network/ to debug content retrieval.

Input

@HamidullahMuslih Can you check if that is your PeerID (ipfs id -f '<id>')?

Recommendations

Enable accelerated DHT client (seems you already did)

is this neccessary to expose port 4001, since in the EKS/pod/svc we are not exposing it.

Yes! Ensure that port 4001 on your IPFS node is reachable from the IPFS node’s public IP (either port forwarding. This is critical for other nodes to be able to retrieve blocks using the Bitswap protocol.

HamidullahMuslih · September 1, 2022, 9:33am

@danieln
yes 12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF id of my IPFS.

but the 4001 is not exposed, then how your node found it?

danieln · September 1, 2022, 10:04am

Because your node is publishing to the DHT both peer records (mapping from your peerId to addresses) and provider records for that CID.

See the Content publishing lifecycle section in the blog post.

ylempereur · September 1, 2022, 3:56pm

As Daniel said, use the following to test if your node is reachable:

IPFS Check

P.S. I just tested your node with it, and it is reachable, but, it’s through a relay, which makes things inefficient. You should open those two ports, things will work better.

HamidullahMuslih · September 2, 2022, 11:06am

@danieln and @ylempereur thank you,

for below content my two IPFSs services did not show any provider
ipfs dht findprovs Qma2L7kLniaafUDWRbGhGy22aiHA9qsv6PD8CzZZMipBhj

but it can be retrieved from below link

https://ipfs.io/ipfs/Qma2L7kLniaafUDWRbGhGy22aiHA9qsv6PD8CzZZMipBhj

can you provide any light how and why ?

danieln · September 2, 2022, 11:23am

This could be due to a combination of two things:

The IPFS node hosting/pinning Qma2L7kLniaafUDWRbGhGy22aiHA9qsv6PD8CzZZMipBhj is not publishing/advertising provider records to the DHT
The same IPFS node is directly peering with the ipfs.io gateway so that it asks for the content directly over BitSwap (without having to look up the DHT)

HamidullahMuslih · September 2, 2022, 7:35pm

@ylempereur as relay you mean there will be one more hop to reach our IPFS.
and the adresses will look like below

		"/ip4/188.166.36.145/tcp/4001/p2p/12D3KooWAtQymAZZ1KwaTqr1oLSomEb45C6oQQokMLkwjeAhdwYq/p2p-circuit/p2p/12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF",
		"/ip4/188.166.36.145/udp/4001/quic/p2p/12D3KooWAtQymAZZ1KwaTqr1oLSomEb45C6oQQokMLkwjeAhdwYq/p2p-circuit/p2p/12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF",

without relay will look like below:

		"/ip4/my-public-ip/tcp/4001/p2p/12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF",
		"/ip4/my-public-ip/udp/4001/quic/p2p/12D3KooWFkPkpLWTKDDbE4Dtdknrhj17KTEsduYp4cnS9aYeUuYF"

ylempereur · September 2, 2022, 8:05pm

Correct.

Using a relay causes the connection establishment to take a lot longer (using hole punching). Once established, it’s a direct connection and works just as fast as any other, though.

So, if it’s an option to set up port forwarding and poke a hole in the firewall, it’s much better than using a relay. Especially if you are running a server, and not just a client on a personal machine.