Random 404 HTTP errors when accessing content via gateway

Help IPFS
1

Hi!

I am getting random 404 errors when accessing the content. It just doesn’t make any sense.

I can easily reproduce it by making subsequent requests, say with wget, like this:

$wget https://libp2p.example.com/img/svg/nat-traversal.svg
--2021-12-09 10:36:26--  https://libp2p.example.com/img/svg/nat-traversal.svg
Resolving libp2p.example.com (libp2p.example.com)... [PUBLIC-IP]
Connecting to libp2p.example.com (libp2p.example.com)|[PUBLIC-IP]|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5821 (5,7K) [image/svg+xml]
Saving to: ‘nat-traversal.svg’

nat-traversal.svg                 100%[=============================================================>]   5,68K  --.-KB/s    in 0s      

2021-12-09 10:36:26 (313 MB/s) - ‘nat-traversal.svg’ saved [5821/5821]

$wget https://libp2p.example.com/img/svg/nat-traversal.svg
--2021-12-09 10:36:28--  https://libp2p.example.com/img/svg/nat-traversal.svg
Resolving libp2p.example.com (libp2p.example.com)... [PUBLIC-IP]
Connecting to libp2p.example.com (libp2p.example.com)|[PUBLIC-IP]|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5821 (5,7K) [image/svg+xml]
Saving to: ‘nat-traversal.svg.1’

nat-traversal.svg.1               100%[=============================================================>]   5,68K  --.-KB/s    in 0,002s  

2021-12-09 10:36:28 (3,17 MB/s) - ‘nat-traversal.svg.1’ saved [5821/5821]

$wget https://libp2p.example.com/img/svg/nat-traversal.svg
--2021-12-09 10:36:30--  https://libp2p.example.com/img/svg/nat-traversal.svg
Resolving libp2p.example.com (libp2p.example.com)... [PUBLIC-IP]
Connecting to libp2p.example.com (libp2p.example.com)|[PUBLIC-IP]|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5821 (5,7K) [image/svg+xml]
Saving to: ‘nat-traversal.svg.2’

nat-traversal.svg.2               100%[=============================================================>]   5,68K  --.-KB/s    in 0s      

2021-12-09 10:36:30 (572 MB/s) - ‘nat-traversal.svg.2’ saved [5821/5821]

$wget https://libp2p.example.com/img/svg/nat-traversal.svg
--2021-12-09 10:36:35--  https://libp2p.example.com/img/svg/nat-traversal.svg
Resolving libp2p.example.com (libp2p.example.com)... [PUBLIC-IP]
Connecting to libp2p.example.com (libp2p.example.com)|[PUBLIC-IP]|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-12-09 10:36:35 ERROR 404: Not Found.

The previous log has been anonymized. I simply mounted libp2p.io on my ipfs gateway/dns.

As you can see the last of subsequent requests responds with 404.

Just making a few requests and eventually 404 comes. Looks like when the IPFS gateway has not been accessed for a while the error is most probable. Happens with pinned content too.

I serve a couple of sites and uptime robot reports 99.3% updatime with frequest 404s on the test url (index).

I use dockerized go-ipfs, latest published image.

Any idea of what could be going on?

2

I have been browsing the go-ipfs repo for situations in which 404 is returned.

I found surprising that 404 seems to be returned from the module: hostname.go

Could that mean that DNS lookup failures result in 404 being returned?

3

DNS failure to resolve libp2p.io would produce 404 in the DNSLink gateway.
That being said, I was not able to reproduce, looks like a problem with your setup.

Local subdomain gateway:

while wget -O /dev/null "http://libp2p.io.ipns.localhost:8080/img/svg/nat-traversal.svg"; do; true; done

Local DNSLink gateway:

while wget -O /dev/null --header "Host: libp2p.io" "http://127.0.0.1:8080/img/svg/nat-traversal.svg"; do; true; done

Both run forever at extreme speeds :wink: and produce no 404s for me.

I suspect 404 comes from the reverse proxy you’ve set up in front go-ipfs rather than DNS failure. You could add DNS cache and see if it helps, otherwise dig into reverse proxy setup.

4

mmm I have other services on the reverse proxy, and they are more solid, I think.

I never see 404 for the other services… but I see timeouts (rare and short) on the uptime monitor.

But we had DNS issues in the past, and if they can translate into 404s that would explain what we are seeing.

I will investigate further.

5

If you could SSH to the machine and talk to go-ipfs directly, and run the one-liners I provided above for a few minutes, that would be a good test.
If any of them produce HTTP 404, you will know it is not reverse proxy bug, but most likely a DNS issue.

6

OK, I will try your test.