Download time (from gw) does not match any value in promethus. Any idea?

Help Kubo
1

ipfs version: 1.14.0

I downloaded a 10M file with curl from gateway. With limit rate It take 13s.
But I did not see any value in /debug/metrics/prometheus close to 13s.

what is ipfs_http_request_duration_seconds{handler="gateway",quantile="0.5"} 4.578398782 means? Should not it record 13s?

curl:

time curl --limit-rate 1000K http://127.0.0.1:8081/ipfs/QmZznELJFyt9gbzThDmabL1hyE32HyCSBJpRR4CxT3FSdr -o temp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10.0M  100 10.0M    0     0   749k      0  0:00:13  0:00:13 --:--:--  629k

real    0m13.710s
user    0m0.021s
sys     0m0.063s

prometheus (restarted ipfs so it only record one download operation):

# HELP ipfs_http_gw_first_content_block_get_latency_seconds The time till the first content block is received on GET from the gateway.
# TYPE ipfs_http_gw_first_content_block_get_latency_seconds histogram
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="0.05"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="0.1"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="0.25"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="0.5"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="1"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="2"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="5"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="10"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="30"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="60"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_bucket{gateway="ipfs",le="+Inf"} 1
ipfs_http_gw_first_content_block_get_latency_seconds_sum{gateway="ipfs"} 0.000279628
ipfs_http_gw_first_content_block_get_latency_seconds_count{gateway="ipfs"} 1
# HELP ipfs_http_gw_unixfs_file_get_duration_seconds The time to serve an entire UnixFS file from the gateway.
# TYPE ipfs_http_gw_unixfs_file_get_duration_seconds histogram
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="0.05"} 0
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="0.1"} 0
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="0.25"} 0
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="0.5"} 0
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="1"} 0
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="2"} 0
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="5"} 1
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="10"} 1
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="30"} 1
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="60"} 1
ipfs_http_gw_unixfs_file_get_duration_seconds_bucket{gateway="ipfs",le="+Inf"} 1
ipfs_http_gw_unixfs_file_get_duration_seconds_sum{gateway="ipfs"} 4.57331979
ipfs_http_gw_unixfs_file_get_duration_seconds_count{gateway="ipfs"} 1
# HELP ipfs_http_request_duration_seconds The HTTP request latencies in seconds.
# TYPE ipfs_http_request_duration_seconds summary
ipfs_http_request_duration_seconds{handler="gateway",quantile="0.5"} 4.578398782
ipfs_http_request_duration_seconds{handler="gateway",quantile="0.9"} 4.578398782
ipfs_http_request_duration_seconds{handler="gateway",quantile="0.99"} 4.578398782
ipfs_http_request_duration_seconds_sum{handler="gateway"} 4.578398782
ipfs_http_request_duration_seconds_count{handler="gateway"} 1
# HELP ipfs_http_request_size_bytes The HTTP request sizes in bytes.
# TYPE ipfs_http_request_size_bytes summary
ipfs_http_request_size_bytes{handler="gateway",quantile="0.5"} 107
ipfs_http_request_size_bytes{handler="gateway",quantile="0.9"} 107
ipfs_http_request_size_bytes{handler="gateway",quantile="0.99"} 107
ipfs_http_request_size_bytes_sum{handler="gateway"} 107
ipfs_http_request_size_bytes_count{handler="gateway"} 1
# HELP ipfs_http_requests_total Total number of HTTP requests made.
# TYPE ipfs_http_requests_total counter
ipfs_http_requests_total{code="200",handler="gateway",method="get"} 1
# HELP ipfs_http_response_size_bytes The HTTP response sizes in bytes.
# TYPE ipfs_http_response_size_bytes summary
ipfs_http_response_size_bytes{handler="gateway",quantile="0.5"} 1.048576e+07
ipfs_http_response_size_bytes{handler="gateway",quantile="0.9"} 1.048576e+07
ipfs_http_response_size_bytes{handler="gateway",quantile="0.99"} 1.048576e+07
ipfs_http_response_size_bytes_sum{handler="gateway"} 1.048576e+07
ipfs_http_response_size_bytes_count{handler="gateway"} 1
# HELP ipfs_http_unixfs_get_latency_seconds The time to receive the first UnixFS node on a GET from the gateway.
# TYPE ipfs_http_unixfs_get_latency_seconds summary
ipfs_http_unixfs_get_latency_seconds_sum{gateway="ipfs"} 0.000279628
ipfs_http_unixfs_get_latency_seconds_count{gateway="ipfs"} 1
2

After some search, I think It’s because tcp send buffer. My host’s send buffer is 4M, somewhat close to the time different. And if I download a 100M file at 1M/s, promethues record a much normal value with only a few second faster than time which I run curl.

Any other suggestion?