Not enough peers to allocate CID

Help IPFS
1

Hi,
I setup a cluster in go, and now in pinning cids it gives me the error:

not enough peers to allocate CID Needed at least: 2. Wanted at most: 6. Valid candidates: 0.

However, when I check with calling http://127.0.0.1:90904/peers it shows me that the cluster has more than 6 peers and also I pinged some of them and more htna 6 are pingable. what can the reason and how to further troubleshoot?

2

ipfs-cluster-ctl health metrics

ipfs-cluster-ctl health metrics freespace

And check that the metrics that play a part in the allocations are there, per the allocator configuration.

3

I don’t see any issues with metrics:

ubuntu@ip-10-0-0-213:~$ curl http://127.0.0.1:9094/monitor/metrics
["ping","pinqueue","tag:group","freespace"]
ubuntu@ip-10-0-0-213:~$ curl http://127.0.0.1:9094/monitor/metrics/freespace
[{"name":"freespace","peer":"12D3KooWBkpV9t98rco4i3HjdyKDH5hspMzo539aNNnEFdV7jaYx","value":"19794815208","expire":1716237015896301374,"valid":true,"weight":19794815208,"partitionable":false,"received_at":1716236985931398290},{"name":"freespace","peer":"12D3KooWEnoexhyw8nfH5vKHNfDoA4QSxDyNKQjnpezAVQUnSXux","value":"9524767101","expire":1716237013651057475,"valid":true,"weight":9524767101,"partitionable":false,"received_at":1716236983651292364},{"name":"freespace","peer":"12D3KooWLHJZNub2y8JkZGhZK3ZRnKchgh3xDf2esWUDARYwoZ4o","value":"19999976304","expire":1716237011842675820,"valid":true,"weight":19999976304,"partitionable":false,"received_at":1716236981853774981},{"name":"freespace","peer":"12D3KooWMYKZFjbWv4JQA7Mg7hE4KKEzfer5SpyZKSfw2So7qtef","value":"19995059949","expire":1716237020557301585,"valid":true,"weight":19995059949,"partitionable":false,"received_at":1716236990587356112},{"name":"freespace","peer":"12D3KooWMpL7AZcsw4tHaUQLoGssuuYJVHqGUNXPgYJLSpynmkGo","value":"19999856685","expire":1716237024094107791,"valid":true,"weight":19999856685,"partitionable":false,"received_at":1716236994129413655},{"name":"freespace","peer":"12D3KooWQueVTT7hhJW9zRG3iLxkohCxMX5mbj8jPhDiMVPYkLFb","value":"19998465313","expire":1716237018978198413,"valid":true,"weight":19998465313,"partitionable":false,"received_at":1716236989006989404},{"name":"freespace","peer":"12D3KooWR7RrYvYN7X9kYKZPB5Zc8KDDi1YuGQLKfix8NL7CvtSH","value":"19579836152","expire":1716237023244536528,"valid":true,"weight":19579836152,"partitionable":false,"received_at":1716236993279470453},{"name":"freespace","peer":"12D3KooWRr6MPQVMCD2mB8GruPWeo6w2wLQQxL2rimVLRvWsVqSk","value":"19794432122","expire":1716237023595255643,"valid":true,"weight":19794432122,"partitionable":false,"received_at":1716236993641956079},{"name":"freespace","peer":"12D3KooWSiFFURqQLGqgqUYiogR7d8vhwfv8fk5FcyJgyDRGdKmZ","value":"19991491196","expire":1716237012197179894,"valid":true,"weight":19991491196,"partitionable":false,"received_at":1716236982226594317}]
4

can it be due to the below section in service.json where there is no definition of numpin in monitor or informer but it is used for the allocator?

"pin_tracker": {
    "stateless": {
      "concurrent_pins": 10,
      "priority_pin_max_age": "24h0m0s",
      "priority_pin_max_retries": 5
    }
  },
  "monitor": {
    "pubsubmon": {
      "check_interval": "15s"
    }
  },
  "allocator": {
    "balanced": {
      "allocate_by": [
        "tag:group",
        "pinqueue",
        "numpin"
      ]
    }
  },
  "informer": {
    "disk": {
      "metric_ttl": "30s",
      "metric_type": "freespace"
    },
    "pinqueue": {
      "metric_ttl": "30s",
      "weight_bucket_size": 100000
    },
    "tags": {
      "metric_ttl": "30s",
      "tags": {
        "group": "default"
      }
    }
  },
  "observations": {
    "metrics": {
      "enable_stats": false,
      "prometheus_endpoint": "/ip4/127.0.0.1/tcp/8888",
      "reporting_interval": "2s"
    },
    "tracing": {
      "enable_tracing": false,
      "jaeger_agent_endpoint": "/ip4/0.0.0.0/udp/6831",
      "sampling_prob": 0.3,
      "service_name": "cluster-daemon"
    }
  }

Because in numpin I get:

ubuntu@ip-10-0-0-213:~$ curl http://127.0.0.1:9094/monitor/metrics/numpin
[]
5

Yes, since the numpin informer is not configured in the informer section, then it is not used.

And also, you probably don’t want to use it unless all your pins are roughly the same size or you are against some peers having more pins than others.

6

The reason I am using it is that in our network we have nodes with storage spaces from 1TB to 70TB. I roughly want to make sure all receive data to pin in a uniform way (so not that someone with a smaller storage needs to wait for hte larger ones to fill before receiving the pins). Any better suggestion for htat?

I now added the numpin in informer:

  "informer": {
    "disk": {
      "metric_ttl": "30s",
      "metric_type": "freespace"
    },
    "pinqueue": {
      "metric_ttl": "30s",
      "weight_bucket_size": 100000
    },
    "numpin": {
      "metric_ttl": "30s"
    },
    "tags": {
      "metric_ttl": "30s",
      "tags": {
        "group": "default"
      }
    }
  }

But I dont see it in metrics. Am I missing somehting?

ubuntu@ip-10-0-0-213:~$ curl http://127.0.0.1:9094/monitor/metrics
["ping","freespace","pinqueue","tag:group"]
7

I also saw that we have access to

"disk": {
      "metric_ttl": "30s",
      "metric_type": "reposize"
    }

Does it mean instad of numpins, I can use reposize in the alloator for the purpose I mentioned to make sure if a repo size grows muhc larger htan other nodes, others are used (instead of relying on number)?

"allocator": {
    "balanced": {
      "allocate_by": [
        "tag:group",
        "pinqueue",
        "reposize"
      ]
    }
  }
8

I think reposize should work for what you want.

I’m bit sure about numpin, but better to avoid as it is also very inefficient in that it lists all pins and counts them every time.