Pinning issue while adding multiple files

Help IPFS Cluster
,
1

Hi Team,

I’m trying to add multiple files simultaneously in IPFS-cluster node but the issue is that only a single CID is being pinned to the cluster while other CIDs are not getting pinned.

Although I’m able to access those files using GET api call but why it isn’t pinning all the CIDs.

2

I believe only the last file gets pinned.

It seems a bit weird to me that it is like this, but I think there must have been some technical reason.

Workaround is to do one request per added file, or to use the wrap-in-directory option to have a single pin covering all.

I’m a bit annoyed by this as it breaks implicit compatibility with ipfs-add and I don’t readily see a good reason to not pin every added file, I suggest you open an issue in the repo, maybe it is fixable.

3

Hi Hector,

Thanks for your response. I have also raised a request in the issues section of IPFS cluster github repo but noone is responding.

Yes, only the last file is getting pinned in the cluster. We can’t process one request for each file and even the wrap-in-directory isn’t helping us as we are processing base64 of the files in our request body so the name field is blank for each of the file and so we aren’t able to map the CIDs for each file individually.

We have debug the whole /add endpoint flow but the pin operation is called internally we suppose so as the client is just calling the IPFS endpoint from doStream func().

So, the challenge for us is if we are adding multiple files using IPFS cluster /add endpoint as only a single last file is getting pinned so there is a possibility if the GC runs in the meantime then it will remove the unpinned hashes from the IPFS cluster cache.

Now when we are trying to add files using the IPFS node http endpoint i.e. on 5001 port with pin flag passed as true(although it’s true by default as well), then all the files are getting pinned on the particular node. So, the issue of gc running after the add operation won’t remove the files.

So, we are thinking like we can call add functionality on IPFS node’s endpoint, then for pinning in the cluster we can call the IPFS cluster node’s /pins/{cid} endpoint.

4

Thanks, I hadn’t seen it. I’ll try to fix for next release.

5

Thanks, Hector.

Actually now we are adding our files on IPFS node’s add endpoint but now we want to implement https based secure connection for the rest API call. We have achieved that in IPFS cluster by adding these env variable in our cluster node:

CLUSTER_RESTAPI_SSLCERTFILE: /path-to-cert/cert.pem
CLUSTER_RESTAPI_SSLKEYFILE: /path-to-key/key.pem

But we are not able to implement that for IPFS node’s http endpoint, so is it possible to do that with some configuration parameters.

6

Kubo doesn’t support SSL endpoints directly and needs to have something else doing ssl termination in front of it.

7

Okay Hector. So we have to use reverse proxies to handle it.