If the file is packaged as car, the file cannot be accessed directly through the gateway

Help IPFS
,
1

I package the file to form car and upload it to my local gateway, but when I want to read the file directly through the root cid of the file, can’t get it, why is this, and I can’t request the block in the car file, what should I do to get it

2

you need to use ipfs dag import instead of ipfs add to import car files.

4

Filebase supports ipfs uploads up-to 1 TB is size using our S3-Compatible API

5

Thank you! I just consulted the Kubo API documentation and found the /api/v0/dag/ endpoint for importing .car files. However, I’ve encountered a problem. I’m using Crust to pin my files, but there’s a limitation of 1GB per file. So, I plan to package the files into a .car file and then split it into multiple 100MB files for upload. However, I’m pinning the split .car files, which results in them being inaccessible. Is there any way to index these blocks so they can be pinned?

6

Now I have progressed to the part of defining dag-pb, to index those blocks saved in crust, but I’m not sure how to define the blocksizes inside. Do you have any reference materials?

7

What do you mean by defining dag-pb blocksizes, the TSize field in the protobuf? That field is generally the cumulative sum of the sizes of the DAGs it references (not counting any potential deduplication).

If so, it sounds like you’re already moving down this path but if you’re writing Go code for this you might consider taking a look at the GitHub - hsanjuan/caribdis: CARibdis swallows IPLD CAR files and does things with them. overlay command. It’s not exactly what you want but it’s in the vicinity so maybe submitting a PR or referencing some of the code would be helpful.

As an FYI to anyone else running into similar splitting constraints with pinning services there is also GitHub - nftstorage/carbites: 🚗 🚙 🚕 Chunking for CAR files. Split a single CAR into multiple CARs. which might be useful. It has a few splitting options based on what you need, although might not totally work here. If you’re looking to construct graphs probably what you’d want to change here is:

  1. The root/overlay nodes should reference the blocks in the DAG using the raw codec rather than their true codec. Without this the DAG could extend beyond just what’s in the emitted CAR file (raw effectively turns off following hash-links because the block is treated as an opaque blob)
  2. There’s a constraint around using a single root block for the “rooted” mode that’s probably fine for you, but if it’s not enough and you’re seeing errors then you want multiple linked nodes for the overlay.