CIDv1.eth : onchain raw/cidv1 library

0xc0de4c0ffee · June 29, 2025, 9:43pm

Hello again from namesys.eth

This is our standalone libraries for onchain raw/cidv1 encoders.

Multicodecs:
dag-cbor, dag-json, dag-pb, json-utf8, raw-bin, unix-fs

Multihashes:
identity-raw, keccak-256, sha-256

test data & CAR files were generated using helia/js client.
Here’s working implementation of json-utf8 using ens wildcards to return onchain data as json/cidv1.

we’ve hit limits on url gateway for raw cidv1..
a) ipfs.io path gateway doesn’t resolve if length > around 165. dweb.io and local subdomain gateways try to force redirect even if cidv1 length > subdomain length limit.
b) eth.limo/DOH added concat support but Kubo can’t handle these long-raw CIDs.
c) ENS isn’t ready for IPLD contenthash support to bypass those limits.

lidel · July 1, 2025, 9:37am

Hi, bunch of things to unpack here, but need to learn more about your use case.

Do you mind sharing example of “long CIDs” that do not work for you?

CIDs for *-256 ones are 32 bytes in length, so how are you producing CIDs that are to long for a single DNS name?

Inlining a lot of data (>32 bytes) using identity one? That sounds like an antipattern but lmk if I misread the situation (CID examples would help a lot).

Two things here:

ipfs.io: mind providing example link? it indeed sounds like a bug, but also could be expected behavior due to explicit hash function safelist / CID data inlining length limits we have hardcoded as a security precaution.
dweb.link: Forced redirect to subdomain is a security feature. Subdomain Gateway Specification was created to ensure separate content roots are not sharing the same Origin. If it allowed for “long CIDs” to be loaded from paths, it would defeat its entire reason to exist.
localhost in Kubo (and IPFS Desktop) is a Subdomain gateway and behaves the same way as dweb.link. If you are just fetching JSON and dont need origin isolation, then use IP (127.0.0.1) – it will act as a Path Gateway similar to ipfs.io

Is there a specification for “concat” approach eth.limo did?

Is the DNS label length limit of 63 characters the only issues?

There is a gap in Subdomain Gateway Specification - we have an open problem of DNS label limit for hash functions longer than 256 bit ones.

There are some notes and Pros / Cons in:

github.com/ipfs/kubo

Subdomain support for CIDs longer than 63

opened 04:11PM - 14 May 20 UTC

lidel

kind/bug status/in-progress topic/gateway P1 topic/cidv1b32 topic/ed25519

> I hoped to punt this until we need to switch away from sha256 in CIDs, but we …may need to solve this problem sooner than expected due to ED25519 keys being new default soon (https://github.com/ipfs/go-ipfs/issues/6916) ## Problem: DNS label limit of 63 > [RFC 1034](https://tools.ietf.org/html/rfc1034#page-7): "each node has a label, which is zero to 63 octets in length" The default CIDv1 Base32 with multihash of sha256 and RSA libp2p-key fits: - Default CID: http://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.dweb.link - RSA: https://bafzbeih7uocwo6vmbjusf4wzw6h5cruhw3gf4jxhevmxssnz5vkgryk2za.ipns.dweb.link but if we use ED25519 libp2p-key then we are 2 characters over the limit: - ED25519 libp2p-key: https://bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link - CID created with `--hash sha2-512` will be even longer: https://bafkrgqe3ohjcjplc6n4f3fwunlj6upltggn7xqujbsvnvyw764srszz4u4rshq6ztos4chl4plgg4ffyyxnayrtdi5oc4xb2332g645433aeg.ipfs.dweb.link Label longer than 63 characters means the hostname can't resolve: ```console $ ping bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link ping: bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link: Name or service not known ``` And links are not picked up by tools like Slack: > ![oops-2020-05-14--17-59-27](https://user-images.githubusercontent.com/157609/81957284-ad54a100-960c-11ea-808f-1de28688f440.png) **Note:** I used ED25519 as an example, but not limited to that single type of CID. Even if we find a way to fit ED25519 in a single label, the problem remains for CIDs with a multihash created with longer hash functions. ## Solved: IPNS-specific fix for ED25519 keys In parallel to the generic fix, we could represent ED25519 keys in a way that fits under 63 characters, solving the UX issue for IPNS websites loaded from public gateways. **Done:** https://github.com/ipfs/go-ipfs/pull/7441 – we support `{cidv1base36}.ipns.dweb.link` which perfectly fits ## Open Problem: generic solution for long CIDs I am happy to open PR with a fix, but unsure if I have the best fix in mind, would love to gather feedback first. ### :question: (A) support split CIDs (but have broken TLS) The first idea I have is to split the label when the max is reached. To maximize entropy for Origin isolation, the remainder should be on the left side: - https://ba.fzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link Pros: - :+1: each long CID gets own Origin – we keep isolation - :+1: path redirect provided by subdomain gateway can take care of splitting - :+1: future-proof solution for longer hashes such as `sha2-512` - the next limit is pretty far away: the maximum length of full domain name: 253 characters, including dots - sha2-512 on dweb.link is 121 characters Cons: - :anger: decreased entropy in security guarantees provided by origin isolation - :anger: wildcard TLS certificate does not pass validation for more than a single level of labels - this will produce annoying UX on public gateways such as dweb.link: TLS warning when opening IPFS website on IPNS. we get the same problem as ENS gateway at `*.eth.link` (https://blog.almonit.eth.link vs https://almonit.eth.link - :anger: copying & pasting CID as-is no longer works on public gateways (user needs to put `.` in the middle etc) - Note: to make it easier UX-wise, we should allow `.` anywhere inside of CID, but internally merge labels, and return a redirect to canonical version that splits at deterministic position (enforcing maximum label for Origin). ### :question: (B) redirect long CIDs to an "insecure" subdomain This would make it possible for content to load, but longer CIDs would not get Origin isolation per CID. To make this bit more clear and idiomatic, we could present this as "cross origin resource sharing" endpoint that allows both CORS requests + supports loading everything from a single origin + has paths locked down in browsers like noted in https://github.com/ipfs/in-web-browsers/issues/157. Think in terms of - `https://dweb.link/ipfs/superlongcid` redirecting to `https://cors.dweb.link/superlongcid` Pros: - :+1: does not break TLS wildcard certs (easy setup for gateway operators) - :+1: useful outside this problem: provides idiomatic way for exposing path gateway on subdomain gateways (for use when origin isolation is not needed) Cons: - :anger: long CIDs don't get Origin isolation ### :question: (C) swap DAG root with CID that uses shorter hash function Pros: - :+1: "just works" Cons: - :anger: decreased entropy - :anger: newly created root blocks need to be persisted somehow: if I bookmark the page loaded via shortened CID and then the root block gets garbage-collected, the address is dead. - potential fix: we could always create redundant sha256 root block for every DAG that uses longer hash function for interop ### :question: (D) leverage HTTP proxy mode (on localhost) When Gateway port is used as HTTP proxy, local client does not perform DNS lookup, but original URL is sent in HTTP request to the proxy for processing. Because HTTP proxy IS go-ipfs node in that scenario, it does not do DNS lookup, but extract original (long) CID and resolves it, without involvement of DNS. As long user agents are not overzealous in validating URLs, this would allow for long (>63) CIDs on subdomains. This is important, because it enables localhost gateway (used by Brave) to resolve long CIDs correctly without any additional hacks. UX details tbd. This could be the solution for localhost gateway, but for public ones we still need something else. ### Other ideas? Would love to find a better way to work around this cc @aschmahmann @Stebalien https://github.com/ipfs/in-web-browsers/issues/89

So far we had no users who asked for addressing that, but if you share example CIDs/links, we could see if there is a path towards addressing the gap.

0xc0de4c0ffee · July 3, 2025, 1:24pm

RFC 4408 used to format longer SPF and TXT records.

Yes, we’re using both inlining and multihashed approach to support dynamic/onchain cidv1. Actual use case would provide both versions as ens contenthash and text records so standalone pinning service can read and pin that long raw data into multihashed ones, but it’s not suitable for more dynamic onchain data updated every block/use case.. Also, we use inline cidv1 to mix offchain pinned ipfs cids with onchain data using dag-cbor link tag.

Looks like that’s case for ipfs.io, it stops resolving after cidv1(base32) length > 463 with 285 utf-8 chars in raw data.. But same longer cids resolve with local 127.0.0.1:8080 path gateway running Kubo v0.35.0

we’ve base16 cids and CAR files in test dir.

github.com/namesys-eth/cidv1.eth

test/data/dag-cbor-test-data.sol

main

// SPDX-License-Identifier: WTFPL.ETH
pragma solidity ^0.8.0;

library DAGCBOR_TestData {
    // Raw strings
    string internal constant Hello = "Hello";
    string internal constant World = "World";
    string internal constant HelloWorld = "Hello World";

    // Raw CIDs with different hashers
    bytes internal constant HelloWorldRaw = hex"0155000b48656c6c6f20576f726c64";
    bytes internal constant HelloWorldRawSha256 =
        hex"01551220a591a6d40bf420404a011733cfb7b190d62c65bf0bcda32b57b277d9ad9f146e";
    bytes internal constant HelloWorldRawKeccak256 =
        hex"01551b20592fa743889fc7f92ac2a37bb1f5ba1daf2a5c84741ca0e0061d243a2e6707ba";

    // JSON CIDs with different hashers
    bytes internal constant HelloWorldJsonRaw = hex"01800400117b2248656c6c6f223a22576f726c64227d";
    bytes internal constant HelloWorldJsonSha256 =
        hex"018004122065a4c4ca5acfc4bf3b34e138c808730f6c9ce141a4c6b889a9345e66c9739d73";

This file has been truncated. show original

Topic		Replies	Views
CID v1 and Base32 encoding in Solidity	1	48	July 19, 2024
Confusion betweed CID and "bare"/"naked"/"raw" Multihash multiformats , multihash	1	796	May 3, 2019
Can you convert CIDV0 into a CIDV1 BASE32? Help	5	2114	August 20, 2020
New Go Library: Human-Readable CID Conversion for IPFS Ecosystem and Usage go-ipfs , use-cases-and-apps	2	48	May 22, 2025
Ipfs cid v1 help me derive cid by hand Help	3	180	March 10, 2024

CIDv1.eth : onchain raw/cidv1 library

Related topics