Filed order of protobuf?

Help
,
#1

Hi IPFS,

I am trying to implement the dag-pb functionality in c++. I almost got it correct (by comparing the dag and mutihash resulted from adding a file), but I’m stuck in a problem.

I think normally the field order after serialization in protobuf is determined by the field number in a .proto file. And IPFS has PBNode defined as

message PBNode {

  // refs to other objects
  repeated PBLink Links = 2;

  // opaque user data
  optional bytes Data = 1;
}

So I expect after serialization a PBNode, the Links field should be after the Data field. And this is the behavior in my implementation. But ipfs block get <cid> | protoc --decode_raw shows the order is not in that order. The following is an example (excerpted).

...
2 {
  1 {
    2: "#\332r\216K~\024M\\\367\234\245\352\234\244\314\303&~\270m+\217Fh0\005D\254\313\322\350"
  }
  2: ""
  3: 523
}
2 {
  1 {
    2: "\212.\371G\202\re\033Ko\240QR\035\371\264\363GZ\355\'l\306\031Q>\250\335\"\262-("
  }
  2: ""
  3: 27
}
1 {
  1: 2
  3: 31763
  4: 512
  4: 512
  4: 512
  4: 512
  4: 512
  4: 512
...

So the Data = 1 filed comes after the Links = 2 field. I believe it is this inconsistency that makes my multihash differs from go-ipfs’ multihash. My question is: is this “mis-ordering” intended by IPFS? How should I make my code consistent with IPFS?

I’m new to protobuf. I tried to google, but didn’t find a solution to change the order. Any help will be appreciated!

#2

This is a long standing “ethereum moment”[1] with DagPB: it was defined over a protobuf encoder that had a bug, encoding objects in the order it saw them. Fixing the ordering to emit “compliant” protobuf would mean making it impossible to generate matching CIDs for the same content (the block content is now different). go-ipfs and js-ipfs bundle patched up encoders that produce this ordering to this day.

Since an upgrade to the UnixFS stack will move everything to Dag-CBOR, there isn’t a compelling argument to changing this: each implementation that wants to converge needs to implement the broken ordering in Dag-Pb :frowning:

I do not have a good answer on how to make a C++ encoder emit “non-standard protobuf”. I would look around github to see if someone already forked and patched up one such encoders, or just do so myself.

Sorry things are not perfect :frowning:

[1] https://ethereum.stackexchange.com/questions/550/which-cryptographic-hash-function-does-ethereum-use

#3

Thanks a lot for fast reply. Now I know it.