Is there an implementation of IPFS that includes erasure coding like Reed-Solomon right now?

Help
,
9

Your analysis it very clearly!
I’m work in progress, but meet some problem just close to what you said.

I found that, but there are several question.

  1. when sharding dag service put the data to ipfs, it ingest block, record the size of rawData, but in the end, pin data is bigger than original data(red label of follow image)? But when use ipfs get to get the file, the file size is equal to the origin file.

  2. why only wrapping data in directory can make cid = lastCid when sharding(blue label of follow image)?

    image
    image1920×415 145 KB

  3. The file is read by multipart.Reader, maybe we only can read data once then use reedsolomon.Encode to generate parity shards (the checksum you mentioned), and get data blocks by DAGService.Add(). So I use that method and generate parity shards during traversal dag. But here comes trouble. After I generate parity shards, I can’t append the parity shards to multipart.reader, Where can I store the parity shards?

    My implementation right now is directly pin the parity shard as a new file(it’s really simple but stupid, may be reuse sharding is better), and record the metadata to new a ClusterDAGPin parity-ClusterDAG links by ClusterDAG.

  4. It sounds great, I will follow your advice!
    btw, I already implement a new get method in ipfs-cluster, never mind. I just want to ask a question. I simple use BlockGet to implement DAGService.Get, and I can get whole file by root cid, but cannot get shard rawdata by shard cid. it show the error below. Maybe I use wrong codec decoder?

    cannot parse 1st data shard(bafyreigia5dxdgizm2uhl577pkpu5afszghtqhsxacq34zhf3oirs7ixti) to rawdata:  protobuf: (PBNode) invalid wireType, expected 2, got 3

That’s all, thanks for your advice!