Data transformation tool on top of IPFS

Hi everyone,

We have just released Holium (holium.org), you can now transform data in a way that enables sharing of its history on IPFS.

We believe this is key to manipulate data with confidence and to unlock widespread trust.

We would love to hear your feedback and ideas for future developments :slight_smile:

2 Likes

A lot of good stuff in there, I like it.

Separating data from meaning/types reminded me of data-oriented design VS OOP for some reason.

Using web assembly modules on IPFS is very good idea. Computation you can trust!

Do you also bundle data+transform in DAGs ? Having one CID for the whole thing would be convenient. edit: just saw, you can, very cool!

Hi SionoiS, thank you for your kind message and glad you found out the answer for yourself! We have designed the protocol so that every element in the DAG has a CID and even the topology of the DAG has a CID. With all this we can have a CID for the whole thing.

Being able to infinitely compose transforms, crowd-source data-sets/transforms and all that being addressable by CID is pretty amazing!

I’m still working my way through the documentation but while the concept is interesting I’m wondering about the performance impact if running everything through a CID. Are there any plans of possibly compilation and optimization of the process graph?

Hi zacharywhitley,

Actually, the design of the protocol ensures a clear distinction between (a) the execution of the pipeline and (b) the storage and sharing of of pipelines and execution artifacts.

When executing processes from the pipeline/graph of processes, many usual optimizations may integrate pretty seamlessly.

Only when execution artifacts have been generated and that the focus is on their sharing, or at least storing in a way that facilitates their sharing, possibly on IPFS, only then are CIDs generated. Therefore in the first implementation of the protocol our priority has been to guarantee the soundness of these representations more than the performance of their representation.

Hi SinoiS, yes there is a lot of potential :slight_smile: Do you have some use cases in mind that could benefit from that?

Thanks. Reading back over my last post I didn’t properly express my enthusiasm for what you’re doing. It’s very cool. I’ve had some ideas that I’ve been exploring along similar lines but with image processing.

Thank you :slight_smile: Would it be possible to give us more details about your idea for image processing? Holium is able to process complex data like images. It could help us a lot to understand your needs!

My use case would be decentralized social media analytics but that’s ways off. I’m still building the social media part. :smiley:

@albertdessaint how do you like Rust?

Wow good luck with that :muscle: We love Rust!

I just wanted to chime in and say: Holium looks really cool. This is exactly the kind of thing IPLD was designed for and “transformations” has been on the TODO list for a long time.

  • Please give feedback to the IPLD team (either here, or by opening issues in Issues · ipld/ipld · GitHub).
  • I’m sure you’ve already seen it, but you should chat with the Qri folks.

Hey, thank you for your kind words Stebalien! I will pass on the message to the technical team :slight_smile: