Best IPFS practice for event or time series data

Ecosystem
1

I have a small Particle network with sensors periodically reporting data with time stamps, temperature, humidity etc. I want to store that data… IPFS seems like a perfect distributed way to do so. And I want to view the last say 48 hours on a web page.

I have a few go-ipfs and ipfs-cluster processes running, so I kind of understand that part.

I “think” the answer has these ingredients:

  1. ipfs gateway (for posting update)
  2. IPNS pubsub to be aware of newly posted data
  3. some kind of ipfs-cluster awareness of updates to ensure distributed storage.

What is the recommended way do go about this?

2

It is difficult to answer this without knowing more about your setup.

  • Are you running IPFS nodes at the places where the data is gathered?
  • What quantities of data are we talking about?

With the ingredients you should definitely be able to roll your own solution. However, you might consider using something like https://github.com/orbitdb/orbit-db if the amount of data is not that high.

3

FYI I have added this to ipfs-cluster:

If you end up coding something along those lines, let us know!

1 Like
4

The data is gathered using a sensor connected to an arduino-ish device e.g. Particle. Via the Particle cloud service, the sensor data is published via HTTP to a data aggregation/collection golang service. The data rate is approx 1K per minute per sensor. I currently only have about 5 sensors… designing for 10^3

Data aggregation/collection options I see:

  1. Use a go-ipfs gateway… where does pubsub fit here?
  2. add IPFS client libs to my golang service… how do I know if/when data is stored?

User interface use cases:

  1. Latest measurement per sensor: Use IPFS JS client to subscribe via pubsub to new data. Whats the high end for number of topics? Is there and advised message/topic ratio? e.g. given 10^3 sensors, create topic per sensor?
  2. Spark line per sensor: Use IPFS JS client to fetch a “window” of data… not sure how to fetch a window. Encode time in file name?

Thank you in advance for any/all time spent thinking about this. If you happen to be in Portland, Oregon I’d be happy to buy you a beer/coffee or two for your time.

1 Like
5

  • So 1K samples per minute per sensor? What are the latency requirements? When do you need to see new data in your web ui?

  • So the “data aggregation/collection golang service” is not really distributed, but a central cluster that can possibly run ipfs?

Regarding number of topics: I would try a single one and see how far that gets you. 1000 messages per second are not exactly much in a data center. And if you don’t have enough network bandwidth, splitting the load over multiple topics is not going to help.

6

Old topic but hey, this might be interesting for people looking into similar needs:
http://blog.klaehn.org/

1 Like
7

( For discoverability:

… blog post was first seen in this sibling thread :

)