IPLD - The Battle between Concurrency vs Streaming

@mikeal I was listening to you on your youtube weekly catch-up. Was very interested in what you were doing with SQL and the battle between concurrency and streaming.

Have you considered a combo?

Index Tables with Stream Chunk Retrieval? As you navigate you can continue to stream.

Basically a variable entry point into your IPLD tables.

Bonus, you will be able to run prestatistics prior to retrieval for truly cost-based tuned optimized Queries.

@mikeal I kept watching the video and you guys started talking about exactly this, but then you guys ended the stream just when it was getting good!!

I do something like that here IPSQL/table.js at master · mikeal/IPSQL · GitHub

I do the index writers concurrently and collect the blocks they emit, when each one finishes i emit any new blocks that had been generated from any concurrent writer.

The reader is a bit trickier because those are concurrent functions that have some merge logic in them.

I think I get the gist of it.

I want to get to the point where I can play around with this data in a table. Still getting my toes wet in this technology.