@mikeal I was listening to you on your youtube weekly catch-up. Was very interested in what you were doing with SQL and the battle between concurrency and streaming.
Have you considered a combo?
Index Tables with Stream Chunk Retrieval? As you navigate you can continue to stream.
Basically a variable entry point into your IPLD tables.
Bonus, you will be able to run prestatistics prior to retrieval for truly cost-based tuned optimized Queries.
@mikeal I kept watching the video and you guys started talking about exactly this, but then you guys ended the stream just when it was getting good!!
I do something like that here IPSQL/table.js at master · mikeal/IPSQL · GitHub
I do the index writers concurrently and collect the blocks they emit, when each one finishes i emit any new blocks that had been generated from any concurrent writer.
The reader is a bit trickier because those are concurrent functions that have some merge logic in them.
I think I get the gist of it.
I want to get to the point where I can play around with this data in a table. Still getting my toes wet in this technology.