The background for this comes from this github issue where i was asked by @vasco-santos to move the discussion to a more public place for a more out in the open discussion.
What i’m trying to do is the following.
I want a website to interact with the IPFS PubSub mechanism. The website should listen to a specific PubSub channel. If an IPFS client posts a message on that specific channel, the site should receive it. And vice versa.
The usecases for this are broad! For instance, think about the following:
Messaging app
or pinterest where, upon browsing, the client requests the next batch from the server.
or a news/post/forum site where the client is notified of a new post to download and display
or a wallpaper site where clicking on a thumbnail asks the PubSub channel which hash belongs to full sized version
etc…
In my very specific usecase i have a site where i want to show some data about movies and series. I first want the user to type in the search query and then fetch the relevant content from “the server”. The “server” here is running a full IPFS node and is subscribed to a specific channel. The client (the website) is also subscribed to the same channel. Both should communicate.
The working principle of this is Socket.io. It works. But the downside is that i need to provide an address there. The complexity becomes a little more when the server is placed in a local network environment. I do not want to provide any addresses as that only is annoying when i change hosts. IPFS is very well suited for this, i think.
Now i thought the libp2p-webrtc-star project was the one that would be able to do this. I had set it up (you can see it at star.sc2.nl). I thought it would serve as a kind of proxy between the backend IPFS host where the libp2p-webrtc-star would be connected to and the site which would be connected to libp2p-webrtc-star. But this proved to be a wrong assumption apparently. Using this project you can connect to other libp2p-webrtc-star nodes, not to IPFS client nodes.
So again, i’m looking for a way to connect IPFS client PubSub to a website.
I’m kind of lost at the moment at what i can still try to get this working.
Perhaps others have bright ideas the get the above desired solution working? I’m fine if i need an intermediary proxy (like star.sc2.nl, but then apparently not with libp2p-webrtc-star).
Also, if you have suggestions for me to try, please be verbose in the examples. I’ve spend way too much time trying to get libp2p-webrtc-star to work (composed from a bunch of different snippets) to figure out that it’s not suitable for my intention. And as i’m now apparently in a very niche area of IPFS, documentation is very difficult to find if existing at all. Therefore, please, be verbose
Hey Mark, feel free to get in touch. Among other things, I’ll be giving a talk on qdesk.org tomorrow in the IPFS meetup. We’re working on a framework to make it easy to build dapps like the one you’re working on (using webrtc star) + our pubsub communication comes with an extra layer of security (rsa/aes encryption and ec signatures). I’ll be busy until after the meetup but then I’m happy to help =)
IPFS client and website seem to be the same piece. Is this assumption correct? If so, let’s call them browser nodes to ease the discussion.
So, webrtc-star will enable you to discover and connect to peers using that transport via WebRTC. In this way, you can have a webrtc signalling server, where your browser nodes can be informed when other peers join and can afterward connect between them. Once they are connected, if they have pubsub enabled, they will be able to exchange pubsub topics based on their subscriptions without further configuration.
The key thing here is to understand why the “Server” node is needed? Do we need it? It seems so, as it seems you will want to have content routing with a set of IPFS Nodes that will be responsible for it.
With this in mind, is the flow you are thinking something like:
Browser Node A has some new content and will add it to the IPFS network
Browser Node A will broadcast via pubsub that it just published new content with its IPFS cid
Browser Node X will receive the pubsub message
Browser Node X will attempt to get the content from the received IPFS cid
Thanks @vascos-santos! I still have to work on my Jake and I didn’t get to do my demo on the machine I was presenting from but I’m happy that you enjoyed the intro =)
@markg85 didn’t get in touch yet and I’m not sure what exactly isn’t working for him. PubSub based sync of chat and (forum) posts were among the use cases he mentioned in his post and they’re, at least in a basic form, in the Quest Network example app.
No, they are not.
I might have phrased it confusingly.
Consider IPFS Client to be an IPFS node running anywhere in the world. A headless node without a browser. You could call it a IPFS Server too though that is wrong too as there are no “servers” in the IPFS world… You see where it gets difficult to phrase it correctly? Well here it’s difficult already
So i think calling it a headless IPFS node is the most accurate i can think of.
Nope, sorry
But i now do see, again, where the confusion comes from.
The Browser node would be a node running in the browser that somehow magically can get the pubsub channel messages from the headless IPFS node. How, i don’t know. Yet. (as all attempts thus far failed). The Browser Node would be using js-ipfs.
A more accurate depiction with the headless IPFS node would be as follows:
headless IPFS node is on the IPFS network.
headless IPFS node adds data to a specific channel and is subscribed to it to listen for incoming data from others besides itself
Browser Node connects to the same channel as the headless IPFS node. It does not connect to headless IPFS node directly by peerid. The only thing connecting the two is the common pubsub channel they both use.
Note though, if connecting to a peer means “bootstrapping” then it can obviously connect to the specific headless IPFS node too. It would merely be one of the bootstrapping nodes. It can connect to that node but shouldn’t need to be.
Browser Node can send new messages on the channel
headless IPFS node receives those messages and can do something with it
I hope that clears it up somewhat?
Right now i only know of one more way to get it potentially working. That would be by creating a nodejs server that internally connects to a pubsub channel and exposes a websocket for websites to connect to it. That would effectively be a proxy (or a pubsub proxy if you will). But this approach would add a single point of entry (and thus potentially failure) which is ok’ish… But not preferred. With this hypothetical approach there would be a 1-on-1 connection between the Browser Node and the headless IPFS node via that nodejs proxy.
Well, i’m not using the Quest Network so that’s why it’s not working for me
I was - thus far - trying to use IPFS native stuff. Not to rule out Quest Network or anything, I just didn’t consider trying it yet.
I’ll watch that meetup later today to see what i missed there.
Anyhow, how can i contact you? Are you on discord somewhere (and what’s your handle there)?
“Well, i’m not using the Quest Network so that’s why it’s not working for me ”
No but you can look at the code and see how we’re doing it. We have this qOS library that runs an IPFS node in the browser and in NodeJS. Writing a NodeJS application with it would essentially create such a headless node and using the library in Angular/Electron allows peers in the browsers to send messages to said headless node. It’s all very basic and should be seen more as kind of a default skeleton and we keep changing stuff a lot until we’re at least at v1 but what we do have is exactly what you described.
Please have a look at how we’re spawning the IPFS node and how we’re setting up and using the PubSub channels and see if that’s not already what you were looking for: qdesk.orgQuestNetwork · GitHub
I just did have a quick look over it.
I’m sure it works as you claim and from what you describe the “concept” is exactly what i need. But that code is a mixture of IPFS, Electron, Angular and a whole slew of other packages. I can’t just “use” that or even use it as a base.
If you have an example respoitory somewhere that is as concise as possible. Meaning it ideally has:
IPFS handling (via it’s API)
Expose that in some web-ready protocol (like websockets)
Also, if i understand the code correctly you do setup n p2p-webrtc-star which isn’t working for me. That only works for sites that need to connect with each other. So for a chat client that would work. But for a “server notifies you of change, site responds to it and vice versa” it won’t work.
Unless… (and that seems to be the route you picked) you let the server (via electron) be handled as if it were a website too. As it too can then run p2p-webrtc-star. That will probably work too but i’m not at all eager to go that route with electron. I prefer to keep electron out of the mix entirely and let the server be headless in concept and in packages used.
To update what i said earlier regarding my assumption with a websocket proxy.
I’m now actually going to take that route.
It is going to:
Wrap the IPFS API calls in websocket (socket.io specifically). I’m focusing only on pubsub at the moment but i will write it in such a way to later extend it with all the other API calls.
Per API call, have a - server side - list of “public keys” that are allowed to execute that API call. There will be a bit of signing involved here.
Note the “server side” is in the same place where you have a full IPFS node running.
The downside is that the website has a single point of failure now, the entry point for that websocket.
The site also isn’t - at all - a participant in the IPFS network. It has no “peerid” of it’s own. Oh well, it doesn’t need it so that’s fine for my case.
The upside is that it is going to allow access on websites to most notable the nice PubSub feature of IPFS! I’m going to host this project here: https://github.com/markg85/ipfs-websocket-api For now there is nothing there yet, but you can expect a very early (alpha) version that has the above for PubSub within a few days. As i simply need it for a website and want to make it
Right now i already have pubsub working where i publish something on my local machine (it has no relation to the website and doesn’t have the ipfs-websocket-api running). That gets neatly broadcasted via PubSub to a machine with IPFS on some vultr node where the wrapper is also active. It catches the message and re-transmits it over websockets which is received by my site (it has a websocket connection to that wrapper) and is then happily displayed
