I’m thinking about implement a web proxy (a forward cache for client side browser) using IPFS as a distributed web cache. The HTML pages are cached in IPFS for future access.
Here the problem is that the web cache is addressed by URL (or URL hash), not the page content hash. Is there a way to be able to tag the page content and make it addressable/searchable by URL?
Use a separate index if you want true search functionality. If exact queries are enough, you can use regular IPFS directories.
http://www.mysite.com/about/about-us.html -> /somedirectory/www.mysite.com/about/about-us.html
How will you handle malicious nodes inserting garbage?
You should also look at https://github.com/oduwsdl/ipwb They’ve been tackling that issue comprehensively in order to build ipfs-based web archives.
How? All I can see is that they extract warc archives to take advantage of IPFS deduplication, they’re still referencing them by hash.
@flyingzumwalt Awesome, ipwb is pretty much what I want. I’ve been looking into OrbitDB (https://github.com/orbitdb/orbit-db) too, to use IPFS as a KV store.
@es_00788224 @flyingzumwalt Thanks a lot!
You are right. ipwb didn’t make uri searchable in IPFS. What I take from ipwb is:
While skimming the code, I thought ipwb uses IPFS directories for CDXJ urls but obviously they didn’t do it. Seems ipwb has very limited functionality.
I’m going back to OrbitDB.
Why do you need orbit-db? A directory is a directory, just treat it as one:
IPWB is just a way to put WARC files on IPFS.
While IPWB is not just a way to put WARC files into IPFS, but for now, this statement holds true to some extent, that is just the indexer part, not the reply side of it. It is not difficult to build a transactional, on-demand, or client-side archiving system using IPWB. However, IPFS is not directly usable for the purpose described in this thread, but some techniques can be borrowed from it to implement such a system. Here are a few assorted thoughts around it, some are utilized in IPWB and some are not.
Via header that describes all the headers that can be used for content negotiation. For example, an exact same URL can be used to serve the content in many different languages using Accept-Language header by the client (if the server advertises that it allows content-negotiation on that header). If a proxy/cache does not consider the Via header, it would overwrite the cache with the content in a different language, if the URL is the same. As far as I know, VA Tech research mentioned above does not consider Via header as it is still only a research project and not a usable product. Additionally, in case of IPWB, archival records are often explored just by a URI and a datetime, so other possible content negotiation dimensions are not part of the index.Hope, this would help you come up with something that you can share with the community.
Add pointers to each CDXJ record at /ipns/YOUR_HASH/THE_URL.
Unless you’re planning to build something extremely large, a simple reverse word index (also distributed over IPFS) searched by some quick JS code with boolean operators is enough for most purposes.
We have thought about a few approaches around the full-text search implementation. here are some assorted points in that regard:
Content Hash => Array of URI-Ms (where a URI-M is a pointer to a memento, i.e., an entry in the CDXJ) can be maintained independently for final presentation purpose which would require some clever ways to rank or bundle competing entries with the same hash (of the same URI-R at different times or different URI-Rs).@mfan if you’re looking at orbit-db you should also look at @pgte’s work with CRDTs on IPFS. It does the same things as orbit-db but uses yjs to handle the CRDTs instead of relying on a homegrown implementation, which makes it easier to use.
@Kubuxu can you link to the code we used to do serverless querying in the wikipedia snapshots?
Blog post with more info about this at: https://ipfs.io/blog/29-js-ipfs-pubsub/
@flyingzumwalt @ibnesayeed @Kubuxu @es_00788224 @daviddias Thanks for all the information. They are very helpful to get me started with some experiments.
For the web proxying, I did a simple experiments with Proxy2 (https://github.com/inaz2/proxy2) on popular websites, e.g. cnn.com. The http and page content are fetched and cached using their URLs, and get replayed at a later time in the browser. It seems relatively easy and no blocking issues:
Caching on IPFS
In my experiment, I used IPFS unixfs-dir and IPNS. It’s very slow.
....
....
# Received a HTTP request, check for page cache in IPFS
uid = hashlib.md5(req.path).hexdigest()
ipfs_path = '/ipns/%s/%s' % (IPFS_BASE, uid)
try:
# Return the page cache (http + page content) to browser.
page_dict = self.api.get_pyobj(ipfs_path)
print 'URL %s found in ipfs: %s' % (req.path, ipfs_path)
print 'UID %s found in ipfs: "%s"' % (uid, page_dict['body'][:60])
print 'UID %s IPFS path: %s ' % (uid, ipfs_path)
self.wfile.write(page_dict['header'])
self.end_headers()
self.wfile.write(page_dict['body'])
self.wfile.flush()
return
except Exception as ex:
# No cache, go fetch the page.
print 'URL %s not found in ipfs: %s' % (req.path, str(ex))
print 'UID %s not found in ipfs.' % (uid,)
print 'UID %s IPFS path: %s ' % (uid, ipfs_path)
print 'Fetching the page... for %s' % (uid,)
...
...
# Fetch page content.
...
...
# Downloaded the page content, store it into IPFS.
uid = hashlib.md5(req.path).hexdigest()
with self.lock:
ipfs_res = self.api.name_resolve('/ipns/%s' % (IPFS_BASE,))
root_hash = ipfs_res['Path']
page_hash = self.api.add_pyobj({
'header': header_file.getvalue(),
'body': page_file.getvalue()
})
ipfs_res = self.api.object_patch_add_link(root_hash, uid, page_hash)
ipfs_hash = ipfs_res['Hash']
self.api.name_publish(ipfs_hash, key=IPFS_BASE_KEY)
I’m afraid this design is not going to work in a production environment, due to the contention when publishing the IPNS name (IPFS_BASE) with a shared key (IPFS_BASE_KEY) from many nodes at the same time. The performance is also not acceptable, it’s extremely slow even with a single flow in my experiment. Any suggestions to fix the contention and have a working design using IPFS directory?
For the project, I’m thinking about using IPFS to help people gain access to blocked internet content.
For a country which has national level Internet censorship, people usually use VPN to workaround the national Firewall to access blocked content or websites. We could use decentralized IPFS to circumvent censorship and make those content available to more people:
The wikipedia snapshot project is great and inspiring. However, many of the wikipedia content in snapshot could be long tail content. Now thinking about using IPFS as a web proxy and provide the Google or Wikipedia search pages to users. We could fetch up-to-date (or almost up-to-date) info those users are seeking, and hot/popular content is guaranteed cached in the swarm.
Since IPFS directory is too slow and not safe for the web cache store, I also looked into IPFS Key-value Stores.
Orbit-db and Tevere are amazing. However, both of them are maintaining a separate local storage other than IPFS swarm. Both require IPFS pubsub to trigger log sync among peers.
A quick skim of go-floodsub code told me that it doesn’t use IPFS store for its messaging and peer management. Guess it’s going to have a hard time to scale and probably won’t work well with nodes have intermittent availability most of time (for better censorship resistant.)
It’s surprising to me that it’s not easy to build a Key-value store upon IPFS. Nobody ever suggested to support “content-tagging or tag-addressed”, in addition to “content-addressed” on IPFS?
I spent some time thinking about content-tagging while reading about IPFS. I start feeling the intense stare of “content-addressed” from the title of Benet’s IPFS paper (DRAFT 3) 
As I understood, if we make content addressable/searchable by its tag, 1) many of the benefits provided by the system will be gone, e.g. Tamper resistance, Dedup; 2) the conflicts caused by a arbitrarily named tag seems not manageable, especially when the data propagated through the network, and how the nodes decide which value should be the latest one. etc. 3) other issues…
However, I have a strong feeling that tagging is very useful and could be a great feature for IPFS even though there is no guarantee to make tagged content consistent across the system.
Are there discussions about similar features in the past? Thanks!
You should use an existing web scraper such as httrack instead of rolling your own.
How are you going to share this key?
No, there are a couple of big issues with this.
ipfs dht findprovs QmOFFENDING_CONTENT to put someone in jail or worse.No, it’s not. Tor can evade GFW/Golden Shield with ease. What countries are you talking about where a VPN ban is enforced?
It’s trivial. You can use traffic patterns, or just block the DHT seed nodes. You could also create some custom implementation of the IPFS DHT that connects to as many nodes as possible and silently adds all the IPs it can find to the IP block list.
That just makes it fault-tolerant.
As I said earlier, they can just block all the IPFS nodes. The ones that really manage to piss them off might get a <img href="1.2.3.4/veryveryverylongstring.png"> tag inserted somewhere, where 1.2.3.4 is their IP. This has been done in the past, and overwhelming a residential connection isn’t very hard.
Why? With Wikipedia, you have the entire database. Build a simple reverse word index instead, this is good enough for most uses. Google doesn’t like people scraping their searches.
You can just use the IPFS directories for that. The key is the file name, the value is what’s referenced by the link in the DAG.
Because it’s a downright horrible idea. You can build a search engine on top, and there are decentralized implementations available in various forms. A simple reverse word index is good enough for most purposes and easy to decentralize.
No, the system will be gone. What you describe is susceptible to DoS attacks.
There was something called ipfs-search, but they shut down. Code available on GitHub.
I copied the IPFS_BASE_KEY into another node 
So far it’s the only way (I figured out) to let many nodes updating the same directory. You mentioned “You can just use the IPFS directories for that. The key is the file name, …” How exactly to do it to allow many nodes to update the same file/directory name?
Distributed web cache
In my experiment, IPFS web cache works fine for dynamic web sites. It cached all HTTP requests including those web services calls.
About Censorship
Searchable tag in IPFS
Now I realize that IPFS knows only the content hash and not content value. I see why “the system will be gone.” However, I don’t thinking searchable tag is a horrible idea in IPFS.
In order to support tag, we need to support tag queries as a new message type than normal content queries in IPFS. We need to do some changes in routing. To support “tag” as a new storage object, we need to extend IPFSLink and IPFSObject and the corresponding protobufs. It’s relatively easy.
Guess we can live with a tag have many different values in the system, it’s up to users to find good use cases for tag. Hopefully people will find creative ways to use tags for their Apps, e.g. tag prefixing, tag namespace. Not used tags will be eventually removed from the system. In order to help publishing tags in the network, we might need to augment routing key to be a pair of keys (hash(tag), hash(content)), or even better (hash(tag), hash(hash(tag), hash(content))) to find a way to help tracking hot tag/value pairs and keep them in the system.
Guess one important use case for searchable tag is a native key-value store in IPFS.