IPFS URL Shortener

Hi there!
I am working on an IPFS based CI tool for solidity smart contracts on Ethereum Blockchain (Ethereum/Solidity Smart Contract CI Toolchain on IPFS! (WIP))

The core component of sharing in this architecture becomes tedious if we donā€™t shorten the looong IPFS URL for each folder under an ipns. :sweat_smile:

Is anyone working on the URL Shortener just for the IPFS links? Let me know in the comments plsā€¦

PS: Happy to contribute(PRs) if any such projects in currentā€¦

1 Like

Thatā€™s actually a perfect use-case for an eth app. Unfortunately, the cost of shortening urls would be prohibitively expensive (and bla bla comparability with existing browsersā€¦). Unfortunately, I donā€™t know of any existing projects.

1 Like

I donā€™t understand why would anyone want that. Creating an URL-shortener for IPFS content is throwing away all the protocol.

1 Like

One fruitful line of investigation might be in how you use the multihash. Trying a hash with less bits will obviously tradeoff shortness for lower security against collision attacks, but this sounds like a tradeoff you may be willing to make. You might also be able to find a more compact representation than Base64. As a trivial example, URL path strings theoretically allow enough characters to support up to Base69 if your HTTP service lets you directly parse the normally reserved character set (domain names only support up to Base36 cleanly due to case insensitivity and the requirement that ā€˜-ā€™ are each surrounded by other legal characters).

I wouldnā€™t want to fill up the finite multihash type table with any domain-specific hashes, but some hash types may (at least theoretically) have different collision characteristics with specific serialized data structures, which could in turn be exploited to only ever use them to hash a limited subset of file types, which in turn further helps limit the risk of global namespace collisions. For example, I imagine a lot of Ethereum smart contract linked files are rather text-heavy in order to retain some semblance of lawyer-readability, so if you could find a low bit total hashing method that is particularly collision resistant only when applied to UTF-8 documents, you could use that multihash on those files independently of the default or more commonly used hash types as a matter of course, both as a shortener and a cross-hash collision avoidance method.

Definitely food for thought for any format-specific crypto hash researchers.

1 Like

I understand your premiseā€¦
But we need the shorting of URLs at some time from the usability perspective, right?

Of course not. IPFS is a protocol, the goal is for everyone to be running their own IPFS nodes. Not to create infrastructure around the ipfs.io gateway (if thatā€™s what you thinking about).

Anyway, I fail to see why from the userā€™s perspective a big identifier is worse than a short identifier.

Even if it was much better (and possible) to have super small identifiers, would you really trade all the decentralization and immutability for that? If yes, why canā€™t you use S3 hosting with bit.ly, then?

3 Likes

I have made attempts to actually shorten the urlā€™s and run it by my own url shortening serviceā€¦ It worked!
But the problem now lies in domain hosting. I wish I could get some help directly from Protocol Labs.(cc @jbenet & @whyrusleeping)

Maybe host this on ipfs.to or some sub-domain of ipfs.io (maybe to.ipfs.io ?)
If someone is serious enough to make a decision, I can put some extra time and host a Redis Cache for a production grade server. :+1:

While I think a URL shortener is an interesting project and an interesting application of IPFS (e.g., by using dnslink + an IPFS directory of all registered paths), I donā€™t think we really want a ā€œblessedā€ centralized URL shortening service.

Really, weā€™d rather not even have to run the gateways (weā€™ve been toying with ways to make them more decentralized). We only do that as a stop-gap. However, a URL shortener like this seems a bit more permanent.

If we ever did introduce something like this as a blessed service, weā€™d definitely want to use some form of decentralized consensus algorithm.

We also need to consider the concrete problem weā€™re trying to solve as there are other solutions. So, why do we want something like this? Whatā€™s wrong with the current system?

IPFS urls look long and unwieldy

  • Solution: Donā€™t show them. Put them in <a> tags, etc. This is my preferred solution. In my perfect world, users would treat IPFS resources as tangible objects. Unfortunately, this would require better browsers and operating systems.
  • Solution: Use an alternative, shorter encoding. E.g., a zero-width encoding that can be copied. If users donā€™t need to read the URLs, thereā€™s no reason show them.

IPFS urls canā€™t be copied from paper

Solution: Use a QR code.

IPFS urls canā€™t be spoken/typed

Story: Professor writes a URL on the board.

Ideally, weā€™d live in a world with augmented reality and, again, we could just treat the URL as a tactile object. Unfortunately, we donā€™t.

This is really the only case where I think we actually need some form of shortener. Well, there are some other fancy solutions with IPNS and QR codes but they have some sever usability drawbacks.

5 Likes

To add some other considerations onto URL shorteners:

  1. They hurt performance by requiring at least one redirect
  2. If (when?) they stop working all of their links are broken
  • Iā€™m not sure how inevitable this would be if this was done in a decentralized way. Doing it through a sub-domain on ipfs.io doesnā€™t sound like such a way, though.
  • Archive Team has spent a lot of time trying to archive shortened URLs for URL shortening services that have come and gone, leaving a trail of broken links in their path.
  1. With IPFS, immutability is lost (as @fiatjaf already mentioned). A link that used to redirect to one hash could theoretically be made to redirect to another.

Unless ā€œthe core component of sharing in this architectureā€ involves manually typing URLs, it would be nice to not increase usersā€™ exposure to the weaknesses of URL shorteners.

2 Likes

None of the currently widely deployed URL-shortening services solve any of the problems youā€™ve mentioned. Who in his right mind (unless itā€™s the only solution and the last) copy shortened URLs from bit.ly or goo.gl to and from paper? Or speak them? Thatā€™s impossible. bit.ly/z30YI8r? no way.

The solution was always to have people link to things. I donā€™t believe IPFS will ever replace the entire web with its hashes, but if it is, it will only be when URLs become, as you say, tangible objects. Until them, we still have domain names and mutable websites that can serve as passage points to IPFS URIs.

If you want to speak an URL today youā€™ll probably have to host it some easily-spoken domain + path and tell people to visit it. You can still do that. Each person may do that in their own way, like theyā€™re doing today.

(Actually, I thought a lot in the last days on the problem of spokenable URLs and in the end produced https://filemap.xyz/, which is a mutable site with a mutable database, but that links to immutable IPFS objects.)

1 Like
  1. People do write down urls like bit.ly/z30YI8r. They donā€™t speak them, but theyā€™ll write them on a whiteboard or print them.
  2. For speakable URLs, there was shoutkey (but that appears to have gone away).

The only other reason to use a URL shortener is to put URLs in tweets.

2 Likes

I enjoy the discussion on the pros and cons of URL shorteners in general, and agree that a sort of ā€œblessed forever-shortenerā€ is impractical to the point of impossibility, but Iā€™m also afraid we may be speaking at cross-purposes here.

Framing the problem in the context of the original question ā€“ use of shortened references in a particular set of Solidity contracts ā€“ I would note that the short-form redirects only need to survive a little longer than the contracts that are referencing them, in the same way that QR codes only need to be valid a little longer than the set of visual media that contain them. Yes I know people want to write contracts that will last forever+1 years, but we do have a thing called ā€œstatute of limitationsā€ in legal frameworks for good reasons. Until Aubrey De Gray formulates the exact correct monkey-paw wish, I donā€™t see permanence as a worthwhile concern in this specific context.

This set-limited impermanence then points to a form of (distributed consensus based) reference counter solution, such that when any short-linkā€™s counter falls to 0, it can safely be recycled. At that point, for safety you could even patch the contracts to utilize the full multihash or CID instead of the shortened version, just in case anyone wants to clone them in the future. At that point the cloners compile their own new shortened links, even preferentially recycling the old ones if they catch them in time. :wink:

1 Like

I still donā€™t understand. If youā€™re going to reference a mutable URL from the Solidity contract, why do you need IPFS?

This is certainly the beginning of a new thought for referring to stuffs on IPFS. Thanks!

Makes sense. But I think the URL Shortening utility is beyond twitter refs :slight_smile:

So hereā€™s the thing - IPFS is permanent web. It doesnā€™t necessarily limit the scope of the data to remain immutable as a stone.

A new line into the document creates a new IPFS hash. (At this point I request you to check the solution architecture for my problem here: Ethereum/Solidity Smart Contract CI Toolchain on IPFS! (WIP))
Atleast this is how I saw its working.

The reason behind choosing IPFS is because we are essentially looking for a distributed, open network.

1 Like

I donā€™t understand your architecture or what exactly IPFS does there, the linked example is not available, apparently.

But hereā€™s the point, again: IPFS is a distributed, open network. A URL shortener is not and will not be, it will be a single server with its own arbitrary rules. As such, the URL shortener can dissappear at any moment or change the shortened URLs to anything it wants. It doesnā€™t change anything if that URL shortener is in an ipfs.io domain.

If you really want an URL shortener, you can create one, but it will not be different from bit.ly or whatever. It is not possible to include an URL shortener in the IPFS protocol.

You can host content on IPFS and then use your own URL shortener (or bit.ly or whatever) to shorten the IPFS URLs, but if you need a distributed, open network, youā€™re not going to have it anymore, as the URL shorteners will be not distributed nor open.

2 Likes

Noted. Iā€™m already looking at the alternative measuresā€¦ But thanks a lot for taking time and explaining! :clap:

IPNS not set to persist the hash object for more than 24 hrs (for now)