# Git on IPFS - Links and References

**URL:** <https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730>\
**Category:** Ecosystem and Usage\
**Tags:** interoperability, use-cases-and-apps\
**Created:** [July 7, 2017, 1:29am UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730 "2017-07-07T01:29:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![flyingzumwalt](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/flyingzumwalt/32/12_2.png) [@flyingzumwalt](https://discuss.ipfs.tech/u/flyingzumwalt)\
**Post date:** [July 7, 2017, 1:29am UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/1 "2017-07-07T01:29:38Z")

</div>

The topic of putting git commits onto IPFS comes up often. Let’s use this thread to help people find related projects and tools.

For example:

- [GitHub - ipfs-shipyard/git-remote-ipld](https://github.com/magik6k/git-remote-ipld)
- [GitHub - larsks/git-remote-ipfs](https://github.com/larsks/git-remote-ipfs) (dormant since 2015 but still cool)

The exciting thing about [git-remote-ipld](https://github.com/magik6k/git-remote-ipld) is that it uses [IPLD](https://ipld.io), which is a data model (and library) that lets you parse hash-linked data structures like git repositories, traverse them, and address items within them _using their native data format_. It lets you treat git repositories as just another data format on the decentralized web. In the same way that a modern OS lets you read the many different data formats for Images, Movies, Spreadsheets, etc, IPLD makes it possible to do the same thing with git repositories, blockchain ledgers (bitcoin, ethereum, zcash, etc), files & directories on IPFS, dat repositories, etc.

---

<div class="post-metadata">

**Author:** ![VictorBjelkholm](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/victorbjelkholm/32/10_2.png) [@VictorBjelkholm](https://discuss.ipfs.tech/u/VictorBjelkholm)\
**Post date:** [July 7, 2017, 9:38am UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/2 "2017-07-07T09:38:23Z")

</div>

Some more projects for reference:

- [https://github.com/cryptix/git-remote-ipfs](https://github.com/cryptix/git-remote-ipfs) - “A ‘native’ git protocol helper to push and pull git repos from ipfs.”
- [https://github.com/whyrusleeping/git-ipfs-rehost](https://github.com/whyrusleeping/git-ipfs-rehost) - “A script to rehost your git repos in ipfs”
- [https://github.com/airalab/github-ipfs](https://github.com/airalab/github-ipfs) - " GitHub -\> IPFS mirroring"

---

<div class="post-metadata">

**Author:** ![Kubuxu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/kubuxu/32/17_2.png) [@Kubuxu](https://discuss.ipfs.tech/u/Kubuxu)\
**Post date:** [July 7, 2017, 2:42pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/3 "2017-07-07T14:42:58Z")

</div>

> [@flyingzumwalt](#):
>
> [GitHub - ipfs-shipyard/git-remote-ipld](https://github.com/magik6k/git-remote-ipld)

should replace most of the other git helpers but one of its problems is that if git repo contains (or contained in history) files bigger than ~2MiB it won’t be able to transfer it (IPFS normally shards files but we can’t do that with Git as it would change the hash). This is security limitation of bitswap we are thinking how to solve.

---

<div class="post-metadata">

**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\
**Post date:** [July 7, 2017, 6:24pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/4 "2017-07-07T18:24:55Z")

</div>

For anyone wondering, the security problem is that bitswap needs to download entire blocks (in this case, git objects) before it can verify their hashes. Unfortunately, git doesn’t break large files into multiple smaller objects. One of the goals here is to be compatible with git hash-for-hash (commit-id for commit-id) so we can’t just change the underlying git object format to keep them small.

# Possible Solutions

In case anyone wants to help solve this problem, here are a few possible solutions to get the discussions started:

## Fail

Given that one shouldn’t be storing large objects in git, we could just fail to checkout such repositories. Unfortunately, GitHub has set it’s max-object size to 100MiB so, if we want to support all GitHub repos, we’d have to support downloading 100MiB blocks 🙁.

## Trusted peers

An alternative is to only download large blocks from trusted peers (e.g., github). This is, unfortunately, not very decentralized.

## Take a vote

One possible (but not great) solution is to trust that N independently selected peers won’t collude. If we do, we could have peers split large blocks up into a merkeltree of smaller blocks and then poll a randomly selected set of N peers for the hash of the root of the merkeltree that corresponds to the block we want. If our randomly selected peers agree, we would then download the entire merkeltree, put it back together, and verify that the reconstructed block matches its hash.

While this would make fetching large blocks slow, this shouldn’t happen _that_ often.

## Extra metadata

We could also require that repositories with large objects also store some extra metadata that allows us to validate these large objects piecewise. That is, we can take the metadata from the “take a vote” solution and check it into the repository. Then, when downloading the repository, we could download the small blocks first, pull out this (small) metadata, and use it to validate the large blocks piece-wise.

IMO, this would be more trouble than it’s worth.

## Crypto Magic

Ideally, we’d be able to progressively validate SHA1 hashes of large objects as we download them by exploiting the fact that SHA1 is a streaming hash function (i.e., somehow verify it in reverse). Unfortunately, I’m pretty sure there’s no way to do this securely.

---

<div class="post-metadata">

**Author:** ![whyrusleeping](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/whyrusleeping/32/34_2.png) [@whyrusleeping](https://discuss.ipfs.tech/u/whyrusleeping)\
**Post date:** [July 9, 2017, 1:56pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/5 "2017-07-09T13:56:22Z")

</div>

Looks like we need to find a crypto magician

---

<div class="post-metadata">

**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\
**Post date:** [July 9, 2017, 6:07pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/6 "2017-07-09T18:07:35Z")

</div>

Actually, this may be doable with zksnarks if we can use them to prove SHA1 hashes “valid”. Of course, I know next to nothing about zksnarks so this may not be possible…

Backing up a bit, the naive way to do this would be to send chunks in reverse order as follows:

```auto
chunks := split(block)
for i = len(chunks)-1; i--; i > 0 {
    send(SHA1(concat(chunks[:i])), chunks[i])
}

```

Given that SHA1 is a streaming hash algorithm, the receiver can validate each chunk as follows:

```auto
hash := hashOfEntireBlock
for {
    hashOfPrefix, chunk := receive()
    assert(Sha1Extend(hashOfPrefix, chunk) == hash)
    hash = hashOfPrefix
}
assert(hash == sha1StartingHash) // all SHA1 hashes start at the same value.

```

Unfortunately, I’m almost positive that it’s entire possible for an attacker cook up chunks for some `targetHash` as follows:

```auto
hash := targetHash
for {
    hash, chunk := solveForHashChunkPair(hash)
    send(hash, chunk)
}

```

Basically, as long as the attacker never tries to “finish” sending the file, they can keep sending us hunks.

However, we may be able to use zksnarks to prove that a sender _knows_ how to finish sending a file (without actually sending it). To do this, we’d need to be able to use use zksnarks to prove that, in order to produce a proof P (zksnark) for some SHA1 hash H, the author of P must have known some D such that `SHA1(D) == H`. This would allow a sender to prove to the receiver that the sender knows of a finite set of steps to finish sending the file.

Note: These proofs only need to be produced the first time an object is added to the network, not every time a peer sends an object to another peer. However, they would have to be produced at least once so adding large objects could take over an hour. The moral of the story is: don’t store 100MiB files in git!

---

<div class="post-metadata">

**Author:** ![Kubuxu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/kubuxu/32/17_2.png) [@Kubuxu](https://discuss.ipfs.tech/u/Kubuxu)\
**Post date:** [July 9, 2017, 7:06pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/7 "2017-07-09T19:06:32Z")

</div>

I was thinking about something similar as you point out with SHA extension but I think it would be vulnerable to an attack.

The internal state is initially defined (in the spec) and after hashing the state is the output. Preimage attack tries to take the output (which is the final, after hashing, state) and find inputs that would turn it back into initial one from the spec.

In this case we have no idea about the “along the way” states. Which also means that the attacker has additional 256 bits of freedom (the “initial” state) so I would expect the attack to be much easier than preimage.

* * *

> However, we may be able to use zksnarks to prove that a sender knows how to finish sending a file.

Having a proof that the sender has knows the steps doesn’t mean that he is willing to share the steps with us. Malicious actor might send us the proof but not the steps required later.

---

<div class="post-metadata">

**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\
**Post date:** [July 10, 2017, 5:02pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/8 "2017-07-10T17:02:14Z")

</div>

> [@Kubuxu](#):
>
> Having a proof that the sender has knows the steps doesn’t mean that he is willing to share the steps with us. Malicious actor might send us the proof but not the steps required later.

Just knowing that _someone_ (anyone) knows the steps should actually be sufficient (i.e., no worse than the current system). It doesn’t prove that the sender is willing to complete the process but _does_ allow one to verify chunks as they are received. If the sender stops sending us blocks, we can just complete the process where we left off with another peer.

Basically, one would:

1. Receive a chunk.
2. Receive a proof that _someone_ knows the set of remaining chunks necessary to make the hash work out. Note: By "proof that someone knows the set of remaining chunks, I mean that someone knows some `remaining_chunks` such that `SHA1(remaining_chunks || this_chunk || previously_received_chunks) == targetHash`.
3. Verify the proof. If it fails, ban the peer.
4. Accept the chunk and go to step 1.

This should allow us to verify chunks as we receive them.

---

<div class="post-metadata">

**Author:** ![Kubuxu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/kubuxu/32/17_2.png) [@Kubuxu](https://discuss.ipfs.tech/u/Kubuxu)\
**Post date:** [July 10, 2017, 5:49pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/9 "2017-07-10T17:49:53Z")

</div>

Hmm, I didn’t think about it as a continuation proof. This in theory could work but I think we both agree that using zkSNARKs or zkSTARKs right now is infeasible.

---

<div class="post-metadata">

**Author:** ![es\_00788224](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@es\_00788224](https://discuss.ipfs.tech/u/es_00788224)\
**Post date:** [July 12, 2017, 3:44pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/10 "2017-07-12T15:44:27Z")

</div>

You’ll need someone to mirror it to IPFS anyway or if the original upload point is IPFS, to have commit access. Can’t you just have them sign mappings of IPFS hashes \<–\> git blob hash with their key? You have to trust them anyway.  
If that’s not decentralized enough, you can use something like Freenet’s [WebOfTrust](https://d6.gnutella2.info/freenet//USK@xedmmitRTj9-PXJxoPbD7RY1gf9pKi0OcsRmjNPPIU4,AzFWTYV~9-I~eXis14tIkJ4XkF17gIgZrB294LjFXjc,AQACAAE/fmsguide/6/trustfaq.html). This would be useful for many other applications as well.  
Here’s how it could be done for this particular use case:

- Start with git blob hash _B_
- Get the IPFS hash mapping from the most trusted node
- If it matches, assign positive trust to all nodes which asserted this was a valid mapping
- If it doesn’t match, assign negative trust to all nodes which asserted this was a valid mapping, return to step 2.
- If you’ve checked all hash mapping assertions from all nodes and none of them are valid, only then you can download it from a centralized server (and of course upload it to IPFS)

Since nobody would waste trusted keys on causing a minor slowdown for someone, you’ll in practice just have to perform steps 1 and 2.

Thoughts?

---

<div class="post-metadata">

**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\
**Post date:** [July 13, 2017, 8:22pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/11 "2017-07-13T20:22:24Z")

</div>

> [@Kubuxu](#):
>
> This in theory could work but I think we both agree that using zkSNARKs or zkSTARKs right now is infeasible.

Totally. Personally, I’d just leave things as-is indefinitely. You shouldn’t be storing large files in GIT.

> [@es\_00788224](#):
>
> You’ll need someone to mirror it to IPFS anyway or if the original upload point is IPFS, to have commit access.

The idea is to make possible to use _existing_ GIT objects as IPLD objects (note, this is IPLD, not IPFS) without converting anything. This way, one can, e.g., call `git show`, send the commit hash to a friend, and let them fetch their git repo using the IPFS daemon.

---

<div class="post-metadata">

**Author:** ![es\_00788224](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@es\_00788224](https://discuss.ipfs.tech/u/es_00788224)\
**Post date:** [July 14, 2017, 12:55pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/12 "2017-07-14T12:55:05Z")

</div>

Sure. But someone needs to upload it to IPFS anyway. How are they going to fetch the git repo with their IPFS daemon if it doesn’t exist on IPFS? Clearly, someone has uploaded it to IPFS some time, and then they could have publish the mapping between git SHA-1 and IPFS SHA-2.

---

<div class="post-metadata">

**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\
**Post date:** [July 14, 2017, 4:58pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/13 "2017-07-14T16:58:52Z")

</div>

Not necessarily. For example, using the up-and-coming plugin interface, I’d like to add a GitHub plugin that fetches unknown git objects directly from GitHub.

---

<div class="post-metadata">

**Author:** ![es\_00788224](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@es\_00788224](https://discuss.ipfs.tech/u/es_00788224)\
**Post date:** [July 15, 2017, 9:05pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/14 "2017-07-15T21:05:59Z")

</div>

If you’re fetching them directly from GitHub, where does IPFS come into play? Does it upload them to IPFS too? In that case, you’ll hash them with the IPFS content hashing and then cryptographically sign the mapping between git and IPFS hash.

---

<div class="post-metadata">

**Author:** ![stebalien](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/stebalien/32/156_2.png) [@stebalien](https://discuss.ipfs.tech/u/stebalien)\
**Post date:** [July 15, 2017, 10:02pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/15 "2017-07-15T22:02:30Z")

</div>

You’re right, you could have a signed mapping from git hashes to IPLD hashes (which you can fully validate once you download the appropriate blocks). When fetching a _large_ IPLD object, the peer could present a signed mapping to a chunked version. However, is there really much benefit to doing this over trusted peers solution above? Storing each mapping along with a bunch of signatures could get very expensive.

---

<div class="post-metadata">

**Author:** ![es\_00788224](https://avatars.discourse-cdn.com/v4/letter/e/e5b9ba/32.png) [@es\_00788224](https://discuss.ipfs.tech/u/es_00788224)\
**Post date:** [July 16, 2017, 3:38pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/16 "2017-07-16T15:38:53Z")

</div>

Much more decentralized. And for some cases, you can’t always use the trusted peers approach. For example, say you want to do this for BitTorrent as well: map info\_hash:start\_offset:end\_offset to IPFS CID. Where’s your trusted peer now?

> Storing each mapping along with a bunch of signatures could get very expensive.

How do you mean? A signature is 256 bytes large for a 2048 bit key. You just need 1 signature for each node, each node could sign a merkle root of all their asserted mappings. This is no huge amount of metadata in any case, even if they sign each mapping individually.

---

<div class="post-metadata">

**Author:** ![balupton](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/balupton/32/211_2.png) [@balupton](https://discuss.ipfs.tech/u/balupton)\
**Post date:** [April 10, 2018, 10:07pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/17 "2018-04-10T22:07:33Z")

</div>

Keybase’s Git solution is based on Keybase FS which is based on IPFS:

[https://keybase.io/blog/encrypted-git-for-everyone](https://keybase.io/blog/encrypted-git-for-everyone)

---

<div class="post-metadata">

**Author:** ![flyingzumwalt](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/flyingzumwalt/32/12_2.png) [@flyingzumwalt](https://discuss.ipfs.tech/u/flyingzumwalt)\
**Post date:** [April 10, 2018, 10:22pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/18 "2018-04-10T22:22:59Z")

</div>

Thanks for the link @balupton

Correction: Keybase FS (KBFS) is not based on IPFS but KBFS and IPFS could potentially be used together. I really like the idea of using IPFS as a storage layer for KBFS and perennially look into how hard it would be to do that integration, but thus far nobody has done that work.

---

<div class="post-metadata">

**Author:** ![balupton](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/balupton/32/211_2.png) [@balupton](https://discuss.ipfs.tech/u/balupton)\
**Post date:** [April 10, 2018, 10:40pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/19 "2018-04-10T22:40:10Z")

</div>

Thanks. Looking into it seems you are right:

> <https://github.com/keybase/kbfs/issues/943>

And that this post is wrong:

> **[Keybase: Encrypted Filesystem on IPFS](https://maidsafeblog.wordpress.com/2016/10/20/keybase-encrypted-filesystem-on-ipfs/)**
>
> SafeNet is (ought to be, to be precise) a distributed replicated storage network with a client-side filesystem overlay called NFS (not that NFS). Each SAFE Network client can have a cached copy of …

---

<div class="post-metadata">

**Author:** ![cogwire](https://avatars.discourse-cdn.com/v4/letter/c/59ef9b/32.png) [@cogwire](https://discuss.ipfs.tech/u/cogwire)\
**Post date:** [June 4, 2018, 10:39pm UTC](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730/20 "2018-06-04T22:39:32Z")

</div>

Hi, just wanted to revive this thread now that Github has been officially overtaken. Not to make it political or anything, but something as important as our social coding infrastructure should not be owned and operated by a commercial entity.

Looked into Git on Keybase and I have two first impressions:

1. the backing storage for Git on Keybase is owned and operated by Keybase the company and might disappear if Keybase were ever to run out of money, or sell. Is this even remotely true?  
[https://news.ycombinator.com/item?id=15401211](https://news.ycombinator.com/item?id=15401211)

2. Git on Keybase really only works for private repos.

3. Is it possible to extend the Keybase experience? To make a Github killer, we would need more functionality related to support software development.

Really interested to hear if anyone has made any progress on this front. Anyone else interested in starting a self-funding Distributed Autonomous Community as an alternative to Github, please send me a message.

-cogwire

[Next page](https://discuss.ipfs.tech/t/git-on-ipfs-links-and-references/730.md?page=2)