IPFS for AI model sharing

I’ve recently been pondering the use of IPFS for transferring large AI models and I came across this very interesting article https://a16z.com/2020/02/16/the-new-business-of-ai-and-how-its-different-from-traditional-software/

The specific section that caught my eye was:

  • We’ve had AI companies tell us that cloud operations can be more complex and costly than traditional approaches, particularly because there aren’t good tools to scale AI models globally. As a result, some AI companies have to routinely transfer trained models across cloud regions – racking up big ingress and egress costs – to improve reliability, latency, and compliance.

It sounds like a problem that’s just asking for IPFS to solve. Any thoughts?

2 Likes

I’m here because I want to find a way to use/share data from the SatNOGS ( https://satnogs.org/ ) network with an AI. I’ve scraped about 1.2 terabytes of a 20+ terabyte archive, which grows about ~1TB/month as far as I can tell. There are other groups that are also scraping the data for their uses.

I’ve looked into setting up p2p proxy for SatNOGS, but I don’t quite have that working yet. I may dump the files here, but I’m not sure the best way to go about it yet, as it seems everytime I add a file, the DNS record needs updating (?):

https://ipfs.spacecruft.org/ipns/spacecruft.org/

I’m proxying the site here too, for example:

https://ipfs.spacecruft.org:4443/observations/1714662/

Here’s the AI I’m working on:
https://wut.spacecruft.org/

https://wut-beta.spacecruft.org/

Source code:

Happy hacking,

-Jeff

1 Like

Take a look at this prototype: https://www.youtube.com/watch?v=aO0CuFyGduc

1 Like

Brilliant. That’s it! Have you thought about using pub/sub to subscribe to the latest model?

1 Like

A similar issue, one of sharing large indexes, is faced by YaCy. YaCy is a Free Software project which aims to provide distributed, user trained, open-source search engine software. If an individual “trains” their instance by web-crawling a large number of sites, the cache can become large, many Gigabytes, and to be of use to others, it needs to be shared peer-to-peer.

IPFS people are cordially invited to take a look at YaCy here:

https://YaCy.net

It is cross-plaform, easy to install, and works right away, though a little configuration is needed to help make contribute resources to the project.

Considering how censorship of searches is happening, I hope that a few more people take a look at the project.

1 Like

I can’t wait to check it out. It looks interesting but one silly question first. How do you pronounce it? YA-See? YAY-See? YAK-y? ya-s-EYE? :slight_smile:

1 Like