Hello! Last week on the Costa Rican hackerspace we started a project to serve the ubuntu archive on top of IPFS. The idea is that it should be faster to download deb packages from an IPFS node that is closer than the closest mirror of the archive. For example, in Costa Rica there is only one mirro…

Well, not GPG. We sign the packages when we are uploading them to the archive. Then the main archive generates hashes, which is what you will see here and in some other dirs: http://archive.ubuntu.com/ubuntu/dists/xenial/ The mirrors of the archive just copy what the main publishes. apt in the cli…

A quick update here. After leaving my machine synchronizing the mirror during all my holidays, I finally have all the files locally. I’m adding them to IPFS, which brought two unexpected problems. First, the recursive add will take a little more than 12 hours :frowning: The Ubuntu mirrors are supp…

[image] elopio: Second, ipfs stores local blocks for each file, so that duplicates the amount of space required for the mirror. It might be worth looking into using IPFS’ filestore functionality to prevent space requirements from being doubled.

hey, that sounds great! That way we also help testing that experimental feature. /me gives it a try.

As said can use the filestore. If somebody ends up here, here is the shortcut: ipfs config --json Experimental.FilestoreEnabled true ipfs add --nocopy $LOVE It should work

No luck with the --no-copy, I got: panic: interface conversion: interface {} is cmdkit.Error, not *coreunix.AddedObject goroutine 37 [running]: github.com/ipfs/go-ipfs/core/commands.glob..func7.1(0xc4202ec060) /cwd/parts/ipfs/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:390 +0x9fc c…

Do you know if there’s a way to resume the ipfs add --recursive? When it fails and I have to re-run it, it seems to start from scratch.

I don’t think it stores the progress anywhere to be able to pick up where it left off. Subsequent runs of ipfs add should be significantly faster (probably not much of a difference with --nocopy though) since the blocks up to the point where it stopped already exist in the datastore and don’t need t…

In terms of add time, a large portion of that is: Our current datastore is slow. If you want something faster, try badgerdb. Unfortunately, that datastore backend is experimental for a reason. One of our huge bottlenecks is telling everyone on the network that you have a file. We’re working on mak…

Thanks for the suggestions @stebalien . We are trying now with --local, and then we can experiment with badgerdb. It says it will be done in 3.5 hours. Edit: This progress bar is the worst liar, it has been running for a long time and it says now 58.85% 4h44m58s :sleeping:

Ubuntu archive on top of IPFS

Ecosystem and Usage

DaniellMesquita December 5, 2019, 10:18pm 39

I have a improved concept for deduplication: (draft) Common Bytes - standard for data deduplication

Topic		Replies	Views
C as a priority for Linux systems Ecosystem and Usage repo , debian	14	132	December 31, 2024
Work-plans for kubo, helia, & other Shipyard IPFS projects in 2025 kubo , helia	12	585	December 12, 2024
Questions after first learning about IPFS Help	12	1722	May 23, 2017
Newbie questions	35	889	June 10, 2022
What motivates people to use IPFS for large volumes of data? Ecosystem and Usage	32	9059	October 26, 2018

Ubuntu archive on top of IPFS

Related topics