Hello, Tl;dr: I would like to run an IPFS gateway which maps IPFS addresses to files which are hosted by a third party. I’m writing to ask for help and advice. I’m a research software engineer at the European Bioinformatics Institute (EMBL-EBI). One of the services offered at the EBI is the Europ…

If I understand correctly, it sounds like you might be describing the IPFS Filestore . Note that it’s currently an experimental feature. [image] rffrancon: However, it’s occurred to me that I can “serve” all of the data from the ENA using IPFS if I can redirect IPFS addresses to the correspondin…

Thank you for your reply @leerspace . I agree with you that IPFS must ingest the data before IPFS multihashes, and the mapping I mentioned above, can be produced. I’m not very concerned about the network IO and file system IO cost of generating this mapping once. But I think that the IO cost of runn…

[image] rffrancon: I’ll take a look at Filestore, I don’t know enough about it right now. I think I was ideally hoping that a request to my IPFS endpoint could be silently redirected to the correct ENA endpoint. What I had in mind with FIlestore is that if you can mount the dataset locally, t…

Sounds like what he would like is to have the IPFS equivalent of a torrent file with a webseed. So he publishes the hashes to the DHT but when people are retrieving the hashes they can fallback to http (or FTP??) to actually transfer the data. I don’t see that fitting in the protocol very well.

Hello @leerspace and @wscott , Thank you both for your replies. As promised, I’ve investigated IPFS Filestore and it’s worked out brilliantly! I can now mount an FTP directory and add directories/files with --nocopy to IPFS. The IPFS init directory takes up a trivial amount of memory on my laptop. …

Hi. I might have a similar usecase to this. Let me explain the scenario a bit. I work with various organizations who have very large sets of data. Normal set might be something like 50TB or so but some could be more or less. The data are large binary blobs by and large but some of them could be t…

[image] timuckun: So basically a bittorrent type of system but I would like to be able to set up different “containers” or fenced in areas where access to data is more restricted. If you’re trying to restrict access to data within IPFS, I don’t think that there are currently good ways to do t…

This subject can be quite interesting for all the biochem industries that do heavy research. There could be even some specialized implementations and functionalities to help their needs. I do happen to have a background in molecular biology and I know people related to biomedical research who are …

Addressing petabytes of genetic data with IPFS

Ecosystem and Usage

eocarragain January 8, 2018, 9:26pm 9

@rffrancon you may be interested in work that @flyingzumwalt is doing on motivating factors for using IPFS with large datasets: What motivates people to use IPFS for large volumes of data?

Topic		Replies	Views
What motivates people to use IPFS for large volumes of data? Ecosystem and Usage	32	9069	October 26, 2018
Content-Location Address Bidirectional Mapping Ecosystem and Usage	9	1505	July 31, 2018
Government Environmental Data on the IPFS Help	2	593	March 29, 2018
Feasibility for Self-Hosting Scientific Datasets? Help go-ipfs , kubo	10	395	December 21, 2023
IPFS for community-led research Help research	6	2156	May 17, 2017

Addressing petabytes of genetic data with IPFS

Related topics