I have N machines to which i want distribute some Big file. I have bootstrap node, which will act as initiator.
A have golang application on each physical machine (node), which is connected to some queue mechanism.
So when i want to distribute some file, i add it to IPFS on bootstrap node and send to all subscribers message over queue to notify about new file. All information to start download will be provided in that message. So after getting the message, each node can start downloading the file.
The thing is, that each node should provide progress of downloading the file in any possible format, percentage, of course, preferred.
Then i should have an option to stop downloading the file and delete it. Also i should have an option to download multiple files.
Here are the questions:
How do i add downloading background job. Right now running ipfs get <filehash> will run synchronous process;
How do i track progress of background downloading jobs?
If there is an ipfs node on each of the target machines, you can do an “ipfs pin add --progress ipfs-path” on each node. That is synchronous to get the progress, but if you wrap it is something else that reads the status and has it available, you can do what you want.
ipfs pin add can pull the file and make it local in the background
–progress will make it synchronous, but will provide status
aborting the ipfs pin add will stop the download (I believe, never tested this)
Once the pin is 100% complete, an ipfs get filehash will complete almost immediately since the file is 100% pinned on the local node. When you no longer need it, you can unpin it.
Oh, and there’s probably a way to do all of the distribution and status with ipfs-cluster, but I’m not familiar with that extension to ipfs.
The reason i need background job is that i don’t want to manually supervisor active downloads. If i have very poor connection, for example 50 kbits/s and connection device could be rebooted in any time, i want ipfs daemon to start download from the progress it’s been stopped.
Any torrent client can do that. I don’t believe such a powerful tool can’t
If there is no way to create background job, may be there is some api to at least check download status. For example when my node was restarted and i need to fetch a list of non finished downloads to resume them
I definitely think you need to look into ipfs-cluster. It will manage the pin for you and make sure the CID is fully distributed even in the event of lost/restored connections. And I think you can query the cluster to know if a given peer has a full copy or not.
The thing is, that i have don’t have centralized server to store files. I have closed local network, where one of equally set up devices will have source file, so it 100% p2p net.
I will look to ipfs cluster, but it already look as big wrapper for such task. Thank you!
One more question here. Could ipfs-cluster be used standalone or it’s more a wrapper over ipfs installation and they should go together?
I am wondering what is your communication layer to send/receive messages between your nodes?
When I need to spread background jobs, I am using a shared directory named with the ID of each node in my swarm. when a node wish to “talk” to another it writes in the “TASK” sub directory of destination node (command file can be ciphered and/or signed… It is using https://cesium.app as swarm control, wallets and messaging. We could talk about our howto ways
