What’s the best way to implement (or simulate) a shared folder on an FTP server where many users can add files? We assume that the number of users is thousands and the number of new files per day is hundreds.
1 Like
Every user manages IPNS that points to “patch” directory tree. Name “patch” stems from nature of the nodes in it - to “patch” shared folder. It keeps db on metadata on files the user added, deleted or renamed.
Every user also runs daemon that listens on “shared folder channel”. There should be some crypto involved on adding new users, deleting users and authenticating users, probably by one or more moderators.
The daemon also coalesces metadata db from all participating users to build “single view” of the filesystem and announce filesystem changes on the channel.
Lots of work here, good luck!
1 Like
Thanks for quick reply!
Does this model imply that the shared folder has an owner (admin) that can make any changes in the folder? If so, is it possible to have such a shared folder without an owner? In the FTP world this would correspond to a public server that somehow magically doesn’t have an admin and everyone has the right to add files (at this point we assume that users are very thoughtful about using this shared space and don’t attempt to misuse it).
I would also like to build a file database, where there is a “public” folder which has admin access, as well as “user” folders where users of the system can store data that could then be shared with other users or also linked to the “public” folder.
I assume that the database part would be one “file” on IPFS?
Perhaps the mental model of a hierarchical system of folders is not right. I’ve found an article that explains a few interesting thoughts about tag-based file systems:
/designing-better-file-organization-around-tags-not-hierarchies
(search for it - not sure if this site allows links)
Perhaps the main idea is that instead of somehow trying to insert a new file into a list of files (a directory) which is immutable by definition (in IPFS), we put our new file into the huge heap of files and also create a “tag” which is another file with two fields:
- The tag name or description.
- The target file hash.
Then the system somehow magically discovers all these tag files and present everything as a hierarchical directory of files.
I think there is a simple way to implement such shared folders:
- The shared folder is a unique id, like a hash of something. It may be represented with a random IPFS file. We only need all the users to know this id. It’s important to note that this shared folder is not an IPFS directory object because IPFS directories are immutable.
- All the users of this shared folder store hashes of the files in that folder. These are regular IPFS hashes, so if someone needs to get the file contents, it’s enough to use
ipfs cat <that file hash>. The list of hashes is stored in a regular local file system folder. - When someone wants to add a file to the shared folder, three things need to be done:
-
ipfs add foo.txtto publishfoo.txtto IPFS - The IPFS hash of this file is added to the local folder.
- The IPFS hash of that file is sent to the peers.
-
- Peers get the hash of
foo.txtand also add it to their local folder.
As you see, we only need a way to message other peers. We can get their addresses from ipfs swarm peers. The messaging daemon will be running next to ipfs daemon and listen on some random port. The messages we need to send will have the following format:
-
[<folder-hash>, <file-hash>].
Nothing stops some users from doing backups of the shared folder: from time to time someone may get all the hashes from its local copy of the shared folder and publish it as a IPFS directory object, maybe linked to an IPFS address, like /ipns/shared-folder-221-backup.
Why not to use this /ipns/... link instead of those local folders? The IPNS link is pinned to a particular user who has the private key: that user would be a super admin. In the model where everyone has a local (maybe partial) copy of the shared folder there are no admins.
It’s not hard to see that a shared folder where everybody can only add files is a basic primitive that can be used to build a much more interesting system.
1 Like
There must be admin that can add or delete users. For truly “public” folder, you can make “admin script” that would simply add whoever requests to join the “folder”. Should someone misbehave, you can manually “remove” him.
This is actually misnomer, to “add” (or “delete”) a user would mean to send signed message on prearranged pubsub channel instructing daemons to pay (or stop paying) attention to “patch tree” of the particular user (or new updates to the same).
The convolution stems from IPFS limitations/features.
The “shared folder” model doesn’t need an admin, but it’s possible to create a shared folder in which there may be an admin or group of admins, which may be fixed or editable, etc.
In the simplest case, someone creates a shared folder, puts there a script, say rules.lua, which does a simple check: a file can be added to the folder only if it’s signed by this public key and the admin has the private key. Once this folder spreads to other users, they will have the admin to sign every file. They can of course modify their copy of rules.lua but that won’t help to convince other users to add random files as their rules.lua is unchanged.
The next level of complexity is a whitelist of users managed by the admin. In this case there are two shared folders: one for the content files, another for the list of users (their ipfs id values or something else). The rules script in the /users folder makes sure that only the admin can add users. The rules script in the /files folder checks that files there are signed by someone from the /users list.
We may want to introduce a blacklist to ban users by ipfs id, by IP address, for a certain period or indefinitely. We just add the 3rd folder: /banned. There’s a problem here, though: if someone gets added to this list accidentally, then there will be no way back - the system doesn’t allow to delete files because otherwise synchronizing the lists of files would be difficult. This may be addressed by saying that we don’t just add files, we rather add “operations” like “add file” or “remove file”. Then whoever wants to restore the latest list of files needs to get the entire chain of operations. But that idea still builds on top of the “shared folder, add only folder”. Or we can write rules.lua in such a way that would allow to just delete a local file as long as the request is signed or meets some criteria. If we want to not only remove a file, but prevent it from being synced back from other peers, we may add the file to another shared folder which lists all the banned files.
In fact, we can set up an arbitrarily complex access control scheme here since the rules for each folder is just a script that can do anything, including displaying captchas.
I think you misunderstand the way ipfs works. It does not have “public folder” that everybody could modify.
It boils down to this: Everybody has his own “folder” that only he can write into. He then announces address of the new version of the “folder” for everybody interested to take into account when simulating “public folder” that has files from many users.
I suggest to read more about IPFS before we can continue the thread productively.
I understand that IPFS deals with immutable files and an IPFS directory is just a file that contains links (hashes) to other IPFS files.
I’m suggesting to introduce a new entity - a shared folder that multiple people can add files to. This shared folder not to be confused with the IPFS directories, which are immutable.
A shared folder has an id. This id can correspond to an IPFS file, but it can be just a random GUID. As long as all the users know this id, we are good.
Every user has a local (and probably incomplete or out of sync) copy of the shared folder. Then we design a protocol that allows users to talk to each other and sync the contents of this folder. This is what I was trying to explain earlier.
In the IPFS system there is no file that would have the reference up-to-date copy of this shared folder. Instead, everybody keeps its own partial copy.
Afaik, this is the simplest way to implement a distributed version of a shared FTP folder.
Ah, sorry, I can discuss shared folder solution only using features already implemented (even with drawbacks) in IPFS…
Couldn’t an alternative system use IPLD or CRDT’s to implement the “folder” (list of hashes)? Similar to how OrbitDB works?
1 Like