Some bash functions to make life easier

Help
1

I badly needed some helper scripts for my ipfs node to manage the MFS, so having written them I thought I’d post them here. I’ll update this post as I go with new functions or bug fixes

#! /bin/bash
function ipfs_add_folder() {
    local file
    local fname
    local folder
    local file_hash
    local existing
    local isExisting=false

    file="${1#'./'}"
    folder="${2}"
    fname=$(basename "${file}")

    while read -r existing
    do
        if [[ "${fname}" = "${existing}" ]]; then
            isExisting=true
        fi
    done < <(ipfs files ls "/${folder}")

    if [[ ${isExisting} = true ]]; then
        echo "Skiping existing file ${fname}"
        return
    fi

    echo "Adding ${fname}"

    file_hash=$(ipfs add --pin=false -Q "${file}")

    ipfs files cp "/ipfs/${file_hash}" "/${folder}/${fname}"

}

function ipfs_find_add_folder() {
    local patern
    local folder

    patern="${1}"
    folder="${2}"

    find . -type f -name "${patern}" -exec bash -c "ipfs_add_folder '{}' '${folder}'" \;

}
#### ipfs dht provide -r (Appears) to hang when trying to process
#### very large folder trees,
#### so here's a simple way to provide each file separately with a 
#### short timeout to prevent getting stuck
function ipfs_provide_folder() {
    local folder
    local fname
    local listing
    local file_hash
    local size
    local len

    folder="${1}"
    echo "Recursing into ${folder}"

    while read -r -a listing
    do
        size=${listing[-1]}
        len=$((${#listing[@]} -2 ))
        fname=$(basename "${listing[*]:0:$len}")
        if [[ "${size}" -eq 0 ]]; then
           ipfs_provide_folder "${folder}/${fname}"
        else
            file_hash=${listing[-2]}
            echo "Providing ${fname} - ${file_hash}"
            ipfs dht provide --timeout 900s "${file_hash}"
        fi
    done < <(ipfs files ls -l "/${folder}")
}
2 Likes
2

Please write to understand.

3

The problem with adding a file in this way, is that you’re checking the MFS for the filename, not for the hash… different files can have the same basename, but never the same multihash. So your script might return that a file already exists in your MFS, but it could actually be a totally different file from the one you’re trying to add. So if you want to add a file to the IPFS, then you need to do it, no matter if a file with the same basename is already in the MFS part of your node. If you add a file to your node that’s already been added before, then (because of deduplication) your node will not grow in size. After adding, you’ll get the multihash, and then you can list directory contents with ipfs files ls -l "/${folder}", and then check the second column of the file info for the hash you just received with the ipfs add command. If there is no hash that matches, then just ipfs files cp the object into that MFS directory. If however there is a hash that matches, then the actual file (with the same hash, not necessarily the same basename) is already in that MFS directory. Then you can, for example, check column 1 of that file’s info, and if the MFS basename is the same as your external basename, then you can return, and if the basenames differ, you can update the MFS basename with the ipfs files mv command, if you like, or leave the basename untouched, and instead create a /dupes subdirectory, and ipfs files cp the object there using the different external basename, maybe even adding -$(date +%s).$extension, so you’ll know when you added the dupe.

4

You’re right of cource, however I just needed something to simply add a real directory from my filesystem to a mfs dir. I already knew that there were no duplicated base names / alternate hashes because that would mean there was a severe problem with the original file sytem. (anyone fancy some raid array corruption)

If you specificaly need those features I can work on implementing a modified function for you

1 Like
5

Thank you, but no need. I have my own functions already. :slight_smile:

6

Please. It would be nice if you could share your scripts :wink:

7

This might be helpful: