Double-hashed entries in denylists

Help
1

Hello,

With the nopfs content blocking feature now available with kubo 0.24, i’m adding support for denylists in galacteek.

Blocking content by creating a custom denylist works great. I’ve also downloaded the badbits list and i understand that all the entries in that list use either “single” sha2_256 hashing of the cid string or double-hashing of the multihash ? Is the goal of the hashing to obfuscate the CIDs so that specific content can be blocked by a party without anyone pointing fingers ? Or is there a more technical reason ?

2

The idea is that by double-hashing CIDs, you can’t trivially go from the list to being able to access abusive content. The last thing you want is to not only not stop badbits entries from being accessible but to actually drive traffic to them.

3

@Zorlin I thought so, thanks for the explanation. Another thing, besides restarting the daemon, is there any other way of notifying kubo that the content of one of the denylists has changed and needs to be parsed again ? I think not … A busy running kubo daemon has more important things to do than parsing potentially huge lists when an entry is added or dropped, doesn’t it …

4
  1. Start Kubo (ipfs daemon). The plugin should be loaded automatically and existing denylists tracked for updates from that point (no restarts required).

I believe it uses inotify or some similar mechanism to track changes automatically and you don’t need to restart Kubo when the list changes, only when adding or removing a list.

5

Read the docs, you’re right, it will track all *.deny files that are present when the daemon is started, but if you add a .deny file after the daemon starts, it won’t scan it, that’s fair. Thanks @Zorlin

Files need to be present before starting the ipfs daemon in order to be watched for updates.

If a new denylist file is added, ipfs daemon needs to be restarted.

6

Hello, it is not clear from the Kubo docs:

Denylist files are watched for anything that is appended to them.

i.e. echo "/ipfs/Qmxxx" >> denylist.deny

appends a new item to the end of the file and this will automatically be read and the block will take effect immediately, no restart required.

Any change to files that is not an append, requires a restart.

The “official” denylist (https://badbits.dwebops.pub/badbits.deny) is updated but not in an append-only fashion. As a stop-gap solution, I created https://denyli.st/badbits.deny which is only updated in an append-only way. https://denyli.st/badbits.deny.txt is the same in a way that it gets displayed in the browser (shenanigans of github pages).

The last thing is something that keeps your local copy of badbits.deny updated to whatever https://denyli.st/badbits.deny has. This “tracking” functionality is not yet in Kubo, but can be achieved by running httpsubs (https://github.com/ipfs-shipyard/nopfs/tree/master/cmd/httpsubs) on the side like httpsubs /home/ipfs/.ipfs/denylists/badbits.deny https://denyli.st/badbits.deny (use your own denylist path).

You can install this with go install github.com/ipfs-shipyard/nopfs/cmd/httpsubs@master.

Once you have all of this you can have automatic tracking of denylists.

3 Likes
7

@hector Thank you very much for these precisions ! So ok, if you insert an entry at the beginning of a denylist once kubo is running, or if you remove some entries, it won’t get scanned again, very good to know !

The way i’ve implemented it is that i fetch the denylist files before starting the ipfs daemon. I use https://denyli.st/badbits.deny.txt right now, is this URL gonna be available long-term, or is it better to use the dwebops.pub url ?

Kudos for httpsubs, very cool. How often is the badbits.deny content updated, every day, every week ?

1 Like
8

is this URL gonna be available long-term, or is it better to use the dwebops.pub url ?

It will be available at least until the dwebops version becomes append-only. Both lists are updated around 20-30 times a day. But sometimes updates are concentrated in a few hours.

3 Likes