# How to measure how many blocks are shared between CIDs?

**URL:** <https://discuss.ipfs.tech/t/how-to-measure-how-many-blocks-are-shared-between-cids/18104>\
**Category:** Help\
**Tags:** use-cases-and-apps\
**Created:** [June 8, 2024, 12:49pm UTC](https://discuss.ipfs.tech/t/how-to-measure-how-many-blocks-are-shared-between-cids/18104 "2024-06-08T12:49:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![meermanr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/meermanr/32/7280_2.png) [@meermanr](https://discuss.ipfs.tech/u/meermanr)\
**Post date:** [June 8, 2024, 12:49pm UTC](https://discuss.ipfs.tech/t/how-to-measure-how-many-blocks-are-shared-between-cids/18104/1 "2024-06-08T12:49:34Z")

</div>

I’d like to use a private IPFS swarm at work to distribute large disk images (~6-10 GiB) between build systems in the cloud and lab equipment in our various offices. I believe that the content of these disk images is largely identical between builds (perhaps 80% unchanged), and so hope that IPFS will save us a lot of network transfers and reduce both time and cost to distribute these images.

I want to maximise deduplication between these similar images, but I’d also like to compress them. I noticed that `zstd` has an `--rsyncable` option and lets you configure the blocksize (with `-B`) so I am hoping that if I match IPFS’s block size of 256kB I might be able to have my cake and eat it.

How can I tell how many blocks are shared between two (or more) CIDs? Are there any tools which can help with this, that can cope with such large images?

* * *

For context and completeness, this is the method I’ve been using to date, but I’ve little confidence in this approach. Please point out anything I am missing.

```bash
ipfs refs ${CID_A} > a.txt
ipfs refs ${CID_B} > b.txt
cat a.txt b.txt | sort | uniq -c | sort -rn | head

```

Which gives me results like this:

```auto
# cat a b | sort | uniq -c | sort -rn | head
 164 bafyb4igsmsho23lsnstheptnxehtrsxag6dejrleftw3mx4dpkjeyw5xvq
   2 bafyb4iew4jo7ncdmfg2omzswignrflkj2nekx6b43uwwtew47m252frj7u
   2 bafyb4icj2iqw3aew5vp6ap4dnfqbvhirdsipanlm3h7q72waelcsoliiiu
   2 bafyb4ibtdmszworl3g4pf5n5gpinsb6lrppgea3xpryoofwjzdcykpteha
   2 bafyb4iar4rrgkliooqvsdihh5vs3cin2pwv4jqrdun7yor26pmysiylw2a
   1 bafyb4ihynsbm6w6mjlcdanpxl5wfhypvbe4tb7khdknui3f2dh2tpuqwsq
   1 bafyb4ihyfusgjxh2ztwi4dfwli6eafqj4yrtnj5tugkeot6vecndustl7u
   1 bafyb4ihxaivofhs7tqz2vyebxtcjb75d6jv7pazyiekzl2ht3wusucanau
   1 bafyb4ihqmz4hcdgwcbghjusx4fgf6pdoq7ptn2oteh4flvtqsw5zztuhs4
   1 bafyb4ihpylrhg3aesspl2v6ume3k2b7p2ps554t3paduu2pmas3duou36y

```

This gives me a rough indication, but I have my doubts about converting this into, say, a percentage of the two files that overlap. I mean, each of these refs could be a subtree, right?

The way I’ve been checking the size of a block is to use `ipfs cat`, and reading the total size from that:

```bash
# ipfs cat bafyb4igsmsho23lsnstheptnxehtrsxag6dejrleftw3mx4dpkjeyw5xvq > /dev/null
 43.50 MiB / 43.50 MiB [==========================================================] 100.00% 0s

```

So my recipe currently is:

```bash
# Dump all blocks referenced for original (old) file
# ipfs refs --recursive bafyb4ihlhd33yczkyehbvt5hoiyyao6m6reqiyqgendho2himmf4krhupq > a

# .. and new file
# ipfs refs --recursive bafyb4ibo7d2bywtaszu5q2pf25eat4akley4ygb7u2tb2gwxibdxtpq4ga > b

# How many blocks are there, total, in both files?
# cat a b | wc -l
   53876

# How many distinct blocks are there?
# cat a b | sort | uniq -c | wc -l
   14938

# .. and how many of them are unique (referenced exactly once)?
# cat a b | sort | uniq -c | grep '^ *1 ' | wc -l
    9178

# So what's the dedupe?
# (53876 - 9178) / 53876
0.8296458534412354

```

Is there a better way?

---

<div class="post-metadata">

**Author:** ![Jorropo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ipfs.tech/jorropo/32/3230_2.png) [@Jorropo](https://discuss.ipfs.tech/u/Jorropo)\
**Post date:** [June 8, 2024, 3:06pm UTC](https://discuss.ipfs.tech/t/how-to-measure-how-many-blocks-are-shared-between-cids/18104/2 "2024-06-08T15:06:06Z")

</div>

`ipfs dag stat ${CID_A} ${CID_B}`