I started digging a bit deeper, some observations:
- Restarting the daemon makes the request http://decentralized.blog:8080/ipfs/QmUAz4unWjfWa7xkHttod8gG3gfyob3HrWFByUvrKH6Toa/ fast again
- IPNS requests are slow whatever I do (doesn’t matter from what gateway the request comes from)
- My IPFS node eats up all memory on my VPS after a few hours
- When that happens the site still loads fast from the official gateway: https://gateway.ipfs.io/ipfs/QmUAz4unWjfWa7xkHttod8gG3gfyob3HrWFByUvrKH6Toa/
- From my own server it’s also still fast. Hmm.
So that doesn’t really lead up to some conclusion 
Does anyone have an idea how to analyze what the cause is? I have some other ideas, but it needs to get slow again