gemini://jacksonchen666.com/posts/2023-11-19/13-34-09/index.gmi

2023-11-19 13:34:09Z

In the last few days of my server, you may have noticed that it was down sometimes. This is because in the last few days, I did multiple things that kept leading to the downtime of my server. Oops.

On 2023-11-16, I was woken up by some emergency monitoring alerts notifications at 04:00Z. I look at it, and the services I host was not available (website, subdomains stuff, etc.).

Because it was 04:00Z at the time, I just decided to put the investigation off later.

Later at about almost 07:00Z, I dig around and check what's available:

Onion services

I2P eep sites

Yggdrasil network stuff

Based on what was available, I concluded that clearnet services (so going to jacksonchen666.com and etc.) were broken.

The issue I found was that ddclient was just... not working for some unknown reason.

ddclient is used to change the IP address for my domains when my public IP for the server changes. However, it just decided to remove the IP address from my domains, resulting in the issue and downtime.

I fixed it by getting my server to do the IP update again. Probably by running ddclient.

So, after that was resolved, we had... the same issue again!

I have no idea why the issue keeps reoccurring. I just tried to do the same thing again, running ddclient and... it immediately exiting. I just hoped it worked.

I continue trying to figure out the issue at around 08:00Z. I still have no idea.

I login to my deSEC account (on 2023-11-16 08:00Z) and look around. Then I get hit with a 24 hour rate limit (on 2023-11-16 08:20Z).

Yes, deSEC has rate limits that apply for 24 hours. In deSEC's documentation for rate limits, there is a rate limit name called user and it gives 2000 requests per day (or 24 hours).

I have somehow exceeded that rate limit. I also did nothing to exceed that rate limit.

It seemed weird, but my server was luckily online with the web, so I just decided to wait out the rate limit.

After waiting for 24 hours (now 2023-11-17 09:30Z), I decided to try to login again. And I am still under a 24 hour rate limit.

At this point, I decided to email deSEC support for this issue, asking why I still have a 24 hour rate limit.

I then receive an email back saying:

> Devices with the IP address [laptop-server IPv6 address 1] and [laptop-server IPv6 address 2] are doing thousands of dynDNS requests, sometimes several times per second.

(That's a partial quote, and I have replaced the IPv6 addresses with the actual device. It was originally an actual IPv6 address)

I checked the server and... oh...

The server has multiple instances of ddclient, all of them quickly retrying (within less than 60 seconds), all trying to do dynDNS requests, then causing my account to be rate limited because of all those requests adding up.

Turns out, ddclient defaults to being a daemon, not being foreground. I assumed that when I ran ddclient, it would just stick output to my terminal. Nope, it just went on its own.

So here's how I got into the rate limit again:

I ran ddclient without the foreground flag

Multiple instances of ddclient were just spamming the deSEC API

That cause it to hit the worst-case rate limits

Repeat

So I just went ahead and stopped the ddclient service, and killed all ddclient processes.

The problem should be solved, so I went ahead and asked deSEC support:

> Can the rate limit on my account be removed earlier?

They responded:

> That's not simple on our side, so we would only do it in extraordinary circumstances.

So I decided to come back after the rate limit again.

I earlier stopped all ddclient processes. However, on 2023-11-18 11:40Z, my server IP address actually changed and there was nothing to fix that issue automatically (not even ddclient because I stopped it).

So of course, when I noticed the issue (at around 2023-11-18 14:10Z), I went ahead and just started ddclient again. And it's fixed.

Now the server should be back to normal, and everything should be back to normal. Things should be running just fine as before the initial issue.

Upon reviewing the past few days and the mistakes I made, I'm not sure if I can say that I am without issues (there's another blog post about another mistake coming soon(tm)).

Server removed IP address

Alerts at 2023-11-16 04:00Z

I fixed the issue

I was rate limited on deSEC (2023-11-16 08:20Z)

Next day, I try to login to deSEC, still rate limited (2023-11-17 09:30Z)

I found the issue for the second rate limit, multiple instances of ddclient

I wait for the rate limit

The server IP changes, and ddclient isn't running (2023-11-18 11:40Z)

I fix that issue (2023-11-18 14:10Z)

The Last Few Days of My Server

The initial issue

More issues arise

My deSEC account, the next day

Even more issues

Hopefully no more issues

Recap