gemini://tilde.team/~easeout/glog/2020-09-07-re-a-wordpress-confession.gmi

You better believe it. I too have done heinous things in live production systems, including outright destroying them by accident. Reading stories about other people learning the hard way helps me relax about the mistakes that haunt me. So this reply is not criticism, just lessons learned.

> After some investigating I realised that our fancy third party theme was storing many of the images as absolute URLs from when they were added to pages. Cripes.

In development we often talk about moving to declarative code from imperative, or preferring composition over inheritance. Wrapped up in those inclinations is the desire to remove implicit dependencies and have less shared mutable state. It makes concurrency safe, for instance. But these concepts also exist outside a program, at the OS level or in this case the network level. In a way, the internet is just a bunch of global variables, and HTML is full of dependencies on them.

Back at a one-program level, one way we ensure quality is by testing functions in isolation. Sometimes a piece of code has implicit dependencies or side effects and we can't easily test it in isolation. We might rearrange code to inject the dependency, or something like that. In any case, we are going to do something to make it isolated so it can be tested, and then we can be confident.

So what dependencies affect a deployed website? What comes to mind in this scenario is DNS. DNS lookups are involved when a browser makes requests for pages and images. DNS changes are often involved when you launch a new site. DNS is a cache with expiration, which often means unwanted surprises. So a browser's view of a URL depends on not only the host machine's DNS cache, but also the caches of upstream servers. It's easy to forget about this because DNS only changes rarely. But it's an implicit dependency on shared mutable state, all the way down.

We do have a way to test in isolation from DNS caches, by editing /etc/hosts. You can alias the production site domains to equal your deployed, but not yet pointed-to, production server. This is no different from mocking a dependency in a unit test. This alias tests two things: 1. From the perspective of the browser, the domain behaves as if it points to the new site. 2. From the perspective of the browser, items on the old site are unreachable.

The second effect is worth considering, because another piece of shared mutable state involved in a browser rendering a website is… every other live web server on the internet right now, and all the URLs they respond to. Yikes. So unless you can get the entire site and all its data running on your own machine and then turn off your wifi, you can't completely test the site until it's launched? Well, maybe you could edit /etc/hosts to map all the domains under your organization's control to a bogus IP so they won't respond, and you can at least test that those domains have no effect on behavior. But all this is one reason that in devops land, we like to do continuous deployment.

> (I was not so ambitious as to do continuous-delivery devops; let's keep expectations low here.)

Again I can't fault the author; I've been there. And the configuration change really does sound like a small adjustment to deploy, not a big bang release. But one warning sign is that there wasn't a production server all along during development, kept in sync with staging in terms of code, but configured with prod domains and SSL and such. The change could have been to just point the gateway at the new production server from the old one, and in that case the production server would have been testable all along using /etc/hosts.

> Many years later I went back to the old website out of curiosity... and some of those images are still coming off the staging domain.

Re: A Wordpress Confession