gemini://michaelnordmeyer.com/gemlog/2022-11-09-bing-and-duckduckgo-remove-indexed-websites.gmi

My website disappeared from DuckDuckGo many months ago. I first noticed it in Spring. It’s easy to verify. Just search for `site:michaelnordmeyer.com` on DuckDuckGo and you will get zero results.

Coincidently, I noticed in fall of 2021 degrading search results on DuckDuckGo. I didn’t think much of it, as DuckDuckGo was never as good as Google, but still okay. That’s why I replaced Google with DDG on all my personal and work devices in 2013 or early 2014, just before Apple officially introduced support in fall 2014. I was very happy about that.

I knew first-hand that DuckDuckGo had a limited search index, because my site – this one – wasn’t fully indexed, even though I had for a long time great on-page SEO, fast load times, great mobile-friendliness, good SSL, and a proper sitemap. But only about half of my pages were indexed. Google on the other hand had my site fully indexed.

It’s easy to imagine that, if this was true for many other smaller sites, that DuckDuckGo’s search results were very “incomplete”. Yes, you probably cannot index the whole Internet, but still — Google was much better.

Especially because I often search for solutions to my software development problems. And those solutions are quite often published by small websites like mine, where a single person publishes a post every once in a while.

Anyway, in spring of 2022 I found out that my website was de-indexed on DuckDuckGo and on Microsoft’s Bing as well.¹

After some searching with a search engine – yes, I know the irony – I found out some others have had the same problem. Jesse Squires had his site de-indexed as well, but was ultimately able to have his site restored on Bing after contacting Bing support.

I tried a couple of my friends’ websites, who also work in software development and are quite competent in this regard. They were de-indexed as well. Okay, seemed to be a real problem on Bing’s part.

This explained the bad search results I got from DuckDuckGo in general, which I had noticed months before. Without knowing all this at the time, I switched in the beginning of 2022 to using Google as my main search engine for work again, because results where just so much better.

After knowing the source of my problems – Bing, I tried to mitigate this problem like Jesse did. But after creating and subsequently being locked out of three different Microsoft accounts for Bing Webmaster Tools, which I only created for this purpose, I gave up.

Maybe because of using good tracking blockers like 1Blocker and uBlock Origin, DoH, and often surfing in private mode in Safari, I was flagged as a bot or something. I even used a Fastmail e-mail address on one occasion, which have very good reputation. But to no avail. So I never was able to contact Bing support because of the locked accounts.

I didn’t want to write about this, even though I was reminded in June by posts from other people having the same problem. In general it’s not important to me anymore how many people read my site. But today I came by a current article by Nicolas Magand, who summed up the problem quite well. He also links to Jesse, who had already updated his posts with more info.

For me it’s especially bad, because I use customized DuckDuckGo search for my site. Right now it always shows no results and there’s not much that I can do to fix this. It’s probably better to remove the search altogether until Bing has fixed its problems.

In the meantime, I use Startpage² on macOS, but still have to use DuckDuckGo on iOS and iPadOS, because Apple doesn’t give users much choice when it comes to search engine alternatives. Even on macOS using Safari, I have to use “Quick Website Search”, which means typing startpage in front of the search query, use cursor down, and then use enter. If I just use enter, I will search DuckDuckGo for “startpage” and my search term. Quite annoying.³

It’s quite tragic that only one good and many not so good search engines exist.

¹ Most of DuckDuckGo’s search results come from Bing.

² Startpage is a privacy-respecting search engine and uses Google search results.

³ Yes, I know that I can use DuckDuckGo’s `!sp` to search on Startpage, but this means three requests, including one redirect on DDG, while exposing my search query to an intermediary. That’s why I’m in the process of switching from those !Bangs to quick website search for Wikipedia, IMDb, and others.

Bing and DuckDuckGo Remove Indexed Websites