-- Leo's gemini proxy

-- Connecting to michaelnordmeyer.com:1965...

-- Connected

-- Sending request

-- Meta line: 20 text/gemini;lang=en-US

Deploying Jekyll with Rake and Notifying Search Engines


Rake is Ruby make. Make is a tool to build source code.


I use it to have all the relevant commands I need to publish the site in one place. Because Jekyll, my blog engine, runs on Ruby, it’s a good fit.


My deployment process looks like this: `build > gzip > rsync > notify`.


It seems, it still is better to notify search engines, if the posts should be indexed timely. Over the last six months, it took Google between two and six weeks to index new posts for me, according to Google Search Console.


The notification process is just telling them, that something has changed. The `sitemap.xml` is a prime candidate for that, because it has all the posts in it. And, at least Google is expecting this file because of that. Bing deprecated this a couple of years ago and only accepts IndexNow request. So I send them the changed sitemap URL.


A `rake deploy` will start the above mentioned process and publish my site, notifying Google and Bing at the end. Here’s the code, which has to be saved as `Rakefile.rb` in the root of the Jekyll blog, and the settings at the beginning of the file have to be adjusted:


## Deployment settings
domain = "example.com"
ssh_user = "username"
ssh_path = "/var/www/#{domain}/"

## Notification settings
# Feed and sitemap take local paths starting with a slash
# E.g. example.com/feed.xml => /feed.xml example.com/feeds/atom.xml => /feeds/atom.xml
feed = "/feed.xml"
sitemap = "/sitemap.xml"
indexnow_sitekey = "xxx"

task :default => ["build"]

desc "Builds and deploys the site and notifies search engines afterwards"
task :deploy do
  puts "==> Builds and deploys the site and notifies search engines afterwards"
  Rake::Task[:build].invoke
  Rake::Task[:rsync].invoke
  Rake::Task[:notify].invoke
end

desc "Builds the site"
task :build do
  puts "==> Building the site..."
  system "bundle exec jekyll build"
end

desc "Serves the site"
task :serve do
  puts "==> Building and serving the site..."
  system "bundle exec jekyll serve"
end

desc "Gzips the site"
task :gzip do
  puts "==> Pre-gzip'ing the site..."
  # FileList won't list .well-known/*
  # files = FileList['_site/**/*.html', '_site/**/*.css', '_site/**/*.js', '_site/**/*.svg', '_site/**/*.xml', '_site/**/*.json', '_site/**/*.txt']
  system 'for file in $(find _site -type f \( -name "*.html" -o -name "*.css" -o -name "*.js" -o -name "*.svg" -o -name "*.xml" -o -name "*.json" -o -name "*.txt" \)); do printf . && gzip -kf "${file}"; done; echo'
end

desc "Deploys the content of ./_site to the server via rsync"
task :rsync => ["gzip"] do
  puts "==> Rsyncing the content of ./_site to #{domain}"
  system "rsync -vaSWhP --delete --chmod=Du=rwx,Dgo=rx \
    --exclude=.DS_Store \
    --exclude=._* \
    --exclude=.git \
    --exclude=.gitignore \
    _site/ \
    #{ssh_user}@#{domain}:#{ssh_path}"
end

task :notify => ["notify:google", "notify:bing"]
desc "Notify various services that the site has been updated"
namespace :notify do
  desc "Notify Google of updated sitemap.xml"
  task :google do
    begin
      require 'net/http'
      require 'cgi'
      puts "==> Notifying Google that #{domain} has been updated..."
      Net::HTTP.get('www.google.com', '/ping?sitemap=' + CGI.escape("https://#{domain}#{sitemap}"))
    rescue LoadError
      puts "! Could not ping Google, because Net::HTTP or URI could not be found."
    end
  end

  desc "Notify Bing of updated sitemap.xml"
  task :bing do
    begin
      require 'net/http'
      require 'cgi'
      puts "==> Notifying Bing that #{domain} has been updated..."
      Net::HTTP.get('www.bing.com', '/indexnow?url=' + CGI.escape("https://#{domain}#{sitemap}&key=#{indexnow_sitekey}"))
    rescue LoadError
      puts "! Could not ping Bing, because Net::HTTP or URI could not be found."
    end
  end
end

Update 2024-01-01


Google has deprecated the sitemap ping endpoint and returns a 404.

-- Response ended

-- Page fetched on Tue May 21 09:36:21 2024