Clean URLs, a Correct sitemap.xml and a Sane robots.txt for a Small Site

Search engines see URLs, not pages. If one article answers at /post, /post/, /post/index.html and on www., that’s four URLs for one page. This guide cleans that up on a small static site. You’ll pick one URL format, redirect the variants, generate a sitemap that lists only the URLs you want shown, check it automatically, and write a robots.txt that Google reads the way you meant. Every script here was run against this site, and the real results are shown, warts included.

Warnings

What You Need

Rules → variants → redirect → sitemap → check → robots.txt → submit

1. 01

   Step 1: Write down your URL rules

   Google’s URL-structure guidance boils down to a few rules worth adopting as house style:

   • Descriptive words instead of ID numbers, e.g. /tutorials/clean-urls-sitemap-robots-txt/.
   • Hyphens, not underscores, between words.
   • One case. Google treats /APPLE and /apple as different URLs, so lowercase everything.
   • Few parameters, and no URL fragments (#/page) that change content.

   Pick one form and stick to it. A static site usually ends up with “folder with a trailing slash, served from folder/index.html”.

2. 02

   Step 2: Test the variants of one URL

   Ask the server what it does with each variant:

   for u in \
     https://example.com/blog/my-post \
     https://example.com/blog/my-post/index.html \
     https://www.example.com/blog/my-post/ \
     http://example.com/blog/my-post/ \
     https://example.com/Blog/My-Post/ ; do
     curl -s -m 10 -o /dev/null -w "%{http_code} %{redirect_url}  <- $u\n" "$u"
   done

   Here’s what this site returned when this guide was written:

   301 https://byteslip.org/tutorials/development/  <- .../tutorials/development
   200   <- .../tutorials/development/index.html
   301 https://byteslip.org/tutorials/development/  <- https://www.byteslip.org/...
   301 https://byteslip.org/tutorials/development/  <- http://byteslip.org/...
   404   <- .../Tutorials/development/

   The no-slash, www and http forms all 301 to the one true URL, which is good. Wrong case gives a 404, which is fine. But /index.html answers 200, which is a duplicate. The self-referential canonical from tutorial #106 already tells Google which URL is preferred, and Google calls a redirect a “strong signal” too, so you can add one.

3. 03

   Step 3: Redirect index.html to the folder (Apache)

   Add this near the top of the site-root .htaccess:

   RewriteEngine On
   # /anything/index.html -> /anything/ (301), only when the visitor asked for it
   RewriteCond %{THE_REQUEST} \s/+(.*/)?index\.html[\s?] [NC]
   RewriteRule ^(.*/)?index\.html$ /%1 [R=301,L,NE]

   Matching on %{THE_REQUEST} (the raw request line) means the rule only fires when a visitor actually typed index.html. It doesn’t fire when Apache serves index.html internally for /folder/, so there’s no redirect loop. Tested on Apache 2.4:

   301 → /                     <- /index.html
   301 → /tutorials/a-b/       <- /tutorials/a-b/index.html
   301 → /tutorials/a-b/?x=1   <- /tutorials/a-b/index.html?x=1
   200                         <- /tutorials/a-b/

   Re-run step 2 afterwards. Every variant should now land on the same URL in one hop.

4. 04

   Step 4: Generate sitemap.xml from your files

   A sitemap should list the canonical URLs you want in search results, as absolute URLs, UTF-8 encoded, up to 50,000 URLs or 50 MB per file. Google recommends generating it rather than hand-editing once you’re past a few dozen URLs. This script walks the site folder and turns every index.html into a clean folder URL:

   #!/usr/bin/env python3
   # usage: python3 mksitemap.py public_html https://example.com > sitemap.xml
   import os, sys, datetime
   from xml.sax.saxutils import escape
   root, base = sys.argv[1], sys.argv[2].rstrip('/')
   SKIP = {'downloads', 'api'}            # folders you don't want listed
   urls = []
   for d, dirs, files in os.walk(root):
       dirs[:] = sorted(x for x in dirs if x not in SKIP and not x.startswith('.'))
       if 'index.html' in files:
           rel = os.path.relpath(d, root).replace(os.sep, '/')
           loc = base + '/' if rel == '.' else f'{base}/{rel}/'
           mtime = os.path.getmtime(os.path.join(d, 'index.html'))
           urls.append((loc, datetime.date.fromtimestamp(mtime).isoformat()))
   print('<?xml version="1.0" encoding="UTF-8"?>')
   print('<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">')
   for loc, lastmod in urls:
       print(f'  <url>\n    <loc>{escape(loc)}</loc>\n    <lastmod>{lastmod}</lastmod>\n  </url>')
   print('</urlset>')
   print(f'{len(urls)} URLs', file=sys.stderr)

   python3 mksitemap.py ~/public_html https://example.com > sitemap.xml

   Two notes from Google’s docs. It ignores <priority> and <changefreq>, so the script leaves them out. It uses <lastmod> only when the dates are consistently accurate, meaning the last significant change. File modification times are a decent stand-in, but re-uploading every file resets them, so don’t blindly re-upload the whole site.

5. 05

   Step 5: Check every URL in the sitemap

   A sitemap that lists redirects or 404s sends mixed signals. This checker fetches each <loc> and reports anything that isn’t a 200 whose canonical points back at itself:

   #!/bin/sh
   # usage: sh checksitemap.sh https://example.com/sitemap.xml
   SM="$1"
   curl -s "$SM" | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' | while read -r url; do
     out=$(curl -s -o /tmp/page.html -w '%{http_code}' "$url")
     canon=$(tr '\n' ' ' < /tmp/page.html | grep -o '<link rel="canonical" href="[^"]*"' | sed 's/.*href="//; s/"$//')
     if [ "$out" != "200" ]; then echo "STATUS $out  $url"
     elif [ -n "$canon" ] && [ "$canon" != "$url" ]; then echo "CANON  $url -> $canon"
     elif [ -z "$canon" ]; then echo "NOCANON $url"
     fi
   done
   echo "checked $(curl -s "$SM" | grep -c '<loc>') URLs"

   Run against this site, it checked 123 URLs in about 30 seconds and found exactly one problem:

   STATUS 301  https://byteslip.org/games
   checked 123 URLs

   That entry should be /games/, the URL it redirects to. That’s the kind of thing nobody notices by eye. Also confirm the file is well-formed XML:

   xmllint --noout sitemap.xml && echo OK

6. 06

   Step 6: Write a robots.txt that means what you think

   robots.txt lives at the site root and controls crawling, not indexing. Google says a blocked URL can still show up in results if other pages link to it. Keep it short:

   User-agent: *
   Disallow: /downloads/
   Disallow: /api/
   
   Sitemap: https://example.com/sitemap.xml

   Two gotchas:

   • Order doesn’t matter to Google. It applies the most specific (longest) matching rule, and when rules tie, the least restrictive one (RFC 9309 says the same). So Allow: / at the top doesn’t cancel Disallow: /downloads/ for Google.
   • Not every tool agrees. Python’s urllib.robotparser, run against this site’s robots.txt (which starts with Allow: /), returned True (crawlable) for /downloads/x.apk. Google’s longest-match rule says it’s blocked. Check with Search Console’s robots.txt report, and don’t trust a generic library’s answer for Google.
   • Don’t block CSS or JS that pages need to render.
7. 07

   Step 7: Tell Google where the sitemap is

   Do either or both:

   • Add a Sitemap: https://example.com/sitemap.xml line anywhere in robots.txt (it must be an absolute URL).
   • Submit it in Search Console’s Sitemaps report, which also shows when Google last read it and any errors.

   Google calls a sitemap a hint, not a guarantee. What matters is that it stays accurate, so re-run the generator and checker whenever you publish.

Tips

FAQ

Sources

• Google Search Central: URL structure best practices
• Google Search Central: Learn about sitemaps
• Google Search Central: Build and submit a sitemap
• Google Search Central: Introduction to robots.txt
• Google: How Google interprets the robots.txt specification
• Google Search Central: Specify a canonical URL (redirects and canonicals)
• sitemaps.org: Sitemaps XML protocol
• RFC 9309: Robots Exclusion Protocol
• Apache HTTP Server: mod_rewrite