Python Sitemap Generator Project

Creating a reliable Python sitemap generator is one of the smartest moves you can make to boost your website’s SEO and improve crawler efficiency. A well‑structured XML sitemap tells search engines exactly which pages to index, how often they change, and how important each page is relative to the rest of your site. In this guide we’ll walk through a complete, production‑ready Python project that automatically crawls a website, builds a standards‑compliant sitemap, and writes it to sitemap.xml. Whether you’re a solo developer, a small business owner, or part of a larger dev team, you’ll get a clear roadmap, ready‑to‑run code snippets, and best‑practice tips that keep your sitemap fresh and search‑engine friendly.

Why a Custom Python Sitemap Generator?

Most CMS platforms offer built‑in sitemap tools, but they often lack flexibility, speed, or the ability to integrate with custom workflows. A Python‑based solution gives you:

• Full control over crawl depth, URL filters, and priority assignment.
• Easy integration with CI/CD pipelines to regenerate the sitemap on each deployment.
• Scalability – you can switch from synchronous requests to asynchronous aiohttp with minimal code changes.
• Transparent, open‑source code that you can audit for security and performance.

Project Overview

The project is organized into three core modules:

1. crawler.py – Handles URL discovery, respects robots.txt, and respects crawl‑delay.
2. sitemap_builder.py – Converts the collected URLs into a valid XML sitemap.
3. run.py – Orchestrates the crawl, builds the sitemap, and writes the output file.

All modules are pure Python, require only a handful of third‑party libraries, and can be extended with plug‑in hooks for authentication, API endpoints, or multi‑domain support.

Setting Up the Development Environment

Follow these steps to get the project up and running on any modern operating system.

# 1. Clone the repository
git clone https://github.com/yourusername/python-sitemap-generator.git
cd python-sitemap-generator

# 2. Create a virtual environment
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

The requirements.txt file typically includes:

• requests – for HTTP requests (synchronous version).
• aiohttp – for optional asynchronous crawling.
• beautifulsoup4 – for parsing HTML and extracting links.
• lxml – for fast XML generation.

Core Logic: Crawling the Site

1. Respecting robots.txt

Before any page is fetched, the crawler checks the site’s robots.txt file. This ensures you stay within the webmaster’s guidelines and avoid accidental bans.

import urllib.robotparser as rp

def can_fetch(url, user_agent='*'):
    parser = rp.RobotFileParser()
    parser.set_url(f"{base_url}/robots.txt")
    parser.read()
    return parser.can_fetch(user_agent, url)

2. Extracting Links

Using BeautifulSoup, the crawler extracts href attributes, normalizes them, and filters out external domains.

from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

def extract_links(html, current_url):
    soup = BeautifulSoup(html, 'html.parser')
    links = set()
    for tag in soup.find_all('a', href=True):
        href = urljoin(current_url, tag['href'])
        if urlparse(href).netloc == urlparse(base_url).netloc:
            links.add(href.split('#')[0])  # Remove fragments
    return links

3. Managing Crawl Depth and Queue

A simple breadth‑first search (BFS) queue guarantees that you never exceed the configured depth.

from collections import deque

def bfs_crawl(start_url, max_depth=3):
    visited = set()
    queue = deque([(start_url, 0)])

    while queue:
        url, depth = queue.popleft()
        if depth > max_depth or url in visited:
            continue
        if not can_fetch(url):
            continue

        response = requests.get(url, timeout=10)
        visited.add(url)

        if 'text/html' in response.headers.get('Content-Type', ''):
            for link in extract_links(response.text, url):
                queue.append((link, depth + 1))
    return visited

Generating a Valid XML Sitemap

The sitemap_builder.py module turns the set of URLs into an XML document that complies with the Sitemap protocol.

from lxml import etree
from datetime import datetime

def build_sitemap(urls, changefreq='weekly', priority=0.5):
    urlset = etree.Element('urlset', xmlns="http://www.sitemaps.org/schemas/sitemap/0.9")
    for url in sorted(urls):
        url_el = etree.SubElement(urlset, 'url')
        etree.SubElement(url_el, 'loc').text = url
        etree.SubElement(url_el, 'lastmod').text = datetime.utcnow().date().isoformat()
        etree.SubElement(url_el, 'changefreq').text = changefreq
        etree.SubElement(url_el, 'priority').text = str(priority)
    return etree.tostring(urlset, pretty_print=True, xml_declaration=True, encoding='UTF-8')

Putting It All Together – The Entry Point

The run.py script ties crawling and sitemap creation into a single, easy‑to‑execute command.

import argparse
from crawler import bfs_crawl
from sitemap_builder import build_sitemap

def main():
    parser = argparse.ArgumentParser(description='Generate an XML sitemap.')
    parser.add_argument('base_url', help='Root URL of the site to crawl')
    parser.add_argument('--depth', type=int, default=3, help='Maximum crawl depth')
    parser.add_argument('--output', default='sitemap.xml', help='Output file name')
    args = parser.parse_args()

    urls = bfs_crawl(args.base_url, max_depth=args.depth)
    sitemap_xml = build_sitemap(urls)

    with open(args.output, 'wb') as f:
        f.write(sitemap_xml)
    print(f"Sitemap generated with {len(urls)} URLs → {args.output}")

if __name__ == '__main__':
    main()

Advanced Features You Can Add

Asynchronous Crawling for Speed

If you need to crawl thousands of pages, replace the synchronous requests calls with aiohttp and an async queue. The core logic stays the same, but you’ll see a 3‑5× speed boost on large sites.

Custom Priority Logic

Assign higher priority to pages that contain key keywords, have a lot of inbound links, or belong to a specific URL pattern. Modify build_sitemap to accept a mapping of url → priority.

Incremental Updates

Store the last crawl’s URL list in a JSON file. On the next run, compare the new list with the old one and only rewrite sitemap.xml if changes are detected. This reduces unnecessary server load and avoids search‑engine churn.

Testing and Validation

Before deploying, run these checks:

• Unit tests – Verify that extract_links correctly normalizes URLs.
• Integration test – Crawl a small staging site and assert that the generated XML validates against the official XSD.
• SEO audit – Use Google Search Console’s “Sitemap” tool to submit the file and watch for parsing errors.

Deploying the Sitemap Generator

There are several ways to keep your sitemap up‑to‑date:

1. CI/CD pipeline – Add a step in GitHub Actions, GitLab CI, or Jenkins that runs python run.py https://example.com --depth 4 after each successful deployment.
2. Scheduled cron job – On a Linux server, create a cron entry like 0 2 * * * /path/to/venv/bin/python /path/to/run.py https://example.com --output /var/www/html/sitemap.xml to regenerate nightly.
3. Serverless function – Deploy the script as an AWS Lambda or Google Cloud Function, trigger it via HTTP, and store the result in a CDN bucket.

SEO Benefits of a Fresh Sitemap

Search engines treat a well‑maintained sitemap as a trust signal. Here’s what you gain:

• Faster indexing of new or updated content.
• Reduced crawl budget waste – bots skip irrelevant pages and focus on high‑value URLs.
• Better visibility for deep‑linked pages that might otherwise be orphaned.
• Clear communication of changefreq and priority, helping search engines allocate resources intelligently.