Python News Aggregator Web App Project

Looking to build a Python news aggregator web app that pulls the latest headlines from multiple sources and presents them in a clean, customizable interface? You’re in the right place. In this comprehensive guide we’ll walk through the project’s core concepts, the tech stack choices, step‑by‑step implementation details, and best practices for scaling and SEO‑optimizing your aggregator. Whether you’re a beginner eager to practice Flask or a seasoned developer planning a Django‑based production system, this article gives you everything you need to turn the idea into a fully functional web app.

Why Build a News Aggregator with Python?

Python’s rich ecosystem of libraries makes it a natural fit for data‑driven web projects. When you combine requests, feedparser, and a modern web framework (Flask or Django), you get a powerful stack that can:

Consume RSS/Atom feeds and public news APIs with minimal code.
Parse, filter, and store articles in a relational database.
Serve dynamic HTML pages that are SEO‑friendly and fast.
Scale horizontally using Docker, Kubernetes, or serverless platforms.

Project Overview

The goal is to create a web application that:

Fetches news articles from at least three different sources (e.g., NewsAPI.org, BBC RSS, TechCrunch API).
Normalizes the data (title, description, URL, publication date, source).
Stores the articles in a SQLite or PostgreSQL database.
Displays the latest headlines on a responsive front‑end with category filters.
Provides a simple admin interface to add or remove sources.

Choosing the Right Framework

Flask – Lightweight and Flexible

If you prefer a minimalistic approach, Flask gives you full control over routing, templating, and background jobs. It’s ideal for:

Rapid prototyping.
Learning how each component fits together.
Deploying as a single‑file app on platforms like Heroku.

Django – Batteries‑Included Solution

Django shines when you need built‑in admin panels, authentication, and ORM capabilities out of the box. Choose Django if you plan to:

Scale to multiple users with personalized feeds.
Leverage Django’s admin to manage sources without writing extra code.
Integrate with third‑party packages such as django-celery-beat for scheduled tasks.

Setting Up the Development Environment

Below is a quick setup script that works for both Flask and Django projects. It creates a virtual environment, installs required packages, and initializes a Git repository.

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

# Choose one of the two stacks:
# Flask stack
pip install Flask requests feedparser SQLAlchemy

# Django stack
pip install Django djangorestframework psycopg2-binary feedparser requests

git init
git add .
git commit -m "Initial commit – project scaffold"

Fetching News – The Core Logic

The aggregator’s heartbeat is a scheduled task that pulls data from external sources. Below is a simplified example using requests and feedparser that works in both frameworks.

import requests
import feedparser
from datetime import datetime

def fetch_rss(url):
    feed = feedparser.parse(url)
    articles = []
    for entry in feed.entries:
        articles.append({
            'title': entry.title,
            'link': entry.link,
            'summary': entry.get('summary', ''),
            'published': datetime(*entry.published_parsed[:6]),
            'source': url
        })
    return articles

def fetch_json_api(endpoint, api_key):
    headers = {'Authorization': f'Bearer {api_key}'}
    resp = requests.get(endpoint, headers=headers)
    resp.raise_for_status()
    data = resp.json()
    articles = []
    for item in data['articles']:
        articles.append({
            'title': item['title'],
            'link': item['url'],
            'summary': item.get('description', ''),
            'published': datetime.fromisoformat(item['publishedAt'].replace('Z', '+00:00')),
            'source': endpoint
        })
    return articles

Storing Articles in the Database

SQLAlchemy Model (Flask)

from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()

class Article(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(255), nullable=False)
    link = db.Column(db.String(500), unique=True, nullable=False)
    summary = db.Column(db.Text)
    published = db.Column(db.DateTime, index=True)
    source = db.Column(db.String(255))

    def __repr__(self):
        return f"<Article {self.title[:50]}...>"

Django Model

from django.db import models

class Article(models.Model):
    title = models.CharField(max_length=255)
    link = models.URLField(unique=True)
    summary = models.TextField(blank=True)
    published = models.DateTimeField(db_index=True)
    source = models.CharField(max_length=255)

    class Meta:
        ordering = ['-published']

    def __str__(self):
        return self.title

Scheduling the Fetcher

To keep the news feed fresh, schedule the fetcher to run every 15‑30 minutes. Two popular options are:

Cron + Flask CLI: Write a custom Flask command and add a cron entry.
Celery Beat + Django: Use django-celery-beat for a database‑backed schedule.

Example: Flask CLI Command

import click
from flask import Flask
from yourapp.models import db, Article
from yourapp.fetcher import fetch_rss, fetch_json_api

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///news.db'
db.init_app(app)

@app.cli.command('update-news')
def update_news():
    sources = [
        {'type': 'rss', 'url': 'https://feeds.bbci.co.uk/news/rss.xml'},
        {'type': 'api', 'url': 'https://newsapi.org/v2/top-headlines?country=us', 'key': 'YOUR_NEWSAPI_KEY'}
    ]
    for src in sources:
        if src['type'] == 'rss':
            articles = fetch_rss(src['url'])
        else:
            articles = fetch_json_api(src['url'], src['key'])
        for art in articles:
            if not Article.query.filter_by(link=art['link']).first():
                db.session.add(Article(**art))
        db.session.commit()
    click.echo('News updated successfully')

Building the Front‑End

SEO matters for a news aggregator because search engines index individual article pages. Use server‑side rendering (SSR) with Jinja2 (Flask) or Django templates to deliver fully‑rendered HTML to crawlers.

Template Structure (Jinja2 Example)

{% extends "base.html" %}
{% block content %}
  <h2>Latest Headlines</h2>
  <ul>
  {% for article in articles %}
    <li>
      <a href="{{ article.link }}" target="_blank">{{ article.title }}</a>
      <small>{{ article.published.strftime('%b %d, %Y %H:%M') }} – {{ article.source }}</small>
    </li>
  {% endfor %}
  </ul>
{% endblock %}

Meta Tags for SEO

Include Open Graph and Twitter Card tags in your <head> section to improve click‑through rates from social platforms.

<meta name="description" content="Free Python news aggregator that pulls the latest headlines from BBC, TechCrunch, and NewsAPI.">
<meta property="og:title" content="Python News Aggregator">
<meta property="og:description" content="Stay updated with curated tech and world news in one place.">
<meta property="og:image" content="{{ url_for('static', filename='logo.png', _external=True) }}">
<meta name="twitter:card" content="summary_large_image">

Adding Search and Category Filters

Enhance user experience by letting visitors search by keyword or filter by category (e.g., Technology, Sports, Politics). Implement a simple query in your view:

def index():
    query = request.args.get('q')
    category = request.args.get('cat')
    articles = Article.query
    if query:
        articles = articles.filter(Article.title.ilike(f'%{query}%'))
    if category:
        articles = articles.filter(Article.source.ilike(f'%{category}%'))
    articles = articles.order_by(Article.published.desc()).limit(50).all()
    return render_template('index.html', articles=articles)

Deploying the Application

For production, consider one of the following deployment strategies:

Docker + Nginx: Containerize the app, serve static files via Nginx, and use Gunicorn (Flask) or Daphne/Uvicorn (Django) as the WSGI server.
Platform‑as‑a‑Service (PaaS): Deploy to Heroku, Render, or Railway with a simple Procfile and environment variables for API keys