Hawker

Fetches articles from 24 curated RSS feeds spanning research labs (OpenAI, DeepMind, Meta AI), developer blogs (Hugging Face, PyTorch), AI newsletters, and tech publications (TechCrunch, MIT News, Wired).

Each feed is parsed with feedparser, then full article text is extracted using newspaper3k with 8 parallel workers. Articles older than 7 days and those that have been crawled in previous runs are dropped before extraction to save bandwidth.

Each article is scored against a curated list of ~50 AI-related terms (machine learning, LLM, transformer, computer vision, etc.). Articles matching at least one keyword pass through.

The keyword score — ratio of distinct matched terms to total terms — is saved and later used as one of three ranking signals. Typical pass rate is ~70%.

Extracts key entities (PERSON, ORG, GPE) using spaCy's en_core_web_sm model from the cleaned article text.

Entities are stored as tags, allowing us to identify the main actors in every story (e.g., 'Sam Altman', 'Google', 'NVIDIA') and use them for topic labeling.

Articles are converted into 384-dim vectors with MiniLM-L6-v2 and grouped into clusters using scikit-learn's K-Means.

Clustering groups semantically similar stories into 'Trending Topics'. Topics are dynamically labeled by looking at the most frequent entities inside each cluster.

Normalized embeddings and topic associations are stored in PostgreSQL using the pgvector extension with an HNSW index.

The system supports both vector searching (for queries) and relational filtering (for topic chips), combined into a single performant pipeline.