Add XHS keyword search source

2026-05-27 16:49:36 +08:00
parent 37b17d8ccf
commit f247cb1a3a
6 changed files with 159 additions and 7 deletions
@@ -0,0 +1,24 @@
+# XHS Search Source Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add `--source search --keyword <term>` to the resumable queue downloader.
+
+**Architecture:** Extend `build_source_url`, CLI parser choices, `run_queue_download` arguments, and README examples. Reuse all queue and download code.
+
+**Tech Stack:** Python 3, unittest, DrissionPage, requests.
+
+---
+
+## Task 1: Search URL and CLI
+
+- [x] Write failing tests for encoded search source URL and CLI keyword plumbing.
+- [x] Implement `build_source_url("search", keyword=...)`, parser support, and queue runner forwarding.
+- [x] Run tests.
+
+## Task 2: Docs and Smoke
+
+- [x] Update README with search examples.
+- [x] Run full unit tests.
+- [x] Run a small search smoke test with `--keyword 猫咪 --target-videos 2`.
+- [ ] Commit and push.
@@ -0,0 +1,27 @@
+# XHS Search Source Design
+
+## Goal
+
+Allow the resumable queue downloader to use Xiaohongshu search results as a source, so queries such as `猫咪` or `猫咪 搞笑` can collect and download related video notes.
+
+## Scope
+
+This feature reuses the existing manually logged-in Chrome, queue persistence, page card collection, detail-page video extraction, validation, and human browsing cadence. It does not automate login, bypass verification, or call hidden APIs directly.
+
+## CLI
+
+```bash
+./.venv/bin/python XHS.py --source search --keyword 猫咪 --target-videos 100 --queue-file data/search_cat_queue.jsonl
+```
+
+## Behavior
+
+- `--source search` requires `--keyword`.
+- The source URL is `https://www.xiaohongshu.com/search_result?keyword=<encoded keyword>&source=web_search_result_notes&type=51`, which opens the video-filtered search results page.
+- Search result cards are collected from both `/explore/<note_id>` and tokenized `/search_result/<note_id>` links.
+- Detail links are polled briefly after navigation because Xiaohongshu search result cards are rendered asynchronously.
+- Queue mode handles videos, images, failures, retries, and resume semantics exactly like other sources.
+
+## Testing
+
+Unit tests cover search URL encoding, parser defaults, queue-mode CLI plumbing for keyword, `/search_result/` note ID extraction, tokenized search link normalization, and async result-link polling.