Add resumable XHS queue downloader

2026-05-27 16:30:06 +08:00
parent ed8357f65a
commit 37b17d8ccf
6 changed files with 480 additions and 0 deletions
@@ -0,0 +1,45 @@
+# XHS Long Queue Downloader Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add a resumable JSONL queue mode so long Xiaohongshu video download jobs can target large counts like 1000 videos.
+
+**Architecture:** Keep `XHS.py` as the CLI entry point. Add queue record helpers, source URL helpers, discovery/processing orchestration, and CLI flags while reusing existing parsing, download validation, shared Chrome, and human browsing cadence.
+
+**Tech Stack:** Python 3, unittest, JSONL files, DrissionPage, requests.
+
+---
+
+## File Structure
+
+- Modify `XHS.py`: queue dataclass/helpers, source selection, queue orchestration, CLI flags.
+- Modify `test_xhs.py`: queue unit tests and CLI plumbing tests.
+- Modify `README.md`: long task command examples.
+
+## Task 1: Queue Persistence
+
+- [ ] Write tests for queue load/save, deduping by note_id, counting downloaded records, and status updates.
+- [ ] Run `python3 -m unittest test_xhs.py -v` and verify failures.
+- [ ] Implement `QueueRecord`, `load_queue`, `save_queue`, `merge_note_urls_into_queue`, `count_queue_status`.
+- [ ] Run tests and verify pass.
+
+## Task 2: Source Selection and CLI
+
+- [ ] Write tests for `build_source_url` and parser defaults for `--source`, `--target-videos`, `--queue-file`, `--retry-limit`.
+- [ ] Run tests and verify failures.
+- [ ] Implement source URL selection and CLI argument plumbing.
+- [ ] Run tests and verify pass.
+
+## Task 3: Queue Processing Orchestration
+
+- [ ] Write tests for pure queue status transitions for success, skipped image, failed retry.
+- [ ] Run tests and verify failures.
+- [ ] Implement queue processing helpers and wire queue mode into `main` when `--queue-file` or `--target-videos` is provided.
+- [ ] Run tests and verify pass.
+
+## Task 4: Docs and Verification
+
+- [ ] Update README with 1000-video queue command and resume behavior.
+- [ ] Run `python3 -m unittest test_xhs.py test_login_xhs.py -v`.
+- [ ] Run a small smoke command with low target and short waits if browser is available.
+- [ ] Commit and push.
@@ -0,0 +1,62 @@
+# XHS Long Queue Downloader Design
+
+## Goal
+
+Add a resumable long-task downloader for collecting large numbers of Xiaohongshu videos, such as 1000 videos, without relying on a single recommendation page pass.
+
+## Scope
+
+The feature stays within the existing manually logged-in browser model. It does not automate login, bypass verification, spoof device fingerprints, or call private APIs directly outside what the loaded web pages expose. It improves task durability, source density, and progress tracking.
+
+## Architecture
+
+The downloader becomes two-phase while preserving the current one-command UX:
+
+1. Queue discovery collects note detail URLs from configured sources and writes them to a JSONL queue.
+2. Queue processing opens pending note URLs, extracts video URLs from page state or feed responses, downloads valid videos, and updates each queue item status.
+
+The queue file stores one JSON object per note:
+
+```json
+{"note_id":"...","url":"...","source":"video-channel","status":"pending","attempts":0,"downloaded_path":"","last_error":"","updated_at":"..."}
+```
+
+Statuses are `pending`, `downloaded`, `skipped_image`, and `failed`.
+
+## Sources
+
+The first implementation supports:
+
+- `explore`: current recommendation page.
+- `video-channel`: `https://www.xiaohongshu.com/explore?channel_id=video` as a best-effort source. If Xiaohongshu redirects or changes channel routing, the collector still reads visible `/explore/` cards.
+- `current-page`: process the current browser page.
+
+Future search keyword sources can be added after the queue engine is stable.
+
+## Runtime Behavior
+
+A command such as:
+
+```bash
+./.venv/bin/python XHS.py --source video-channel --target-videos 1000 --queue-file data/xhs_queue.jsonl --max-runtime 7200
+```
+
+will:
+
+1. Load existing queue records.
+2. Count already downloaded items.
+3. Open the selected source page and collect visible note URLs.
+4. Append new pending records, preserving existing statuses.
+5. Process pending records until `target_videos`, `max_runtime`, or queue exhaustion.
+6. If queue is exhausted before target, return to source, scroll, collect more URLs, and continue.
+
+## Error Handling
+
+- Non-video notes become `skipped_image`.
+- Download failures increment attempts and become `failed` after retry limit.
+- The queue is rewritten atomically after status changes.
+- Progress logs include downloaded count, skipped count, failed count, and pending count.
+
+## Testing
+
+Unit tests cover JSONL queue load/save, deduplication, status updates, source URL selection, target counting, and CLI argument plumbing. Existing download and parsing tests remain in place.