feat: add zero-argument douyin target detection
This commit is contained in:
@@ -0,0 +1,154 @@
|
||||
# Douyin Zero-Argument Target Detection Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make `Douyin.py` work with zero arguments by default, auto-detect the current browser page target, and keep a single manual fallback input for creator URLs, video URLs, or `aweme_id`.
|
||||
|
||||
**Architecture:** Add a target-resolution layer ahead of the existing crawl logic. Route resolved targets into either a visible-only creator flow or a single-video flow, keeping browser-attach checks and download primitives reusable.
|
||||
|
||||
**Tech Stack:** Python 3, `argparse`, `re`, `socket`, `pathlib`, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Revise the requirements and freeze the contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `externaldocs/2026-04-17-douyin-targeted-crawling-requirements.md`
|
||||
- Create: `docs/superpowers/specs/2026-04-17-douyin-zero-arg-target-detection-design.md`
|
||||
|
||||
- [ ] **Step 1: Align the requirements doc with the approved UX**
|
||||
|
||||
Document that `./.venv/bin/python Douyin.py` is the primary command and that manual input is fallback-only.
|
||||
|
||||
- [ ] **Step 2: Save the approved design as a spec**
|
||||
|
||||
Write the validated design into `docs/superpowers/specs/2026-04-17-douyin-zero-arg-target-detection-design.md`.
|
||||
|
||||
- [ ] **Step 3: Review both docs locally**
|
||||
|
||||
Read both files and confirm the language matches the agreed zero-argument flow and visible-only scope.
|
||||
|
||||
### Task 2: Add failing tests for target parsing and target resolution
|
||||
|
||||
**Files:**
|
||||
- Modify: `test_douyin.py`
|
||||
- Modify: `Douyin.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Add tests for:
|
||||
|
||||
- `is_creator_url()` accepts supported creator URLs
|
||||
- `is_video_url()` accepts supported video URLs
|
||||
- `is_aweme_id()` accepts numeric IDs
|
||||
- `parse_target_input()` classifies creator URLs, video URLs, and `aweme_id`
|
||||
- `resolve_target()` uses the active browser page when CLI input is absent
|
||||
- `resolve_target()` raises a readable error when neither the current page nor the manual input is supported
|
||||
|
||||
- [ ] **Step 2: Run the focused tests to verify RED**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: FAIL because the new target-resolution helpers do not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write the minimal implementation**
|
||||
|
||||
Implement the smallest set of pure helper functions and a compact parsed-target structure in `Douyin.py`.
|
||||
|
||||
- [ ] **Step 4: Run the focused tests to verify GREEN**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: PASS
|
||||
|
||||
### Task 3: Add failing tests for current-page behavior and visible-only creator flow
|
||||
|
||||
**Files:**
|
||||
- Modify: `test_douyin.py`
|
||||
- Modify: `Douyin.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Add tests for:
|
||||
|
||||
- current-page creator mode does not auto-scroll by default
|
||||
- creator flow reports a clear error when no aweme items are available
|
||||
|
||||
- [ ] **Step 2: Run the focused tests to verify RED**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: FAIL because the current creator flow still scrolls automatically.
|
||||
|
||||
- [ ] **Step 3: Write the minimal implementation**
|
||||
|
||||
Split creator crawling so the default path only processes the currently loaded response set and does not call scroll helpers automatically.
|
||||
|
||||
- [ ] **Step 4: Run the focused tests to verify GREEN**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: PASS
|
||||
|
||||
### Task 4: Add failing tests for single-video flow
|
||||
|
||||
**Files:**
|
||||
- Modify: `test_douyin.py`
|
||||
- Modify: `Douyin.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Add tests for:
|
||||
|
||||
- resolving a video URL leads to a single-video target
|
||||
- resolving an `aweme_id` leads to a single-video target
|
||||
- single-video flow downloads exactly one file
|
||||
|
||||
- [ ] **Step 2: Run the focused tests to verify RED**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: FAIL because single-video execution path does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write the minimal implementation**
|
||||
|
||||
Implement single-video resolution and a narrow download path that saves one mp4 file.
|
||||
|
||||
- [ ] **Step 4: Run the focused tests to verify GREEN**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: PASS
|
||||
|
||||
### Task 5: Update CLI entry behavior and verify end to end
|
||||
|
||||
**Files:**
|
||||
- Modify: `Douyin.py`
|
||||
- Modify: `test_douyin.py`
|
||||
- Modify: `README.md`
|
||||
- Modify: `externaldocs/beginner-guide.md`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Add tests for:
|
||||
|
||||
- default CLI invocation with no positional target chooses current-page resolution
|
||||
- unsupported current page produces a fallback hint
|
||||
- manual positional target overrides the current page
|
||||
|
||||
- [ ] **Step 2: Run the focused tests to verify RED**
|
||||
|
||||
Run: `python3 -m unittest test_douyin.py -q`
|
||||
Expected: FAIL because the CLI still assumes a default hardcoded creator URL.
|
||||
|
||||
- [ ] **Step 3: Write the minimal implementation**
|
||||
|
||||
Update the parser and `main()` flow so zero-argument execution becomes the default, while keeping the manual positional target as fallback.
|
||||
|
||||
- [ ] **Step 4: Update user docs**
|
||||
|
||||
Revise `README.md` and `externaldocs/beginner-guide.md` to show the new default flow:
|
||||
|
||||
```bash
|
||||
./.venv/bin/python login_douyin.py
|
||||
./.venv/bin/python Douyin.py
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run full verification**
|
||||
|
||||
Run: `python3 -m unittest -q`
|
||||
Expected: PASS for the full test suite.
|
||||
@@ -0,0 +1,173 @@
|
||||
# Douyin Zero-Argument Target Detection Design
|
||||
|
||||
## Goal
|
||||
|
||||
将当前抓取入口从“主要依赖手动传 URL 或页码参数”收敛为“默认零参数运行”，让用户在最常见场景下只需要：
|
||||
|
||||
```bash
|
||||
./.venv/bin/python login_douyin.py
|
||||
./.venv/bin/python Douyin.py
|
||||
```
|
||||
|
||||
脚本会优先附着已登录浏览器，并自动识别当前活动标签页是博主主页还是单视频页。
|
||||
|
||||
## Current Context
|
||||
|
||||
- `login_douyin.py` 已能启动带调试端口的可见 Chrome。
|
||||
- `Douyin.py` 已能附着浏览器、监听博主作品列表接口并下载视频。
|
||||
- 当前抓取入口仍偏向“显式传 URL + 可选翻页参数”。
|
||||
- 当前实现默认会继续滚动翻页，这与“只按当前浏览器已加载状态工作”的目标不一致。
|
||||
|
||||
## User Experience
|
||||
|
||||
### Default Flow
|
||||
|
||||
1. 用户运行 `./.venv/bin/python login_douyin.py`
|
||||
2. 用户在 Chrome 里完成抖音登录、验证码，并停留在目标页面
|
||||
3. 用户运行 `./.venv/bin/python Douyin.py`
|
||||
|
||||
### Default Crawl Behavior
|
||||
|
||||
- 如果当前活动标签页是博主主页：
|
||||
- 只抓当前页面已经加载出来的作品
|
||||
- 不自动继续翻多页
|
||||
- 如果当前活动标签页是单视频页：
|
||||
- 只下载这一条视频
|
||||
- 如果当前活动标签页不是受支持页面：
|
||||
- 明确报错
|
||||
- 提示用户手动传入链接或 `aweme_id`
|
||||
|
||||
### Manual Fallback
|
||||
|
||||
用户仅在自动判断失败或不想切换浏览器页面时，才传一个简单目标值：
|
||||
|
||||
```bash
|
||||
./.venv/bin/python Douyin.py "https://www.douyin.com/user/..."
|
||||
./.venv/bin/python Douyin.py "https://www.douyin.com/video/..."
|
||||
./.venv/bin/python Douyin.py "7619989983668240802"
|
||||
```
|
||||
|
||||
## Chosen Approach
|
||||
|
||||
采用“内部显式分类，外部最小参数”的方案：
|
||||
|
||||
- 对用户：
|
||||
- 默认零参数
|
||||
- 保留一个位置参数作为手动兜底入口
|
||||
- 对代码：
|
||||
- 先统一解析输入来源和目标类型
|
||||
- 再分发到“博主页抓取”或“单视频下载”路径
|
||||
|
||||
这样既避免 `--mode --target` 的使用负担，也保留明确的内部边界，便于测试。
|
||||
|
||||
## Internal Model
|
||||
|
||||
建议在 `Douyin.py` 内新增一层目标解析：
|
||||
|
||||
- `creator`
|
||||
- 来源可以是当前活动页 URL 或手动传入的博主页 URL
|
||||
- `single-video`
|
||||
- 来源可以是当前活动页 URL、手动传入的视频 URL 或 `aweme_id`
|
||||
- `unsupported`
|
||||
- 当前页或手动输入都不符合支持规则
|
||||
|
||||
建议新增的小函数边界：
|
||||
|
||||
- `parse_target_input(value: str) -> ParsedTarget`
|
||||
- `resolve_target(page: Any, cli_target: str | None) -> ParsedTarget`
|
||||
- `is_creator_url(url: str) -> bool`
|
||||
- `is_video_url(url: str) -> bool`
|
||||
- `is_aweme_id(value: str) -> bool`
|
||||
- `get_active_page_url(page: Any) -> str`
|
||||
|
||||
## Crawl Paths
|
||||
|
||||
### Creator Path
|
||||
|
||||
- 复用当前 `web/aweme/post/` 监听和解析能力
|
||||
- 默认只处理当前已加载内容
|
||||
- 不再在默认主流程中自动继续滚动翻页
|
||||
- 若页面未加载出可用作品数据，明确提示用户先在浏览器中完成加载或筛选后再重试
|
||||
|
||||
### Single Video Path
|
||||
|
||||
- 若目标是视频 URL，先解析出 `aweme_id`
|
||||
- 若目标本身就是 `aweme_id`，直接进入单视频下载
|
||||
- 下载结果仍写入 `video/`
|
||||
- 最终只落地一个 mp4 文件
|
||||
|
||||
## CLI Design
|
||||
|
||||
### Keep
|
||||
|
||||
- 保留 `--browser-port`
|
||||
- 保留 `--output-dir`
|
||||
- 保留 `--timeout`
|
||||
|
||||
### Change
|
||||
|
||||
- 位置参数从“默认博主页 URL”改为“可选手动目标值”
|
||||
- 默认不再依赖硬编码博主页 URL
|
||||
|
||||
### De-Emphasize
|
||||
|
||||
- `--pages` 不再作为默认主流程的核心参数
|
||||
- 如果保留兼容性，也不应影响零参数主路径的“visible-only”语义
|
||||
|
||||
## Error Handling
|
||||
|
||||
- 浏览器端口不可用：提示先运行 `login_douyin.py` 并确认 Chrome 仍在运行
|
||||
- 当前活动标签页不是受支持页面：提示切到目标页面，或手动传入链接/`aweme_id`
|
||||
- 手动输入既不是博主页 URL、视频 URL，也不是 `aweme_id`：明确报错
|
||||
- 博主页没有加载出作品数据：提示用户先在浏览器页面完成滚动或等待加载
|
||||
- 视频标识无法解析：明确报错
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests First
|
||||
|
||||
第一轮 TDD 先覆盖目标识别和错误路径：
|
||||
|
||||
1. 当前页是博主页 URL 时，识别为 `creator`
|
||||
2. 当前页是单视频页 URL 时，识别为 `single-video`
|
||||
3. 手动传入博主页 URL 时，识别为 `creator`
|
||||
4. 手动传入视频 URL 时，识别为 `single-video`
|
||||
5. 手动传入 `aweme_id` 时，识别为 `single-video`
|
||||
6. 当前页不支持且没传参数时，抛出可读错误
|
||||
7. 手动输入不支持时，抛出可读错误
|
||||
|
||||
第二轮 TDD 再覆盖执行路径：
|
||||
|
||||
1. 博主页默认不自动继续翻页
|
||||
2. 单视频路径最终只下载一条
|
||||
3. 端口不可用时提前失败
|
||||
|
||||
## Implementation Boundaries
|
||||
|
||||
本次实现只做：
|
||||
|
||||
- 零参数目标识别
|
||||
- 当前活动标签页自动判断
|
||||
- 单位置参数兜底输入
|
||||
- 博主页 visible-only 默认行为
|
||||
- 单视频下载路径
|
||||
- 对应自动化测试
|
||||
|
||||
本次不做：
|
||||
|
||||
- 自动登录
|
||||
- 自动搜索博主
|
||||
- 自动筛选页面条件
|
||||
- 抓完整个博主历史内容
|
||||
- 任意网页抓取
|
||||
|
||||
## Success Criteria
|
||||
|
||||
满足以下条件即视为完成：
|
||||
|
||||
1. 用户默认执行 `./.venv/bin/python Douyin.py` 可以按当前活动标签页工作
|
||||
2. 当前页为博主页时，只抓当前已加载内容
|
||||
3. 当前页为单视频页时，只下载这一条
|
||||
4. 当前页不支持时，提示用户手动传链接或 `aweme_id`
|
||||
5. 手动传入博主页 URL、视频 URL 或 `aweme_id` 时可正常工作
|
||||
6. 关键路径都有自动化测试覆盖，并按 TDD 落地
|
||||
Reference in New Issue
Block a user