feat: initial douyin crawler
This commit is contained in:
@@ -0,0 +1,198 @@
|
||||
# Douyin Login Entry Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a dedicated browser-login launcher and a clearer attach-port check so the Douyin crawler has a stable two-step workflow: login first, crawl second.
|
||||
|
||||
**Architecture:** Keep browser-launch responsibilities in a new `login_douyin.py` script and keep crawl responsibilities in `Douyin.py`. Add a small socket-based port readiness check before attaching to Chrome, and cover the new behavior with unit tests before implementing production code.
|
||||
|
||||
**Tech Stack:** Python 3, `argparse`, `pathlib`, `subprocess`, `socket`, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Write failing tests for the new login launcher
|
||||
|
||||
**Files:**
|
||||
- Create: `login_douyin.py`
|
||||
- Create: `test_login_douyin.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
def test_build_login_command_uses_expected_chrome_arguments(self) -> None:
|
||||
module = importlib.import_module("login_douyin")
|
||||
command = module.build_login_command(
|
||||
chrome_path="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
profile_dir=Path("/tmp/douyin-profile"),
|
||||
browser_port=9223,
|
||||
user_url="https://www.douyin.com/user/example",
|
||||
)
|
||||
self.assertEqual(
|
||||
command,
|
||||
[
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
"--user-data-dir=/tmp/douyin-profile",
|
||||
"--remote-debugging-port=9223",
|
||||
"https://www.douyin.com/user/example",
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_login_douyin.py -v`
|
||||
Expected: FAIL because `login_douyin.py` does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Create `login_douyin.py` with:
|
||||
|
||||
- `DEFAULT_CHROME_PATH`
|
||||
- `DEFAULT_BROWSER_PORT = 9223`
|
||||
- `DEFAULT_PROFILE_DIR`
|
||||
- `build_login_command(...)`
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_login_douyin.py -v`
|
||||
Expected: PASS for the command-building test.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
Not applicable here because the workspace is not a git repository.
|
||||
|
||||
### Task 2: Add tests and implementation for launcher validation and user guidance
|
||||
|
||||
**Files:**
|
||||
- Modify: `login_douyin.py`
|
||||
- Modify: `test_login_douyin.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Add tests for:
|
||||
|
||||
- parser defaults use `9223`
|
||||
- `main()` creates the profile dir
|
||||
- `main()` prints the follow-up crawl command
|
||||
- `main()` returns non-zero with a readable message when the Chrome path does not exist
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_login_douyin.py -v`
|
||||
Expected: FAIL because validation and guidance behavior is not implemented yet.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `login_douyin.py`:
|
||||
|
||||
- `build_parser()`
|
||||
- `launch_browser(...)`
|
||||
- `main(...)`
|
||||
- readable `SystemExit`/stderr-style messaging through printed output and return codes
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_login_douyin.py -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
Not applicable here because the workspace is not a git repository.
|
||||
|
||||
### Task 3: Write failing tests for attach-port readiness in the crawler
|
||||
|
||||
**Files:**
|
||||
- Modify: `Douyin.py`
|
||||
- Modify: `test_douyin.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Add tests for:
|
||||
|
||||
- `ensure_browser_debug_port_ready()` returns successfully when a temporary local server is listening
|
||||
- `ensure_browser_debug_port_ready()` raises a readable `RuntimeError` when the port is unavailable
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_douyin.py -v`
|
||||
Expected: FAIL because the function does not exist yet.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `Douyin.py`:
|
||||
|
||||
- socket-based readiness helper
|
||||
- call it in `collect_videos()` before `create_page(...)` when `browser_port` is provided
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_douyin.py -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
Not applicable here because the workspace is not a git repository.
|
||||
|
||||
### Task 4: Update usage documentation
|
||||
|
||||
**Files:**
|
||||
- Modify: `抖音爬取视频.md`
|
||||
|
||||
- [ ] **Step 1: Write the failing doc expectation**
|
||||
|
||||
Define the required doc updates:
|
||||
|
||||
- explicit step 1 command for `login_douyin.py`
|
||||
- explicit step 2 command for `Douyin.py --browser-port 9223`
|
||||
- short note that login state is kept in the dedicated profile dir
|
||||
|
||||
- [ ] **Step 2: Verify current doc is incomplete**
|
||||
|
||||
Run: `rg -n "login_douyin.py|--browser-port 9223" 抖音爬取视频.md`
|
||||
Expected: no matches or incomplete guidance
|
||||
|
||||
- [ ] **Step 3: Write minimal documentation update**
|
||||
|
||||
Append a short “推荐流程” section to `抖音爬取视频.md`.
|
||||
|
||||
- [ ] **Step 4: Verify the doc contains the new commands**
|
||||
|
||||
Run: `rg -n "login_douyin.py|--browser-port 9223" 抖音爬取视频.md`
|
||||
Expected: matches for both commands
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
Not applicable here because the workspace is not a git repository.
|
||||
|
||||
### Task 5: Run full verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `Douyin.py`
|
||||
- Modify: `login_douyin.py`
|
||||
- Modify: `test_douyin.py`
|
||||
- Modify: `test_login_douyin.py`
|
||||
- Modify: `抖音爬取视频.md`
|
||||
|
||||
- [ ] **Step 1: Run the full unit test suite**
|
||||
|
||||
Run: `./.venv/bin/python -m unittest test_douyin.py test_login_douyin.py -v`
|
||||
Expected: all tests pass
|
||||
|
||||
- [ ] **Step 2: Run the login launcher manually**
|
||||
|
||||
Run: `./.venv/bin/python login_douyin.py --browser-port 9223`
|
||||
Expected: visible Chrome launches and prints the next crawl command
|
||||
|
||||
- [ ] **Step 3: Run the crawler against the logged-in browser**
|
||||
|
||||
Run: `./.venv/bin/python Douyin.py --pages 1 --timeout 20 --browser-port 9223`
|
||||
Expected: videos are downloaded to `video/`
|
||||
|
||||
- [ ] **Step 4: Review changed files for scope drift**
|
||||
|
||||
Run: `rg --files`
|
||||
Expected: only the planned files changed or were added
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
Not applicable here because the workspace is not a git repository.
|
||||
@@ -0,0 +1,191 @@
|
||||
# Douyin Login Entry Design
|
||||
|
||||
## Goal
|
||||
|
||||
将当前“手动先开浏览器登录，再让 `Douyin.py` 附着到调试端口抓取”的隐式流程，固化为稳定、明确、可复用的两步式命令行入口。
|
||||
|
||||
## Current Context
|
||||
|
||||
- 现有抓取实现位于 `Douyin.py`。
|
||||
- `Douyin.py` 已支持通过 `--browser-port` 附着到已启动的 Chrome 调试端口。
|
||||
- 本次实测已经证明：用户先在可见 Chrome 中登录抖音并通过验证码后，`Douyin.py --browser-port 9223` 可以成功抓到 `web/aweme/post/` 接口并下载视频。
|
||||
- 当前缺少一个明确的“登录准备入口”，导致可操作性依赖人工记忆和临时命令。
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional
|
||||
|
||||
1. 提供一个独立脚本，用于启动可见 Chrome，并固定：
|
||||
- 调试端口，默认 `9223`
|
||||
- 用户数据目录，默认使用一个项目约定路径
|
||||
- 打开的初始 URL，默认指向现有抖音博主页
|
||||
2. 登录脚本只负责“打开浏览器并提示用户手动登录”，不负责抓取。
|
||||
3. `Douyin.py` 继续负责抓取，并保持“附着已有浏览器”的职责边界。
|
||||
4. 当 `Douyin.py` 指定了 `--browser-port` 但端口不可连通时，应给出清晰错误，提示先运行登录脚本。
|
||||
5. 文档应给出最短可执行流程：
|
||||
- 第一步：启动浏览器并登录
|
||||
- 第二步：运行抓取命令
|
||||
|
||||
### Non-Functional
|
||||
|
||||
1. 不改变现有抓包、解析、下载的主逻辑。
|
||||
2. 保持现有命令参数兼容。
|
||||
3. 入口职责清晰，便于排查“登录问题”和“抓取问题”。
|
||||
4. 新增行为应具备可自动化测试的核心单元。
|
||||
|
||||
## Chosen Approach
|
||||
|
||||
采用双脚本方案：
|
||||
|
||||
- 新增 `login_douyin.py`
|
||||
- 负责启动可见 Chrome
|
||||
- 固定 remote debugging port
|
||||
- 固定 profile 目录
|
||||
- 打开目标用户主页
|
||||
- 输出明确提示，引导用户完成手动登录和验证码
|
||||
- 保留 `Douyin.py`
|
||||
- 继续承担附着浏览器、监听接口、下载视频的职责
|
||||
- 增强附着前检查与报错信息
|
||||
|
||||
## Rejected Alternatives
|
||||
|
||||
### Alternative 1: 将“启动浏览器”直接并入 `Douyin.py`
|
||||
|
||||
不采用。原因：
|
||||
|
||||
- 会让 `Douyin.py` 同时承担登录准备和抓取职责。
|
||||
- 错误定位会变差，用户更难区分是登录失败还是抓取失败。
|
||||
- 未来若需要“先登录、稍后再抓”，这种合并入口不灵活。
|
||||
|
||||
### Alternative 2: 只写 shell 脚本串联所有步骤
|
||||
|
||||
不采用。原因：
|
||||
|
||||
- 逻辑容易散落在 shell 中，测试性差。
|
||||
- 浏览器启动参数、等待逻辑和抓取命令耦合度高。
|
||||
- 后续若要扩展默认参数或跨平台兼容，shell 方案维护成本更高。
|
||||
|
||||
## Proposed CLI UX
|
||||
|
||||
### Step 1: 启动登录浏览器
|
||||
|
||||
```bash
|
||||
./.venv/bin/python login_douyin.py
|
||||
```
|
||||
|
||||
默认行为：
|
||||
|
||||
- 启动可见 Chrome
|
||||
- 调试端口为 `9223`
|
||||
- profile 目录为项目约定的本地路径
|
||||
- 打开默认的抖音主页 URL
|
||||
- 输出“请在浏览器中完成登录/验证码，然后再运行抓取命令”
|
||||
|
||||
可选扩展参数：
|
||||
|
||||
- `--browser-port`
|
||||
- `--profile-dir`
|
||||
- `--user-url`
|
||||
- `--chrome-path`
|
||||
|
||||
### Step 2: 运行抓取
|
||||
|
||||
```bash
|
||||
./.venv/bin/python Douyin.py --pages 1 --browser-port 9223
|
||||
```
|
||||
|
||||
## Design Details
|
||||
|
||||
### `login_douyin.py`
|
||||
|
||||
建议拆分为可测试的小函数：
|
||||
|
||||
- `build_login_command(...)`
|
||||
- 输入 Chrome 路径、profile 目录、端口、URL
|
||||
- 输出适合 `subprocess.Popen(...)` 的参数列表
|
||||
- `launch_browser(...)`
|
||||
- 调用 `subprocess.Popen(...)`
|
||||
- `build_parser()`
|
||||
- 定义 CLI 参数
|
||||
- `main()`
|
||||
- 解析参数
|
||||
- 启动浏览器
|
||||
- 打印下一步指引
|
||||
|
||||
### `Douyin.py`
|
||||
|
||||
新增一个显式的端口检查函数，例如：
|
||||
|
||||
- `ensure_browser_debug_port_ready(browser_port: int) -> None`
|
||||
|
||||
行为：
|
||||
|
||||
- 仅当用户传入 `--browser-port` 时执行
|
||||
- 尝试连接 `127.0.0.1:<port>`
|
||||
- 若失败，抛出清晰错误，提示：
|
||||
- 先启动 `login_douyin.py`
|
||||
- 确认 Chrome 仍在运行
|
||||
- 确认端口与抓取命令一致
|
||||
|
||||
## Error Handling
|
||||
|
||||
### 登录脚本
|
||||
|
||||
- Chrome 可执行文件不存在：直接报错并退出。
|
||||
- 浏览器启动失败：输出异常原因并返回非零退出码。
|
||||
- profile 目录不存在：自动创建。
|
||||
|
||||
### 抓取脚本
|
||||
|
||||
- 指定 `--browser-port` 但端口不可达：立即失败，不进入抓取流程。
|
||||
- 登录未完成导致页面异常：保留现有抓包等待与警告逻辑。
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
新增或扩展 `test_douyin.py`，覆盖：
|
||||
|
||||
1. `build_login_command()` 生成的命令参数正确。
|
||||
2. 默认调试地址仍为 `127.0.0.1:<port>`。
|
||||
3. `ensure_browser_debug_port_ready()` 在端口不可达时抛出可读错误。
|
||||
4. `ensure_browser_debug_port_ready()` 在端口可达时正常返回。
|
||||
|
||||
如测试边界过大，可新增 `test_login_douyin.py`。
|
||||
|
||||
### Manual Verification
|
||||
|
||||
1. 运行 `./.venv/bin/python login_douyin.py`
|
||||
2. 在打开的 Chrome 中登录抖音并通过验证码
|
||||
3. 运行 `./.venv/bin/python Douyin.py --pages 1 --browser-port 9223`
|
||||
4. 确认 `video/` 下生成新的 mp4 文件
|
||||
|
||||
## Implementation Boundaries
|
||||
|
||||
本次只做以下改动：
|
||||
|
||||
- 新增登录入口脚本
|
||||
- 为抓取入口补充附着前端口检查
|
||||
- 更新测试
|
||||
- 更新使用文档
|
||||
|
||||
本次不做以下改动：
|
||||
|
||||
- 不重写抓取主流程
|
||||
- 不改成单命令自动等待登录
|
||||
- 不引入 Playwright 作为正式运行时依赖
|
||||
- 不增加下载调度、断点续传或批量任务管理
|
||||
|
||||
## Risks
|
||||
|
||||
1. 本机 Chrome 路径可能与预设不同，因此需要保留 `--chrome-path` 覆盖能力。
|
||||
2. profile 目录固定后，用户可能重复复用登录态，这是预期行为，但文档需说明。
|
||||
3. 若目标端口被其他进程占用，登录脚本需要给出可诊断的失败信息或允许端口覆盖。
|
||||
|
||||
## Success Criteria
|
||||
|
||||
满足以下条件即视为完成：
|
||||
|
||||
1. 用户可以通过固定命令启动登录浏览器。
|
||||
2. 用户登录完成后，可通过固定命令让 `Douyin.py` 成功附着并抓取。
|
||||
3. 当浏览器未启动或端口错误时，抓取脚本会给出明确提示，而不是模糊失败。
|
||||
Reference in New Issue
Block a user