wangshaoqing/douyin-crawler-poc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

添加学习教程目录(learning/)及对应单元测试

Browse Source 
- 新增 DrissionPage 基础教程(01-05)
- 新增 Playwright 基础教程(01-05)
- 新增网络基础教程(01-05)
- 新增 test_learning_examples.py 单元测试
- 更新 .gitignore 忽略 learning/*/output/ 目录
This commit is contained in:
wangshaoqing
2026-05-06 16:39:55 +08:00
parent 84bcc4ac71
commit 321bbff1c1
21 changed files with 1587 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
learning/playwright_basics/01_open_page.py
+36
View File
@@ -0,0 +1,36 @@
"""
练习 01: 用 Playwright 打开一个网页。
目标:
1. 自己导入 `sync_playwright`
2. 启动一个可见的 Chromium 浏览器
3. 新建一个页面并打开 URL
4. 打印当前页面标题和 URL
建议:
- 第一版先不要封装太多函数
- 先把浏览器正常打开，再补打印信息
- 写完后记得主动关闭浏览器
运行:
./.venv/bin/python learning/playwright_basics/01_open_page.py
"""
DEFAULT_URL = "https://example.com/"
def main() -> None:
from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=False)
page = browser.new_page()
page.goto(DEFAULT_URL)
print(f"[INFO] 已打开 URL: {page.url}")
print(f"[INFO] 页面标题: {page.title()}")
input("按回车退出...")
browser.close()
if __name__ == "__main__":
main()
learning/playwright_basics/02_persistent_context.py
+56
View File
@@ -0,0 +1,56 @@
"""
练习 02: 用 Playwright 持久化浏览器目录保留登录态。
目标:
1. 自己导入 `sync_playwright`
2. 学会 `launch_persistent_context(user_data_dir=...)`
3. 打开一个固定页面后，手动确认这个浏览器目录会被复用
4. 理解为什么不要直接复用你日常 Chrome 默认资料目录
建议:
- 第一版先把浏览器正常启动起来
- `user_data_dir` 建议放在项目目录里，便于观察
- 如果你想验证登录态，连续运行两次并观察 cookie / 登录状态变化
运行:
./.venv/bin/python learning/playwright_basics/02_persistent_context.py
"""
from __future__ import annotations
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
USER_DATA_DIR = PROJECT_ROOT / ".playwright-douyin-profile"
DEFAULT_URL = "https://www.douyin.com/"
def get_or_create_page(context):
if getattr(context, "pages", None):
return context.pages[0]
return context.new_page()
def main() -> None:
from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
context = playwright.chromium.launch_persistent_context(
user_data_dir=str(USER_DATA_DIR),
headless=False,
)
page = get_or_create_page(context)
page.goto(DEFAULT_URL)
print(f"[INFO] 用户目录: {USER_DATA_DIR}")
print(f"[INFO] 当前 URL: {page.url}")
print(f"[INFO] 页面标题: {page.title()}")
input("观察浏览器状态后按回车退出...")
context.close()
if __name__ == "__main__":
main()
learning/playwright_basics/03_wait_and_locate.py
+43
View File
@@ -0,0 +1,43 @@
"""
练习 03: 用 Playwright 等待元素并定位元素。
目标:
1. 练会 `page.locator(...)`
2. 练会 `page.get_by_role(...)`
3. 练会等待元素出现
4. 成功打印一个稳定元素的文本内容
建议:
- 先选一个结构稳定的页面，不要一上来就拿复杂站点练
- 如果定位不到元素，先打印页面标题和 URL，确认你打开的是对的页面
- 第一版优先使用 role / text 这种更直观的定位方式
运行:
./.venv/bin/python learning/playwright_basics/03_wait_and_locate.py
"""
DEFAULT_URL = "https://example.com/"
def main() -> None:
from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
browser = playwright.chromium.launch(headless=False)
page = browser.new_page()
page.goto(DEFAULT_URL)
heading = page.get_by_role("heading", name="Example Domain")
heading.wait_for()
print(f"[INFO] 标题文本: {heading.text_content()}")
first_paragraph = page.locator("p").first
first_paragraph.wait_for()
print(f"[INFO] 第一段文本: {first_paragraph.text_content()}")
input("按回车退出...")
browser.close()
if __name__ == "__main__":
main()
learning/playwright_basics/04_listen_response.py
+81
View File
@@ -0,0 +1,81 @@
"""
练习 04: 用 Playwright 监听响应。
前置:
1. 你已经会用 `02_persistent_context.py` 启动带用户目录的浏览器
2. 最好先手动完成一次抖音登录
3. 打开一个抖音博主主页，或者让脚本自己打开
目标:
1. 学会 `page.on("response", callback)`
2. 学会筛选你关心的接口 URL
3. 打印响应状态码和 URL
4. 尝试读取 JSON 响应体
建议:
- 第一版先只打印 URL，别急着做完整解析
- 如果没看到目标接口，手动滚动页面触发加载
- 你也可以对比试一下 `page.expect_response(...)` 的写法
运行:
./.venv/bin/python learning/playwright_basics/04_listen_response.py
"""
from __future__ import annotations
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from Douyin import DEFAULT_USER_URL
USER_DATA_DIR = PROJECT_ROOT / ".playwright-douyin-profile"
LISTEN_TARGET = "web/aweme/post/"
def is_target_response_url(url: str) -> bool:
return LISTEN_TARGET in url
def try_read_json_payload(response):
try:
payload = response.json()
except Exception:
return None
return payload if isinstance(payload, dict) else None
def main() -> None:
from playwright.sync_api import sync_playwright
def on_response(response) -> None:
if not is_target_response_url(response.url):
return
print(f"[INFO] 命中目标响应: {response.status} {response.url}")
payload = try_read_json_payload(response)
if payload is None:
print("[WARN] 这个响应不是可直接读取的 JSON 字典。")
return
print(f"[INFO] JSON keys: {list(payload.keys())[:10]}")
with sync_playwright() as playwright:
context = playwright.chromium.launch_persistent_context(
user_data_dir=str(USER_DATA_DIR),
headless=False,
)
page = context.pages[0] if context.pages else context.new_page()
page.on("response", on_response)
page.goto(DEFAULT_USER_URL)
print(f"[INFO] 当前 URL: {page.url}")
print("[INFO] 已开始监听响应。请在页面中滚动一下，观察是否出现目标接口。")
input("观察输出后按回车退出...")
context.close()
if __name__ == "__main__":
main()
learning/playwright_basics/05_download_video.py
+87
View File
@@ -0,0 +1,87 @@
"""
练习 05: 用 Playwright 监听到的接口数据配合 requests 下载视频。
目标:
1. 用 Playwright 找到目标接口响应
2. 从 JSON 里提取第一个可下载视频
3. 用 `requests` 把 mp4 写到本地
4. 输出最终文件路径
建议:
- 浏览器负责“拿到页面里的接口数据”
- `requests` 负责“把真实 mp4 下载下来”
- 先拿第一条视频练通，不要一开始就做批量下载
运行:
./.venv/bin/python learning/playwright_basics/05_download_video.py
"""
from __future__ import annotations
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from Douyin import build_headers
from Douyin import DEFAULT_USER_URL
from Douyin import download_video
from Douyin import parse_aweme_items
from Douyin import sanitize_filename
OUTPUT_DIR = Path("learning/playwright_basics/output")
USER_DATA_DIR = PROJECT_ROOT / ".playwright-douyin-profile"
LISTEN_TARGET = "web/aweme/post/"
def build_output_path(title: str, video_id: str, output_dir: Path = OUTPUT_DIR) -> Path:
safe_title = sanitize_filename(title, fallback="playwright-video")
return output_dir / f"{safe_title}-{video_id}.mp4"
def extract_first_item_from_payload(payload) -> dict[str, str]:
items = parse_aweme_items(payload)
if not items:
raise RuntimeError("当前接口里没有可下载视频，请先确认页面已加载出作品。")
return items[0]
def main() -> None:
import requests
from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
context = playwright.chromium.launch_persistent_context(
user_data_dir=str(USER_DATA_DIR),
headless=False,
)
page = context.pages[0] if context.pages else context.new_page()
page.goto(DEFAULT_USER_URL)
print("[INFO] 页面已打开。请在必要时滚动一下，等待作品接口出现。")
with page.expect_response(lambda response: LISTEN_TARGET in response.url, timeout=20000) as response_info:
page.reload()
response = response_info.value
payload = response.json()
first_item = extract_first_item_from_payload(payload)
output_path = build_output_path(
title=first_item["title"],
video_id=first_item["video_id"],
output_dir=OUTPUT_DIR,
)
headers = build_headers(page.url)
download_video(
requests_module=requests,
headers=headers,
video_url=first_item["video_url"],
output_path=output_path,
)
print(f"[OK] 已下载到: {output_path}")
context.close()
if __name__ == "__main__":
main()
learning/playwright_basics/README.md
+119
View File
@@ -0,0 +1,119 @@
# Playwright Basics
这个目录是你的第二阶段练习。
目标不是立刻重写整个项目，而是先把 `Playwright Python` 最常用的几个能力拆开练一遍，再回头和现在的 `DrissionPage` 实现对照。
这里的文件故意保留了 `TODO`，你需要自己把它们补完整。
## 建议学习顺序
1. `01_open_page.py`
2. `02_persistent_context.py`
3. `03_wait_and_locate.py`
4. `04_listen_response.py`
5. `05_download_video.py`
## 开始前先准备
在项目根目录执行：
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install requests playwright
./.venv/bin/python -m playwright install chromium
```
## 练习原则
- 不要一开始就追求“项目可直接替换”
- 每次只补一个文件
- 跑起来后优先 `print()` 看页面对象、locator、response、json 数据长什么样
- 先写最小可运行版本，再慢慢整理函数
- 如果遇到元素定位失败，先确认页面是不是你以为的那个页面
## 每个文件你应该关注什么
### `01_open_page.py`
- 学会 `sync_playwright()`
- 学会 `browser = playwright.chromium.launch(...)`
- 学会 `page.goto(url)`
- 学会打印标题和 URL
运行：
```bash
./.venv/bin/python learning/playwright_basics/01_open_page.py
```
### `02_persistent_context.py`
- 学会 `launch_persistent_context(user_data_dir=...)`
- 理解“浏览器用户目录”和“保留登录态”的关系
- 学会复用同一个 Playwright 浏览器资料目录
运行：
```bash
./.venv/bin/python learning/playwright_basics/02_persistent_context.py
```
### `03_wait_and_locate.py`
- 学会 `locator(...)`
- 学会 `get_by_role(...)`
- 学会等待元素出现后再读文本
- 学会先确认页面结构，再决定怎么定位
运行：
```bash
./.venv/bin/python learning/playwright_basics/03_wait_and_locate.py
```
### `04_listen_response.py`
- 学会 `page.on("response", ...)`
- 学会只关注你想看的接口 URL
- 学会打印 `response.status`、`response.url`
- 学会尝试 `response.json()` 看接口结构
运行：
```bash
./.venv/bin/python learning/playwright_basics/04_listen_response.py
```
### `05_download_video.py`
- 学会把 Playwright 监听到的接口 JSON 接到下载逻辑
- 学会把浏览器控制和文件下载拆成两段
- 学会把结果保存到 `learning/playwright_basics/output/`
运行：
```bash
./.venv/bin/python learning/playwright_basics/05_download_video.py
```
## 推荐你的手动练法
1. 先补 `01`，只做到“能打开页面”
2. 再补 `02`，只做到“能复用同一个用户目录”
3. 再补 `03`，只做到“能定位到一个稳定元素并打印文本”
4. 再补 `04`，只做到“能看到目标响应并打印部分 JSON”
5. 最后补 `05`，把一个真实视频链接下载到本地
## 练完后建议你对照这些内容
- [Douyin.py](/Users/wangshaoqing/Desktop/MiaoSi/Study/douyin-crawler-poc/Douyin.py)
- [learning/drissionpage_basics/03_listen_api.py](/Users/wangshaoqing/Desktop/MiaoSi/Study/douyin-crawler-poc/learning/drissionpage_basics/03_listen_api.py)
- [learning/drissionpage_basics/05_download_video.py](/Users/wangshaoqing/Desktop/MiaoSi/Study/douyin-crawler-poc/learning/drissionpage_basics/05_download_video.py)
你重点对照的是：
- `DrissionPage` 的附着方式和 `Playwright` 的持久化上下文有什么差别
- `page.listen.wait(...)` 和 `page.on("response", ...)` / `expect_response(...)` 的心智模型有什么差别
- 为什么下载文件时经常还是会回到 `requests`