Commit Graph

6 Commits

Author SHA1 Message Date
JamesFlare1212
fb68c1ad5d refactor(scan): remove multi-thread scan logic, use sequential processing 2026-04-08 12:04:27 -04:00
JamesFlare1212
0a133159e8 重构 scan: 实现多线程并发爬虫功能
- 新增 Semaphore 信号量类控制并发数
- 新增 BatchProcessor 批量处理器带进度回调
- 重构 initializeClubCache 和 updateStaleClubs 为并发模式
- 修复 Cookie 4xx 判断逻辑(仅 401/403 触发重新登录)
- 添加环境变量配置:CONCURRENT_API_CALLS 等
- 新增并发功能测试脚本 test-concurrency.ts

性能提升:从串行处理提升至可配置的并发处理(默认 8 线程)
修复问题:404 错误不再误判为认证失败
2026-04-07 18:18:18 -04:00
JamesFlare1212
32dee6b161 fix(cache): resolve scanning stop issue and add cache TTL management
- Fix Redis SCAN cursor type conversion (Buffer to String) to prevent early termination
- Add progress logging in initializeClubCache (every 100 activities with summary)
- Add Redis memory limits (512MB with LRU eviction policy)
- Implement cache TTL: 24h for normal data, 1h for error states (allows retry)
- Fix Docker permission issue by running app container as root
- Add TTL configuration to .env and example.env

Root cause: SCAN cursor comparison failed due to type mismatch (Buffer vs String)
Impact: Scanning now processes all 5000+ IDs instead of stopping at ~300
2026-04-06 21:03:30 -04:00
JamesFlare1212
4a97057825 feat: 添加可选的代理功能支持
新增功能:
- 集成 Cloudflare WARP socks5 代理服务
- 通过环境变量 USE_PROXY 控制代理开关
- 支持自定义 HTTP/HTTPS/SOCKS5 代理服务器
- 使用 docker compose profile 管理 proxy 服务

配置方式:
- USE_PROXY=true 启用代理
- ALL_PROXY/HTTP_PROXY/HTTPS_PROXY 自定义代理
- docker compose --profile proxy up 启动 warp 服务

文件变更:
- docker-compose.yaml: 添加 warp-proxy 服务
- playwright-auth.ts: 添加代理配置逻辑
- example.env: 添加代理环境变量
- PROXY.md: 使用文档
2026-04-06 16:37:54 -04:00
JamesFlare1212
b18b8a85e0 feat new s3 public url option 2026-03-15 19:40:59 -04:00
JamesFlare1212
2543e56ec4 init: port to typescript and bun 2025-05-10 23:39:39 -04:00