Skip to content

feat: richText resource extraction and download_file_to_file - #4

Merged
haofeng0705 merged 3 commits into
DingTalk-Real-AI:mainfrom
typefield:feat/richtext-resources-and-download-to-file
Sep 20, 2026
Merged

haofeng0705 merged 3 commits into
DingTalk-Real-AI:mainfrom
typefield:feat/richtext-resources-and-download-to-file

Conversation

@typefield

Copy link
Copy Markdown
Contributor

为钉钉 richText 消息补齐两项媒体能力:

1. richText 附件资源提取

richText 消息中的 picture/file 段此前被解析器直接丢弃,现在提取进 IncomingMessage.resources:

  • picture 段值即钉钉下载码 → {"type": "image", "downloadCode": ...}
  • file 段(downloadCode + fileName)→ {"type": "file", ...}
  • 脏数据防御:段值非字符串(isinstance 严格判断)或下载码为空时跳过该段,不影响其余段落;同一下载码单条消息内去重

2. DingTalkChannel.download_file_to_file(流式落盘)

流式下载文件到本地路径,不整块载入内存(64 KiB 分块):

  • SSRF 防护与 download_file 完全一致(ssrf_allowlist 白名单语义不变)
  • 同目录临时文件 + 原子 os.replace,失败不落半截文件
  • 父目录必须已存在(否则 FileNotFoundError);返回写入字节数

验证

pytest:113 个测试全部通过(含新增:资源提取/去重/脏数据、流式落盘成功/父目录缺失/SSRF 拦截)。

与 #2 互不冲突,可任意顺序合并。

typefield and others added 2 commits September 16, 2026 23:44
- normalize: richText picture/file segments now extract into
  IncomingMessage.resources. Defensive narrowing via isinstance checks;
  unknown/dirty segments are skipped without affecting the rest;
  download codes deduped per message.
- DingTalkChannel.download_file_to_file: streaming download to a local
  path (64 KiB chunks, no whole-file buffering), SSRF-guarded like
  download_file, same-dir temp file + atomic os.replace, parent dir must
  exist, no partial file left behind. Returns bytes written.
- Tests: resource extraction/dedup/dirty-data; streaming download success,
  missing-parent-dir, and SSRF-block regressions.
…m lark channel-sdk

- normalize: richText picture/file segments now extract into
  IncomingMessage.resources (analog of lark's post attachment zone).
  Defensive narrowing via isinstance checks; unknown/dirty segments are
  skipped without affecting the rest; download codes deduped per message.
- DingTalkChannel.download_file_to_file: streaming download to a local
  path (64 KiB chunks, no whole-file buffering), SSRF-guarded like
  download_file, same-dir temp file + atomic os.replace, parent dir must
  exist, no partial file left behind. Returns bytes written.
- Tests: resource extraction/dedup/dirty-data; streaming download success,
  missing-parent-dir, and SSRF-block regressions.

@haofeng0705 haofeng0705 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复查结论:请求修改,当前不建议合入。

  1. 阻塞 normalize/converters/richtext.py:33 从 item[\"picture\"] 读取 richText 图片下载码;真实回调使用 downloadCode / pictureDownloadCode,当前测试使用了错误字段。
  2. 阻塞 channel.py:231-245 只校验初始 URL,urlopen 自动跟随重定向,重定向目标未重新校验;已复现公网 URL 重定向后读取回环地址。
  3. 阻塞 channel.py:247-252 未验证 Content-Length/响应完整性,服务提前 EOF 时仍会用截断文件覆盖原目标;已复现。
  4. channel.py:264 中取消协程不会停止 to_thread 下载,调用方收到 CancelledError 后后台线程仍可能覆盖目标文件。
  5. 项目声明支持 Python 3.8,但 asyncio.to_thread 需要 Python 3.9+。

正常下载和 HTTP 500 路径已做定向验证;完整 pytest 因基线缺少 websockets 依赖未能执行。请修复以上问题并补充对应回归测试后再合入。

…le_to_file

- Support downloadCode and pictureDownloadCode with fallback to picture for richText images
- Use POST with JSON body for messageFiles/download in reply
- Prevent SSRF redirect bypass with custom HTTPRedirectHandler
- Check Content-Length to prevent overwriting with truncated files on early EOF
- Add cancellation check to prevent background thread overwriting target
- Provide to_thread compatibility helper for Python 3.8
- Add websockets to dependencies
- Add GitHub Actions CI workflow

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@haofeng0705 haofeng0705 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复查通过。

  • richText 图片字段已兼容 downloadCode / pictureDownloadCode / picture(src/dingtalk_channel_sdk/normalize/converters/richtext.py:32-37)。
  • SSRF 重定向绕过已修复:_SSRFSafeRedirectHandler.redirect_request 逐跳校验(src/dingtalk_channel_sdk/channel.py:46-53),测试覆盖。
  • 截断响应检测:流式下载在 Content-Length 存在时校验实际写入字节数(src/dingtalk_channel_sdk/channel.py:270-286)。
  • 取消后覆盖问题已修复:通过 threading.Event 在写入前后检查取消状态(src/dingtalk_channel_sdk/channel.py:256-306)。
  • Python 3.8 兼容性:新增 compat.to_thread 替代 asyncio.to_thread(src/dingtalk_channel_sdk/compat.py)。
  • pyproject.toml 已补齐 websockets 依赖和 pytest-asyncio dev 依赖。
  • 新增 CI workflow(Python 3.8-3.12)。
  • pytest 本地 116/116 通过。

可合入。

@haofeng0705
haofeng0705 merged commit 0b55603 into DingTalk-Real-AI:main Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants