Skip to content

feat(fetchers): add public social media metadata support - #188

Merged
chaliy merged 2 commits into
mainfrom
codex/social-fetchers
Oct 28, 2026
Merged

chaliy merged 2 commits into
mainfrom
codex/social-fetchers

Conversation

@chaliy

@chaliy chaliy commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Fetch public TikTok and Vimeo videos and SoundCloud tracks/playlists as bounded Markdown metadata through their official oEmbed endpoints. The shared registry enables this automatically in Rust, CLI, MCP and Python. YouTube now recognizes Shorts, live, embed and privacy-enhanced embed URLs alongside watch/short links.

Titles, authors, thumbnails and descriptions appear when supplied; descriptions become Markdown. Embed HTML is ignored. Output explicitly states that media/transcripts are not retrieved by the new fetcher. TikTok currently supplies only minimal embed metadata, so it can return the platform and canonical URL without a title. Existing YouTube captions remain best-effort.

Why

Social pages commonly need JavaScript or show login/navigation shells. Public oEmbed metadata provides useful structured output without credentials or a browser. Supported URL matching stays narrow; unsupported pages retain the default HTML fetcher.

Before / After

Before: TikTok, Vimeo and SoundCloud had no specialized fetcher; YouTube Shorts/live/embed paths fell through to HTML fetching. YouTube matched watch and youtu.be URLs only.

After: real release-CLI runs (fetchkit fetch URL --output json) returned:

URL HTTP Format Heading
https://www.tiktok.com/@scout2015/video/6718335390845095173 200 tiktok_video TikTok (provider omitted title/author)
https://www.youtube.com/shorts/jNQXAC9IVRw 200 youtube_video Me at the zoo
https://vimeo.com/863362136 200 vimeo_video Inside Vimeo Staff Picks
https://soundcloud.com/forss/flickermood 200 soundcloud_track Flickermood by Forss

SoundCloud description output now includes [https://www.dswshoes.com/tv_commercial.jsp?m=october2007](https://www.dswshoes.com/tv_commercial.jsp?m=october2007) rather than raw anchor HTML. The old Vimeo sample 76979871 returned HTTP 404 and Vimeo oEmbed unavailable: HTTP 404, confirming failures are preserved.

Validation: 597 workspace tests passed (2 existing ignored); all 18 live fetcher tests passed with no skips; formatting, Clippy -D warnings, warning-free docs, release build excluding Python, and both OKF checks passed. Offline regression coverage includes registration, canonicalization, metadata rendering, malformed/incomplete JSON, output caps, HEAD, provider errors, policy enforcement and YouTube variants/query injection.

Two stale live fixtures now use Rust’s current main branch and httpbin’s dedicated HTML endpoint, retaining HTTP/content assertions.

Risk

  • Medium: public provider responses/availability can change. No media download, login bypass, API credentials or short-link resolution is added.
  • Original and complete API URLs obey host/port/prefix policy. Fixed endpoints use DNS-pinned transport without redirect following; returned media/author/embed URLs are never fetched.
  • Incomplete provider bodies are rejected, even when a valid JSON prefix fits within the byte limit. Rendered output is UTF-8 safely capped. No new dependencies.

Checklist

  • Unit tests are passed
  • Smoke tests are passed
  • Documentation is updated
  • Knowledge is up to date and not in conflict

@chaliy
chaliy merged commit be86550 into main Oct 28, 2026
12 checks passed
@chaliy
chaliy deleted the codex/social-fetchers branch October 28, 2026 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant