Repository navigation
feat(fetchers): add public social media metadata support - #188
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Fetch public TikTok and Vimeo videos and SoundCloud tracks/playlists as bounded Markdown metadata through their official oEmbed endpoints. The shared registry enables this automatically in Rust, CLI, MCP and Python. YouTube now recognizes Shorts, live, embed and privacy-enhanced embed URLs alongside watch/short links.
Titles, authors, thumbnails and descriptions appear when supplied; descriptions become Markdown. Embed HTML is ignored. Output explicitly states that media/transcripts are not retrieved by the new fetcher. TikTok currently supplies only minimal embed metadata, so it can return the platform and canonical URL without a title. Existing YouTube captions remain best-effort.
Why
Social pages commonly need JavaScript or show login/navigation shells. Public oEmbed metadata provides useful structured output without credentials or a browser. Supported URL matching stays narrow; unsupported pages retain the default HTML fetcher.
Before / After
Before: TikTok, Vimeo and SoundCloud had no specialized fetcher; YouTube Shorts/live/embed paths fell through to HTML fetching. YouTube matched watch and youtu.be URLs only.
After: real release-CLI runs (
fetchkit fetch URL --output json) returned:https://www.tiktok.com/@scout2015/video/6718335390845095173tiktok_videohttps://www.youtube.com/shorts/jNQXAC9IVRwyoutube_videohttps://vimeo.com/863362136vimeo_videohttps://soundcloud.com/forss/flickermoodsoundcloud_trackSoundCloud description output now includes
[https://www.dswshoes.com/tv_commercial.jsp?m=october2007](https://www.dswshoes.com/tv_commercial.jsp?m=october2007)rather than raw anchor HTML. The old Vimeo sample76979871returned HTTP 404 andVimeo oEmbed unavailable: HTTP 404, confirming failures are preserved.Validation: 597 workspace tests passed (2 existing ignored); all 18 live fetcher tests passed with no skips; formatting, Clippy
-D warnings, warning-free docs, release build excluding Python, and both OKF checks passed. Offline regression coverage includes registration, canonicalization, metadata rendering, malformed/incomplete JSON, output caps, HEAD, provider errors, policy enforcement and YouTube variants/query injection.Two stale live fixtures now use Rust’s current
mainbranch and httpbin’s dedicated HTML endpoint, retaining HTTP/content assertions.Risk
Checklist