Repository navigation
Sudden problem indexing a particular channel #283
Description
TubeSync has been reliably downloading videos daily from this channel since I started using it (September 3rd based on when I filed my first issue), but suddenly, it has trouble indexing the latest videos every so often. The channel releases at least one new video per day, but attached is a screenshot of my Latest Downloads box that shows it took three days to recognize Episode 1909 (and then caught up by downloading 1910 and 1911 during the same task), and currently, Episodes 1914 through 1916 are completely missing in action despite clearly being available on the channel. Will TubeSync resume recognizing episodes tomorrow? Has the channel stumped it permanently this time? Only time or meeb will tell.
I've restarted TubeSync, I've run "reset tasks", I've combed through the task log and confirmed that the episodes aren't being spotted at all, I've combed through the Docker log for anything noteworthy to no avail, and I've confirmed I'm still running the same version I've been running all along. If there's anything else you need me to provide, let me know.
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
meeb commentedon Nov 4, 2022
Is this a channel which is streamed live then VoD are available later?
blackwind commentedon Nov 4, 2022
Yep. But it's available right after the stream, not days later.
meeb commentedon Nov 4, 2022
Typically the issue occasionally here is that YouTube makes live streams available as VoDs immediately with a very limited number of formats and then adds to these formats over the next several days. For example the video may only be available under H264 1080p for 2-3 days and then all the other transcoded formats appear later. Additionally, some of the immediately available stream formats may be DASH encoded and fail to be merged back into a single video file with ffmpeg due to a historical bug. Thanks to some PRs by others recently the DASH issue is likely to be fixed shortly.
Try going to a missing media item that's 3-4 days old and clicking "skip and delete", wait a few second and click "unskip/download" again. If the item then downloads then it's a live stream format availability issue at YouTube.
blackwind commentedon Nov 4, 2022
That's the thing, the items aren't being indexed at all. I had the problem you note initially when I was asking TubeSync to fetch AVC files but only VP9 were available sometimes (in the end, I just changed my setting to VP9 and lived with it), but here, TubeSync doesn't see the episodes at all.
meeb commentedon Nov 4, 2022
OK, the media items aren't even in your library at all? Can you give me a URL to a video which is on YouTube but missing from your local TubeSync index?
blackwind commentedon Nov 4, 2022
Episode 1914: https://youtu.be/Pm-ejSLIrzA
Episode 1915: https://youtu.be/LWj9BY1Jn3s
Episode 1916: https://youtu.be/-Aua8cY-0PU
meeb commentedon Nov 4, 2022
Thanks. What TubeSync does to dramatically improve indexing speeds and reduce requests to YouTube is when it indexes a channel it stops indexing once it finds a media item it's already indexed.
If there is a situation where YouTube has a list of media items, say item 1, 5, 9 in a list and TubeSync indexes up to 9, then YouTube adds in more items so the list is 1, 2, 3, 4, 5, 9 then the new items, 2, 3, 4 would not be indexed. Basically if YouTube retroactively injects media out of order into the video index there is a chance the media items might be missed.
This is something I've not seen before so I'll check the channel you've mentioned and see if I can find it. This may require a "items are missing, make sure you do a full index" button to be created if this is indeed the issue.
blackwind commentedon Nov 6, 2022
Yep, she's dead, Jim. We're up to Episode 1919 now and it just cannot find these episodes anymore.
meeb commentedon Nov 6, 2022
Fair enough, from what I can tell it looks like non-chronologically added media items as discussed above. I'll look into adding a quick
--full-reindexstyle command as a quick hack for now which should fix this particular source at least.blackwind commentedon Nov 7, 2022
Is there no sort order parameter in the YouTube API? Or a way to limit the returned results to the last 30 days (or whichever time period was configured in TubeSync)? The issue as you've described it makes sense to me, but it feels like there should be an even simpler proper solution.
meeb commentedon Nov 7, 2022
youtube-dland forks includingyt-dlpdo not use the YouTube APIs, which require Google accounts and API keys. They basically just scrape the front end of YouTube with various methods. There are some URL parameters you can specify to tweak ordering on some URLs but that's about it. Generally what you get on the/videosURL for your channel or playlist is what TubeSync can index.blackwind commentedon Nov 8, 2022
So, YouTube has split its Videos tab into Videos, Live, and Shorts tabs, and this is apparently why yt-dlp isn't picking up what I want anymore -- it only downloads from the Videos tab which no longer contains completed streams. The proper solution is to call the URL for each tab when indexing. But yt-dlp has trouble indexing tabs where the URL and tab name don't match (the Live tab's URL is /streams), so
--compat-options no-youtube-channel-redirectneeds to be passed.See these issues:
yt-dlp/yt-dlp#5419
yt-dlp/yt-dlp#5430
blackwind commentedon Nov 8, 2022
Oh, and most critically:
--full-reindex, therefore, should have no effect.I think there are two solutions here:
meeb commentedon Nov 9, 2022
Thanks for the investigative work! To set this up properly in TubeSync it's going to be a reasonable amount of work. Allowing people to enter a freeform text box is going to result in a lot of channels breaking for people as it really isn't entirely clear what to enter into the box unless you really know what you're doing. This is why it's as guided as possible right now and probably isn't sensible to change for everyone just for channels that have streams.
This likely needs an extra "also index streams" tick-box per source which also indexes
/streamsas well as/videosfor a channel and in addition per-source TubeSync would need to store the "last indexed stream" media ID to prevent full re-indexing when updating the media items synced into TubeSync.Any additional flags or arguments that might be required also need to be mapped into the
yt-dlpembedded Python API as TubeSync doesn't callyt-dlpwith flags.I'll pop this on the wishlist, happy to take PRs as well of course.
blackwind commentedon Nov 9, 2022
Solution 3: Apparently they just fixed it in
masterand it should work exactly as it did before:yt-dlp/yt-dlp#5439
Should just need to release a new version of TubeSync with the latest yt-dlp bundled and all will be well again.
meeb commentedon Nov 10, 2022
I don't believe this patch to
yt-dlpwill fix this issue, but it will be in the next TubeSync update when I next syncyt-dlpanyway. TubeSync specifically requests the/videostab as the others do not seem to be, or at least last time I looked, chronologically order-able reliably. The only way TubeSync can detect new media items when using tabs with arbitrarily ordered media is to index everything on the channel every time. This is useful and the default behavior ofyt-dlpto download a channel but not particularly useful if you want to just detect when new media has been added to a channel to sync it PVR-style as full channel indexes can quickly results in thousands of requests to YouTube and getting your IP throttled or blocked.This will probably still require a "last stream media ID" to be recorded per source, and possibly a more general arbitrary tabs support per source to work.
tcely commentedon Jun 11, 2025
I've made significant changes to indexing recently.
#1078 and #1044, in particular, should be providing much improved indexing of sources.
meeb commentedon Mar 19, 2026
I'll close this for now as it's probably either resolved, or, things have moved on enough that it's no longer relevant. Please open a new issue if you continue to experience problems.