Don't refetch sitemaps that were already fetched

[pypt]

2019-07-19 14:26:46,279 INFO mediawords.util.sitemap.media [95859/MainThread]: Fetching sitemap pages for media ID 10 (https://globalvoices.org/)...
2019-07-19 14:26:46,282 INFO usp.fetch_parse [95859/MainThread]: Fetching level 0 sitemap from https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,282 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,800 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,801 INFO usp.fetch_parse [95859/MainThread]: Fetching level 0 sitemap from https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,802 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,963 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,979 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:46,979 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:47,407 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:47,413 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:26:47,414 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:26:48,998 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
<...>
2019-07-19 14:54:06,695 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,696 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,869 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,871 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:54:06,872 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:54:07,033 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
<...>

[a-kuchynski]

@pypt

I'm experiencing the same issue with multiple websites. It also happens on multilingual websites where the sitemap structure is very complex and having 2+ levels of sitemap. The processing of it simply goes into the infinite loop.

Example: f5.com

from usp.tree import sitemap_tree_for_homepage

tree = sitemap_tree_for_homepage('https://www.f5.com/')

for page in tree.all_pages():
    print(page.url)

[pypt]

Sorry @a-kuchynski, I'm no longer working on this project. Maybe you could vibecode a fix and suggest it to the current maintainers?

[freddyheppell]

It shouldn't ever enter an infinite loop, as it'll refuse to parse a sitemap if its path matches any of its parents (which can be seen at the top of the logs you posted).

What might happen is that the sub-sitemaps each contain a reference to the same structure. Currently it will parse each of these separately as USP parses strictly as a tree.

I can have a look at f5.com and see what's causing it to get stuck or appear to do so.