Fix MyHeritage with new design

[GuyKh]

Still work in progress.... DO NOT MERGE YET

Solves #110

[eljeffeg]

If it helps, here was my work in progress that I haven't had time to finish.
myheritage.js.zip

[GuyKh]

@eljeffeg
The main problem from what I can tell is the fact that the relative URLs are coming from GraphQL queries:
https://familygraphql.myheritage.com/profile_header_data/
https://familygraphql.myheritage.com/profile_details_data/

e.g. for https://www.myheritage.com/profile-420551751-1500056/dov-kushnir:

{
    "data": {
        "profile": {
            "individual": {
                "family_groups": [
                    {
                        "type": "parent",
                        "is_parent_family": true,
                        "father": {
                            "relationship_description": "His father",
                            "relationship_type": "father",
                            "individual": {
                                "id": "individual-420551751-1500056",
                                "name": "DOV KUSHNIR",
                                "gender": "M",
                                "age_group": "A",
                                "lifespan": "Died: 1943",
                                "personal_photo": null,
                                "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500056\/dov-kushnir"
                            }
                        },
                        "mother": {
                            "relationship_description": "His mother",
                            "relationship_type": "mother",
                            "individual": {
                                "id": "individual-420551751-1500057",
                                "name": "GITL KUSHNIR (\u05dc\u05d1\u05d9\u05ea GUTMAN)",
                                "gender": "F",
                                "age_group": "A",
                                "lifespan": "Died: 1943",
                                "personal_photo": null,
                                "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500057\/gitl-kushnir-%D7%9C%D7%91%D7%99%D7%AA-gutman"
                            }
                        },
                        "siblings": [
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500066",
                                    "name": "ZEEV KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "1911 - 2011",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500066\/zeev-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His sister",
                                "relationship_type": "sister",
                                "individual": {
                                    "id": "individual-420551751-1500060",
                                    "name": "ROCHL KUSHNER",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500060\/rochl-kushner"
                                }
                            },
                            {
                                "relationship_description": "His sister",
                                "relationship_type": "sister",
                                "individual": {
                                    "id": "individual-420551751-1500061",
                                    "name": "TAYBL TILLIE GOLDMAN (\u05dc\u05d1\u05d9\u05ea KUSHNER)",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500061\/taybl-tillie-goldman-%D7%9C%D7%91%D7%99%D7%AA-kushner"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500062",
                                    "name": "AVRAHAM KUSHNER",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500062\/avraham-kushner"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500063",
                                    "name": "BARUCH KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500063\/baruch-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500064",
                                    "name": "SHAMAI KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Died: 1943",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500064\/shamai-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500065",
                                    "name": "HILEL KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500065\/hilel-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His sister",
                                "relationship_type": "sister",
                                "individual": {
                                    "id": "individual-420551751-1500067",
                                    "name": "HANA RABINOVICH (\u05dc\u05d1\u05d9\u05ea KUSHNIR)",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500067\/hana-rabinovich-%D7%9C%D7%91%D7%99%D7%AA-kushnir"
                                }
                            }
                        ],
                        "spouse": null,
                        "children": null
                    },
                    {
                        "type": "spouse",
                        "is_parent_family": false,
                        "father": null,
                        "mother": null,
                        "siblings": null,
                        "spouse": {
                            "relationship_description": "His wife",
                            "relationship_type": "wife",
                            "individual": {
                                "id": "individual-420551751-1500014",
                                "name": "\u05d7\u05de\u05d3\u05d4 \u05e7\u05d5\u05e9\u05e0\u05d9\u05e8",
                                "gender": "F",
                                "age_group": "A",
                                "lifespan": "Deceased",
                                "personal_photo": null,
                                "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500014\/%D7%97%D7%9E%D7%93%D7%94-%D7%A7%D7%95%D7%A9%D7%A0%D7%99%D7%A8"
                            }
                        },
                        "children": [
                            {
                                "relationship_description": "His son",
                                "relationship_type": "son",
                                "individual": {
                                    "id": "individual-420551751-1500005",
                                    "name": "<Private> <Private>",
                                    "gender": "U",
                                    "age_group": "A",
                                    "lifespan": null,
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500005\/private"
                                }
                            }
                        ]
                    }
                ],
                "event_facts": {
                    "data": [
                        {
                            "id": "event-420551751-1500016",
                            "type": "BIRT",
                            "title": "Birth",
                            "is_family_fact": false,
                            "is_fact_of_relative": false,
                            "date": null,
                            "year": null,
                            "formatted_age": null,
                            "formatted_place": "\u05e4\u05d5\u05dc\u05d9\u05df",
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-420551751-1500015"
                            },
                            "relative": null,
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        },
                        {
                            "id": "event-420551751-1500017",
                            "type": "DEAT",
                            "title": "Death",
                            "is_family_fact": false,
                            "is_fact_of_relative": false,
                            "date": null,
                            "year": null,
                            "formatted_age": null,
                            "formatted_place": null,
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-420551751-1500015"
                            },
                            "relative": null,
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        }
                    ]
                },
                "insights": {
                    "confirmed_record_matches_summary": null,
                    "consistency_issues_summary": null,
                    "relative_hints": null
                },
                "map_pins": {
                    "data": [
                        {
                            "location": {
                                "name": "Poland",
                                "point": {
                                    "lat": 51.919438,
                                    "lng": 19.145136
                                },
                                "bounds": {
                                    "north_east": {
                                        "lat": 54.835784,
                                        "lng": 24.1458931
                                    },
                                    "south_west": {
                                        "lat": 49.0020252,
                                        "lng": 14.1228641
                                    }
                                }
                            },
                            "facts": {
                                "data": [
                                    {
                                        "id": "event-420551751-1500016",
                                        "is_fact_of_relative": false,
                                        "is_family_fact": false,
                                        "title": "Birth",
                                        "date": null,
                                        "formatted_place": "\u05e4\u05d5\u05dc\u05d9\u05df",
                                        "individual": {
                                            "id": "individual-420551751-1500015"
                                        },
                                        "relative": null,
                                        "spouse": null
                                    }
                                ]
                            }
                        }
                    ]
                }
            },
            "site_membership": null,
            "user": null,
            "birthday_greeting": null,
            "anniversary_greeting": null
        }
    }
}

Unfortunately, the GraphQL request is sent with a bearer_token so it'll be quite impossible to reproduce it, and also the url (i.e. link_in_profile_page in the response) is nowhere to be seen in the HTML)

[GuyKh]

Ideally speaking - we need to tap on these responses and everything you need is there but I'm not sure how to fetch this data from JQuery/HTML

[GuyKh]

I found this code:

        function Ty(e) {
            var t = e.individual
              , n = e.type
              , r = e.relationship;
            e.familyType;
            return Boolean(t) && a().createElement("div", {
                className: Ti(["family_relative", n]),
                onClick: function() {
                    K().location.href = t.link_in_profile_page
                }
            }

It seems that they generate the link on the onClick event. But I have no idea how to fetch that outside

[GuyKh]

I fixed various things -
One thing that's missing, I think is relatives Birth/Death which is being ignored today. and the freaking URL (which is a blocker) :/

[GuyKh]

OK @eljeffeg , I think we're in a whole new realm here.
Using React DevTools I was able to find this React Component hierarchy:

As you can see Fragment key=family_0 is the whole "Immediate Family" list,
where for each sub-component (including the parent) - there's a component (Cy - but I guess the name may change) which contains a prop with all the data - name, relation and link.

The question now is... it seems that we can't just fetch this from the HTML itself (as the logic happens today) - and some React analysis is required.

Here's what Google Gemini said:

Q: I'm developing a chrome extension. To show some data, I need to query the react component tree and find a Fragment component by key and fetch it's props. How to do that?
A:
There are two main approaches to achieve this functionality in a Chrome extension:

1. Content scripts:

• Content scripts are JavaScript programs that run directly within the webpage's DOM. They can access and modify the webpage's content.

Here's how you can use a content script to achieve your goal:

   - **Inject a script:**  Use the `chrome.tabs.executeScript` API to inject a content script into the webpage where you want to query the React component tree.

   - **Traverse the React tree:**  Within the content script, you can access the React component hierarchy using the browser's developer tools API (`chrome.devtools.inspectedWindow.evaluate`). This API allows you to execute code within the context of the webpage.

   - **Find Fragment by key:**  Use the `document.querySelector` function (or similar methods) to navigate the component tree based on the provided key. Keep in mind that directly querying React components by key might not always be reliable, as keys are internal implementation details.

   - **Fetch props:**  Once you have a reference to the Fragment component, you can access its props using the standard JavaScript property access syntax (e.g., `component.props.propName`).

2. Background script and devtools extension:

• Background scripts run in the background process of your extension and can communicate with webpages and other scripts using message passing.
• Devtools extensions leverage the Chrome DevTools Extension API to interact with the browser's developer tools.

Here's how you can use this approach:

    - **Background script communication:**  The devtools panel can communicate with a background script using the `runtime.onMessage` API.

    - **Content script injection:**  The background script can inject a content script similar to the first approach, but with the key information received from the devtools panel.

    - **Data retrieval and response:**  The content script retrieves the Fragment's props and sends them back to the background script using message passing.

    - **Display data in devtools panel:**  The background script receives the data from the content script and updates the devtools panel UI to display the retrieved props.

Important considerations:

• Content scripts can be limited in their ability to access and modify certain parts of the webpage due to security restrictions.
• Directly querying React components by key is not the most reliable approach as keys might not be unique or publicly exposed.
• Consider using a library like react-devtools (if allowed by Chrome extension guidelines) to interact with the React component tree in a more robust way.

Remember that modifying the DOM or interacting with webpages requires careful consideration of security implications and potential conflicts with other extensions. Make sure your extension adheres to Chrome extension development guidelines.

[eljeffeg]

Yeah, I see what you mean with https://familygraphql.myheritage.com. That's suppose to be MyHeritage's API - wonder if it's still an accessible option. For the react, a content script probably wouldn't work for this part of SmartCopy, but a background script might. We do have a background script that runs from the popup in background.js, which can make xhr requests and such. Perhaps that would be a way to get the data.

[eljeffeg]

What you have is a lot better than the failure now, so I wonder if we should just commit that and let everyone know we're still working out the dates. We might be able to at least put the years in there, since that appears to be in the data.

[GuyKh]

@eljeffeg - the problem is not the dates rather than the urls. Without it, SmartCopy (atleast today's implementation) doesn't "recognize" the family members - which is, I think, a crucial part.

I'd definitely leave the dates out, but in my opinion, the URLs are a must....

[GuyKh]

In a perfect world, I'd somehow reuse the bearer token and call the GraphQL myself (rather than parsing the HTML) and fetch all the data from there.

However, couple of thoughts on this:

1. There's a MyHeritage rate limiter (unfortunately I've seen it wayyyy to many times) - so calling APIs a second time would just bring us closer to the limit
2. Not sure how trivial it is to fetch the bearer token since the extension doesn't monitor the network calls....
3. Perhaps some data is being fetched from different APIs or so. So a single (or even 2) call(s) to the GraphQL API won't be enough.

On the other hand, I have no clear idea about accessing React Component's state from within an Extension.

[alvestrand]

Commenting without too much knowledge ... this is about translating the list of family members into queries that fetch the information about the family members? Is the URL you need the one for the family member's page?

Wild thought - is it possible to just follow the relative links from within the page and intercept the queries so that you can figure out what the complete URL has become? (the interception could also return a 5xx code to avoid bumping the rate limiter)? Or is the result of following the relative link a different URL from the one you need?

[GuyKh]

this is about translating the list of family members into queries that fetch the information about the family members? Is the URL you need the one for the family member's page?

The problem here is how to get the links - the way this is implemented is through React Components, when the page is built - each family member in the list has a onClick event hander which sends the page to the relevant link (when the user clicks anywhere on the compnent).
This is unlike the previous (and standard) behavior where an object has a <a href... tag - which is easily "read" when parsing the HTML.

The trouble here is how to get these links... React BS...

[GuyKh]

See this
and this commit specifically

[GuyKh]

@eljeffeg as a temp solution. We can set the url as the current page url for all family members, so they'll be added but referenced to the source person and not the actual person. WDYT?

[alvestrand]

If we can add "referenced in " to the text of the URL in family members, that would make it clear what's happening.
I'd love to have my MyHeritage copy tool back soon!

[alvestrand]

any chance of getting this merged? I'm still hurting from the lack of MyHeritage SmartCopy.

[GuyKh]

Still hoping to get it working with actual URLs.

[eljeffeg]

As we update the parsers for particular sites, we may want to consider implementing a model based on beautifulsoup. https://beautiful-soup-4.readthedocs.io/en/latest/

[GuyKh]

@eljeffeg any references in the project?

Isn't beautiful soup is a python tool?

[eljeffeg]

Ha, you're right. I work with Python so much I forgot that this project is pretty much just JS.

[GuyKh]

OK. I give up. I'll set the url field as "" and let's see what happens

[alvestrand]

Is this ready to merge, then?

[GuyKh]

@alvestrand - this won't do, I'm afraid.
Still looking for a solution...

[GuyKh]

OK @eljeffeg , I think we're in a whole new realm here. Using React DevTools I was able to find this React Component hierarchy:

As you can see Fragment key=family_0 is the whole "Immediate Family" list, where for each sub-component (including the parent) - there's a component (Cy - but I guess the name may change) which contains a prop with all the data - name, relation and link.

The question now is... it seems that we can't just fetch this from the HTML itself (as the logic happens today) - and some React analysis is required.

Here's what Google Gemini said:

Q: I'm developing a chrome extension. To show some data, I need to query the react component tree and find a Fragment component by key and fetch it's props. How to do that?
A:
There are two main approaches to achieve this functionality in a Chrome extension:

1. Content scripts:

• Content scripts are JavaScript programs that run directly within the webpage's DOM. They can access and modify the webpage's content.

Here's how you can use a content script to achieve your goal:

   - **Inject a script:**  Use the `chrome.tabs.executeScript` API to inject a content script into the webpage where you want to query the React component tree.

   - **Traverse the React tree:**  Within the content script, you can access the React component hierarchy using the browser's developer tools API (`chrome.devtools.inspectedWindow.evaluate`). This API allows you to execute code within the context of the webpage.

   - **Find Fragment by key:**  Use the `document.querySelector` function (or similar methods) to navigate the component tree based on the provided key. Keep in mind that directly querying React components by key might not always be reliable, as keys are internal implementation details.

   - **Fetch props:**  Once you have a reference to the Fragment component, you can access its props using the standard JavaScript property access syntax (e.g., `component.props.propName`).

2. Background script and devtools extension:

• Background scripts run in the background process of your extension and can communicate with webpages and other scripts using message passing.
• Devtools extensions leverage the Chrome DevTools Extension API to interact with the browser's developer tools.

Here's how you can use this approach:

    - **Background script communication:**  The devtools panel can communicate with a background script using the `runtime.onMessage` API.

    - **Content script injection:**  The background script can inject a content script similar to the first approach, but with the key information received from the devtools panel.

    - **Data retrieval and response:**  The content script retrieves the Fragment's props and sends them back to the background script using message passing.

    - **Display data in devtools panel:**  The background script receives the data from the content script and updates the devtools panel UI to display the retrieved props.

Important considerations:

• Content scripts can be limited in their ability to access and modify certain parts of the webpage due to security restrictions.
• Directly querying React components by key is not the most reliable approach as keys might not be unique or publicly exposed.
• Consider using a library like react-devtools (if allowed by Chrome extension guidelines) to interact with the React component tree in a more robust way.

Remember that modifying the DOM or interacting with webpages requires careful consideration of security implications and potential conflicts with other extensions. Make sure your extension adheres to Chrome extension development guidelines.

Structure was change a bit:

Still no luck :(

[GuyKh]

I see that when loading - the page does a GraphQL Query:

This is for profile-171591352-1500891 and requires a Bearer Token

Request to **https://familygraphql.myheritage.com/profile_header_data**

Request

curl --location 'https://familygraphql.myheritage.com/profile_header_data/' \
--header 'accept: application/json, text/plain, */*' \
--header 'accept-language: en-US,en;q=0.9,fr;q=0.8,he;q=0.7' \
--header 'origin: https://www.myheritage.com' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36' \
--form 'bearer_token="4.179cNOTTODAY"' \
--form 'query="\"{profile(id:\\\"profile-171591352-1500891-0\\\",lang:\\\"EN\\\"){name first_name last_name gender age_group age{text}personal_photo{...personal_photo_fragment}is_prefer_user can_current_user_view_discoveries can_current_user_manage_photos can_current_user_edit_personal_photo can_current_user_invite_individual tabs{name total counters}recent_individuals{data{...history_fragment}}favorite_individuals{data{...history_fragment}}individual{...individual_fragment}user{...user_fragment}site_membership{member_id site_id can_user_contact_member member_joined_date member_last_visit_date role_sentence{text}}tree{is_imported_using_family_search_sync}site{name}}}fragment history_fragment on Individual{id name gender age_group lifespan personal_photo{...personal_photo_fragment}tree_relationship{description}link_in_profile_page}fragment individual_fragment on Individual{id name first_name gender personal_photo{...personal_photo_fragment}is_privatized religious_name former_name namesake alternate_names birth_date{text}birth_place death_date{text}is_alive is_likely_deceased death_place burial_place cause_of_death is_cause_of_death_holocaust age{text}tree_relationship{description is_blood_relative is_biological_blood_relative blood_relative_description hour_glass_color_code is_path_to_self is_path_cannot_decide_if_related}can_edit link_in_tree link_in_pedigree_tree link_in_fan_view link_in_research_this_person link_template_in_edit_profile}fragment personal_photo_fragment on Photo{thumbnails(thumbnail_size:\\\"136x136c\\\"){url}}fragment user_fragment on User{name first_name crown_status country country_code birth_date{text}age{text}age_group_in_years show_age is_public is_privatized nickname created_time}\""' \
--form 'operation=""' \
--form 'variables=""' \
--form 'description="profile header data"'

Response

Response

{
    "data": {
        "profile": {
            "name": "שושנה רוזנפלד (לבית ימפולסקי)",
            "first_name": "שושנה",
            "last_name": "ימפולסקי",
            "gender": "F",
            "age_group": "A",
            "age": null,
            "personal_photo": null,
            "is_prefer_user": false,
            "can_current_user_view_discoveries": false,
            "can_current_user_manage_photos": false,
            "can_current_user_edit_personal_photo": false,
            "can_current_user_invite_individual": false,
            "tabs": [
                {
                    "name": "details",
                    "total": 0,
                    "counters": null
                },
                {
                    "name": "biography",
                    "total": 0,
                    "counters": null
                },
                {
                    "name": "sources",
                    "total": 1,
                    "counters": null
                }
            ],
            "recent_individuals": {
                "data": [
                    {
                        "id": "individual-171591352-1500890",
                        "name": "אהרון רוזנפלד",
                        "gender": "M",
                        "age_group": "A",
                        "lifespan": "Deceased",
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": "Great-grandfather's half brother's father-in-law (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1500890/%D7%90%D7%94%D7%A8%D7%95%D7%9F-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                    },
                    {
                        "id": "individual-171591352-1501337",
                        "name": "Efraim Rozenfld",
                        "gender": "M",
                        "age_group": "A",
                        "lifespan": "1876 - 1960",
                        "personal_photo": {
                            "thumbnails": [
                                {
                                    "url": "https://sites-cf.mhcache.com/t/591/352/171591352/501/501578_9778695jfa3598ge725b6u_V_111x111C/dcVCgF5LyBZt8E0Z66CxzH-FUV8=/.jpg"
                                }
                            ]
                        },
                        "tree_relationship": {
                            "description": " (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1501337/efraim-rozenfld"
                    },
                    {
                        "id": "individual-171591352-1501958",
                        "name": "Shoshana Rosenfeld (לבית Yampolski)",
                        "gender": "F",
                        "age_group": "A",
                        "lifespan": "Deceased",
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": " (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1501958/shoshana-rosenfeld-%D7%9C%D7%91%D7%99%D7%AA-yampolski"
                    },
                    {
                        "id": "individual-171591352-1000227",
                        "name": "בלה אלברט (לבית רוזנפלד)",
                        "gender": "F",
                        "age_group": "A",
                        "lifespan": "1874 - 1958",
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": "Great-grandfather's half brother's wife (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000227/%D7%91%D7%9C%D7%94-%D7%90%D7%9C%D7%91%D7%A8%D7%98-%D7%9C%D7%91%D7%99%D7%AA-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                    },
                    {
                        "id": "individual-171591352-1000626",
                        "name": "<Private> nagid",
                        "gender": "U",
                        "age_group": "A",
                        "lifespan": null,
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": " (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000626/private-nagid"
                    }
                ]
            },
            "favorite_individuals": {
                "data": null
            },
            "individual": {
                "id": "individual-171591352-1500891",
                "name": "שושנה רוזנפלד (לבית ימפולסקי)",
                "first_name": "שושנה",
                "gender": "F",
                "personal_photo": null,
                "is_privatized": false,
                "religious_name": null,
                "former_name": null,
                "namesake": null,
                "alternate_names": null,
                "birth_date": null,
                "birth_place": null,
                "death_date": null,
                "is_alive": false,
                "is_likely_deceased": true,
                "death_place": null,
                "burial_place": null,
                "cause_of_death": null,
                "is_cause_of_death_holocaust": false,
                "age": null,
                "tree_relationship": {
                    "description": "Great-grandfather's half brother's mother-in-law (of site manager)",
                    "is_blood_relative": false,
                    "is_biological_blood_relative": false,
                    "blood_relative_description": null,
                    "hour_glass_color_code": null,
                    "is_path_to_self": false,
                    "is_path_cannot_decide_if_related": false
                },
                "can_edit": false,
                "link_in_tree": "https://www.myheritage.com/site-family-tree-171591352/lerner?rootIndividualID=1500891&familyTreeID=1",
                "link_in_pedigree_tree": "https://www.myheritage.com/pedigree-tree-171591352-1500891/lerner?familyTreeID=1",
                "link_in_fan_view": "https://www.myheritage.com/fan-view-171591352-1500891/lerner?familyTreeID=1",
                "link_in_research_this_person": "https://www.myheritage.com/research/individual-1500891/?s=171591352&rfr=profile_page",
                "link_template_in_edit_profile": null
            },
            "user": null,
            "site_membership": null,
            "tree": {
                "is_imported_using_family_search_sync": false
            },
            "site": {
                "name": "Lerner Web Site"
            }
        }
    }
}

Request to **https://familygraphql.myheritage.com/profile_details_data**

Request

curl --location 'https://familygraphql.myheritage.com/profile_details_data/' \
--header 'accept: application/json, text/plain, */*' \
--header 'accept-language: en-US,en;q=0.9,fr;q=0.8,he;q=0.7' \
--header 'origin: https://www.myheritage.com' \
--header 'referer: https://www.myheritage.com/' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36' \
--form 'bearer_token="4.179c5NotToday"' \
--form 'query="\"{profile(id:\\\"profile-171591352-1500891-0\\\",lang:\\\"EN\\\"){individual{family_groups(relationship_prefix:\\\"auto\\\"){type is_parent_family father{...family_member_fragment}mother{...family_member_fragment}siblings(include_half_siblings:true){...family_member_fragment}spouse{...family_member_fragment}children{...family_member_fragment}}event_facts(hints:3){data{...fact_fragment}}insights{confirmed_record_matches_summary{...insight_summary_fragment}consistency_issues_summary{...insight_summary_fragment}relative_hints{...hint_fragment}}map_pins{data{...map_pin_fragment}}}site_membership{...site_membership_fragment}user{surname_research}birthday_greeting{...greeting_fragment}anniversary_greeting{...greeting_fragment}}}fragment fact_fragment on Fact{id type title is_family_fact is_fact_of_relative date{text}year formatted_age formatted_place cause_of_death content additional_content individual{id}relative{...fact_relative_fragment}spouse{...fact_relative_fragment}hint{...hint_fragment}citations{data{...citation_fragment}}notes{data{...note_fragment}}media{data{name link thumbnails(thumbnail_size:\\\"96x96c\\\"){url}}}}fragment citation_fragment on Citation{id page confidence event{id title}family_event{id title}date{text}formatted_text page_link{url name image}source{name smart_matching_site{id}image link}extended_citation{reference comment reason}smart_matching_individual{id name}}fragment note_fragment on Note{id type text subject body}fragment family_member_fragment on Relationship{relationship_description relationship_type individual{id name gender age_group lifespan personal_photo{...personal_photo_fragment}link_in_profile_page}}fragment personal_photo_fragment on Photo{thumbnails(thumbnail_size:\\\"136x136c\\\"){url}}fragment fact_relative_fragment on Individual{id name gender age_group personal_photo{...personal_photo_fragment}link_in_profile_page}fragment insight_summary_fragment on InsightSummary{type status count link is_accessible fields{id label value}}fragment hint_fragment on InsightHint{factor key modifier count first_source_name image}fragment map_pin_fragment on FactMapPin{location{name point{lat lng}bounds{north_east{lat lng}south_west{lat lng}}}facts{data{id is_fact_of_relative is_family_fact title date{text}formatted_place individual{id}relative{id name}spouse{name}}}}fragment sentence_fragment on StorySentence{text tokens{type text value link}}fragment site_membership_fragment on ProfileSiteMembership{member_id member_gender site_id site_creator_id role_sentence{...sentence_fragment}visit_sentence{...sentence_fragment}join_sentence{...sentence_fragment}request_sentence{...sentence_fragment}is_current_user_member_in_site can_user_contact_member can_user_contact_site_manager can_user_promote_member_to_site_manager can_user_demote_member_from_site_manager can_user_remind_member_to_visit can_user_change_member_email_for_remind_to_visit can_user_review_membership_request review_membership_request_link can_user_remove_member_from_site can_user_identify_member_in_tree can_user_edit_member_profile edit_member_profile_link can_user_edit_member_site_preferences edit_member_site_preferences_link can_user_edit_member_privacy_preferences edit_member_privacy_preferences_link can_user_change_member_email_and_password change_member_email_and_password_link can_user_view_member_public_profile view_member_public_profile_link can_user_associate_member_in_tree other_site_memberships{data{site_name site_link role}}}fragment greeting_fragment on ProfileGreeting{type date title label link}\""' \
--form 'operation=""' \
--form 'variables=""' \
--form 'description="profile details data"'

Response

Response

{
    "data": {
        "profile": {
            "individual": {
                "family_groups": [
                    {
                        "type": "spouse",
                        "is_parent_family": false,
                        "father": null,
                        "mother": null,
                        "siblings": null,
                        "spouse": {
                            "relationship_description": "Her husband",
                            "relationship_type": "husband",
                            "individual": {
                                "id": "individual-171591352-1500890",
                                "name": "אהרון רוזנפלד",
                                "gender": "M",
                                "age_group": "A",
                                "lifespan": "Deceased",
                                "personal_photo": null,
                                "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1500890/%D7%90%D7%94%D7%A8%D7%95%D7%9F-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                            }
                        },
                        "children": [
                            {
                                "relationship_description": "Her daughter",
                                "relationship_type": "daughter",
                                "individual": {
                                    "id": "individual-171591352-1000227",
                                    "name": "בלה אלברט (לבית רוזנפלד)",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "1874 - 1958",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000227/%D7%91%D7%9C%D7%94-%D7%90%D7%9C%D7%91%D7%A8%D7%98-%D7%9C%D7%91%D7%99%D7%AA-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                                }
                            }
                        ]
                    }
                ],
                "event_facts": {
                    "data": [
                        {
                            "id": "event-171591352-1000251",
                            "type": "BIRT",
                            "title": "Birth of daughter:",
                            "is_family_fact": false,
                            "is_fact_of_relative": true,
                            "date": {
                                "text": "1874"
                            },
                            "year": "1874",
                            "formatted_age": null,
                            "formatted_place": "סבסטופול, רוסיה",
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-171591352-1500891"
                            },
                            "relative": {
                                "id": "individual-171591352-1000227",
                                "name": "בלה אלברט (לבית רוזנפלד)",
                                "gender": "F",
                                "age_group": "A",
                                "personal_photo": null,
                                "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000227/%D7%91%D7%9C%D7%94-%D7%90%D7%9C%D7%91%D7%A8%D7%98-%D7%9C%D7%91%D7%99%D7%AA-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                            },
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        },
                        {
                            "id": "event-171591352-1501837",
                            "type": "DEAT",
                            "title": "Death",
                            "is_family_fact": false,
                            "is_fact_of_relative": false,
                            "date": null,
                            "year": null,
                            "formatted_age": null,
                            "formatted_place": null,
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-171591352-1500891"
                            },
                            "relative": null,
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        }
                    ]
                },
                "insights": {
                    "confirmed_record_matches_summary": null,
                    "consistency_issues_summary": null,
                    "relative_hints": null
                },
                "map_pins": {
                    "data": [
                        {
                            "location": {
                                "name": "Russia",
                                "point": {
                                    "lat": 61.52401,
                                    "lng": 105.318756
                                },
                                "bounds": {
                                    "north_east": {
                                        "lat": 70,
                                        "lng": 179
                                    },
                                    "south_west": {
                                        "lat": 40,
                                        "lng": 27
                                    }
                                }
                            },
                            "facts": {
                                "data": [
                                    {
                                        "id": "event-171591352-1000251",
                                        "is_fact_of_relative": true,
                                        "is_family_fact": false,
                                        "title": "Birth of daughter:",
                                        "date": {
                                            "text": "1874"
                                        },
                                        "formatted_place": "סבסטופול, רוסיה",
                                        "individual": {
                                            "id": "individual-171591352-1500891"
                                        },
                                        "relative": {
                                            "id": "individual-171591352-1000227",
                                            "name": "בלה אלברט (לבית רוזנפלד)"
                                        },
                                        "spouse": null
                                    }
                                ]
                            }
                        }
                    ]
                }
            },
            "site_membership": null,
            "user": null,
            "birthday_greeting": null,
            "anniversary_greeting": null
        }
    }
}

This could be a potential to use it - but we need to extract the BearerToken from the page somehow

[eljeffeg]

Yeah, if you can use the background script, it's not too hard to call listener functions in that and wait for the response.

[GuyKh]

Yeah, if you can use the background script, it's not too hard to call listener functions in that and wait for the response.

I'd love to see an example

[stbodie]

Superseded by #179. This PR's approach (parsing the new page design in place) got the focus person's own data working, but left family-member linking as a known gap (const url = ""; // How to get URLs?) - the new page's relative cards genuinely have no <a href>, so that stub was correct to flag it as unsolved rather than guess.

#179 solves it: spouse/children links come from the Facts section's real anchor tags (and, as a bonus, those facts already carry full birth date+place for children), while parents/siblings - who have no link anywhere in the rendered HTML at all - are recovered by walking React's internal fiber tree in a MAIN-world content script, since the link only exists as an unrendered link_in_profile_page prop on the card's React component.

Thank you for the initial exploration here - it was very helpful reference for what the new page's markup looks like.