Skip to content

Checkout failure: "Git fetch failed with exit code 128" #9009

Open

Description

@carlossanlop

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

  • Open the failing build above and investigate
  • Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

  1. Add the label "Known Build Error"
  2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation
{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build Repository Step Name Console log Pull Request
1687534 dotnet/maui Checkout dotnet/maui@refs/pull/40369/merge to s Log dotnet/maui#40369
1687513 dotnet/maui Checkout dotnet/maui@refs/pull/40368/merge to s Log dotnet/maui#40368
1687370 dotnet/maui Checkout dotnet/maui@refs/pull/40366/merge to s Log dotnet/maui#40366
1687346 dotnet/maui Checkout dotnet/maui@refs/pull/40284/merge to s Log dotnet/maui#40284
1687306 dotnet/maui Checkout dotnet/maui@refs/pull/39975/merge to s Log dotnet/maui#39975
1687312 dotnet/maui Checkout dotnet/maui@refs/pull/40234/merge to s Log dotnet/maui#40234
1687242 dotnet/maui Checkout dotnet/maui@refs/pull/40363/merge to s Log dotnet/maui#40363
1687266 dotnet/maui Checkout dotnet/maui@refs/pull/40295/merge to s Log dotnet/maui#40295
1687285 dotnet/maui Checkout dotnet/maui@refs/pull/40365/merge to s Log dotnet/maui#40365
1687225 dotnet/maui Checkout dotnet/maui@refs/pull/39979/merge to s Log dotnet/maui#39979
1687003 dotnet/maui Checkout dotnet/maui@refs/pull/40318/merge to s Log dotnet/maui#40318
1686708 dotnet/maui Checkout dotnet/maui@candidate/11.0.1xx-sr2 to s Log
3138748 dotnet-dotnet Checkout dotnet-dotnet@refs/pull/68652/merge to s Log #68652
1685697 dotnet/maui Checkout dotnet/maui@refs/pull/40334/merge to s Log dotnet/maui#40334
1685551 dotnet/maui Checkout dotnet/maui@refs/pull/40284/merge to s Log dotnet/maui#40284
1685442 dotnet/maui Checkout dotnet/maui@refs/pull/40133/merge to s Log dotnet/maui#40133
1685390 dotnet/maui Checkout dotnet/maui@refs/pull/40234/merge to s Log dotnet/maui#40234
1685397 dotnet/maui Checkout dotnet/maui@refs/pull/40291/merge to s Log dotnet/maui#40291
3138494 dotnet-runtime Checkout dotnet-runtime@main to s Log
1685276 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1685291 dotnet/maui Checkout dotnet/maui@refs/pull/40316/merge to s Log dotnet/maui#40316
1685312 dotnet/maui Checkout dotnet/maui@refs/pull/39917/merge to s Log dotnet/maui#39917
1685121 dotnet/maui Checkout dotnet/maui@refs/pull/40153/merge to s Log dotnet/maui#40153
1684940 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1685100 dotnet/maui Checkout dotnet/maui@refs/pull/40324/merge to s Log dotnet/maui#40324
1685030 dotnet/maui Checkout dotnet/maui@refs/pull/40283/merge to s Log dotnet/maui#40283
1685074 dotnet/maui Checkout dotnet/maui@refs/pull/40295/merge to s Log dotnet/maui#40295
1685023 dotnet/maui Checkout dotnet/maui@refs/pull/39975/merge to s Log dotnet/maui#39975
1684830 dotnet/maui Checkout dotnet/maui@refs/pull/40307/merge to s Log dotnet/maui#40307
1684827 dotnet/maui Checkout dotnet/maui@refs/pull/40316/merge to s Log dotnet/maui#40316
1684652 dotnet/maui Checkout dotnet/maui@refs/pull/40318/merge to s Log dotnet/maui#40318
1684820 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1684363 dotnet/maui Checkout dotnet/maui@refs/pull/40311/merge to s Log dotnet/maui#40311
1684332 dotnet/maui Checkout dotnet/maui@refs/pull/40309/merge to s Log dotnet/maui#40309
1684349 dotnet/maui Checkout dotnet/maui@refs/pull/40310/merge to s Log dotnet/maui#40310
1684308 dotnet/runtime Checkout dotnet/runtime@refs/pull/137691/merge to s Log dotnet/runtime#137691
1684233 dotnet/runtime Checkout dotnet/runtime@refs/pull/137685/merge to s Log dotnet/runtime#137685
1684371 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1684339 dotnet/maui Checkout dotnet/maui@refs/pull/40305/merge to s Log dotnet/maui#40305
1684527 dotnet/arcade-services Checkout dotnet/arcade-services@refs/pull/6895/merge to s Log dotnet/arcade-services#6895
1684468 dotnet/maui Checkout dotnet/maui@refs/pull/40312/merge to s Log dotnet/maui#40312
1684279 dotnet/maui Checkout dotnet/maui@inflight/current to s Log
1684337 dotnet/maui Checkout dotnet/maui@refs/pull/40305/merge to s Log dotnet/maui#40305
1684196 dotnet/maui Checkout dotnet/maui@candidate/11.0.1xx-sr2 to s Log
1683498 dotnet/maui Checkout dotnet/maui@refs/pull/40295/merge to s Log dotnet/maui#40295
3137858 dotnet-runtime Checkout dotnet-runtime@main to s Log
1684035 dotnet/maui Checkout dotnet/maui@refs/pull/40304/merge to s Log dotnet/maui#40304
1683994 dotnet/maui Checkout dotnet/maui@candidate/11.0.1xx-sr2 to s Log
1683495 dotnet/maui Checkout dotnet/maui@refs/pull/39979/merge to s Log dotnet/maui#39979
1683482 dotnet/maui Checkout dotnet/maui@refs/pull/40283/merge to s Log dotnet/maui#40283
1683828 dotnet/machinelearning Checkout dotnet/machinelearning@refs/pull/7918/merge to s Log dotnet/machinelearning#7918
1683391 dotnet/maui Checkout dotnet/maui@refs/pull/39974/merge to s Log dotnet/maui#39974
1683386 dotnet/maui Checkout dotnet/maui@refs/pull/40284/merge to s Log dotnet/maui#40284
1683329 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1683480 dotnet/maui Checkout dotnet/maui@refs/pull/40283/merge to s Log dotnet/maui#40283
1683163 dotnet/maui Checkout dotnet/maui@refs/pull/39975/merge to s Log dotnet/maui#39975
1683015 dotnet/maui Checkout dotnet/maui@refs/pull/40153/merge to s Log dotnet/maui#40153
1683169 dotnet/maui Checkout dotnet/maui@refs/pull/40234/merge to s Log dotnet/maui#40234
1683011 dotnet/maui Checkout dotnet/maui@refs/pull/40133/merge to s Log dotnet/maui#40133
1682962 dotnet/maui Checkout dotnet/maui@refs/pull/40283/merge to s Log dotnet/maui#40283
1683005 dotnet/maui Checkout dotnet/maui@refs/pull/40284/merge to s Log dotnet/maui#40284
1683060 dotnet/sdk Checkout dotnet/sdk@main to s Log
1682949 dotnet/maui Checkout dotnet/maui@refs/pull/39655/merge to s Log dotnet/maui#39655
1682433 dotnet/maui Checkout dotnet/maui@refs/pull/40263/merge to s Log dotnet/maui#40263
1682374 dotnet/runtime Checkout dotnet/runtime@refs/pull/137594/merge to s Log dotnet/runtime#137594
1682345 dotnet/maui Checkout dotnet/maui@refs/pull/40272/merge to s Log dotnet/maui#40272
1682246 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1682348 dotnet/maui Checkout dotnet/maui@refs/pull/40263/merge to s Log dotnet/maui#40263
1682306 dotnet/maui Checkout dotnet/maui@refs/pull/40182/merge to s Log dotnet/maui#40182
1682296 dotnet/maui Checkout dotnet/maui@refs/pull/40271/merge to s Log dotnet/maui#40271
1682299 dotnet/maui Checkout dotnet/maui@refs/pull/40270/merge to s Log dotnet/maui#40270
1682257 dotnet/maui Checkout dotnet/maui@refs/pull/40188/merge to s Log dotnet/maui#40188
3136602 dotnet-dotnet Checkout dotnet-dotnet@release/11.0.1xx to s Log
1682094 dotnet/maui Checkout dotnet/maui@refs/pull/40263/merge to s Log dotnet/maui#40263
1682087 dotnet/maui Checkout dotnet/maui@refs/pull/40261/merge to s Log dotnet/maui#40261
1682044 dotnet/maui Checkout dotnet/maui@refs/pull/40263/merge to s Log dotnet/maui#40263
1681969 dotnet/maui Checkout dotnet/maui@refs/pull/39974/merge to s Log dotnet/maui#39974
1681712 dotnet/maui Checkout dotnet/maui@refs/pull/40188/merge to s Log dotnet/maui#40188
1681675 dotnet/maui Checkout dotnet/maui@refs/pull/40263/merge to s Log dotnet/maui#40263
1681699 dotnet/diagnostics Checkout dotnet/diagnostics@refs/pull/6248/merge to s Log dotnet/diagnostics#6248
1681588 dotnet/maui Checkout dotnet/maui@refs/pull/40188/merge to s Log dotnet/maui#40188
1681382 dotnet/maui Checkout dotnet/maui@refs/pull/40254/merge to s Log dotnet/maui#40254
1681510 dotnet/runtime Checkout dotnet/runtime@main to s Log
1681463 dotnet/runtime Checkout dotnet/runtime@refs/pull/137548/merge to s Log dotnet/runtime#137548
1681453 dotnet/maui Checkout dotnet/maui@refs/pull/40256/merge to s Log dotnet/maui#40256
1681198 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1681388 dotnet/maui Checkout dotnet/maui@refs/pull/40255/merge to s Log dotnet/maui#40255
1681353 dotnet/maui Checkout dotnet/maui@refs/pull/40182/merge to s Log dotnet/maui#40182
3136080 dotnet-dotnet Checkout dotnet-dotnet@release/11.0.2xx to s Log
1681126 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1681125 dotnet/diagnostics Checkout dotnet/diagnostics@refs/pull/6239/merge to s Log dotnet/diagnostics#6239
1681124 dotnet/runtime Checkout dotnet/runtime@main to s Log
1681039 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1680949 dotnet/maui Checkout dotnet/maui@refs/pull/40060/merge to s Log dotnet/maui#40060
1680590 dotnet/maui Checkout dotnet/maui@refs/pull/40183/merge to s Log dotnet/maui#40183
1680449 dotnet/maui Checkout dotnet/maui@refs/pull/40166/merge to s Log dotnet/maui#40166
1680593 dotnet/maui Checkout dotnet/maui@refs/pull/40182/merge to s Log dotnet/maui#40182
1680587 dotnet/maui Checkout dotnet/maui@refs/pull/40184/merge to s Log dotnet/maui#40184
1680579 dotnet/maui Checkout dotnet/maui@refs/pull/40221/merge to s Log dotnet/maui#40221
1680701 dotnet/runtime Checkout dotnet/runtime@refs/pull/137520/merge to s Log dotnet/runtime#137520
Displaying 100 of 463 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
16 179 463

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

Activity

  1. ChadNedzlek commented on Apr 12, 2022

    @ChadNedzlek
    Contributor

    So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

    @ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

  2. ulisesh commented on Apr 13, 2022

    @ulisesh
    Contributor

    I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

    In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

    I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

  3. markwilkie commented on Apr 13, 2022

    @markwilkie
    Contributor

    So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

  4. ulisesh commented on Apr 13, 2022

    @ulisesh
    Contributor

    I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

    This is the query:

    KnownIssues
    | where IssueId == 9009 and IssueRepository == "dotnet/arcade"

  5. self-assigned this
    on Apr 19, 2022
  6. removed their assignment
    on Jun 22, 2022
  7. changed the title [-]Checkout dotnet/runtime step failure: "Git fetch failed with exit code 128"[/-] [+]Checkout failure: "Git fetch failed with exit code 128"[/+] on Oct 7, 2022
  8. ViktorHofer commented on Jan 11, 2023

    @ViktorHofer
    Member

    This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?

  9. premun commented on Jan 11, 2023

    @premun
    Member

    GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b
    We saw problems cloning all over. Maybe it's related to that?

  10. dougbu commented on Jan 8, 2024

    @dougbu
    Contributor

    This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.com and a few also showed fatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.

    Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including "BuildRetry": true in the JSON content in this description❔

  11. carlossanlop commented on Jan 8, 2024

    @carlossanlop
    ContributorAuthor

    It's fine, we can try that, as long as we monitor it closely, because if we keep seeing the failure even with a retry, I think we should just go back to "BuildRetry": false, to avoid wasting resources.

  12. dougbu commented on Jan 8, 2024

    @dougbu
    Contributor

    you mean "BuildRetry": true was tried in the past and didn't help much @carlossanlop❔ if the extra time it takes PR creators to click the Rerun failed builds actually helps avoid issues in the retry, I'd lean toward leaving things as-is.

  13. carlossanlop commented on Jan 8, 2024

    @carlossanlop
    ContributorAuthor

    I do not know if it was tried in the past.

  14. AlitzelMendez commented on Jan 8, 2024

    @AlitzelMendez
    Member

    sounds like a good plan @dougbu, let's try "BuildRetry": true, I will put together a query to monitor if it is helpful for this issue

  15. dougbu commented on Jan 8, 2024

    @dougbu
    Contributor

    updated the description to try retries

  16. AlitzelMendez commented on Jan 8, 2024

    @AlitzelMendez
    Member

    query:

    let buildIds =
    KnownIssues
    | where ingestion_time() > datetime(2024-01-08 23:10:44.8683418)
    | where IssueId  == 9009
    | distinct BuildId;
    TimelineBuilds
    | where BuildId  in (buildIds)
    | summarize arg_max(FinishTime, *) by BuildId
    | summarize count() by Result, bin(FinishTime, 1d)
    | order by FinishTime asc 
    | render columnchart with (xcolumn=FinishTime)
    

    I will check this on Monday to see how it is going, posting here the query so I don't loose it :)

  17. AlitzelMendez commented on Jan 16, 2024

    @AlitzelMendez
    Member

    the success rate (having a green build) after the retry was between 30% to 50% per day, so this doesn't solve the issue, it just helps to mitigate it a little bit.

  18. removed their assignment
    on Jan 16, 2024
  19. added
    Ops - FacilelyOperations issues that are easily accomplished or attained
    on Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions