Repository navigation
Open
Labels
Known Build ErrorOps - FacilelyOperations issues that are easily accomplished or attainedOperations issues that are easily accomplished or attained
Description
Build
https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results
Build leg reported
Libraries Test Run release coreclr Linux_musl x64 Release
Pull Request
Rolling build for runtime
Action required for the engineering services team
To triage this issue (First Responder / @dotnet/dnceng):
- Open the failing build above and investigate
- Add a comment explaining your findings
If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:
- Add the label "Known Build Error"
- Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation
{
"ErrorMessage" : "Git fetch failed with exit code: 128",
"BuildRetry": true
}Additional information about the issue reported
The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:
remote: Compressing objects: 29% (30070/103687)
remote: Compressing objects: 30% (31107/103687)
remote: Compressing objects: 31% (32143/103687)
remote: Compressing objects: 31% (32322/103687)
remote: Compressing objects: 32% (33180/103687)
remote: Compressing objects: 33% (34217/103687)
remote: Compressing objects: 33% (35247/103687)
remote: Compressing objects: 34% (35254/103687)
remote: Compressing objects: 35% (36291/103687)
remote: Compressing objects: 36% (37328/103687)
remote: Compressing objects: 37% (38365/103687)
remote: Compressing objects: 37% (38926/103687)
remote: Compressing objects: 38% (39402/103687)
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s
Report
| Build | Repository | Step Name | Console log | Pull Request |
|---|---|---|---|---|
| 1687534 | dotnet/maui | Checkout dotnet/maui@refs/pull/40369/merge to s | Log | dotnet/maui#40369 |
| 1687513 | dotnet/maui | Checkout dotnet/maui@refs/pull/40368/merge to s | Log | dotnet/maui#40368 |
| 1687370 | dotnet/maui | Checkout dotnet/maui@refs/pull/40366/merge to s | Log | dotnet/maui#40366 |
| 1687346 | dotnet/maui | Checkout dotnet/maui@refs/pull/40284/merge to s | Log | dotnet/maui#40284 |
| 1687306 | dotnet/maui | Checkout dotnet/maui@refs/pull/39975/merge to s | Log | dotnet/maui#39975 |
| 1687312 | dotnet/maui | Checkout dotnet/maui@refs/pull/40234/merge to s | Log | dotnet/maui#40234 |
| 1687242 | dotnet/maui | Checkout dotnet/maui@refs/pull/40363/merge to s | Log | dotnet/maui#40363 |
| 1687266 | dotnet/maui | Checkout dotnet/maui@refs/pull/40295/merge to s | Log | dotnet/maui#40295 |
| 1687285 | dotnet/maui | Checkout dotnet/maui@refs/pull/40365/merge to s | Log | dotnet/maui#40365 |
| 1687225 | dotnet/maui | Checkout dotnet/maui@refs/pull/39979/merge to s | Log | dotnet/maui#39979 |
| 1687003 | dotnet/maui | Checkout dotnet/maui@refs/pull/40318/merge to s | Log | dotnet/maui#40318 |
| 1686708 | dotnet/maui | Checkout dotnet/maui@candidate/11.0.1xx-sr2 to s | Log | |
| 3138748 | dotnet-dotnet | Checkout dotnet-dotnet@refs/pull/68652/merge to s | Log | #68652 |
| 1685697 | dotnet/maui | Checkout dotnet/maui@refs/pull/40334/merge to s | Log | dotnet/maui#40334 |
| 1685551 | dotnet/maui | Checkout dotnet/maui@refs/pull/40284/merge to s | Log | dotnet/maui#40284 |
| 1685442 | dotnet/maui | Checkout dotnet/maui@refs/pull/40133/merge to s | Log | dotnet/maui#40133 |
| 1685390 | dotnet/maui | Checkout dotnet/maui@refs/pull/40234/merge to s | Log | dotnet/maui#40234 |
| 1685397 | dotnet/maui | Checkout dotnet/maui@refs/pull/40291/merge to s | Log | dotnet/maui#40291 |
| 3138494 | dotnet-runtime | Checkout dotnet-runtime@main to s | Log | |
| 1685276 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1685291 | dotnet/maui | Checkout dotnet/maui@refs/pull/40316/merge to s | Log | dotnet/maui#40316 |
| 1685312 | dotnet/maui | Checkout dotnet/maui@refs/pull/39917/merge to s | Log | dotnet/maui#39917 |
| 1685121 | dotnet/maui | Checkout dotnet/maui@refs/pull/40153/merge to s | Log | dotnet/maui#40153 |
| 1684940 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1685100 | dotnet/maui | Checkout dotnet/maui@refs/pull/40324/merge to s | Log | dotnet/maui#40324 |
| 1685030 | dotnet/maui | Checkout dotnet/maui@refs/pull/40283/merge to s | Log | dotnet/maui#40283 |
| 1685074 | dotnet/maui | Checkout dotnet/maui@refs/pull/40295/merge to s | Log | dotnet/maui#40295 |
| 1685023 | dotnet/maui | Checkout dotnet/maui@refs/pull/39975/merge to s | Log | dotnet/maui#39975 |
| 1684830 | dotnet/maui | Checkout dotnet/maui@refs/pull/40307/merge to s | Log | dotnet/maui#40307 |
| 1684827 | dotnet/maui | Checkout dotnet/maui@refs/pull/40316/merge to s | Log | dotnet/maui#40316 |
| 1684652 | dotnet/maui | Checkout dotnet/maui@refs/pull/40318/merge to s | Log | dotnet/maui#40318 |
| 1684820 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1684363 | dotnet/maui | Checkout dotnet/maui@refs/pull/40311/merge to s | Log | dotnet/maui#40311 |
| 1684332 | dotnet/maui | Checkout dotnet/maui@refs/pull/40309/merge to s | Log | dotnet/maui#40309 |
| 1684349 | dotnet/maui | Checkout dotnet/maui@refs/pull/40310/merge to s | Log | dotnet/maui#40310 |
| 1684308 | dotnet/runtime | Checkout dotnet/runtime@refs/pull/137691/merge to s | Log | dotnet/runtime#137691 |
| 1684233 | dotnet/runtime | Checkout dotnet/runtime@refs/pull/137685/merge to s | Log | dotnet/runtime#137685 |
| 1684371 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1684339 | dotnet/maui | Checkout dotnet/maui@refs/pull/40305/merge to s | Log | dotnet/maui#40305 |
| 1684527 | dotnet/arcade-services | Checkout dotnet/arcade-services@refs/pull/6895/merge to s | Log | dotnet/arcade-services#6895 |
| 1684468 | dotnet/maui | Checkout dotnet/maui@refs/pull/40312/merge to s | Log | dotnet/maui#40312 |
| 1684279 | dotnet/maui | Checkout dotnet/maui@inflight/current to s | Log | |
| 1684337 | dotnet/maui | Checkout dotnet/maui@refs/pull/40305/merge to s | Log | dotnet/maui#40305 |
| 1684196 | dotnet/maui | Checkout dotnet/maui@candidate/11.0.1xx-sr2 to s | Log | |
| 1683498 | dotnet/maui | Checkout dotnet/maui@refs/pull/40295/merge to s | Log | dotnet/maui#40295 |
| 3137858 | dotnet-runtime | Checkout dotnet-runtime@main to s | Log | |
| 1684035 | dotnet/maui | Checkout dotnet/maui@refs/pull/40304/merge to s | Log | dotnet/maui#40304 |
| 1683994 | dotnet/maui | Checkout dotnet/maui@candidate/11.0.1xx-sr2 to s | Log | |
| 1683495 | dotnet/maui | Checkout dotnet/maui@refs/pull/39979/merge to s | Log | dotnet/maui#39979 |
| 1683482 | dotnet/maui | Checkout dotnet/maui@refs/pull/40283/merge to s | Log | dotnet/maui#40283 |
| 1683828 | dotnet/machinelearning | Checkout dotnet/machinelearning@refs/pull/7918/merge to s | Log | dotnet/machinelearning#7918 |
| 1683391 | dotnet/maui | Checkout dotnet/maui@refs/pull/39974/merge to s | Log | dotnet/maui#39974 |
| 1683386 | dotnet/maui | Checkout dotnet/maui@refs/pull/40284/merge to s | Log | dotnet/maui#40284 |
| 1683329 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1683480 | dotnet/maui | Checkout dotnet/maui@refs/pull/40283/merge to s | Log | dotnet/maui#40283 |
| 1683163 | dotnet/maui | Checkout dotnet/maui@refs/pull/39975/merge to s | Log | dotnet/maui#39975 |
| 1683015 | dotnet/maui | Checkout dotnet/maui@refs/pull/40153/merge to s | Log | dotnet/maui#40153 |
| 1683169 | dotnet/maui | Checkout dotnet/maui@refs/pull/40234/merge to s | Log | dotnet/maui#40234 |
| 1683011 | dotnet/maui | Checkout dotnet/maui@refs/pull/40133/merge to s | Log | dotnet/maui#40133 |
| 1682962 | dotnet/maui | Checkout dotnet/maui@refs/pull/40283/merge to s | Log | dotnet/maui#40283 |
| 1683005 | dotnet/maui | Checkout dotnet/maui@refs/pull/40284/merge to s | Log | dotnet/maui#40284 |
| 1683060 | dotnet/sdk | Checkout dotnet/sdk@main to s | Log | |
| 1682949 | dotnet/maui | Checkout dotnet/maui@refs/pull/39655/merge to s | Log | dotnet/maui#39655 |
| 1682433 | dotnet/maui | Checkout dotnet/maui@refs/pull/40263/merge to s | Log | dotnet/maui#40263 |
| 1682374 | dotnet/runtime | Checkout dotnet/runtime@refs/pull/137594/merge to s | Log | dotnet/runtime#137594 |
| 1682345 | dotnet/maui | Checkout dotnet/maui@refs/pull/40272/merge to s | Log | dotnet/maui#40272 |
| 1682246 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1682348 | dotnet/maui | Checkout dotnet/maui@refs/pull/40263/merge to s | Log | dotnet/maui#40263 |
| 1682306 | dotnet/maui | Checkout dotnet/maui@refs/pull/40182/merge to s | Log | dotnet/maui#40182 |
| 1682296 | dotnet/maui | Checkout dotnet/maui@refs/pull/40271/merge to s | Log | dotnet/maui#40271 |
| 1682299 | dotnet/maui | Checkout dotnet/maui@refs/pull/40270/merge to s | Log | dotnet/maui#40270 |
| 1682257 | dotnet/maui | Checkout dotnet/maui@refs/pull/40188/merge to s | Log | dotnet/maui#40188 |
| 3136602 | dotnet-dotnet | Checkout dotnet-dotnet@release/11.0.1xx to s | Log | |
| 1682094 | dotnet/maui | Checkout dotnet/maui@refs/pull/40263/merge to s | Log | dotnet/maui#40263 |
| 1682087 | dotnet/maui | Checkout dotnet/maui@refs/pull/40261/merge to s | Log | dotnet/maui#40261 |
| 1682044 | dotnet/maui | Checkout dotnet/maui@refs/pull/40263/merge to s | Log | dotnet/maui#40263 |
| 1681969 | dotnet/maui | Checkout dotnet/maui@refs/pull/39974/merge to s | Log | dotnet/maui#39974 |
| 1681712 | dotnet/maui | Checkout dotnet/maui@refs/pull/40188/merge to s | Log | dotnet/maui#40188 |
| 1681675 | dotnet/maui | Checkout dotnet/maui@refs/pull/40263/merge to s | Log | dotnet/maui#40263 |
| 1681699 | dotnet/diagnostics | Checkout dotnet/diagnostics@refs/pull/6248/merge to s | Log | dotnet/diagnostics#6248 |
| 1681588 | dotnet/maui | Checkout dotnet/maui@refs/pull/40188/merge to s | Log | dotnet/maui#40188 |
| 1681382 | dotnet/maui | Checkout dotnet/maui@refs/pull/40254/merge to s | Log | dotnet/maui#40254 |
| 1681510 | dotnet/runtime | Checkout dotnet/runtime@main to s | Log | |
| 1681463 | dotnet/runtime | Checkout dotnet/runtime@refs/pull/137548/merge to s | Log | dotnet/runtime#137548 |
| 1681453 | dotnet/maui | Checkout dotnet/maui@refs/pull/40256/merge to s | Log | dotnet/maui#40256 |
| 1681198 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1681388 | dotnet/maui | Checkout dotnet/maui@refs/pull/40255/merge to s | Log | dotnet/maui#40255 |
| 1681353 | dotnet/maui | Checkout dotnet/maui@refs/pull/40182/merge to s | Log | dotnet/maui#40182 |
| 3136080 | dotnet-dotnet | Checkout dotnet-dotnet@release/11.0.2xx to s | Log | |
| 1681126 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1681125 | dotnet/diagnostics | Checkout dotnet/diagnostics@refs/pull/6239/merge to s | Log | dotnet/diagnostics#6239 |
| 1681124 | dotnet/runtime | Checkout dotnet/runtime@main to s | Log | |
| 1681039 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1680949 | dotnet/maui | Checkout dotnet/maui@refs/pull/40060/merge to s | Log | dotnet/maui#40060 |
| 1680590 | dotnet/maui | Checkout dotnet/maui@refs/pull/40183/merge to s | Log | dotnet/maui#40183 |
| 1680449 | dotnet/maui | Checkout dotnet/maui@refs/pull/40166/merge to s | Log | dotnet/maui#40166 |
| 1680593 | dotnet/maui | Checkout dotnet/maui@refs/pull/40182/merge to s | Log | dotnet/maui#40182 |
| 1680587 | dotnet/maui | Checkout dotnet/maui@refs/pull/40184/merge to s | Log | dotnet/maui#40184 |
| 1680579 | dotnet/maui | Checkout dotnet/maui@refs/pull/40221/merge to s | Log | dotnet/maui#40221 |
| 1680701 | dotnet/runtime | Checkout dotnet/runtime@refs/pull/137520/merge to s | Log | dotnet/runtime#137520 |
| Displaying 100 of 463 results |
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 16 | 179 | 463 |
Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC
Metadata
Metadata
Assignees
Labels
Known Build ErrorOps - FacilelyOperations issues that are easily accomplished or attainedOperations issues that are easily accomplished or attained
Activity
ChadNedzlek commentedon Apr 12, 2022
So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.
@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.
ulisesh commentedon Apr 13, 2022
I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.
In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.
I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.
markwilkie commentedon Apr 13, 2022
So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)
ulisesh commentedon Apr 13, 2022
I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄
This is the query:
KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"
[-]Checkout dotnet/runtime step failure: "Git fetch failed with exit code 128"[/-][+]Checkout failure: "Git fetch failed with exit code 128"[/+]ViktorHofer commentedon Jan 11, 2023
This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?
premun commentedon Jan 11, 2023
GitHub reported degraded git operations yesterday: https://www.githubstatus.com/incidents/00km0kwpz23b
We saw problems cloning all over. Maybe it's related to that?
dougbu commentedon Jan 8, 2024
This issue may have started in early 2022 but seems to remain w/ 29 matching failures in the last 24 hours. https://www.githubstatus.com/ and various Azure / Azure DevOps status pages aren't showing any current or recent degradations. Unclear what's going on this time beyond what appears to be DNS flakiness. I looked at the last 7 reports; all but one showed
fatal: unable to access 'https://github.com/dotnet/runtime/': Could not resolve host: github.comand a few also showedfatal: unable to access 'https://github.com/dotnet/runtime/': Failed to connect to github.com port 443 after 75012 ms: Couldn't connect to server.Without more information, I think the best we could do is auto retry when the problem occurs. @AlitzelMendez and @carlossanlop what are your thoughts on including
"BuildRetry": truein the JSON content in this description❔carlossanlop commentedon Jan 8, 2024
It's fine, we can try that, as long as we monitor it closely, because if we keep seeing the failure even with a retry, I think we should just go back to
"BuildRetry": false, to avoid wasting resources.dougbu commentedon Jan 8, 2024
you mean
"BuildRetry": truewas tried in the past and didn't help much @carlossanlop❔ if the extra time it takes PR creators to click theRerun failed buildsactually helps avoid issues in the retry, I'd lean toward leaving things as-is.carlossanlop commentedon Jan 8, 2024
I do not know if it was tried in the past.
AlitzelMendez commentedon Jan 8, 2024
sounds like a good plan @dougbu, let's try
"BuildRetry": true, I will put together a query to monitor if it is helpful for this issuedougbu commentedon Jan 8, 2024
updated the description to try retries
AlitzelMendez commentedon Jan 8, 2024
query:
I will check this on Monday to see how it is going, posting here the query so I don't loose it :)
AlitzelMendez commentedon Jan 16, 2024
the success rate (having a green build) after the retry was between 30% to 50% per day, so this doesn't solve the issue, it just helps to mitigate it a little bit.