Repository navigation
Description
When I use the Checkov tool to scan a cloned Azure DevOps Git repo with an azure-pipelines.yml, I can see CKV_AZUREPIPELINES_ results, but when I use MicrosoftSecurityDevOps@1 in a pipeline for that very same repo, scanning the whole repo, I don't get these CKV_AZUREPIPELINES_ errors.
Don't get me wrong but, this seems like a missed easy win?
Thus, I'm not sure if this is a bug with my implementation or a missing feature. As such here are some details on my implementation, please let me know if I'm missing something obvious.
Here's my .gdcnconfig, derived from https://github.com/microsoft/security-devops-azdevops/wiki#checkov-gdnconfig-sample:
{
"tools": [
{
"tool": {
"name": "checkov",
"version": "Latest"
},
"arguments": {
"Directory": "$(Checkov.DefaultTargetDirectory)",
"Help": false,
"Version": false,
"OutputType": "sarif",
"List": false,
"Quiet": false,
"Compact": true,
"RunAllExternalChecks": false,
"Soft": false,
"ShowConfig": false,
"CreateBaseline": false,
"OutputBaselineAsSkipped": false,
"NoFailOnCrash": false,
"EnableSecretScanAllFiles": true
},
"outputExtension": "sarif",
"successfulExitCodes": [
0
],
"errorExitCodes": {
"1": "Checkov tool found issues.",
"2": "An error has occurred running the Checkov tool."
},
"outputPaths": []
}
]
}And the pipeline task:
- task: MicrosoftSecurityDevOps@1
env:
GDN_RUN_WORKINGDIRECTORY: $(Build.SourcesDirectory)
inputs:
break: true
publish: falseI do get the other results from Checkov, although not always on par with calling standalone Checkov itself.
There seems to be some behind-the-scenes magic at work. For instance, when using MicrosoftSecurityDevOps@1 I found CKV_AZURE_177 downgraded to a warning, where it's an error when I call checkov standalone to perform what I think is the same scan.
Here's how I've been performing a manual scan with Checkov, that seems to look deeper and be more strict than the task's call does:
checkov -d "$(Build.SourcesDirectory)" --repo-root-for-plan-enrichment "$(Build.SourcesDirectory)" --deep-analysis -o cli -o sarif --output-file-path console,$results
Activity
ncook-hxgn commentedon May 13, 2026
.. OK fine, I was using
TargetDirectory. When I usedDirectory, MSDO returns the following:So, the documented example on the wiki seems broken.
DimaBir commentedon May 13, 2026
Hi ncook-hxgn — thanks for the detailed report and the side-by-side comparison, that's very helpful.
A few things worth checking before we dig deeper:
Argument key name. In your
.gdnconfigthe Checkov argument is named"Directory", but the wrapper's schema expects"TargetDirectory"(it translates to--directory). When the key doesn't match, the value is silently ignored and Guardian falls back to its default working directory. The wiki sample usesTargetDirectory, and the same gotcha came up in Help me with the example of using environment variables with values for checkov #121. Could you rename it and re-run?The actual Checkov invocation. In the pipeline logs, look for the line that starts with
…/Microsoft.Guardian.CheckovRedist_…/tools/dist/checkov …. That line shows the exact CLI Guardian is executing — bundled version, directory, frameworks, soft-fail, etc. Sharing it (and the resulting.gdn/.r/checkov/001/checkov.sarif) would let us tell quickly whetherazure_pipelinesframework results are being produced and filtered out downstream, or simply never produced.--deep-analysis/--repo-root-for-plan-enrichment. These flags are not currently surfaced by the wrapper's argument schema, so the wrapped invocation won't pass them. They primarily affect Terraform plan enrichment rather than theCKV_AZUREPIPELINES_*rule family, so I wouldn't expect them to explain the missing Azure Pipelines results — but worth flagging.CKV_AZURE_177severity. The task applies Guardian's severity mapping on top of Checkov's output, which can diverge from Checkov's defaults. That's by design rather than a bug, but if you have a specific case where the downgrade hides something material, please call that out separately.If you can share (a) the log line above, (b) the SARIF the task produces, and (c) the result of changing
Directory→TargetDirectory, that should be enough to confirm whether this is a config issue, a wrapper issue, or a Guardian-side filter. Thanks!DimaBir commentedon May 13, 2026
You're right, and apologies for the confused direction — I had your
.gdnconfigand the wiki sample mixed up in my head. To confirm what you've established:TargetDirectoryis correct (matchessamples/configs/checkov.gdnconfigin this repo).Directoryis rejected by the wrapper withUnrecognizedArgumentException— that's the error you just hit."Directory": ""while the env-var table immediately below mapsDirectory → GDN_CHECKOV_TARGETDIRECTORY. The JSON snippet is the broken half — it should read"TargetDirectory". I'll get a wiki fix in for that. Thanks for catching it.Now, back to the original
CKV_AZUREPIPELINES_*question — since your real config (withTargetDirectory) does scan and does return Checkov results, the most useful artifact would still be:…/Microsoft.Guardian.CheckovRedist_…/tools/dist/checkov …invocation line from your task logs (so we can see the bundled version and the exact flags Guardian passed)..gdn/.r/checkov/001/checkov.sarif(or at least itsruns[0].tool.driver.rules[*].idlist, filtered toCKV_AZUREPIPELINES_*) so we can tell whether the rules were executed but filtered out downstream vs never executed at all.That'll let me say definitively whether this is a framework/version gap in the bundled Checkov, a Guardian post-processing filter, or something else. Sorry again for the detour.
DimaBir commentedon May 13, 2026
Quick follow-up: the wiki is updated and live — the Checkov gdnconfig sample and the env-var table now both say
TargetDirectory. Thanks again for catching that.The
CKV_AZUREPIPELINES_*question is still open — when you have a chance, the Guardian invocation log line and.gdn/.r/checkov/001/checkov.sariffrom a real run would let me give you a definitive answer.ncook-hxgn commentedon May 13, 2026
1., 2. So, I found a combination of what I had working before + what I saw in the wiki example that surfaces* the
CKV_AZUREPIPELINES_*errors - I changed from$(Checkov.DefaultTargetDirectory)to$(WorkingDirectory)and returned toTargetDirectoryrather than the documentedDirectoryfrom the sample.* I can see them in the cli output from Checkov as called by guardian,
but they aren't showing up in the SAST Scans tab collapse for
checkov. They are present in theCheckovcollapse from my standalone call though.. (I can't get good screen shots of the omissions from the SAST Scans tab as I have quite a lot of results)..Standalone call's sarif in Scans tab:
MicrosoftSecurityDevOps@1's results in Scans tab:
So, whilst the following .gdnconfig gets Checkov to find my azure-pipelines.yml and checkov does discover and record pipeline issues to CLI, I can't see them in the final sarif.
CKV_AZUREPIPELINES_*results from the final sarif that is rendered by the Scans tab?{ "tool": { "name": "checkov", "version": "Latest" }, "arguments": { "TargetDirectory": "$(WorkingDirectory)", "Help": false, "Version": false, "OutputType": "sarif", "List": false, "Quiet": false, "Compact": true, "RunAllExternalChecks": false, "Soft": false, "ShowConfig": false, "CreateBaseline": false, "OutputBaselineAsSkipped": false, "NoFailOnCrash": false, "EnableSecretScanAllFiles": true }, "outputExtension": "sarif", "successfulExitCodes": [ 0 ], "errorExitCodes": { "1": "Checkov tool found issues.", "2": "An error has occurred running the Checkov tool." }, "outputPaths": [] }ncook-hxgn commentedon May 13, 2026
And here is the requested guardian call for checkov, output from pipeline log:
ncook-hxgn commentedon May 13, 2026
Let me see if I can do some jq stuff in the pipeline to get this info, hang tight :)
ncook-hxgn commentedon May 13, 2026
Here's the output from the query of the checkov sarif file.
Sorry about the wonky filename, I've been playing with
outputPathsin the .gdnconfigsI digress. I also managed to extract the ruleids from the msdo.sarif emitted by the task - the
CKV_AZUREPIPELINES_1andCKV_AZUREPIPELINES_2are present..but yet not visible in the scans tab. That's super weird, I'm not sure how to explain that. Somehow the checkov-results.sarif file from standalone checkov calls, when uploaded to CodeAnalysisLogs, definitely shows the CSKV_AZUREPIPELINES_* stuff in the SARIF Scans tab.
DimaBir commentedon May 13, 2026
Thanks for the thorough data — this changes the diagnosis.
Both queries confirm
CKV_AZUREPIPELINES_1andCKV_AZUREPIPELINES_2do survive intomsdo.sarif. So the drop isn't happening in Guardian / MSDO's SARIF processing — those findings are in the publishedCodeAnalysisLogsartifact. The disappearance is in the rendering layer readingmsdo.sarif(the SARIF Scans Tab extension).Two structural hypotheses, in priority order:
Orphaned
results[].ruleIdreferences. SARIF allowsresults[].ruleIdto refer to a rule that isn't defined inruns[*].tool.driver.rules[]. Most SARIF viewers — including the Scans Tab — silently skip results whose ruleId doesn't resolve to a rule definition. Your jq queriedresults[].ruleIdinmsdo.sarifbut did not querytool.driver.rules[].id. If MSDO's SARIF merge keeps the results but loses some rule definitions, AZUREPIPELINES findings would be present-but-invisible.Multi-run SARIF, viewer reading only one run. Your
msdo.sarifjq used.runs[](plural). The duplicateruleIds in your dump (CKV_AZURE_221× 3,CKV_SECRET_6× 3) are a strong signal that MSDO is concatenating multiple Checkov runs into one file. If the Scans Tab renders onlyruns[0](or one run per tool name), and the AZUREPIPELINES results live in a non-first run, they'd be invisible regardless of correctness.A single jq query disambiguates these. Could you run this against
msdo.sarif(not the raw Checkov one) and share the output?For each Checkov run in
msdo.sarifit returns the rules defined in that run and the unique ruleIds the results reference. Interpretation:CKV_AZUREPIPELINES_*appears inresult_ids_uniquefor some run but not in that same run'srule_ids→ hypothesis 1 (orphaned references).Either way, the fix would be on our side (or on the Scans Tab extension's side); your config is doing what it should. Thanks again for the careful reproduction — this is rapidly converging on a precise root cause.
ncook-hxgn commentedon May 13, 2026
Our .gdn config has configurations for trivy, terrascan, and checkov, so there are indeed three sets of results to be fed into msdo.sarif, alas, they aren't all checkov.
brb :)
ncook-hxgn commentedon May 13, 2026
ncook-hxgn commentedon May 13, 2026
This isn't quite correct: we run trivy, terrascan and now checkov with MicrosoftSecurityDevOps@1, and we get results for all three of those tools in the Scans tab, only the checkov results omit the AZUREPIPELINES result.
When I run checkov "standalone" and upload the generated .sarif, I do get the CKV_AZUREPIPELINES_* issues rendered in the Scans tab.
Yeilds:
As such, I'm not convinced that it's the Scans tab that's at fault.
Edit:
Comparing the sarif results file from checkov itself against msdo.sarif, I found both instances of the CKV_AZUREPIPELINES_2 issue. It seems I can't see it because Guardian has downgraded this to a
notewhere checkov initially raises this as an error:Ahhh - the penny drops. Because they are 'note' in Guardian's opinion, the Scans tab filters them out. When I adjust the filter to include Note level issues, they appear!
ncook-hxgn commentedon May 13, 2026
So, I think I would reposit my overall question: please could CKV_AZUREPIPELINES_ be promoted to at least
warning?With all the checkmarx and trivy malarkey that's gone on recently, I think avoiding the
latesttag on container images used by pipelines is probably more important than it was when the logic to downgrade this to a note was decided on originally.DimaBir commentedon May 13, 2026
You nailed it — and apologies for the detour. Looking back, my very first reply flagged Guardian's severity mapping (point 4) as a possible factor, but I dropped that thread and chased structural SARIF hypotheses instead. You did the work my point 4 should have done.
To summarize where we landed for anyone reading later:
CKV_AZUREPIPELINES_*tolevel: "note"in the publishedmsdo.sarif. The SARIF Scans Tab extension's default severity filter hides notes, so the findings render only when you explicitly includeNotein the filter.msdo.sarifand inCodeAnalysisLogs; they're just below the default visibility threshold.On your actual request — promoting
CKV_AZUREPIPELINES_*to at leastwarning: that's a reasonable ask, and your reasoning lines up with where the threat landscape has moved (pinning images vslatest, untrusted third-party tasks, etc.). The severity mapping lives in Guardian's policy rather than in this task's argument schema, so it isn't something we can flip via a config option in.gdnconfig— it needs a policy update on the Guardian side. I'll flag this with the team and treat this issue as the feature/policy request going forward.Separately, the "findings present in the file but hidden by the default Scans Tab filter" UX is worth a follow-up of its own — even users who want note-level findings won't necessarily think to change the filter. I'll mention that alongside the severity ask.
Re-labeled accordingly. Thanks again for the careful repro and the patience through my wrong turn.
DimaBir commentedon May 13, 2026
CC: James Brotsos (@jbrotsos)
ncook-hxgn commentedon May 14, 2026
Much appreciated Dima Birenbaum (@DimaBir)
DimaBir commentedon May 16, 2026
Filed the spec as a separate trackable issue so it can be assigned and worked independently: #164. This issue (#163) will close automatically when #164 is merged per the spec's definition of done.