Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,17 @@ jobs:
if errorlevel 1 exit /b %errorlevel%
dir

- name: dotnet test
shell: powershell
run: |
$testProject = "$env:project_root\CollectSFData.Tests\CollectSFData.Tests.csproj"
$runSettings = "$env:project_root\CollectSFData.Tests\CollectSFData.Tests.runsettings"
foreach ($framework in @("net8.0", "net48")) {
write-host "dotnet test $testProject -c Release -f $framework --settings $runSettings"
dotnet test $testProject -c Release -f $framework --settings $runSettings --logger "console;verbosity=normal"
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
}

- name: build output
shell: cmd
run: |
Expand Down
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -329,3 +329,15 @@ ASALocalRun/
# MFractors (Xamarin productivity tool) working folder
.mfractor/
**/tst/*
# Test audit & coverage artifacts (added by automated test auditing fixture)
test-audits/
latest.json
coverage.opencover.xml
coverage.cobertura.xml
coverage.info
*.trx
# Potential crash / blame dumps
*.dmp

# IDE language-service caches
*.lscache
104 changes: 102 additions & 2 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ To report issue with collectsfdata, create a new issue in github repo [new issue
[CI CollectSFData](../.github/workflows/workflow.yml) builds pull requests targeting
`main` or `master` on opening, reopening, and subsequent updates with a read-only
contents token and no persisted checkout credentials. The existing documentation,
configuration, and Kusto path exclusions still apply.
configuration, and Kusto path exclusions still apply. After the build, CI runs the
`CollectSFData.Tests` project on net8.0 and net48 in the Release configuration;
a test failure fails the run.

Pushes to `main` or `master` build the exact pushed commit independently of PR
artifacts. A release is published only after that build succeeds and the GitHub
Expand Down Expand Up @@ -265,7 +267,105 @@ data ingestion uri: https://ingest-collectsfdataunittest.eastus.kusto.windows.ne

## Testing

todo
### Test Project Targets

The test project targets `net48` and `net8.0`, one full-framework and one modern runtime, with C# latest, nullable and implicit usings enabled for both.

### Running Tests (Quick Start)

```powershell
# Fast run for latest framework only
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -f net8.0

# Run all frameworks (slower)
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj

# Use custom runsettings (stabilizes host & disables parallelization)
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -f net8.0 --settings src/CollectSFData.Tests/CollectSFData.Tests.runsettings

# Filter to a subset (example new process tests)
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -f net8.0 --settings src/CollectSFData.Tests/CollectSFData.Tests.runsettings --filter "FullyQualifiedName~FileManagerProcessTests"
```

### Runsettings Rationale

`CollectSFData.Tests.runsettings` was added to mitigate intermittent file locking and PowerShell runspace errors by:

- Disabling AppDomain reuse (`DisableAppDomain=true`) so each run uses a clean test host.
- Forcing single-threaded execution (`MaxCpuCount=1`, xUnit parallelization off) to avoid simultaneous copies of large dependent assemblies.
- Increasing session timeout to allow slower PowerShell-invoking tests to complete without abrupt host termination.

Enable code coverage later by uncommenting the `XPlat code coverage` collector block inside the runsettings file.

### Test Audit Output (opt-in)

Set `COLLECTSFDATA_TEST_AUDIT=1` to have the test run write per-run metadata
(run id, OS, framework, git commit, coverage and TRX summaries) under the
gitignored `test-audits/` folder. Without the variable nothing is written.

### File Lock (MSB3021 / MSB3027) Mitigation Guide

Symptoms:

- Build warnings `MSB3026` escalating to errors `MSB3027` / `MSB3021` mentioning `CollectSFDataDll.dll` locked by `testhost`.
- Subsequent builds for the same TFM fail until the locking process exits.

Root Causes:

- Parallel test executions across target frameworks holding open file handles.
- Post-build PowerShell scripts executing while the prior testhost still loads the DLL.

Mitigations (apply in order):

1. Use single target with `--framework net8.0` while iterating.
2. Add `--no-build` when re-running quickly after a successful build.
3. Use the provided runsettings to disable parallelization & AppDomain reuse.
4. If locks persist, close IDE test sessions or manually terminate `testhost.exe` processes (outside constrained environments).
5. For CI: run target frameworks sequentially (`dotnet test -f net48` then `-f net8.0`) instead of simultaneous matrix on a single workspace if locks reappear.

### PowerShell Runspace Exception

Error: `System.Management.Automation.PSInvalidOperationException: There is no Runspace available…`

Causes & Remedies:

- Host teardown while a timer in a PowerShell-invoking component still references a disposed runspace.
- Long-running or asynchronous PowerShell script triggered during test cleanup.

Workarounds implemented:

- Longer `TestSessionTimeout` (10 minutes) to reduce abrupt cancellation.
- Serial execution eliminating overlapping teardown phases.

### Adding New Tests – Guidelines

Focus areas still needing coverage:

- Serialization chunking thresholds (CSV & JSON) for large collections.
- Compression branches (when configuration enables compression).
- Observer/ETL timeout & error pathways.
- Ingestion mode branching (Kusto vs Log Analytics vs none) with mocked dependencies.
- Concurrency safety of collection modifications under simulated parallel enumeration.

Use patterns already present (FluentAssertions + xUnit). Prefer small, fast unit tests; only integrate PowerShell/IO when asserting critical behaviors.

### Traits / Filtering (Optional Enhancement)

Introduce `[Trait("Category","Fast")]` vs `"Integration"` on future tests to enable:

```powershell
dotnet test --filter "Category=Fast"
```

### Coverage (Future)

Once stable, enable coverage collection:

```powershell
dotnet test -f net8.0 --settings src/CollectSFData.Tests/CollectSFData.Tests.runsettings /p:CollectCoverage=true
```

Or uncomment the coverage collector in runsettings for portable output formats (cobertura, lcov, opencover).

## Creating pull requests

Expand Down
107 changes: 107 additions & 0 deletions docs/testing-roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Testing & Quality Improvement Roadmap

Status: Active (Phase 2 in progress)

Owner: jagilber

Last Updated: 2026-10-21

## Goals

Establish sustainable, high-confidence test & quality framework for CollectSFData enabling safe refactors, feature velocity, and reliability.

## Success Metrics

- Progressive code coverage thresholds (core logic): 50% → 70% → 85%+
- <5% flaky test rate over rolling 30 days
- Zero secret tokens/keys in logs or saved configs (automated grep gate)
- Mutation score (core) ≥65% by Phase 5
- Time to add new ingestion type reduced (qualitative dev feedback)

## Phase Plan

| Phase | Focus | Deliverables | Target Coverage | Status |
|-------|-------|-------------|-----------------|--------|
| 0 | Baseline setup | Test infra, initial unit tests (time, file type, table naming), coverage reporting | ~10-15% | ✅ Complete |
| 1 | Seams & Decomposition | Interfaces (IClock, IFileSystem, IHttp), config refactor facade | 25-35% | ✅ Complete |
| 2 | Core Logic | Validation tests, merge precedence, golden-file parser fixtures | 40-50% | 🔄 In Progress |
| 3 | Integration Harness | Azurite setup, Kusto stub, E2E CLI happy/empty/error | 55-65% | Pending |
| 4 | Concurrency & Reliability | Replace CustomTaskManager, cancellation & timeout tests | 65-75% | Pending |
| 5 | Performance & Mutation | Benchmarks, Stryker, hotspot optimizations | 70-80% | Pending |
| 6 | Security & Hardening | Secret redaction, threat model checklist | 75-85% | Pending |
| 7 | Legacy Rationalization | TFM review (drop net462?), simplify conditionals | 80-85% | Pending |
| 8 | Governance & Polish | Analyzer enforcement, docs, contributor test guidelines | 85%+ | Pending |

## Immediate Tasks (Phase 0)

- [ ] Add test project `CollectSFData.Tests`
- [ ] Add baseline unit tests:
- [ ] `ConfigurationOptions.ConvertToUtcTime`
- [ ] `ConfigurationOptions.CleanTableName` (indirect via reflection or wrapper)
- [ ] File type parsing (private ConvertFileType via public side-effect of GatherType)
- [ ] Add coverage tooling (Coverlet + ReportGenerator)
- [ ] Add CI step (deferred to follow-up PR) for `dotnet test --collect:"XPlat Code Coverage"`
- [ ] Add README section pointing to roadmap
- [ ] Add redaction helper placeholder & tests (ensure no direct secret echo)

## Risk Register

| Risk | Impact | Mitigation |
|------|--------|------------|
| Static singletons limit test isolation | Slows test adoption | Introduce seam interfaces early (Phase 1) |
| Custom scheduler complexity | Flaky concurrency tests | Replace with channels workers (Phase 4) |
| Large refactor churn | Merge conflicts | Short-lived incremental PRs per phase |
| Legacy TFMs blocking APIs | Limits modernization | Evaluate usage metrics before removal |

## Test Strategy Summary

See prior analysis for detailed rationale. Key layers: unit (pure logic), integration (local Azurite + stubs), E2E CLI, performance, mutation, security redaction.

## Progress Log

| Date | Phase | Change | Author |
|------|-------|--------|--------|
| 2025-09-11 | 0 | Roadmap file created, test project scaffolded | (fill) |
| 2025-09-11 | 0 | Multi-target framework support (net462;net48;net6.0;net8.0) | (fill) |
| 2025-09-11 | 0 | Initial ConfigurationOptions unit tests (time parsing, file type, table naming) | (fill) |
| 2025-09-11 | 0→1 | Tests passing on modern frameworks (net6.0, net8.0); legacy signing issue identified | (fill) |
| 2025-09-11 | 1 | Implemented dependency injection pattern with ITimeProvider and IFileSystemAbstraction | (fill) |
| 2025-09-11 | 1 | Created TestableConfigurationOptions wrapper for dependency injection testing | (fill) |
| 2025-09-11 | 1 | Added Moq-based testing with mocked dependencies, 12 tests passing on modern frameworks | (fill) |
| 2025-09-11 | 1 | Phase 1 complete: Added test project to solution, ready for Phase 2 core logic expansion | (fill) |

## Current Status (2025-09-11)

✅ **Phase 0 & 1 Completed:**

- Test project `CollectSFData.Tests` created with multi-target framework support (net462;net48;net6.0;net8.0)
- 12 baseline unit tests implemented for `ConfigurationOptions` using dependency injection pattern
- Coverage collection infrastructure in place (Coverlet + TRX)
- Modern frameworks (net6.0, net8.0) passing all tests (12/12)
- Dependency injection interfaces: `ITimeProvider`, `IFileSystemAbstraction`
- Testable wrapper: `TestableConfigurationOptions` with Moq-based testing
- Test project added to solution file for IDE integration
- Branch `feature/testing-roadmap` active and stable

⚠️ **Known Issues:**

- Legacy frameworks (net462, net48) failing due to strong name verification
- Error: "Strong name signature could not be verified" for CollectSFDataDll assembly
- Expected behavior - test project signing disabled for easier testing

🔄 **Phase 2 Next Steps:**

1. Expand ConfigurationOptions test coverage to validation methods
2. Add tests for configuration merging and precedence logic
3. Test edge cases, malformed inputs, and error recovery
4. Generate coverage baseline for Phase 2 target (40-50%)

## Next Steps

1. Scaffold test project & add first three unit tests.
2. Integrate Coverlet & initial coverage badge generation script.
3. Document redaction approach & add failing test (TDD) before implementation.

---

This document is the authoritative plan; update table & log on each material change.
48 changes: 48 additions & 0 deletions src/CollectSFData.Tests/AuditFixtureTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
using System;
using System.IO;
using System.Linq;
using Xunit;

namespace CollectSFData.Tests
{
[Collection("TestAuditCollection")] // ensure fixture has run
public class AuditFixtureTests
{
[Fact]
public void AuditLatestJsonExists()
{
if (!TestAuditFixture.Enabled)
{
return; // audit output is opt-in, nothing to verify
}

// Discover repo root similarly to fixture logic (climb for .git or solution)
var dir = new DirectoryInfo(AppContext.BaseDirectory);
DirectoryInfo? solutionRoot = null;
DirectoryInfo? repoRoot = null;
for (int i = 0; i < 12 && dir != null; i++)
{
if (solutionRoot == null && dir.GetFiles("*.sln").Any())
{
solutionRoot = dir;
}
if (dir.GetDirectories(".git").Any())
{
repoRoot = dir;
break;
}
dir = dir.Parent;
}
repoRoot ??= solutionRoot ?? new DirectoryInfo(AppContext.BaseDirectory);
var auditsDir = Path.Combine(repoRoot.FullName, "test-audits");

// The fixture writes latest.json during construction (phase = start), so it should exist now.
var latestPath = Path.Combine(auditsDir, "latest.json");
Assert.True(File.Exists(latestPath), $"Expected audit latest.json at {latestPath} but it was not found.");

// Basic sanity: file should have non-trivial content
var length = new FileInfo(latestPath).Length;
Assert.True(length > 50, $"latest.json seems unexpectedly small ({length} bytes).");
}
}
}
39 changes: 39 additions & 0 deletions src/CollectSFData.Tests/CollectSFData.Tests.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<!-- One full-framework and one modern target cover both runtime families of CollectSFDataDll. -->
<TargetFrameworks>net48;net8.0</TargetFrameworks>
<LangVersion>latest</LangVersion>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<SignAssembly>false</SignAssembly>

<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="coverlet.collector" Version="6.0.0">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="FluentAssertions" Version="6.12.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="Moq" Version="4.20.70" />
<PackageReference Include="xunit" Version="2.5.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.3" />
</ItemGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net48'">
<PackageReference Include="System.Text.Json" Version="8.0.5" />
</ItemGroup>

<ItemGroup>
<Using Include="Xunit" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\CollectSFDataDll\CollectSFDataDll.csproj" />
</ItemGroup>

</Project>
41 changes: 41 additions & 0 deletions src/CollectSFData.Tests/CollectSFData.Tests.runsettings
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<DisableAppDomain>true</DisableAppDomain>
<MaxCpuCount>1</MaxCpuCount>
<TargetFrameworkVersion>.NETCoreApp,Version=v8.0</TargetFrameworkVersion>
<TestSessionTimeout>600000</TestSessionTimeout>
<CollectSourceInformation>true</CollectSourceInformation>
<ResultsDirectory>TestResults</ResultsDirectory>
<TreatNoTestsAsError>false</TreatNoTestsAsError>
<!-- Increase console verbosity for richer audit trail in CI logs -->
<ConsoleLoggerParameters>verbosity=detailed</ConsoleLoggerParameters>
</RunConfiguration>
<RunConfiguration>
<!-- legacy element duplication avoided by single block; ensuring minimal host reuse issues -->
</RunConfiguration>
<Loggers>
<!-- Standard TRX logger (default) plus an explicit console logger config -->
<Logger friendlyName="trx" enabled="true" />
<Logger friendlyName="console" enabled="true" />
</Loggers>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
<Configuration>
<Format>cobertura,lcov,opencover</Format>
</Configuration>
</DataCollector>
<DataCollector friendlyName="blame" enabled="true">
<Configuration>
<CollectDump DumpType="mini" CollectAlways="false" />
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
<xunit>
<ParallelizeTestCollections>false</ParallelizeTestCollections>
<ParallelizeAssemblies>false</ParallelizeAssemblies>
<MaxParallelThreads>1</MaxParallelThreads>
</xunit>
</RunSettings>
Loading
Loading