SQE Local Lab

Overview

SQE Local Lab is a local .NET application that emulates the core behavior of Geneva Standing Query Extension (SQE).

The project demonstrates how telemetry events are:

1. Generated
2. Serialized to JSON
3. Written to an event stream
4. Evaluated against standing queries
5. Converted into metric points
6. Aggregated by dimensions
7. Emitted as metric outputs

Everything runs locally without:

• Geneva
• Monitoring Agent
• ETW
• Kusto
• Azure dependencies

This project is intended purely for learning and understanding the SQE processing model.

---

High-Level Architecture

OrderEvent
    ↓
StandingQueryDefinition
    ↓
LocalTelemetryPipeline
    ↓
LocalStandingQueryEvaluator
    ↓
QueryEvaluationResult
    ↓
LocalMetricPointFactory
    ↓
MetricPoint
    ↓
LocalMetricAggregator
    ↓
AggregatedMetricPoint

---

Project Structure

SqeLocalLab
│
├── Config
│   └── queries.json
│
├── Engine
│   ├── LocalMetricAggregator.cs
│   ├── LocalMetricPointFactory.cs
│   ├── LocalStandingQueryEvaluator.cs
│   └── QueryEvaluationResult.cs
│
├── Models
│   ├── AggregatedMetricPoint.cs
│   ├── MetricPoint.cs
│   ├── OrderEvent.cs
│   └── StandingQueryDefinition.cs
│
├── Pipeline
│   ├── LocalTelemetryPipeline.cs
│   └── PipelineExecutionResult.cs
│
├── output
│   ├── events.jsonl
│   ├── metrics.jsonl
│   └── aggregated-metrics.jsonl
│
└── Program.cs

---

Processing Flow

1. Create Events

Telemetry events are represented using:

OrderEvent

Example:

OrderProcessed
StatusCode = 500
OrderId = ORD-002

---

2. Write Event Stream

Events are serialized using:

JsonSerializer.Serialize(...)

and written to:

output/events.jsonl

Example:

{"statusCode":200}
{"statusCode":500}
{"statusCode":503}

Purpose:

Simulate an incoming telemetry stream

---

3. Load Standing Queries

Queries are loaded from:

Config/queries.json

Example:

{
  "name": "FailedOrders",
  "filter": {
    "eventName": "OrderProcessed",
    "minimumStatusCode": 500
  }
}

Purpose:

Separate business rules from code

---

4. Evaluate Events

Events are evaluated using:

LocalStandingQueryEvaluator

Result:

QueryEvaluationResult

Possible outcomes:

MATCHED
NOT MATCHED

Example:

StatusCode = 200
→ NOT MATCHED

StatusCode = 500
→ MATCHED

StatusCode = 503
→ MATCHED

---

5. Generate Metric Points

Matched evaluations are converted into:

MetricPoint

using:

LocalMetricPointFactory

Output:

output/metrics.jsonl

Example:

{
  "name": "FailedOrderCount",
  "value": 1
}

Concept:

Match
    →
Metric Contribution

---

6. Extract Dimensions

The metric factory extracts configured dimensions.

Example dimensions:

data.operationName
data.statusCode
ext.environment

Possible result:

operationName = CreateOrder
statusCode = 500
environment = Local

---

7. Aggregate Metrics

Raw metric points are aggregated using:

LocalMetricAggregator

Result:

AggregatedMetricPoint

Output:

output/aggregated-metrics.jsonl

---

Important Metric Concept

A metric series is determined by:

Metric Name
    +
Dimension Set
    =
Time Series

Example:

Scenario A

Dimensions:

operationName
environment

Results:

MetricPoint #1
CreateOrder
Local

MetricPoint #2
CreateOrder
Local

Aggregate:

FailedOrderCount = 2

---

Scenario B

Dimensions:

operationName
statusCode
environment

Results:

CreateOrder
500
Local

and

CreateOrder
503
Local

Aggregate:

FailedOrderCount(status=500) = 1

FailedOrderCount(status=503) = 1

Two separate metric series are produced because the dimension values differ.

---

LocalTelemetryPipeline

The processing engine is encapsulated inside:

LocalTelemetryPipeline

Purpose:

Program.cs
    ↓
Pipeline
        ↓
Evaluator
        ↓
Metric Factory
        ↓
Aggregator

This separates orchestration from processing logic.

---

Output Files

events.jsonl

Contains all generated events.

Example:

{"statusCode":200}
{"statusCode":500}
{"statusCode":503}

---

metrics.jsonl

Contains raw metric contributions.

Example:

{
  "name":"FailedOrderCount",
  "value":1
}

One line per metric contribution.

---

aggregated-metrics.jsonl

Contains aggregated metrics.

Example:

{
  "name":"FailedOrderCount",
  "value":2
}

One line per final metric series.

---

Automated Testing

A separate project validates all behavior:

SqeLocalLab.SelfTests

Result:

Passed: 8
Failed: 0

---

Test Cases

1. SuccessfulOrderDoesNotMatch

Input:

StatusCode = 200

Expected:

NOT MATCHED

Validates:

Query filtering

---

2. FailedOrderMatches

Input:

StatusCode = 500

Expected:

MATCHED

Validates:

Boundary condition

---

3. MatchedResultProducesMetricPoint

Expected:

MetricPoint created
Value = 1

Validates:

Metric generation

---

4. MetricPointContainsDimensions

Expected:

Required dimensions exist

Validates:

Dimension extraction

---

5. EqualDimensionsAggregateTogether

Expected:

1 aggregate
Value = 2

Validates:

Grouping behavior

---

6. DifferentDimensionsCreateSeparateAggregates

Expected:

2 aggregates

Validates:

Dimension-based separation

---

7. UnknownDimensionPathIsRejected

Expected:

InvalidOperationException

Validates:

Configuration validation

---

8. NonMatchingResultCannotProduceMetric

Expected:

Metric creation blocked

Validates:

System correctness

---

Key Learning Outcomes

By completing this project, you implemented and understood:

• Telemetry event modeling
• JSON serialization
• JSONL event streams
• Standing query configuration
• Query evaluation
• Metric generation
• Dimension extraction
• Metric aggregation
• Time-series concepts
• Pipeline architecture
• Automated testing
• Configuration validation

---

Current Status

✅ Local telemetry event stream

✅ Standing query engine

✅ Metric generation

✅ Dimension extraction

✅ Metric aggregation

✅ Pipeline abstraction

✅ Automated validation

✅ 8/8 tests passing

✅ Fully local execution

The application now serves as a small but complete SQE-style telemetry processing engine that can be extended with additional query operators, aggregations, dimensions, time-windowing, and more advanced event sources.