Skip to content

feat(data-masking): add Data Masking utility (Erase, Encrypt/Decrypt, Logging integration) - #1277

Open
nicolastarzia wants to merge 5 commits into
aws-powertools:developfrom
nicolastarzia:data-masking
Open

nicolastarzia wants to merge 5 commits into
aws-powertools:developfrom
nicolastarzia:data-masking

Conversation

@nicolastarzia

Copy link
Copy Markdown

Issue number: #1257
Closes #1257

Summary

Changes

Adds a new AWS.Lambda.Powertools.DataMasking utility that protects sensitive data (PII, credentials, financial records) before it is logged, stored, or forwarded — helping meet GDPR/HIPAA/PCI-DSS requirements. Ported from the Powertools for Python/TypeScript design.

Delivered in three phases:

  • Erase (irreversible masking). DataMasking.Erase(...) masks fields in place while keeping the surrounding payload readable. Overloads for string and JsonNode (AOT-safe), object (annotated), and object + JsonTypeInfo (AOT-safe via source generator). MaskingOptions supports a fixed mask (default *****), length-preserving mask, and regex + replacement.
  • Encrypt / Decrypt (reversible). Pluggable IDataMaskingProvider; default AwsEncryptionSdkProvider performs AWS KMS envelope encryption via the AWS Encryption SDK for .NET, with encryption-context binding/validation and single- or multi-key keyrings. EncryptAsync/DecryptAsync support whole-payload and field-level operations.
  • Logging integration. EraseToNode(...) returns a JsonNode ready for structured logging, so users can Logger.LogInformation(masker.EraseToNode(json, fields)) without coupling the Data Masking and Logging packages.

Field selection uses simple dotted paths (for example address.street). Also includes a runnable example under examples/DataMasking and unit tests.

User experience

Before: .NET customers hand-roll masking/encryption logic, which is inconsistent across teams and easy to get wrong (especially field-level encryption that preserves structure, and KMS envelope encryption with data-key handling).

After: a single, consistent utility:

var masker = new DataMasking();
var masked = masker.Erase(json, new[] { "customer.ssn", "payment.creditCard" });

var secure = new DataMasking(new AwsEncryptionSdkProvider(KMS_KEY_ARN));
var encrypted = await secure.EncryptAsync(json, new[] { "customer.ssn" },
    encryptionContext: new() { ["tenantId"] = "acme-corp" });
var decrypted = await secure.DecryptAsync(encrypted, new[] { "customer.ssn" },
    encryptionContext: new() { ["tenantId"] = "acme-corp" });

Logger.LogInformation(masker.EraseToNode(json, new[] { "customer.ssn" }));

Checklist

Testing

  • 25 unit tests pass on net10.0 (13 Erase + 8 Encrypt/Decrypt + 4 EraseToNode). Library builds clean multi-targeting net8.0;net10.0.
  • The AwsEncryptionSdkProvider itself is not unit-tested as it requires live AWS KMS credentials; the DataMasking Encrypt/Decrypt orchestration is tested against a reversible in-memory fake provider.

Open design questions for maintainers

Since this is a new public utility, a few choices would benefit from your input:

  1. Field paths: Phase 1 uses simple dotted paths over JsonNode. Full JMESPath support (the repo already ships AWS.Lambda.Powertools.JMESPath, which currently only extracts, not sets) is deferred — extend JMESPath or keep an internal setter?
  2. Async shape: the AWS Encryption SDK API is synchronous; IDataMaskingProvider exposes EncryptAsync/DecryptAsync for future-proofing (the default provider wraps sync calls). Keep async or expose sync?
  3. AOT: the AwsEncryptionSdkProvider depends on the SDK's Dafny runtime (reflection) and is annotated [RequiresDynamicCode]/[RequiresUnreferencedCode]; Erase/EraseToNode remain AOT-safe.
  4. Is phasing acceptable, or do you prefer a different split?
Is this a breaking change?

No. This adds a new package; no existing library code or public API is modified.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

Nicolas Tarzia added 3 commits September 23, 2026 23:34
Introduces the AWS.Lambda.Powertools.DataMasking package with the Erase
operation for irreversibly masking sensitive fields in a payload.

- DataMasking.Erase overloads: string and JsonNode (AOT-safe), object
  (annotated RequiresUnreferencedCode/RequiresDynamicCode), and
  object + JsonTypeInfo (AOT-safe via source generator)
- MaskingOptions: default *****, fixed value, length-preserving, and
  regex + replacement rules
- Field selection via simple dotted paths over System.Text.Json.Nodes
  (JMESPath expression support deferred to a later phase)
- Unit tests (13) and an examples/DataMasking sample function

Relates to aws-powertools#1257. Phase 1 (Erase) only; encryption/decryption via a
pluggable provider (AWS Encryption SDK + KMS) planned for later phases.
Adds reversible encryption to the Data Masking utility via a pluggable
provider, with a default AWS KMS envelope-encryption implementation.

- IDataMaskingProvider abstraction (EncryptAsync/DecryptAsync with
  encryption context and cancellation token)
- AwsEncryptionSdkProvider: default provider using the AWS Encryption SDK
  for .NET; single-key (CreateAwsKmsKeyring) or multi-key (multi-keyring);
  supports a custom KMS client. Annotated RequiresUnreferencedCode/
  RequiresDynamicCode since the SDK's Dafny runtime is not AOT-safe.
- DataMasking: optional ctor(IDataMaskingProvider) plus EncryptAsync/
  DecryptAsync for whole-payload and field-level encryption over JsonNode
- JsonNodeMasker.TransformFieldsAsync for async field transforms
- Central Package Management: AWS.Cryptography.EncryptionSDK 5.0.0 and
  AWSSDK.KeyManagementService
- Tests: FakeDataMaskingProvider (reversible, context-binding) and 8
  Encrypt/Decrypt tests (round-trip, field-level, context validation,
  missing-provider guards). 21 tests total pass on net10.0.

Relates to aws-powertools#1257. Phase 2 (Encrypt/Decrypt) building on Phase 1 (Erase).
The AwsEncryptionSdkProvider is not covered by unit tests as it requires
live AWS KMS credentials.
Adds an ergonomic way to redact sensitive fields before they reach logs,
without coupling the Data Masking and Logging packages.

- DataMasking.EraseToNode(string json, string[] fields, MaskingOptions?)
  returns a JsonNode ready for structured logging, e.g.
  Logger.LogInformation(masker.EraseToNode(json, fields)) — no extra
  string round-trip. Fully trimming/AOT safe.
- Example updated to log the masked payload via EraseToNode
- Package and example READMEs updated with the Logging integration

Relates to aws-powertools#1257. Phase 3 (Logging integration + docs) completing the
Erase / Encrypt / Decrypt trio. 25 unit tests pass on net10.0.
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 24, 2026
@boring-cyborg boring-cyborg Bot added the tests label Sep 24, 2026
Comment thread examples/DataMasking/src/HelloWorld/Function.cs Fixed
Nicolas Tarzia added 2 commits September 24, 2026 00:29
- Add a timeout to the example Regex to avoid potential ReDoS (fixes the
  Security rating on new code)
- Reduce code duplication in DataMasking by extracting MaskRoot/Serialize/
  TransformFieldsToStringAsync helpers (fixes duplication > 3% on new code)
- Declare an explicit CloudWatch Log Group in the example SAM template
  (with 7-day retention), matching the BatchProcessing example

All 25 unit tests still pass on net10.0.
Extract a shared TryResolveLeaf (and ReadRawValue) helper so the sync mask
and async transform paths no longer duplicate the dotted-path walk, bringing
new-code duplication under the SonarCloud threshold. Behavior unchanged; 25
tests still pass.
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: Add Data Masking utility for encrypting and masking sensitive data

2 participants