Skip to content

feat: column-level Dynamic Data Masking (masked_with / masks)#746

Open
Benjamin-Knight wants to merge 1 commit into
dbt-msft:masterfrom
Benjamin-Knight:feat/dynamic-data-masking
Open

feat: column-level Dynamic Data Masking (masked_with / masks)#746
Benjamin-Knight wants to merge 1 commit into
dbt-msft:masterfrom
Benjamin-Knight:feat/dynamic-data-masking

Conversation

@Benjamin-Knight

@Benjamin-Knight Benjamin-Knight commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Applies SQL Server Dynamic Data Masking to columns as part of the materialization, so masks are re-applied on every build and survive a full-refresh rebuild. A principal with SELECT but not UNMASK sees masked values. Base tables only (table / incremental / snapshot); requires SQL Server 2016+. Resolve #572

Config — two surfaces, use either or both

  • Column-level masked_with: — a first-class column property in schema YAML:
    columns:
      - name: nhs_number
        masked_with: 'partial(0,"XXXXXXXXXX",0)'
  • Model-level masks — a {column: function} dict in {{ config() }}, .yml config:, or a dbt_project.yml directory default. Merges key-wise across those levels (like meta) rather than clobbering:
    {{ config(masks={'surname': "default()", 'nhs_number': 'partial(0,"XXXXXXXXXX",0)'}) }}

Behaviour

  • Precedence: column-level masked_with wins over a model-level masks entry; conflicts are warned.
  • Opt out: masked_with: null removes an inherited default.
  • Idempotent: diffs against sys.masked_columns and emits only changed ADD / MASKED WITH / DROP MASKED DDL.
  • Validation: unknown columns are skipped with a warning; unmaskable column types error.
  • Indexes: masks are applied before rowstore index creation so masked index-key columns work on fresh builds; adding a new mask to an existing index key on a persisted table errors with the documented workaround.

Apply SQL Server Dynamic Data Masking as a post-materialization step,
modelled on apply_grants, so masks are re-applied on every build and
survive dbt's drop-and-recreate on full refresh.

Two config surfaces resolve to one per-column mask map:
- column-level `masked_with:` property in schema YAML (null opts out of an
  inherited default), and
- model-level `masks` dict, registered with MergeBehavior.Update so it
  merges key-wise across dbt_project.yml / .yml / in-file config().

Column-level wins over model-level (warned); unknown columns are skipped
with a warning; computed/FILESTREAM/COLUMN_SET/Always Encrypted columns
error. The macro diffs desired masks against sys.masked_columns and emits
only changed ADD / MASKED WITH / DROP MASKED DDL (idempotent). Wired into
table, incremental, snapshot and both dml-refresh sub-paths, with masks
applied before index creation on fresh builds (a mask cannot be added to a
column an index depends on) and after reconcile on persisted tables.
Base tables only: views inherit masking, seeds are not masked. Requires
SQL Server 2016+.

Tests: 18 unit (pure resolve/diff, red->green) + 10 functional against SQL
Server (both surfaces, precedence, opt-out, validation, full-refresh
survival, masked-user read, incremental idempotency, snapshot). Docs in
README + CHANGELOG.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add dynamic column masking to tables

1 participant