Add Purview plugin - #102
Open
johnhayes-squaredup wants to merge 7 commits into
Open
Conversation
- Add dataSourceName field to Scans on Data Source stream, read directly from the API response rather than parsed from the scan id - Fix lastRunResult fields in dataSourceScans and scansForDataSource: status was scanRunStatus, times were nested under discoveryExecutionDetails, and runType was scanLevelType — all previously broken due to incorrect field name assumptions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix classification summary (count from assets instead of missing facet count field) - Add Classified Assets stream (all classifications, one row per asset-classification pair) - Add Assets with Custom Classifications stream (non-Microsoft classifications only) - Add Custom Classifications stream (account-wide, available on Collection and Data Source) - Update Scans on Data Source stream with dataSourceName field and postRequestScript - Add Duration (s) numeric column to Scan History for average queries - Restrict Scan History to single object selection - Remove Collection and Data Source out-of-the-box dashboards Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resize Classification Coverage and Metadata Policies tiles to 2x2 and stack them on the right side of the Asset Count by Entity Type tile. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add Governance Domains stream (Unified Catalog API, account-wide) - Add Glossary Terms stream scoped to Purview Governance Domain objects - Index Governance Domains as a new object type - Strip HTML from description fields in both streams - Update README with Unified Catalog permission requirements - Add Purview Governance Domain to objectTypes and What gets indexed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
Next review available in: 25 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe Microsoft Purview v1 plugin adds OAuth configuration, validation, indexed collections, data sources, and governance domains. It also adds on-demand asset, lineage, classification, policy, glossary, and scan streams, response scripts, an Overview dashboard, scopes, and documentation. ChangesMicrosoft Purview plugin foundation
Indexed entities and operational streams
On-demand asset analytics
Dashboard and scope wiring
Sequence Diagram(s)sequenceDiagram
participant PluginConfiguration
participant ConfigValidation
participant PurviewAPI
participant IndexDefinition
participant Dashboard
PluginConfiguration->>ConfigValidation: validate configured credentials
ConfigValidation->>PurviewAPI: request collections and data sources
PurviewAPI-->>ConfigValidation: return validation results
IndexDefinition->>PurviewAPI: retrieve indexed Purview entities
PurviewAPI-->>IndexDefinition: return collections, data sources, and domains
Dashboard->>PurviewAPI: request on-demand asset analytics
PurviewAPI-->>Dashboard: return transformed stream data
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 15
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/MicrosoftPurview/v1/dataStreams/assetLineage.json`:
- Around line 28-32: Constrain the depth field in the asset lineage
configuration to valid Purview lineage hop counts: require numeric integer
input, enforce a minimum of 0, and add an appropriate maximum if supported by
the API. Keep the existing defaultValue of 3 and help text unchanged.
- Around line 7-14: Update the assetGuid field validation in the UI
configuration to require a valid GUID format before the value is used in the
lineage endpoint path. Keep the field required and ensure values containing path
characters such as /, ?, or # are rejected.
In `@plugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.json`:
- Around line 13-17: Configure token-based pagination in the postRequest
configuration for assetsByEntityTypeGlobal.json (lines 13-17) and
classificationCoverage.json (lines 13-17), matching the contract used by
assetsByCollection.json. Ensure all Purview response pages are combined before
assetsByEntityTypeGlobal.js and classificationCoverage.js aggregate and count
results.
In `@plugins/MicrosoftPurview/v1/dataStreams/classificationTypes.json`:
- Around line 10-13: Update the getArgs entry in classificationTypes.json to use
the documented uppercase CLASSIFICATION value for the type filter, while leaving
the classificationTypes.js postRequestScript unchanged.
In `@plugins/MicrosoftPurview/v1/dataStreams/collections.json`:
- Around line 28-68: Replace the mustache-style valueExpression transformations
in plugins/MicrosoftPurview/v1/dataStreams/collections.json lines 28-68 with map
transformations for collectionName, displayName, parentCollectionName, and
sourceType, preserving alias, fallback, nested-field, and constant-value
behavior. Apply the same conversion in
plugins/MicrosoftPurview/v1/dataStreams/listDataSources.json lines 24-78 for the
nested collection, resource group, subscription, portal URL, and sourceType
fields; both sites require direct changes.
In `@plugins/MicrosoftPurview/v1/dataStreams/dataSourceScans.json`:
- Around line 29-34: Update the normalization script in dataSourceScans.js to
emit collection from props.collection?.referenceName, then change the collection
entry in dataSourceScans.json to a normal metadata field without computed or
valueExpression properties. Remove the dotted implementation-field derivation
and preserve the existing display name.
In `@plugins/MicrosoftPurview/v1/dataStreams/scansForDataSource.json`:
- Around line 2-6: Add the mandatory tags property to the scansForDataSource
stream definition, reusing the existing Data Sources tag used by scanHistory.
Keep the current metadata and hidden visibility unchanged.
In `@plugins/MicrosoftPurview/v1/dataStreams/scripts/accountAssetCount.js`:
- Around line 1-2: Update the asset-count result construction around
`@search.count` to detect `data?.['`@search.count.approximate`'] === true` and
avoid presenting the estimated value as the exact “Total Assets” KPI, using an
appropriate status field/label or equivalent non-exact representation. Increment
the plugin `version` in `metadata.json` for this change.
In `@plugins/MicrosoftPurview/v1/dataStreams/scripts/assetLineage.js`:
- Around line 7-16: Update the lineage processing around the relations loop and
role assignment to traverse the graph from baseGuid, collecting every reachable
upstream and downstream node across multiple hops rather than only direct
neighbors. Preserve Base for baseGuid and assign roles from the complete
traversal sets, then increment the version in metadata.json to reflect the
behavior change.
In `@plugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.js`:
- Around line 3-12: The custom-classification output must expose category and
typeVersion. In
plugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.js lines
3-12, update the normalized rows produced by the defs filter/map to include both
fields; in plugins/MicrosoftPurview/v1/dataStreams/customClassifications.json
lines 16-22, declare metadata for those same fields so Category and Version are
available to consumers.
In `@plugins/MicrosoftPurview/v1/dataStreams/searchAssets.json`:
- Around line 9-13: Update the keywords text field in searchAssets.json so its
placeholder includes an example search keyword, while retaining that the field
is optional.
- Around line 39-42: Update the postBody limit expression in the Purview search
request to coerce the input, constrain it to the valid 1–1000 range, and forward
at most 1000 while preserving the existing default behavior. Also constrain the
associated UI number field, if present in the diff, with minimum 1 and maximum
1000.
In `@plugins/MicrosoftPurview/v1/indexDefinitions/default.json`:
- Around line 2-50: Implement scan indexing by adding a scan-stream import step
with a stable ID, scan label, and “Purview Scan” source type in
plugins/MicrosoftPurview/v1/indexDefinitions/default.json lines 2-50; add
“Purview Scan” to objectTypes in plugins/MicrosoftPurview/v1/metadata.json lines
24-28, and retain the corresponding scan-indexing and object-type documentation
in plugins/MicrosoftPurview/v1/docs/README.md lines 7-7 and 63-70.
In `@plugins/MicrosoftPurview/v1/metadata.json`:
- Around line 45-52: The Microsoft Purview metadata configuration currently
routes all streams through the account-scoped baseUrl. Add a separate
authenticated base configuration for https://api.purview-service.microsoft.com/
using the existing OAuth2 settings, then update the governanceDomains and
glossaryTerms stream definitions to reference that base while leaving other
streams on the account-scoped endpoint.
In `@plugins/MicrosoftPurview/v1/ui.json`:
- Around line 12-37: Update the tenantId, clientId, and clientSecret fields in
the UI configuration with example GUID-based placeholders for the IDs and
concise help text describing where to obtain each credential. Preserve the
existing labels, field types, and required validation.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 97e34756-0fb3-4a75-a5c5-73ed08e038f5
⛔ Files ignored due to path filters (1)
plugins/MicrosoftPurview/v1/icon.svgis excluded by!**/*.svg
📒 Files selected for processing (47)
plugins/MicrosoftPurview/v1/configValidation.jsonplugins/MicrosoftPurview/v1/custom_types.jsonplugins/MicrosoftPurview/v1/dataStreams/accountAssetCount.jsonplugins/MicrosoftPurview/v1/dataStreams/assetCountByEntityType.jsonplugins/MicrosoftPurview/v1/dataStreams/assetLineage.jsonplugins/MicrosoftPurview/v1/dataStreams/assetsByCollection.jsonplugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.jsonplugins/MicrosoftPurview/v1/dataStreams/classificationCoverage.jsonplugins/MicrosoftPurview/v1/dataStreams/classificationTypes.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionClassificationSummary.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionClassifiedAssets.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionCustomClassifiedAssets.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionPolicyCoverage.jsonplugins/MicrosoftPurview/v1/dataStreams/collections.jsonplugins/MicrosoftPurview/v1/dataStreams/customClassifications.jsonplugins/MicrosoftPurview/v1/dataStreams/dataSourceScans.jsonplugins/MicrosoftPurview/v1/dataStreams/glossaryTerms.jsonplugins/MicrosoftPurview/v1/dataStreams/governanceDomains.jsonplugins/MicrosoftPurview/v1/dataStreams/listDataSources.jsonplugins/MicrosoftPurview/v1/dataStreams/metadataPolicies.jsonplugins/MicrosoftPurview/v1/dataStreams/scanHistory.jsonplugins/MicrosoftPurview/v1/dataStreams/scansForDataSource.jsonplugins/MicrosoftPurview/v1/dataStreams/scripts/accountAssetCount.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetCountByEntityType.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetLineage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetsByCollection.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetsByEntityTypeGlobal.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/classificationCoverage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/classificationTypes.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionClassificationSummary.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionClassifiedAssets.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionCustomClassifiedAssets.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionPolicyCoverage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/dataSourceScans.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/glossaryTerms.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/governanceDomains.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/metadataPolicies.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/searchAssets.jsplugins/MicrosoftPurview/v1/dataStreams/searchAssets.jsonplugins/MicrosoftPurview/v1/defaultContent/manifest.jsonplugins/MicrosoftPurview/v1/defaultContent/overview.dash.jsonplugins/MicrosoftPurview/v1/defaultContent/scopes.jsonplugins/MicrosoftPurview/v1/docs/README.mdplugins/MicrosoftPurview/v1/indexDefinitions/default.jsonplugins/MicrosoftPurview/v1/metadata.jsonplugins/MicrosoftPurview/v1/ui.json
- Add GUID regex validation to assetGuid field; add min/max constraints to depth field (assetLineage) - Add token-based pagination to assetsByEntityTypeGlobal and classificationCoverage so all pages are aggregated before counting - Fix classificationTypes type filter to uppercase CLASSIFICATION per API spec - Replace mustache valueExpression computed fields with .map() postRequestScripts in collections and listDataSources; remove expandInnerObjects from listDataSources - Fix dataSourceScans to emit collection directly from props in the script, removing dotted-path computed field pattern - Add tags to scansForDataSource (was missing mandatory tags property) - Detect @search.count.approximate in accountAssetCount to avoid presenting estimated counts as exact - Upgrade assetLineage traversal from single-hop to multi-hop BFS so all reachable upstream/downstream nodes are correctly classified - Expose category and typeVersion in customClassifications script and metadata - Add example placeholder and 1–1000 clamping to searchAssets keywords and limit fields - Index Purview Scans: add scan step to default.json, emit sourceType from dataSourceScans script, add Purview Scan to objectTypes - Add GUID-format placeholders and help text to tenantId and clientId fields in ui.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. An unexpected error occurred while generating fixes: Not Found - https://docs.github.com/rest/git/refs#get-a-reference |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/MicrosoftPurview/v1/indexDefinitions/default.json`:
- Around line 54-56: Update the default objectMapping so indexed scan IDs are
globally unique by storing the API scan ID in the data stream or mapping a
composite of dataSourceName and name, rather than using name alone. Ensure
identical scan names from different data sources produce distinct indexed
objects, and increment the plugin version in metadata.json for this change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 86052599-ed62-4d79-adde-ff2268c95559
📒 Files selected for processing (19)
plugins/MicrosoftPurview/v1/dataStreams/assetLineage.jsonplugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.jsonplugins/MicrosoftPurview/v1/dataStreams/classificationCoverage.jsonplugins/MicrosoftPurview/v1/dataStreams/classificationTypes.jsonplugins/MicrosoftPurview/v1/dataStreams/collections.jsonplugins/MicrosoftPurview/v1/dataStreams/customClassifications.jsonplugins/MicrosoftPurview/v1/dataStreams/dataSourceScans.jsonplugins/MicrosoftPurview/v1/dataStreams/listDataSources.jsonplugins/MicrosoftPurview/v1/dataStreams/scansForDataSource.jsonplugins/MicrosoftPurview/v1/dataStreams/scripts/accountAssetCount.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetLineage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collections.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/dataSourceScans.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/listDataSources.jsplugins/MicrosoftPurview/v1/dataStreams/searchAssets.jsonplugins/MicrosoftPurview/v1/indexDefinitions/default.jsonplugins/MicrosoftPurview/v1/metadata.jsonplugins/MicrosoftPurview/v1/ui.json
Scan names alone are not unique across data sources (e.g. two sources can both have a scan named "DefaultScan"). Emit a composite id of dataSourceName/name from the script, declare it as sourceId in the metadata, and map id in the index definition objectMapping so scans from different data sources never collide. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
johnhayes-squaredup
marked this pull request as ready for review
July 28, 2026 16:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔌 Plugin overview
🖼️ Plugin screenshots
Plugin configuration
Default dashboards
🧪 Testing
I have carried out functional testing to ensure that:
This is a first iteration of the plugin and I have encountered some data hierarchy issues which mean that data joins do not work as elegantly as I would like. For example, it is not possible to see all Scans at once - they can only be viewed per data source.
For the purpose of this exercise I have created a free Purview instance. This means that I do not have access to Enterprise features so these are not included in the data source
Some Purview functionality is in preview mode only and is not yet supported by the API
I have only been able to test against a small dataset. I don't now how the data source will perform under large loads
In a future iteration I would like to:
📚 Checklist
Summary by CodeRabbit