From efa9d4aecd60790dfeb8da8893d1b2b11a5919a6 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 09:32:23 +0000 Subject: [PATCH 01/35] Make building blocks and icon collections resolvable by bare DESCRIBE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `describe Module.Name` reported "no describable document named ..." for building blocks and icon collections, even though `describe building block Module.Name` and `describe icon collection Module.Name` both worked. Two lists in different packages had drifted apart: bare DESCRIBE resolves a type by looking the qualified name up in the catalog `objects` view and mapping its ObjectType through `objectTypeToDescribeKind`. - Building blocks were built into `building_blocks_data` but never joined into the `objects` view union, so the lookup found no row. - Icon collections had no catalog table at all, which left the existing `ICON_COLLECTION` entry in the map as dead code — nothing could emit it. Measured against a project carrying seven real marketplace modules: 43 of 251 documents (40 building blocks, 3 icon collections) were unreachable this way. Auto-detect coverage goes from 204/251 (81%) to 247/251 (98%), verified end-to-end over the same documents with no new name ambiguity. The remaining 4 are separate defects: import/export mapping describe errors, and menu documents have no DESCRIBE at all. The denominator comes from raw MPR unit types rather than the catalog, which indexes only describable types and so reports full coverage by construction. `TestDescribeAutoCoversCatalogObjectTypes` guards the drift: it reads the ObjectType literals out of the objects view's own SQL and fails when one has neither a describe kind nor an entry in an explicit exemption list. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/fix-issue.md | 1 + .../PROPOSAL_marketplace_module_upgrade.md | 92 +++++++++++++- mdl/catalog/builder.go | 4 + mdl/catalog/catalog.go | 1 + mdl/catalog/catalog_test.go | 4 + mdl/catalog/tables.go | 21 ++++ mdl/executor/describe_auto.go | 1 + mdl/executor/describe_auto_test.go | 114 ++++++++++++++++++ 8 files changed, 237 insertions(+), 1 deletion(-) create mode 100644 mdl/executor/describe_auto_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index e2417a57b..5acbbfac7 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -461,3 +461,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A microflow's `StableId` is a different GUID after every mxcli write, so a microflow always differs from itself and no amount of no-op elision can settle it. Downstream: every `callMicroflow` entry in `deployment/model/operations.json` gets a new `operationId` on each build | `microflow_write.go` registered `StableId` as a `codec` `FreshGUIDField`, minting a fresh GUID per write. It is the one field on a microflow whose *stated purpose* is not to change: Mendix declares it `ModelPropertyAttribute("StableId", RetentionType.DesignTime)` with `IsIdentifier = true`, seeds it once via the one-time `MicroflowStableIdConversion`, and transplants it across a marketplace module update in `PackageUtils.RescueStableIDs` | `modelsdk/canon/identity.go` (`identityFields`, `CarryIdentity`) — the stored value is carried onto the rebuilt document before the comparison, so an otherwise-unchanged microflow compares equal and is elided | **Establish that a property is an identity before adding a row to `identityFields`** — do not assume it from the name. The method that settled this one: strict-boundary `strings` scan of every runtime jar (a substring match gives false positives — `newPersistableIds` contains `stableId`); `monodis` the modeler assembly and read the `ModelPropertyAttribute` blob; look for an `IOneTimeConversion` named after the field; search the packaging assembly for a get→set pair; then build with `mxbuild --target=deploy` and reproduce the derivation against `deployment/`. That last step is what proved the value escapes the model: `operationId == base64(uuid5(projectId, StableId).bytes_le)`, 10 of 10 exact. **Carry only a key both documents already have** — never invent one, and dispatch on the stored `$Type` (CLAUDE.md, overlay writes). **Identity preservation is not gated by `MXCLI_ALWAYS_WRITE`**: that flag turns off eliding a write, not preserving what the document is. Tests `modelsdk/canon/identity_test.go`, `sdk/mpr/writer_elision_test.go`. ADR-0008 | | `call workflow`, `get workflow data/workflows/activity records`, `open`/`lock`/`unlock workflow` and `workflow operation` all DESCRIBE as a placeholder, so a describe→edit→exec cycle deletes them — while authoring and building them works fine | The modelsdk reader (the DEFAULT engine) had no case for any of the eight; each read back with a nil Action. The formatters and grammar were already there, so only `actionFromGen` was missing | New `mdl/backend/modelsdk/microflow_workflow_read.go` (+ dispatch in `microflow_read_actions.go`), `mdl/grammar/domains/MDLMicroflow.g4` + `mdl/visitor/visitor_microflow_workflow.go` (positional `call workflow` form), `mdl/executor/cmd_microflows_builder_workflow.go` (abort reason) | **Read against the WRITER's keys, not gen's** — gen binds every "get" action's result as `VariableName` while the model stores `OutputVariableName`, so an accessor-based reader silently drops the variable; round-trip tests (not reader-only tests on hand-written BSON) are what catch it. **`WorkflowSelection` has two variants and the object form uses `WorkflowDefinitionVariable`**, not `WorkflowVariable`; its ABSENCE is meaningful (the all-workflows case), so do not synthesise one. **Dispatch the operation on `$Type` before reading fields** — only Abort carries a Reason. **Finishing the reader exposes what could not be seen while nothing rendered**: `call workflow` described to a positional form the grammar rejected (the model never stores the parameter NAME, so DESCRIBE cannot emit the named form), and the abort reason stored the expression *including its quotes* into a StringTemplate Text, so Mendix rendered the quotes at runtime and every round trip doubled them. **Still open**: `lock/unlock workflow all` writes an activity mxbuild rejects with CE1825 — a lock always needs a specific definition, and supplying an empty selection does not satisfy it. Tests `microflow_workflow_read_test.go`, example `mdl-examples/bug-tests/workflow-actions-describe.mdl` | | `transform $In with Module.Transformer` reports "Created microflow" and the build then fails `[CE0008] "No action defined."` — the activity is in the model with no action inside it | The modelsdk writer (the DEFAULT engine) had no `TransformJsonAction` case, so it fell through to `default: return nil`. The grammar, builder, DESCRIBE formatter and the LEGACY writer all handled it, which is why it looked supported. The reader was missing too, so even a correctly written action described as a placeholder | `mdl/backend/modelsdk/microflow_write.go` (writer case), `mdl/backend/modelsdk/microflow_read_actions.go` (reader cases for `TransformJsonAction`, `CallExternalAction`, `RestOperationCallAction`) | **A missing WRITER case and a missing READER case present identically in a coverage audit** — grepping for reader cases found `transform` "authorable but unreadable" when it was actually unwritable, a strictly worse bug. Check both directions before sizing the work. **Mirror the legacy serializer for keys** (`InputVariableName`/`OutputVariableName`/`Transformation`); for the other two, note `CallExternalAction` stores its result under `VariableName` (not `ResultVariableName`) and REST's two mapping lists are NOT symmetric — the query list keys its name as `QueryParameter`. **Do not reconstruct `CallExternalAction.ResultDataType`**: it is resolved from the consumed service's cached `$metadata` at write time, so reading it back would let a stale value round-trip as if authored. **CE0008 turning into CE1613 is progress, not a new bug** — the action now exists and the build has moved on to validating its reference. Tests `microflow_integration_actions_test.go`, example `mdl-examples/bug-tests/transform-json-write-and-describe.mdl` | +| `describe Module.Name` reports `no describable document named "..."` for a document that plainly exists and that `describe building block Module.Name` / `describe icon collection Module.Name` describes fine when the type is named explicitly | Two lists in different packages drifted apart. Bare DESCRIBE resolves the type by looking the qualified name up in the catalog `objects` view and mapping its `ObjectType` through `objectTypeToDescribeKind`. Building blocks were built into `building_blocks_data` but never joined into the `objects` view union, so the lookup returned no row; icon collections had **no catalog table at all**, leaving the `ICON_COLLECTION` entry in the map as dead code that nothing could ever emit. Measured on a stock 7-module marketplace project: 43 of 251 documents (40 building blocks + 3 icon collections) were unreachable this way — auto-detect coverage 81%, explicit-type coverage 98% | `mdl/executor/describe_auto.go` (`objectTypeToDescribeKind`), `mdl/catalog/tables.go` (the `objects` view union + the `*_data` table), `mdl/catalog/builder.go` (`buildSimpleNamedDocs` call), `mdl/catalog/catalog.go` (the `CATALOG.*` table list) | Add the type in **all four** places — a map entry alone does nothing if the view never emits the `ObjectType`, and a view row alone does nothing if the map has no kind. For a document with just name/folder/documentation, the builder is one `buildSimpleNamedDocs("", "", "
", "
' page +-- * a leaf with a microflow target -> menu item '' microflow +-- * a leaf with no action -> menu item '' +-- * a sub-menu -> menu '' ( ...nested items... ) +-- +-- An `icon` clause is emitted only for Forms$IconCollectionIcon, the one icon +-- variant MDL can name. A glyph icon (numeric code) or an image icon is +-- reported on its own comment line rather than dropped silently, so what is +-- missing is visible: +-- +-- -- icon a numeric glyph code (Forms$GlyphIcon) is not reproducible by MDL; +-- -- set it in Studio Pro diff --git a/mdl/ast/ast_query.go b/mdl/ast/ast_query.go index d2b80be1f..d1bba5232 100644 --- a/mdl/ast/ast_query.go +++ b/mdl/ast/ast_query.go @@ -330,6 +330,7 @@ const ( DescribeConsumedMCPService // DESCRIBE CONSUMED MCP SERVICE Module.Name (agent-editor MCP document) DescribeJarDependency // DESCRIBE JAR DEPENDENCY ModuleName 'group:artifact' DescribeBuildingBlock // DESCRIBE BUILDING BLOCK Module.Name + DescribeMenu // DESCRIBE MENU Module.Name (standalone Menus$MenuDocument) DescribeAuto // DESCRIBE Module.Name — type auto-detected at execution time ) @@ -418,6 +419,8 @@ func (t DescribeObjectType) String() string { return "JAR DEPENDENCY" case DescribeBuildingBlock: return "BUILDING BLOCK" + case DescribeMenu: + return "MENU" case DescribeAuto: return "AUTO" default: diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index 637e5e5d6..54bccf0ba 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -484,6 +484,11 @@ func (unsupportedBackend) GetMendixVersion() (r0 string, err1 error) { return } +func (unsupportedBackend) GetMenuDocumentByQualifiedName(_ string, _ string) (r0 *types.MenuDocument, err1 error) { + err1 = errUnsupported("GetMenuDocumentByQualifiedName") + return +} + func (unsupportedBackend) GetMicroflow(_ model.ID) (r0 *microflows.Microflow, err1 error) { err1 = errUnsupported("GetMicroflow") return @@ -707,6 +712,11 @@ func (unsupportedBackend) ListLayouts() (r0 []*pages.Layout, err1 error) { return } +func (unsupportedBackend) ListMenuDocuments() (r0 []*types.MenuDocument, err1 error) { + err1 = errUnsupported("ListMenuDocuments") + return +} + func (unsupportedBackend) ListMicroflows() (r0 []*microflows.Microflow, err1 error) { err1 = errUnsupported("ListMicroflows") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index b2d9e8c62..508364375 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -244,11 +244,14 @@ type MockBackend struct { UpdateProjectSettingsFunc func(ps *model.ProjectSettings) error // ImageBackend - ListImageCollectionsFunc func() ([]*types.ImageCollection, error) - ListIconCollectionsFunc func() ([]*types.IconCollection, error) - CreateImageCollectionFunc func(ic *types.ImageCollection) error - UpdateImageCollectionFunc func(ic *types.ImageCollection) error - DeleteImageCollectionFunc func(id string) error + ListImageCollectionsFunc func() ([]*types.ImageCollection, error) + ListIconCollectionsFunc func() ([]*types.IconCollection, error) + + ListMenuDocumentsFunc func() ([]*types.MenuDocument, error) + GetMenuDocumentByQualifiedNameFunc func(moduleName, name string) (*types.MenuDocument, error) + CreateImageCollectionFunc func(ic *types.ImageCollection) error + UpdateImageCollectionFunc func(ic *types.ImageCollection) error + DeleteImageCollectionFunc func(id string) error // ScheduledEventBackend ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) diff --git a/mdl/backend/mock/mock_navigation.go b/mdl/backend/mock/mock_navigation.go index b0940d9a1..794e8e687 100644 --- a/mdl/backend/mock/mock_navigation.go +++ b/mdl/backend/mock/mock_navigation.go @@ -3,6 +3,8 @@ package mock import ( + "fmt" + "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/model" ) @@ -27,3 +29,17 @@ func (m *MockBackend) UpdateNavigationProfile(navDocID model.ID, profileName str } return nil } + +func (m *MockBackend) ListMenuDocuments() ([]*types.MenuDocument, error) { + if m.ListMenuDocumentsFunc != nil { + return m.ListMenuDocumentsFunc() + } + return nil, fmt.Errorf("MockBackend.ListMenuDocuments not configured") +} + +func (m *MockBackend) GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) { + if m.GetMenuDocumentByQualifiedNameFunc != nil { + return m.GetMenuDocumentByQualifiedNameFunc(moduleName, name) + } + return nil, fmt.Errorf("MockBackend.GetMenuDocumentByQualifiedName not configured") +} diff --git a/mdl/backend/modelsdk/menu_read.go b/mdl/backend/modelsdk/menu_read.go new file mode 100644 index 000000000..768db8ebb --- /dev/null +++ b/mdl/backend/modelsdk/menu_read.go @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + genMenus "github.com/mendixlabs/mxcli/modelsdk/gen/menus" + "github.com/mendixlabs/mxcli/modelsdk/mprread" +) + +// ListMenuDocuments reads every standalone Menus$MenuDocument unit. +// +// A menu document's entries are the same Menus$MenuItem elements a navigation +// profile holds, so the recursive conversion reuses navMenuItemFromGen — which +// already handles the three icon variants and the client-action dispatch. The +// only structural difference is that a menu document wraps its entries in a +// Menus$MenuItemCollection rather than holding them directly. +func (b *Backend) ListMenuDocuments() ([]*types.MenuDocument, error) { + if b.reader == nil { + return nil, fmt.Errorf("ListMenuDocuments: not connected") + } + units, err := mprread.ListUnitsWithContainer[*genMenus.MenuDocument](b.reader) + if err != nil { + return nil, err + } + + out := make([]*types.MenuDocument, 0, len(units)) + for _, u := range units { + g := u.Element + md := &types.MenuDocument{ + ID: model.ID(g.ID()), + ContainerID: model.ID(u.ContainerID), + Name: g.Name(), + Documentation: g.Documentation(), + ExportLevel: g.ExportLevel(), + Excluded: g.Excluded(), + } + if coll, ok := g.ItemCollection().(*genMenus.MenuItemCollection); ok && coll != nil { + for _, el := range coll.ItemsItems() { + if item := navMenuItemFromGen(el); item != nil { + md.Items = append(md.Items, item) + } + } + } + out = append(out, md) + } + return out, nil +} + +// GetMenuDocumentByQualifiedName finds a menu document by module + name. +func (b *Backend) GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) { + all, err := b.ListMenuDocuments() + if err != nil { + return nil, err + } + for _, md := range all { + if md.Name == name && b.moduleNameFor(md.ID) == moduleName { + return md, nil + } + } + return nil, fmt.Errorf("menu not found: %s.%s", moduleName, name) +} diff --git a/mdl/backend/modelsdk/module_resolution_test.go b/mdl/backend/modelsdk/module_resolution_test.go index c9c5e5224..78b82cf87 100644 --- a/mdl/backend/modelsdk/module_resolution_test.go +++ b/mdl/backend/modelsdk/module_resolution_test.go @@ -68,3 +68,39 @@ func TestByQualifiedNameRejectsWrongModule(t *testing.T) { t.Error("expected an error for a mapping that lives in a different module, got nil") } } + +// TestListMenuDocuments reads the standalone Menus$MenuDocument units from the +// vendored fixture. Atlas_Core ships Phone_Menu and Tablet_Menu, both foldered, +// so this also exercises the module-resolution walk above. +func TestListMenuDocuments(t *testing.T) { + b := New() + if err := b.Connect(fixture); err != nil { + t.Fatalf("Connect(%s): %v", fixture, err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + all, err := b.ListMenuDocuments() + if err != nil { + t.Fatalf("ListMenuDocuments: %v", err) + } + if len(all) != 2 { + t.Fatalf("got %d menu documents, want 2 (Phone_Menu, Tablet_Menu)", len(all)) + } + + md, err := b.GetMenuDocumentByQualifiedName("Atlas_Core", "Phone_Menu") + if err != nil { + t.Fatalf("GetMenuDocumentByQualifiedName: %v", err) + } + if len(md.Items) != 4 { + t.Fatalf("got %d top-level items, want 4", len(md.Items)) + } + // Items must carry their caption and icon, not just exist — an item parsed + // into an empty struct would still satisfy a count assertion. + first := md.Items[0] + if first.Caption != "Home" { + t.Errorf("first item caption = %q, want Home", first.Caption) + } + if first.Icon != "Atlas_Core.Atlas_Filled.home" { + t.Errorf("first item icon = %q, want Atlas_Core.Atlas_Filled.home", first.Icon) + } +} diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 4daeaa705..108b91dd7 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -871,3 +871,10 @@ func (b *MprBackend) useCallMicroflowActivityName() bool { func (b *MprBackend) SerializeWorkflowActivity(a workflows.WorkflowActivity) (any, error) { return mpr.SerializeWorkflowActivity(a, b.useCallMicroflowActivityName()), nil } + +func (b *MprBackend) ListMenuDocuments() ([]*types.MenuDocument, error) { + return b.reader.ListMenuDocuments() +} +func (b *MprBackend) GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) { + return b.reader.GetMenuDocumentByQualifiedName(moduleName, name) +} diff --git a/mdl/backend/navigation.go b/mdl/backend/navigation.go index 3c47cc0de..3d23c3d1c 100644 --- a/mdl/backend/navigation.go +++ b/mdl/backend/navigation.go @@ -12,4 +12,11 @@ type NavigationBackend interface { ListNavigationDocuments() ([]*types.NavigationDocument, error) GetNavigation() (*types.NavigationDocument, error) UpdateNavigationProfile(navDocID model.ID, profileName string, spec types.NavigationProfileSpec) error + + // Menu documents are standalone reusable menus (Menus$MenuDocument), not + // the menu embedded in a navigation profile. They are read-only: Mendix + // offers no way to author one outside Studio Pro, so there is deliberately + // no Create/Update/Delete here. + ListMenuDocuments() ([]*types.MenuDocument, error) + GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) } diff --git a/mdl/catalog/builder.go b/mdl/catalog/builder.go index 07aa1c4f7..3f25464bd 100644 --- a/mdl/catalog/builder.go +++ b/mdl/catalog/builder.go @@ -402,6 +402,10 @@ func (b *Builder) Build(progress ProgressFunc) error { return fmt.Errorf("failed to build icon collections: %w", err) } + if err := b.buildSimpleNamedDocs("Menus$MenuDocument", "menus", "Menus"); err != nil { + return fmt.Errorf("failed to build menus: %w", err) + } + if err := b.buildSimpleNamedDocs("DataTransformers$DataTransformer", "data_transformers", "Data Transformers"); err != nil { return fmt.Errorf("failed to build data transformers: %w", err) } diff --git a/mdl/catalog/catalog.go b/mdl/catalog/catalog.go index 5f7b3f783..b9ffbb457 100644 --- a/mdl/catalog/catalog.go +++ b/mdl/catalog/catalog.go @@ -116,6 +116,7 @@ func (c *Catalog) Tables() []string { "CATALOG.JAVASCRIPT_ACTIONS", "CATALOG.IMAGE_COLLECTIONS", "CATALOG.ICON_COLLECTIONS", + "CATALOG.MENUS", "CATALOG.DATA_TRANSFORMERS", "CATALOG.AGENTS", "CATALOG.AI_MODELS", diff --git a/mdl/catalog/catalog_test.go b/mdl/catalog/catalog_test.go index 73a2c3ff9..953f993c9 100644 --- a/mdl/catalog/catalog_test.go +++ b/mdl/catalog/catalog_test.go @@ -235,6 +235,7 @@ func TestObjectsView_IncludesNewDocumentTypes(t *testing.T) { // view, so bare `DESCRIBE Module.Name` could not resolve them. {"building_blocks", "BUILDING_BLOCK"}, {"icon_collections", "ICON_COLLECTION"}, + {"menus", "MENU"}, {"data_transformers", "DATA_TRANSFORMER"}, {"agents", "AGENT"}, {"ai_models", "AI_MODEL"}, diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index 11df21bae..3e6a65a4e 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -325,6 +325,19 @@ func (c *Catalog) createTables() error { )`, viewWithFullSnapshot("icon_collections"), + // menus (standalone Menus$MenuDocument; read-only reusable menus) + `CREATE TABLE IF NOT EXISTS menus_data ( + Id TEXT PRIMARY KEY, + Name TEXT, + QualifiedName TEXT, + ModuleName TEXT, + Folder TEXT, + Description TEXT, + ProjectId TEXT, + SnapshotId TEXT + )`, + viewWithFullSnapshot("menus"), + // data_transformers `CREATE TABLE IF NOT EXISTS data_transformers_data ( Id TEXT PRIMARY KEY, @@ -941,6 +954,10 @@ func (c *Catalog) createTables() error { ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource FROM icon_collections UNION ALL + SELECT Id, 'MENU' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, + ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource + FROM menus + UNION ALL SELECT Id, 'DATA_TRANSFORMER' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource FROM data_transformers diff --git a/mdl/executor/cmd_menus.go b/mdl/executor/cmd_menus.go new file mode 100644 index 000000000..9bac9202f --- /dev/null +++ b/mdl/executor/cmd_menus.go @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" +) + +// describeMenu renders a standalone Menus$MenuDocument — the reusable menu a +// menu widget points at, as opposed to the menu embedded in a navigation +// profile. Atlas_Core ships Phone_Menu and Tablet_Menu. +// +// The entries are ordinary Menus$MenuItem elements, so the tree is rendered by +// printMenuMDL, the same renderer DESCRIBE NAVIGATION uses. Output is +// deliberately not re-executable: Mendix offers no way to author a menu document +// outside Studio Pro, so there is no CREATE MENU for it to round-trip into. That +// is stated in the header rather than left for the reader to discover, following +// the DESCRIBE BUILDING BLOCK precedent. +func describeMenu(ctx *ExecContext, name ast.QualifiedName) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + + md, err := ctx.Backend.GetMenuDocumentByQualifiedName(name.Module, name.Name) + if err != nil { + if strings.Contains(err.Error(), "not found") { + return mdlerrors.NewNotFound("menu", name.String()) + } + return mdlerrors.NewBackend("get menu", err) + } + + if md.Documentation != "" { + fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", + strings.ReplaceAll(md.Documentation, "\n", "\n * ")) + } + + fmt.Fprintf(ctx.Output, "-- Menu: %s.%s (%d top-level item(s))\n", + name.Module, md.Name, len(md.Items)) + if md.ExportLevel != "" { + fmt.Fprintf(ctx.Output, "-- Export level: %s\n", md.ExportLevel) + } + if md.Excluded { + fmt.Fprintln(ctx.Output, "-- Excluded from the project") + } + fmt.Fprintln(ctx.Output, "-- Menus are read-only; they cannot be created via MDL.") + + if len(md.Items) == 0 { + fmt.Fprintln(ctx.Output, "{ }") + return nil + } + + fmt.Fprintln(ctx.Output, "{") + printMenuMDL(ctx.Output, md.Items, 1, "MDL") + fmt.Fprintln(ctx.Output, "}") + return nil +} diff --git a/mdl/executor/cmd_menus_mock_test.go b/mdl/executor/cmd_menus_mock_test.go new file mode 100644 index 000000000..fa92224b0 --- /dev/null +++ b/mdl/executor/cmd_menus_mock_test.go @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// menuBackend returns a MockBackend serving one menu document. +func menuBackend(md *types.MenuDocument) *mock.MockBackend { + return &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetMenuDocumentByQualifiedNameFunc: func(moduleName, name string) (*types.MenuDocument, error) { + if md != nil && md.Name == name { + return md, nil + } + return nil, fmt.Errorf("menu not found: %s.%s", moduleName, name) + }, + } +} + +// TestDescribeMenu_Nested is the renderer's real exercise. The vendored fixture's +// Atlas menus are flat — every MenuItem's Items array holds only the list marker +// — so recursion, page/microflow targets and the non-round-trippable icon note +// have no coverage there. This builds a menu that has all of them. +func TestDescribeMenu_Nested(t *testing.T) { + md := &types.MenuDocument{ + Name: "Main_Menu", + ExportLevel: "Hidden", + Items: []*types.NavMenuItem{ + { + Caption: "Home", + Page: "MyModule.Home_Web", + IconType: "Forms$IconCollectionIcon", + Icon: "Atlas_Core.Atlas.home", + }, + { + Caption: "Admin", + Items: []*types.NavMenuItem{ + {Caption: "Accounts", Page: "Administration.Account_Overview"}, + {Caption: "Rebuild", Microflow: "Administration.Rebuild"}, + }, + }, + { + // A glyph icon carries a numeric Code and no name, so it cannot be + // expressed in MDL. It must be flagged, not silently dropped. + Caption: "Settings", + IconType: "Forms$GlyphIcon", + }, + }, + } + + ctx, buf := newMockCtx(t, withBackend(menuBackend(md))) + assertNoError(t, describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Main_Menu"})) + out := buf.String() + + assertContainsStr(t, out, "-- Menu: Atlas_Core.Main_Menu (3 top-level item(s))") + assertContainsStr(t, out, "read-only") + assertContainsStr(t, out, "menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home;") + + // A sub-menu opens a nested block and its children are indented one level in. + assertContainsStr(t, out, "menu 'Admin' (") + assertContainsStr(t, out, " menu item 'Accounts' page Administration.Account_Overview;") + assertContainsStr(t, out, " menu item 'Rebuild' microflow Administration.Rebuild;") + + // The glyph icon is reported rather than dropped, and names no statement that + // could author it — a menu document cannot be authored at all. + assertContainsStr(t, out, "is not reproducible by MDL") + if strings.Contains(out, "CREATE NAVIGATION") { + t.Errorf("menu output should not point at CREATE NAVIGATION, which cannot author a menu document:\n%s", out) + } +} + +func TestDescribeMenu_Empty(t *testing.T) { + md := &types.MenuDocument{Name: "Empty_Menu"} + ctx, buf := newMockCtx(t, withBackend(menuBackend(md))) + assertNoError(t, describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Empty_Menu"})) + assertContainsStr(t, buf.String(), "{ }") +} + +func TestDescribeMenu_NotFound(t *testing.T) { + ctx, _ := newMockCtx(t, withBackend(menuBackend(nil))) + err := describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Nope"}) + if err == nil { + t.Fatal("expected an error for a menu that does not exist") + } + if !strings.Contains(err.Error(), "not found") { + t.Errorf("error = %q, want it to report the menu was not found", err) + } +} + +// TestDescribeMenu_Documentation checks the doc comment is emitted, since a +// menu document carries Documentation like any other document. +func TestDescribeMenu_Documentation(t *testing.T) { + md := &types.MenuDocument{Name: "Doc_Menu", Documentation: "Phone navigation."} + ctx, buf := newMockCtx(t, withBackend(menuBackend(md))) + assertNoError(t, describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Doc_Menu"})) + assertContainsStr(t, buf.String(), "Phone navigation.") +} diff --git a/mdl/executor/cmd_navigation.go b/mdl/executor/cmd_navigation.go index c01caccb1..32a040c4d 100644 --- a/mdl/executor/cmd_navigation.go +++ b/mdl/executor/cmd_navigation.go @@ -292,7 +292,7 @@ func outputNavigationProfile(ctx *ExecContext, p *types.NavigationProfile) { // Menu items if len(p.MenuItems) > 0 { fmt.Fprintln(ctx.Output, " menu (") - printMenuMDL(ctx.Output, p.MenuItems, 2) + printMenuMDL(ctx.Output, p.MenuItems, 2, "CREATE NAVIGATION") fmt.Fprintln(ctx.Output, " )") } @@ -344,15 +344,17 @@ func menuItemTarget(item *types.NavMenuItem) string { return "" } -// printMenuMDL prints menu items in MDL-style format. -func printMenuMDL(w io.Writer, items []*types.NavMenuItem, depth int) { +// printMenuMDL prints menu items in MDL-style format. reproducer names the +// construct an icon note should point at — navigation menus are authored by +// CREATE NAVIGATION, while a standalone menu document cannot be authored at all. +func printMenuMDL(w io.Writer, items []*types.NavMenuItem, depth int, reproducer string) { indent := strings.Repeat(" ", depth) for _, item := range items { icon := menuItemIconMDL(item) if len(item.Items) > 0 { // Sub-menu container fmt.Fprintf(w, "%smenu '%s'%s (\n", indent, item.Caption, icon) - printMenuMDL(w, item.Items, depth+1) + printMenuMDL(w, item.Items, depth+1, reproducer) fmt.Fprintf(w, "%s);\n", indent) } else if item.Page != "" { fmt.Fprintf(w, "%smenu item '%s' page %s%s;\n", indent, item.Caption, item.Page, icon) @@ -361,7 +363,7 @@ func printMenuMDL(w io.Writer, items []*types.NavMenuItem, depth int) { } else { fmt.Fprintf(w, "%smenu item '%s'%s;\n", indent, item.Caption, icon) } - if note := menuItemIconNote(item); note != "" { + if note := menuItemIconNote(item, reproducer); note != "" { fmt.Fprintf(w, "%s%s\n", indent, note) } } @@ -381,7 +383,7 @@ func menuItemIconMDL(item *types.NavMenuItem) string { // Forms$IconCollectionIcon; a glyph icon (numeric Code) or an image icon // (pointing into an image collection, not an icon collection) is a different // element and would have to be guessed at. -func menuItemIconNote(item *types.NavMenuItem) string { +func menuItemIconNote(item *types.NavMenuItem, reproducer string) string { if item.IconType == "" || strings.HasSuffix(item.IconType, "IconCollectionIcon") { return "" } @@ -389,6 +391,6 @@ func menuItemIconNote(item *types.NavMenuItem) string { if target == "" { target = "a numeric glyph code" } - return fmt.Sprintf("-- icon %s (%s) is not reproducible by CREATE NAVIGATION; set it in Studio Pro", - target, item.IconType) + return fmt.Sprintf("-- icon %s (%s) is not reproducible by %s; set it in Studio Pro", + target, item.IconType, reproducer) } diff --git a/mdl/executor/cmd_navigation_icon_test.go b/mdl/executor/cmd_navigation_icon_test.go index fbb9986d8..8f6cc5189 100644 --- a/mdl/executor/cmd_navigation_icon_test.go +++ b/mdl/executor/cmd_navigation_icon_test.go @@ -15,7 +15,7 @@ import ( // menuMDL renders items through the DESCRIBE emitter. func menuMDL(items []*types.NavMenuItem) string { var b bytes.Buffer - printMenuMDL(&b, items, 0) + printMenuMDL(&b, items, 0, "CREATE NAVIGATION") return b.String() } diff --git a/mdl/executor/describe_auto.go b/mdl/executor/describe_auto.go index d6bbb6321..cef567b79 100644 --- a/mdl/executor/describe_auto.go +++ b/mdl/executor/describe_auto.go @@ -26,6 +26,7 @@ var objectTypeToDescribeKind = map[string]ast.DescribeObjectType{ "PAGE": ast.DescribePage, "SNIPPET": ast.DescribeSnippet, "BUILDING_BLOCK": ast.DescribeBuildingBlock, + "MENU": ast.DescribeMenu, "LAYOUT": ast.DescribeLayout, "ENUMERATION": ast.DescribeEnumeration, "CONSTANT": ast.DescribeConstant, diff --git a/mdl/executor/executor_query.go b/mdl/executor/executor_query.go index 2ba2519b6..aac4ac99d 100644 --- a/mdl/executor/executor_query.go +++ b/mdl/executor/executor_query.go @@ -261,6 +261,8 @@ func execDescribe(ctx *ExecContext, s *ast.DescribeStmt) error { return describeImportMapping(ctx, s.Name) case ast.DescribeExportMapping: return describeExportMapping(ctx, s.Name) + case ast.DescribeMenu: + return describeMenu(ctx, s.Name) case ast.DescribeJarDependency: return execDescribeJarDependency(ctx, s.Name.String(), s.Qualifier) default: @@ -352,6 +354,8 @@ func describeObjectTypeLabel(t ast.DescribeObjectType) string { return "importmapping" case ast.DescribeExportMapping: return "exportmapping" + case ast.DescribeMenu: + return "menu" default: return "unknown" } diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index f5e606fae..8ea56633e 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -146,6 +146,7 @@ describeStatement | DESCRIBE PAGE qualifiedName | DESCRIBE SNIPPET qualifiedName | DESCRIBE BUILDING BLOCK qualifiedName + | DESCRIBE MENU_KW qualifiedName | DESCRIBE LAYOUT qualifiedName | DESCRIBE ENUMERATION qualifiedName | DESCRIBE CONSTANT qualifiedName diff --git a/mdl/types/navigation.go b/mdl/types/navigation.go index dc8187a5f..8fd704117 100644 --- a/mdl/types/navigation.go +++ b/mdl/types/navigation.go @@ -59,6 +59,26 @@ type NavMenuItem struct { Items []*NavMenuItem `json:"items,omitempty"` } +// MenuDocument is a standalone `Menus$MenuDocument` — a reusable menu that menu +// widgets point at, stored as its own document rather than inside a navigation +// profile. Atlas_Core ships two of them (Phone_Menu, Tablet_Menu). +// +// Its entries are the same `Menus$MenuItem` elements a navigation profile holds, +// so they are modelled as NavMenuItem rather than a parallel type — one item +// shape, one parser, one renderer. +type MenuDocument struct { + ID model.ID `json:"id"` + ContainerID model.ID `json:"containerId"` + Name string `json:"name"` + Documentation string `json:"documentation,omitempty"` + ExportLevel string `json:"exportLevel,omitempty"` + Excluded bool `json:"excluded,omitempty"` + Items []*NavMenuItem `json:"items,omitempty"` +} + +// GetName returns the menu document's name. +func (m *MenuDocument) GetName() string { return m.Name } + // NavOfflineEntity declares offline sync rules for an entity. type NavOfflineEntity struct { Entity string `json:"entity"` diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 79170dfae..1209ca3c4 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -1010,6 +1010,11 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { ObjectType: ast.DescribeBuildingBlock, Name: name, }) + } else if ctx.MENU_KW() != nil { + b.statements = append(b.statements, &ast.DescribeStmt{ + ObjectType: ast.DescribeMenu, + Name: name, + }) } else if ctx.SNIPPET() != nil { b.statements = append(b.statements, &ast.DescribeStmt{ ObjectType: ast.DescribeSnippet, diff --git a/sdk/mpr/reader_documents.go b/sdk/mpr/reader_documents.go index a8b735bb4..45f1e7f3c 100644 --- a/sdk/mpr/reader_documents.go +++ b/sdk/mpr/reader_documents.go @@ -1010,3 +1010,62 @@ func parseJarDependencyExclusion(raw map[string]any) *types.JarDependencyExclusi ArtifactID: extractString(raw["ArtifactId"]), } } + +// ListMenuDocuments returns all standalone Menus$MenuDocument documents. +// +// A menu document holds its entries in a Menus$MenuItemCollection rather than +// directly, but the entries themselves are ordinary Menus$MenuItem elements, so +// the recursive conversion reuses parseNavMenuItem. +func (r *Reader) ListMenuDocuments() ([]*types.MenuDocument, error) { + units, err := r.listUnitsByType("Menus$MenuDocument") + if err != nil { + return nil, err + } + + result := make([]*types.MenuDocument, 0, len(units)) + for _, u := range units { + var raw map[string]any + if err := bson.Unmarshal(u.Contents, &raw); err != nil { + return nil, fmt.Errorf("failed to parse menu document %s: %w", u.ID, err) + } + md := &types.MenuDocument{ + ID: model.ID(u.ID), + ContainerID: model.ID(u.ContainerID), + Name: extractString(raw["Name"]), + Documentation: extractString(raw["Documentation"]), + ExportLevel: extractString(raw["ExportLevel"]), + } + if b, ok := raw["Excluded"].(bool); ok { + md.Excluded = b + } + if coll, ok := raw["ItemCollection"].(map[string]any); ok { + for _, item := range extractBsonArray(coll["Items"]) { + if m, ok := item.(map[string]any); ok { + if mi := parseNavMenuItem(m); mi != nil { + md.Items = append(md.Items, mi) + } + } + } + } + result = append(result, md) + } + return result, nil +} + +// GetMenuDocumentByQualifiedName finds a menu document by module + name. +func (r *Reader) GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) { + all, err := r.ListMenuDocuments() + if err != nil { + return nil, err + } + moduleMap, err := r.buildContainerModuleNameMap() + if err != nil { + return nil, err + } + for _, md := range all { + if md.Name == name && moduleMap[md.ContainerID] == moduleName { + return md, nil + } + } + return nil, fmt.Errorf("menu not found: %s.%s", moduleName, name) +} From 53f444e5219af29e34e76f7ec2ae0334cc2d203b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 12:32:54 +0000 Subject: [PATCH 04/35] Add CREATE OR MODIFY MENU and DROP MENU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Menus were shipped read-only an hour ago on the claim that Mendix offers no way to author a menu document outside Studio Pro. That was wrong — Studio Pro creates them via Add > Menu. The gap was mxcli's missing writer, not a platform limitation, so this adds the writer and corrects the claim wherever it was written down. Syntax reuses the existing navMenuItemDef grammar rule, the same one CREATE NAVIGATION's MENU block uses, so a menu item is written identically wherever it appears and DESCRIBE output feeds straight back into CREATE: create or modify menu MyModule.Main_Menu ( menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home; menu 'Admin' ( menu item 'Accounts' page Administration.Account_Overview; ); ); DESCRIBE MENU now emits that statement rather than an informational block, so describe -> exec -> describe is a fixed point (verified byte-identical). The writer goes through gen + codec rather than hand-built BSON, and that is load-bearing: Studio Pro's menu documents carry typed-array marker 3 on both the item collection and each item's sub-items, which is the codec's default. The navigation writers build menu items by hand with marker 1. Whether that is a latent navigation bug or a genuine difference is unverified — this fixture stores no navigation menu items — so navigation is left untouched rather than changed on a guess. OR MODIFY replaces the item list wholesale, like CREATE NAVIGATION, while preserving the document's ID, container and export level so menu widgets pointing at it keep working. Authoring is modelsdk-only; the legacy engine refuses rather than writing a differently-shaped document. Verified against mxbuild 11.10.0 on an 11.6.6 project: create, modify and drop each leave the app at 0 errors, with a clean-fixture baseline as the control. An earlier run pointed a menu item at a page with a required parameter and mxbuild reported CE1571 at "Menu item" — proof the check actually inspects what was written rather than passing vacuously. That gotcha is documented in the example and syntax help. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/syntax/features_page.go | 44 +++-- docs/01-project/MDL_QUICK_REFERENCE.md | 8 +- .../doctype-tests/26-menu-examples.mdl | 110 +++++++----- mdl/ast/ast_navigation.go | 22 +++ mdl/backend/mcp/unsupported_gen.go | 15 ++ mdl/backend/mock/backend.go | 3 + mdl/backend/mock/mock_navigation.go | 21 +++ mdl/backend/modelsdk/menu_write.go | 164 ++++++++++++++++++ mdl/backend/mpr/backend.go | 14 ++ mdl/backend/navigation.go | 7 +- mdl/executor/cmd_menus.go | 114 ++++++++++-- mdl/executor/cmd_menus_mock_test.go | 115 +++++++++++- mdl/executor/register_stubs.go | 6 + mdl/executor/registry_test.go | 2 + mdl/grammar/MDLParser.g4 | 10 ++ mdl/visitor/visitor_entity.go | 4 + mdl/visitor/visitor_menu.go | 33 ++++ 17 files changed, 604 insertions(+), 88 deletions(-) create mode 100644 mdl/backend/modelsdk/menu_write.go create mode 100644 mdl/visitor/visitor_menu.go diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index 0a7e11ae0..d2e20cbc1 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -183,24 +183,36 @@ func init() { }) Register(SyntaxFeature{ - Path: "menu.describe", - Summary: "Describe a standalone menu document (read-only)", + Path: "menu", + Summary: "Create, describe and drop standalone menu documents", Keywords: []string{ - "describe menu", "menu", "menus", "menu document", "menu item", + "create menu", "describe menu", "drop menu", + "menu", "menus", "menu document", "menu item", }, - Syntax: "DESCRIBE MENU Module.Name;", - Example: "DESCRIBE MENU Atlas_Core.Phone_Menu;\n\n" + - "-- Output is informational; menus cannot be authored via MDL:\n" + - "-- -- Menu: Atlas_Core.Phone_Menu (4 top-level item(s))\n" + - "-- {\n" + - "-- menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home;\n" + - "-- menu 'Admin' (\n" + - "-- menu item 'Accounts' page Administration.Account_Overview;\n" + - "-- );\n" + - "-- }\n\n" + - "-- A menu document is the reusable menu a menu widget points at. It is NOT\n" + - "-- the menu inside a navigation profile — for that use SHOW NAVIGATION MENU\n" + - "-- and ALTER NAVIGATION.", + Syntax: "CREATE [OR MODIFY] MENU Module.Name (\n" + + " MENU ITEM '' [PAGE Module.Page | MICROFLOW Module.Flow] [ICON Module.Collection.name];\n" + + " MENU '' [ICON Module.Collection.name] ( );\n" + + ");\n" + + "DESCRIBE MENU Module.Name;\n" + + "DROP MENU Module.Name;", + Example: "CREATE OR MODIFY MENU MyModule.Main_Menu (\n" + + " menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home;\n" + + " menu item 'Run' microflow MyModule.DoThing;\n" + + " menu 'Admin' (\n" + + " menu item 'Accounts' page Administration.Account_Overview;\n" + + " );\n" + + " menu item 'Plain';\n" + + ");\n\n" + + "-- Notes:\n" + + "-- * A menu document is the reusable menu a menu widget points at. It is\n" + + "-- NOT the menu inside a navigation profile — for that use\n" + + "-- SHOW NAVIGATION MENU and ALTER NAVIGATION. Both use these same items.\n" + + "-- * OR MODIFY replaces the item list wholesale; an omitted item is removed.\n" + + "-- The document's identity and export level are preserved.\n" + + "-- * ICON names an icon collection entry. A glyph or image icon cannot be\n" + + "-- expressed in MDL; DESCRIBE flags those rather than dropping them silently.\n" + + "-- * A page with required parameters cannot be opened from a menu item\n" + + "-- without an argument — Mendix reports CE1571.", SeeAlso: []string{"navigation", "page.show"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 9cfd8987d..20e9e6410 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -962,8 +962,12 @@ MDL uses explicit property declarations for pages: | Describe snippet | `describe snippet Module.Name;` | Round-trippable MDL output | | List building blocks | `show building blocks [in module];` | Read-only; cannot be authored via MDL | | Describe building block | `describe building block Module.Name;` | Informational (header comment + widget tree), not a `create` statement | -| Describe menu | `describe menu Module.Name;` | Standalone `Menus$MenuDocument` (e.g. `Atlas_Core.Phone_Menu`), read-only. Not the navigation-profile menu — see `show navigation menu` | -| Describe menu | `describe menu Module.Name;` | Standalone `Menus$MenuDocument` (e.g. `Atlas_Core.Phone_Menu`), read-only. Not the navigation-profile menu — see `show navigation menu` | +| Create menu | `create [or modify] menu Module.Name ( );` | Standalone `Menus$MenuDocument`. Full replacement: the item list is the document's complete contents | +| Describe menu | `describe menu Module.Name;` | Round-trippable MDL. Not the navigation-profile menu — see `show navigation menu` | +| Drop menu | `drop menu Module.Name;` | | +| Create menu | `create [or modify] menu Module.Name ( );` | Standalone `Menus$MenuDocument`. Full replacement: the item list is the document's complete contents | +| Describe menu | `describe menu Module.Name;` | Round-trippable MDL. Not the navigation-profile menu — see `show navigation menu` | +| Drop menu | `drop menu Module.Name;` | | **DataGrid Column Properties:** diff --git a/mdl-examples/doctype-tests/26-menu-examples.mdl b/mdl-examples/doctype-tests/26-menu-examples.mdl index 3b52473a3..ca55a49bf 100644 --- a/mdl-examples/doctype-tests/26-menu-examples.mdl +++ b/mdl-examples/doctype-tests/26-menu-examples.mdl @@ -4,56 +4,78 @@ -- -- A menu document (Menus$MenuDocument) is a standalone, reusable menu that a -- menu widget points at. Atlas_Core ships two of them, Phone_Menu and --- Tablet_Menu. +-- Tablet_Menu, and Studio Pro creates them via Add > Menu. -- -- It is NOT the menu inside a navigation profile. The two are easy to confuse --- because both are built from the same Menus$MenuItem elements: +-- because both are built from the same Menus$MenuItem elements — which is why +-- the item syntax below is identical to the MENU (...) block of +-- CREATE NAVIGATION: -- --- describe menu Atlas_Core.Phone_Menu; -- this file: a standalone document --- show navigation menu; -- the menu inside a profile --- --- Menus are READ-ONLY. Mendix offers no way to author a menu document outside --- Studio Pro, so there is deliberately no `create menu` / `alter menu` / --- `drop menu`. DESCRIBE output is informational and is not re-executable — it --- opens with a `-- Menu:` header saying so, in the same way DESCRIBE BUILDING --- BLOCK does. +-- create or modify menu Atlas_Core.Phone_Menu ( ... ); -- this file +-- show navigation menu; -- profile menu -- ============================================================================ --- Describe a menu by naming the type explicitly. -describe menu Atlas_Core.Phone_Menu; +-- ---------------------------------------------------------------------------- +-- Create +-- ---------------------------------------------------------------------------- +-- Item forms: +-- * page target -> menu item '' page +-- * microflow target -> menu item '' microflow +-- * no action -> menu item '' +-- * sub-menu -> menu '' ( ...nested items... ) +-- +-- ICON names an entry in an icon collection. It is optional on every form. +create menu MyFirstModule.Main_Menu ( + menu item 'Home' page MyFirstModule.Home_Web icon Atlas_Core.Atlas_Filled.home; + menu item 'Run' microflow MyFirstModule.MyFirstLogic; + menu 'Admin' ( + menu item 'Accounts' page Administration.Account_Overview; + ); + menu item 'Plain'; +); + +-- OR MODIFY replaces the item list wholesale, exactly like CREATE NAVIGATION: +-- the list given is the document's complete contents, so an omitted item is a +-- removed item. The document's identity and its export level are preserved, so +-- menu widgets pointing at it keep working. +create or modify menu MyFirstModule.Main_Menu ( + menu item 'Home' page MyFirstModule.Home_Web icon Atlas_Core.Atlas_Filled.home; + menu item 'Run' microflow MyFirstModule.MyFirstLogic; +); + +-- ---------------------------------------------------------------------------- +-- Describe +-- ---------------------------------------------------------------------------- +-- DESCRIBE emits a re-executable CREATE OR MODIFY statement, so +-- describe -> exec -> describe is a fixed point. +describe menu MyFirstModule.Main_Menu; -describe menu Atlas_Core.Tablet_Menu; +-- The type is auto-detected too, as for any other document type. +describe MyFirstModule.Main_Menu; --- The type can also be auto-detected, as for any other document type. -describe Atlas_Core.Phone_Menu; +-- ---------------------------------------------------------------------------- +-- Drop +-- ---------------------------------------------------------------------------- +drop menu MyFirstModule.Main_Menu; -- ---------------------------------------------------------------------------- --- Shape of the output --- ---------------------------------------------------------------------------- --- --- -- Menu: Atlas_Core.Phone_Menu (4 top-level item(s)) --- -- Export level: Hidden --- -- Menus are read-only; they cannot be created via MDL. --- { --- menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home; --- menu item 'Rebuild' microflow Administration.Rebuild; --- menu 'Admin' ( --- menu item 'Accounts' page Administration.Account_Overview; --- ); --- menu item 'Settings'; --- } --- --- Items render with the same syntax navigation menus use, so the two read --- alike: --- * a leaf with a page target -> menu item '' page --- * a leaf with a microflow target -> menu item '' microflow --- * a leaf with no action -> menu item '' --- * a sub-menu -> menu '' ( ...nested items... ) --- --- An `icon` clause is emitted only for Forms$IconCollectionIcon, the one icon --- variant MDL can name. A glyph icon (numeric code) or an image icon is --- reported on its own comment line rather than dropped silently, so what is --- missing is visible: --- --- -- icon a numeric glyph code (Forms$GlyphIcon) is not reproducible by MDL; --- -- set it in Studio Pro +-- Gotchas +-- ---------------------------------------------------------------------------- +-- +-- 1. A page with required parameters cannot be opened from a menu item without +-- supplying an argument. Mendix rejects it with CE1571 ("No argument has been +-- selected for parameter ..."), which `mx check` reports at "Menu item". +-- Point the item at a parameterless page, or open the page from a microflow. +-- +-- 2. Only Forms$IconCollectionIcon can be expressed by the ICON clause. A glyph +-- icon (numeric code) or an image icon is reported by DESCRIBE on its own +-- comment line rather than dropped silently: +-- +-- -- icon a numeric glyph code (Forms$GlyphIcon) is not reproducible by +-- -- CREATE MENU; set it in Studio Pro +-- +-- Re-running such output therefore loses that icon — visibly, not silently. +-- +-- 3. Authoring requires the default (modelsdk) engine. Under +-- MXCLI_ENGINE=legacy, create/modify/drop refuse rather than write a +-- differently-shaped document. diff --git a/mdl/ast/ast_navigation.go b/mdl/ast/ast_navigation.go index 7ce9c6715..20232428d 100644 --- a/mdl/ast/ast_navigation.go +++ b/mdl/ast/ast_navigation.go @@ -31,3 +31,25 @@ type NavMenuItemDef struct { Icon string // ICON 'Module.Collection.name', empty for none Items []NavMenuItemDef // Sub-items (for MENU 'caption' (...)) } + +// CreateMenuStmt is `create [or modify] menu Module.Name ( )` — a +// standalone Menus$MenuDocument, not the menu inside a navigation profile. +// Items reuse NavMenuItemDef so both constructs share one item syntax. +// +// Like CREATE NAVIGATION, this is a full replacement: the item list given is the +// document's complete contents, so an omitted item is a removed item. +type CreateMenuStmt struct { + Name QualifiedName + Items []NavMenuItemDef + CreateOrModify bool // CREATE OR MODIFY / OR REPLACE + Documentation string +} + +func (s *CreateMenuStmt) isStatement() {} + +// DropMenuStmt is `drop menu Module.Name`. +type DropMenuStmt struct { + Name QualifiedName +} + +func (s *DropMenuStmt) isStatement() {} diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index 54bccf0ba..755d6c19f 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -184,6 +184,11 @@ func (unsupportedBackend) CreateLayout(_ *pages.Layout) (err0 error) { return } +func (unsupportedBackend) CreateMenuDocument(_ *types.MenuDocument) (err0 error) { + err0 = errUnsupported("CreateMenuDocument") + return +} + func (unsupportedBackend) CreateMicroflow(_ *microflows.Microflow) (err0 error) { err0 = errUnsupported("CreateMicroflow") return @@ -354,6 +359,11 @@ func (unsupportedBackend) DeleteLayout(_ model.ID) (err0 error) { return } +func (unsupportedBackend) DeleteMenuDocument(_ model.ID) (err0 error) { + err0 = errUnsupported("DeleteMenuDocument") + return +} + func (unsupportedBackend) DeleteMicroflow(_ model.ID) (err0 error) { err0 = errUnsupported("DeleteMicroflow") return @@ -1132,6 +1142,11 @@ func (unsupportedBackend) UpdateLayout(_ *pages.Layout) (err0 error) { return } +func (unsupportedBackend) UpdateMenuDocument(_ *types.MenuDocument) (err0 error) { + err0 = errUnsupported("UpdateMenuDocument") + return +} + func (unsupportedBackend) UpdateMicroflow(_ *microflows.Microflow) (err0 error) { err0 = errUnsupported("UpdateMicroflow") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index 508364375..70ed12609 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -249,6 +249,9 @@ type MockBackend struct { ListMenuDocumentsFunc func() ([]*types.MenuDocument, error) GetMenuDocumentByQualifiedNameFunc func(moduleName, name string) (*types.MenuDocument, error) + CreateMenuDocumentFunc func(md *types.MenuDocument) error + UpdateMenuDocumentFunc func(md *types.MenuDocument) error + DeleteMenuDocumentFunc func(id model.ID) error CreateImageCollectionFunc func(ic *types.ImageCollection) error UpdateImageCollectionFunc func(ic *types.ImageCollection) error DeleteImageCollectionFunc func(id string) error diff --git a/mdl/backend/mock/mock_navigation.go b/mdl/backend/mock/mock_navigation.go index 794e8e687..60962ee66 100644 --- a/mdl/backend/mock/mock_navigation.go +++ b/mdl/backend/mock/mock_navigation.go @@ -43,3 +43,24 @@ func (m *MockBackend) GetMenuDocumentByQualifiedName(moduleName, name string) (* } return nil, fmt.Errorf("MockBackend.GetMenuDocumentByQualifiedName not configured") } + +func (m *MockBackend) CreateMenuDocument(md *types.MenuDocument) error { + if m.CreateMenuDocumentFunc != nil { + return m.CreateMenuDocumentFunc(md) + } + return fmt.Errorf("MockBackend.CreateMenuDocument not configured") +} + +func (m *MockBackend) UpdateMenuDocument(md *types.MenuDocument) error { + if m.UpdateMenuDocumentFunc != nil { + return m.UpdateMenuDocumentFunc(md) + } + return fmt.Errorf("MockBackend.UpdateMenuDocument not configured") +} + +func (m *MockBackend) DeleteMenuDocument(id model.ID) error { + if m.DeleteMenuDocumentFunc != nil { + return m.DeleteMenuDocumentFunc(id) + } + return fmt.Errorf("MockBackend.DeleteMenuDocument not configured") +} diff --git a/mdl/backend/modelsdk/menu_write.go b/mdl/backend/modelsdk/menu_write.go new file mode 100644 index 000000000..cc321887f --- /dev/null +++ b/mdl/backend/modelsdk/menu_write.go @@ -0,0 +1,164 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/modelsdk/element" + genMenus "github.com/mendixlabs/mxcli/modelsdk/gen/menus" + genPages "github.com/mendixlabs/mxcli/modelsdk/gen/pages" + genTexts "github.com/mendixlabs/mxcli/modelsdk/gen/texts" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// menuDocumentType is the BSON storage name of a standalone menu document. +const menuDocumentType = "Menus$MenuDocument" + +// CreateMenuDocument writes a new Menus$MenuDocument unit. +// +// This goes through gen + codec rather than hand-built BSON, which is what makes +// the typed-array markers come out right: the codec emits the default marker 3 +// for a PartList, and Studio Pro's own menu documents carry 3 on both the +// collection's Items and each item's sub-Items (verified against Atlas_Core's +// Phone_Menu / Tablet_Menu). The navigation writers build their menu items by +// hand and use 1 — that difference is deliberate here and not copied. +func (b *Backend) CreateMenuDocument(md *types.MenuDocument) error { + if md == nil { + return fmt.Errorf("CreateMenuDocument: nil menu") + } + if b.writer == nil { + return fmt.Errorf("CreateMenuDocument: not connected for writing") + } + if md.ID == "" { + md.ID = model.ID(mmpr.GenerateID()) + } + contents, err := encodeMenuDocument(md) + if err != nil { + return fmt.Errorf("CreateMenuDocument: encode: %w", err) + } + return b.writer.InsertUnit(string(md.ID), string(md.ContainerID), "Documents", menuDocumentType, contents) +} + +// UpdateMenuDocument rebuilds a menu document and rewrites its unit. The document +// is small and rebuilt wholesale, matching how enumerations are updated. +func (b *Backend) UpdateMenuDocument(md *types.MenuDocument) error { + if md == nil { + return fmt.Errorf("UpdateMenuDocument: nil menu") + } + if b.writer == nil { + return fmt.Errorf("UpdateMenuDocument: not connected for writing") + } + if md.ID == "" { + return fmt.Errorf("UpdateMenuDocument: menu %q has no ID", md.Name) + } + contents, err := encodeMenuDocument(md) + if err != nil { + return fmt.Errorf("UpdateMenuDocument: encode: %w", err) + } + return b.writer.UpdateRawUnit(string(md.ID), contents) +} + +// DeleteMenuDocument removes a menu document unit by ID. +func (b *Backend) DeleteMenuDocument(id model.ID) error { + if b.writer == nil { + return fmt.Errorf("DeleteMenuDocument: not connected for writing") + } + return b.writer.DeleteUnit(string(id)) +} + +func encodeMenuDocument(md *types.MenuDocument) ([]byte, error) { + g := genMenus.NewMenuDocument() + g.SetID(element.ID(md.ID)) + g.SetName(md.Name) + g.SetDocumentation(md.Documentation) + g.SetExcluded(md.Excluded) + exportLevel := md.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + g.SetExportLevel(exportLevel) + + coll := genMenus.NewMenuItemCollection() + coll.SetID(element.ID(mmpr.GenerateID())) + for _, item := range md.Items { + coll.AddItems(menuItemToGen(item)) + } + g.SetItemCollection(coll) + + return (&codec.Encoder{}).Encode(g) +} + +// menuItemToGen converts one semantic menu item (and its sub-items) to gen. It is +// the inverse of navMenuItemFromGen, and deliberately mirrors the same three +// concerns: caption, icon, action. +func menuItemToGen(item *types.NavMenuItem) element.Element { + g := genMenus.NewMenuItem() + g.SetID(element.ID(mmpr.GenerateID())) + g.SetCaption(menuCaptionToGen(item.Caption)) + g.SetIcon(menuIconToGen(item)) + g.SetAction(menuActionToGen(item)) + for _, sub := range item.Items { + g.AddItems(menuItemToGen(sub)) + } + return g +} + +// menuCaptionToGen builds the Texts$Text / Texts$Translation pair a caption is +// stored as. en_US matches what the navigation writers emit. +func menuCaptionToGen(caption string) element.Element { + t := genTexts.NewText() + t.SetID(element.ID(mmpr.GenerateID())) + tr := genTexts.NewTranslation() + tr.SetID(element.ID(mmpr.GenerateID())) + tr.SetLanguageCode("en_US") + tr.SetText(caption) + t.AddTranslations(tr) + return t +} + +// menuIconToGen emits only Forms$IconCollectionIcon, the one variant MDL can +// name. A glyph icon carries a numeric code and an image icon points into an +// image collection; neither is expressible in the ICON clause, so an item that +// had one keeps no icon rather than getting a wrong one. DESCRIBE flags those on +// the way out, so the loss is visible rather than silent. +func menuIconToGen(item *types.NavMenuItem) element.Element { + if item.Icon == "" { + return nil + } + icon := genPages.NewIconCollectionIcon() + icon.SetID(element.ID(mmpr.GenerateID())) + icon.SetImageQualifiedName(item.Icon) + return icon +} + +// menuActionToGen builds the item's client action. The gen type names are the SDK +// names; their storage names are what Mendix writes — PageClientAction is +// Forms$FormAction, NoClientAction is Forms$NoAction. +func menuActionToGen(item *types.NavMenuItem) element.Element { + switch { + case item.Page != "": + a := genPages.NewPageClientAction() + a.SetID(element.ID(mmpr.GenerateID())) + ps := genPages.NewPageSettings() + ps.SetID(element.ID(mmpr.GenerateID())) + ps.SetPageQualifiedName(item.Page) + a.SetPageSettings(ps) + return a + case item.Microflow != "": + a := genPages.NewMicroflowClientAction() + a.SetID(element.ID(mmpr.GenerateID())) + ms := genPages.NewMicroflowSettings() + ms.SetID(element.ID(mmpr.GenerateID())) + ms.SetMicroflowQualifiedName(item.Microflow) + a.SetMicroflowSettings(ms) + return a + default: + a := genPages.NewNoClientAction() + a.SetID(element.ID(mmpr.GenerateID())) + return a + } +} diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 108b91dd7..f045a6a6a 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -878,3 +878,17 @@ func (b *MprBackend) ListMenuDocuments() ([]*types.MenuDocument, error) { func (b *MprBackend) GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) { return b.reader.GetMenuDocumentByQualifiedName(moduleName, name) } + +// Menu-document writes are implemented on the modelsdk engine only. The legacy +// writer builds menu items by hand with typed-array marker 1, which does not +// match what Studio Pro stores in a menu document (3) — rather than ship a +// second, differently-shaped writer, this refuses so the caller is told plainly. +func (b *MprBackend) CreateMenuDocument(md *types.MenuDocument) error { + return errors.New("creating a menu requires the modelsdk engine — rerun without MXCLI_ENGINE=legacy") +} +func (b *MprBackend) UpdateMenuDocument(md *types.MenuDocument) error { + return errors.New("modifying a menu requires the modelsdk engine — rerun without MXCLI_ENGINE=legacy") +} +func (b *MprBackend) DeleteMenuDocument(id model.ID) error { + return errors.New("dropping a menu requires the modelsdk engine — rerun without MXCLI_ENGINE=legacy") +} diff --git a/mdl/backend/navigation.go b/mdl/backend/navigation.go index 3d23c3d1c..0318e5651 100644 --- a/mdl/backend/navigation.go +++ b/mdl/backend/navigation.go @@ -14,9 +14,10 @@ type NavigationBackend interface { UpdateNavigationProfile(navDocID model.ID, profileName string, spec types.NavigationProfileSpec) error // Menu documents are standalone reusable menus (Menus$MenuDocument), not - // the menu embedded in a navigation profile. They are read-only: Mendix - // offers no way to author one outside Studio Pro, so there is deliberately - // no Create/Update/Delete here. + // the menu embedded in a navigation profile. ListMenuDocuments() ([]*types.MenuDocument, error) GetMenuDocumentByQualifiedName(moduleName, name string) (*types.MenuDocument, error) + CreateMenuDocument(md *types.MenuDocument) error + UpdateMenuDocument(md *types.MenuDocument) error + DeleteMenuDocument(id model.ID) error } diff --git a/mdl/executor/cmd_menus.go b/mdl/executor/cmd_menus.go index 9bac9202f..d4ddecbb8 100644 --- a/mdl/executor/cmd_menus.go +++ b/mdl/executor/cmd_menus.go @@ -8,6 +8,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" ) // describeMenu renders a standalone Menus$MenuDocument — the reusable menu a @@ -15,11 +16,8 @@ import ( // profile. Atlas_Core ships Phone_Menu and Tablet_Menu. // // The entries are ordinary Menus$MenuItem elements, so the tree is rendered by -// printMenuMDL, the same renderer DESCRIBE NAVIGATION uses. Output is -// deliberately not re-executable: Mendix offers no way to author a menu document -// outside Studio Pro, so there is no CREATE MENU for it to round-trip into. That -// is stated in the header rather than left for the reader to discover, following -// the DESCRIBE BUILDING BLOCK precedent. +// printMenuMDL, the same renderer DESCRIBE NAVIGATION uses — and the same syntax +// CREATE MENU accepts, so the output round-trips. func describeMenu(ctx *ExecContext, name ast.QualifiedName) error { if !ctx.Connected() { return mdlerrors.NewNotConnected() @@ -38,23 +36,107 @@ func describeMenu(ctx *ExecContext, name ast.QualifiedName) error { strings.ReplaceAll(md.Documentation, "\n", "\n * ")) } - fmt.Fprintf(ctx.Output, "-- Menu: %s.%s (%d top-level item(s))\n", - name.Module, md.Name, len(md.Items)) - if md.ExportLevel != "" { - fmt.Fprintf(ctx.Output, "-- Export level: %s\n", md.ExportLevel) - } if md.Excluded { fmt.Fprintln(ctx.Output, "-- Excluded from the project") } - fmt.Fprintln(ctx.Output, "-- Menus are read-only; they cannot be created via MDL.") - if len(md.Items) == 0 { - fmt.Fprintln(ctx.Output, "{ }") + // Output is re-executable: the item syntax is the same one CREATE MENU + // accepts, so describe → exec → describe is a fixed point. + fmt.Fprintf(ctx.Output, "create or modify menu %s.%s (\n", name.Module, md.Name) + printMenuMDL(ctx.Output, md.Items, 1, "CREATE MENU") + fmt.Fprintln(ctx.Output, ");") + return nil +} + +// execCreateMenu handles CREATE [OR MODIFY] MENU Module.Name ( items ). +// +// Like CREATE NAVIGATION, the item list is the document's complete contents, so +// a modify replaces the items wholesale rather than merging. The existing +// document's $ID is reused on modify, so references to the menu survive and the +// unit is rewritten in place rather than replaced. +func execCreateMenu(ctx *ExecContext, s *ast.CreateMenuStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + + mod, err := ctx.Backend.GetModuleByName(s.Name.Module) + if err != nil || mod == nil { + return mdlerrors.NewNotFound("module", s.Name.Module) + } + + existing, _ := ctx.Backend.GetMenuDocumentByQualifiedName(s.Name.Module, s.Name.Name) + if existing != nil && !s.CreateOrModify { + return mdlerrors.NewAlreadyExists("menu", s.Name.String()) + } + + md := &types.MenuDocument{ + Name: s.Name.Name, + ContainerID: mod.ID, + Documentation: s.Documentation, + Items: menuItemsFromAST(s.Items), + } + + if existing != nil { + // Preserve the document's identity and the properties MDL does not + // author, so a modify does not silently reset them. + md.ID = existing.ID + md.ContainerID = existing.ContainerID + md.ExportLevel = existing.ExportLevel + md.Excluded = existing.Excluded + if md.Documentation == "" { + md.Documentation = existing.Documentation + } + if err := ctx.Backend.UpdateMenuDocument(md); err != nil { + return mdlerrors.NewBackend("update menu", err) + } + fmt.Fprintf(ctx.Output, "Modified menu %s\n", s.Name.String()) return nil } - fmt.Fprintln(ctx.Output, "{") - printMenuMDL(ctx.Output, md.Items, 1, "MDL") - fmt.Fprintln(ctx.Output, "}") + if err := ctx.Backend.CreateMenuDocument(md); err != nil { + return mdlerrors.NewBackend("create menu", err) + } + fmt.Fprintf(ctx.Output, "Created menu %s\n", s.Name.String()) return nil } + +// execDropMenu handles DROP MENU Module.Name. +func execDropMenu(ctx *ExecContext, s *ast.DropMenuStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + md, err := ctx.Backend.GetMenuDocumentByQualifiedName(s.Name.Module, s.Name.Name) + if err != nil || md == nil { + return mdlerrors.NewNotFound("menu", s.Name.String()) + } + if err := ctx.Backend.DeleteMenuDocument(md.ID); err != nil { + return mdlerrors.NewBackend("drop menu", err) + } + fmt.Fprintf(ctx.Output, "Dropped menu %s\n", s.Name.String()) + return nil +} + +// menuItemsFromAST converts parsed menu items to the semantic model. The AST and +// semantic shapes differ only in how the target is held (pointer vs string), so +// this stays a direct mapping rather than acquiring behaviour. +func menuItemsFromAST(defs []ast.NavMenuItemDef) []*types.NavMenuItem { + var out []*types.NavMenuItem + for _, d := range defs { + item := &types.NavMenuItem{Caption: d.Caption, Icon: d.Icon} + if d.Icon != "" { + item.IconType = "Forms$IconCollectionIcon" + } + if d.Page != nil { + item.Page = d.Page.String() + item.ActionType = "PageAction" + } else if d.Microflow != nil { + item.Microflow = d.Microflow.String() + item.ActionType = "MicroflowAction" + } else { + item.ActionType = "NoAction" + } + item.Items = menuItemsFromAST(d.Items) + out = append(out, item) + } + return out +} diff --git a/mdl/executor/cmd_menus_mock_test.go b/mdl/executor/cmd_menus_mock_test.go index fa92224b0..95dd5c0e5 100644 --- a/mdl/executor/cmd_menus_mock_test.go +++ b/mdl/executor/cmd_menus_mock_test.go @@ -10,6 +10,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" "github.com/mendixlabs/mxcli/mdl/backend/mock" "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" ) // menuBackend returns a MockBackend serving one menu document. @@ -60,8 +61,8 @@ func TestDescribeMenu_Nested(t *testing.T) { assertNoError(t, describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Main_Menu"})) out := buf.String() - assertContainsStr(t, out, "-- Menu: Atlas_Core.Main_Menu (3 top-level item(s))") - assertContainsStr(t, out, "read-only") + // The output is re-executable, so it opens with the statement that recreates it. + assertContainsStr(t, out, "create or modify menu Atlas_Core.Main_Menu (") assertContainsStr(t, out, "menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home;") // A sub-menu opens a nested block and its children are indented one level in. @@ -69,11 +70,11 @@ func TestDescribeMenu_Nested(t *testing.T) { assertContainsStr(t, out, " menu item 'Accounts' page Administration.Account_Overview;") assertContainsStr(t, out, " menu item 'Rebuild' microflow Administration.Rebuild;") - // The glyph icon is reported rather than dropped, and names no statement that - // could author it — a menu document cannot be authored at all. - assertContainsStr(t, out, "is not reproducible by MDL") + // The glyph icon is reported rather than dropped, and points at the statement + // that would have to reproduce it — CREATE MENU, not CREATE NAVIGATION. + assertContainsStr(t, out, "is not reproducible by CREATE MENU") if strings.Contains(out, "CREATE NAVIGATION") { - t.Errorf("menu output should not point at CREATE NAVIGATION, which cannot author a menu document:\n%s", out) + t.Errorf("menu output should not point at CREATE NAVIGATION, which authors a profile menu, not a menu document:\n%s", out) } } @@ -81,7 +82,9 @@ func TestDescribeMenu_Empty(t *testing.T) { md := &types.MenuDocument{Name: "Empty_Menu"} ctx, buf := newMockCtx(t, withBackend(menuBackend(md))) assertNoError(t, describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Empty_Menu"})) - assertContainsStr(t, buf.String(), "{ }") + // An empty menu still describes to a statement that recreates it. + assertContainsStr(t, buf.String(), "create or modify menu Atlas_Core.Empty_Menu (") + assertContainsStr(t, buf.String(), ");") } func TestDescribeMenu_NotFound(t *testing.T) { @@ -103,3 +106,101 @@ func TestDescribeMenu_Documentation(t *testing.T) { assertNoError(t, describeMenu(ctx, ast.QualifiedName{Module: "Atlas_Core", Name: "Doc_Menu"})) assertContainsStr(t, buf.String(), "Phone navigation.") } + +// TestCreateMenu_RejectsDuplicateWithoutOrModify pins the create/modify split: +// a plain CREATE against an existing menu must fail rather than silently +// replacing a document the author did not mean to touch. +func TestCreateMenu_RejectsDuplicateWithoutOrModify(t *testing.T) { + existing := &types.MenuDocument{ID: "menu-1", Name: "Main_Menu"} + mb := menuBackend(existing) + mb.GetModuleByNameFunc = func(name string) (*model.Module, error) { + return &model.Module{BaseElement: model.BaseElement{ID: "mod-1"}, Name: name}, nil + } + + ctx, _ := newMockCtx(t, withBackend(mb)) + err := execCreateMenu(ctx, &ast.CreateMenuStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Main_Menu"}, + }) + if err == nil { + t.Fatal("expected CREATE MENU on an existing menu to fail without OR MODIFY") + } +} + +// TestCreateMenu_OrModifyPreservesIdentity checks that a modify rewrites the +// stored document rather than minting a new one, and does not reset the +// properties MDL does not author. Losing the ID would break every menu widget +// pointing at it; resetting ExportLevel would silently change the module's API. +func TestCreateMenu_OrModifyPreservesIdentity(t *testing.T) { + existing := &types.MenuDocument{ + ID: "menu-1", ContainerID: "folder-9", Name: "Main_Menu", + ExportLevel: "Public", Documentation: "kept", + } + var updated *types.MenuDocument + mb := menuBackend(existing) + mb.GetModuleByNameFunc = func(name string) (*model.Module, error) { + return &model.Module{BaseElement: model.BaseElement{ID: "mod-1"}, Name: name}, nil + } + mb.UpdateMenuDocumentFunc = func(md *types.MenuDocument) error { updated = md; return nil } + mb.CreateMenuDocumentFunc = func(md *types.MenuDocument) error { + t.Error("OR MODIFY must update the existing menu, not create a second one") + return nil + } + + ctx, _ := newMockCtx(t, withBackend(mb)) + assertNoError(t, execCreateMenu(ctx, &ast.CreateMenuStmt{ + Name: ast.QualifiedName{Module: "MyModule", Name: "Main_Menu"}, + CreateOrModify: true, + Items: []ast.NavMenuItemDef{{Caption: "Home"}}, + })) + + if updated == nil { + t.Fatal("UpdateMenuDocument was not called") + } + if updated.ID != "menu-1" { + t.Errorf("ID = %q, want the stored menu-1 — a fresh ID orphans every reference", updated.ID) + } + if updated.ContainerID != "folder-9" { + t.Errorf("ContainerID = %q, want folder-9 — a modify must not move the document", updated.ContainerID) + } + if updated.ExportLevel != "Public" { + t.Errorf("ExportLevel = %q, want Public preserved", updated.ExportLevel) + } + if updated.Documentation != "kept" { + t.Errorf("Documentation = %q, want the stored text preserved", updated.Documentation) + } + if len(updated.Items) != 1 || updated.Items[0].Caption != "Home" { + t.Errorf("items were not replaced by the statement's list: %+v", updated.Items) + } +} + +// TestMenuItemsFromAST_Nested covers the AST→model mapping the executor owns, +// including the recursion the fixture's flat Atlas menus cannot exercise. +func TestMenuItemsFromAST_Nested(t *testing.T) { + page := ast.QualifiedName{Module: "M", Name: "P"} + mf := ast.QualifiedName{Module: "M", Name: "F"} + items := menuItemsFromAST([]ast.NavMenuItemDef{{ + Caption: "Top", + Items: []ast.NavMenuItemDef{ + {Caption: "Pg", Page: &page, Icon: "M.C.i"}, + {Caption: "Mf", Microflow: &mf}, + {Caption: "Plain"}, + }, + }}) + + if len(items) != 1 || len(items[0].Items) != 3 { + t.Fatalf("expected 1 top item with 3 children, got %+v", items) + } + kids := items[0].Items + if kids[0].Page != "M.P" || kids[0].ActionType != "PageAction" { + t.Errorf("page child = %+v", kids[0]) + } + if kids[0].IconType != "Forms$IconCollectionIcon" { + t.Errorf("an ICON clause must record the icon type it round-trips as, got %q", kids[0].IconType) + } + if kids[1].Microflow != "M.F" || kids[1].ActionType != "MicroflowAction" { + t.Errorf("microflow child = %+v", kids[1]) + } + if kids[2].ActionType != "NoAction" { + t.Errorf("a target-less item must be NoAction, got %q", kids[2].ActionType) + } +} diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index bae5ee4fb..82482785b 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -205,6 +205,12 @@ func registerNavigationHandlers(r *Registry) { r.Register(&ast.AlterNavigationStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execAlterNavigation(ctx, stmt.(*ast.AlterNavigationStmt)) }) + r.Register(&ast.CreateMenuStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execCreateMenu(ctx, stmt.(*ast.CreateMenuStmt)) + }) + r.Register(&ast.DropMenuStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDropMenu(ctx, stmt.(*ast.DropMenuStmt)) + }) } func registerImageHandlers(r *Registry) { diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 538505250..e0a2998e4 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -167,6 +167,8 @@ func allKnownStatements() []ast.Statement { &ast.AlterModelStmt{}, &ast.AlterModuleJarDepStmt{}, &ast.AlterNavigationStmt{}, + &ast.CreateMenuStmt{}, + &ast.DropMenuStmt{}, &ast.AlterODataClientStmt{}, &ast.AlterODataServiceStmt{}, &ast.AlterPageStmt{}, diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index 7b7d9e63f..ebab6307f 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -126,6 +126,7 @@ createStatement | createKnowledgeBaseStatement | createAgentStatement | createNanoflowStatement + | createMenuStatement ) ; @@ -297,6 +298,14 @@ navMenuItemDef | MENU_KW STRING_LITERAL (ICON qualifiedName)? LPAREN navMenuItemDef* RPAREN SEMICOLON? ; +// A standalone menu document (Menus$MenuDocument) — the reusable menu a menu +// widget points at, as opposed to the menu inside a navigation profile. Both are +// built from the same items, so this reuses navMenuItemDef rather than defining a +// second item syntax. +createMenuStatement + : MENU_KW qualifiedName LPAREN navMenuItemDef* RPAREN + ; + dropStatement : DROP ENTITY qualifiedName | DROP ASSOCIATION qualifiedName @@ -306,6 +315,7 @@ dropStatement | DROP NANOFLOW qualifiedName | DROP PAGE qualifiedName | DROP SNIPPET qualifiedName + | DROP MENU_KW qualifiedName | DROP MODULE qualifiedName | DROP NOTEBOOK qualifiedName | DROP JAVA ACTION qualifiedName diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index c4b043d6e..0ed3fe93e 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -801,6 +801,10 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { b.statements = append(b.statements, &ast.DropSnippetStmt{ Name: buildQualifiedName(names[0]), }) + } else if ctx.MENU_KW() != nil { + b.statements = append(b.statements, &ast.DropMenuStmt{ + Name: buildQualifiedName(names[0]), + }) } else if ctx.JAVASCRIPT() != nil && ctx.ACTION() != nil { b.statements = append(b.statements, &ast.DropJavaScriptActionStmt{ Name: buildQualifiedName(names[0]), diff --git a/mdl/visitor/visitor_menu.go b/mdl/visitor/visitor_menu.go new file mode 100644 index 000000000..295466adc --- /dev/null +++ b/mdl/visitor/visitor_menu.go @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// ExitCreateMenuStatement handles CREATE [OR MODIFY] MENU Module.Name ( items ). +// +// The items reuse navMenuItemDef, the same rule CREATE NAVIGATION's MENU block +// uses, so buildNavMenuItemDef is reused verbatim — a menu item is written the +// same way wherever it appears. +func (b *Builder) ExitCreateMenuStatement(ctx *parser.CreateMenuStatementContext) { + qn := ctx.QualifiedName() + if qn == nil { + return + } + + stmt := &ast.CreateMenuStmt{Name: buildQualifiedName(qn)} + for _, itemCtx := range ctx.AllNavMenuItemDef() { + stmt.Items = append(stmt.Items, buildNavMenuItemDef(itemCtx)) + } + + if createStmt := findParentCreateStatement(ctx); createStmt != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { + stmt.CreateOrModify = true + } + } + + b.statements = append(b.statements, stmt) +} From 7d6cd0b334a43d1689317b9c6086f6fe264c25e7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 12:44:41 +0000 Subject: [PATCH 05/35] Wire menus into the CLI describe command, skill, docs-site and status MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answering "did the docs get done": examples, syntax help and the quick reference did; the skill, the docs-site page and the standalone `mxcli describe` command did not. The last of those was a functional gap, not a documentation one. `mxcli describe` is a separate Cobra command with its own type list and its own $Type/ObjectType maps, independent of the executor's DESCRIBE. Menus were in neither, so `mxcli describe menu Atlas_Core.Phone_Menu` failed with "Unknown type: menu" and the bare `mxcli describe Atlas_Core.Phone_Menu` failed to resolve, even though both worked from the REPL and from exec. Both paths now work. The syntax topic moves from `menu` to `navigation.menu-document`. At the top level it resolved but never appeared in any listing, because the top-level index is a curated list; under `navigation` it shows up next to the profile menu it is most often confused with. The skill section goes in manage-navigation.md, which already covers profile menus, and leads with how to tell the two apart — `show navigation menu` for the menu inside a profile, `describe menu` for a standalone document. It carries the CE1571 parameterised-page trap and the icon round-trip limit, both of which cost a debugging cycle to find. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/mendix/manage-navigation.md | 52 ++++++++ CLAUDE.md | 1 + cmd/mxcli/cmd_describe.go | 7 +- cmd/mxcli/syntax/features_page.go | 6 +- docs-site/src/SUMMARY.md | 1 + docs-site/src/reference/navigation/README.md | 1 + docs-site/src/reference/navigation/menu.md | 131 +++++++++++++++++++ 7 files changed, 195 insertions(+), 4 deletions(-) create mode 100644 docs-site/src/reference/navigation/menu.md diff --git a/.claude/skills/mendix/manage-navigation.md b/.claude/skills/mendix/manage-navigation.md index 8dce073cf..563833b7d 100644 --- a/.claude/skills/mendix/manage-navigation.md +++ b/.claude/skills/mendix/manage-navigation.md @@ -18,6 +18,7 @@ Use when the user asks to: - **Home Page** — The default page shown after login. Can be a PAGE or MICROFLOW. - **Role-Based Home Pages** — Override the default home page per user role (e.g., admins see a dashboard, users see a task list). - **Menu Items** — Hierarchical menu tree. Each item has a caption and optionally targets a PAGE or MICROFLOW. Sub-menus nest with `menu 'caption' (...)`. +- **Menu Documents** — A *separate* document type (`Menus$MenuDocument`) holding a reusable menu that a menu widget points at, e.g. Atlas_Core's `Phone_Menu`. Not the same thing as a profile's menu, though both are built from the same items, so the item syntax is identical. Managed with `create/describe/drop menu` — see below. - **Login Page** — Custom login page (optional; Mendix provides a default). - **Not-Found Page** — Custom 404 page (optional). @@ -244,6 +245,55 @@ create or replace navigation Responsive ); ``` +## Menu Documents (standalone, reusable) + +A profile menu lives *inside* a navigation profile and is edited with +`create or replace navigation`. A **menu document** is its own document, and a +menu widget on a page points at it. Atlas_Core ships `Phone_Menu` and +`Tablet_Menu`. + +Tell them apart by which command reads them: + +```sql +show navigation menu; -- the menu inside each profile +describe menu Atlas_Core.Phone_Menu; -- a standalone menu document +``` + +Menu documents use the same item syntax as the profile `menu (...)` block: + +```sql +create or modify menu MyModule.Main_Menu ( + menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home; + menu item 'Run' microflow MyModule.DoThing; + menu 'Admin' ( + menu item 'Accounts' page Administration.Account_Overview; + ); + menu item 'Plain'; +); + +drop menu MyModule.Main_Menu; +``` + +`describe menu` emits a re-executable `create or modify` statement, so +describe → edit → exec is the normal editing loop. + +**`or modify` replaces the whole item list.** An omitted item is a removed item, +exactly as with `create or replace navigation`. The document's identity and +export level are preserved, so menu widgets pointing at it keep working. + +### Gotchas + +- **A menu item cannot open a page that takes a required parameter.** There is + nowhere to supply the argument, and Mendix reports **CE1571** ("No argument has + been selected for parameter …") against `Menu item`. Point the item at a + parameterless page, or call a microflow that opens the page. +- **Only icon-collection icons round-trip.** A glyph icon (numeric code) or an + image icon cannot be written by MDL; `describe` flags those on their own + comment line rather than dropping them silently, so re-running the output + loses that icon visibly. +- **Authoring needs the default engine.** Under `MXCLI_ENGINE=legacy`, + create/modify/drop refuse rather than writing a differently-shaped document. + ## Checklist - [ ] Profile name matches an existing profile (Responsive, Phone, Tablet, or a native profile) @@ -254,3 +304,5 @@ create or replace navigation Responsive - [ ] `icon` is a qualified name (not a string); hyphenated segments are double-quoted - [ ] The icon exists — check with `describe icon collection Module.Name`, do not guess - [ ] Use `describe navigation` to verify changes after applying +- [ ] For a **menu document**, confirm you want `create menu` and not a profile menu — `show navigation menu` vs `describe menu` tells them apart +- [ ] No menu item targets a page with required parameters (CE1571) diff --git a/CLAUDE.md b/CLAUDE.md index d1767a6ab..32a26adae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -668,6 +668,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - ALTER WORKFLOW (SET properties, INSERT/DROP/REPLACE activities, outcomes, paths, conditions, boundary events) - CALCULATED BY microflow syntax for calculated attributes - Image collections (SHOW/DESCRIBE/CREATE/DROP) +- Menu documents (CREATE OR MODIFY/DESCRIBE/DROP MENU): standalone `Menus$MenuDocument`, the reusable menu a menu widget points at (Atlas_Core's `Phone_Menu`/`Tablet_Menu`) — **not** the menu inside a navigation profile, though both are built from the same items, so the item syntax is shared with `CREATE NAVIGATION`'s `MENU (...)` block. DESCRIBE is round-trippable. Written through gen+codec, which is load-bearing: Studio Pro's menu documents carry typed-array marker **3** on the item collection and each item's sub-items (the codec default), while the navigation writers hand-build items with marker **1** — unverified whether that is a latent navigation bug or a real difference, so navigation is left alone. Authoring is modelsdk-only; legacy refuses. Two traps: a menu item cannot open a page with required parameters (**CE1571**), and only `Forms$IconCollectionIcon` round-trips (glyph/image icons are flagged by DESCRIBE, not dropped silently) - AI agent documents: Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) - OData contract browsing (SHOW/DESCRIBE CONTRACT ENTITIES/ACTIONS FROM cached $metadata) - AsyncAPI contract browsing (SHOW/DESCRIBE CONTRACT CHANNELS/MESSAGES FROM cached AsyncAPI) diff --git a/cmd/mxcli/cmd_describe.go b/cmd/mxcli/cmd_describe.go index bfeec9091..e8a4bcd14 100644 --- a/cmd/mxcli/cmd_describe.go +++ b/cmd/mxcli/cmd_describe.go @@ -47,6 +47,7 @@ Types: odataclient Describe a consumed OData service odataservice Describe a published OData service imagecollection Describe an image collection (also: "image collection") + menu Describe a standalone menu document businesseventservice Describe a business event service (also: "business event service") databaseconnection Describe a database connection (also: "database connection") agent Describe an AI agent (also: "agent") @@ -166,6 +167,8 @@ Example: mdlCmd = fmt.Sprintf("DESCRIBE ODATA SERVICE %s", name) case "IMAGECOLLECTION", "IMAGE COLLECTION": mdlCmd = fmt.Sprintf("DESCRIBE IMAGE COLLECTION %s", name) + case "MENU": + mdlCmd = fmt.Sprintf("DESCRIBE MENU %s", name) case "BUSINESSEVENTSERVICE", "BUSINESS EVENT SERVICE": mdlCmd = fmt.Sprintf("DESCRIBE BUSINESS EVENT SERVICE %s", name) case "DATABASECONNECTION", "DATABASE CONNECTION": @@ -190,7 +193,7 @@ Example: mdlCmd = "" // handled directly by format-specific path default: fmt.Fprintf(os.Stderr, "Unknown type: %s\n", strings.Join(args[:len(args)-1], " ")) - fmt.Fprintln(os.Stderr, "Valid types: module, entity, association, enumeration, constant, microflow, nanoflow, workflow, page, snippet, layout, javaaction, jsonstructure, importmapping, exportmapping, restclient, odataclient, odataservice, imagecollection, businesseventservice, databaseconnection, agent, aimodel, knowledgebase, consumedmcpservice, datatransformer, modulerole, userrole, projectsecurity, settings, demouser, navigation, systemoverview") + fmt.Fprintln(os.Stderr, "Valid types: module, entity, association, enumeration, constant, microflow, nanoflow, workflow, page, snippet, layout, javaaction, jsonstructure, importmapping, exportmapping, restclient, odataclient, odataservice, imagecollection, menu, businesseventservice, databaseconnection, agent, aimodel, knowledgebase, consumedmcpservice, datatransformer, modulerole, userrole, projectsecurity, settings, demouser, navigation, systemoverview") fmt.Fprintln(os.Stderr, "Multi-word types also accepted: json structure, import mapping, export mapping, rest client, image collection, business event service, agent, model, knowledge base, consumed mcp service, data transformer, etc.") os.Exit(1) } @@ -309,6 +312,7 @@ var objectTypeToDescribe = map[string]string{ "BUSINESS_EVENT_SERVICE": "businesseventservice", "DATABASE_CONNECTION": "databaseconnection", "IMAGE_COLLECTION": "imagecollection", + "MENU": "menu", "DATA_TRANSFORMER": "datatransformer", "AGENT": "agent", "AI_MODEL": "model", @@ -337,6 +341,7 @@ var unitTypeToDescribe = map[string]string{ "ImportMappings$ImportMapping": "importmapping", "ExportMappings$ExportMapping": "exportmapping", "Images$ImageCollection": "imagecollection", + "Menus$MenuDocument": "menu", "Workflows$Workflow": "workflow", } diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index d2e20cbc1..b283d454f 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -183,8 +183,8 @@ func init() { }) Register(SyntaxFeature{ - Path: "menu", - Summary: "Create, describe and drop standalone menu documents", + Path: "navigation.menu-document", + Summary: "Create, describe and drop standalone menu documents (Menus$MenuDocument)", Keywords: []string{ "create menu", "describe menu", "drop menu", "menu", "menus", "menu document", "menu item", @@ -213,7 +213,7 @@ func init() { "-- expressed in MDL; DESCRIBE flags those rather than dropping them silently.\n" + "-- * A page with required parameters cannot be opened from a menu item\n" + "-- without an argument — Mendix reports CE1571.", - SeeAlso: []string{"navigation", "page.show"}, + SeeAlso: []string{"navigation.create", "navigation.show", "page.show"}, }) // ── Fragment ────────────────────────────────────────────────────────── diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 0ea07706e..1171b22a7 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -256,6 +256,7 @@ - [Navigation Statements](reference/navigation/README.md) - [ALTER NAVIGATION](reference/navigation/alter-navigation.md) - [SHOW NAVIGATION](reference/navigation/show-navigation.md) + - [CREATE MENU](reference/navigation/menu.md) - [Workflow Statements](reference/workflow/README.md) - [CREATE WORKFLOW](reference/workflow/create-workflow.md) - [DROP WORKFLOW](reference/workflow/drop-workflow.md) diff --git a/docs-site/src/reference/navigation/README.md b/docs-site/src/reference/navigation/README.md index e1004b088..180a281be 100644 --- a/docs-site/src/reference/navigation/README.md +++ b/docs-site/src/reference/navigation/README.md @@ -10,6 +10,7 @@ Mendix applications can have multiple navigation profiles (Responsive, Tablet, P |-----------|-------------| | [ALTER NAVIGATION](alter-navigation.md) | Create or replace a navigation profile with home pages, login page, and menus | | [SHOW NAVIGATION](show-navigation.md) | Display navigation profiles, menus, and home page assignments | +| [CREATE MENU](menu.md) | Create, describe and drop standalone menu documents (`Menus$MenuDocument`) | ## Related Statements diff --git a/docs-site/src/reference/navigation/menu.md b/docs-site/src/reference/navigation/menu.md new file mode 100644 index 000000000..8e3fecec8 --- /dev/null +++ b/docs-site/src/reference/navigation/menu.md @@ -0,0 +1,131 @@ +# CREATE MENU + +## Synopsis + +```sql +CREATE [ OR MODIFY ] MENU module.name ( menu_item [ menu_item ... ] ) +DESCRIBE MENU module.name +DROP MENU module.name +``` + +Where each `menu_item` is one of: + +```sql +MENU ITEM 'caption' [ PAGE module.page | MICROFLOW module.microflow ] [ ICON module.collection.icon ] ; +MENU 'caption' [ ICON module.collection.icon ] ( nested_items ) ; +``` + +## Description + +Manages standalone **menu documents** (`Menus$MenuDocument`) — reusable menus that a +menu widget on a page points at. Atlas_Core ships two of them, `Phone_Menu` and +`Tablet_Menu`. + +A menu document is **not** the menu inside a navigation profile, although the two +are easy to confuse: both are built from the same menu items, which is why the item +syntax here is identical to the `MENU (...)` block of +[ALTER NAVIGATION](alter-navigation.md). They differ in where they live and how they +are read: + +| | Profile menu | Menu document | +|---|---|---| +| Lives in | a navigation profile | its own document | +| Used by | the app's navigation | a menu widget you place on a page | +| Read with | `SHOW NAVIGATION MENU` | `DESCRIBE MENU module.name` | +| Written with | `CREATE OR REPLACE NAVIGATION` | `CREATE OR MODIFY MENU` | + +`OR MODIFY` replaces the item list **wholesale**, exactly as `CREATE OR REPLACE +NAVIGATION` does: the list you give is the document's complete contents, so an +omitted item is a removed item. The document's identity, container and export level +are preserved, so menu widgets pointing at it keep working. + +`DESCRIBE MENU` emits a re-executable `CREATE OR MODIFY MENU` statement, so +describe → edit → exec is the normal editing loop, and describe → exec → describe is +a fixed point. + +## Parameters + +`module.name` +: Qualified name of the menu document. + +`'caption'` +: The item's label, in single quotes. + +`PAGE` / `MICROFLOW` +: Optional target opened when the item is clicked. An item with neither is inert + (stored as `Forms$NoAction`), which is normal for an item that only groups + sub-items. + +`ICON` +: Optional icon, given as a qualified name into an icon collection — not a string. + Hyphenated segments are double-quoted: `Atlas_Core.Atlas."layout-2"`. + +## Examples + +Create a menu with a nested sub-menu: + +```sql +CREATE MENU MyModule.Main_Menu ( + menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home; + menu item 'Run' microflow MyModule.DoThing; + menu 'Admin' ( + menu item 'Accounts' page Administration.Account_Overview; + ); + menu item 'Plain'; +); +``` + +Replace its contents (the two omitted items are removed): + +```sql +CREATE OR MODIFY MENU MyModule.Main_Menu ( + menu item 'Home' page MyModule.Home_Web icon Atlas_Core.Atlas.home; + menu item 'Run' microflow MyModule.DoThing; +); +``` + +Read one back — including Atlas's own: + +```sql +DESCRIBE MENU Atlas_Core.Phone_Menu; +``` + +The type is auto-detected too, so the bare form works: + +```sql +DESCRIBE Atlas_Core.Phone_Menu; +``` + +From the command line: + +```bash +mxcli describe menu Atlas_Core.Phone_Menu -p app.mpr +``` + +## Notes + +**A menu item cannot open a page that takes a required parameter.** There is nowhere +to supply the argument, and Mendix rejects the model with **CE1571** ("No argument +has been selected for parameter …") reported against `Menu item`. Point the item at a +parameterless page, or call a microflow that opens the page with its argument. + +**Only icon-collection icons round-trip.** `ICON` writes a +`Forms$IconCollectionIcon`. A glyph icon (which carries a numeric code) or an image +icon cannot be expressed in MDL, so `DESCRIBE` reports those on their own comment +line rather than dropping them silently: + +``` +-- icon a numeric glyph code (Forms$GlyphIcon) is not reproducible by CREATE MENU; +-- set it in Studio Pro +``` + +Re-running such output therefore loses that icon — visibly, not silently. + +**Authoring requires the default engine.** Under `MXCLI_ENGINE=legacy`, +create/modify/drop refuse rather than writing a differently-shaped document. Reading +(`DESCRIBE MENU`) works on both engines. + +## See Also + +- [ALTER NAVIGATION](alter-navigation.md) — the menu inside a navigation profile +- [SHOW NAVIGATION](show-navigation.md) — read profile menus and home pages From 4cedd3c34c80a27257de7e4adb87c94541eee2ef Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 12:59:50 +0000 Subject: [PATCH 06/35] =?UTF-8?q?Emit=20module=20roles=20from=20DESCRIBE?= =?UTF-8?q?=20MODULE,=20and=20correct=20=C2=A77's=20security=20claim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §7 recorded module security as invisible to a describe-based comparison and ranked it the largest remaining risk to the marketplace differ. Re-measuring shows that was wrong, and wrong in a way worth writing down. Three of the four parts of module security were already emitted: entity access rules by DESCRIBE ENTITY, page access by DESCRIBE PAGE, and microflow access by DESCRIBE MICROFLOW. Only the module's role list was missing — it lives in the module's own Security$ModuleSecurity unit and belongs to no document, so no document describe could ever reach it. DESCRIBE MODULE now emits it. The original error came from grepping describe output for "role|access| allowed", a pattern that cannot match the line it was looking for: grant view on page Administration.Account_Overview to Administration.Administrator; Nothing in that statement contains any of those words. The search returned nothing and the absence was read as a missing feature rather than a bad pattern. §7 now records that, since the same mistake is easy to repeat. Roles are sorted before emission: the reader's order is not guaranteed, and an unsorted list would show up as a phantom change in exactly the comparison this exists to support. A backend that cannot read module security degrades to the old output rather than failing the describe. With this and the two fixes before it, all of §7's Phase 1 prerequisites are closed except the page/template conflation, which does not mislead the differ. Phase 1 is unblocked. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../PROPOSAL_marketplace_module_upgrade.md | 39 ++++++--- mdl/executor/cmd_modules.go | 34 ++++++++ mdl/executor/cmd_modules_security_test.go | 85 +++++++++++++++++++ 3 files changed, 148 insertions(+), 10 deletions(-) create mode 100644 mdl/executor/cmd_modules_security_test.go diff --git a/docs/11-proposals/PROPOSAL_marketplace_module_upgrade.md b/docs/11-proposals/PROPOSAL_marketplace_module_upgrade.md index 661801e40..782d4dccb 100644 --- a/docs/11-proposals/PROPOSAL_marketplace_module_upgrade.md +++ b/docs/11-proposals/PROPOSAL_marketplace_module_upgrade.md @@ -381,11 +381,21 @@ The remaining 4 are two genuine defects, both out of scope for this proposal: reported type is wrong, re-executing the output would create a Page rather than a template, and `show modules` consequently reports Atlas_Web_Content as having 46 pages when it has zero. -- **Module security is invisible.** `describe module Administration` emits exactly - `create module Administration;`; the 8 `Security$ModuleSecurity` units are not - reachable. A marketplace update routinely changes module roles, so by this - proposal's own honesty rule module security must be reported **unknown** until this - is closed. This is the largest remaining blind spot. +- **Module roles were invisible** (closed 2026-08-11). This was originally recorded + here as "module security is invisible", which was wrong and briefly made the + security hole look like the largest risk in the proposal. Re-measured: three of the + four parts of module security were **already** in the describe surface — + entity access rules in `DESCRIBE ENTITY` (`grant on (...) where + ''`), page access in `DESCRIBE PAGE` (`grant view on page ...`), and + microflow access in `DESCRIBE MICROFLOW` (`grant execute on microflow ...`). Only + the module's **role list** was missing, because it lives in the module's own + `Security$ModuleSecurity` unit and belongs to no document. `DESCRIBE MODULE` now + emits it, sorted for stable comparison. + + The original error came from grepping describe output for `role|access|allowed` — + a pattern that cannot match `grant view on page P to Administration.Administrator;`. + Absence of evidence from a search is not evidence of absence: check the emitter, or + grep for the statement you expect to see rather than for words describing it. Folders need no separate coverage: folder membership is captured inside each document's describe (`Folder: 'Phone/PageTemplates/Form'`), so a document moved @@ -490,13 +500,22 @@ Phase 1 is the whole of this proposal; phase 2 is named only to show where it le documents in a seven-module marketplace project, so the design below is viable as written. Three prerequisites fall out of that measurement, in priority order: -1. **Report module security as unknown** (or close the gap). `describe module` does - not reach `Security$ModuleSecurity`, and marketplace updates change module roles. - This is the one gap that can make the tool *wrong* rather than incomplete. -2. **Fix import/export mapping describe** — 2 documents, currently erroring. +1. ~~**Report module security as unknown** (or close the gap).~~ **Closed.** The gap + was narrower than first recorded — only the module role list, not module security + as a whole; see the correction above. `DESCRIBE MODULE` now emits roles. +2. ~~**Fix import/export mapping describe.**~~ **Closed** — the cause was + `moduleNameFor` reading a unit's direct container instead of walking to the + enclosing module, so every foldered document missed. 3. **Distinguish page templates from pages**, or accept and document the conflation. + The only one still open, and the least severe: both sides of a comparison conflate + identically, so it does not mislead the differ. It does make `show modules` report + 46 pages for a module with none, which is worth fixing on its own terms. -The bare-DESCRIBE auto-detect gap that §7 found (43 documents) is already closed. +Also closed since: the bare-DESCRIBE auto-detect gap (43 documents), and menu +documents, which had no DESCRIBE at all and now have full CRUD. Bare-DESCRIBE +coverage over the fixture is 251/251. + +**Phase 1 is therefore unblocked.** | File | Change | |------|--------| diff --git a/mdl/executor/cmd_modules.go b/mdl/executor/cmd_modules.go index e4e102339..99cb3e454 100644 --- a/mdl/executor/cmd_modules.go +++ b/mdl/executor/cmd_modules.go @@ -15,6 +15,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/domainmodel" + "github.com/mendixlabs/mxcli/sdk/security" ) // execCreateModule handles CREATE MODULE statements. @@ -601,6 +602,12 @@ func describeModule(ctx *ExecContext, moduleName string, withAll bool) error { // Output basic CREATE MODULE statement fmt.Fprintf(ctx.Output, "create module %s;\n", targetModule.Name) + // Module roles live in the module's own Security$ModuleSecurity unit rather + // than in any document, so a sweep that describes every document in a module + // still never reports them. Emitting them here is what lets a describe-based + // comparison see a marketplace update that adds, removes or renames a role. + describeModuleRoles(ctx, targetModule) + if !withAll { fmt.Fprintln(ctx.Output, "/") return nil @@ -1203,3 +1210,30 @@ func jarDepIdxByCoord(deps []*types.JarDependency, coordinate string) int { } // Executor method wrappers for callers in unmigrated files. + +// describeModuleRoles emits the module's roles as re-executable statements. +// +// Failures are deliberately silent: DESCRIBE MODULE is useful without the roles, +// and a backend that cannot read module security (the MCP backend, for one) +// should not turn a working describe into an error. +func describeModuleRoles(ctx *ExecContext, mod *model.Module) { + ms, err := ctx.Backend.GetModuleSecurity(mod.ID) + if err != nil || ms == nil || len(ms.ModuleRoles) == 0 { + return + } + // Sort so the output is stable: a differ comparing two describes must not see + // a change because the reader returned roles in a different order. + roles := append([]*security.ModuleRole(nil), ms.ModuleRoles...) + sort.Slice(roles, func(i, j int) bool { return roles[i].Name < roles[j].Name }) + + for _, r := range roles { + // Same shape DESCRIBE MODULE ROLE emits, so the two agree. + fmt.Fprintf(ctx.Output, "create module role %s.%s", mod.Name, r.Name) + if r.Description != "" { + // Double embedded quotes; a description containing one would + // otherwise terminate the literal and produce unparseable output. + fmt.Fprintf(ctx.Output, " description '%s'", strings.ReplaceAll(r.Description, "'", "''")) + } + fmt.Fprintln(ctx.Output, ";") + } +} diff --git a/mdl/executor/cmd_modules_security_test.go b/mdl/executor/cmd_modules_security_test.go new file mode 100644 index 000000000..ac3c85f81 --- /dev/null +++ b/mdl/executor/cmd_modules_security_test.go @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/security" +) + +// TestDescribeModule_EmitsModuleRoles covers the one part of module security that +// no other describe reaches. +// +// Entity access rules already surface in DESCRIBE ENTITY as `grant ... on ...`, +// page access in DESCRIBE PAGE as `grant view on page ...`, and microflow access +// in DESCRIBE MICROFLOW as `grant execute on microflow ...`. Module *roles* live +// in the module's own Security$ModuleSecurity unit and belong to no document, so +// before this they were invisible to any describe-based comparison — a +// marketplace update that added or renamed a role would read as no change. +func TestDescribeModule_EmitsModuleRoles(t *testing.T) { + mod := mkModule("Administration") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleSecurityFunc: func(id model.ID) (*security.ModuleSecurity, error) { + // Deliberately out of alphabetical order: the emitter must sort, or a + // differ sees a phantom change when the reader's order shifts. + return &security.ModuleSecurity{ + ModuleRoles: []*security.ModuleRole{ + {Name: "User"}, + {Name: "Administrator", Description: "Full access"}, + }, + }, nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb)) + assertNoError(t, describeModule(ctx, "Administration", false)) + out := buf.String() + + assertContainsStr(t, out, "create module role Administration.Administrator description 'Full access';") + assertContainsStr(t, out, "create module role Administration.User;") + + if strings.Index(out, "Administration.Administrator") > strings.Index(out, "Administration.User") { + t.Errorf("module roles must be emitted in a stable sorted order, got:\n%s", out) + } +} + +// TestDescribeModule_RoleDescriptionQuoting guards the literal: an apostrophe in a +// description would otherwise close the string and make the output unparseable. +func TestDescribeModule_RoleDescriptionQuoting(t *testing.T) { + mod := mkModule("Sales") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + GetModuleSecurityFunc: func(id model.ID) (*security.ModuleSecurity, error) { + return &security.ModuleSecurity{ + ModuleRoles: []*security.ModuleRole{{Name: "Rep", Description: "it's theirs"}}, + }, nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb)) + assertNoError(t, describeModule(ctx, "Sales", false)) + assertContainsStr(t, buf.String(), "description 'it''s theirs'") +} + +// TestDescribeModule_SurvivesUnreadableSecurity keeps DESCRIBE MODULE working on a +// backend that cannot read module security (the MCP backend, for one) rather than +// turning a useful describe into an error. +func TestDescribeModule_SurvivesUnreadableSecurity(t *testing.T) { + mod := mkModule("Administration") + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + // GetModuleSecurityFunc left nil: the mock's default returns an error. + } + + ctx, buf := newMockCtx(t, withBackend(mb)) + assertNoError(t, describeModule(ctx, "Administration", false)) + assertContainsStr(t, buf.String(), "create module Administration;") +} From a3da553336793a4b58ff3c27b4ff722477dba5c2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 13:48:18 +0000 Subject: [PATCH 07/35] Make the menu doctype example self-contained and skip it on legacy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI has been red since afc857aa on `make test-integration`, a target I never ran locally — `make test`, `check-mdl` and `lint` were all green, which is why this shipped. TestMxCheck_DoctypeScripts executes every doctype-tests/*.mdl in a **fresh blank project** and validates the result with `mx check`. When menus became authorable the example changed from describe-only to CREATE statements, and it referenced documents a blank project does not have — Administration.* is a marketplace module, and MyFirstModule.MyFirstLogic exists only in the vendored fixture I had been testing against by hand. The example now creates its own module, pages and microflow, matching how every other doctype script is written, and points its menu items at those. It also runs under both engines, and menu authoring is modelsdk-only by design: the legacy backend refuses it rather than writing item lists with typed-array marker 1 where Studio Pro stores 3. That refusal cannot pass the gate, so the script is registered in engineScriptSkip for legacy with the reason — the same mechanism the SOAP and chart-template splits use. Verified by running the failing test rather than inferring: the menu script now passes on modelsdk with 0 errors from mx check, and skips on legacy. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../doctype-tests/26-menu-examples.mdl | 59 +++++++++++++++---- mdl/executor/roundtrip_doctype_test.go | 7 +++ 2 files changed, 53 insertions(+), 13 deletions(-) diff --git a/mdl-examples/doctype-tests/26-menu-examples.mdl b/mdl-examples/doctype-tests/26-menu-examples.mdl index ca55a49bf..32e839450 100644 --- a/mdl-examples/doctype-tests/26-menu-examples.mdl +++ b/mdl-examples/doctype-tests/26-menu-examples.mdl @@ -11,10 +11,42 @@ -- the item syntax below is identical to the MENU (...) block of -- CREATE NAVIGATION: -- --- create or modify menu Atlas_Core.Phone_Menu ( ... ); -- this file --- show navigation menu; -- profile menu +-- create or modify menu MenuTest.Main_Menu ( ... ); -- this file +-- show navigation menu; -- profile menu +-- +-- This script is self-contained: it creates the module, and the page and +-- microflow the menu points at, so it runs against a fresh project. -- ============================================================================ +create module MenuTest; + +-- Targets for the menu items below. The page deliberately takes no parameters: +-- a menu item has nowhere to supply an argument, so opening a parameterised +-- page from one is rejected by Mendix with CE1571 (see Gotchas). +create page MenuTest.Landing +( + title: 'Landing', + layout: Atlas_Core.Atlas_Default, + url: 'menutest_landing' +) +{ + DYNAMICTEXT welcome (Content: 'Welcome', RenderMode: H2) +} + +create page MenuTest.Reports +( + title: 'Reports', + layout: Atlas_Core.Atlas_Default, + url: 'menutest_reports' +) +{ + DYNAMICTEXT heading (Content: 'Reports', RenderMode: H2) +} + +create microflow MenuTest.Rebuild () +begin +end; + -- ---------------------------------------------------------------------------- -- Create -- ---------------------------------------------------------------------------- @@ -25,11 +57,11 @@ -- * sub-menu -> menu '' ( ...nested items... ) -- -- ICON names an entry in an icon collection. It is optional on every form. -create menu MyFirstModule.Main_Menu ( - menu item 'Home' page MyFirstModule.Home_Web icon Atlas_Core.Atlas_Filled.home; - menu item 'Run' microflow MyFirstModule.MyFirstLogic; +create menu MenuTest.Main_Menu ( + menu item 'Home' page MenuTest.Landing icon Atlas_Core.Atlas_Filled.home; + menu item 'Rebuild' microflow MenuTest.Rebuild; menu 'Admin' ( - menu item 'Accounts' page Administration.Account_Overview; + menu item 'Reports' page MenuTest.Reports; ); menu item 'Plain'; ); @@ -38,9 +70,9 @@ create menu MyFirstModule.Main_Menu ( -- the list given is the document's complete contents, so an omitted item is a -- removed item. The document's identity and its export level are preserved, so -- menu widgets pointing at it keep working. -create or modify menu MyFirstModule.Main_Menu ( - menu item 'Home' page MyFirstModule.Home_Web icon Atlas_Core.Atlas_Filled.home; - menu item 'Run' microflow MyFirstModule.MyFirstLogic; +create or modify menu MenuTest.Main_Menu ( + menu item 'Home' page MenuTest.Landing icon Atlas_Core.Atlas_Filled.home; + menu item 'Rebuild' microflow MenuTest.Rebuild; ); -- ---------------------------------------------------------------------------- @@ -48,15 +80,15 @@ create or modify menu MyFirstModule.Main_Menu ( -- ---------------------------------------------------------------------------- -- DESCRIBE emits a re-executable CREATE OR MODIFY statement, so -- describe -> exec -> describe is a fixed point. -describe menu MyFirstModule.Main_Menu; +describe menu MenuTest.Main_Menu; -- The type is auto-detected too, as for any other document type. -describe MyFirstModule.Main_Menu; +describe MenuTest.Main_Menu; -- ---------------------------------------------------------------------------- -- Drop -- ---------------------------------------------------------------------------- -drop menu MyFirstModule.Main_Menu; +drop menu MenuTest.Main_Menu; -- ---------------------------------------------------------------------------- -- Gotchas @@ -78,4 +110,5 @@ drop menu MyFirstModule.Main_Menu; -- -- 3. Authoring requires the default (modelsdk) engine. Under -- MXCLI_ENGINE=legacy, create/modify/drop refuse rather than write a --- differently-shaped document. +-- differently-shaped document — which is why this script is skipped for the +-- legacy engine in the doctype gate. diff --git a/mdl/executor/roundtrip_doctype_test.go b/mdl/executor/roundtrip_doctype_test.go index e613d8832..89f3ffbc8 100644 --- a/mdl/executor/roundtrip_doctype_test.go +++ b/mdl/executor/roundtrip_doctype_test.go @@ -50,6 +50,13 @@ var engineScriptSkip = map[string]string{ // The legacy widget builder has no `barchart` pluggable-widget template, so // page build fails ("template not found: barchart"). Passes on modelsdk. "legacy/34-chart-widget-examples.mdl": "legacy widget builder lacks the barchart template (works on modelsdk); tracked", + // Menu-document authoring is modelsdk-only *by design*, not a gap: Studio Pro + // stores a menu document's item lists with typed-array marker 3, which the + // codec emits by default, while the legacy navigation writer hand-builds menu + // items with marker 1. Rather than ship a second writer of unverified shape, + // the legacy backend refuses create/modify/drop — so the script cannot pass + // there and the refusal is the intended behaviour. + "legacy/26-menu-examples.mdl": "menu authoring is modelsdk-only by design; the legacy backend refuses it", // The legacy widget builder has no `linechart` template either, so the OL08 // LineChart object-list example (added in 6b837ad7) fails page build // ("template not found: linechart"). Passes on modelsdk. Same class as the From 3130ca7789c3e87bd2c5d64543f84fcb5008dfeb Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 14:05:06 +0000 Subject: [PATCH 08/35] Apply gofmt to ten files that were checked in unformatted `make lint` rewrites these on every run, so they show up as uncommitted changes in every working tree and had to be reverted out of each commit in this branch to keep the diffs to one thing each. Committing the formatting once stops that recurring. Mechanical only. The visible half is alignment and import ordering; the rest is gofmt splitting semicolon-packed one-liners in the enginecompare tests onto separate lines, e.g. lp := copyProject(t); if _,err:=Run(Legacy,lp,s);err!=nil{...} becomes the same statements one per line. No statement is added, removed or reordered. Verified with `gofmt -l` (clean afterwards), `go build ./...`, `go vet`, and the full unit suite. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- mdl/backend/modelsdk/constant.go | 2 +- .../modelsdk/widget_write_navlist_test.go | 2 +- mdl/catalog/builder_pages_test.go | 6 ++-- mdl/enginecompare/bsoncompare.go | 4 +-- mdl/enginecompare/write_gen_test.go | 26 +++++++++++++---- mdl/enginecompare/write_valid_test.go | 26 +++++++++++++---- mdl/exprcheck/unknown_funcs_test.go | 22 +++++++-------- mdl/types/unit_types.go | 28 +++++++++---------- modelsdk/widgets/augment_metadata_test.go | 2 +- modelsdk/widgets/dirty_template_test.go | 2 +- 10 files changed, 76 insertions(+), 44 deletions(-) diff --git a/mdl/backend/modelsdk/constant.go b/mdl/backend/modelsdk/constant.go index b2cc74358..aa11a877a 100644 --- a/mdl/backend/modelsdk/constant.go +++ b/mdl/backend/modelsdk/constant.go @@ -3,9 +3,9 @@ package modelsdkbackend import ( + "github.com/mendixlabs/mxcli/modelsdk/element" genConst "github.com/mendixlabs/mxcli/modelsdk/gen/constants" genDT "github.com/mendixlabs/mxcli/modelsdk/gen/datatypes" - "github.com/mendixlabs/mxcli/modelsdk/element" "github.com/mendixlabs/mxcli/modelsdk/mprread" "github.com/mendixlabs/mxcli/model" diff --git a/mdl/backend/modelsdk/widget_write_navlist_test.go b/mdl/backend/modelsdk/widget_write_navlist_test.go index d75e8acb2..c74769c0e 100644 --- a/mdl/backend/modelsdk/widget_write_navlist_test.go +++ b/mdl/backend/modelsdk/widget_write_navlist_test.go @@ -14,7 +14,7 @@ import ( // TestNavListItemToGen_WritesNames guards ledger finding #24: the modelsdk // writer must emit the navigation item's Name and give the caption's generated // DynamicText a name — otherwise Studio Pro rejects the project with CE7247 -// "name cannot be empty" / CE0495 "duplicate name ''". +// "name cannot be empty" / CE0495 "duplicate name ”". func TestNavListItemToGen_WritesNames(t *testing.T) { item := &pages.NavigationListItem{ Name: "itemTransactions", diff --git a/mdl/catalog/builder_pages_test.go b/mdl/catalog/builder_pages_test.go index 838c5063d..8e173ee60 100644 --- a/mdl/catalog/builder_pages_test.go +++ b/mdl/catalog/builder_pages_test.go @@ -245,9 +245,9 @@ func TestExtractWidgetsRecursive(t *testing.T) { t.Run("indexes a styled conditionalVisibilityWidget (not transparent)", func(t *testing.T) { // If a wrapper-named container carries styling it is a real user widget. w := map[string]any{ - "$ID": "div3", - "Name": "conditionalVisibilityWidget9", - "$Type": "Forms$DivContainer", + "$ID": "div3", + "Name": "conditionalVisibilityWidget9", + "$Type": "Forms$DivContainer", "Appearance": map[string]any{"Class": "card"}, } got := extractWidgetsRecursive(w) diff --git a/mdl/enginecompare/bsoncompare.go b/mdl/enginecompare/bsoncompare.go index 2cfdcb3ae..8de761e1a 100644 --- a/mdl/enginecompare/bsoncompare.go +++ b/mdl/enginecompare/bsoncompare.go @@ -10,13 +10,13 @@ import ( "go.mongodb.org/mongo-driver/v2/bson" "github.com/mendixlabs/mxcli/modelsdk/codec" - genDm "github.com/mendixlabs/mxcli/modelsdk/gen/domainmodels" genConst "github.com/mendixlabs/mxcli/modelsdk/gen/constants" _ "github.com/mendixlabs/mxcli/modelsdk/gen/datatypes" // register DataTypes$* for constant decode + genDm "github.com/mendixlabs/mxcli/modelsdk/gen/domainmodels" genEnum "github.com/mendixlabs/mxcli/modelsdk/gen/enumerations" genMf "github.com/mendixlabs/mxcli/modelsdk/gen/microflows" - "github.com/mendixlabs/mxcli/modelsdk/mprread" mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" + "github.com/mendixlabs/mxcli/modelsdk/mprread" ) // MicroflowCanonBSON returns the canonicalized raw BSON of a named microflow unit diff --git a/mdl/enginecompare/write_gen_test.go b/mdl/enginecompare/write_gen_test.go index 38d502165..8c5ae8344 100644 --- a/mdl/enginecompare/write_gen_test.go +++ b/mdl/enginecompare/write_gen_test.go @@ -1,10 +1,26 @@ package enginecompare + import "testing" + func TestWriteParity_Generalization(t *testing.T) { const s = "CREATE PERSISTENT ENTITY MyFirstModule.GenParent;CREATE PERSISTENT ENTITY MyFirstModule.GenChild EXTENDS MyFirstModule.GenParent;" - lp := copyProject(t); if _,err:=Run(Legacy,lp,s);err!=nil{t.Fatalf("legacy: %v",err)} - mp := copyProject(t); if _,err:=Run(ModelSDK,mp,s);err!=nil{t.Fatalf("modelsdk: %v",err)} - leg,err:=EntityCanonBSON(lp,"MyFirstModule","GenChild"); if err!=nil{t.Fatalf("leg: %v",err)} - msd,err:=EntityCanonBSON(mp,"MyFirstModule","GenChild"); if err!=nil{t.Fatalf("msd: %v",err)} - if leg!=msd { t.Errorf("Generalization divergence:\nlegacy: %s\nmodelsdk: %s", leg, msd) } + lp := copyProject(t) + if _, err := Run(Legacy, lp, s); err != nil { + t.Fatalf("legacy: %v", err) + } + mp := copyProject(t) + if _, err := Run(ModelSDK, mp, s); err != nil { + t.Fatalf("modelsdk: %v", err) + } + leg, err := EntityCanonBSON(lp, "MyFirstModule", "GenChild") + if err != nil { + t.Fatalf("leg: %v", err) + } + msd, err := EntityCanonBSON(mp, "MyFirstModule", "GenChild") + if err != nil { + t.Fatalf("msd: %v", err) + } + if leg != msd { + t.Errorf("Generalization divergence:\nlegacy: %s\nmodelsdk: %s", leg, msd) + } } diff --git a/mdl/enginecompare/write_valid_test.go b/mdl/enginecompare/write_valid_test.go index 15222bb6c..9a77b51ff 100644 --- a/mdl/enginecompare/write_valid_test.go +++ b/mdl/enginecompare/write_valid_test.go @@ -1,11 +1,27 @@ package enginecompare + import "testing" + func TestWriteParity_ValidationRules(t *testing.T) { const s = "CREATE PERSISTENT ENTITY MyFirstModule.ValTest " + "( Name: string(100) not null error 'Name is required', Code: string(20) unique error 'Code must be unique' )" - lp := copyProject(t); if _,e:=Run(Legacy,lp,s);e!=nil{t.Fatalf("legacy: %v",e)} - mp := copyProject(t); if _,e:=Run(ModelSDK,mp,s);e!=nil{t.Fatalf("modelsdk: %v",e)} - leg,e:=EntityCanonBSON(lp,"MyFirstModule","ValTest"); if e!=nil{t.Fatalf("leg: %v",e)} - msd,e:=EntityCanonBSON(mp,"MyFirstModule","ValTest"); if e!=nil{t.Fatalf("msd: %v",e)} - if leg!=msd { t.Errorf("ValidationRules divergence:\nlegacy: %s\nmodelsdk: %s", leg, msd) } + lp := copyProject(t) + if _, e := Run(Legacy, lp, s); e != nil { + t.Fatalf("legacy: %v", e) + } + mp := copyProject(t) + if _, e := Run(ModelSDK, mp, s); e != nil { + t.Fatalf("modelsdk: %v", e) + } + leg, e := EntityCanonBSON(lp, "MyFirstModule", "ValTest") + if e != nil { + t.Fatalf("leg: %v", e) + } + msd, e := EntityCanonBSON(mp, "MyFirstModule", "ValTest") + if e != nil { + t.Fatalf("msd: %v", e) + } + if leg != msd { + t.Errorf("ValidationRules divergence:\nlegacy: %s\nmodelsdk: %s", leg, msd) + } } diff --git a/mdl/exprcheck/unknown_funcs_test.go b/mdl/exprcheck/unknown_funcs_test.go index 3905801a7..be428c8f3 100644 --- a/mdl/exprcheck/unknown_funcs_test.go +++ b/mdl/exprcheck/unknown_funcs_test.go @@ -11,10 +11,10 @@ func TestUnknownFunctionCalls(t *testing.T) { wantSuggat string // expected suggestion (if any) }{ {"randomInt(9)", "randomInt", "random"}, - {"round(random() * 8)", "", ""}, // all known - {"toUpperCase($x)", "", ""}, // known - {"secondsBetween($a, $b)", "", ""}, // known - {"$a + length($s)", "", ""}, // known nested + {"round(random() * 8)", "", ""}, // all known + {"toUpperCase($x)", "", ""}, // known + {"secondsBetween($a, $b)", "", ""}, // known + {"$a + length($s)", "", ""}, // known nested {"if $a then floor($b) else ceil($c)", "", ""}, {"totallyMadeUpFn($x)", "totallyMadeUpFn", ""}, // no close match } @@ -41,13 +41,13 @@ func TestSourceRejectedForIntegerTarget(t *testing.T) { src string want bool }{ - {"$a div $b", true}, // arithmetic Decimal - {"secondsBetween($d1, $d2)", true}, // Decimal-returning func - {"random()", true}, // Decimal-returning func - {"round(random() * 8)", false}, // rounding → accepted - {"floor($a div $b)", false}, // rounding → accepted - {"$a + $b", false}, // Integer arithmetic - {"length($s)", false}, // Integer-returning func + {"$a div $b", true}, // arithmetic Decimal + {"secondsBetween($d1, $d2)", true}, // Decimal-returning func + {"random()", true}, // Decimal-returning func + {"round(random() * 8)", false}, // rounding → accepted + {"floor($a div $b)", false}, // rounding → accepted + {"$a + $b", false}, // Integer arithmetic + {"length($s)", false}, // Integer-returning func {"calendarMonthsBetween($d1, $d2)", false}, // Integer-returning func {"", false}, } diff --git a/mdl/types/unit_types.go b/mdl/types/unit_types.go index b41b7cfeb..d6f2feff5 100644 --- a/mdl/types/unit_types.go +++ b/mdl/types/unit_types.go @@ -8,35 +8,35 @@ package types // when switching on UnitInfo.Type or unitCache entries. const ( // Core project structure - UnitTypeModule = "Projects$ModuleImpl" - UnitTypeModuleSettings = "Projects$ModuleSettings" - UnitTypeFolder = "Projects$Folder" + UnitTypeModule = "Projects$ModuleImpl" + UnitTypeModuleSettings = "Projects$ModuleSettings" + UnitTypeFolder = "Projects$Folder" UnitTypeProjectSettings = "Settings$ProjectSettings" // Domain model - UnitTypeDomainModel = "DomainModels$DomainModel" - UnitTypeEnumeration = "Enumerations$Enumeration" - UnitTypeConstant = "Constants$Constant" + UnitTypeDomainModel = "DomainModels$DomainModel" + UnitTypeEnumeration = "Enumerations$Enumeration" + UnitTypeConstant = "Constants$Constant" // Flows - UnitTypeMicroflow = "Microflows$Microflow" - UnitTypeNanoflow = "Microflows$Nanoflow" - UnitTypeRule = "Microflows$Rule" + UnitTypeMicroflow = "Microflows$Microflow" + UnitTypeNanoflow = "Microflows$Nanoflow" + UnitTypeRule = "Microflows$Rule" // Pages / UI - UnitTypePage = "Forms$Page" - UnitTypeLayout = "Forms$Layout" - UnitTypeSnippet = "Forms$Snippet" + UnitTypePage = "Forms$Page" + UnitTypeLayout = "Forms$Layout" + UnitTypeSnippet = "Forms$Snippet" // Java / JavaScript actions UnitTypeJavaAction = "JavaActions$JavaAction" UnitTypeJavaScriptAction = "JavaActions$JavaScriptAction" // Workflows - UnitTypeWorkflow = "Workflows$Workflow" + UnitTypeWorkflow = "Workflows$Workflow" // REST / OData / Business events - UnitTypePublishedRestService = "Rest$PublishedRestService" + UnitTypePublishedRestService = "Rest$PublishedRestService" UnitTypePublishedODataService = "ODataPublish$PublishedODataService" UnitTypeConsumedODataService = "Rest$ConsumedODataService" UnitTypeBusinessEvent = "BusinessEvents$" // prefix — multiple sub-types diff --git a/modelsdk/widgets/augment_metadata_test.go b/modelsdk/widgets/augment_metadata_test.go index e18c12ada..5adc4b500 100644 --- a/modelsdk/widgets/augment_metadata_test.go +++ b/modelsdk/widgets/augment_metadata_test.go @@ -25,7 +25,7 @@ func TestReconcilePropertyMetadata(t *testing.T) { map[string]any{ "$Type": "CustomWidgets$WidgetPropertyType", "PropertyKey": "pagingPosition", - "Category": "General::Pagination", // stale + "Category": "General::Pagination", // stale "Caption": "Position of pagination", // stale "ValueType": map[string]any{ "$Type": "CustomWidgets$WidgetValueType", diff --git a/modelsdk/widgets/dirty_template_test.go b/modelsdk/widgets/dirty_template_test.go index 507c2842c..16fb696dd 100644 --- a/modelsdk/widgets/dirty_template_test.go +++ b/modelsdk/widgets/dirty_template_test.go @@ -168,7 +168,7 @@ func TestDirtyBindings_CleanIsClean(t *testing.T) { "Properties": []any{float64(2), map[string]any{ "$Type": "CustomWidgets$WidgetProperty", "Value": map[string]any{ - "$Type": "CustomWidgets$WidgetValue", + "$Type": "CustomWidgets$WidgetValue", "AttributeRef": nil, "DataSource": map[string]any{ // empty source slot — legitimate on data widgets "$Type": "CustomWidgets$CustomWidgetXPathSource", From 12a07fb2e5036ab491f10b008b498222de72556c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 14:39:48 +0000 Subject: [PATCH 09/35] Phase 1 slice 1: module snapshot + drift comparison engine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The part of `marketplace diff` that carries the design risk: capture what a module looks like, and decide whether two captures differ. Comparison is on DESCRIBE output rather than BSON, per the proposal's §3 measurement — an *untouched* module differs from its own published package in ~15,000 BSON paths, because the installed copy carries whole subtrees the package does not. DESCRIBE discards exactly those artefacts. Elements are enumerated from the catalog `objects` view and described through executor.DescribeKindFor, newly exported so this resolves types through the same table bare DESCRIBE uses. Three copies of that mapping already exist and each has drifted once; a fourth was not worth the convenience. The honesty rule is the load-bearing behaviour and has its own tests: an element that cannot be described on either side is Unknown, never Unchanged, and a report containing one is not Clean. "We could not tell" is deliberately a different answer from "nothing changed" — collapsing them would clear a module for a destructive upgrade on the strength of a gap in coverage. Verified against the vendored 7-module marketplace fixture, not just synthetic snapshots: - Administration snapshots 21 of 21 elements, all describable. - Two snapshots of the same module compare clean. This is the control the design rests on: if DESCRIBE output varied run to run, every result would be noise. Two separate copies are used, so a path dependency would fail it. - A real edit (`alter entity Administration.Account add attribute ...`) is reported as exactly one modified element, ENTITY Account, and nothing else. Both halves matter — missing it is unsafe, and false positives make the report ignorable. The backend factory is injected rather than chosen here: the engine is a global CLI concern, and comparing both sides with the same engine is what makes the result meaningful. Slices 2 (scratch project from a downloaded .mpk) and 3 (the CLI command) follow; the marketplace API is reachable from this environment, verified against DataWidgets 116540. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/marketplace/compare.go | 146 +++++++++++++++++ cmd/mxcli/marketplace/compare_test.go | 171 +++++++++++++++++++ cmd/mxcli/marketplace/snapshot.go | 218 +++++++++++++++++++++++++ cmd/mxcli/marketplace/snapshot_test.go | 159 ++++++++++++++++++ mdl/executor/describe_auto.go | 13 ++ 5 files changed, 707 insertions(+) create mode 100644 cmd/mxcli/marketplace/compare.go create mode 100644 cmd/mxcli/marketplace/compare_test.go create mode 100644 cmd/mxcli/marketplace/snapshot.go create mode 100644 cmd/mxcli/marketplace/snapshot_test.go diff --git a/cmd/mxcli/marketplace/compare.go b/cmd/mxcli/marketplace/compare.go new file mode 100644 index 000000000..e108a7e9a --- /dev/null +++ b/cmd/mxcli/marketplace/compare.go @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import "sort" + +// Verdict is what the comparison concluded about one element. +type Verdict string + +const ( + // Unchanged: both sides described identically. + Unchanged Verdict = "unchanged" + // Modified: both sides described, and the MDL differs. + Modified Verdict = "modified" + // OnlyInstalled: present in the project, absent from the package. + OnlyInstalled Verdict = "only-installed" + // OnlyPackage: present in the package, absent from the project. + OnlyPackage Verdict = "only-package" + // Unknown: at least one side could not be described, so nothing can be + // concluded. Never collapsed into Unchanged — an un-describable element + // reported as clean is the failure mode that would make this dangerous + // rather than merely incomplete. + Unknown Verdict = "unknown" +) + +// Finding is the comparison's conclusion for one element. +type Finding struct { + Key ElementKey + Verdict Verdict + // Reason explains an Unknown verdict; empty otherwise. + Reason string + // InstalledMDL / PackageMDL are the two descriptions, carried so a caller can + // show the actual difference. Empty where the side is absent or unreadable. + InstalledMDL string + PackageMDL string +} + +// Report is the full comparison of an installed module against a package. +type Report struct { + Module string + Findings []Finding +} + +// Counts tallies the report by verdict. +func (r *Report) Counts() map[Verdict]int { + out := make(map[Verdict]int, 5) + for _, f := range r.Findings { + out[f.Verdict]++ + } + return out +} + +// LocallyModified reports whether anything was changed in the project relative +// to the package. Unknown does not count as modified — but it does not count as +// clean either, which is why Clean is a separate question. +func (r *Report) LocallyModified() bool { + for _, f := range r.Findings { + if f.Verdict == Modified || f.Verdict == OnlyInstalled { + return true + } + } + return false +} + +// Clean reports whether the module can be said to be untouched. It requires +// every element to be positively verified as unchanged: a single Unknown makes +// the answer "we cannot tell", which is deliberately not the same as yes. +func (r *Report) Clean() bool { + for _, f := range r.Findings { + if f.Verdict != Unchanged { + return false + } + } + return true +} + +// Compare matches two snapshots of the same module by name+type and classifies +// each element. +// +// installed is the copy in the user's project; pkg is the copy built from the +// published marketplace package. The asymmetry matters for reporting: an element +// only in the project is something the user added, while one only in the package +// is something they deleted (or the version differs). +func Compare(installed, pkg *Snapshot) *Report { + rep := &Report{Module: installed.Module} + + seen := make(map[ElementKey]bool, len(installed.Elements)+len(pkg.Elements)) + keys := make([]ElementKey, 0, len(seen)) + for _, k := range installed.Keys() { + seen[k] = true + keys = append(keys, k) + } + for _, k := range pkg.Keys() { + if !seen[k] { + keys = append(keys, k) + } + } + sort.Slice(keys, func(i, j int) bool { + if keys[i].Type != keys[j].Type { + return keys[i].Type < keys[j].Type + } + return keys[i].Name < keys[j].Name + }) + + for _, k := range keys { + rep.Findings = append(rep.Findings, classify(k, installed.Elements[k], pkg.Elements[k], + hasKey(installed, k), hasKey(pkg, k))) + } + return rep +} + +func hasKey(s *Snapshot, k ElementKey) bool { + _, ok := s.Elements[k] + return ok +} + +func classify(k ElementKey, inst, pkg Element, hasInst, hasPkg bool) Finding { + switch { + case hasInst && !hasPkg: + // An element the user added is a local modification even though there is + // nothing to compare it against, so it is not Unknown. + return Finding{Key: k, Verdict: OnlyInstalled, InstalledMDL: inst.MDL} + case !hasInst && hasPkg: + return Finding{Key: k, Verdict: OnlyPackage, PackageMDL: pkg.MDL} + } + + // Present on both sides: it can only be compared if both described. + if !inst.Describable() || !pkg.Describable() { + return Finding{Key: k, Verdict: Unknown, Reason: unknownReason(inst, pkg)} + } + if inst.MDL == pkg.MDL { + return Finding{Key: k, Verdict: Unchanged} + } + return Finding{Key: k, Verdict: Modified, InstalledMDL: inst.MDL, PackageMDL: pkg.MDL} +} + +func unknownReason(inst, pkg Element) string { + switch { + case !inst.Describable() && !pkg.Describable(): + return "not describable on either side: " + inst.Err + case !inst.Describable(): + return "not describable in the project: " + inst.Err + default: + return "not describable in the package: " + pkg.Err + } +} diff --git a/cmd/mxcli/marketplace/compare_test.go b/cmd/mxcli/marketplace/compare_test.go new file mode 100644 index 000000000..bcf7833ed --- /dev/null +++ b/cmd/mxcli/marketplace/compare_test.go @@ -0,0 +1,171 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import "testing" + +func el(t, n, mdl string) Element { + return Element{Key: ElementKey{Type: t, Name: n}, MDL: mdl} +} + +func errEl(t, n, why string) Element { + return Element{Key: ElementKey{Type: t, Name: n}, Err: why} +} + +func snap(module string, els ...Element) *Snapshot { + s := &Snapshot{Module: module, Elements: map[ElementKey]Element{}} + for _, e := range els { + s.Elements[e.Key] = e + } + return s +} + +func findingFor(t *testing.T, r *Report, typ, name string) Finding { + t.Helper() + for _, f := range r.Findings { + if f.Key.Type == typ && f.Key.Name == name { + return f + } + } + t.Fatalf("no finding for %s %s in %+v", typ, name, r.Findings) + return Finding{} +} + +func TestCompare_Verdicts(t *testing.T) { + installed := snap("Administration", + el("PAGE", "Account_Overview", "create page A;"), + el("PAGE", "Account_Edit", "create page EDITED;"), + el("MICROFLOW", "LocalOnly", "create microflow L;"), + ) + pkg := snap("Administration", + el("PAGE", "Account_Overview", "create page A;"), + el("PAGE", "Account_Edit", "create page ORIGINAL;"), + el("MICROFLOW", "PackageOnly", "create microflow P;"), + ) + + rep := Compare(installed, pkg) + + if got := findingFor(t, rep, "PAGE", "Account_Overview").Verdict; got != Unchanged { + t.Errorf("identical MDL should be unchanged, got %s", got) + } + if got := findingFor(t, rep, "PAGE", "Account_Edit").Verdict; got != Modified { + t.Errorf("differing MDL should be modified, got %s", got) + } + if got := findingFor(t, rep, "MICROFLOW", "LocalOnly").Verdict; got != OnlyInstalled { + t.Errorf("element added by the user should be only-installed, got %s", got) + } + if got := findingFor(t, rep, "MICROFLOW", "PackageOnly").Verdict; got != OnlyPackage { + t.Errorf("element missing from the project should be only-package, got %s", got) + } + + if !rep.LocallyModified() { + t.Error("a modified element must make the module count as locally modified") + } + if rep.Clean() { + t.Error("a module with modifications must not report clean") + } +} + +// TestCompare_UnknownIsNeverClean is the honesty rule, and the single most +// important behaviour here: an element that cannot be described must not be +// silently treated as unchanged. Getting this wrong would tell a user their +// module is untouched when the tool simply could not look. +func TestCompare_UnknownIsNeverClean(t *testing.T) { + installed := snap("M", + el("PAGE", "Same", "x"), + errEl("MENU", "Opaque", "no DESCRIBE support for MENU"), + ) + pkg := snap("M", + el("PAGE", "Same", "x"), + errEl("MENU", "Opaque", "no DESCRIBE support for MENU"), + ) + + rep := Compare(installed, pkg) + + f := findingFor(t, rep, "MENU", "Opaque") + if f.Verdict != Unknown { + t.Fatalf("an un-describable element must be unknown, got %s", f.Verdict) + } + if f.Reason == "" { + t.Error("an unknown verdict must say why, or the user cannot judge the risk") + } + if rep.Clean() { + t.Error("a module containing an unknown element must not report clean — " + + "'we could not tell' is not 'nothing changed'") + } + // It is not evidence of modification either. + if rep.LocallyModified() { + t.Error("unknown must not be reported as a local modification either") + } +} + +// TestCompare_UnknownWhenOneSideFails covers the asymmetric case: describable +// in the project, not in the package (or vice versa). Comparing a real +// description against nothing would report a spurious modification. +func TestCompare_UnknownWhenOneSideFails(t *testing.T) { + installed := snap("M", el("PAGE", "P", "create page P;")) + pkg := snap("M", errEl("PAGE", "P", "boom")) + + f := findingFor(t, Compare(installed, pkg), "PAGE", "P") + if f.Verdict != Unknown { + t.Errorf("one unreadable side must yield unknown, not a spurious diff; got %s", f.Verdict) + } + if f.Reason == "" { + t.Error("expected a reason naming which side failed") + } +} + +// TestCompare_IdenticalSnapshotsAreClean is the control: a module compared with +// itself must report no drift at all. If this ever fails, the normaliser or the +// describe path is non-deterministic and every other result is noise. +func TestCompare_IdenticalSnapshotsAreClean(t *testing.T) { + s := snap("M", + el("PAGE", "A", "create page A;"), + el("MICROFLOW", "B", "create microflow B;"), + el("ENTITY", "C", "create entity C;"), + ) + + rep := Compare(s, s) + if !rep.Clean() { + t.Fatalf("a module compared with itself must be clean, got %+v", rep.Findings) + } + if rep.LocallyModified() { + t.Error("a module compared with itself must not report modifications") + } + if n := rep.Counts()[Unchanged]; n != 3 { + t.Errorf("expected 3 unchanged, got %d", n) + } +} + +// TestCompare_SameNameDifferentTypesDoNotCollide guards the join key. A module +// may hold a page and a microflow of the same name; keying on name alone would +// compare one against the other and report both as modified. +func TestCompare_SameNameDifferentTypesDoNotCollide(t *testing.T) { + installed := snap("M", + el("PAGE", "Overview", "page body"), + el("MICROFLOW", "Overview", "microflow body"), + ) + pkg := snap("M", + el("PAGE", "Overview", "page body"), + el("MICROFLOW", "Overview", "microflow body"), + ) + + rep := Compare(installed, pkg) + if !rep.Clean() { + t.Errorf("same-named elements of different types must not collide, got %+v", rep.Findings) + } +} + +func TestNormalizeMDL_IgnoresOnlyIncidentalFormatting(t *testing.T) { + a := normalizeMDL("create page P;\n\n \n title: 'x'; \n") + b := normalizeMDL("create page P;\n title: 'x';\n\n") + if a != b { + t.Errorf("blank lines and trailing spaces must not read as a difference:\n%q\nvs\n%q", a, b) + } + + // Indentation is meaningful in describe output (it shows nesting), so it must + // survive normalisation. + if normalizeMDL(" nested") == normalizeMDL("nested") { + t.Error("leading indentation must be preserved — it encodes widget nesting") + } +} diff --git a/cmd/mxcli/marketplace/snapshot.go b/cmd/mxcli/marketplace/snapshot.go new file mode 100644 index 000000000..f58025994 --- /dev/null +++ b/cmd/mxcli/marketplace/snapshot.go @@ -0,0 +1,218 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package marketplace implements drift detection for installed marketplace +// modules: capture what a module looks like now, capture what the published +// package looks like, and report which elements the user has changed. +// +// The comparison is on DESCRIBE output, not on BSON. A path-level BSON diff of +// an *untouched* module against its own published package differs in ~15,000 +// paths (PROPOSAL_marketplace_module_upgrade.md §3), because the installed copy +// carries whole subtrees the package does not. DESCRIBE discards exactly those +// artefacts — $IDs, storage envelopes, widget-internal representation — so two +// elements that describe alike are the same element as far as an author is +// concerned. +package marketplace + +import ( + "bytes" + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/executor" +) + +// ElementKey identifies an element within a module. Name+type is a sound join +// key: measured against a blank project's Administration module and its own +// published .mpk, every one of 27 elements matched with zero orphans on either +// side (§3). +type ElementKey struct { + // Type is the catalog `objects` view ObjectType, e.g. PAGE or MICROFLOW. + Type string + // Name is the element name, unqualified — the module name is not part of the + // key because the two sides are the same module by construction. + Name string +} + +func (k ElementKey) String() string { return k.Type + " " + k.Name } + +// Element is one described element of a module. +type Element struct { + Key ElementKey + // MDL is the DESCRIBE output, normalised for comparison. Empty when Err is set. + MDL string + // Err records why the element could not be described. An element with Err set + // is reported as unknown and never as unchanged — treating an un-describable + // element as clean is the one failure mode that would make this dangerous + // rather than merely incomplete. + Err string +} + +// Describable reports whether the element could be read at all. +func (e Element) Describable() bool { return e.Err == "" } + +// Snapshot is the describable content of one module at one point in time. +type Snapshot struct { + Module string + Elements map[ElementKey]Element +} + +// Keys returns the snapshot's element keys in a stable order. +func (s *Snapshot) Keys() []ElementKey { + keys := make([]ElementKey, 0, len(s.Elements)) + for k := range s.Elements { + keys = append(keys, k) + } + sort.Slice(keys, func(i, j int) bool { + if keys[i].Type != keys[j].Type { + return keys[i].Type < keys[j].Type + } + return keys[i].Name < keys[j].Name + }) + return keys +} + +// SnapshotModule connects to a project and captures DESCRIBE output for every +// element of one module. +// +// Elements are enumerated from the catalog's `objects` view, which indexes every +// describable top-level document type. That is deliberately the same source the +// DESCRIBE auto-detect uses, so the enumeration and the describe agree on what +// exists; an element the catalog does not index is invisible here, which is the +// coverage bound recorded in §7 of the proposal. +// newBackend supplies the engine to read with. It is injected rather than chosen +// here because the engine is a global CLI concern (--engine / MXCLI_ENGINE) that +// this package has no business deciding — and because comparing the two sides +// with the *same* engine is what makes the result meaningful. +func SnapshotModule(mprPath, moduleName string, newBackend func() backend.FullBackend) (*Snapshot, error) { + var sink bytes.Buffer + exec := executor.New(&sink) + defer exec.Close() + if newBackend != nil { + exec.SetBackendFactory(newBackend) + } + + if err := exec.Execute(&ast.ConnectStmt{Path: mprPath}); err != nil { + return nil, fmt.Errorf("connect %s: %w", mprPath, err) + } + + // The catalog is built lazily by the statements that need it, and the + // enumeration below reads it directly rather than through a statement, so + // build it explicitly. Fast mode is enough: the `objects` index it populates + // is exactly what this needs, and full mode additionally parses every + // activity and widget for cross-references nothing here uses. + if err := exec.Execute(&ast.RefreshCatalogStmt{}); err != nil { + return nil, fmt.Errorf("build catalog for %s: %w", mprPath, err) + } + + rows, err := queryModuleObjects(exec, moduleName) + if err != nil { + return nil, err + } + + snap := &Snapshot{Module: moduleName, Elements: make(map[ElementKey]Element, len(rows))} + for _, row := range rows { + key := ElementKey{Type: row.objectType, Name: row.name} + // A module can legitimately hold two elements of different types with the + // same name; the type is part of the key so they do not collide. Two of the + // same type and name cannot exist, so a repeat means the enumeration + // returned a duplicate — keep the first and move on rather than pick + // arbitrarily. + if _, seen := snap.Elements[key]; seen { + continue + } + snap.Elements[key] = describeElement(exec, &sink, row) + } + return snap, nil +} + +// moduleObject is one row of the enumeration. +type moduleObject struct { + objectType string + name string + qualifiedName string +} + +func queryModuleObjects(exec *executor.Executor, moduleName string) ([]moduleObject, error) { + cat := exec.Catalog() + if cat == nil { + return nil, fmt.Errorf("no catalog available for %s", moduleName) + } + + q := "SELECT ObjectType, Name, QualifiedName FROM objects WHERE ModuleName = '" + + strings.ReplaceAll(moduleName, "'", "''") + "' ORDER BY ObjectType, Name" + res, err := cat.Query(q) + if err != nil { + return nil, fmt.Errorf("enumerate module %s: %w", moduleName, err) + } + + out := make([]moduleObject, 0, len(res.Rows)) + for _, row := range res.Rows { + if len(row) < 3 { + continue + } + out = append(out, moduleObject{ + objectType: fmt.Sprintf("%v", row[0]), + name: fmt.Sprintf("%v", row[1]), + qualifiedName: fmt.Sprintf("%v", row[2]), + }) + } + return out, nil +} + +// describeElement runs one DESCRIBE and captures its output. +func describeElement(exec *executor.Executor, sink *bytes.Buffer, obj moduleObject) Element { + key := ElementKey{Type: obj.objectType, Name: obj.name} + + kind, ok := executor.DescribeKindFor(obj.objectType) + if !ok { + return Element{Key: key, Err: "no DESCRIBE support for " + obj.objectType} + } + + qn, err := splitQualified(obj.qualifiedName) + if err != nil { + return Element{Key: key, Err: err.Error()} + } + + sink.Reset() + if err := exec.Execute(&ast.DescribeStmt{ObjectType: kind, Name: qn}); err != nil { + return Element{Key: key, Err: err.Error()} + } + out := normalizeMDL(sink.String()) + if out == "" { + return Element{Key: key, Err: "DESCRIBE produced no output"} + } + return Element{Key: key, MDL: out} +} + +// splitQualified splits Module.Name. Some catalog rows carry a dotted service +// prefix (Service.Action), so the split is on the first separator and the +// remainder is the name. +func splitQualified(qualified string) (ast.QualifiedName, error) { + i := strings.Index(qualified, ".") + if i <= 0 || i == len(qualified)-1 { + return ast.QualifiedName{}, fmt.Errorf("not a qualified name: %q", qualified) + } + return ast.QualifiedName{Module: qualified[:i], Name: qualified[i+1:]}, nil +} + +// normalizeMDL makes DESCRIBE output comparable across two projects. +// +// Only incidental formatting is removed. Nothing that could carry a real edit is +// touched: a normaliser that stripped, say, property values would report a +// modified element as clean, which is the failure this whole design exists to +// avoid. +func normalizeMDL(s string) string { + lines := strings.Split(s, "\n") + kept := make([]string, 0, len(lines)) + for _, ln := range lines { + ln = strings.TrimRight(ln, " \t\r") + if strings.TrimSpace(ln) == "" { + continue + } + kept = append(kept, ln) + } + return strings.Join(kept, "\n") +} diff --git a/cmd/mxcli/marketplace/snapshot_test.go b/cmd/mxcli/marketplace/snapshot_test.go new file mode 100644 index 000000000..4b2639dc1 --- /dev/null +++ b/cmd/mxcli/marketplace/snapshot_test.go @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "bytes" + "os" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// testBackend is the default engine, the one users get. +func testBackend() backend.FullBackend { return modelsdkbackend.New() } + +const fixtureDir = "../../../testdata/expr-checker" + +// copyFixture makes a throwaway copy of the vendored project. Snapshotting builds +// a catalog beside the .mpr, and the mutation test writes to the model, so +// neither may touch the checked-in fixture. +func copyFixture(t *testing.T) string { + t.Helper() + dst := t.TempDir() + if err := os.CopyFS(dst, os.DirFS(fixtureDir)); err != nil { + t.Fatalf("copy fixture: %v", err) + } + return filepath.Join(dst, "minimal.mpr") +} + +// execMDL applies MDL text to a project through the executor — the same path a +// user's edit takes. Written as MDL rather than hand-built AST so the test keeps +// exercising the real parser and does not silently rot when AST fields change. +func execMDL(t *testing.T, mprPath, mdl string) { + t.Helper() + prog, errs := visitor.Build(mdl) + if len(errs) > 0 { + t.Fatalf("parse %q: %v", mdl, errs) + } + + var sink bytes.Buffer + exec := executor.New(&sink) + exec.SetBackendFactory(testBackend) + defer exec.Close() + if err := exec.Execute(&ast.ConnectStmt{Path: mprPath}); err != nil { + t.Fatalf("connect: %v", err) + } + for _, s := range prog.Statements { + if err := exec.Execute(s); err != nil { + t.Fatalf("execute: %v\noutput: %s", err, sink.String()) + } + } +} + +// TestSnapshotModule_ReadsRealModule checks the enumeration and describe capture +// actually work against a real project before anything is compared. +func TestSnapshotModule_ReadsRealModule(t *testing.T) { + snap, err := SnapshotModule(copyFixture(t), "Administration", testBackend) + if err != nil { + t.Fatalf("SnapshotModule: %v", err) + } + if len(snap.Elements) == 0 { + t.Fatal("no elements captured for Administration") + } + + // Spot-check that content was captured, not just keys: an empty MDL body + // would satisfy a count assertion while comparing everything as equal. + var described int + for _, e := range snap.Elements { + if e.Describable() && len(e.MDL) > 0 { + described++ + } + } + if described == 0 { + t.Fatalf("every element failed to describe: %+v", snap.Elements) + } + t.Logf("Administration: %d elements, %d described", len(snap.Elements), described) +} + +// TestSnapshotModule_IsDeterministic is the control the whole design rests on. +// +// Two snapshots of the same unmodified module must be identical. If DESCRIBE +// output varies between runs — a map iterated without sorting, a fresh $ID +// leaking into the text — then every comparison is noise and no result from this +// tool can be believed. Snapshotting two *separate copies* rather than the same +// file twice also catches anything that depends on the project's path. +func TestSnapshotModule_IsDeterministic(t *testing.T) { + a, err := SnapshotModule(copyFixture(t), "Administration", testBackend) + if err != nil { + t.Fatalf("snapshot A: %v", err) + } + b, err := SnapshotModule(copyFixture(t), "Administration", testBackend) + if err != nil { + t.Fatalf("snapshot B: %v", err) + } + + rep := Compare(a, b) + if !rep.Clean() { + for _, f := range rep.Findings { + if f.Verdict != Unchanged { + t.Errorf("%s: %s (%s)", f.Key, f.Verdict, f.Reason) + if f.Verdict == Modified { + t.Errorf(" A: %s", f.InstalledMDL) + t.Errorf(" B: %s", f.PackageMDL) + } + } + } + t.Fatal("two snapshots of the same module must compare clean") + } +} + +// TestSnapshotModule_DetectsARealEdit is the money test: the differ must notice a +// genuine local modification, and must not report anything else as changed. +// +// Both halves matter. Missing the edit makes the tool unsafe — it would clear a +// module for a destructive upgrade. Reporting unrelated elements as modified +// makes it useless, because a report full of false positives gets ignored. +func TestSnapshotModule_DetectsARealEdit(t *testing.T) { + pristine := copyFixture(t) + edited := copyFixture(t) + + // A minimal, unambiguous edit to one element of the module — the kind of local + // change a user makes to a marketplace module and then forgets about. + execMDL(t, edited, "alter entity Administration.Account add attribute LocalNote: String(100);") + + before, err := SnapshotModule(pristine, "Administration", testBackend) + if err != nil { + t.Fatalf("snapshot pristine: %v", err) + } + after, err := SnapshotModule(edited, "Administration", testBackend) + if err != nil { + t.Fatalf("snapshot edited: %v", err) + } + + rep := Compare(after, before) + if rep.Clean() { + t.Fatal("an added attribute must show up as a local modification") + } + if !rep.LocallyModified() { + t.Error("LocallyModified must be true after a real edit") + } + + var modified []string + for _, f := range rep.Findings { + if f.Verdict == Modified || f.Verdict == OnlyInstalled { + modified = append(modified, f.Key.String()) + } + } + if len(modified) != 1 { + t.Fatalf("exactly one element should differ, got %d: %v", len(modified), modified) + } + if modified[0] != "ENTITY Account" { + t.Errorf("the changed element should be ENTITY Account, got %s", modified[0]) + } +} diff --git a/mdl/executor/describe_auto.go b/mdl/executor/describe_auto.go index cef567b79..a9870cc6f 100644 --- a/mdl/executor/describe_auto.go +++ b/mdl/executor/describe_auto.go @@ -51,6 +51,19 @@ var objectTypeToDescribeKind = map[string]ast.DescribeObjectType{ "CONSUMED_MCP_SERVICE": ast.DescribeConsumedMCPService, } +// DescribeKindFor maps a catalog `objects` view ObjectType to the DESCRIBE kind +// that renders it, reporting false for a type with no describe handler. +// +// Exported so callers outside the executor (the marketplace differ, which walks +// a whole module) resolve types through the same table bare DESCRIBE uses, +// rather than keeping a fourth copy of this mapping. Three copies already exist +// — this map, the catalog's objects view, and cmd/mxcli's own dispatch — and +// they have drifted apart once each. +func DescribeKindFor(objectType string) (ast.DescribeObjectType, bool) { + kind, ok := objectTypeToDescribeKind[objectType] + return kind, ok +} + // resolveDescribeAuto auto-detects the type of a bare `DESCRIBE Module.Name` // against the connected project. It builds (fast mode) the catalog if needed and // looks the qualified name up in the `objects` index — a complete index for every From 031e63238fdece3c0771f5ae407053c28957da0c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 14:56:10 +0000 Subject: [PATCH 10/35] Phase 1 slice 2: build a reference project from a marketplace package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the version a module was installed from, creates a project at the consuming project's Mendix version, and imports the published .mpk into it — the baseline slice 1 compares against. Three things were found by running it rather than reasoning about it, and each changes the code: **A blank Mendix project is not empty.** The create-project template already ships Administration, Atlas_Core and friends, so `mx module-import` refuses with "Module 'Administration' already exist in the app" (exit 47) — for precisely the module the field report cares about. The template's copy is now removed first, through mxcli's own DROP MODULE rather than a file edit, because that also unpicks the references the template set up ("Removed Administration.User from 1 user role(s)"); leaving those dangling would hand module-import an inconsistent model. **ResolveMxForVersion silently falls back to any cached mxbuild.** Asking for 11.6.6 on a machine holding 11.12.1 returns 11.12.1 without complaint, and create-project stamps the project with whatever binary ran it. The reference would be built Mendix versions away from the project under comparison, and every platform migration between them would read as a user edit — false findings indistinguishable from real ones. The stamped version is now read back from the created project and a mismatch is refused, not warned about, with the setup command in the message. **mx create-project dies with PathTooLongException under a deep path.** It extracts its template with .NET path handling; t.TempDir() nested in a long module path breaks it where /tmp/xxxx works. Documented on the parameter, and the tests use a short dir. Also recorded: a .mpk's package.xml carries no version, so `mx module-import` stamps the module's *internal* version, not the marketplace release — importing Administration 4.3.2 records 2.0.1. That stamp never reaches DESCRIBE output so the comparison is unaffected, but it means the reference project's recorded version is not evidence of which package was imported. Verified end to end against real marketplace content (Administration 4.3.2, content 23513, at Mendix 11.12.1): 21 elements captured from the package, two independently built references compare clean — so the baseline is reproducible and a diff against it carries no noise — and a single added attribute is reported as exactly `ENTITY Account` modified. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/marketplace/scratch.go | 244 ++++++++++++++++++ .../marketplace/scratch_integration_test.go | 201 +++++++++++++++ 2 files changed, 445 insertions(+) create mode 100644 cmd/mxcli/marketplace/scratch.go create mode 100644 cmd/mxcli/marketplace/scratch_integration_test.go diff --git a/cmd/mxcli/marketplace/scratch.go b/cmd/mxcli/marketplace/scratch.go new file mode 100644 index 000000000..ed3302192 --- /dev/null +++ b/cmd/mxcli/marketplace/scratch.go @@ -0,0 +1,244 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "archive/zip" + "bytes" + "context" + "encoding/xml" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/executor" +) + +// InstalledModule reports what the project records about a marketplace module: +// the version it was installed from, and the project's own Mendix version. +// +// The recorded AppStoreVersion is what makes "have I changed this?" answerable +// at all — it names the published package the installed copy started life as. +// A module with no recorded version was not installed from the marketplace (or +// was imported by hand), and cannot be compared against anything. +func InstalledModule(mprPath, moduleName string) (appStoreVersion, mendixVersion string, err error) { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return "", "", fmt.Errorf("open %s: %w", mprPath, err) + } + defer reader.Close() + + mendixVersion, _ = reader.GetMendixVersion() + + mods, err := reader.ListModules() + if err != nil { + return "", "", fmt.Errorf("list modules: %w", err) + } + for _, m := range mods { + if !strings.EqualFold(m.Name, moduleName) { + continue + } + if m.AppStoreVersion == "" { + return "", mendixVersion, fmt.Errorf( + "module %q records no marketplace version, so there is no published package to compare it against", + m.Name) + } + return m.AppStoreVersion, mendixVersion, nil + } + return "", mendixVersion, fmt.Errorf("module %q not found in %s", moduleName, filepath.Base(mprPath)) +} + +// PackageProject builds a throwaway project containing nothing but the module +// from mpkPath, and returns the path to its .mpr. +// +// The project is created at mendixVersion — the consuming project's version, not +// the latest — so the imported module goes through the same conversion the +// installed copy went through. Comparing against a package converted to a +// different Mendix version would report the platform's own migrations as user +// edits. +// +// workDir must already exist and is written to freely; callers own its lifetime +// (t.TempDir in tests, an os.MkdirTemp the command removes in production). +// **Keep it short**: mx create-project extracts its template with .NET path +// handling and fails with PathTooLongException under a deep directory, so a +// nested scratch path breaks it where /tmp/xxxx works. +// +// newBackend is used to remove a module the template already ships (see below). +func PackageProject(ctx context.Context, mpkPath, mendixVersion, workDir string, newBackend func() backend.FullBackend) (string, error) { + mxPath, err := docker.ResolveMxForVersion("", mendixVersion) + if err != nil { + return "", fmt.Errorf("locate mx for Mendix %s: %w\n"+ + "hint: run 'mxcli setup mxbuild --version %s'", mendixVersion, err, mendixVersion) + } + + const appName = "PackageRef" + create := exec.CommandContext(ctx, mxPath, "create-project", "--app-name", appName) + create.Dir = workDir + docker.PrepareMxCommand(create) + if out, err := create.CombinedOutput(); err != nil { + return "", fmt.Errorf("mx create-project failed: %w\n%s", err, strings.TrimSpace(string(out))) + } + + mprPath, err := findScratchMpr(workDir, appName) + if err != nil { + return "", err + } + + // Verify the reference project really is at the requested version. + // + // This is not paranoia: ResolveMxForVersion falls back to any cached mxbuild + // when the requested version is missing — asking for 11.6.6 on a machine + // holding 11.12.1 returns 11.12.1 without complaint. mx create-project stamps + // the project with the version of the binary that ran it, so the reference + // would silently be built one or more Mendix versions away from the project + // being compared, and every platform migration between them would surface as + // a user edit. A tool whose false positives are indistinguishable from real + // findings is worse than no tool, so this refuses instead of warning. + if err := verifyProjectVersion(mprPath, mendixVersion); err != nil { + return "", err + } + + // A "blank" Mendix project is not empty: the template ships Administration, + // Atlas_Core, Atlas_Web_Content and friends already installed. module-import + // refuses a name that already exists ("Module 'X' already exist in the app", + // exit 47) — and Administration is exactly the module the field report cares + // about. So remove the template's copy first, then import the published one. + // + // mxcli's own DROP MODULE is used rather than a file edit because it also + // unpicks the references the template set up (it reports e.g. "Removed + // Administration.User from 1 user role(s)"); leaving those dangling would + // give module-import an inconsistent model to write into. + if name, err := moduleNameInPackage(mpkPath); err == nil && name != "" { + if err := dropModuleIfPresent(mprPath, name, newBackend); err != nil { + return "", err + } + } + + imp := exec.CommandContext(ctx, mxPath, "module-import", mpkPath, mprPath) + docker.PrepareMxCommand(imp) + if out, err := imp.CombinedOutput(); err != nil { + return "", fmt.Errorf("mx module-import failed: %w\n%s", err, strings.TrimSpace(string(out))) + } + return mprPath, nil +} + +// findScratchMpr locates the project mx just created. The name follows +// --app-name, but that is not contractual, so fall back to whatever .mpr exists. +func findScratchMpr(workDir, appName string) (string, error) { + named := filepath.Join(workDir, appName+".mpr") + if _, err := os.Stat(named); err == nil { + return named, nil + } + matches, _ := filepath.Glob(filepath.Join(workDir, "*.mpr")) + if len(matches) > 0 { + return matches[0], nil + } + // mx may nest the project one level down under the app name. + matches, _ = filepath.Glob(filepath.Join(workDir, "*", "*.mpr")) + if len(matches) > 0 { + return matches[0], nil + } + return "", fmt.Errorf("mx create-project produced no .mpr under %s", workDir) +} + +// verifyProjectVersion checks that a freshly created project is stamped with the +// version we asked for, and returns an actionable error when it is not. +// +// The stamped version is read from the project rather than inferred from the mx +// binary's path, because the path is a cache-layout detail while the stamp is +// what the comparison actually depends on. +func verifyProjectVersion(mprPath, want string) error { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return fmt.Errorf("open reference project: %w", err) + } + got, _ := reader.GetMendixVersion() + _ = reader.Close() + + if got == want { + return nil + } + return fmt.Errorf( + "reference project was built at Mendix %s but the project under comparison is %s.\n"+ + "Comparing across versions reports Mendix's own conversions as local edits, so this is refused.\n"+ + "hint: run 'mxcli setup mxbuild --version %s' to fetch the matching toolchain", + orUnknown(got), want, want) +} + +func orUnknown(v string) string { + if v == "" { + return "an unknown version" + } + return v +} + +// moduleNameInPackage reads the module name a .mpk declares. package.xml is the +// manifest mx itself reads; note it carries no version — the AppStoreVersion a +// project ends up with after `mx module-import` is the module's *internal* +// version (set by its author), not the marketplace release number. Importing +// Administration 4.3.2 stamps 2.0.1. That stamp never reaches DESCRIBE output, +// so it does not affect the comparison, but it does mean the reference project's +// recorded version is not evidence of which package was imported. +func moduleNameInPackage(mpkPath string) (string, error) { + zr, err := zip.OpenReader(mpkPath) + if err != nil { + return "", fmt.Errorf("open package %s: %w", filepath.Base(mpkPath), err) + } + defer zr.Close() + + for _, f := range zr.File { + if f.Name != "package.xml" { + continue + } + rc, err := f.Open() + if err != nil { + return "", err + } + defer rc.Close() + + var manifest struct { + Module struct { + Name string `xml:"name,attr"` + } `xml:"modelerProject>module"` + } + if err := xml.NewDecoder(rc).Decode(&manifest); err != nil { + return "", fmt.Errorf("parse package.xml: %w", err) + } + return manifest.Module.Name, nil + } + return "", fmt.Errorf("no package.xml in %s", filepath.Base(mpkPath)) +} + +// dropModuleIfPresent removes a module from the reference project when the +// template already provides it. A module that is not there is not an error. +func dropModuleIfPresent(mprPath, moduleName string, newBackend func() backend.FullBackend) error { + var sink bytes.Buffer + ex := executor.New(&sink) + defer ex.Close() + if newBackend != nil { + ex.SetBackendFactory(newBackend) + } + if err := ex.Execute(&ast.ConnectStmt{Path: mprPath}); err != nil { + return fmt.Errorf("connect reference project: %w", err) + } + + mods, err := ex.Backend().ListModules() + if err != nil { + return fmt.Errorf("list reference modules: %w", err) + } + for _, m := range mods { + if strings.EqualFold(m.Name, moduleName) { + if err := ex.Execute(&ast.DropModuleStmt{Name: m.Name}); err != nil { + return fmt.Errorf("remove the template's %s before importing the package: %w", m.Name, err) + } + return nil + } + } + return nil +} diff --git a/cmd/mxcli/marketplace/scratch_integration_test.go b/cmd/mxcli/marketplace/scratch_integration_test.go new file mode 100644 index 000000000..09bb917e9 --- /dev/null +++ b/cmd/mxcli/marketplace/scratch_integration_test.go @@ -0,0 +1,201 @@ +//go:build integration + +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" +) + +// shortTemp returns a work dir with a short path. mx create-project extracts its +// template with .NET path handling and dies with PathTooLongException under a +// deep directory, so t.TempDir() nested inside a long module path is not safe +// here — this was found the hard way. +func shortTemp(t *testing.T) string { + d, err := os.MkdirTemp("", "mxref") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.RemoveAll(d) }) + return d +} + +// mxVersionAvailable reports whether the exact mxbuild for want is cached. +// ResolveMxForVersion falls back to any cached version, so its success is not +// evidence — the cache path is checked directly. +func mxVersionAvailable(want string) bool { + p := docker.CachedMxPath(want) + if p == "" { + return false + } + _, err := os.Stat(p) + return err == nil +} + +// TestPackageProject_RefusesAVersionMismatch is the guard that matters most here, +// and it is the one case this environment can always exercise. +// +// ResolveMxForVersion silently falls back to any cached mxbuild: asking for a +// version that is not installed returns a different one, and mx create-project +// stamps the reference project with whatever binary ran it. Building the +// baseline one Mendix version away from the project under comparison would make +// the platform's own conversions look like user edits — false findings that are +// indistinguishable from real ones. PackageProject must refuse, not warn. +func TestPackageProject_RefusesAVersionMismatch(t *testing.T) { + const absent = "9.0.0" // old enough that no cache in CI or dev holds it + if mxVersionAvailable(absent) { + t.Skipf("Mendix %s is cached here, so the mismatch path cannot be exercised", absent) + } + if docker.AnyCachedMxPath() == "" { + t.Skip("no mxbuild cached; nothing to fall back to") + } + + _, err := PackageProject(context.Background(), "", absent, t.TempDir(), testBackend) + if err == nil { + t.Fatal("PackageProject must refuse when the reference project cannot be built at the requested version") + } + // The message has to name both versions and the fix, or the user cannot act. + for _, want := range []string{absent, "setup mxbuild"} { + if !contains(err.Error(), want) { + t.Errorf("error should mention %q so the user can act on it; got:\n%s", want, err) + } + } +} + +// TestPackageProject_BuildsAReferenceProject exercises the happy path end to end +// against a real .mpk and a real mx, and then snapshots the imported module — +// proving slice 2 hands slice 1 something it can actually read. +// +// It needs a package to import. MXCLI_TEST_MPK points at one; without it the +// test skips rather than reaching for the network, so the suite stays hermetic +// by default. +func TestPackageProject_BuildsAReferenceProject(t *testing.T) { + mpk := os.Getenv("MXCLI_TEST_MPK") + if mpk == "" { + t.Skip("set MXCLI_TEST_MPK to a downloaded .mpk to run this") + } + version := os.Getenv("MXCLI_TEST_MPK_MENDIX") + if version == "" { + t.Skip("set MXCLI_TEST_MPK_MENDIX to the Mendix version the .mpk should be imported at") + } + if !mxVersionAvailable(version) { + t.Skipf("mxbuild %s is not cached; run 'mxcli setup mxbuild --version %s'", version, version) + } + if _, err := os.Stat(mpk); err != nil { + t.Skipf("MXCLI_TEST_MPK does not exist: %v", err) + } + + mprPath, err := PackageProject(context.Background(), mpk, version, shortTemp(t), testBackend) + if err != nil { + t.Fatalf("PackageProject: %v", err) + } + if _, err := os.Stat(mprPath); err != nil { + t.Fatalf("reference project not on disk: %v", err) + } + t.Logf("reference project: %s", filepath.Base(mprPath)) + + module := os.Getenv("MXCLI_TEST_MPK_MODULE") + if module == "" { + return + } + snap, err := SnapshotModule(mprPath, module, testBackend) + if err != nil { + t.Fatalf("snapshot the imported module: %v", err) + } + if len(snap.Elements) == 0 { + t.Fatalf("imported module %q described no elements — the import produced nothing to compare", module) + } + t.Logf("%s in the reference project: %d elements", module, len(snap.Elements)) +} + +func contains(haystack, needle string) bool { + return len(needle) == 0 || (len(haystack) >= len(needle) && + (haystack == needle || indexOf(haystack, needle) >= 0)) +} + +func indexOf(h, n string) int { + for i := 0; i+len(n) <= len(h); i++ { + if h[i:i+len(n)] == n { + return i + } + } + return -1 +} + +// TestPackageProject_ReferenceIsReproducibleAndDiffable is the end-to-end proof +// of the whole Phase 1 read path against real marketplace content. +// +// Two reference projects are built independently from the *same* .mpk. They must +// compare clean: if building the baseline were not reproducible, every diff +// against it would carry noise the user cannot distinguish from their own edits. +// Then one side is edited, and the differ must report exactly that element. +// +// Measured with Administration 4.3.2 (content 23513) at Mendix 11.12.1: 21 +// elements captured, 21 unchanged between the two references, and a single added +// attribute reported as `ENTITY Account` modified. +func TestPackageProject_ReferenceIsReproducibleAndDiffable(t *testing.T) { + mpk := os.Getenv("MXCLI_TEST_MPK") + version := os.Getenv("MXCLI_TEST_MPK_MENDIX") + module := os.Getenv("MXCLI_TEST_MPK_MODULE") + if mpk == "" || version == "" || module == "" { + t.Skip("set MXCLI_TEST_MPK, MXCLI_TEST_MPK_MENDIX and MXCLI_TEST_MPK_MODULE to run this") + } + if !mxVersionAvailable(version) { + t.Skipf("mxbuild %s is not cached; run 'mxcli setup mxbuild --version %s'", version, version) + } + + ctx := context.Background() + refA, err := PackageProject(ctx, mpk, version, shortTemp(t), testBackend) + if err != nil { + t.Fatalf("reference A: %v", err) + } + refB, err := PackageProject(ctx, mpk, version, shortTemp(t), testBackend) + if err != nil { + t.Fatalf("reference B: %v", err) + } + + snapA, err := SnapshotModule(refA, module, testBackend) + if err != nil { + t.Fatalf("snapshot A: %v", err) + } + snapB, err := SnapshotModule(refB, module, testBackend) + if err != nil { + t.Fatalf("snapshot B: %v", err) + } + if len(snapA.Elements) == 0 { + t.Fatalf("the imported module described no elements — nothing to compare") + } + + if rep := Compare(snapA, snapB); !rep.Clean() { + for _, f := range rep.Findings { + if f.Verdict != Unchanged { + t.Errorf("%s: %s %s", f.Key, f.Verdict, f.Reason) + } + } + t.Fatal("two references built from the same package must compare clean") + } + t.Logf("%s: %d elements, reproducible", module, len(snapA.Elements)) + + // A real local edit must be reported, and nothing else with it. + execMDL(t, refA, "alter entity "+module+".Account add attribute DiffProbe: String(50);") + edited, err := SnapshotModule(refA, module, testBackend) + if err != nil { + t.Fatalf("snapshot after edit: %v", err) + } + + var changed []string + for _, f := range Compare(edited, snapB).Findings { + if f.Verdict != Unchanged { + changed = append(changed, f.Key.String()+"="+string(f.Verdict)) + } + } + if len(changed) != 1 || changed[0] != "ENTITY Account=modified" { + t.Errorf("expected exactly ENTITY Account=modified, got %v", changed) + } +} From 99aec6690789be00b4020872f976fff30a558244 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 15:23:07 +0000 Subject: [PATCH 11/35] Paginate the marketplace versions endpoint /v1/content/{id}/versions returns 10 items when given no paging parameters and caps `limit` at 20, but Client.Versions asked once and trusted the answer. Every content item therefore appeared to have exactly ten versions, and looking up an older one reported it as not published -- Data Widgets has 131 published releases and mxcli could see 10. Walk pages until one comes back short. `marketplace versions`, `download` and `install` all resolve older versions correctly now. The cap here is not the /v1/content cap (100), so the existing pageSize constant could not be reused; the test's mock clamps `limit` the way the server does, and fails a client that makes a single request. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/fix-issue.md | 1 + internal/marketplace/client.go | 32 +++++++++++++-- internal/marketplace/client_test.go | 62 +++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+), 4 deletions(-) diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 4bd5830f9..8184c19a9 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -463,3 +463,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `transform $In with Module.Transformer` reports "Created microflow" and the build then fails `[CE0008] "No action defined."` — the activity is in the model with no action inside it | The modelsdk writer (the DEFAULT engine) had no `TransformJsonAction` case, so it fell through to `default: return nil`. The grammar, builder, DESCRIBE formatter and the LEGACY writer all handled it, which is why it looked supported. The reader was missing too, so even a correctly written action described as a placeholder | `mdl/backend/modelsdk/microflow_write.go` (writer case), `mdl/backend/modelsdk/microflow_read_actions.go` (reader cases for `TransformJsonAction`, `CallExternalAction`, `RestOperationCallAction`) | **A missing WRITER case and a missing READER case present identically in a coverage audit** — grepping for reader cases found `transform` "authorable but unreadable" when it was actually unwritable, a strictly worse bug. Check both directions before sizing the work. **Mirror the legacy serializer for keys** (`InputVariableName`/`OutputVariableName`/`Transformation`); for the other two, note `CallExternalAction` stores its result under `VariableName` (not `ResultVariableName`) and REST's two mapping lists are NOT symmetric — the query list keys its name as `QueryParameter`. **Do not reconstruct `CallExternalAction.ResultDataType`**: it is resolved from the consumed service's cached `$metadata` at write time, so reading it back would let a stale value round-trip as if authored. **CE0008 turning into CE1613 is progress, not a new bug** — the action now exists and the build has moved on to validating its reference. Tests `microflow_integration_actions_test.go`, example `mdl-examples/bug-tests/transform-json-write-and-describe.mdl` | | `describe Module.Name` reports `no describable document named "..."` for a document that plainly exists and that `describe building block Module.Name` / `describe icon collection Module.Name` describes fine when the type is named explicitly | Two lists in different packages drifted apart. Bare DESCRIBE resolves the type by looking the qualified name up in the catalog `objects` view and mapping its `ObjectType` through `objectTypeToDescribeKind`. Building blocks were built into `building_blocks_data` but never joined into the `objects` view union, so the lookup returned no row; icon collections had **no catalog table at all**, leaving the `ICON_COLLECTION` entry in the map as dead code that nothing could ever emit. Measured on a stock 7-module marketplace project: 43 of 251 documents (40 building blocks + 3 icon collections) were unreachable this way — auto-detect coverage 81%, explicit-type coverage 98% | `mdl/executor/describe_auto.go` (`objectTypeToDescribeKind`), `mdl/catalog/tables.go` (the `objects` view union + the `*_data` table), `mdl/catalog/builder.go` (`buildSimpleNamedDocs` call), `mdl/catalog/catalog.go` (the `CATALOG.*` table list) | Add the type in **all four** places — a map entry alone does nothing if the view never emits the `ObjectType`, and a view row alone does nothing if the map has no kind. For a document with just name/folder/documentation, the builder is one `buildSimpleNamedDocs("", "", "
", "
[where ...];` | SQL against project metadata | -| Show callers | `show callers of Module.Name;` | What calls this element | +| Show callers | `show callers of Module.Name;` | What INVOKES this element: microflow call activities, page action buttons and other widget actions, calculated attributes, and navigation entries. A page counts as a caller of the microflow its button runs, and of the page that button opens | | Show callees | `show callees of Module.Name;` | What this element calls | | Show references | `show references of Module.Name;` | All references to/from | | Show impact | `show impact of Module.Name;` | Impact analysis | @@ -1174,6 +1174,8 @@ CLI subcommand: `mxcli sql --driver postgres --dsn '...' "select 1"` (see `mxcli Cross-reference commands require `refresh catalog full` to populate reference data. +`show callers` covers invocation only. A document that merely *uses a type* — an entity as a page datasource, a microflow parameter, an entity's generalization — is not a caller of it; `show references to` lists those. + ## Connection & Session | Statement | Syntax | Notes | diff --git a/mdl/catalog/builder_pages.go b/mdl/catalog/builder_pages.go index 7cc9d8758..99f8f78a9 100644 --- a/mdl/catalog/builder_pages.go +++ b/mdl/catalog/builder_pages.go @@ -34,9 +34,9 @@ func (b *Builder) buildPages() error { if b.fullMode { widgetStmt, err = b.tx.Prepare(` INSERT INTO widgets_data (Id, Name, WidgetType, ContainerId, ContainerQualifiedName, ContainerType, - ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, Description, + ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, PageRef, Description, ProjectId, SnapshotId) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `) if err != nil { return err @@ -114,6 +114,7 @@ func (b *Builder) buildPages() error { w.AttributeRef, w.MicroflowRef, w.NanoflowRef, + w.PageRef, "", projectID, snapshotID, ); err != nil { @@ -156,9 +157,9 @@ func (b *Builder) buildSnippets() error { if b.fullMode { widgetStmt, err = b.tx.Prepare(` INSERT INTO widgets_data (Id, Name, WidgetType, ContainerId, ContainerQualifiedName, ContainerType, - ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, Description, + ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, PageRef, Description, ProjectId, SnapshotId) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `) if err != nil { return err @@ -205,7 +206,7 @@ func (b *Builder) buildSnippets() error { w.ID, w.Name, w.WidgetType, string(sn.ID), qualifiedName, "SNIPPET", moduleName, folder, - w.EntityRef, w.AttributeRef, w.MicroflowRef, w.NanoflowRef, "", + w.EntityRef, w.AttributeRef, w.MicroflowRef, w.NanoflowRef, w.PageRef, "", projectID, snapshotID, ); err != nil { return fmt.Errorf("insert widget %s for snippet %s: %w", w.Name, qualifiedName, err) @@ -292,6 +293,7 @@ type rawWidgetInfo struct { AttributeRef string MicroflowRef string // action/datasource microflow (Forms$MicroflowSettings.Microflow, …) NanoflowRef string // action/datasource nanoflow + PageRef string // action page (Forms$PageSettings.Form) — see scanWidgetOwnRefs } // widgetChildKeys are the keys under which a widget nests *other* widgets. The @@ -306,8 +308,8 @@ var widgetChildKeys = map[string]bool{ // Returns the lexicographically smallest match of each kind so the result is // deterministic regardless of BSON map iteration order; page-level dedup in the // refs projection makes the per-widget choice immaterial to the final graph. -func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow string) { - var ents, mfs, nfs []string +func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow, page string) { + var ents, mfs, nfs, forms []string var walk func(v any) walk = func(v any) { switch x := v.(type) { @@ -321,6 +323,14 @@ func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow string) { if s, ok := x["Nanoflow"].(string); ok && s != "" { nfs = append(nfs, s) } + // A page reference is stored under "Form" — Mendix's original word for + // a page, the same rename behind ShowFormAction/CloseFormAction. Not + // collecting it is why an action button that opens a page created no + // reference, so the page reported "(no callers found)" however many + // buttons pointed at it (issue #773). + if s, ok := x["Form"].(string); ok && s != "" { + forms = append(forms, s) + } for k, val := range x { if widgetChildKeys[k] { continue @@ -346,7 +356,7 @@ func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow string) { } return m } - return min(ents), min(mfs), min(nfs) + return min(ents), min(mfs), min(nfs), min(forms) } // extractLayoutRef extracts the layout reference from raw page BSON. Regular @@ -461,7 +471,7 @@ func extractWidgetsRecursive(w map[string]any) []rawWidgetInfo { // Extract datasource entity + action microflow/nanoflow references from this // widget's own content (not its child widgets). - widget.EntityRef, widget.MicroflowRef, widget.NanoflowRef = scanWidgetOwnRefs(w) + widget.EntityRef, widget.MicroflowRef, widget.NanoflowRef, widget.PageRef = scanWidgetOwnRefs(w) // Index user-authored containers, but skip the synthetic // "conditionalVisibilityWidget*" wrapper that mxcli / Studio Pro insert as a diff --git a/mdl/catalog/builder_pages_test.go b/mdl/catalog/builder_pages_test.go index 838c5063d..b565d2fbc 100644 --- a/mdl/catalog/builder_pages_test.go +++ b/mdl/catalog/builder_pages_test.go @@ -24,7 +24,7 @@ func TestScanWidgetOwnRefs(t *testing.T) { }, }, } - ent, mf, nf := scanWidgetOwnRefs(dataView) + ent, mf, nf, _ := scanWidgetOwnRefs(dataView) if ent != "Sales.Order" { t.Errorf("entity = %q, want Sales.Order", ent) } @@ -41,7 +41,7 @@ func TestScanWidgetOwnRefs(t *testing.T) { "Action": map[string]any{"Settings": map[string]any{"Microflow": "M.DoThing"}}, "Extra": map[string]any{"Nanoflow": "M.DoNano"}, } - if _, mf, nf := scanWidgetOwnRefs(button); mf != "M.DoThing" || nf != "M.DoNano" { + if _, mf, nf, _ := scanWidgetOwnRefs(button); mf != "M.DoThing" || nf != "M.DoNano" { t.Errorf("button refs = (%q,%q), want (M.DoThing, M.DoNano)", mf, nf) } @@ -52,13 +52,13 @@ func TestScanWidgetOwnRefs(t *testing.T) { "B": map[string]any{"Microflow": "M.Alpha"}, } for range 5 { - if _, mf, _ := scanWidgetOwnRefs(multi); mf != "M.Alpha" { + if _, mf, _, _ := scanWidgetOwnRefs(multi); mf != "M.Alpha" { t.Fatalf("non-deterministic microflow pick: got %q, want M.Alpha", mf) } } // No refs. - if e, m, n := scanWidgetOwnRefs(map[string]any{"$Type": "Forms$Label"}); e != "" || m != "" || n != "" { + if e, m, n, p := scanWidgetOwnRefs(map[string]any{"$Type": "Forms$Label"}); e != "" || m != "" || n != "" || p != "" { t.Errorf("expected no refs, got (%q,%q,%q)", e, m, n) } } @@ -245,9 +245,9 @@ func TestExtractWidgetsRecursive(t *testing.T) { t.Run("indexes a styled conditionalVisibilityWidget (not transparent)", func(t *testing.T) { // If a wrapper-named container carries styling it is a real user widget. w := map[string]any{ - "$ID": "div3", - "Name": "conditionalVisibilityWidget9", - "$Type": "Forms$DivContainer", + "$ID": "div3", + "Name": "conditionalVisibilityWidget9", + "$Type": "Forms$DivContainer", "Appearance": map[string]any{"Class": "card"}, } got := extractWidgetsRecursive(w) @@ -557,3 +557,65 @@ func TestBytesToHex(t *testing.T) { }) } } + +// upstream #773: a widget action that OPENS a page created no reference at all, +// so a page reachable only from a button reported "(no callers found)" — a false +// negative that reads as "safe to delete". +// +// The cause was one missing key. scanWidgetOwnRefs collected Entity, Microflow +// and Nanoflow but not `Form`, which is where a page reference lives — "Form" +// being Mendix's original word for a page, the same rename behind +// ShowFormAction/CloseFormAction. +func TestScanWidgetOwnRefs_PageReference(t *testing.T) { + // An action button that opens a page: `show_page Module.Target`. + showPage := map[string]any{ + "$Type": "Forms$ActionButton", + "Action": map[string]any{ + "$Type": "Forms$ShowPageClientAction", + "PageSettings": map[string]any{"$Type": "Forms$PageSettings", "Form": "Sales.OrderDetail"}, + }, + } + if _, _, _, page := scanWidgetOwnRefs(showPage); page != "Sales.OrderDetail" { + t.Errorf("page = %q, want Sales.OrderDetail", page) + } + + // The compound the issue reported — "create object … then open page" — is ONE + // action carrying BOTH an entity and a page. Collecting only the entity (which + // is what happened) still leaves the page with no inbound reference. + createThenShow := map[string]any{ + "$Type": "Forms$ActionButton", + "Action": map[string]any{ + "$Type": "Forms$CreateObjectClientAction", + "Entity": "Sales.Order", + "PageSettings": map[string]any{"$Type": "Forms$PageSettings", "Form": "Sales.OrderDetail"}, + }, + } + ent, _, _, page := scanWidgetOwnRefs(createThenShow) + if ent != "Sales.Order" { + t.Errorf("entity = %q, want Sales.Order", ent) + } + if page != "Sales.OrderDetail" { + t.Errorf("page = %q, want Sales.OrderDetail — the page half of "+ + "'create object … then open page' is the reference the issue reported missing", page) + } + + // A child widget's page must not be attributed to its parent, same as the + // existing microflow rule. + container := map[string]any{ + "$Type": "Forms$DivContainer", + "Widgets": []any{ + map[string]any{ + "$Type": "Forms$ActionButton", + "Action": map[string]any{"PageSettings": map[string]any{"Form": "Sales.ChildPage"}}, + }, + }, + } + if _, _, _, page := scanWidgetOwnRefs(container); page != "" { + t.Errorf("page = %q, want empty — a child button's page must not leak to the container", page) + } + + // A widget with no action references no page. + if _, _, _, page := scanWidgetOwnRefs(map[string]any{"$Type": "Forms$Label"}); page != "" { + t.Errorf("page = %q, want empty", page) + } +} diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index 26519d0be..b85da8997 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -366,6 +366,10 @@ func (b *Builder) buildReferences() error { {"EntityRef", "ENTITY", RefKindDatasource}, {"MicroflowRef", "MICROFLOW", RefKindAction}, {"NanoflowRef", "NANOFLOW", RefKindAction}, + // A widget action that opens a page. Without this row, a page reachable + // only from a button had no inbound reference and `show callers` / + // `show references` reported it as unused (issue #773). + {"PageRef", "PAGE", RefKindShowPage}, } for _, p := range widgetProjections { res, perr := b.tx.Exec( diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index b39426c6b..583cc993d 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -416,6 +416,11 @@ func (c *Catalog) createTables() error { AttributeRef TEXT, MicroflowRef TEXT, NanoflowRef TEXT, + -- The page a widget's action opens (show_page, and the page half of + -- "create object … then open page"). Without it a page reachable only + -- from a button had no inbound reference at all and read as dead code + -- (issue #773). + PageRef TEXT, Description TEXT, ProjectId TEXT, SnapshotId TEXT diff --git a/mdl/executor/cmd_search.go b/mdl/executor/cmd_search.go index 497ec70c6..689528ecd 100644 --- a/mdl/executor/cmd_search.go +++ b/mdl/executor/cmd_search.go @@ -11,6 +11,52 @@ import ( ) // execShowCallers handles SHOW CALLERS OF Module.Microflow [TRANSITIVE]. +// callerRefKinds are the reference kinds that mean "this thing invokes the +// target". `show callers` filtered on 'call' alone, which is the kind a MICROFLOW +// activity produces — so a microflow called from a page action button (kind +// 'action') reported "(no callers found)" even though the reference was sitting +// in the refs table, and a page opened by a button or a menu was equally +// invisible (issue #773). +// +// A false negative here reads as "nothing uses this", which is the answer +// somebody acts on before deleting a document — so the set errs toward +// including a kind rather than omitting it. +// +// Deliberately excluded: 'datasource', 'parameter', 'return', 'retrieve', +// 'create', 'change', 'delete', 'associate', 'generalize' and 'layout'. Those +// are uses of a TYPE or a LAYOUT, not invocations, and folding them in would +// make `show callers of ` a synonym for `show references to`. +var callerRefKinds = []string{ + RefKindCallerCall, // microflow/nanoflow call activity + RefKindCallerAction, // widget action: button, on-change, on-click + RefKindCallerShowPage, // microflow show-page activity, or a widget action opening a page + RefKindCallerCalculate, // calculated attribute + RefKindCallerHomePage, // navigation + RefKindCallerLoginPage, + RefKindCallerMenuItem, +} + +// Kind literals, kept next to the set that uses them so the SQL below cannot +// drift from mdl/catalog's constants unnoticed. +const ( + RefKindCallerCall = "call" + RefKindCallerAction = "action" + RefKindCallerShowPage = "show_page" + RefKindCallerCalculate = "calculate" + RefKindCallerHomePage = "home_page" + RefKindCallerLoginPage = "login_page" + RefKindCallerMenuItem = "menu_item" +) + +// callerRefKindsSQL renders callerRefKinds as a SQL IN list. +func callerRefKindsSQL() string { + quoted := make([]string, len(callerRefKinds)) + for i, k := range callerRefKinds { + quoted[i] = "'" + k + "'" + } + return "(" + strings.Join(quoted, ", ") + ")" +} + func execShowCallers(ctx *ExecContext, s *ast.ShowStmt) error { if s.Name == nil { return mdlerrors.NewValidation("target name required for show callers") @@ -36,12 +82,12 @@ func execShowCallers(ctx *ExecContext, s *ast.ShowStmt) error { with RECURSIVE callers_cte as ( select SourceName as Caller, 1 as Depth from refs - where TargetName = ? and RefKind = 'call' + where TargetName = ? and RefKind in ` + callerRefKindsSQL() + ` union all select r.SourceName, c.Depth + 1 from refs r join callers_cte c on r.TargetName = c.Caller - where r.RefKind = 'call' and c.Depth < 10 + where r.RefKind in ` + callerRefKindsSQL() + ` and c.Depth < 10 ) select distinct Caller, min(Depth) as Depth from callers_cte @@ -53,7 +99,7 @@ func execShowCallers(ctx *ExecContext, s *ast.ShowStmt) error { query = ` select distinct SourceName as Caller, 1 as Depth from refs - where TargetName = ? and RefKind = 'call' + where TargetName = ? and RefKind in ` + callerRefKindsSQL() + ` ORDER by Caller ` } diff --git a/mdl/executor/cmd_search_callers_test.go b/mdl/executor/cmd_search_callers_test.go new file mode 100644 index 000000000..a9998eb77 --- /dev/null +++ b/mdl/executor/cmd_search_callers_test.go @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// upstream #773, second half. `show callers` filtered on RefKind = 'call' — the +// kind a MICROFLOW call activity produces. A microflow invoked from a page action +// button is recorded as 'action', and that row was already sitting in the refs +// table: the reference existed, the query hid it. A page opened by a button or a +// navigation menu was invisible for the same reason. +// +// A false negative here reads as "nothing uses this", which is the answer +// somebody acts on before deleting a document — so the set errs toward including +// a kind rather than omitting it. +func TestCallerRefKinds(t *testing.T) { + in := map[string]bool{} + for _, k := range callerRefKinds { + in[k] = true + } + + // Every way one document can INVOKE another. + for _, k := range []string{ + "call", // microflow/nanoflow call activity — the only one that used to count + "action", // widget action: the reported button case + "show_page", // a microflow, or a button, opening a page + "calculate", // calculated attribute running a microflow + "home_page", // navigation + "login_page", // + "menu_item", // + } { + if !in[k] { + t.Errorf("%q means one document invokes another and must count as a caller — "+ + "omitting it reports the target as unused, which is what #773 was", k) + } + } + + // Uses of a TYPE or a LAYOUT are NOT invocations. Folding these in would make + // `show callers of ` a synonym for `show references to` and destroy + // the distinction between the two commands. + for _, k := range []string{ + "datasource", "parameter", "return", "retrieve", + "create", "change", "delete", "associate", "generalize", "layout", + } { + if in[k] { + t.Errorf("%q is a use of a type or layout, not an invocation — including it "+ + "collapses `show callers` into `show references to`", k) + } + } +} + +// The kind list reaches SQLite as an IN list, so a malformed render silently +// matches nothing — the same "(no callers found)" the issue reported, from a +// different cause. +func TestCallerRefKindsSQL(t *testing.T) { + got := callerRefKindsSQL() + if !strings.HasPrefix(got, "(") || !strings.HasSuffix(got, ")") { + t.Fatalf("not a SQL IN list: %q", got) + } + for _, k := range callerRefKinds { + if !strings.Contains(got, "'"+k+"'") { + t.Errorf("%q is not quoted in the IN list %q", k, got) + } + } + if n := strings.Count(got, ","); n != len(callerRefKinds)-1 { + t.Errorf("%d separators for %d kinds: %q", n, len(callerRefKinds), got) + } +} From 5f82ea39376beda8e2293a08eeec315b0b5ea377 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 20:23:23 +0000 Subject: [PATCH 21/35] Phase 2 slice 1: capture a module's GUID identities MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The load-bearing safety mechanism of a module update. §8 measured that the runtime keys entities and attributes on the model's GUID, so a replace that does not carry every existing GUID onto its replacement destroys that module's data on the next deploy -- silently, with a valid model and a green build. CaptureIdentities walks a module's units and records every element that carries a GUID, keyed by its path of names (Account, Account/FullName) rather than by name alone: two entities each having a Name attribute is ordinary, and a name-keyed map would transplant one entity's identity onto another's column. The path is also what survives an update, since a replace renumbers every $ID and keeps every name. The walk is type-agnostic -- any node carrying both Name and GUID is recorded, wherever it sits -- because the set of GUID-carrying types is not enumerable from the metamodel. It resolves module membership by walking the containment chain, not by reading one level, which is the defect that made foldered documents invisible to DESCRIBE (#759). Corroboration: the fixture's Administration yields 9 identities (2 entities, 6 attributes, 1 association), the same count §4 measured Studio Pro transplanting on a different project at a different Mendix version. The test asserts Account's captured GUID is byte-identical to the mendixsystem$entity.id read from the live database in §8. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/marketplace/identity.go | 177 +++++++++++++++++++++++++ cmd/mxcli/marketplace/identity_test.go | 101 ++++++++++++++ 2 files changed, 278 insertions(+) create mode 100644 cmd/mxcli/marketplace/identity.go create mode 100644 cmd/mxcli/marketplace/identity_test.go diff --git a/cmd/mxcli/marketplace/identity.go b/cmd/mxcli/marketplace/identity.go new file mode 100644 index 000000000..97fe9cfc2 --- /dev/null +++ b/cmd/mxcli/marketplace/identity.go @@ -0,0 +1,177 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "fmt" + "sort" + "strings" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" +) + +// Identities maps an element's path within its module to the `GUID` the stored +// model holds for it. +// +// This is the load-bearing safety mechanism of a module update. The database +// keys entities and attributes on the model's `GUID` — measured, see +// PROPOSAL_marketplace_module_upgrade.md §8 — so an update that replaces a +// module's documents must carry every existing `GUID` onto its replacement or +// the next deploy silently destroys that module's data. Studio Pro's own update +// does exactly this transplant (§4: 9 of 9 preserved, while all 94 `$ID`s are +// renumbered). +type Identities map[string][]byte + +// Paths returns the recorded paths in a stable order. +func (ids Identities) Paths() []string { + out := make([]string, 0, len(ids)) + for p := range ids { + out = append(out, p) + } + sort.Strings(out) + return out +} + +// CaptureIdentities records the `GUID` of every element of a module that carries +// one. +// +// Elements are keyed by their **path of names** within the document — an entity +// is `Account`, its attribute is `Account/FullName` — rather than by name alone, +// because names repeat: two entities in one domain model both having a `Name` +// attribute is the normal case, not an edge case. The path is also what survives +// the update, since a replace renumbers every `$ID` and keeps every name. +// +// The walk is deliberately type-agnostic: any BSON node carrying both a `Name` +// and a `GUID` is recorded, wherever it sits. New document types therefore need +// no registration here, which matters because the set of `GUID`-carrying types +// is not something mxcli can enumerate from the metamodel. +func CaptureIdentities(mprPath, moduleName string) (Identities, error) { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return nil, fmt.Errorf("open %s: %w", mprPath, err) + } + defer reader.Close() + + units, err := reader.ListUnits() + if err != nil { + return nil, fmt.Errorf("list units: %w", err) + } + + // Resolve which units belong to the module. A document nests in folders, so + // this walks the containment chain rather than reading one level — the same + // trap that made foldered documents invisible to DESCRIBE (#759). + inModule, err := unitsOfModule(reader, units, moduleName) + if err != nil { + return nil, err + } + if len(inModule) == 0 { + return nil, fmt.Errorf("module %q has no units, so there is nothing to preserve", moduleName) + } + + ids := Identities{} + for _, unitID := range inModule { + raw, err := reader.GetRawUnitBytes(model.ID(unitID)) + if err != nil || len(raw) == 0 { + continue + } + var doc bson.D + if err := bson.Unmarshal(raw, &doc); err != nil { + continue + } + collectIdentities(doc, nil, ids) + } + return ids, nil +} + +// collectIdentities walks a decoded document, recording every Name+GUID pair +// under its path of enclosing names. +func collectIdentities(v any, trail []string, out Identities) { + switch t := v.(type) { + case bson.D: + name, guid, hasGUID := nameAndGUID(t) + next := trail + if name != "" { + next = append(append([]string{}, trail...), name) + if hasGUID { + out[strings.Join(next, "/")] = guid + } + } + for _, e := range t { + collectIdentities(e.Value, next, out) + } + case bson.A: + for _, e := range t { + collectIdentities(e, trail, out) + } + } +} + +func nameAndGUID(d bson.D) (name string, guid []byte, hasGUID bool) { + for _, e := range d { + switch e.Key { + case "Name": + if s, ok := e.Value.(string); ok { + name = s + } + case "GUID": + if b, ok := e.Value.(primitive.Binary); ok && len(b.Data) > 0 { + guid, hasGUID = b.Data, true + } + } + } + return name, guid, hasGUID +} + +// unitsOfModule returns the IDs of every unit contained in the named module, +// directly or through any depth of folders. +func unitsOfModule(reader *modelsdk.Reader, units []*types.UnitInfo, moduleName string) ([]string, error) { + parent := make(map[string]string, len(units)) + for _, u := range units { + parent[string(u.ID)] = string(u.ContainerID) + } + + mods, err := reader.ListModules() + if err != nil { + return nil, err + } + var moduleID string + for _, m := range mods { + if strings.EqualFold(m.Name, moduleName) { + moduleID = string(m.ID) + break + } + } + if moduleID == "" { + return nil, fmt.Errorf("module %q not found", moduleName) + } + + var out []string + for _, u := range units { + if string(u.ID) == moduleID || descendsFrom(string(u.ID), moduleID, parent) { + out = append(out, string(u.ID)) + } + } + return out, nil +} + +// descendsFrom reports whether id sits under ancestor. The walk is bounded +// because the project root is its own container — an unguarded loop hangs there. +func descendsFrom(id, ancestor string, parent map[string]string) bool { + seen := map[string]bool{} + for cur := id; cur != "" && !seen[cur]; { + seen[cur] = true + p, ok := parent[cur] + if !ok || p == cur { + return false + } + if p == ancestor { + return true + } + cur = p + } + return false +} diff --git a/cmd/mxcli/marketplace/identity_test.go b/cmd/mxcli/marketplace/identity_test.go new file mode 100644 index 000000000..85dc4c41c --- /dev/null +++ b/cmd/mxcli/marketplace/identity_test.go @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "encoding/hex" + "testing" +) + +// TestCaptureIdentities_RecordsTheDatabaseKey is the assertion the whole update +// path rests on. +// +// The runtime keys entities and attributes on the model's GUID — measured +// against a live PostgreSQL, see PROPOSAL §8, where mendixsystem$entity.id for +// Administration.Account is b16e49ea-91df-4caa-aed8-6ba4c4e133c5. The value +// captured here must be exactly that, byte-for-byte in stored (.NET) order, +// because a module update that carries anything else destroys the table's data +// on the next deploy. +func TestCaptureIdentities_RecordsTheDatabaseKey(t *testing.T) { + ids, err := CaptureIdentities(copyFixture(t), "Administration") + if err != nil { + t.Fatalf("CaptureIdentities: %v", err) + } + + // Stored .NET byte order for b16e49ea-91df-4caa-aed8-6ba4c4e133c5: the first + // three groups are little-endian, the rest is as-is. + const accountGUID = "ea496eb1df91aa4caed86ba4c4e133c5" + + got, ok := ids["Account"] + if !ok { + t.Fatalf("entity Account has no recorded identity; captured: %v", ids.Paths()) + } + if hex.EncodeToString(got) != accountGUID { + t.Errorf("Account GUID = %s, want %s (the id the runtime keys the table on)", + hex.EncodeToString(got), accountGUID) + } +} + +// TestCaptureIdentities_KeysByPathNotName — names repeat. Two entities each +// having an attribute of the same name is ordinary, so a name-keyed map would +// silently collapse them and transplant one entity's identity onto another's +// column. The path is what keeps them apart. +func TestCaptureIdentities_KeysByPathNotName(t *testing.T) { + ids, err := CaptureIdentities(copyFixture(t), "Administration") + if err != nil { + t.Fatalf("CaptureIdentities: %v", err) + } + + // Attributes are recorded under their entity. + if _, ok := ids["Account/FullName"]; !ok { + t.Errorf("expected Account/FullName to be recorded; captured: %v", ids.Paths()) + } + // The bare attribute name must not be a key of its own. + if _, ok := ids["FullName"]; ok { + t.Error("attributes must be keyed under their entity, not by bare name") + } + + // Every recorded GUID must be a full 16 bytes: a truncated identity is worse + // than none, because it would be transplanted and silently wrong. + for _, p := range ids.Paths() { + if len(ids[p]) != 16 { + t.Errorf("%s: GUID is %d bytes, want 16", p, len(ids[p])) + } + } +} + +// TestCaptureIdentities_CoversTheWholeModule checks the capture reaches +// documents nested in folders, not just those sitting directly under the module. +// In a real marketplace module almost everything is foldered — reading one level +// is the defect that made DESCRIBE miss 15 of 16 microflows (#759). +func TestCaptureIdentities_CoversTheWholeModule(t *testing.T) { + ids, err := CaptureIdentities(copyFixture(t), "Administration") + if err != nil { + t.Fatalf("CaptureIdentities: %v", err) + } + // The fixture's Administration yields 9: two entities, six attributes and one + // association. That is the same count §4 measured Studio Pro transplanting + // ("Elements carrying a GUID: 9, GUID preserved: 9") on a different project at + // a different Mendix version — so the walk is finding the set Mendix itself + // treats as identity-bearing, not merely a plausible-looking subset. + // + // Asserted as a floor rather than equality: a fixture gaining an attribute + // should not fail this test, while a walk that stopped at the domain model's + // direct children would come in far below. + if len(ids) < 8 { + t.Errorf("captured only %d identities (%v) — the walk is not reaching the whole module", + len(ids), ids.Paths()) + } + if _, ok := ids["AccountPasswordData"]; !ok { + t.Errorf("second entity not captured; got %v", ids.Paths()) + } +} + +// TestCaptureIdentities_RefusesAnUnknownModule — an update must not proceed on a +// module it could not read identities for, so this has to be an error rather +// than an empty map that a caller could mistake for "nothing to preserve". +func TestCaptureIdentities_RefusesAnUnknownModule(t *testing.T) { + if _, err := CaptureIdentities(copyFixture(t), "NoSuchModule"); err == nil { + t.Fatal("capturing identities for a module that does not exist must be an error") + } +} From 47f7015ff245f7596ba522d25556c711f563d2a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 20:30:15 +0000 Subject: [PATCH 22/35] Phase 2 slice 2: transplant captured GUIDs onto a replaced module MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The other half of the pair. ApplyIdentities writes recorded GUIDs back onto a module's elements, matched by the path capture recorded them under, so that after a module's documents are replaced every element that existed before still carries the identity the database keys on (§8). Three behaviours are deliberate and tested: - An element with no recorded identity keeps its freshly minted GUID. It is new in the target version, and inheriting an old one would make two elements the same entity as far as the runtime is concerned. - A recorded path that no longer exists is reported as missing rather than swallowed. That is an element the new version removed, and its column is about to disappear -- a data decision the caller has to make, not something this function should absorb. - Only units that actually changed are written, so a module whose identities already match is not rewritten (ADR-0008). The round-trip test scrambles every GUID before restoring them. Without that step it would pass whether or not the code did anything, since the identities already matched. Slice 3 is the unit copy itself: reading a module's units out of a package and writing them into the target with mxcli's own writer, which is what keeps the project in MPR v2 (option A). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/marketplace/identity.go | 136 +++++++++++++++++++++++++ cmd/mxcli/marketplace/identity_test.go | 106 +++++++++++++++++++ 2 files changed, 242 insertions(+) diff --git a/cmd/mxcli/marketplace/identity.go b/cmd/mxcli/marketplace/identity.go index 97fe9cfc2..25d297017 100644 --- a/cmd/mxcli/marketplace/identity.go +++ b/cmd/mxcli/marketplace/identity.go @@ -175,3 +175,139 @@ func descendsFrom(id, ancestor string, parent map[string]string) bool { } return false } + +// ApplyIdentities writes recorded `GUID`s back onto a module's elements, matched +// by path, and reports what it could not place. +// +// This is the transplant half of the update: after a module's documents are +// replaced, every element that existed before must carry the `GUID` it had +// before, or the runtime treats it as a new entity and drops the old table +// (§8). Studio Pro does the same thing — its update renumbers all 94 `$ID`s and +// preserves all 9 `GUID`s (§4). +// +// Elements with no recorded identity are left exactly as they are, with their +// freshly minted `GUID`s: those are genuinely new in the target version, and a +// new element must not inherit an old one's identity. Recorded paths that no +// longer exist are returned as `missing` — an element the new version removed, +// which is information the caller needs rather than an error here. +// +// The write is per unit and only for units that actually changed, so a module +// whose identities all already match is not rewritten at all (ADR-0008). +func ApplyIdentities(mprPath, moduleName string, ids Identities) (applied int, missing []string, err error) { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return 0, nil, fmt.Errorf("open %s: %w", mprPath, err) + } + units, err := reader.ListUnits() + if err != nil { + reader.Close() + return 0, nil, fmt.Errorf("list units: %w", err) + } + inModule, err := unitsOfModule(reader, units, moduleName) + if err != nil { + reader.Close() + return 0, nil, err + } + + placed := map[string]bool{} + type pending struct { + id string + contents []byte + } + var writes []pending + + for _, unitID := range inModule { + raw, rerr := reader.GetRawUnitBytes(model.ID(unitID)) + if rerr != nil || len(raw) == 0 { + continue + } + var doc bson.D + if bson.Unmarshal(raw, &doc) != nil { + continue + } + n := applyIdentities(doc, nil, ids, placed) + if n == 0 { + continue + } + encoded, merr := bson.Marshal(doc) + if merr != nil { + reader.Close() + return 0, nil, fmt.Errorf("re-encode unit %s: %w", unitID, merr) + } + writes = append(writes, pending{unitID, encoded}) + applied += n + } + reader.Close() + + for _, p := range ids.Paths() { + if !placed[p] { + missing = append(missing, p) + } + } + if len(writes) == 0 { + return applied, missing, nil + } + + writer, err := modelsdk.OpenForWriting(mprPath) + if err != nil { + return 0, nil, fmt.Errorf("open %s for writing: %w", mprPath, err) + } + defer writer.Close() + for _, w := range writes { + if err := writer.UpdateRawUnit(w.id, w.contents); err != nil { + return 0, nil, fmt.Errorf("write identities into unit %s: %w", w.id, err) + } + } + return applied, missing, nil +} + +// applyIdentities rewrites GUIDs in place, returning how many it changed and +// recording which recorded paths it found. +func applyIdentities(v any, trail []string, ids Identities, placed map[string]bool) int { + changed := 0 + switch t := v.(type) { + case bson.D: + name, _, hasGUID := nameAndGUID(t) + next := trail + if name != "" { + next = append(append([]string{}, trail...), name) + if hasGUID { + path := strings.Join(next, "/") + if want, ok := ids[path]; ok { + placed[path] = true + for i, e := range t { + if e.Key != "GUID" { + continue + } + b := e.Value.(primitive.Binary) + if !bytesEqual(b.Data, want) { + t[i].Value = primitive.Binary{Subtype: b.Subtype, Data: append([]byte{}, want...)} + changed++ + } + break + } + } + } + } + for _, e := range t { + changed += applyIdentities(e.Value, next, ids, placed) + } + case bson.A: + for _, e := range t { + changed += applyIdentities(e, trail, ids, placed) + } + } + return changed +} + +func bytesEqual(a, b []byte) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} diff --git a/cmd/mxcli/marketplace/identity_test.go b/cmd/mxcli/marketplace/identity_test.go index 85dc4c41c..dfe24a132 100644 --- a/cmd/mxcli/marketplace/identity_test.go +++ b/cmd/mxcli/marketplace/identity_test.go @@ -99,3 +99,109 @@ func TestCaptureIdentities_RefusesAnUnknownModule(t *testing.T) { t.Fatal("capturing identities for a module that does not exist must be an error") } } + +// TestApplyIdentities_RestoresTheCapturedGUIDs is the round trip the update +// rests on: capture from the copy that has the data, apply onto the copy that +// replaced it, and every element must end up carrying the identity it had +// before. +// +// The two projects here stand in for before-and-after an update. Mutating the +// GUIDs first is what makes the assertion mean something — applying identities +// that already match would pass whether or not the code did anything. +func TestApplyIdentities_RestoresTheCapturedGUIDs(t *testing.T) { + original := copyFixture(t) + replaced := copyFixture(t) + + before, err := CaptureIdentities(original, "Administration") + if err != nil { + t.Fatalf("capture: %v", err) + } + + // Stand in for "the module was replaced": every identity is now different. + scrambled := Identities{} + for p, g := range before { + alt := append([]byte{}, g...) + alt[0] ^= 0xFF + scrambled[p] = alt + } + if _, _, err := ApplyIdentities(replaced, "Administration", scrambled); err != nil { + t.Fatalf("scramble: %v", err) + } + mid, err := CaptureIdentities(replaced, "Administration") + if err != nil { + t.Fatalf("capture after scramble: %v", err) + } + if bytesEqual(mid["Account"], before["Account"]) { + t.Fatal("the scramble did not take, so the restore below would prove nothing") + } + + // Now the transplant. + applied, missing, err := ApplyIdentities(replaced, "Administration", before) + if err != nil { + t.Fatalf("ApplyIdentities: %v", err) + } + if applied != len(before) { + t.Errorf("applied %d of %d identities", applied, len(before)) + } + if len(missing) != 0 { + t.Errorf("nothing should be missing between identical modules; got %v", missing) + } + + after, err := CaptureIdentities(replaced, "Administration") + if err != nil { + t.Fatalf("capture after restore: %v", err) + } + for _, p := range before.Paths() { + if !bytesEqual(after[p], before[p]) { + t.Errorf("%s: identity not restored (%x vs %x)", p, after[p], before[p]) + } + } +} + +// TestApplyIdentities_ReportsWhatItCouldNotPlace — a recorded path that no +// longer exists means the new version removed that element. The caller has to +// know: its column is about to disappear, which is a data decision rather than +// something to silently absorb. +func TestApplyIdentities_ReportsWhatItCouldNotPlace(t *testing.T) { + mpr := copyFixture(t) + ids, err := CaptureIdentities(mpr, "Administration") + if err != nil { + t.Fatalf("capture: %v", err) + } + ids["Account/AttributeThatWasRemoved"] = make([]byte, 16) + + _, missing, err := ApplyIdentities(mpr, "Administration", ids) + if err != nil { + t.Fatalf("ApplyIdentities: %v", err) + } + if len(missing) != 1 || missing[0] != "Account/AttributeThatWasRemoved" { + t.Errorf("missing = %v, want exactly the removed attribute", missing) + } +} + +// TestApplyIdentities_LeavesNewElementsAlone — an element with no recorded +// identity is new in the target version and must keep its freshly minted GUID. +// Inheriting an old one would make two elements the same entity to the runtime. +func TestApplyIdentities_LeavesNewElementsAlone(t *testing.T) { + mpr := copyFixture(t) + full, err := CaptureIdentities(mpr, "Administration") + if err != nil { + t.Fatalf("capture: %v", err) + } + + // Apply a map that knows about only one element. + partial := Identities{"Account": full["Account"]} + if _, _, err := ApplyIdentities(mpr, "Administration", partial); err != nil { + t.Fatalf("ApplyIdentities: %v", err) + } + + after, err := CaptureIdentities(mpr, "Administration") + if err != nil { + t.Fatalf("capture: %v", err) + } + for _, p := range full.Paths() { + if !bytesEqual(after[p], full[p]) { + t.Errorf("%s changed, but only Account was in the map", p) + } + } +} From bac4f3fa02b6f1fb369b25bdfdedbab0e15d5e08 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 21:04:14 +0000 Subject: [PATCH 23/35] Phase 2 slice 3: transplant a module's units without mx module-import MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The import step, done with mxcli's own writer rather than `mx module-import`. That command rewrites an MPR v2 project as v1 -- measured, and refused outright by `marketplace install` -- and it also refuses theme modules. Copying the units directly avoids both and keeps the destination in whatever format it already uses, since the writer handles v1 and v2 alike. Units are copied verbatim, unit IDs included. That is sound because the destination's copy of the module is removed first, so the IDs are free, and because no element $ID pointer crosses a unit boundary (§4) -- so a unit is either copied whole or not at all, never rewritten, which is what ADR-0008 requires. Only the module unit is re-parented, onto the destination's project unit under the "Modules" containment every module uses. Verified where it counts. A round trip through mxcli's own reader proves only that mxcli agrees with itself; the failure to rule out is a model that reads back fine and that mxbuild rejects. Measured on 11.12.1 with two blank projects: 28 units copied, and `mx check` reports "The app contains: 0 errors." That run is now an integration test. Transplant deliberately does not preserve identities -- the copied module carries the package's GUIDs, and the caller pairs it with CaptureIdentities before and ApplyIdentities after. Keeping the two separate is what lets the identity step be tested against a scrambled source, where a no-op would otherwise look like success. Still open before this is a command: project-level references (a user role's grant of a module role) are unpicked by DROP MODULE and not restored by the copy, and the conflict case -- what to do when diff says the user edited an element the update replaces -- is undecided. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .../marketplace/scratch_integration_test.go | 79 ++++++++ cmd/mxcli/marketplace/transplant.go | 182 ++++++++++++++++++ cmd/mxcli/marketplace/transplant_test.go | 108 +++++++++++ sdk/mpr/writer_units.go | 15 ++ 4 files changed, 384 insertions(+) create mode 100644 cmd/mxcli/marketplace/transplant.go create mode 100644 cmd/mxcli/marketplace/transplant_test.go diff --git a/cmd/mxcli/marketplace/scratch_integration_test.go b/cmd/mxcli/marketplace/scratch_integration_test.go index 09bb917e9..073af030d 100644 --- a/cmd/mxcli/marketplace/scratch_integration_test.go +++ b/cmd/mxcli/marketplace/scratch_integration_test.go @@ -7,7 +7,9 @@ package marketplace import ( "context" "os" + "os/exec" "path/filepath" + "strings" "testing" "github.com/mendixlabs/mxcli/cmd/mxcli/docker" @@ -199,3 +201,80 @@ func TestPackageProject_ReferenceIsReproducibleAndDiffable(t *testing.T) { t.Errorf("expected exactly ENTITY Account=modified, got %v", changed) } } + +// TestTransplantModule_ProducesAProjectMxbuildAccepts is the check the unit +// tests cannot make. +// +// A transplant that round-trips through mxcli's own reader proves only that +// mxcli agrees with itself. Units are copied verbatim with their unit IDs and +// only the module unit is re-parented, so the failure mode to rule out is a +// model that reads back fine and that mxbuild rejects — a dangling container, a +// containment name Mendix does not expect, a duplicated identity. +// +// Measured on Mendix 11.12.1 with two blank projects: 28 units copied, and +// `mx check` reports "The app contains: 0 errors." +func TestTransplantModule_ProducesAProjectMxbuildAccepts(t *testing.T) { + const version = "11.12.1" + if !mxVersionAvailable(version) { + t.Skipf("mxbuild %s is not cached; run 'mxcli setup mxbuild --version %s'", version, version) + } + mxPath := docker.CachedMxPath(version) + + ctx := context.Background() + source := blankProject(t, ctx, mxPath, "S") + target := blankProject(t, ctx, mxPath, "T") + const module = "Administration" + + ids, err := CaptureIdentities(target, module) + if err != nil { + t.Fatalf("capture: %v", err) + } + execMDL(t, target, "drop module "+module+";") + + copied, err := TransplantModule(source, target, module) + if err != nil { + t.Fatalf("TransplantModule: %v", err) + } + if copied == 0 { + t.Fatal("no units copied") + } + if _, _, err := ApplyIdentities(target, module, ids); err != nil { + t.Fatalf("ApplyIdentities: %v", err) + } + + cmd := exec.CommandContext(ctx, mxPath, "check", "-p", target) + docker.PrepareMxCommand(cmd) + out, _ := cmd.CombinedOutput() + // mx check exits non-zero on any error, including the CE0462 widget errors a + // project without its widgets/ folder always produces, so the assertion is on + // the error count rather than the exit code. + if !strings.Contains(string(out), "The app contains: 0 errors") { + t.Errorf("mxbuild rejected the transplanted project:\n%s", tail(string(out), 30)) + } + t.Logf("%d units transplanted, mx check clean", copied) +} + +// blankProject creates a fresh project with mx and returns its .mpr path. +func blankProject(t *testing.T, ctx context.Context, mxPath, name string) string { + t.Helper() + dir := shortTemp(t) + cmd := exec.CommandContext(ctx, mxPath, "create-project", "--app-name", name) + cmd.Dir = dir + docker.PrepareMxCommand(cmd) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("create-project: %v\n%s", err, out) + } + mpr, err := findScratchMpr(dir, name) + if err != nil { + t.Fatal(err) + } + return mpr +} + +func tail(s string, lines int) string { + parts := strings.Split(strings.TrimSpace(s), "\n") + if len(parts) > lines { + parts = parts[len(parts)-lines:] + } + return strings.Join(parts, "\n") +} diff --git a/cmd/mxcli/marketplace/transplant.go b/cmd/mxcli/marketplace/transplant.go new file mode 100644 index 000000000..017e3bf0e --- /dev/null +++ b/cmd/mxcli/marketplace/transplant.go @@ -0,0 +1,182 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "fmt" + + "strings" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/model" +) + +// projectContainment is the containment name a module sits under on the project +// unit. Measured on a real project: every Projects$ModuleImpl hangs off the +// single Projects$Project unit under "Modules". +const projectContainment = "Modules" + +// TransplantModule copies a module and everything under it from one project into +// another, using mxcli's own writer. +// +// This is the import step of a module update, and it deliberately does not use +// `mx module-import`. That command rewrites an MPR v2 project as v1 — measured, +// and refused outright by `marketplace install` — and it also refuses theme +// modules. Copying the units directly avoids both, and keeps the destination in +// whatever format it already uses, because the writer handles v1 and v2 alike. +// +// Units are copied verbatim, including their unit IDs. That is sound because the +// destination's copy of the module has been removed first, so the IDs are free, +// and because no element `$ID` pointer crosses a unit boundary (§4). Only the +// module unit is re-parented, onto the destination's project unit. +// +// It does NOT preserve identities: the copied module carries the *package's* +// GUIDs. Pair it with CaptureIdentities before and ApplyIdentities after, or the +// update destroys the module's data on the next deploy (§8). +func TransplantModule(srcMpr, dstMpr, moduleName string) (copied int, err error) { + src, err := modelsdk.Open(srcMpr) + if err != nil { + return 0, fmt.Errorf("open source %s: %w", srcMpr, err) + } + defer src.Close() + + srcUnits, err := src.ListUnits() + if err != nil { + return 0, fmt.Errorf("list source units: %w", err) + } + moduleUnits, err := unitsOfModule(src, srcUnits, moduleName) + if err != nil { + return 0, err + } + + byID := make(map[string]*unitCopy, len(moduleUnits)) + order := make([]string, 0, len(moduleUnits)) + for _, u := range srcUnits { + id := string(u.ID) + if !containsStr(moduleUnits, id) { + continue + } + raw, rerr := src.GetRawUnitBytes(model.ID(id)) + if rerr != nil || len(raw) == 0 { + return 0, fmt.Errorf("read source unit %s: %w", id, rerr) + } + byID[id] = &unitCopy{ + id: id, + containerID: string(u.ContainerID), + containment: u.ContainmentName, + unitType: u.Type, + contents: raw, + } + order = append(order, id) + } + + moduleUnitID, err := moduleUnitIDOf(src, moduleName) + if err != nil { + return 0, err + } + + dstProjectID, err := projectUnitID(dstMpr) + if err != nil { + return 0, err + } + if err := refuseIfPresent(dstMpr, moduleName); err != nil { + return 0, err + } + + writer, err := modelsdk.OpenForWriting(dstMpr) + if err != nil { + return 0, fmt.Errorf("open destination %s for writing: %w", dstMpr, err) + } + defer writer.Close() + + for _, id := range order { + u := byID[id] + container := u.containerID + containment := u.containment + if id == moduleUnitID { + // The only re-parenting: the module attaches to the destination's + // project unit. Everything below it keeps its existing container, + // which is a unit being copied in the same pass. + container = dstProjectID + containment = projectContainment + } + if err := writer.AddRawUnit(id, container, containment, u.unitType, u.contents); err != nil { + return copied, fmt.Errorf("copy unit %s: %w", id, err) + } + copied++ + } + return copied, nil +} + +type unitCopy struct { + id, containerID, containment, unitType string + contents []byte +} + +func containsStr(haystack []string, needle string) bool { + for _, s := range haystack { + if s == needle { + return true + } + } + return false +} + +func moduleUnitIDOf(reader *modelsdk.Reader, moduleName string) (string, error) { + mods, err := reader.ListModules() + if err != nil { + return "", err + } + for _, m := range mods { + if equalFold(m.Name, moduleName) { + return string(m.ID), nil + } + } + return "", fmt.Errorf("module %q not found in the source project", moduleName) +} + +// projectUnitID finds the destination's Projects$Project unit — the container +// every module hangs off. +func projectUnitID(mprPath string) (string, error) { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return "", fmt.Errorf("open %s: %w", mprPath, err) + } + defer reader.Close() + + units, err := reader.ListUnits() + if err != nil { + return "", err + } + for _, u := range units { + if u.Type == "Projects$Project" { + return string(u.ID), nil + } + } + return "", fmt.Errorf("%s has no project unit to attach a module to", mprPath) +} + +// refuseIfPresent stops a transplant onto a module that is still there. Copying +// on top would leave two modules of the same name, which is a corrupt model +// rather than an update — the destination's copy must be removed first. +func refuseIfPresent(mprPath, moduleName string) error { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return fmt.Errorf("open %s: %w", mprPath, err) + } + defer reader.Close() + + mods, err := reader.ListModules() + if err != nil { + return err + } + for _, m := range mods { + if equalFold(m.Name, moduleName) { + return fmt.Errorf("module %q is still present in %s; remove it before transplanting, "+ + "or the project ends up with two modules of the same name", moduleName, mprPath) + } + } + return nil +} + +func equalFold(a, b string) bool { return strings.EqualFold(a, b) } diff --git a/cmd/mxcli/marketplace/transplant_test.go b/cmd/mxcli/marketplace/transplant_test.go new file mode 100644 index 000000000..95a85550e --- /dev/null +++ b/cmd/mxcli/marketplace/transplant_test.go @@ -0,0 +1,108 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "strings" + "testing" +) + +// TestTransplantModule_MovesTheWholeModuleAndPreservesIdentity is the shape of a +// real update, run end to end on the model: capture the identities, remove the +// module, copy a replacement in, transplant the identities back. +// +// The destination keeps its MPR v2 layout throughout, which is the whole reason +// this path exists rather than `mx module-import` — that command rewrites a v2 +// project as v1 and refuses theme modules outright. +func TestTransplantModule_MovesTheWholeModuleAndPreservesIdentity(t *testing.T) { + source := copyFixture(t) // stands in for the package's project + target := copyFixture(t) // stands in for the user's project + const module = "Administration" + + before, err := CaptureIdentities(target, module) + if err != nil { + t.Fatalf("capture: %v", err) + } + + // Make the source's identities differ, so a transplant that silently did + // nothing would be indistinguishable from success. + scrambled := Identities{} + for p, g := range before { + alt := append([]byte{}, g...) + alt[0] ^= 0xFF + scrambled[p] = alt + } + if _, _, err := ApplyIdentities(source, module, scrambled); err != nil { + t.Fatalf("scramble source: %v", err) + } + + execMDL(t, target, "drop module "+module+";") + + copied, err := TransplantModule(source, target, module) + if err != nil { + t.Fatalf("TransplantModule: %v", err) + } + if copied < 3 { + t.Fatalf("copied only %d units; a module carries at least its own unit, a domain model and security", copied) + } + + // The module is readable in the target, and carries the package's identities. + moved, err := CaptureIdentities(target, module) + if err != nil { + t.Fatalf("capture after transplant: %v", err) + } + if len(moved) != len(before) { + t.Errorf("transplanted module has %d identities, source had %d", len(moved), len(before)) + } + if bytesEqual(moved["Account"], before["Account"]) { + t.Fatal("the transplanted module already carries the target's old identity; " + + "the restore below would prove nothing") + } + + // Now the step that makes an update data-safe. + applied, missing, err := ApplyIdentities(target, module, before) + if err != nil { + t.Fatalf("ApplyIdentities: %v", err) + } + if len(missing) != 0 { + t.Errorf("nothing should be missing when the versions match; got %v", missing) + } + if applied != len(before) { + t.Errorf("applied %d of %d identities", applied, len(before)) + } + + after, err := CaptureIdentities(target, module) + if err != nil { + t.Fatalf("capture after restore: %v", err) + } + for _, p := range before.Paths() { + if !bytesEqual(after[p], before[p]) { + t.Errorf("%s: identity not preserved across the update (%x vs %x)", + p, after[p], before[p]) + } + } +} + +// TestTransplantModule_RefusesWhenTheModuleIsStillThere — copying on top of a +// live module leaves two of the same name, which is a corrupt model rather than +// an update. The caller has to remove it first, deliberately. +func TestTransplantModule_RefusesWhenTheModuleIsStillThere(t *testing.T) { + source := copyFixture(t) + target := copyFixture(t) // still has Administration + + _, err := TransplantModule(source, target, "Administration") + if err == nil { + t.Fatal("transplanting onto a module that is still present must be refused") + } + if !strings.Contains(err.Error(), "still present") { + t.Errorf("the error should say why; got: %v", err) + } +} + +// TestTransplantModule_RefusesAnUnknownSourceModule keeps a typo from producing +// an empty, silent success. +func TestTransplantModule_RefusesAnUnknownSourceModule(t *testing.T) { + if _, err := TransplantModule(copyFixture(t), copyFixture(t), "NoSuchModule"); err == nil { + t.Fatal("a module that is not in the source must be an error") + } +} diff --git a/sdk/mpr/writer_units.go b/sdk/mpr/writer_units.go index d9e26cd85..69c7f3336 100644 --- a/sdk/mpr/writer_units.go +++ b/sdk/mpr/writer_units.go @@ -198,6 +198,21 @@ func (w *Writer) updateUnit(unitID string, contents []byte) error { // UpdateRawUnit saves raw BSON bytes for a unit, bypassing deserialization. // Used by ALTER PAGE to modify the BSON widget tree directly. +// AddRawUnit inserts a unit verbatim: same contents, same containment name, no +// re-encoding. It is the primitive a module transplant is built from. +// +// Copying a unit wholesale is safe because element `$ID` pointers do not cross +// unit boundaries — measured at 0 of 9,910 in a real project (PROPOSAL +// marketplace_module_upgrade §4) — and cross-unit references are qualified-name +// strings. Rewriting the contents, by contrast, would risk exactly the +// intra-unit pointer inconsistency ADR-0008 forbids. +// +// The caller owns uniqueness of unitID. Inserting an ID the project already +// holds is a caller error, not something this can repair. +func (w *Writer) AddRawUnit(unitID, containerID, containmentName, unitType string, contents []byte) error { + return w.insertUnit(unitID, containerID, containmentName, unitType, contents) +} + func (w *Writer) UpdateRawUnit(unitID string, contents []byte) error { return w.updateUnit(unitID, contents) } From b1a138c117aaa90cb43f15c9788fc3d079db3c4a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 22:34:07 +0000 Subject: [PATCH 24/35] feat(queues): author task queues in MDL, and refuse the rewrite that dropped them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A microflow call bound to a task queue in Studio Pro lost that binding on `create or replace microflow` — silently. Worse, `mx check` went from [CE1613] "The selected task queue 'X' no longer exists." to 0 errors, because mxcli had deleted the configuration the error was about. `describe microflow` never showed the binding either, so the loss was invisible from every angle: the write reported success, the describe looked complete, and the build looked healthier than before. Two defects behind one report, fixed here: 1. MDL had no queue surface at all, so a script could not restate a binding even in principle. CREATE [OR MODIFY] / DROP / SHOW / DESCRIBE QUEUE now exist, wired through grammar → AST → visitor → executor → backend on both engines. The BSON is pinned against the four Studio Pro-authored queues in Mendix Business Events 3.12.1, which agree exactly: Config is a Queues$BasicQueueConfig whose ParallelismExpression is a STRING, with the sibling int32 `Parallelism` absent in all four. Parallelism is therefore an expression everywhere in MDL, not a number. 2. Rewriting a microflow whose stored calls are queued is now refused, naming the queues that would be lost. Binding a call to a queue is deliberately still unimplemented — the Retry shape has no Studio Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. Refusing is the only option that does not lose data (ADR-0005). QUEUE/QUEUES are added to the `keyword` rule, so `queue` remains usable as an ordinary attribute name. Verified on Mendix 11.13: the example script leaves a clean project at 0 errors under both engines, and describe → exec round-trips. The guard's test fails (the rewrite runs, the binding goes) when its call site is disabled. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/lsp_completions_gen.go | 2 + cmd/mxcli/syntax/features_misc.go | 46 ++++ docs/01-project/MDL_QUICK_REFERENCE.md | 28 ++ .../queue-authoring-and-rewrite-guard.mdl | 61 +++++ mdl/ast/ast_queue.go | 42 +++ mdl/backend/backend.go | 1 + mdl/backend/infrastructure.go | 8 + mdl/backend/mcp/unsupported_gen.go | 20 ++ mdl/backend/mock/backend.go | 6 + mdl/backend/mock/mock_queue.go | 43 +++ mdl/backend/modelsdk/queue_write.go | 123 +++++++++ mdl/backend/modelsdk/queue_write_test.go | 129 +++++++++ mdl/backend/modelsdk/unimplemented_gen.go | 17 ++ mdl/backend/mpr/backend.go | 17 ++ mdl/executor/cmd_microflows_create.go | 8 + mdl/executor/cmd_queues.go | 183 +++++++++++++ mdl/executor/cmd_queues_mock_test.go | 247 ++++++++++++++++++ mdl/executor/register_stubs.go | 15 ++ mdl/executor/registry.go | 1 + mdl/executor/registry_test.go | 4 + mdl/executor/validate_queued_calls.go | 103 ++++++++ mdl/executor/validate_queued_calls_test.go | 160 ++++++++++++ mdl/grammar/MDLLexer.g4 | 2 + mdl/grammar/MDLParser.g4 | 2 + mdl/grammar/domains/MDLCatalog.g4 | 2 + mdl/grammar/domains/MDLDomainModel.g4 | 23 ++ mdl/grammar/domains/MDLSettings.g4 | 2 +- mdl/types/queue.go | 49 ++++ mdl/visitor/visitor_entity.go | 4 + mdl/visitor/visitor_query.go | 18 ++ mdl/visitor/visitor_queue.go | 72 +++++ mdl/visitor/visitor_queue_test.go | 127 +++++++++ sdk/mpr/queues.go | 109 ++++++++ 34 files changed, 1674 insertions(+), 1 deletion(-) create mode 100644 mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl create mode 100644 mdl/ast/ast_queue.go create mode 100644 mdl/backend/mock/mock_queue.go create mode 100644 mdl/backend/modelsdk/queue_write.go create mode 100644 mdl/backend/modelsdk/queue_write_test.go create mode 100644 mdl/executor/cmd_queues.go create mode 100644 mdl/executor/cmd_queues_mock_test.go create mode 100644 mdl/executor/validate_queued_calls.go create mode 100644 mdl/executor/validate_queued_calls_test.go create mode 100644 mdl/types/queue.go create mode 100644 mdl/visitor/visitor_queue.go create mode 100644 mdl/visitor/visitor_queue_test.go create mode 100644 sdk/mpr/queues.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index f75d4e0bc..dd3c1dd49 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -472,3 +472,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A datagrid column bound to an association — `column c (attribute: Order_Customer)` — reports success from `mxcli exec` and then fails the build with `[error] [CE1613] "The selected attribute 'Mod.Order.Order_Customer' no longer exists." at Columns (1/1) of data grid 2`. Separately, there is no MDL spelling for the drop-down filter's association mode: `mxcli check` says `[MDL-WIDGET01] has no property \`refEntity\`` | Two unrelated defects behind one report. (1) The reference is **not representable**: `CustomWidgets$WidgetValue.AttributeRef` is typed `AttributeRef`, not the polymorphic `MemberRef`, so the association was qualified like an attribute and written as a dangling `AttributeRef`. (2) `dropdownfilter.def.json` mapped only `attrChoice`/`attributes`/`defaultFilter`, so every `baseType: 'ref'` property was unmapped and dropped | `mdl/executor/cmd_pages_builder_input.go` (`rejectAssociationAsAttribute`, `entityInChain`) wired into `mdl/executor/widget_engine.go` (the objectlist `attribute` case and the `Attribute` source); `sdk/widgets/definitions/dropdownfilter.def.json` (association mode); `mdl/executor/cmd_pages_describe_parse.go` + `_pluggable.go` + `_output.go` (round-trip) | **Establish that a shape is unrepresentable before designing a fix for it** — hand-patch the BSON and run `mx check`. A `DomainModels$AssociationRef` in that slot makes the project **UNLOADABLE** (`ArgumentException: Object of type 'AssociationRef' cannot be converted to type 'AttributeRef'`), and the assembly defining the type (`Mendix.Modeler.WebUI.dll`) has no `AssociationRef` member at all — so the only correct outcome is a refusal carrying both working forms. **`` on an ATTRIBUTE-typed widget property is permission to TRAVERSE a reference, not to bind one** — `attribute: Assoc/Attr` already worked and is what the XML is advertising; the DataGrid column is the only shipped widget where the two are easy to confuse. **A def.json `mode` is the whole feature** for an unauthorable widget mode — the engine already had the `association` operation and the `hasDataSource` condition, so the second half was a data change plus its DESCRIBE reader (without which describe→edit→exec silently reverts the filter to attribute mode). **0 errors from `mx check` does not prove the properties landed** — an unmapped property is silently dropped and the build is just as green; read them back with `mx dump-mpr`. Tests `cmd_pages_builder_assoc_as_attribute_test.go`, `widget_dropdownfilter_assoc_test.go`, example `mdl-examples/bug-tests/830-datagrid-association-filter.mdl`. upstream #830 | | An association's line anchors — where the connector attaches to the entity boxes in the domain model editor — are absent from `DESCRIBE ASSOCIATION`, and manual adjustments made in Studio Pro do not survive an mxcli round trip | `DomainModels$Association.ParentConnection`/`ChildConnection` (the string `"x;y"`) were **hardcoded** to `"0;50"`/`"100;50"` in BOTH writers and never read by either parser. Because every association write rebuilds the whole element, this was not an omission but active destruction: a documentation-only `alter association … set comment` reset them | `sdk/domainmodel/connection.go` (new: `ParseConnectionPoint`/`FormatConnectionPoint`, `Default*Connection`), `sdk/domainmodel/domainmodel.go` (fields → `*model.Point`), `sdk/mpr/parser_domainmodel.go` + `sdk/mpr/writer_domainmodel.go`, `mdl/backend/modelsdk/domainmodel.go` + `domainmodel_write.go`, `mdl/executor/cmd_associations.go` (`describeConnectionPoints`) | **A feature request that says "X is not exposed" may be hiding "X is destroyed"** — check the write path before scoping the read path. The A/B that settled it: a blank 11.13 app's own `Administration.AccountPasswordData_Account` stores `0;54/100;54`, so a Studio-Pro-authored association is a free fixture for "did mxcli overwrite this?" — no Studio Pro needed. **Learn the value's constraints from the LOADER, not from the shape**: hand-patch and run `mx check` — `"0.5;50"` dies with `StorageLoadException` (integers required) while `"0;500"` and `"-20;50"` load with 0 errors (no range check), so out-of-range values must round-trip untouched. **A zero value is not an absent value** — `{0,0}` is a real anchor (top-left), which forces the field to be a POINTER; a plain `model.Point` cannot distinguish "unset" from "top-left" and would silently rewrite it. **Fix both engines**: they share the semantic model, and a fix in one is invisible to a user on the other. **Emit unauthorable data as a COMMENT** — DESCRIBE output must stay re-executable, and inventing syntax (`@anchor(parent: bottom-left, …)`) would bake in a vocabulary the storage does not have: the pair is CONTINUOUS, not 8 named anchors (observed x values 0 9 11 17 18 47 49 50 65 77 78 84 87 100). **The marketplace is the sample** when you need to know what Studio Pro actually writes: `mxcli marketplace download ` gives real Mendix-authored models, and a module .mpk holds either a raw BSON `project.mpr` or an MPR v1 SQLite one — 88 coordinate pairs from three modules turned "looks like percentages" into a measurement (all 0..100; 85 of 88 pin one coordinate to exactly 0 or 100). **Rule a unit out from the model, not the values**: pixels is impossible because `DomainModels$EntityImpl` stores only `Location` and NO size — the box is sized by the editor from the name and attribute list, so a pixel anchor would have nothing to measure against. Not applicable to `CrossAssociation`, which has no connection properties and crashes Studio Pro if given them (#50). Tests `sdk/domainmodel/connection_test.go`, `mdl/backend/modelsdk/association_connection_test.go`, `sdk/mpr/writer_domainmodel_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | +| A microflow call bound to a task queue in Studio Pro loses that binding on `create or replace microflow` — silently, and `mx check` goes from `[CE1613] "The selected task queue no longer exists"` to **0 errors**, because the configuration the error was about has been deleted | Two defects behind one report. (1) `Microflows$MicroflowCall` has `QueueSettings` in `NullFields` of `codec.RegisterTypeDefaults` (and the legacy writer hardcodes null too) — correct for a newly authored call, destructive for a stored one, and since a CREATE OR REPLACE rebuilds the whole microflow it fires on every rewrite. (2) MDL had **no queue surface at all**, so a script could not restate the binding even in principle | `mdl/executor/validate_queued_calls.go` (new: the refusal), `mdl/executor/cmd_microflows_create.go` (call site), `mdl/types/queue.go` + `mdl/backend/infrastructure.go` (`QueueBackend`), `mdl/backend/modelsdk/queue_write.go` + `sdk/mpr/queues.go` (both engines), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4`, `mdl/executor/cmd_queues.go` | **A green build can be the bug**: the fix made `mx check` report MORE errors than before (CE1613 came back), because the binding it complains about now survives. Any "error count went down" check would have scored the data loss as a fix. **`describe` showing nothing is not evidence of nothing** — the binding was invisible from every angle (describe omitted it, check went quiet, the write reported success), which is why it needed a stored-BSON probe: inject `Queue`+`QueueSettings` into a stored call, rewrite, and dump. **Isolate which property Mendix acts on before guarding on it**: a call carrying only `Queue` (with `QueueSettings` null) draws NO complaint from `mx check` — `QueueSettings` is the load-bearing one, so a guard keyed on `Queue` alone would have both missed the real case and fired on inert ones. **Get the BSON shape from a Mendix-authored model, not from the metamodel**: `mxcli marketplace download 202649` (Business Events 3.12.1) has four real queues, and all four agree — `Config` is a `Queues$BasicQueueConfig` whose `ParallelismExpression` is a **STRING**, with the sibling int32 `Parallelism` absent in every one. Writing the int (which the metamodel makes look equally valid) is inventing a property Mendix does not write. **Refuse, do not half-author** (ADR-0005): `in queue` on a call is deliberately still unimplemented, because the `Retry` shape has no Studio-Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. **Adding a keyword costs an identifier** — `QUEUE`/`QUEUES` had to go into the `keyword` rule or `queue` would have stopped working as an attribute name. Tests `mdl/backend/modelsdk/queue_write_test.go`, `mdl/executor/validate_queued_calls_test.go`, `mdl/executor/cmd_queues_mock_test.go`, `mdl/visitor/visitor_queue_test.go`, example `mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl` | diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index f9b9b8564..7016b7d5d 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -266,6 +266,8 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "ATTRIBUTES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "FILTERTYPE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "IMAGE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "QUEUE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "QUEUES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "JAR", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCY", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCIES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 70371f13b..191cbcfac 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -207,6 +207,52 @@ CREATE CONFIGURATION 'Production' SeeAlso: []string{"settings.show"}, }) + // ── Queues ────────────────────────────────────────────────────────── + + Register(SyntaxFeature{ + Path: "queue", + Summary: "Task queues — bound concurrency for queued microflow calls", + Keywords: []string{ + "queue", "queues", "task queue", "create queue", "drop queue", + "describe queue", "show queues", "parallelism", "cluster wide", + "background", "async microflow", + }, + Syntax: `CREATE [OR MODIFY] QUEUE Module.Name [( : , ... )]; +SHOW QUEUES [IN ]; +LIST QUEUES [IN ]; +DESCRIBE QUEUE Module.Name; +DROP QUEUE Module.Name; + +Properties: + Parallelism how many tasks run at once. This is an EXPRESSION, not a + number — Mendix stores it as a string. A bare integer is the + common case; quote anything else. Defaults to 1. + ClusterWide true = the limit applies across the cluster, false (default) + = per runtime instance. + Documentation free text. + +Binding a call to a queue is not yet expressible in MDL. Because a rebuild +would drop an existing binding, mxcli REFUSES to CREATE OR REPLACE/MODIFY a +microflow whose stored calls are queued — change those in Studio Pro.`, + Example: `CREATE QUEUE Ops.OrderProcessing ( + Parallelism: 3, + ClusterWide: true +); + +-- Defaults: parallelism 1, per-instance. +CREATE QUEUE Ops.Mail; + +-- An expression is legal wherever a number is. +CREATE OR MODIFY QUEUE Ops.OrderProcessing ( + Parallelism: '$MyModule.Workers', + ClusterWide: true +); + +SHOW QUEUES IN Ops; +DESCRIBE QUEUE Ops.OrderProcessing; +DROP QUEUE Ops.Mail;`, + }) + // ── Structure ─────────────────────────────────────────────────────── Register(SyntaxFeature{ diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index d4cb8d818..1aded5443 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -138,6 +138,34 @@ create constant MyModule.MaxRetries type integer default 3; create constant MyModule.EnableLogging type boolean default true; ``` +## Task Queues + +| Statement | Syntax | Notes | +|-----------|--------|-------| +| Show queues | `show queues [in module];` (`list queues` too) | Parallelism + cluster-wide flag | +| Describe queue | `describe queue Module.Name;` | Re-executable MDL | +| Create queue | `create [or modify] queue Module.Name ( Parallelism: 3, ClusterWide: true );` | Body optional; defaults `1` / `false` | +| Drop queue | `drop queue Module.Name;` | | + +`Parallelism` is an **expression**, not a number — Mendix stores it as a string +(`Queues$BasicQueueConfig.ParallelismExpression`). A bare integer is the common +case; quote anything else. + +Binding a microflow **call** to a queue is not yet expressible in MDL. Because a +rebuild would drop an existing binding, `create or replace|modify microflow` is +**refused** when the stored microflow has a queued call — change those in Studio +Pro. (Without the refusal the binding was written back as null and `mx check` +stopped reporting CE1613, so the project looked healthy while the configuration +was gone.) + +**Example:** +```sql +create queue Ops.OrderProcessing ( Parallelism: 3, ClusterWide: true ); +create queue Ops.Mail; +create or modify queue Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); +drop queue Ops.Mail; +``` + ## OData Clients, Services & External Entities | Statement | Syntax | Notes | diff --git a/mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl b/mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl new file mode 100644 index 000000000..5887a3354 --- /dev/null +++ b/mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl @@ -0,0 +1,61 @@ +-- ============================================================================ +-- Task queues: authorable in MDL, and never silently dropped by a rewrite +-- ============================================================================ +-- +-- Symptom (measured on Mendix 11.13): +-- A microflow call bound to a task queue in Studio Pro stores a +-- Queues$QueueSettings node on the call. `create or replace microflow` rebuilt +-- the microflow and both engines wrote `QueueSettings: null` — the binding was +-- gone. Nothing reported it. Worse, `mx check` went from +-- [CE1613] "The selected task queue no longer exists" +-- to 0 errors, because mxcli had deleted the configuration that the error was +-- about. `describe microflow` never showed the binding either, so the loss was +-- invisible from every angle. +-- +-- Two root causes, two fixes: +-- 1. MDL could not author a queue at all, so a script could not restate one. +-- CREATE/DROP/SHOW/DESCRIBE QUEUE now exist. The BSON is pinned against the +-- four Studio Pro-authored queues in Mendix Business Events 3.12.1 +-- (Consumer_Queue, Consumer_Processor_Queue, Producer_Queue, +-- Outbox_Cleanup_Queue), which agree exactly: Config is a +-- Queues$BasicQueueConfig carrying ParallelismExpression as a STRING, with +-- the int32 `Parallelism` absent in all four. +-- 2. MDL still cannot author a *queued call*. Until it can, rewriting a +-- microflow that has one is refused rather than performed lossily +-- (guard-don't-drop, ADR-0005). +-- +-- Verify: +-- mxcli exec mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl -p app.mpr +-- mx check app.mpr # 0 errors +-- mxcli -p app.mpr -c "list queues" +-- mxcli -p app.mpr -c "describe queue Ops.OrderProcessing" # re-executable MDL +-- ============================================================================ + +create module Ops; +/ +-- Parallelism is an EXPRESSION, not a number. A bare integer is the common case; +-- quote anything else. +create queue Ops.OrderProcessing ( + Parallelism: 3, + ClusterWide: true +); +/ +-- No properties: the defaults ("1", not cluster-wide) match a queue created in +-- Studio Pro with nothing changed. +create queue Ops.Mail; +/ +-- Re-running is idempotent with OR MODIFY, and the stored $ID is reused so any +-- call already bound to the queue keeps pointing at it. +create or modify queue Ops.OrderProcessing ( + Parallelism: 8, + ClusterWide: true +); +/ +-- "queue" is a plausible attribute name; adding the keyword must not take it +-- away (it is listed in the `keyword` rule, so it still parses as an identifier). +create entity Ops.Job ( + queue: String(100), + Status: String(50) +); +/ +drop queue Ops.Mail; diff --git a/mdl/ast/ast_queue.go b/mdl/ast/ast_queue.go new file mode 100644 index 000000000..1ad13ee0b --- /dev/null +++ b/mdl/ast/ast_queue.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package ast + +// CreateQueueStmt represents: +// +// CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( Parallelism: 3, ClusterWide: true ); +type CreateQueueStmt struct { + Name QualifiedName + Documentation string + // Parallelism is kept as written. Mendix stores it as an expression string + // (Queues$BasicQueueConfig.ParallelismExpression), so `3` and `'3'` are the + // same thing and an arbitrary expression is legal. + Parallelism string + ClusterWide bool + ExportLevel string + // CreateOrModify is set by the shared CREATE OR REPLACE/MODIFY prefix. + CreateOrModify bool +} + +func (s *CreateQueueStmt) isStatement() {} + +// DropQueueStmt represents: DROP QUEUE Module.Name; +type DropQueueStmt struct { + Name QualifiedName +} + +func (s *DropQueueStmt) isStatement() {} + +// ShowQueuesStmt represents: SHOW|LIST QUEUES [IN Module]; +type ShowQueuesStmt struct { + Module string +} + +func (s *ShowQueuesStmt) isStatement() {} + +// DescribeQueueStmt represents: DESCRIBE QUEUE Module.Name; +type DescribeQueueStmt struct { + Name QualifiedName +} + +func (s *DescribeQueueStmt) isStatement() {} diff --git a/mdl/backend/backend.go b/mdl/backend/backend.go index 69bf47719..c5ebf95d2 100644 --- a/mdl/backend/backend.go +++ b/mdl/backend/backend.go @@ -26,6 +26,7 @@ type FullBackend interface { WorkflowBackend SettingsBackend ImageBackend + QueueBackend ScheduledEventBackend RenameBackend RawUnitBackend diff --git a/mdl/backend/infrastructure.go b/mdl/backend/infrastructure.go index 254b6366e..9f3f822ca 100644 --- a/mdl/backend/infrastructure.go +++ b/mdl/backend/infrastructure.go @@ -84,6 +84,14 @@ type ImageBackend interface { ListIconCollections() ([]*types.IconCollection, error) } +// QueueBackend provides task queue (Queues$Queue) operations. +type QueueBackend interface { + ListQueues() ([]*types.Queue, error) + CreateQueue(q *types.Queue) error + UpdateQueue(q *types.Queue) error + DeleteQueue(id string) error +} + // ScheduledEventBackend provides scheduled event operations. type ScheduledEventBackend interface { ListScheduledEvents() ([]*model.ScheduledEvent, error) diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index 637e5e5d6..5fbba52d6 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -214,6 +214,11 @@ func (unsupportedBackend) CreatePublishedRestService(_ *model.PublishedRestServi return } +func (unsupportedBackend) CreateQueue(_ *types.Queue) (err0 error) { + err0 = errUnsupported("CreateQueue") + return +} + func (unsupportedBackend) CreateSnippet(_ *pages.Snippet) (err0 error) { err0 = errUnsupported("CreateSnippet") return @@ -389,6 +394,11 @@ func (unsupportedBackend) DeletePublishedRestService(_ model.ID) (err0 error) { return } +func (unsupportedBackend) DeleteQueue(_ string) (err0 error) { + err0 = errUnsupported("DeleteQueue") + return +} + func (unsupportedBackend) DeleteSnippet(_ model.ID) (err0 error) { err0 = errUnsupported("DeleteSnippet") return @@ -757,6 +767,11 @@ func (unsupportedBackend) ListPublishedRestServices() (r0 []*model.PublishedRest return } +func (unsupportedBackend) ListQueues() (r0 []*types.Queue, err1 error) { + err1 = errUnsupported("ListQueues") + return +} + func (unsupportedBackend) ListRawUnits(_ string) (r0 []*types.RawUnitInfo, err1 error) { err1 = errUnsupported("ListRawUnits") return @@ -1182,6 +1197,11 @@ func (unsupportedBackend) UpdateQualifiedNameInAllUnits(_ string, _ string) (r0 return } +func (unsupportedBackend) UpdateQueue(_ *types.Queue) (err0 error) { + err0 = errUnsupported("UpdateQueue") + return +} + func (unsupportedBackend) UpdateRawUnit(_ string, _ []uint8) (err0 error) { err0 = errUnsupported("UpdateRawUnit") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index b2d9e8c62..3af35404e 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -250,6 +250,12 @@ type MockBackend struct { UpdateImageCollectionFunc func(ic *types.ImageCollection) error DeleteImageCollectionFunc func(id string) error + // QueueBackend + ListQueuesFunc func() ([]*types.Queue, error) + CreateQueueFunc func(q *types.Queue) error + UpdateQueueFunc func(q *types.Queue) error + DeleteQueueFunc func(id string) error + // ScheduledEventBackend ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) GetScheduledEventFunc func(id model.ID) (*model.ScheduledEvent, error) diff --git a/mdl/backend/mock/mock_queue.go b/mdl/backend/mock/mock_queue.go new file mode 100644 index 000000000..e57de049f --- /dev/null +++ b/mdl/backend/mock/mock_queue.go @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mock + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// QueueBackend — task queues (Queues$Queue). +// +// The write methods default to a descriptive error rather than nil so a test +// that forgets to configure one fails on the missing stub instead of silently +// reporting a write that never happened. + +func (m *MockBackend) ListQueues() ([]*types.Queue, error) { + if m.ListQueuesFunc != nil { + return m.ListQueuesFunc() + } + return nil, nil +} + +func (m *MockBackend) CreateQueue(q *types.Queue) error { + if m.CreateQueueFunc != nil { + return m.CreateQueueFunc(q) + } + return fmt.Errorf("MockBackend.CreateQueue not configured") +} + +func (m *MockBackend) UpdateQueue(q *types.Queue) error { + if m.UpdateQueueFunc != nil { + return m.UpdateQueueFunc(q) + } + return fmt.Errorf("MockBackend.UpdateQueue not configured") +} + +func (m *MockBackend) DeleteQueue(id string) error { + if m.DeleteQueueFunc != nil { + return m.DeleteQueueFunc(id) + } + return fmt.Errorf("MockBackend.DeleteQueue not configured") +} diff --git a/mdl/backend/modelsdk/queue_write.go b/mdl/backend/modelsdk/queue_write.go new file mode 100644 index 000000000..9f31b5b93 --- /dev/null +++ b/mdl/backend/modelsdk/queue_write.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/bsonutil" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// queueTypeName is the BSON storage name for a task queue document. +const queueTypeName = "Queues$Queue" + +// ListQueues reads every Queues$Queue unit. +func (b *Backend) ListQueues() ([]*types.Queue, error) { + units, err := b.reader.ListRawUnitsByType(queueTypeName) + if err != nil { + return nil, err + } + out := make([]*types.Queue, 0, len(units)) + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + return nil, fmt.Errorf("unmarshal queue %s: %w", u.ID, err) + } + q := &types.Queue{ContainerID: model.ID(u.ContainerID)} + q.ID = model.ID(u.ID) + q.TypeName = queueTypeName + q.Name, _ = doc["Name"].(string) + q.Documentation, _ = doc["Documentation"].(string) + q.Excluded, _ = doc["Excluded"].(bool) + q.ExportLevel, _ = doc["ExportLevel"].(string) + // Parallelism and cluster scope live on the nested Config node. + if cfg, ok := doc["Config"].(bson.M); ok { + q.Parallelism, _ = cfg["ParallelismExpression"].(string) + q.ClusterWide, _ = cfg["ClusterWide"].(bool) + } + out = append(out, q) + } + return out, nil +} + +// CreateQueue inserts a new Queues$Queue document. +func (b *Backend) CreateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("CreateQueue: nil queue") + } + if b.writer == nil { + return fmt.Errorf("CreateQueue: not connected for writing") + } + if q.ID == "" { + q.ID = model.ID(mmpr.GenerateID()) + } + return b.writer.InsertUnit(string(q.ID), string(q.ContainerID), "Documents", queueTypeName, serializeQueue(q)) +} + +// UpdateQueue rewrites an existing queue in place. +func (b *Backend) UpdateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("UpdateQueue: nil queue") + } + if b.writer == nil { + return fmt.Errorf("UpdateQueue: not connected for writing") + } + return b.writer.UpdateRawUnit(string(q.ID), serializeQueue(q)) +} + +// DeleteQueue removes a queue unit by ID. +func (b *Backend) DeleteQueue(id string) error { + if b.writer == nil { + return fmt.Errorf("DeleteQueue: not connected for writing") + } + return b.writer.DeleteUnit(id) +} + +// serializeQueue writes the document in the shape Studio Pro produces. +// +// Keys are alphabetical, matching every other writer here and the observed +// documents. Two things are deliberate: +// +// - ParallelismExpression is a STRING. Queues$BasicQueueConfig also declares an +// int32 Parallelism, but Studio Pro wrote it in none of the four reference +// documents, so writing it would be inventing a property. +// - An empty Parallelism becomes "1", which is what every observed queue has +// and what Mendix treats as the default; an empty expression is not a +// meaningful queue configuration. +func serializeQueue(q *types.Queue) []byte { + parallelism := q.Parallelism + if parallelism == "" { + parallelism = "1" + } + exportLevel := q.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + config := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(mmpr.GenerateID())}, + {Key: "$Type", Value: "Queues$BasicQueueConfig"}, + {Key: "ClusterWide", Value: q.ClusterWide}, + {Key: "ParallelismExpression", Value: parallelism}, + } + doc := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(string(q.ID))}, + {Key: "$Type", Value: queueTypeName}, + {Key: "Config", Value: config}, + {Key: "Documentation", Value: q.Documentation}, + {Key: "Excluded", Value: q.Excluded}, + {Key: "ExportLevel", Value: exportLevel}, + {Key: "Name", Value: q.Name}, + } + out, err := bson.Marshal(doc) + if err != nil { + // bson.Marshal on a fixed bson.D of primitives cannot fail; a nil return + // would be written as an empty unit, so surface it loudly instead. + panic(fmt.Sprintf("serializeQueue: marshal %q: %v", q.Name, err)) + } + return out +} diff --git a/mdl/backend/modelsdk/queue_write_test.go b/mdl/backend/modelsdk/queue_write_test.go new file mode 100644 index 000000000..29064256f --- /dev/null +++ b/mdl/backend/modelsdk/queue_write_test.go @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// TestSerializeQueue_MatchesStudioProShape pins the document against four real +// Studio Pro-authored queues from the Mendix Business Events module +// (Consumer_Queue, Consumer_Processor_Queue, Producer_Queue, +// Outbox_Cleanup_Queue), which agree exactly on this shape. +// +// Two assertions are the whole point of the test: +// +// - ParallelismExpression is a STRING. Queues$BasicQueueConfig also declares an +// int32 `Parallelism`, and writing that instead (or as well) would be +// inventing a property Mendix does not write. +// - `Parallelism` must be ABSENT. It appeared in none of the four references. +func TestSerializeQueue_MatchesStudioProShape(t *testing.T) { + q := &types.Queue{Name: "OrderProcessing", Parallelism: "3", ClusterWide: true} + q.ID = "11111111-1111-1111-1111-111111111111" + + var doc bson.M + if err := bson.Unmarshal(serializeQueue(q), &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + + if got := doc["$Type"]; got != "Queues$Queue" { + t.Errorf("$Type = %v, want Queues$Queue", got) + } + if got := doc["Name"]; got != "OrderProcessing" { + t.Errorf("Name = %v", got) + } + if got := doc["ExportLevel"]; got != "Hidden" { + t.Errorf("ExportLevel = %v, want Hidden (every reference document uses it)", got) + } + if _, ok := doc["Excluded"].(bool); !ok { + t.Errorf("Excluded missing or not a bool: %v", doc["Excluded"]) + } + + cfg, ok := doc["Config"].(bson.M) + if !ok { + t.Fatalf("Config is not a document: %T", doc["Config"]) + } + if got := cfg["$Type"]; got != "Queues$BasicQueueConfig" { + t.Errorf("Config.$Type = %v", got) + } + if got, ok := cfg["ParallelismExpression"].(string); !ok || got != "3" { + t.Errorf("ParallelismExpression = %#v, want the string \"3\" — Mendix stores an expression, not a number", cfg["ParallelismExpression"]) + } + if _, present := cfg["Parallelism"]; present { + t.Error("Parallelism must not be written: Studio Pro wrote it in none of the four reference queues") + } + if got := cfg["ClusterWide"]; got != true { + t.Errorf("ClusterWide = %v, want true", got) + } +} + +// TestSerializeQueue_Defaults covers the empty case: every observed queue has a +// parallelism, so an unspecified one becomes "1" rather than an empty +// expression, which is not a meaningful configuration. +func TestSerializeQueue_Defaults(t *testing.T) { + q := &types.Queue{Name: "Plain"} + q.ID = "22222222-2222-2222-2222-222222222222" + + var doc bson.M + if err := bson.Unmarshal(serializeQueue(q), &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + cfg := doc["Config"].(bson.M) + if got := cfg["ParallelismExpression"]; got != "1" { + t.Errorf("default ParallelismExpression = %v, want \"1\"", got) + } + if got := cfg["ClusterWide"]; got != false { + t.Errorf("default ClusterWide = %v, want false", got) + } + if got := doc["ExportLevel"]; got != "Hidden" { + t.Errorf("default ExportLevel = %v, want Hidden", got) + } +} + +// TestQueueRoundTrip writes a queue and reads it back through the real backend. +func TestQueueRoundTrip(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + if err := b.CreateQueue(&types.Queue{ + ContainerID: mod.ID, Name: "ZzQueue", Parallelism: "5", ClusterWide: true, + }); err != nil { + t.Fatalf("CreateQueue: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + queues, err := b2.ListQueues() + if err != nil { + t.Fatalf("ListQueues: %v", err) + } + for _, q := range queues { + if q.Name != "ZzQueue" { + continue + } + if q.Parallelism != "5" { + t.Errorf("Parallelism = %q, want 5", q.Parallelism) + } + if !q.ClusterWide { + t.Error("ClusterWide did not round-trip") + } + return + } + t.Fatalf("ZzQueue not found after create (got %d queues)", len(queues)) +} diff --git a/mdl/backend/modelsdk/unimplemented_gen.go b/mdl/backend/modelsdk/unimplemented_gen.go index ce24b4208..62cea55be 100644 --- a/mdl/backend/modelsdk/unimplemented_gen.go +++ b/mdl/backend/modelsdk/unimplemented_gen.go @@ -180,6 +180,10 @@ func (unimplemented) CreatePublishedRestService(_ *model.PublishedRestService) e return errUnimplemented("CreatePublishedRestService") } +func (unimplemented) CreateQueue(_ *types.Queue) error { + return errUnimplemented("CreateQueue") +} + func (unimplemented) CreateSnippet(_ *pages.Snippet) error { return errUnimplemented("CreateSnippet") } @@ -321,6 +325,10 @@ func (unimplemented) DeletePublishedRestService(_ model.ID) error { return errUnimplemented("DeletePublishedRestService") } +func (unimplemented) DeleteQueue(_ string) error { + return errUnimplemented("DeleteQueue") +} + func (unimplemented) DeleteSnippet(_ model.ID) error { return errUnimplemented("DeleteSnippet") } @@ -685,6 +693,11 @@ func (unimplemented) ListPublishedRestServices() ([]*model.PublishedRestService, return r0, errUnimplemented("ListPublishedRestServices") } +func (unimplemented) ListQueues() ([]*types.Queue, error) { + var r0 []*types.Queue + return r0, errUnimplemented("ListQueues") +} + func (unimplemented) ListRawUnits(_ string) ([]*types.RawUnitInfo, error) { var r0 []*types.RawUnitInfo return r0, errUnimplemented("ListRawUnits") @@ -1061,6 +1074,10 @@ func (unimplemented) UpdateQualifiedNameInAllUnits(_ string, _ string) (int, err return r0, errUnimplemented("UpdateQualifiedNameInAllUnits") } +func (unimplemented) UpdateQueue(_ *types.Queue) error { + return errUnimplemented("UpdateQueue") +} + func (unimplemented) UpdateRawUnit(_ string, _ []uint8) error { return errUnimplemented("UpdateRawUnit") } diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 4daeaa705..44050fa80 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -702,6 +702,23 @@ func (b *MprBackend) ListIconCollections() ([]*types.IconCollection, error) { return b.reader.ListIconCollections() } +// --------------------------------------------------------------------------- +// QueueBackend +// --------------------------------------------------------------------------- + +func (b *MprBackend) ListQueues() ([]*types.Queue, error) { + return b.reader.ListQueues() +} +func (b *MprBackend) CreateQueue(q *types.Queue) error { + return b.writer.CreateQueue(q) +} +func (b *MprBackend) UpdateQueue(q *types.Queue) error { + return b.writer.UpdateQueue(q) +} +func (b *MprBackend) DeleteQueue(id string) error { + return b.writer.DeleteQueue(id) +} + // --------------------------------------------------------------------------- // ScheduledEventBackend // --------------------------------------------------------------------------- diff --git a/mdl/executor/cmd_microflows_create.go b/mdl/executor/cmd_microflows_create.go index d48a134bc..81aa0238d 100644 --- a/mdl/executor/cmd_microflows_create.go +++ b/mdl/executor/cmd_microflows_create.go @@ -91,6 +91,14 @@ func execCreateMicroflow(ctx *ExecContext, s *ast.CreateMicroflowStmt) error { // For CREATE OR REPLACE/MODIFY, reuse the existing ID to preserve references qualifiedName := s.Name.Module + "." + s.Name.Name + + // Refuse before writing if the stored microflow has a call bound to a task + // queue: the rebuild would null it out and nothing downstream would notice. + if existingID != "" { + if err := checkNoQueuedCalls(ctx, existingID, qualifiedName); err != nil { + return err + } + } microflowID := model.ID(types.GenerateID()) if existingID != "" { microflowID = existingID diff --git a/mdl/executor/cmd_queues.go b/mdl/executor/cmd_queues.go new file mode 100644 index 000000000..76600c65b --- /dev/null +++ b/mdl/executor/cmd_queues.go @@ -0,0 +1,183 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package executor — task queue commands (CREATE/DROP/SHOW/DESCRIBE QUEUE). +// +// A Mendix task queue (Queues$Queue) governs how many instances of a queued +// microflow call run at once, and whether that limit is per runtime instance or +// cluster-wide. +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// findQueue returns the queue with the given module-qualified name, or nil. +func findQueue(ctx *ExecContext, moduleName, name string) *types.Queue { + queues, err := ctx.Backend.ListQueues() + if err != nil { + return nil + } + h, err := getHierarchy(ctx) + if err != nil { + return nil + } + for _, q := range queues { + if !strings.EqualFold(q.Name, name) { + continue + } + mod := h.GetModuleName(h.FindModuleID(q.ContainerID)) + if strings.EqualFold(mod, moduleName) { + return q + } + } + return nil +} + +// execCreateQueue handles CREATE [OR REPLACE|MODIFY] QUEUE Module.Name (...). +func execCreateQueue(ctx *ExecContext, s *ast.CreateQueueStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + + module, err := findOrCreateModule(ctx, s.Name.Module) + if err != nil { + return err + } + + existing := findQueue(ctx, s.Name.Module, s.Name.Name) + if existing != nil && !s.CreateOrModify { + return mdlerrors.NewAlreadyExists("queue", s.Name.String()) + } + + containerID := module.ID + if existing != nil { + containerID = existing.ContainerID + } + + q := &types.Queue{ + ContainerID: containerID, + Name: s.Name.Name, + Documentation: s.Documentation, + Parallelism: s.Parallelism, + ClusterWide: s.ClusterWide, + ExportLevel: s.ExportLevel, + } + + if existing != nil { + q.ID = existing.ID + if err := ctx.Backend.UpdateQueue(q); err != nil { + return mdlerrors.NewBackend("update queue", err) + } + fmt.Fprintf(ctx.Output, "Modified queue: %s\n", s.Name.String()) + return nil + } + if err := ctx.Backend.CreateQueue(q); err != nil { + return mdlerrors.NewBackend("create queue", err) + } + fmt.Fprintf(ctx.Output, "Created queue: %s\n", s.Name.String()) + return nil +} + +// execDropQueue handles DROP QUEUE Module.Name. +func execDropQueue(ctx *ExecContext, s *ast.DropQueueStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + existing := findQueue(ctx, s.Name.Module, s.Name.Name) + if existing == nil { + return mdlerrors.NewNotFound("queue", s.Name.String()) + } + if err := ctx.Backend.DeleteQueue(string(existing.ID)); err != nil { + return mdlerrors.NewBackend("drop queue", err) + } + fmt.Fprintf(ctx.Output, "Dropped queue: %s\n", s.Name.String()) + return nil +} + +// execShowQueues handles SHOW|LIST QUEUES [IN Module]. +func execShowQueues(ctx *ExecContext, s *ast.ShowQueuesStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + queues, err := ctx.Backend.ListQueues() + if err != nil { + return mdlerrors.NewBackend("list queues", err) + } + h, err := getHierarchy(ctx) + if err != nil { + return mdlerrors.NewBackend("build hierarchy", err) + } + + type row struct{ qualified, parallelism, clusterWide string } + var rows []row + for _, q := range queues { + mod := h.GetModuleName(h.FindModuleID(q.ContainerID)) + if s.Module != "" && !strings.EqualFold(mod, s.Module) { + continue + } + rows = append(rows, row{ + qualified: mod + "." + q.Name, + parallelism: q.Parallelism, + clusterWide: fmt.Sprintf("%t", q.ClusterWide), + }) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].qualified < rows[j].qualified }) + + result := &TableResult{ + Columns: []string{"Queue", "Parallelism", "Cluster Wide"}, + Summary: fmt.Sprintf("(%d queue(s))", len(rows)), + } + for _, r := range rows { + result.Rows = append(result.Rows, []any{r.qualified, r.parallelism, r.clusterWide}) + } + return writeResult(ctx, result) +} + +// execDescribeQueue handles DESCRIBE QUEUE Module.Name, emitting re-executable +// MDL so describe → exec round-trips. +func execDescribeQueue(ctx *ExecContext, s *ast.DescribeQueueStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + q := findQueue(ctx, s.Name.Module, s.Name.Name) + if q == nil { + return mdlerrors.NewNotFound("queue", s.Name.String()) + } + + if q.Documentation != "" { + fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", q.Documentation) + } + fmt.Fprintf(ctx.Output, "create or modify queue %s (\n", s.Name.String()) + // Parallelism is an expression string; quote it unless it is a plain integer, + // so an expression survives the round-trip. + fmt.Fprintf(ctx.Output, " Parallelism: %s,\n", formatParallelism(q.Parallelism)) + fmt.Fprintf(ctx.Output, " ClusterWide: %t,\n", q.ClusterWide) + fmt.Fprint(ctx.Output, ");\n") + return nil +} + +// formatParallelism renders the expression bare when it is a plain integer and +// quoted otherwise. +func formatParallelism(expr string) string { + if expr == "" { + return "1" + } + for _, r := range expr { + if r < '0' || r > '9' { + return "'" + strings.ReplaceAll(expr, "'", "''") + "'" + } + } + return expr +} diff --git a/mdl/executor/cmd_queues_mock_test.go b/mdl/executor/cmd_queues_mock_test.go new file mode 100644 index 000000000..d3424afeb --- /dev/null +++ b/mdl/executor/cmd_queues_mock_test.go @@ -0,0 +1,247 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +func mkQueue(containerID model.ID, name, parallelism string, clusterWide bool) *types.Queue { + q := &types.Queue{ + ContainerID: containerID, + Name: name, + Parallelism: parallelism, + ClusterWide: clusterWide, + } + q.ID = nextID("queue") + return q +} + +func TestShowQueues_Mock(t *testing.T) { + mod := mkModule("Ops") + q1 := mkQueue(mod.ID, "OrderProcessing", "3", true) + q2 := mkQueue(mod.ID, "Mail", "1", false) + + h := mkHierarchy(mod) + withContainer(h, q1.ContainerID, mod.ID) + withContainer(h, q2.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{q1, q2}, nil }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execShowQueues(ctx, &ast.ShowQueuesStmt{})) + + out := buf.String() + assertContainsStr(t, out, "Ops.OrderProcessing") + assertContainsStr(t, out, "Ops.Mail") + assertContainsStr(t, out, "(2 queue(s))") +} + +func TestShowQueues_Mock_FilterByModule(t *testing.T) { + alpha := mkModule("Alpha") + beta := mkModule("Beta") + q1 := mkQueue(alpha.ID, "One", "1", false) + q2 := mkQueue(beta.ID, "Two", "2", false) + + h := mkHierarchy(alpha, beta) + withContainer(h, q1.ContainerID, alpha.ID) + withContainer(h, q2.ContainerID, beta.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{q1, q2}, nil }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execShowQueues(ctx, &ast.ShowQueuesStmt{Module: "Beta"})) + + out := buf.String() + assertNotContainsStr(t, out, "Alpha.One") + assertContainsStr(t, out, "Beta.Two") +} + +// TestCreateQueue_Mock_PassesParallelismThrough checks that the expression +// reaches the backend as written. Parallelism is an expression, so it must not +// be parsed into a number anywhere on the way down. +func TestCreateQueue_Mock_PassesParallelismThrough(t *testing.T) { + mod := mkModule("Ops") + h := mkHierarchy(mod) + + var created *types.Queue + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + CreateQueueFunc: func(q *types.Queue) error { + created = q + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.CreateQueueStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}, + Parallelism: "$Config/Workers", + ClusterWide: true, + } + assertNoError(t, execCreateQueue(ctx, stmt)) + + if created == nil { + t.Fatal("CreateQueue was not called") + } + if created.Parallelism != "$Config/Workers" { + t.Errorf("Parallelism = %q, want the expression verbatim", created.Parallelism) + } + if !created.ClusterWide { + t.Error("ClusterWide did not reach the backend") + } + assertContainsStr(t, buf.String(), "Created queue: Ops.OrderProcessing") +} + +func TestCreateQueue_Mock_DuplicateWithoutOrModify(t *testing.T) { + mod := mkModule("Ops") + existing := mkQueue(mod.ID, "OrderProcessing", "1", false) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + CreateQueueFunc: func(q *types.Queue) error { + t.Error("CreateQueue must not be called for a duplicate") + return nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.CreateQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}} + if err := execCreateQueue(ctx, stmt); err == nil { + t.Fatal("expected an already-exists error") + } +} + +func TestCreateQueue_Mock_OrModifyUpdates(t *testing.T) { + mod := mkModule("Ops") + existing := mkQueue(mod.ID, "OrderProcessing", "1", false) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var updated *types.Queue + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + UpdateQueueFunc: func(q *types.Queue) error { + updated = q + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.CreateQueueStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}, + Parallelism: "8", + CreateOrModify: true, + } + assertNoError(t, execCreateQueue(ctx, stmt)) + + if updated == nil { + t.Fatal("UpdateQueue was not called") + } + // The stored ID must be reused, or the update becomes a second queue. + if updated.ID != existing.ID { + t.Errorf("ID = %q, want the existing %q", updated.ID, existing.ID) + } + if updated.Parallelism != "8" { + t.Errorf("Parallelism = %q, want 8", updated.Parallelism) + } + assertContainsStr(t, buf.String(), "Modified queue: Ops.OrderProcessing") +} + +func TestDropQueue_Mock(t *testing.T) { + mod := mkModule("Ops") + existing := mkQueue(mod.ID, "OrderProcessing", "1", false) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var deleted string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + DeleteQueueFunc: func(id string) error { + deleted = id + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.DropQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}} + assertNoError(t, execDropQueue(ctx, stmt)) + + if deleted != string(existing.ID) { + t.Errorf("deleted %q, want %q", deleted, existing.ID) + } + assertContainsStr(t, buf.String(), "Dropped queue: Ops.OrderProcessing") +} + +func TestDropQueue_Mock_NotFound(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, + DeleteQueueFunc: func(id string) error { + t.Error("DeleteQueue must not be called when the queue does not exist") + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy())) + stmt := &ast.DropQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "Missing"}} + if err := execDropQueue(ctx, stmt); err == nil { + t.Fatal("expected a not-found error") + } +} + +// TestDescribeQueue_Mock_RoundTrips checks that DESCRIBE emits MDL that can be +// fed straight back in, including quoting a non-numeric parallelism expression. +func TestDescribeQueue_Mock_RoundTrips(t *testing.T) { + mod := mkModule("Ops") + q := mkQueue(mod.ID, "OrderProcessing", "$Config/Workers", true) + h := mkHierarchy(mod) + withContainer(h, q.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{q}, nil }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.DescribeQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}} + assertNoError(t, execDescribeQueue(ctx, stmt)) + + out := buf.String() + assertContainsStr(t, out, "create or modify queue Ops.OrderProcessing (") + assertContainsStr(t, out, "Parallelism: '$Config/Workers',") + assertContainsStr(t, out, "ClusterWide: true,") +} + +func TestFormatParallelism(t *testing.T) { + tests := []struct{ in, want string }{ + {"", "1"}, + {"3", "3"}, + {"$Config/Workers", "'$Config/Workers'"}, + {"it's", "'it''s'"}, // Mendix escapes a quote by doubling it, never with a backslash + } + for _, tt := range tests { + if got := formatParallelism(tt.in); got != tt.want { + t.Errorf("formatParallelism(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index bae5ee4fb..044f9aead 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -207,6 +207,21 @@ func registerNavigationHandlers(r *Registry) { }) } +func registerQueueHandlers(r *Registry) { + r.Register(&ast.CreateQueueStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execCreateQueue(ctx, stmt.(*ast.CreateQueueStmt)) + }) + r.Register(&ast.DropQueueStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDropQueue(ctx, stmt.(*ast.DropQueueStmt)) + }) + r.Register(&ast.ShowQueuesStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execShowQueues(ctx, stmt.(*ast.ShowQueuesStmt)) + }) + r.Register(&ast.DescribeQueueStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDescribeQueue(ctx, stmt.(*ast.DescribeQueueStmt)) + }) +} + func registerImageHandlers(r *Registry) { r.Register(&ast.CreateImageCollectionStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execCreateImageCollection(ctx, stmt.(*ast.CreateImageCollectionStmt)) diff --git a/mdl/executor/registry.go b/mdl/executor/registry.go index 8756717f8..761a87fc6 100644 --- a/mdl/executor/registry.go +++ b/mdl/executor/registry.go @@ -38,6 +38,7 @@ func NewRegistry() *Registry { registerSecurityHandlers(r) registerNavigationHandlers(r) registerImageHandlers(r) + registerQueueHandlers(r) registerWorkflowHandlers(r) registerBusinessEventHandlers(r) registerSettingsHandlers(r) diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 538505250..5ec9c72ab 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -206,6 +206,7 @@ func allKnownStatements() []ast.Statement { &ast.CreateODataServiceStmt{}, &ast.CreatePageStmtV3{}, &ast.CreatePublishedRestServiceStmt{}, + &ast.CreateQueueStmt{}, &ast.CreateRestClientStmt{}, &ast.CreateSnippetStmtV3{}, &ast.CreateUserRoleStmt{}, @@ -215,6 +216,7 @@ func allKnownStatements() []ast.Statement { &ast.DescribeCatalogTableStmt{}, &ast.DescribeContractFromOpenAPIStmt{}, &ast.DescribeFragmentFromStmt{}, + &ast.DescribeQueueStmt{}, &ast.DescribeStmt{}, &ast.DescribeStylingStmt{}, &ast.DisconnectStmt{}, @@ -245,6 +247,7 @@ func allKnownStatements() []ast.Statement { &ast.DropODataServiceStmt{}, &ast.DropPageStmt{}, &ast.DropPublishedRestServiceStmt{}, + &ast.DropQueueStmt{}, &ast.DropRestClientStmt{}, &ast.DropSnippetStmt{}, &ast.DropUserRoleStmt{}, @@ -278,6 +281,7 @@ func allKnownStatements() []ast.Statement { &ast.SetStmt{}, &ast.ShowDesignPropertiesStmt{}, &ast.ShowFeaturesStmt{}, + &ast.ShowQueuesStmt{}, &ast.ShowStmt{}, &ast.ShowWidgetsStmt{}, &ast.SQLConnectionsStmt{}, diff --git a/mdl/executor/validate_queued_calls.go b/mdl/executor/validate_queued_calls.go new file mode 100644 index 000000000..fbbf2ea3f --- /dev/null +++ b/mdl/executor/validate_queued_calls.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" +) + +// checkNoQueuedCalls refuses to rewrite a microflow that has a call bound to a +// task queue, because the rewrite would silently drop that binding. +// +// A CREATE OR REPLACE/MODIFY rebuilds the microflow from the statement, and both +// engines hardcode QueueSettings to null on a call — correct for a newly +// authored call, wrong for one that was already queued: +// +// codec.RegisterTypeDefaults("Microflows$MicroflowCall", codec.TypeDefaults{ +// NullFields: []string{"QueueSettings"}, ... +// +// Nothing signals the loss afterwards. Measured on Mendix 11.13: with the +// binding present `mx check` reports CE1613 ("The selected task queue … no +// longer exists") on the call activity; after the rewrite it reports 0 errors. +// So mxcli "fixes" the build by deleting the user's configuration, and the +// project then looks healthy. +// +// MDL cannot yet author a queued call, so the binding cannot be restated in the +// script either — refusing is the only option that does not lose data +// (guard-don't-drop, ADR-0005). Remove this guard when `in queue` exists and the +// rebuild carries the binding through. +func checkNoQueuedCalls(ctx *ExecContext, microflowID model.ID, qualifiedName string) error { + raw, err := ctx.Backend.GetRawUnit(microflowID) + if err != nil { + // Unreadable stored unit is not this guard's business; the rewrite path + // reports its own errors. + return nil + } + queues := queuedCallTargets(raw) + if len(queues) == 0 { + return nil + } + sort.Strings(queues) + return mdlerrors.NewUnsupported(fmt.Sprintf( + "microflow %s has %d call(s) bound to a task queue (%s), and rewriting it would silently "+ + "drop that binding — MDL cannot express a queued call yet, so the queue cannot be restated "+ + "in this script.\n"+ + " Change the microflow in Studio Pro, or remove the task queue from the call first "+ + "(the binding lives on the call activity, not the microflow).", + qualifiedName, len(queues), strings.Join(queues, ", "))) +} + +// queuedCallTargets walks a stored microflow document and returns the queue +// bound to each call that has one. +// +// The binding that matters is QueueSettings — a Queues$QueueSettings node whose +// own Queue property names the queue. The call's top-level Queue property is +// also read, because it is in the metamodel, but on its own it is inert: +// measured on 11.13, a call carrying only Queue (with QueueSettings null) draws +// no complaint from mx check at all, while one carrying QueueSettings does. +func queuedCallTargets(v any) []string { + var out []string + switch t := v.(type) { + case map[string]any: + if qs, ok := t["QueueSettings"].(map[string]any); ok && qs != nil { + name, _ := qs["Queue"].(string) + if name == "" { + name = "(unnamed queue)" + } + out = append(out, name) + } else if name, ok := t["Queue"].(string); ok && name != "" { + // Present without QueueSettings: not something Mendix acts on, but + // still authored state that the rewrite would drop. + out = append(out, name) + } + for _, val := range t { + out = append(out, queuedCallTargets(val)...) + } + case []any: + for _, el := range t { + out = append(out, queuedCallTargets(el)...) + } + } + return dedupeStrings(out) +} + +func dedupeStrings(in []string) []string { + if len(in) < 2 { + return in + } + seen := make(map[string]bool, len(in)) + out := in[:0] + for _, s := range in { + if seen[s] { + continue + } + seen[s] = true + out = append(out, s) + } + return out +} diff --git a/mdl/executor/validate_queued_calls_test.go b/mdl/executor/validate_queued_calls_test.go new file mode 100644 index 000000000..b44d4066c --- /dev/null +++ b/mdl/executor/validate_queued_calls_test.go @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// storedCall builds the shape a stored MicroflowCall has once a task queue is +// bound to it in Studio Pro. The binding lives two levels down (Queues$QueueSettings +// inside the call's QueueSettings property), so the walk has to be recursive. +func storedCall(queueSettings map[string]any, topLevelQueue any) map[string]any { + return map[string]any{ + "$Type": "Microflows$Microflow", + "ObjectCollection": map[string]any{ + "Objects": []any{ + map[string]any{ + "$Type": "Microflows$ActionActivity", + "Action": map[string]any{ + "$Type": "Microflows$MicroflowCall", + "Microflow": "Q.Target", + "Queue": topLevelQueue, + "QueueSettings": queueSettings, + }, + }, + }, + }, + } +} + +func TestQueuedCallTargets_FindsQueueSettings(t *testing.T) { + got := queuedCallTargets(storedCall(map[string]any{ + "$Type": "Queues$QueueSettings", + "Queue": "Q.MyQueue", + }, nil)) + if len(got) != 1 || got[0] != "Q.MyQueue" { + t.Fatalf("queuedCallTargets = %v, want [Q.MyQueue]", got) + } +} + +func TestQueuedCallTargets_NoBinding(t *testing.T) { + if got := queuedCallTargets(storedCall(nil, nil)); len(got) != 0 { + t.Fatalf("queuedCallTargets = %v, want none for an unqueued call", got) + } +} + +// A call can carry a bare Queue with QueueSettings null. Measured on 11.13 that +// is inert (mx check does not complain), but it is still authored state, and a +// rewrite would drop it. +func TestQueuedCallTargets_BareQueue(t *testing.T) { + got := queuedCallTargets(storedCall(nil, "Q.MyQueue")) + if len(got) != 1 || got[0] != "Q.MyQueue" { + t.Fatalf("queuedCallTargets = %v, want [Q.MyQueue]", got) + } +} + +func TestQueuedCallTargets_Dedupes(t *testing.T) { + doc := map[string]any{"Objects": []any{ + map[string]any{"QueueSettings": map[string]any{"Queue": "Q.A"}}, + map[string]any{"QueueSettings": map[string]any{"Queue": "Q.A"}}, + map[string]any{"QueueSettings": map[string]any{"Queue": "Q.B"}}, + }} + got := queuedCallTargets(doc) + if len(got) != 2 { + t.Fatalf("queuedCallTargets = %v, want 2 distinct queues", got) + } +} + +// TestCheckNoQueuedCalls_Refuses is the guard against the data loss itself: a +// CREATE OR REPLACE of a microflow with a queued call used to succeed and write +// QueueSettings back as null, so mx check went from CE1613 to 0 errors by +// deleting the user's configuration. +func TestCheckNoQueuedCalls_Refuses(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(map[string]any{"Queue": "Q.MyQueue"}, nil), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + + err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller") + if err == nil { + t.Fatal("expected a refusal for a microflow with a queued call") + } + msg := err.Error() + for _, want := range []string{"Q.ACT_Caller", "Q.MyQueue", "task queue"} { + if !strings.Contains(msg, want) { + t.Errorf("error message missing %q:\n%s", want, msg) + } + } +} + +func TestCheckNoQueuedCalls_AllowsUnqueued(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(nil, nil), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + if err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller"); err != nil { + t.Fatalf("unqueued microflow must still be rewritable: %v", err) + } +} + +// An unreadable unit is not this guard's business — the rewrite path reports its +// own errors, and failing here would block writes for an unrelated reason. +func TestCheckNoQueuedCalls_UnreadableUnitDoesNotBlock(t *testing.T) { + ctx, _ := newMockCtx(t) // default mock: GetRawUnit is not configured, so it errors + if err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller"); err != nil { + t.Fatalf("unreadable unit must not block the write: %v", err) + } +} + +// TestCreateOrModifyMicroflow_RefusesQueuedCall drives the guard through the +// real CREATE OR MODIFY path. This is the test that fails if the call site in +// execCreateMicroflow is removed — checkNoQueuedCalls passing on its own proves +// nothing about whether anything calls it. +func TestCreateOrModifyMicroflow_RefusesQueuedCall(t *testing.T) { + mod := mkModule("Q") + h := mkHierarchy(mod) + + existing := µflows.Microflow{Name: "ACT_Caller"} + existing.ID = "mf-existing" + existing.ContainerID = mod.ID + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListMicroflowsFunc: func() ([]*microflows.Microflow, error) { + return []*microflows.Microflow{existing}, nil + }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(map[string]any{"Queue": "Q.MyQueue"}, nil), nil + }, + UpdateMicroflowFunc: func(mf *microflows.Microflow) error { + t.Error("UpdateMicroflow must not run: the rewrite would drop the queue binding") + return nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + err := execCreateMicroflow(ctx, &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "Q", Name: "ACT_Caller"}, + CreateOrModify: true, + }) + if err == nil { + t.Fatal("expected a refusal, got a successful rewrite") + } + if !strings.Contains(err.Error(), "Q.MyQueue") { + t.Errorf("error should name the queue that would be lost:\n%s", err) + } +} diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index 61a832e43..87fe9226c 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -354,6 +354,8 @@ READONLY: R E A D O N L Y; ATTRIBUTES: A T T R I B U T E S; FILTERTYPE: F I L T E R T Y P E; IMAGE: I M A G E; +QUEUE: Q U E U E; +QUEUES: Q U E U E S; COLLECTION: C O L L E C T I O N S?; // accept singular + plural ("collection(s)") JAR: J A R; DEPENDENCY: D E P E N D E N C Y; diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index cebf49dd0..fe263a5dc 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -115,6 +115,7 @@ createStatement | createUserRoleStatement | createDemoUserStatement | createImageCollectionStatement + | createQueueStatement | createJsonStructureStatement | createImportMappingStatement | createExportMappingStatement @@ -309,6 +310,7 @@ dropStatement | DROP SNIPPET qualifiedName | DROP MODULE qualifiedName | DROP NOTEBOOK qualifiedName + | DROP QUEUE qualifiedName | DROP JAVA ACTION qualifiedName | DROP JAVASCRIPT ACTION qualifiedName | DROP INDEX qualifiedName ON qualifiedName diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index f5e606fae..c747df1a7 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -35,6 +35,7 @@ showStatement | showOrList CONSTANT VALUES (IN (qualifiedName | IDENTIFIER))? | showOrList LAYOUTS (IN (qualifiedName | IDENTIFIER))? | showOrList NOTEBOOKS (IN (qualifiedName | IDENTIFIER))? + | showOrList QUEUES (IN (qualifiedName | IDENTIFIER))? | showOrList JAVA ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList JAVASCRIPT ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList IMAGE COLLECTION (IN (qualifiedName | IDENTIFIER))? @@ -162,6 +163,7 @@ describeStatement | DESCRIBE STYLING ON (PAGE | SNIPPET) qualifiedName (WIDGET IDENTIFIER)? // DESCRIBE STYLING ON PAGE Module.Page [WIDGET name] | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name + | DESCRIBE QUEUE qualifiedName // DESCRIBE QUEUE Module.Name | DESCRIBE DATABASE CONNECTION qualifiedName // DESCRIBE DATABASE CONNECTION Module.Name | DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)? // DESCRIBE SETTINGS [CONFIGURATION 'Default'] | DESCRIBE FRAGMENT FROM PAGE qualifiedName WIDGET identifierOrKeyword // DESCRIBE FRAGMENT FROM PAGE Module.Page WIDGET name diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index e7e31d312..3ad98dfc5 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -331,6 +331,29 @@ enumerationOption | FOLDER STRING_LITERAL // place the enumeration in a module folder (Bug 12b) ; +// ============================================================================= +// TASK QUEUE CREATION +// ============================================================================= + +/** + * CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( Parallelism: 3, ClusterWide: true ); + * + * Parallelism is stored by Mendix as an EXPRESSION string + * (Queues$BasicQueueConfig.ParallelismExpression), so it accepts a number or a + * quoted expression. + */ +createQueueStatement + : QUEUE qualifiedName queueBody? + ; + +queueBody + : LPAREN (queueProperty (COMMA queueProperty)* COMMA?)? RPAREN + ; + +queueProperty + : identifierOrKeyword COLON (NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | identifierOrKeyword) + ; + // ============================================================================= // IMAGE COLLECTION CREATION // ============================================================================= diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index 9db3b5d0f..a18faa5bb 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -486,7 +486,7 @@ annotationParenValue */ keyword // DDL / DML - : ADD | ALTER | BATCH | BROWSER | CHANGE | CLOSE | COMMIT | CREATE | DECLARE | DELETE | DESCRIBE + : QUEUE | QUEUES | ADD | ALTER | BATCH | BROWSER | CHANGE | CLOSE | COMMIT | CREATE | DECLARE | DELETE | DESCRIBE | DOWNLOAD | DROP | EXECUTE | EXPORT | GENERATE | IMPORT | INSERT | INTO | MODIFY | MOVE | REFRESH | SYNCHRONIZE | UNSYNCHRONIZED | REMOVE | RENAME | REPLACE | RETRIEVE | RETURN | ROLLBACK | SET | UPDATE diff --git a/mdl/types/queue.go b/mdl/types/queue.go new file mode 100644 index 000000000..1560bd681 --- /dev/null +++ b/mdl/types/queue.go @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: Apache-2.0 + +package types + +import "github.com/mendixlabs/mxcli/model" + +// Queue is a Mendix task queue (Queues$Queue) — the configuration that governs +// how many instances of a queued microflow call run at once, and whether that +// limit is per-node or cluster-wide. +// +// The shape here follows four Studio Pro-authored queues from the Mendix +// Business Events module (Consumer_Queue, Consumer_Processor_Queue, +// Producer_Queue, Outbox_Cleanup_Queue), which agree exactly: +// +// { "$Type": "Queues$Queue", "Name": "Consumer_Queue", +// "Documentation": "", "Excluded": false, "ExportLevel": "Hidden", +// "Config": { "$Type": "Queues$BasicQueueConfig", +// "ClusterWide": false, "ParallelismExpression": "1" } } +// +// Two details are load-bearing and are not guesses: +// +// 1. Parallelism is stored as ParallelismExpression, a STRING holding a Mendix +// expression ("1"). Queues$BasicQueueConfig also declares an int32 +// `Parallelism`, but Studio Pro wrote it zero times out of four — so it is +// not written here either. +// 2. Config was Queues$BasicQueueConfig in all four, and it is the only config +// type in the metamodel, so there is no variant to dispatch on. +type Queue struct { + model.BaseElement + ContainerID model.ID `json:"containerId"` + Name string `json:"name"` + Documentation string `json:"documentation,omitempty"` + Excluded bool `json:"excluded,omitempty"` + ExportLevel string `json:"exportLevel,omitempty"` + + // Parallelism is the expression form (Config.ParallelismExpression), kept as + // a string because that is how Mendix stores it: usually a literal like "3", + // but any Mendix expression is valid. + Parallelism string `json:"parallelism,omitempty"` + // ClusterWide makes the parallelism limit apply across the cluster rather + // than per runtime instance (Config.ClusterWide). + ClusterWide bool `json:"clusterWide,omitempty"` +} + +// GetName returns the queue's name. +func (q *Queue) GetName() string { return q.Name } + +// GetContainerID returns the container ID. +func (q *Queue) GetContainerID() model.ID { return q.ContainerID } diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index c4b043d6e..24598a7ed 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -829,6 +829,10 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { b.statements = append(b.statements, &ast.DropImageCollectionStmt{ Name: buildQualifiedName(names[0]), }) + } else if ctx.QUEUE() != nil { + b.statements = append(b.statements, &ast.DropQueueStmt{ + Name: buildQualifiedName(names[0]), + }) } else if ctx.MODEL() != nil { b.statements = append(b.statements, &ast.DropModelStmt{ Name: buildQualifiedName(names[0]), diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 79170dfae..596cb5d6d 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -204,6 +204,16 @@ func (b *Builder) ExitShowStatement(ctx *parser.ShowStatementContext) { } } b.statements = append(b.statements, stmt) + } else if ctx.QUEUES() != nil { + stmt := &ast.ShowQueuesStmt{} + if ctx.IN() != nil { + if qn := ctx.QualifiedName(); qn != nil { + stmt.Module = getQualifiedNameText(qn) + } else if id := ctx.IDENTIFIER(); id != nil { + stmt.Module = id.GetText() + } + } + b.statements = append(b.statements, stmt) } else if ctx.LAYOUTS() != nil { stmt := &ast.ShowStmt{ObjectType: ast.ShowLayouts} if ctx.IN() != nil { @@ -694,6 +704,14 @@ func (b *Builder) ExitCatalogSelectQuery(ctx *parser.CatalogSelectQueryContext) // ExitDescribeStatement handles DESCRIBE ENTITY/ASSOCIATION/ENUMERATION/MODULE func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { + // DESCRIBE QUEUE Module.Name + if ctx.QUEUE() != nil { + if qn := ctx.QualifiedName(); qn != nil { + b.statements = append(b.statements, &ast.DescribeQueueStmt{Name: buildQualifiedName(qn)}) + } + return + } + // Handle DESCRIBE MODULE ROLE (uses qualifiedName) if ctx.MODULE() != nil && ctx.ROLE() != nil { if qn := ctx.QualifiedName(); qn != nil { diff --git a/mdl/visitor/visitor_queue.go b/mdl/visitor/visitor_queue.go new file mode 100644 index 000000000..8da08ffea --- /dev/null +++ b/mdl/visitor/visitor_queue.go @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// ExitCreateQueueStatement builds a CreateQueueStmt from +// CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( ... ). +func (b *Builder) ExitCreateQueueStatement(ctx *parser.CreateQueueStatementContext) { + stmt := &ast.CreateQueueStmt{ + Name: buildQualifiedName(ctx.QualifiedName()), + Documentation: findDocCommentText(ctx), + } + if createStmt := findParentCreateStatement(ctx); createStmt != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { + stmt.CreateOrModify = true + } + } + + if body := ctx.QueueBody(); body != nil { + bodyCtx := body.(*parser.QueueBodyContext) + for _, prop := range bodyCtx.AllQueueProperty() { + pc, ok := prop.(*parser.QueuePropertyContext) + if !ok || pc == nil { + continue + } + iok := pc.IdentifierOrKeyword(0) + if iok == nil { + continue + } + key := strings.ToLower(identifierOrKeywordText(iok)) + switch key { + case "parallelism": + stmt.Parallelism = queuePropertyText(pc) + case "clusterwide": + stmt.ClusterWide = strings.EqualFold(queuePropertyText(pc), "true") + case "exportlevel": + stmt.ExportLevel = queuePropertyText(pc) + case "documentation": + stmt.Documentation = queuePropertyText(pc) + } + } + } + + b.statements = append(b.statements, stmt) +} + +// queuePropertyText returns the value side of a queue property, unquoted. +// +// The value alternatives are NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | +// identifierOrKeyword. The key is identifierOrKeyword(0), so an identifier value +// is index 1 — reading index 0 would echo the key back as the value. +func queuePropertyText(pc *parser.QueuePropertyContext) string { + if n := pc.NUMBER_LITERAL(); n != nil { + return n.GetText() + } + if s := pc.STRING_LITERAL(); s != nil { + return unquoteString(s.GetText()) + } + if bl := pc.BooleanLiteral(); bl != nil { + return bl.GetText() + } + if v := pc.IdentifierOrKeyword(1); v != nil { + return identifierOrKeywordText(v) + } + return "" +} diff --git a/mdl/visitor/visitor_queue_test.go b/mdl/visitor/visitor_queue_test.go new file mode 100644 index 000000000..831d9bba1 --- /dev/null +++ b/mdl/visitor/visitor_queue_test.go @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func TestCreateQueue(t *testing.T) { + input := `CREATE QUEUE Ops.OrderProcessing ( + Parallelism: 3, + ClusterWide: true + );` + prog, errs := Build(input) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateQueueStmt) + if !ok { + t.Fatalf("expected CreateQueueStmt, got %T", prog.Statements[0]) + } + if stmt.Name.Module != "Ops" || stmt.Name.Name != "OrderProcessing" { + t.Errorf("Name = %+v", stmt.Name) + } + if stmt.Parallelism != "3" { + t.Errorf("Parallelism = %q, want 3", stmt.Parallelism) + } + if !stmt.ClusterWide { + t.Error("ClusterWide = false, want true") + } + if stmt.CreateOrModify { + t.Error("CreateOrModify set without OR MODIFY") + } +} + +// TestCreateQueue_ExpressionParallelism covers the reason Parallelism is a +// string all the way down: Mendix stores an expression, not a number. +func TestCreateQueue_ExpressionParallelism(t *testing.T) { + prog, errs := Build(`CREATE OR MODIFY QUEUE Ops.Q ( Parallelism: '$Config/Workers' );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateQueueStmt) + if stmt.Parallelism != "$Config/Workers" { + t.Errorf("Parallelism = %q, want the unquoted expression", stmt.Parallelism) + } + if !stmt.CreateOrModify { + t.Error("CreateOrModify not set for CREATE OR MODIFY") + } +} + +func TestCreateQueue_Defaults(t *testing.T) { + prog, errs := Build(`CREATE QUEUE Ops.Q ();`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateQueueStmt) + if stmt.Parallelism != "" { + t.Errorf("Parallelism = %q, want empty (the backend supplies the default)", stmt.Parallelism) + } + if stmt.ClusterWide { + t.Error("ClusterWide defaulted to true") + } +} + +func TestDropQueue(t *testing.T) { + prog, errs := Build(`DROP QUEUE Ops.OrderProcessing;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.DropQueueStmt) + if !ok { + t.Fatalf("expected DropQueueStmt, got %T", prog.Statements[0]) + } + if stmt.Name.String() != "Ops.OrderProcessing" { + t.Errorf("Name = %s", stmt.Name.String()) + } +} + +func TestShowAndDescribeQueues(t *testing.T) { + prog, errs := Build(`SHOW QUEUES; LIST QUEUES IN Ops; DESCRIBE QUEUE Ops.OrderProcessing;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 3 { + t.Fatalf("expected 3 statements, got %d", len(prog.Statements)) + } + if _, ok := prog.Statements[0].(*ast.ShowQueuesStmt); !ok { + t.Errorf("statement 0 = %T, want ShowQueuesStmt", prog.Statements[0]) + } + s1, ok := prog.Statements[1].(*ast.ShowQueuesStmt) + if !ok { + t.Fatalf("statement 1 = %T, want ShowQueuesStmt", prog.Statements[1]) + } + if s1.Module != "Ops" { + t.Errorf("Module = %q, want Ops", s1.Module) + } + s2, ok := prog.Statements[2].(*ast.DescribeQueueStmt) + if !ok { + t.Fatalf("statement 2 = %T, want DescribeQueueStmt", prog.Statements[2]) + } + if s2.Name.String() != "Ops.OrderProcessing" { + t.Errorf("Name = %s", s2.Name.String()) + } +} + +// TestQueueKeywordStillUsableAsIdentifier guards the cost of adding QUEUE to the +// lexer: a new keyword stops being usable as an ordinary name unless it is also +// listed in the `keyword` rule. "queue" is a plausible attribute name. +func TestQueueKeywordStillUsableAsIdentifier(t *testing.T) { + prog, errs := Build(`CREATE ENTITY Ops.Job ( queue: String(100), queues: String(100) );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateEntityStmt) + if !ok { + t.Fatalf("expected CreateEntityStmt, got %T", prog.Statements[0]) + } + if len(stmt.Attributes) != 2 { + t.Fatalf("expected 2 attributes, got %d", len(stmt.Attributes)) + } + if stmt.Attributes[0].Name != "queue" || stmt.Attributes[1].Name != "queues" { + t.Errorf("attribute names = %q, %q", stmt.Attributes[0].Name, stmt.Attributes[1].Name) + } +} diff --git a/sdk/mpr/queues.go b/sdk/mpr/queues.go new file mode 100644 index 000000000..907690da8 --- /dev/null +++ b/sdk/mpr/queues.go @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +// Task queues (Queues$Queue). The document is small and flat apart from a single +// nested Config node, so it is read and written as raw BSON here rather than +// through a dedicated element type. +// +// The shape follows four Studio Pro-authored queues from the Mendix Business +// Events module. Note that Queues$BasicQueueConfig declares an int32 +// `Parallelism` in addition to `ParallelismExpression`, and Studio Pro wrote it +// in none of them — so only the expression is read and written. + +const queueUnitType = "Queues$Queue" + +// ListQueues reads every task queue in the project. +func (r *Reader) ListQueues() ([]*types.Queue, error) { + units, err := r.ListRawUnitsByType(queueUnitType) + if err != nil { + return nil, err + } + out := make([]*types.Queue, 0, len(units)) + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + return nil, fmt.Errorf("unmarshal queue %s: %w", u.ID, err) + } + q := &types.Queue{ContainerID: model.ID(u.ContainerID)} + q.ID = model.ID(u.ID) + q.TypeName = queueUnitType + q.Name, _ = doc["Name"].(string) + q.Documentation, _ = doc["Documentation"].(string) + q.Excluded, _ = doc["Excluded"].(bool) + q.ExportLevel, _ = doc["ExportLevel"].(string) + if cfg, ok := doc["Config"].(bson.M); ok { + q.Parallelism, _ = cfg["ParallelismExpression"].(string) + q.ClusterWide, _ = cfg["ClusterWide"].(bool) + } + out = append(out, q) + } + return out, nil +} + +// CreateQueue inserts a new task queue document. +func (w *Writer) CreateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("CreateQueue: nil queue") + } + if q.ID == "" { + q.ID = model.ID(generateUUID()) + } + contents, err := serializeQueueUnit(q) + if err != nil { + return err + } + return w.insertUnit(string(q.ID), string(q.ContainerID), "Documents", queueUnitType, contents) +} + +// UpdateQueue rewrites an existing task queue in place. +func (w *Writer) UpdateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("UpdateQueue: nil queue") + } + contents, err := serializeQueueUnit(q) + if err != nil { + return err + } + return w.UpdateRawUnit(string(q.ID), contents) +} + +// DeleteQueue removes a task queue by ID. +func (w *Writer) DeleteQueue(id string) error { + return w.deleteUnit(id) +} + +func serializeQueueUnit(q *types.Queue) ([]byte, error) { + parallelism := q.Parallelism + if parallelism == "" { + parallelism = "1" + } + exportLevel := q.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + doc := bson.D{ + {Key: "$ID", Value: idToBsonBinary(string(q.ID))}, + {Key: "$Type", Value: queueUnitType}, + {Key: "Config", Value: bson.D{ + {Key: "$ID", Value: idToBsonBinary(generateUUID())}, + {Key: "$Type", Value: "Queues$BasicQueueConfig"}, + {Key: "ClusterWide", Value: q.ClusterWide}, + {Key: "ParallelismExpression", Value: parallelism}, + }}, + {Key: "Documentation", Value: q.Documentation}, + {Key: "Excluded", Value: q.Excluded}, + {Key: "ExportLevel", Value: exportLevel}, + {Key: "Name", Value: q.Name}, + } + return marshalUnitIDFirst(doc) +} From 7d4867820fd416aae95328c13b67818a3e704b09 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 23:08:25 +0000 Subject: [PATCH 25/35] feat(scheduled-events): author Mendix's cron in MDL, and read back what it stores MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scheduled events were read-only: ScheduledEventBackend had List/Get and nothing else, and there was no MDL surface at all. A project's cron had to be added in Studio Pro even though everything around it could be scripted. CREATE [OR MODIFY] / DROP / SHOW / DESCRIBE SCHEDULED EVENT now exist, wired through grammar → AST → visitor → executor → backend on both engines. The repeat rule is stored as one of eight ScheduledEvents$*Schedule types that differ in WHICH fields they carry, not just in their values. MDL mirrors that: `Repeat: Daily` names the variant and then only that variant's fields are accepted — a Multiplier on a Daily repeat is refused, by `mxcli check` (MDL-SCHED01) and by exec, which call the same function so they cannot drift. Merging the field sets is what produces a document mxbuild accepts and Studio Pro cannot open. The read was lossy in both engines and is fixed here too. The legacy parser never looked at Schedule at all, and the modelsdk reader went through modelsdk/gen, whose generated types disagree with what Studio Pro writes on two properties: every integer is stored as int64 where gen declares int32, and StartDateTime is a BSON datetime where gen declares a string. Both engines now share one codec (mdl/scheduledevents), pinned by re-serializing three whole Studio Pro-authored documents — from Workflow Commons 4.11.0, OIDC SSO 4.6.0 and SAML 4.2.1 — element by element, in order, with matching BSON types. Those four references cover the Daily and Hourly variants; the other six are derived from the metamodel and verified to load with 0 errors from `mx check`. Interval/IntervalType are legacy siblings of Schedule that Studio Pro writes and does not keep in sync (one shipped module stores 0/"Minute" beside a daily schedule of 01:00). MDL has no syntax for them: a new event gets the pair matching its repeat — an empty IntervalType is not valid, the enumeration has no such member — and OR MODIFY carries the stored pair through untouched. Verified on Mendix 11.13 under both engines: all eight variants leave a clean project at 0 errors, DESCRIBE output re-parses and re-validates for every variant, and a re-run writes nothing (the MXCLI_ALWAYS_WRITE=1 control changes 9 units where the normal run changes 0). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/cmd_check.go | 6 + cmd/mxcli/lsp_completions_gen.go | 1 + cmd/mxcli/syntax/features_misc.go | 83 +++ docs-site/src/SUMMARY.md | 1 + docs-site/src/language/scheduled-events.md | 185 ++++++ docs/01-project/MDL_QUICK_REFERENCE.md | 52 ++ .../scheduled-event-repeat-fields.fail.mdl | 25 + .../doctype-tests/scheduled-events.mdl | 133 ++++ mdl/ast/ast_scheduledevent.go | 75 +++ mdl/backend/infrastructure.go | 3 + mdl/backend/mcp/unsupported_gen.go | 15 + mdl/backend/mock/backend.go | 7 +- mdl/backend/mock/mock_workflow.go | 25 + mdl/backend/modelsdk/scheduledevent_read.go | 90 ++- .../modelsdk/scheduledevent_read_test.go | 129 ++-- mdl/backend/modelsdk/unimplemented_gen.go | 12 + mdl/backend/mpr/backend.go | 9 + mdl/executor/cmd_queues_mock_test.go | 18 +- mdl/executor/cmd_scheduledevents.go | 586 ++++++++++++++++++ mdl/executor/cmd_scheduledevents_test.go | 456 ++++++++++++++ mdl/executor/register_stubs.go | 15 + mdl/executor/registry.go | 1 + mdl/executor/registry_test.go | 4 + mdl/executor/validate_scheduled_events.go | 41 ++ mdl/grammar/MDLLexer.g4 | 1 + mdl/grammar/MDLParser.g4 | 2 + mdl/grammar/domains/MDLCatalog.g4 | 2 + mdl/grammar/domains/MDLDomainModel.g4 | 22 + mdl/grammar/domains/MDLSettings.g4 | 2 +- mdl/scheduledevents/codec.go | 295 +++++++++ mdl/scheduledevents/codec_test.go | 314 ++++++++++ mdl/visitor/visitor_entity.go | 4 + mdl/visitor/visitor_query.go | 18 + mdl/visitor/visitor_queue.go | 2 +- mdl/visitor/visitor_scheduledevent.go | 133 ++++ mdl/visitor/visitor_scheduledevent_test.go | 150 +++++ model/types.go | 66 ++ sdk/mpr/parser_enumeration.go | 36 +- sdk/mpr/scheduledevents.go | 52 ++ 40 files changed, 2952 insertions(+), 120 deletions(-) create mode 100644 docs-site/src/language/scheduled-events.md create mode 100644 mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl create mode 100644 mdl-examples/doctype-tests/scheduled-events.mdl create mode 100644 mdl/ast/ast_scheduledevent.go create mode 100644 mdl/executor/cmd_scheduledevents.go create mode 100644 mdl/executor/cmd_scheduledevents_test.go create mode 100644 mdl/executor/validate_scheduled_events.go create mode 100644 mdl/scheduledevents/codec.go create mode 100644 mdl/scheduledevents/codec_test.go create mode 100644 mdl/visitor/visitor_scheduledevent.go create mode 100644 mdl/visitor/visitor_scheduledevent_test.go create mode 100644 sdk/mpr/scheduledevents.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index dd3c1dd49..c5c287cda 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -473,3 +473,4 @@ extracting `OffsetExpression`/`LimitExpression`. | An association's line anchors — where the connector attaches to the entity boxes in the domain model editor — are absent from `DESCRIBE ASSOCIATION`, and manual adjustments made in Studio Pro do not survive an mxcli round trip | `DomainModels$Association.ParentConnection`/`ChildConnection` (the string `"x;y"`) were **hardcoded** to `"0;50"`/`"100;50"` in BOTH writers and never read by either parser. Because every association write rebuilds the whole element, this was not an omission but active destruction: a documentation-only `alter association … set comment` reset them | `sdk/domainmodel/connection.go` (new: `ParseConnectionPoint`/`FormatConnectionPoint`, `Default*Connection`), `sdk/domainmodel/domainmodel.go` (fields → `*model.Point`), `sdk/mpr/parser_domainmodel.go` + `sdk/mpr/writer_domainmodel.go`, `mdl/backend/modelsdk/domainmodel.go` + `domainmodel_write.go`, `mdl/executor/cmd_associations.go` (`describeConnectionPoints`) | **A feature request that says "X is not exposed" may be hiding "X is destroyed"** — check the write path before scoping the read path. The A/B that settled it: a blank 11.13 app's own `Administration.AccountPasswordData_Account` stores `0;54/100;54`, so a Studio-Pro-authored association is a free fixture for "did mxcli overwrite this?" — no Studio Pro needed. **Learn the value's constraints from the LOADER, not from the shape**: hand-patch and run `mx check` — `"0.5;50"` dies with `StorageLoadException` (integers required) while `"0;500"` and `"-20;50"` load with 0 errors (no range check), so out-of-range values must round-trip untouched. **A zero value is not an absent value** — `{0,0}` is a real anchor (top-left), which forces the field to be a POINTER; a plain `model.Point` cannot distinguish "unset" from "top-left" and would silently rewrite it. **Fix both engines**: they share the semantic model, and a fix in one is invisible to a user on the other. **Emit unauthorable data as a COMMENT** — DESCRIBE output must stay re-executable, and inventing syntax (`@anchor(parent: bottom-left, …)`) would bake in a vocabulary the storage does not have: the pair is CONTINUOUS, not 8 named anchors (observed x values 0 9 11 17 18 47 49 50 65 77 78 84 87 100). **The marketplace is the sample** when you need to know what Studio Pro actually writes: `mxcli marketplace download ` gives real Mendix-authored models, and a module .mpk holds either a raw BSON `project.mpr` or an MPR v1 SQLite one — 88 coordinate pairs from three modules turned "looks like percentages" into a measurement (all 0..100; 85 of 88 pin one coordinate to exactly 0 or 100). **Rule a unit out from the model, not the values**: pixels is impossible because `DomainModels$EntityImpl` stores only `Location` and NO size — the box is sized by the editor from the name and attribute list, so a pixel anchor would have nothing to measure against. Not applicable to `CrossAssociation`, which has no connection properties and crashes Studio Pro if given them (#50). Tests `sdk/domainmodel/connection_test.go`, `mdl/backend/modelsdk/association_connection_test.go`, `sdk/mpr/writer_domainmodel_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | A microflow call bound to a task queue in Studio Pro loses that binding on `create or replace microflow` — silently, and `mx check` goes from `[CE1613] "The selected task queue no longer exists"` to **0 errors**, because the configuration the error was about has been deleted | Two defects behind one report. (1) `Microflows$MicroflowCall` has `QueueSettings` in `NullFields` of `codec.RegisterTypeDefaults` (and the legacy writer hardcodes null too) — correct for a newly authored call, destructive for a stored one, and since a CREATE OR REPLACE rebuilds the whole microflow it fires on every rewrite. (2) MDL had **no queue surface at all**, so a script could not restate the binding even in principle | `mdl/executor/validate_queued_calls.go` (new: the refusal), `mdl/executor/cmd_microflows_create.go` (call site), `mdl/types/queue.go` + `mdl/backend/infrastructure.go` (`QueueBackend`), `mdl/backend/modelsdk/queue_write.go` + `sdk/mpr/queues.go` (both engines), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4`, `mdl/executor/cmd_queues.go` | **A green build can be the bug**: the fix made `mx check` report MORE errors than before (CE1613 came back), because the binding it complains about now survives. Any "error count went down" check would have scored the data loss as a fix. **`describe` showing nothing is not evidence of nothing** — the binding was invisible from every angle (describe omitted it, check went quiet, the write reported success), which is why it needed a stored-BSON probe: inject `Queue`+`QueueSettings` into a stored call, rewrite, and dump. **Isolate which property Mendix acts on before guarding on it**: a call carrying only `Queue` (with `QueueSettings` null) draws NO complaint from `mx check` — `QueueSettings` is the load-bearing one, so a guard keyed on `Queue` alone would have both missed the real case and fired on inert ones. **Get the BSON shape from a Mendix-authored model, not from the metamodel**: `mxcli marketplace download 202649` (Business Events 3.12.1) has four real queues, and all four agree — `Config` is a `Queues$BasicQueueConfig` whose `ParallelismExpression` is a **STRING**, with the sibling int32 `Parallelism` absent in every one. Writing the int (which the metamodel makes look equally valid) is inventing a property Mendix does not write. **Refuse, do not half-author** (ADR-0005): `in queue` on a call is deliberately still unimplemented, because the `Retry` shape has no Studio-Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. **Adding a keyword costs an identifier** — `QUEUE`/`QUEUES` had to go into the `keyword` rule or `queue` would have stopped working as an attribute name. Tests `mdl/backend/modelsdk/queue_write_test.go`, `mdl/executor/validate_queued_calls_test.go`, `mdl/executor/cmd_queues_mock_test.go`, `mdl/visitor/visitor_queue_test.go`, example `mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl` | +| Scheduled events are read-only in MDL — a project's cron cannot be scripted at all, so `mxcli new` + a script leaves you opening Studio Pro to add the one thing that makes a batch job run. Separately, `describe` of an existing one omits its schedule entirely | Two gaps behind one request. (1) No write path: `ScheduledEventBackend` had List/Get and nothing else, and there was no grammar/AST/visitor/executor. (2) The READ was lossy in both engines — the legacy `parseScheduledEvent` never looked at `Schedule`, and the modelsdk reader went through `modelsdk/gen`, whose generated types **disagree with what Studio Pro writes** on two properties | `mdl/scheduledevents/codec.go` (new, shared by both engines), `mdl/backend/modelsdk/scheduledevent_read.go` + `sdk/mpr/scheduledevents.go` + `sdk/mpr/parser_enumeration.go` (both engines through the one codec), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4` + `MDLParser.g4`, `mdl/ast/ast_scheduledevent.go`, `mdl/visitor/visitor_scheduledevent.go`, `mdl/executor/cmd_scheduledevents.go`, `mdl/executor/validate_scheduled_events.go` (MDL-SCHED01) | **The generated metamodel is not the storage's ground truth** — `modelsdk/gen` declares `Interval`/`HourOfDay`/… as **int32** where Studio Pro writes **int64**, and `StartDateTime` as a **string** where Studio Pro writes a **BSON datetime**. This is the same mismatch as #585 (a reader that asserted int32), so the pattern is now twice-confirmed: when gen and an observed document disagree about a numeric width or a date, the document is right. **A byte-level pin beats field assertions**: three whole documents are hex-embedded in `codec_test.go` and re-serialized element by element in ORDER, so key order, key SET and BSON TYPE are all covered — a field-by-field test passes happily while writing int32 and an empty enum. **`mxcli marketplace download` is the Studio Pro substitute**: Business Events and Community Commons have none, but Workflow Commons 4.11.0, OIDC SSO 4.6.0 and SAML 4.2.1 have four between them (only Day and Hour variants — the other six had to be metamodel-derived, then verified to load). **An empty string is not "unset" for an enum property**: the first draft wrote `IntervalType: ""` and `mx check` reported 0 errors, but the enumeration has no such member. **`Interval`/`IntervalType` are LEGACY siblings of `Schedule` that Studio Pro does not keep in sync** — Workflow Commons stores `0`/`Minute` beside a `DaySchedule` of 01:00 — so they are derived on CREATE and carried through untouched on MODIFY, never re-derived. **A polymorphic child must be dispatched on `$Type`, and its field sets never merged** (the ADR-0005 rule): the eight variants differ in arity, so the executor refuses a field belonging to another Repeat rather than dropping it. **Validate at check time, not only at exec** — the validation is decidable from the statement, so it runs in the no-project pass and `check` and `exec` call the SAME function. **Prove the describe round-trips by PARSING its own output** for every variant, not by asserting on strings. Idempotence came free (ADR-0008 canon elides the regenerated `Schedule` `$ID`) — verified with the `MXCLI_ALWAYS_WRITE=1` control, which changes 9 units where the normal run changes 0. Tests `mdl/scheduledevents/codec_test.go`, `mdl/executor/cmd_scheduledevents_test.go`, `mdl/visitor/visitor_scheduledevent_test.go`, `mdl/backend/modelsdk/scheduledevent_read_test.go`, example `mdl-examples/doctype-tests/scheduled-events.mdl` | diff --git a/cmd/mxcli/cmd_check.go b/cmd/mxcli/cmd_check.go index f76861a97..d42da4257 100644 --- a/cmd/mxcli/cmd_check.go +++ b/cmd/mxcli/cmd_check.go @@ -216,6 +216,12 @@ Examples: // operation, so the mapping would be dropped in silence (#843). violations = append(violations, executor.ValidateRestClientMappings(prog)...) + // Flag a scheduled event whose Repeat and fields disagree (a Multiplier on + // a Daily repeat, an HourOfDay of 99). Decidable from the statement, so it + // runs here rather than at exec, where the script would already have + // passed check. + violations = append(violations, executor.ValidateScheduledEvents(prog)...) + if isStructured { // Always emit structured output (even when clean) formatter.Format(violations, os.Stderr) diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index 7016b7d5d..79749cd18 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -268,6 +268,7 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "IMAGE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "QUEUE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "QUEUES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "SCHEDULED", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "JAR", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCY", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCIES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 191cbcfac..088aeac6f 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -253,6 +253,89 @@ DESCRIBE QUEUE Ops.OrderProcessing; DROP QUEUE Ops.Mail;`, }) + // ── Scheduled events ──────────────────────────────────────────────── + + Register(SyntaxFeature{ + Path: "scheduled-event", + Summary: "Scheduled events — Mendix's cron: run a microflow on a repeating schedule", + Keywords: []string{ + "scheduled event", "scheduled events", "schedule", "cron", "recurring", + "create scheduled event", "drop scheduled event", "describe scheduled event", + "repeat", "daily", "hourly", "weekly", "monthly", "yearly", "timer", "batch job", + }, + Syntax: `CREATE [OR MODIFY] SCHEDULED EVENT Module.Name ( : , ... ); +SHOW SCHEDULED EVENTS [IN ]; +LIST SCHEDULED EVENTS [IN ]; +DESCRIBE SCHEDULED EVENT Module.Name; +DROP SCHEDULED EVENT Module.Name; + +Always required: + Microflow the microflow to run, as a qualified name + Repeat which schedule to use (below) + +Each Repeat takes ONLY its own fields; anything else is refused: + Minutely Multiplier + Hourly Multiplier, MinuteOffset + Daily HourOfDay, MinuteOfHour + Weekly Weekdays, HourOfDay, MinuteOfHour + MonthlyByDate Multiplier, MonthOffset, DayOfMonth, HourOfDay, MinuteOfHour + MonthlyByWeekday Multiplier, MonthOffset, DaySelector, Weekday, HourOfDay, MinuteOfHour + YearlyByDate Month, DayOfMonth, HourOfDay, MinuteOfHour + YearlyByWeekday Month, DaySelector, Weekday, HourOfDay, MinuteOfHour + + Weekdays is a quoted list: 'Monday, Friday'. DaySelector is First, Second, + Third, Fourth or Last. Weekday is Sunday..Saturday. Month and DayOfMonth are + numbers (1-12, 1-31). MonthOffset picks which month of a multi-month cycle + fires (0-based). + +Optional on any repeat: + Enabled true or false (default false) + OnOverlap DelayNext (default) or SkipNext — what happens when a run is + still going when the next one is due. This is a scheduled + event's own concurrency control; it does not use a task queue. + TimeZone UTC (default) or Server + StartDateTime an RFC 3339 timestamp; the event does not run before it + Documentation free text`, + Example: `CREATE SCHEDULED EVENT Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +CREATE SCHEDULED EVENT Ops.HourlyPing ( + Microflow: Ops.SE_Ping, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); + +CREATE SCHEDULED EVENT Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); + +CREATE SCHEDULED EVENT Ops.QuarterEnd ( + Microflow: Ops.SE_Close, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18 +); + +SHOW SCHEDULED EVENTS IN Ops; +DESCRIBE SCHEDULED EVENT Ops.NightlyCleanup; +DROP SCHEDULED EVENT Ops.HourlyPing;`, + SeeAlso: []string{"queue"}, + }) + // ── Structure ─────────────────────────────────────────────────────── Register(SyntaxFeature{ diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 0ea07706e..30656cbd1 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -111,6 +111,7 @@ - [Business Events](language/business-events.md) - [Event Services](language/event-services.md) - [Publishing and Consuming Events](language/pub-sub-events.md) +- [Scheduled Events and Task Queues](language/scheduled-events.md) - [Image Collections](language/image-collections.md) --- diff --git a/docs-site/src/language/scheduled-events.md b/docs-site/src/language/scheduled-events.md new file mode 100644 index 000000000..f6da79561 --- /dev/null +++ b/docs-site/src/language/scheduled-events.md @@ -0,0 +1,185 @@ +# Scheduled Events and Task Queues + +Two Mendix features for work that runs outside a user request. They are unrelated +and easy to confuse: + +- A **scheduled event** is Mendix's cron — it runs a microflow on a repeating + schedule. +- A **task queue** bounds how many queued microflow calls run at once. + +A scheduled event does **not** go through a task queue. Its own concurrency +control is `OnOverlap`, which decides what happens when a run is still going when +the next one is due. + +## Scheduled Events + +### Inspecting + +```sql +-- All scheduled events, or one module's +LIST SCHEDULED EVENTS; +LIST SCHEDULED EVENTS IN Ops; + +-- Re-executable MDL for one event +DESCRIBE SCHEDULED EVENT Ops.NightlyCleanup; +``` + +`SHOW` is accepted as a synonym for `LIST`. + +### CREATE SCHEDULED EVENT + +```sql +CREATE [OR MODIFY] SCHEDULED EVENT . ( + Microflow: ., + Repeat: , + , + +); +``` + +`Microflow` and `Repeat` are always required. + +#### Repeat variants + +The repeat rule is stored as one of eight Mendix schedule types, and they differ +in **which fields they carry** — not just in their values. MDL mirrors that: each +repeat takes only its own fields, and a field belonging to another repeat is +refused by both `mxcli check` (rule `MDL-SCHED01`) and `mxcli exec` rather than +silently dropped. + +| Repeat | Fields | Reads as | +|--------|--------|----------| +| `Minutely` | `Multiplier` | every N minutes | +| `Hourly` | `Multiplier`, `MinuteOffset` | every N hours, at :MM | +| `Daily` | `HourOfDay`, `MinuteOfHour` | every day at HH:MM | +| `Weekly` | `Weekdays`, `HourOfDay`, `MinuteOfHour` | on the named days at HH:MM | +| `MonthlyByDate` | `Multiplier`, `MonthOffset`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | the Dth of every N months | +| `MonthlyByWeekday` | `Multiplier`, `MonthOffset`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the last Friday of every N months | +| `YearlyByDate` | `Month`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | every 2 January | +| `YearlyByWeekday` | `Month`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the first Monday of March | + +Field values: + +| Field | Value | +|-------|-------| +| `Multiplier` | how many units between runs (1 or more; defaults to 1) | +| `MinuteOffset` | 0–59, the minute past the hour | +| `MonthOffset` | 0-based, which month of a multi-month cycle fires | +| `HourOfDay` / `MinuteOfHour` | 0–23 / 0–59 | +| `DayOfMonth` / `Month` | 1–31 / 1–12 | +| `Weekdays` | a quoted list, e.g. `'Monday, Friday'` (case-insensitive) | +| `DaySelector` | `First`, `Second`, `Third`, `Fourth`, `Last` | +| `Weekday` | `Sunday` … `Saturday` | + +A value outside its range is an error, not a truncation: a schedule that is +stored but can never fire is worse than a refusal, because nothing downstream +reports it. + +#### Optional properties + +| Property | Values | Default | +|----------|--------|---------| +| `Enabled` | `true` / `false` | `false` | +| `OnOverlap` | `DelayNext` / `SkipNext` | `DelayNext` | +| `TimeZone` | `UTC` / `Server` | `UTC` | +| `StartDateTime` | an RFC 3339 timestamp; the event does not run before it | none | +| `Documentation` | free text | none | + +`SkipNext` drops a run that would overlap the previous one; `DelayNext` queues it +until the previous one finishes. + +### Examples + +```sql +-- Every night at 04:00 in the server's timezone +CREATE SCHEDULED EVENT Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +-- Every two hours, 23 minutes past +CREATE SCHEDULED EVENT Ops.HourlyPing ( + Microflow: Ops.SE_Ping, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); + +-- Mondays and Fridays at 09:30 +CREATE SCHEDULED EVENT Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); + +-- The last Friday of every third month, at 18:00 +CREATE SCHEDULED EVENT Ops.QuarterEnd ( + Microflow: Ops.SE_Close, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18 +); + +DROP SCHEDULED EVENT Ops.HourlyPing; +``` + +### A note on `Interval` / `IntervalType` + +Stored events also carry an `Interval` and `IntervalType` pair. These predate the +`Schedule` child and Studio Pro does **not** keep them in sync with it — a real +Mendix module ships an event storing `0` / `Minute` next to a daily schedule of +01:00. MDL has no syntax for them: a new event gets the pair that matches its +repeat, and `CREATE OR MODIFY` carries whatever is stored through untouched. +`DESCRIBE` reports them as a comment so the output stays re-executable. + +## Task Queues + +A task queue bounds how many instances of a queued microflow call run at once. + +```sql +CREATE [OR MODIFY] QUEUE . [( + Parallelism: , + ClusterWide: true|false, + Documentation: '' +)]; + +LIST QUEUES [IN ]; +DESCRIBE QUEUE .; +DROP QUEUE .; +``` + +| Property | Meaning | Default | +|----------|---------|---------| +| `Parallelism` | how many tasks run at once — an **expression**, not a number | `1` | +| `ClusterWide` | `true` applies the limit across the cluster; `false` per runtime instance | `false` | + +Mendix stores parallelism as an expression string, so a bare integer and a quoted +one mean the same thing and an arbitrary expression is legal: + +```sql +CREATE QUEUE Ops.OrderProcessing ( Parallelism: 3, ClusterWide: true ); +CREATE QUEUE Ops.Mail; -- defaults: 1, per-instance +CREATE OR MODIFY QUEUE Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); +``` + +### Binding a call to a queue is not yet expressible + +MDL cannot yet author a *queued call* — the binding lives on the call activity +inside a microflow, not on the queue. Because rebuilding a microflow would drop +an existing binding, `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the +stored microflow has a queued call, naming the queues that would be lost. Change +those microflows in Studio Pro. + +Without that refusal the binding was written back as null and the project then +looked *healthier* than before — `mx check` stopped reporting +`CE1613 "The selected task queue no longer exists"`, because the configuration +the error was about had been deleted. diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 1aded5443..08d2da8b0 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -166,6 +166,58 @@ create or modify queue Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); drop queue Ops.Mail; ``` +## Scheduled Events + +Mendix's cron: run a microflow on a repeating schedule. + +| Statement | Syntax | Notes | +|-----------|--------|-------| +| Show scheduled events | `show scheduled events [in module];` (`list` too) | Repeat, microflow, enabled | +| Describe scheduled event | `describe scheduled event Module.Name;` | Re-executable MDL | +| Create scheduled event | `create [or modify] scheduled event Module.Name ( Microflow: ..., Repeat: ..., ... );` | | +| Drop scheduled event | `drop scheduled event Module.Name;` | | + +`Microflow` and `Repeat` are always required. Each repeat takes **only** its own +fields — anything else is refused by `mxcli check` (MDL-SCHED01) and by `exec`: + +| Repeat | Fields | +|--------|--------| +| `Minutely` | `Multiplier` | +| `Hourly` | `Multiplier`, `MinuteOffset` | +| `Daily` | `HourOfDay`, `MinuteOfHour` | +| `Weekly` | `Weekdays`, `HourOfDay`, `MinuteOfHour` | +| `MonthlyByDate` | `Multiplier`, `MonthOffset`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | +| `MonthlyByWeekday` | `Multiplier`, `MonthOffset`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | +| `YearlyByDate` | `Month`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | +| `YearlyByWeekday` | `Month`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | + +Optional on any repeat: `Enabled` (default false), `OnOverlap` +(`DelayNext` default / `SkipNext`), `TimeZone` (`UTC` default / `Server`), +`StartDateTime` (RFC 3339), `Documentation`. + +`OnOverlap` is a scheduled event's own concurrency control — scheduled events do +**not** go through a task queue. + +**Example:** +```sql +create scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +create scheduled event Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); +``` + ## OData Clients, Services & External Entities | Statement | Syntax | Notes | diff --git a/mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl b/mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl new file mode 100644 index 000000000..d1c869094 --- /dev/null +++ b/mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl @@ -0,0 +1,25 @@ +-- Scheduled event: a field from the wrong Repeat must be rejected by `check`. +-- +-- NEGATIVE TEST (.fail.mdl) — EXPECTED to fail `mxcli check`. +-- `make check-mdl` inverts the exit code for .fail.mdl files: an unexpected +-- pass would be reported as a regression of the MDL-SCHED01 rule. +-- +-- The eight ScheduledEvents$*Schedule variants differ in WHICH fields they +-- carry, not just in their values. A DaySchedule has no Multiplier, so writing +-- one would put a property on the node that the type does not declare — the +-- shape that mxbuild accepts and Studio Pro cannot open +-- (System.InvalidOperationException at MprProperty). +-- +-- The rule runs in the no-project pass, so a plain `mxcli check` catches it — +-- it is decidable from the statement alone, and exec calls the same function, +-- so check and exec cannot drift. +-- +-- Correct form: drop Multiplier (Daily takes HourOfDay and MinuteOfHour), or +-- switch to a Repeat that has one. + +create scheduled event Ops.Bad ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + Multiplier: 3, + HourOfDay: 4 +); diff --git a/mdl-examples/doctype-tests/scheduled-events.mdl b/mdl-examples/doctype-tests/scheduled-events.mdl new file mode 100644 index 000000000..b4d3ed555 --- /dev/null +++ b/mdl-examples/doctype-tests/scheduled-events.mdl @@ -0,0 +1,133 @@ +-- ============================================================================ +-- Scheduled events — all eight repeat variants +-- ============================================================================ +-- +-- Mendix's cron. A scheduled event runs a microflow on a repeating schedule; +-- the repeat rule is a ScheduledEvents$Schedule child with eight variants that +-- differ in WHICH fields they carry, so MDL names the variant (Repeat) and then +-- takes only that variant's fields. A field from another variant is refused — +-- writing a merged field set produces a document mxbuild accepts and Studio Pro +-- cannot open. +-- +-- Scheduled events do NOT go through a task queue: OnOverlap +-- (DelayNext | SkipNext) is their own concurrency control. +-- +-- The document shape is pinned against four Studio Pro-authored events (Workflow +-- Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1 ×2). Those cover Daily and Hourly; +-- the other six variants are metamodel-derived and verified to load with 0 +-- errors from `mx check`. +-- +-- Verify: +-- mxcli exec mdl-examples/doctype-tests/scheduled-events.mdl -p app.mpr +-- mx check app.mpr # 0 errors +-- mxcli -p app.mpr -c "list scheduled events" +-- mxcli -p app.mpr -c "describe scheduled event Ops.NightlyCleanup" +-- ============================================================================ + +create module Ops; +/ +create microflow Ops.SE_Cleanup() +begin + return; +end; +/ +-- Daily: a fixed time of day, no multiplier (the variant has none). +create scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true, + StartDateTime: '2026-01-01T04:00:00Z' +); +/ +-- Hourly: every N hours, at a fixed minute past the hour. +create scheduled event Ops.HourlyPing ( + Microflow: Ops.SE_Cleanup, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); +/ +-- Minutely: the only field is the multiplier. +create scheduled event Ops.EveryFiveMinutes ( + Microflow: Ops.SE_Cleanup, + Repeat: Minutely, + Multiplier: 5 +); +/ +-- Weekly: a day list plus a time. Weekday names are case-insensitive. +create scheduled event Ops.WeeklyReport ( + Microflow: Ops.SE_Cleanup, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); +/ +-- MonthlyByDate: the Nth day of every M months. +create scheduled event Ops.MidMonth ( + Microflow: Ops.SE_Cleanup, + Repeat: MonthlyByDate, + Multiplier: 1, + DayOfMonth: 15, + HourOfDay: 2, + MinuteOfHour: 0 +); +/ +-- MonthlyByWeekday: "the last Friday of every third month". MonthOffset picks +-- which month of the three-month cycle fires (0-based). +create scheduled event Ops.QuarterEnd ( + Microflow: Ops.SE_Cleanup, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18, + MinuteOfHour: 0 +); +/ +-- YearlyByDate: a calendar date. Month is 1-12. +create scheduled event Ops.YearOpen ( + Microflow: Ops.SE_Cleanup, + Repeat: YearlyByDate, + Month: 1, + DayOfMonth: 2, + HourOfDay: 8, + MinuteOfHour: 0 +); +/ +-- YearlyByWeekday: "the first Monday of March". +create scheduled event Ops.YearWeek ( + Microflow: Ops.SE_Cleanup, + Repeat: YearlyByWeekday, + Month: 3, + DaySelector: First, + Weekday: Monday, + HourOfDay: 7, + MinuteOfHour: 15 +); +/ +-- Re-running with OR MODIFY reuses the stored $ID and preserves the legacy +-- Interval/IntervalType pair, which Studio Pro writes but does not keep in sync +-- with Schedule and MDL cannot author. +create or modify scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 5, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true, + OnOverlap: SkipNext +); +/ +-- "scheduled" and "event" stay usable as ordinary names: both are in the +-- grammar's `keyword` rule, so adding the tokens did not take them away. +create entity Ops.Job ( + scheduled: Boolean, + event: String(50) +); +/ +drop scheduled event Ops.EveryFiveMinutes; diff --git a/mdl/ast/ast_scheduledevent.go b/mdl/ast/ast_scheduledevent.go new file mode 100644 index 000000000..7e97bf0ca --- /dev/null +++ b/mdl/ast/ast_scheduledevent.go @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: Apache-2.0 + +package ast + +// CreateScheduledEventStmt represents: +// +// CREATE [OR REPLACE|MODIFY] SCHEDULED EVENT Module.Name ( +// Microflow: Module.MF, Repeat: Daily, HourOfDay: 4, ... +// ); +// +// Properties are kept as written and validated by the executor, which knows +// which fields the chosen Repeat actually has. Numeric fields are pointers so +// "not mentioned" stays distinguishable from "mentioned as 0" — 0 is a real +// hour, minute and month offset. +type CreateScheduledEventStmt struct { + Name QualifiedName + Documentation string + // Microflow is the qualified name of the microflow to run. + Microflow string + // Repeat names the schedule variant: Minutely, Hourly, Daily, Weekly, + // MonthlyByDate, MonthlyByWeekday, YearlyByDate, YearlyByWeekday. + Repeat string + + Multiplier *int + MinuteOffset *int + MonthOffset *int + HourOfDay *int + MinuteOfHour *int + DayOfMonth *int + Month *int + + // Weekdays is the day list of a Weekly repeat, as written + // ("Monday, Friday"); DaySelector/Weekday are the two fields of the + // ByWeekday repeats. + Weekdays string + DaySelector string + Weekday string + + // StartDateTime is an RFC 3339 timestamp; TimeZone is UTC or Server. + StartDateTime string + TimeZone string + // OnOverlap is DelayNext or SkipNext — what happens when a run is still + // going when the next one is due. + OnOverlap string + + Enabled *bool + Excluded *bool + ExportLevel string + + // CreateOrModify is set by the shared CREATE OR REPLACE/MODIFY prefix. + CreateOrModify bool +} + +func (s *CreateScheduledEventStmt) isStatement() {} + +// DropScheduledEventStmt represents: DROP SCHEDULED EVENT Module.Name; +type DropScheduledEventStmt struct { + Name QualifiedName +} + +func (s *DropScheduledEventStmt) isStatement() {} + +// ShowScheduledEventsStmt represents: SHOW|LIST SCHEDULED EVENTS [IN Module]; +type ShowScheduledEventsStmt struct { + Module string +} + +func (s *ShowScheduledEventsStmt) isStatement() {} + +// DescribeScheduledEventStmt represents: DESCRIBE SCHEDULED EVENT Module.Name; +type DescribeScheduledEventStmt struct { + Name QualifiedName +} + +func (s *DescribeScheduledEventStmt) isStatement() {} diff --git a/mdl/backend/infrastructure.go b/mdl/backend/infrastructure.go index 9f3f822ca..ff88a0ddb 100644 --- a/mdl/backend/infrastructure.go +++ b/mdl/backend/infrastructure.go @@ -96,4 +96,7 @@ type QueueBackend interface { type ScheduledEventBackend interface { ListScheduledEvents() ([]*model.ScheduledEvent, error) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, error) + CreateScheduledEvent(ev *model.ScheduledEvent) error + UpdateScheduledEvent(ev *model.ScheduledEvent) error + DeleteScheduledEvent(id string) error } diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index 5fbba52d6..d39502993 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -219,6 +219,11 @@ func (unsupportedBackend) CreateQueue(_ *types.Queue) (err0 error) { return } +func (unsupportedBackend) CreateScheduledEvent(_ *model.ScheduledEvent) (err0 error) { + err0 = errUnsupported("CreateScheduledEvent") + return +} + func (unsupportedBackend) CreateSnippet(_ *pages.Snippet) (err0 error) { err0 = errUnsupported("CreateSnippet") return @@ -399,6 +404,11 @@ func (unsupportedBackend) DeleteQueue(_ string) (err0 error) { return } +func (unsupportedBackend) DeleteScheduledEvent(_ string) (err0 error) { + err0 = errUnsupported("DeleteScheduledEvent") + return +} + func (unsupportedBackend) DeleteSnippet(_ model.ID) (err0 error) { err0 = errUnsupported("DeleteSnippet") return @@ -1207,6 +1217,11 @@ func (unsupportedBackend) UpdateRawUnit(_ string, _ []uint8) (err0 error) { return } +func (unsupportedBackend) UpdateScheduledEvent(_ *model.ScheduledEvent) (err0 error) { + err0 = errUnsupported("UpdateScheduledEvent") + return +} + func (unsupportedBackend) UpdateSnippet(_ *pages.Snippet) (err0 error) { err0 = errUnsupported("UpdateSnippet") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index 3af35404e..bf72ffb19 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -257,8 +257,11 @@ type MockBackend struct { DeleteQueueFunc func(id string) error // ScheduledEventBackend - ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) - GetScheduledEventFunc func(id model.ID) (*model.ScheduledEvent, error) + ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) + GetScheduledEventFunc func(id model.ID) (*model.ScheduledEvent, error) + CreateScheduledEventFunc func(ev *model.ScheduledEvent) error + UpdateScheduledEventFunc func(ev *model.ScheduledEvent) error + DeleteScheduledEventFunc func(id string) error // RenameBackend UpdateQualifiedNameInAllUnitsFunc func(oldName, newName string) (int, error) diff --git a/mdl/backend/mock/mock_workflow.go b/mdl/backend/mock/mock_workflow.go index 84cf8464b..4d792482f 100644 --- a/mdl/backend/mock/mock_workflow.go +++ b/mdl/backend/mock/mock_workflow.go @@ -123,3 +123,28 @@ func (m *MockBackend) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, err } return nil, nil } + +// The write methods default to a descriptive error rather than nil so a test +// that forgets to configure one fails on the missing stub instead of silently +// reporting a write that never happened. + +func (m *MockBackend) CreateScheduledEvent(ev *model.ScheduledEvent) error { + if m.CreateScheduledEventFunc != nil { + return m.CreateScheduledEventFunc(ev) + } + return fmt.Errorf("MockBackend.CreateScheduledEvent not configured") +} + +func (m *MockBackend) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + if m.UpdateScheduledEventFunc != nil { + return m.UpdateScheduledEventFunc(ev) + } + return fmt.Errorf("MockBackend.UpdateScheduledEvent not configured") +} + +func (m *MockBackend) DeleteScheduledEvent(id string) error { + if m.DeleteScheduledEventFunc != nil { + return m.DeleteScheduledEventFunc(id) + } + return fmt.Errorf("MockBackend.DeleteScheduledEvent not configured") +} diff --git a/mdl/backend/modelsdk/scheduledevent_read.go b/mdl/backend/modelsdk/scheduledevent_read.go index 8bf928674..9d4e8727a 100644 --- a/mdl/backend/modelsdk/scheduledevent_read.go +++ b/mdl/backend/modelsdk/scheduledevent_read.go @@ -5,60 +5,88 @@ package modelsdkbackend import ( "fmt" - genSched "github.com/mendixlabs/mxcli/modelsdk/gen/scheduledevents" - "github.com/mendixlabs/mxcli/modelsdk/mprread" + "go.mongodb.org/mongo-driver/bson" + sched "github.com/mendixlabs/mxcli/mdl/scheduledevents" "github.com/mendixlabs/mxcli/model" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" ) -// Codec-native scheduled-event read. Used by SHOW STRUCTURE (per-module counts) -// and the project tree; the modelsdk engine previously left these unimplemented, -// so scheduled events were silently undercounted (the callers swallow the -// not-implemented error). Unlike JavaScript actions, the gen codec decodes -// ScheduledEvent under the same storage keys the legacy parser uses (Name / -// Documentation / Microflow / Enabled / Interval / IntervalType), so gen -// accessors are correct here — no raw-key reads needed. +// Scheduled events. Used by SHOW STRUCTURE (per-module counts), the project +// tree, and SHOW/DESCRIBE/CREATE/DROP SCHEDULED EVENT. +// +// The document is read and written as raw BSON through mdl/scheduledevents +// rather than through modelsdk/gen, because gen's generated types disagree with +// what Studio Pro actually writes on two properties: every integer is stored as +// int64 while gen declares int32 (the mismatch behind issue #585), and +// StartDateTime is a BSON UTC datetime while gen declares a string. func (b *Backend) ListScheduledEvents() ([]*model.ScheduledEvent, error) { - units, err := mprread.ListUnitsWithContainer[*genSched.ScheduledEvent](b.reader) + units, err := b.reader.ListRawUnitsByType(sched.TypeName) if err != nil { return nil, err } out := make([]*model.ScheduledEvent, 0, len(units)) for _, u := range units { - out = append(out, scheduledEventFromGen(u.Element, u.ContainerID)) + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + return nil, fmt.Errorf("unmarshal scheduled event %s: %w", u.ID, err) + } + out = append(out, sched.Parse(doc, model.ID(u.ID), model.ID(u.ContainerID))) } return out, nil } func (b *Backend) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, error) { - units, err := mprread.ListUnitsWithContainer[*genSched.ScheduledEvent](b.reader) + events, err := b.ListScheduledEvents() if err != nil { return nil, err } - for _, u := range units { - if model.ID(u.Element.ID()) == id { - return scheduledEventFromGen(u.Element, u.ContainerID), nil + for _, ev := range events { + if ev.ID == id { + return ev, nil } } return nil, fmt.Errorf("scheduled event not found: %s", id) } -// scheduledEventFromGen converts a gen ScheduledEvent to the semantic type. Mirrors -// the legacy parseScheduledEvent field set: MicroflowID holds the by-name microflow -// reference (BSON "Microflow"), and Interval comes through the int32 decoder (which -// accepts the int64 Studio Pro actually writes — issue #585). -func scheduledEventFromGen(g *genSched.ScheduledEvent, containerID model.ID) *model.ScheduledEvent { - ev := &model.ScheduledEvent{ - ContainerID: containerID, - Name: g.Name(), - Documentation: g.Documentation(), - MicroflowID: model.ID(g.MicroflowQualifiedName()), - Interval: int(g.Interval()), - IntervalType: g.IntervalType(), - Enabled: g.Enabled(), +// CreateScheduledEvent inserts a new scheduled event document. +func (b *Backend) CreateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("CreateScheduledEvent: nil event") + } + if b.writer == nil { + return fmt.Errorf("CreateScheduledEvent: not connected for writing") + } + if ev.ID == "" { + ev.ID = model.ID(mmpr.GenerateID()) + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return b.writer.InsertUnit(string(ev.ID), string(ev.ContainerID), "Documents", sched.TypeName, contents) +} + +// UpdateScheduledEvent rewrites an existing scheduled event in place. +func (b *Backend) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("UpdateScheduledEvent: nil event") + } + if b.writer == nil { + return fmt.Errorf("UpdateScheduledEvent: not connected for writing") + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return b.writer.UpdateRawUnit(string(ev.ID), contents) +} + +// DeleteScheduledEvent removes a scheduled event unit by ID. +func (b *Backend) DeleteScheduledEvent(id string) error { + if b.writer == nil { + return fmt.Errorf("DeleteScheduledEvent: not connected for writing") } - ev.ID = model.ID(g.ID()) - ev.TypeName = "ScheduledEvents$ScheduledEvent" - return ev + return b.writer.DeleteUnit(id) } diff --git a/mdl/backend/modelsdk/scheduledevent_read_test.go b/mdl/backend/modelsdk/scheduledevent_read_test.go index bb97048f6..01ef70971 100644 --- a/mdl/backend/modelsdk/scheduledevent_read_test.go +++ b/mdl/backend/modelsdk/scheduledevent_read_test.go @@ -4,58 +4,11 @@ package modelsdkbackend import ( "testing" + "time" "github.com/mendixlabs/mxcli/model" - genSched "github.com/mendixlabs/mxcli/modelsdk/gen/scheduledevents" ) -// TestScheduledEventFromGen guards the gen→semantic mapping the modelsdk read -// relies on. (No committed fixture contains scheduled events, and modelsdk has no -// scheduled-event write path, so the converter is exercised directly; the -// List/Get plumbing reuses the ListUnitsWithContainer pattern covered by the -// page/java reads.) The key mappings: MicroflowID carries the by-name microflow -// reference (BSON "Microflow"), Interval narrows the int32 accessor to int. -func TestScheduledEventFromGen(t *testing.T) { - g := genSched.NewScheduledEvent() - g.SetID("evt-1") - g.SetName("SE_Cleanup") - g.SetDocumentation("nightly cleanup") - g.SetMicroflowQualifiedName("MyModule.DoCleanup") - g.SetInterval(86400) - g.SetIntervalType("Day") - g.SetEnabled(true) - - ev := scheduledEventFromGen(g, model.ID("mod-1")) - - if ev.ID != "evt-1" { - t.Errorf("ID = %q, want evt-1", ev.ID) - } - if ev.ContainerID != "mod-1" { - t.Errorf("ContainerID = %q, want mod-1", ev.ContainerID) - } - if ev.TypeName != "ScheduledEvents$ScheduledEvent" { - t.Errorf("TypeName = %q", ev.TypeName) - } - if ev.Name != "SE_Cleanup" { - t.Errorf("Name = %q, want SE_Cleanup", ev.Name) - } - if ev.Documentation != "nightly cleanup" { - t.Errorf("Documentation = %q", ev.Documentation) - } - if ev.MicroflowID != "MyModule.DoCleanup" { - t.Errorf("MicroflowID = %q, want MyModule.DoCleanup", ev.MicroflowID) - } - if ev.Interval != 86400 { - t.Errorf("Interval = %d, want 86400", ev.Interval) - } - if ev.IntervalType != "Day" { - t.Errorf("IntervalType = %q, want Day", ev.IntervalType) - } - if !ev.Enabled { - t.Error("Enabled = false, want true") - } -} - // TestListScheduledEvents_Empty confirms the read returns an empty (not error) // result on a project with no scheduled events — the minimal fixture — so SHOW // STRUCTURE no longer swallows a not-implemented error. @@ -74,3 +27,83 @@ func TestListScheduledEvents_Empty(t *testing.T) { t.Errorf("got %d scheduled events, want 0 (minimal fixture has none)", len(events)) } } + +// TestScheduledEventRoundTrip writes an event and reads it back through the real +// backend. A round trip (rather than a reader-only test on hand-written BSON) is +// what catches a reader keyed on different property names than the writer +// produces — the failure mode recorded for the workflow-activity reads. +func TestScheduledEventRoundTrip(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + start := time.Date(2026, 1, 1, 4, 0, 0, 0, time.UTC) + want := &model.ScheduledEvent{ + ContainerID: mod.ID, + Name: "ZzNightly", + Documentation: "nightly cleanup", + MicroflowID: "MyFirstModule.DoCleanup", + StartDateTime: &start, + TimeZone: "Server", + OnOverlap: "SkipNext", + Interval: 1, + IntervalType: "Day", + Enabled: true, + Schedule: &model.Schedule{ + Kind: model.ScheduleMonthWeekday, Multiplier: 3, MonthOffset: 2, + DaySelector: "Last", Weekday: "Friday", HourOfDay: 18, MinuteOfHour: 30, + }, + } + if err := b.CreateScheduledEvent(want); err != nil { + t.Fatalf("CreateScheduledEvent: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + events, err := b2.ListScheduledEvents() + if err != nil { + t.Fatalf("ListScheduledEvents: %v", err) + } + for _, got := range events { + if got.Name != "ZzNightly" { + continue + } + if got.MicroflowID != want.MicroflowID { + t.Errorf("MicroflowID = %q", got.MicroflowID) + } + if got.Documentation != want.Documentation { + t.Errorf("Documentation = %q", got.Documentation) + } + if got.TimeZone != "Server" || got.OnOverlap != "SkipNext" { + t.Errorf("TimeZone/OnOverlap = %q/%q", got.TimeZone, got.OnOverlap) + } + if got.Interval != 1 || got.IntervalType != "Day" { + t.Errorf("legacy pair = %d/%q", got.Interval, got.IntervalType) + } + if !got.Enabled { + t.Error("Enabled did not round-trip") + } + if got.StartDateTime == nil || !got.StartDateTime.Equal(start) { + t.Errorf("StartDateTime = %v, want %v", got.StartDateTime, start) + } + if got.Schedule == nil { + t.Fatal("Schedule did not round-trip") + } + if *got.Schedule != *want.Schedule { + t.Errorf("Schedule = %+v, want %+v", *got.Schedule, *want.Schedule) + } + return + } + t.Fatalf("ZzNightly not found after create (got %d events)", len(events)) +} diff --git a/mdl/backend/modelsdk/unimplemented_gen.go b/mdl/backend/modelsdk/unimplemented_gen.go index 62cea55be..f3ceed68d 100644 --- a/mdl/backend/modelsdk/unimplemented_gen.go +++ b/mdl/backend/modelsdk/unimplemented_gen.go @@ -184,6 +184,10 @@ func (unimplemented) CreateQueue(_ *types.Queue) error { return errUnimplemented("CreateQueue") } +func (unimplemented) CreateScheduledEvent(_ *model.ScheduledEvent) error { + return errUnimplemented("CreateScheduledEvent") +} + func (unimplemented) CreateSnippet(_ *pages.Snippet) error { return errUnimplemented("CreateSnippet") } @@ -329,6 +333,10 @@ func (unimplemented) DeleteQueue(_ string) error { return errUnimplemented("DeleteQueue") } +func (unimplemented) DeleteScheduledEvent(_ string) error { + return errUnimplemented("DeleteScheduledEvent") +} + func (unimplemented) DeleteSnippet(_ model.ID) error { return errUnimplemented("DeleteSnippet") } @@ -1082,6 +1090,10 @@ func (unimplemented) UpdateRawUnit(_ string, _ []uint8) error { return errUnimplemented("UpdateRawUnit") } +func (unimplemented) UpdateScheduledEvent(_ *model.ScheduledEvent) error { + return errUnimplemented("UpdateScheduledEvent") +} + func (unimplemented) UpdateSnippet(_ *pages.Snippet) error { return errUnimplemented("UpdateSnippet") } diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 44050fa80..a539d898d 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -729,6 +729,15 @@ func (b *MprBackend) ListScheduledEvents() ([]*model.ScheduledEvent, error) { func (b *MprBackend) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, error) { return b.reader.GetScheduledEvent(id) } +func (b *MprBackend) CreateScheduledEvent(ev *model.ScheduledEvent) error { + return b.writer.CreateScheduledEvent(ev) +} +func (b *MprBackend) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + return b.writer.UpdateScheduledEvent(ev) +} +func (b *MprBackend) DeleteScheduledEvent(id string) error { + return b.writer.DeleteScheduledEvent(id) +} // --------------------------------------------------------------------------- // RenameBackend diff --git a/mdl/executor/cmd_queues_mock_test.go b/mdl/executor/cmd_queues_mock_test.go index d3424afeb..c004b8df8 100644 --- a/mdl/executor/cmd_queues_mock_test.go +++ b/mdl/executor/cmd_queues_mock_test.go @@ -77,9 +77,9 @@ func TestCreateQueue_Mock_PassesParallelismThrough(t *testing.T) { var created *types.Queue mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, CreateQueueFunc: func(q *types.Queue) error { created = q return nil @@ -113,9 +113,9 @@ func TestCreateQueue_Mock_DuplicateWithoutOrModify(t *testing.T) { withContainer(h, existing.ContainerID, mod.ID) mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, CreateQueueFunc: func(q *types.Queue) error { t.Error("CreateQueue must not be called for a duplicate") return nil @@ -137,9 +137,9 @@ func TestCreateQueue_Mock_OrModifyUpdates(t *testing.T) { var updated *types.Queue mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, UpdateQueueFunc: func(q *types.Queue) error { updated = q return nil diff --git a/mdl/executor/cmd_scheduledevents.go b/mdl/executor/cmd_scheduledevents.go new file mode 100644 index 000000000..a89b2c4ac --- /dev/null +++ b/mdl/executor/cmd_scheduledevents.go @@ -0,0 +1,586 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package executor — scheduled event commands (CREATE/DROP/SHOW/DESCRIBE +// SCHEDULED EVENT). +// +// A Mendix scheduled event runs a microflow on a repeating schedule. It is +// Mendix's cron: the repeat rule is a ScheduledEvents$Schedule child with eight +// variants, and OnOverlap (DelayNext|SkipNext) is its own concurrency control — +// scheduled events do not go through a task queue. +package executor + +import ( + "fmt" + "sort" + "strings" + "time" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" +) + +// repeatKinds maps the MDL Repeat value to the storage's schedule variant. +// The MDL spellings are the calendar words a reader expects; the storage names +// are Mendix's. +var repeatKinds = map[string]model.ScheduleKind{ + "minutely": model.ScheduleMinute, + "hourly": model.ScheduleHour, + "daily": model.ScheduleDay, + "weekly": model.ScheduleWeek, + "monthlybydate": model.ScheduleMonthDate, + "monthlybyweekday": model.ScheduleMonthWeekday, + "yearlybydate": model.ScheduleYearDate, + "yearlybyweekday": model.ScheduleYearWeekday, +} + +// repeatNames is the reverse map, for DESCRIBE. +var repeatNames = map[model.ScheduleKind]string{ + model.ScheduleMinute: "Minutely", + model.ScheduleHour: "Hourly", + model.ScheduleDay: "Daily", + model.ScheduleWeek: "Weekly", + model.ScheduleMonthDate: "MonthlyByDate", + model.ScheduleMonthWeekday: "MonthlyByWeekday", + model.ScheduleYearDate: "YearlyByDate", + model.ScheduleYearWeekday: "YearlyByWeekday", +} + +// repeatFields lists the properties each repeat actually uses. A property that +// belongs to a different variant is REFUSED rather than ignored: the variants +// differ in which fields they carry, so silently dropping one would write a +// schedule that does not do what the script says. +var repeatFields = map[model.ScheduleKind][]string{ + model.ScheduleMinute: {"Multiplier"}, + model.ScheduleHour: {"Multiplier", "MinuteOffset"}, + model.ScheduleDay: {"HourOfDay", "MinuteOfHour"}, + model.ScheduleWeek: {"Weekdays", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthDate: {"Multiplier", "MonthOffset", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthWeekday: {"Multiplier", "MonthOffset", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearDate: {"Month", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearWeekday: {"Month", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, +} + +var daySelectors = []string{"First", "Second", "Third", "Fourth", "Last"} + +var weekdayNames = []string{"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"} + +// findScheduledEvent returns the event with the given module-qualified name, or nil. +func findScheduledEvent(ctx *ExecContext, moduleName, name string) *model.ScheduledEvent { + events, err := ctx.Backend.ListScheduledEvents() + if err != nil { + return nil + } + h, err := getHierarchy(ctx) + if err != nil { + return nil + } + for _, ev := range events { + if !strings.EqualFold(ev.Name, name) { + continue + } + mod := h.GetModuleName(h.FindModuleID(ev.ContainerID)) + if strings.EqualFold(mod, moduleName) { + return ev + } + } + return nil +} + +// execCreateScheduledEvent handles CREATE [OR REPLACE|MODIFY] SCHEDULED EVENT. +func execCreateScheduledEvent(ctx *ExecContext, s *ast.CreateScheduledEventStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + + module, err := findOrCreateModule(ctx, s.Name.Module) + if err != nil { + return err + } + + existing := findScheduledEvent(ctx, s.Name.Module, s.Name.Name) + if existing != nil && !s.CreateOrModify { + return mdlerrors.NewAlreadyExists("scheduled event", s.Name.String()) + } + + ev, err := scheduledEventFromStmt(s) + if err != nil { + return err + } + ev.ContainerID = module.ID + if existing != nil { + ev.ID = existing.ID + ev.ContainerID = existing.ContainerID + // Interval/IntervalType are legacy siblings of Schedule that Studio Pro + // writes but does not keep in sync, and MDL has no syntax for them. + // Carry the stored values so a modify does not invent new ones. + ev.Interval = existing.Interval + ev.IntervalType = existing.IntervalType + if err := ctx.Backend.UpdateScheduledEvent(ev); err != nil { + return mdlerrors.NewBackend("update scheduled event", err) + } + fmt.Fprintf(ctx.Output, "Modified scheduled event: %s\n", s.Name.String()) + return nil + } + if err := ctx.Backend.CreateScheduledEvent(ev); err != nil { + return mdlerrors.NewBackend("create scheduled event", err) + } + fmt.Fprintf(ctx.Output, "Created scheduled event: %s\n", s.Name.String()) + return nil +} + +// scheduledEventFromStmt validates the statement and builds the semantic event. +func scheduledEventFromStmt(s *ast.CreateScheduledEventStmt) (*model.ScheduledEvent, error) { + if strings.TrimSpace(s.Microflow) == "" { + return nil, mdlerrors.NewValidation( + "scheduled event " + s.Name.String() + " has no Microflow — a scheduled event runs a microflow, " + + "so add `Microflow: Module.MyMicroflow` to the property list") + } + ev := &model.ScheduledEvent{ + Name: s.Name.Name, + Documentation: s.Documentation, + MicroflowID: model.ID(s.Microflow), + TimeZone: s.TimeZone, + OnOverlap: s.OnOverlap, + ExportLevel: s.ExportLevel, + } + if s.Enabled != nil { + ev.Enabled = *s.Enabled + } + if s.Excluded != nil { + ev.Excluded = *s.Excluded + } + if err := validateEnumProperty("TimeZone", ev.TimeZone, []string{"UTC", "Server"}); err != nil { + return nil, err + } + if err := validateEnumProperty("OnOverlap", ev.OnOverlap, []string{"DelayNext", "SkipNext"}); err != nil { + return nil, err + } + if s.StartDateTime != "" { + t, err := time.Parse(time.RFC3339, s.StartDateTime) + if err != nil { + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "StartDateTime %q is not an RFC 3339 timestamp (e.g. '2026-01-01T04:00:00Z')", s.StartDateTime)) + } + utc := t.UTC() + ev.StartDateTime = &utc + } + + sched, err := scheduleFromStmt(s) + if err != nil { + return nil, err + } + ev.Schedule = sched + ev.Interval, ev.IntervalType = legacyIntervalFor(sched) + return ev, nil +} + +// legacyIntervalFor derives the Interval/IntervalType pair a new event gets. +// +// These are legacy siblings of Schedule with no MDL syntax, but they are not +// optional: IntervalType is a Mendix enumeration, and writing an empty string +// puts a value in the model that the enumeration does not have. Every Studio +// Pro-authored event carries a real one. +// +// The mapping is what the two self-consistent references show (SAML: Day/1 +// beside a DaySchedule; OIDC: Hour/1 beside an HourSchedule with Multiplier 1). +// It is only used on CREATE — a modify carries the stored pair through +// untouched, because Studio Pro does NOT keep these in sync with Schedule (the +// Workflow Commons event stores Minute/0 next to a DaySchedule) and re-deriving +// them would overwrite whatever the developer's Studio Pro left behind. +func legacyIntervalFor(s *model.Schedule) (int, string) { + if s == nil { + return 1, "Day" + } + switch s.Kind { + case model.ScheduleMinute: + return s.Multiplier, "Minute" + case model.ScheduleHour: + return s.Multiplier, "Hour" + case model.ScheduleDay: + return 1, "Day" + case model.ScheduleWeek: + return 1, "Week" + case model.ScheduleMonthDate, model.ScheduleMonthWeekday: + return s.Multiplier, "Month" + case model.ScheduleYearDate, model.ScheduleYearWeekday: + return 1, "Year" + } + return 1, "Day" +} + +// scheduleFromStmt builds the repeat rule, refusing any field that does not +// belong to the chosen Repeat. +func scheduleFromStmt(s *ast.CreateScheduledEventStmt) (*model.Schedule, error) { + if strings.TrimSpace(s.Repeat) == "" { + return nil, mdlerrors.NewValidation( + "scheduled event " + s.Name.String() + " has no Repeat — add one of " + + strings.Join(sortedRepeatNames(), ", ")) + } + kind, ok := repeatKinds[strings.ToLower(s.Repeat)] + if !ok { + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "unknown Repeat %q — expected one of %s", s.Repeat, strings.Join(sortedRepeatNames(), ", "))) + } + + // Which properties the statement actually set, so a field belonging to a + // different variant is reported instead of silently dropped. + set := map[string]bool{ + "Multiplier": s.Multiplier != nil, + "MinuteOffset": s.MinuteOffset != nil, + "MonthOffset": s.MonthOffset != nil, + "HourOfDay": s.HourOfDay != nil, + "MinuteOfHour": s.MinuteOfHour != nil, + "DayOfMonth": s.DayOfMonth != nil, + "Month": s.Month != nil, + "Weekdays": s.Weekdays != "", + "DaySelector": s.DaySelector != "", + "Weekday": s.Weekday != "", + } + allowed := map[string]bool{} + for _, f := range repeatFields[kind] { + allowed[f] = true + } + var stray []string + for f, isSet := range set { + if isSet && !allowed[f] { + stray = append(stray, f) + } + } + if len(stray) > 0 { + sort.Strings(stray) + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "Repeat %s does not have %s — it takes %s", + s.Repeat, strings.Join(stray, ", "), strings.Join(repeatFields[kind], ", "))) + } + + sched := &model.Schedule{Kind: kind} + if s.Multiplier != nil { + sched.Multiplier = *s.Multiplier + } else if allowed["Multiplier"] { + // "every 1 " — an unstated multiplier of 0 would never fire. + sched.Multiplier = 1 + } + if s.MinuteOffset != nil { + sched.MinuteOffset = *s.MinuteOffset + } + if s.MonthOffset != nil { + sched.MonthOffset = *s.MonthOffset + } + if s.HourOfDay != nil { + sched.HourOfDay = *s.HourOfDay + } + if s.MinuteOfHour != nil { + sched.MinuteOfHour = *s.MinuteOfHour + } + if s.DayOfMonth != nil { + sched.DayOfMonth = *s.DayOfMonth + } else if allowed["DayOfMonth"] { + sched.DayOfMonth = 1 + } + if s.Month != nil { + sched.Month = *s.Month + } else if allowed["Month"] { + sched.Month = 1 + } + sched.DaySelector = s.DaySelector + sched.Weekday = s.Weekday + + if err := validateScheduleRanges(sched, allowed); err != nil { + return nil, err + } + if allowed["Weekdays"] { + days, err := parseWeekdays(s.Weekdays) + if err != nil { + return nil, err + } + sched.Weekdays = days + } + if allowed["DaySelector"] { + if err := validateEnumProperty("DaySelector", sched.DaySelector, daySelectors); err != nil { + return nil, err + } + if sched.DaySelector == "" { + sched.DaySelector = "First" + } + if err := validateEnumProperty("Weekday", sched.Weekday, weekdayNames); err != nil { + return nil, err + } + if sched.Weekday == "" { + sched.Weekday = "Monday" + } + } + return sched, nil +} + +// validateScheduleRanges rejects values Mendix's editor would not let you enter. +// A schedule that is stored but can never fire is worse than a refusal, because +// nothing downstream reports it. +func validateScheduleRanges(s *model.Schedule, allowed map[string]bool) error { + checks := []struct { + name string + value int + min, max int + }{ + {"Multiplier", s.Multiplier, 1, 1_000_000}, + {"MinuteOffset", s.MinuteOffset, 0, 59}, + {"MonthOffset", s.MonthOffset, 0, 11}, + {"HourOfDay", s.HourOfDay, 0, 23}, + {"MinuteOfHour", s.MinuteOfHour, 0, 59}, + {"DayOfMonth", s.DayOfMonth, 1, 31}, + {"Month", s.Month, 1, 12}, + } + for _, c := range checks { + if !allowed[c.name] { + continue + } + if c.value < c.min || c.value > c.max { + return mdlerrors.NewValidation(fmt.Sprintf( + "%s is %d — it must be between %d and %d", c.name, c.value, c.min, c.max)) + } + } + return nil +} + +// parseWeekdays turns "Monday, Friday" into the seven flags of a WeekSchedule. +func parseWeekdays(spec string) ([7]bool, error) { + var days [7]bool + if strings.TrimSpace(spec) == "" { + return days, mdlerrors.NewValidation( + "Repeat Weekly needs Weekdays — e.g. Weekdays: 'Monday, Friday'") + } + for _, part := range strings.Split(spec, ",") { + name := strings.TrimSpace(part) + if name == "" { + continue + } + idx := -1 + for i, w := range weekdayNames { + if strings.EqualFold(w, name) { + idx = i + break + } + } + if idx < 0 { + return days, mdlerrors.NewValidation(fmt.Sprintf( + "unknown weekday %q in Weekdays — expected one of %s", name, strings.Join(weekdayNames, ", "))) + } + days[idx] = true + } + return days, nil +} + +// validateEnumProperty accepts an empty value (the writer supplies the default) +// and otherwise requires an exact match, normalising nothing — an enum value +// Mendix does not know produces a document that loads and misbehaves. +func validateEnumProperty(name, value string, allowed []string) error { + if value == "" { + return nil + } + for _, a := range allowed { + if a == value { + return nil + } + } + for _, a := range allowed { + if strings.EqualFold(a, value) { + return mdlerrors.NewValidation(fmt.Sprintf( + "%s %q has the wrong casing — Mendix stores it as %q", name, value, a)) + } + } + return mdlerrors.NewValidation(fmt.Sprintf( + "unknown %s %q — expected one of %s", name, value, strings.Join(allowed, ", "))) +} + +func sortedRepeatNames() []string { + out := make([]string, 0, len(repeatNames)) + for _, n := range repeatNames { + out = append(out, n) + } + sort.Strings(out) + return out +} + +// execDropScheduledEvent handles DROP SCHEDULED EVENT Module.Name. +func execDropScheduledEvent(ctx *ExecContext, s *ast.DropScheduledEventStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + existing := findScheduledEvent(ctx, s.Name.Module, s.Name.Name) + if existing == nil { + return mdlerrors.NewNotFound("scheduled event", s.Name.String()) + } + if err := ctx.Backend.DeleteScheduledEvent(string(existing.ID)); err != nil { + return mdlerrors.NewBackend("drop scheduled event", err) + } + fmt.Fprintf(ctx.Output, "Dropped scheduled event: %s\n", s.Name.String()) + return nil +} + +// execShowScheduledEvents handles SHOW|LIST SCHEDULED EVENTS [IN Module]. +func execShowScheduledEvents(ctx *ExecContext, s *ast.ShowScheduledEventsStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + events, err := ctx.Backend.ListScheduledEvents() + if err != nil { + return mdlerrors.NewBackend("list scheduled events", err) + } + h, err := getHierarchy(ctx) + if err != nil { + return mdlerrors.NewBackend("build hierarchy", err) + } + + type row struct{ qualified, repeat, microflow, enabled string } + var rows []row + for _, ev := range events { + mod := h.GetModuleName(h.FindModuleID(ev.ContainerID)) + if s.Module != "" && !strings.EqualFold(mod, s.Module) { + continue + } + rows = append(rows, row{ + qualified: mod + "." + ev.Name, + repeat: describeRepeat(ev.Schedule), + microflow: string(ev.MicroflowID), + enabled: fmt.Sprintf("%t", ev.Enabled), + }) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].qualified < rows[j].qualified }) + + result := &TableResult{ + Columns: []string{"Scheduled Event", "Repeat", "Microflow", "Enabled"}, + Summary: fmt.Sprintf("(%d scheduled event(s))", len(rows)), + } + for _, r := range rows { + result.Rows = append(result.Rows, []any{r.qualified, r.repeat, r.microflow, r.enabled}) + } + return writeResult(ctx, result) +} + +// describeRepeat renders the schedule as a short human phrase for the listing. +func describeRepeat(s *model.Schedule) string { + if s == nil { + return "(none)" + } + at := fmt.Sprintf("%02d:%02d", s.HourOfDay, s.MinuteOfHour) + switch s.Kind { + case model.ScheduleMinute: + return fmt.Sprintf("every %d min", s.Multiplier) + case model.ScheduleHour: + return fmt.Sprintf("every %dh at :%02d", s.Multiplier, s.MinuteOffset) + case model.ScheduleDay: + return "daily at " + at + case model.ScheduleWeek: + return "weekly " + strings.Join(setWeekdayNames(s.Weekdays), "/") + " at " + at + case model.ScheduleMonthDate: + return fmt.Sprintf("every %d month(s) on day %d at %s", s.Multiplier, s.DayOfMonth, at) + case model.ScheduleMonthWeekday: + return fmt.Sprintf("every %d month(s) on the %s %s at %s", s.Multiplier, s.DaySelector, s.Weekday, at) + case model.ScheduleYearDate: + return fmt.Sprintf("yearly on %d/%d at %s", s.Month, s.DayOfMonth, at) + case model.ScheduleYearWeekday: + return fmt.Sprintf("yearly on the %s %s of month %d at %s", s.DaySelector, s.Weekday, s.Month, at) + } + return string(s.Kind) +} + +func setWeekdayNames(days [7]bool) []string { + var out []string + for i, on := range days { + if on { + out = append(out, weekdayNames[i][:3]) + } + } + if out == nil { + return []string{"(no days)"} + } + return out +} + +// execDescribeScheduledEvent handles DESCRIBE SCHEDULED EVENT Module.Name, +// emitting re-executable MDL so describe → exec round-trips. +func execDescribeScheduledEvent(ctx *ExecContext, s *ast.DescribeScheduledEventStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + ev := findScheduledEvent(ctx, s.Name.Module, s.Name.Name) + if ev == nil { + return mdlerrors.NewNotFound("scheduled event", s.Name.String()) + } + + if ev.Documentation != "" { + fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", ev.Documentation) + } + fmt.Fprintf(ctx.Output, "create or modify scheduled event %s (\n", s.Name.String()) + fmt.Fprintf(ctx.Output, " Microflow: %s,\n", ev.MicroflowID) + for _, line := range describeScheduleProperties(ev.Schedule) { + fmt.Fprintf(ctx.Output, " %s,\n", line) + } + fmt.Fprintf(ctx.Output, " Enabled: %t,\n", ev.Enabled) + if ev.OnOverlap != "" { + fmt.Fprintf(ctx.Output, " OnOverlap: %s,\n", ev.OnOverlap) + } + if ev.TimeZone != "" { + fmt.Fprintf(ctx.Output, " TimeZone: %s,\n", ev.TimeZone) + } + // A zero StartDateTime is how "no start restriction" is stored (Mendix's own + // DateTime.MinValue), not a date anyone chose — emitting it would put + // '0001-01-01T00:00:00Z' into every describe. + if ev.StartDateTime != nil && !ev.StartDateTime.IsZero() { + fmt.Fprintf(ctx.Output, " StartDateTime: '%s',\n", ev.StartDateTime.Format(time.RFC3339)) + } + fmt.Fprint(ctx.Output, ");\n") + // Interval/IntervalType have no MDL syntax: they are legacy siblings of + // Schedule that Studio Pro writes without keeping in sync. Reporting them as + // a comment keeps the output honest without making it non-re-executable. + if ev.IntervalType != "" { + fmt.Fprintf(ctx.Output, "-- legacy (preserved, not authorable): Interval %d %s\n", ev.Interval, ev.IntervalType) + } + return nil +} + +// describeScheduleProperties emits the Repeat plus exactly the fields that +// repeat uses, in the order repeatFields lists them. +func describeScheduleProperties(s *model.Schedule) []string { + if s == nil { + return nil + } + out := []string{"Repeat: " + repeatNames[s.Kind]} + for _, f := range repeatFields[s.Kind] { + switch f { + case "Multiplier": + out = append(out, fmt.Sprintf("Multiplier: %d", s.Multiplier)) + case "MinuteOffset": + out = append(out, fmt.Sprintf("MinuteOffset: %d", s.MinuteOffset)) + case "MonthOffset": + out = append(out, fmt.Sprintf("MonthOffset: %d", s.MonthOffset)) + case "HourOfDay": + out = append(out, fmt.Sprintf("HourOfDay: %d", s.HourOfDay)) + case "MinuteOfHour": + out = append(out, fmt.Sprintf("MinuteOfHour: %d", s.MinuteOfHour)) + case "DayOfMonth": + out = append(out, fmt.Sprintf("DayOfMonth: %d", s.DayOfMonth)) + case "Month": + out = append(out, fmt.Sprintf("Month: %d", s.Month)) + case "DaySelector": + out = append(out, "DaySelector: "+s.DaySelector) + case "Weekday": + out = append(out, "Weekday: "+s.Weekday) + case "Weekdays": + var names []string + for i, on := range s.Weekdays { + if on { + names = append(names, weekdayNames[i]) + } + } + out = append(out, "Weekdays: '"+strings.Join(names, ", ")+"'") + } + } + return out +} diff --git a/mdl/executor/cmd_scheduledevents_test.go b/mdl/executor/cmd_scheduledevents_test.go new file mode 100644 index 000000000..bae0dbc0b --- /dev/null +++ b/mdl/executor/cmd_scheduledevents_test.go @@ -0,0 +1,456 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" +) + +func mkScheduledEvent(containerID model.ID, name string, s *model.Schedule) *model.ScheduledEvent { + ev := &model.ScheduledEvent{ + ContainerID: containerID, + Name: name, + MicroflowID: "Ops.SE_Cleanup", + Schedule: s, + OnOverlap: "DelayNext", + TimeZone: "UTC", + } + ev.ID = nextID("sched") + return ev +} + +func intPtr(v int) *int { return &v } + +// TestScheduleFromStmt_EachRepeatTakesOnlyItsOwnFields is the load-bearing test: +// the eight ScheduledEvents$*Schedule variants differ in WHICH fields they +// carry, and a merged field set is the shape mxbuild accepts and Studio Pro +// refuses to open. +func TestScheduleFromStmt_EachRepeatTakesOnlyItsOwnFields(t *testing.T) { + tests := []struct { + repeat string + stray string // a property that belongs to a different repeat + apply func(*ast.CreateScheduledEventStmt) + }{ + {"Daily", "Multiplier", func(s *ast.CreateScheduledEventStmt) { s.Multiplier = intPtr(3) }}, + {"Minutely", "HourOfDay", func(s *ast.CreateScheduledEventStmt) { s.HourOfDay = intPtr(4) }}, + {"Hourly", "DayOfMonth", func(s *ast.CreateScheduledEventStmt) { s.DayOfMonth = intPtr(15) }}, + {"YearlyByDate", "MonthOffset", func(s *ast.CreateScheduledEventStmt) { s.MonthOffset = intPtr(1) }}, + {"MonthlyByDate", "DaySelector", func(s *ast.CreateScheduledEventStmt) { s.DaySelector = "Last" }}, + {"Daily", "Weekdays", func(s *ast.CreateScheduledEventStmt) { s.Weekdays = "Monday" }}, + } + for _, tt := range tests { + t.Run(tt.repeat+"/"+tt.stray, func(t *testing.T) { + s := &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, + Microflow: "Ops.MF", + Repeat: tt.repeat, + } + tt.apply(s) + _, err := scheduleFromStmt(s) + if err == nil { + t.Fatalf("%s accepted %s, which belongs to a different repeat", tt.repeat, tt.stray) + } + if !strings.Contains(err.Error(), tt.stray) { + t.Errorf("error should name the stray property:\n%s", err) + } + }) + } +} + +func TestScheduleFromStmt_RangesAreEnforced(t *testing.T) { + tests := []struct { + name string + repeat string + apply func(*ast.CreateScheduledEventStmt) + }{ + {"HourOfDay 24", "Daily", func(s *ast.CreateScheduledEventStmt) { s.HourOfDay = intPtr(24) }}, + {"MinuteOfHour 60", "Daily", func(s *ast.CreateScheduledEventStmt) { s.MinuteOfHour = intPtr(60) }}, + {"DayOfMonth 0", "MonthlyByDate", func(s *ast.CreateScheduledEventStmt) { s.DayOfMonth = intPtr(0) }}, + {"Month 13", "YearlyByDate", func(s *ast.CreateScheduledEventStmt) { s.Month = intPtr(13) }}, + {"Multiplier 0", "Minutely", func(s *ast.CreateScheduledEventStmt) { s.Multiplier = intPtr(0) }}, + {"MinuteOffset 60", "Hourly", func(s *ast.CreateScheduledEventStmt) { s.MinuteOffset = intPtr(60) }}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, + Microflow: "Ops.MF", + Repeat: tt.repeat, + } + tt.apply(s) + if _, err := scheduleFromStmt(s); err == nil { + t.Fatalf("%s was accepted — a schedule that can never fire is worse than a refusal", tt.name) + } + }) + } +} + +func TestScheduleFromStmt_Defaults(t *testing.T) { + // An unstated multiplier of 0 would never fire, so it defaults to 1. + got, err := scheduleFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", Repeat: "Hourly", + }) + if err != nil { + t.Fatalf("scheduleFromStmt: %v", err) + } + if got.Multiplier != 1 { + t.Errorf("Multiplier = %d, want 1", got.Multiplier) + } + if got.Kind != model.ScheduleHour { + t.Errorf("Kind = %v", got.Kind) + } +} + +func TestScheduleFromStmt_Weekdays(t *testing.T) { + got, err := scheduleFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Weekly", Weekdays: "monday, FRIDAY", HourOfDay: intPtr(9), + }) + if err != nil { + t.Fatalf("scheduleFromStmt: %v", err) + } + want := [7]bool{false, true, false, false, false, true, false} + if got.Weekdays != want { + t.Errorf("Weekdays = %v, want %v (case-insensitive names)", got.Weekdays, want) + } +} + +func TestScheduleFromStmt_UnknownWeekday(t *testing.T) { + _, err := scheduleFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Weekly", Weekdays: "Moonday", + }) + if err == nil || !strings.Contains(err.Error(), "Moonday") { + t.Fatalf("expected an unknown-weekday error, got %v", err) + } +} + +// TestScheduledEventFromStmt_EnumCasing checks that a near-miss enum value is +// reported rather than normalised: Mendix stores the exact spelling, and a value +// the enumeration does not have produces a document that loads and misbehaves. +func TestScheduledEventFromStmt_EnumCasing(t *testing.T) { + _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Daily", TimeZone: "server", + }) + if err == nil || !strings.Contains(err.Error(), "Server") { + t.Fatalf("expected a casing error naming the stored spelling, got %v", err) + } +} + +func TestScheduledEventFromStmt_RequiresMicroflowAndRepeat(t *testing.T) { + if _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Repeat: "Daily", + }); err == nil { + t.Error("a scheduled event with no Microflow was accepted") + } + if _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + }); err == nil { + t.Error("a scheduled event with no Repeat was accepted") + } +} + +func TestScheduledEventFromStmt_StartDateTime(t *testing.T) { + ev, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Daily", StartDateTime: "2026-01-01T04:00:00Z", + }) + if err != nil { + t.Fatalf("scheduledEventFromStmt: %v", err) + } + if ev.StartDateTime == nil || ev.StartDateTime.Year() != 2026 { + t.Errorf("StartDateTime = %v", ev.StartDateTime) + } + if _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Daily", StartDateTime: "1 January 2026", + }); err == nil { + t.Error("a non-RFC-3339 StartDateTime was accepted") + } +} + +// TestLegacyIntervalFor pins the Interval/IntervalType pair a new event gets. +// IntervalType is a Mendix enumeration, so an empty string is not a valid +// "unset" — every Studio Pro-authored event carries a real value. +func TestLegacyIntervalFor(t *testing.T) { + tests := []struct { + kind model.ScheduleKind + mult int + wantN int + wantType string + }{ + {model.ScheduleMinute, 5, 5, "Minute"}, + {model.ScheduleHour, 2, 2, "Hour"}, + {model.ScheduleDay, 0, 1, "Day"}, + {model.ScheduleWeek, 0, 1, "Week"}, + {model.ScheduleMonthDate, 3, 3, "Month"}, + {model.ScheduleYearWeekday, 0, 1, "Year"}, + } + for _, tt := range tests { + n, typ := legacyIntervalFor(&model.Schedule{Kind: tt.kind, Multiplier: tt.mult}) + if n != tt.wantN || typ != tt.wantType { + t.Errorf("%s -> %d/%q, want %d/%q", tt.kind, n, typ, tt.wantN, tt.wantType) + } + if typ == "" { + t.Errorf("%s produced an empty IntervalType — not a value the enumeration has", tt.kind) + } + } +} + +func TestShowScheduledEvents_Mock(t *testing.T) { + mod := mkModule("Ops") + e1 := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay, HourOfDay: 4}) + e1.Enabled = true + e2 := mkScheduledEvent(mod.ID, "Hourly", &model.Schedule{Kind: model.ScheduleHour, Multiplier: 2, MinuteOffset: 23}) + + h := mkHierarchy(mod) + withContainer(h, e1.ContainerID, mod.ID) + withContainer(h, e2.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{e1, e2}, nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execShowScheduledEvents(ctx, &ast.ShowScheduledEventsStmt{})) + + out := buf.String() + assertContainsStr(t, out, "Ops.Nightly") + assertContainsStr(t, out, "daily at 04:00") + assertContainsStr(t, out, "every 2h at :23") + assertContainsStr(t, out, "(2 scheduled event(s))") +} + +func TestCreateScheduledEvent_Mock(t *testing.T) { + mod := mkModule("Ops") + h := mkHierarchy(mod) + + var created *model.ScheduledEvent + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { return nil, nil }, + CreateScheduledEventFunc: func(ev *model.ScheduledEvent) error { + created = ev + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execCreateScheduledEvent(ctx, &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, + Microflow: "Ops.SE_Cleanup", + Repeat: "Daily", + HourOfDay: intPtr(4), + MinuteOfHour: intPtr(0), + Enabled: boolPtr(true), + })) + + if created == nil { + t.Fatal("CreateScheduledEvent was not called") + } + if created.Schedule == nil || created.Schedule.Kind != model.ScheduleDay { + t.Fatalf("Schedule = %+v", created.Schedule) + } + if created.Schedule.HourOfDay != 4 { + t.Errorf("HourOfDay = %d", created.Schedule.HourOfDay) + } + if created.IntervalType == "" { + t.Error("IntervalType is empty — it is an enumeration, not an optional string") + } + assertContainsStr(t, buf.String(), "Created scheduled event: Ops.Nightly") +} + +// TestModifyScheduledEvent_Mock_PreservesLegacyInterval: Studio Pro does not +// keep Interval/IntervalType in sync with Schedule and MDL cannot author them, +// so a modify must carry the stored pair through rather than re-derive it. +func TestModifyScheduledEvent_Mock_PreservesLegacyInterval(t *testing.T) { + mod := mkModule("Ops") + existing := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay, HourOfDay: 1}) + existing.Interval = 0 + existing.IntervalType = "Minute" // the Workflow Commons shape: stale, but real + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var updated *model.ScheduledEvent + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{existing}, nil + }, + UpdateScheduledEventFunc: func(ev *model.ScheduledEvent) error { + updated = ev + return nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execCreateScheduledEvent(ctx, &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, + Microflow: "Ops.SE_Cleanup", + Repeat: "Daily", + HourOfDay: intPtr(6), + CreateOrModify: true, + })) + + if updated == nil { + t.Fatal("UpdateScheduledEvent was not called") + } + if updated.ID != existing.ID { + t.Errorf("ID = %q, want the existing %q", updated.ID, existing.ID) + } + if updated.Interval != 0 || updated.IntervalType != "Minute" { + t.Errorf("legacy pair = %d/%q, want the stored 0/\"Minute\"", updated.Interval, updated.IntervalType) + } + if updated.Schedule.HourOfDay != 6 { + t.Errorf("HourOfDay = %d, want the new 6", updated.Schedule.HourOfDay) + } +} + +func TestCreateScheduledEvent_Mock_DuplicateWithoutOrModify(t *testing.T) { + mod := mkModule("Ops") + existing := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay}) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{existing}, nil + }, + CreateScheduledEventFunc: func(ev *model.ScheduledEvent) error { + t.Error("CreateScheduledEvent must not be called for a duplicate") + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + if err := execCreateScheduledEvent(ctx, &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, Microflow: "Ops.MF", Repeat: "Daily", + }); err == nil { + t.Fatal("expected an already-exists error") + } +} + +func TestDropScheduledEvent_Mock(t *testing.T) { + mod := mkModule("Ops") + existing := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay}) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var deleted string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{existing}, nil + }, + DeleteScheduledEventFunc: func(id string) error { + deleted = id + return nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDropScheduledEvent(ctx, &ast.DropScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, + })) + if deleted != string(existing.ID) { + t.Errorf("deleted %q, want %q", deleted, existing.ID) + } + assertContainsStr(t, buf.String(), "Dropped scheduled event: Ops.Nightly") +} + +// TestDescribeScheduledEvent_Mock_RoundTrips checks that DESCRIBE emits exactly +// the fields the repeat has, and nothing from another variant. +func TestDescribeScheduledEvent_Mock_RoundTrips(t *testing.T) { + mod := mkModule("Ops") + ev := mkScheduledEvent(mod.ID, "QuarterEnd", &model.Schedule{ + Kind: model.ScheduleMonthWeekday, Multiplier: 3, MonthOffset: 2, + DaySelector: "Last", Weekday: "Friday", HourOfDay: 18, + }) + h := mkHierarchy(mod) + withContainer(h, ev.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{ev}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDescribeScheduledEvent(ctx, &ast.DescribeScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "QuarterEnd"}, + })) + + out := buf.String() + assertContainsStr(t, out, "create or modify scheduled event Ops.QuarterEnd (") + assertContainsStr(t, out, "Repeat: MonthlyByWeekday,") + assertContainsStr(t, out, "DaySelector: Last,") + assertContainsStr(t, out, "Weekday: Friday,") + // Fields of other variants must not appear, or the output would not + // re-execute: the executor refuses a field the repeat does not have. + assertNotContainsStr(t, out, "DayOfMonth") + assertNotContainsStr(t, out, "Weekdays:") + assertNotContainsStr(t, out, "MinuteOffset") +} + +// TestDescribeScheduledEvent_Mock_OutputParses is the real round-trip proof: +// asserting on strings passes against a formatter emitting something nothing +// can read. +func TestDescribeScheduledEvent_Mock_OutputParses(t *testing.T) { + mod := mkModule("Ops") + for _, sched := range []*model.Schedule{ + {Kind: model.ScheduleMinute, Multiplier: 5}, + {Kind: model.ScheduleHour, Multiplier: 2, MinuteOffset: 23}, + {Kind: model.ScheduleDay, HourOfDay: 4}, + {Kind: model.ScheduleWeek, Weekdays: [7]bool{false, true, false, false, false, true, false}, HourOfDay: 9}, + {Kind: model.ScheduleMonthDate, Multiplier: 1, DayOfMonth: 15}, + {Kind: model.ScheduleMonthWeekday, Multiplier: 3, DaySelector: "Last", Weekday: "Friday"}, + {Kind: model.ScheduleYearDate, Month: 1, DayOfMonth: 2}, + {Kind: model.ScheduleYearWeekday, Month: 3, DaySelector: "First", Weekday: "Monday"}, + } { + t.Run(string(sched.Kind), func(t *testing.T) { + ev := mkScheduledEvent(mod.ID, "E", sched) + h := mkHierarchy(mod) + withContainer(h, ev.ContainerID, mod.ID) + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{ev}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDescribeScheduledEvent(ctx, &ast.DescribeScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, + })) + + prog, errs := visitor.Build(buf.String()) + if len(errs) > 0 { + t.Fatalf("describe output does not parse: %v\n%s", errs, buf.String()) + } + stmt, ok := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if !ok { + t.Fatalf("statement 0 = %T", prog.Statements[0]) + } + // And the parsed statement must be accepted by the same validation + // that guards a hand-written one. + back, err := scheduleFromStmt(stmt) + if err != nil { + t.Fatalf("describe output is refused on re-execution: %v\n%s", err, buf.String()) + } + if back.Kind != sched.Kind { + t.Errorf("round-tripped kind = %v, want %v", back.Kind, sched.Kind) + } + }) + } +} diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index 044f9aead..010c20383 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -222,6 +222,21 @@ func registerQueueHandlers(r *Registry) { }) } +func registerScheduledEventHandlers(r *Registry) { + r.Register(&ast.CreateScheduledEventStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execCreateScheduledEvent(ctx, stmt.(*ast.CreateScheduledEventStmt)) + }) + r.Register(&ast.DropScheduledEventStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDropScheduledEvent(ctx, stmt.(*ast.DropScheduledEventStmt)) + }) + r.Register(&ast.ShowScheduledEventsStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execShowScheduledEvents(ctx, stmt.(*ast.ShowScheduledEventsStmt)) + }) + r.Register(&ast.DescribeScheduledEventStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDescribeScheduledEvent(ctx, stmt.(*ast.DescribeScheduledEventStmt)) + }) +} + func registerImageHandlers(r *Registry) { r.Register(&ast.CreateImageCollectionStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execCreateImageCollection(ctx, stmt.(*ast.CreateImageCollectionStmt)) diff --git a/mdl/executor/registry.go b/mdl/executor/registry.go index 761a87fc6..7055b6184 100644 --- a/mdl/executor/registry.go +++ b/mdl/executor/registry.go @@ -39,6 +39,7 @@ func NewRegistry() *Registry { registerNavigationHandlers(r) registerImageHandlers(r) registerQueueHandlers(r) + registerScheduledEventHandlers(r) registerWorkflowHandlers(r) registerBusinessEventHandlers(r) registerSettingsHandlers(r) diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 5ec9c72ab..1a777fb2b 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -207,6 +207,7 @@ func allKnownStatements() []ast.Statement { &ast.CreatePageStmtV3{}, &ast.CreatePublishedRestServiceStmt{}, &ast.CreateQueueStmt{}, + &ast.CreateScheduledEventStmt{}, &ast.CreateRestClientStmt{}, &ast.CreateSnippetStmtV3{}, &ast.CreateUserRoleStmt{}, @@ -217,6 +218,7 @@ func allKnownStatements() []ast.Statement { &ast.DescribeContractFromOpenAPIStmt{}, &ast.DescribeFragmentFromStmt{}, &ast.DescribeQueueStmt{}, + &ast.DescribeScheduledEventStmt{}, &ast.DescribeStmt{}, &ast.DescribeStylingStmt{}, &ast.DisconnectStmt{}, @@ -248,6 +250,7 @@ func allKnownStatements() []ast.Statement { &ast.DropPageStmt{}, &ast.DropPublishedRestServiceStmt{}, &ast.DropQueueStmt{}, + &ast.DropScheduledEventStmt{}, &ast.DropRestClientStmt{}, &ast.DropSnippetStmt{}, &ast.DropUserRoleStmt{}, @@ -282,6 +285,7 @@ func allKnownStatements() []ast.Statement { &ast.ShowDesignPropertiesStmt{}, &ast.ShowFeaturesStmt{}, &ast.ShowQueuesStmt{}, + &ast.ShowScheduledEventsStmt{}, &ast.ShowStmt{}, &ast.ShowWidgetsStmt{}, &ast.SQLConnectionsStmt{}, diff --git a/mdl/executor/validate_scheduled_events.go b/mdl/executor/validate_scheduled_events.go new file mode 100644 index 000000000..32c37d90b --- /dev/null +++ b/mdl/executor/validate_scheduled_events.go @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation for CREATE SCHEDULED EVENT. +// +// The repeat rule is a polymorphic child with eight variants that differ in +// which fields they carry, so most of what can go wrong — a field that belongs +// to a different repeat, an hour of 99, a misspelled weekday — is decidable from +// the statement alone. Running it here means a plain `mxcli check` reports it, +// instead of the script passing check and failing at exec. +package executor + +import ( + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// ValidateScheduledEvents reports (MDL-SCHED01) a CREATE SCHEDULED EVENT whose +// properties do not describe a schedule Mendix can store. +// +// It reuses the executor's own builder, so check and exec cannot drift: there is +// one implementation of what a valid statement is, and this pass is the same +// function exec calls before writing. +func ValidateScheduledEvents(prog *ast.Program) []linter.Violation { + var out []linter.Violation + for _, stmt := range prog.Statements { + s, ok := stmt.(*ast.CreateScheduledEventStmt) + if !ok { + continue + } + if _, err := scheduledEventFromStmt(s); err != nil { + out = append(out, linter.Violation{ + RuleID: "MDL-SCHED01", + Severity: linter.SeverityError, + Message: err.Error(), + Suggestion: "Each Repeat takes only its own fields — see `mxcli syntax scheduled-event` " + + "for the field list of each one.", + }) + } + } + return out +} diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index 87fe9226c..ea8995d3c 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -356,6 +356,7 @@ FILTERTYPE: F I L T E R T Y P E; IMAGE: I M A G E; QUEUE: Q U E U E; QUEUES: Q U E U E S; +SCHEDULED: S C H E D U L E D; COLLECTION: C O L L E C T I O N S?; // accept singular + plural ("collection(s)") JAR: J A R; DEPENDENCY: D E P E N D E N C Y; diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index fe263a5dc..f1309162d 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -116,6 +116,7 @@ createStatement | createDemoUserStatement | createImageCollectionStatement | createQueueStatement + | createScheduledEventStatement | createJsonStructureStatement | createImportMappingStatement | createExportMappingStatement @@ -311,6 +312,7 @@ dropStatement | DROP MODULE qualifiedName | DROP NOTEBOOK qualifiedName | DROP QUEUE qualifiedName + | DROP SCHEDULED EVENT qualifiedName | DROP JAVA ACTION qualifiedName | DROP JAVASCRIPT ACTION qualifiedName | DROP INDEX qualifiedName ON qualifiedName diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index c747df1a7..0ed72c210 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -36,6 +36,7 @@ showStatement | showOrList LAYOUTS (IN (qualifiedName | IDENTIFIER))? | showOrList NOTEBOOKS (IN (qualifiedName | IDENTIFIER))? | showOrList QUEUES (IN (qualifiedName | IDENTIFIER))? + | showOrList SCHEDULED EVENTS (IN (qualifiedName | IDENTIFIER))? | showOrList JAVA ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList JAVASCRIPT ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList IMAGE COLLECTION (IN (qualifiedName | IDENTIFIER))? @@ -164,6 +165,7 @@ describeStatement | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name | DESCRIBE QUEUE qualifiedName // DESCRIBE QUEUE Module.Name + | DESCRIBE SCHEDULED EVENT qualifiedName // DESCRIBE SCHEDULED EVENT Module.Name | DESCRIBE DATABASE CONNECTION qualifiedName // DESCRIBE DATABASE CONNECTION Module.Name | DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)? // DESCRIBE SETTINGS [CONFIGURATION 'Default'] | DESCRIBE FRAGMENT FROM PAGE qualifiedName WIDGET identifierOrKeyword // DESCRIBE FRAGMENT FROM PAGE Module.Page WIDGET name diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index 3ad98dfc5..c86129b2a 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -354,6 +354,28 @@ queueProperty : identifierOrKeyword COLON (NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | identifierOrKeyword) ; +// ============================================================================= +// SCHEDULED EVENT CREATION +// ============================================================================= +// +// The repeat rule is a property (Repeat: Daily) plus the fields that rule uses, +// rather than an English clause, because the eight ScheduledEvents$*Schedule +// variants differ in WHICH fields they carry — a labelled property list keeps +// the storage's own vocabulary and lets the executor reject a field that does +// not belong to the chosen repeat. + +createScheduledEventStatement + : SCHEDULED EVENT qualifiedName scheduledEventBody? + ; + +scheduledEventBody + : LPAREN (scheduledEventProperty (COMMA scheduledEventProperty)* COMMA?)? RPAREN + ; + +scheduledEventProperty + : identifierOrKeyword COLON (qualifiedName | NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | identifierOrKeyword) + ; + // ============================================================================= // IMAGE COLLECTION CREATION // ============================================================================= diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index a18faa5bb..b12db73a8 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -624,7 +624,7 @@ keyword | UNLOCK | UNPAUSE | WAIT | WORKFLOW | WORKFLOWS // Business events / settings - | BUSINESS | CONFIGURATION | EVENT | EVENTS | HANDLER | SETTINGS | SUBSCRIBE + | BUSINESS | CONFIGURATION | EVENT | EVENTS | HANDLER | SCHEDULED | SETTINGS | SUBSCRIBE // Code search / analysis | BACKGROUND | CALLERS | CALLEES | DEPTH | IMPACT | REFERENCES diff --git a/mdl/scheduledevents/codec.go b/mdl/scheduledevents/codec.go new file mode 100644 index 000000000..95a353731 --- /dev/null +++ b/mdl/scheduledevents/codec.go @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package scheduledevents holds the BSON codec for scheduled events +// (ScheduledEvents$ScheduledEvent) and their polymorphic Schedule child. +// +// It lives outside both engines because both write this document, and the +// schedule has eight variants that differ in which fields they carry — a shape +// worth getting right once. The package depends only on the model types and the +// BSON driver, so either backend can call it. +package scheduledevents + +import ( + "fmt" + "time" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/bsonutil" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +// TypeName is the BSON storage name for a scheduled event document. +const TypeName = "ScheduledEvents$ScheduledEvent" + +// ScheduleTypeNames maps a schedule kind to its BSON storage name. The variants +// differ in arity, not just in values, so the kind is dispatched before any +// field is read or written — see Serialize and ParseSchedule. +var ScheduleTypeNames = map[model.ScheduleKind]string{ + model.ScheduleMinute: "ScheduledEvents$MinuteSchedule", + model.ScheduleHour: "ScheduledEvents$HourSchedule", + model.ScheduleDay: "ScheduledEvents$DaySchedule", + model.ScheduleWeek: "ScheduledEvents$WeekSchedule", + model.ScheduleMonthDate: "ScheduledEvents$MonthDateSchedule", + model.ScheduleMonthWeekday: "ScheduledEvents$MonthWeekdaySchedule", + model.ScheduleYearDate: "ScheduledEvents$YearDateSchedule", + model.ScheduleYearWeekday: "ScheduledEvents$YearWeekdaySchedule", +} + +// WeekdayNames indexes model.Schedule.Weekdays: Sunday first, matching the +// Mendix Weekday enumeration and the BSON property names of WeekSchedule. +var WeekdayNames = [7]string{"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"} + +// Serialize writes the document in the shape Studio Pro produces. +// +// Pinned against four Studio Pro-authored events (Workflow Commons 4.11.0, OIDC +// SSO 4.6.0, SAML 4.2.1 ×2). All four agree on the property set and on three +// types the generated metamodel would lead you to get wrong: +// +// - every integer is BSON int64, while modelsdk/gen declares int32 — the same +// mismatch that made the READER fail on Studio Pro documents in issue #585; +// - StartDateTime is a BSON UTC datetime, while gen declares a string; +// - Microflow is a by-name reference (the qualified name), not an element ID. +// +// Interval/IntervalType are legacy siblings of Schedule that Studio Pro still +// writes and does NOT keep in sync — the Workflow Commons event stores +// Interval 0 / "Minute" next to a DaySchedule of 01:00. They are therefore +// carried through from the caller rather than derived from the schedule, so a +// round trip cannot invent a value Mendix did not have. +func Serialize(ev *model.ScheduledEvent) ([]byte, error) { + exportLevel := ev.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + onOverlap := ev.OnOverlap + if onOverlap == "" { + // Every reference document uses DelayNext, and it is the safer default: + // SkipNext silently drops a run. + onOverlap = "DelayNext" + } + timeZone := ev.TimeZone + if timeZone == "" { + timeZone = "UTC" + } + var start time.Time + if ev.StartDateTime != nil { + start = ev.StartDateTime.UTC() + } + + doc := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(string(ev.ID))}, + {Key: "$Type", Value: TypeName}, + {Key: "Documentation", Value: ev.Documentation}, + {Key: "Enabled", Value: ev.Enabled}, + {Key: "Excluded", Value: ev.Excluded}, + {Key: "ExportLevel", Value: exportLevel}, + {Key: "Interval", Value: int64(ev.Interval)}, + {Key: "IntervalType", Value: ev.IntervalType}, + {Key: "Microflow", Value: string(ev.MicroflowID)}, + {Key: "Name", Value: ev.Name}, + {Key: "OnOverlap", Value: onOverlap}, + } + if ev.Schedule != nil { + sched, err := SerializeSchedule(ev.Schedule) + if err != nil { + return nil, err + } + doc = append(doc, bson.E{Key: "Schedule", Value: sched}) + } + doc = append(doc, + bson.E{Key: "StartDateTime", Value: primitive.DateTime(start.UnixMilli())}, + bson.E{Key: "TimeZone", Value: timeZone}, + ) + + out, err := bson.Marshal(doc) + if err != nil { + return nil, fmt.Errorf("serialize scheduled event %q: %w", ev.Name, err) + } + return out, nil +} + +// SerializeSchedule writes the ScheduledEvents$*Schedule child. +// +// Each variant writes only the fields its type declares. Writing a field the +// type does not carry is the failure mode that produces a document mxbuild +// accepts and Studio Pro cannot open (System.InvalidOperationException at +// MprProperty), so this dispatches on the kind and never merges field sets. +// Keys are alphabetical within each variant, matching the observed documents. +func SerializeSchedule(s *model.Schedule) (bson.D, error) { + typeName, ok := ScheduleTypeNames[s.Kind] + if !ok { + return nil, fmt.Errorf("unknown schedule kind %q", s.Kind) + } + head := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(types.GenerateID())}, + {Key: "$Type", Value: typeName}, + } + + var fields bson.D + switch s.Kind { + case model.ScheduleMinute: + fields = bson.D{{Key: "Multiplier", Value: int64(s.Multiplier)}} + case model.ScheduleHour: + fields = bson.D{ + {Key: "MinuteOffset", Value: int64(s.MinuteOffset)}, + {Key: "Multiplier", Value: int64(s.Multiplier)}, + } + case model.ScheduleDay: + fields = bson.D{ + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + } + case model.ScheduleWeek: + // Alphabetical, so the seven day flags interleave with the times rather + // than grouping — Friday, HourOfDay, MinuteOfHour, Monday, ... + fields = bson.D{ + {Key: "Friday", Value: s.Weekdays[5]}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "Monday", Value: s.Weekdays[1]}, + {Key: "Saturday", Value: s.Weekdays[6]}, + {Key: "Sunday", Value: s.Weekdays[0]}, + {Key: "Thursday", Value: s.Weekdays[4]}, + {Key: "Tuesday", Value: s.Weekdays[2]}, + {Key: "Wednesday", Value: s.Weekdays[3]}, + } + case model.ScheduleMonthDate: + fields = bson.D{ + {Key: "DayOfMonth", Value: int64(s.DayOfMonth)}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "MonthOffset", Value: int64(s.MonthOffset)}, + {Key: "Multiplier", Value: int64(s.Multiplier)}, + } + case model.ScheduleMonthWeekday: + fields = bson.D{ + {Key: "DaySelector", Value: s.DaySelector}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "MonthOffset", Value: int64(s.MonthOffset)}, + {Key: "Multiplier", Value: int64(s.Multiplier)}, + {Key: "Weekday", Value: s.Weekday}, + } + case model.ScheduleYearDate: + fields = bson.D{ + {Key: "DayOfMonth", Value: int64(s.DayOfMonth)}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "Month", Value: int64(s.Month)}, + } + case model.ScheduleYearWeekday: + fields = bson.D{ + {Key: "DaySelector", Value: s.DaySelector}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "Month", Value: int64(s.Month)}, + {Key: "Weekday", Value: s.Weekday}, + } + } + return append(head, fields...), nil +} + +// Parse converts a stored document to the semantic type. MicroflowID holds the +// by-name microflow reference (BSON "Microflow"), not an element ID. +func Parse(doc bson.M, id, containerID model.ID) *model.ScheduledEvent { + ev := &model.ScheduledEvent{ContainerID: containerID} + ev.ID = id + ev.TypeName = TypeName + ev.Name, _ = doc["Name"].(string) + ev.Documentation, _ = doc["Documentation"].(string) + if mf, ok := doc["Microflow"].(string); ok { + ev.MicroflowID = model.ID(mf) + } + ev.Enabled, _ = doc["Enabled"].(bool) + ev.Excluded, _ = doc["Excluded"].(bool) + ev.ExportLevel, _ = doc["ExportLevel"].(string) + ev.OnOverlap, _ = doc["OnOverlap"].(string) + ev.TimeZone, _ = doc["TimeZone"].(string) + ev.IntervalType, _ = doc["IntervalType"].(string) + ev.Interval = anyInt(doc["Interval"]) + if dt, ok := doc["StartDateTime"].(primitive.DateTime); ok { + t := time.UnixMilli(int64(dt)).UTC() + ev.StartDateTime = &t + } + if sched, ok := doc["Schedule"].(bson.M); ok { + ev.Schedule = ParseSchedule(sched) + } + return ev +} + +// ParseSchedule dispatches on $Type before reading any field: the variants +// differ in arity, so inferring the kind from which keys are present would +// mis-read a schedule whose fields overlap another's. +func ParseSchedule(doc bson.M) *model.Schedule { + typeName, _ := doc["$Type"].(string) + var kind model.ScheduleKind + for k, name := range ScheduleTypeNames { + if name == typeName { + kind = k + break + } + } + if kind == "" { + return nil + } + s := &model.Schedule{Kind: kind} + switch kind { + case model.ScheduleMinute: + s.Multiplier = anyInt(doc["Multiplier"]) + case model.ScheduleHour: + s.Multiplier = anyInt(doc["Multiplier"]) + s.MinuteOffset = anyInt(doc["MinuteOffset"]) + case model.ScheduleDay: + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleWeek: + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + for i, day := range WeekdayNames { + s.Weekdays[i], _ = doc[day].(bool) + } + case model.ScheduleMonthDate: + s.Multiplier = anyInt(doc["Multiplier"]) + s.MonthOffset = anyInt(doc["MonthOffset"]) + s.DayOfMonth = anyInt(doc["DayOfMonth"]) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleMonthWeekday: + s.Multiplier = anyInt(doc["Multiplier"]) + s.MonthOffset = anyInt(doc["MonthOffset"]) + s.DaySelector, _ = doc["DaySelector"].(string) + s.Weekday, _ = doc["Weekday"].(string) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleYearDate: + s.Month = anyInt(doc["Month"]) + s.DayOfMonth = anyInt(doc["DayOfMonth"]) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleYearWeekday: + s.Month = anyInt(doc["Month"]) + s.DaySelector, _ = doc["DaySelector"].(string) + s.Weekday, _ = doc["Weekday"].(string) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + } + return s +} + +// anyInt accepts every numeric width a writer might have produced. Studio Pro +// writes int64; older mxcli builds and hand-made fixtures write int32 (#585). +func anyInt(v any) int { + switch n := v.(type) { + case int64: + return int(n) + case int32: + return int(n) + case int: + return n + case float64: + return int(n) + } + return 0 +} diff --git a/mdl/scheduledevents/codec_test.go b/mdl/scheduledevents/codec_test.go new file mode 100644 index 000000000..dce733cdb --- /dev/null +++ b/mdl/scheduledevents/codec_test.go @@ -0,0 +1,314 @@ +// SPDX-License-Identifier: Apache-2.0 + +package scheduledevents + +import ( + "encoding/hex" + "testing" + "time" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/model" +) + +// Three complete, unedited scheduled-event documents lifted out of Mendix's own +// marketplace modules — the only Studio Pro-authored references available +// without Studio Pro. Between them they cover both schedule variants that ship +// in Mendix modules (Day and Hour), both TimeZone values, both Enabled values, +// documented and undocumented, and a zero Interval. +// +// oidcHex OIDC SSO 4.6.0 CleanupOldAuthAttempts HourSchedule +// samlHex SAML 4.2.1 SE_LogCleanUp DaySchedule, TimeZone Server +// wfcHex Workflow Commons 4.11.0 SE_WorkflowAuditTrailRecord… DaySchedule, Interval 0 +const ( + oidcHex = "c00100000524494400100000000059533bc5d678bb40835cd6028304521f022454797065001f0000005363686564756c65644576656e7473245363686564756c65644576656e740002446f63756d656e746174696f6e00010000000008456e61626c65640001084578636c756465640000024578706f72744c6576656c000700000048696464656e0012496e74657276616c00010000000000000002496e74657276616c547970650005000000486f757200024d6963726f666c6f7700200000004f4944432e5355425f436c65616e75704f6c6441757468417474656d70747300024e616d650017000000436c65616e75704f6c6441757468417474656d70747300024f6e4f7665726c6170000a00000044656c61794e65787400035363686564756c65007100000005244944001000000000c154dede71230641b986faef8e304829022454797065001d0000005363686564756c65644576656e747324486f75725363686564756c6500124d696e7574654f6666736574001700000000000000124d756c7469706c696572000100000000000000000953746172744461746554696d650058a17738730100000254696d655a6f6e6500040000005554430000" + samlHex = "63020000052449440010000000007e890ddd9ec5a444b417ef5992c27101022454797065001f0000005363686564756c65644576656e7473245363686564756c65644576656e740002446f63756d656e746174696f6e00b80000005468697320616374696f6e2077696c6c20636c65616e757020746865206c6f676c696e65732e20546865206c6f677320656e74726965732077696c6c20626520617263686976656420666f722074686520616d6f756e74206f6620646179732073706563696669656420696e2074686520636f6e66696775726174696f6e2c20746865206c6f6720656e74726965732074686174207265616368656420746865206c696d69742077696c6c2062652064656c657465642e0008456e61626c65640000084578636c756465640000024578706f72744c6576656c000700000048696464656e0012496e74657276616c00010000000000000002496e74657276616c54797065000400000044617900024d6963726f666c6f77001500000053414d4c32302e53455f4c6f67436c65616e557000024e616d65000e00000053455f4c6f67436c65616e557000024f6e4f7665726c6170000a00000044656c61794e65787400035363686564756c65006f00000005244944001000000000df856dc51b45c24d89325077b18be82b022454797065001c0000005363686564756c65644576656e7473244461795363686564756c650012486f75724f66446179000400000000000000124d696e7574654f66486f7572000000000000000000000953746172744461746554696d6500002ea2b8390100000254696d655a6f6e6500070000005365727665720000" + wfcHex = "e1010000052449440010000000003d7bd7ba3c94a84d816c130543adf825022454797065001f0000005363686564756c65644576656e7473245363686564756c65644576656e740002446f63756d656e746174696f6e00010000000008456e61626c65640000084578636c756465640000024578706f72744c6576656c000700000048696464656e0012496e74657276616c00000000000000000002496e74657276616c5479706500070000004d696e75746500024d6963726f666c6f770034000000576f726b666c6f77436f6d6d6f6e732e53455f576f726b666c6f774175646974547261696c5265636f72645f436c65616e557000024e616d65002400000053455f576f726b666c6f774175646974547261696c5265636f72645f436c65616e557000024f6e4f7665726c6170000a00000044656c61794e65787400035363686564756c65006f0000000524494400100000000081a8e58306a48748bdc93d240d67c64b022454797065001c0000005363686564756c65644576656e7473244461795363686564756c650012486f75724f66446179000100000000000000124d696e7574654f66486f7572000000000000000000000953746172744461746554696d6500db6b4292900100000254696d655a6f6e6500040000005554430000" +) + +func mustDecode(t *testing.T, h string) bson.M { + t.Helper() + raw, err := hex.DecodeString(h) + if err != nil { + t.Fatalf("hex: %v", err) + } + var doc bson.M + if err := bson.Unmarshal(raw, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + return doc +} + +// TestParseStudioProDocuments checks the reader against the real documents, +// including the two properties the generated metamodel gets wrong: every +// integer is int64 (gen says int32) and StartDateTime is a BSON datetime (gen +// says string). +func TestParseStudioProDocuments(t *testing.T) { + tests := []struct { + name string + hexDoc string + want model.ScheduledEvent + wantSched model.Schedule + }{ + { + name: "OIDC CleanupOldAuthAttempts", + hexDoc: oidcHex, + want: model.ScheduledEvent{ + Name: "CleanupOldAuthAttempts", MicroflowID: "OIDC.SUB_CleanupOldAuthAttempts", + Enabled: true, Interval: 1, IntervalType: "Hour", + OnOverlap: "DelayNext", TimeZone: "UTC", ExportLevel: "Hidden", + }, + wantSched: model.Schedule{Kind: model.ScheduleHour, Multiplier: 1, MinuteOffset: 23}, + }, + { + name: "SAML SE_LogCleanUp", + hexDoc: samlHex, + want: model.ScheduledEvent{ + Name: "SE_LogCleanUp", MicroflowID: "SAML20.SE_LogCleanUp", + Enabled: false, Interval: 1, IntervalType: "Day", + OnOverlap: "DelayNext", TimeZone: "Server", ExportLevel: "Hidden", + }, + wantSched: model.Schedule{Kind: model.ScheduleDay, HourOfDay: 4, MinuteOfHour: 0}, + }, + { + name: "Workflow Commons cleanup", + hexDoc: wfcHex, + want: model.ScheduledEvent{ + Name: "SE_WorkflowAuditTrailRecord_CleanUp", + // Interval 0 / "Minute" alongside a DaySchedule of 01:00: Studio + // Pro does not keep the legacy pair in sync with Schedule, which + // is why the writer carries them through instead of deriving them. + MicroflowID: "WorkflowCommons.SE_WorkflowAuditTrailRecord_CleanUp", + Enabled: false, Interval: 0, IntervalType: "Minute", + OnOverlap: "DelayNext", TimeZone: "UTC", ExportLevel: "Hidden", + }, + wantSched: model.Schedule{Kind: model.ScheduleDay, HourOfDay: 1, MinuteOfHour: 0}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ev := Parse(mustDecode(t, tt.hexDoc), "id-1", "mod-1") + if ev.Name != tt.want.Name { + t.Errorf("Name = %q, want %q", ev.Name, tt.want.Name) + } + if ev.MicroflowID != tt.want.MicroflowID { + t.Errorf("MicroflowID = %q, want %q", ev.MicroflowID, tt.want.MicroflowID) + } + if ev.Enabled != tt.want.Enabled { + t.Errorf("Enabled = %v", ev.Enabled) + } + if ev.Interval != tt.want.Interval { + t.Errorf("Interval = %d, want %d (stored as int64)", ev.Interval, tt.want.Interval) + } + if ev.IntervalType != tt.want.IntervalType { + t.Errorf("IntervalType = %q", ev.IntervalType) + } + if ev.OnOverlap != tt.want.OnOverlap { + t.Errorf("OnOverlap = %q", ev.OnOverlap) + } + if ev.TimeZone != tt.want.TimeZone { + t.Errorf("TimeZone = %q", ev.TimeZone) + } + if ev.ExportLevel != tt.want.ExportLevel { + t.Errorf("ExportLevel = %q", ev.ExportLevel) + } + if ev.StartDateTime == nil { + t.Error("StartDateTime not read — it is a BSON datetime, not the string gen declares") + } + if ev.Schedule == nil { + t.Fatal("Schedule not read") + } + if *ev.Schedule != tt.wantSched { + t.Errorf("Schedule = %+v, want %+v", *ev.Schedule, tt.wantSched) + } + }) + } +} + +// TestSerializeMatchesStudioProDocuments is the shape pin: read each real +// document, write it back, and require the result to be identical apart from +// the two generated $ID values. It compares raw BSON elements, so key ORDER and +// BSON TYPE (int64 vs int32, datetime vs string) are covered as well as values — +// none of which a field-by-field assertion would catch. +func TestSerializeMatchesStudioProDocuments(t *testing.T) { + for _, tt := range []struct{ name, hexDoc string }{ + {"OIDC", oidcHex}, + {"SAML", samlHex}, + {"WorkflowCommons", wfcHex}, + } { + t.Run(tt.name, func(t *testing.T) { + originalBytes, err := hex.DecodeString(tt.hexDoc) + if err != nil { + t.Fatalf("hex: %v", err) + } + ev := Parse(mustDecode(t, tt.hexDoc), "11111111-1111-1111-1111-111111111111", "mod-1") + + out, err := Serialize(ev) + if err != nil { + t.Fatalf("Serialize: %v", err) + } + assertSameExceptIDs(t, "", bson.Raw(originalBytes), bson.Raw(out)) + }) + } +} + +// assertSameExceptIDs compares two raw documents element by element, in order, +// requiring the same keys in the same positions with the same BSON types and +// values. $ID is compared for presence only: it is regenerated on every write. +func assertSameExceptIDs(t *testing.T, path string, want, got bson.Raw) { + t.Helper() + wantEls, err := want.Elements() + if err != nil { + t.Fatalf("%s: elements: %v", path, err) + } + gotEls, err := got.Elements() + if err != nil { + t.Fatalf("%s: elements: %v", path, err) + } + if len(wantEls) != len(gotEls) { + t.Fatalf("%s: wrote %d properties, Studio Pro wrote %d\n want %v\n got %v", + path, len(gotEls), len(wantEls), keysOf(wantEls), keysOf(gotEls)) + } + for i, we := range wantEls { + ge := gotEls[i] + if we.Key() != ge.Key() { + t.Errorf("%sproperty %d is %q, want %q (order differs)", path, i, ge.Key(), we.Key()) + continue + } + wv, gv := we.Value(), ge.Value() + if wv.Type != gv.Type { + t.Errorf("%s%s: BSON type %s, want %s", path, we.Key(), gv.Type, wv.Type) + continue + } + if we.Key() == "$ID" { + continue + } + if wv.Type == bson.TypeEmbeddedDocument { + assertSameExceptIDs(t, path+we.Key()+".", wv.Document(), gv.Document()) + continue + } + if !wv.Equal(gv) { + t.Errorf("%s%s = %v, want %v", path, we.Key(), gv, wv) + } + } +} + +func keysOf(els []bson.RawElement) []string { + out := make([]string, len(els)) + for i, e := range els { + out[i] = e.Key() + } + return out +} + +// TestSerializeSchedule_AllVariants covers the six variants no Mendix module +// ships, so the field sets are metamodel-derived rather than observed. The +// assertion that matters is that each variant writes ONLY its own fields: a +// merged field set is the shape that mxbuild accepts and Studio Pro refuses to +// open. +func TestSerializeSchedule_AllVariants(t *testing.T) { + full := model.Schedule{ + Multiplier: 2, MinuteOffset: 5, MonthOffset: 1, + HourOfDay: 6, MinuteOfHour: 30, DayOfMonth: 15, Month: 3, + DaySelector: "Last", Weekday: "Friday", + Weekdays: [7]bool{false, true, false, false, false, true, false}, + } + wantKeys := map[model.ScheduleKind][]string{ + model.ScheduleMinute: {"Multiplier"}, + model.ScheduleHour: {"MinuteOffset", "Multiplier"}, + model.ScheduleDay: {"HourOfDay", "MinuteOfHour"}, + model.ScheduleWeek: {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthDate: {"Multiplier", "MonthOffset", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthWeekday: {"Multiplier", "MonthOffset", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearDate: {"Month", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearWeekday: {"Month", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, + } + + for kind, keys := range wantKeys { + t.Run(string(kind), func(t *testing.T) { + s := full + s.Kind = kind + doc, err := SerializeSchedule(&s) + if err != nil { + t.Fatalf("SerializeSchedule: %v", err) + } + got := map[string]bool{} + for _, e := range doc { + got[e.Key] = true + } + if !got["$ID"] || !got["$Type"] { + t.Error("missing $ID/$Type") + } + delete(got, "$ID") + delete(got, "$Type") + + for _, k := range keys { + if !got[k] { + t.Errorf("missing %s", k) + } + delete(got, k) + } + for k := range got { + t.Errorf("%s is not a property of %s — writing it produces a document Studio Pro cannot open", k, ScheduleTypeNames[kind]) + } + }) + } +} + +func TestSerializeSchedule_UnknownKindIsRefused(t *testing.T) { + if _, err := SerializeSchedule(&model.Schedule{Kind: "Fortnightly"}); err == nil { + t.Fatal("expected an error for an unknown schedule kind") + } +} + +// TestSerialize_Defaults documents the values supplied when the caller leaves +// them empty, all taken from the reference documents. +func TestSerialize_Defaults(t *testing.T) { + ev := &model.ScheduledEvent{Name: "SE_Plain", MicroflowID: "M.MF"} + ev.ID = "22222222-2222-2222-2222-222222222222" + + out, err := Serialize(ev) + if err != nil { + t.Fatalf("Serialize: %v", err) + } + var doc bson.M + if err := bson.Unmarshal(out, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if doc["ExportLevel"] != "Hidden" { + t.Errorf("ExportLevel = %v, want Hidden", doc["ExportLevel"]) + } + if doc["OnOverlap"] != "DelayNext" { + t.Errorf("OnOverlap = %v, want DelayNext (SkipNext silently drops a run)", doc["OnOverlap"]) + } + if doc["TimeZone"] != "UTC" { + t.Errorf("TimeZone = %v, want UTC", doc["TimeZone"]) + } + if _, ok := doc["Interval"].(int64); !ok { + t.Errorf("Interval = %#v, want an int64 — Studio Pro writes int64, gen declares int32", doc["Interval"]) + } + if _, ok := doc["Schedule"]; ok { + t.Error("a nil schedule must not be written as an empty child") + } +} + +func TestSerialize_StartDateTimeRoundTrips(t *testing.T) { + want := time.Date(2024, 7, 8, 12, 12, 24, 923_000_000, time.UTC) + ev := &model.ScheduledEvent{Name: "SE", MicroflowID: "M.MF", StartDateTime: &want} + ev.ID = "33333333-3333-3333-3333-333333333333" + + out, err := Serialize(ev) + if err != nil { + t.Fatalf("Serialize: %v", err) + } + var doc bson.M + if err := bson.Unmarshal(out, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + got := Parse(doc, "id", "mod") + if got.StartDateTime == nil || !got.StartDateTime.Equal(want) { + t.Errorf("StartDateTime = %v, want %v", got.StartDateTime, want) + } +} diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index 24598a7ed..65c870b40 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -833,6 +833,10 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { b.statements = append(b.statements, &ast.DropQueueStmt{ Name: buildQualifiedName(names[0]), }) + } else if ctx.SCHEDULED() != nil && ctx.EVENT() != nil { + b.statements = append(b.statements, &ast.DropScheduledEventStmt{ + Name: buildQualifiedName(names[0]), + }) } else if ctx.MODEL() != nil { b.statements = append(b.statements, &ast.DropModelStmt{ Name: buildQualifiedName(names[0]), diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 596cb5d6d..5a3e4d157 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -214,6 +214,16 @@ func (b *Builder) ExitShowStatement(ctx *parser.ShowStatementContext) { } } b.statements = append(b.statements, stmt) + } else if ctx.SCHEDULED() != nil && ctx.EVENTS() != nil { + stmt := &ast.ShowScheduledEventsStmt{} + if ctx.IN() != nil { + if qn := ctx.QualifiedName(); qn != nil { + stmt.Module = getQualifiedNameText(qn) + } else if id := ctx.IDENTIFIER(); id != nil { + stmt.Module = id.GetText() + } + } + b.statements = append(b.statements, stmt) } else if ctx.LAYOUTS() != nil { stmt := &ast.ShowStmt{ObjectType: ast.ShowLayouts} if ctx.IN() != nil { @@ -712,6 +722,14 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { return } + // DESCRIBE SCHEDULED EVENT Module.Name + if ctx.SCHEDULED() != nil && ctx.EVENT() != nil { + if qn := ctx.QualifiedName(); qn != nil { + b.statements = append(b.statements, &ast.DescribeScheduledEventStmt{Name: buildQualifiedName(qn)}) + } + return + } + // Handle DESCRIBE MODULE ROLE (uses qualifiedName) if ctx.MODULE() != nil && ctx.ROLE() != nil { if qn := ctx.QualifiedName(); qn != nil { diff --git a/mdl/visitor/visitor_queue.go b/mdl/visitor/visitor_queue.go index 8da08ffea..a7ff58cc1 100644 --- a/mdl/visitor/visitor_queue.go +++ b/mdl/visitor/visitor_queue.go @@ -13,7 +13,7 @@ import ( // CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( ... ). func (b *Builder) ExitCreateQueueStatement(ctx *parser.CreateQueueStatementContext) { stmt := &ast.CreateQueueStmt{ - Name: buildQualifiedName(ctx.QualifiedName()), + Name: buildQualifiedName(ctx.QualifiedName()), Documentation: findDocCommentText(ctx), } if createStmt := findParentCreateStatement(ctx); createStmt != nil { diff --git a/mdl/visitor/visitor_scheduledevent.go b/mdl/visitor/visitor_scheduledevent.go new file mode 100644 index 000000000..5a52640b3 --- /dev/null +++ b/mdl/visitor/visitor_scheduledevent.go @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strconv" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// ExitCreateScheduledEventStatement builds a CreateScheduledEventStmt from +// CREATE [OR REPLACE|MODIFY] SCHEDULED EVENT Module.Name ( ... ). +// +// Every property is carried through as written; the executor decides which ones +// the chosen Repeat actually has and rejects the rest. Numeric properties are +// stored as pointers so an explicit 0 (a real hour, minute or month offset) is +// distinguishable from an omitted one. +func (b *Builder) ExitCreateScheduledEventStatement(ctx *parser.CreateScheduledEventStatementContext) { + stmt := &ast.CreateScheduledEventStmt{ + Name: buildQualifiedName(ctx.QualifiedName()), + Documentation: findDocCommentText(ctx), + } + if createStmt := findParentCreateStatement(ctx); createStmt != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { + stmt.CreateOrModify = true + } + } + + if body := ctx.ScheduledEventBody(); body != nil { + bodyCtx := body.(*parser.ScheduledEventBodyContext) + for _, prop := range bodyCtx.AllScheduledEventProperty() { + pc, ok := prop.(*parser.ScheduledEventPropertyContext) + if !ok || pc == nil { + continue + } + iok := pc.IdentifierOrKeyword(0) + if iok == nil { + continue + } + key := strings.ToLower(identifierOrKeywordText(iok)) + val := scheduledEventPropertyText(pc) + switch key { + case "microflow": + stmt.Microflow = val + case "repeat": + stmt.Repeat = val + case "multiplier": + stmt.Multiplier = parseIntPtr(val) + case "minuteoffset": + stmt.MinuteOffset = parseIntPtr(val) + case "monthoffset": + stmt.MonthOffset = parseIntPtr(val) + case "hourofday": + stmt.HourOfDay = parseIntPtr(val) + case "minuteofhour": + stmt.MinuteOfHour = parseIntPtr(val) + case "dayofmonth": + stmt.DayOfMonth = parseIntPtr(val) + case "month": + stmt.Month = parseIntPtr(val) + case "weekdays": + stmt.Weekdays = val + case "dayselector": + stmt.DaySelector = val + case "weekday": + stmt.Weekday = val + case "startdatetime": + stmt.StartDateTime = val + case "timezone": + stmt.TimeZone = val + case "onoverlap": + stmt.OnOverlap = val + case "enabled": + stmt.Enabled = parseBoolPtr(val) + case "excluded": + stmt.Excluded = parseBoolPtr(val) + case "exportlevel": + stmt.ExportLevel = val + case "documentation": + stmt.Documentation = val + } + } + } + + b.statements = append(b.statements, stmt) +} + +// scheduledEventPropertyText returns the value side of a property, unquoted. +// +// The key is identifierOrKeyword(0), so an identifier value is index 1 — +// reading index 0 would echo the key back as the value. +func scheduledEventPropertyText(pc *parser.ScheduledEventPropertyContext) string { + if qn := pc.QualifiedName(); qn != nil { + return getQualifiedNameText(qn) + } + if n := pc.NUMBER_LITERAL(); n != nil { + return n.GetText() + } + if s := pc.STRING_LITERAL(); s != nil { + return unquoteString(s.GetText()) + } + if bl := pc.BooleanLiteral(); bl != nil { + return bl.GetText() + } + if v := pc.IdentifierOrKeyword(1); v != nil { + return identifierOrKeywordText(v) + } + return "" +} + +// parseIntPtr returns nil for anything that is not an integer, so a malformed +// value is reported by the executor's validation rather than silently becoming 0. +func parseIntPtr(s string) *int { + n, err := strconv.Atoi(strings.TrimSpace(s)) + if err != nil { + return nil + } + return &n +} + +func parseBoolPtr(s string) *bool { + switch strings.ToLower(strings.TrimSpace(s)) { + case "true": + v := true + return &v + case "false": + v := false + return &v + } + return nil +} diff --git a/mdl/visitor/visitor_scheduledevent_test.go b/mdl/visitor/visitor_scheduledevent_test.go new file mode 100644 index 000000000..0a091ed75 --- /dev/null +++ b/mdl/visitor/visitor_scheduledevent_test.go @@ -0,0 +1,150 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func TestCreateScheduledEvent(t *testing.T) { + input := `CREATE SCHEDULED EVENT Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true, + StartDateTime: '2026-01-01T04:00:00Z' + );` + prog, errs := Build(input) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if !ok { + t.Fatalf("expected CreateScheduledEventStmt, got %T", prog.Statements[0]) + } + if stmt.Name.String() != "Ops.NightlyCleanup" { + t.Errorf("Name = %s", stmt.Name.String()) + } + if stmt.Microflow != "Ops.SE_Cleanup" { + t.Errorf("Microflow = %q — a qualified name must survive as one", stmt.Microflow) + } + if stmt.Repeat != "Daily" { + t.Errorf("Repeat = %q", stmt.Repeat) + } + if stmt.HourOfDay == nil || *stmt.HourOfDay != 4 { + t.Errorf("HourOfDay = %v", stmt.HourOfDay) + } + // 0 is a real minute, so an explicit 0 must not be indistinguishable from + // an omitted property — that is why these fields are pointers. + if stmt.MinuteOfHour == nil || *stmt.MinuteOfHour != 0 { + t.Errorf("MinuteOfHour = %v, want an explicit 0", stmt.MinuteOfHour) + } + if stmt.TimeZone != "Server" { + t.Errorf("TimeZone = %q", stmt.TimeZone) + } + if stmt.Enabled == nil || !*stmt.Enabled { + t.Errorf("Enabled = %v", stmt.Enabled) + } + if stmt.StartDateTime != "2026-01-01T04:00:00Z" { + t.Errorf("StartDateTime = %q", stmt.StartDateTime) + } +} + +func TestCreateScheduledEvent_OmittedFieldsStayNil(t *testing.T) { + prog, errs := Build(`CREATE SCHEDULED EVENT Ops.E ( Microflow: Ops.MF, Repeat: Minutely, Multiplier: 5 );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if stmt.Multiplier == nil || *stmt.Multiplier != 5 { + t.Errorf("Multiplier = %v", stmt.Multiplier) + } + if stmt.HourOfDay != nil { + t.Errorf("HourOfDay = %v, want nil for an omitted property", stmt.HourOfDay) + } + if stmt.Enabled != nil { + t.Errorf("Enabled = %v, want nil for an omitted property", stmt.Enabled) + } +} + +func TestCreateScheduledEvent_WeeklyAndSelectors(t *testing.T) { + prog, errs := Build(`CREATE OR MODIFY SCHEDULED EVENT Ops.E ( + Microflow: Ops.MF, + Repeat: MonthlyByWeekday, + Multiplier: 3, + DaySelector: Last, + Weekday: Friday, + Weekdays: 'Monday, Friday' + );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if !stmt.CreateOrModify { + t.Error("CreateOrModify not set for CREATE OR MODIFY") + } + if stmt.DaySelector != "Last" || stmt.Weekday != "Friday" { + t.Errorf("DaySelector/Weekday = %q/%q", stmt.DaySelector, stmt.Weekday) + } + if stmt.Weekdays != "Monday, Friday" { + t.Errorf("Weekdays = %q", stmt.Weekdays) + } +} + +func TestDropScheduledEvent(t *testing.T) { + prog, errs := Build(`DROP SCHEDULED EVENT Ops.NightlyCleanup;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.DropScheduledEventStmt) + if !ok { + t.Fatalf("expected DropScheduledEventStmt, got %T", prog.Statements[0]) + } + if stmt.Name.String() != "Ops.NightlyCleanup" { + t.Errorf("Name = %s", stmt.Name.String()) + } +} + +func TestShowAndDescribeScheduledEvents(t *testing.T) { + prog, errs := Build(`SHOW SCHEDULED EVENTS; LIST SCHEDULED EVENTS IN Ops; DESCRIBE SCHEDULED EVENT Ops.NightlyCleanup;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 3 { + t.Fatalf("expected 3 statements, got %d", len(prog.Statements)) + } + if _, ok := prog.Statements[0].(*ast.ShowScheduledEventsStmt); !ok { + t.Errorf("statement 0 = %T", prog.Statements[0]) + } + s1, ok := prog.Statements[1].(*ast.ShowScheduledEventsStmt) + if !ok { + t.Fatalf("statement 1 = %T", prog.Statements[1]) + } + if s1.Module != "Ops" { + t.Errorf("Module = %q", s1.Module) + } + if _, ok := prog.Statements[2].(*ast.DescribeScheduledEventStmt); !ok { + t.Errorf("statement 2 = %T", prog.Statements[2]) + } +} + +// TestScheduledKeywordStillUsableAsIdentifier guards the cost of adding +// SCHEDULED to the lexer: a new keyword stops being usable as an ordinary name +// unless it is also listed in the `keyword` rule. +func TestScheduledKeywordStillUsableAsIdentifier(t *testing.T) { + prog, errs := Build(`CREATE ENTITY Ops.Job ( scheduled: Boolean, event: String(50) );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateEntityStmt) + if len(stmt.Attributes) != 2 { + t.Fatalf("expected 2 attributes, got %d", len(stmt.Attributes)) + } + if stmt.Attributes[0].Name != "scheduled" || stmt.Attributes[1].Name != "event" { + t.Errorf("attribute names = %q, %q", stmt.Attributes[0].Name, stmt.Attributes[1].Name) + } +} diff --git a/model/types.go b/model/types.go index bdfde6ad1..908ec38a6 100644 --- a/model/types.go +++ b/model/types.go @@ -244,6 +244,63 @@ func (r *RegularExpression) GetContainerID() ID { return r.ContainerID } +// ScheduleKind identifies which ScheduledEvents$*Schedule variant a schedule is. +// The variants differ in which fields they carry, so the kind has to be decided +// before any field is read or written — see Schedule. +type ScheduleKind string + +const ( + ScheduleMinute ScheduleKind = "Minute" + ScheduleHour ScheduleKind = "Hour" + ScheduleDay ScheduleKind = "Day" + ScheduleWeek ScheduleKind = "Week" + ScheduleMonthDate ScheduleKind = "MonthDate" + ScheduleMonthWeekday ScheduleKind = "MonthWeekday" + ScheduleYearDate ScheduleKind = "YearDate" + ScheduleYearWeekday ScheduleKind = "YearWeekday" +) + +// Schedule is the repeat rule of a scheduled event — the polymorphic +// ScheduledEvents$Schedule child, flattened into one struct with Kind saying +// which variant it is. +// +// It is flat rather than eight types because the fields overlap heavily and the +// consumers (a formatter, a serializer, a describe) all switch on the kind +// anyway. Only the fields listed for a kind are written; the rest are ignored: +// +// Minute Multiplier +// Hour Multiplier, MinuteOffset +// Day HourOfDay, MinuteOfHour +// Week Weekdays, HourOfDay, MinuteOfHour +// MonthDate Multiplier, MonthOffset, DayOfMonth, HourOfDay, MinuteOfHour +// MonthWeekday Multiplier, MonthOffset, DaySelector, Weekday, HourOfDay, MinuteOfHour +// YearDate Month, DayOfMonth, HourOfDay, MinuteOfHour +// YearWeekday Month, DaySelector, Weekday, HourOfDay, MinuteOfHour +type Schedule struct { + Kind ScheduleKind `json:"kind"` + + // Multiplier is the repeat count ("every N minutes/hours/months"). + // Day, Week and the two Year variants have no multiplier in the metamodel. + Multiplier int `json:"multiplier,omitempty"` + // MinuteOffset is the minute within the hour, for the Hour variant only. + MinuteOffset int `json:"minuteOffset,omitempty"` + // MonthOffset selects which month of a multi-month cycle fires. + MonthOffset int `json:"monthOffset,omitempty"` + + HourOfDay int `json:"hourOfDay,omitempty"` + MinuteOfHour int `json:"minuteOfHour,omitempty"` + + // DayOfMonth is 1-31; Month is 1-12. + DayOfMonth int `json:"dayOfMonth,omitempty"` + Month int `json:"month,omitempty"` + + // Weekdays holds the seven per-day flags of the Week variant, Sunday first. + Weekdays [7]bool `json:"weekdays,omitempty"` + // DaySelector is First|Second|Third|Fourth|Last, Weekday is Sunday..Saturday. + DaySelector string `json:"daySelector,omitempty"` + Weekday string `json:"weekday,omitempty"` +} + // ScheduledEvent represents a scheduled event. type ScheduledEvent struct { BaseElement @@ -256,6 +313,15 @@ type ScheduledEvent struct { Interval int `json:"interval,omitempty"` IntervalType string `json:"intervalType,omitempty"` Enabled bool `json:"enabled"` + // Schedule is the repeat rule. Every Studio Pro-authored event carries one; + // nil means the document did not have the child. + Schedule *Schedule `json:"schedule,omitempty"` + // OnOverlap is SkipNext or DelayNext — what happens when a run is still + // going when the next one is due. This is a scheduled event's own + // concurrency control; it does not go through a task queue. + OnOverlap string `json:"onOverlap,omitempty"` + ExportLevel string `json:"exportLevel,omitempty"` + Excluded bool `json:"excluded,omitempty"` } // GetName returns the scheduled event's name. diff --git a/sdk/mpr/parser_enumeration.go b/sdk/mpr/parser_enumeration.go index dfaac3bc9..964ae548e 100644 --- a/sdk/mpr/parser_enumeration.go +++ b/sdk/mpr/parser_enumeration.go @@ -5,6 +5,7 @@ package mpr import ( "fmt" + "github.com/mendixlabs/mxcli/mdl/scheduledevents" "github.com/mendixlabs/mxcli/model" "go.mongodb.org/mongo-driver/bson" @@ -175,38 +176,17 @@ func (r *Reader) parseScheduledEvent(unitID, containerID string, contents []byte return nil, err } - var raw map[string]any + var raw bson.M if err := bson.Unmarshal(contents, &raw); err != nil { return nil, fmt.Errorf("failed to unmarshal BSON: %w", err) } - event := &model.ScheduledEvent{} - event.ID = model.ID(unitID) - event.TypeName = "ScheduledEvents$ScheduledEvent" - event.ContainerID = model.ID(containerID) - - if name, ok := raw["Name"].(string); ok { - event.Name = name - } - if doc, ok := raw["Documentation"].(string); ok { - event.Documentation = doc - } - if mfID, ok := raw["Microflow"].(string); ok { - event.MicroflowID = model.ID(mfID) - } - if enabled, ok := raw["Enabled"].(bool); ok { - event.Enabled = enabled - } - // Issue #585: Studio Pro stores Interval as BSON int64; extractInt - // also accepts int32/int/float64 emitted by other writers. - if _, ok := raw["Interval"]; ok { - event.Interval = extractInt(raw["Interval"]) - } - if intervalType, ok := raw["IntervalType"].(string); ok { - event.IntervalType = intervalType - } - - return event, nil + // Shared with the modelsdk engine so both read the same keys the shared + // writer produces — including the polymorphic Schedule child, which this + // parser used to drop, and StartDateTime, which Studio Pro stores as a BSON + // datetime. (Interval is int64 in Studio Pro documents; the codec accepts + // every numeric width — issue #585.) + return scheduledevents.Parse(raw, model.ID(unitID), model.ID(containerID)), nil } // resolveContents handles MPR v2 external file references. diff --git a/sdk/mpr/scheduledevents.go b/sdk/mpr/scheduledevents.go new file mode 100644 index 000000000..21aaa1fb0 --- /dev/null +++ b/sdk/mpr/scheduledevents.go @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "fmt" + + sched "github.com/mendixlabs/mxcli/mdl/scheduledevents" + "github.com/mendixlabs/mxcli/model" +) + +// Scheduled events (ScheduledEvents$ScheduledEvent) for the legacy engine. +// +// The document shape lives in mdl/scheduledevents so both engines write exactly +// the same bytes — the schedule child has eight variants that differ in which +// fields they carry, and two copies of that dispatch would eventually disagree. +// +// The legacy READER for scheduled events is parseScheduledEvent (in +// parser_enumeration.go), which predates this and covers only the flat legacy +// fields; the write path here round-trips through the shared codec. + +// CreateScheduledEvent inserts a new scheduled event document. +func (w *Writer) CreateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("CreateScheduledEvent: nil event") + } + if ev.ID == "" { + ev.ID = model.ID(generateUUID()) + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return w.insertUnit(string(ev.ID), string(ev.ContainerID), "Documents", sched.TypeName, contents) +} + +// UpdateScheduledEvent rewrites an existing scheduled event in place. +func (w *Writer) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("UpdateScheduledEvent: nil event") + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return w.UpdateRawUnit(string(ev.ID), contents) +} + +// DeleteScheduledEvent removes a scheduled event by ID. +func (w *Writer) DeleteScheduledEvent(id string) error { + return w.deleteUnit(id) +} From 3317ffc1009acabcb3f5074a5dd2a8da6fda0fdc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 05:12:42 +0000 Subject: [PATCH 26/35] Phase 2 slice 4: preserve a module's role grants across an update The second thing an update has to carry, alongside the GUIDs. A user role's grant of a module role lives in the *project's* security document, not in the module, so removing the module takes it away and putting the module back does not return it. Measured on a blank 11.12.1 app: dropping Administration left Administrator holding 2 module roles instead of 3, and User 3 instead of 4. Nothing complains -- the app builds, and users quietly lose access. CaptureRoleGrants records which of one module's roles each user role grants; RestoreRoleGrants re-grants them afterwards. Only that module's roles are recorded, because an update touches one module and restoring a grant it never removed would be a write nobody asked for. A recorded role the new version no longer defines is reported as dropped rather than skipped. Someone had that access and now cannot, which is a permission change the operator has to see rather than something to hide behind a successful-looking update. Restoring goes through MDL (ALTER USER ROLE ... ADD MODULE ROLES) so it uses the same validated write path a user would, instead of a second hand-rolled security writer. The test asserts the loss before asserting the repair -- grants gone after the drop, still gone after the transplant, restored only by the restore. Without those two intermediate checks it would pass against code that did nothing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/marketplace/grants.go | 173 +++++++++++++++++++++++++++ cmd/mxcli/marketplace/grants_test.go | 114 ++++++++++++++++++ 2 files changed, 287 insertions(+) create mode 100644 cmd/mxcli/marketplace/grants.go create mode 100644 cmd/mxcli/marketplace/grants_test.go diff --git a/cmd/mxcli/marketplace/grants.go b/cmd/mxcli/marketplace/grants.go new file mode 100644 index 000000000..38786784b --- /dev/null +++ b/cmd/mxcli/marketplace/grants.go @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "bytes" + "fmt" + "sort" + "strings" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/executor" + "github.com/mendixlabs/mxcli/mdl/visitor" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// RoleGrants records which of a module's roles each application user role +// grants: user role name → module role names, unqualified. +// +// These are the second thing an update must carry, alongside the `GUID`s. They +// live in the project's security document rather than in the module, so removing +// the module takes them with it — measured on a blank 11.12.1 app, where +// dropping Administration left Administrator with 2 module roles instead of 3 +// and User with 3 instead of 4. Restoring the module does not bring them back, +// and the loss is quiet: the app builds, and users simply lose access. +type RoleGrants map[string][]string + +// UserRoles returns the recorded user role names in a stable order. +func (g RoleGrants) UserRoles() []string { + out := make([]string, 0, len(g)) + for r := range g { + out = append(out, r) + } + sort.Strings(out) + return out +} + +// CaptureRoleGrants records every grant of one module's roles. +// +// Grants of *other* modules' roles are deliberately not recorded: an update +// touches one module, and restoring a grant this function never removed would +// be a write nobody asked for. +func CaptureRoleGrants(mprPath, moduleName string) (RoleGrants, error) { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return nil, fmt.Errorf("open %s: %w", mprPath, err) + } + defer reader.Close() + + sec, err := reader.GetProjectSecurity() + if err != nil { + return nil, fmt.Errorf("read project security: %w", err) + } + if sec == nil { + return RoleGrants{}, nil + } + + prefix := strings.ToLower(moduleName) + "." + grants := RoleGrants{} + for _, ur := range sec.UserRoles { + var mine []string + for _, mr := range ur.ModuleRoles { + if strings.HasPrefix(strings.ToLower(mr), prefix) { + mine = append(mine, mr[len(prefix):]) + } + } + if len(mine) > 0 { + sort.Strings(mine) + grants[ur.Name] = mine + } + } + return grants, nil +} + +// RestoreRoleGrants re-grants recorded module roles, and reports the ones it +// could not. +// +// A recorded role the new version no longer defines is returned in `dropped` +// rather than silently skipped: someone had that access and now cannot, which +// is a change the operator has to see. Restoring goes through MDL so it uses +// the same validated write path a user would, rather than a second +// hand-rolled security writer. +func RestoreRoleGrants(mprPath, moduleName string, grants RoleGrants, newBackend func() backend.FullBackend) (restored int, dropped []string, err error) { + if len(grants) == 0 { + return 0, nil, nil + } + + available, err := moduleRoleNames(mprPath, moduleName) + if err != nil { + return 0, nil, err + } + + var stmts []string + for _, userRole := range grants.UserRoles() { + var keep []string + for _, role := range grants[userRole] { + if available[strings.ToLower(role)] { + keep = append(keep, moduleName+"."+role) + continue + } + dropped = append(dropped, fmt.Sprintf("%s: %s.%s", userRole, moduleName, role)) + } + if len(keep) == 0 { + continue + } + stmts = append(stmts, fmt.Sprintf("ALTER USER ROLE %s ADD MODULE ROLES (%s);", + userRole, strings.Join(keep, ", "))) + restored += len(keep) + } + sort.Strings(dropped) + + if len(stmts) == 0 { + return 0, dropped, nil + } + if err := execStatements(mprPath, strings.Join(stmts, "\n"), newBackend); err != nil { + return 0, dropped, err + } + return restored, dropped, nil +} + +// moduleRoleNames returns the roles the module currently defines, lowercased. +func moduleRoleNames(mprPath, moduleName string) (map[string]bool, error) { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return nil, fmt.Errorf("open %s: %w", mprPath, err) + } + defer reader.Close() + + mods, err := reader.ListModules() + if err != nil { + return nil, err + } + for _, m := range mods { + if !strings.EqualFold(m.Name, moduleName) { + continue + } + sec, serr := reader.GetModuleSecurity(m.ID) + if serr != nil || sec == nil { + return map[string]bool{}, nil //nolint:nilerr // no security document means no roles + } + out := make(map[string]bool, len(sec.ModuleRoles)) + for _, r := range sec.ModuleRoles { + out[strings.ToLower(r.Name)] = true + } + return out, nil + } + return nil, fmt.Errorf("module %q not found", moduleName) +} + +// execStatements runs MDL against a project through the normal executor. +func execStatements(mprPath, mdl string, newBackend func() backend.FullBackend) error { + prog, errs := visitor.Build(mdl) + if len(errs) > 0 { + return fmt.Errorf("build restore statements: %v", errs) + } + var sink bytes.Buffer + ex := executor.New(&sink) + defer ex.Close() + if newBackend != nil { + ex.SetBackendFactory(newBackend) + } + if err := ex.Execute(&ast.ConnectStmt{Path: mprPath}); err != nil { + return fmt.Errorf("connect %s: %w", mprPath, err) + } + for _, s := range prog.Statements { + if err := ex.Execute(s); err != nil { + return fmt.Errorf("%w\noutput: %s", err, sink.String()) + } + } + return nil +} diff --git a/cmd/mxcli/marketplace/grants_test.go b/cmd/mxcli/marketplace/grants_test.go new file mode 100644 index 000000000..a2e86883f --- /dev/null +++ b/cmd/mxcli/marketplace/grants_test.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "strings" + "testing" +) + +// TestRoleGrants_SurviveADropAndTransplant is the whole point of this pair. +// +// A module's role grants live in the *project's* security document, not in the +// module, so dropping the module takes them with it and putting the module back +// does not return them. The app still builds; users just quietly lose access. +// Measured on a blank 11.12.1 project: dropping Administration left Administrator +// with 2 module roles instead of 3, and User with 3 instead of 4. +func TestRoleGrants_SurviveADropAndTransplant(t *testing.T) { + source := copyFixture(t) + target := copyFixture(t) + const module = "Administration" + + before, err := CaptureRoleGrants(target, module) + if err != nil { + t.Fatalf("capture: %v", err) + } + if len(before) == 0 { + t.Skip("the fixture grants none of this module's roles; nothing to preserve") + } + + execMDL(t, target, "drop module "+module+";") + + // The grants are gone — this is the loss being repaired, and asserting it + // keeps the restore below from passing vacuously. + gone, err := CaptureRoleGrants(target, module) + if err != nil { + t.Fatalf("capture after drop: %v", err) + } + if len(gone) != 0 { + t.Fatalf("expected the drop to remove the grants; %d user role(s) still hold them", len(gone)) + } + + if _, err := TransplantModule(source, target, module); err != nil { + t.Fatalf("TransplantModule: %v", err) + } + // Still gone after the module is back: the module returning does not restore + // grants that live outside it. + if after, _ := CaptureRoleGrants(target, module); len(after) != 0 { + t.Fatalf("transplant unexpectedly restored grants; the restore step would be untested") + } + + restored, dropped, err := RestoreRoleGrants(target, module, before, testBackend) + if err != nil { + t.Fatalf("RestoreRoleGrants: %v", err) + } + if len(dropped) != 0 { + t.Errorf("no role should be missing when the versions match; got %v", dropped) + } + + after, err := CaptureRoleGrants(target, module) + if err != nil { + t.Fatalf("capture after restore: %v", err) + } + for _, ur := range before.UserRoles() { + want, got := strings.Join(before[ur], ","), strings.Join(after[ur], ",") + if want != got { + t.Errorf("%s: grants = [%s], want [%s]", ur, got, want) + } + } + if restored == 0 { + t.Error("restored count should be non-zero") + } +} + +// TestRestoreRoleGrants_ReportsARoleTheNewVersionRemoved — someone had that +// access and now cannot. Silently skipping it would hide a real permission +// change behind a successful-looking update. +func TestRestoreRoleGrants_ReportsARoleTheNewVersionRemoved(t *testing.T) { + mpr := copyFixture(t) + const module = "Administration" + + grants, err := CaptureRoleGrants(mpr, module) + if err != nil { + t.Fatalf("capture: %v", err) + } + if len(grants) == 0 { + t.Skip("fixture grants none of this module's roles") + } + first := grants.UserRoles()[0] + grants[first] = append(grants[first], "RoleRemovedInTheNewVersion") + + _, dropped, err := RestoreRoleGrants(mpr, module, grants, testBackend) + if err != nil { + t.Fatalf("RestoreRoleGrants: %v", err) + } + if len(dropped) != 1 || !strings.Contains(dropped[0], "RoleRemovedInTheNewVersion") { + t.Errorf("dropped = %v, want the one role the module no longer defines", dropped) + } +} + +// TestCaptureRoleGrants_IgnoresOtherModules — an update touches one module, so +// restoring a grant it never removed would be a write nobody asked for. +func TestCaptureRoleGrants_IgnoresOtherModules(t *testing.T) { + grants, err := CaptureRoleGrants(copyFixture(t), "Administration") + if err != nil { + t.Fatalf("capture: %v", err) + } + for ur, roles := range grants { + for _, r := range roles { + if strings.Contains(r, ".") { + t.Errorf("%s: %q is qualified, so a role from another module leaked in", ur, r) + } + } + } +} From e2d31dd01ae113dfda6c69ef4b8beeae078c7911 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 05:16:39 +0000 Subject: [PATCH 27/35] docs(queues,scheduled-events): wire the new document types into every help surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two features were implemented but only half-discoverable. Filling the gaps the CLAUDE.md checklist calls for: - `mxcli describe` did not know either type. Both are now in the type list, the dispatch, the error message, and the live-reader auto-detect map — so `mxcli describe -p app.mpr Ops.NightlyCleanup` resolves without naming a type, as it already does for every other document. - `show structure` counted scheduled events but not queues, so a queue was invisible at every depth. Added to the depth-1 counts, the compact summary and the depth-2/3 listings, and to `project-tree` (which feeds the VS Code tree). - New user-facing skill `.claude/skills/mendix/scheduled-events-and-queues.md`, indexed in the skills README and its LLM loading guide, so `mxcli init` ships it into user projects. It leads with the trap: pick the Repeat first, then use only that repeat's fields. - CLAUDE.md: both features added to the implementation status, and the skill to the read-this-first list. - VS Code extension: tree icons, terminal-link type inference, the describe fallback type list, and the context-menu viewItem pattern. Already in place from the feature commits: `mxcli syntax queue` / `mxcli syntax scheduled-event`, MDL_QUICK_REFERENCE, the docs-site page, the symptom-table rows and the MDL examples. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/skills/mendix/README.md | 4 + .../mendix/scheduled-events-and-queues.md | 186 ++++++++++++++++++ CLAUDE.md | 3 + cmd/mxcli/cmd_describe.go | 42 ++-- cmd/mxcli/project_tree.go | 11 ++ cmd/mxcli/tui/icons.go | 1 + mdl/executor/cmd_structure.go | 49 ++++- vscode-mdl/package.json | 2 +- vscode-mdl/src/extension.ts | 2 +- vscode-mdl/src/projectTreeProvider.ts | 2 + vscode-mdl/src/terminalLinkProvider.ts | 1 + 11 files changed, 283 insertions(+), 20 deletions(-) create mode 100644 .claude/skills/mendix/scheduled-events-and-queues.md diff --git a/.claude/skills/mendix/README.md b/.claude/skills/mendix/README.md index 54cf5c71d..d9d85ec2b 100644 --- a/.claude/skills/mendix/README.md +++ b/.claude/skills/mendix/README.md @@ -23,6 +23,7 @@ Detailed syntax for each MDL document type: | [write-oql-queries.md](write-oql-queries.md) | OQL query syntax | Creating VIEW entities | | [create-page.md](create-page.md) | Page and widget syntax | Creating pages | | [fragments.md](fragments.md) | Fragment (reusable widget group) syntax | Reusing widget patterns across pages | +| [scheduled-events-and-queues.md](scheduled-events-and-queues.md) | Scheduled event (cron) and task queue syntax | Running a microflow on a schedule; bounding background concurrency | ## Patterns (By Use Case) @@ -92,6 +93,9 @@ Load skills based on the task: | "Create export mapping" | `json-structures-and-mappings.md` | | "Map JSON to entities" | `json-structures-and-mappings.md` | | "Seed/populate test data" | `demo-data.md` | +| "Run a microflow nightly / hourly / on a schedule" | `scheduled-events-and-queues.md` | +| "Add a cron job / batch job / recurring task" | `scheduled-events-and-queues.md` | +| "Limit how many background tasks run at once" | `scheduled-events-and-queues.md` | | "Update widget properties" | `bulk-widget-updates.md` | | "Change widgets in bulk" | `bulk-widget-updates.md` | | "Reuse widgets across pages" | `fragments.md` | diff --git a/.claude/skills/mendix/scheduled-events-and-queues.md b/.claude/skills/mendix/scheduled-events-and-queues.md new file mode 100644 index 000000000..a6ebf19e1 --- /dev/null +++ b/.claude/skills/mendix/scheduled-events-and-queues.md @@ -0,0 +1,186 @@ +# Scheduled Events and Task Queues + +## When to Use This Skill + +Use this skill when the user wants to: +- Run a microflow on a schedule ("every night at 4", "hourly", "cron", "batch job") +- Inspect or change an existing scheduled event +- Limit how many background tasks run at once (a task queue) +- Understand why `mxcli` refuses to rewrite a microflow that has a queued call + +**These two features are unrelated.** A scheduled event does **not** go through a +task queue. Its own concurrency control is `OnOverlap`. + +## Scheduled Events + +Mendix's cron: run a microflow on a repeating schedule. + +```sql +-- Inspect +list scheduled events; +list scheduled events in Ops; +describe scheduled event Ops.NightlyCleanup; -- re-executable MDL + +-- Create +create scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +drop scheduled event Ops.NightlyCleanup; +``` + +`Microflow` and `Repeat` are **always required**. `show` is a synonym for `list`. + +### Pick the Repeat first, then use only its fields + +Mendix stores the repeat rule as one of eight types, and they differ in **which +fields they carry** — not just in their values. Naming a field from another +repeat is an error, not a no-op: + +``` +Error: Repeat Daily does not have Multiplier — it takes HourOfDay, MinuteOfHour +``` + +| Repeat | Fields | Means | +|--------|--------|-------| +| `Minutely` | `Multiplier` | every N minutes | +| `Hourly` | `Multiplier`, `MinuteOffset` | every N hours, at :MM past | +| `Daily` | `HourOfDay`, `MinuteOfHour` | every day at HH:MM (**no multiplier**) | +| `Weekly` | `Weekdays`, `HourOfDay`, `MinuteOfHour` | on the named days at HH:MM | +| `MonthlyByDate` | `Multiplier`, `MonthOffset`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | the Dth of every N months | +| `MonthlyByWeekday` | `Multiplier`, `MonthOffset`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the last Friday of every N months | +| `YearlyByDate` | `Month`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | every 2 January | +| `YearlyByWeekday` | `Month`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the first Monday of March | + +Field values: + +| Field | Value | +|-------|-------| +| `Multiplier` | 1 or more (defaults to 1) | +| `MinuteOffset` | 0–59 | +| `MonthOffset` | 0-based: which month of a multi-month cycle fires | +| `HourOfDay` / `MinuteOfHour` | 0–23 / 0–59 | +| `DayOfMonth` / `Month` | 1–31 / 1–12 | +| `Weekdays` | quoted list: `'Monday, Friday'` (case-insensitive) | +| `DaySelector` | `First`, `Second`, `Third`, `Fourth`, `Last` | +| `Weekday` | `Sunday` … `Saturday` | + +Optional on any repeat: + +| Property | Values | Default | +|----------|--------|---------| +| `Enabled` | `true` / `false` | `false` — **a new event does not run until you enable it** | +| `OnOverlap` | `DelayNext` / `SkipNext` | `DelayNext` | +| `TimeZone` | `UTC` / `Server` | `UTC` | +| `StartDateTime` | RFC 3339, e.g. `'2026-01-01T04:00:00Z'` | none | +| `Documentation` | free text | none | + +`SkipNext` drops a run that would overlap the previous one; `DelayNext` waits. + +### More examples + +```sql +-- Every two hours, 23 minutes past +create scheduled event Ops.HourlyPing ( + Microflow: Ops.SE_Ping, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); + +-- Mondays and Fridays at 09:30 +create scheduled event Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); + +-- The last Friday of every third month, 18:00 +create scheduled event Ops.QuarterEnd ( + Microflow: Ops.SE_Close, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18 +); +``` + +## Task Queues + +A task queue bounds how many queued microflow calls run at once. + +```sql +list queues; +describe queue Ops.OrderProcessing; + +create queue Ops.OrderProcessing ( Parallelism: 3, ClusterWide: true ); +create queue Ops.Mail; -- defaults: parallelism 1, per-instance + +create or modify queue Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); +drop queue Ops.Mail; +``` + +| Property | Meaning | Default | +|----------|---------|---------| +| `Parallelism` | how many run at once — an **expression**, not a number | `1` | +| `ClusterWide` | `true` = across the cluster, `false` = per runtime instance | `false` | + +Mendix stores parallelism as an expression string, so `3` and `'3'` are the same +thing and an arbitrary expression is legal. + +## Common Mistakes + +| Mistake | Symptom | Fix | +|---------|---------|-----| +| `Multiplier` on a `Daily` repeat | `Repeat Daily does not have Multiplier` | Daily has no multiplier — use `HourOfDay`/`MinuteOfHour`, or switch to `Hourly` | +| Forgetting `Enabled: true` | The event is in the model but never runs | Set `Enabled: true` (the default is false) | +| `TimeZone: server` | `has the wrong casing — Mendix stores it as "Server"` | Use the exact spelling: `Server`, `UTC`, `DelayNext`, `SkipNext`, `Last`, `Friday` | +| `HourOfDay: 24` | `it must be between 0 and 23` | Hours are 0–23; midnight is `0` | +| Expecting a queue to throttle a scheduled event | Nothing changes | They are unrelated — use `OnOverlap` | + +## Rewriting a Microflow with a Queued Call Is Refused + +MDL cannot yet author a *queued call* — the binding lives on the call activity +inside a microflow, not on the queue. So `create or replace|modify microflow` is +refused when the stored microflow has one: + +``` +Error: microflow Ops.ACT_Caller has 1 call(s) bound to a task queue (Ops.MyQueue), +and rewriting it would silently drop that binding +``` + +This is deliberate. Change that microflow in Studio Pro, or remove the task queue +from the call first. Without the refusal the binding was written back as null and +the project then looked *healthier* than before — `mx check` stopped reporting +`CE1613 "The selected task queue no longer exists"`, because the configuration +the error was about had been deleted. + +## Validation Checklist + +Before presenting a script: + +```bash +mxcli check script.mdl # catches wrong-repeat fields (MDL-SCHED01) +mxcli check script.mdl -p app.mpr --references +``` + +- [ ] Every scheduled event has `Microflow` and `Repeat` +- [ ] Only that repeat's fields are used +- [ ] `Enabled: true` if it is meant to run +- [ ] Enum values spelled exactly (`Server`, `DelayNext`, `Last`, `Monday`) +- [ ] The target microflow exists and takes no parameters + +## Related + +- `mxcli syntax scheduled-event`, `mxcli syntax queue` — full syntax reference +- `write-microflows.md` — writing the microflow the event calls +- `project-settings.md` — after-startup / before-shutdown microflows diff --git a/CLAUDE.md b/CLAUDE.md index d1767a6ab..9bcdf0fbe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -603,6 +603,7 @@ Regenerate after modifying `MDLLexer.g4`, `MDLParser.g4`, or any `domains/*.g4` - `.claude/skills/overview-pages.md` - CRUD page patterns - `.claude/skills/master-detail-pages.md` - Master-detail page patterns - `.claude/skills/generate-domain-model.md` - Entity/Association syntax +- `.claude/skills/mendix/scheduled-events-and-queues.md` - **Scheduled events (Mendix's cron) and task queues**: the eight Repeat variants and which fields each one takes, why a queue does NOT throttle a scheduled event, and why rewriting a microflow with a queued call is refused - `.claude/skills/check-syntax.md` - Pre-flight validation checklist - `.claude/skills/organize-project.md` - Folders, MOVE command, project structure conventions - `.claude/skills/manage-security.md` - Security roles, access control, GRANT/REVOKE patterns @@ -668,6 +669,8 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - ALTER WORKFLOW (SET properties, INSERT/DROP/REPLACE activities, outcomes, paths, conditions, boundary events) - CALCULATED BY microflow syntax for calculated attributes - Image collections (SHOW/DESCRIBE/CREATE/DROP) +- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. See `.claude/skills/mendix/scheduled-events-and-queues.md` +- Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005): the rebuild used to write `QueueSettings` back as null, which made `mx check` go from CE1613 to 0 errors by deleting the user's configuration - AI agent documents: Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) - OData contract browsing (SHOW/DESCRIBE CONTRACT ENTITIES/ACTIONS FROM cached $metadata) - AsyncAPI contract browsing (SHOW/DESCRIBE CONTRACT CHANNELS/MESSAGES FROM cached AsyncAPI) diff --git a/cmd/mxcli/cmd_describe.go b/cmd/mxcli/cmd_describe.go index bfeec9091..776c4beb2 100644 --- a/cmd/mxcli/cmd_describe.go +++ b/cmd/mxcli/cmd_describe.go @@ -47,6 +47,8 @@ Types: odataclient Describe a consumed OData service odataservice Describe a published OData service imagecollection Describe an image collection (also: "image collection") + queue Describe a task queue + scheduledevent Describe a scheduled event (also: "scheduled event") businesseventservice Describe a business event service (also: "business event service") databaseconnection Describe a database connection (also: "database connection") agent Describe an AI agent (also: "agent") @@ -166,6 +168,10 @@ Example: mdlCmd = fmt.Sprintf("DESCRIBE ODATA SERVICE %s", name) case "IMAGECOLLECTION", "IMAGE COLLECTION": mdlCmd = fmt.Sprintf("DESCRIBE IMAGE COLLECTION %s", name) + case "QUEUE": + mdlCmd = fmt.Sprintf("DESCRIBE QUEUE %s", name) + case "SCHEDULEDEVENT", "SCHEDULED EVENT": + mdlCmd = fmt.Sprintf("DESCRIBE SCHEDULED EVENT %s", name) case "BUSINESSEVENTSERVICE", "BUSINESS EVENT SERVICE": mdlCmd = fmt.Sprintf("DESCRIBE BUSINESS EVENT SERVICE %s", name) case "DATABASECONNECTION", "DATABASE CONNECTION": @@ -190,8 +196,8 @@ Example: mdlCmd = "" // handled directly by format-specific path default: fmt.Fprintf(os.Stderr, "Unknown type: %s\n", strings.Join(args[:len(args)-1], " ")) - fmt.Fprintln(os.Stderr, "Valid types: module, entity, association, enumeration, constant, microflow, nanoflow, workflow, page, snippet, layout, javaaction, jsonstructure, importmapping, exportmapping, restclient, odataclient, odataservice, imagecollection, businesseventservice, databaseconnection, agent, aimodel, knowledgebase, consumedmcpservice, datatransformer, modulerole, userrole, projectsecurity, settings, demouser, navigation, systemoverview") - fmt.Fprintln(os.Stderr, "Multi-word types also accepted: json structure, import mapping, export mapping, rest client, image collection, business event service, agent, model, knowledge base, consumed mcp service, data transformer, etc.") + fmt.Fprintln(os.Stderr, "Valid types: module, entity, association, enumeration, constant, microflow, nanoflow, workflow, page, snippet, layout, javaaction, jsonstructure, importmapping, exportmapping, restclient, odataclient, odataservice, imagecollection, queue, scheduledevent, businesseventservice, databaseconnection, agent, aimodel, knowledgebase, consumedmcpservice, datatransformer, modulerole, userrole, projectsecurity, settings, demouser, navigation, systemoverview") + fmt.Fprintln(os.Stderr, "Multi-word types also accepted: json structure, import mapping, export mapping, rest client, image collection, scheduled event, business event service, agent, model, knowledge base, consumed mcp service, data transformer, etc.") os.Exit(1) } @@ -323,21 +329,23 @@ var objectTypeToDescribe = map[string]string{ // types are listed (page templates, building blocks, rules, etc. are absent so // they don't resolve). var unitTypeToDescribe = map[string]string{ - "Microflows$Microflow": "microflow", - "Microflows$Nanoflow": "nanoflow", - "Forms$Page": "page", - "Forms$Snippet": "snippet", - "Pages$BuildingBlock": "buildingblock", - "Forms$BuildingBlock": "buildingblock", - "Forms$Layout": "layout", - "Enumerations$Enumeration": "enumeration", - "Constants$Constant": "constant", - "JavaActions$JavaAction": "javaaction", - "JsonStructures$JsonStructure": "jsonstructure", - "ImportMappings$ImportMapping": "importmapping", - "ExportMappings$ExportMapping": "exportmapping", - "Images$ImageCollection": "imagecollection", - "Workflows$Workflow": "workflow", + "Microflows$Microflow": "microflow", + "Microflows$Nanoflow": "nanoflow", + "Forms$Page": "page", + "Forms$Snippet": "snippet", + "Pages$BuildingBlock": "buildingblock", + "Forms$BuildingBlock": "buildingblock", + "Forms$Layout": "layout", + "Enumerations$Enumeration": "enumeration", + "Constants$Constant": "constant", + "JavaActions$JavaAction": "javaaction", + "JsonStructures$JsonStructure": "jsonstructure", + "ImportMappings$ImportMapping": "importmapping", + "ExportMappings$ExportMapping": "exportmapping", + "Images$ImageCollection": "imagecollection", + "Workflows$Workflow": "workflow", + "Queues$Queue": "queue", + "ScheduledEvents$ScheduledEvent": "scheduledevent", } // resolveDescribeType auto-detects the `describe` type for a qualified document diff --git a/cmd/mxcli/project_tree.go b/cmd/mxcli/project_tree.go index d617da3e7..a2f99eb6a 100644 --- a/cmd/mxcli/project_tree.go +++ b/cmd/mxcli/project_tree.go @@ -241,6 +241,17 @@ func buildProjectTree(projectPath string) ([]*TreeNode, error) { md.documents = append(md.documents, treeElement{Name: se.Name, ContainerID: se.ContainerID, Type: "scheduledevent"}) } + // Collect task queues + queues, _ := reader.ListQueues() + for _, q := range queues { + modID := h.FindModuleID(q.ContainerID) + md, ok := modData[modID] + if !ok { + continue + } + md.documents = append(md.documents, treeElement{Name: q.Name, ContainerID: q.ContainerID, Type: "queue"}) + } + // Collect JavaScript actions jsas, _ := reader.ListJavaScriptActions() for _, jsa := range jsas { diff --git a/cmd/mxcli/tui/icons.go b/cmd/mxcli/tui/icons.go index 6e0b0dd0c..4106d39ee 100644 --- a/cmd/mxcli/tui/icons.go +++ b/cmd/mxcli/tui/icons.go @@ -35,6 +35,7 @@ var typeIconMap = map[string]string{ // Constants & events "constant": "π", "scheduledevent": "⏰", + "queue": "🧵", // Actions "javaaction": "☕", diff --git a/mdl/executor/cmd_structure.go b/mdl/executor/cmd_structure.go index 2c177c205..09fba87db 100644 --- a/mdl/executor/cmd_structure.go +++ b/mdl/executor/cmd_structure.go @@ -9,6 +9,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/domainmodel" "github.com/mendixlabs/mxcli/sdk/javaactions" @@ -77,12 +78,13 @@ func structureDepth1JSON(ctx *ExecContext, modules []structureModule) error { beServiceCounts := queryCountByModule(ctx, "business_event_services") constantCounts := countByModuleFromBackend(ctx, "constants") scheduledEventCounts := countByModuleFromBackend(ctx, "scheduled_events") + queueCounts := countByModuleFromBackend(ctx, "queues") tr := &TableResult{ Columns: []string{ "Module", "Entities", "Enumerations", "Microflows", "Nanoflows", "Workflows", "Pages", "Snippets", "JavaActions", "Constants", - "ScheduledEvents", "ODataClients", "ODataServices", "BusinessEventServices", + "ScheduledEvents", "Queues", "ODataClients", "ODataServices", "BusinessEventServices", }, } for _, m := range modules { @@ -98,6 +100,7 @@ func structureDepth1JSON(ctx *ExecContext, modules []structureModule) error { jaCounts[m.Name], constantCounts[m.Name], scheduledEventCounts[m.Name], + queueCounts[m.Name], odataClientCounts[m.Name], odataServiceCounts[m.Name], beServiceCounts[m.Name], @@ -198,6 +201,7 @@ func structureDepth1(ctx *ExecContext, modules []structureModule) error { // Get constants and scheduled events from backend (no catalog tables) constantCounts := countByModuleFromBackend(ctx, "constants") scheduledEventCounts := countByModuleFromBackend(ctx, "scheduled_events") + queueCounts := countByModuleFromBackend(ctx, "queues") // Calculate name column width for alignment nameWidth := 0 @@ -240,6 +244,9 @@ func structureDepth1(ctx *ExecContext, modules []structureModule) error { if c := scheduledEventCounts[m.Name]; c > 0 { parts = append(parts, pluralize(c, "scheduled event", "scheduled events")) } + if c := queueCounts[m.Name]; c > 0 { + parts = append(parts, pluralize(c, "queue", "queues")) + } if c := odataClientCounts[m.Name]; c > 0 { parts = append(parts, pluralize(c, "odata client", "odata clients")) } @@ -297,6 +304,14 @@ func countByModuleFromBackend(ctx *ExecContext, kind string) map[string]int { counts[modName]++ } } + case "queues": + if queues, err := ctx.Backend.ListQueues(); err == nil { + for _, q := range queues { + modID := h.FindModuleID(q.ContainerID) + modName := h.GetModuleName(modID) + counts[modName]++ + } + } } return counts } @@ -374,6 +389,14 @@ func structureDepth2(ctx *ExecContext, modules []structureModule) error { eventsByModule[modName] = append(eventsByModule[modName], ev) } + allQueues, _ := ctx.Backend.ListQueues() + queuesByModule := make(map[string][]*types.Queue) + for _, q := range allQueues { + modID := h.FindModuleID(q.ContainerID) + modName := h.GetModuleName(modID) + queuesByModule[modName] = append(queuesByModule[modName], q) + } + // Load java actions for parameter types allJavaActions, _ := ctx.Backend.ListJavaActionsFull() jaByModule := make(map[string][]*javaactions.JavaAction) @@ -457,6 +480,14 @@ func structureDepth2(ctx *ExecContext, modules []structureModule) error { } } + // Task Queues + if queues, ok := queuesByModule[m.Name]; ok { + sort.Slice(queues, func(i, j int) bool { return queues[i].Name < queues[j].Name }) + for _, q := range queues { + fmt.Fprintf(ctx.Output, " Queue %s.%s\n", m.Name, q.Name) + } + } + // OData Clients structureODataClients(ctx, m.Name) @@ -529,6 +560,14 @@ func structureDepth3(ctx *ExecContext, modules []structureModule) error { eventsByModule[modName] = append(eventsByModule[modName], ev) } + allQueues, _ := ctx.Backend.ListQueues() + queuesByModule := make(map[string][]*types.Queue) + for _, q := range allQueues { + modID := h.FindModuleID(q.ContainerID) + modName := h.GetModuleName(modID) + queuesByModule[modName] = append(queuesByModule[modName], q) + } + allJavaActions, _ := ctx.Backend.ListJavaActionsFull() jaByModule := make(map[string][]*javaactions.JavaAction) for _, ja := range allJavaActions { @@ -615,6 +654,14 @@ func structureDepth3(ctx *ExecContext, modules []structureModule) error { } } + // Task Queues + if queues, ok := queuesByModule[m.Name]; ok { + sort.Slice(queues, func(i, j int) bool { return queues[i].Name < queues[j].Name }) + for _, q := range queues { + fmt.Fprintf(ctx.Output, " Queue %s.%s\n", m.Name, q.Name) + } + } + // OData structureODataClients(ctx, m.Name) structureODataServices(ctx, m.Name) diff --git a/vscode-mdl/package.json b/vscode-mdl/package.json index 73a3030e4..7499ed69e 100644 --- a/vscode-mdl/package.json +++ b/vscode-mdl/package.json @@ -152,7 +152,7 @@ }, { "command": "mendix.openElement", - "when": "view == mendixProjectTree && viewItem =~ /^(entity|microflow|nanoflow|page|snippet|enumeration|association|workflow|constant|layout|javaaction|javascriptaction|scheduledevent|buildingblock|pagetemplate|imagecollection|jsonstructure|importmapping|exportmapping|restclient|businesseventservice|databaseconnection|publishedrestservice|odataclient|odataservice|modulerole|userrole|projectsecurity|demouser|agent|aimodel|knowledgebase|consumedmcpservice|datatransformer)$/", + "when": "view == mendixProjectTree && viewItem =~ /^(entity|microflow|nanoflow|page|snippet|enumeration|association|workflow|constant|layout|javaaction|javascriptaction|scheduledevent|queue|buildingblock|pagetemplate|imagecollection|jsonstructure|importmapping|exportmapping|restclient|businesseventservice|databaseconnection|publishedrestservice|odataclient|odataservice|modulerole|userrole|projectsecurity|demouser|agent|aimodel|knowledgebase|consumedmcpservice|datatransformer)$/", "group": "mendix@6" }, { diff --git a/vscode-mdl/src/extension.ts b/vscode-mdl/src/extension.ts index 79f56824a..289638bcf 100644 --- a/vscode-mdl/src/extension.ts +++ b/vscode-mdl/src/extension.ts @@ -118,7 +118,7 @@ export function activate(context: vscode.ExtensionContext) { } // Try the given type first, then fallback to other common types - const fallbackTypes = ['entity', 'microflow', 'nanoflow', 'page', 'enumeration', 'snippet', 'constant', 'javaaction', 'javascriptaction', 'scheduledevent', 'buildingblock', 'pagetemplate', 'imagecollection', 'businesseventservice', 'databaseconnection', 'publishedrestservice', 'workflow', 'layout', 'importmapping', 'exportmapping', 'restclient', 'jsonstructure', 'agent', 'aimodel', 'knowledgebase', 'consumedmcpservice', 'datatransformer']; + const fallbackTypes = ['entity', 'microflow', 'nanoflow', 'page', 'enumeration', 'snippet', 'constant', 'javaaction', 'javascriptaction', 'scheduledevent', 'queue', 'buildingblock', 'pagetemplate', 'imagecollection', 'businesseventservice', 'databaseconnection', 'publishedrestservice', 'workflow', 'layout', 'importmapping', 'exportmapping', 'restclient', 'jsonstructure', 'agent', 'aimodel', 'knowledgebase', 'consumedmcpservice', 'datatransformer']; const typesToTry = [type, ...fallbackTypes.filter(t => t !== type)]; for (const tryType of typesToTry) { diff --git a/vscode-mdl/src/projectTreeProvider.ts b/vscode-mdl/src/projectTreeProvider.ts index 57c5d9da7..317b0e33e 100644 --- a/vscode-mdl/src/projectTreeProvider.ts +++ b/vscode-mdl/src/projectTreeProvider.ts @@ -253,6 +253,8 @@ export class MendixProjectTreeProvider implements vscode.TreeDataProvider = { 'JavaAction': 'javaaction', 'Constant': 'constant', 'ScheduledEvent': 'scheduledevent', + 'Queue': 'queue', 'ODataClient': 'odataclient', 'ODataService': 'odataservice', }; From 68df51dfc93f65513de0f581c78a903b2b2cec6b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 05:36:47 +0000 Subject: [PATCH 28/35] Phase 2 slice 5: marketplace update, with --save-edits and --force MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The command. It refuses when the module has local edits, --save-edits writes them out as re-executable MDL first, and --force proceeds -- so a destructive update becomes park, replace, replay. Measured end to end on real packages, Administration 4.3.2 → 4.5.0 in a blank 11.12.1 app with one locally added attribute: 28 units copied, 9 element identities preserved, 2 role grants restored Removed in 4.5.0 (1): Account/MyLocalEdit Account's GUID after the update is ea496eb1df91aa4caed86ba4c4e133c5 -- byte-identical to the mendixsystem$entity.id read from the live database in §8, so the table survives. The role grants are back at 3 and 4, the counts before the drop. Replaying the parked file restores MyLocalEdit and its access rules. Running it for real found a defect no fixture would have. The reference project is built with `mx module-import`, which stamps a module with the author's INTERNAL version rather than the marketplace release: after updating to 4.5.0 the project recorded AppStoreVersion 2.0.1 and an unrelated GUID. That is not cosmetic -- diff and update identify a module by its AppStoreGuid, so the next update could not have found it at all. StampMarketplaceVersion fixes it, and a second update now correctly reports "already at 4.5.0; nothing to do". Also measured: the update leaves 11 CE0463 widget errors, which `mx update-widgets` clears to 0. That is the known resync after any headless module install (mxcli-formula1 FINDINGS §53), not a fault here, so the command says so rather than leaving the user to diagnose it. Not done: this does not roll back. A failure partway leaves the module removed, and the command says to work on a copy. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/cmd_marketplace_update.go | 249 ++++++++++++++++++++++++++++ cmd/mxcli/marketplace/update.go | 242 +++++++++++++++++++++++++++ docs-site/src/guides/marketplace.md | 42 +++++ 3 files changed, 533 insertions(+) create mode 100644 cmd/mxcli/cmd_marketplace_update.go create mode 100644 cmd/mxcli/marketplace/update.go diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go new file mode 100644 index 000000000..59c5ccf7f --- /dev/null +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -0,0 +1,249 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "os" + + "github.com/mendixlabs/mxcli/cmd/mxcli/marketplace" + "github.com/mendixlabs/mxcli/internal/auth" + "github.com/spf13/cobra" +) + +var marketplaceUpdateCmd = &cobra.Command{ + Use: "update -p --to ", + Short: "Replace an installed marketplace module with a newer version", + Long: `Replace an installed marketplace module with another published version, +preserving the two things a plain replace destroys. + +The first is element identity. The runtime keys entities and their attributes on +the model's GUID, so a module whose documents are replaced without carrying the +old GUIDs is a different module as far as the database is concerned, and its +tables are dropped on the next deploy. Studio Pro's own update transplants them; +so does this. + +The second is access. A user role's grant of a module role lives in the project's +security document rather than in the module, so removing the module takes the +grants with it and putting the module back does not return them. + +Local edits are NOT preserved, and by default the update refuses when it finds +any. Use --save-edits to write them out as re-executable MDL first, and --force +to proceed. Studio Pro discards them without asking; this at least tells you what +they were. + +This does not roll back. Work on a copy or have the project in version control: +if a step fails partway, the module has already been removed.`, + Example: ` # What would this update touch, and have I edited any of it? + mxcli marketplace diff 23513 -p app.mpr --to 4.5.0 + + # Park local edits, then update over them + mxcli marketplace update 23513 -p app.mpr --to 4.5.0 --save-edits ./local-edits + mxcli marketplace update 23513 -p app.mpr --to 4.5.0 --force + mxcli exec ./local-edits/entity-Account.mdl -p app.mpr`, + Args: cobra.ExactArgs(1), + RunE: runMarketplaceUpdate, +} + +func runMarketplaceUpdate(cmd *cobra.Command, args []string) error { + contentID, err := parseContentID(args[0]) + if err != nil { + return err + } + mprPath, _ := cmd.Flags().GetString("project") + if mprPath == "" { + return fmt.Errorf("-p/--project is required") + } + target, _ := cmd.Flags().GetString("to") + if target == "" { + return fmt.Errorf("--to is required: an update needs a version to move to") + } + moduleName, _ := cmd.Flags().GetString("module") + saveEdits, _ := cmd.Flags().GetString("save-edits") + force, _ := cmd.Flags().GetBool("force") + + ctx := cmd.Context() + out := cmd.OutOrStdout() + client, err := newMarketplaceClient(ctx, cmd) + if err != nil { + return err + } + versions, err := client.Versions(ctx, contentID) + if err != nil { + return err + } + + work, err := os.MkdirTemp("", "mxupdate") + if err != nil { + return err + } + defer os.RemoveAll(work) + + var installedVersionID string + if moduleName == "" { + moduleName, installedVersionID, err = marketplace.ModuleForVersionIDs(mprPath, versionIDs(versions.Items)) + if err != nil { + return fmt.Errorf("%w\nhint: pass --module with the module's name in the project", err) + } + } + installedVersion, mendixVersion, err := marketplace.InstalledModule(mprPath, moduleName) + if err != nil { + return err + } + if installedVersion == target { + fmt.Fprintf(out, "%s is already at %s; nothing to do.\n", moduleName, target) + return nil + } + + // Has anyone edited this module? Answering needs the version it was installed + // from, built as a reference exactly as `marketplace diff` does. + base, err := pickVersion(versions.Items, installedVersionID, installedVersion) + if err != nil { + return err + } + baseRef, basePkgModule, err := referenceFor(ctx, client, base, mendixVersion, work, "base") + if err != nil { + return err + } + installed, err := marketplace.SnapshotModule(mprPath, moduleName, newBackendFactory()) + if err != nil { + return fmt.Errorf("read %s from the project: %w", moduleName, err) + } + published, err := marketplace.SnapshotModule(baseRef, basePkgModule, newBackendFactory()) + if err != nil { + return fmt.Errorf("read %s from its published package: %w", basePkgModule, err) + } + drift := marketplace.Compare(installed, published) + + if saveEdits != "" { + written, unsaved, serr := marketplace.SaveEdits(saveEdits, drift) + if serr != nil { + return serr + } + reportSavedEdits(out, saveEdits, written, unsaved) + } + + if err := gateOnLocalEdits(out, drift, force, saveEdits); err != nil { + return err + } + + // Build the version being moved to, and replace. + targetVersion, err := pickVersion(versions.Items, "", target) + if err != nil { + return err + } + targetRef, _, err := referenceFor(ctx, client, targetVersion, mendixVersion, work, "target") + if err != nil { + return err + } + + fmt.Fprintf(out, "\nUpdating %s %s → %s...\n", moduleName, installedVersion, target) + res, err := marketplace.PerformUpdate(mprPath, targetRef, moduleName, installedVersion, target, + targetVersion.VersionID, newBackendFactory()) + if err != nil { + return fmt.Errorf("%w\n\nThe project may be mid-update — %s could be missing. Restore from version control", + err, moduleName) + } + reportUpdate(out, res) + return nil +} + +// gateOnLocalEdits refuses an update that would destroy local work, unless the +// user has said to proceed. The refusal names the elements rather than just +// their count: "3 elements were modified" is not enough to decide with. +func gateOnLocalEdits(out io.Writer, drift *marketplace.Report, force bool, saveEdits string) error { + result := marketplace.NewDiffResult("", "", "", drift) + edited := append(append([]string{}, result.Modified...), result.OnlyInstalled...) + + if len(edited) == 0 { + if !result.Verified { + // Unknown elements are not proof of safety. Say so, and let --force + // carry the decision, rather than reporting a clean bill of health. + if !force { + return fmt.Errorf("%d element(s) could not be read, so it cannot be shown that the module is unedited.\n"+ + "Re-run with --force to update anyway", len(result.Unknown)) + } + fmt.Fprintf(out, "Note: %d element(s) could not be read; proceeding under --force.\n", len(result.Unknown)) + } + return nil + } + + if !force { + msg := fmt.Sprintf("refusing to update: %d element(s) have been changed locally and the update would discard them:\n", len(edited)) + for _, e := range edited { + msg += " " + e + "\n" + } + if saveEdits == "" { + msg += "\n - Save them first: --save-edits \n - Then update: --force" + } else { + msg += "\n Saved above. Re-run with --force to proceed." + } + return fmt.Errorf("%s", msg) + } + + fmt.Fprintf(out, "Proceeding under --force; %d locally changed element(s) will be replaced:\n", len(edited)) + for _, e := range edited { + fmt.Fprintf(out, " %s\n", e) + } + return nil +} + +func reportSavedEdits(out io.Writer, dir string, written, unsaved []string) { + if len(written) > 0 { + fmt.Fprintf(out, "Saved %d locally changed element(s) to %s:\n", len(written), dir) + for _, f := range written { + fmt.Fprintf(out, " %s\n", f) + } + fmt.Fprintln(out, " Replay after the update with 'mxcli exec -p '.") + fmt.Fprintln(out, " These are resulting states, not diffs: an edit that REMOVED something") + fmt.Fprintln(out, " will not be restored by replaying them.") + } + if len(unsaved) > 0 { + fmt.Fprintf(out, "\n Could not be saved (%d) — these edits are not recoverable this way:\n", len(unsaved)) + for _, u := range unsaved { + fmt.Fprintf(out, " %s\n", u) + } + } +} + +func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { + fmt.Fprintf(out, "\n%s updated %s → %s\n", r.Module, r.FromVersion, r.ToVersion) + fmt.Fprintf(out, " %d units copied, %d element identities preserved, %d role grant(s) restored.\n", + r.UnitsCopied, r.IdentitiesKept, r.GrantsRestored) + + if len(r.IdentitiesLost) > 0 { + fmt.Fprintf(out, "\n Removed in %s (%d) — their database columns or tables will go on the next deploy:\n", + r.ToVersion, len(r.IdentitiesLost)) + for _, e := range r.IdentitiesLost { + fmt.Fprintf(out, " %s\n", e) + } + } + if len(r.GrantsDropped) > 0 { + fmt.Fprintf(out, "\n Role grants that could not be restored (%d) — those users lose that access:\n", + len(r.GrantsDropped)) + for _, g := range r.GrantsDropped { + fmt.Fprintf(out, " %s\n", g) + } + } + // A newer module's pages reference widget definitions the project has not + // resynced, so `mx check` reports CE0463 until it is told to. Measured on + // Administration 4.3.2 → 4.5.0: 11 CE0463 errors, and 0 after update-widgets. + // Saying so here is the difference between a two-command fix and a day in + // diagnose-ce0463.md, which is where that error normally leads. + fmt.Fprintln(out, "\n Next: resync widget definitions, or 'mx check' will report CE0463 on the") + fmt.Fprintln(out, " new version's pages (this is expected after any headless module install):") + fmt.Fprintln(out, " mx update-widgets ") + fmt.Fprintln(out, "\n Then review with 'mxcli diff-local' and validate with 'mxcli docker check'.") +} + +func init() { + marketplaceUpdateCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") + marketplaceUpdateCmd.Flags().String("to", "", "version to update to (required)") + marketplaceUpdateCmd.Flags().String("module", "", "module name in the project, when it cannot be identified automatically") + marketplaceUpdateCmd.Flags().String("save-edits", "", "write locally changed elements to this directory as re-executable MDL") + marketplaceUpdateCmd.Flags().Bool("force", false, "update even though local edits will be discarded") + marketplaceUpdateCmd.Flags().String("profile", auth.ProfileDefault, "credential profile") + + marketplaceCmd.AddCommand(marketplaceUpdateCmd) +} diff --git a/cmd/mxcli/marketplace/update.go b/cmd/mxcli/marketplace/update.go new file mode 100644 index 000000000..40e4c72f0 --- /dev/null +++ b/cmd/mxcli/marketplace/update.go @@ -0,0 +1,242 @@ +// SPDX-License-Identifier: Apache-2.0 + +package marketplace + +import ( + "fmt" + "os" + "path/filepath" + "regexp" + "strings" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/mdl/backend" + "go.mongodb.org/mongo-driver/bson" +) + +// SaveEdits writes each locally modified element's current definition to dir as +// re-executable MDL, and returns the files written. +// +// This is what makes `--force` recoverable: the update discards local edits, so +// parking them first turns "your work is gone" into "replay these and review the +// result". The text is the same DESCRIBE output the comparison ran on, which +// re-executes — an entity comes back as `create or modify persistent entity ...` +// with its access rules, and `mxcli check` accepts it. +// +// Two limits are inherent and are the caller's to communicate: +// +// - `create or modify` MERGES. An edit that *removed* something — a deleted +// attribute, a tightened grant — is not in this text, because the text is +// the resulting state rather than a diff. Additions and changes replay; +// removals do not. +// - An element that could not be described has nothing to save. Those are +// reported rather than skipped, because "no file written" and "nothing was +// changed here" must not look the same. +func SaveEdits(dir string, rep *Report) (written []string, unsaved []string, err error) { + var wanted []Finding + for _, f := range rep.Findings { + switch f.Verdict { + case Modified, OnlyInstalled: + wanted = append(wanted, f) + case Unknown: + unsaved = append(unsaved, fmt.Sprintf("%s (%s)", f.Key, f.Reason)) + } + } + if len(wanted) == 0 { + return nil, unsaved, nil + } + if err := os.MkdirAll(dir, 0o755); err != nil { + return nil, unsaved, fmt.Errorf("create %s: %w", dir, err) + } + + for _, f := range wanted { + body := replayable(f.InstalledMDL) + if strings.TrimSpace(body) == "" { + unsaved = append(unsaved, f.Key.String()+" (no re-executable definition)") + continue + } + name := filepath.Join(dir, safeFileName(f.Key)+".mdl") + header := fmt.Sprintf("-- %s, as it stands in your project before the update.\n"+ + "-- Replay with: mxcli exec %s -p \n"+ + "-- Note: this is the element's resulting state, not a diff — an edit that\n"+ + "-- REMOVED something will not be restored by replaying it.\n\n", f.Key, name) + if err := os.WriteFile(name, []byte(header+body+"\n"), 0o644); err != nil { + return written, unsaved, fmt.Errorf("write %s: %w", name, err) + } + written = append(written, name) + } + return written, unsaved, nil +} + +// trailingSeparator matches the lone "/" the executor prints after DESCRIBE +// output. It is harmless in a comparison, where both sides carry it, but it is +// noise in a file a user is meant to replay. +var trailingSeparator = regexp.MustCompile(`(?m)^\s*/\s*$`) + +func replayable(mdl string) string { + return strings.TrimSpace(trailingSeparator.ReplaceAllString(mdl, "")) +} + +// safeFileName turns an element key into something usable on disk. +func safeFileName(k ElementKey) string { + s := strings.ToLower(k.Type) + "-" + k.Name + return strings.Map(func(r rune) rune { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_': + return r + default: + return '-' + } + }, s) +} + +// UpdateResult is what an update did. +type UpdateResult struct { + Module string + FromVersion string + ToVersion string + UnitsCopied int + IdentitiesKept int + IdentitiesLost []string + GrantsRestored int + GrantsDropped []string + ForcedOverEdits []string +} + +// PerformUpdate replaces an installed module with the copy in referenceMpr, +// preserving the two things that do not survive a plain replace: the `GUID`s the +// database keys on (§8) and the user-role grants of the module's roles. +// +// The order is not arbitrary. Identities and grants are read *before* the module +// is removed, because removing it destroys both. The module is then dropped +// through mxcli's own DROP MODULE so project references are unpicked cleanly, +// the replacement is copied in with mxcli's writer (never `mx module-import`, +// which would rewrite an MPR v2 project as v1), and only then are identity and +// access put back. +// +// A failure partway leaves the project in a broken state. The caller must work +// on a copy, or hold a backup: this does not roll back. +func PerformUpdate(mprPath, referenceMpr, moduleName, fromVersion, toVersion, toVersionID string, + newBackend func() backend.FullBackend) (*UpdateResult, error) { + + ids, err := CaptureIdentities(mprPath, moduleName) + if err != nil { + return nil, fmt.Errorf("record the identities the database keys on: %w", err) + } + grants, err := CaptureRoleGrants(mprPath, moduleName) + if err != nil { + return nil, fmt.Errorf("record role grants: %w", err) + } + + if err := execStatements(mprPath, "drop module "+moduleName+";", newBackend); err != nil { + return nil, fmt.Errorf("remove the installed module: %w", err) + } + + copied, err := TransplantModule(referenceMpr, mprPath, moduleName) + if err != nil { + return nil, fmt.Errorf("copy in the new version (the project no longer has the module): %w", err) + } + + applied, missing, err := ApplyIdentities(mprPath, moduleName, ids) + if err != nil { + return nil, fmt.Errorf("restore identities (the project's data mapping is at risk): %w", err) + } + restored, dropped, err := RestoreRoleGrants(mprPath, moduleName, grants, newBackend) + if err != nil { + return nil, fmt.Errorf("restore role grants: %w", err) + } + if err := StampMarketplaceVersion(mprPath, moduleName, toVersion, toVersionID); err != nil { + return nil, fmt.Errorf("record the installed version: %w", err) + } + + return &UpdateResult{ + Module: moduleName, + FromVersion: fromVersion, + ToVersion: toVersion, + UnitsCopied: copied, + IdentitiesKept: applied, + IdentitiesLost: missing, + GrantsRestored: restored, + GrantsDropped: dropped, + }, nil +} + +// StampMarketplaceVersion records which published version a module now is. +// +// Necessary because the reference project is built with `mx module-import`, +// which stamps the module with the *author's internal* version rather than the +// marketplace release number: importing Administration 4.5.0 leaves +// AppStoreVersion "2.0.1" and an unrelated AppStoreGuid. Transplanting carries +// that stamp across, so without this the project claims a version it does not +// have. +// +// It is not cosmetic. `marketplace diff` and `marketplace update` identify a +// module by its AppStoreGuid — the marketplace *version* UUID — so a wrong stamp +// makes the module unrecognisable to the next update, which then reports that no +// module in the project came from this content. +func StampMarketplaceVersion(mprPath, moduleName, versionNumber, versionID string) error { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return fmt.Errorf("open %s: %w", mprPath, err) + } + units, err := reader.ListRawUnitsByType("Projects$ModuleImpl") + if err != nil { + reader.Close() + return fmt.Errorf("read module documents: %w", err) + } + + var unitID string + var contents []byte + for _, u := range units { + var doc bson.D + if bson.Unmarshal(u.Contents, &doc) != nil { + continue + } + name, _, _ := nameAndGUID(doc) + if !strings.EqualFold(name, moduleName) { + continue + } + setStringField(doc, "AppStoreVersion", versionNumber) + setStringField(doc, "AppStoreGuid", versionID) + setBoolField(doc, "FromAppStore", true) + enc, merr := bson.Marshal(doc) + if merr != nil { + reader.Close() + return fmt.Errorf("re-encode module document: %w", merr) + } + unitID, contents = string(u.ID), enc + break + } + reader.Close() + + if unitID == "" { + return fmt.Errorf("module %q not found when stamping its version", moduleName) + } + writer, err := modelsdk.OpenForWriting(mprPath) + if err != nil { + return fmt.Errorf("open %s for writing: %w", mprPath, err) + } + defer writer.Close() + return writer.UpdateRawUnit(unitID, contents) +} + +// setStringField assigns an existing key, and only an existing key. Inventing a +// property Mendix does not store for this version is the failure mode ADR-0005 +// warns about: mxbuild tolerates it and Studio Pro refuses to open the document. +func setStringField(doc bson.D, key, value string) { + for i, e := range doc { + if e.Key == key { + doc[i].Value = value + return + } + } +} + +func setBoolField(doc bson.D, key string, value bool) { + for i, e := range doc { + if e.Key == key { + doc[i].Value = value + return + } + } +} diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index 4f6c43f97..1da35dd5e 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -108,6 +108,48 @@ Two reasons make automatic in-place module updates unsafe: Studio Pro's Marketplace **Update** performs an ID-preserving merge that the `mx` CLI does not expose, so module updates are left to Studio Pro for now. +## Updating a module (`marketplace update`) + +`update` replaces an installed module with another published version, preserving the two things a plain replace destroys. + +```bash +# Refuses if you have edited the module, naming what it would discard +mxcli marketplace update 23513 -p app.mpr --to 4.5.0 + +# Park those edits as re-executable MDL, then update over them +mxcli marketplace update 23513 -p app.mpr --to 4.5.0 --save-edits ./local-edits +mxcli marketplace update 23513 -p app.mpr --to 4.5.0 --force +mxcli exec ./local-edits/entity-Account.mdl -p app.mpr +``` + +```text +Administration updated 4.3.2 → 4.5.0 + 28 units copied, 9 element identities preserved, 2 role grant(s) restored. + + Removed in 4.5.0 (1) — their database columns or tables will go on the next deploy: + Account/MyLocalEdit +``` + +### What it preserves, and why + +- **Element identity.** The runtime keys entities and their attributes on the model's `GUID`, so a module whose documents are replaced without carrying the old GUIDs is a *different* module to the database and its tables are dropped on the next deploy. Studio Pro transplants them; so does this. +- **Access.** A user role's grant of a module role lives in the project's security document, not the module, so removing the module takes the grants with it and putting it back does not return them. + +It does not use `mx module-import`, which would rewrite an MPR v2 project as v1 and refuses theme modules. Units are copied with mxcli's own writer, so the project keeps its format. + +### Local edits + +Local edits are **not** preserved. `update` refuses when it finds any, `--save-edits` writes them out first, and `--force` proceeds. Two limits on the saved files: + +- They are the element's **resulting state, not a diff**, so replaying restores additions and changes but not removals. +- An element that could not be described has nothing to save, and is reported rather than skipped. + +### Afterwards + +Run `mx update-widgets `. A newer module's pages reference widget definitions the project has not resynced, so `mx check` reports CE0463 until told to — measured on Administration 4.3.2 → 4.5.0: 11 errors before, 0 after. This is expected after any headless module install, not a fault in the update. + +`update` does **not** roll back. Work on a copy or have the project in version control. + ## Has this module been edited? (`marketplace diff`) Studio Pro's Marketplace **Update** replaces the module and discards local edits without asking. `marketplace diff` answers the question that decides whether that is safe: From f47d8bfa15ec8377e225629c31c0004088f6001f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 05:49:39 +0000 Subject: [PATCH 29/35] Install a module's widget binaries during an update Testing slice 5 against DataWidgets 3.5.0 -> 3.11.3 found a gap that Administration could not expose, because Administration ships no widgets. The update moved only the model. A widget module's .mpk carries its widget binaries under widgets/, and copying units out of a reference project never touches them -- so the project reported 3.11.3, its pages referenced 3.11.3's widget definitions, and all ten binaries on disk were still 3.5.0 (Datagrid.mpk project=216193 vs package=166933). No build error names that. InstallPackageWidgets takes them from the package rather than from the reference project, whose widgets/ also holds the blank template's widgets -- copying those would overwrite widgets the update has nothing to do with. Zero-size directory entries are skipped: the 3.11.3 package has ten widgets/ entries and nine real files. Re-run: 9 of 9 match 3.11.3. Also corrects the post-update guidance. `mx update-widgets` clears the CE0463 resync as before, but DataWidgets 3.11.3 then leaves 29 x CE6083 -- its widgets want design properties an older Atlas does not define. `mx rename-design-properties` renames 0 and changes nothing, so this is a cross-module dependency rather than a resync, and the command now says so instead of implying two commands will clear everything. Administration 4.3.2 -> 4.5.0 still reaches 0 errors. DataWidgets does not, and that is a property of the content rather than of the update. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/cmd_marketplace_update.go | 17 ++++- cmd/mxcli/marketplace/update.go | 97 +++++++++++++++++++++++------ 3 files changed, 95 insertions(+), 20 deletions(-) diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 62ea5bafa..2abc04436 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -468,3 +468,4 @@ extracting `OffsetExpression`/`LimitExpression`. | After `mxcli marketplace install -p app.mpr`, the project's `mprcontents/` folder is gone, the `.mpr` has grown from tens of KB to tens of MB, and `mxcli diff-local` no longer works. Git shows hundreds of deleted `.mxunit` files and one enormous binary | `mx module-import` rewrites an MPR **v2** project as **v1** — silently, and one-way. Measured on a blank 11.12.1 app: 69,632-byte `.mpr` + 341 `.mxunit` → 14,295,040-byte `.mpr` + 0, with the `_Transaction` table dropped. Reproduced independently on 11.13.0, so it is not version-specific. mxcli's install command shells straight out to it against the **user's** project | `cmd/mxcli/cmd_marketplace_install.go` (`checkStorageFormatPreserved`, `isMPRv2`, `reportFormatChange`, `--allow-format-change`) | **Refuse, do not warn** — the conversion cannot be undone (`mx convert` targets Mendix versions, not storage formats) and it destroys the per-document layout `diff-local`, `.mxunit` merges and ADR-0008's observable idempotence all rest on. **Detect the format the way the readers do** (presence of `mprcontents/`), not with a second definition. **Check the guard's negative case**: a test that only asserts "v2 is refused" also passes a guard that refuses everything — assert a v1 project still imports. **Do not guard every `module-import` call site**: `marketplace diff` runs one against a throwaway reference project, where the format is irrelevant (and the resulting v1 reference still diffs exactly against a v2 project, because DESCRIBE output does not depend on storage). Related, same family: `mx module-import` also refuses any module whose `Projects$ModuleImpl → IsThemeModule` is `true` (exit 112), which is the sole gate — flipping that one boolean makes an otherwise identical package import cleanly. Tests `cmd/mxcli/cmd_marketplace_install_test.go`. Reported in mxcli-formula1 FINDINGS §53 | | A .mpr extracted from a `.mpk` (or copied anywhere beside an existing project) reads with the wrong unit contents: edits appear to succeed, `ListRawUnitsByType` returns plausible documents, and a write lands nowhere the file will carry. Downstream the operation fails exactly as if nothing had been attempted | The MPR format test is **adjacency, not content**: an `.mpr` is treated as v2 when an `mprcontents/` directory sits next to it (`sdk/mpr/reader.go` `OpenWithOptions`). A package's v1 `project.mpr` unpacked into a work directory that already holds a scratch v2 project is therefore read as v2, and every unit resolves against the *other* project's `.mxunit` files | `cmd/mxcli/marketplace/theme.go` (`makeImportable` unpacks into its own `os.MkdirTemp` subdirectory) | **Unpack any foreign .mpr into a directory of its own** — never beside a project, even a throwaway one. **The failure is silent in both directions**: the read returns real-looking documents (from the wrong model) and the write reports success, so nothing surfaces until a later step refuses. The tell is a fix that provably works in isolation (`mx module-import` accepted the hand-built package) but does nothing in situ — that gap is environment, not logic. Test `TestMakeImportable_UnpacksAwayFromTheScratchProject` asserts the unpack does not land in the work-directory root | | `show features` lists nothing for agents or MCP, so `create agent` / `create model` on a pre-11.9 project runs without complaint. Reported alongside it: "`CREATE MODEL` can only author one provider — the writer assigns `MxCloudGenAI` unconditionally" | Two different things, and only the first was real. **The version gap**: the `agent_documents` area did not exist in `sdk/versions/mendix-11.yaml`, so `checkFeature()` had nothing to consult. **The provider claim was wrong**: the writer's assignment is `if m.Provider == ""` — a default, not an override. `Provider: OpenAI` parses, writes and round-trips through `describe model`; so does `Provider: TotallyMadeUp` | `sdk/versions/mendix-11.yaml` (new `agent_documents` area), `mdl/executor/cmd_agenteditor_models.go` + `cmd_agenteditor_write.go` (four `checkFeature` gates), `.claude/skills/mendix/agents.md` + `version-awareness.md` | **Read the assignment's guard before believing "hardcoded"** — `if x == "" { x = default }` and `x = default` are one character apart in a grep and opposite in meaning; the round-trip test settles it in a minute. **Agent doctypes are the one version gate with no downstream safety net**: the documents are custom blobs, mxbuild contains no agent-editor strings at all, so an ungated project builds green and fails only when Studio Pro opens it. **Do not invent the provider allowlist** — the enum lives in a Studio Pro *extension*, not in `generated/metamodel` and not in mxbuild, so a guessed list would reject values Mendix accepts; document that nothing validates it instead. Test `TestAgentDocumentsAreGated`. Reported in mxcli-formula1 FINDINGS §53 | +| A marketplace module update leaves the app running the OLD widget code: `show modules` reports the new version, pages reference new widget definitions, and the binaries in `widgets/` are unchanged. Symptoms are whatever the version gap causes at render time, with no build error naming the cause | The update moved only the **model**. A widget module's `.mpk` ships its widget binaries under `widgets/` inside the package, and copying units out of a reference project never touches them. Measured on DataWidgets 3.5.0 → 3.11.3: model updated cleanly, all 10 widget binaries still 3.5.0 (`Datagrid.mpk` project=216193 vs package=166933) | `cmd/mxcli/marketplace/update.go` (`InstallPackageWidgets`, called from `PerformUpdate`) | **Take the widget files from the `.mpk`, not from the reference project** — the reference is a blank app plus the module, so its `widgets/` also holds the template's widgets and copying those overwrites widgets the update has nothing to do with. **Skip zero-size directory entries** in the zip or you write a stray file (the 3.11.3 package has 10 `widgets/` entries and 9 real files). **A module with no widgets cannot expose this**: Administration updated end-to-end and looked completely correct, which is why the gap survived a full slice. Pick a second subject with a different shape before believing an update path works. Related but distinct: after any headless module install `mx update-widgets` is needed (CE0463); and a newer module can want a newer Atlas than the project has (29 × CE6083 on DataWidgets 3.11.3), which `rename-design-properties` does NOT fix — that is a dependency, not a resync | diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index 59c5ccf7f..c16d72505 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -6,6 +6,7 @@ import ( "fmt" "io" "os" + "path/filepath" "github.com/mendixlabs/mxcli/cmd/mxcli/marketplace" "github.com/mendixlabs/mxcli/internal/auth" @@ -137,9 +138,13 @@ func runMarketplaceUpdate(cmd *cobra.Command, args []string) error { if err != nil { return err } + // referenceFor downloaded the package into the work directory under this + // slot; the widgets are taken from it rather than from the reference project, + // whose widgets/ also holds the blank template's. + targetMpk := filepath.Join(work, "target.mpk") fmt.Fprintf(out, "\nUpdating %s %s → %s...\n", moduleName, installedVersion, target) - res, err := marketplace.PerformUpdate(mprPath, targetRef, moduleName, installedVersion, target, + res, err := marketplace.PerformUpdate(mprPath, targetRef, targetMpk, moduleName, installedVersion, target, targetVersion.VersionID, newBackendFactory()) if err != nil { return fmt.Errorf("%w\n\nThe project may be mid-update — %s could be missing. Restore from version control", @@ -211,6 +216,9 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { fmt.Fprintf(out, "\n%s updated %s → %s\n", r.Module, r.FromVersion, r.ToVersion) fmt.Fprintf(out, " %d units copied, %d element identities preserved, %d role grant(s) restored.\n", r.UnitsCopied, r.IdentitiesKept, r.GrantsRestored) + if len(r.WidgetsInstalled) > 0 { + fmt.Fprintf(out, " %d widget package(s) replaced in widgets/.\n", len(r.WidgetsInstalled)) + } if len(r.IdentitiesLost) > 0 { fmt.Fprintf(out, "\n Removed in %s (%d) — their database columns or tables will go on the next deploy:\n", @@ -234,7 +242,12 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { fmt.Fprintln(out, "\n Next: resync widget definitions, or 'mx check' will report CE0463 on the") fmt.Fprintln(out, " new version's pages (this is expected after any headless module install):") fmt.Fprintln(out, " mx update-widgets ") - fmt.Fprintln(out, "\n Then review with 'mxcli diff-local' and validate with 'mxcli docker check'.") + fmt.Fprintln(out, "\n Then check the app. A newer module can need newer companions — measured on") + fmt.Fprintln(out, " DataWidgets 3.11.3, whose widgets want design properties an older Atlas does") + fmt.Fprintln(out, " not define (29 × CE6083). That is a dependency to resolve, not something") + fmt.Fprintln(out, " this update can fix:") + fmt.Fprintln(out, " mxcli docker check -p ") + fmt.Fprintln(out, "\n Review the change with 'mxcli diff-local'.") } func init() { diff --git a/cmd/mxcli/marketplace/update.go b/cmd/mxcli/marketplace/update.go index 40e4c72f0..1b30504d1 100644 --- a/cmd/mxcli/marketplace/update.go +++ b/cmd/mxcli/marketplace/update.go @@ -3,10 +3,13 @@ package marketplace import ( + "archive/zip" "fmt" + "io" "os" "path/filepath" "regexp" + "sort" "strings" modelsdk "github.com/mendixlabs/mxcli" @@ -92,15 +95,16 @@ func safeFileName(k ElementKey) string { // UpdateResult is what an update did. type UpdateResult struct { - Module string - FromVersion string - ToVersion string - UnitsCopied int - IdentitiesKept int - IdentitiesLost []string - GrantsRestored int - GrantsDropped []string - ForcedOverEdits []string + Module string + FromVersion string + ToVersion string + UnitsCopied int + IdentitiesKept int + IdentitiesLost []string + GrantsRestored int + GrantsDropped []string + WidgetsInstalled []string + ForcedOverEdits []string } // PerformUpdate replaces an installed module with the copy in referenceMpr, @@ -116,7 +120,7 @@ type UpdateResult struct { // // A failure partway leaves the project in a broken state. The caller must work // on a copy, or hold a backup: this does not roll back. -func PerformUpdate(mprPath, referenceMpr, moduleName, fromVersion, toVersion, toVersionID string, +func PerformUpdate(mprPath, referenceMpr, targetMpk, moduleName, fromVersion, toVersion, toVersionID string, newBackend func() backend.FullBackend) (*UpdateResult, error) { ids, err := CaptureIdentities(mprPath, moduleName) @@ -148,16 +152,21 @@ func PerformUpdate(mprPath, referenceMpr, moduleName, fromVersion, toVersion, to if err := StampMarketplaceVersion(mprPath, moduleName, toVersion, toVersionID); err != nil { return nil, fmt.Errorf("record the installed version: %w", err) } + widgets, err := InstallPackageWidgets(targetMpk, filepath.Dir(mprPath)) + if err != nil { + return nil, fmt.Errorf("install the new version's widgets: %w", err) + } return &UpdateResult{ - Module: moduleName, - FromVersion: fromVersion, - ToVersion: toVersion, - UnitsCopied: copied, - IdentitiesKept: applied, - IdentitiesLost: missing, - GrantsRestored: restored, - GrantsDropped: dropped, + Module: moduleName, + FromVersion: fromVersion, + ToVersion: toVersion, + UnitsCopied: copied, + IdentitiesKept: applied, + IdentitiesLost: missing, + GrantsRestored: restored, + GrantsDropped: dropped, + WidgetsInstalled: widgets, }, nil } @@ -240,3 +249,55 @@ func setBoolField(doc bson.D, key string, value bool) { } } } + +// InstallPackageWidgets copies a package's bundled widget .mpk files into the +// project's widgets/ folder, and reports what it wrote. +// +// A module update that moves only the model is wrong for any module shipping +// widgets. Measured on DataWidgets 3.5.0 → 3.11.3: the model updated cleanly and +// all ten widget binaries were still the old version, so the project claimed +// 3.11.3 while running 3.5.0's widget code. Administration has no widgets, which +// is exactly why the first end-to-end run did not catch it. +// +// The files come from the package rather than from the reference project, +// because the reference is a blank app plus the module and its widgets/ folder +// therefore also holds the template's widgets — copying those would overwrite +// widgets this update has nothing to do with. +func InstallPackageWidgets(mpkPath, projectDir string) (written []string, err error) { + zr, err := zip.OpenReader(mpkPath) + if err != nil { + return nil, fmt.Errorf("open package %s: %w", filepath.Base(mpkPath), err) + } + defer zr.Close() + + for _, f := range zr.File { + if !strings.HasPrefix(f.Name, "widgets/") || f.FileInfo().IsDir() { + continue + } + base := filepath.Base(f.Name) + if base == "" || base == "." { + continue + } + dstDir := filepath.Join(projectDir, "widgets") + if err := os.MkdirAll(dstDir, 0o755); err != nil { + return written, fmt.Errorf("create widgets directory: %w", err) + } + dst := filepath.Join(dstDir, base) + + rc, oerr := f.Open() + if oerr != nil { + return written, fmt.Errorf("read %s from the package: %w", f.Name, oerr) + } + body, rerr := io.ReadAll(rc) + _ = rc.Close() + if rerr != nil { + return written, fmt.Errorf("read %s: %w", f.Name, rerr) + } + if err := os.WriteFile(dst, body, 0o644); err != nil { + return written, fmt.Errorf("write %s: %w", dst, err) + } + written = append(written, base) + } + sort.Strings(written) + return written, nil +} From 5d07764a4e5df6435be3d9b81376f80f45585236 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 06:39:40 +0000 Subject: [PATCH 30/35] fix(catalog,linter): catalog scheduled events and queues, and stop calling scheduled microflows dead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Neither document type was in the catalog, and the consequence was worse than a missing table. A scheduled event produced no edge in the reference graph, so a microflow run only by one was reported as unused from three directions at once: show callers of Ops.SE_Cleanup -> (no callers found) CATALOG.GRAPH_DEAD_ASSETS -> Ops.SE_Cleanup mxcli lint -> [QUAL004] Microflow 'SE_Cleanup' is not called from anywhere. -> Remove if unused on a microflow that runs nightly. The table is an absence somebody notices; the missing edge is a wrong answer three tools state confidently, one of which recommends deleting the file. Changes: - CATALOG.SCHEDULED_EVENTS (microflow, repeat, a readable schedule phrase, interval, enabled, timezone, on-overlap) and CATALOG.QUEUES (parallelism as a string — Mendix stores an expression — and cluster-wide), both registered in Tables() so SHOW CATALOG TABLES lists them. - A `schedule` edge from each event to its microflow, added to callerRefKinds, graphRefKinds and the QUAL004 rule — three consumers, none sharing the others' list. - The linter's interval_seconds is now derived from the Schedule child instead of the stored Interval/IntervalType pair. Studio Pro writes that pair and does not keep it in sync (Workflow Commons ships 0/"Minute" beside a daily schedule), so a "fires too often" rule read a nightly job as every 0 seconds. Rules can now also branch on repeat / on_overlap / time_zone, and iterate queues(). Also fixes a regression from the queue feature: QUEUES became a lexer keyword, so `select * from CATALOG.QUEUES` parsed to nothing — no error, no output — until QUEUES was added to catalogTableName, the same trap COMMUNITIES hit before. Both fixes have a control: reverting the interval derivation fails the schedule tests, and removing the ref edge takes QUAL004 back from 1 to 2 on the test project. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/lint-rules/orphaned_elements.star | 7 +- .claude/skills/fix-issue.md | 1 + .../mendix/scheduled-events-and-queues.md | 22 ++ CLAUDE.md | 2 +- docs-site/src/tools/catalog-tables.md | 56 +++++ mdl/catalog/builder.go | 13 + mdl/catalog/builder_graph.go | 4 + mdl/catalog/builder_references.go | 26 ++ mdl/catalog/builder_scheduling.go | 227 ++++++++++++++++++ mdl/catalog/builder_scheduling_test.go | 92 +++++++ mdl/catalog/catalog.go | 2 + mdl/catalog/tables.go | 46 ++++ mdl/executor/cmd_search.go | 7 +- mdl/grammar/domains/MDLCatalog.g4 | 1 + mdl/linter/context.go | 131 ++++++++-- mdl/linter/starlark.go | 37 ++- mdl/linter/starlark_scheduledevents_test.go | 186 +++++++++----- mdl/visitor/visitor_catalog_test.go | 12 + 18 files changed, 784 insertions(+), 88 deletions(-) create mode 100644 mdl/catalog/builder_scheduling.go create mode 100644 mdl/catalog/builder_scheduling_test.go diff --git a/.claude/lint-rules/orphaned_elements.star b/.claude/lint-rules/orphaned_elements.star index e3d16cdf5..1aa790cb2 100644 --- a/.claude/lint-rules/orphaned_elements.star +++ b/.claude/lint-rules/orphaned_elements.star @@ -54,10 +54,13 @@ def check(): # Get references to this microflow refs = refs_to(mf.qualified_name) - # Check if any reference is a call + # A scheduled event is an entry point: it runs the microflow without + # anything "calling" it, so a 'schedule' edge counts as a caller. Without + # this, a microflow that runs nightly in production was reported as + # orphaned — with the suggestion "Remove if unused". has_callers = False for ref in refs: - if ref.ref_kind == "call": + if ref.ref_kind == "call" or ref.ref_kind == "schedule": has_callers = True break diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index db8459c9b..033993449 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -474,5 +474,6 @@ extracting `OffsetExpression`/`LimitExpression`. | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | A microflow call bound to a task queue in Studio Pro loses that binding on `create or replace microflow` — silently, and `mx check` goes from `[CE1613] "The selected task queue no longer exists"` to **0 errors**, because the configuration the error was about has been deleted | Two defects behind one report. (1) `Microflows$MicroflowCall` has `QueueSettings` in `NullFields` of `codec.RegisterTypeDefaults` (and the legacy writer hardcodes null too) — correct for a newly authored call, destructive for a stored one, and since a CREATE OR REPLACE rebuilds the whole microflow it fires on every rewrite. (2) MDL had **no queue surface at all**, so a script could not restate the binding even in principle | `mdl/executor/validate_queued_calls.go` (new: the refusal), `mdl/executor/cmd_microflows_create.go` (call site), `mdl/types/queue.go` + `mdl/backend/infrastructure.go` (`QueueBackend`), `mdl/backend/modelsdk/queue_write.go` + `sdk/mpr/queues.go` (both engines), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4`, `mdl/executor/cmd_queues.go` | **A green build can be the bug**: the fix made `mx check` report MORE errors than before (CE1613 came back), because the binding it complains about now survives. Any "error count went down" check would have scored the data loss as a fix. **`describe` showing nothing is not evidence of nothing** — the binding was invisible from every angle (describe omitted it, check went quiet, the write reported success), which is why it needed a stored-BSON probe: inject `Queue`+`QueueSettings` into a stored call, rewrite, and dump. **Isolate which property Mendix acts on before guarding on it**: a call carrying only `Queue` (with `QueueSettings` null) draws NO complaint from `mx check` — `QueueSettings` is the load-bearing one, so a guard keyed on `Queue` alone would have both missed the real case and fired on inert ones. **Get the BSON shape from a Mendix-authored model, not from the metamodel**: `mxcli marketplace download 202649` (Business Events 3.12.1) has four real queues, and all four agree — `Config` is a `Queues$BasicQueueConfig` whose `ParallelismExpression` is a **STRING**, with the sibling int32 `Parallelism` absent in every one. Writing the int (which the metamodel makes look equally valid) is inventing a property Mendix does not write. **Refuse, do not half-author** (ADR-0005): `in queue` on a call is deliberately still unimplemented, because the `Retry` shape has no Studio-Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. **Adding a keyword costs an identifier** — `QUEUE`/`QUEUES` had to go into the `keyword` rule or `queue` would have stopped working as an attribute name. Tests `mdl/backend/modelsdk/queue_write_test.go`, `mdl/executor/validate_queued_calls_test.go`, `mdl/executor/cmd_queues_mock_test.go`, `mdl/visitor/visitor_queue_test.go`, example `mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl` | | Scheduled events are read-only in MDL — a project's cron cannot be scripted at all, so `mxcli new` + a script leaves you opening Studio Pro to add the one thing that makes a batch job run. Separately, `describe` of an existing one omits its schedule entirely | Two gaps behind one request. (1) No write path: `ScheduledEventBackend` had List/Get and nothing else, and there was no grammar/AST/visitor/executor. (2) The READ was lossy in both engines — the legacy `parseScheduledEvent` never looked at `Schedule`, and the modelsdk reader went through `modelsdk/gen`, whose generated types **disagree with what Studio Pro writes** on two properties | `mdl/scheduledevents/codec.go` (new, shared by both engines), `mdl/backend/modelsdk/scheduledevent_read.go` + `sdk/mpr/scheduledevents.go` + `sdk/mpr/parser_enumeration.go` (both engines through the one codec), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4` + `MDLParser.g4`, `mdl/ast/ast_scheduledevent.go`, `mdl/visitor/visitor_scheduledevent.go`, `mdl/executor/cmd_scheduledevents.go`, `mdl/executor/validate_scheduled_events.go` (MDL-SCHED01) | **The generated metamodel is not the storage's ground truth** — `modelsdk/gen` declares `Interval`/`HourOfDay`/… as **int32** where Studio Pro writes **int64**, and `StartDateTime` as a **string** where Studio Pro writes a **BSON datetime**. This is the same mismatch as #585 (a reader that asserted int32), so the pattern is now twice-confirmed: when gen and an observed document disagree about a numeric width or a date, the document is right. **A byte-level pin beats field assertions**: three whole documents are hex-embedded in `codec_test.go` and re-serialized element by element in ORDER, so key order, key SET and BSON TYPE are all covered — a field-by-field test passes happily while writing int32 and an empty enum. **`mxcli marketplace download` is the Studio Pro substitute**: Business Events and Community Commons have none, but Workflow Commons 4.11.0, OIDC SSO 4.6.0 and SAML 4.2.1 have four between them (only Day and Hour variants — the other six had to be metamodel-derived, then verified to load). **An empty string is not "unset" for an enum property**: the first draft wrote `IntervalType: ""` and `mx check` reported 0 errors, but the enumeration has no such member. **`Interval`/`IntervalType` are LEGACY siblings of `Schedule` that Studio Pro does not keep in sync** — Workflow Commons stores `0`/`Minute` beside a `DaySchedule` of 01:00 — so they are derived on CREATE and carried through untouched on MODIFY, never re-derived. **A polymorphic child must be dispatched on `$Type`, and its field sets never merged** (the ADR-0005 rule): the eight variants differ in arity, so the executor refuses a field belonging to another Repeat rather than dropping it. **Validate at check time, not only at exec** — the validation is decidable from the statement, so it runs in the no-project pass and `check` and `exec` call the SAME function. **Prove the describe round-trips by PARSING its own output** for every variant, not by asserting on strings. Idempotence came free (ADR-0008 canon elides the regenerated `Schedule` `$ID`) — verified with the `MXCLI_ALWAYS_WRITE=1` control, which changes 9 units where the normal run changes 0. Tests `mdl/scheduledevents/codec_test.go`, `mdl/executor/cmd_scheduledevents_test.go`, `mdl/visitor/visitor_scheduledevent_test.go`, `mdl/backend/modelsdk/scheduledevent_read_test.go`, example `mdl-examples/doctype-tests/scheduled-events.mdl` | +| A microflow run only by a scheduled event is reported as unused from three directions at once: `show callers of Mod.SE_Cleanup` says `(no callers found)`, `CATALOG.GRAPH_DEAD_ASSETS` lists it, and `mxcli lint` emits `[QUAL004] Microflow 'SE_Cleanup' is not called from anywhere.` with the suggestion **"Remove if unused"** — on a microflow that runs nightly in production | The catalog had no scheduled-event table and, more importantly, **no `refs` edge** from an event to the microflow it runs. Every consumer of the reference graph therefore agreed the microflow was dead. Separately, the linter's `ScheduledEvents()` iterator computed `interval_seconds` from the stored `Interval`/`IntervalType` pair | `mdl/catalog/tables.go` + `catalog.go` (new `scheduled_events`/`queues` views, registered in `Tables()`), `mdl/catalog/builder_scheduling.go` (new), `mdl/catalog/builder_references.go` (`RefKindSchedule` + `extractScheduledEventRefs`), `mdl/catalog/builder_graph.go` (`graphRefKinds`), `mdl/executor/cmd_search.go` (`callerRefKinds`), `.claude/lint-rules/orphaned_elements.star`, `mdl/linter/context.go` (`scheduleSeconds`, `Queues()`), `mdl/linter/starlark.go` | **A missing edge is worse than a missing table** — the table is an absence a user notices, the edge is a *wrong answer* three tools state confidently, and one of them recommends deleting the file. When adding a document type that references another, add the `refs` row in the same change. **An entry point is not a caller but must count as one**: `schedule` had to go into `callerRefKinds`, `graphRefKinds` AND the QUAL004 rule — three independent consumers, none of which shares the other's list. **Derive from the live property, not its legacy twin**: `interval_seconds` came from `Interval`/`IntervalType`, which Studio Pro writes and does not keep in sync with `Schedule` (Workflow Commons stores 0/"Minute" beside a DaySchedule of 01:00), so a "fires too often" rule read a nightly job as every 0 seconds; the old tests pinned the wrong source and had to be rewritten, which is the tell. **Adding a plural keyword breaks `CATALOG.`**: `QUEUES` became a lexer token and `select * from CATALOG.QUEUES` then parsed to NOTHING — no error, no output, just silence — until `QUEUES` was added to `catalogTableName` (`COMMUNITIES` had been through this before). `SCHEDULED_EVENTS` escaped only because the underscore makes it one IDENTIFIER. **`cmd/mxcli/lint-rules/` is gitignored and regenerated** from `.claude/lint-rules/` by `make build`, exactly like `cmd/mxcli/skills/` — editing the embed dir silently reverts on the next build. **`TestTables_CoversAllViews` catches a view missing from `Tables()`**, so a new catalog table is invisible to `SHOW CATALOG TABLES` until registered. Tests `mdl/catalog/builder_scheduling_test.go`, `mdl/linter/starlark_scheduledevents_test.go`, `mdl/visitor/visitor_catalog_test.go` | | `mxcli new` into a deep output directory dies with `System.IO.PathTooLongException` and leaves the directory holding ~259 files and no `.mpr` — output that looks like a project until you try to open it | MxToolset refuses any full destination path over **259 characters** (its own Windows-compatibility limit, not the filesystem's) and aborts extraction PART WAY THROUGH. mxcli pointed `mx create-project` straight at the user's output directory, so the abort happened there | `cmd/mxcli/newproject_paths.go` (new: `stagedProjectDirs`, `longestRelativePath`, `warnIfPathTooLongForStudioPro`, `moveProject`), wired into `cmd/mxcli/cmd_new.go` step 2 | **Stage the work somewhere safe and move it in, rather than validating a path you could just avoid** — creating in a short temp dir and renaming makes ANY depth work (POSIX allows 4096) instead of merely failing politely, and the destination is never partially populated because nothing is written there until creation succeeded. Check the relocation is safe first: `grep -rl ` returned **0 files**, so a fresh Mendix project embeds no absolute paths. **Bisect for the real threshold instead of trusting arithmetic** — 77 characters creates a project on 11.13.0, 78 fails leaving 259 files, which pins `len(dest) + 1 + longest ≤ 259` exactly. **Measure the template, don't hardcode it**: the longest relative path is 181 on 11.13.0 and 182 on 11.12.0, so walk the staged tree and use the real number, or the reported budget drifts a character per release. **Warn, don't refuse, when the finished path is over budget** — the project works on POSIX, so refusing would block a machine where it is fine; but Studio Pro on Windows would not open it, so silence would be worse. Cross-device staging needs an `os.CopyFS` fallback: `os.Rename` cannot cross filesystems. Tests `cmd/mxcli/newproject_paths_test.go`. upstream #825 | | `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | diff --git a/.claude/skills/mendix/scheduled-events-and-queues.md b/.claude/skills/mendix/scheduled-events-and-queues.md index a6ebf19e1..187b06ecf 100644 --- a/.claude/skills/mendix/scheduled-events-and-queues.md +++ b/.claude/skills/mendix/scheduled-events-and-queues.md @@ -179,6 +179,28 @@ mxcli check script.mdl -p app.mpr --references - [ ] Enum values spelled exactly (`Server`, `DelayNext`, `Last`, `Monday`) - [ ] The target microflow exists and takes no parameters +## Querying and Linting + +Both document types are in the catalog after `refresh catalog`: + +```sql +-- Anything that fires more often than once a minute +select QualifiedName, RepeatDescription, Microflow +from CATALOG.SCHEDULED_EVENTS +where Enabled = 1 and IntervalSeconds < 60; + +select QualifiedName, Parallelism, ClusterWide from CATALOG.QUEUES; +``` + +A scheduled event counts as a caller of the microflow it runs, so +`show callers of Ops.SE_Cleanup` lists it and the lint rule for orphaned +microflows (QUAL004) does not flag it. `IntervalSeconds` is derived from the +schedule, not from the legacy `Interval`/`IntervalType` pair Mendix also stores. + +Starlark lint rules can iterate both: `scheduled_events()` yields +`repeat`, `interval_seconds`, `on_overlap`, `time_zone`, `enabled`, and +`microflow_name`; `queues()` yields `parallelism` (a string) and `cluster_wide`. + ## Related - `mxcli syntax scheduled-event`, `mxcli syntax queue` — full syntax reference diff --git a/CLAUDE.md b/CLAUDE.md index 9bcdf0fbe..c08a53743 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -669,7 +669,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - ALTER WORKFLOW (SET properties, INSERT/DROP/REPLACE activities, outcomes, paths, conditions, boundary events) - CALCULATED BY microflow syntax for calculated attributes - Image collections (SHOW/DESCRIBE/CREATE/DROP) -- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. See `.claude/skills/mendix/scheduled-events-and-queues.md` +- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. Both are in the catalog (`CATALOG.SCHEDULED_EVENTS`, `CATALOG.QUEUES`) and a scheduled event emits a `schedule` edge into `CATALOG.REFS` — without it a microflow run only by a scheduled event was reported as dead by `show callers`, `GRAPH_DEAD_ASSETS` and lint rule QUAL004. See `.claude/skills/mendix/scheduled-events-and-queues.md` - Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005): the rebuild used to write `QueueSettings` back as null, which made `mx check` go from CE1613 to 0 errors by deleting the user's configuration - AI agent documents: Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) - OData contract browsing (SHOW/DESCRIBE CONTRACT ENTITIES/ACTIONS FROM cached $metadata) diff --git a/docs-site/src/tools/catalog-tables.md b/docs-site/src/tools/catalog-tables.md index d6bdbfe39..6f5d20825 100644 --- a/docs-site/src/tools/catalog-tables.md +++ b/docs-site/src/tools/catalog-tables.md @@ -130,6 +130,62 @@ JOIN CATALOG.ENTITIES e ON ar.EntityId = e.Id WHERE e.ModuleName = 'Sales'; ``` +### CATALOG.SCHEDULED_EVENTS + +Scheduled events — Mendix's cron. + +| Column | Description | +|--------|-------------| +| `Name`, `QualifiedName`, `ModuleName`, `Folder` | Identity | +| `Microflow` | Qualified name of the microflow the event runs | +| `Repeat` | Schedule variant: `Minute`, `Hour`, `Day`, `Week`, `MonthDate`, `MonthWeekday`, `YearDate`, `YearWeekday` | +| `RepeatDescription` | The schedule as a phrase, e.g. `weekly Mon/Fri at 09:30` | +| `IntervalSeconds` | Gap between runs, derived from the schedule | +| `Enabled` | 1 if the event runs | +| `TimeZone` | `UTC` or `Server` | +| `OnOverlap` | `DelayNext` or `SkipNext` | + +`IntervalSeconds` comes from the schedule, **not** from the stored +`Interval`/`IntervalType` pair. Those are a legacy sibling that Studio Pro writes +and does not keep in sync — a shipped Mendix module stores `0`/`Minute` beside a +daily schedule — so a query keyed on them would read a nightly job as firing +every 0 seconds. Month and year figures are averages (30 and 365 days): the +column is for thresholds and ordering, not calendar arithmetic. + +```sql +-- Anything that fires more often than once a minute +select QualifiedName, RepeatDescription, Microflow +from CATALOG.SCHEDULED_EVENTS +where Enabled = 1 and IntervalSeconds < 60; + +-- Scheduled events whose microflow no longer exists +select se.QualifiedName, se.Microflow +from CATALOG.SCHEDULED_EVENTS se +left join CATALOG.MICROFLOWS m on m.QualifiedName = se.Microflow +where m.Id is null; +``` + +A scheduled event also produces a `schedule` row in `CATALOG.REFS`, so +`show callers of ` and the dead-asset analysis both see it. Without +that edge a microflow run only by a scheduled event looked unreferenced. + +### CATALOG.QUEUES + +Task queues. + +| Column | Description | +|--------|-------------| +| `Name`, `QualifiedName`, `ModuleName`, `Folder` | Identity | +| `Parallelism` | How many tasks run at once — an **expression string**, not a number | +| `ClusterWide` | 1 if the limit applies across the cluster | + +`Parallelism` is stored as text because Mendix stores an expression: a query must +not assume it parses as an integer. + +```sql +select QualifiedName, Parallelism, ClusterWide from CATALOG.QUEUES; +``` + ## Graph-Analysis Tables The dependency graph (`CATALOG.REFS`, full refresh) is analysed by a family of diff --git a/mdl/catalog/builder.go b/mdl/catalog/builder.go index d3aeafc87..0540318a6 100644 --- a/mdl/catalog/builder.go +++ b/mdl/catalog/builder.go @@ -91,6 +91,11 @@ type Builder struct { resolution float64 // Leiden resolution for the graph-analysis pass describeFunc DescribeFunc + // Scheduled event → microflow edges, collected while cataloguing the events + // and emitted by buildReferences (a later pass). Carried on the Builder + // rather than re-queried because CatalogTx has no Query. + scheduledEventRefs []scheduledEventRef + // Built-in widget definitions supplied by the caller — used to populate // the widget_definitions catalog table alongside project widgets/. builtinWidgetMetas []WidgetDefinitionMeta @@ -402,6 +407,14 @@ func (b *Builder) Build(progress ProgressFunc) error { return fmt.Errorf("failed to build data transformers: %w", err) } + if err := b.buildScheduledEvents(); err != nil { + return fmt.Errorf("failed to build scheduled events: %w", err) + } + + if err := b.buildQueues(); err != nil { + return fmt.Errorf("failed to build queues: %w", err) + } + if err := b.buildAgentEditorDocs(); err != nil { return fmt.Errorf("failed to build agent-editor documents: %w", err) } diff --git a/mdl/catalog/builder_graph.go b/mdl/catalog/builder_graph.go index 12cc298c0..eb926f4ea 100644 --- a/mdl/catalog/builder_graph.go +++ b/mdl/catalog/builder_graph.go @@ -15,6 +15,10 @@ import ( var graphRefKinds = []string{ "call", "retrieve", "create", "change", "delete", "associate", "generalize", "parameter", "return", + // A scheduled event is an entry point: the microflow it runs is reachable + // even though nothing calls it. Without this kind, GRAPH_DEAD_ASSETS reports + // every scheduled microflow as dead. + "schedule", } // betweennessNodeCap bounds the O(V*E) betweenness computation. Above it, diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index b85da8997..4b840db91 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -32,6 +32,7 @@ const ( RefKindDelete = "delete" // Microflow deletes an entity object RefKindCalculate = "calculate" // Calculated attribute uses a microflow RefKindReturn = "return" // Microflow/nanoflow returns an entity type + RefKindSchedule = "schedule" // Scheduled event runs a microflow ) // collectActionActivities returns all ActionActivity objects from an ObjectCollection, @@ -481,10 +482,35 @@ func (b *Builder) buildReferences() error { } } + // Scheduled events run a microflow. Without this edge the microflow looks + // unreferenced: `show callers` reported none, GRAPH_DEAD_ASSETS listed it, + // and QUAL004 said "is not called from anywhere" with the suggestion + // "Remove if unused" — on a microflow that runs nightly in production. + refCount += b.extractScheduledEventRefs(stmt, projectID, snapshotID) + b.report("References", refCount) return nil } +// extractScheduledEventRefs emits one `schedule` edge per scheduled event, from +// the event to the microflow it runs. +// +// The edges are collected by buildScheduledEvents, which runs earlier in the +// same transaction. +func (b *Builder) extractScheduledEventRefs(stmt *sql.Stmt, projectID, snapshotID string) int { + count := 0 + for _, r := range b.scheduledEventRefs { + if _, err := stmt.Exec( + "SCHEDULED_EVENT", "", r.qualifiedName, + "MICROFLOW", "", r.microflow, + RefKindSchedule, r.moduleName, projectID, snapshotID, + ); err == nil { + count++ + } + } + return count +} + // extractMenuItemRefs extracts page and microflow references from menu items recursively. func (b *Builder) extractMenuItemRefs(stmt *sql.Stmt, items []*types.NavMenuItem, sourceName, projectID, snapshotID string) int { refCount := 0 diff --git a/mdl/catalog/builder_scheduling.go b/mdl/catalog/builder_scheduling.go new file mode 100644 index 000000000..c7a7cbd59 --- /dev/null +++ b/mdl/catalog/builder_scheduling.go @@ -0,0 +1,227 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "fmt" + "strings" + + "go.mongodb.org/mongo-driver/bson" + + sched "github.com/mendixlabs/mxcli/mdl/scheduledevents" + "github.com/mendixlabs/mxcli/model" +) + +// Catalog rows for the two scheduling document types. +// +// Both read through the raw-unit surface and decode with the same codec the +// writers use, so the catalog cannot drift from what is stored. Scheduled events +// get their own builder rather than going through buildSimpleNamedDocs because +// the interesting columns — which microflow runs, how often, whether it is +// enabled — are the whole reason to query them. + +// scheduledEventRef is one event → microflow edge, held until buildReferences. +type scheduledEventRef struct{ qualifiedName, moduleName, microflow string } + +// buildScheduledEvents catalogs ScheduledEvents$ScheduledEvent units. +func (b *Builder) buildScheduledEvents() error { + units, err := b.reader.ListRawUnitsByType(sched.TypeName) + if err != nil { + return err + } + + stmt, err := b.tx.Prepare( + `INSERT INTO scheduled_events_data + (Id, Name, QualifiedName, ModuleName, Folder, Description, Microflow, + Repeat, RepeatDescription, IntervalSeconds, Enabled, TimeZone, OnOverlap, + ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + if err != nil { + return err + } + defer stmt.Close() + + projectID, snapshotID := b.snapshotMeta() + + count := 0 + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + continue + } + ev := sched.Parse(doc, model.ID(u.ID), model.ID(u.ContainerID)) + if ev.Name == "" { + continue + } + moduleID := b.hierarchy.findModuleID(u.ContainerID) + moduleName := b.hierarchy.getModuleName(moduleID) + + repeat, desc := "", "" + if ev.Schedule != nil { + repeat = string(ev.Schedule.Kind) + desc = describeSchedule(ev.Schedule) + } + if _, err := stmt.Exec( + string(u.ID), ev.Name, moduleName+"."+ev.Name, moduleName, + b.hierarchy.buildFolderPath(u.ContainerID), ev.Documentation, + string(ev.MicroflowID), repeat, desc, + scheduleIntervalSeconds(ev.Schedule), boolToInt(ev.Enabled), + ev.TimeZone, ev.OnOverlap, projectID, snapshotID, + ); err != nil { + return err + } + if ev.MicroflowID != "" { + b.scheduledEventRefs = append(b.scheduledEventRefs, scheduledEventRef{ + qualifiedName: moduleName + "." + ev.Name, + moduleName: moduleName, + microflow: string(ev.MicroflowID), + }) + } + count++ + } + + b.report("Scheduled Events", count) + return nil +} + +// buildQueues catalogs Queues$Queue units. +func (b *Builder) buildQueues() error { + units, err := b.reader.ListRawUnitsByType("Queues$Queue") + if err != nil { + return err + } + + stmt, err := b.tx.Prepare( + `INSERT INTO queues_data + (Id, Name, QualifiedName, ModuleName, Folder, Description, Parallelism, + ClusterWide, ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + if err != nil { + return err + } + defer stmt.Close() + + projectID, snapshotID := b.snapshotMeta() + + count := 0 + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + continue + } + name, _ := doc["Name"].(string) + if name == "" { + continue + } + documentation, _ := doc["Documentation"].(string) + // Parallelism lives on the nested Config node and is an EXPRESSION + // string, so it is stored as text rather than an integer column. + parallelism := "" + clusterWide := false + if cfg, ok := doc["Config"].(bson.M); ok { + parallelism, _ = cfg["ParallelismExpression"].(string) + clusterWide, _ = cfg["ClusterWide"].(bool) + } + moduleID := b.hierarchy.findModuleID(u.ContainerID) + moduleName := b.hierarchy.getModuleName(moduleID) + + if _, err := stmt.Exec( + string(u.ID), name, moduleName+"."+name, moduleName, + b.hierarchy.buildFolderPath(u.ContainerID), documentation, + parallelism, boolToInt(clusterWide), projectID, snapshotID, + ); err != nil { + return err + } + count++ + } + + b.report("Task Queues", count) + return nil +} + +// scheduleIntervalSeconds is how often the event fires, derived from the +// Schedule child. +// +// It deliberately does NOT use the stored Interval/IntervalType pair, which is a +// legacy sibling of Schedule that Studio Pro writes and does not keep in sync — +// Workflow Commons ships an event storing 0/"Minute" beside a DaySchedule of +// 01:00, which would catalog as "fires every 0 seconds". +// +// The month and year figures are averages (30 and 365 days); the column is for +// ordering and thresholds ("anything under a minute"), not for arithmetic on +// calendar dates. +func scheduleIntervalSeconds(s *model.Schedule) int64 { + if s == nil { + return 0 + } + mult := int64(s.Multiplier) + if mult < 1 { + mult = 1 + } + const day = int64(86400) + switch s.Kind { + case model.ScheduleMinute: + return mult * 60 + case model.ScheduleHour: + return mult * 3600 + case model.ScheduleDay: + return day + case model.ScheduleWeek: + // A weekly schedule can name several days, so the gap between runs is + // the week divided by how many are selected. + n := int64(0) + for _, on := range s.Weekdays { + if on { + n++ + } + } + if n == 0 { + return 7 * day + } + return (7 * day) / n + case model.ScheduleMonthDate, model.ScheduleMonthWeekday: + return mult * 30 * day + case model.ScheduleYearDate, model.ScheduleYearWeekday: + return 365 * day + } + return 0 +} + +// describeSchedule renders the repeat rule as a short human phrase, so a catalog +// query is readable without decoding the variant's fields. +func describeSchedule(s *model.Schedule) string { + if s == nil { + return "" + } + at := fmt.Sprintf("%02d:%02d", s.HourOfDay, s.MinuteOfHour) + switch s.Kind { + case model.ScheduleMinute: + return fmt.Sprintf("every %d min", s.Multiplier) + case model.ScheduleHour: + return fmt.Sprintf("every %dh at :%02d", s.Multiplier, s.MinuteOffset) + case model.ScheduleDay: + return "daily at " + at + case model.ScheduleWeek: + var days []string + for i, on := range s.Weekdays { + if on { + days = append(days, sched.WeekdayNames[i][:3]) + } + } + if days == nil { + days = []string{"(no days)"} + } + return "weekly " + strings.Join(days, "/") + " at " + at + case model.ScheduleMonthDate: + return fmt.Sprintf("every %d month(s) on day %d at %s", s.Multiplier, s.DayOfMonth, at) + case model.ScheduleMonthWeekday: + return fmt.Sprintf("every %d month(s) on the %s %s at %s", s.Multiplier, s.DaySelector, s.Weekday, at) + case model.ScheduleYearDate: + return fmt.Sprintf("yearly on %d/%d at %s", s.Month, s.DayOfMonth, at) + case model.ScheduleYearWeekday: + return fmt.Sprintf("yearly on the %s %s of month %d at %s", s.DaySelector, s.Weekday, s.Month, at) + } + return string(s.Kind) +} diff --git a/mdl/catalog/builder_scheduling_test.go b/mdl/catalog/builder_scheduling_test.go new file mode 100644 index 000000000..ad22ec327 --- /dev/null +++ b/mdl/catalog/builder_scheduling_test.go @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" +) + +// TestScheduleIntervalSeconds pins the interval a catalog query sees. +// +// It is derived from the Schedule child on purpose. The stored +// Interval/IntervalType pair is a legacy sibling that Studio Pro writes and does +// NOT keep in sync — Workflow Commons 4.11.0 ships an event storing 0/"Minute" +// beside a DaySchedule of 01:00 — so a threshold query keyed on the pair would +// read a nightly job as firing every 0 seconds. +func TestScheduleIntervalSeconds(t *testing.T) { + const day = int64(86400) + tests := []struct { + name string + sched *model.Schedule + want int64 + }{ + {"nil", nil, 0}, + {"every 2 minutes", &model.Schedule{Kind: model.ScheduleMinute, Multiplier: 2}, 120}, + {"every 3 hours", &model.Schedule{Kind: model.ScheduleHour, Multiplier: 3}, 10800}, + {"daily", &model.Schedule{Kind: model.ScheduleDay}, day}, + {"weekly, one day", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, false, false}}, 7 * day}, + // Two selected days means it fires twice a week, not once. + {"weekly, two days", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, true, false}}, (7 * day) / 2}, + {"weekly, no days", &model.Schedule{Kind: model.ScheduleWeek}, 7 * day}, + {"every 3 months", &model.Schedule{Kind: model.ScheduleMonthDate, Multiplier: 3}, 3 * 30 * day}, + {"yearly", &model.Schedule{Kind: model.ScheduleYearWeekday}, 365 * day}, + // An unstated multiplier is 1, not 0 — 0 would read as "never fires". + {"multiplier defaults to 1", &model.Schedule{Kind: model.ScheduleHour}, 3600}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := scheduleIntervalSeconds(tt.sched); got != tt.want { + t.Errorf("scheduleIntervalSeconds = %d, want %d", got, tt.want) + } + }) + } +} + +// TestDescribeSchedule checks the human phrase each variant produces, and that +// only that variant's fields reach it — a MonthWeekday must not print a day of +// the month, or the catalog would describe a schedule the model does not have. +func TestDescribeSchedule(t *testing.T) { + full := model.Schedule{ + Multiplier: 3, MinuteOffset: 23, MonthOffset: 2, + HourOfDay: 18, MinuteOfHour: 30, DayOfMonth: 15, Month: 3, + DaySelector: "Last", Weekday: "Friday", + Weekdays: [7]bool{false, true, false, false, false, true, false}, + } + tests := []struct { + kind model.ScheduleKind + want string + mustNot []string + }{ + {model.ScheduleMinute, "every 3 min", []string{"18:30", "Friday"}}, + {model.ScheduleHour, "every 3h at :23", []string{"18:30", "Friday"}}, + {model.ScheduleDay, "daily at 18:30", []string{"Friday", "3 month"}}, + {model.ScheduleWeek, "weekly Mon/Fri at 18:30", []string{"Last"}}, + {model.ScheduleMonthDate, "every 3 month(s) on day 15 at 18:30", []string{"Friday"}}, + {model.ScheduleMonthWeekday, "every 3 month(s) on the Last Friday at 18:30", []string{"day 15"}}, + {model.ScheduleYearDate, "yearly on 3/15 at 18:30", []string{"Friday"}}, + {model.ScheduleYearWeekday, "yearly on the Last Friday of month 3 at 18:30", []string{"day 15"}}, + } + for _, tt := range tests { + t.Run(string(tt.kind), func(t *testing.T) { + s := full + s.Kind = tt.kind + got := describeSchedule(&s) + if got != tt.want { + t.Errorf("describeSchedule = %q, want %q", got, tt.want) + } + for _, bad := range tt.mustNot { + if strings.Contains(got, bad) { + t.Errorf("describeSchedule = %q — %q belongs to a different variant", got, bad) + } + } + }) + } + if got := describeSchedule(nil); got != "" { + t.Errorf("describeSchedule(nil) = %q, want empty", got) + } +} diff --git a/mdl/catalog/catalog.go b/mdl/catalog/catalog.go index 07383a3d7..83eaf8815 100644 --- a/mdl/catalog/catalog.go +++ b/mdl/catalog/catalog.go @@ -115,6 +115,8 @@ func (c *Catalog) Tables() []string { "CATALOG.JAVA_ACTION_PARAMETERS", "CATALOG.JAVASCRIPT_ACTIONS", "CATALOG.IMAGE_COLLECTIONS", + "CATALOG.SCHEDULED_EVENTS", + "CATALOG.QUEUES", "CATALOG.DATA_TRANSFORMERS", "CATALOG.AGENTS", "CATALOG.AI_MODELS", diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index 583cc993d..c76dc769d 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -312,6 +312,44 @@ func (c *Catalog) createTables() error { )`, viewWithFullSnapshot("image_collections"), + // scheduled_events — Mendix's cron. Repeat/RepeatDescription come from the + // Schedule child; IntervalSeconds is derived from it, NOT from the legacy + // Interval/IntervalType pair (Studio Pro does not keep those in sync). + `CREATE TABLE IF NOT EXISTS scheduled_events_data ( + Id TEXT PRIMARY KEY, + Name TEXT, + QualifiedName TEXT, + ModuleName TEXT, + Folder TEXT, + Description TEXT, + Microflow TEXT, + Repeat TEXT, + RepeatDescription TEXT, + IntervalSeconds INTEGER, + Enabled INTEGER, + TimeZone TEXT, + OnOverlap TEXT, + ProjectId TEXT, + SnapshotId TEXT + )`, + viewWithFullSnapshot("scheduled_events"), + + // queues — task queues (Queues$Queue). Parallelism is an expression + // string, not a number. + `CREATE TABLE IF NOT EXISTS queues_data ( + Id TEXT PRIMARY KEY, + Name TEXT, + QualifiedName TEXT, + ModuleName TEXT, + Folder TEXT, + Description TEXT, + Parallelism TEXT, + ClusterWide INTEGER, + ProjectId TEXT, + SnapshotId TEXT + )`, + viewWithFullSnapshot("queues"), + // data_transformers `CREATE TABLE IF NOT EXISTS data_transformers_data ( Id TEXT PRIMARY KEY, @@ -925,6 +963,14 @@ func (c *Catalog) createTables() error { ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource FROM image_collections UNION ALL + SELECT Id, 'SCHEDULED_EVENT' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, + ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource + FROM scheduled_events + UNION ALL + SELECT Id, 'QUEUE' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, + ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource + FROM queues + UNION ALL SELECT Id, 'DATA_TRANSFORMER' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource FROM data_transformers diff --git a/mdl/executor/cmd_search.go b/mdl/executor/cmd_search.go index 689528ecd..dec48316f 100644 --- a/mdl/executor/cmd_search.go +++ b/mdl/executor/cmd_search.go @@ -20,7 +20,10 @@ import ( // // A false negative here reads as "nothing uses this", which is the answer // somebody acts on before deleting a document — so the set errs toward -// including a kind rather than omitting it. +// including a kind rather than omitting it. 'schedule' is the same shape: a +// microflow run only by a scheduled event reported "(no callers found)", was +// listed in GRAPH_DEAD_ASSETS, and drew QUAL004 "is not called from anywhere" +// with the suggestion "Remove if unused" — on a microflow that runs nightly. // // Deliberately excluded: 'datasource', 'parameter', 'return', 'retrieve', // 'create', 'change', 'delete', 'associate', 'generalize' and 'layout'. Those @@ -34,6 +37,7 @@ var callerRefKinds = []string{ RefKindCallerHomePage, // navigation RefKindCallerLoginPage, RefKindCallerMenuItem, + RefKindCallerSchedule, // scheduled event: the microflow it runs } // Kind literals, kept next to the set that uses them so the SQL below cannot @@ -46,6 +50,7 @@ const ( RefKindCallerHomePage = "home_page" RefKindCallerLoginPage = "login_page" RefKindCallerMenuItem = "menu_item" + RefKindCallerSchedule = "schedule" ) // callerRefKindsSQL renders callerRefKinds as a SQL IN list. diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index 0ed72c210..9581d0061 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -220,6 +220,7 @@ catalogTableName | CONSTANTS // keyword token — must be listed explicitly | OBJECTS // keyword token — must be listed explicitly | COMMUNITIES // keyword token (SHOW COMMUNITIES) — must be listed explicitly for CATALOG.COMMUNITIES + | QUEUES // keyword token (SHOW QUEUES) — must be listed explicitly for CATALOG.QUEUES | SOURCE_KW // For CATALOG.SOURCE FTS table | ODATA // For CATALOG.ODATA_CLIENTS and CATALOG.ODATA_SERVICES (via IDENTIFIER) | IDENTIFIER // For tables like activities, xpath_expressions, projects, snapshots, refs, strings, odata_clients, odata_services, java_actions diff --git a/mdl/linter/context.go b/mdl/linter/context.go index e38e774fb..d0b08b438 100644 --- a/mdl/linter/context.go +++ b/mdl/linter/context.go @@ -814,32 +814,77 @@ func (ctx *LintContext) Snippets() iter.Seq[Snippet] { // ScheduledEvent represents a scheduled event document. type ScheduledEvent struct { - Name string - QualifiedName string - ModuleName string - MicroflowName string // qualified name of the microflow to execute + Name string + QualifiedName string + ModuleName string + MicroflowName string // qualified name of the microflow to execute + // IntervalSeconds is how often the event fires, derived from the Schedule + // child — NOT from the stored Interval/IntervalType pair, which Studio Pro + // writes and does not keep in sync with Schedule. Workflow Commons ships an + // event storing 0/"Minute" beside a DaySchedule of 01:00, so a rule keyed on + // the legacy pair would read it as "fires every 0 seconds". IntervalSeconds int - Enabled bool -} - -// intervalToSeconds converts a Mendix interval value and type to seconds. -// Returns 0 for unrecognised interval types (treated as "not convertible"). -func intervalToSeconds(interval int, intervalType string) int { - multipliers := map[string]int{ - "Second": 1, - "Minute": 60, - "Hour": 3600, - "Day": 86400, - "Week": 604800, - "Month": 2592000, - "Year": 31536000, + // Repeat is the schedule variant (Minute, Hour, Day, Week, MonthDate, + // MonthWeekday, YearDate, YearWeekday); empty when the event has no + // Schedule child. + Repeat string + // OnOverlap is DelayNext or SkipNext — the event's own concurrency control. + OnOverlap string + TimeZone string + Enabled bool +} + +// scheduleSeconds is the gap between runs implied by a schedule. +// +// The month and year figures are averages (30 and 365 days): the value is for +// thresholds and ordering ("anything that fires more often than a minute"), not +// for calendar arithmetic. +func scheduleSeconds(s *model.Schedule) int { + if s == nil { + return 0 + } + mult := s.Multiplier + if mult < 1 { + mult = 1 } - if mult, ok := multipliers[intervalType]; ok { - return interval * mult + const day = 86400 + switch s.Kind { + case model.ScheduleMinute: + return mult * 60 + case model.ScheduleHour: + return mult * 3600 + case model.ScheduleDay: + return day + case model.ScheduleWeek: + n := 0 + for _, on := range s.Weekdays { + if on { + n++ + } + } + if n == 0 { + return 7 * day + } + return (7 * day) / n + case model.ScheduleMonthDate, model.ScheduleMonthWeekday: + return mult * 30 * day + case model.ScheduleYearDate, model.ScheduleYearWeekday: + return 365 * day } return 0 } +// Queue represents a task queue document. +type Queue struct { + Name string + QualifiedName string + ModuleName string + // Parallelism is an EXPRESSION, not a number — Mendix stores it as a string, + // so a rule must not assume it parses as an integer. + Parallelism string + ClusterWide bool +} + // ScheduledEvents returns an iterator over all scheduled events (excluding system modules). // Returns an empty iterator if no reader is available. func (ctx *LintContext) ScheduledEvents() iter.Seq[ScheduledEvent] { @@ -889,12 +934,19 @@ func (ctx *LintContext) ScheduledEvents() iter.Seq[ScheduledEvent] { if mfName == "" { mfName = string(e.MicroflowID) } + repeat := "" + if e.Schedule != nil { + repeat = string(e.Schedule.Kind) + } se := ScheduledEvent{ Name: e.Name, QualifiedName: moduleName + "." + e.Name, ModuleName: moduleName, MicroflowName: mfName, - IntervalSeconds: intervalToSeconds(e.Interval, e.IntervalType), + IntervalSeconds: scheduleSeconds(e.Schedule), + Repeat: repeat, + OnOverlap: e.OnOverlap, + TimeZone: e.TimeZone, Enabled: e.Enabled, } if !yield(se) { @@ -981,6 +1033,43 @@ type DatabaseConnection struct { } // DatabaseConnections returns an iterator over all database connections (excluding system modules). +// Queues returns an iterator over all task queues (excluding platform modules). +// +// Backed by the catalog rather than the reader, so no LintReader change is +// needed; the catalog is already built whenever rules run. +func (ctx *LintContext) Queues() iter.Seq[Queue] { + return func(yield func(Queue) bool) { + rows, err := ctx.db.Query(fmt.Sprintf(` + SELECT q.Name, q.QualifiedName, q.ModuleName, q.Parallelism, q.ClusterWide + FROM queues q + LEFT JOIN modules m ON q.ModuleName = m.Name + WHERE %s + ORDER BY q.ModuleName, q.Name + `, notPlatformModule("m"))) + if err != nil { + ctx.recordQueryError("Queues", err) + return + } + defer rows.Close() + + for rows.Next() { + var q Queue + var clusterWide int + if err := rows.Scan(&q.Name, &q.QualifiedName, &q.ModuleName, &q.Parallelism, &clusterWide); err != nil { + ctx.recordQueryError("Queues (row scan)", err) + continue + } + q.ClusterWide = clusterWide != 0 + if ctx.IsExcluded(q.ModuleName) { + continue + } + if !yield(q) { + return + } + } + } +} + func (ctx *LintContext) DatabaseConnections() iter.Seq[DatabaseConnection] { return func(yield func(DatabaseConnection) bool) { rows, err := ctx.db.Query(fmt.Sprintf(` diff --git a/mdl/linter/starlark.go b/mdl/linter/starlark.go index 88623fa7f..a1d23e880 100644 --- a/mdl/linter/starlark.go +++ b/mdl/linter/starlark.go @@ -326,6 +326,7 @@ func (r *StarlarkRule) buildPredeclared() starlark.StringDict { "refs_from": starlark.NewBuiltin("refs_from", r.builtinRefsFrom), "attributes_for": starlark.NewBuiltin("attributes_for", r.builtinAttributesFor), "scheduled_events": starlark.NewBuiltin("scheduled_events", r.builtinScheduledEvents), + "queues": starlark.NewBuiltin("queues", r.builtinQueues), // Graph-analysis facts (populated by `refresh catalog communities`). "community_of": starlark.NewBuiltin("community_of", r.builtinCommunityOf), @@ -1016,15 +1017,43 @@ func databaseConnectionToStarlark(dc DatabaseConnection) starlark.Value { func scheduledEventToStarlark(se ScheduledEvent) starlark.Value { return starlarkstruct.FromStringDict(starlark.String("scheduled_event"), starlark.StringDict{ - "name": starlark.String(se.Name), - "qualified_name": starlark.String(se.QualifiedName), - "module_name": starlark.String(se.ModuleName), - "microflow_name": starlark.String(se.MicroflowName), + "name": starlark.String(se.Name), + "qualified_name": starlark.String(se.QualifiedName), + "module_name": starlark.String(se.ModuleName), + "microflow_name": starlark.String(se.MicroflowName), + // Derived from the Schedule child, not the legacy Interval/IntervalType + // pair — see ScheduledEvent.IntervalSeconds. "interval_seconds": starlark.MakeInt(se.IntervalSeconds), + "repeat": starlark.String(se.Repeat), + "on_overlap": starlark.String(se.OnOverlap), + "time_zone": starlark.String(se.TimeZone), "enabled": starlark.Bool(se.Enabled), }) } +// builtinQueues returns all task queues. +func (r *StarlarkRule) builtinQueues(_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { + if r.ctx == nil { + return starlark.NewList(nil), nil + } + var result []starlark.Value + for q := range r.ctx.Queues() { + result = append(result, queueToStarlark(q)) + } + return starlark.NewList(result), nil +} + +func queueToStarlark(q Queue) starlark.Value { + return starlarkstruct.FromStringDict(starlark.String("queue"), starlark.StringDict{ + "name": starlark.String(q.Name), + "qualified_name": starlark.String(q.QualifiedName), + "module_name": starlark.String(q.ModuleName), + // An EXPRESSION string, not a number. + "parallelism": starlark.String(q.Parallelism), + "cluster_wide": starlark.Bool(q.ClusterWide), + }) +} + // builtinXPathExpressions returns all XPath expression entries from the catalog. func (r *StarlarkRule) builtinXPathExpressions(_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { if r.ctx == nil { diff --git a/mdl/linter/starlark_scheduledevents_test.go b/mdl/linter/starlark_scheduledevents_test.go index d27e3e3b8..c4112c5cf 100644 --- a/mdl/linter/starlark_scheduledevents_test.go +++ b/mdl/linter/starlark_scheduledevents_test.go @@ -40,62 +40,127 @@ func (m *minimalReader) ListScheduledEvents() ([]*model.ScheduledEvent, error) { return nil, nil } -// TestIntervalToSeconds is a white-box test; we call it via the exported -// ScheduledEvents iterator rather than calling the unexported helper directly. -// The expected IntervalSeconds values verify all multipliers and the unknown-type fallback. -func TestIntervalToSeconds(t *testing.T) { +// TestScheduleSeconds checks the interval a rule sees, exercised through the +// exported ScheduledEvents iterator. +// +// It is derived from the Schedule child, NOT from the stored +// Interval/IntervalType pair: Studio Pro writes that pair and does not keep it +// in sync with Schedule — Workflow Commons 4.11.0 ships an event storing +// 0/"Minute" beside a DaySchedule of 01:00 — so a rule keyed on the legacy pair +// reads a daily job as firing every 0 seconds. TestScheduleSeconds_IgnoresStaleLegacyPair +// below is that exact document. +func TestScheduleSeconds(t *testing.T) { tests := []struct { - interval int - intervalType string - want int + name string + sched *model.Schedule + want int }{ - {1, "Second", 1}, - {2, "Minute", 120}, - {3, "Hour", 10800}, - {1, "Day", 86400}, - {1, "Week", 604800}, - {1, "Month", 2592000}, - {1, "Year", 31536000}, - {5, "Unknown", 0}, // unrecognised type → 0 - {5, "", 0}, // empty type → 0 - } - - containerID := model.ID("mod-1") + {"minutely x2", &model.Schedule{Kind: model.ScheduleMinute, Multiplier: 2}, 120}, + {"hourly x3", &model.Schedule{Kind: model.ScheduleHour, Multiplier: 3}, 10800}, + {"daily", &model.Schedule{Kind: model.ScheduleDay}, 86400}, + {"weekly, one day", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, false, false}}, 604800}, + // Two selected days means it fires twice a week. + {"weekly, two days", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, true, false}}, 302400}, + {"monthly", &model.Schedule{Kind: model.ScheduleMonthDate, Multiplier: 1}, 2592000}, + {"yearly", &model.Schedule{Kind: model.ScheduleYearDate}, 31536000}, + // An unstated multiplier is 1, not 0 — a 0 would read as "never". + {"multiplier defaults to 1", &model.Schedule{Kind: model.ScheduleHour}, 3600}, + {"no schedule", nil, 0}, + } + for _, tt := range tests { - reader := &minimalReader{ - listScheduledEvents: func() ([]*model.ScheduledEvent, error) { - return []*model.ScheduledEvent{{ - ContainerID: containerID, - Name: "SE", - Interval: tt.interval, - IntervalType: tt.intervalType, - Enabled: true, - }}, nil - }, - } + t.Run(tt.name, func(t *testing.T) { + if got := scheduledEventInterval(t, &model.ScheduledEvent{ + ContainerID: model.ID("mod-1"), Name: "SE", Schedule: tt.sched, Enabled: true, + }); got != tt.want { + t.Errorf("IntervalSeconds = %d, want %d", got, tt.want) + } + }) + } +} - cat, err := catalog.NewFromFile(filepath.Join(t.TempDir(), "cat.db")) - if err != nil { - t.Fatalf("NewFromFile: %v", err) - } - db := cat.CatalogDB() - if _, err := db.Exec( - `INSERT INTO modules_data (Id, Name, ProjectId, SnapshotId) VALUES (?,?,?,?)`, - string(containerID), "MyModule", "default", "s1", - ); err != nil { - t.Fatalf("insert module: %v", err) - } - cat.Close() +// TestScheduleSeconds_IgnoresStaleLegacyPair is the regression: this is the +// Workflow Commons document, whose legacy pair disagrees with its Schedule. +// Reading the pair gives 0; reading the Schedule gives a day. +func TestScheduleSeconds_IgnoresStaleLegacyPair(t *testing.T) { + got := scheduledEventInterval(t, &model.ScheduledEvent{ + ContainerID: model.ID("mod-1"), + Name: "SE_WorkflowAuditTrailRecord_CleanUp", + Interval: 0, + IntervalType: "Minute", + Schedule: &model.Schedule{Kind: model.ScheduleDay, HourOfDay: 1}, + Enabled: false, + }) + if got != 86400 { + t.Errorf("IntervalSeconds = %d, want 86400 — the legacy Interval/IntervalType pair must not be used", got) + } +} - ctx := linter.NewLintContext(cat, reader) - var got int - for se := range ctx.ScheduledEvents() { - got = se.IntervalSeconds +// TestScheduledEventExposesSchedule checks the fields a rule can branch on. +func TestScheduledEventExposesSchedule(t *testing.T) { + reader := &minimalReader{ + listScheduledEvents: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{{ + ContainerID: model.ID("mod-1"), Name: "SE", + Schedule: &model.Schedule{Kind: model.ScheduleMonthWeekday, Multiplier: 3}, + OnOverlap: "SkipNext", TimeZone: "Server", Enabled: true, + }}, nil + }, + } + ctx := linter.NewLintContext(newSingleModuleCatalog(t, model.ID("mod-1")), reader) + found := false + for se := range ctx.ScheduledEvents() { + found = true + if se.Repeat != "MonthWeekday" { + t.Errorf("Repeat = %q", se.Repeat) + } + if se.OnOverlap != "SkipNext" { + t.Errorf("OnOverlap = %q", se.OnOverlap) } - if got != tt.want { - t.Errorf("interval=%d type=%q: IntervalSeconds=%d, want %d", tt.interval, tt.intervalType, got, tt.want) + if se.TimeZone != "Server" { + t.Errorf("TimeZone = %q", se.TimeZone) } } + if !found { + t.Fatal("no scheduled event yielded") + } +} + +// scheduledEventInterval runs one event through the iterator and returns the +// interval a rule would see. +func scheduledEventInterval(t *testing.T, ev *model.ScheduledEvent) int { + t.Helper() + reader := &minimalReader{ + listScheduledEvents: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{ev}, nil + }, + } + ctx := linter.NewLintContext(newSingleModuleCatalog(t, ev.ContainerID), reader) + got := 0 + for se := range ctx.ScheduledEvents() { + got = se.IntervalSeconds + } + return got +} + +// newSingleModuleCatalog builds a catalog holding one module, so the iterator +// can resolve the container to a module name. +func newSingleModuleCatalog(t *testing.T, containerID model.ID) *catalog.Catalog { + t.Helper() + cat, err := catalog.NewFromFile(filepath.Join(t.TempDir(), "cat.db")) + if err != nil { + t.Fatalf("NewFromFile: %v", err) + } + if _, err := cat.CatalogDB().Exec( + `INSERT INTO modules_data (Id, Name, ProjectId, SnapshotId) VALUES (?,?,?,?)`, + string(containerID), "MyModule", "default", "s1", + ); err != nil { + t.Fatalf("insert module: %v", err) + } + cat.Close() + return cat } func TestScheduledEvents_MicroflowNameResolution(t *testing.T) { @@ -169,11 +234,10 @@ func TestScheduledEvents_ExcludedModules(t *testing.T) { reader := &minimalReader{ listScheduledEvents: func() ([]*model.ScheduledEvent, error) { return []*model.ScheduledEvent{{ - ContainerID: containerID, - Name: "ExcludedSE", - Interval: 1, - IntervalType: "Day", - Enabled: true, + ContainerID: containerID, + Name: "ExcludedSE", + Schedule: &model.Schedule{Kind: model.ScheduleDay}, + Enabled: true, }}, nil }, } @@ -211,8 +275,8 @@ func TestScheduledEvents_IncludedModules(t *testing.T) { reader := &minimalReader{ listScheduledEvents: func() ([]*model.ScheduledEvent, error) { return []*model.ScheduledEvent{ - {ContainerID: modA, Name: "SE_A", Interval: 1, IntervalType: "Hour", Enabled: true}, - {ContainerID: modB, Name: "SE_B", Interval: 1, IntervalType: "Hour", Enabled: true}, + {ContainerID: modA, Name: "SE_A", Schedule: &model.Schedule{Kind: model.ScheduleHour, Multiplier: 1}, Enabled: true}, + {ContainerID: modB, Name: "SE_B", Schedule: &model.Schedule{Kind: model.ScheduleHour, Multiplier: 1}, Enabled: true}, }, nil }, } @@ -289,11 +353,15 @@ func TestStarlarkScheduledEventsBuiltin(t *testing.T) { reader := &minimalReader{ listScheduledEvents: func() ([]*model.ScheduledEvent, error) { return []*model.ScheduledEvent{{ - ContainerID: containerID, - Name: "DailySE", - MicroflowID: mfID, + ContainerID: containerID, + Name: "DailySE", + MicroflowID: mfID, + // The interval a rule sees comes from Schedule. The legacy pair + // is left deliberately inconsistent here (it claims 2 days) to + // prove the builtin does not read it. Interval: 2, IntervalType: "Day", + Schedule: &model.Schedule{Kind: model.ScheduleHour, Multiplier: 2}, Enabled: true, }}, nil }, @@ -343,7 +411,7 @@ func TestStarlarkScheduledEventsBuiltin(t *testing.T) { for _, want := range []string{ "se Billing.DailySE", "mf Billing.SUB_DailyJob", - "secs 172800", // 2 * 86400 + "secs 7200", // 2 hours, from the Schedule — NOT the 172800 the legacy pair implies "enabled yes", } { if !strings.Contains(joined, want) { diff --git a/mdl/visitor/visitor_catalog_test.go b/mdl/visitor/visitor_catalog_test.go index b8599c969..8eba1f62b 100644 --- a/mdl/visitor/visitor_catalog_test.go +++ b/mdl/visitor/visitor_catalog_test.go @@ -68,6 +68,18 @@ func TestSelectFromCatalog(t *testing.T) { {"communities keyword table", "SELECT * FROM CATALOG.COMMUNITIES;"}, {"communities lowercase", "select * from catalog.communities;"}, {"communities with where", "SELECT AssetName FROM CATALOG.COMMUNITIES WHERE CommunityId = 1;"}, + // Same trap, hit again when QUEUES became a lexer keyword (SHOW QUEUES): + // CATALOG.QUEUES stopped parsing and the SELECT produced no statement and + // no output — no error, just silence. Every new plural keyword that names + // a catalog table has to be added to catalogTableName. + {"queues keyword table", "SELECT * FROM CATALOG.QUEUES;"}, + {"queues lowercase", "select * from catalog.queues;"}, + {"queues with where", "SELECT Name FROM CATALOG.QUEUES WHERE ClusterWide = 1;"}, + // SCHEDULED_EVENTS lexes as one IDENTIFIER (the underscore joins it), so + // the SCHEDULED keyword does not reach it — asserted so a future lexer + // change that splits it is caught here. + {"scheduled events table", "SELECT * FROM CATALOG.SCHEDULED_EVENTS;"}, + {"scheduled events with where", "SELECT Name FROM CATALOG.SCHEDULED_EVENTS WHERE Enabled = 1;"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { From 43e5624caad21da137c625021452e426994b96e2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 08:54:38 +0000 Subject: [PATCH 31/35] Copy every file a package ships, not just its widgets The CE6083 design-property errors left over from DataWidgets 3.5.0 -> 3.11.3 were diagnosed in the previous commit as a cross-module dependency on a newer Atlas. That was wrong, and the correction is the point of this one. DataWidgets ships its OWN themesource/datawidgets/web/design-properties.json. The properties CE6083 named -- Grid spacing, Hover, Pagination on Gallery -- are declared there, not by Atlas, and the project still held the 3.5.0 copy. Updating Atlas would have changed nothing. `mx rename-design-properties` renaming 0 was the tell: it renames properties between Atlas generations, so nothing to rename means the declaration was absent rather than moved. This was the third instance of one gap, presenting as three problems: the model was copied, then widgets were added because the binaries were stale, and themesource was still missing. Enumerating the directories that seem to matter is what produced that sequence. InstallPackageFiles copies everything the package contains, excluding only project.mpr and package.xml -- manifest rather than payload -- with a guard against paths escaping the project. Measured: DataWidgets 3.5.0 -> 3.11.3 replaces 49 bundled files, and after `mx update-widgets` the app contains 0 errors. Administration 4.3.2 -> 4.5.0 remains at 0. Both subjects now check clean. The post-update guidance drops the claim that a check may surface dependency errors, since that came from the wrong diagnosis. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/cmd_marketplace_update.go | 11 ++- cmd/mxcli/marketplace/update.go | 100 ++++++++++++++++------------ docs-site/src/guides/marketplace.md | 4 ++ 4 files changed, 67 insertions(+), 49 deletions(-) diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 2abc04436..67f4340e1 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -469,3 +469,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A .mpr extracted from a `.mpk` (or copied anywhere beside an existing project) reads with the wrong unit contents: edits appear to succeed, `ListRawUnitsByType` returns plausible documents, and a write lands nowhere the file will carry. Downstream the operation fails exactly as if nothing had been attempted | The MPR format test is **adjacency, not content**: an `.mpr` is treated as v2 when an `mprcontents/` directory sits next to it (`sdk/mpr/reader.go` `OpenWithOptions`). A package's v1 `project.mpr` unpacked into a work directory that already holds a scratch v2 project is therefore read as v2, and every unit resolves against the *other* project's `.mxunit` files | `cmd/mxcli/marketplace/theme.go` (`makeImportable` unpacks into its own `os.MkdirTemp` subdirectory) | **Unpack any foreign .mpr into a directory of its own** — never beside a project, even a throwaway one. **The failure is silent in both directions**: the read returns real-looking documents (from the wrong model) and the write reports success, so nothing surfaces until a later step refuses. The tell is a fix that provably works in isolation (`mx module-import` accepted the hand-built package) but does nothing in situ — that gap is environment, not logic. Test `TestMakeImportable_UnpacksAwayFromTheScratchProject` asserts the unpack does not land in the work-directory root | | `show features` lists nothing for agents or MCP, so `create agent` / `create model` on a pre-11.9 project runs without complaint. Reported alongside it: "`CREATE MODEL` can only author one provider — the writer assigns `MxCloudGenAI` unconditionally" | Two different things, and only the first was real. **The version gap**: the `agent_documents` area did not exist in `sdk/versions/mendix-11.yaml`, so `checkFeature()` had nothing to consult. **The provider claim was wrong**: the writer's assignment is `if m.Provider == ""` — a default, not an override. `Provider: OpenAI` parses, writes and round-trips through `describe model`; so does `Provider: TotallyMadeUp` | `sdk/versions/mendix-11.yaml` (new `agent_documents` area), `mdl/executor/cmd_agenteditor_models.go` + `cmd_agenteditor_write.go` (four `checkFeature` gates), `.claude/skills/mendix/agents.md` + `version-awareness.md` | **Read the assignment's guard before believing "hardcoded"** — `if x == "" { x = default }` and `x = default` are one character apart in a grep and opposite in meaning; the round-trip test settles it in a minute. **Agent doctypes are the one version gate with no downstream safety net**: the documents are custom blobs, mxbuild contains no agent-editor strings at all, so an ungated project builds green and fails only when Studio Pro opens it. **Do not invent the provider allowlist** — the enum lives in a Studio Pro *extension*, not in `generated/metamodel` and not in mxbuild, so a guessed list would reject values Mendix accepts; document that nothing validates it instead. Test `TestAgentDocumentsAreGated`. Reported in mxcli-formula1 FINDINGS §53 | | A marketplace module update leaves the app running the OLD widget code: `show modules` reports the new version, pages reference new widget definitions, and the binaries in `widgets/` are unchanged. Symptoms are whatever the version gap causes at render time, with no build error naming the cause | The update moved only the **model**. A widget module's `.mpk` ships its widget binaries under `widgets/` inside the package, and copying units out of a reference project never touches them. Measured on DataWidgets 3.5.0 → 3.11.3: model updated cleanly, all 10 widget binaries still 3.5.0 (`Datagrid.mpk` project=216193 vs package=166933) | `cmd/mxcli/marketplace/update.go` (`InstallPackageWidgets`, called from `PerformUpdate`) | **Take the widget files from the `.mpk`, not from the reference project** — the reference is a blank app plus the module, so its `widgets/` also holds the template's widgets and copying those overwrites widgets the update has nothing to do with. **Skip zero-size directory entries** in the zip or you write a stray file (the 3.11.3 package has 10 `widgets/` entries and 9 real files). **A module with no widgets cannot expose this**: Administration updated end-to-end and looked completely correct, which is why the gap survived a full slice. Pick a second subject with a different shape before believing an update path works. Related but distinct: after any headless module install `mx update-widgets` is needed (CE0463); and a newer module can want a newer Atlas than the project has (29 × CE6083 on DataWidgets 3.11.3), which `rename-design-properties` does NOT fix — that is a dependency, not a resync | +| After a marketplace module update, `mx check` reports `CE6083` "Design property X is not supported by your theme" on the updated module's own widgets, and it survives BOTH `mx update-widgets` and `mx rename-design-properties` (which renames 0) | The update copied only the model and `widgets/`. The design properties are declared in the module's **own** `themesource//web/design-properties.json`, which was still the old version's copy — so the model referenced properties the shipped theme file did not define | `cmd/mxcli/marketplace/update.go` (`InstallPackageFiles` replaces `InstallPackageWidgets`) | **Copy everything the package ships**, excluding only `project.mpr` and `package.xml` — enumerating the directories that seem to matter is what produced two bugs in a row (widgets, then themesource). **CE6083 surviving `rename-design-properties` is the tell**: that command renames properties between Atlas generations, so 0 renamed means the declaration is missing entirely rather than renamed. **Do not diagnose this as a cross-module Atlas dependency** — that was the wrong call here; the declaring module was the one being updated. Guard against `..` in package paths while copying. Measured: DataWidgets 3.5.0 → 3.11.3, 49 files replaced, 29 errors → 0 | diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index c16d72505..fa7b5515d 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -216,8 +216,8 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { fmt.Fprintf(out, "\n%s updated %s → %s\n", r.Module, r.FromVersion, r.ToVersion) fmt.Fprintf(out, " %d units copied, %d element identities preserved, %d role grant(s) restored.\n", r.UnitsCopied, r.IdentitiesKept, r.GrantsRestored) - if len(r.WidgetsInstalled) > 0 { - fmt.Fprintf(out, " %d widget package(s) replaced in widgets/.\n", len(r.WidgetsInstalled)) + if len(r.FilesInstalled) > 0 { + fmt.Fprintf(out, " %d bundled file(s) replaced (widgets, themesource, ...).\n", len(r.FilesInstalled)) } if len(r.IdentitiesLost) > 0 { @@ -242,12 +242,9 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { fmt.Fprintln(out, "\n Next: resync widget definitions, or 'mx check' will report CE0463 on the") fmt.Fprintln(out, " new version's pages (this is expected after any headless module install):") fmt.Fprintln(out, " mx update-widgets ") - fmt.Fprintln(out, "\n Then check the app. A newer module can need newer companions — measured on") - fmt.Fprintln(out, " DataWidgets 3.11.3, whose widgets want design properties an older Atlas does") - fmt.Fprintln(out, " not define (29 × CE6083). That is a dependency to resolve, not something") - fmt.Fprintln(out, " this update can fix:") + fmt.Fprintln(out, "\n Then validate and review:") fmt.Fprintln(out, " mxcli docker check -p ") - fmt.Fprintln(out, "\n Review the change with 'mxcli diff-local'.") + fmt.Fprintln(out, " mxcli diff-local -p ") } func init() { diff --git a/cmd/mxcli/marketplace/update.go b/cmd/mxcli/marketplace/update.go index 1b30504d1..8c016f4fe 100644 --- a/cmd/mxcli/marketplace/update.go +++ b/cmd/mxcli/marketplace/update.go @@ -95,16 +95,16 @@ func safeFileName(k ElementKey) string { // UpdateResult is what an update did. type UpdateResult struct { - Module string - FromVersion string - ToVersion string - UnitsCopied int - IdentitiesKept int - IdentitiesLost []string - GrantsRestored int - GrantsDropped []string - WidgetsInstalled []string - ForcedOverEdits []string + Module string + FromVersion string + ToVersion string + UnitsCopied int + IdentitiesKept int + IdentitiesLost []string + GrantsRestored int + GrantsDropped []string + FilesInstalled []string + ForcedOverEdits []string } // PerformUpdate replaces an installed module with the copy in referenceMpr, @@ -152,21 +152,21 @@ func PerformUpdate(mprPath, referenceMpr, targetMpk, moduleName, fromVersion, to if err := StampMarketplaceVersion(mprPath, moduleName, toVersion, toVersionID); err != nil { return nil, fmt.Errorf("record the installed version: %w", err) } - widgets, err := InstallPackageWidgets(targetMpk, filepath.Dir(mprPath)) + files, err := InstallPackageFiles(targetMpk, filepath.Dir(mprPath)) if err != nil { - return nil, fmt.Errorf("install the new version's widgets: %w", err) + return nil, fmt.Errorf("install the new version's bundled files: %w", err) } return &UpdateResult{ - Module: moduleName, - FromVersion: fromVersion, - ToVersion: toVersion, - UnitsCopied: copied, - IdentitiesKept: applied, - IdentitiesLost: missing, - GrantsRestored: restored, - GrantsDropped: dropped, - WidgetsInstalled: widgets, + Module: moduleName, + FromVersion: fromVersion, + ToVersion: toVersion, + UnitsCopied: copied, + IdentitiesKept: applied, + IdentitiesLost: missing, + GrantsRestored: restored, + GrantsDropped: dropped, + FilesInstalled: files, }, nil } @@ -250,20 +250,30 @@ func setBoolField(doc bson.D, key string, value bool) { } } -// InstallPackageWidgets copies a package's bundled widget .mpk files into the -// project's widgets/ folder, and reports what it wrote. +// InstallPackageFiles copies every non-model file a package ships into the +// project, and reports what it wrote. // -// A module update that moves only the model is wrong for any module shipping -// widgets. Measured on DataWidgets 3.5.0 → 3.11.3: the model updated cleanly and -// all ten widget binaries were still the old version, so the project claimed -// 3.11.3 while running 3.5.0's widget code. Administration has no widgets, which -// is exactly why the first end-to-end run did not catch it. +// A module is not only its model. The .mpk carries widget binaries under +// widgets/, styling and design-property declarations under themesource/, and +// whatever else the module needs; only project.mpr and package.xml are +// manifest rather than payload. An update that moves the model alone leaves all +// of it at the old version. // -// The files come from the package rather than from the reference project, -// because the reference is a blank app plus the module and its widgets/ folder -// therefore also holds the template's widgets — copying those would overwrite -// widgets this update has nothing to do with. -func InstallPackageWidgets(mpkPath, projectDir string) (written []string, err error) { +// Both halves of that were measured on DataWidgets 3.5.0 → 3.11.3, one after the +// other: +// +// - all ten widget binaries stayed at 3.5.0 while the model said 3.11.3, so +// the app ran old widget code with new definitions; +// - and 29 × CE6083 ("design property not supported by your theme") persisted +// through `mx update-widgets` AND `mx rename-design-properties`, because the +// properties Gallery wants are declared in the module's *own* +// themesource/datawidgets/web/design-properties.json, which was still the +// 3.5.0 copy. +// +// The second looked like a cross-module dependency on a newer Atlas. It was not. +// Copying everything the package ships, rather than enumerating the directories +// that seem to matter, is what stops there being a third instance. +func InstallPackageFiles(mpkPath, projectDir string) (written []string, err error) { zr, err := zip.OpenReader(mpkPath) if err != nil { return nil, fmt.Errorf("open package %s: %w", filepath.Base(mpkPath), err) @@ -271,19 +281,25 @@ func InstallPackageWidgets(mpkPath, projectDir string) (written []string, err er defer zr.Close() for _, f := range zr.File { - if !strings.HasPrefix(f.Name, "widgets/") || f.FileInfo().IsDir() { + if f.FileInfo().IsDir() { continue } - base := filepath.Base(f.Name) - if base == "" || base == "." { - continue + switch f.Name { + case packageProjectEntry, "package.xml": + continue // the model and its manifest, handled by the transplant } - dstDir := filepath.Join(projectDir, "widgets") - if err := os.MkdirAll(dstDir, 0o755); err != nil { - return written, fmt.Errorf("create widgets directory: %w", err) + // Refuse a path that escapes the project. Nothing in a Mendix package + // should contain "..", and honouring one would let a package write + // anywhere on disk. + clean := filepath.Clean(f.Name) + if strings.HasPrefix(clean, "..") || filepath.IsAbs(clean) { + return written, fmt.Errorf("package entry %q would write outside the project", f.Name) } - dst := filepath.Join(dstDir, base) + dst := filepath.Join(projectDir, clean) + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return written, fmt.Errorf("create %s: %w", filepath.Dir(dst), err) + } rc, oerr := f.Open() if oerr != nil { return written, fmt.Errorf("read %s from the package: %w", f.Name, oerr) @@ -296,7 +312,7 @@ func InstallPackageWidgets(mpkPath, projectDir string) (written []string, err er if err := os.WriteFile(dst, body, 0o644); err != nil { return written, fmt.Errorf("write %s: %w", dst, err) } - written = append(written, base) + written = append(written, clean) } sort.Strings(written) return written, nil diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index 1da35dd5e..9bbca9a68 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -134,9 +134,11 @@ Administration updated 4.3.2 → 4.5.0 - **Element identity.** The runtime keys entities and their attributes on the model's `GUID`, so a module whose documents are replaced without carrying the old GUIDs is a *different* module to the database and its tables are dropped on the next deploy. Studio Pro transplants them; so does this. - **Access.** A user role's grant of a module role lives in the project's security document, not the module, so removing the module takes the grants with it and putting it back does not return them. +- **Everything else the package ships.** A module is not only its model: the `.mpk` carries widget binaries under `widgets/`, styling and design-property declarations under `themesource/`, and whatever else it needs. All of it is replaced — only `project.mpr` and `package.xml` are manifest rather than payload. DataWidgets 3.11.3 replaces 49 such files, and skipping them leaves the app running old widget code and reporting `CE6083` for design properties the module itself declares. It does not use `mx module-import`, which would rewrite an MPR v2 project as v1 and refuses theme modules. Units are copied with mxcli's own writer, so the project keeps its format. + ### Local edits Local edits are **not** preserved. `update` refuses when it finds any, `--save-edits` writes them out first, and `--force` proceeds. Two limits on the saved files: @@ -148,6 +150,8 @@ Local edits are **not** preserved. `update` refuses when it finds any, `--save-e Run `mx update-widgets `. A newer module's pages reference widget definitions the project has not resynced, so `mx check` reports CE0463 until told to — measured on Administration 4.3.2 → 4.5.0: 11 errors before, 0 after. This is expected after any headless module install, not a fault in the update. +Then `mxcli docker check -p ` and `mxcli diff-local -p `. Measured after that resync: Administration 4.3.2 → 4.5.0 and DataWidgets 3.5.0 → 3.11.3 both reach **0 errors**. + `update` does **not** roll back. Work on a copy or have the project in version control. ## Has this module been edited? (`marketplace diff`) From 8860e7c830d470bfc5519dc14d6957f2f16d23f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 12:34:34 +0000 Subject: [PATCH 32/35] Install modules with mxcli's writer instead of mx module-import MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Installing a NEW module was refused on any MPR v2 project, because the only path was `mx module-import` and that rewrites v2 as v1. Since a vanilla `mxcli new` app is v2 and every module is a new install there, the headless install path was effectively closed for exactly the case it matters in -- the guard was right and the capability was missing. Slice 3 already built the missing piece. Installing is the update path minus the drop and minus identity capture: build a reference project from the package, copy the module's units in with mxcli's own writer, stamp the marketplace version, install the bundled files. The format is preserved because the writer handles v1 and v2 alike, and theme modules work because the reference builder clears the flag module-import refuses on. Measured on a vanilla 11.12.1 app: CommunityCommons 11.5.1 installs as 128 units and 126 bundled files, mprcontents/ grows from 369 to 497 .mxunit files -- so the project is still v2 -- and after `mx update-widgets` the app contains 0 errors. --allow-format-change still selects the legacy module-import path for anyone who wants exact mx semantics, and the v2 guard still covers it. This is what unblocks installing the Agents Kit 2 modules headlessly, including Conversational UI, which mxcli-formula1 FINDINGS §53 recorded as impossible by any headless path because module-import rejects theme modules outright. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- cmd/mxcli/cmd_marketplace_install.go | 56 +++++++++++++++++++---- cmd/mxcli/cmd_marketplace_install_test.go | 4 ++ cmd/mxcli/marketplace/update.go | 30 ++++++++++++ docs-site/src/guides/marketplace.md | 10 ++-- 4 files changed, 87 insertions(+), 13 deletions(-) diff --git a/cmd/mxcli/cmd_marketplace_install.go b/cmd/mxcli/cmd_marketplace_install.go index 680be1932..d0c195d7f 100644 --- a/cmd/mxcli/cmd_marketplace_install.go +++ b/cmd/mxcli/cmd_marketplace_install.go @@ -15,6 +15,7 @@ import ( modelsdk "github.com/mendixlabs/mxcli" "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + mp "github.com/mendixlabs/mxcli/cmd/mxcli/marketplace" "github.com/mendixlabs/mxcli/internal/marketplace" "github.com/spf13/cobra" ) @@ -26,18 +27,21 @@ var marketplaceInstallCmd = &cobra.Command{ Install is type-aware: - Widget copied into the project's widgets/ folder (overwrites on update) - - Module imported via 'mx module-import' (new modules only) + - Module copied in with mxcli's own writer (new modules only) - other types downloaded to disk with import instructions Updating a module that is already present is NOT done automatically: it could discard local edits and, for modules with persistent entities, change entity IDs (which loses data). Such updates are reported and left to Studio Pro. -Module import into an MPR v2 project is refused. 'mx module-import' rewrites a -v2 project as v1 — mprcontents/ is collapsed into a single binary .mpr — which -loses the per-document files 'mxcli diff-local' and git review depend on, and -the conversion is one-way. Import in Studio Pro, or pass --allow-format-change -to accept it.`, +A module is installed by copying its units with mxcli's own writer rather than +by running 'mx module-import'. That preserves the project's storage format — +module-import rewrites MPR v2 as v1, collapsing mprcontents/ into a single +binary .mpr, one-way — and it works for theme modules, which module-import +refuses outright. Everything the package ships (widgets, themesource, ...) is +installed alongside the model. + +--allow-format-change selects the legacy module-import path instead.`, Example: ` mxcli marketplace install 20 -p app.mpr mxcli marketplace install 2888 --version 7.0.3 -p app.mpr`, Args: cobra.ExactArgs(1), @@ -49,7 +53,7 @@ func init() { marketplaceInstallCmd.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") marketplaceInstallCmd.Flags().String("version", "", "version number to install (default: latest)") marketplaceInstallCmd.Flags().Bool("allow-format-change", false, - "permit the import to rewrite an MPR v2 project as v1 (one-way; loses mprcontents/)") + "use the legacy 'mx module-import' path, which rewrites an MPR v2 project as v1 (one-way)") _ = marketplaceInstallCmd.MarkFlagRequired("project") marketplaceCmd.AddCommand(marketplaceInstallCmd) @@ -162,8 +166,20 @@ func installModule(ctx context.Context, client *marketplace.Client, v *marketpla return nil } - if err := checkStorageFormatPreserved(mprPath, allowFormatChange); err != nil { - return err + // Default path: copy the module in with mxcli's own writer, which preserves + // the project's storage format and works for theme modules. --allow-format-change + // selects the legacy `mx module-import`, which does neither. + if !allowFormatChange { + res, ierr := installByTransplant(ctx, mpkPath, mprPath, moduleName, mendixVer, v) + if ierr != nil { + return ierr + } + fmt.Fprintf(out, "Installed module %q version %s into %s\n", + moduleName, v.VersionNumber, filepath.Base(mprPath)) + fmt.Fprintf(out, " %d units copied, %d bundled file(s) installed.\n", + res.UnitsCopied, len(res.FilesInstalled)) + fmt.Fprintln(out, "\n Next: 'mx update-widgets ', then 'mxcli docker check -p '.") + return nil } mxPath, err := docker.ResolveMxForVersion("", mendixVer) @@ -182,6 +198,28 @@ func installModule(ctx context.Context, client *marketplace.Client, v *marketpla return nil } +// installByTransplant builds a reference project from the package and copies the +// module out of it, so the destination keeps its MPR format. +func installByTransplant(ctx context.Context, mpkPath, mprPath, moduleName, mendixVer string, + v *marketplace.Version) (*mp.UpdateResult, error) { + + work, err := os.MkdirTemp("", "mxinstall") + if err != nil { + return nil, err + } + defer os.RemoveAll(work) + + refDir := filepath.Join(work, "ref") + if err := os.MkdirAll(refDir, 0o755); err != nil { + return nil, err + } + refMpr, err := mp.PackageProject(ctx, mpkPath, mendixVer, refDir, newBackendFactory()) + if err != nil { + return nil, fmt.Errorf("build a reference project from the package: %w", err) + } + return mp.PerformInstall(mprPath, refMpr, mpkPath, moduleName, v.VersionNumber, v.VersionID, newBackendFactory()) +} + // isMPRv2 reports whether the project at mprPath uses the MPR v2 storage format: // a small .mpr holding metadata beside an mprcontents/ tree of one .mxunit file // per document. The presence of that directory is how the readers themselves diff --git a/cmd/mxcli/cmd_marketplace_install_test.go b/cmd/mxcli/cmd_marketplace_install_test.go index d434bcb4b..57ff7a8e4 100644 --- a/cmd/mxcli/cmd_marketplace_install_test.go +++ b/cmd/mxcli/cmd_marketplace_install_test.go @@ -77,6 +77,10 @@ func TestModuleNameFromMpk_NoPackageXML(t *testing.T) { // TestCheckStorageFormatPreserved_RefusesV2 is the guard for the silent MPR // v2→v1 collapse. // +// Still enforced, but no longer on the default path: a module install now copies +// units with mxcli's own writer and preserves the format, so this guards only +// the legacy --allow-format-change route through `mx module-import`. +// // `mx module-import` rewrites a v2 project as v1: measured on a blank Mendix // 11.12.1 app, one import turned a 69 KB .mpr plus 341 .mxunit files into a // single 14 MB blob with no mprcontents/. That destroys the per-document files diff --git a/cmd/mxcli/marketplace/update.go b/cmd/mxcli/marketplace/update.go index 8c016f4fe..4fad86c31 100644 --- a/cmd/mxcli/marketplace/update.go +++ b/cmd/mxcli/marketplace/update.go @@ -317,3 +317,33 @@ func InstallPackageFiles(mpkPath, projectDir string) (written []string, err erro sort.Strings(written) return written, nil } + +// PerformInstall adds a module that is not yet in the project, copying it from +// referenceMpr and installing everything the package ships. +// +// It is PerformUpdate without the parts that only make sense for a replace: +// nothing to capture, nothing to drop. The reason it exists separately from +// `mx module-import` is what makes it worth having — it preserves the project's +// storage format, where module-import rewrites MPR v2 as v1, and it works for +// theme modules, which module-import refuses outright. +func PerformInstall(mprPath, referenceMpr, packageMpk, moduleName, version, versionID string, + newBackend func() backend.FullBackend) (*UpdateResult, error) { + + copied, err := TransplantModule(referenceMpr, mprPath, moduleName) + if err != nil { + return nil, fmt.Errorf("copy the module in: %w", err) + } + if err := StampMarketplaceVersion(mprPath, moduleName, version, versionID); err != nil { + return nil, fmt.Errorf("record the installed version: %w", err) + } + files, err := InstallPackageFiles(packageMpk, filepath.Dir(mprPath)) + if err != nil { + return nil, fmt.Errorf("install the package's bundled files: %w", err) + } + return &UpdateResult{ + Module: moduleName, + ToVersion: version, + UnitsCopied: copied, + FilesInstalled: files, + }, nil +} diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index 9bbca9a68..92033b52a 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -61,11 +61,11 @@ mxcli marketplace install 2888 --version 7.0.3 -p app.mpr # a module | Content type | What `install` does | |---|---| | **Widget** | Copies the `.mpk` into the project's `widgets/` folder (overwrites on update). Reload in Studio Pro or run `mx update-widgets` to pick it up. | -| **Module** (new) | Imports it via `mx module-import` (requires a matching mxbuild — run `mxcli setup mxbuild -p app.mpr` if missing). **Refused on an MPR v2 project** — see below. | +| **Module** (new) | Copies the module in with mxcli's own writer, preserving the project's MPR format, plus everything the package ships (widgets, themesource, ...). Requires a matching mxbuild — run `mxcli setup mxbuild -p app.mpr` if missing. | | **Module** (already present) | **Reported, not modified** — see below. | | Theme / Starter App / Sample | Downloaded to disk with import instructions (import via Studio Pro). | -## Module import is refused on MPR v2 projects +## Why installs do not use `mx module-import` `mx module-import` rewrites an MPR v2 project as v1. Measured on a blank Mendix 11.12.1 app, a single import turned a 69 KB `.mpr` plus 341 `.mxunit` files into one 14 MB SQLite blob with no `mprcontents/` — and the same was observed independently on 11.13.0, so it is not version-specific: @@ -76,7 +76,9 @@ after .mpr 14,295,040 bytes + 0 .mxunit tables: Unit, _MetaData That is not cosmetic. The v2 layout is what makes the model diffable and mergeable per document: it is what [`mxcli diff-local`](../tools/diff.md) reads, and what makes an idempotent re-run observable as "no files changed". The conversion is **one-way** — `mx convert` targets Mendix *versions*, not storage formats. -So `install` refuses rather than warning: +So `install` copies the module's units directly instead, which keeps the project in whatever format it already uses and also works for theme modules (`module-import` refuses those outright). Measured: CommunityCommons 11.5.1 into a vanilla 11.12.1 app — 128 units and 126 bundled files, `mprcontents/` grew from 369 to 497 `.mxunit` files, and `mx check` reports 0 errors. + +`--allow-format-change` selects the legacy `module-import` path, which still refuses to run silently: ```text refusing to import: app.mpr uses the MPR v2 storage format, and 'mx module-import' @@ -86,7 +88,7 @@ would rewrite it as v1. - pass --allow-format-change to accept the conversion to MPR v1. ``` -Import the module in **Studio Pro**, which preserves the format. If your project is not kept in git and the v1 layout is fine, `--allow-format-change` accepts the conversion — and the command then states plainly that the project is now v1. +If you take that route the command states plainly that the project is now v1. ## Updating an existing module From 93e21822a0c7b403696731b93fcdac0ea7935e20 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 12:46:35 +0000 Subject: [PATCH 33/35] Document the marketplace install/update lifecycle in a skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The marketplace skill stopped at "install a new module; for updates, use Studio Pro", which is two commands out of date. It now covers discover → download → install → diff → update, with the measurements behind each step and the limits stated up front (no dependency resolution, no rollback, local edits not preserved). It also corrects a hint that was actively destructive. After a headless module install the project's widget definitions are stale and a check reports CE0463, and both marketplace commands told the user to fix that with bare `mx update-widgets`. That command does perform the resync, and rewrites an MPR v2 project as v1 while doing it — measured on 11.12.1, 200 .mxunit files to 0 and a 69,632-byte index to 14 MB. The same collapse mxcli refuses to perform through `mx module-import` was being recommended in prose. `mxcli docker check` is the v2-safe route: it runs the same step under a storage-format snapshot, so the check sees the resynced model and the project keeps its format (verified: 200 units and 69,632 bytes before and after). That also means the resync is not persisted, which the docs now say rather than implying it sticks. --- .claude/skills/fix-issue.md | 1 + .../mendix/download-marketplace-content.md | 240 ++++++++++++++++-- cmd/mxcli/cmd_marketplace_install.go | 8 +- cmd/mxcli/cmd_marketplace_update.go | 10 +- docs-site/src/guides/marketplace.md | 8 +- 5 files changed, 237 insertions(+), 30 deletions(-) diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 67f4340e1..4a7b0c8a8 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -470,3 +470,4 @@ extracting `OffsetExpression`/`LimitExpression`. | `show features` lists nothing for agents or MCP, so `create agent` / `create model` on a pre-11.9 project runs without complaint. Reported alongside it: "`CREATE MODEL` can only author one provider — the writer assigns `MxCloudGenAI` unconditionally" | Two different things, and only the first was real. **The version gap**: the `agent_documents` area did not exist in `sdk/versions/mendix-11.yaml`, so `checkFeature()` had nothing to consult. **The provider claim was wrong**: the writer's assignment is `if m.Provider == ""` — a default, not an override. `Provider: OpenAI` parses, writes and round-trips through `describe model`; so does `Provider: TotallyMadeUp` | `sdk/versions/mendix-11.yaml` (new `agent_documents` area), `mdl/executor/cmd_agenteditor_models.go` + `cmd_agenteditor_write.go` (four `checkFeature` gates), `.claude/skills/mendix/agents.md` + `version-awareness.md` | **Read the assignment's guard before believing "hardcoded"** — `if x == "" { x = default }` and `x = default` are one character apart in a grep and opposite in meaning; the round-trip test settles it in a minute. **Agent doctypes are the one version gate with no downstream safety net**: the documents are custom blobs, mxbuild contains no agent-editor strings at all, so an ungated project builds green and fails only when Studio Pro opens it. **Do not invent the provider allowlist** — the enum lives in a Studio Pro *extension*, not in `generated/metamodel` and not in mxbuild, so a guessed list would reject values Mendix accepts; document that nothing validates it instead. Test `TestAgentDocumentsAreGated`. Reported in mxcli-formula1 FINDINGS §53 | | A marketplace module update leaves the app running the OLD widget code: `show modules` reports the new version, pages reference new widget definitions, and the binaries in `widgets/` are unchanged. Symptoms are whatever the version gap causes at render time, with no build error naming the cause | The update moved only the **model**. A widget module's `.mpk` ships its widget binaries under `widgets/` inside the package, and copying units out of a reference project never touches them. Measured on DataWidgets 3.5.0 → 3.11.3: model updated cleanly, all 10 widget binaries still 3.5.0 (`Datagrid.mpk` project=216193 vs package=166933) | `cmd/mxcli/marketplace/update.go` (`InstallPackageWidgets`, called from `PerformUpdate`) | **Take the widget files from the `.mpk`, not from the reference project** — the reference is a blank app plus the module, so its `widgets/` also holds the template's widgets and copying those overwrites widgets the update has nothing to do with. **Skip zero-size directory entries** in the zip or you write a stray file (the 3.11.3 package has 10 `widgets/` entries and 9 real files). **A module with no widgets cannot expose this**: Administration updated end-to-end and looked completely correct, which is why the gap survived a full slice. Pick a second subject with a different shape before believing an update path works. Related but distinct: after any headless module install `mx update-widgets` is needed (CE0463); and a newer module can want a newer Atlas than the project has (29 × CE6083 on DataWidgets 3.11.3), which `rename-design-properties` does NOT fix — that is a dependency, not a resync | | After a marketplace module update, `mx check` reports `CE6083` "Design property X is not supported by your theme" on the updated module's own widgets, and it survives BOTH `mx update-widgets` and `mx rename-design-properties` (which renames 0) | The update copied only the model and `widgets/`. The design properties are declared in the module's **own** `themesource//web/design-properties.json`, which was still the old version's copy — so the model referenced properties the shipped theme file did not define | `cmd/mxcli/marketplace/update.go` (`InstallPackageFiles` replaces `InstallPackageWidgets`) | **Copy everything the package ships**, excluding only `project.mpr` and `package.xml` — enumerating the directories that seem to matter is what produced two bugs in a row (widgets, then themesource). **CE6083 surviving `rename-design-properties` is the tell**: that command renames properties between Atlas generations, so 0 renamed means the declaration is missing entirely rather than renamed. **Do not diagnose this as a cross-module Atlas dependency** — that was the wrong call here; the declaring module was the one being updated. Guard against `..` in package paths while copying. Measured: DataWidgets 3.5.0 → 3.11.3, 49 files replaced, 29 errors → 0 | +| A project that was MPR v2 is v1 after a marketplace install or update: `mprcontents/` is empty or gone, the `.mpr` jumped to tens of MB, `mxcli diff-local` stops working. The user followed the command's own "Next:" hint, or the docs | The hint said to run bare `mx update-widgets ` to clear the CE0463 a headless install leaves behind. That command performs the resync **and** rewrites a v2 project as v1 — measured on 11.12.1: 200 `.mxunit` files → 0, a 69,632-byte index → 14,405,632 bytes. Same family as the `mx module-import` collapse, arriving through advice rather than through a call | `cmd/mxcli/cmd_marketplace_install.go` + `cmd_marketplace_update.go` (the "Next:" lines now name `mxcli docker check`), `docs-site/src/guides/marketplace.md`, `.claude/skills/mendix/download-marketplace-content.md` | **`mxcli docker check` is the v2-safe resync** — it runs the same `mx update-widgets` under a storage-format snapshot (`cmd/mxcli/docker/update_widgets.go`, #808), so the check sees the resynced model and the project keeps its format. Verified: 200 `.mxunit` / 69,632 bytes before and after. **The resync is then not persisted** — the stored model still holds the old widget definitions and a later `mx check` reports CE0463 again; Studio Pro's "Update all widgets" persists it in v2, and `mxcli widget sync` is the partial headless equivalent (7 of 40 on the reference fixture). Say that rather than implying the resync sticks. **A guard in the code does not cover a string in the help text**: #808 already protected every call site mxcli owns, and the collapse came back as a sentence telling the user to bypass it | diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index e5e87a105..3b7daf0b4 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -1,13 +1,17 @@ -# Download and Install Marketplace Content +# Download, Install and Update Marketplace Content -This skill covers discovering, downloading, and installing Mendix Marketplace content (modules and widgets) with `mxcli marketplace`. These are **CLI commands**, not MDL statements. +This skill covers the full lifecycle of Mendix Marketplace content (modules and widgets) +from the command line: discover → download → install → check for local edits → update. +These are **CLI commands**, not MDL statements. ## When to Use This Skill - User wants to add a marketplace module or widget to a project +- User wants to upgrade a module that is already installed +- User asks whether a marketplace module has been edited locally, or what an upgrade would overwrite - User asks to download a specific `.mpk` (e.g. for CI, or to import in Studio Pro) - User asks which versions of a marketplace item are compatible with their Mendix version -- User asks why a marketplace module did not update +- User reports `CE0463` right after installing or updating a module ## Prerequisites: Authenticate @@ -23,7 +27,10 @@ mxcli auth status # verify it validates Credentials are stored at `~/.mxcli/auth.json` (mode `0600`). -## Discover +Module installs also need the mxbuild toolchain for the project's Mendix version: +`mxcli setup mxbuild -p app.mpr`. + +## Step 1 — Discover ```bash mxcli marketplace search "database connector" # find content by name/publisher @@ -32,25 +39,42 @@ mxcli marketplace versions 2888 # available versions mxcli marketplace versions 2888 --min-mendix 10.24.0 # compatible versions only ``` -The numeric **content id** (from `search`/`info`) is what `download`/`install` take. +The numeric **content id** (from `search`/`info`) is what every other command takes. **Search caching.** The Content API has no server-side search, so the first `search` fetches the whole catalog (tens of seconds) and caches it under `~/.mxcli/` for 24h; later searches are instant. If the first search seems slow, it is scanning the catalog — -let it finish. Pass `--refresh` to bypass the cache and re-fetch (e.g. for a brand-new -module). If `search` returns nothing, the content may be private or named differently — -look it up by id with `info ` (ids come from the marketplace URL `.../link/component/`). +let it finish. Pass `--refresh` to bypass the cache (e.g. for a brand-new module). If +`search` returns nothing, the content may be private or listed under a different name — +look it up by id with `info ` (ids come from the marketplace URL +`.../link/component/`). + +**The listing name is not the module name.** Content 23513 is listed as "Administration +module" and installs a module called `Administration`; "Data Widgets" installs +`DataWidgets`. Never match a module to its marketplace listing by name — the commands +below identify it by the marketplace **version UUID** the project records per module. + +Content ids that come up often: + +| Content | Id | Installs module | +|---|---|---| +| Administration | 23513 | `Administration` | +| Community Commons | 170 | `CommunityCommons` | +| Data Widgets | 116540 | `DataWidgets` | +| Atlas Core | 117187 | `Atlas_Core` (theme module) | +| Atlas Web Content | 117183 | `Atlas_Web_Content` (theme module) | -## Download a `.mpk` to disk +## Step 2 — Download a `.mpk` to disk (optional) ```bash -mxcli marketplace download 2888 # latest, CDN filename +mxcli marketplace download 2888 # latest, CDN filename mxcli marketplace download 2888 --version 7.0.2 -o dbc.mpk # specific version + path ``` -Use this when you only want the file (e.g. to commit to `mx-modules/`, or import in Studio Pro yourself). +Use this when you only want the file (to commit to `mx-modules/`, or to import in Studio +Pro yourself). `download` needs no project; `install` does. -## Install into a project +## Step 3 — Install into a project ```bash mxcli marketplace install -p app.mpr [--version X.Y.Z] @@ -58,23 +82,193 @@ mxcli marketplace install -p app.mpr [--version X.Y.Z] `install` is **type-aware**: -| Content type | Behavior | +| Content type | Behaviour | |---|---| -| **Widget** | Copied into `widgets/` (overwrites on update). Reload in Studio Pro, or run `mxcli docker check`/`build` to normalize (v2-safe). Do **not** run bare `mx update-widgets` on an `mprcontents/` project — it converts to v1 and deletes `mprcontents/`. | -| **Module** (new) | Imported via `mx module-import` — needs a matching mxbuild (`mxcli setup mxbuild -p app.mpr`). | -| **Module** (already present) | **Reported, not modified** — see the caveat below. | +| **Widget** | Copied into `widgets/` (overwrites on update). | +| **Module** (new) | Copied in with mxcli's own writer — every unit, plus everything else the package ships (`widgets/`, `themesource/`, `javasource/`, ...). Preserves the project's storage format, and works for theme modules. | +| **Module** (already present) | **Reported, not modified.** Use `marketplace update` (step 5). | | Theme / Starter App / Sample | Downloaded with import instructions (import via Studio Pro). | -## IMPORTANT: module updates are not automatic +Measured: CommunityCommons 11.5.1 into a vanilla 11.12.1 app — 128 units and 126 bundled +files, `mprcontents/` grew from 369 to 497 `.mxunit` files, `mx check` reports 0 errors. + +### Do not use `mx module-import` + +`mx module-import` rewrites an **MPR v2** project as v1: one import turned a 69 KB `.mpr` +plus 341 `.mxunit` files into a single 14 MB SQLite blob with no `mprcontents/` +(measured on 11.12.1 and again on 11.13.0). The conversion is one-way — `mx convert` +targets Mendix *versions*, not storage formats — and it takes `mxcli diff-local`, per-document +git diffs and mergeability with it. `mx module-import` also refuses theme modules outright +("Importing theme module is not supported"). + +`install` therefore copies the units itself. `--allow-format-change` selects the legacy +`module-import` path; without it, that path refuses to run on a v2 project rather than +converting silently. + +### Dependencies are not resolved + +`install` installs exactly the content you name. A module whose dependencies are missing +produces a large error count that only shrinks as you add them, and the count is **not +monotonic** — adding a module can raise it before it falls (observed on a real agentic +stack: 156 → 16 → 227 → 211 → 1 → 0). Install one module at a time, re-check after each, +and read the remaining errors to find the next missing dependency rather than treating a +rising count as a regression. + +## Step 4 — Resync widget definitions (required after any headless install) + +```bash +mxcli docker check -p app.mpr +``` + +A freshly installed or updated module's pages reference widget definitions the project has +not resynced, so a check reports **CE0463** ("the definition of this widget has changed") +until it is told to. Measured on Administration 4.3.2 → 4.5.0: 11 errors before the +resync, 0 after. This is expected after any headless module install — it is **not** a +mxcli defect, and it is not the CE0463 that `.claude/skills/diagnose-ce0463.md` is for. + +**Never run bare `mx update-widgets` on an MPR v2 project.** It performs the resync and +converts the project to v1 in the process — measured on 11.12.1: 200 `.mxunit` files +became 0, and a 69,632-byte index became 14 MB. `mxcli docker check` runs the same +`mx update-widgets` step with the v2 storage snapshotted and restored around it, so the +check sees the resynced model and the project keeps its format. + +The resync is therefore not *persisted*: the check passes, and the stored model still +holds the pre-resync widget definitions, so a later `mx check` reports CE0463 again. +To persist it, open the project in Studio Pro once and use **Update all widgets**. +`mxcli widget sync -p app.mpr` is the headless equivalent, but is **partial** — on the +reference fixture it clears 7 of 40. + +## Step 5 — Before updating: has the module been edited? + +Studio Pro's Marketplace **Update** replaces the module and discards local edits without +asking. `marketplace diff` answers the question that decides whether that is safe: + +```bash +mxcli marketplace diff 23513 -p app.mpr # what have I changed? +mxcli marketplace diff 23513 -p app.mpr --to 4.5.0 # ...and what would an upgrade touch? +mxcli marketplace diff 23513 -p app.mpr --json # for a CI gate +``` + +```text +Administration — installed 4.3.2 (Mendix 11.12.1) + + Locally modified (1 of 21 elements): + changed ENTITY Account + + Upgrading to 4.5.0 would touch 5 element(s), 1 of which you have modified: + CONFLICT ENTITY Account +``` + +It downloads the installed version's `.mpk`, imports it into a throwaway reference project +built **at the project's own Mendix version** (a mismatch is refused, not warned about — +Mendix's own conversions would otherwise read as your edits), and compares `DESCRIBE` +output on both sides. -If the module is **already in the project**, `install` will NOT replace it. It reports the installed vs target version and stops. Do not try to force an update by deleting the module and re-importing — that is unsafe: +**Read `verified`, not just `locallyModified`.** An element that cannot be described is +reported as `unknown`, never as unchanged, and `verified: false` means "no modifications +found" is not a conclusion: + +```text + No local modifications found, but 46 of 89 elements could not be read — + this is not a clean bill of health. +``` + +Flags: `-p/--project` (required), `--to `, `--module ` (when the project +records no marketplace version for it, i.e. a hand-imported copy), `--json`, +`--profile`. + +## Step 6 — Update an installed module + +```bash +# Refuses if you have edited the module, naming what it would discard +mxcli marketplace update 23513 -p app.mpr --to 4.5.0 + +# Park those edits as re-executable MDL, then update over them +mxcli marketplace update 23513 -p app.mpr --to 4.5.0 --save-edits ./local-edits +mxcli marketplace update 23513 -p app.mpr --to 4.5.0 --force +mxcli exec ./local-edits/entity-Account.mdl -p app.mpr +``` + +```text +Administration updated 4.3.2 → 4.5.0 + 28 units copied, 9 element identities preserved, 2 role grant(s) restored. +``` + +Flags: `-p/--project`, `--to ` (required), `--module `, +`--save-edits `, `--force`, `--profile`. + +### What it preserves, and why it matters + +- **Element identity (`GUID`).** The runtime keys entities and attributes on the model's + `GUID` — `mendixsystem$entity.id` holds it verbatim. A module whose documents are + replaced without carrying the old `GUID`s is a *different* module to the database, and + its tables are dropped on the next deploy. `$ID` renumbering is irrelevant here; `GUID` + is everything. This is why deleting a module and re-importing it is never a valid + update. +- **Role grants.** A user role's grant of a module role lives in the *project's* security + document, not the module, so removing the module takes the grants with it and putting + it back does not return them. +- **Everything else the package ships.** Widget binaries under `widgets/`, styling and + design-property declarations under `themesource/`, and so on — only `project.mpr` and + `package.xml` are manifest rather than payload. DataWidgets 3.11.3 replaces 49 such + files; skipping them leaves the app running old widget code and reporting `CE6083` for + design properties the module itself declares. + +### Limits — state these to the user before running it + +- **Local edits are not preserved.** `update` refuses when it finds any; `--save-edits` + writes them out first; `--force` proceeds. Saved files are the element's **resulting + state, not a diff**, so replaying restores additions and changes but not removals, and + an element that could not be described has nothing to save (it is reported, not skipped). +- **No rollback.** Work on a copy, or have the project committed to version control first. +- **No dependency resolution** (same as `install`). +- **`update` does not run `mx check` itself** — do step 4 afterwards. + +### Afterwards + +```bash +mxcli docker check -p app.mpr # resyncs widgets; expect 0 errors +mxcli diff-local -p app.mpr # review what landed, per document +``` + +Measured after the resync: Administration 4.3.2 → 4.5.0 (28 units, 9 identities, 2 grants) +and DataWidgets 3.5.0 → 3.11.3 (49 files) both reach **0 errors**. + +## Worked example: the agent-editor stack on a vanilla app + +The seven modules in `.claude/skills/mendix/agents.md` must all be present before any +`create agent` statement will build. Install them one at a time, checking between: + +| Listing | Id | Module | +|---|---|---| +| GenAI Commons | 239448 | `GenAICommons` | +| Mendix Cloud GenAI Connector | 239449 | `MxGenAIConnector` | +| Agent Commons | 240371 | `AgentCommons` | +| Agent Editor | 257918 | `AgentEditorCommons` | +| MCP Client | 244893 | `MCPClient` | +| Conversational UI | 239450 | `ConversationalUI` | +| Encryption | 1011 | `Encryption` | + +```bash +mxcli new MyAgentApp --version 11.12.1 +cd MyAgentApp +for id in 239448 239449 240371 244893 239450 1011 257918; do + mxcli marketplace install "$id" -p MyAgentApp.mpr + mxcli docker check -p MyAgentApp.mpr | tail -3 +done +``` -1. **Local edits** to the module would be discarded. -2. **Persistent-entity `$ID`s** would change. The runtime database keys data by entity ID, so a re-import makes the runtime treat the entities as new ones and **data is lost**. +Install `AgentEditorCommons` (257918) **last** — it depends transitively on the other six. +Expect the error count to move non-monotonically until the last dependency lands. -Studio Pro's Marketplace **Update** does an ID-preserving merge that the CLI cannot. For module updates, tell the user to update via Studio Pro. +The ids above were resolved with `mxcli marketplace search` and are a convenience, not an +authority: confirm with `search`/`info` rather than trusting them from memory, and note +that the listing name never matches the module name. ## Notes -- `install` requires `-p `; `download` does not (it just fetches the file). -- Both require `mxcli auth login` first; an expired/missing PAT gives an auth error with a login hint. +- `install`/`update`/`diff` require `-p `; `download` does not. +- All of them require `mxcli auth login` first; an expired or missing PAT gives an auth + error with a login hint. +- Marketplace CDN TLS handshakes time out occasionally. Retry once before reporting a + failure. diff --git a/cmd/mxcli/cmd_marketplace_install.go b/cmd/mxcli/cmd_marketplace_install.go index d0c195d7f..5c72afc82 100644 --- a/cmd/mxcli/cmd_marketplace_install.go +++ b/cmd/mxcli/cmd_marketplace_install.go @@ -125,7 +125,7 @@ func installWidget(ctx context.Context, client *marketplace.Client, v *marketpla return err } fmt.Fprintf(out, "Installed widget %s into %s\n", v.VersionNumber, dest) - fmt.Fprintln(out, "Reload the project in Studio Pro (or run 'mx update-widgets') to pick it up.") + fmt.Fprintln(out, "Run 'mxcli docker check -p ' (or reload in Studio Pro) to pick it up.") return nil } @@ -178,7 +178,11 @@ func installModule(ctx context.Context, client *marketplace.Client, v *marketpla moduleName, v.VersionNumber, filepath.Base(mprPath)) fmt.Fprintf(out, " %d units copied, %d bundled file(s) installed.\n", res.UnitsCopied, len(res.FilesInstalled)) - fmt.Fprintln(out, "\n Next: 'mx update-widgets ', then 'mxcli docker check -p '.") + // 'mxcli docker check' resyncs widget definitions (clearing CE0463 on the + // module's pages) and restores the MPR v2 storage format afterwards; bare + // 'mx update-widgets' does the resync but leaves the project as v1. + fmt.Fprintln(out, "\n Next: 'mxcli docker check -p ' (resyncs widget definitions,") + fmt.Fprintln(out, " which a headless install leaves stale, and reports CE0463 until it runs).") return nil } diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index fa7b5515d..3412b1a50 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -236,14 +236,18 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { } // A newer module's pages reference widget definitions the project has not // resynced, so `mx check` reports CE0463 until it is told to. Measured on - // Administration 4.3.2 → 4.5.0: 11 CE0463 errors, and 0 after update-widgets. + // Administration 4.3.2 → 4.5.0: 11 CE0463 errors, and 0 after the resync. // Saying so here is the difference between a two-command fix and a day in // diagnose-ce0463.md, which is where that error normally leads. + // + // The resync is named as `mxcli docker check`, not as bare `mx update-widgets`: + // the latter rewrites an MPR v2 project as v1 (measured on 11.12.1 — 200 + // .mxunit files to 0, a 69 KB index to 14 MB), while docker check runs the + // same step under a storage-format snapshot (#808). fmt.Fprintln(out, "\n Next: resync widget definitions, or 'mx check' will report CE0463 on the") fmt.Fprintln(out, " new version's pages (this is expected after any headless module install):") - fmt.Fprintln(out, " mx update-widgets ") - fmt.Fprintln(out, "\n Then validate and review:") fmt.Fprintln(out, " mxcli docker check -p ") + fmt.Fprintln(out, "\n Then review what landed:") fmt.Fprintln(out, " mxcli diff-local -p ") } diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index 92033b52a..e2de093ee 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -150,9 +150,13 @@ Local edits are **not** preserved. `update` refuses when it finds any, `--save-e ### Afterwards -Run `mx update-widgets `. A newer module's pages reference widget definitions the project has not resynced, so `mx check` reports CE0463 until told to — measured on Administration 4.3.2 → 4.5.0: 11 errors before, 0 after. This is expected after any headless module install, not a fault in the update. +Run `mxcli docker check -p `, then `mxcli diff-local -p `. -Then `mxcli docker check -p ` and `mxcli diff-local -p `. Measured after that resync: Administration 4.3.2 → 4.5.0 and DataWidgets 3.5.0 → 3.11.3 both reach **0 errors**. +A newer module's pages reference widget definitions the project has not resynced, so a check reports CE0463 until it is told to — measured on Administration 4.3.2 → 4.5.0: 11 errors before the resync, 0 after. This is expected after any headless module install, not a fault in the update. Measured after the resync, Administration 4.3.2 → 4.5.0 and DataWidgets 3.5.0 → 3.11.3 both reach **0 errors**. + +**Do not run bare `mx update-widgets` on an MPR v2 project.** It performs the resync but rewrites the project as v1 — measured on 11.12.1, 200 `.mxunit` files became 0 and a 69,632-byte index became 14 MB. `mxcli docker check` runs the same `mx update-widgets` step with the v2 storage snapshotted and restored around it, so the check sees the resynced model and the project keeps its format. + +The consequence is that the resync is not *persisted*: the check passes, and the stored model still holds the pre-resync widget definitions, so a later `mx check` reports CE0463 again. Opening the project in Studio Pro once and using **Update all widgets** persists it in v2. `mxcli widget sync` is the headless equivalent but is partial — on the reference fixture it clears 7 of 40. `update` does **not** roll back. Work on a copy or have the project in version control. From 1e18ba7a630c1c13c001d858f0c0b874dbca6e85 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 13:18:17 +0000 Subject: [PATCH 34/35] Refuse a marketplace version the project's Mendix cannot import MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running the documented install workflow end-to-end on a fresh 11.12.1 app failed on the first command. The marketplace publishes new releases against the newest Studio Pro patch within days of it shipping, and `install` with no --version resolves to the latest — so on any project not on the very newest patch, the default is routinely the one version that cannot be installed. All six agent-editor stack modules had a latest release requiring 11.12.2, published five days earlier. The refusal already happened, three layers down: after the download and after building a reference project, as `mx module-import` exit 117, under the command's full flag list. Every version the API returns already carries minSupportedMendixVersion; nothing consulted it. Now install and update check it first and name the version to use instead. SilenceErrors goes with SilenceUsage because main() already prints what Execute returns, so silencing only usage left every refusal printed twice. The rest of the run is recorded in the skill: 8 modules and 2 widget packages into a vanilla app, error count 0 → 15 → 0 → 18 → 1 → 22 → 1, staying MPR v2 throughout (1,869 .mxunit files), ending with a working `create agent`. Dependencies are not resolved and include widget content, and the count rises before it falls — both now stated with the measured sequence rather than in the abstract. Two corrections. The .mxunit counts published yesterday were shard directories, not files: the update-widgets collapse is 370 files to 0, not 200. And CE6087 has no headless fix — `mx rename-design-properties` does real work (149 properties across 41 documents) and collapses v2 like its siblings, but unlike the widget resync its result must persist, so the snapshot-and-restore trick does not transfer. Documented as open rather than papered over. --- .claude/skills/fix-issue.md | 4 +- .../mendix/download-marketplace-content.md | 116 +++++++++++++----- cmd/mxcli/cmd_marketplace_diff.go | 6 + cmd/mxcli/cmd_marketplace_install.go | 14 +++ cmd/mxcli/cmd_marketplace_update.go | 14 +++ cmd/mxcli/marketplace_compat.go | 80 ++++++++++++ cmd/mxcli/marketplace_compat_test.go | 99 +++++++++++++++ docs-site/src/guides/marketplace.md | 27 +++- 8 files changed, 327 insertions(+), 33 deletions(-) create mode 100644 cmd/mxcli/marketplace_compat.go create mode 100644 cmd/mxcli/marketplace_compat_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 4a7b0c8a8..c2feec0c3 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -470,4 +470,6 @@ extracting `OffsetExpression`/`LimitExpression`. | `show features` lists nothing for agents or MCP, so `create agent` / `create model` on a pre-11.9 project runs without complaint. Reported alongside it: "`CREATE MODEL` can only author one provider — the writer assigns `MxCloudGenAI` unconditionally" | Two different things, and only the first was real. **The version gap**: the `agent_documents` area did not exist in `sdk/versions/mendix-11.yaml`, so `checkFeature()` had nothing to consult. **The provider claim was wrong**: the writer's assignment is `if m.Provider == ""` — a default, not an override. `Provider: OpenAI` parses, writes and round-trips through `describe model`; so does `Provider: TotallyMadeUp` | `sdk/versions/mendix-11.yaml` (new `agent_documents` area), `mdl/executor/cmd_agenteditor_models.go` + `cmd_agenteditor_write.go` (four `checkFeature` gates), `.claude/skills/mendix/agents.md` + `version-awareness.md` | **Read the assignment's guard before believing "hardcoded"** — `if x == "" { x = default }` and `x = default` are one character apart in a grep and opposite in meaning; the round-trip test settles it in a minute. **Agent doctypes are the one version gate with no downstream safety net**: the documents are custom blobs, mxbuild contains no agent-editor strings at all, so an ungated project builds green and fails only when Studio Pro opens it. **Do not invent the provider allowlist** — the enum lives in a Studio Pro *extension*, not in `generated/metamodel` and not in mxbuild, so a guessed list would reject values Mendix accepts; document that nothing validates it instead. Test `TestAgentDocumentsAreGated`. Reported in mxcli-formula1 FINDINGS §53 | | A marketplace module update leaves the app running the OLD widget code: `show modules` reports the new version, pages reference new widget definitions, and the binaries in `widgets/` are unchanged. Symptoms are whatever the version gap causes at render time, with no build error naming the cause | The update moved only the **model**. A widget module's `.mpk` ships its widget binaries under `widgets/` inside the package, and copying units out of a reference project never touches them. Measured on DataWidgets 3.5.0 → 3.11.3: model updated cleanly, all 10 widget binaries still 3.5.0 (`Datagrid.mpk` project=216193 vs package=166933) | `cmd/mxcli/marketplace/update.go` (`InstallPackageWidgets`, called from `PerformUpdate`) | **Take the widget files from the `.mpk`, not from the reference project** — the reference is a blank app plus the module, so its `widgets/` also holds the template's widgets and copying those overwrites widgets the update has nothing to do with. **Skip zero-size directory entries** in the zip or you write a stray file (the 3.11.3 package has 10 `widgets/` entries and 9 real files). **A module with no widgets cannot expose this**: Administration updated end-to-end and looked completely correct, which is why the gap survived a full slice. Pick a second subject with a different shape before believing an update path works. Related but distinct: after any headless module install `mx update-widgets` is needed (CE0463); and a newer module can want a newer Atlas than the project has (29 × CE6083 on DataWidgets 3.11.3), which `rename-design-properties` does NOT fix — that is a dependency, not a resync | | After a marketplace module update, `mx check` reports `CE6083` "Design property X is not supported by your theme" on the updated module's own widgets, and it survives BOTH `mx update-widgets` and `mx rename-design-properties` (which renames 0) | The update copied only the model and `widgets/`. The design properties are declared in the module's **own** `themesource//web/design-properties.json`, which was still the old version's copy — so the model referenced properties the shipped theme file did not define | `cmd/mxcli/marketplace/update.go` (`InstallPackageFiles` replaces `InstallPackageWidgets`) | **Copy everything the package ships**, excluding only `project.mpr` and `package.xml` — enumerating the directories that seem to matter is what produced two bugs in a row (widgets, then themesource). **CE6083 surviving `rename-design-properties` is the tell**: that command renames properties between Atlas generations, so 0 renamed means the declaration is missing entirely rather than renamed. **Do not diagnose this as a cross-module Atlas dependency** — that was the wrong call here; the declaring module was the one being updated. Guard against `..` in package paths while copying. Measured: DataWidgets 3.5.0 → 3.11.3, 49 files replaced, 29 errors → 0 | -| A project that was MPR v2 is v1 after a marketplace install or update: `mprcontents/` is empty or gone, the `.mpr` jumped to tens of MB, `mxcli diff-local` stops working. The user followed the command's own "Next:" hint, or the docs | The hint said to run bare `mx update-widgets ` to clear the CE0463 a headless install leaves behind. That command performs the resync **and** rewrites a v2 project as v1 — measured on 11.12.1: 200 `.mxunit` files → 0, a 69,632-byte index → 14,405,632 bytes. Same family as the `mx module-import` collapse, arriving through advice rather than through a call | `cmd/mxcli/cmd_marketplace_install.go` + `cmd_marketplace_update.go` (the "Next:" lines now name `mxcli docker check`), `docs-site/src/guides/marketplace.md`, `.claude/skills/mendix/download-marketplace-content.md` | **`mxcli docker check` is the v2-safe resync** — it runs the same `mx update-widgets` under a storage-format snapshot (`cmd/mxcli/docker/update_widgets.go`, #808), so the check sees the resynced model and the project keeps its format. Verified: 200 `.mxunit` / 69,632 bytes before and after. **The resync is then not persisted** — the stored model still holds the old widget definitions and a later `mx check` reports CE0463 again; Studio Pro's "Update all widgets" persists it in v2, and `mxcli widget sync` is the partial headless equivalent (7 of 40 on the reference fixture). Say that rather than implying the resync sticks. **A guard in the code does not cover a string in the help text**: #808 already protected every call site mxcli owns, and the collapse came back as a sentence telling the user to bypass it | +| A project that was MPR v2 is v1 after a marketplace install or update: `mprcontents/` is empty or gone, the `.mpr` jumped to tens of MB, `mxcli diff-local` stops working. The user followed the command's own "Next:" hint, or the docs | The hint said to run bare `mx update-widgets ` to clear the CE0463 a headless install leaves behind. That command performs the resync **and** rewrites a v2 project as v1 — measured on 11.12.1: 370 `.mxunit` files → 0, a 69,632-byte index → 14,405,632 bytes. Same family as the `mx module-import` collapse, arriving through advice rather than through a call | `cmd/mxcli/cmd_marketplace_install.go` + `cmd_marketplace_update.go` (the "Next:" lines now name `mxcli docker check`), `docs-site/src/guides/marketplace.md`, `.claude/skills/mendix/download-marketplace-content.md` | **`mxcli docker check` is the v2-safe resync** — it runs the same `mx update-widgets` under a storage-format snapshot (`cmd/mxcli/docker/update_widgets.go`, #808), so the check sees the resynced model and the project keeps its format. Verified: 370 `.mxunit` / 69,632 bytes before and after. **The resync is then not persisted** — the stored model still holds the old widget definitions and a later `mx check` reports CE0463 again; Studio Pro's "Update all widgets" persists it in v2, and `mxcli widget sync` is the partial headless equivalent (7 of 40 on the reference fixture). Say that rather than implying the resync sticks. **A guard in the code does not cover a string in the help text**: #808 already protected every call site mxcli owns, and the collapse came back as a sentence telling the user to bypass it | +| `mxcli marketplace install -p app.mpr` fails with `mx module-import failed: exit status 117` — "The package could not be imported, because it was created with a newer version of Mendix Studio Pro" — after downloading the package and building a reference project, and the message is buried under the command's full flag list | The marketplace publishes new releases against the newest Studio Pro patch within days, and `install` with no `--version` resolves to the latest, so on any project not on the very newest patch the **default** version is the one that cannot be installed. Measured 2026-08-12 on an 11.12.1 project: the latest release of all six agent-stack modules required 11.12.2, published five days earlier. Every version the API returns already carries `minSupportedMendixVersion` — nothing consulted it | `cmd/mxcli/marketplace_compat.go` (`checkMendixCompatibility`, `newestCompatibleVersion`, `mendixVersionOf`), called from `cmd_marketplace_install.go` and `cmd_marketplace_update.go`; `SilenceUsage`/`SilenceErrors` on install/update/diff | **Check what the API already told you before spending a download and a reference build on it** — the refusal was always going to happen, three layers lower and in mxcli's voice rather than Mendix's. **Name the version to use, not just the problem**: `--version 4.1.0 (the newest release built for 11.12.1 or older)` is the whole fix. **Skip, do not refuse, when the check cannot evaluate** (no project version, no published minimum) — same rule as `checkFeature`. **`SilenceErrors` belongs with `SilenceUsage`**: `main()` already prints what `Execute` returns, so silencing only usage leaves every refusal printed twice. Tests `cmd/mxcli/marketplace_compat_test.go`, including the negative case — a check that refuses everything passes the refusal test | +| After a headless module install, `mx check` reports project-level `CE6087` "Design properties have been renamed in your theme and need to be updated" with an empty location, and no mxcli command clears it | The module ships its own design properties; Mendix's fix is `mx rename-design-properties`, which mxcli never runs. Measured on 11.12.1: it renames real work (149 design properties across 41 documents) **and** collapses MPR v2 — 1,866 `.mxunit` files → 0, a 249,856-byte index → 39,895,040 bytes. Third member of the family, after `mx module-import` and `mx update-widgets` | *(open — documented, not fixed)* `.claude/skills/mendix/download-marketplace-content.md`, `docs-site/src/guides/marketplace.md` | **The `update-widgets` snapshot trick does not transfer**: that one restores the pre-run storage because the resync only has to hold for the duration of the check, whereas these renames must **persist**, so restoring undoes the fix. A v2-safe path needs harvesting the changed units out of the collapsed v1 file and writing them back through mxcli's writer — no such helper exists (grep for a v1→v2 conversion returns nothing). **Do not claim a headless fix exists**; on v2 the choices are Studio Pro or accepting the conversion. Distinguish from CE6083, which is a *missing* declaration (fixed by copying the package's `themesource/`) — CE6087 is a *renamed* one | diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index 3b7daf0b4..652c3f764 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -105,14 +105,39 @@ git diffs and mergeability with it. `mx module-import` also refuses theme module `module-import` path; without it, that path refuses to run on a v2 project rather than converting silently. +### The latest version is usually NOT the one to install + +New releases are published against the newest Studio Pro patch within days of it shipping, +and `install` with no `--version` resolves to the latest. On any project that is not on the +very newest patch, the default is therefore routinely the one version that cannot be +installed. Measured 2026-08-12 on an 11.12.1 project: the latest release of **all six** +agent-stack modules required 11.12.2, published five days earlier. + +`install` and `update` refuse it up front and name the version to use instead: + +```text +Agent Commons 4.2.0 requires Mendix 11.12.2, and the project is 11.12.1 + hint: install --version 4.1.0 (the newest release built for 11.12.1 or older) +``` + +Run `mxcli marketplace versions ` first and read the `MIN MENDIX` column, or just act +on the refusal. + ### Dependencies are not resolved -`install` installs exactly the content you name. A module whose dependencies are missing -produces a large error count that only shrinks as you add them, and the count is **not -monotonic** — adding a module can raise it before it falls (observed on a real agentic -stack: 156 → 16 → 227 → 211 → 1 → 0). Install one module at a time, re-check after each, -and read the remaining errors to find the next missing dependency rather than treating a -rising count as a regression. +`install` installs exactly the content you name, and its dependencies are neither fetched +nor named. Read the check errors after each install — they identify what is missing by +qualified name (`CommunityCommons.RandomHash`, `MCPClient.ConsumedMCPService`), which +tells you the module to add next. + +The error count is **not monotonic**: adding a module can raise it before it falls, because +a module brings its own unmet dependencies with it. Measured on a vanilla 11.12.1 app while +installing the agent-editor stack: 0 → 15 → 0 → 18 → 1 → 22 → 1 → 1. A rising count is +progress, not a regression. + +Dependencies include **widget content**, not only modules — `ConversationalUI` needs the +`Markdown viewer` (230248) and `Events` (224259) widget packages, which surface as +`CE0462 "Could not find widget ... in the 'widgets' directory"`. ## Step 4 — Resync widget definitions (required after any headless install) @@ -127,8 +152,8 @@ resync, 0 after. This is expected after any headless module install — it is ** mxcli defect, and it is not the CE0463 that `.claude/skills/diagnose-ce0463.md` is for. **Never run bare `mx update-widgets` on an MPR v2 project.** It performs the resync and -converts the project to v1 in the process — measured on 11.12.1: 200 `.mxunit` files -became 0, and a 69,632-byte index became 14 MB. `mxcli docker check` runs the same +converts the project to v1 in the process — measured on 11.12.1: 370 `.mxunit` files +became 0, and a 69,632-byte index became 14,405,632 bytes. `mxcli docker check` runs the same `mx update-widgets` step with the v2 storage snapshotted and restored around it, so the check sees the resynced model and the project keeps its format. @@ -138,6 +163,22 @@ To persist it, open the project in Studio Pro once and use **Update all widgets* `mxcli widget sync -p app.mpr` is the headless equivalent, but is **partial** — on the reference fixture it clears 7 of 40. +### CE6087 has no headless fix today — know this before you promise one + +`CE6087 "Design properties have been renamed in your theme and need to be updated"` appears +after installing modules that ship their own design properties. Mendix's fix is +`mx rename-design-properties`, and it collapses MPR v2 exactly like `update-widgets` does — +measured on 11.12.1: 1,866 `.mxunit` files → 0, a 249,856-byte index → 39,895,040 bytes, +having renamed 149 design properties across 41 documents. + +Unlike `update-widgets`, **mxcli has no protected path for it**: `docker check` does not run +it, and snapshot-and-restore would not help anyway, because the renames have to *persist* +where the widget resync does not. So on an MPR v2 project the choices are Studio Pro, or +accepting the v1 conversion. Do not tell a user a headless fix exists. + +The error is project-level (its location in the check output is empty), so it cannot be +traced to the module that caused it from the message alone. + ## Step 5 — Before updating: has the module been edited? Studio Pro's Marketplace **Update** replaces the module and discards local edits without @@ -236,34 +277,47 @@ and DataWidgets 3.5.0 → 3.11.3 (49 files) both reach **0 errors**. ## Worked example: the agent-editor stack on a vanilla app -The seven modules in `.claude/skills/mendix/agents.md` must all be present before any -`create agent` statement will build. Install them one at a time, checking between: - -| Listing | Id | Module | -|---|---|---| -| GenAI Commons | 239448 | `GenAICommons` | -| Mendix Cloud GenAI Connector | 239449 | `MxGenAIConnector` | -| Agent Commons | 240371 | `AgentCommons` | -| Agent Editor | 257918 | `AgentEditorCommons` | -| MCP Client | 244893 | `MCPClient` | -| Conversational UI | 239450 | `ConversationalUI` | -| Encryption | 1011 | `Encryption` | +Run end-to-end on 2026-08-12 against a fresh `mxcli new … --version 11.12.1` app. The +modules in `.claude/skills/mendix/agents.md` must all be present before any `create agent` +statement will build, and two of the dependencies are neither listed there nor modules. + +| Step | Content | Id | `--version` | Units | Errors after check | +|---|---|---|---|---|---| +| 0 | *(vanilla app)* | — | — | 370 files | 0 | +| 1 | GenAI Commons | 239448 | 7.1.1 | 214 | 15 — needs CommunityCommons | +| 2 | Community Commons | 170 | latest | 128 | 0 | +| 3 | Mendix Cloud GenAI Connector | 239449 | 7.1.0 | 223 | 18 — needs Encryption | +| 4 | Encryption | 1011 | latest | 61 | 1 — CE6087 | +| 5 | Agent Commons | 240371 | 4.1.0 | 385 | 22 — needs MCPClient + ConversationalUI | +| 6 | MCP Client | 244893 | 4.1.0 | 82 | — | +| 7 | Conversational UI | 239450 | 7.1.0 | 345 | 22 — CE0462, missing widgets | +| 8 | Markdown viewer *(widget)* | 230248 | latest | — | — | +| 9 | Events *(widget)* | 224259 | latest | — | 1 — CE6087 | +| 10 | Agent Editor | 257918 | 2.1.0 | 58 | 1 — CE6087 | ```bash -mxcli new MyAgentApp --version 11.12.1 -cd MyAgentApp -for id in 239448 239449 240371 244893 239450 1011 257918; do - mxcli marketplace install "$id" -p MyAgentApp.mpr - mxcli docker check -p MyAgentApp.mpr | tail -3 -done +mxcli new MyAgentApp --version 11.12.1 && cd MyAgentApp +mxcli marketplace install 239448 --version 7.1.1 -p MyAgentApp.mpr +mxcli docker check -p MyAgentApp.mpr # read the errors; they name what is missing +# ...repeat per row... ``` -Install `AgentEditorCommons` (257918) **last** — it depends transitively on the other six. -Expect the error count to move non-monotonically until the last dependency lands. +Then authoring works — `create constant` + `create model` + `create agent` executed and +added 3 units, with `show features in agent_documents` reporting all four document types +available on 11.12.1. + +Four things this run established, none of them obvious from the command list: + +1. **Install `AgentEditorCommons` last** — it depends transitively on the rest. +2. **`--version` is mandatory in practice.** Every agent-stack module's latest release + required 11.12.2 against an 11.12.1 project. +3. **Two dependencies are widgets, and two more are modules the agent skill does not + list** (CommunityCommons, and the widget packages). Let the check errors drive it. +4. **The end state is 1 error, not 0** — CE6087, which has no headless fix (above). The + project stays MPR v2 throughout: 1,869 `.mxunit` files, a 249,856-byte index. -The ids above were resolved with `mxcli marketplace search` and are a convenience, not an -authority: confirm with `search`/`info` rather than trusting them from memory, and note -that the listing name never matches the module name. +The ids are a convenience, not an authority: confirm with `search`/`info` rather than +trusting them from memory, and note that the listing name never matches the module name. ## Notes diff --git a/cmd/mxcli/cmd_marketplace_diff.go b/cmd/mxcli/cmd_marketplace_diff.go index 82f23e20f..8578fe215 100644 --- a/cmd/mxcli/cmd_marketplace_diff.go +++ b/cmd/mxcli/cmd_marketplace_diff.go @@ -49,6 +49,12 @@ Requires the mxbuild toolchain for the project's Mendix version: mxcli marketplace diff 23513 -p app.mpr --json`, Args: cobra.ExactArgs(1), RunE: runMarketplaceDiff, + // A failed install/update/diff is a runtime failure, not a misuse of the + // command: printing the full flag list on top of the error buries it. + // SilenceErrors too, because main() already prints what Execute returns — + // without it every refusal is printed twice. + SilenceUsage: true, + SilenceErrors: true, } func runMarketplaceDiff(cmd *cobra.Command, args []string) error { diff --git a/cmd/mxcli/cmd_marketplace_install.go b/cmd/mxcli/cmd_marketplace_install.go index 5c72afc82..e30655eba 100644 --- a/cmd/mxcli/cmd_marketplace_install.go +++ b/cmd/mxcli/cmd_marketplace_install.go @@ -46,6 +46,12 @@ installed alongside the model. mxcli marketplace install 2888 --version 7.0.3 -p app.mpr`, Args: cobra.ExactArgs(1), RunE: runMarketplaceInstall, + // A failed install/update/diff is a runtime failure, not a misuse of the + // command: printing the full flag list on top of the error buries it. + // SilenceErrors too, because main() already prints what Execute returns — + // without it every refusal is printed twice. + SilenceUsage: true, + SilenceErrors: true, } func init() { @@ -99,6 +105,14 @@ func runMarketplaceInstall(cmd *cobra.Command, args []string) error { case "widget": return installWidget(cmd.Context(), client, version, projDir, out) case "module": + // Refuse a version the project's Mendix cannot import, before spending a + // download and a reference build on it. Only modules go through + // module-import, so only modules are gated here. + if projectVer := mendixVersionOf(mprPath); projectVer != "" { + if err := checkMendixCompatibility(version, verList.Items, projectVer, version.Name); err != nil { + return err + } + } return installModule(cmd.Context(), client, version, mprPath, allowFormatChange, out) default: // Theme / Starter App / Sample / unknown: download + instruct rather diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index 3412b1a50..5f0cbff0d 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -45,6 +45,12 @@ if a step fails partway, the module has already been removed.`, mxcli exec ./local-edits/entity-Account.mdl -p app.mpr`, Args: cobra.ExactArgs(1), RunE: runMarketplaceUpdate, + // A failed install/update/diff is a runtime failure, not a misuse of the + // command: printing the full flag list on top of the error buries it. + // SilenceErrors too, because main() already prints what Execute returns — + // without it every refusal is printed twice. + SilenceUsage: true, + SilenceErrors: true, } func runMarketplaceUpdate(cmd *cobra.Command, args []string) error { @@ -97,6 +103,14 @@ func runMarketplaceUpdate(cmd *cobra.Command, args []string) error { return nil } + // Refuse a target the project's Mendix cannot import, before building two + // reference projects to discover it from `mx module-import`'s exit code. + if tv, verr := pickVersion(versions.Items, "", target); verr == nil { + if err := checkMendixCompatibility(tv, versions.Items, mendixVersion, moduleName); err != nil { + return err + } + } + // Has anyone edited this module? Answering needs the version it was installed // from, built as a reference exactly as `marketplace diff` does. base, err := pickVersion(versions.Items, installedVersionID, installedVersion) diff --git a/cmd/mxcli/marketplace_compat.go b/cmd/mxcli/marketplace_compat.go new file mode 100644 index 000000000..8cba679ba --- /dev/null +++ b/cmd/mxcli/marketplace_compat.go @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "strings" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/internal/marketplace" +) + +// mendixVersionOf reports the project's Mendix version, or "" when the project +// cannot be opened or records none. Callers treat "" as "cannot evaluate". +func mendixVersionOf(mprPath string) string { + reader, err := modelsdk.Open(mprPath) + if err != nil { + return "" + } + defer reader.Close() + v, _ := reader.GetMendixVersion() + return v +} + +// checkMendixCompatibility refuses a marketplace version the project's Mendix +// version cannot import, and names the newest version that it can. +// +// Every version the API returns carries `minSupportedMendixVersion`, and the +// marketplace publishes new releases against the newest Studio Pro patch within +// days of it shipping — so on any project that is not on the very latest patch, +// the *default* (latest) version is routinely the one that cannot be installed. +// Measured 2026-08-12 on a project at 11.12.1: the latest release of all six +// agent-stack modules required 11.12.2, published five days earlier. +// +// Without this check the refusal still happens, but three layers down and in a +// form that reads like an mxcli failure: the package is downloaded, a reference +// project is built, and `mx module-import` exits 117 with "the package ... was +// created with a newer version of Mendix Studio Pro". Checking here turns that +// into one line naming the version to pass instead. +// +// A version whose minimum cannot be parsed is allowed through rather than +// refused — the check exists to give a better error, never to block an install +// it cannot evaluate. +func checkMendixCompatibility(v *marketplace.Version, all []marketplace.Version, projectVersion, contentName string) error { + if v == nil || v.MinSupportedMendixVersion == "" || projectVersion == "" { + return nil + } + if compareSemverLike(v.MinSupportedMendixVersion, projectVersion) <= 0 { + return nil + } + + var b strings.Builder + fmt.Fprintf(&b, "%s %s requires Mendix %s, and the project is %s", + contentName, v.VersionNumber, v.MinSupportedMendixVersion, projectVersion) + if best := newestCompatibleVersion(all, projectVersion); best != "" { + fmt.Fprintf(&b, "\n hint: install --version %s (the newest release built for %s or older)", + best, projectVersion) + } else { + fmt.Fprintf(&b, "\n hint: no published version supports Mendix %s; upgrade the project first", + projectVersion) + } + return fmt.Errorf("%s", b.String()) +} + +// newestCompatibleVersion returns the highest version number whose minimum +// Mendix version the project satisfies, or "" when none does. The API returns +// versions newest-first, so the first match is the answer; it is not re-sorted +// here, because version numbers are publisher-controlled strings and the +// publication order is the only ranking the API actually guarantees. +func newestCompatibleVersion(all []marketplace.Version, projectVersion string) string { + for i := range all { + if all[i].MinSupportedMendixVersion == "" { + continue + } + if compareSemverLike(all[i].MinSupportedMendixVersion, projectVersion) <= 0 { + return all[i].VersionNumber + } + } + return "" +} diff --git a/cmd/mxcli/marketplace_compat_test.go b/cmd/mxcli/marketplace_compat_test.go new file mode 100644 index 000000000..6c9d98dee --- /dev/null +++ b/cmd/mxcli/marketplace_compat_test.go @@ -0,0 +1,99 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/internal/marketplace" +) + +// The shape measured on 2026-08-12: the latest release of every agent-stack +// module required Mendix 11.12.2, five days after that patch shipped, while the +// project under test was on 11.12.1. `install` with no --version resolves to the +// latest, so this is the default path, not an edge case. +func agentStackVersions() []marketplace.Version { + return []marketplace.Version{ + {VersionNumber: "7.2.0", MinSupportedMendixVersion: "11.12.2"}, + {VersionNumber: "7.1.1", MinSupportedMendixVersion: "11.12.1"}, + {VersionNumber: "7.0.0", MinSupportedMendixVersion: "11.12.0"}, + {VersionNumber: "6.2.1", MinSupportedMendixVersion: "10.24.13"}, + } +} + +func TestCheckMendixCompatibility_RefusesAndNamesTheVersionToUse(t *testing.T) { + all := agentStackVersions() + err := checkMendixCompatibility(&all[0], all, "11.12.1", "GenAI Commons") + if err == nil { + t.Fatal("a version requiring 11.12.2 was accepted for an 11.12.1 project; " + + "without this the refusal surfaces as 'mx module-import' exit 117, three layers down") + } + msg := err.Error() + for _, want := range []string{"11.12.2", "11.12.1", "--version 7.1.1"} { + if !strings.Contains(msg, want) { + t.Errorf("error must contain %q so the user can act on it, got:\n%s", want, msg) + } + } +} + +// The negative case: a check that refuses everything would also pass the test +// above. A compatible version must install. +func TestCheckMendixCompatibility_AllowsACompatibleVersion(t *testing.T) { + all := agentStackVersions() + for _, i := range []int{1, 2, 3} { + if err := checkMendixCompatibility(&all[i], all, "11.12.1", "GenAI Commons"); err != nil { + t.Errorf("version %s (min %s) refused on an 11.12.1 project: %v", + all[i].VersionNumber, all[i].MinSupportedMendixVersion, err) + } + } +} + +// A check that cannot evaluate must not block. Both an unknown project version +// and an unpublished minimum fall through, matching how checkFeature treats a +// backend that reports no version. +func TestCheckMendixCompatibility_SkipsWhenItCannotEvaluate(t *testing.T) { + all := agentStackVersions() + if err := checkMendixCompatibility(&all[0], all, "", "GenAI Commons"); err != nil { + t.Errorf("refused with an unknown project version: %v", err) + } + noMin := marketplace.Version{VersionNumber: "9.9.9"} + if err := checkMendixCompatibility(&noMin, all, "11.12.1", "GenAI Commons"); err != nil { + t.Errorf("refused a version publishing no minimum: %v", err) + } + if err := checkMendixCompatibility(nil, all, "11.12.1", "GenAI Commons"); err != nil { + t.Errorf("refused a nil version: %v", err) + } +} + +// When nothing is compatible the hint must not point at a version that does not +// exist — an empty suggestion is worse than none. +func TestCheckMendixCompatibility_NoCompatibleVersionSaysSo(t *testing.T) { + all := []marketplace.Version{ + {VersionNumber: "2.0.0", MinSupportedMendixVersion: "11.12.2"}, + {VersionNumber: "1.0.0", MinSupportedMendixVersion: "11.12.2"}, + } + err := checkMendixCompatibility(&all[0], all, "10.24.0", "Agent Editor") + if err == nil { + t.Fatal("expected a refusal") + } + if strings.Contains(err.Error(), "--version") { + t.Errorf("suggested a --version when none is compatible:\n%s", err) + } + if !strings.Contains(err.Error(), "upgrade the project") { + t.Errorf("expected the message to name the only remaining option, got:\n%s", err) + } +} + +func TestNewestCompatibleVersion_PrefersPublicationOrder(t *testing.T) { + all := agentStackVersions() + if got := newestCompatibleVersion(all, "11.12.1"); got != "7.1.1" { + t.Errorf("newestCompatibleVersion = %q, want 7.1.1", got) + } + if got := newestCompatibleVersion(all, "10.24.13"); got != "6.2.1" { + t.Errorf("newestCompatibleVersion = %q, want 6.2.1", got) + } + if got := newestCompatibleVersion(all, "9.0.0"); got != "" { + t.Errorf("newestCompatibleVersion = %q, want an empty result", got) + } +} diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index e2de093ee..d7799f2c4 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -65,6 +65,23 @@ mxcli marketplace install 2888 --version 7.0.3 -p app.mpr # a module | **Module** (already present) | **Reported, not modified** — see below. | | Theme / Starter App / Sample | Downloaded to disk with import instructions (import via Studio Pro). | +### The latest version is often not installable + +New releases are published against the newest Studio Pro patch within days of it shipping, and `install` with no `--version` resolves to the latest — so on a project that is not on the very newest patch, the default is routinely the one version that cannot be imported. Measured on an 11.12.1 project: the latest release of all six agent-editor stack modules required 11.12.2, published five days earlier. + +`install` and `update` check the version's published minimum before downloading anything, and name the version to use instead: + +```text +Agent Commons 4.2.0 requires Mendix 11.12.2, and the project is 11.12.1 + hint: install --version 4.1.0 (the newest release built for 11.12.1 or older) +``` + +`mxcli marketplace versions ` shows the same information as a `MIN MENDIX` column. + +### Dependencies are not resolved + +`install` installs exactly the content you name. Its dependencies are neither fetched nor named — read the check errors after each install, which identify what is missing by qualified name. The error count is **not monotonic**, because each new module brings its own unmet dependencies: installing the agent-editor stack into a vanilla 11.12.1 app went 0 → 15 → 0 → 18 → 1 → 22 → 1 → 1. Dependencies include widget content as well as modules (`CE0462 "Could not find widget ..."`). + ## Why installs do not use `mx module-import` `mx module-import` rewrites an MPR v2 project as v1. Measured on a blank Mendix 11.12.1 app, a single import turned a 69 KB `.mpr` plus 341 `.mxunit` files into one 14 MB SQLite blob with no `mprcontents/` — and the same was observed independently on 11.13.0, so it is not version-specific: @@ -154,10 +171,18 @@ Run `mxcli docker check -p `, then `mxcli diff-local -p Date: Wed, 12 Aug 2026 13:51:43 +0000 Subject: [PATCH 35/35] Persist Mendix's model repairs without collapsing MPR v2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mx update-widgets` and `mx rename-design-properties` each fix something only Mendix can fix — CE0463 and CE6087, the normal aftermath of a headless module install — and each rewrites an MPR v2 project into the single-file v1 format while doing it. Measured on 11.12.1: update-widgets took 369 .mxunit files to 0; rename-design-properties took 1,865 to 0 and a 249,856-byte index to 39,895,040 bytes, having renamed 149 design properties across 41 documents. The snapshot-and-restore that already protects update-widgets (#808) cannot be reused. That one is allowed to throw the tool's output away once the check has run; these renames have to persist, so restoring the snapshot would restore the un-renamed model with it. So harvest instead: let the tool convert the project, read every unit back out of the converted file, restore the v2 storage, and write the changed units into it through mxcli's own writer — which is also where canon.Reconcile preserves identity fields and elides the units the tool rewrote without changing anything (ADR-0008), so a second run writes nothing. Copying whole units is safe for the same reason a module transplant is: no binary $ID crosses a unit boundary. Exposed as `mxcli fix widgets` and `mxcli fix design-properties`. Both print the storage count before and after, because a collapse shows up there as a zero and a success message without its own evidence is how this shipped the first time. Every failure path after the tool has run restores the format first, so a failed command leaves the project as it was rather than as a half-converted v1 file. Measured end to end on the vanilla 11.12.1 agent-stack app: a plain `mx check` reported 203 errors (202 CE0463 + 1 CE6087) and 0 after the two commands, with the project still v2 at 1,868 units — reproduced from a restored pre-fix snapshot, so the fix is shown to be the cause. Under MXCLI_ALWAYS_WRITE every one of the 1,868 units round-trips and the model still checks clean. Each new test was verified to fail with the reported symptom when the fix is stubbed out. Also corrects an off-by-one in yesterday's counts: mprcontents/ holds an `mprname` file beside the units, so a raw file count is one higher than the unit count. unitCount now counts only .mxunit, with a test tying it to what the reader reports. --- .claude/skills/fix-issue.md | 4 +- .../mendix/download-marketplace-content.md | 80 ++--- CLAUDE.md | 1 + cmd/mxcli/cmd_fix.go | 154 ++++++++++ cmd/mxcli/cmd_marketplace_install.go | 14 +- cmd/mxcli/cmd_marketplace_update.go | 7 +- cmd/mxcli/docker/check_test.go | 2 +- cmd/mxcli/docker/harvest.go | 260 ++++++++++++++++ cmd/mxcli/docker/harvest_test.go | 283 ++++++++++++++++++ cmd/mxcli/docker/update_widgets.go | 12 +- docs-site/src/guides/marketplace.md | 40 ++- 11 files changed, 796 insertions(+), 61 deletions(-) create mode 100644 cmd/mxcli/cmd_fix.go create mode 100644 cmd/mxcli/docker/harvest.go create mode 100644 cmd/mxcli/docker/harvest_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index c2feec0c3..787158e0e 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -470,6 +470,6 @@ extracting `OffsetExpression`/`LimitExpression`. | `show features` lists nothing for agents or MCP, so `create agent` / `create model` on a pre-11.9 project runs without complaint. Reported alongside it: "`CREATE MODEL` can only author one provider — the writer assigns `MxCloudGenAI` unconditionally" | Two different things, and only the first was real. **The version gap**: the `agent_documents` area did not exist in `sdk/versions/mendix-11.yaml`, so `checkFeature()` had nothing to consult. **The provider claim was wrong**: the writer's assignment is `if m.Provider == ""` — a default, not an override. `Provider: OpenAI` parses, writes and round-trips through `describe model`; so does `Provider: TotallyMadeUp` | `sdk/versions/mendix-11.yaml` (new `agent_documents` area), `mdl/executor/cmd_agenteditor_models.go` + `cmd_agenteditor_write.go` (four `checkFeature` gates), `.claude/skills/mendix/agents.md` + `version-awareness.md` | **Read the assignment's guard before believing "hardcoded"** — `if x == "" { x = default }` and `x = default` are one character apart in a grep and opposite in meaning; the round-trip test settles it in a minute. **Agent doctypes are the one version gate with no downstream safety net**: the documents are custom blobs, mxbuild contains no agent-editor strings at all, so an ungated project builds green and fails only when Studio Pro opens it. **Do not invent the provider allowlist** — the enum lives in a Studio Pro *extension*, not in `generated/metamodel` and not in mxbuild, so a guessed list would reject values Mendix accepts; document that nothing validates it instead. Test `TestAgentDocumentsAreGated`. Reported in mxcli-formula1 FINDINGS §53 | | A marketplace module update leaves the app running the OLD widget code: `show modules` reports the new version, pages reference new widget definitions, and the binaries in `widgets/` are unchanged. Symptoms are whatever the version gap causes at render time, with no build error naming the cause | The update moved only the **model**. A widget module's `.mpk` ships its widget binaries under `widgets/` inside the package, and copying units out of a reference project never touches them. Measured on DataWidgets 3.5.0 → 3.11.3: model updated cleanly, all 10 widget binaries still 3.5.0 (`Datagrid.mpk` project=216193 vs package=166933) | `cmd/mxcli/marketplace/update.go` (`InstallPackageWidgets`, called from `PerformUpdate`) | **Take the widget files from the `.mpk`, not from the reference project** — the reference is a blank app plus the module, so its `widgets/` also holds the template's widgets and copying those overwrites widgets the update has nothing to do with. **Skip zero-size directory entries** in the zip or you write a stray file (the 3.11.3 package has 10 `widgets/` entries and 9 real files). **A module with no widgets cannot expose this**: Administration updated end-to-end and looked completely correct, which is why the gap survived a full slice. Pick a second subject with a different shape before believing an update path works. Related but distinct: after any headless module install `mx update-widgets` is needed (CE0463); and a newer module can want a newer Atlas than the project has (29 × CE6083 on DataWidgets 3.11.3), which `rename-design-properties` does NOT fix — that is a dependency, not a resync | | After a marketplace module update, `mx check` reports `CE6083` "Design property X is not supported by your theme" on the updated module's own widgets, and it survives BOTH `mx update-widgets` and `mx rename-design-properties` (which renames 0) | The update copied only the model and `widgets/`. The design properties are declared in the module's **own** `themesource//web/design-properties.json`, which was still the old version's copy — so the model referenced properties the shipped theme file did not define | `cmd/mxcli/marketplace/update.go` (`InstallPackageFiles` replaces `InstallPackageWidgets`) | **Copy everything the package ships**, excluding only `project.mpr` and `package.xml` — enumerating the directories that seem to matter is what produced two bugs in a row (widgets, then themesource). **CE6083 surviving `rename-design-properties` is the tell**: that command renames properties between Atlas generations, so 0 renamed means the declaration is missing entirely rather than renamed. **Do not diagnose this as a cross-module Atlas dependency** — that was the wrong call here; the declaring module was the one being updated. Guard against `..` in package paths while copying. Measured: DataWidgets 3.5.0 → 3.11.3, 49 files replaced, 29 errors → 0 | -| A project that was MPR v2 is v1 after a marketplace install or update: `mprcontents/` is empty or gone, the `.mpr` jumped to tens of MB, `mxcli diff-local` stops working. The user followed the command's own "Next:" hint, or the docs | The hint said to run bare `mx update-widgets ` to clear the CE0463 a headless install leaves behind. That command performs the resync **and** rewrites a v2 project as v1 — measured on 11.12.1: 370 `.mxunit` files → 0, a 69,632-byte index → 14,405,632 bytes. Same family as the `mx module-import` collapse, arriving through advice rather than through a call | `cmd/mxcli/cmd_marketplace_install.go` + `cmd_marketplace_update.go` (the "Next:" lines now name `mxcli docker check`), `docs-site/src/guides/marketplace.md`, `.claude/skills/mendix/download-marketplace-content.md` | **`mxcli docker check` is the v2-safe resync** — it runs the same `mx update-widgets` under a storage-format snapshot (`cmd/mxcli/docker/update_widgets.go`, #808), so the check sees the resynced model and the project keeps its format. Verified: 370 `.mxunit` / 69,632 bytes before and after. **The resync is then not persisted** — the stored model still holds the old widget definitions and a later `mx check` reports CE0463 again; Studio Pro's "Update all widgets" persists it in v2, and `mxcli widget sync` is the partial headless equivalent (7 of 40 on the reference fixture). Say that rather than implying the resync sticks. **A guard in the code does not cover a string in the help text**: #808 already protected every call site mxcli owns, and the collapse came back as a sentence telling the user to bypass it | +| A project that was MPR v2 is v1 after a marketplace install or update: `mprcontents/` is empty or gone, the `.mpr` jumped to tens of MB, `mxcli diff-local` stops working. The user followed the command's own "Next:" hint, or the docs | The hint said to run bare `mx update-widgets ` to clear the CE0463 a headless install leaves behind. That command performs the resync **and** rewrites a v2 project as v1 — measured on 11.12.1: 369 `.mxunit` files → 0, a 69,632-byte index → 14,405,632 bytes. Same family as the `mx module-import` collapse, arriving through advice rather than through a call | `cmd/mxcli/cmd_marketplace_install.go` + `cmd_marketplace_update.go` (the "Next:" lines now name `mxcli docker check`), `docs-site/src/guides/marketplace.md`, `.claude/skills/mendix/download-marketplace-content.md` | **`mxcli docker check` is the v2-safe resync** — it runs the same `mx update-widgets` under a storage-format snapshot (`cmd/mxcli/docker/update_widgets.go`, #808), so the check sees the resynced model and the project keeps its format. Verified: 369 `.mxunit` / 69,632 bytes before and after. **The resync is then not persisted** — the stored model still holds the old widget definitions and a later `mx check` reports CE0463 again; Studio Pro's "Update all widgets" persists it in v2, and `mxcli widget sync` is the partial headless equivalent (7 of 40 on the reference fixture). Say that rather than implying the resync sticks. **A guard in the code does not cover a string in the help text**: #808 already protected every call site mxcli owns, and the collapse came back as a sentence telling the user to bypass it | | `mxcli marketplace install -p app.mpr` fails with `mx module-import failed: exit status 117` — "The package could not be imported, because it was created with a newer version of Mendix Studio Pro" — after downloading the package and building a reference project, and the message is buried under the command's full flag list | The marketplace publishes new releases against the newest Studio Pro patch within days, and `install` with no `--version` resolves to the latest, so on any project not on the very newest patch the **default** version is the one that cannot be installed. Measured 2026-08-12 on an 11.12.1 project: the latest release of all six agent-stack modules required 11.12.2, published five days earlier. Every version the API returns already carries `minSupportedMendixVersion` — nothing consulted it | `cmd/mxcli/marketplace_compat.go` (`checkMendixCompatibility`, `newestCompatibleVersion`, `mendixVersionOf`), called from `cmd_marketplace_install.go` and `cmd_marketplace_update.go`; `SilenceUsage`/`SilenceErrors` on install/update/diff | **Check what the API already told you before spending a download and a reference build on it** — the refusal was always going to happen, three layers lower and in mxcli's voice rather than Mendix's. **Name the version to use, not just the problem**: `--version 4.1.0 (the newest release built for 11.12.1 or older)` is the whole fix. **Skip, do not refuse, when the check cannot evaluate** (no project version, no published minimum) — same rule as `checkFeature`. **`SilenceErrors` belongs with `SilenceUsage`**: `main()` already prints what `Execute` returns, so silencing only usage leaves every refusal printed twice. Tests `cmd/mxcli/marketplace_compat_test.go`, including the negative case — a check that refuses everything passes the refusal test | -| After a headless module install, `mx check` reports project-level `CE6087` "Design properties have been renamed in your theme and need to be updated" with an empty location, and no mxcli command clears it | The module ships its own design properties; Mendix's fix is `mx rename-design-properties`, which mxcli never runs. Measured on 11.12.1: it renames real work (149 design properties across 41 documents) **and** collapses MPR v2 — 1,866 `.mxunit` files → 0, a 249,856-byte index → 39,895,040 bytes. Third member of the family, after `mx module-import` and `mx update-widgets` | *(open — documented, not fixed)* `.claude/skills/mendix/download-marketplace-content.md`, `docs-site/src/guides/marketplace.md` | **The `update-widgets` snapshot trick does not transfer**: that one restores the pre-run storage because the resync only has to hold for the duration of the check, whereas these renames must **persist**, so restoring undoes the fix. A v2-safe path needs harvesting the changed units out of the collapsed v1 file and writing them back through mxcli's writer — no such helper exists (grep for a v1→v2 conversion returns nothing). **Do not claim a headless fix exists**; on v2 the choices are Studio Pro or accepting the conversion. Distinguish from CE6083, which is a *missing* declaration (fixed by copying the package's `themesource/`) — CE6087 is a *renamed* one | +| After a headless module install, `mx check` reports project-level `CE6087` "Design properties have been renamed in your theme and need to be updated" with an empty location, and no mxcli command clears it — or the same for `CE0463`, which `mxcli docker check` makes pass without actually fixing | The module ships its own design properties; Mendix's fix is `mx rename-design-properties`, which mxcli never ran. Measured on 11.12.1: it does real work (149 design properties across 41 documents) **and** collapses MPR v2 — 1,865 `.mxunit` files to 0, a 249,856-byte index to 39,895,040 bytes. Third member of the family, after `mx module-import` and `mx update-widgets` | `cmd/mxcli/docker/harvest.go` (`RunToolPreservingFormat`), `cmd/mxcli/cmd_fix.go` (`mxcli fix design-properties` / `fix widgets`) | **The `update-widgets` snapshot trick does not transfer** — that one restores the pre-run storage because the resync only has to hold for the duration of the check, whereas these renames must **persist**, so restoring undoes the fix. **Harvest instead**: let the tool convert the project, read every unit back out of the converted file, restore the v2 storage, then write the changed units into it through the writer — which is also where `canon.Reconcile` elides the units the tool rewrote without changing (ADR-0008), so a second run writes 0. Safe for the same reason a module transplant is: no binary `$ID` crosses a unit boundary. **Restore on every failure path after the tool has run**, or a failed command leaves a half-converted v1 file. **Print the storage count before and after** — the collapse this prevents shows up there as a zero, and a success message without its own evidence is how it shipped the first time. **`mprcontents/` is not all units**: it also holds an `mprname` file, so a naive file count is one high and reads as a dropped unit. Measured: 203 errors (202 CE0463 + 1 CE6087) to 0 on a vanilla 11.12.1 app, MPR v2 intact at 1,868 units, reproduced from a restored pre-fix snapshot. Tests `cmd/mxcli/docker/harvest_test.go` — each verified to fail with the reported symptom when the fix is stubbed out | diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index 652c3f764..64700722b 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -11,7 +11,7 @@ These are **CLI commands**, not MDL statements. - User asks whether a marketplace module has been edited locally, or what an upgrade would overwrite - User asks to download a specific `.mpk` (e.g. for CI, or to import in Studio Pro) - User asks which versions of a marketplace item are compatible with their Mendix version -- User reports `CE0463` right after installing or updating a module +- User reports `CE0463` or `CE6087` right after installing or updating a module ## Prerequisites: Authenticate @@ -139,45 +139,51 @@ Dependencies include **widget content**, not only modules — `ConversationalUI` `Markdown viewer` (230248) and `Events` (224259) widget packages, which surface as `CE0462 "Could not find widget ... in the 'widgets' directory"`. -## Step 4 — Resync widget definitions (required after any headless install) +## Step 4 — Repair the model after the install (required, headless) ```bash -mxcli docker check -p app.mpr +mxcli fix widgets -p app.mpr # clears CE0463 +mxcli fix design-properties -p app.mpr # clears CE6087 +mxcli docker check -p app.mpr # confirm ``` -A freshly installed or updated module's pages reference widget definitions the project has -not resynced, so a check reports **CE0463** ("the definition of this widget has changed") -until it is told to. Measured on Administration 4.3.2 → 4.5.0: 11 errors before the -resync, 0 after. This is expected after any headless module install — it is **not** a -mxcli defect, and it is not the CE0463 that `.claude/skills/diagnose-ce0463.md` is for. +A headless install leaves two things for Mendix's own tools to finish, and **neither is +an mxcli defect**: -**Never run bare `mx update-widgets` on an MPR v2 project.** It performs the resync and -converts the project to v1 in the process — measured on 11.12.1: 370 `.mxunit` files -became 0, and a 69,632-byte index became 14,405,632 bytes. `mxcli docker check` runs the same -`mx update-widgets` step with the v2 storage snapshotted and restored around it, so the -check sees the resynced model and the project keeps its format. +- **CE0463** "the definition of this widget has changed" — the project's stored widget + instances are older than the widget packages now sitting beside them. This is not the + CE0463 that `.claude/skills/diagnose-ce0463.md` is about. +- **CE6087** "design properties have been renamed in your theme" — a module references + design properties an older Atlas spelled differently. Project-level: the location in the + check output is empty, so the message alone does not say which module caused it. -The resync is therefore not *persisted*: the check passes, and the stored model still -holds the pre-resync widget definitions, so a later `mx check` reports CE0463 again. -To persist it, open the project in Studio Pro once and use **Update all widgets**. -`mxcli widget sync -p app.mpr` is the headless equivalent, but is **partial** — on the -reference fixture it clears 7 of 40. +Measured end to end on a vanilla 11.12.1 app carrying the agent-editor stack: `mx check` +reported **203 errors** (202 × CE0463 + 1 × CE6087) and **0** after the two commands, with +the project still MPR v2 — 1,868 `.mxunit` files, a 249,856-byte index, before and after. -### CE6087 has no headless fix today — know this before you promise one +### Never run the bare `mx` commands on an MPR v2 project -`CE6087 "Design properties have been renamed in your theme and need to be updated"` appears -after installing modules that ship their own design properties. Mendix's fix is -`mx rename-design-properties`, and it collapses MPR v2 exactly like `update-widgets` does — -measured on 11.12.1: 1,866 `.mxunit` files → 0, a 249,856-byte index → 39,895,040 bytes, -having renamed 149 design properties across 41 documents. +`mx update-widgets` and `mx rename-design-properties` each do the repair **and** rewrite +the project into the single-file v1 format. Measured on 11.12.1: `update-widgets` took 369 +`.mxunit` files to 0 and a 69,632-byte index to 14,405,632 bytes; `rename-design-properties` +took 1,865 files to 0 and a 249,856-byte index to 39,895,040 bytes, while renaming 149 +design properties across 41 documents. The conversion is one-way. -Unlike `update-widgets`, **mxcli has no protected path for it**: `docker check` does not run -it, and snapshot-and-restore would not help anyway, because the renames have to *persist* -where the widget resync does not. So on an MPR v2 project the choices are Studio Pro, or -accepting the v1 conversion. Do not tell a user a headless fix exists. +`mxcli fix …` runs the same tool, reads its result back out, restores the v2 storage, and +writes the changed units into it. It reports the storage count before and after for exactly +that reason — a collapse shows up there as a zero. Re-running is free: the second run +reports 0 units changed (ADR-0008 elision), so the `.mpr` is left byte-identical. -The error is project-level (its location in the check output is empty), so it cannot be -traced to the module that caused it from the message alone. +An MPR v1 project is passed straight through, since these tools write v1 natively. + +### Related commands, and when each is right + +| Command | Persists? | Use | +|---|---|---| +| `mxcli fix widgets` | **yes** | the fix — after any headless install | +| `mxcli fix design-properties` | **yes** | the fix — after any headless install | +| `mxcli docker check` | no | runs the widget resync under a snapshot so the *check* is not tripped by CE0463; the stored model stays stale | +| `mxcli widget sync` | yes, partial | reconciles widget schemas in mxcli's own code; clears 7 of 40 on the reference fixture | ## Step 5 — Before updating: has the module been edited? @@ -268,11 +274,13 @@ Flags: `-p/--project`, `--to ` (required), `--module `, ### Afterwards ```bash -mxcli docker check -p app.mpr # resyncs widgets; expect 0 errors -mxcli diff-local -p app.mpr # review what landed, per document +mxcli fix widgets -p app.mpr # step 4 applies to updates too +mxcli fix design-properties -p app.mpr +mxcli docker check -p app.mpr # expect 0 errors +mxcli diff-local -p app.mpr # review what landed, per document ``` -Measured after the resync: Administration 4.3.2 → 4.5.0 (28 units, 9 identities, 2 grants) +Measured after the repair: Administration 4.3.2 → 4.5.0 (28 units, 9 identities, 2 grants) and DataWidgets 3.5.0 → 3.11.3 (49 files) both reach **0 errors**. ## Worked example: the agent-editor stack on a vanilla app @@ -313,8 +321,10 @@ Four things this run established, none of them obvious from the command list: required 11.12.2 against an 11.12.1 project. 3. **Two dependencies are widgets, and two more are modules the agent skill does not list** (CommunityCommons, and the widget packages). Let the check errors drive it. -4. **The end state is 1 error, not 0** — CE6087, which has no headless fix (above). The - project stays MPR v2 throughout: 1,869 `.mxunit` files, a 249,856-byte index. +4. **The install leaves the model needing repair, and step 4 is not optional.** A plain + `mx check` on the finished project reported 203 errors (202 × CE0463, 1 × CE6087); + `mxcli fix widgets` (62 units) and `mxcli fix design-properties` (42 units) took it to + **0**. The project stays MPR v2 throughout: 1,868 `.mxunit` files, a 249,856-byte index. The ids are a convenience, not an authority: confirm with `search`/`info` rather than trusting them from memory, and note that the listing name never matches the module name. diff --git a/CLAUDE.md b/CLAUDE.md index 308e881f7..6c901add1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -573,6 +573,7 @@ go build -o bin/mxcli ./cmd/mxcli | **Data import** | `import from query '...' into Module.Entity map (...)` | Import from external DB into Mendix app PostgreSQL (batch insert with ID generation) | | **Connector gen** | `sql generate connector into [tables (...)] [views (...)] [exec]` | Auto-generate Database Connector MDL from discovered schema | | **Marketplace drift** | `mxcli marketplace diff -p app.mpr [--to V] [--json]` | Which elements of an installed marketplace module have been edited locally, and what an upgrade would overwrite | +| **Model repair** | `mxcli fix widgets`, `mxcli fix design-properties` | Runs `mx update-widgets` / `mx rename-design-properties` and **persists** the result without their MPR v2 → v1 collapse (harvest: let the tool convert, read the units back, restore v2, write the changed ones through mxcli's writer). Clears CE0463 / CE6087 after a headless install — measured 203 → 0 errors on a vanilla 11.12.1 app | | **Diagnostics** | `mxcli diag [--bundle]` | Session logs, version info, bug report bundles | | **New project** | `mxcli new --version X.Y.Z [--output-dir dir] [--theme none]` | Downloads mxbuild, creates blank project, applies default styling, runs init, installs Linux mxcli for devcontainer | | **Default styling** | `mxcli theme list\|show\|apply\|remove` | Applies a built-in theme (signal/ledger/console) — files under `theme/` only, the model is never touched | diff --git a/cmd/mxcli/cmd_fix.go b/cmd/mxcli/cmd_fix.go new file mode 100644 index 000000000..e018c04e4 --- /dev/null +++ b/cmd/mxcli/cmd_fix.go @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "os" + + modelsdk "github.com/mendixlabs/mxcli" + "github.com/mendixlabs/mxcli/cmd/mxcli/docker" + "github.com/spf13/cobra" +) + +// cmd_fix.go exposes Mendix's own model-fixing tools without their side effect. +// +// `mx rename-design-properties` and `mx update-widgets` each fix something only +// Mendix can fix, and each rewrites an MPR v2 project as v1 while doing it. The +// commands here run the tool, harvest its output, and put it back into the v2 +// project through mxcli's writer — see docker.RunToolPreservingFormat. +// +// Both errors these clear are the normal aftermath of a headless module install, +// not defects: CE0463 because the project has not resynced widget definitions, +// CE6087 because a module's design properties were renamed in a newer Atlas. + +var fixCmd = &cobra.Command{ + Use: "fix", + Short: "Apply Mendix's model-fixing tools while preserving the MPR v2 storage format", + Long: `Run Mendix's own model-fixing tools and keep the project's storage format. + +'mx update-widgets' and 'mx rename-design-properties' both do work that only +Mendix can do, and both rewrite an MPR v2 project into the single-file v1 format +as a side effect — measured on 11.12.1, rename-design-properties turned 1,866 +.mxunit files into 0 and a 69 KB index into 39 MB. + +These subcommands run the tool, read its result back out, restore the v2 +storage, and write the changed units into it with mxcli's own writer. The fix +persists; the format survives. An MPR v1 project is passed straight through.`, +} + +var fixDesignPropertiesCmd = &cobra.Command{ + Use: "design-properties", + Short: "Update renamed design properties (clears CE6087) without collapsing MPR v2", + Long: `Apply 'mx rename-design-properties' to the project, preserving MPR v2. + +CE6087 "Design properties have been renamed in your theme and need to be +updated" appears when a module references design properties an older Atlas +spelled differently. It is the normal aftermath of installing a module that +ships its own design properties, and Mendix's rename tool is the only thing that +fixes it. + +Unlike the widget resync, this fix has to persist, so it cannot be run under a +snapshot-and-restore: the restore would undo the renames. The renamed units are +read back out of the converted file and written into the restored v2 project +instead.`, + Example: ` mxcli fix design-properties -p app.mpr`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runFixTool(cmd, "rename-design-properties", "design properties") + }, + SilenceUsage: true, + SilenceErrors: true, +} + +var fixWidgetsCmd = &cobra.Command{ + Use: "widgets", + Short: "Resync widget definitions (clears CE0463) without collapsing MPR v2", + Long: `Apply 'mx update-widgets' to the project, preserving MPR v2. + +CE0463 "The definition of this widget has changed" is what a project reports +when its stored widget instances are older than the widget packages installed +beside them — the normal state after any headless module or widget install. + +'mxcli docker check' already runs this step, but under a snapshot that is +restored afterwards, so the check passes and the stored model stays stale. This +persists the resync instead, which is what Studio Pro's "Update all widgets" +does. It is also more complete than 'mxcli widget sync', which reconciles widget +schemas itself and clears only part of the same errors.`, + Example: ` mxcli fix widgets -p app.mpr`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runFixTool(cmd, "update-widgets", "widget definitions") + }, + SilenceUsage: true, + SilenceErrors: true, +} + +func init() { + for _, c := range []*cobra.Command{fixDesignPropertiesCmd, fixWidgetsCmd} { + c.Flags().StringP("project", "p", "", "path to the Mendix project (.mpr)") + _ = c.MarkFlagRequired("project") + c.Flags().String("mx", "", "path to the mx binary (default: resolved from the project's Mendix version)") + fixCmd.AddCommand(c) + } + rootCmd.AddCommand(fixCmd) +} + +func runFixTool(cmd *cobra.Command, subcommand, what string) error { + mprPath, _ := cmd.Flags().GetString("project") + if _, err := os.Stat(mprPath); err != nil { + return fmt.Errorf("project not found: %s", mprPath) + } + mxOverride, _ := cmd.Flags().GetString("mx") + out := cmd.OutOrStdout() + + reader, err := modelsdk.Open(mprPath) + if err != nil { + return fmt.Errorf("open project: %w", err) + } + mendixVer, _ := reader.GetMendixVersion() + _ = reader.Close() + + mxPath, err := docker.ResolveMxForVersion(mxOverride, mendixVer) + if err != nil { + return fmt.Errorf("locate mx for Mendix %s: %w\nhint: run 'mxcli setup mxbuild -p %s'", mendixVer, err, mprPath) + } + + fmt.Fprintf(out, "Updating %s in %s (Mendix %s)...\n", what, mprPath, mendixVer) + res, err := docker.RunToolPreservingFormat(mxPath, mprPath, subcommand, out, cmd.ErrOrStderr()) + if err != nil { + return err + } + reportFix(out, what, res) + return nil +} + +func reportFix(out io.Writer, what string, res *docker.HarvestResult) { + if !res.Harvested { + fmt.Fprintf(out, "\nUpdated %s. The project is MPR v1, which this tool writes natively.\n", what) + return + } + + fmt.Fprintf(out, "\nUpdated %s: %d unit(s) changed.\n", what, res.UnitsWritten) + if res.UnitsWritten == 0 { + fmt.Fprintln(out, " Nothing to update — the model was already in sync.") + } + // Print the storage numbers unconditionally. The failure this mechanism + // exists to prevent shows up here as a zero, and a success message that does + // not carry its own evidence is how the collapse shipped in the first place. + fmt.Fprintf(out, " Storage: %d .mxunit file(s), unchanged from %d before (MPR v2 preserved).\n", + res.StorageFiles, res.StorageFilesBefore) + + if !res.StillV2 || res.StorageFiles == 0 { + fmt.Fprintln(out, "\n WARNING: the project is no longer in the MPR v2 layout. Restore it from") + fmt.Fprintln(out, " version control and report this — the storage format should have survived.") + } + for _, a := range res.Added { + fmt.Fprintf(out, " Note: the tool added a unit this did not carry over: %s\n", a) + } + for _, r := range res.Removed { + fmt.Fprintf(out, " Note: the tool removed a unit this did not remove: %s\n", r) + } + fmt.Fprintln(out, "\n Next: 'mxcli docker check -p '") +} diff --git a/cmd/mxcli/cmd_marketplace_install.go b/cmd/mxcli/cmd_marketplace_install.go index e30655eba..3836eb363 100644 --- a/cmd/mxcli/cmd_marketplace_install.go +++ b/cmd/mxcli/cmd_marketplace_install.go @@ -139,7 +139,7 @@ func installWidget(ctx context.Context, client *marketplace.Client, v *marketpla return err } fmt.Fprintf(out, "Installed widget %s into %s\n", v.VersionNumber, dest) - fmt.Fprintln(out, "Run 'mxcli docker check -p ' (or reload in Studio Pro) to pick it up.") + fmt.Fprintln(out, "Run 'mxcli fix widgets -p ' (or reload in Studio Pro) to pick it up.") return nil } @@ -192,11 +192,13 @@ func installModule(ctx context.Context, client *marketplace.Client, v *marketpla moduleName, v.VersionNumber, filepath.Base(mprPath)) fmt.Fprintf(out, " %d units copied, %d bundled file(s) installed.\n", res.UnitsCopied, len(res.FilesInstalled)) - // 'mxcli docker check' resyncs widget definitions (clearing CE0463 on the - // module's pages) and restores the MPR v2 storage format afterwards; bare - // 'mx update-widgets' does the resync but leaves the project as v1. - fmt.Fprintln(out, "\n Next: 'mxcli docker check -p ' (resyncs widget definitions,") - fmt.Fprintln(out, " which a headless install leaves stale, and reports CE0463 until it runs).") + // A headless install leaves the model needing two repairs only Mendix's own + // tools can make (CE0463, CE6087). 'mxcli fix' runs them without the v2 -> + // v1 conversion the bare mx commands perform. + fmt.Fprintln(out, "\n Next, repair what a headless install leaves for Studio Pro to finish:") + fmt.Fprintln(out, " mxcli fix widgets -p # CE0463") + fmt.Fprintln(out, " mxcli fix design-properties -p # CE6087") + fmt.Fprintln(out, " mxcli docker check -p ") return nil } diff --git a/cmd/mxcli/cmd_marketplace_update.go b/cmd/mxcli/cmd_marketplace_update.go index 5f0cbff0d..b66da731c 100644 --- a/cmd/mxcli/cmd_marketplace_update.go +++ b/cmd/mxcli/cmd_marketplace_update.go @@ -258,10 +258,11 @@ func reportUpdate(out io.Writer, r *marketplace.UpdateResult) { // the latter rewrites an MPR v2 project as v1 (measured on 11.12.1 — 200 // .mxunit files to 0, a 69 KB index to 14 MB), while docker check runs the // same step under a storage-format snapshot (#808). - fmt.Fprintln(out, "\n Next: resync widget definitions, or 'mx check' will report CE0463 on the") - fmt.Fprintln(out, " new version's pages (this is expected after any headless module install):") + fmt.Fprintln(out, "\n Next, repair what a headless update leaves behind (expected, not a fault):") + fmt.Fprintln(out, " mxcli fix widgets -p # CE0463") + fmt.Fprintln(out, " mxcli fix design-properties -p # CE6087") + fmt.Fprintln(out, "\n Then validate and review:") fmt.Fprintln(out, " mxcli docker check -p ") - fmt.Fprintln(out, "\n Then review what landed:") fmt.Fprintln(out, " mxcli diff-local -p ") } diff --git a/cmd/mxcli/docker/check_test.go b/cmd/mxcli/docker/check_test.go index 2cabda87f..9ce1d2a4a 100644 --- a/cmd/mxcli/docker/check_test.go +++ b/cmd/mxcli/docker/check_test.go @@ -68,7 +68,7 @@ func TestSnapshotStorageFormat_RestoresV2AfterConversion(t *testing.T) { snapshotGlob := filepath.Join(os.TempDir(), "mxcli-mpr-snapshot-*") leakBefore, _ := filepath.Glob(snapshotGlob) - restore, err := snapshotStorageFormat(mprPath, contentsDir) + _, restore, err := snapshotStorageFormat(mprPath, contentsDir) if err != nil { t.Fatalf("snapshotStorageFormat: %v", err) } diff --git a/cmd/mxcli/docker/harvest.go b/cmd/mxcli/docker/harvest.go new file mode 100644 index 000000000..4c94f0190 --- /dev/null +++ b/cmd/mxcli/docker/harvest.go @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "sort" + + "github.com/mendixlabs/mxcli/modelsdk/canon" + "github.com/mendixlabs/mxcli/sdk/mpr" +) + +// HarvestResult reports what a model-fixing mx tool changed. +type HarvestResult struct { + // Harvested is true when the tool's output had to be carried back into an + // MPR v2 project; false when the project was already v1 and the tool's write + // landed directly. + Harvested bool + // UnitsWritten counts units whose content actually changed. Units the tool + // rewrote without changing anything (a fresh $ID per sub-element, which every + // rebuild mints) are elided by canon.Reconcile and not counted — ADR-0008. + UnitsWritten int + // Added and Removed name units the tool created or deleted, as + // " ". These are reported rather than applied: the tools this runs + // rewrite documents in place, so a non-empty list means the tool did + // something this mechanism was not built for and the caller should say so. + Added []string + Removed []string + // StorageFiles is the number of .mxunit files on disk after the run, and + // StorageFilesBefore what it was before. They are the assertion this whole + // mechanism exists for: a v2 -> v1 conversion takes the count to zero, so a + // caller that prints both cannot report success on a collapsed project. + StorageFilesBefore int + StorageFiles int + // StillV2 is false if the project is not in the v2 layout afterwards. It + // should never be false when Harvested is true; the commands check it rather + // than assume it. + StillV2 bool +} + +// mxToolCmd runs the mx invocation. A package variable so tests can substitute a +// stub that simulates the v2 -> v1 conversion without needing mx on the box. +var mxToolCmd = func(mxPath string, args []string, w, stderr io.Writer) error { + cmd := exec.Command(mxPath, args...) + cmd.Stdout = w + cmd.Stderr = stderr + PrepareMxCommand(cmd) + return cmd.Run() +} + +// RunToolPreservingFormat runs an mx model-fixing tool (`rename-design-properties`, +// `update-widgets`, ...) and lands its result in the project **without** leaving +// the project converted to MPR v1. +// +// Three of Mendix's own commands rewrite an MPR v2 project as v1 as a side +// effect of doing their job — `module-import`, `update-widgets` and +// `rename-design-properties`. Measured on 11.12.1, `rename-design-properties` +// turned 1,865 `.mxunit` files into 0 and a 249,856-byte index into 39,895,040 +// bytes, while renaming 149 design properties across 41 documents. The renames +// are real work that only Mendix can do; the conversion is collateral. +// +// `runUpdateWidgets` solves its half of this by snapshotting the v2 storage and +// restoring it afterwards, which is correct **there** because the widget resync +// only has to hold for the duration of the check that follows. That trick does +// not transfer to a fix whose result must persist: restoring the snapshot would +// restore the un-renamed model too. +// +// So this harvests instead. The tool is allowed to convert the project, every +// unit is read back out of the converted file, the v2 storage is restored, and +// the changed units are written into it through mxcli's own writer — which is +// also what keeps the write honest, because that writer is the choke point where +// canon.Reconcile preserves identity fields and elides units that did not really +// change (ADR-0008). Copying whole units is safe for the same reason a module +// transplant is: no binary `$ID` pointer crosses a unit boundary. +// +// An MPR v1 project needs none of this and is handed straight to the tool. +// +// On any failure after the tool has run, the v2 storage is restored before +// returning, so a failed harvest leaves the project as it was rather than as a +// half-converted v1 file. +func RunToolPreservingFormat(mxPath, projectPath, subcommand string, w, stderr io.Writer) (*HarvestResult, error) { + abs, err := filepath.Abs(projectPath) + if err != nil { + abs = projectPath + } + args := []string{subcommand, abs} + + reader, err := mpr.Open(projectPath) + if err != nil { + return nil, fmt.Errorf("open %s: %w", projectPath, err) + } + isV2 := reader.Version() == mpr.MPRVersionV2 + contentsDir := reader.ContentsDir() + _ = reader.Close() + + if !isV2 { + // v1 is what these tools already produce; nothing to protect. + if err := mxToolCmd(mxPath, args, w, stderr); err != nil { + return nil, fmt.Errorf("mx %s: %w", subcommand, err) + } + return &HarvestResult{}, nil + } + before := unitCount(contentsDir) + + _, restore, err := snapshotStorageFormat(projectPath, contentsDir) + if err != nil { + // Without a snapshot the conversion would be unrecoverable. Refuse rather + // than trade the storage format for the fix. + return nil, fmt.Errorf("snapshot MPR v2 storage: %w\n"+ + " refusing to run 'mx %s', which would convert the project to MPR v1 with no way back", err, subcommand) + } + + if err := mxToolCmd(mxPath, args, w, stderr); err != nil { + restore() + return nil, fmt.Errorf("mx %s: %w", subcommand, err) + } + + fixed, order, err := readAllUnits(projectPath) + if err != nil { + restore() + return nil, fmt.Errorf("read back what 'mx %s' produced: %w", subcommand, err) + } + + // Back to v2, pre-fix, with the tool's output held in memory. + restore() + + res, err := applyHarvest(projectPath, fixed, order) + if err != nil { + return nil, err + } + res.StorageFilesBefore = before + res.StorageFiles = unitCount(contentsDir) + res.StillV2 = verifyStillV2(projectPath) + return res, nil +} + +// harvestedUnit is a unit as the tool left it, with the placement metadata an +// insert needs. +type harvestedUnit struct { + contents []byte + containerID string + containmentName string + unitType string +} + +// readAllUnits reads every unit out of a project, returning them by ID plus a +// stable ordering. The order is the reader's, which is the storage order — not +// map order, which would make the write sequence (and any failure point) +// unreproducible. +func readAllUnits(projectPath string) (map[string]harvestedUnit, []string, error) { + reader, err := mpr.Open(projectPath) + if err != nil { + return nil, nil, err + } + defer reader.Close() + + units, err := reader.ListUnits() + if err != nil { + return nil, nil, err + } + + out := make(map[string]harvestedUnit, len(units)) + order := make([]string, 0, len(units)) + for _, u := range units { + raw, rerr := reader.GetRawUnitBytes(u.ID) + if rerr != nil || len(raw) == 0 { + // A unit that cannot be read cannot be carried over. Skipping it leaves + // the stored version in place, which is the safe direction. + continue + } + id := string(u.ID) + out[id] = harvestedUnit{ + contents: append([]byte{}, raw...), + containerID: string(u.ContainerID), + containmentName: u.ContainmentName, + unitType: u.Type, + } + order = append(order, id) + } + return out, order, nil +} + +// applyHarvest writes the tool's units into the restored v2 project, skipping +// the ones that did not really change. +func applyHarvest(projectPath string, fixed map[string]harvestedUnit, order []string) (*HarvestResult, error) { + res := &HarvestResult{Harvested: true} + + writer, err := mpr.NewWriter(projectPath) + if err != nil { + return nil, fmt.Errorf("open %s for writing: %w", projectPath, err) + } + defer writer.Close() + + stored, _, err := readAllUnits(projectPath) + if err != nil { + return nil, fmt.Errorf("read the restored project: %w", err) + } + + for _, id := range order { + h := fixed[id] + prev, ok := stored[id] + if !ok { + res.Added = append(res.Added, fmt.Sprintf("%s %s", h.unitType, id)) + continue + } + out, unchanged := canon.Reconcile(h.contents, prev.contents) + if unchanged { + continue + } + if err := writer.UpdateRawUnit(id, out); err != nil { + return nil, fmt.Errorf("write unit %s: %w", id, err) + } + res.UnitsWritten++ + } + + for id, s := range stored { + if _, ok := fixed[id]; !ok { + res.Removed = append(res.Removed, fmt.Sprintf("%s %s", s.unitType, id)) + } + } + sort.Strings(res.Added) + sort.Strings(res.Removed) + return res, nil +} + +// verifyStillV2 reports whether the project is on disk in the MPR v2 layout. The +// commands built on RunToolPreservingFormat assert this after they run: the +// whole point is that the format survives, and an assertion is cheaper than a +// bug report. +func verifyStillV2(projectPath string) bool { + reader, err := mpr.Open(projectPath) + if err != nil { + return false + } + defer reader.Close() + return reader.Version() == mpr.MPRVersionV2 +} + +// unitCount is used by the commands to report the storage size before and after, +// which is the number a v2 -> v1 conversion moves to zero. +// +// Only `.mxunit` files are counted. `mprcontents/` also holds an `mprname` +// metadata file, and counting it makes the reported storage size one higher than +// the unit count every other part of mxcli prints — which reads as a unit having +// been dropped, and costs whoever notices an investigation to find out it was +// not. +func unitCount(contentsDir string) int { + n := 0 + _ = filepath.WalkDir(contentsDir, func(path string, d os.DirEntry, err error) error { + if err == nil && !d.IsDir() && filepath.Ext(path) == ".mxunit" { + n++ + } + return nil + }) + return n +} diff --git a/cmd/mxcli/docker/harvest_test.go b/cmd/mxcli/docker/harvest_test.go new file mode 100644 index 000000000..496135c16 --- /dev/null +++ b/cmd/mxcli/docker/harvest_test.go @@ -0,0 +1,283 @@ +// SPDX-License-Identifier: Apache-2.0 + +// `mx rename-design-properties` fixes something only Mendix can fix (CE6087) and +// rewrites an MPR v2 project as v1 while doing it — measured on 11.12.1, 1,865 +// .mxunit files to 0. The snapshot-and-restore that protects `update-widgets` +// (#808) cannot be reused, because that one is allowed to throw the tool's output +// away once the check has run, while these renames have to persist. +// +// So RunToolPreservingFormat harvests: it lets the tool convert the project, +// reads every unit back out, restores the v2 storage, and writes the changed +// units into it. These tests pin the three things that makes load-bearing — the +// tool's change survives, the format survives, and a failure anywhere leaves the +// project as it was rather than as a half-converted v1 file. +package docker + +import ( + "bytes" + "errors" + "io" + "os" + "path/filepath" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/mpr" + "go.mongodb.org/mongo-driver/bson" +) + +// stubMxTool swaps the mx invocation for the duration of a test. +func stubMxTool(t *testing.T, fn func(mxPath string, args []string, w, stderr io.Writer) error) { + t.Helper() + prev := mxToolCmd + mxToolCmd = fn + t.Cleanup(func() { mxToolCmd = prev }) +} + +// firstUnitID returns some unit of the project, for a test to mutate. +func firstUnitID(t *testing.T, mprPath string) string { + t.Helper() + reader, err := mpr.Open(mprPath) + if err != nil { + t.Fatalf("open %s: %v", mprPath, err) + } + defer reader.Close() + units, err := reader.ListUnits() + if err != nil || len(units) == 0 { + t.Fatalf("fixture has no units (err=%v)", err) + } + return string(units[0].ID) +} + +// markUnit is what the tool "doing its job" looks like: a content change to one +// unit. A canonical comparison must see it — it is a property, not an $ID. +func markUnit(t *testing.T, mprPath, unitID, marker string) { + t.Helper() + reader, err := mpr.Open(mprPath) + if err != nil { + t.Fatalf("open for marking: %v", err) + } + raw, err := reader.GetRawUnitBytes(model.ID(unitID)) + _ = reader.Close() + if err != nil { + t.Fatalf("read unit %s: %v", unitID, err) + } + var doc bson.D + if err := bson.Unmarshal(raw, &doc); err != nil { + t.Fatalf("decode unit %s: %v", unitID, err) + } + doc = append(doc, bson.E{Key: "HarvestTestMarker", Value: marker}) + encoded, err := bson.Marshal(doc) + if err != nil { + t.Fatalf("encode unit %s: %v", unitID, err) + } + writer, err := mpr.NewWriter(mprPath) + if err != nil { + t.Fatalf("open for writing: %v", err) + } + defer writer.Close() + if err := writer.UpdateRawUnit(unitID, encoded); err != nil { + t.Fatalf("write unit %s: %v", unitID, err) + } +} + +// unitMarker reads back what markUnit wrote, or "" when absent. +func unitMarker(t *testing.T, mprPath, unitID string) string { + t.Helper() + reader, err := mpr.Open(mprPath) + if err != nil { + t.Fatalf("open for reading marker: %v", err) + } + defer reader.Close() + raw, err := reader.GetRawUnitBytes(model.ID(unitID)) + if err != nil { + t.Fatalf("read unit %s: %v", unitID, err) + } + var doc bson.D + if err := bson.Unmarshal(raw, &doc); err != nil { + t.Fatalf("decode unit %s: %v", unitID, err) + } + for _, e := range doc { + if e.Key == "HarvestTestMarker" { + s, _ := e.Value.(string) + return s + } + } + return "" +} + +// The core property: the tool's change lands in the project even though the v2 +// storage is restored from a snapshot taken before the tool ran. Get this wrong +// and the command reports success while silently discarding the fix — which is +// exactly what would happen if runUpdateWidgets' snapshot/restore were reused +// here unchanged. +func TestRunToolPreservingFormat_CarriesTheToolsChangeBack(t *testing.T) { + mprPath := v2Fixture(t) + unitID := firstUnitID(t, mprPath) + + stubMxTool(t, func(_ string, _ []string, _, _ io.Writer) error { + markUnit(t, mprPath, unitID, "renamed") + return nil + }) + + res, err := RunToolPreservingFormat("mx", mprPath, "rename-design-properties", io.Discard, io.Discard) + if err != nil { + t.Fatalf("RunToolPreservingFormat: %v", err) + } + if got := unitMarker(t, mprPath, unitID); got != "renamed" { + t.Errorf("the tool's change did not survive the restore: marker=%q, want %q", got, "renamed") + } + if res.UnitsWritten != 1 { + t.Errorf("UnitsWritten = %d, want 1 — only the unit the tool touched should be rewritten", res.UnitsWritten) + } + if !res.Harvested || !res.StillV2 { + t.Errorf("Harvested=%v StillV2=%v, want both true", res.Harvested, res.StillV2) + } + if v := storageVersion(t, mprPath); v != mpr.MPRVersionV2 { + t.Errorf("project is %v afterwards, want MPRv2 — the format is the whole point", v) + } +} + +// The negative half. A mechanism that rewrites every unit would also pass the +// test above; ADR-0008 requires that a unit whose content did not change is not +// written, or a fix run against an in-sync project churns the whole model. +func TestRunToolPreservingFormat_WritesNothingWhenTheToolChangedNothing(t *testing.T) { + mprPath := v2Fixture(t) + + stubMxTool(t, func(_ string, _ []string, _, _ io.Writer) error { return nil }) + + res, err := RunToolPreservingFormat("mx", mprPath, "update-widgets", io.Discard, io.Discard) + if err != nil { + t.Fatalf("RunToolPreservingFormat: %v", err) + } + if res.UnitsWritten != 0 { + t.Errorf("UnitsWritten = %d, want 0 — a no-op tool run must not rewrite the model", res.UnitsWritten) + } + if res.StorageFiles == 0 || res.StorageFiles != res.StorageFilesBefore { + t.Errorf("storage went %d -> %d; a v2 -> v1 collapse shows up exactly here", + res.StorageFilesBefore, res.StorageFiles) + } +} + +// The control ADR-0008 requires for any "nothing changed" assertion: with +// elision off, the same run writes everything. Without this, the test above also +// passes a harvest that read no units at all and had nothing to offer. +func TestRunToolPreservingFormat_AlwaysWriteProvesUnitsWereRead(t *testing.T) { + mprPath := v2Fixture(t) + t.Setenv("MXCLI_ALWAYS_WRITE", "1") + + stubMxTool(t, func(_ string, _ []string, _, _ io.Writer) error { return nil }) + + res, err := RunToolPreservingFormat("mx", mprPath, "update-widgets", io.Discard, io.Discard) + if err != nil { + t.Fatalf("RunToolPreservingFormat: %v", err) + } + if res.UnitsWritten == 0 { + t.Fatal("MXCLI_ALWAYS_WRITE wrote 0 units — the harvest is reading nothing, " + + "so the 0 in the elision test means 'no data', not 'no change'") + } + if res.UnitsWritten != res.StorageFiles { + t.Errorf("wrote %d of %d units under MXCLI_ALWAYS_WRITE; every unit should be offered", + res.UnitsWritten, res.StorageFiles) + } +} + +// A tool that fails after converting the project must not leave it converted. +// This is the difference between a failed command and a broken repository. +func TestRunToolPreservingFormat_RestoresFormatWhenTheToolFails(t *testing.T) { + mprPath := v2Fixture(t) + contentsDir := filepath.Join(filepath.Dir(mprPath), "mprcontents") + + stubMxTool(t, func(_ string, _ []string, _, _ io.Writer) error { + convertToV1(t, mprPath) + return errors.New("mx exploded") + }) + + if _, err := RunToolPreservingFormat("mx", mprPath, "rename-design-properties", io.Discard, io.Discard); err == nil { + t.Fatal("expected the tool's failure to surface") + } + if _, err := os.Stat(contentsDir); err != nil { + t.Fatalf("mprcontents/ was not restored after a failed tool run: %v", err) + } + if v := storageVersion(t, mprPath); v != mpr.MPRVersionV2 { + t.Errorf("project left as %v after a failed run, want MPRv2", v) + } +} + +// Same, one step later: the tool succeeds but leaves something the reader cannot +// open. The harvest has to give up *and* put the format back. +func TestRunToolPreservingFormat_RestoresFormatWhenTheOutputIsUnreadable(t *testing.T) { + mprPath := v2Fixture(t) + + stubMxTool(t, func(_ string, _ []string, _, _ io.Writer) error { + convertToV1(t, mprPath) // writes a .mpr that is not a database at all + return nil + }) + + if _, err := RunToolPreservingFormat("mx", mprPath, "rename-design-properties", io.Discard, io.Discard); err == nil { + t.Fatal("expected unreadable tool output to be an error, not a silent no-op") + } + if v := storageVersion(t, mprPath); v != mpr.MPRVersionV2 { + t.Errorf("project left as %v after an unreadable harvest, want MPRv2", v) + } +} + +// An MPR v1 project needs no protection: these tools write v1 natively. Snapshot +// and harvest would be pure cost, and the result must say so rather than claim a +// format was preserved that was never at risk. +func TestRunToolPreservingFormat_PassesV1StraightThrough(t *testing.T) { + mprPath := v1Fixture(t) + + ran := false + stubMxTool(t, func(_ string, args []string, _, _ io.Writer) error { + ran = true + if len(args) == 0 || args[0] != "update-widgets" { + t.Errorf("args = %v, want the subcommand first", args) + } + if !filepath.IsAbs(args[len(args)-1]) { + t.Errorf("project path %q is not absolute; MxToolset skips the step on a bare filename", args[len(args)-1]) + } + return nil + }) + + res, err := RunToolPreservingFormat("mx", mprPath, "update-widgets", io.Discard, io.Discard) + if err != nil { + t.Fatalf("RunToolPreservingFormat: %v", err) + } + if !ran { + t.Fatal("the tool was not run on an MPRv1 project") + } + if res.Harvested { + t.Error("Harvested = true on an MPRv1 project; nothing needed harvesting") + } +} + +// The reported storage count must match what the rest of mxcli calls a unit. +// mprcontents/ also holds an `mprname` metadata file; counting it makes every +// report one too high, which reads as a dropped unit. +func TestUnitCount_CountsOnlyMxunitFiles(t *testing.T) { + mprPath := v2Fixture(t) + contentsDir := filepath.Join(filepath.Dir(mprPath), "mprcontents") + + reader, err := mpr.Open(mprPath) + if err != nil { + t.Fatal(err) + } + units, err := reader.ListUnits() + _ = reader.Close() + if err != nil { + t.Fatal(err) + } + + if got := unitCount(contentsDir); got != len(units) { + var all bytes.Buffer + _ = filepath.WalkDir(contentsDir, func(p string, d os.DirEntry, err error) error { + if err == nil && !d.IsDir() && filepath.Ext(p) != ".mxunit" { + all.WriteString(" " + filepath.Base(p)) + } + return nil + }) + t.Errorf("unitCount = %d but the project has %d units; non-.mxunit files present:%s", + got, len(units), all.String()) + } +} diff --git a/cmd/mxcli/docker/update_widgets.go b/cmd/mxcli/docker/update_widgets.go index 748bc8998..8dcb1c6a7 100644 --- a/cmd/mxcli/docker/update_widgets.go +++ b/cmd/mxcli/docker/update_widgets.go @@ -74,7 +74,7 @@ func runUpdateWidgets(mxPath, projectPath string, w, stderr io.Writer) (restore contentsDir := reader.ContentsDir() reader.Close() if isV2 { - snapRestore, snapErr := snapshotStorageFormat(projectPath, contentsDir) + _, snapRestore, snapErr := snapshotStorageFormat(projectPath, contentsDir) if snapErr != nil { // Can't protect the format — skip update-widgets rather than risk an // unrecoverable v2 -> v1 conversion. A CE0463 false positive is the @@ -103,22 +103,22 @@ func runUpdateWidgets(mxPath, projectPath string, w, stderr io.Writer) (restore // restore function removes the temp directory and is safe to defer; it best-effort // restores and never panics. mprPath and contentsDir come from an mpr.Reader on a // project already known to be MPRv2. -func snapshotStorageFormat(mprPath, contentsDir string) (restore func(), err error) { +func snapshotStorageFormat(mprPath, contentsDir string) (dir string, restore func(), err error) { tmp, err := os.MkdirTemp("", "mxcli-mpr-snapshot-*") if err != nil { - return nil, err + return "", nil, err } mprBackup := filepath.Join(tmp, filepath.Base(mprPath)) if err := copyFile(mprPath, mprBackup); err != nil { os.RemoveAll(tmp) - return nil, err + return "", nil, err } contentsBackup := filepath.Join(tmp, "mprcontents") if err := copyDir(contentsDir, contentsBackup); err != nil { os.RemoveAll(tmp) - return nil, err + return "", nil, err } restore = func() { @@ -130,5 +130,5 @@ func snapshotStorageFormat(mprPath, contentsDir string) (restore func(), err err _ = os.RemoveAll(contentsDir) _ = copyDir(contentsBackup, contentsDir) } - return restore, nil + return tmp, restore, nil } diff --git a/docs-site/src/guides/marketplace.md b/docs-site/src/guides/marketplace.md index d7799f2c4..fb5628ace 100644 --- a/docs-site/src/guides/marketplace.md +++ b/docs-site/src/guides/marketplace.md @@ -167,21 +167,45 @@ Local edits are **not** preserved. `update` refuses when it finds any, `--save-e ### Afterwards -Run `mxcli docker check -p `, then `mxcli diff-local -p `. +```bash +mxcli fix widgets -p app.mpr +mxcli fix design-properties -p app.mpr +mxcli docker check -p app.mpr +mxcli diff-local -p app.mpr +``` + +A headless install or update leaves two repairs for Mendix's own tools: **CE0463** (the project's stored widget instances are older than the widget packages beside them) and **CE6087** (a module references design properties an older Atlas spelled differently). Both are expected, not faults in the install. See [`mxcli fix`](#repairing-the-model-mxcli-fix) below. + +Measured: Administration 4.3.2 → 4.5.0 and DataWidgets 3.5.0 → 3.11.3 both reach **0 errors** afterwards. + +## Repairing the model (`mxcli fix`) -A newer module's pages reference widget definitions the project has not resynced, so a check reports CE0463 until it is told to — measured on Administration 4.3.2 → 4.5.0: 11 errors before the resync, 0 after. This is expected after any headless module install, not a fault in the update. Measured after the resync, Administration 4.3.2 → 4.5.0 and DataWidgets 3.5.0 → 3.11.3 both reach **0 errors**. +`mx update-widgets` and `mx rename-design-properties` each fix something only Mendix can fix, and each rewrites an MPR v2 project into the single-file v1 format while doing it. Measured on 11.12.1: `update-widgets` took 369 `.mxunit` files to 0 and a 69,632-byte index to 14,405,632 bytes; `rename-design-properties` took 1,865 files to 0 and a 249,856-byte index to 39,895,040 bytes, having renamed 149 design properties across 41 documents. The conversion is one-way. -**Do not run bare `mx update-widgets` on an MPR v2 project.** It performs the resync but rewrites the project as v1 — measured on 11.12.1, 370 `.mxunit` files became 0 and a 69,632-byte index became 14,405,632 bytes. `mxcli docker check` runs the same `mx update-widgets` step with the v2 storage snapshotted and restored around it, so the check sees the resynced model and the project keeps its format. +```bash +mxcli fix widgets -p app.mpr # CE0463 +mxcli fix design-properties -p app.mpr # CE6087 +``` + +```text +Updated design properties: 42 unit(s) changed. + Storage: 1868 .mxunit file(s), unchanged from 1868 before (MPR v2 preserved). +``` -The consequence is that the resync is not *persisted*: the check passes, and the stored model still holds the pre-resync widget definitions, so a later `mx check` reports CE0463 again. Opening the project in Studio Pro once and using **Update all widgets** persists it in v2. `mxcli widget sync` is the headless equivalent but is partial — on the reference fixture it clears 7 of 40. +Each runs the same Mendix tool, reads every unit back out of the converted file, restores the v2 storage, and writes the changed units into it through mxcli's own writer. The storage count is printed before and after because that is where the failure this exists to prevent would show up — as a zero. -### CE6087 has no headless fix +Measured end to end on a vanilla 11.12.1 app carrying the agent-editor stack: `mx check` reported **203 errors** (202 × CE0463 + 1 × CE6087), and **0** after the two commands, with the project still MPR v2 (1,868 `.mxunit` files) — reproduced from a restored pre-fix snapshot. -Installing modules that ship their own design properties can leave a project-level `CE6087` — "Design properties have been renamed in your theme and need to be updated". Mendix's fix is `mx rename-design-properties`, and it collapses MPR v2 the same way: measured on 11.12.1, 1,866 `.mxunit` files became 0 and a 249,856-byte index became 39,895,040 bytes, having renamed 149 design properties across 41 documents. +Re-running is free: a second run reports 0 units changed, because [idempotent writes](../internals/idempotent-writes.md) elide a unit whose content did not really change. An MPR v1 project is passed straight through, since these tools write v1 natively. -The snapshot-and-restore that protects `update-widgets` does not transfer here, because these renames have to persist where a widget resync does not. On an MPR v2 project the options today are Studio Pro, or accepting the conversion to v1. +| Command | Persists? | Use | +|---|---|---| +| `mxcli fix widgets` | **yes** | the fix — after any headless install | +| `mxcli fix design-properties` | **yes** | the fix — after any headless install | +| `mxcli docker check` | no | runs the widget resync under a snapshot so the *check* is not tripped by CE0463; the stored model stays stale | +| `mxcli widget sync` | yes, partial | reconciles widget schemas in mxcli's own code; clears 7 of 40 on the reference fixture | -This is distinct from `CE6083`, which is a *missing* design-property declaration and is fixed by installing everything the package ships — something `install` and `update` already do. +CE6087 is distinct from `CE6083`, which is a *missing* design-property declaration and is fixed by installing everything the package ships — something `install` and `update` already do. `update` does **not** roll back. Work on a copy or have the project in version control.