From cff682382918c2565263d85dd8c3a25bac37b7fa Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 19:38:09 +0000 Subject: [PATCH 1/7] fix(new): create the project in a staging directory, so a deep --output-dir works MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mxcli new` pointed `mx create-project` straight at the output directory. MxToolset refuses any full destination path over 259 characters — its own Windows-compatibility limit, not the filesystem's — and aborts extraction PART WAY THROUGH when it hits that, so a deep output directory was left holding a few hundred files and no .mpr: output that looks like a project until you open it. Bisecting the output-directory length on mxbuild 11.13.0 pins the rule exactly: 77 characters -> project created, .mpr present 78 characters -> PathTooLongException, 259 files left behind, no .mpr The blank template's longest relative path is 181 characters there, and 77 + 1 + 181 = 259. The reporter measured 182 on 11.12.0 and saw 258 files, so the constant drifts a character per release. Rather than validate a path we can avoid, create the project in a short staging directory and move it into place. Two things follow: * A deep destination WORKS — the limit is MxToolset's, and POSIX allows 4096. Relocation is safe because a fresh project embeds no absolute paths: grepping a blank 11.13 project for its own directory matches zero files. * The destination is never partially populated. Nothing is written there until creation has already succeeded, so a failure leaves the user's directory exactly as it was. The template's longest path is measured from the staged tree rather than hardcoded, and used for a warning — not a refusal — when the finished project exceeds 259: it works here, but Studio Pro on Windows may not open it, and refusing would block a machine where it is fine. The warning carries both numbers and the budget, so nobody has to derive the limit themselves the way the reporter did. Verified against mxbuild 11.13.0 at the length that used to fail: build exit result pre-fix 1 PathTooLongException, 259 orphaned files, 0 .mpr fixed 0 project created, warning emitted, `mx check` 0 errors Cross-device staging falls back to os.CopyFS, which os.Rename cannot do. Fixes #825 --- .claude/skills/fix-issue.md | 1 + .claude/skills/verify-in-runtime.md | 16 ++- cmd/mxcli/cmd_new.go | 37 ++++++- cmd/mxcli/newproject_paths.go | 141 +++++++++++++++++++++++++ cmd/mxcli/newproject_paths_test.go | 154 ++++++++++++++++++++++++++++ 5 files changed, 342 insertions(+), 7 deletions(-) create mode 100644 cmd/mxcli/newproject_paths.go create mode 100644 cmd/mxcli/newproject_paths_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index f75d4e0bc..26a1f45b3 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -472,3 +472,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A datagrid column bound to an association — `column c (attribute: Order_Customer)` — reports success from `mxcli exec` and then fails the build with `[error] [CE1613] "The selected attribute 'Mod.Order.Order_Customer' no longer exists." at Columns (1/1) of data grid 2`. Separately, there is no MDL spelling for the drop-down filter's association mode: `mxcli check` says `[MDL-WIDGET01] has no property \`refEntity\`` | Two unrelated defects behind one report. (1) The reference is **not representable**: `CustomWidgets$WidgetValue.AttributeRef` is typed `AttributeRef`, not the polymorphic `MemberRef`, so the association was qualified like an attribute and written as a dangling `AttributeRef`. (2) `dropdownfilter.def.json` mapped only `attrChoice`/`attributes`/`defaultFilter`, so every `baseType: 'ref'` property was unmapped and dropped | `mdl/executor/cmd_pages_builder_input.go` (`rejectAssociationAsAttribute`, `entityInChain`) wired into `mdl/executor/widget_engine.go` (the objectlist `attribute` case and the `Attribute` source); `sdk/widgets/definitions/dropdownfilter.def.json` (association mode); `mdl/executor/cmd_pages_describe_parse.go` + `_pluggable.go` + `_output.go` (round-trip) | **Establish that a shape is unrepresentable before designing a fix for it** — hand-patch the BSON and run `mx check`. A `DomainModels$AssociationRef` in that slot makes the project **UNLOADABLE** (`ArgumentException: Object of type 'AssociationRef' cannot be converted to type 'AttributeRef'`), and the assembly defining the type (`Mendix.Modeler.WebUI.dll`) has no `AssociationRef` member at all — so the only correct outcome is a refusal carrying both working forms. **`` on an ATTRIBUTE-typed widget property is permission to TRAVERSE a reference, not to bind one** — `attribute: Assoc/Attr` already worked and is what the XML is advertising; the DataGrid column is the only shipped widget where the two are easy to confuse. **A def.json `mode` is the whole feature** for an unauthorable widget mode — the engine already had the `association` operation and the `hasDataSource` condition, so the second half was a data change plus its DESCRIBE reader (without which describe→edit→exec silently reverts the filter to attribute mode). **0 errors from `mx check` does not prove the properties landed** — an unmapped property is silently dropped and the build is just as green; read them back with `mx dump-mpr`. Tests `cmd_pages_builder_assoc_as_attribute_test.go`, `widget_dropdownfilter_assoc_test.go`, example `mdl-examples/bug-tests/830-datagrid-association-filter.mdl`. upstream #830 | | An association's line anchors — where the connector attaches to the entity boxes in the domain model editor — are absent from `DESCRIBE ASSOCIATION`, and manual adjustments made in Studio Pro do not survive an mxcli round trip | `DomainModels$Association.ParentConnection`/`ChildConnection` (the string `"x;y"`) were **hardcoded** to `"0;50"`/`"100;50"` in BOTH writers and never read by either parser. Because every association write rebuilds the whole element, this was not an omission but active destruction: a documentation-only `alter association … set comment` reset them | `sdk/domainmodel/connection.go` (new: `ParseConnectionPoint`/`FormatConnectionPoint`, `Default*Connection`), `sdk/domainmodel/domainmodel.go` (fields → `*model.Point`), `sdk/mpr/parser_domainmodel.go` + `sdk/mpr/writer_domainmodel.go`, `mdl/backend/modelsdk/domainmodel.go` + `domainmodel_write.go`, `mdl/executor/cmd_associations.go` (`describeConnectionPoints`) | **A feature request that says "X is not exposed" may be hiding "X is destroyed"** — check the write path before scoping the read path. The A/B that settled it: a blank 11.13 app's own `Administration.AccountPasswordData_Account` stores `0;54/100;54`, so a Studio-Pro-authored association is a free fixture for "did mxcli overwrite this?" — no Studio Pro needed. **Learn the value's constraints from the LOADER, not from the shape**: hand-patch and run `mx check` — `"0.5;50"` dies with `StorageLoadException` (integers required) while `"0;500"` and `"-20;50"` load with 0 errors (no range check), so out-of-range values must round-trip untouched. **A zero value is not an absent value** — `{0,0}` is a real anchor (top-left), which forces the field to be a POINTER; a plain `model.Point` cannot distinguish "unset" from "top-left" and would silently rewrite it. **Fix both engines**: they share the semantic model, and a fix in one is invisible to a user on the other. **Emit unauthorable data as a COMMENT** — DESCRIBE output must stay re-executable, and inventing syntax (`@anchor(parent: bottom-left, …)`) would bake in a vocabulary the storage does not have: the pair is CONTINUOUS, not 8 named anchors (observed x values 0 9 11 17 18 47 49 50 65 77 78 84 87 100). **The marketplace is the sample** when you need to know what Studio Pro actually writes: `mxcli marketplace download ` gives real Mendix-authored models, and a module .mpk holds either a raw BSON `project.mpr` or an MPR v1 SQLite one — 88 coordinate pairs from three modules turned "looks like percentages" into a measurement (all 0..100; 85 of 88 pin one coordinate to exactly 0 or 100). **Rule a unit out from the model, not the values**: pixels is impossible because `DomainModels$EntityImpl` stores only `Location` and NO size — the box is sized by the editor from the name and attribute list, so a pixel anchor would have nothing to measure against. Not applicable to `CrossAssociation`, which has no connection properties and crashes Studio Pro if given them (#50). Tests `sdk/domainmodel/connection_test.go`, `mdl/backend/modelsdk/association_connection_test.go`, `sdk/mpr/writer_domainmodel_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | +| `mxcli new` into a deep output directory dies with `System.IO.PathTooLongException` and leaves the directory holding ~259 files and no `.mpr` — output that looks like a project until you try to open it | MxToolset refuses any full destination path over **259 characters** (its own Windows-compatibility limit, not the filesystem's) and aborts extraction PART WAY THROUGH. mxcli pointed `mx create-project` straight at the user's output directory, so the abort happened there | `cmd/mxcli/newproject_paths.go` (new: `stagedProjectDirs`, `longestRelativePath`, `warnIfPathTooLongForStudioPro`, `moveProject`), wired into `cmd/mxcli/cmd_new.go` step 2 | **Stage the work somewhere safe and move it in, rather than validating a path you could just avoid** — creating in a short temp dir and renaming makes ANY depth work (POSIX allows 4096) instead of merely failing politely, and the destination is never partially populated because nothing is written there until creation succeeded. Check the relocation is safe first: `grep -rl ` returned **0 files**, so a fresh Mendix project embeds no absolute paths. **Bisect for the real threshold instead of trusting arithmetic** — 77 characters creates a project on 11.13.0, 78 fails leaving 259 files, which pins `len(dest) + 1 + longest ≤ 259` exactly. **Measure the template, don't hardcode it**: the longest relative path is 181 on 11.13.0 and 182 on 11.12.0, so walk the staged tree and use the real number, or the reported budget drifts a character per release. **Warn, don't refuse, when the finished path is over budget** — the project works on POSIX, so refusing would block a machine where it is fine; but Studio Pro on Windows would not open it, so silence would be worse. Cross-device staging needs an `os.CopyFS` fallback: `os.Rename` cannot cross filesystems. Tests `cmd/mxcli/newproject_paths_test.go`. upstream #825 | diff --git a/.claude/skills/verify-in-runtime.md b/.claude/skills/verify-in-runtime.md index f18c7ac71..f7d7807be 100644 --- a/.claude/skills/verify-in-runtime.md +++ b/.claude/skills/verify-in-runtime.md @@ -49,10 +49,18 @@ whatever happens to be cached. mxcli new PopupDemo --version 11.12.2 --output-dir /root/pd ``` -> **Trap: `PathTooLongException`.** Mendix's package extractor fails on deep paths. -> A scratchpad path like -> `/tmp/claude-...//scratchpad/proj` is already too long and -> `mx create-project` dies. Use a short root (`/root/pd`). The same applies to Go +> **Trap: `PathTooLongException`.** MxToolset refuses any full path over **259 +> characters** — its own Windows-compatibility limit, not the filesystem's — and +> aborts extraction part way through, leaving ~259 files and no `.mpr`. With the +> blank 11.13 template's longest relative path at 181 characters, the output +> directory gets **77**. A scratchpad path like +> `/tmp/claude-...//scratchpad/proj` blows that on its own. +> +> `mxcli new` handles this since #825 — it creates the project in a short staging +> directory and moves it into place, so any depth works and a failure never leaves +> partial output. It warns when the final path exceeds 259 that Studio Pro on +> Windows may not open the project. **Calling `mx create-project` directly still +> dies**, so keep using a short root (`/root/pd`) for that. The same applies to Go > tests: set `TMPDIR=/root/t` so `t.TempDir()` stays short, or the scaffolding fails > and the test **skips** — which is indistinguishable from passing. diff --git a/cmd/mxcli/cmd_new.go b/cmd/mxcli/cmd_new.go index f2a060e3d..5171aca5e 100644 --- a/cmd/mxcli/cmd_new.go +++ b/cmd/mxcli/cmd_new.go @@ -28,6 +28,12 @@ This command performs the following steps: 5. Runs one build so generated sources are settled (--skip-build to skip) 6. Links this mxcli into the project (or downloads a Linux build on macOS/Windows) +The project is created in a short temporary directory and moved into place, so +--output-dir may be as deep as your filesystem allows and a failure never leaves +partial output behind. Mendix tooling itself refuses paths over 259 characters, +so a project deeper than that is created but reported as one Studio Pro on +Windows may not open. + Examples: mxcli new MyApp mxcli new MyApp --version 11.8.0 @@ -87,22 +93,47 @@ Examples: os.Exit(1) } - // Step 2: Create project + // Step 2: Create project. + // + // Created in a SHORT staging directory and moved into place afterwards. + // MxToolset refuses to extract past a 259-character full path and aborts + // PART WAY THROUGH when it hits that, so pointing it straight at a deep + // output directory left the user with a few hundred orphaned files and no + // .mpr (issue #825). Staging removes the limit from the equation entirely — + // the destination may be as deep as the filesystem allows — and means the + // destination is only ever written to after creation has succeeded. fmt.Printf("\nStep 2/6: Creating Mendix project '%s'...\n", appName) if err := os.MkdirAll(absDir, 0755); err != nil { fmt.Fprintf(os.Stderr, "Error creating directory: %v\n", err) os.Exit(1) } + stageDir, cleanupStage, err := stagedProjectDirs() + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + defer cleanupStage() + mxCmd := exec.Command(mxPath, "create-project", "--app-name", appName) - mxCmd.Dir = absDir + mxCmd.Dir = stageDir mxCmd.Stdout = os.Stdout mxCmd.Stderr = os.Stderr docker.PrepareMxCommand(mxCmd) if err := mxCmd.Run(); err != nil { - fmt.Fprintf(os.Stderr, "Error creating project: %v\n", err) + fmt.Fprintf(os.Stderr, "Error creating project: %v\n", describeCreateProjectFailure(stageDir, err)) + os.Exit(1) + } + + // Measure the template's deepest path before moving, so the portability + // warning uses this version's real number rather than a constant that + // drifts (181 characters on 11.13.0, 182 on 11.12.0). + longest, longestPath := longestRelativePath(stageDir) + if err := moveProject(stageDir, absDir); err != nil { + fmt.Fprintf(os.Stderr, "Error moving project into place: %v\n", err) os.Exit(1) } + warnIfPathTooLongForStudioPro(os.Stdout, absDir, longest, longestPath) // Clean up duplicate locale files that mx create-project generates. // MxBuild's AtlasPlugin.LoadTranslations crashes with "An item with the same diff --git a/cmd/mxcli/newproject_paths.go b/cmd/mxcli/newproject_paths.go new file mode 100644 index 000000000..43ac77f49 --- /dev/null +++ b/cmd/mxcli/newproject_paths.go @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "fmt" + "io" + "os" + "path/filepath" +) + +// MxToolset refuses to extract a file whose full destination path exceeds this, +// regardless of what the filesystem allows — it is a Windows-compatibility limit +// carried by the tool, not by the OS. Exceeding it aborts extraction PART WAY +// THROUGH, so the output directory is left holding a few hundred files and no +// .mpr. +// +// Measured on mxbuild 11.13.0 by bisecting the output-directory length: a +// 77-character absolute directory creates the project; 78 fails with +// +// System.IO.PathTooLongException: The specified file name or path is too long +// +// leaving 259 files and no .mpr — matching what issue #825 reported from 11.12. +// The blank template's longest relative path was 181 characters there +// (…/RNCAsyncStorage.xcodeproj/xcshareddata/xcschemes/RNCAsyncStorage-macOS.xcscheme), +// and 77 + 1 + 181 = 259 exactly. +const mxToolsetMaxPath = 259 + +// stagedProjectDirs returns a short staging directory to create the project in, +// plus a cleanup func. +// +// mxcli creates the project in the staging directory and moves it to the +// destination afterwards, rather than pointing `mx create-project` at a deep path +// and hoping. Two things fall out of that: +// +// - A deep destination WORKS. The limit is MxToolset's own, and a created +// project embeds no absolute paths (verified: zero files in a blank 11.13 +// project mention their own directory), so relocating it is safe. +// - The destination is never partially populated. Nothing is written there +// until creation has already succeeded, so a failure leaves the user's +// directory exactly as it was instead of with 259 orphaned files. +// +// (issue #825) +func stagedProjectDirs() (stage string, cleanup func(), err error) { + stage, err = os.MkdirTemp("", "mxcli-new-") + if err != nil { + return "", func() {}, fmt.Errorf("create staging directory: %w", err) + } + return stage, func() { _ = os.RemoveAll(stage) }, nil +} + +// longestRelativePath returns the length of the longest path inside root, +// relative to root, and the path itself. +// +// Measured rather than hardcoded: the template changes between Mendix versions +// (181 characters on 11.13.0, 182 reported on 11.12.0), so a constant would drift +// and quietly mis-report the budget. +func longestRelativePath(root string) (int, string) { + longest, longestPath := 0, "" + _ = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() { + return nil //nolint:nilerr // a partial walk still yields a usable bound + } + rel, relErr := filepath.Rel(root, path) + if relErr != nil { + return nil + } + if len(rel) > longest { + longest, longestPath = len(rel), rel + } + return nil + }) + return longest, longestPath +} + +// warnIfPathTooLongForStudioPro reports when the finished project sits deep +// enough that MxToolset's own limit would be exceeded. +// +// A warning rather than a refusal: the project has already been created and works +// here — the limit is a Windows-compatibility one, and POSIX allows far longer +// paths — but Studio Pro on Windows would not open it, and a `mx` invocation +// against it can fail the same way mxcli's own creation would have. Saying so is +// more useful than either silently shipping an unportable project or refusing to +// create one that is fine on this machine. (issue #825) +func warnIfPathTooLongForStudioPro(w io.Writer, dest string, longest int, longestPath string) bool { + total := len(dest) + 1 + longest + if total <= mxToolsetMaxPath { + return false + } + budget := mxToolsetMaxPath - 1 - longest + fmt.Fprintf(w, "\nWarning: this project's path is %d characters longer than Mendix tooling allows.\n", total-mxToolsetMaxPath) + fmt.Fprintf(w, " Output directory: %s (%d characters)\n", dest, len(dest)) + fmt.Fprintf(w, " Longest file below: %s (%d)\n", longestPath, longest) + fmt.Fprintf(w, " Total %d exceeds the %d-character limit MxToolset enforces.\n", total, mxToolsetMaxPath) + fmt.Fprintf(w, " The project was created and works on this machine, but Studio Pro on Windows\n") + fmt.Fprintf(w, " may refuse to open it, and `mx` commands against it can fail with\n") + fmt.Fprintf(w, " PathTooLongException. Move it to a directory of at most %d characters.\n", budget) + return true +} + +// moveProject moves a created project from the staging directory into dest. +// Falls back to copy-then-remove when the two are on different filesystems, +// which os.Rename cannot cross. +func moveProject(stage, dest string) error { + if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil { + return fmt.Errorf("create parent directory: %w", err) + } + // Rename onto an existing empty directory fails, so clear the placeholder the + // caller may have created. Only ever an empty directory: `mxcli new` refuses a + // destination that already has content. + if entries, err := os.ReadDir(dest); err == nil && len(entries) == 0 { + _ = os.Remove(dest) + } + if err := os.Rename(stage, dest); err == nil { + return nil + } + // Cross-device: copy, then drop the staging copy. + if err := os.MkdirAll(dest, 0o755); err != nil { + return fmt.Errorf("create destination: %w", err) + } + if err := os.CopyFS(dest, os.DirFS(stage)); err != nil { + return fmt.Errorf("copy project into place: %w", err) + } + _ = os.RemoveAll(stage) + return nil +} + +// describeCreateProjectFailure adds the path-length explanation to a +// create-project failure when the staging path was the plausible cause. +// +// Staging makes this nearly unreachable — the staging root is short — but if +// someone's TMPDIR is itself deep, the same failure returns with a far more +// confusing message, so name the cause rather than surfacing a bare exit status. +func describeCreateProjectFailure(stage string, runErr error) error { + if len(stage) > mxToolsetMaxPath-1-200 { + return fmt.Errorf("%w\n The staging directory (%s, %d characters) may be too deep for Mendix tooling,\n"+ + " which refuses paths over %d characters. Set TMPDIR to a shorter directory and retry", + runErr, stage, len(stage), mxToolsetMaxPath) + } + return runErr +} diff --git a/cmd/mxcli/newproject_paths_test.go b/cmd/mxcli/newproject_paths_test.go new file mode 100644 index 000000000..5d7b9c611 --- /dev/null +++ b/cmd/mxcli/newproject_paths_test.go @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "os" + "path/filepath" + "strings" + "testing" +) + +// upstream #825: `mxcli new` pointed `mx create-project` straight at the output +// directory. MxToolset refuses to extract past a 259-character full path and +// aborts PART WAY THROUGH when it hits that, so a deep output directory left the +// user with a few hundred orphaned files and no .mpr. +// +// Measured on mxbuild 11.13.0 by bisecting the output-directory length: +// +// 77 characters → project created, .mpr present +// 78 characters → PathTooLongException, 259 files left behind, no .mpr +// +// The blank template's longest relative path is 181 characters there, and +// 77 + 1 + 181 = 259 exactly — so the arithmetic below is the tool's real rule, +// not a guess. +func TestPathBudgetArithmeticMatchesMeasuredThreshold(t *testing.T) { + const measuredLongest = 181 // blank 11.13.0 template + // The last output directory length that worked, and the first that did not. + for _, tc := range []struct { + destLen int + wantWarn bool + }{ + {77, false}, + {78, true}, + } { + dest := "/" + strings.Repeat("x", tc.destLen-1) + var buf bytes.Buffer + got := warnIfPathTooLongForStudioPro(&buf, dest, measuredLongest, "some/deep/file") + if got != tc.wantWarn { + t.Errorf("dest of %d characters: warned=%v, want %v (77 created a project on 11.13.0, 78 did not)", + tc.destLen, got, tc.wantWarn) + } + if got && !strings.Contains(buf.String(), "at most 77") { + t.Errorf("the warning must name the budget the user can actually hit, got:\n%s", buf.String()) + } + } +} + +// The warning has to carry enough to act on: both numbers that make up the +// total, and what to do. A bare "path too long" reproduces the original problem +// — the reporter had to work out the 259 limit and the 182-character template +// path themselves. +func TestPathWarningIsActionable(t *testing.T) { + var buf bytes.Buffer + if !warnIfPathTooLongForStudioPro(&buf, "/"+strings.Repeat("d", 99), 182, "deep/template/path.xcscheme") { + t.Fatal("expected a warning for a 100-character destination with a 182-character template path") + } + out := buf.String() + for _, want := range []string{"100 characters", "182", "259", "deep/template/path.xcscheme", "at most 76"} { + if !strings.Contains(out, want) { + t.Errorf("warning should mention %q, got:\n%s", want, out) + } + } +} + +// longestRelativePath is measured rather than hardcoded because the template +// changes between Mendix versions (181 on 11.13.0, 182 reported on 11.12.0) — a +// constant would drift and mis-report the budget. +func TestLongestRelativePath(t *testing.T) { + root := t.TempDir() + deep := filepath.Join(root, "a", "bb", "ccc", "dddd") + if err := os.MkdirAll(deep, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(deep, "leaf.txt"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "short.txt"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + n, p := longestRelativePath(root) + want := filepath.Join("a", "bb", "ccc", "dddd", "leaf.txt") + if p != want || n != len(want) { + t.Errorf("longestRelativePath = (%d, %q), want (%d, %q)", n, p, len(want), want) + } +} + +// The property that matters most: a FAILED creation must leave the user's +// directory exactly as it was. Before staging, the destination held 259 orphaned +// files and no .mpr — output that looks like a project until you try to open it. +func TestStagingLeavesDestinationUntouchedOnFailure(t *testing.T) { + stage, cleanup, err := stagedProjectDirs() + if err != nil { + t.Fatalf("stagedProjectDirs: %v", err) + } + // Simulate a partial extraction: files land in the staging directory. + if err := os.WriteFile(filepath.Join(stage, "partial.txt"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + + dest := filepath.Join(t.TempDir(), "project") + if err := os.MkdirAll(dest, 0o755); err != nil { + t.Fatal(err) + } + + cleanup() // what the deferred cleanup does when create-project fails + + if _, err := os.Stat(stage); !os.IsNotExist(err) { + t.Errorf("staging directory survived cleanup: %v", err) + } + entries, err := os.ReadDir(dest) + if err != nil { + t.Fatalf("read destination: %v", err) + } + if len(entries) != 0 { + t.Errorf("destination holds %d entries after a failed creation, want 0 — "+ + "the whole point of staging is that a failure leaves nothing behind", len(entries)) + } +} + +// A successful creation moves the tree into place, including through the +// copy fallback that a cross-filesystem staging directory forces. +func TestMoveProject(t *testing.T) { + stage, cleanup, err := stagedProjectDirs() + if err != nil { + t.Fatalf("stagedProjectDirs: %v", err) + } + defer cleanup() + + if err := os.MkdirAll(filepath.Join(stage, "mprcontents", "aa"), 0o755); err != nil { + t.Fatal(err) + } + for _, f := range []string{"App.mpr", filepath.Join("mprcontents", "aa", "unit.mxunit")} { + if err := os.WriteFile(filepath.Join(stage, f), []byte("content"), 0o644); err != nil { + t.Fatal(err) + } + } + + // The caller creates the destination before staging runs, so moveProject has + // to cope with an existing empty directory — os.Rename onto one fails. + dest := filepath.Join(t.TempDir(), "project") + if err := os.MkdirAll(dest, 0o755); err != nil { + t.Fatal(err) + } + + if err := moveProject(stage, dest); err != nil { + t.Fatalf("moveProject: %v", err) + } + for _, f := range []string{"App.mpr", filepath.Join("mprcontents", "aa", "unit.mxunit")} { + if _, err := os.Stat(filepath.Join(dest, f)); err != nil { + t.Errorf("%s did not arrive at the destination: %v", f, err) + } + } +} From 5cea8cad5362ba10ebdae5aab0b600ece5e09526 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 19:38:18 +0000 Subject: [PATCH 2/7] ci(docs): only publish Pages where a Pages site exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every push to main on this fork left a red X on "Deploy Documentation": Failed to create deployment (status: 404) ... Ensure GitHub Pages has been enabled: https://github.com/ako/mxcli/settings/pages A fork inherits the workflow but not the upstream Pages configuration, so actions/deploy-pages 404s on something unrelated to the change being pushed. The API confirms both halves for this repo: fork=true and has_pages=false. It has failed on all three runs since the fork was active, while Build/Test/Lint is 23/23 green — a permanently red workflow is how a real failure gets missed. The upload and deploy steps are now gated on the repository not being a fork, with an opt-out for a fork that does want to publish its own copy (enable Pages, then set the repository variable DEPLOY_DOCS=true). The book is still BUILT everywhere — on forks and on pull requests, as before — so a docs change is still proven to compile wherever it is pushed. Only publishing is skipped, and a skipped job does not fail the run. Nothing changes upstream: mendixlabs/mxcli is not a fork, so the condition is true there exactly as it is today. This is worth sending upstream on its own merits, since any fork of the repo currently hits the same 404. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .github/workflows/docs.yml | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 7d18c8a48..03f275b5d 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -21,6 +21,23 @@ concurrency: group: "pages" cancel-in-progress: false +# Publishing is gated on the repository actually having a Pages site. +# +# A fork inherits this workflow but not the upstream Pages configuration, so +# actions/deploy-pages fails there with a 404 that has nothing to do with the +# change being pushed: +# +# Failed to create deployment (status: 404) ... Ensure GitHub Pages has been +# enabled: https://github.com//mxcli/settings/pages +# +# Every push to a fork's main therefore left a permanent red X, which is how a +# real failure gets missed. The book is still BUILT on forks (and on pull +# requests) — only the upload and deploy are skipped, so a docs change is still +# proven to compile wherever it is pushed. +# +# A fork that wants to publish its own copy: enable Pages (Settings → Pages → +# Source: GitHub Actions), then set the repository variable DEPLOY_DOCS=true +# (Settings → Secrets and variables → Actions → Variables). jobs: build: runs-on: ubuntu-latest @@ -37,13 +54,19 @@ jobs: run: mdbook build docs-site - name: Upload artifact - if: github.ref == 'refs/heads/main' && github.event_name == 'push' + if: >- + github.ref == 'refs/heads/main' && github.event_name == 'push' + && (github.event.repository.fork != true || vars.DEPLOY_DOCS == 'true') uses: actions/upload-pages-artifact@v5 with: path: docs-site/book deploy: - if: github.ref == 'refs/heads/main' && github.event_name == 'push' + # Same condition as the upload above: with no artifact there is nothing to + # deploy, and deploy-pages fails rather than no-ops when Pages is absent. + if: >- + github.ref == 'refs/heads/main' && github.event_name == 'push' + && (github.event.repository.fork != true || vars.DEPLOY_DOCS == 'true') needs: build runs-on: ubuntu-latest environment: From f38f02b295a934a329abfb0650d8ced74b719cf1 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 20:18:06 +0000 Subject: [PATCH 3/7] fix(catalog): index page references from widget actions, and stop `show callers` hiding them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `show callers` and `show references` missed anything reached only from a page action button, reporting "(no callers found)" — a false negative that reads as "safe to delete". Two independent defects behind one symptom, and fixing either alone closes half the issue. 1. The page reference was never recorded. scanWidgetOwnRefs collects a widget's Entity, Microflow and Nanoflow out of raw BSON but not `Form`, which is where a PAGE reference lives — "Form" being Mendix's original word for a page, the same rename behind ShowFormAction/CloseFormAction. So widgets_data had no page column and the refs projection had no page row. `create object … then open page` is one action carrying TWO references, and only the entity survived. 2. The microflow reference was recorded and then hidden by the query. execShowCallers filtered RefKind = 'call', the kind a microflow CALL ACTIVITY produces; a button's microflow is 'action', and that row was sitting in the refs table all along. widgets_data gains a PageRef column, the projection a PAGE/show_page row, and `show callers` an explicit set of invocation kinds — call, action, show_page, calculate, and the three navigation kinds. `show callers` stays narrower than `show references to`: datasource, parameter, return, retrieve, create, change, delete, associate, generalize and layout are uses of a TYPE or a LAYOUT, not invocations, and folding them in would make the two commands synonyms. Both the included and the excluded set are pinned by test. Verified on a project reproducing the reporter's two scenarios: before: (no callers found) for both the button-called microflow and the button-opened page after: the page is a caller of its button's microflow; the overview page is a caller of the page its "create object … then open page" button opens; transitive finds the overview at depth 2 The microflow→microflow control still resolves, and `show callers of ` is still empty while `show references to ` shows the datasource use. Fixes #773 --- .claude/skills/fix-issue.md | 1 + docs/01-project/MDL_QUICK_REFERENCE.md | 4 +- mdl/catalog/builder_pages.go | 28 ++++++--- mdl/catalog/builder_pages_test.go | 76 ++++++++++++++++++++++--- mdl/catalog/builder_references.go | 4 ++ mdl/catalog/tables.go | 5 ++ mdl/executor/cmd_search.go | 52 ++++++++++++++++- mdl/executor/cmd_search_callers_test.go | 71 +++++++++++++++++++++++ 8 files changed, 221 insertions(+), 20 deletions(-) create mode 100644 mdl/executor/cmd_search_callers_test.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 26a1f45b3..0af259e2a 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -473,3 +473,4 @@ extracting `OffsetExpression`/`LimitExpression`. | An association's line anchors — where the connector attaches to the entity boxes in the domain model editor — are absent from `DESCRIBE ASSOCIATION`, and manual adjustments made in Studio Pro do not survive an mxcli round trip | `DomainModels$Association.ParentConnection`/`ChildConnection` (the string `"x;y"`) were **hardcoded** to `"0;50"`/`"100;50"` in BOTH writers and never read by either parser. Because every association write rebuilds the whole element, this was not an omission but active destruction: a documentation-only `alter association … set comment` reset them | `sdk/domainmodel/connection.go` (new: `ParseConnectionPoint`/`FormatConnectionPoint`, `Default*Connection`), `sdk/domainmodel/domainmodel.go` (fields → `*model.Point`), `sdk/mpr/parser_domainmodel.go` + `sdk/mpr/writer_domainmodel.go`, `mdl/backend/modelsdk/domainmodel.go` + `domainmodel_write.go`, `mdl/executor/cmd_associations.go` (`describeConnectionPoints`) | **A feature request that says "X is not exposed" may be hiding "X is destroyed"** — check the write path before scoping the read path. The A/B that settled it: a blank 11.13 app's own `Administration.AccountPasswordData_Account` stores `0;54/100;54`, so a Studio-Pro-authored association is a free fixture for "did mxcli overwrite this?" — no Studio Pro needed. **Learn the value's constraints from the LOADER, not from the shape**: hand-patch and run `mx check` — `"0.5;50"` dies with `StorageLoadException` (integers required) while `"0;500"` and `"-20;50"` load with 0 errors (no range check), so out-of-range values must round-trip untouched. **A zero value is not an absent value** — `{0,0}` is a real anchor (top-left), which forces the field to be a POINTER; a plain `model.Point` cannot distinguish "unset" from "top-left" and would silently rewrite it. **Fix both engines**: they share the semantic model, and a fix in one is invisible to a user on the other. **Emit unauthorable data as a COMMENT** — DESCRIBE output must stay re-executable, and inventing syntax (`@anchor(parent: bottom-left, …)`) would bake in a vocabulary the storage does not have: the pair is CONTINUOUS, not 8 named anchors (observed x values 0 9 11 17 18 47 49 50 65 77 78 84 87 100). **The marketplace is the sample** when you need to know what Studio Pro actually writes: `mxcli marketplace download ` gives real Mendix-authored models, and a module .mpk holds either a raw BSON `project.mpr` or an MPR v1 SQLite one — 88 coordinate pairs from three modules turned "looks like percentages" into a measurement (all 0..100; 85 of 88 pin one coordinate to exactly 0 or 100). **Rule a unit out from the model, not the values**: pixels is impossible because `DomainModels$EntityImpl` stores only `Location` and NO size — the box is sized by the editor from the name and attribute list, so a pixel anchor would have nothing to measure against. Not applicable to `CrossAssociation`, which has no connection properties and crashes Studio Pro if given them (#50). Tests `sdk/domainmodel/connection_test.go`, `mdl/backend/modelsdk/association_connection_test.go`, `sdk/mpr/writer_domainmodel_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | `mxcli new` into a deep output directory dies with `System.IO.PathTooLongException` and leaves the directory holding ~259 files and no `.mpr` — output that looks like a project until you try to open it | MxToolset refuses any full destination path over **259 characters** (its own Windows-compatibility limit, not the filesystem's) and aborts extraction PART WAY THROUGH. mxcli pointed `mx create-project` straight at the user's output directory, so the abort happened there | `cmd/mxcli/newproject_paths.go` (new: `stagedProjectDirs`, `longestRelativePath`, `warnIfPathTooLongForStudioPro`, `moveProject`), wired into `cmd/mxcli/cmd_new.go` step 2 | **Stage the work somewhere safe and move it in, rather than validating a path you could just avoid** — creating in a short temp dir and renaming makes ANY depth work (POSIX allows 4096) instead of merely failing politely, and the destination is never partially populated because nothing is written there until creation succeeded. Check the relocation is safe first: `grep -rl ` returned **0 files**, so a fresh Mendix project embeds no absolute paths. **Bisect for the real threshold instead of trusting arithmetic** — 77 characters creates a project on 11.13.0, 78 fails leaving 259 files, which pins `len(dest) + 1 + longest ≤ 259` exactly. **Measure the template, don't hardcode it**: the longest relative path is 181 on 11.13.0 and 182 on 11.12.0, so walk the staged tree and use the real number, or the reported budget drifts a character per release. **Warn, don't refuse, when the finished path is over budget** — the project works on POSIX, so refusing would block a machine where it is fine; but Studio Pro on Windows would not open it, so silence would be worse. Cross-device staging needs an `os.CopyFS` fallback: `os.Rename` cannot cross filesystems. Tests `cmd/mxcli/newproject_paths_test.go`. upstream #825 | +| `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index d4cb8d818..45da7ed6f 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1165,7 +1165,7 @@ CLI subcommand: `mxcli sql --driver postgres --dsn '...' "select 1"` (see `mxcli | Refresh with refs | `refresh catalog full;` | Include cross-references and source | | Show catalog tables | `show catalog tables;` | List available queryable tables | | Query catalog | `select ... from CATALOG. [where ...];` | SQL against project metadata | -| Show callers | `show callers of Module.Name;` | What calls this element | +| Show callers | `show callers of Module.Name;` | What INVOKES this element: microflow call activities, page action buttons and other widget actions, calculated attributes, and navigation entries. A page counts as a caller of the microflow its button runs, and of the page that button opens | | Show callees | `show callees of Module.Name;` | What this element calls | | Show references | `show references of Module.Name;` | All references to/from | | Show impact | `show impact of Module.Name;` | Impact analysis | @@ -1174,6 +1174,8 @@ CLI subcommand: `mxcli sql --driver postgres --dsn '...' "select 1"` (see `mxcli Cross-reference commands require `refresh catalog full` to populate reference data. +`show callers` covers invocation only. A document that merely *uses a type* — an entity as a page datasource, a microflow parameter, an entity's generalization — is not a caller of it; `show references to` lists those. + ## Connection & Session | Statement | Syntax | Notes | diff --git a/mdl/catalog/builder_pages.go b/mdl/catalog/builder_pages.go index 7cc9d8758..99f8f78a9 100644 --- a/mdl/catalog/builder_pages.go +++ b/mdl/catalog/builder_pages.go @@ -34,9 +34,9 @@ func (b *Builder) buildPages() error { if b.fullMode { widgetStmt, err = b.tx.Prepare(` INSERT INTO widgets_data (Id, Name, WidgetType, ContainerId, ContainerQualifiedName, ContainerType, - ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, Description, + ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, PageRef, Description, ProjectId, SnapshotId) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `) if err != nil { return err @@ -114,6 +114,7 @@ func (b *Builder) buildPages() error { w.AttributeRef, w.MicroflowRef, w.NanoflowRef, + w.PageRef, "", projectID, snapshotID, ); err != nil { @@ -156,9 +157,9 @@ func (b *Builder) buildSnippets() error { if b.fullMode { widgetStmt, err = b.tx.Prepare(` INSERT INTO widgets_data (Id, Name, WidgetType, ContainerId, ContainerQualifiedName, ContainerType, - ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, Description, + ModuleName, Folder, EntityRef, AttributeRef, MicroflowRef, NanoflowRef, PageRef, Description, ProjectId, SnapshotId) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) `) if err != nil { return err @@ -205,7 +206,7 @@ func (b *Builder) buildSnippets() error { w.ID, w.Name, w.WidgetType, string(sn.ID), qualifiedName, "SNIPPET", moduleName, folder, - w.EntityRef, w.AttributeRef, w.MicroflowRef, w.NanoflowRef, "", + w.EntityRef, w.AttributeRef, w.MicroflowRef, w.NanoflowRef, w.PageRef, "", projectID, snapshotID, ); err != nil { return fmt.Errorf("insert widget %s for snippet %s: %w", w.Name, qualifiedName, err) @@ -292,6 +293,7 @@ type rawWidgetInfo struct { AttributeRef string MicroflowRef string // action/datasource microflow (Forms$MicroflowSettings.Microflow, …) NanoflowRef string // action/datasource nanoflow + PageRef string // action page (Forms$PageSettings.Form) — see scanWidgetOwnRefs } // widgetChildKeys are the keys under which a widget nests *other* widgets. The @@ -306,8 +308,8 @@ var widgetChildKeys = map[string]bool{ // Returns the lexicographically smallest match of each kind so the result is // deterministic regardless of BSON map iteration order; page-level dedup in the // refs projection makes the per-widget choice immaterial to the final graph. -func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow string) { - var ents, mfs, nfs []string +func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow, page string) { + var ents, mfs, nfs, forms []string var walk func(v any) walk = func(v any) { switch x := v.(type) { @@ -321,6 +323,14 @@ func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow string) { if s, ok := x["Nanoflow"].(string); ok && s != "" { nfs = append(nfs, s) } + // A page reference is stored under "Form" — Mendix's original word for + // a page, the same rename behind ShowFormAction/CloseFormAction. Not + // collecting it is why an action button that opens a page created no + // reference, so the page reported "(no callers found)" however many + // buttons pointed at it (issue #773). + if s, ok := x["Form"].(string); ok && s != "" { + forms = append(forms, s) + } for k, val := range x { if widgetChildKeys[k] { continue @@ -346,7 +356,7 @@ func scanWidgetOwnRefs(w map[string]any) (entity, microflow, nanoflow string) { } return m } - return min(ents), min(mfs), min(nfs) + return min(ents), min(mfs), min(nfs), min(forms) } // extractLayoutRef extracts the layout reference from raw page BSON. Regular @@ -461,7 +471,7 @@ func extractWidgetsRecursive(w map[string]any) []rawWidgetInfo { // Extract datasource entity + action microflow/nanoflow references from this // widget's own content (not its child widgets). - widget.EntityRef, widget.MicroflowRef, widget.NanoflowRef = scanWidgetOwnRefs(w) + widget.EntityRef, widget.MicroflowRef, widget.NanoflowRef, widget.PageRef = scanWidgetOwnRefs(w) // Index user-authored containers, but skip the synthetic // "conditionalVisibilityWidget*" wrapper that mxcli / Studio Pro insert as a diff --git a/mdl/catalog/builder_pages_test.go b/mdl/catalog/builder_pages_test.go index 838c5063d..b565d2fbc 100644 --- a/mdl/catalog/builder_pages_test.go +++ b/mdl/catalog/builder_pages_test.go @@ -24,7 +24,7 @@ func TestScanWidgetOwnRefs(t *testing.T) { }, }, } - ent, mf, nf := scanWidgetOwnRefs(dataView) + ent, mf, nf, _ := scanWidgetOwnRefs(dataView) if ent != "Sales.Order" { t.Errorf("entity = %q, want Sales.Order", ent) } @@ -41,7 +41,7 @@ func TestScanWidgetOwnRefs(t *testing.T) { "Action": map[string]any{"Settings": map[string]any{"Microflow": "M.DoThing"}}, "Extra": map[string]any{"Nanoflow": "M.DoNano"}, } - if _, mf, nf := scanWidgetOwnRefs(button); mf != "M.DoThing" || nf != "M.DoNano" { + if _, mf, nf, _ := scanWidgetOwnRefs(button); mf != "M.DoThing" || nf != "M.DoNano" { t.Errorf("button refs = (%q,%q), want (M.DoThing, M.DoNano)", mf, nf) } @@ -52,13 +52,13 @@ func TestScanWidgetOwnRefs(t *testing.T) { "B": map[string]any{"Microflow": "M.Alpha"}, } for range 5 { - if _, mf, _ := scanWidgetOwnRefs(multi); mf != "M.Alpha" { + if _, mf, _, _ := scanWidgetOwnRefs(multi); mf != "M.Alpha" { t.Fatalf("non-deterministic microflow pick: got %q, want M.Alpha", mf) } } // No refs. - if e, m, n := scanWidgetOwnRefs(map[string]any{"$Type": "Forms$Label"}); e != "" || m != "" || n != "" { + if e, m, n, p := scanWidgetOwnRefs(map[string]any{"$Type": "Forms$Label"}); e != "" || m != "" || n != "" || p != "" { t.Errorf("expected no refs, got (%q,%q,%q)", e, m, n) } } @@ -245,9 +245,9 @@ func TestExtractWidgetsRecursive(t *testing.T) { t.Run("indexes a styled conditionalVisibilityWidget (not transparent)", func(t *testing.T) { // If a wrapper-named container carries styling it is a real user widget. w := map[string]any{ - "$ID": "div3", - "Name": "conditionalVisibilityWidget9", - "$Type": "Forms$DivContainer", + "$ID": "div3", + "Name": "conditionalVisibilityWidget9", + "$Type": "Forms$DivContainer", "Appearance": map[string]any{"Class": "card"}, } got := extractWidgetsRecursive(w) @@ -557,3 +557,65 @@ func TestBytesToHex(t *testing.T) { }) } } + +// upstream #773: a widget action that OPENS a page created no reference at all, +// so a page reachable only from a button reported "(no callers found)" — a false +// negative that reads as "safe to delete". +// +// The cause was one missing key. scanWidgetOwnRefs collected Entity, Microflow +// and Nanoflow but not `Form`, which is where a page reference lives — "Form" +// being Mendix's original word for a page, the same rename behind +// ShowFormAction/CloseFormAction. +func TestScanWidgetOwnRefs_PageReference(t *testing.T) { + // An action button that opens a page: `show_page Module.Target`. + showPage := map[string]any{ + "$Type": "Forms$ActionButton", + "Action": map[string]any{ + "$Type": "Forms$ShowPageClientAction", + "PageSettings": map[string]any{"$Type": "Forms$PageSettings", "Form": "Sales.OrderDetail"}, + }, + } + if _, _, _, page := scanWidgetOwnRefs(showPage); page != "Sales.OrderDetail" { + t.Errorf("page = %q, want Sales.OrderDetail", page) + } + + // The compound the issue reported — "create object … then open page" — is ONE + // action carrying BOTH an entity and a page. Collecting only the entity (which + // is what happened) still leaves the page with no inbound reference. + createThenShow := map[string]any{ + "$Type": "Forms$ActionButton", + "Action": map[string]any{ + "$Type": "Forms$CreateObjectClientAction", + "Entity": "Sales.Order", + "PageSettings": map[string]any{"$Type": "Forms$PageSettings", "Form": "Sales.OrderDetail"}, + }, + } + ent, _, _, page := scanWidgetOwnRefs(createThenShow) + if ent != "Sales.Order" { + t.Errorf("entity = %q, want Sales.Order", ent) + } + if page != "Sales.OrderDetail" { + t.Errorf("page = %q, want Sales.OrderDetail — the page half of "+ + "'create object … then open page' is the reference the issue reported missing", page) + } + + // A child widget's page must not be attributed to its parent, same as the + // existing microflow rule. + container := map[string]any{ + "$Type": "Forms$DivContainer", + "Widgets": []any{ + map[string]any{ + "$Type": "Forms$ActionButton", + "Action": map[string]any{"PageSettings": map[string]any{"Form": "Sales.ChildPage"}}, + }, + }, + } + if _, _, _, page := scanWidgetOwnRefs(container); page != "" { + t.Errorf("page = %q, want empty — a child button's page must not leak to the container", page) + } + + // A widget with no action references no page. + if _, _, _, page := scanWidgetOwnRefs(map[string]any{"$Type": "Forms$Label"}); page != "" { + t.Errorf("page = %q, want empty", page) + } +} diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index 26519d0be..b85da8997 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -366,6 +366,10 @@ func (b *Builder) buildReferences() error { {"EntityRef", "ENTITY", RefKindDatasource}, {"MicroflowRef", "MICROFLOW", RefKindAction}, {"NanoflowRef", "NANOFLOW", RefKindAction}, + // A widget action that opens a page. Without this row, a page reachable + // only from a button had no inbound reference and `show callers` / + // `show references` reported it as unused (issue #773). + {"PageRef", "PAGE", RefKindShowPage}, } for _, p := range widgetProjections { res, perr := b.tx.Exec( diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index b39426c6b..583cc993d 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -416,6 +416,11 @@ func (c *Catalog) createTables() error { AttributeRef TEXT, MicroflowRef TEXT, NanoflowRef TEXT, + -- The page a widget's action opens (show_page, and the page half of + -- "create object … then open page"). Without it a page reachable only + -- from a button had no inbound reference at all and read as dead code + -- (issue #773). + PageRef TEXT, Description TEXT, ProjectId TEXT, SnapshotId TEXT diff --git a/mdl/executor/cmd_search.go b/mdl/executor/cmd_search.go index 497ec70c6..689528ecd 100644 --- a/mdl/executor/cmd_search.go +++ b/mdl/executor/cmd_search.go @@ -11,6 +11,52 @@ import ( ) // execShowCallers handles SHOW CALLERS OF Module.Microflow [TRANSITIVE]. +// callerRefKinds are the reference kinds that mean "this thing invokes the +// target". `show callers` filtered on 'call' alone, which is the kind a MICROFLOW +// activity produces — so a microflow called from a page action button (kind +// 'action') reported "(no callers found)" even though the reference was sitting +// in the refs table, and a page opened by a button or a menu was equally +// invisible (issue #773). +// +// A false negative here reads as "nothing uses this", which is the answer +// somebody acts on before deleting a document — so the set errs toward +// including a kind rather than omitting it. +// +// Deliberately excluded: 'datasource', 'parameter', 'return', 'retrieve', +// 'create', 'change', 'delete', 'associate', 'generalize' and 'layout'. Those +// are uses of a TYPE or a LAYOUT, not invocations, and folding them in would +// make `show callers of ` a synonym for `show references to`. +var callerRefKinds = []string{ + RefKindCallerCall, // microflow/nanoflow call activity + RefKindCallerAction, // widget action: button, on-change, on-click + RefKindCallerShowPage, // microflow show-page activity, or a widget action opening a page + RefKindCallerCalculate, // calculated attribute + RefKindCallerHomePage, // navigation + RefKindCallerLoginPage, + RefKindCallerMenuItem, +} + +// Kind literals, kept next to the set that uses them so the SQL below cannot +// drift from mdl/catalog's constants unnoticed. +const ( + RefKindCallerCall = "call" + RefKindCallerAction = "action" + RefKindCallerShowPage = "show_page" + RefKindCallerCalculate = "calculate" + RefKindCallerHomePage = "home_page" + RefKindCallerLoginPage = "login_page" + RefKindCallerMenuItem = "menu_item" +) + +// callerRefKindsSQL renders callerRefKinds as a SQL IN list. +func callerRefKindsSQL() string { + quoted := make([]string, len(callerRefKinds)) + for i, k := range callerRefKinds { + quoted[i] = "'" + k + "'" + } + return "(" + strings.Join(quoted, ", ") + ")" +} + func execShowCallers(ctx *ExecContext, s *ast.ShowStmt) error { if s.Name == nil { return mdlerrors.NewValidation("target name required for show callers") @@ -36,12 +82,12 @@ func execShowCallers(ctx *ExecContext, s *ast.ShowStmt) error { with RECURSIVE callers_cte as ( select SourceName as Caller, 1 as Depth from refs - where TargetName = ? and RefKind = 'call' + where TargetName = ? and RefKind in ` + callerRefKindsSQL() + ` union all select r.SourceName, c.Depth + 1 from refs r join callers_cte c on r.TargetName = c.Caller - where r.RefKind = 'call' and c.Depth < 10 + where r.RefKind in ` + callerRefKindsSQL() + ` and c.Depth < 10 ) select distinct Caller, min(Depth) as Depth from callers_cte @@ -53,7 +99,7 @@ func execShowCallers(ctx *ExecContext, s *ast.ShowStmt) error { query = ` select distinct SourceName as Caller, 1 as Depth from refs - where TargetName = ? and RefKind = 'call' + where TargetName = ? and RefKind in ` + callerRefKindsSQL() + ` ORDER by Caller ` } diff --git a/mdl/executor/cmd_search_callers_test.go b/mdl/executor/cmd_search_callers_test.go new file mode 100644 index 000000000..a9998eb77 --- /dev/null +++ b/mdl/executor/cmd_search_callers_test.go @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// upstream #773, second half. `show callers` filtered on RefKind = 'call' — the +// kind a MICROFLOW call activity produces. A microflow invoked from a page action +// button is recorded as 'action', and that row was already sitting in the refs +// table: the reference existed, the query hid it. A page opened by a button or a +// navigation menu was invisible for the same reason. +// +// A false negative here reads as "nothing uses this", which is the answer +// somebody acts on before deleting a document — so the set errs toward including +// a kind rather than omitting it. +func TestCallerRefKinds(t *testing.T) { + in := map[string]bool{} + for _, k := range callerRefKinds { + in[k] = true + } + + // Every way one document can INVOKE another. + for _, k := range []string{ + "call", // microflow/nanoflow call activity — the only one that used to count + "action", // widget action: the reported button case + "show_page", // a microflow, or a button, opening a page + "calculate", // calculated attribute running a microflow + "home_page", // navigation + "login_page", // + "menu_item", // + } { + if !in[k] { + t.Errorf("%q means one document invokes another and must count as a caller — "+ + "omitting it reports the target as unused, which is what #773 was", k) + } + } + + // Uses of a TYPE or a LAYOUT are NOT invocations. Folding these in would make + // `show callers of ` a synonym for `show references to` and destroy + // the distinction between the two commands. + for _, k := range []string{ + "datasource", "parameter", "return", "retrieve", + "create", "change", "delete", "associate", "generalize", "layout", + } { + if in[k] { + t.Errorf("%q is a use of a type or layout, not an invocation — including it "+ + "collapses `show callers` into `show references to`", k) + } + } +} + +// The kind list reaches SQLite as an IN list, so a malformed render silently +// matches nothing — the same "(no callers found)" the issue reported, from a +// different cause. +func TestCallerRefKindsSQL(t *testing.T) { + got := callerRefKindsSQL() + if !strings.HasPrefix(got, "(") || !strings.HasSuffix(got, ")") { + t.Fatalf("not a SQL IN list: %q", got) + } + for _, k := range callerRefKinds { + if !strings.Contains(got, "'"+k+"'") { + t.Errorf("%q is not quoted in the IN list %q", k, got) + } + } + if n := strings.Count(got, ","); n != len(callerRefKinds)-1 { + t.Errorf("%d separators for %d kinds: %q", n, len(callerRefKinds), got) + } +} From b1a138c117aaa90cb43f15c9788fc3d079db3c4a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 22:34:07 +0000 Subject: [PATCH 4/7] feat(queues): author task queues in MDL, and refuse the rewrite that dropped them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A microflow call bound to a task queue in Studio Pro lost that binding on `create or replace microflow` — silently. Worse, `mx check` went from [CE1613] "The selected task queue 'X' no longer exists." to 0 errors, because mxcli had deleted the configuration the error was about. `describe microflow` never showed the binding either, so the loss was invisible from every angle: the write reported success, the describe looked complete, and the build looked healthier than before. Two defects behind one report, fixed here: 1. MDL had no queue surface at all, so a script could not restate a binding even in principle. CREATE [OR MODIFY] / DROP / SHOW / DESCRIBE QUEUE now exist, wired through grammar → AST → visitor → executor → backend on both engines. The BSON is pinned against the four Studio Pro-authored queues in Mendix Business Events 3.12.1, which agree exactly: Config is a Queues$BasicQueueConfig whose ParallelismExpression is a STRING, with the sibling int32 `Parallelism` absent in all four. Parallelism is therefore an expression everywhere in MDL, not a number. 2. Rewriting a microflow whose stored calls are queued is now refused, naming the queues that would be lost. Binding a call to a queue is deliberately still unimplemented — the Retry shape has no Studio Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. Refusing is the only option that does not lose data (ADR-0005). QUEUE/QUEUES are added to the `keyword` rule, so `queue` remains usable as an ordinary attribute name. Verified on Mendix 11.13: the example script leaves a clean project at 0 errors under both engines, and describe → exec round-trips. The guard's test fails (the rewrite runs, the binding goes) when its call site is disabled. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/lsp_completions_gen.go | 2 + cmd/mxcli/syntax/features_misc.go | 46 ++++ docs/01-project/MDL_QUICK_REFERENCE.md | 28 ++ .../queue-authoring-and-rewrite-guard.mdl | 61 +++++ mdl/ast/ast_queue.go | 42 +++ mdl/backend/backend.go | 1 + mdl/backend/infrastructure.go | 8 + mdl/backend/mcp/unsupported_gen.go | 20 ++ mdl/backend/mock/backend.go | 6 + mdl/backend/mock/mock_queue.go | 43 +++ mdl/backend/modelsdk/queue_write.go | 123 +++++++++ mdl/backend/modelsdk/queue_write_test.go | 129 +++++++++ mdl/backend/modelsdk/unimplemented_gen.go | 17 ++ mdl/backend/mpr/backend.go | 17 ++ mdl/executor/cmd_microflows_create.go | 8 + mdl/executor/cmd_queues.go | 183 +++++++++++++ mdl/executor/cmd_queues_mock_test.go | 247 ++++++++++++++++++ mdl/executor/register_stubs.go | 15 ++ mdl/executor/registry.go | 1 + mdl/executor/registry_test.go | 4 + mdl/executor/validate_queued_calls.go | 103 ++++++++ mdl/executor/validate_queued_calls_test.go | 160 ++++++++++++ mdl/grammar/MDLLexer.g4 | 2 + mdl/grammar/MDLParser.g4 | 2 + mdl/grammar/domains/MDLCatalog.g4 | 2 + mdl/grammar/domains/MDLDomainModel.g4 | 23 ++ mdl/grammar/domains/MDLSettings.g4 | 2 +- mdl/types/queue.go | 49 ++++ mdl/visitor/visitor_entity.go | 4 + mdl/visitor/visitor_query.go | 18 ++ mdl/visitor/visitor_queue.go | 72 +++++ mdl/visitor/visitor_queue_test.go | 127 +++++++++ sdk/mpr/queues.go | 109 ++++++++ 34 files changed, 1674 insertions(+), 1 deletion(-) create mode 100644 mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl create mode 100644 mdl/ast/ast_queue.go create mode 100644 mdl/backend/mock/mock_queue.go create mode 100644 mdl/backend/modelsdk/queue_write.go create mode 100644 mdl/backend/modelsdk/queue_write_test.go create mode 100644 mdl/executor/cmd_queues.go create mode 100644 mdl/executor/cmd_queues_mock_test.go create mode 100644 mdl/executor/validate_queued_calls.go create mode 100644 mdl/executor/validate_queued_calls_test.go create mode 100644 mdl/types/queue.go create mode 100644 mdl/visitor/visitor_queue.go create mode 100644 mdl/visitor/visitor_queue_test.go create mode 100644 sdk/mpr/queues.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index f75d4e0bc..dd3c1dd49 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -472,3 +472,4 @@ extracting `OffsetExpression`/`LimitExpression`. | A datagrid column bound to an association — `column c (attribute: Order_Customer)` — reports success from `mxcli exec` and then fails the build with `[error] [CE1613] "The selected attribute 'Mod.Order.Order_Customer' no longer exists." at Columns (1/1) of data grid 2`. Separately, there is no MDL spelling for the drop-down filter's association mode: `mxcli check` says `[MDL-WIDGET01] has no property \`refEntity\`` | Two unrelated defects behind one report. (1) The reference is **not representable**: `CustomWidgets$WidgetValue.AttributeRef` is typed `AttributeRef`, not the polymorphic `MemberRef`, so the association was qualified like an attribute and written as a dangling `AttributeRef`. (2) `dropdownfilter.def.json` mapped only `attrChoice`/`attributes`/`defaultFilter`, so every `baseType: 'ref'` property was unmapped and dropped | `mdl/executor/cmd_pages_builder_input.go` (`rejectAssociationAsAttribute`, `entityInChain`) wired into `mdl/executor/widget_engine.go` (the objectlist `attribute` case and the `Attribute` source); `sdk/widgets/definitions/dropdownfilter.def.json` (association mode); `mdl/executor/cmd_pages_describe_parse.go` + `_pluggable.go` + `_output.go` (round-trip) | **Establish that a shape is unrepresentable before designing a fix for it** — hand-patch the BSON and run `mx check`. A `DomainModels$AssociationRef` in that slot makes the project **UNLOADABLE** (`ArgumentException: Object of type 'AssociationRef' cannot be converted to type 'AttributeRef'`), and the assembly defining the type (`Mendix.Modeler.WebUI.dll`) has no `AssociationRef` member at all — so the only correct outcome is a refusal carrying both working forms. **`` on an ATTRIBUTE-typed widget property is permission to TRAVERSE a reference, not to bind one** — `attribute: Assoc/Attr` already worked and is what the XML is advertising; the DataGrid column is the only shipped widget where the two are easy to confuse. **A def.json `mode` is the whole feature** for an unauthorable widget mode — the engine already had the `association` operation and the `hasDataSource` condition, so the second half was a data change plus its DESCRIBE reader (without which describe→edit→exec silently reverts the filter to attribute mode). **0 errors from `mx check` does not prove the properties landed** — an unmapped property is silently dropped and the build is just as green; read them back with `mx dump-mpr`. Tests `cmd_pages_builder_assoc_as_attribute_test.go`, `widget_dropdownfilter_assoc_test.go`, example `mdl-examples/bug-tests/830-datagrid-association-filter.mdl`. upstream #830 | | An association's line anchors — where the connector attaches to the entity boxes in the domain model editor — are absent from `DESCRIBE ASSOCIATION`, and manual adjustments made in Studio Pro do not survive an mxcli round trip | `DomainModels$Association.ParentConnection`/`ChildConnection` (the string `"x;y"`) were **hardcoded** to `"0;50"`/`"100;50"` in BOTH writers and never read by either parser. Because every association write rebuilds the whole element, this was not an omission but active destruction: a documentation-only `alter association … set comment` reset them | `sdk/domainmodel/connection.go` (new: `ParseConnectionPoint`/`FormatConnectionPoint`, `Default*Connection`), `sdk/domainmodel/domainmodel.go` (fields → `*model.Point`), `sdk/mpr/parser_domainmodel.go` + `sdk/mpr/writer_domainmodel.go`, `mdl/backend/modelsdk/domainmodel.go` + `domainmodel_write.go`, `mdl/executor/cmd_associations.go` (`describeConnectionPoints`) | **A feature request that says "X is not exposed" may be hiding "X is destroyed"** — check the write path before scoping the read path. The A/B that settled it: a blank 11.13 app's own `Administration.AccountPasswordData_Account` stores `0;54/100;54`, so a Studio-Pro-authored association is a free fixture for "did mxcli overwrite this?" — no Studio Pro needed. **Learn the value's constraints from the LOADER, not from the shape**: hand-patch and run `mx check` — `"0.5;50"` dies with `StorageLoadException` (integers required) while `"0;500"` and `"-20;50"` load with 0 errors (no range check), so out-of-range values must round-trip untouched. **A zero value is not an absent value** — `{0,0}` is a real anchor (top-left), which forces the field to be a POINTER; a plain `model.Point` cannot distinguish "unset" from "top-left" and would silently rewrite it. **Fix both engines**: they share the semantic model, and a fix in one is invisible to a user on the other. **Emit unauthorable data as a COMMENT** — DESCRIBE output must stay re-executable, and inventing syntax (`@anchor(parent: bottom-left, …)`) would bake in a vocabulary the storage does not have: the pair is CONTINUOUS, not 8 named anchors (observed x values 0 9 11 17 18 47 49 50 65 77 78 84 87 100). **The marketplace is the sample** when you need to know what Studio Pro actually writes: `mxcli marketplace download ` gives real Mendix-authored models, and a module .mpk holds either a raw BSON `project.mpr` or an MPR v1 SQLite one — 88 coordinate pairs from three modules turned "looks like percentages" into a measurement (all 0..100; 85 of 88 pin one coordinate to exactly 0 or 100). **Rule a unit out from the model, not the values**: pixels is impossible because `DomainModels$EntityImpl` stores only `Location` and NO size — the box is sized by the editor from the name and attribute list, so a pixel anchor would have nothing to measure against. Not applicable to `CrossAssociation`, which has no connection properties and crashes Studio Pro if given them (#50). Tests `sdk/domainmodel/connection_test.go`, `mdl/backend/modelsdk/association_connection_test.go`, `sdk/mpr/writer_domainmodel_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | +| A microflow call bound to a task queue in Studio Pro loses that binding on `create or replace microflow` — silently, and `mx check` goes from `[CE1613] "The selected task queue no longer exists"` to **0 errors**, because the configuration the error was about has been deleted | Two defects behind one report. (1) `Microflows$MicroflowCall` has `QueueSettings` in `NullFields` of `codec.RegisterTypeDefaults` (and the legacy writer hardcodes null too) — correct for a newly authored call, destructive for a stored one, and since a CREATE OR REPLACE rebuilds the whole microflow it fires on every rewrite. (2) MDL had **no queue surface at all**, so a script could not restate the binding even in principle | `mdl/executor/validate_queued_calls.go` (new: the refusal), `mdl/executor/cmd_microflows_create.go` (call site), `mdl/types/queue.go` + `mdl/backend/infrastructure.go` (`QueueBackend`), `mdl/backend/modelsdk/queue_write.go` + `sdk/mpr/queues.go` (both engines), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4`, `mdl/executor/cmd_queues.go` | **A green build can be the bug**: the fix made `mx check` report MORE errors than before (CE1613 came back), because the binding it complains about now survives. Any "error count went down" check would have scored the data loss as a fix. **`describe` showing nothing is not evidence of nothing** — the binding was invisible from every angle (describe omitted it, check went quiet, the write reported success), which is why it needed a stored-BSON probe: inject `Queue`+`QueueSettings` into a stored call, rewrite, and dump. **Isolate which property Mendix acts on before guarding on it**: a call carrying only `Queue` (with `QueueSettings` null) draws NO complaint from `mx check` — `QueueSettings` is the load-bearing one, so a guard keyed on `Queue` alone would have both missed the real case and fired on inert ones. **Get the BSON shape from a Mendix-authored model, not from the metamodel**: `mxcli marketplace download 202649` (Business Events 3.12.1) has four real queues, and all four agree — `Config` is a `Queues$BasicQueueConfig` whose `ParallelismExpression` is a **STRING**, with the sibling int32 `Parallelism` absent in every one. Writing the int (which the metamodel makes look equally valid) is inventing a property Mendix does not write. **Refuse, do not half-author** (ADR-0005): `in queue` on a call is deliberately still unimplemented, because the `Retry` shape has no Studio-Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. **Adding a keyword costs an identifier** — `QUEUE`/`QUEUES` had to go into the `keyword` rule or `queue` would have stopped working as an attribute name. Tests `mdl/backend/modelsdk/queue_write_test.go`, `mdl/executor/validate_queued_calls_test.go`, `mdl/executor/cmd_queues_mock_test.go`, `mdl/visitor/visitor_queue_test.go`, example `mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl` | diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index f9b9b8564..7016b7d5d 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -266,6 +266,8 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "ATTRIBUTES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "FILTERTYPE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "IMAGE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "QUEUE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "QUEUES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "JAR", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCY", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCIES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 70371f13b..191cbcfac 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -207,6 +207,52 @@ CREATE CONFIGURATION 'Production' SeeAlso: []string{"settings.show"}, }) + // ── Queues ────────────────────────────────────────────────────────── + + Register(SyntaxFeature{ + Path: "queue", + Summary: "Task queues — bound concurrency for queued microflow calls", + Keywords: []string{ + "queue", "queues", "task queue", "create queue", "drop queue", + "describe queue", "show queues", "parallelism", "cluster wide", + "background", "async microflow", + }, + Syntax: `CREATE [OR MODIFY] QUEUE Module.Name [( : , ... )]; +SHOW QUEUES [IN ]; +LIST QUEUES [IN ]; +DESCRIBE QUEUE Module.Name; +DROP QUEUE Module.Name; + +Properties: + Parallelism how many tasks run at once. This is an EXPRESSION, not a + number — Mendix stores it as a string. A bare integer is the + common case; quote anything else. Defaults to 1. + ClusterWide true = the limit applies across the cluster, false (default) + = per runtime instance. + Documentation free text. + +Binding a call to a queue is not yet expressible in MDL. Because a rebuild +would drop an existing binding, mxcli REFUSES to CREATE OR REPLACE/MODIFY a +microflow whose stored calls are queued — change those in Studio Pro.`, + Example: `CREATE QUEUE Ops.OrderProcessing ( + Parallelism: 3, + ClusterWide: true +); + +-- Defaults: parallelism 1, per-instance. +CREATE QUEUE Ops.Mail; + +-- An expression is legal wherever a number is. +CREATE OR MODIFY QUEUE Ops.OrderProcessing ( + Parallelism: '$MyModule.Workers', + ClusterWide: true +); + +SHOW QUEUES IN Ops; +DESCRIBE QUEUE Ops.OrderProcessing; +DROP QUEUE Ops.Mail;`, + }) + // ── Structure ─────────────────────────────────────────────────────── Register(SyntaxFeature{ diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index d4cb8d818..1aded5443 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -138,6 +138,34 @@ create constant MyModule.MaxRetries type integer default 3; create constant MyModule.EnableLogging type boolean default true; ``` +## Task Queues + +| Statement | Syntax | Notes | +|-----------|--------|-------| +| Show queues | `show queues [in module];` (`list queues` too) | Parallelism + cluster-wide flag | +| Describe queue | `describe queue Module.Name;` | Re-executable MDL | +| Create queue | `create [or modify] queue Module.Name ( Parallelism: 3, ClusterWide: true );` | Body optional; defaults `1` / `false` | +| Drop queue | `drop queue Module.Name;` | | + +`Parallelism` is an **expression**, not a number — Mendix stores it as a string +(`Queues$BasicQueueConfig.ParallelismExpression`). A bare integer is the common +case; quote anything else. + +Binding a microflow **call** to a queue is not yet expressible in MDL. Because a +rebuild would drop an existing binding, `create or replace|modify microflow` is +**refused** when the stored microflow has a queued call — change those in Studio +Pro. (Without the refusal the binding was written back as null and `mx check` +stopped reporting CE1613, so the project looked healthy while the configuration +was gone.) + +**Example:** +```sql +create queue Ops.OrderProcessing ( Parallelism: 3, ClusterWide: true ); +create queue Ops.Mail; +create or modify queue Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); +drop queue Ops.Mail; +``` + ## OData Clients, Services & External Entities | Statement | Syntax | Notes | diff --git a/mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl b/mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl new file mode 100644 index 000000000..5887a3354 --- /dev/null +++ b/mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl @@ -0,0 +1,61 @@ +-- ============================================================================ +-- Task queues: authorable in MDL, and never silently dropped by a rewrite +-- ============================================================================ +-- +-- Symptom (measured on Mendix 11.13): +-- A microflow call bound to a task queue in Studio Pro stores a +-- Queues$QueueSettings node on the call. `create or replace microflow` rebuilt +-- the microflow and both engines wrote `QueueSettings: null` — the binding was +-- gone. Nothing reported it. Worse, `mx check` went from +-- [CE1613] "The selected task queue no longer exists" +-- to 0 errors, because mxcli had deleted the configuration that the error was +-- about. `describe microflow` never showed the binding either, so the loss was +-- invisible from every angle. +-- +-- Two root causes, two fixes: +-- 1. MDL could not author a queue at all, so a script could not restate one. +-- CREATE/DROP/SHOW/DESCRIBE QUEUE now exist. The BSON is pinned against the +-- four Studio Pro-authored queues in Mendix Business Events 3.12.1 +-- (Consumer_Queue, Consumer_Processor_Queue, Producer_Queue, +-- Outbox_Cleanup_Queue), which agree exactly: Config is a +-- Queues$BasicQueueConfig carrying ParallelismExpression as a STRING, with +-- the int32 `Parallelism` absent in all four. +-- 2. MDL still cannot author a *queued call*. Until it can, rewriting a +-- microflow that has one is refused rather than performed lossily +-- (guard-don't-drop, ADR-0005). +-- +-- Verify: +-- mxcli exec mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl -p app.mpr +-- mx check app.mpr # 0 errors +-- mxcli -p app.mpr -c "list queues" +-- mxcli -p app.mpr -c "describe queue Ops.OrderProcessing" # re-executable MDL +-- ============================================================================ + +create module Ops; +/ +-- Parallelism is an EXPRESSION, not a number. A bare integer is the common case; +-- quote anything else. +create queue Ops.OrderProcessing ( + Parallelism: 3, + ClusterWide: true +); +/ +-- No properties: the defaults ("1", not cluster-wide) match a queue created in +-- Studio Pro with nothing changed. +create queue Ops.Mail; +/ +-- Re-running is idempotent with OR MODIFY, and the stored $ID is reused so any +-- call already bound to the queue keeps pointing at it. +create or modify queue Ops.OrderProcessing ( + Parallelism: 8, + ClusterWide: true +); +/ +-- "queue" is a plausible attribute name; adding the keyword must not take it +-- away (it is listed in the `keyword` rule, so it still parses as an identifier). +create entity Ops.Job ( + queue: String(100), + Status: String(50) +); +/ +drop queue Ops.Mail; diff --git a/mdl/ast/ast_queue.go b/mdl/ast/ast_queue.go new file mode 100644 index 000000000..1ad13ee0b --- /dev/null +++ b/mdl/ast/ast_queue.go @@ -0,0 +1,42 @@ +// SPDX-License-Identifier: Apache-2.0 + +package ast + +// CreateQueueStmt represents: +// +// CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( Parallelism: 3, ClusterWide: true ); +type CreateQueueStmt struct { + Name QualifiedName + Documentation string + // Parallelism is kept as written. Mendix stores it as an expression string + // (Queues$BasicQueueConfig.ParallelismExpression), so `3` and `'3'` are the + // same thing and an arbitrary expression is legal. + Parallelism string + ClusterWide bool + ExportLevel string + // CreateOrModify is set by the shared CREATE OR REPLACE/MODIFY prefix. + CreateOrModify bool +} + +func (s *CreateQueueStmt) isStatement() {} + +// DropQueueStmt represents: DROP QUEUE Module.Name; +type DropQueueStmt struct { + Name QualifiedName +} + +func (s *DropQueueStmt) isStatement() {} + +// ShowQueuesStmt represents: SHOW|LIST QUEUES [IN Module]; +type ShowQueuesStmt struct { + Module string +} + +func (s *ShowQueuesStmt) isStatement() {} + +// DescribeQueueStmt represents: DESCRIBE QUEUE Module.Name; +type DescribeQueueStmt struct { + Name QualifiedName +} + +func (s *DescribeQueueStmt) isStatement() {} diff --git a/mdl/backend/backend.go b/mdl/backend/backend.go index 69bf47719..c5ebf95d2 100644 --- a/mdl/backend/backend.go +++ b/mdl/backend/backend.go @@ -26,6 +26,7 @@ type FullBackend interface { WorkflowBackend SettingsBackend ImageBackend + QueueBackend ScheduledEventBackend RenameBackend RawUnitBackend diff --git a/mdl/backend/infrastructure.go b/mdl/backend/infrastructure.go index 254b6366e..9f3f822ca 100644 --- a/mdl/backend/infrastructure.go +++ b/mdl/backend/infrastructure.go @@ -84,6 +84,14 @@ type ImageBackend interface { ListIconCollections() ([]*types.IconCollection, error) } +// QueueBackend provides task queue (Queues$Queue) operations. +type QueueBackend interface { + ListQueues() ([]*types.Queue, error) + CreateQueue(q *types.Queue) error + UpdateQueue(q *types.Queue) error + DeleteQueue(id string) error +} + // ScheduledEventBackend provides scheduled event operations. type ScheduledEventBackend interface { ListScheduledEvents() ([]*model.ScheduledEvent, error) diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index 637e5e5d6..5fbba52d6 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -214,6 +214,11 @@ func (unsupportedBackend) CreatePublishedRestService(_ *model.PublishedRestServi return } +func (unsupportedBackend) CreateQueue(_ *types.Queue) (err0 error) { + err0 = errUnsupported("CreateQueue") + return +} + func (unsupportedBackend) CreateSnippet(_ *pages.Snippet) (err0 error) { err0 = errUnsupported("CreateSnippet") return @@ -389,6 +394,11 @@ func (unsupportedBackend) DeletePublishedRestService(_ model.ID) (err0 error) { return } +func (unsupportedBackend) DeleteQueue(_ string) (err0 error) { + err0 = errUnsupported("DeleteQueue") + return +} + func (unsupportedBackend) DeleteSnippet(_ model.ID) (err0 error) { err0 = errUnsupported("DeleteSnippet") return @@ -757,6 +767,11 @@ func (unsupportedBackend) ListPublishedRestServices() (r0 []*model.PublishedRest return } +func (unsupportedBackend) ListQueues() (r0 []*types.Queue, err1 error) { + err1 = errUnsupported("ListQueues") + return +} + func (unsupportedBackend) ListRawUnits(_ string) (r0 []*types.RawUnitInfo, err1 error) { err1 = errUnsupported("ListRawUnits") return @@ -1182,6 +1197,11 @@ func (unsupportedBackend) UpdateQualifiedNameInAllUnits(_ string, _ string) (r0 return } +func (unsupportedBackend) UpdateQueue(_ *types.Queue) (err0 error) { + err0 = errUnsupported("UpdateQueue") + return +} + func (unsupportedBackend) UpdateRawUnit(_ string, _ []uint8) (err0 error) { err0 = errUnsupported("UpdateRawUnit") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index b2d9e8c62..3af35404e 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -250,6 +250,12 @@ type MockBackend struct { UpdateImageCollectionFunc func(ic *types.ImageCollection) error DeleteImageCollectionFunc func(id string) error + // QueueBackend + ListQueuesFunc func() ([]*types.Queue, error) + CreateQueueFunc func(q *types.Queue) error + UpdateQueueFunc func(q *types.Queue) error + DeleteQueueFunc func(id string) error + // ScheduledEventBackend ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) GetScheduledEventFunc func(id model.ID) (*model.ScheduledEvent, error) diff --git a/mdl/backend/mock/mock_queue.go b/mdl/backend/mock/mock_queue.go new file mode 100644 index 000000000..e57de049f --- /dev/null +++ b/mdl/backend/mock/mock_queue.go @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mock + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// QueueBackend — task queues (Queues$Queue). +// +// The write methods default to a descriptive error rather than nil so a test +// that forgets to configure one fails on the missing stub instead of silently +// reporting a write that never happened. + +func (m *MockBackend) ListQueues() ([]*types.Queue, error) { + if m.ListQueuesFunc != nil { + return m.ListQueuesFunc() + } + return nil, nil +} + +func (m *MockBackend) CreateQueue(q *types.Queue) error { + if m.CreateQueueFunc != nil { + return m.CreateQueueFunc(q) + } + return fmt.Errorf("MockBackend.CreateQueue not configured") +} + +func (m *MockBackend) UpdateQueue(q *types.Queue) error { + if m.UpdateQueueFunc != nil { + return m.UpdateQueueFunc(q) + } + return fmt.Errorf("MockBackend.UpdateQueue not configured") +} + +func (m *MockBackend) DeleteQueue(id string) error { + if m.DeleteQueueFunc != nil { + return m.DeleteQueueFunc(id) + } + return fmt.Errorf("MockBackend.DeleteQueue not configured") +} diff --git a/mdl/backend/modelsdk/queue_write.go b/mdl/backend/modelsdk/queue_write.go new file mode 100644 index 000000000..9f31b5b93 --- /dev/null +++ b/mdl/backend/modelsdk/queue_write.go @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/bsonutil" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" +) + +// queueTypeName is the BSON storage name for a task queue document. +const queueTypeName = "Queues$Queue" + +// ListQueues reads every Queues$Queue unit. +func (b *Backend) ListQueues() ([]*types.Queue, error) { + units, err := b.reader.ListRawUnitsByType(queueTypeName) + if err != nil { + return nil, err + } + out := make([]*types.Queue, 0, len(units)) + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + return nil, fmt.Errorf("unmarshal queue %s: %w", u.ID, err) + } + q := &types.Queue{ContainerID: model.ID(u.ContainerID)} + q.ID = model.ID(u.ID) + q.TypeName = queueTypeName + q.Name, _ = doc["Name"].(string) + q.Documentation, _ = doc["Documentation"].(string) + q.Excluded, _ = doc["Excluded"].(bool) + q.ExportLevel, _ = doc["ExportLevel"].(string) + // Parallelism and cluster scope live on the nested Config node. + if cfg, ok := doc["Config"].(bson.M); ok { + q.Parallelism, _ = cfg["ParallelismExpression"].(string) + q.ClusterWide, _ = cfg["ClusterWide"].(bool) + } + out = append(out, q) + } + return out, nil +} + +// CreateQueue inserts a new Queues$Queue document. +func (b *Backend) CreateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("CreateQueue: nil queue") + } + if b.writer == nil { + return fmt.Errorf("CreateQueue: not connected for writing") + } + if q.ID == "" { + q.ID = model.ID(mmpr.GenerateID()) + } + return b.writer.InsertUnit(string(q.ID), string(q.ContainerID), "Documents", queueTypeName, serializeQueue(q)) +} + +// UpdateQueue rewrites an existing queue in place. +func (b *Backend) UpdateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("UpdateQueue: nil queue") + } + if b.writer == nil { + return fmt.Errorf("UpdateQueue: not connected for writing") + } + return b.writer.UpdateRawUnit(string(q.ID), serializeQueue(q)) +} + +// DeleteQueue removes a queue unit by ID. +func (b *Backend) DeleteQueue(id string) error { + if b.writer == nil { + return fmt.Errorf("DeleteQueue: not connected for writing") + } + return b.writer.DeleteUnit(id) +} + +// serializeQueue writes the document in the shape Studio Pro produces. +// +// Keys are alphabetical, matching every other writer here and the observed +// documents. Two things are deliberate: +// +// - ParallelismExpression is a STRING. Queues$BasicQueueConfig also declares an +// int32 Parallelism, but Studio Pro wrote it in none of the four reference +// documents, so writing it would be inventing a property. +// - An empty Parallelism becomes "1", which is what every observed queue has +// and what Mendix treats as the default; an empty expression is not a +// meaningful queue configuration. +func serializeQueue(q *types.Queue) []byte { + parallelism := q.Parallelism + if parallelism == "" { + parallelism = "1" + } + exportLevel := q.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + config := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(mmpr.GenerateID())}, + {Key: "$Type", Value: "Queues$BasicQueueConfig"}, + {Key: "ClusterWide", Value: q.ClusterWide}, + {Key: "ParallelismExpression", Value: parallelism}, + } + doc := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(string(q.ID))}, + {Key: "$Type", Value: queueTypeName}, + {Key: "Config", Value: config}, + {Key: "Documentation", Value: q.Documentation}, + {Key: "Excluded", Value: q.Excluded}, + {Key: "ExportLevel", Value: exportLevel}, + {Key: "Name", Value: q.Name}, + } + out, err := bson.Marshal(doc) + if err != nil { + // bson.Marshal on a fixed bson.D of primitives cannot fail; a nil return + // would be written as an empty unit, so surface it loudly instead. + panic(fmt.Sprintf("serializeQueue: marshal %q: %v", q.Name, err)) + } + return out +} diff --git a/mdl/backend/modelsdk/queue_write_test.go b/mdl/backend/modelsdk/queue_write_test.go new file mode 100644 index 000000000..29064256f --- /dev/null +++ b/mdl/backend/modelsdk/queue_write_test.go @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// TestSerializeQueue_MatchesStudioProShape pins the document against four real +// Studio Pro-authored queues from the Mendix Business Events module +// (Consumer_Queue, Consumer_Processor_Queue, Producer_Queue, +// Outbox_Cleanup_Queue), which agree exactly on this shape. +// +// Two assertions are the whole point of the test: +// +// - ParallelismExpression is a STRING. Queues$BasicQueueConfig also declares an +// int32 `Parallelism`, and writing that instead (or as well) would be +// inventing a property Mendix does not write. +// - `Parallelism` must be ABSENT. It appeared in none of the four references. +func TestSerializeQueue_MatchesStudioProShape(t *testing.T) { + q := &types.Queue{Name: "OrderProcessing", Parallelism: "3", ClusterWide: true} + q.ID = "11111111-1111-1111-1111-111111111111" + + var doc bson.M + if err := bson.Unmarshal(serializeQueue(q), &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + + if got := doc["$Type"]; got != "Queues$Queue" { + t.Errorf("$Type = %v, want Queues$Queue", got) + } + if got := doc["Name"]; got != "OrderProcessing" { + t.Errorf("Name = %v", got) + } + if got := doc["ExportLevel"]; got != "Hidden" { + t.Errorf("ExportLevel = %v, want Hidden (every reference document uses it)", got) + } + if _, ok := doc["Excluded"].(bool); !ok { + t.Errorf("Excluded missing or not a bool: %v", doc["Excluded"]) + } + + cfg, ok := doc["Config"].(bson.M) + if !ok { + t.Fatalf("Config is not a document: %T", doc["Config"]) + } + if got := cfg["$Type"]; got != "Queues$BasicQueueConfig" { + t.Errorf("Config.$Type = %v", got) + } + if got, ok := cfg["ParallelismExpression"].(string); !ok || got != "3" { + t.Errorf("ParallelismExpression = %#v, want the string \"3\" — Mendix stores an expression, not a number", cfg["ParallelismExpression"]) + } + if _, present := cfg["Parallelism"]; present { + t.Error("Parallelism must not be written: Studio Pro wrote it in none of the four reference queues") + } + if got := cfg["ClusterWide"]; got != true { + t.Errorf("ClusterWide = %v, want true", got) + } +} + +// TestSerializeQueue_Defaults covers the empty case: every observed queue has a +// parallelism, so an unspecified one becomes "1" rather than an empty +// expression, which is not a meaningful configuration. +func TestSerializeQueue_Defaults(t *testing.T) { + q := &types.Queue{Name: "Plain"} + q.ID = "22222222-2222-2222-2222-222222222222" + + var doc bson.M + if err := bson.Unmarshal(serializeQueue(q), &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + cfg := doc["Config"].(bson.M) + if got := cfg["ParallelismExpression"]; got != "1" { + t.Errorf("default ParallelismExpression = %v, want \"1\"", got) + } + if got := cfg["ClusterWide"]; got != false { + t.Errorf("default ClusterWide = %v, want false", got) + } + if got := doc["ExportLevel"]; got != "Hidden" { + t.Errorf("default ExportLevel = %v, want Hidden", got) + } +} + +// TestQueueRoundTrip writes a queue and reads it back through the real backend. +func TestQueueRoundTrip(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + if err := b.CreateQueue(&types.Queue{ + ContainerID: mod.ID, Name: "ZzQueue", Parallelism: "5", ClusterWide: true, + }); err != nil { + t.Fatalf("CreateQueue: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + queues, err := b2.ListQueues() + if err != nil { + t.Fatalf("ListQueues: %v", err) + } + for _, q := range queues { + if q.Name != "ZzQueue" { + continue + } + if q.Parallelism != "5" { + t.Errorf("Parallelism = %q, want 5", q.Parallelism) + } + if !q.ClusterWide { + t.Error("ClusterWide did not round-trip") + } + return + } + t.Fatalf("ZzQueue not found after create (got %d queues)", len(queues)) +} diff --git a/mdl/backend/modelsdk/unimplemented_gen.go b/mdl/backend/modelsdk/unimplemented_gen.go index ce24b4208..62cea55be 100644 --- a/mdl/backend/modelsdk/unimplemented_gen.go +++ b/mdl/backend/modelsdk/unimplemented_gen.go @@ -180,6 +180,10 @@ func (unimplemented) CreatePublishedRestService(_ *model.PublishedRestService) e return errUnimplemented("CreatePublishedRestService") } +func (unimplemented) CreateQueue(_ *types.Queue) error { + return errUnimplemented("CreateQueue") +} + func (unimplemented) CreateSnippet(_ *pages.Snippet) error { return errUnimplemented("CreateSnippet") } @@ -321,6 +325,10 @@ func (unimplemented) DeletePublishedRestService(_ model.ID) error { return errUnimplemented("DeletePublishedRestService") } +func (unimplemented) DeleteQueue(_ string) error { + return errUnimplemented("DeleteQueue") +} + func (unimplemented) DeleteSnippet(_ model.ID) error { return errUnimplemented("DeleteSnippet") } @@ -685,6 +693,11 @@ func (unimplemented) ListPublishedRestServices() ([]*model.PublishedRestService, return r0, errUnimplemented("ListPublishedRestServices") } +func (unimplemented) ListQueues() ([]*types.Queue, error) { + var r0 []*types.Queue + return r0, errUnimplemented("ListQueues") +} + func (unimplemented) ListRawUnits(_ string) ([]*types.RawUnitInfo, error) { var r0 []*types.RawUnitInfo return r0, errUnimplemented("ListRawUnits") @@ -1061,6 +1074,10 @@ func (unimplemented) UpdateQualifiedNameInAllUnits(_ string, _ string) (int, err return r0, errUnimplemented("UpdateQualifiedNameInAllUnits") } +func (unimplemented) UpdateQueue(_ *types.Queue) error { + return errUnimplemented("UpdateQueue") +} + func (unimplemented) UpdateRawUnit(_ string, _ []uint8) error { return errUnimplemented("UpdateRawUnit") } diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 4daeaa705..44050fa80 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -702,6 +702,23 @@ func (b *MprBackend) ListIconCollections() ([]*types.IconCollection, error) { return b.reader.ListIconCollections() } +// --------------------------------------------------------------------------- +// QueueBackend +// --------------------------------------------------------------------------- + +func (b *MprBackend) ListQueues() ([]*types.Queue, error) { + return b.reader.ListQueues() +} +func (b *MprBackend) CreateQueue(q *types.Queue) error { + return b.writer.CreateQueue(q) +} +func (b *MprBackend) UpdateQueue(q *types.Queue) error { + return b.writer.UpdateQueue(q) +} +func (b *MprBackend) DeleteQueue(id string) error { + return b.writer.DeleteQueue(id) +} + // --------------------------------------------------------------------------- // ScheduledEventBackend // --------------------------------------------------------------------------- diff --git a/mdl/executor/cmd_microflows_create.go b/mdl/executor/cmd_microflows_create.go index d48a134bc..81aa0238d 100644 --- a/mdl/executor/cmd_microflows_create.go +++ b/mdl/executor/cmd_microflows_create.go @@ -91,6 +91,14 @@ func execCreateMicroflow(ctx *ExecContext, s *ast.CreateMicroflowStmt) error { // For CREATE OR REPLACE/MODIFY, reuse the existing ID to preserve references qualifiedName := s.Name.Module + "." + s.Name.Name + + // Refuse before writing if the stored microflow has a call bound to a task + // queue: the rebuild would null it out and nothing downstream would notice. + if existingID != "" { + if err := checkNoQueuedCalls(ctx, existingID, qualifiedName); err != nil { + return err + } + } microflowID := model.ID(types.GenerateID()) if existingID != "" { microflowID = existingID diff --git a/mdl/executor/cmd_queues.go b/mdl/executor/cmd_queues.go new file mode 100644 index 000000000..76600c65b --- /dev/null +++ b/mdl/executor/cmd_queues.go @@ -0,0 +1,183 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package executor — task queue commands (CREATE/DROP/SHOW/DESCRIBE QUEUE). +// +// A Mendix task queue (Queues$Queue) governs how many instances of a queued +// microflow call run at once, and whether that limit is per runtime instance or +// cluster-wide. +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// findQueue returns the queue with the given module-qualified name, or nil. +func findQueue(ctx *ExecContext, moduleName, name string) *types.Queue { + queues, err := ctx.Backend.ListQueues() + if err != nil { + return nil + } + h, err := getHierarchy(ctx) + if err != nil { + return nil + } + for _, q := range queues { + if !strings.EqualFold(q.Name, name) { + continue + } + mod := h.GetModuleName(h.FindModuleID(q.ContainerID)) + if strings.EqualFold(mod, moduleName) { + return q + } + } + return nil +} + +// execCreateQueue handles CREATE [OR REPLACE|MODIFY] QUEUE Module.Name (...). +func execCreateQueue(ctx *ExecContext, s *ast.CreateQueueStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + + module, err := findOrCreateModule(ctx, s.Name.Module) + if err != nil { + return err + } + + existing := findQueue(ctx, s.Name.Module, s.Name.Name) + if existing != nil && !s.CreateOrModify { + return mdlerrors.NewAlreadyExists("queue", s.Name.String()) + } + + containerID := module.ID + if existing != nil { + containerID = existing.ContainerID + } + + q := &types.Queue{ + ContainerID: containerID, + Name: s.Name.Name, + Documentation: s.Documentation, + Parallelism: s.Parallelism, + ClusterWide: s.ClusterWide, + ExportLevel: s.ExportLevel, + } + + if existing != nil { + q.ID = existing.ID + if err := ctx.Backend.UpdateQueue(q); err != nil { + return mdlerrors.NewBackend("update queue", err) + } + fmt.Fprintf(ctx.Output, "Modified queue: %s\n", s.Name.String()) + return nil + } + if err := ctx.Backend.CreateQueue(q); err != nil { + return mdlerrors.NewBackend("create queue", err) + } + fmt.Fprintf(ctx.Output, "Created queue: %s\n", s.Name.String()) + return nil +} + +// execDropQueue handles DROP QUEUE Module.Name. +func execDropQueue(ctx *ExecContext, s *ast.DropQueueStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + existing := findQueue(ctx, s.Name.Module, s.Name.Name) + if existing == nil { + return mdlerrors.NewNotFound("queue", s.Name.String()) + } + if err := ctx.Backend.DeleteQueue(string(existing.ID)); err != nil { + return mdlerrors.NewBackend("drop queue", err) + } + fmt.Fprintf(ctx.Output, "Dropped queue: %s\n", s.Name.String()) + return nil +} + +// execShowQueues handles SHOW|LIST QUEUES [IN Module]. +func execShowQueues(ctx *ExecContext, s *ast.ShowQueuesStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + queues, err := ctx.Backend.ListQueues() + if err != nil { + return mdlerrors.NewBackend("list queues", err) + } + h, err := getHierarchy(ctx) + if err != nil { + return mdlerrors.NewBackend("build hierarchy", err) + } + + type row struct{ qualified, parallelism, clusterWide string } + var rows []row + for _, q := range queues { + mod := h.GetModuleName(h.FindModuleID(q.ContainerID)) + if s.Module != "" && !strings.EqualFold(mod, s.Module) { + continue + } + rows = append(rows, row{ + qualified: mod + "." + q.Name, + parallelism: q.Parallelism, + clusterWide: fmt.Sprintf("%t", q.ClusterWide), + }) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].qualified < rows[j].qualified }) + + result := &TableResult{ + Columns: []string{"Queue", "Parallelism", "Cluster Wide"}, + Summary: fmt.Sprintf("(%d queue(s))", len(rows)), + } + for _, r := range rows { + result.Rows = append(result.Rows, []any{r.qualified, r.parallelism, r.clusterWide}) + } + return writeResult(ctx, result) +} + +// execDescribeQueue handles DESCRIBE QUEUE Module.Name, emitting re-executable +// MDL so describe → exec round-trips. +func execDescribeQueue(ctx *ExecContext, s *ast.DescribeQueueStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + q := findQueue(ctx, s.Name.Module, s.Name.Name) + if q == nil { + return mdlerrors.NewNotFound("queue", s.Name.String()) + } + + if q.Documentation != "" { + fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", q.Documentation) + } + fmt.Fprintf(ctx.Output, "create or modify queue %s (\n", s.Name.String()) + // Parallelism is an expression string; quote it unless it is a plain integer, + // so an expression survives the round-trip. + fmt.Fprintf(ctx.Output, " Parallelism: %s,\n", formatParallelism(q.Parallelism)) + fmt.Fprintf(ctx.Output, " ClusterWide: %t,\n", q.ClusterWide) + fmt.Fprint(ctx.Output, ");\n") + return nil +} + +// formatParallelism renders the expression bare when it is a plain integer and +// quoted otherwise. +func formatParallelism(expr string) string { + if expr == "" { + return "1" + } + for _, r := range expr { + if r < '0' || r > '9' { + return "'" + strings.ReplaceAll(expr, "'", "''") + "'" + } + } + return expr +} diff --git a/mdl/executor/cmd_queues_mock_test.go b/mdl/executor/cmd_queues_mock_test.go new file mode 100644 index 000000000..d3424afeb --- /dev/null +++ b/mdl/executor/cmd_queues_mock_test.go @@ -0,0 +1,247 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +func mkQueue(containerID model.ID, name, parallelism string, clusterWide bool) *types.Queue { + q := &types.Queue{ + ContainerID: containerID, + Name: name, + Parallelism: parallelism, + ClusterWide: clusterWide, + } + q.ID = nextID("queue") + return q +} + +func TestShowQueues_Mock(t *testing.T) { + mod := mkModule("Ops") + q1 := mkQueue(mod.ID, "OrderProcessing", "3", true) + q2 := mkQueue(mod.ID, "Mail", "1", false) + + h := mkHierarchy(mod) + withContainer(h, q1.ContainerID, mod.ID) + withContainer(h, q2.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{q1, q2}, nil }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execShowQueues(ctx, &ast.ShowQueuesStmt{})) + + out := buf.String() + assertContainsStr(t, out, "Ops.OrderProcessing") + assertContainsStr(t, out, "Ops.Mail") + assertContainsStr(t, out, "(2 queue(s))") +} + +func TestShowQueues_Mock_FilterByModule(t *testing.T) { + alpha := mkModule("Alpha") + beta := mkModule("Beta") + q1 := mkQueue(alpha.ID, "One", "1", false) + q2 := mkQueue(beta.ID, "Two", "2", false) + + h := mkHierarchy(alpha, beta) + withContainer(h, q1.ContainerID, alpha.ID) + withContainer(h, q2.ContainerID, beta.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{q1, q2}, nil }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execShowQueues(ctx, &ast.ShowQueuesStmt{Module: "Beta"})) + + out := buf.String() + assertNotContainsStr(t, out, "Alpha.One") + assertContainsStr(t, out, "Beta.Two") +} + +// TestCreateQueue_Mock_PassesParallelismThrough checks that the expression +// reaches the backend as written. Parallelism is an expression, so it must not +// be parsed into a number anywhere on the way down. +func TestCreateQueue_Mock_PassesParallelismThrough(t *testing.T) { + mod := mkModule("Ops") + h := mkHierarchy(mod) + + var created *types.Queue + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + CreateQueueFunc: func(q *types.Queue) error { + created = q + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.CreateQueueStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}, + Parallelism: "$Config/Workers", + ClusterWide: true, + } + assertNoError(t, execCreateQueue(ctx, stmt)) + + if created == nil { + t.Fatal("CreateQueue was not called") + } + if created.Parallelism != "$Config/Workers" { + t.Errorf("Parallelism = %q, want the expression verbatim", created.Parallelism) + } + if !created.ClusterWide { + t.Error("ClusterWide did not reach the backend") + } + assertContainsStr(t, buf.String(), "Created queue: Ops.OrderProcessing") +} + +func TestCreateQueue_Mock_DuplicateWithoutOrModify(t *testing.T) { + mod := mkModule("Ops") + existing := mkQueue(mod.ID, "OrderProcessing", "1", false) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + CreateQueueFunc: func(q *types.Queue) error { + t.Error("CreateQueue must not be called for a duplicate") + return nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.CreateQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}} + if err := execCreateQueue(ctx, stmt); err == nil { + t.Fatal("expected an already-exists error") + } +} + +func TestCreateQueue_Mock_OrModifyUpdates(t *testing.T) { + mod := mkModule("Ops") + existing := mkQueue(mod.ID, "OrderProcessing", "1", false) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var updated *types.Queue + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + UpdateQueueFunc: func(q *types.Queue) error { + updated = q + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.CreateQueueStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}, + Parallelism: "8", + CreateOrModify: true, + } + assertNoError(t, execCreateQueue(ctx, stmt)) + + if updated == nil { + t.Fatal("UpdateQueue was not called") + } + // The stored ID must be reused, or the update becomes a second queue. + if updated.ID != existing.ID { + t.Errorf("ID = %q, want the existing %q", updated.ID, existing.ID) + } + if updated.Parallelism != "8" { + t.Errorf("Parallelism = %q, want 8", updated.Parallelism) + } + assertContainsStr(t, buf.String(), "Modified queue: Ops.OrderProcessing") +} + +func TestDropQueue_Mock(t *testing.T) { + mod := mkModule("Ops") + existing := mkQueue(mod.ID, "OrderProcessing", "1", false) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var deleted string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + DeleteQueueFunc: func(id string) error { + deleted = id + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.DropQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}} + assertNoError(t, execDropQueue(ctx, stmt)) + + if deleted != string(existing.ID) { + t.Errorf("deleted %q, want %q", deleted, existing.ID) + } + assertContainsStr(t, buf.String(), "Dropped queue: Ops.OrderProcessing") +} + +func TestDropQueue_Mock_NotFound(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, + DeleteQueueFunc: func(id string) error { + t.Error("DeleteQueue must not be called when the queue does not exist") + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(mkHierarchy())) + stmt := &ast.DropQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "Missing"}} + if err := execDropQueue(ctx, stmt); err == nil { + t.Fatal("expected a not-found error") + } +} + +// TestDescribeQueue_Mock_RoundTrips checks that DESCRIBE emits MDL that can be +// fed straight back in, including quoting a non-numeric parallelism expression. +func TestDescribeQueue_Mock_RoundTrips(t *testing.T) { + mod := mkModule("Ops") + q := mkQueue(mod.ID, "OrderProcessing", "$Config/Workers", true) + h := mkHierarchy(mod) + withContainer(h, q.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{q}, nil }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + stmt := &ast.DescribeQueueStmt{Name: ast.QualifiedName{Module: "Ops", Name: "OrderProcessing"}} + assertNoError(t, execDescribeQueue(ctx, stmt)) + + out := buf.String() + assertContainsStr(t, out, "create or modify queue Ops.OrderProcessing (") + assertContainsStr(t, out, "Parallelism: '$Config/Workers',") + assertContainsStr(t, out, "ClusterWide: true,") +} + +func TestFormatParallelism(t *testing.T) { + tests := []struct{ in, want string }{ + {"", "1"}, + {"3", "3"}, + {"$Config/Workers", "'$Config/Workers'"}, + {"it's", "'it''s'"}, // Mendix escapes a quote by doubling it, never with a backslash + } + for _, tt := range tests { + if got := formatParallelism(tt.in); got != tt.want { + t.Errorf("formatParallelism(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index bae5ee4fb..044f9aead 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -207,6 +207,21 @@ func registerNavigationHandlers(r *Registry) { }) } +func registerQueueHandlers(r *Registry) { + r.Register(&ast.CreateQueueStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execCreateQueue(ctx, stmt.(*ast.CreateQueueStmt)) + }) + r.Register(&ast.DropQueueStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDropQueue(ctx, stmt.(*ast.DropQueueStmt)) + }) + r.Register(&ast.ShowQueuesStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execShowQueues(ctx, stmt.(*ast.ShowQueuesStmt)) + }) + r.Register(&ast.DescribeQueueStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDescribeQueue(ctx, stmt.(*ast.DescribeQueueStmt)) + }) +} + func registerImageHandlers(r *Registry) { r.Register(&ast.CreateImageCollectionStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execCreateImageCollection(ctx, stmt.(*ast.CreateImageCollectionStmt)) diff --git a/mdl/executor/registry.go b/mdl/executor/registry.go index 8756717f8..761a87fc6 100644 --- a/mdl/executor/registry.go +++ b/mdl/executor/registry.go @@ -38,6 +38,7 @@ func NewRegistry() *Registry { registerSecurityHandlers(r) registerNavigationHandlers(r) registerImageHandlers(r) + registerQueueHandlers(r) registerWorkflowHandlers(r) registerBusinessEventHandlers(r) registerSettingsHandlers(r) diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 538505250..5ec9c72ab 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -206,6 +206,7 @@ func allKnownStatements() []ast.Statement { &ast.CreateODataServiceStmt{}, &ast.CreatePageStmtV3{}, &ast.CreatePublishedRestServiceStmt{}, + &ast.CreateQueueStmt{}, &ast.CreateRestClientStmt{}, &ast.CreateSnippetStmtV3{}, &ast.CreateUserRoleStmt{}, @@ -215,6 +216,7 @@ func allKnownStatements() []ast.Statement { &ast.DescribeCatalogTableStmt{}, &ast.DescribeContractFromOpenAPIStmt{}, &ast.DescribeFragmentFromStmt{}, + &ast.DescribeQueueStmt{}, &ast.DescribeStmt{}, &ast.DescribeStylingStmt{}, &ast.DisconnectStmt{}, @@ -245,6 +247,7 @@ func allKnownStatements() []ast.Statement { &ast.DropODataServiceStmt{}, &ast.DropPageStmt{}, &ast.DropPublishedRestServiceStmt{}, + &ast.DropQueueStmt{}, &ast.DropRestClientStmt{}, &ast.DropSnippetStmt{}, &ast.DropUserRoleStmt{}, @@ -278,6 +281,7 @@ func allKnownStatements() []ast.Statement { &ast.SetStmt{}, &ast.ShowDesignPropertiesStmt{}, &ast.ShowFeaturesStmt{}, + &ast.ShowQueuesStmt{}, &ast.ShowStmt{}, &ast.ShowWidgetsStmt{}, &ast.SQLConnectionsStmt{}, diff --git a/mdl/executor/validate_queued_calls.go b/mdl/executor/validate_queued_calls.go new file mode 100644 index 000000000..fbbf2ea3f --- /dev/null +++ b/mdl/executor/validate_queued_calls.go @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" +) + +// checkNoQueuedCalls refuses to rewrite a microflow that has a call bound to a +// task queue, because the rewrite would silently drop that binding. +// +// A CREATE OR REPLACE/MODIFY rebuilds the microflow from the statement, and both +// engines hardcode QueueSettings to null on a call — correct for a newly +// authored call, wrong for one that was already queued: +// +// codec.RegisterTypeDefaults("Microflows$MicroflowCall", codec.TypeDefaults{ +// NullFields: []string{"QueueSettings"}, ... +// +// Nothing signals the loss afterwards. Measured on Mendix 11.13: with the +// binding present `mx check` reports CE1613 ("The selected task queue … no +// longer exists") on the call activity; after the rewrite it reports 0 errors. +// So mxcli "fixes" the build by deleting the user's configuration, and the +// project then looks healthy. +// +// MDL cannot yet author a queued call, so the binding cannot be restated in the +// script either — refusing is the only option that does not lose data +// (guard-don't-drop, ADR-0005). Remove this guard when `in queue` exists and the +// rebuild carries the binding through. +func checkNoQueuedCalls(ctx *ExecContext, microflowID model.ID, qualifiedName string) error { + raw, err := ctx.Backend.GetRawUnit(microflowID) + if err != nil { + // Unreadable stored unit is not this guard's business; the rewrite path + // reports its own errors. + return nil + } + queues := queuedCallTargets(raw) + if len(queues) == 0 { + return nil + } + sort.Strings(queues) + return mdlerrors.NewUnsupported(fmt.Sprintf( + "microflow %s has %d call(s) bound to a task queue (%s), and rewriting it would silently "+ + "drop that binding — MDL cannot express a queued call yet, so the queue cannot be restated "+ + "in this script.\n"+ + " Change the microflow in Studio Pro, or remove the task queue from the call first "+ + "(the binding lives on the call activity, not the microflow).", + qualifiedName, len(queues), strings.Join(queues, ", "))) +} + +// queuedCallTargets walks a stored microflow document and returns the queue +// bound to each call that has one. +// +// The binding that matters is QueueSettings — a Queues$QueueSettings node whose +// own Queue property names the queue. The call's top-level Queue property is +// also read, because it is in the metamodel, but on its own it is inert: +// measured on 11.13, a call carrying only Queue (with QueueSettings null) draws +// no complaint from mx check at all, while one carrying QueueSettings does. +func queuedCallTargets(v any) []string { + var out []string + switch t := v.(type) { + case map[string]any: + if qs, ok := t["QueueSettings"].(map[string]any); ok && qs != nil { + name, _ := qs["Queue"].(string) + if name == "" { + name = "(unnamed queue)" + } + out = append(out, name) + } else if name, ok := t["Queue"].(string); ok && name != "" { + // Present without QueueSettings: not something Mendix acts on, but + // still authored state that the rewrite would drop. + out = append(out, name) + } + for _, val := range t { + out = append(out, queuedCallTargets(val)...) + } + case []any: + for _, el := range t { + out = append(out, queuedCallTargets(el)...) + } + } + return dedupeStrings(out) +} + +func dedupeStrings(in []string) []string { + if len(in) < 2 { + return in + } + seen := make(map[string]bool, len(in)) + out := in[:0] + for _, s := range in { + if seen[s] { + continue + } + seen[s] = true + out = append(out, s) + } + return out +} diff --git a/mdl/executor/validate_queued_calls_test.go b/mdl/executor/validate_queued_calls_test.go new file mode 100644 index 000000000..b44d4066c --- /dev/null +++ b/mdl/executor/validate_queued_calls_test.go @@ -0,0 +1,160 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/microflows" +) + +// storedCall builds the shape a stored MicroflowCall has once a task queue is +// bound to it in Studio Pro. The binding lives two levels down (Queues$QueueSettings +// inside the call's QueueSettings property), so the walk has to be recursive. +func storedCall(queueSettings map[string]any, topLevelQueue any) map[string]any { + return map[string]any{ + "$Type": "Microflows$Microflow", + "ObjectCollection": map[string]any{ + "Objects": []any{ + map[string]any{ + "$Type": "Microflows$ActionActivity", + "Action": map[string]any{ + "$Type": "Microflows$MicroflowCall", + "Microflow": "Q.Target", + "Queue": topLevelQueue, + "QueueSettings": queueSettings, + }, + }, + }, + }, + } +} + +func TestQueuedCallTargets_FindsQueueSettings(t *testing.T) { + got := queuedCallTargets(storedCall(map[string]any{ + "$Type": "Queues$QueueSettings", + "Queue": "Q.MyQueue", + }, nil)) + if len(got) != 1 || got[0] != "Q.MyQueue" { + t.Fatalf("queuedCallTargets = %v, want [Q.MyQueue]", got) + } +} + +func TestQueuedCallTargets_NoBinding(t *testing.T) { + if got := queuedCallTargets(storedCall(nil, nil)); len(got) != 0 { + t.Fatalf("queuedCallTargets = %v, want none for an unqueued call", got) + } +} + +// A call can carry a bare Queue with QueueSettings null. Measured on 11.13 that +// is inert (mx check does not complain), but it is still authored state, and a +// rewrite would drop it. +func TestQueuedCallTargets_BareQueue(t *testing.T) { + got := queuedCallTargets(storedCall(nil, "Q.MyQueue")) + if len(got) != 1 || got[0] != "Q.MyQueue" { + t.Fatalf("queuedCallTargets = %v, want [Q.MyQueue]", got) + } +} + +func TestQueuedCallTargets_Dedupes(t *testing.T) { + doc := map[string]any{"Objects": []any{ + map[string]any{"QueueSettings": map[string]any{"Queue": "Q.A"}}, + map[string]any{"QueueSettings": map[string]any{"Queue": "Q.A"}}, + map[string]any{"QueueSettings": map[string]any{"Queue": "Q.B"}}, + }} + got := queuedCallTargets(doc) + if len(got) != 2 { + t.Fatalf("queuedCallTargets = %v, want 2 distinct queues", got) + } +} + +// TestCheckNoQueuedCalls_Refuses is the guard against the data loss itself: a +// CREATE OR REPLACE of a microflow with a queued call used to succeed and write +// QueueSettings back as null, so mx check went from CE1613 to 0 errors by +// deleting the user's configuration. +func TestCheckNoQueuedCalls_Refuses(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(map[string]any{"Queue": "Q.MyQueue"}, nil), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + + err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller") + if err == nil { + t.Fatal("expected a refusal for a microflow with a queued call") + } + msg := err.Error() + for _, want := range []string{"Q.ACT_Caller", "Q.MyQueue", "task queue"} { + if !strings.Contains(msg, want) { + t.Errorf("error message missing %q:\n%s", want, msg) + } + } +} + +func TestCheckNoQueuedCalls_AllowsUnqueued(t *testing.T) { + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(nil, nil), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + if err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller"); err != nil { + t.Fatalf("unqueued microflow must still be rewritable: %v", err) + } +} + +// An unreadable unit is not this guard's business — the rewrite path reports its +// own errors, and failing here would block writes for an unrelated reason. +func TestCheckNoQueuedCalls_UnreadableUnitDoesNotBlock(t *testing.T) { + ctx, _ := newMockCtx(t) // default mock: GetRawUnit is not configured, so it errors + if err := checkNoQueuedCalls(ctx, "mf-1", "Q.ACT_Caller"); err != nil { + t.Fatalf("unreadable unit must not block the write: %v", err) + } +} + +// TestCreateOrModifyMicroflow_RefusesQueuedCall drives the guard through the +// real CREATE OR MODIFY path. This is the test that fails if the call site in +// execCreateMicroflow is removed — checkNoQueuedCalls passing on its own proves +// nothing about whether anything calls it. +func TestCreateOrModifyMicroflow_RefusesQueuedCall(t *testing.T) { + mod := mkModule("Q") + h := mkHierarchy(mod) + + existing := µflows.Microflow{Name: "ACT_Caller"} + existing.ID = "mf-existing" + existing.ContainerID = mod.ID + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListMicroflowsFunc: func() ([]*microflows.Microflow, error) { + return []*microflows.Microflow{existing}, nil + }, + GetRawUnitFunc: func(id model.ID) (map[string]any, error) { + return storedCall(map[string]any{"Queue": "Q.MyQueue"}, nil), nil + }, + UpdateMicroflowFunc: func(mf *microflows.Microflow) error { + t.Error("UpdateMicroflow must not run: the rewrite would drop the queue binding") + return nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + err := execCreateMicroflow(ctx, &ast.CreateMicroflowStmt{ + Name: ast.QualifiedName{Module: "Q", Name: "ACT_Caller"}, + CreateOrModify: true, + }) + if err == nil { + t.Fatal("expected a refusal, got a successful rewrite") + } + if !strings.Contains(err.Error(), "Q.MyQueue") { + t.Errorf("error should name the queue that would be lost:\n%s", err) + } +} diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index 61a832e43..87fe9226c 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -354,6 +354,8 @@ READONLY: R E A D O N L Y; ATTRIBUTES: A T T R I B U T E S; FILTERTYPE: F I L T E R T Y P E; IMAGE: I M A G E; +QUEUE: Q U E U E; +QUEUES: Q U E U E S; COLLECTION: C O L L E C T I O N S?; // accept singular + plural ("collection(s)") JAR: J A R; DEPENDENCY: D E P E N D E N C Y; diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index cebf49dd0..fe263a5dc 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -115,6 +115,7 @@ createStatement | createUserRoleStatement | createDemoUserStatement | createImageCollectionStatement + | createQueueStatement | createJsonStructureStatement | createImportMappingStatement | createExportMappingStatement @@ -309,6 +310,7 @@ dropStatement | DROP SNIPPET qualifiedName | DROP MODULE qualifiedName | DROP NOTEBOOK qualifiedName + | DROP QUEUE qualifiedName | DROP JAVA ACTION qualifiedName | DROP JAVASCRIPT ACTION qualifiedName | DROP INDEX qualifiedName ON qualifiedName diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index f5e606fae..c747df1a7 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -35,6 +35,7 @@ showStatement | showOrList CONSTANT VALUES (IN (qualifiedName | IDENTIFIER))? | showOrList LAYOUTS (IN (qualifiedName | IDENTIFIER))? | showOrList NOTEBOOKS (IN (qualifiedName | IDENTIFIER))? + | showOrList QUEUES (IN (qualifiedName | IDENTIFIER))? | showOrList JAVA ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList JAVASCRIPT ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList IMAGE COLLECTION (IN (qualifiedName | IDENTIFIER))? @@ -162,6 +163,7 @@ describeStatement | DESCRIBE STYLING ON (PAGE | SNIPPET) qualifiedName (WIDGET IDENTIFIER)? // DESCRIBE STYLING ON PAGE Module.Page [WIDGET name] | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name + | DESCRIBE QUEUE qualifiedName // DESCRIBE QUEUE Module.Name | DESCRIBE DATABASE CONNECTION qualifiedName // DESCRIBE DATABASE CONNECTION Module.Name | DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)? // DESCRIBE SETTINGS [CONFIGURATION 'Default'] | DESCRIBE FRAGMENT FROM PAGE qualifiedName WIDGET identifierOrKeyword // DESCRIBE FRAGMENT FROM PAGE Module.Page WIDGET name diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index e7e31d312..3ad98dfc5 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -331,6 +331,29 @@ enumerationOption | FOLDER STRING_LITERAL // place the enumeration in a module folder (Bug 12b) ; +// ============================================================================= +// TASK QUEUE CREATION +// ============================================================================= + +/** + * CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( Parallelism: 3, ClusterWide: true ); + * + * Parallelism is stored by Mendix as an EXPRESSION string + * (Queues$BasicQueueConfig.ParallelismExpression), so it accepts a number or a + * quoted expression. + */ +createQueueStatement + : QUEUE qualifiedName queueBody? + ; + +queueBody + : LPAREN (queueProperty (COMMA queueProperty)* COMMA?)? RPAREN + ; + +queueProperty + : identifierOrKeyword COLON (NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | identifierOrKeyword) + ; + // ============================================================================= // IMAGE COLLECTION CREATION // ============================================================================= diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index 9db3b5d0f..a18faa5bb 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -486,7 +486,7 @@ annotationParenValue */ keyword // DDL / DML - : ADD | ALTER | BATCH | BROWSER | CHANGE | CLOSE | COMMIT | CREATE | DECLARE | DELETE | DESCRIBE + : QUEUE | QUEUES | ADD | ALTER | BATCH | BROWSER | CHANGE | CLOSE | COMMIT | CREATE | DECLARE | DELETE | DESCRIBE | DOWNLOAD | DROP | EXECUTE | EXPORT | GENERATE | IMPORT | INSERT | INTO | MODIFY | MOVE | REFRESH | SYNCHRONIZE | UNSYNCHRONIZED | REMOVE | RENAME | REPLACE | RETRIEVE | RETURN | ROLLBACK | SET | UPDATE diff --git a/mdl/types/queue.go b/mdl/types/queue.go new file mode 100644 index 000000000..1560bd681 --- /dev/null +++ b/mdl/types/queue.go @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: Apache-2.0 + +package types + +import "github.com/mendixlabs/mxcli/model" + +// Queue is a Mendix task queue (Queues$Queue) — the configuration that governs +// how many instances of a queued microflow call run at once, and whether that +// limit is per-node or cluster-wide. +// +// The shape here follows four Studio Pro-authored queues from the Mendix +// Business Events module (Consumer_Queue, Consumer_Processor_Queue, +// Producer_Queue, Outbox_Cleanup_Queue), which agree exactly: +// +// { "$Type": "Queues$Queue", "Name": "Consumer_Queue", +// "Documentation": "", "Excluded": false, "ExportLevel": "Hidden", +// "Config": { "$Type": "Queues$BasicQueueConfig", +// "ClusterWide": false, "ParallelismExpression": "1" } } +// +// Two details are load-bearing and are not guesses: +// +// 1. Parallelism is stored as ParallelismExpression, a STRING holding a Mendix +// expression ("1"). Queues$BasicQueueConfig also declares an int32 +// `Parallelism`, but Studio Pro wrote it zero times out of four — so it is +// not written here either. +// 2. Config was Queues$BasicQueueConfig in all four, and it is the only config +// type in the metamodel, so there is no variant to dispatch on. +type Queue struct { + model.BaseElement + ContainerID model.ID `json:"containerId"` + Name string `json:"name"` + Documentation string `json:"documentation,omitempty"` + Excluded bool `json:"excluded,omitempty"` + ExportLevel string `json:"exportLevel,omitempty"` + + // Parallelism is the expression form (Config.ParallelismExpression), kept as + // a string because that is how Mendix stores it: usually a literal like "3", + // but any Mendix expression is valid. + Parallelism string `json:"parallelism,omitempty"` + // ClusterWide makes the parallelism limit apply across the cluster rather + // than per runtime instance (Config.ClusterWide). + ClusterWide bool `json:"clusterWide,omitempty"` +} + +// GetName returns the queue's name. +func (q *Queue) GetName() string { return q.Name } + +// GetContainerID returns the container ID. +func (q *Queue) GetContainerID() model.ID { return q.ContainerID } diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index c4b043d6e..24598a7ed 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -829,6 +829,10 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { b.statements = append(b.statements, &ast.DropImageCollectionStmt{ Name: buildQualifiedName(names[0]), }) + } else if ctx.QUEUE() != nil { + b.statements = append(b.statements, &ast.DropQueueStmt{ + Name: buildQualifiedName(names[0]), + }) } else if ctx.MODEL() != nil { b.statements = append(b.statements, &ast.DropModelStmt{ Name: buildQualifiedName(names[0]), diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 79170dfae..596cb5d6d 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -204,6 +204,16 @@ func (b *Builder) ExitShowStatement(ctx *parser.ShowStatementContext) { } } b.statements = append(b.statements, stmt) + } else if ctx.QUEUES() != nil { + stmt := &ast.ShowQueuesStmt{} + if ctx.IN() != nil { + if qn := ctx.QualifiedName(); qn != nil { + stmt.Module = getQualifiedNameText(qn) + } else if id := ctx.IDENTIFIER(); id != nil { + stmt.Module = id.GetText() + } + } + b.statements = append(b.statements, stmt) } else if ctx.LAYOUTS() != nil { stmt := &ast.ShowStmt{ObjectType: ast.ShowLayouts} if ctx.IN() != nil { @@ -694,6 +704,14 @@ func (b *Builder) ExitCatalogSelectQuery(ctx *parser.CatalogSelectQueryContext) // ExitDescribeStatement handles DESCRIBE ENTITY/ASSOCIATION/ENUMERATION/MODULE func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { + // DESCRIBE QUEUE Module.Name + if ctx.QUEUE() != nil { + if qn := ctx.QualifiedName(); qn != nil { + b.statements = append(b.statements, &ast.DescribeQueueStmt{Name: buildQualifiedName(qn)}) + } + return + } + // Handle DESCRIBE MODULE ROLE (uses qualifiedName) if ctx.MODULE() != nil && ctx.ROLE() != nil { if qn := ctx.QualifiedName(); qn != nil { diff --git a/mdl/visitor/visitor_queue.go b/mdl/visitor/visitor_queue.go new file mode 100644 index 000000000..8da08ffea --- /dev/null +++ b/mdl/visitor/visitor_queue.go @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// ExitCreateQueueStatement builds a CreateQueueStmt from +// CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( ... ). +func (b *Builder) ExitCreateQueueStatement(ctx *parser.CreateQueueStatementContext) { + stmt := &ast.CreateQueueStmt{ + Name: buildQualifiedName(ctx.QualifiedName()), + Documentation: findDocCommentText(ctx), + } + if createStmt := findParentCreateStatement(ctx); createStmt != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { + stmt.CreateOrModify = true + } + } + + if body := ctx.QueueBody(); body != nil { + bodyCtx := body.(*parser.QueueBodyContext) + for _, prop := range bodyCtx.AllQueueProperty() { + pc, ok := prop.(*parser.QueuePropertyContext) + if !ok || pc == nil { + continue + } + iok := pc.IdentifierOrKeyword(0) + if iok == nil { + continue + } + key := strings.ToLower(identifierOrKeywordText(iok)) + switch key { + case "parallelism": + stmt.Parallelism = queuePropertyText(pc) + case "clusterwide": + stmt.ClusterWide = strings.EqualFold(queuePropertyText(pc), "true") + case "exportlevel": + stmt.ExportLevel = queuePropertyText(pc) + case "documentation": + stmt.Documentation = queuePropertyText(pc) + } + } + } + + b.statements = append(b.statements, stmt) +} + +// queuePropertyText returns the value side of a queue property, unquoted. +// +// The value alternatives are NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | +// identifierOrKeyword. The key is identifierOrKeyword(0), so an identifier value +// is index 1 — reading index 0 would echo the key back as the value. +func queuePropertyText(pc *parser.QueuePropertyContext) string { + if n := pc.NUMBER_LITERAL(); n != nil { + return n.GetText() + } + if s := pc.STRING_LITERAL(); s != nil { + return unquoteString(s.GetText()) + } + if bl := pc.BooleanLiteral(); bl != nil { + return bl.GetText() + } + if v := pc.IdentifierOrKeyword(1); v != nil { + return identifierOrKeywordText(v) + } + return "" +} diff --git a/mdl/visitor/visitor_queue_test.go b/mdl/visitor/visitor_queue_test.go new file mode 100644 index 000000000..831d9bba1 --- /dev/null +++ b/mdl/visitor/visitor_queue_test.go @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func TestCreateQueue(t *testing.T) { + input := `CREATE QUEUE Ops.OrderProcessing ( + Parallelism: 3, + ClusterWide: true + );` + prog, errs := Build(input) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateQueueStmt) + if !ok { + t.Fatalf("expected CreateQueueStmt, got %T", prog.Statements[0]) + } + if stmt.Name.Module != "Ops" || stmt.Name.Name != "OrderProcessing" { + t.Errorf("Name = %+v", stmt.Name) + } + if stmt.Parallelism != "3" { + t.Errorf("Parallelism = %q, want 3", stmt.Parallelism) + } + if !stmt.ClusterWide { + t.Error("ClusterWide = false, want true") + } + if stmt.CreateOrModify { + t.Error("CreateOrModify set without OR MODIFY") + } +} + +// TestCreateQueue_ExpressionParallelism covers the reason Parallelism is a +// string all the way down: Mendix stores an expression, not a number. +func TestCreateQueue_ExpressionParallelism(t *testing.T) { + prog, errs := Build(`CREATE OR MODIFY QUEUE Ops.Q ( Parallelism: '$Config/Workers' );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateQueueStmt) + if stmt.Parallelism != "$Config/Workers" { + t.Errorf("Parallelism = %q, want the unquoted expression", stmt.Parallelism) + } + if !stmt.CreateOrModify { + t.Error("CreateOrModify not set for CREATE OR MODIFY") + } +} + +func TestCreateQueue_Defaults(t *testing.T) { + prog, errs := Build(`CREATE QUEUE Ops.Q ();`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateQueueStmt) + if stmt.Parallelism != "" { + t.Errorf("Parallelism = %q, want empty (the backend supplies the default)", stmt.Parallelism) + } + if stmt.ClusterWide { + t.Error("ClusterWide defaulted to true") + } +} + +func TestDropQueue(t *testing.T) { + prog, errs := Build(`DROP QUEUE Ops.OrderProcessing;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.DropQueueStmt) + if !ok { + t.Fatalf("expected DropQueueStmt, got %T", prog.Statements[0]) + } + if stmt.Name.String() != "Ops.OrderProcessing" { + t.Errorf("Name = %s", stmt.Name.String()) + } +} + +func TestShowAndDescribeQueues(t *testing.T) { + prog, errs := Build(`SHOW QUEUES; LIST QUEUES IN Ops; DESCRIBE QUEUE Ops.OrderProcessing;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 3 { + t.Fatalf("expected 3 statements, got %d", len(prog.Statements)) + } + if _, ok := prog.Statements[0].(*ast.ShowQueuesStmt); !ok { + t.Errorf("statement 0 = %T, want ShowQueuesStmt", prog.Statements[0]) + } + s1, ok := prog.Statements[1].(*ast.ShowQueuesStmt) + if !ok { + t.Fatalf("statement 1 = %T, want ShowQueuesStmt", prog.Statements[1]) + } + if s1.Module != "Ops" { + t.Errorf("Module = %q, want Ops", s1.Module) + } + s2, ok := prog.Statements[2].(*ast.DescribeQueueStmt) + if !ok { + t.Fatalf("statement 2 = %T, want DescribeQueueStmt", prog.Statements[2]) + } + if s2.Name.String() != "Ops.OrderProcessing" { + t.Errorf("Name = %s", s2.Name.String()) + } +} + +// TestQueueKeywordStillUsableAsIdentifier guards the cost of adding QUEUE to the +// lexer: a new keyword stops being usable as an ordinary name unless it is also +// listed in the `keyword` rule. "queue" is a plausible attribute name. +func TestQueueKeywordStillUsableAsIdentifier(t *testing.T) { + prog, errs := Build(`CREATE ENTITY Ops.Job ( queue: String(100), queues: String(100) );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateEntityStmt) + if !ok { + t.Fatalf("expected CreateEntityStmt, got %T", prog.Statements[0]) + } + if len(stmt.Attributes) != 2 { + t.Fatalf("expected 2 attributes, got %d", len(stmt.Attributes)) + } + if stmt.Attributes[0].Name != "queue" || stmt.Attributes[1].Name != "queues" { + t.Errorf("attribute names = %q, %q", stmt.Attributes[0].Name, stmt.Attributes[1].Name) + } +} diff --git a/sdk/mpr/queues.go b/sdk/mpr/queues.go new file mode 100644 index 000000000..907690da8 --- /dev/null +++ b/sdk/mpr/queues.go @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "fmt" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +// Task queues (Queues$Queue). The document is small and flat apart from a single +// nested Config node, so it is read and written as raw BSON here rather than +// through a dedicated element type. +// +// The shape follows four Studio Pro-authored queues from the Mendix Business +// Events module. Note that Queues$BasicQueueConfig declares an int32 +// `Parallelism` in addition to `ParallelismExpression`, and Studio Pro wrote it +// in none of them — so only the expression is read and written. + +const queueUnitType = "Queues$Queue" + +// ListQueues reads every task queue in the project. +func (r *Reader) ListQueues() ([]*types.Queue, error) { + units, err := r.ListRawUnitsByType(queueUnitType) + if err != nil { + return nil, err + } + out := make([]*types.Queue, 0, len(units)) + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + return nil, fmt.Errorf("unmarshal queue %s: %w", u.ID, err) + } + q := &types.Queue{ContainerID: model.ID(u.ContainerID)} + q.ID = model.ID(u.ID) + q.TypeName = queueUnitType + q.Name, _ = doc["Name"].(string) + q.Documentation, _ = doc["Documentation"].(string) + q.Excluded, _ = doc["Excluded"].(bool) + q.ExportLevel, _ = doc["ExportLevel"].(string) + if cfg, ok := doc["Config"].(bson.M); ok { + q.Parallelism, _ = cfg["ParallelismExpression"].(string) + q.ClusterWide, _ = cfg["ClusterWide"].(bool) + } + out = append(out, q) + } + return out, nil +} + +// CreateQueue inserts a new task queue document. +func (w *Writer) CreateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("CreateQueue: nil queue") + } + if q.ID == "" { + q.ID = model.ID(generateUUID()) + } + contents, err := serializeQueueUnit(q) + if err != nil { + return err + } + return w.insertUnit(string(q.ID), string(q.ContainerID), "Documents", queueUnitType, contents) +} + +// UpdateQueue rewrites an existing task queue in place. +func (w *Writer) UpdateQueue(q *types.Queue) error { + if q == nil { + return fmt.Errorf("UpdateQueue: nil queue") + } + contents, err := serializeQueueUnit(q) + if err != nil { + return err + } + return w.UpdateRawUnit(string(q.ID), contents) +} + +// DeleteQueue removes a task queue by ID. +func (w *Writer) DeleteQueue(id string) error { + return w.deleteUnit(id) +} + +func serializeQueueUnit(q *types.Queue) ([]byte, error) { + parallelism := q.Parallelism + if parallelism == "" { + parallelism = "1" + } + exportLevel := q.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + doc := bson.D{ + {Key: "$ID", Value: idToBsonBinary(string(q.ID))}, + {Key: "$Type", Value: queueUnitType}, + {Key: "Config", Value: bson.D{ + {Key: "$ID", Value: idToBsonBinary(generateUUID())}, + {Key: "$Type", Value: "Queues$BasicQueueConfig"}, + {Key: "ClusterWide", Value: q.ClusterWide}, + {Key: "ParallelismExpression", Value: parallelism}, + }}, + {Key: "Documentation", Value: q.Documentation}, + {Key: "Excluded", Value: q.Excluded}, + {Key: "ExportLevel", Value: exportLevel}, + {Key: "Name", Value: q.Name}, + } + return marshalUnitIDFirst(doc) +} From 7d4867820fd416aae95328c13b67818a3e704b09 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 23:08:25 +0000 Subject: [PATCH 5/7] feat(scheduled-events): author Mendix's cron in MDL, and read back what it stores MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scheduled events were read-only: ScheduledEventBackend had List/Get and nothing else, and there was no MDL surface at all. A project's cron had to be added in Studio Pro even though everything around it could be scripted. CREATE [OR MODIFY] / DROP / SHOW / DESCRIBE SCHEDULED EVENT now exist, wired through grammar → AST → visitor → executor → backend on both engines. The repeat rule is stored as one of eight ScheduledEvents$*Schedule types that differ in WHICH fields they carry, not just in their values. MDL mirrors that: `Repeat: Daily` names the variant and then only that variant's fields are accepted — a Multiplier on a Daily repeat is refused, by `mxcli check` (MDL-SCHED01) and by exec, which call the same function so they cannot drift. Merging the field sets is what produces a document mxbuild accepts and Studio Pro cannot open. The read was lossy in both engines and is fixed here too. The legacy parser never looked at Schedule at all, and the modelsdk reader went through modelsdk/gen, whose generated types disagree with what Studio Pro writes on two properties: every integer is stored as int64 where gen declares int32, and StartDateTime is a BSON datetime where gen declares a string. Both engines now share one codec (mdl/scheduledevents), pinned by re-serializing three whole Studio Pro-authored documents — from Workflow Commons 4.11.0, OIDC SSO 4.6.0 and SAML 4.2.1 — element by element, in order, with matching BSON types. Those four references cover the Daily and Hourly variants; the other six are derived from the metamodel and verified to load with 0 errors from `mx check`. Interval/IntervalType are legacy siblings of Schedule that Studio Pro writes and does not keep in sync (one shipped module stores 0/"Minute" beside a daily schedule of 01:00). MDL has no syntax for them: a new event gets the pair matching its repeat — an empty IntervalType is not valid, the enumeration has no such member — and OR MODIFY carries the stored pair through untouched. Verified on Mendix 11.13 under both engines: all eight variants leave a clean project at 0 errors, DESCRIBE output re-parses and re-validates for every variant, and a re-run writes nothing (the MXCLI_ALWAYS_WRITE=1 control changes 9 units where the normal run changes 0). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/skills/fix-issue.md | 1 + cmd/mxcli/cmd_check.go | 6 + cmd/mxcli/lsp_completions_gen.go | 1 + cmd/mxcli/syntax/features_misc.go | 83 +++ docs-site/src/SUMMARY.md | 1 + docs-site/src/language/scheduled-events.md | 185 ++++++ docs/01-project/MDL_QUICK_REFERENCE.md | 52 ++ .../scheduled-event-repeat-fields.fail.mdl | 25 + .../doctype-tests/scheduled-events.mdl | 133 ++++ mdl/ast/ast_scheduledevent.go | 75 +++ mdl/backend/infrastructure.go | 3 + mdl/backend/mcp/unsupported_gen.go | 15 + mdl/backend/mock/backend.go | 7 +- mdl/backend/mock/mock_workflow.go | 25 + mdl/backend/modelsdk/scheduledevent_read.go | 90 ++- .../modelsdk/scheduledevent_read_test.go | 129 ++-- mdl/backend/modelsdk/unimplemented_gen.go | 12 + mdl/backend/mpr/backend.go | 9 + mdl/executor/cmd_queues_mock_test.go | 18 +- mdl/executor/cmd_scheduledevents.go | 586 ++++++++++++++++++ mdl/executor/cmd_scheduledevents_test.go | 456 ++++++++++++++ mdl/executor/register_stubs.go | 15 + mdl/executor/registry.go | 1 + mdl/executor/registry_test.go | 4 + mdl/executor/validate_scheduled_events.go | 41 ++ mdl/grammar/MDLLexer.g4 | 1 + mdl/grammar/MDLParser.g4 | 2 + mdl/grammar/domains/MDLCatalog.g4 | 2 + mdl/grammar/domains/MDLDomainModel.g4 | 22 + mdl/grammar/domains/MDLSettings.g4 | 2 +- mdl/scheduledevents/codec.go | 295 +++++++++ mdl/scheduledevents/codec_test.go | 314 ++++++++++ mdl/visitor/visitor_entity.go | 4 + mdl/visitor/visitor_query.go | 18 + mdl/visitor/visitor_queue.go | 2 +- mdl/visitor/visitor_scheduledevent.go | 133 ++++ mdl/visitor/visitor_scheduledevent_test.go | 150 +++++ model/types.go | 66 ++ sdk/mpr/parser_enumeration.go | 36 +- sdk/mpr/scheduledevents.go | 52 ++ 40 files changed, 2952 insertions(+), 120 deletions(-) create mode 100644 docs-site/src/language/scheduled-events.md create mode 100644 mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl create mode 100644 mdl-examples/doctype-tests/scheduled-events.mdl create mode 100644 mdl/ast/ast_scheduledevent.go create mode 100644 mdl/executor/cmd_scheduledevents.go create mode 100644 mdl/executor/cmd_scheduledevents_test.go create mode 100644 mdl/executor/validate_scheduled_events.go create mode 100644 mdl/scheduledevents/codec.go create mode 100644 mdl/scheduledevents/codec_test.go create mode 100644 mdl/visitor/visitor_scheduledevent.go create mode 100644 mdl/visitor/visitor_scheduledevent_test.go create mode 100644 sdk/mpr/scheduledevents.go diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index dd3c1dd49..c5c287cda 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -473,3 +473,4 @@ extracting `OffsetExpression`/`LimitExpression`. | An association's line anchors — where the connector attaches to the entity boxes in the domain model editor — are absent from `DESCRIBE ASSOCIATION`, and manual adjustments made in Studio Pro do not survive an mxcli round trip | `DomainModels$Association.ParentConnection`/`ChildConnection` (the string `"x;y"`) were **hardcoded** to `"0;50"`/`"100;50"` in BOTH writers and never read by either parser. Because every association write rebuilds the whole element, this was not an omission but active destruction: a documentation-only `alter association … set comment` reset them | `sdk/domainmodel/connection.go` (new: `ParseConnectionPoint`/`FormatConnectionPoint`, `Default*Connection`), `sdk/domainmodel/domainmodel.go` (fields → `*model.Point`), `sdk/mpr/parser_domainmodel.go` + `sdk/mpr/writer_domainmodel.go`, `mdl/backend/modelsdk/domainmodel.go` + `domainmodel_write.go`, `mdl/executor/cmd_associations.go` (`describeConnectionPoints`) | **A feature request that says "X is not exposed" may be hiding "X is destroyed"** — check the write path before scoping the read path. The A/B that settled it: a blank 11.13 app's own `Administration.AccountPasswordData_Account` stores `0;54/100;54`, so a Studio-Pro-authored association is a free fixture for "did mxcli overwrite this?" — no Studio Pro needed. **Learn the value's constraints from the LOADER, not from the shape**: hand-patch and run `mx check` — `"0.5;50"` dies with `StorageLoadException` (integers required) while `"0;500"` and `"-20;50"` load with 0 errors (no range check), so out-of-range values must round-trip untouched. **A zero value is not an absent value** — `{0,0}` is a real anchor (top-left), which forces the field to be a POINTER; a plain `model.Point` cannot distinguish "unset" from "top-left" and would silently rewrite it. **Fix both engines**: they share the semantic model, and a fix in one is invisible to a user on the other. **Emit unauthorable data as a COMMENT** — DESCRIBE output must stay re-executable, and inventing syntax (`@anchor(parent: bottom-left, …)`) would bake in a vocabulary the storage does not have: the pair is CONTINUOUS, not 8 named anchors (observed x values 0 9 11 17 18 47 49 50 65 77 78 84 87 100). **The marketplace is the sample** when you need to know what Studio Pro actually writes: `mxcli marketplace download ` gives real Mendix-authored models, and a module .mpk holds either a raw BSON `project.mpr` or an MPR v1 SQLite one — 88 coordinate pairs from three modules turned "looks like percentages" into a measurement (all 0..100; 85 of 88 pin one coordinate to exactly 0 or 100). **Rule a unit out from the model, not the values**: pixels is impossible because `DomainModels$EntityImpl` stores only `Location` and NO size — the box is sized by the editor from the name and attribute list, so a pixel anchor would have nothing to measure against. Not applicable to `CrossAssociation`, which has no connection properties and crashes Studio Pro if given them (#50). Tests `sdk/domainmodel/connection_test.go`, `mdl/backend/modelsdk/association_connection_test.go`, `sdk/mpr/writer_domainmodel_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | A microflow call bound to a task queue in Studio Pro loses that binding on `create or replace microflow` — silently, and `mx check` goes from `[CE1613] "The selected task queue no longer exists"` to **0 errors**, because the configuration the error was about has been deleted | Two defects behind one report. (1) `Microflows$MicroflowCall` has `QueueSettings` in `NullFields` of `codec.RegisterTypeDefaults` (and the legacy writer hardcodes null too) — correct for a newly authored call, destructive for a stored one, and since a CREATE OR REPLACE rebuilds the whole microflow it fires on every rewrite. (2) MDL had **no queue surface at all**, so a script could not restate the binding even in principle | `mdl/executor/validate_queued_calls.go` (new: the refusal), `mdl/executor/cmd_microflows_create.go` (call site), `mdl/types/queue.go` + `mdl/backend/infrastructure.go` (`QueueBackend`), `mdl/backend/modelsdk/queue_write.go` + `sdk/mpr/queues.go` (both engines), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4`, `mdl/executor/cmd_queues.go` | **A green build can be the bug**: the fix made `mx check` report MORE errors than before (CE1613 came back), because the binding it complains about now survives. Any "error count went down" check would have scored the data loss as a fix. **`describe` showing nothing is not evidence of nothing** — the binding was invisible from every angle (describe omitted it, check went quiet, the write reported success), which is why it needed a stored-BSON probe: inject `Queue`+`QueueSettings` into a stored call, rewrite, and dump. **Isolate which property Mendix acts on before guarding on it**: a call carrying only `Queue` (with `QueueSettings` null) draws NO complaint from `mx check` — `QueueSettings` is the load-bearing one, so a guard keyed on `Queue` alone would have both missed the real case and fired on inert ones. **Get the BSON shape from a Mendix-authored model, not from the metamodel**: `mxcli marketplace download 202649` (Business Events 3.12.1) has four real queues, and all four agree — `Config` is a `Queues$BasicQueueConfig` whose `ParallelismExpression` is a **STRING**, with the sibling int32 `Parallelism` absent in every one. Writing the int (which the metamodel makes look equally valid) is inventing a property Mendix does not write. **Refuse, do not half-author** (ADR-0005): `in queue` on a call is deliberately still unimplemented, because the `Retry` shape has no Studio-Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. **Adding a keyword costs an identifier** — `QUEUE`/`QUEUES` had to go into the `keyword` rule or `queue` would have stopped working as an attribute name. Tests `mdl/backend/modelsdk/queue_write_test.go`, `mdl/executor/validate_queued_calls_test.go`, `mdl/executor/cmd_queues_mock_test.go`, `mdl/visitor/visitor_queue_test.go`, example `mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl` | +| Scheduled events are read-only in MDL — a project's cron cannot be scripted at all, so `mxcli new` + a script leaves you opening Studio Pro to add the one thing that makes a batch job run. Separately, `describe` of an existing one omits its schedule entirely | Two gaps behind one request. (1) No write path: `ScheduledEventBackend` had List/Get and nothing else, and there was no grammar/AST/visitor/executor. (2) The READ was lossy in both engines — the legacy `parseScheduledEvent` never looked at `Schedule`, and the modelsdk reader went through `modelsdk/gen`, whose generated types **disagree with what Studio Pro writes** on two properties | `mdl/scheduledevents/codec.go` (new, shared by both engines), `mdl/backend/modelsdk/scheduledevent_read.go` + `sdk/mpr/scheduledevents.go` + `sdk/mpr/parser_enumeration.go` (both engines through the one codec), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4` + `MDLParser.g4`, `mdl/ast/ast_scheduledevent.go`, `mdl/visitor/visitor_scheduledevent.go`, `mdl/executor/cmd_scheduledevents.go`, `mdl/executor/validate_scheduled_events.go` (MDL-SCHED01) | **The generated metamodel is not the storage's ground truth** — `modelsdk/gen` declares `Interval`/`HourOfDay`/… as **int32** where Studio Pro writes **int64**, and `StartDateTime` as a **string** where Studio Pro writes a **BSON datetime**. This is the same mismatch as #585 (a reader that asserted int32), so the pattern is now twice-confirmed: when gen and an observed document disagree about a numeric width or a date, the document is right. **A byte-level pin beats field assertions**: three whole documents are hex-embedded in `codec_test.go` and re-serialized element by element in ORDER, so key order, key SET and BSON TYPE are all covered — a field-by-field test passes happily while writing int32 and an empty enum. **`mxcli marketplace download` is the Studio Pro substitute**: Business Events and Community Commons have none, but Workflow Commons 4.11.0, OIDC SSO 4.6.0 and SAML 4.2.1 have four between them (only Day and Hour variants — the other six had to be metamodel-derived, then verified to load). **An empty string is not "unset" for an enum property**: the first draft wrote `IntervalType: ""` and `mx check` reported 0 errors, but the enumeration has no such member. **`Interval`/`IntervalType` are LEGACY siblings of `Schedule` that Studio Pro does not keep in sync** — Workflow Commons stores `0`/`Minute` beside a `DaySchedule` of 01:00 — so they are derived on CREATE and carried through untouched on MODIFY, never re-derived. **A polymorphic child must be dispatched on `$Type`, and its field sets never merged** (the ADR-0005 rule): the eight variants differ in arity, so the executor refuses a field belonging to another Repeat rather than dropping it. **Validate at check time, not only at exec** — the validation is decidable from the statement, so it runs in the no-project pass and `check` and `exec` call the SAME function. **Prove the describe round-trips by PARSING its own output** for every variant, not by asserting on strings. Idempotence came free (ADR-0008 canon elides the regenerated `Schedule` `$ID`) — verified with the `MXCLI_ALWAYS_WRITE=1` control, which changes 9 units where the normal run changes 0. Tests `mdl/scheduledevents/codec_test.go`, `mdl/executor/cmd_scheduledevents_test.go`, `mdl/visitor/visitor_scheduledevent_test.go`, `mdl/backend/modelsdk/scheduledevent_read_test.go`, example `mdl-examples/doctype-tests/scheduled-events.mdl` | diff --git a/cmd/mxcli/cmd_check.go b/cmd/mxcli/cmd_check.go index f76861a97..d42da4257 100644 --- a/cmd/mxcli/cmd_check.go +++ b/cmd/mxcli/cmd_check.go @@ -216,6 +216,12 @@ Examples: // operation, so the mapping would be dropped in silence (#843). violations = append(violations, executor.ValidateRestClientMappings(prog)...) + // Flag a scheduled event whose Repeat and fields disagree (a Multiplier on + // a Daily repeat, an HourOfDay of 99). Decidable from the statement, so it + // runs here rather than at exec, where the script would already have + // passed check. + violations = append(violations, executor.ValidateScheduledEvents(prog)...) + if isStructured { // Always emit structured output (even when clean) formatter.Format(violations, os.Stderr) diff --git a/cmd/mxcli/lsp_completions_gen.go b/cmd/mxcli/lsp_completions_gen.go index 7016b7d5d..79749cd18 100644 --- a/cmd/mxcli/lsp_completions_gen.go +++ b/cmd/mxcli/lsp_completions_gen.go @@ -268,6 +268,7 @@ var mdlGeneratedKeywords = []protocol.CompletionItem{ {Label: "IMAGE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "QUEUE", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "QUEUES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, + {Label: "SCHEDULED", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "JAR", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCY", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, {Label: "DEPENDENCIES", Kind: protocol.CompletionItemKindKeyword, Detail: "Widget keyword"}, diff --git a/cmd/mxcli/syntax/features_misc.go b/cmd/mxcli/syntax/features_misc.go index 191cbcfac..088aeac6f 100644 --- a/cmd/mxcli/syntax/features_misc.go +++ b/cmd/mxcli/syntax/features_misc.go @@ -253,6 +253,89 @@ DESCRIBE QUEUE Ops.OrderProcessing; DROP QUEUE Ops.Mail;`, }) + // ── Scheduled events ──────────────────────────────────────────────── + + Register(SyntaxFeature{ + Path: "scheduled-event", + Summary: "Scheduled events — Mendix's cron: run a microflow on a repeating schedule", + Keywords: []string{ + "scheduled event", "scheduled events", "schedule", "cron", "recurring", + "create scheduled event", "drop scheduled event", "describe scheduled event", + "repeat", "daily", "hourly", "weekly", "monthly", "yearly", "timer", "batch job", + }, + Syntax: `CREATE [OR MODIFY] SCHEDULED EVENT Module.Name ( : , ... ); +SHOW SCHEDULED EVENTS [IN ]; +LIST SCHEDULED EVENTS [IN ]; +DESCRIBE SCHEDULED EVENT Module.Name; +DROP SCHEDULED EVENT Module.Name; + +Always required: + Microflow the microflow to run, as a qualified name + Repeat which schedule to use (below) + +Each Repeat takes ONLY its own fields; anything else is refused: + Minutely Multiplier + Hourly Multiplier, MinuteOffset + Daily HourOfDay, MinuteOfHour + Weekly Weekdays, HourOfDay, MinuteOfHour + MonthlyByDate Multiplier, MonthOffset, DayOfMonth, HourOfDay, MinuteOfHour + MonthlyByWeekday Multiplier, MonthOffset, DaySelector, Weekday, HourOfDay, MinuteOfHour + YearlyByDate Month, DayOfMonth, HourOfDay, MinuteOfHour + YearlyByWeekday Month, DaySelector, Weekday, HourOfDay, MinuteOfHour + + Weekdays is a quoted list: 'Monday, Friday'. DaySelector is First, Second, + Third, Fourth or Last. Weekday is Sunday..Saturday. Month and DayOfMonth are + numbers (1-12, 1-31). MonthOffset picks which month of a multi-month cycle + fires (0-based). + +Optional on any repeat: + Enabled true or false (default false) + OnOverlap DelayNext (default) or SkipNext — what happens when a run is + still going when the next one is due. This is a scheduled + event's own concurrency control; it does not use a task queue. + TimeZone UTC (default) or Server + StartDateTime an RFC 3339 timestamp; the event does not run before it + Documentation free text`, + Example: `CREATE SCHEDULED EVENT Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +CREATE SCHEDULED EVENT Ops.HourlyPing ( + Microflow: Ops.SE_Ping, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); + +CREATE SCHEDULED EVENT Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); + +CREATE SCHEDULED EVENT Ops.QuarterEnd ( + Microflow: Ops.SE_Close, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18 +); + +SHOW SCHEDULED EVENTS IN Ops; +DESCRIBE SCHEDULED EVENT Ops.NightlyCleanup; +DROP SCHEDULED EVENT Ops.HourlyPing;`, + SeeAlso: []string{"queue"}, + }) + // ── Structure ─────────────────────────────────────────────────────── Register(SyntaxFeature{ diff --git a/docs-site/src/SUMMARY.md b/docs-site/src/SUMMARY.md index 0ea07706e..30656cbd1 100644 --- a/docs-site/src/SUMMARY.md +++ b/docs-site/src/SUMMARY.md @@ -111,6 +111,7 @@ - [Business Events](language/business-events.md) - [Event Services](language/event-services.md) - [Publishing and Consuming Events](language/pub-sub-events.md) +- [Scheduled Events and Task Queues](language/scheduled-events.md) - [Image Collections](language/image-collections.md) --- diff --git a/docs-site/src/language/scheduled-events.md b/docs-site/src/language/scheduled-events.md new file mode 100644 index 000000000..f6da79561 --- /dev/null +++ b/docs-site/src/language/scheduled-events.md @@ -0,0 +1,185 @@ +# Scheduled Events and Task Queues + +Two Mendix features for work that runs outside a user request. They are unrelated +and easy to confuse: + +- A **scheduled event** is Mendix's cron — it runs a microflow on a repeating + schedule. +- A **task queue** bounds how many queued microflow calls run at once. + +A scheduled event does **not** go through a task queue. Its own concurrency +control is `OnOverlap`, which decides what happens when a run is still going when +the next one is due. + +## Scheduled Events + +### Inspecting + +```sql +-- All scheduled events, or one module's +LIST SCHEDULED EVENTS; +LIST SCHEDULED EVENTS IN Ops; + +-- Re-executable MDL for one event +DESCRIBE SCHEDULED EVENT Ops.NightlyCleanup; +``` + +`SHOW` is accepted as a synonym for `LIST`. + +### CREATE SCHEDULED EVENT + +```sql +CREATE [OR MODIFY] SCHEDULED EVENT . ( + Microflow: ., + Repeat: , + , + +); +``` + +`Microflow` and `Repeat` are always required. + +#### Repeat variants + +The repeat rule is stored as one of eight Mendix schedule types, and they differ +in **which fields they carry** — not just in their values. MDL mirrors that: each +repeat takes only its own fields, and a field belonging to another repeat is +refused by both `mxcli check` (rule `MDL-SCHED01`) and `mxcli exec` rather than +silently dropped. + +| Repeat | Fields | Reads as | +|--------|--------|----------| +| `Minutely` | `Multiplier` | every N minutes | +| `Hourly` | `Multiplier`, `MinuteOffset` | every N hours, at :MM | +| `Daily` | `HourOfDay`, `MinuteOfHour` | every day at HH:MM | +| `Weekly` | `Weekdays`, `HourOfDay`, `MinuteOfHour` | on the named days at HH:MM | +| `MonthlyByDate` | `Multiplier`, `MonthOffset`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | the Dth of every N months | +| `MonthlyByWeekday` | `Multiplier`, `MonthOffset`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the last Friday of every N months | +| `YearlyByDate` | `Month`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | every 2 January | +| `YearlyByWeekday` | `Month`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the first Monday of March | + +Field values: + +| Field | Value | +|-------|-------| +| `Multiplier` | how many units between runs (1 or more; defaults to 1) | +| `MinuteOffset` | 0–59, the minute past the hour | +| `MonthOffset` | 0-based, which month of a multi-month cycle fires | +| `HourOfDay` / `MinuteOfHour` | 0–23 / 0–59 | +| `DayOfMonth` / `Month` | 1–31 / 1–12 | +| `Weekdays` | a quoted list, e.g. `'Monday, Friday'` (case-insensitive) | +| `DaySelector` | `First`, `Second`, `Third`, `Fourth`, `Last` | +| `Weekday` | `Sunday` … `Saturday` | + +A value outside its range is an error, not a truncation: a schedule that is +stored but can never fire is worse than a refusal, because nothing downstream +reports it. + +#### Optional properties + +| Property | Values | Default | +|----------|--------|---------| +| `Enabled` | `true` / `false` | `false` | +| `OnOverlap` | `DelayNext` / `SkipNext` | `DelayNext` | +| `TimeZone` | `UTC` / `Server` | `UTC` | +| `StartDateTime` | an RFC 3339 timestamp; the event does not run before it | none | +| `Documentation` | free text | none | + +`SkipNext` drops a run that would overlap the previous one; `DelayNext` queues it +until the previous one finishes. + +### Examples + +```sql +-- Every night at 04:00 in the server's timezone +CREATE SCHEDULED EVENT Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +-- Every two hours, 23 minutes past +CREATE SCHEDULED EVENT Ops.HourlyPing ( + Microflow: Ops.SE_Ping, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); + +-- Mondays and Fridays at 09:30 +CREATE SCHEDULED EVENT Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); + +-- The last Friday of every third month, at 18:00 +CREATE SCHEDULED EVENT Ops.QuarterEnd ( + Microflow: Ops.SE_Close, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18 +); + +DROP SCHEDULED EVENT Ops.HourlyPing; +``` + +### A note on `Interval` / `IntervalType` + +Stored events also carry an `Interval` and `IntervalType` pair. These predate the +`Schedule` child and Studio Pro does **not** keep them in sync with it — a real +Mendix module ships an event storing `0` / `Minute` next to a daily schedule of +01:00. MDL has no syntax for them: a new event gets the pair that matches its +repeat, and `CREATE OR MODIFY` carries whatever is stored through untouched. +`DESCRIBE` reports them as a comment so the output stays re-executable. + +## Task Queues + +A task queue bounds how many instances of a queued microflow call run at once. + +```sql +CREATE [OR MODIFY] QUEUE . [( + Parallelism: , + ClusterWide: true|false, + Documentation: '' +)]; + +LIST QUEUES [IN ]; +DESCRIBE QUEUE .; +DROP QUEUE .; +``` + +| Property | Meaning | Default | +|----------|---------|---------| +| `Parallelism` | how many tasks run at once — an **expression**, not a number | `1` | +| `ClusterWide` | `true` applies the limit across the cluster; `false` per runtime instance | `false` | + +Mendix stores parallelism as an expression string, so a bare integer and a quoted +one mean the same thing and an arbitrary expression is legal: + +```sql +CREATE QUEUE Ops.OrderProcessing ( Parallelism: 3, ClusterWide: true ); +CREATE QUEUE Ops.Mail; -- defaults: 1, per-instance +CREATE OR MODIFY QUEUE Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); +``` + +### Binding a call to a queue is not yet expressible + +MDL cannot yet author a *queued call* — the binding lives on the call activity +inside a microflow, not on the queue. Because rebuilding a microflow would drop +an existing binding, `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the +stored microflow has a queued call, naming the queues that would be lost. Change +those microflows in Studio Pro. + +Without that refusal the binding was written back as null and the project then +looked *healthier* than before — `mx check` stopped reporting +`CE1613 "The selected task queue no longer exists"`, because the configuration +the error was about had been deleted. diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index 1aded5443..08d2da8b0 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -166,6 +166,58 @@ create or modify queue Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); drop queue Ops.Mail; ``` +## Scheduled Events + +Mendix's cron: run a microflow on a repeating schedule. + +| Statement | Syntax | Notes | +|-----------|--------|-------| +| Show scheduled events | `show scheduled events [in module];` (`list` too) | Repeat, microflow, enabled | +| Describe scheduled event | `describe scheduled event Module.Name;` | Re-executable MDL | +| Create scheduled event | `create [or modify] scheduled event Module.Name ( Microflow: ..., Repeat: ..., ... );` | | +| Drop scheduled event | `drop scheduled event Module.Name;` | | + +`Microflow` and `Repeat` are always required. Each repeat takes **only** its own +fields — anything else is refused by `mxcli check` (MDL-SCHED01) and by `exec`: + +| Repeat | Fields | +|--------|--------| +| `Minutely` | `Multiplier` | +| `Hourly` | `Multiplier`, `MinuteOffset` | +| `Daily` | `HourOfDay`, `MinuteOfHour` | +| `Weekly` | `Weekdays`, `HourOfDay`, `MinuteOfHour` | +| `MonthlyByDate` | `Multiplier`, `MonthOffset`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | +| `MonthlyByWeekday` | `Multiplier`, `MonthOffset`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | +| `YearlyByDate` | `Month`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | +| `YearlyByWeekday` | `Month`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | + +Optional on any repeat: `Enabled` (default false), `OnOverlap` +(`DelayNext` default / `SkipNext`), `TimeZone` (`UTC` default / `Server`), +`StartDateTime` (RFC 3339), `Documentation`. + +`OnOverlap` is a scheduled event's own concurrency control — scheduled events do +**not** go through a task queue. + +**Example:** +```sql +create scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +create scheduled event Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); +``` + ## OData Clients, Services & External Entities | Statement | Syntax | Notes | diff --git a/mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl b/mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl new file mode 100644 index 000000000..d1c869094 --- /dev/null +++ b/mdl-examples/bug-tests/scheduled-event-repeat-fields.fail.mdl @@ -0,0 +1,25 @@ +-- Scheduled event: a field from the wrong Repeat must be rejected by `check`. +-- +-- NEGATIVE TEST (.fail.mdl) — EXPECTED to fail `mxcli check`. +-- `make check-mdl` inverts the exit code for .fail.mdl files: an unexpected +-- pass would be reported as a regression of the MDL-SCHED01 rule. +-- +-- The eight ScheduledEvents$*Schedule variants differ in WHICH fields they +-- carry, not just in their values. A DaySchedule has no Multiplier, so writing +-- one would put a property on the node that the type does not declare — the +-- shape that mxbuild accepts and Studio Pro cannot open +-- (System.InvalidOperationException at MprProperty). +-- +-- The rule runs in the no-project pass, so a plain `mxcli check` catches it — +-- it is decidable from the statement alone, and exec calls the same function, +-- so check and exec cannot drift. +-- +-- Correct form: drop Multiplier (Daily takes HourOfDay and MinuteOfHour), or +-- switch to a Repeat that has one. + +create scheduled event Ops.Bad ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + Multiplier: 3, + HourOfDay: 4 +); diff --git a/mdl-examples/doctype-tests/scheduled-events.mdl b/mdl-examples/doctype-tests/scheduled-events.mdl new file mode 100644 index 000000000..b4d3ed555 --- /dev/null +++ b/mdl-examples/doctype-tests/scheduled-events.mdl @@ -0,0 +1,133 @@ +-- ============================================================================ +-- Scheduled events — all eight repeat variants +-- ============================================================================ +-- +-- Mendix's cron. A scheduled event runs a microflow on a repeating schedule; +-- the repeat rule is a ScheduledEvents$Schedule child with eight variants that +-- differ in WHICH fields they carry, so MDL names the variant (Repeat) and then +-- takes only that variant's fields. A field from another variant is refused — +-- writing a merged field set produces a document mxbuild accepts and Studio Pro +-- cannot open. +-- +-- Scheduled events do NOT go through a task queue: OnOverlap +-- (DelayNext | SkipNext) is their own concurrency control. +-- +-- The document shape is pinned against four Studio Pro-authored events (Workflow +-- Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1 ×2). Those cover Daily and Hourly; +-- the other six variants are metamodel-derived and verified to load with 0 +-- errors from `mx check`. +-- +-- Verify: +-- mxcli exec mdl-examples/doctype-tests/scheduled-events.mdl -p app.mpr +-- mx check app.mpr # 0 errors +-- mxcli -p app.mpr -c "list scheduled events" +-- mxcli -p app.mpr -c "describe scheduled event Ops.NightlyCleanup" +-- ============================================================================ + +create module Ops; +/ +create microflow Ops.SE_Cleanup() +begin + return; +end; +/ +-- Daily: a fixed time of day, no multiplier (the variant has none). +create scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true, + StartDateTime: '2026-01-01T04:00:00Z' +); +/ +-- Hourly: every N hours, at a fixed minute past the hour. +create scheduled event Ops.HourlyPing ( + Microflow: Ops.SE_Cleanup, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); +/ +-- Minutely: the only field is the multiplier. +create scheduled event Ops.EveryFiveMinutes ( + Microflow: Ops.SE_Cleanup, + Repeat: Minutely, + Multiplier: 5 +); +/ +-- Weekly: a day list plus a time. Weekday names are case-insensitive. +create scheduled event Ops.WeeklyReport ( + Microflow: Ops.SE_Cleanup, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); +/ +-- MonthlyByDate: the Nth day of every M months. +create scheduled event Ops.MidMonth ( + Microflow: Ops.SE_Cleanup, + Repeat: MonthlyByDate, + Multiplier: 1, + DayOfMonth: 15, + HourOfDay: 2, + MinuteOfHour: 0 +); +/ +-- MonthlyByWeekday: "the last Friday of every third month". MonthOffset picks +-- which month of the three-month cycle fires (0-based). +create scheduled event Ops.QuarterEnd ( + Microflow: Ops.SE_Cleanup, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18, + MinuteOfHour: 0 +); +/ +-- YearlyByDate: a calendar date. Month is 1-12. +create scheduled event Ops.YearOpen ( + Microflow: Ops.SE_Cleanup, + Repeat: YearlyByDate, + Month: 1, + DayOfMonth: 2, + HourOfDay: 8, + MinuteOfHour: 0 +); +/ +-- YearlyByWeekday: "the first Monday of March". +create scheduled event Ops.YearWeek ( + Microflow: Ops.SE_Cleanup, + Repeat: YearlyByWeekday, + Month: 3, + DaySelector: First, + Weekday: Monday, + HourOfDay: 7, + MinuteOfHour: 15 +); +/ +-- Re-running with OR MODIFY reuses the stored $ID and preserves the legacy +-- Interval/IntervalType pair, which Studio Pro writes but does not keep in sync +-- with Schedule and MDL cannot author. +create or modify scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 5, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true, + OnOverlap: SkipNext +); +/ +-- "scheduled" and "event" stay usable as ordinary names: both are in the +-- grammar's `keyword` rule, so adding the tokens did not take them away. +create entity Ops.Job ( + scheduled: Boolean, + event: String(50) +); +/ +drop scheduled event Ops.EveryFiveMinutes; diff --git a/mdl/ast/ast_scheduledevent.go b/mdl/ast/ast_scheduledevent.go new file mode 100644 index 000000000..7e97bf0ca --- /dev/null +++ b/mdl/ast/ast_scheduledevent.go @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: Apache-2.0 + +package ast + +// CreateScheduledEventStmt represents: +// +// CREATE [OR REPLACE|MODIFY] SCHEDULED EVENT Module.Name ( +// Microflow: Module.MF, Repeat: Daily, HourOfDay: 4, ... +// ); +// +// Properties are kept as written and validated by the executor, which knows +// which fields the chosen Repeat actually has. Numeric fields are pointers so +// "not mentioned" stays distinguishable from "mentioned as 0" — 0 is a real +// hour, minute and month offset. +type CreateScheduledEventStmt struct { + Name QualifiedName + Documentation string + // Microflow is the qualified name of the microflow to run. + Microflow string + // Repeat names the schedule variant: Minutely, Hourly, Daily, Weekly, + // MonthlyByDate, MonthlyByWeekday, YearlyByDate, YearlyByWeekday. + Repeat string + + Multiplier *int + MinuteOffset *int + MonthOffset *int + HourOfDay *int + MinuteOfHour *int + DayOfMonth *int + Month *int + + // Weekdays is the day list of a Weekly repeat, as written + // ("Monday, Friday"); DaySelector/Weekday are the two fields of the + // ByWeekday repeats. + Weekdays string + DaySelector string + Weekday string + + // StartDateTime is an RFC 3339 timestamp; TimeZone is UTC or Server. + StartDateTime string + TimeZone string + // OnOverlap is DelayNext or SkipNext — what happens when a run is still + // going when the next one is due. + OnOverlap string + + Enabled *bool + Excluded *bool + ExportLevel string + + // CreateOrModify is set by the shared CREATE OR REPLACE/MODIFY prefix. + CreateOrModify bool +} + +func (s *CreateScheduledEventStmt) isStatement() {} + +// DropScheduledEventStmt represents: DROP SCHEDULED EVENT Module.Name; +type DropScheduledEventStmt struct { + Name QualifiedName +} + +func (s *DropScheduledEventStmt) isStatement() {} + +// ShowScheduledEventsStmt represents: SHOW|LIST SCHEDULED EVENTS [IN Module]; +type ShowScheduledEventsStmt struct { + Module string +} + +func (s *ShowScheduledEventsStmt) isStatement() {} + +// DescribeScheduledEventStmt represents: DESCRIBE SCHEDULED EVENT Module.Name; +type DescribeScheduledEventStmt struct { + Name QualifiedName +} + +func (s *DescribeScheduledEventStmt) isStatement() {} diff --git a/mdl/backend/infrastructure.go b/mdl/backend/infrastructure.go index 9f3f822ca..ff88a0ddb 100644 --- a/mdl/backend/infrastructure.go +++ b/mdl/backend/infrastructure.go @@ -96,4 +96,7 @@ type QueueBackend interface { type ScheduledEventBackend interface { ListScheduledEvents() ([]*model.ScheduledEvent, error) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, error) + CreateScheduledEvent(ev *model.ScheduledEvent) error + UpdateScheduledEvent(ev *model.ScheduledEvent) error + DeleteScheduledEvent(id string) error } diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index 5fbba52d6..d39502993 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -219,6 +219,11 @@ func (unsupportedBackend) CreateQueue(_ *types.Queue) (err0 error) { return } +func (unsupportedBackend) CreateScheduledEvent(_ *model.ScheduledEvent) (err0 error) { + err0 = errUnsupported("CreateScheduledEvent") + return +} + func (unsupportedBackend) CreateSnippet(_ *pages.Snippet) (err0 error) { err0 = errUnsupported("CreateSnippet") return @@ -399,6 +404,11 @@ func (unsupportedBackend) DeleteQueue(_ string) (err0 error) { return } +func (unsupportedBackend) DeleteScheduledEvent(_ string) (err0 error) { + err0 = errUnsupported("DeleteScheduledEvent") + return +} + func (unsupportedBackend) DeleteSnippet(_ model.ID) (err0 error) { err0 = errUnsupported("DeleteSnippet") return @@ -1207,6 +1217,11 @@ func (unsupportedBackend) UpdateRawUnit(_ string, _ []uint8) (err0 error) { return } +func (unsupportedBackend) UpdateScheduledEvent(_ *model.ScheduledEvent) (err0 error) { + err0 = errUnsupported("UpdateScheduledEvent") + return +} + func (unsupportedBackend) UpdateSnippet(_ *pages.Snippet) (err0 error) { err0 = errUnsupported("UpdateSnippet") return diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index 3af35404e..bf72ffb19 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -257,8 +257,11 @@ type MockBackend struct { DeleteQueueFunc func(id string) error // ScheduledEventBackend - ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) - GetScheduledEventFunc func(id model.ID) (*model.ScheduledEvent, error) + ListScheduledEventsFunc func() ([]*model.ScheduledEvent, error) + GetScheduledEventFunc func(id model.ID) (*model.ScheduledEvent, error) + CreateScheduledEventFunc func(ev *model.ScheduledEvent) error + UpdateScheduledEventFunc func(ev *model.ScheduledEvent) error + DeleteScheduledEventFunc func(id string) error // RenameBackend UpdateQualifiedNameInAllUnitsFunc func(oldName, newName string) (int, error) diff --git a/mdl/backend/mock/mock_workflow.go b/mdl/backend/mock/mock_workflow.go index 84cf8464b..4d792482f 100644 --- a/mdl/backend/mock/mock_workflow.go +++ b/mdl/backend/mock/mock_workflow.go @@ -123,3 +123,28 @@ func (m *MockBackend) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, err } return nil, nil } + +// The write methods default to a descriptive error rather than nil so a test +// that forgets to configure one fails on the missing stub instead of silently +// reporting a write that never happened. + +func (m *MockBackend) CreateScheduledEvent(ev *model.ScheduledEvent) error { + if m.CreateScheduledEventFunc != nil { + return m.CreateScheduledEventFunc(ev) + } + return fmt.Errorf("MockBackend.CreateScheduledEvent not configured") +} + +func (m *MockBackend) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + if m.UpdateScheduledEventFunc != nil { + return m.UpdateScheduledEventFunc(ev) + } + return fmt.Errorf("MockBackend.UpdateScheduledEvent not configured") +} + +func (m *MockBackend) DeleteScheduledEvent(id string) error { + if m.DeleteScheduledEventFunc != nil { + return m.DeleteScheduledEventFunc(id) + } + return fmt.Errorf("MockBackend.DeleteScheduledEvent not configured") +} diff --git a/mdl/backend/modelsdk/scheduledevent_read.go b/mdl/backend/modelsdk/scheduledevent_read.go index 8bf928674..9d4e8727a 100644 --- a/mdl/backend/modelsdk/scheduledevent_read.go +++ b/mdl/backend/modelsdk/scheduledevent_read.go @@ -5,60 +5,88 @@ package modelsdkbackend import ( "fmt" - genSched "github.com/mendixlabs/mxcli/modelsdk/gen/scheduledevents" - "github.com/mendixlabs/mxcli/modelsdk/mprread" + "go.mongodb.org/mongo-driver/bson" + sched "github.com/mendixlabs/mxcli/mdl/scheduledevents" "github.com/mendixlabs/mxcli/model" + mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" ) -// Codec-native scheduled-event read. Used by SHOW STRUCTURE (per-module counts) -// and the project tree; the modelsdk engine previously left these unimplemented, -// so scheduled events were silently undercounted (the callers swallow the -// not-implemented error). Unlike JavaScript actions, the gen codec decodes -// ScheduledEvent under the same storage keys the legacy parser uses (Name / -// Documentation / Microflow / Enabled / Interval / IntervalType), so gen -// accessors are correct here — no raw-key reads needed. +// Scheduled events. Used by SHOW STRUCTURE (per-module counts), the project +// tree, and SHOW/DESCRIBE/CREATE/DROP SCHEDULED EVENT. +// +// The document is read and written as raw BSON through mdl/scheduledevents +// rather than through modelsdk/gen, because gen's generated types disagree with +// what Studio Pro actually writes on two properties: every integer is stored as +// int64 while gen declares int32 (the mismatch behind issue #585), and +// StartDateTime is a BSON UTC datetime while gen declares a string. func (b *Backend) ListScheduledEvents() ([]*model.ScheduledEvent, error) { - units, err := mprread.ListUnitsWithContainer[*genSched.ScheduledEvent](b.reader) + units, err := b.reader.ListRawUnitsByType(sched.TypeName) if err != nil { return nil, err } out := make([]*model.ScheduledEvent, 0, len(units)) for _, u := range units { - out = append(out, scheduledEventFromGen(u.Element, u.ContainerID)) + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + return nil, fmt.Errorf("unmarshal scheduled event %s: %w", u.ID, err) + } + out = append(out, sched.Parse(doc, model.ID(u.ID), model.ID(u.ContainerID))) } return out, nil } func (b *Backend) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, error) { - units, err := mprread.ListUnitsWithContainer[*genSched.ScheduledEvent](b.reader) + events, err := b.ListScheduledEvents() if err != nil { return nil, err } - for _, u := range units { - if model.ID(u.Element.ID()) == id { - return scheduledEventFromGen(u.Element, u.ContainerID), nil + for _, ev := range events { + if ev.ID == id { + return ev, nil } } return nil, fmt.Errorf("scheduled event not found: %s", id) } -// scheduledEventFromGen converts a gen ScheduledEvent to the semantic type. Mirrors -// the legacy parseScheduledEvent field set: MicroflowID holds the by-name microflow -// reference (BSON "Microflow"), and Interval comes through the int32 decoder (which -// accepts the int64 Studio Pro actually writes — issue #585). -func scheduledEventFromGen(g *genSched.ScheduledEvent, containerID model.ID) *model.ScheduledEvent { - ev := &model.ScheduledEvent{ - ContainerID: containerID, - Name: g.Name(), - Documentation: g.Documentation(), - MicroflowID: model.ID(g.MicroflowQualifiedName()), - Interval: int(g.Interval()), - IntervalType: g.IntervalType(), - Enabled: g.Enabled(), +// CreateScheduledEvent inserts a new scheduled event document. +func (b *Backend) CreateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("CreateScheduledEvent: nil event") + } + if b.writer == nil { + return fmt.Errorf("CreateScheduledEvent: not connected for writing") + } + if ev.ID == "" { + ev.ID = model.ID(mmpr.GenerateID()) + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return b.writer.InsertUnit(string(ev.ID), string(ev.ContainerID), "Documents", sched.TypeName, contents) +} + +// UpdateScheduledEvent rewrites an existing scheduled event in place. +func (b *Backend) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("UpdateScheduledEvent: nil event") + } + if b.writer == nil { + return fmt.Errorf("UpdateScheduledEvent: not connected for writing") + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return b.writer.UpdateRawUnit(string(ev.ID), contents) +} + +// DeleteScheduledEvent removes a scheduled event unit by ID. +func (b *Backend) DeleteScheduledEvent(id string) error { + if b.writer == nil { + return fmt.Errorf("DeleteScheduledEvent: not connected for writing") } - ev.ID = model.ID(g.ID()) - ev.TypeName = "ScheduledEvents$ScheduledEvent" - return ev + return b.writer.DeleteUnit(id) } diff --git a/mdl/backend/modelsdk/scheduledevent_read_test.go b/mdl/backend/modelsdk/scheduledevent_read_test.go index bb97048f6..01ef70971 100644 --- a/mdl/backend/modelsdk/scheduledevent_read_test.go +++ b/mdl/backend/modelsdk/scheduledevent_read_test.go @@ -4,58 +4,11 @@ package modelsdkbackend import ( "testing" + "time" "github.com/mendixlabs/mxcli/model" - genSched "github.com/mendixlabs/mxcli/modelsdk/gen/scheduledevents" ) -// TestScheduledEventFromGen guards the gen→semantic mapping the modelsdk read -// relies on. (No committed fixture contains scheduled events, and modelsdk has no -// scheduled-event write path, so the converter is exercised directly; the -// List/Get plumbing reuses the ListUnitsWithContainer pattern covered by the -// page/java reads.) The key mappings: MicroflowID carries the by-name microflow -// reference (BSON "Microflow"), Interval narrows the int32 accessor to int. -func TestScheduledEventFromGen(t *testing.T) { - g := genSched.NewScheduledEvent() - g.SetID("evt-1") - g.SetName("SE_Cleanup") - g.SetDocumentation("nightly cleanup") - g.SetMicroflowQualifiedName("MyModule.DoCleanup") - g.SetInterval(86400) - g.SetIntervalType("Day") - g.SetEnabled(true) - - ev := scheduledEventFromGen(g, model.ID("mod-1")) - - if ev.ID != "evt-1" { - t.Errorf("ID = %q, want evt-1", ev.ID) - } - if ev.ContainerID != "mod-1" { - t.Errorf("ContainerID = %q, want mod-1", ev.ContainerID) - } - if ev.TypeName != "ScheduledEvents$ScheduledEvent" { - t.Errorf("TypeName = %q", ev.TypeName) - } - if ev.Name != "SE_Cleanup" { - t.Errorf("Name = %q, want SE_Cleanup", ev.Name) - } - if ev.Documentation != "nightly cleanup" { - t.Errorf("Documentation = %q", ev.Documentation) - } - if ev.MicroflowID != "MyModule.DoCleanup" { - t.Errorf("MicroflowID = %q, want MyModule.DoCleanup", ev.MicroflowID) - } - if ev.Interval != 86400 { - t.Errorf("Interval = %d, want 86400", ev.Interval) - } - if ev.IntervalType != "Day" { - t.Errorf("IntervalType = %q, want Day", ev.IntervalType) - } - if !ev.Enabled { - t.Error("Enabled = false, want true") - } -} - // TestListScheduledEvents_Empty confirms the read returns an empty (not error) // result on a project with no scheduled events — the minimal fixture — so SHOW // STRUCTURE no longer swallows a not-implemented error. @@ -74,3 +27,83 @@ func TestListScheduledEvents_Empty(t *testing.T) { t.Errorf("got %d scheduled events, want 0 (minimal fixture has none)", len(events)) } } + +// TestScheduledEventRoundTrip writes an event and reads it back through the real +// backend. A round trip (rather than a reader-only test on hand-written BSON) is +// what catches a reader keyed on different property names than the writer +// produces — the failure mode recorded for the workflow-activity reads. +func TestScheduledEventRoundTrip(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + start := time.Date(2026, 1, 1, 4, 0, 0, 0, time.UTC) + want := &model.ScheduledEvent{ + ContainerID: mod.ID, + Name: "ZzNightly", + Documentation: "nightly cleanup", + MicroflowID: "MyFirstModule.DoCleanup", + StartDateTime: &start, + TimeZone: "Server", + OnOverlap: "SkipNext", + Interval: 1, + IntervalType: "Day", + Enabled: true, + Schedule: &model.Schedule{ + Kind: model.ScheduleMonthWeekday, Multiplier: 3, MonthOffset: 2, + DaySelector: "Last", Weekday: "Friday", HourOfDay: 18, MinuteOfHour: 30, + }, + } + if err := b.CreateScheduledEvent(want); err != nil { + t.Fatalf("CreateScheduledEvent: %v", err) + } + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + events, err := b2.ListScheduledEvents() + if err != nil { + t.Fatalf("ListScheduledEvents: %v", err) + } + for _, got := range events { + if got.Name != "ZzNightly" { + continue + } + if got.MicroflowID != want.MicroflowID { + t.Errorf("MicroflowID = %q", got.MicroflowID) + } + if got.Documentation != want.Documentation { + t.Errorf("Documentation = %q", got.Documentation) + } + if got.TimeZone != "Server" || got.OnOverlap != "SkipNext" { + t.Errorf("TimeZone/OnOverlap = %q/%q", got.TimeZone, got.OnOverlap) + } + if got.Interval != 1 || got.IntervalType != "Day" { + t.Errorf("legacy pair = %d/%q", got.Interval, got.IntervalType) + } + if !got.Enabled { + t.Error("Enabled did not round-trip") + } + if got.StartDateTime == nil || !got.StartDateTime.Equal(start) { + t.Errorf("StartDateTime = %v, want %v", got.StartDateTime, start) + } + if got.Schedule == nil { + t.Fatal("Schedule did not round-trip") + } + if *got.Schedule != *want.Schedule { + t.Errorf("Schedule = %+v, want %+v", *got.Schedule, *want.Schedule) + } + return + } + t.Fatalf("ZzNightly not found after create (got %d events)", len(events)) +} diff --git a/mdl/backend/modelsdk/unimplemented_gen.go b/mdl/backend/modelsdk/unimplemented_gen.go index 62cea55be..f3ceed68d 100644 --- a/mdl/backend/modelsdk/unimplemented_gen.go +++ b/mdl/backend/modelsdk/unimplemented_gen.go @@ -184,6 +184,10 @@ func (unimplemented) CreateQueue(_ *types.Queue) error { return errUnimplemented("CreateQueue") } +func (unimplemented) CreateScheduledEvent(_ *model.ScheduledEvent) error { + return errUnimplemented("CreateScheduledEvent") +} + func (unimplemented) CreateSnippet(_ *pages.Snippet) error { return errUnimplemented("CreateSnippet") } @@ -329,6 +333,10 @@ func (unimplemented) DeleteQueue(_ string) error { return errUnimplemented("DeleteQueue") } +func (unimplemented) DeleteScheduledEvent(_ string) error { + return errUnimplemented("DeleteScheduledEvent") +} + func (unimplemented) DeleteSnippet(_ model.ID) error { return errUnimplemented("DeleteSnippet") } @@ -1082,6 +1090,10 @@ func (unimplemented) UpdateRawUnit(_ string, _ []uint8) error { return errUnimplemented("UpdateRawUnit") } +func (unimplemented) UpdateScheduledEvent(_ *model.ScheduledEvent) error { + return errUnimplemented("UpdateScheduledEvent") +} + func (unimplemented) UpdateSnippet(_ *pages.Snippet) error { return errUnimplemented("UpdateSnippet") } diff --git a/mdl/backend/mpr/backend.go b/mdl/backend/mpr/backend.go index 44050fa80..a539d898d 100644 --- a/mdl/backend/mpr/backend.go +++ b/mdl/backend/mpr/backend.go @@ -729,6 +729,15 @@ func (b *MprBackend) ListScheduledEvents() ([]*model.ScheduledEvent, error) { func (b *MprBackend) GetScheduledEvent(id model.ID) (*model.ScheduledEvent, error) { return b.reader.GetScheduledEvent(id) } +func (b *MprBackend) CreateScheduledEvent(ev *model.ScheduledEvent) error { + return b.writer.CreateScheduledEvent(ev) +} +func (b *MprBackend) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + return b.writer.UpdateScheduledEvent(ev) +} +func (b *MprBackend) DeleteScheduledEvent(id string) error { + return b.writer.DeleteScheduledEvent(id) +} // --------------------------------------------------------------------------- // RenameBackend diff --git a/mdl/executor/cmd_queues_mock_test.go b/mdl/executor/cmd_queues_mock_test.go index d3424afeb..c004b8df8 100644 --- a/mdl/executor/cmd_queues_mock_test.go +++ b/mdl/executor/cmd_queues_mock_test.go @@ -77,9 +77,9 @@ func TestCreateQueue_Mock_PassesParallelismThrough(t *testing.T) { var created *types.Queue mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return nil, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, CreateQueueFunc: func(q *types.Queue) error { created = q return nil @@ -113,9 +113,9 @@ func TestCreateQueue_Mock_DuplicateWithoutOrModify(t *testing.T) { withContainer(h, existing.ContainerID, mod.ID) mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, CreateQueueFunc: func(q *types.Queue) error { t.Error("CreateQueue must not be called for a duplicate") return nil @@ -137,9 +137,9 @@ func TestCreateQueue_Mock_OrModifyUpdates(t *testing.T) { var updated *types.Queue mb := &mock.MockBackend{ - IsConnectedFunc: func() bool { return true }, - ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, - ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + IsConnectedFunc: func() bool { return true }, + ListQueuesFunc: func() ([]*types.Queue, error) { return []*types.Queue{existing}, nil }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, UpdateQueueFunc: func(q *types.Queue) error { updated = q return nil diff --git a/mdl/executor/cmd_scheduledevents.go b/mdl/executor/cmd_scheduledevents.go new file mode 100644 index 000000000..a89b2c4ac --- /dev/null +++ b/mdl/executor/cmd_scheduledevents.go @@ -0,0 +1,586 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package executor — scheduled event commands (CREATE/DROP/SHOW/DESCRIBE +// SCHEDULED EVENT). +// +// A Mendix scheduled event runs a microflow on a repeating schedule. It is +// Mendix's cron: the repeat rule is a ScheduledEvents$Schedule child with eight +// variants, and OnOverlap (DelayNext|SkipNext) is its own concurrency control — +// scheduled events do not go through a task queue. +package executor + +import ( + "fmt" + "sort" + "strings" + "time" + + "github.com/mendixlabs/mxcli/mdl/ast" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" +) + +// repeatKinds maps the MDL Repeat value to the storage's schedule variant. +// The MDL spellings are the calendar words a reader expects; the storage names +// are Mendix's. +var repeatKinds = map[string]model.ScheduleKind{ + "minutely": model.ScheduleMinute, + "hourly": model.ScheduleHour, + "daily": model.ScheduleDay, + "weekly": model.ScheduleWeek, + "monthlybydate": model.ScheduleMonthDate, + "monthlybyweekday": model.ScheduleMonthWeekday, + "yearlybydate": model.ScheduleYearDate, + "yearlybyweekday": model.ScheduleYearWeekday, +} + +// repeatNames is the reverse map, for DESCRIBE. +var repeatNames = map[model.ScheduleKind]string{ + model.ScheduleMinute: "Minutely", + model.ScheduleHour: "Hourly", + model.ScheduleDay: "Daily", + model.ScheduleWeek: "Weekly", + model.ScheduleMonthDate: "MonthlyByDate", + model.ScheduleMonthWeekday: "MonthlyByWeekday", + model.ScheduleYearDate: "YearlyByDate", + model.ScheduleYearWeekday: "YearlyByWeekday", +} + +// repeatFields lists the properties each repeat actually uses. A property that +// belongs to a different variant is REFUSED rather than ignored: the variants +// differ in which fields they carry, so silently dropping one would write a +// schedule that does not do what the script says. +var repeatFields = map[model.ScheduleKind][]string{ + model.ScheduleMinute: {"Multiplier"}, + model.ScheduleHour: {"Multiplier", "MinuteOffset"}, + model.ScheduleDay: {"HourOfDay", "MinuteOfHour"}, + model.ScheduleWeek: {"Weekdays", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthDate: {"Multiplier", "MonthOffset", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthWeekday: {"Multiplier", "MonthOffset", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearDate: {"Month", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearWeekday: {"Month", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, +} + +var daySelectors = []string{"First", "Second", "Third", "Fourth", "Last"} + +var weekdayNames = []string{"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"} + +// findScheduledEvent returns the event with the given module-qualified name, or nil. +func findScheduledEvent(ctx *ExecContext, moduleName, name string) *model.ScheduledEvent { + events, err := ctx.Backend.ListScheduledEvents() + if err != nil { + return nil + } + h, err := getHierarchy(ctx) + if err != nil { + return nil + } + for _, ev := range events { + if !strings.EqualFold(ev.Name, name) { + continue + } + mod := h.GetModuleName(h.FindModuleID(ev.ContainerID)) + if strings.EqualFold(mod, moduleName) { + return ev + } + } + return nil +} + +// execCreateScheduledEvent handles CREATE [OR REPLACE|MODIFY] SCHEDULED EVENT. +func execCreateScheduledEvent(ctx *ExecContext, s *ast.CreateScheduledEventStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + + module, err := findOrCreateModule(ctx, s.Name.Module) + if err != nil { + return err + } + + existing := findScheduledEvent(ctx, s.Name.Module, s.Name.Name) + if existing != nil && !s.CreateOrModify { + return mdlerrors.NewAlreadyExists("scheduled event", s.Name.String()) + } + + ev, err := scheduledEventFromStmt(s) + if err != nil { + return err + } + ev.ContainerID = module.ID + if existing != nil { + ev.ID = existing.ID + ev.ContainerID = existing.ContainerID + // Interval/IntervalType are legacy siblings of Schedule that Studio Pro + // writes but does not keep in sync, and MDL has no syntax for them. + // Carry the stored values so a modify does not invent new ones. + ev.Interval = existing.Interval + ev.IntervalType = existing.IntervalType + if err := ctx.Backend.UpdateScheduledEvent(ev); err != nil { + return mdlerrors.NewBackend("update scheduled event", err) + } + fmt.Fprintf(ctx.Output, "Modified scheduled event: %s\n", s.Name.String()) + return nil + } + if err := ctx.Backend.CreateScheduledEvent(ev); err != nil { + return mdlerrors.NewBackend("create scheduled event", err) + } + fmt.Fprintf(ctx.Output, "Created scheduled event: %s\n", s.Name.String()) + return nil +} + +// scheduledEventFromStmt validates the statement and builds the semantic event. +func scheduledEventFromStmt(s *ast.CreateScheduledEventStmt) (*model.ScheduledEvent, error) { + if strings.TrimSpace(s.Microflow) == "" { + return nil, mdlerrors.NewValidation( + "scheduled event " + s.Name.String() + " has no Microflow — a scheduled event runs a microflow, " + + "so add `Microflow: Module.MyMicroflow` to the property list") + } + ev := &model.ScheduledEvent{ + Name: s.Name.Name, + Documentation: s.Documentation, + MicroflowID: model.ID(s.Microflow), + TimeZone: s.TimeZone, + OnOverlap: s.OnOverlap, + ExportLevel: s.ExportLevel, + } + if s.Enabled != nil { + ev.Enabled = *s.Enabled + } + if s.Excluded != nil { + ev.Excluded = *s.Excluded + } + if err := validateEnumProperty("TimeZone", ev.TimeZone, []string{"UTC", "Server"}); err != nil { + return nil, err + } + if err := validateEnumProperty("OnOverlap", ev.OnOverlap, []string{"DelayNext", "SkipNext"}); err != nil { + return nil, err + } + if s.StartDateTime != "" { + t, err := time.Parse(time.RFC3339, s.StartDateTime) + if err != nil { + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "StartDateTime %q is not an RFC 3339 timestamp (e.g. '2026-01-01T04:00:00Z')", s.StartDateTime)) + } + utc := t.UTC() + ev.StartDateTime = &utc + } + + sched, err := scheduleFromStmt(s) + if err != nil { + return nil, err + } + ev.Schedule = sched + ev.Interval, ev.IntervalType = legacyIntervalFor(sched) + return ev, nil +} + +// legacyIntervalFor derives the Interval/IntervalType pair a new event gets. +// +// These are legacy siblings of Schedule with no MDL syntax, but they are not +// optional: IntervalType is a Mendix enumeration, and writing an empty string +// puts a value in the model that the enumeration does not have. Every Studio +// Pro-authored event carries a real one. +// +// The mapping is what the two self-consistent references show (SAML: Day/1 +// beside a DaySchedule; OIDC: Hour/1 beside an HourSchedule with Multiplier 1). +// It is only used on CREATE — a modify carries the stored pair through +// untouched, because Studio Pro does NOT keep these in sync with Schedule (the +// Workflow Commons event stores Minute/0 next to a DaySchedule) and re-deriving +// them would overwrite whatever the developer's Studio Pro left behind. +func legacyIntervalFor(s *model.Schedule) (int, string) { + if s == nil { + return 1, "Day" + } + switch s.Kind { + case model.ScheduleMinute: + return s.Multiplier, "Minute" + case model.ScheduleHour: + return s.Multiplier, "Hour" + case model.ScheduleDay: + return 1, "Day" + case model.ScheduleWeek: + return 1, "Week" + case model.ScheduleMonthDate, model.ScheduleMonthWeekday: + return s.Multiplier, "Month" + case model.ScheduleYearDate, model.ScheduleYearWeekday: + return 1, "Year" + } + return 1, "Day" +} + +// scheduleFromStmt builds the repeat rule, refusing any field that does not +// belong to the chosen Repeat. +func scheduleFromStmt(s *ast.CreateScheduledEventStmt) (*model.Schedule, error) { + if strings.TrimSpace(s.Repeat) == "" { + return nil, mdlerrors.NewValidation( + "scheduled event " + s.Name.String() + " has no Repeat — add one of " + + strings.Join(sortedRepeatNames(), ", ")) + } + kind, ok := repeatKinds[strings.ToLower(s.Repeat)] + if !ok { + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "unknown Repeat %q — expected one of %s", s.Repeat, strings.Join(sortedRepeatNames(), ", "))) + } + + // Which properties the statement actually set, so a field belonging to a + // different variant is reported instead of silently dropped. + set := map[string]bool{ + "Multiplier": s.Multiplier != nil, + "MinuteOffset": s.MinuteOffset != nil, + "MonthOffset": s.MonthOffset != nil, + "HourOfDay": s.HourOfDay != nil, + "MinuteOfHour": s.MinuteOfHour != nil, + "DayOfMonth": s.DayOfMonth != nil, + "Month": s.Month != nil, + "Weekdays": s.Weekdays != "", + "DaySelector": s.DaySelector != "", + "Weekday": s.Weekday != "", + } + allowed := map[string]bool{} + for _, f := range repeatFields[kind] { + allowed[f] = true + } + var stray []string + for f, isSet := range set { + if isSet && !allowed[f] { + stray = append(stray, f) + } + } + if len(stray) > 0 { + sort.Strings(stray) + return nil, mdlerrors.NewValidation(fmt.Sprintf( + "Repeat %s does not have %s — it takes %s", + s.Repeat, strings.Join(stray, ", "), strings.Join(repeatFields[kind], ", "))) + } + + sched := &model.Schedule{Kind: kind} + if s.Multiplier != nil { + sched.Multiplier = *s.Multiplier + } else if allowed["Multiplier"] { + // "every 1 " — an unstated multiplier of 0 would never fire. + sched.Multiplier = 1 + } + if s.MinuteOffset != nil { + sched.MinuteOffset = *s.MinuteOffset + } + if s.MonthOffset != nil { + sched.MonthOffset = *s.MonthOffset + } + if s.HourOfDay != nil { + sched.HourOfDay = *s.HourOfDay + } + if s.MinuteOfHour != nil { + sched.MinuteOfHour = *s.MinuteOfHour + } + if s.DayOfMonth != nil { + sched.DayOfMonth = *s.DayOfMonth + } else if allowed["DayOfMonth"] { + sched.DayOfMonth = 1 + } + if s.Month != nil { + sched.Month = *s.Month + } else if allowed["Month"] { + sched.Month = 1 + } + sched.DaySelector = s.DaySelector + sched.Weekday = s.Weekday + + if err := validateScheduleRanges(sched, allowed); err != nil { + return nil, err + } + if allowed["Weekdays"] { + days, err := parseWeekdays(s.Weekdays) + if err != nil { + return nil, err + } + sched.Weekdays = days + } + if allowed["DaySelector"] { + if err := validateEnumProperty("DaySelector", sched.DaySelector, daySelectors); err != nil { + return nil, err + } + if sched.DaySelector == "" { + sched.DaySelector = "First" + } + if err := validateEnumProperty("Weekday", sched.Weekday, weekdayNames); err != nil { + return nil, err + } + if sched.Weekday == "" { + sched.Weekday = "Monday" + } + } + return sched, nil +} + +// validateScheduleRanges rejects values Mendix's editor would not let you enter. +// A schedule that is stored but can never fire is worse than a refusal, because +// nothing downstream reports it. +func validateScheduleRanges(s *model.Schedule, allowed map[string]bool) error { + checks := []struct { + name string + value int + min, max int + }{ + {"Multiplier", s.Multiplier, 1, 1_000_000}, + {"MinuteOffset", s.MinuteOffset, 0, 59}, + {"MonthOffset", s.MonthOffset, 0, 11}, + {"HourOfDay", s.HourOfDay, 0, 23}, + {"MinuteOfHour", s.MinuteOfHour, 0, 59}, + {"DayOfMonth", s.DayOfMonth, 1, 31}, + {"Month", s.Month, 1, 12}, + } + for _, c := range checks { + if !allowed[c.name] { + continue + } + if c.value < c.min || c.value > c.max { + return mdlerrors.NewValidation(fmt.Sprintf( + "%s is %d — it must be between %d and %d", c.name, c.value, c.min, c.max)) + } + } + return nil +} + +// parseWeekdays turns "Monday, Friday" into the seven flags of a WeekSchedule. +func parseWeekdays(spec string) ([7]bool, error) { + var days [7]bool + if strings.TrimSpace(spec) == "" { + return days, mdlerrors.NewValidation( + "Repeat Weekly needs Weekdays — e.g. Weekdays: 'Monday, Friday'") + } + for _, part := range strings.Split(spec, ",") { + name := strings.TrimSpace(part) + if name == "" { + continue + } + idx := -1 + for i, w := range weekdayNames { + if strings.EqualFold(w, name) { + idx = i + break + } + } + if idx < 0 { + return days, mdlerrors.NewValidation(fmt.Sprintf( + "unknown weekday %q in Weekdays — expected one of %s", name, strings.Join(weekdayNames, ", "))) + } + days[idx] = true + } + return days, nil +} + +// validateEnumProperty accepts an empty value (the writer supplies the default) +// and otherwise requires an exact match, normalising nothing — an enum value +// Mendix does not know produces a document that loads and misbehaves. +func validateEnumProperty(name, value string, allowed []string) error { + if value == "" { + return nil + } + for _, a := range allowed { + if a == value { + return nil + } + } + for _, a := range allowed { + if strings.EqualFold(a, value) { + return mdlerrors.NewValidation(fmt.Sprintf( + "%s %q has the wrong casing — Mendix stores it as %q", name, value, a)) + } + } + return mdlerrors.NewValidation(fmt.Sprintf( + "unknown %s %q — expected one of %s", name, value, strings.Join(allowed, ", "))) +} + +func sortedRepeatNames() []string { + out := make([]string, 0, len(repeatNames)) + for _, n := range repeatNames { + out = append(out, n) + } + sort.Strings(out) + return out +} + +// execDropScheduledEvent handles DROP SCHEDULED EVENT Module.Name. +func execDropScheduledEvent(ctx *ExecContext, s *ast.DropScheduledEventStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + existing := findScheduledEvent(ctx, s.Name.Module, s.Name.Name) + if existing == nil { + return mdlerrors.NewNotFound("scheduled event", s.Name.String()) + } + if err := ctx.Backend.DeleteScheduledEvent(string(existing.ID)); err != nil { + return mdlerrors.NewBackend("drop scheduled event", err) + } + fmt.Fprintf(ctx.Output, "Dropped scheduled event: %s\n", s.Name.String()) + return nil +} + +// execShowScheduledEvents handles SHOW|LIST SCHEDULED EVENTS [IN Module]. +func execShowScheduledEvents(ctx *ExecContext, s *ast.ShowScheduledEventsStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + events, err := ctx.Backend.ListScheduledEvents() + if err != nil { + return mdlerrors.NewBackend("list scheduled events", err) + } + h, err := getHierarchy(ctx) + if err != nil { + return mdlerrors.NewBackend("build hierarchy", err) + } + + type row struct{ qualified, repeat, microflow, enabled string } + var rows []row + for _, ev := range events { + mod := h.GetModuleName(h.FindModuleID(ev.ContainerID)) + if s.Module != "" && !strings.EqualFold(mod, s.Module) { + continue + } + rows = append(rows, row{ + qualified: mod + "." + ev.Name, + repeat: describeRepeat(ev.Schedule), + microflow: string(ev.MicroflowID), + enabled: fmt.Sprintf("%t", ev.Enabled), + }) + } + sort.Slice(rows, func(i, j int) bool { return rows[i].qualified < rows[j].qualified }) + + result := &TableResult{ + Columns: []string{"Scheduled Event", "Repeat", "Microflow", "Enabled"}, + Summary: fmt.Sprintf("(%d scheduled event(s))", len(rows)), + } + for _, r := range rows { + result.Rows = append(result.Rows, []any{r.qualified, r.repeat, r.microflow, r.enabled}) + } + return writeResult(ctx, result) +} + +// describeRepeat renders the schedule as a short human phrase for the listing. +func describeRepeat(s *model.Schedule) string { + if s == nil { + return "(none)" + } + at := fmt.Sprintf("%02d:%02d", s.HourOfDay, s.MinuteOfHour) + switch s.Kind { + case model.ScheduleMinute: + return fmt.Sprintf("every %d min", s.Multiplier) + case model.ScheduleHour: + return fmt.Sprintf("every %dh at :%02d", s.Multiplier, s.MinuteOffset) + case model.ScheduleDay: + return "daily at " + at + case model.ScheduleWeek: + return "weekly " + strings.Join(setWeekdayNames(s.Weekdays), "/") + " at " + at + case model.ScheduleMonthDate: + return fmt.Sprintf("every %d month(s) on day %d at %s", s.Multiplier, s.DayOfMonth, at) + case model.ScheduleMonthWeekday: + return fmt.Sprintf("every %d month(s) on the %s %s at %s", s.Multiplier, s.DaySelector, s.Weekday, at) + case model.ScheduleYearDate: + return fmt.Sprintf("yearly on %d/%d at %s", s.Month, s.DayOfMonth, at) + case model.ScheduleYearWeekday: + return fmt.Sprintf("yearly on the %s %s of month %d at %s", s.DaySelector, s.Weekday, s.Month, at) + } + return string(s.Kind) +} + +func setWeekdayNames(days [7]bool) []string { + var out []string + for i, on := range days { + if on { + out = append(out, weekdayNames[i][:3]) + } + } + if out == nil { + return []string{"(no days)"} + } + return out +} + +// execDescribeScheduledEvent handles DESCRIBE SCHEDULED EVENT Module.Name, +// emitting re-executable MDL so describe → exec round-trips. +func execDescribeScheduledEvent(ctx *ExecContext, s *ast.DescribeScheduledEventStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + ev := findScheduledEvent(ctx, s.Name.Module, s.Name.Name) + if ev == nil { + return mdlerrors.NewNotFound("scheduled event", s.Name.String()) + } + + if ev.Documentation != "" { + fmt.Fprintf(ctx.Output, "/**\n * %s\n */\n", ev.Documentation) + } + fmt.Fprintf(ctx.Output, "create or modify scheduled event %s (\n", s.Name.String()) + fmt.Fprintf(ctx.Output, " Microflow: %s,\n", ev.MicroflowID) + for _, line := range describeScheduleProperties(ev.Schedule) { + fmt.Fprintf(ctx.Output, " %s,\n", line) + } + fmt.Fprintf(ctx.Output, " Enabled: %t,\n", ev.Enabled) + if ev.OnOverlap != "" { + fmt.Fprintf(ctx.Output, " OnOverlap: %s,\n", ev.OnOverlap) + } + if ev.TimeZone != "" { + fmt.Fprintf(ctx.Output, " TimeZone: %s,\n", ev.TimeZone) + } + // A zero StartDateTime is how "no start restriction" is stored (Mendix's own + // DateTime.MinValue), not a date anyone chose — emitting it would put + // '0001-01-01T00:00:00Z' into every describe. + if ev.StartDateTime != nil && !ev.StartDateTime.IsZero() { + fmt.Fprintf(ctx.Output, " StartDateTime: '%s',\n", ev.StartDateTime.Format(time.RFC3339)) + } + fmt.Fprint(ctx.Output, ");\n") + // Interval/IntervalType have no MDL syntax: they are legacy siblings of + // Schedule that Studio Pro writes without keeping in sync. Reporting them as + // a comment keeps the output honest without making it non-re-executable. + if ev.IntervalType != "" { + fmt.Fprintf(ctx.Output, "-- legacy (preserved, not authorable): Interval %d %s\n", ev.Interval, ev.IntervalType) + } + return nil +} + +// describeScheduleProperties emits the Repeat plus exactly the fields that +// repeat uses, in the order repeatFields lists them. +func describeScheduleProperties(s *model.Schedule) []string { + if s == nil { + return nil + } + out := []string{"Repeat: " + repeatNames[s.Kind]} + for _, f := range repeatFields[s.Kind] { + switch f { + case "Multiplier": + out = append(out, fmt.Sprintf("Multiplier: %d", s.Multiplier)) + case "MinuteOffset": + out = append(out, fmt.Sprintf("MinuteOffset: %d", s.MinuteOffset)) + case "MonthOffset": + out = append(out, fmt.Sprintf("MonthOffset: %d", s.MonthOffset)) + case "HourOfDay": + out = append(out, fmt.Sprintf("HourOfDay: %d", s.HourOfDay)) + case "MinuteOfHour": + out = append(out, fmt.Sprintf("MinuteOfHour: %d", s.MinuteOfHour)) + case "DayOfMonth": + out = append(out, fmt.Sprintf("DayOfMonth: %d", s.DayOfMonth)) + case "Month": + out = append(out, fmt.Sprintf("Month: %d", s.Month)) + case "DaySelector": + out = append(out, "DaySelector: "+s.DaySelector) + case "Weekday": + out = append(out, "Weekday: "+s.Weekday) + case "Weekdays": + var names []string + for i, on := range s.Weekdays { + if on { + names = append(names, weekdayNames[i]) + } + } + out = append(out, "Weekdays: '"+strings.Join(names, ", ")+"'") + } + } + return out +} diff --git a/mdl/executor/cmd_scheduledevents_test.go b/mdl/executor/cmd_scheduledevents_test.go new file mode 100644 index 000000000..bae0dbc0b --- /dev/null +++ b/mdl/executor/cmd_scheduledevents_test.go @@ -0,0 +1,456 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/model" +) + +func mkScheduledEvent(containerID model.ID, name string, s *model.Schedule) *model.ScheduledEvent { + ev := &model.ScheduledEvent{ + ContainerID: containerID, + Name: name, + MicroflowID: "Ops.SE_Cleanup", + Schedule: s, + OnOverlap: "DelayNext", + TimeZone: "UTC", + } + ev.ID = nextID("sched") + return ev +} + +func intPtr(v int) *int { return &v } + +// TestScheduleFromStmt_EachRepeatTakesOnlyItsOwnFields is the load-bearing test: +// the eight ScheduledEvents$*Schedule variants differ in WHICH fields they +// carry, and a merged field set is the shape mxbuild accepts and Studio Pro +// refuses to open. +func TestScheduleFromStmt_EachRepeatTakesOnlyItsOwnFields(t *testing.T) { + tests := []struct { + repeat string + stray string // a property that belongs to a different repeat + apply func(*ast.CreateScheduledEventStmt) + }{ + {"Daily", "Multiplier", func(s *ast.CreateScheduledEventStmt) { s.Multiplier = intPtr(3) }}, + {"Minutely", "HourOfDay", func(s *ast.CreateScheduledEventStmt) { s.HourOfDay = intPtr(4) }}, + {"Hourly", "DayOfMonth", func(s *ast.CreateScheduledEventStmt) { s.DayOfMonth = intPtr(15) }}, + {"YearlyByDate", "MonthOffset", func(s *ast.CreateScheduledEventStmt) { s.MonthOffset = intPtr(1) }}, + {"MonthlyByDate", "DaySelector", func(s *ast.CreateScheduledEventStmt) { s.DaySelector = "Last" }}, + {"Daily", "Weekdays", func(s *ast.CreateScheduledEventStmt) { s.Weekdays = "Monday" }}, + } + for _, tt := range tests { + t.Run(tt.repeat+"/"+tt.stray, func(t *testing.T) { + s := &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, + Microflow: "Ops.MF", + Repeat: tt.repeat, + } + tt.apply(s) + _, err := scheduleFromStmt(s) + if err == nil { + t.Fatalf("%s accepted %s, which belongs to a different repeat", tt.repeat, tt.stray) + } + if !strings.Contains(err.Error(), tt.stray) { + t.Errorf("error should name the stray property:\n%s", err) + } + }) + } +} + +func TestScheduleFromStmt_RangesAreEnforced(t *testing.T) { + tests := []struct { + name string + repeat string + apply func(*ast.CreateScheduledEventStmt) + }{ + {"HourOfDay 24", "Daily", func(s *ast.CreateScheduledEventStmt) { s.HourOfDay = intPtr(24) }}, + {"MinuteOfHour 60", "Daily", func(s *ast.CreateScheduledEventStmt) { s.MinuteOfHour = intPtr(60) }}, + {"DayOfMonth 0", "MonthlyByDate", func(s *ast.CreateScheduledEventStmt) { s.DayOfMonth = intPtr(0) }}, + {"Month 13", "YearlyByDate", func(s *ast.CreateScheduledEventStmt) { s.Month = intPtr(13) }}, + {"Multiplier 0", "Minutely", func(s *ast.CreateScheduledEventStmt) { s.Multiplier = intPtr(0) }}, + {"MinuteOffset 60", "Hourly", func(s *ast.CreateScheduledEventStmt) { s.MinuteOffset = intPtr(60) }}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, + Microflow: "Ops.MF", + Repeat: tt.repeat, + } + tt.apply(s) + if _, err := scheduleFromStmt(s); err == nil { + t.Fatalf("%s was accepted — a schedule that can never fire is worse than a refusal", tt.name) + } + }) + } +} + +func TestScheduleFromStmt_Defaults(t *testing.T) { + // An unstated multiplier of 0 would never fire, so it defaults to 1. + got, err := scheduleFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", Repeat: "Hourly", + }) + if err != nil { + t.Fatalf("scheduleFromStmt: %v", err) + } + if got.Multiplier != 1 { + t.Errorf("Multiplier = %d, want 1", got.Multiplier) + } + if got.Kind != model.ScheduleHour { + t.Errorf("Kind = %v", got.Kind) + } +} + +func TestScheduleFromStmt_Weekdays(t *testing.T) { + got, err := scheduleFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Weekly", Weekdays: "monday, FRIDAY", HourOfDay: intPtr(9), + }) + if err != nil { + t.Fatalf("scheduleFromStmt: %v", err) + } + want := [7]bool{false, true, false, false, false, true, false} + if got.Weekdays != want { + t.Errorf("Weekdays = %v, want %v (case-insensitive names)", got.Weekdays, want) + } +} + +func TestScheduleFromStmt_UnknownWeekday(t *testing.T) { + _, err := scheduleFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Weekly", Weekdays: "Moonday", + }) + if err == nil || !strings.Contains(err.Error(), "Moonday") { + t.Fatalf("expected an unknown-weekday error, got %v", err) + } +} + +// TestScheduledEventFromStmt_EnumCasing checks that a near-miss enum value is +// reported rather than normalised: Mendix stores the exact spelling, and a value +// the enumeration does not have produces a document that loads and misbehaves. +func TestScheduledEventFromStmt_EnumCasing(t *testing.T) { + _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Daily", TimeZone: "server", + }) + if err == nil || !strings.Contains(err.Error(), "Server") { + t.Fatalf("expected a casing error naming the stored spelling, got %v", err) + } +} + +func TestScheduledEventFromStmt_RequiresMicroflowAndRepeat(t *testing.T) { + if _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Repeat: "Daily", + }); err == nil { + t.Error("a scheduled event with no Microflow was accepted") + } + if _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + }); err == nil { + t.Error("a scheduled event with no Repeat was accepted") + } +} + +func TestScheduledEventFromStmt_StartDateTime(t *testing.T) { + ev, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Daily", StartDateTime: "2026-01-01T04:00:00Z", + }) + if err != nil { + t.Fatalf("scheduledEventFromStmt: %v", err) + } + if ev.StartDateTime == nil || ev.StartDateTime.Year() != 2026 { + t.Errorf("StartDateTime = %v", ev.StartDateTime) + } + if _, err := scheduledEventFromStmt(&ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, Microflow: "Ops.MF", + Repeat: "Daily", StartDateTime: "1 January 2026", + }); err == nil { + t.Error("a non-RFC-3339 StartDateTime was accepted") + } +} + +// TestLegacyIntervalFor pins the Interval/IntervalType pair a new event gets. +// IntervalType is a Mendix enumeration, so an empty string is not a valid +// "unset" — every Studio Pro-authored event carries a real value. +func TestLegacyIntervalFor(t *testing.T) { + tests := []struct { + kind model.ScheduleKind + mult int + wantN int + wantType string + }{ + {model.ScheduleMinute, 5, 5, "Minute"}, + {model.ScheduleHour, 2, 2, "Hour"}, + {model.ScheduleDay, 0, 1, "Day"}, + {model.ScheduleWeek, 0, 1, "Week"}, + {model.ScheduleMonthDate, 3, 3, "Month"}, + {model.ScheduleYearWeekday, 0, 1, "Year"}, + } + for _, tt := range tests { + n, typ := legacyIntervalFor(&model.Schedule{Kind: tt.kind, Multiplier: tt.mult}) + if n != tt.wantN || typ != tt.wantType { + t.Errorf("%s -> %d/%q, want %d/%q", tt.kind, n, typ, tt.wantN, tt.wantType) + } + if typ == "" { + t.Errorf("%s produced an empty IntervalType — not a value the enumeration has", tt.kind) + } + } +} + +func TestShowScheduledEvents_Mock(t *testing.T) { + mod := mkModule("Ops") + e1 := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay, HourOfDay: 4}) + e1.Enabled = true + e2 := mkScheduledEvent(mod.ID, "Hourly", &model.Schedule{Kind: model.ScheduleHour, Multiplier: 2, MinuteOffset: 23}) + + h := mkHierarchy(mod) + withContainer(h, e1.ContainerID, mod.ID) + withContainer(h, e2.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{e1, e2}, nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execShowScheduledEvents(ctx, &ast.ShowScheduledEventsStmt{})) + + out := buf.String() + assertContainsStr(t, out, "Ops.Nightly") + assertContainsStr(t, out, "daily at 04:00") + assertContainsStr(t, out, "every 2h at :23") + assertContainsStr(t, out, "(2 scheduled event(s))") +} + +func TestCreateScheduledEvent_Mock(t *testing.T) { + mod := mkModule("Ops") + h := mkHierarchy(mod) + + var created *model.ScheduledEvent + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { return nil, nil }, + CreateScheduledEventFunc: func(ev *model.ScheduledEvent) error { + created = ev + return nil + }, + } + + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execCreateScheduledEvent(ctx, &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, + Microflow: "Ops.SE_Cleanup", + Repeat: "Daily", + HourOfDay: intPtr(4), + MinuteOfHour: intPtr(0), + Enabled: boolPtr(true), + })) + + if created == nil { + t.Fatal("CreateScheduledEvent was not called") + } + if created.Schedule == nil || created.Schedule.Kind != model.ScheduleDay { + t.Fatalf("Schedule = %+v", created.Schedule) + } + if created.Schedule.HourOfDay != 4 { + t.Errorf("HourOfDay = %d", created.Schedule.HourOfDay) + } + if created.IntervalType == "" { + t.Error("IntervalType is empty — it is an enumeration, not an optional string") + } + assertContainsStr(t, buf.String(), "Created scheduled event: Ops.Nightly") +} + +// TestModifyScheduledEvent_Mock_PreservesLegacyInterval: Studio Pro does not +// keep Interval/IntervalType in sync with Schedule and MDL cannot author them, +// so a modify must carry the stored pair through rather than re-derive it. +func TestModifyScheduledEvent_Mock_PreservesLegacyInterval(t *testing.T) { + mod := mkModule("Ops") + existing := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay, HourOfDay: 1}) + existing.Interval = 0 + existing.IntervalType = "Minute" // the Workflow Commons shape: stale, but real + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var updated *model.ScheduledEvent + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{existing}, nil + }, + UpdateScheduledEventFunc: func(ev *model.ScheduledEvent) error { + updated = ev + return nil + }, + } + + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execCreateScheduledEvent(ctx, &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, + Microflow: "Ops.SE_Cleanup", + Repeat: "Daily", + HourOfDay: intPtr(6), + CreateOrModify: true, + })) + + if updated == nil { + t.Fatal("UpdateScheduledEvent was not called") + } + if updated.ID != existing.ID { + t.Errorf("ID = %q, want the existing %q", updated.ID, existing.ID) + } + if updated.Interval != 0 || updated.IntervalType != "Minute" { + t.Errorf("legacy pair = %d/%q, want the stored 0/\"Minute\"", updated.Interval, updated.IntervalType) + } + if updated.Schedule.HourOfDay != 6 { + t.Errorf("HourOfDay = %d, want the new 6", updated.Schedule.HourOfDay) + } +} + +func TestCreateScheduledEvent_Mock_DuplicateWithoutOrModify(t *testing.T) { + mod := mkModule("Ops") + existing := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay}) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{existing}, nil + }, + CreateScheduledEventFunc: func(ev *model.ScheduledEvent) error { + t.Error("CreateScheduledEvent must not be called for a duplicate") + return nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + if err := execCreateScheduledEvent(ctx, &ast.CreateScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, Microflow: "Ops.MF", Repeat: "Daily", + }); err == nil { + t.Fatal("expected an already-exists error") + } +} + +func TestDropScheduledEvent_Mock(t *testing.T) { + mod := mkModule("Ops") + existing := mkScheduledEvent(mod.ID, "Nightly", &model.Schedule{Kind: model.ScheduleDay}) + h := mkHierarchy(mod) + withContainer(h, existing.ContainerID, mod.ID) + + var deleted string + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{existing}, nil + }, + DeleteScheduledEventFunc: func(id string) error { + deleted = id + return nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDropScheduledEvent(ctx, &ast.DropScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "Nightly"}, + })) + if deleted != string(existing.ID) { + t.Errorf("deleted %q, want %q", deleted, existing.ID) + } + assertContainsStr(t, buf.String(), "Dropped scheduled event: Ops.Nightly") +} + +// TestDescribeScheduledEvent_Mock_RoundTrips checks that DESCRIBE emits exactly +// the fields the repeat has, and nothing from another variant. +func TestDescribeScheduledEvent_Mock_RoundTrips(t *testing.T) { + mod := mkModule("Ops") + ev := mkScheduledEvent(mod.ID, "QuarterEnd", &model.Schedule{ + Kind: model.ScheduleMonthWeekday, Multiplier: 3, MonthOffset: 2, + DaySelector: "Last", Weekday: "Friday", HourOfDay: 18, + }) + h := mkHierarchy(mod) + withContainer(h, ev.ContainerID, mod.ID) + + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{ev}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDescribeScheduledEvent(ctx, &ast.DescribeScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "QuarterEnd"}, + })) + + out := buf.String() + assertContainsStr(t, out, "create or modify scheduled event Ops.QuarterEnd (") + assertContainsStr(t, out, "Repeat: MonthlyByWeekday,") + assertContainsStr(t, out, "DaySelector: Last,") + assertContainsStr(t, out, "Weekday: Friday,") + // Fields of other variants must not appear, or the output would not + // re-execute: the executor refuses a field the repeat does not have. + assertNotContainsStr(t, out, "DayOfMonth") + assertNotContainsStr(t, out, "Weekdays:") + assertNotContainsStr(t, out, "MinuteOffset") +} + +// TestDescribeScheduledEvent_Mock_OutputParses is the real round-trip proof: +// asserting on strings passes against a formatter emitting something nothing +// can read. +func TestDescribeScheduledEvent_Mock_OutputParses(t *testing.T) { + mod := mkModule("Ops") + for _, sched := range []*model.Schedule{ + {Kind: model.ScheduleMinute, Multiplier: 5}, + {Kind: model.ScheduleHour, Multiplier: 2, MinuteOffset: 23}, + {Kind: model.ScheduleDay, HourOfDay: 4}, + {Kind: model.ScheduleWeek, Weekdays: [7]bool{false, true, false, false, false, true, false}, HourOfDay: 9}, + {Kind: model.ScheduleMonthDate, Multiplier: 1, DayOfMonth: 15}, + {Kind: model.ScheduleMonthWeekday, Multiplier: 3, DaySelector: "Last", Weekday: "Friday"}, + {Kind: model.ScheduleYearDate, Month: 1, DayOfMonth: 2}, + {Kind: model.ScheduleYearWeekday, Month: 3, DaySelector: "First", Weekday: "Monday"}, + } { + t.Run(string(sched.Kind), func(t *testing.T) { + ev := mkScheduledEvent(mod.ID, "E", sched) + h := mkHierarchy(mod) + withContainer(h, ev.ContainerID, mod.ID) + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListScheduledEventsFunc: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{ev}, nil + }, + } + ctx, buf := newMockCtx(t, withBackend(mb), withHierarchy(h)) + assertNoError(t, execDescribeScheduledEvent(ctx, &ast.DescribeScheduledEventStmt{ + Name: ast.QualifiedName{Module: "Ops", Name: "E"}, + })) + + prog, errs := visitor.Build(buf.String()) + if len(errs) > 0 { + t.Fatalf("describe output does not parse: %v\n%s", errs, buf.String()) + } + stmt, ok := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if !ok { + t.Fatalf("statement 0 = %T", prog.Statements[0]) + } + // And the parsed statement must be accepted by the same validation + // that guards a hand-written one. + back, err := scheduleFromStmt(stmt) + if err != nil { + t.Fatalf("describe output is refused on re-execution: %v\n%s", err, buf.String()) + } + if back.Kind != sched.Kind { + t.Errorf("round-tripped kind = %v, want %v", back.Kind, sched.Kind) + } + }) + } +} diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index 044f9aead..010c20383 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -222,6 +222,21 @@ func registerQueueHandlers(r *Registry) { }) } +func registerScheduledEventHandlers(r *Registry) { + r.Register(&ast.CreateScheduledEventStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execCreateScheduledEvent(ctx, stmt.(*ast.CreateScheduledEventStmt)) + }) + r.Register(&ast.DropScheduledEventStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDropScheduledEvent(ctx, stmt.(*ast.DropScheduledEventStmt)) + }) + r.Register(&ast.ShowScheduledEventsStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execShowScheduledEvents(ctx, stmt.(*ast.ShowScheduledEventsStmt)) + }) + r.Register(&ast.DescribeScheduledEventStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execDescribeScheduledEvent(ctx, stmt.(*ast.DescribeScheduledEventStmt)) + }) +} + func registerImageHandlers(r *Registry) { r.Register(&ast.CreateImageCollectionStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execCreateImageCollection(ctx, stmt.(*ast.CreateImageCollectionStmt)) diff --git a/mdl/executor/registry.go b/mdl/executor/registry.go index 761a87fc6..7055b6184 100644 --- a/mdl/executor/registry.go +++ b/mdl/executor/registry.go @@ -39,6 +39,7 @@ func NewRegistry() *Registry { registerNavigationHandlers(r) registerImageHandlers(r) registerQueueHandlers(r) + registerScheduledEventHandlers(r) registerWorkflowHandlers(r) registerBusinessEventHandlers(r) registerSettingsHandlers(r) diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 5ec9c72ab..1a777fb2b 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -207,6 +207,7 @@ func allKnownStatements() []ast.Statement { &ast.CreatePageStmtV3{}, &ast.CreatePublishedRestServiceStmt{}, &ast.CreateQueueStmt{}, + &ast.CreateScheduledEventStmt{}, &ast.CreateRestClientStmt{}, &ast.CreateSnippetStmtV3{}, &ast.CreateUserRoleStmt{}, @@ -217,6 +218,7 @@ func allKnownStatements() []ast.Statement { &ast.DescribeContractFromOpenAPIStmt{}, &ast.DescribeFragmentFromStmt{}, &ast.DescribeQueueStmt{}, + &ast.DescribeScheduledEventStmt{}, &ast.DescribeStmt{}, &ast.DescribeStylingStmt{}, &ast.DisconnectStmt{}, @@ -248,6 +250,7 @@ func allKnownStatements() []ast.Statement { &ast.DropPageStmt{}, &ast.DropPublishedRestServiceStmt{}, &ast.DropQueueStmt{}, + &ast.DropScheduledEventStmt{}, &ast.DropRestClientStmt{}, &ast.DropSnippetStmt{}, &ast.DropUserRoleStmt{}, @@ -282,6 +285,7 @@ func allKnownStatements() []ast.Statement { &ast.ShowDesignPropertiesStmt{}, &ast.ShowFeaturesStmt{}, &ast.ShowQueuesStmt{}, + &ast.ShowScheduledEventsStmt{}, &ast.ShowStmt{}, &ast.ShowWidgetsStmt{}, &ast.SQLConnectionsStmt{}, diff --git a/mdl/executor/validate_scheduled_events.go b/mdl/executor/validate_scheduled_events.go new file mode 100644 index 000000000..32c37d90b --- /dev/null +++ b/mdl/executor/validate_scheduled_events.go @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Check-time (no-project) validation for CREATE SCHEDULED EVENT. +// +// The repeat rule is a polymorphic child with eight variants that differ in +// which fields they carry, so most of what can go wrong — a field that belongs +// to a different repeat, an hour of 99, a misspelled weekday — is decidable from +// the statement alone. Running it here means a plain `mxcli check` reports it, +// instead of the script passing check and failing at exec. +package executor + +import ( + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" +) + +// ValidateScheduledEvents reports (MDL-SCHED01) a CREATE SCHEDULED EVENT whose +// properties do not describe a schedule Mendix can store. +// +// It reuses the executor's own builder, so check and exec cannot drift: there is +// one implementation of what a valid statement is, and this pass is the same +// function exec calls before writing. +func ValidateScheduledEvents(prog *ast.Program) []linter.Violation { + var out []linter.Violation + for _, stmt := range prog.Statements { + s, ok := stmt.(*ast.CreateScheduledEventStmt) + if !ok { + continue + } + if _, err := scheduledEventFromStmt(s); err != nil { + out = append(out, linter.Violation{ + RuleID: "MDL-SCHED01", + Severity: linter.SeverityError, + Message: err.Error(), + Suggestion: "Each Repeat takes only its own fields — see `mxcli syntax scheduled-event` " + + "for the field list of each one.", + }) + } + } + return out +} diff --git a/mdl/grammar/MDLLexer.g4 b/mdl/grammar/MDLLexer.g4 index 87fe9226c..ea8995d3c 100644 --- a/mdl/grammar/MDLLexer.g4 +++ b/mdl/grammar/MDLLexer.g4 @@ -356,6 +356,7 @@ FILTERTYPE: F I L T E R T Y P E; IMAGE: I M A G E; QUEUE: Q U E U E; QUEUES: Q U E U E S; +SCHEDULED: S C H E D U L E D; COLLECTION: C O L L E C T I O N S?; // accept singular + plural ("collection(s)") JAR: J A R; DEPENDENCY: D E P E N D E N C Y; diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index fe263a5dc..f1309162d 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -116,6 +116,7 @@ createStatement | createDemoUserStatement | createImageCollectionStatement | createQueueStatement + | createScheduledEventStatement | createJsonStructureStatement | createImportMappingStatement | createExportMappingStatement @@ -311,6 +312,7 @@ dropStatement | DROP MODULE qualifiedName | DROP NOTEBOOK qualifiedName | DROP QUEUE qualifiedName + | DROP SCHEDULED EVENT qualifiedName | DROP JAVA ACTION qualifiedName | DROP JAVASCRIPT ACTION qualifiedName | DROP INDEX qualifiedName ON qualifiedName diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index c747df1a7..0ed72c210 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -36,6 +36,7 @@ showStatement | showOrList LAYOUTS (IN (qualifiedName | IDENTIFIER))? | showOrList NOTEBOOKS (IN (qualifiedName | IDENTIFIER))? | showOrList QUEUES (IN (qualifiedName | IDENTIFIER))? + | showOrList SCHEDULED EVENTS (IN (qualifiedName | IDENTIFIER))? | showOrList JAVA ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList JAVASCRIPT ACTIONS (IN (qualifiedName | IDENTIFIER))? | showOrList IMAGE COLLECTION (IN (qualifiedName | IDENTIFIER))? @@ -164,6 +165,7 @@ describeStatement | DESCRIBE CATALOG DOT (catalogTableName) // DESCRIBE CATALOG.ENTITIES | DESCRIBE BUSINESS EVENT SERVICE qualifiedName // DESCRIBE BUSINESS EVENT SERVICE Module.Name | DESCRIBE QUEUE qualifiedName // DESCRIBE QUEUE Module.Name + | DESCRIBE SCHEDULED EVENT qualifiedName // DESCRIBE SCHEDULED EVENT Module.Name | DESCRIBE DATABASE CONNECTION qualifiedName // DESCRIBE DATABASE CONNECTION Module.Name | DESCRIBE SETTINGS (CONFIGURATION STRING_LITERAL)? // DESCRIBE SETTINGS [CONFIGURATION 'Default'] | DESCRIBE FRAGMENT FROM PAGE qualifiedName WIDGET identifierOrKeyword // DESCRIBE FRAGMENT FROM PAGE Module.Page WIDGET name diff --git a/mdl/grammar/domains/MDLDomainModel.g4 b/mdl/grammar/domains/MDLDomainModel.g4 index 3ad98dfc5..c86129b2a 100644 --- a/mdl/grammar/domains/MDLDomainModel.g4 +++ b/mdl/grammar/domains/MDLDomainModel.g4 @@ -354,6 +354,28 @@ queueProperty : identifierOrKeyword COLON (NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | identifierOrKeyword) ; +// ============================================================================= +// SCHEDULED EVENT CREATION +// ============================================================================= +// +// The repeat rule is a property (Repeat: Daily) plus the fields that rule uses, +// rather than an English clause, because the eight ScheduledEvents$*Schedule +// variants differ in WHICH fields they carry — a labelled property list keeps +// the storage's own vocabulary and lets the executor reject a field that does +// not belong to the chosen repeat. + +createScheduledEventStatement + : SCHEDULED EVENT qualifiedName scheduledEventBody? + ; + +scheduledEventBody + : LPAREN (scheduledEventProperty (COMMA scheduledEventProperty)* COMMA?)? RPAREN + ; + +scheduledEventProperty + : identifierOrKeyword COLON (qualifiedName | NUMBER_LITERAL | STRING_LITERAL | booleanLiteral | identifierOrKeyword) + ; + // ============================================================================= // IMAGE COLLECTION CREATION // ============================================================================= diff --git a/mdl/grammar/domains/MDLSettings.g4 b/mdl/grammar/domains/MDLSettings.g4 index a18faa5bb..b12db73a8 100644 --- a/mdl/grammar/domains/MDLSettings.g4 +++ b/mdl/grammar/domains/MDLSettings.g4 @@ -624,7 +624,7 @@ keyword | UNLOCK | UNPAUSE | WAIT | WORKFLOW | WORKFLOWS // Business events / settings - | BUSINESS | CONFIGURATION | EVENT | EVENTS | HANDLER | SETTINGS | SUBSCRIBE + | BUSINESS | CONFIGURATION | EVENT | EVENTS | HANDLER | SCHEDULED | SETTINGS | SUBSCRIBE // Code search / analysis | BACKGROUND | CALLERS | CALLEES | DEPTH | IMPACT | REFERENCES diff --git a/mdl/scheduledevents/codec.go b/mdl/scheduledevents/codec.go new file mode 100644 index 000000000..95a353731 --- /dev/null +++ b/mdl/scheduledevents/codec.go @@ -0,0 +1,295 @@ +// SPDX-License-Identifier: Apache-2.0 + +// Package scheduledevents holds the BSON codec for scheduled events +// (ScheduledEvents$ScheduledEvent) and their polymorphic Schedule child. +// +// It lives outside both engines because both write this document, and the +// schedule has eight variants that differ in which fields they carry — a shape +// worth getting right once. The package depends only on the model types and the +// BSON driver, so either backend can call it. +package scheduledevents + +import ( + "fmt" + "time" + + "go.mongodb.org/mongo-driver/bson" + "go.mongodb.org/mongo-driver/bson/primitive" + + "github.com/mendixlabs/mxcli/mdl/bsonutil" + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/model" +) + +// TypeName is the BSON storage name for a scheduled event document. +const TypeName = "ScheduledEvents$ScheduledEvent" + +// ScheduleTypeNames maps a schedule kind to its BSON storage name. The variants +// differ in arity, not just in values, so the kind is dispatched before any +// field is read or written — see Serialize and ParseSchedule. +var ScheduleTypeNames = map[model.ScheduleKind]string{ + model.ScheduleMinute: "ScheduledEvents$MinuteSchedule", + model.ScheduleHour: "ScheduledEvents$HourSchedule", + model.ScheduleDay: "ScheduledEvents$DaySchedule", + model.ScheduleWeek: "ScheduledEvents$WeekSchedule", + model.ScheduleMonthDate: "ScheduledEvents$MonthDateSchedule", + model.ScheduleMonthWeekday: "ScheduledEvents$MonthWeekdaySchedule", + model.ScheduleYearDate: "ScheduledEvents$YearDateSchedule", + model.ScheduleYearWeekday: "ScheduledEvents$YearWeekdaySchedule", +} + +// WeekdayNames indexes model.Schedule.Weekdays: Sunday first, matching the +// Mendix Weekday enumeration and the BSON property names of WeekSchedule. +var WeekdayNames = [7]string{"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"} + +// Serialize writes the document in the shape Studio Pro produces. +// +// Pinned against four Studio Pro-authored events (Workflow Commons 4.11.0, OIDC +// SSO 4.6.0, SAML 4.2.1 ×2). All four agree on the property set and on three +// types the generated metamodel would lead you to get wrong: +// +// - every integer is BSON int64, while modelsdk/gen declares int32 — the same +// mismatch that made the READER fail on Studio Pro documents in issue #585; +// - StartDateTime is a BSON UTC datetime, while gen declares a string; +// - Microflow is a by-name reference (the qualified name), not an element ID. +// +// Interval/IntervalType are legacy siblings of Schedule that Studio Pro still +// writes and does NOT keep in sync — the Workflow Commons event stores +// Interval 0 / "Minute" next to a DaySchedule of 01:00. They are therefore +// carried through from the caller rather than derived from the schedule, so a +// round trip cannot invent a value Mendix did not have. +func Serialize(ev *model.ScheduledEvent) ([]byte, error) { + exportLevel := ev.ExportLevel + if exportLevel == "" { + exportLevel = "Hidden" + } + onOverlap := ev.OnOverlap + if onOverlap == "" { + // Every reference document uses DelayNext, and it is the safer default: + // SkipNext silently drops a run. + onOverlap = "DelayNext" + } + timeZone := ev.TimeZone + if timeZone == "" { + timeZone = "UTC" + } + var start time.Time + if ev.StartDateTime != nil { + start = ev.StartDateTime.UTC() + } + + doc := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(string(ev.ID))}, + {Key: "$Type", Value: TypeName}, + {Key: "Documentation", Value: ev.Documentation}, + {Key: "Enabled", Value: ev.Enabled}, + {Key: "Excluded", Value: ev.Excluded}, + {Key: "ExportLevel", Value: exportLevel}, + {Key: "Interval", Value: int64(ev.Interval)}, + {Key: "IntervalType", Value: ev.IntervalType}, + {Key: "Microflow", Value: string(ev.MicroflowID)}, + {Key: "Name", Value: ev.Name}, + {Key: "OnOverlap", Value: onOverlap}, + } + if ev.Schedule != nil { + sched, err := SerializeSchedule(ev.Schedule) + if err != nil { + return nil, err + } + doc = append(doc, bson.E{Key: "Schedule", Value: sched}) + } + doc = append(doc, + bson.E{Key: "StartDateTime", Value: primitive.DateTime(start.UnixMilli())}, + bson.E{Key: "TimeZone", Value: timeZone}, + ) + + out, err := bson.Marshal(doc) + if err != nil { + return nil, fmt.Errorf("serialize scheduled event %q: %w", ev.Name, err) + } + return out, nil +} + +// SerializeSchedule writes the ScheduledEvents$*Schedule child. +// +// Each variant writes only the fields its type declares. Writing a field the +// type does not carry is the failure mode that produces a document mxbuild +// accepts and Studio Pro cannot open (System.InvalidOperationException at +// MprProperty), so this dispatches on the kind and never merges field sets. +// Keys are alphabetical within each variant, matching the observed documents. +func SerializeSchedule(s *model.Schedule) (bson.D, error) { + typeName, ok := ScheduleTypeNames[s.Kind] + if !ok { + return nil, fmt.Errorf("unknown schedule kind %q", s.Kind) + } + head := bson.D{ + {Key: "$ID", Value: bsonutil.IDToBsonBinary(types.GenerateID())}, + {Key: "$Type", Value: typeName}, + } + + var fields bson.D + switch s.Kind { + case model.ScheduleMinute: + fields = bson.D{{Key: "Multiplier", Value: int64(s.Multiplier)}} + case model.ScheduleHour: + fields = bson.D{ + {Key: "MinuteOffset", Value: int64(s.MinuteOffset)}, + {Key: "Multiplier", Value: int64(s.Multiplier)}, + } + case model.ScheduleDay: + fields = bson.D{ + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + } + case model.ScheduleWeek: + // Alphabetical, so the seven day flags interleave with the times rather + // than grouping — Friday, HourOfDay, MinuteOfHour, Monday, ... + fields = bson.D{ + {Key: "Friday", Value: s.Weekdays[5]}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "Monday", Value: s.Weekdays[1]}, + {Key: "Saturday", Value: s.Weekdays[6]}, + {Key: "Sunday", Value: s.Weekdays[0]}, + {Key: "Thursday", Value: s.Weekdays[4]}, + {Key: "Tuesday", Value: s.Weekdays[2]}, + {Key: "Wednesday", Value: s.Weekdays[3]}, + } + case model.ScheduleMonthDate: + fields = bson.D{ + {Key: "DayOfMonth", Value: int64(s.DayOfMonth)}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "MonthOffset", Value: int64(s.MonthOffset)}, + {Key: "Multiplier", Value: int64(s.Multiplier)}, + } + case model.ScheduleMonthWeekday: + fields = bson.D{ + {Key: "DaySelector", Value: s.DaySelector}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "MonthOffset", Value: int64(s.MonthOffset)}, + {Key: "Multiplier", Value: int64(s.Multiplier)}, + {Key: "Weekday", Value: s.Weekday}, + } + case model.ScheduleYearDate: + fields = bson.D{ + {Key: "DayOfMonth", Value: int64(s.DayOfMonth)}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "Month", Value: int64(s.Month)}, + } + case model.ScheduleYearWeekday: + fields = bson.D{ + {Key: "DaySelector", Value: s.DaySelector}, + {Key: "HourOfDay", Value: int64(s.HourOfDay)}, + {Key: "MinuteOfHour", Value: int64(s.MinuteOfHour)}, + {Key: "Month", Value: int64(s.Month)}, + {Key: "Weekday", Value: s.Weekday}, + } + } + return append(head, fields...), nil +} + +// Parse converts a stored document to the semantic type. MicroflowID holds the +// by-name microflow reference (BSON "Microflow"), not an element ID. +func Parse(doc bson.M, id, containerID model.ID) *model.ScheduledEvent { + ev := &model.ScheduledEvent{ContainerID: containerID} + ev.ID = id + ev.TypeName = TypeName + ev.Name, _ = doc["Name"].(string) + ev.Documentation, _ = doc["Documentation"].(string) + if mf, ok := doc["Microflow"].(string); ok { + ev.MicroflowID = model.ID(mf) + } + ev.Enabled, _ = doc["Enabled"].(bool) + ev.Excluded, _ = doc["Excluded"].(bool) + ev.ExportLevel, _ = doc["ExportLevel"].(string) + ev.OnOverlap, _ = doc["OnOverlap"].(string) + ev.TimeZone, _ = doc["TimeZone"].(string) + ev.IntervalType, _ = doc["IntervalType"].(string) + ev.Interval = anyInt(doc["Interval"]) + if dt, ok := doc["StartDateTime"].(primitive.DateTime); ok { + t := time.UnixMilli(int64(dt)).UTC() + ev.StartDateTime = &t + } + if sched, ok := doc["Schedule"].(bson.M); ok { + ev.Schedule = ParseSchedule(sched) + } + return ev +} + +// ParseSchedule dispatches on $Type before reading any field: the variants +// differ in arity, so inferring the kind from which keys are present would +// mis-read a schedule whose fields overlap another's. +func ParseSchedule(doc bson.M) *model.Schedule { + typeName, _ := doc["$Type"].(string) + var kind model.ScheduleKind + for k, name := range ScheduleTypeNames { + if name == typeName { + kind = k + break + } + } + if kind == "" { + return nil + } + s := &model.Schedule{Kind: kind} + switch kind { + case model.ScheduleMinute: + s.Multiplier = anyInt(doc["Multiplier"]) + case model.ScheduleHour: + s.Multiplier = anyInt(doc["Multiplier"]) + s.MinuteOffset = anyInt(doc["MinuteOffset"]) + case model.ScheduleDay: + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleWeek: + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + for i, day := range WeekdayNames { + s.Weekdays[i], _ = doc[day].(bool) + } + case model.ScheduleMonthDate: + s.Multiplier = anyInt(doc["Multiplier"]) + s.MonthOffset = anyInt(doc["MonthOffset"]) + s.DayOfMonth = anyInt(doc["DayOfMonth"]) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleMonthWeekday: + s.Multiplier = anyInt(doc["Multiplier"]) + s.MonthOffset = anyInt(doc["MonthOffset"]) + s.DaySelector, _ = doc["DaySelector"].(string) + s.Weekday, _ = doc["Weekday"].(string) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleYearDate: + s.Month = anyInt(doc["Month"]) + s.DayOfMonth = anyInt(doc["DayOfMonth"]) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + case model.ScheduleYearWeekday: + s.Month = anyInt(doc["Month"]) + s.DaySelector, _ = doc["DaySelector"].(string) + s.Weekday, _ = doc["Weekday"].(string) + s.HourOfDay = anyInt(doc["HourOfDay"]) + s.MinuteOfHour = anyInt(doc["MinuteOfHour"]) + } + return s +} + +// anyInt accepts every numeric width a writer might have produced. Studio Pro +// writes int64; older mxcli builds and hand-made fixtures write int32 (#585). +func anyInt(v any) int { + switch n := v.(type) { + case int64: + return int(n) + case int32: + return int(n) + case int: + return n + case float64: + return int(n) + } + return 0 +} diff --git a/mdl/scheduledevents/codec_test.go b/mdl/scheduledevents/codec_test.go new file mode 100644 index 000000000..dce733cdb --- /dev/null +++ b/mdl/scheduledevents/codec_test.go @@ -0,0 +1,314 @@ +// SPDX-License-Identifier: Apache-2.0 + +package scheduledevents + +import ( + "encoding/hex" + "testing" + "time" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/model" +) + +// Three complete, unedited scheduled-event documents lifted out of Mendix's own +// marketplace modules — the only Studio Pro-authored references available +// without Studio Pro. Between them they cover both schedule variants that ship +// in Mendix modules (Day and Hour), both TimeZone values, both Enabled values, +// documented and undocumented, and a zero Interval. +// +// oidcHex OIDC SSO 4.6.0 CleanupOldAuthAttempts HourSchedule +// samlHex SAML 4.2.1 SE_LogCleanUp DaySchedule, TimeZone Server +// wfcHex Workflow Commons 4.11.0 SE_WorkflowAuditTrailRecord… DaySchedule, Interval 0 +const ( + oidcHex = "c00100000524494400100000000059533bc5d678bb40835cd6028304521f022454797065001f0000005363686564756c65644576656e7473245363686564756c65644576656e740002446f63756d656e746174696f6e00010000000008456e61626c65640001084578636c756465640000024578706f72744c6576656c000700000048696464656e0012496e74657276616c00010000000000000002496e74657276616c547970650005000000486f757200024d6963726f666c6f7700200000004f4944432e5355425f436c65616e75704f6c6441757468417474656d70747300024e616d650017000000436c65616e75704f6c6441757468417474656d70747300024f6e4f7665726c6170000a00000044656c61794e65787400035363686564756c65007100000005244944001000000000c154dede71230641b986faef8e304829022454797065001d0000005363686564756c65644576656e747324486f75725363686564756c6500124d696e7574654f6666736574001700000000000000124d756c7469706c696572000100000000000000000953746172744461746554696d650058a17738730100000254696d655a6f6e6500040000005554430000" + samlHex = "63020000052449440010000000007e890ddd9ec5a444b417ef5992c27101022454797065001f0000005363686564756c65644576656e7473245363686564756c65644576656e740002446f63756d656e746174696f6e00b80000005468697320616374696f6e2077696c6c20636c65616e757020746865206c6f676c696e65732e20546865206c6f677320656e74726965732077696c6c20626520617263686976656420666f722074686520616d6f756e74206f6620646179732073706563696669656420696e2074686520636f6e66696775726174696f6e2c20746865206c6f6720656e74726965732074686174207265616368656420746865206c696d69742077696c6c2062652064656c657465642e0008456e61626c65640000084578636c756465640000024578706f72744c6576656c000700000048696464656e0012496e74657276616c00010000000000000002496e74657276616c54797065000400000044617900024d6963726f666c6f77001500000053414d4c32302e53455f4c6f67436c65616e557000024e616d65000e00000053455f4c6f67436c65616e557000024f6e4f7665726c6170000a00000044656c61794e65787400035363686564756c65006f00000005244944001000000000df856dc51b45c24d89325077b18be82b022454797065001c0000005363686564756c65644576656e7473244461795363686564756c650012486f75724f66446179000400000000000000124d696e7574654f66486f7572000000000000000000000953746172744461746554696d6500002ea2b8390100000254696d655a6f6e6500070000005365727665720000" + wfcHex = "e1010000052449440010000000003d7bd7ba3c94a84d816c130543adf825022454797065001f0000005363686564756c65644576656e7473245363686564756c65644576656e740002446f63756d656e746174696f6e00010000000008456e61626c65640000084578636c756465640000024578706f72744c6576656c000700000048696464656e0012496e74657276616c00000000000000000002496e74657276616c5479706500070000004d696e75746500024d6963726f666c6f770034000000576f726b666c6f77436f6d6d6f6e732e53455f576f726b666c6f774175646974547261696c5265636f72645f436c65616e557000024e616d65002400000053455f576f726b666c6f774175646974547261696c5265636f72645f436c65616e557000024f6e4f7665726c6170000a00000044656c61794e65787400035363686564756c65006f0000000524494400100000000081a8e58306a48748bdc93d240d67c64b022454797065001c0000005363686564756c65644576656e7473244461795363686564756c650012486f75724f66446179000100000000000000124d696e7574654f66486f7572000000000000000000000953746172744461746554696d6500db6b4292900100000254696d655a6f6e6500040000005554430000" +) + +func mustDecode(t *testing.T, h string) bson.M { + t.Helper() + raw, err := hex.DecodeString(h) + if err != nil { + t.Fatalf("hex: %v", err) + } + var doc bson.M + if err := bson.Unmarshal(raw, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + return doc +} + +// TestParseStudioProDocuments checks the reader against the real documents, +// including the two properties the generated metamodel gets wrong: every +// integer is int64 (gen says int32) and StartDateTime is a BSON datetime (gen +// says string). +func TestParseStudioProDocuments(t *testing.T) { + tests := []struct { + name string + hexDoc string + want model.ScheduledEvent + wantSched model.Schedule + }{ + { + name: "OIDC CleanupOldAuthAttempts", + hexDoc: oidcHex, + want: model.ScheduledEvent{ + Name: "CleanupOldAuthAttempts", MicroflowID: "OIDC.SUB_CleanupOldAuthAttempts", + Enabled: true, Interval: 1, IntervalType: "Hour", + OnOverlap: "DelayNext", TimeZone: "UTC", ExportLevel: "Hidden", + }, + wantSched: model.Schedule{Kind: model.ScheduleHour, Multiplier: 1, MinuteOffset: 23}, + }, + { + name: "SAML SE_LogCleanUp", + hexDoc: samlHex, + want: model.ScheduledEvent{ + Name: "SE_LogCleanUp", MicroflowID: "SAML20.SE_LogCleanUp", + Enabled: false, Interval: 1, IntervalType: "Day", + OnOverlap: "DelayNext", TimeZone: "Server", ExportLevel: "Hidden", + }, + wantSched: model.Schedule{Kind: model.ScheduleDay, HourOfDay: 4, MinuteOfHour: 0}, + }, + { + name: "Workflow Commons cleanup", + hexDoc: wfcHex, + want: model.ScheduledEvent{ + Name: "SE_WorkflowAuditTrailRecord_CleanUp", + // Interval 0 / "Minute" alongside a DaySchedule of 01:00: Studio + // Pro does not keep the legacy pair in sync with Schedule, which + // is why the writer carries them through instead of deriving them. + MicroflowID: "WorkflowCommons.SE_WorkflowAuditTrailRecord_CleanUp", + Enabled: false, Interval: 0, IntervalType: "Minute", + OnOverlap: "DelayNext", TimeZone: "UTC", ExportLevel: "Hidden", + }, + wantSched: model.Schedule{Kind: model.ScheduleDay, HourOfDay: 1, MinuteOfHour: 0}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ev := Parse(mustDecode(t, tt.hexDoc), "id-1", "mod-1") + if ev.Name != tt.want.Name { + t.Errorf("Name = %q, want %q", ev.Name, tt.want.Name) + } + if ev.MicroflowID != tt.want.MicroflowID { + t.Errorf("MicroflowID = %q, want %q", ev.MicroflowID, tt.want.MicroflowID) + } + if ev.Enabled != tt.want.Enabled { + t.Errorf("Enabled = %v", ev.Enabled) + } + if ev.Interval != tt.want.Interval { + t.Errorf("Interval = %d, want %d (stored as int64)", ev.Interval, tt.want.Interval) + } + if ev.IntervalType != tt.want.IntervalType { + t.Errorf("IntervalType = %q", ev.IntervalType) + } + if ev.OnOverlap != tt.want.OnOverlap { + t.Errorf("OnOverlap = %q", ev.OnOverlap) + } + if ev.TimeZone != tt.want.TimeZone { + t.Errorf("TimeZone = %q", ev.TimeZone) + } + if ev.ExportLevel != tt.want.ExportLevel { + t.Errorf("ExportLevel = %q", ev.ExportLevel) + } + if ev.StartDateTime == nil { + t.Error("StartDateTime not read — it is a BSON datetime, not the string gen declares") + } + if ev.Schedule == nil { + t.Fatal("Schedule not read") + } + if *ev.Schedule != tt.wantSched { + t.Errorf("Schedule = %+v, want %+v", *ev.Schedule, tt.wantSched) + } + }) + } +} + +// TestSerializeMatchesStudioProDocuments is the shape pin: read each real +// document, write it back, and require the result to be identical apart from +// the two generated $ID values. It compares raw BSON elements, so key ORDER and +// BSON TYPE (int64 vs int32, datetime vs string) are covered as well as values — +// none of which a field-by-field assertion would catch. +func TestSerializeMatchesStudioProDocuments(t *testing.T) { + for _, tt := range []struct{ name, hexDoc string }{ + {"OIDC", oidcHex}, + {"SAML", samlHex}, + {"WorkflowCommons", wfcHex}, + } { + t.Run(tt.name, func(t *testing.T) { + originalBytes, err := hex.DecodeString(tt.hexDoc) + if err != nil { + t.Fatalf("hex: %v", err) + } + ev := Parse(mustDecode(t, tt.hexDoc), "11111111-1111-1111-1111-111111111111", "mod-1") + + out, err := Serialize(ev) + if err != nil { + t.Fatalf("Serialize: %v", err) + } + assertSameExceptIDs(t, "", bson.Raw(originalBytes), bson.Raw(out)) + }) + } +} + +// assertSameExceptIDs compares two raw documents element by element, in order, +// requiring the same keys in the same positions with the same BSON types and +// values. $ID is compared for presence only: it is regenerated on every write. +func assertSameExceptIDs(t *testing.T, path string, want, got bson.Raw) { + t.Helper() + wantEls, err := want.Elements() + if err != nil { + t.Fatalf("%s: elements: %v", path, err) + } + gotEls, err := got.Elements() + if err != nil { + t.Fatalf("%s: elements: %v", path, err) + } + if len(wantEls) != len(gotEls) { + t.Fatalf("%s: wrote %d properties, Studio Pro wrote %d\n want %v\n got %v", + path, len(gotEls), len(wantEls), keysOf(wantEls), keysOf(gotEls)) + } + for i, we := range wantEls { + ge := gotEls[i] + if we.Key() != ge.Key() { + t.Errorf("%sproperty %d is %q, want %q (order differs)", path, i, ge.Key(), we.Key()) + continue + } + wv, gv := we.Value(), ge.Value() + if wv.Type != gv.Type { + t.Errorf("%s%s: BSON type %s, want %s", path, we.Key(), gv.Type, wv.Type) + continue + } + if we.Key() == "$ID" { + continue + } + if wv.Type == bson.TypeEmbeddedDocument { + assertSameExceptIDs(t, path+we.Key()+".", wv.Document(), gv.Document()) + continue + } + if !wv.Equal(gv) { + t.Errorf("%s%s = %v, want %v", path, we.Key(), gv, wv) + } + } +} + +func keysOf(els []bson.RawElement) []string { + out := make([]string, len(els)) + for i, e := range els { + out[i] = e.Key() + } + return out +} + +// TestSerializeSchedule_AllVariants covers the six variants no Mendix module +// ships, so the field sets are metamodel-derived rather than observed. The +// assertion that matters is that each variant writes ONLY its own fields: a +// merged field set is the shape that mxbuild accepts and Studio Pro refuses to +// open. +func TestSerializeSchedule_AllVariants(t *testing.T) { + full := model.Schedule{ + Multiplier: 2, MinuteOffset: 5, MonthOffset: 1, + HourOfDay: 6, MinuteOfHour: 30, DayOfMonth: 15, Month: 3, + DaySelector: "Last", Weekday: "Friday", + Weekdays: [7]bool{false, true, false, false, false, true, false}, + } + wantKeys := map[model.ScheduleKind][]string{ + model.ScheduleMinute: {"Multiplier"}, + model.ScheduleHour: {"MinuteOffset", "Multiplier"}, + model.ScheduleDay: {"HourOfDay", "MinuteOfHour"}, + model.ScheduleWeek: {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthDate: {"Multiplier", "MonthOffset", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleMonthWeekday: {"Multiplier", "MonthOffset", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearDate: {"Month", "DayOfMonth", "HourOfDay", "MinuteOfHour"}, + model.ScheduleYearWeekday: {"Month", "DaySelector", "Weekday", "HourOfDay", "MinuteOfHour"}, + } + + for kind, keys := range wantKeys { + t.Run(string(kind), func(t *testing.T) { + s := full + s.Kind = kind + doc, err := SerializeSchedule(&s) + if err != nil { + t.Fatalf("SerializeSchedule: %v", err) + } + got := map[string]bool{} + for _, e := range doc { + got[e.Key] = true + } + if !got["$ID"] || !got["$Type"] { + t.Error("missing $ID/$Type") + } + delete(got, "$ID") + delete(got, "$Type") + + for _, k := range keys { + if !got[k] { + t.Errorf("missing %s", k) + } + delete(got, k) + } + for k := range got { + t.Errorf("%s is not a property of %s — writing it produces a document Studio Pro cannot open", k, ScheduleTypeNames[kind]) + } + }) + } +} + +func TestSerializeSchedule_UnknownKindIsRefused(t *testing.T) { + if _, err := SerializeSchedule(&model.Schedule{Kind: "Fortnightly"}); err == nil { + t.Fatal("expected an error for an unknown schedule kind") + } +} + +// TestSerialize_Defaults documents the values supplied when the caller leaves +// them empty, all taken from the reference documents. +func TestSerialize_Defaults(t *testing.T) { + ev := &model.ScheduledEvent{Name: "SE_Plain", MicroflowID: "M.MF"} + ev.ID = "22222222-2222-2222-2222-222222222222" + + out, err := Serialize(ev) + if err != nil { + t.Fatalf("Serialize: %v", err) + } + var doc bson.M + if err := bson.Unmarshal(out, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if doc["ExportLevel"] != "Hidden" { + t.Errorf("ExportLevel = %v, want Hidden", doc["ExportLevel"]) + } + if doc["OnOverlap"] != "DelayNext" { + t.Errorf("OnOverlap = %v, want DelayNext (SkipNext silently drops a run)", doc["OnOverlap"]) + } + if doc["TimeZone"] != "UTC" { + t.Errorf("TimeZone = %v, want UTC", doc["TimeZone"]) + } + if _, ok := doc["Interval"].(int64); !ok { + t.Errorf("Interval = %#v, want an int64 — Studio Pro writes int64, gen declares int32", doc["Interval"]) + } + if _, ok := doc["Schedule"]; ok { + t.Error("a nil schedule must not be written as an empty child") + } +} + +func TestSerialize_StartDateTimeRoundTrips(t *testing.T) { + want := time.Date(2024, 7, 8, 12, 12, 24, 923_000_000, time.UTC) + ev := &model.ScheduledEvent{Name: "SE", MicroflowID: "M.MF", StartDateTime: &want} + ev.ID = "33333333-3333-3333-3333-333333333333" + + out, err := Serialize(ev) + if err != nil { + t.Fatalf("Serialize: %v", err) + } + var doc bson.M + if err := bson.Unmarshal(out, &doc); err != nil { + t.Fatalf("unmarshal: %v", err) + } + got := Parse(doc, "id", "mod") + if got.StartDateTime == nil || !got.StartDateTime.Equal(want) { + t.Errorf("StartDateTime = %v, want %v", got.StartDateTime, want) + } +} diff --git a/mdl/visitor/visitor_entity.go b/mdl/visitor/visitor_entity.go index 24598a7ed..65c870b40 100644 --- a/mdl/visitor/visitor_entity.go +++ b/mdl/visitor/visitor_entity.go @@ -833,6 +833,10 @@ func (b *Builder) ExitDropStatement(ctx *parser.DropStatementContext) { b.statements = append(b.statements, &ast.DropQueueStmt{ Name: buildQualifiedName(names[0]), }) + } else if ctx.SCHEDULED() != nil && ctx.EVENT() != nil { + b.statements = append(b.statements, &ast.DropScheduledEventStmt{ + Name: buildQualifiedName(names[0]), + }) } else if ctx.MODEL() != nil { b.statements = append(b.statements, &ast.DropModelStmt{ Name: buildQualifiedName(names[0]), diff --git a/mdl/visitor/visitor_query.go b/mdl/visitor/visitor_query.go index 596cb5d6d..5a3e4d157 100644 --- a/mdl/visitor/visitor_query.go +++ b/mdl/visitor/visitor_query.go @@ -214,6 +214,16 @@ func (b *Builder) ExitShowStatement(ctx *parser.ShowStatementContext) { } } b.statements = append(b.statements, stmt) + } else if ctx.SCHEDULED() != nil && ctx.EVENTS() != nil { + stmt := &ast.ShowScheduledEventsStmt{} + if ctx.IN() != nil { + if qn := ctx.QualifiedName(); qn != nil { + stmt.Module = getQualifiedNameText(qn) + } else if id := ctx.IDENTIFIER(); id != nil { + stmt.Module = id.GetText() + } + } + b.statements = append(b.statements, stmt) } else if ctx.LAYOUTS() != nil { stmt := &ast.ShowStmt{ObjectType: ast.ShowLayouts} if ctx.IN() != nil { @@ -712,6 +722,14 @@ func (b *Builder) ExitDescribeStatement(ctx *parser.DescribeStatementContext) { return } + // DESCRIBE SCHEDULED EVENT Module.Name + if ctx.SCHEDULED() != nil && ctx.EVENT() != nil { + if qn := ctx.QualifiedName(); qn != nil { + b.statements = append(b.statements, &ast.DescribeScheduledEventStmt{Name: buildQualifiedName(qn)}) + } + return + } + // Handle DESCRIBE MODULE ROLE (uses qualifiedName) if ctx.MODULE() != nil && ctx.ROLE() != nil { if qn := ctx.QualifiedName(); qn != nil { diff --git a/mdl/visitor/visitor_queue.go b/mdl/visitor/visitor_queue.go index 8da08ffea..a7ff58cc1 100644 --- a/mdl/visitor/visitor_queue.go +++ b/mdl/visitor/visitor_queue.go @@ -13,7 +13,7 @@ import ( // CREATE [OR REPLACE|MODIFY] QUEUE Module.Name ( ... ). func (b *Builder) ExitCreateQueueStatement(ctx *parser.CreateQueueStatementContext) { stmt := &ast.CreateQueueStmt{ - Name: buildQualifiedName(ctx.QualifiedName()), + Name: buildQualifiedName(ctx.QualifiedName()), Documentation: findDocCommentText(ctx), } if createStmt := findParentCreateStatement(ctx); createStmt != nil { diff --git a/mdl/visitor/visitor_scheduledevent.go b/mdl/visitor/visitor_scheduledevent.go new file mode 100644 index 000000000..5a52640b3 --- /dev/null +++ b/mdl/visitor/visitor_scheduledevent.go @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "strconv" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// ExitCreateScheduledEventStatement builds a CreateScheduledEventStmt from +// CREATE [OR REPLACE|MODIFY] SCHEDULED EVENT Module.Name ( ... ). +// +// Every property is carried through as written; the executor decides which ones +// the chosen Repeat actually has and rejects the rest. Numeric properties are +// stored as pointers so an explicit 0 (a real hour, minute or month offset) is +// distinguishable from an omitted one. +func (b *Builder) ExitCreateScheduledEventStatement(ctx *parser.CreateScheduledEventStatementContext) { + stmt := &ast.CreateScheduledEventStmt{ + Name: buildQualifiedName(ctx.QualifiedName()), + Documentation: findDocCommentText(ctx), + } + if createStmt := findParentCreateStatement(ctx); createStmt != nil { + if createStmt.OR() != nil && (createStmt.MODIFY() != nil || createStmt.REPLACE() != nil) { + stmt.CreateOrModify = true + } + } + + if body := ctx.ScheduledEventBody(); body != nil { + bodyCtx := body.(*parser.ScheduledEventBodyContext) + for _, prop := range bodyCtx.AllScheduledEventProperty() { + pc, ok := prop.(*parser.ScheduledEventPropertyContext) + if !ok || pc == nil { + continue + } + iok := pc.IdentifierOrKeyword(0) + if iok == nil { + continue + } + key := strings.ToLower(identifierOrKeywordText(iok)) + val := scheduledEventPropertyText(pc) + switch key { + case "microflow": + stmt.Microflow = val + case "repeat": + stmt.Repeat = val + case "multiplier": + stmt.Multiplier = parseIntPtr(val) + case "minuteoffset": + stmt.MinuteOffset = parseIntPtr(val) + case "monthoffset": + stmt.MonthOffset = parseIntPtr(val) + case "hourofday": + stmt.HourOfDay = parseIntPtr(val) + case "minuteofhour": + stmt.MinuteOfHour = parseIntPtr(val) + case "dayofmonth": + stmt.DayOfMonth = parseIntPtr(val) + case "month": + stmt.Month = parseIntPtr(val) + case "weekdays": + stmt.Weekdays = val + case "dayselector": + stmt.DaySelector = val + case "weekday": + stmt.Weekday = val + case "startdatetime": + stmt.StartDateTime = val + case "timezone": + stmt.TimeZone = val + case "onoverlap": + stmt.OnOverlap = val + case "enabled": + stmt.Enabled = parseBoolPtr(val) + case "excluded": + stmt.Excluded = parseBoolPtr(val) + case "exportlevel": + stmt.ExportLevel = val + case "documentation": + stmt.Documentation = val + } + } + } + + b.statements = append(b.statements, stmt) +} + +// scheduledEventPropertyText returns the value side of a property, unquoted. +// +// The key is identifierOrKeyword(0), so an identifier value is index 1 — +// reading index 0 would echo the key back as the value. +func scheduledEventPropertyText(pc *parser.ScheduledEventPropertyContext) string { + if qn := pc.QualifiedName(); qn != nil { + return getQualifiedNameText(qn) + } + if n := pc.NUMBER_LITERAL(); n != nil { + return n.GetText() + } + if s := pc.STRING_LITERAL(); s != nil { + return unquoteString(s.GetText()) + } + if bl := pc.BooleanLiteral(); bl != nil { + return bl.GetText() + } + if v := pc.IdentifierOrKeyword(1); v != nil { + return identifierOrKeywordText(v) + } + return "" +} + +// parseIntPtr returns nil for anything that is not an integer, so a malformed +// value is reported by the executor's validation rather than silently becoming 0. +func parseIntPtr(s string) *int { + n, err := strconv.Atoi(strings.TrimSpace(s)) + if err != nil { + return nil + } + return &n +} + +func parseBoolPtr(s string) *bool { + switch strings.ToLower(strings.TrimSpace(s)) { + case "true": + v := true + return &v + case "false": + v := false + return &v + } + return nil +} diff --git a/mdl/visitor/visitor_scheduledevent_test.go b/mdl/visitor/visitor_scheduledevent_test.go new file mode 100644 index 000000000..0a091ed75 --- /dev/null +++ b/mdl/visitor/visitor_scheduledevent_test.go @@ -0,0 +1,150 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +func TestCreateScheduledEvent(t *testing.T) { + input := `CREATE SCHEDULED EVENT Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true, + StartDateTime: '2026-01-01T04:00:00Z' + );` + prog, errs := Build(input) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if !ok { + t.Fatalf("expected CreateScheduledEventStmt, got %T", prog.Statements[0]) + } + if stmt.Name.String() != "Ops.NightlyCleanup" { + t.Errorf("Name = %s", stmt.Name.String()) + } + if stmt.Microflow != "Ops.SE_Cleanup" { + t.Errorf("Microflow = %q — a qualified name must survive as one", stmt.Microflow) + } + if stmt.Repeat != "Daily" { + t.Errorf("Repeat = %q", stmt.Repeat) + } + if stmt.HourOfDay == nil || *stmt.HourOfDay != 4 { + t.Errorf("HourOfDay = %v", stmt.HourOfDay) + } + // 0 is a real minute, so an explicit 0 must not be indistinguishable from + // an omitted property — that is why these fields are pointers. + if stmt.MinuteOfHour == nil || *stmt.MinuteOfHour != 0 { + t.Errorf("MinuteOfHour = %v, want an explicit 0", stmt.MinuteOfHour) + } + if stmt.TimeZone != "Server" { + t.Errorf("TimeZone = %q", stmt.TimeZone) + } + if stmt.Enabled == nil || !*stmt.Enabled { + t.Errorf("Enabled = %v", stmt.Enabled) + } + if stmt.StartDateTime != "2026-01-01T04:00:00Z" { + t.Errorf("StartDateTime = %q", stmt.StartDateTime) + } +} + +func TestCreateScheduledEvent_OmittedFieldsStayNil(t *testing.T) { + prog, errs := Build(`CREATE SCHEDULED EVENT Ops.E ( Microflow: Ops.MF, Repeat: Minutely, Multiplier: 5 );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if stmt.Multiplier == nil || *stmt.Multiplier != 5 { + t.Errorf("Multiplier = %v", stmt.Multiplier) + } + if stmt.HourOfDay != nil { + t.Errorf("HourOfDay = %v, want nil for an omitted property", stmt.HourOfDay) + } + if stmt.Enabled != nil { + t.Errorf("Enabled = %v, want nil for an omitted property", stmt.Enabled) + } +} + +func TestCreateScheduledEvent_WeeklyAndSelectors(t *testing.T) { + prog, errs := Build(`CREATE OR MODIFY SCHEDULED EVENT Ops.E ( + Microflow: Ops.MF, + Repeat: MonthlyByWeekday, + Multiplier: 3, + DaySelector: Last, + Weekday: Friday, + Weekdays: 'Monday, Friday' + );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateScheduledEventStmt) + if !stmt.CreateOrModify { + t.Error("CreateOrModify not set for CREATE OR MODIFY") + } + if stmt.DaySelector != "Last" || stmt.Weekday != "Friday" { + t.Errorf("DaySelector/Weekday = %q/%q", stmt.DaySelector, stmt.Weekday) + } + if stmt.Weekdays != "Monday, Friday" { + t.Errorf("Weekdays = %q", stmt.Weekdays) + } +} + +func TestDropScheduledEvent(t *testing.T) { + prog, errs := Build(`DROP SCHEDULED EVENT Ops.NightlyCleanup;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt, ok := prog.Statements[0].(*ast.DropScheduledEventStmt) + if !ok { + t.Fatalf("expected DropScheduledEventStmt, got %T", prog.Statements[0]) + } + if stmt.Name.String() != "Ops.NightlyCleanup" { + t.Errorf("Name = %s", stmt.Name.String()) + } +} + +func TestShowAndDescribeScheduledEvents(t *testing.T) { + prog, errs := Build(`SHOW SCHEDULED EVENTS; LIST SCHEDULED EVENTS IN Ops; DESCRIBE SCHEDULED EVENT Ops.NightlyCleanup;`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + if len(prog.Statements) != 3 { + t.Fatalf("expected 3 statements, got %d", len(prog.Statements)) + } + if _, ok := prog.Statements[0].(*ast.ShowScheduledEventsStmt); !ok { + t.Errorf("statement 0 = %T", prog.Statements[0]) + } + s1, ok := prog.Statements[1].(*ast.ShowScheduledEventsStmt) + if !ok { + t.Fatalf("statement 1 = %T", prog.Statements[1]) + } + if s1.Module != "Ops" { + t.Errorf("Module = %q", s1.Module) + } + if _, ok := prog.Statements[2].(*ast.DescribeScheduledEventStmt); !ok { + t.Errorf("statement 2 = %T", prog.Statements[2]) + } +} + +// TestScheduledKeywordStillUsableAsIdentifier guards the cost of adding +// SCHEDULED to the lexer: a new keyword stops being usable as an ordinary name +// unless it is also listed in the `keyword` rule. +func TestScheduledKeywordStillUsableAsIdentifier(t *testing.T) { + prog, errs := Build(`CREATE ENTITY Ops.Job ( scheduled: Boolean, event: String(50) );`) + if len(errs) > 0 { + t.Fatalf("parse errors: %v", errs) + } + stmt := prog.Statements[0].(*ast.CreateEntityStmt) + if len(stmt.Attributes) != 2 { + t.Fatalf("expected 2 attributes, got %d", len(stmt.Attributes)) + } + if stmt.Attributes[0].Name != "scheduled" || stmt.Attributes[1].Name != "event" { + t.Errorf("attribute names = %q, %q", stmt.Attributes[0].Name, stmt.Attributes[1].Name) + } +} diff --git a/model/types.go b/model/types.go index bdfde6ad1..908ec38a6 100644 --- a/model/types.go +++ b/model/types.go @@ -244,6 +244,63 @@ func (r *RegularExpression) GetContainerID() ID { return r.ContainerID } +// ScheduleKind identifies which ScheduledEvents$*Schedule variant a schedule is. +// The variants differ in which fields they carry, so the kind has to be decided +// before any field is read or written — see Schedule. +type ScheduleKind string + +const ( + ScheduleMinute ScheduleKind = "Minute" + ScheduleHour ScheduleKind = "Hour" + ScheduleDay ScheduleKind = "Day" + ScheduleWeek ScheduleKind = "Week" + ScheduleMonthDate ScheduleKind = "MonthDate" + ScheduleMonthWeekday ScheduleKind = "MonthWeekday" + ScheduleYearDate ScheduleKind = "YearDate" + ScheduleYearWeekday ScheduleKind = "YearWeekday" +) + +// Schedule is the repeat rule of a scheduled event — the polymorphic +// ScheduledEvents$Schedule child, flattened into one struct with Kind saying +// which variant it is. +// +// It is flat rather than eight types because the fields overlap heavily and the +// consumers (a formatter, a serializer, a describe) all switch on the kind +// anyway. Only the fields listed for a kind are written; the rest are ignored: +// +// Minute Multiplier +// Hour Multiplier, MinuteOffset +// Day HourOfDay, MinuteOfHour +// Week Weekdays, HourOfDay, MinuteOfHour +// MonthDate Multiplier, MonthOffset, DayOfMonth, HourOfDay, MinuteOfHour +// MonthWeekday Multiplier, MonthOffset, DaySelector, Weekday, HourOfDay, MinuteOfHour +// YearDate Month, DayOfMonth, HourOfDay, MinuteOfHour +// YearWeekday Month, DaySelector, Weekday, HourOfDay, MinuteOfHour +type Schedule struct { + Kind ScheduleKind `json:"kind"` + + // Multiplier is the repeat count ("every N minutes/hours/months"). + // Day, Week and the two Year variants have no multiplier in the metamodel. + Multiplier int `json:"multiplier,omitempty"` + // MinuteOffset is the minute within the hour, for the Hour variant only. + MinuteOffset int `json:"minuteOffset,omitempty"` + // MonthOffset selects which month of a multi-month cycle fires. + MonthOffset int `json:"monthOffset,omitempty"` + + HourOfDay int `json:"hourOfDay,omitempty"` + MinuteOfHour int `json:"minuteOfHour,omitempty"` + + // DayOfMonth is 1-31; Month is 1-12. + DayOfMonth int `json:"dayOfMonth,omitempty"` + Month int `json:"month,omitempty"` + + // Weekdays holds the seven per-day flags of the Week variant, Sunday first. + Weekdays [7]bool `json:"weekdays,omitempty"` + // DaySelector is First|Second|Third|Fourth|Last, Weekday is Sunday..Saturday. + DaySelector string `json:"daySelector,omitempty"` + Weekday string `json:"weekday,omitempty"` +} + // ScheduledEvent represents a scheduled event. type ScheduledEvent struct { BaseElement @@ -256,6 +313,15 @@ type ScheduledEvent struct { Interval int `json:"interval,omitempty"` IntervalType string `json:"intervalType,omitempty"` Enabled bool `json:"enabled"` + // Schedule is the repeat rule. Every Studio Pro-authored event carries one; + // nil means the document did not have the child. + Schedule *Schedule `json:"schedule,omitempty"` + // OnOverlap is SkipNext or DelayNext — what happens when a run is still + // going when the next one is due. This is a scheduled event's own + // concurrency control; it does not go through a task queue. + OnOverlap string `json:"onOverlap,omitempty"` + ExportLevel string `json:"exportLevel,omitempty"` + Excluded bool `json:"excluded,omitempty"` } // GetName returns the scheduled event's name. diff --git a/sdk/mpr/parser_enumeration.go b/sdk/mpr/parser_enumeration.go index dfaac3bc9..964ae548e 100644 --- a/sdk/mpr/parser_enumeration.go +++ b/sdk/mpr/parser_enumeration.go @@ -5,6 +5,7 @@ package mpr import ( "fmt" + "github.com/mendixlabs/mxcli/mdl/scheduledevents" "github.com/mendixlabs/mxcli/model" "go.mongodb.org/mongo-driver/bson" @@ -175,38 +176,17 @@ func (r *Reader) parseScheduledEvent(unitID, containerID string, contents []byte return nil, err } - var raw map[string]any + var raw bson.M if err := bson.Unmarshal(contents, &raw); err != nil { return nil, fmt.Errorf("failed to unmarshal BSON: %w", err) } - event := &model.ScheduledEvent{} - event.ID = model.ID(unitID) - event.TypeName = "ScheduledEvents$ScheduledEvent" - event.ContainerID = model.ID(containerID) - - if name, ok := raw["Name"].(string); ok { - event.Name = name - } - if doc, ok := raw["Documentation"].(string); ok { - event.Documentation = doc - } - if mfID, ok := raw["Microflow"].(string); ok { - event.MicroflowID = model.ID(mfID) - } - if enabled, ok := raw["Enabled"].(bool); ok { - event.Enabled = enabled - } - // Issue #585: Studio Pro stores Interval as BSON int64; extractInt - // also accepts int32/int/float64 emitted by other writers. - if _, ok := raw["Interval"]; ok { - event.Interval = extractInt(raw["Interval"]) - } - if intervalType, ok := raw["IntervalType"].(string); ok { - event.IntervalType = intervalType - } - - return event, nil + // Shared with the modelsdk engine so both read the same keys the shared + // writer produces — including the polymorphic Schedule child, which this + // parser used to drop, and StartDateTime, which Studio Pro stores as a BSON + // datetime. (Interval is int64 in Studio Pro documents; the codec accepts + // every numeric width — issue #585.) + return scheduledevents.Parse(raw, model.ID(unitID), model.ID(containerID)), nil } // resolveContents handles MPR v2 external file references. diff --git a/sdk/mpr/scheduledevents.go b/sdk/mpr/scheduledevents.go new file mode 100644 index 000000000..21aaa1fb0 --- /dev/null +++ b/sdk/mpr/scheduledevents.go @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "fmt" + + sched "github.com/mendixlabs/mxcli/mdl/scheduledevents" + "github.com/mendixlabs/mxcli/model" +) + +// Scheduled events (ScheduledEvents$ScheduledEvent) for the legacy engine. +// +// The document shape lives in mdl/scheduledevents so both engines write exactly +// the same bytes — the schedule child has eight variants that differ in which +// fields they carry, and two copies of that dispatch would eventually disagree. +// +// The legacy READER for scheduled events is parseScheduledEvent (in +// parser_enumeration.go), which predates this and covers only the flat legacy +// fields; the write path here round-trips through the shared codec. + +// CreateScheduledEvent inserts a new scheduled event document. +func (w *Writer) CreateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("CreateScheduledEvent: nil event") + } + if ev.ID == "" { + ev.ID = model.ID(generateUUID()) + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return w.insertUnit(string(ev.ID), string(ev.ContainerID), "Documents", sched.TypeName, contents) +} + +// UpdateScheduledEvent rewrites an existing scheduled event in place. +func (w *Writer) UpdateScheduledEvent(ev *model.ScheduledEvent) error { + if ev == nil { + return fmt.Errorf("UpdateScheduledEvent: nil event") + } + contents, err := sched.Serialize(ev) + if err != nil { + return err + } + return w.UpdateRawUnit(string(ev.ID), contents) +} + +// DeleteScheduledEvent removes a scheduled event by ID. +func (w *Writer) DeleteScheduledEvent(id string) error { + return w.deleteUnit(id) +} From e2d31dd01ae113dfda6c69ef4b8beeae078c7911 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 05:16:39 +0000 Subject: [PATCH 6/7] docs(queues,scheduled-events): wire the new document types into every help surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two features were implemented but only half-discoverable. Filling the gaps the CLAUDE.md checklist calls for: - `mxcli describe` did not know either type. Both are now in the type list, the dispatch, the error message, and the live-reader auto-detect map — so `mxcli describe -p app.mpr Ops.NightlyCleanup` resolves without naming a type, as it already does for every other document. - `show structure` counted scheduled events but not queues, so a queue was invisible at every depth. Added to the depth-1 counts, the compact summary and the depth-2/3 listings, and to `project-tree` (which feeds the VS Code tree). - New user-facing skill `.claude/skills/mendix/scheduled-events-and-queues.md`, indexed in the skills README and its LLM loading guide, so `mxcli init` ships it into user projects. It leads with the trap: pick the Repeat first, then use only that repeat's fields. - CLAUDE.md: both features added to the implementation status, and the skill to the read-this-first list. - VS Code extension: tree icons, terminal-link type inference, the describe fallback type list, and the context-menu viewItem pattern. Already in place from the feature commits: `mxcli syntax queue` / `mxcli syntax scheduled-event`, MDL_QUICK_REFERENCE, the docs-site page, the symptom-table rows and the MDL examples. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/skills/mendix/README.md | 4 + .../mendix/scheduled-events-and-queues.md | 186 ++++++++++++++++++ CLAUDE.md | 3 + cmd/mxcli/cmd_describe.go | 42 ++-- cmd/mxcli/project_tree.go | 11 ++ cmd/mxcli/tui/icons.go | 1 + mdl/executor/cmd_structure.go | 49 ++++- vscode-mdl/package.json | 2 +- vscode-mdl/src/extension.ts | 2 +- vscode-mdl/src/projectTreeProvider.ts | 2 + vscode-mdl/src/terminalLinkProvider.ts | 1 + 11 files changed, 283 insertions(+), 20 deletions(-) create mode 100644 .claude/skills/mendix/scheduled-events-and-queues.md diff --git a/.claude/skills/mendix/README.md b/.claude/skills/mendix/README.md index 54cf5c71d..d9d85ec2b 100644 --- a/.claude/skills/mendix/README.md +++ b/.claude/skills/mendix/README.md @@ -23,6 +23,7 @@ Detailed syntax for each MDL document type: | [write-oql-queries.md](write-oql-queries.md) | OQL query syntax | Creating VIEW entities | | [create-page.md](create-page.md) | Page and widget syntax | Creating pages | | [fragments.md](fragments.md) | Fragment (reusable widget group) syntax | Reusing widget patterns across pages | +| [scheduled-events-and-queues.md](scheduled-events-and-queues.md) | Scheduled event (cron) and task queue syntax | Running a microflow on a schedule; bounding background concurrency | ## Patterns (By Use Case) @@ -92,6 +93,9 @@ Load skills based on the task: | "Create export mapping" | `json-structures-and-mappings.md` | | "Map JSON to entities" | `json-structures-and-mappings.md` | | "Seed/populate test data" | `demo-data.md` | +| "Run a microflow nightly / hourly / on a schedule" | `scheduled-events-and-queues.md` | +| "Add a cron job / batch job / recurring task" | `scheduled-events-and-queues.md` | +| "Limit how many background tasks run at once" | `scheduled-events-and-queues.md` | | "Update widget properties" | `bulk-widget-updates.md` | | "Change widgets in bulk" | `bulk-widget-updates.md` | | "Reuse widgets across pages" | `fragments.md` | diff --git a/.claude/skills/mendix/scheduled-events-and-queues.md b/.claude/skills/mendix/scheduled-events-and-queues.md new file mode 100644 index 000000000..a6ebf19e1 --- /dev/null +++ b/.claude/skills/mendix/scheduled-events-and-queues.md @@ -0,0 +1,186 @@ +# Scheduled Events and Task Queues + +## When to Use This Skill + +Use this skill when the user wants to: +- Run a microflow on a schedule ("every night at 4", "hourly", "cron", "batch job") +- Inspect or change an existing scheduled event +- Limit how many background tasks run at once (a task queue) +- Understand why `mxcli` refuses to rewrite a microflow that has a queued call + +**These two features are unrelated.** A scheduled event does **not** go through a +task queue. Its own concurrency control is `OnOverlap`. + +## Scheduled Events + +Mendix's cron: run a microflow on a repeating schedule. + +```sql +-- Inspect +list scheduled events; +list scheduled events in Ops; +describe scheduled event Ops.NightlyCleanup; -- re-executable MDL + +-- Create +create scheduled event Ops.NightlyCleanup ( + Microflow: Ops.SE_Cleanup, + Repeat: Daily, + HourOfDay: 4, + MinuteOfHour: 0, + TimeZone: Server, + Enabled: true +); + +drop scheduled event Ops.NightlyCleanup; +``` + +`Microflow` and `Repeat` are **always required**. `show` is a synonym for `list`. + +### Pick the Repeat first, then use only its fields + +Mendix stores the repeat rule as one of eight types, and they differ in **which +fields they carry** — not just in their values. Naming a field from another +repeat is an error, not a no-op: + +``` +Error: Repeat Daily does not have Multiplier — it takes HourOfDay, MinuteOfHour +``` + +| Repeat | Fields | Means | +|--------|--------|-------| +| `Minutely` | `Multiplier` | every N minutes | +| `Hourly` | `Multiplier`, `MinuteOffset` | every N hours, at :MM past | +| `Daily` | `HourOfDay`, `MinuteOfHour` | every day at HH:MM (**no multiplier**) | +| `Weekly` | `Weekdays`, `HourOfDay`, `MinuteOfHour` | on the named days at HH:MM | +| `MonthlyByDate` | `Multiplier`, `MonthOffset`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | the Dth of every N months | +| `MonthlyByWeekday` | `Multiplier`, `MonthOffset`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the last Friday of every N months | +| `YearlyByDate` | `Month`, `DayOfMonth`, `HourOfDay`, `MinuteOfHour` | every 2 January | +| `YearlyByWeekday` | `Month`, `DaySelector`, `Weekday`, `HourOfDay`, `MinuteOfHour` | the first Monday of March | + +Field values: + +| Field | Value | +|-------|-------| +| `Multiplier` | 1 or more (defaults to 1) | +| `MinuteOffset` | 0–59 | +| `MonthOffset` | 0-based: which month of a multi-month cycle fires | +| `HourOfDay` / `MinuteOfHour` | 0–23 / 0–59 | +| `DayOfMonth` / `Month` | 1–31 / 1–12 | +| `Weekdays` | quoted list: `'Monday, Friday'` (case-insensitive) | +| `DaySelector` | `First`, `Second`, `Third`, `Fourth`, `Last` | +| `Weekday` | `Sunday` … `Saturday` | + +Optional on any repeat: + +| Property | Values | Default | +|----------|--------|---------| +| `Enabled` | `true` / `false` | `false` — **a new event does not run until you enable it** | +| `OnOverlap` | `DelayNext` / `SkipNext` | `DelayNext` | +| `TimeZone` | `UTC` / `Server` | `UTC` | +| `StartDateTime` | RFC 3339, e.g. `'2026-01-01T04:00:00Z'` | none | +| `Documentation` | free text | none | + +`SkipNext` drops a run that would overlap the previous one; `DelayNext` waits. + +### More examples + +```sql +-- Every two hours, 23 minutes past +create scheduled event Ops.HourlyPing ( + Microflow: Ops.SE_Ping, + Repeat: Hourly, + Multiplier: 2, + MinuteOffset: 23 +); + +-- Mondays and Fridays at 09:30 +create scheduled event Ops.WeeklyReport ( + Microflow: Ops.SE_Report, + Repeat: Weekly, + Weekdays: 'Monday, Friday', + HourOfDay: 9, + MinuteOfHour: 30 +); + +-- The last Friday of every third month, 18:00 +create scheduled event Ops.QuarterEnd ( + Microflow: Ops.SE_Close, + Repeat: MonthlyByWeekday, + Multiplier: 3, + MonthOffset: 2, + DaySelector: Last, + Weekday: Friday, + HourOfDay: 18 +); +``` + +## Task Queues + +A task queue bounds how many queued microflow calls run at once. + +```sql +list queues; +describe queue Ops.OrderProcessing; + +create queue Ops.OrderProcessing ( Parallelism: 3, ClusterWide: true ); +create queue Ops.Mail; -- defaults: parallelism 1, per-instance + +create or modify queue Ops.OrderProcessing ( Parallelism: '$MyModule.Workers' ); +drop queue Ops.Mail; +``` + +| Property | Meaning | Default | +|----------|---------|---------| +| `Parallelism` | how many run at once — an **expression**, not a number | `1` | +| `ClusterWide` | `true` = across the cluster, `false` = per runtime instance | `false` | + +Mendix stores parallelism as an expression string, so `3` and `'3'` are the same +thing and an arbitrary expression is legal. + +## Common Mistakes + +| Mistake | Symptom | Fix | +|---------|---------|-----| +| `Multiplier` on a `Daily` repeat | `Repeat Daily does not have Multiplier` | Daily has no multiplier — use `HourOfDay`/`MinuteOfHour`, or switch to `Hourly` | +| Forgetting `Enabled: true` | The event is in the model but never runs | Set `Enabled: true` (the default is false) | +| `TimeZone: server` | `has the wrong casing — Mendix stores it as "Server"` | Use the exact spelling: `Server`, `UTC`, `DelayNext`, `SkipNext`, `Last`, `Friday` | +| `HourOfDay: 24` | `it must be between 0 and 23` | Hours are 0–23; midnight is `0` | +| Expecting a queue to throttle a scheduled event | Nothing changes | They are unrelated — use `OnOverlap` | + +## Rewriting a Microflow with a Queued Call Is Refused + +MDL cannot yet author a *queued call* — the binding lives on the call activity +inside a microflow, not on the queue. So `create or replace|modify microflow` is +refused when the stored microflow has one: + +``` +Error: microflow Ops.ACT_Caller has 1 call(s) bound to a task queue (Ops.MyQueue), +and rewriting it would silently drop that binding +``` + +This is deliberate. Change that microflow in Studio Pro, or remove the task queue +from the call first. Without the refusal the binding was written back as null and +the project then looked *healthier* than before — `mx check` stopped reporting +`CE1613 "The selected task queue no longer exists"`, because the configuration +the error was about had been deleted. + +## Validation Checklist + +Before presenting a script: + +```bash +mxcli check script.mdl # catches wrong-repeat fields (MDL-SCHED01) +mxcli check script.mdl -p app.mpr --references +``` + +- [ ] Every scheduled event has `Microflow` and `Repeat` +- [ ] Only that repeat's fields are used +- [ ] `Enabled: true` if it is meant to run +- [ ] Enum values spelled exactly (`Server`, `DelayNext`, `Last`, `Monday`) +- [ ] The target microflow exists and takes no parameters + +## Related + +- `mxcli syntax scheduled-event`, `mxcli syntax queue` — full syntax reference +- `write-microflows.md` — writing the microflow the event calls +- `project-settings.md` — after-startup / before-shutdown microflows diff --git a/CLAUDE.md b/CLAUDE.md index d1767a6ab..9bcdf0fbe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -603,6 +603,7 @@ Regenerate after modifying `MDLLexer.g4`, `MDLParser.g4`, or any `domains/*.g4` - `.claude/skills/overview-pages.md` - CRUD page patterns - `.claude/skills/master-detail-pages.md` - Master-detail page patterns - `.claude/skills/generate-domain-model.md` - Entity/Association syntax +- `.claude/skills/mendix/scheduled-events-and-queues.md` - **Scheduled events (Mendix's cron) and task queues**: the eight Repeat variants and which fields each one takes, why a queue does NOT throttle a scheduled event, and why rewriting a microflow with a queued call is refused - `.claude/skills/check-syntax.md` - Pre-flight validation checklist - `.claude/skills/organize-project.md` - Folders, MOVE command, project structure conventions - `.claude/skills/manage-security.md` - Security roles, access control, GRANT/REVOKE patterns @@ -668,6 +669,8 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - ALTER WORKFLOW (SET properties, INSERT/DROP/REPLACE activities, outcomes, paths, conditions, boundary events) - CALCULATED BY microflow syntax for calculated attributes - Image collections (SHOW/DESCRIBE/CREATE/DROP) +- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. See `.claude/skills/mendix/scheduled-events-and-queues.md` +- Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005): the rebuild used to write `QueueSettings` back as null, which made `mx check` go from CE1613 to 0 errors by deleting the user's configuration - AI agent documents: Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) - OData contract browsing (SHOW/DESCRIBE CONTRACT ENTITIES/ACTIONS FROM cached $metadata) - AsyncAPI contract browsing (SHOW/DESCRIBE CONTRACT CHANNELS/MESSAGES FROM cached AsyncAPI) diff --git a/cmd/mxcli/cmd_describe.go b/cmd/mxcli/cmd_describe.go index bfeec9091..776c4beb2 100644 --- a/cmd/mxcli/cmd_describe.go +++ b/cmd/mxcli/cmd_describe.go @@ -47,6 +47,8 @@ Types: odataclient Describe a consumed OData service odataservice Describe a published OData service imagecollection Describe an image collection (also: "image collection") + queue Describe a task queue + scheduledevent Describe a scheduled event (also: "scheduled event") businesseventservice Describe a business event service (also: "business event service") databaseconnection Describe a database connection (also: "database connection") agent Describe an AI agent (also: "agent") @@ -166,6 +168,10 @@ Example: mdlCmd = fmt.Sprintf("DESCRIBE ODATA SERVICE %s", name) case "IMAGECOLLECTION", "IMAGE COLLECTION": mdlCmd = fmt.Sprintf("DESCRIBE IMAGE COLLECTION %s", name) + case "QUEUE": + mdlCmd = fmt.Sprintf("DESCRIBE QUEUE %s", name) + case "SCHEDULEDEVENT", "SCHEDULED EVENT": + mdlCmd = fmt.Sprintf("DESCRIBE SCHEDULED EVENT %s", name) case "BUSINESSEVENTSERVICE", "BUSINESS EVENT SERVICE": mdlCmd = fmt.Sprintf("DESCRIBE BUSINESS EVENT SERVICE %s", name) case "DATABASECONNECTION", "DATABASE CONNECTION": @@ -190,8 +196,8 @@ Example: mdlCmd = "" // handled directly by format-specific path default: fmt.Fprintf(os.Stderr, "Unknown type: %s\n", strings.Join(args[:len(args)-1], " ")) - fmt.Fprintln(os.Stderr, "Valid types: module, entity, association, enumeration, constant, microflow, nanoflow, workflow, page, snippet, layout, javaaction, jsonstructure, importmapping, exportmapping, restclient, odataclient, odataservice, imagecollection, businesseventservice, databaseconnection, agent, aimodel, knowledgebase, consumedmcpservice, datatransformer, modulerole, userrole, projectsecurity, settings, demouser, navigation, systemoverview") - fmt.Fprintln(os.Stderr, "Multi-word types also accepted: json structure, import mapping, export mapping, rest client, image collection, business event service, agent, model, knowledge base, consumed mcp service, data transformer, etc.") + fmt.Fprintln(os.Stderr, "Valid types: module, entity, association, enumeration, constant, microflow, nanoflow, workflow, page, snippet, layout, javaaction, jsonstructure, importmapping, exportmapping, restclient, odataclient, odataservice, imagecollection, queue, scheduledevent, businesseventservice, databaseconnection, agent, aimodel, knowledgebase, consumedmcpservice, datatransformer, modulerole, userrole, projectsecurity, settings, demouser, navigation, systemoverview") + fmt.Fprintln(os.Stderr, "Multi-word types also accepted: json structure, import mapping, export mapping, rest client, image collection, scheduled event, business event service, agent, model, knowledge base, consumed mcp service, data transformer, etc.") os.Exit(1) } @@ -323,21 +329,23 @@ var objectTypeToDescribe = map[string]string{ // types are listed (page templates, building blocks, rules, etc. are absent so // they don't resolve). var unitTypeToDescribe = map[string]string{ - "Microflows$Microflow": "microflow", - "Microflows$Nanoflow": "nanoflow", - "Forms$Page": "page", - "Forms$Snippet": "snippet", - "Pages$BuildingBlock": "buildingblock", - "Forms$BuildingBlock": "buildingblock", - "Forms$Layout": "layout", - "Enumerations$Enumeration": "enumeration", - "Constants$Constant": "constant", - "JavaActions$JavaAction": "javaaction", - "JsonStructures$JsonStructure": "jsonstructure", - "ImportMappings$ImportMapping": "importmapping", - "ExportMappings$ExportMapping": "exportmapping", - "Images$ImageCollection": "imagecollection", - "Workflows$Workflow": "workflow", + "Microflows$Microflow": "microflow", + "Microflows$Nanoflow": "nanoflow", + "Forms$Page": "page", + "Forms$Snippet": "snippet", + "Pages$BuildingBlock": "buildingblock", + "Forms$BuildingBlock": "buildingblock", + "Forms$Layout": "layout", + "Enumerations$Enumeration": "enumeration", + "Constants$Constant": "constant", + "JavaActions$JavaAction": "javaaction", + "JsonStructures$JsonStructure": "jsonstructure", + "ImportMappings$ImportMapping": "importmapping", + "ExportMappings$ExportMapping": "exportmapping", + "Images$ImageCollection": "imagecollection", + "Workflows$Workflow": "workflow", + "Queues$Queue": "queue", + "ScheduledEvents$ScheduledEvent": "scheduledevent", } // resolveDescribeType auto-detects the `describe` type for a qualified document diff --git a/cmd/mxcli/project_tree.go b/cmd/mxcli/project_tree.go index d617da3e7..a2f99eb6a 100644 --- a/cmd/mxcli/project_tree.go +++ b/cmd/mxcli/project_tree.go @@ -241,6 +241,17 @@ func buildProjectTree(projectPath string) ([]*TreeNode, error) { md.documents = append(md.documents, treeElement{Name: se.Name, ContainerID: se.ContainerID, Type: "scheduledevent"}) } + // Collect task queues + queues, _ := reader.ListQueues() + for _, q := range queues { + modID := h.FindModuleID(q.ContainerID) + md, ok := modData[modID] + if !ok { + continue + } + md.documents = append(md.documents, treeElement{Name: q.Name, ContainerID: q.ContainerID, Type: "queue"}) + } + // Collect JavaScript actions jsas, _ := reader.ListJavaScriptActions() for _, jsa := range jsas { diff --git a/cmd/mxcli/tui/icons.go b/cmd/mxcli/tui/icons.go index 6e0b0dd0c..4106d39ee 100644 --- a/cmd/mxcli/tui/icons.go +++ b/cmd/mxcli/tui/icons.go @@ -35,6 +35,7 @@ var typeIconMap = map[string]string{ // Constants & events "constant": "π", "scheduledevent": "⏰", + "queue": "🧵", // Actions "javaaction": "☕", diff --git a/mdl/executor/cmd_structure.go b/mdl/executor/cmd_structure.go index 2c177c205..09fba87db 100644 --- a/mdl/executor/cmd_structure.go +++ b/mdl/executor/cmd_structure.go @@ -9,6 +9,7 @@ import ( "github.com/mendixlabs/mxcli/mdl/ast" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/mdl/types" "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/sdk/domainmodel" "github.com/mendixlabs/mxcli/sdk/javaactions" @@ -77,12 +78,13 @@ func structureDepth1JSON(ctx *ExecContext, modules []structureModule) error { beServiceCounts := queryCountByModule(ctx, "business_event_services") constantCounts := countByModuleFromBackend(ctx, "constants") scheduledEventCounts := countByModuleFromBackend(ctx, "scheduled_events") + queueCounts := countByModuleFromBackend(ctx, "queues") tr := &TableResult{ Columns: []string{ "Module", "Entities", "Enumerations", "Microflows", "Nanoflows", "Workflows", "Pages", "Snippets", "JavaActions", "Constants", - "ScheduledEvents", "ODataClients", "ODataServices", "BusinessEventServices", + "ScheduledEvents", "Queues", "ODataClients", "ODataServices", "BusinessEventServices", }, } for _, m := range modules { @@ -98,6 +100,7 @@ func structureDepth1JSON(ctx *ExecContext, modules []structureModule) error { jaCounts[m.Name], constantCounts[m.Name], scheduledEventCounts[m.Name], + queueCounts[m.Name], odataClientCounts[m.Name], odataServiceCounts[m.Name], beServiceCounts[m.Name], @@ -198,6 +201,7 @@ func structureDepth1(ctx *ExecContext, modules []structureModule) error { // Get constants and scheduled events from backend (no catalog tables) constantCounts := countByModuleFromBackend(ctx, "constants") scheduledEventCounts := countByModuleFromBackend(ctx, "scheduled_events") + queueCounts := countByModuleFromBackend(ctx, "queues") // Calculate name column width for alignment nameWidth := 0 @@ -240,6 +244,9 @@ func structureDepth1(ctx *ExecContext, modules []structureModule) error { if c := scheduledEventCounts[m.Name]; c > 0 { parts = append(parts, pluralize(c, "scheduled event", "scheduled events")) } + if c := queueCounts[m.Name]; c > 0 { + parts = append(parts, pluralize(c, "queue", "queues")) + } if c := odataClientCounts[m.Name]; c > 0 { parts = append(parts, pluralize(c, "odata client", "odata clients")) } @@ -297,6 +304,14 @@ func countByModuleFromBackend(ctx *ExecContext, kind string) map[string]int { counts[modName]++ } } + case "queues": + if queues, err := ctx.Backend.ListQueues(); err == nil { + for _, q := range queues { + modID := h.FindModuleID(q.ContainerID) + modName := h.GetModuleName(modID) + counts[modName]++ + } + } } return counts } @@ -374,6 +389,14 @@ func structureDepth2(ctx *ExecContext, modules []structureModule) error { eventsByModule[modName] = append(eventsByModule[modName], ev) } + allQueues, _ := ctx.Backend.ListQueues() + queuesByModule := make(map[string][]*types.Queue) + for _, q := range allQueues { + modID := h.FindModuleID(q.ContainerID) + modName := h.GetModuleName(modID) + queuesByModule[modName] = append(queuesByModule[modName], q) + } + // Load java actions for parameter types allJavaActions, _ := ctx.Backend.ListJavaActionsFull() jaByModule := make(map[string][]*javaactions.JavaAction) @@ -457,6 +480,14 @@ func structureDepth2(ctx *ExecContext, modules []structureModule) error { } } + // Task Queues + if queues, ok := queuesByModule[m.Name]; ok { + sort.Slice(queues, func(i, j int) bool { return queues[i].Name < queues[j].Name }) + for _, q := range queues { + fmt.Fprintf(ctx.Output, " Queue %s.%s\n", m.Name, q.Name) + } + } + // OData Clients structureODataClients(ctx, m.Name) @@ -529,6 +560,14 @@ func structureDepth3(ctx *ExecContext, modules []structureModule) error { eventsByModule[modName] = append(eventsByModule[modName], ev) } + allQueues, _ := ctx.Backend.ListQueues() + queuesByModule := make(map[string][]*types.Queue) + for _, q := range allQueues { + modID := h.FindModuleID(q.ContainerID) + modName := h.GetModuleName(modID) + queuesByModule[modName] = append(queuesByModule[modName], q) + } + allJavaActions, _ := ctx.Backend.ListJavaActionsFull() jaByModule := make(map[string][]*javaactions.JavaAction) for _, ja := range allJavaActions { @@ -615,6 +654,14 @@ func structureDepth3(ctx *ExecContext, modules []structureModule) error { } } + // Task Queues + if queues, ok := queuesByModule[m.Name]; ok { + sort.Slice(queues, func(i, j int) bool { return queues[i].Name < queues[j].Name }) + for _, q := range queues { + fmt.Fprintf(ctx.Output, " Queue %s.%s\n", m.Name, q.Name) + } + } + // OData structureODataClients(ctx, m.Name) structureODataServices(ctx, m.Name) diff --git a/vscode-mdl/package.json b/vscode-mdl/package.json index 73a3030e4..7499ed69e 100644 --- a/vscode-mdl/package.json +++ b/vscode-mdl/package.json @@ -152,7 +152,7 @@ }, { "command": "mendix.openElement", - "when": "view == mendixProjectTree && viewItem =~ /^(entity|microflow|nanoflow|page|snippet|enumeration|association|workflow|constant|layout|javaaction|javascriptaction|scheduledevent|buildingblock|pagetemplate|imagecollection|jsonstructure|importmapping|exportmapping|restclient|businesseventservice|databaseconnection|publishedrestservice|odataclient|odataservice|modulerole|userrole|projectsecurity|demouser|agent|aimodel|knowledgebase|consumedmcpservice|datatransformer)$/", + "when": "view == mendixProjectTree && viewItem =~ /^(entity|microflow|nanoflow|page|snippet|enumeration|association|workflow|constant|layout|javaaction|javascriptaction|scheduledevent|queue|buildingblock|pagetemplate|imagecollection|jsonstructure|importmapping|exportmapping|restclient|businesseventservice|databaseconnection|publishedrestservice|odataclient|odataservice|modulerole|userrole|projectsecurity|demouser|agent|aimodel|knowledgebase|consumedmcpservice|datatransformer)$/", "group": "mendix@6" }, { diff --git a/vscode-mdl/src/extension.ts b/vscode-mdl/src/extension.ts index 79f56824a..289638bcf 100644 --- a/vscode-mdl/src/extension.ts +++ b/vscode-mdl/src/extension.ts @@ -118,7 +118,7 @@ export function activate(context: vscode.ExtensionContext) { } // Try the given type first, then fallback to other common types - const fallbackTypes = ['entity', 'microflow', 'nanoflow', 'page', 'enumeration', 'snippet', 'constant', 'javaaction', 'javascriptaction', 'scheduledevent', 'buildingblock', 'pagetemplate', 'imagecollection', 'businesseventservice', 'databaseconnection', 'publishedrestservice', 'workflow', 'layout', 'importmapping', 'exportmapping', 'restclient', 'jsonstructure', 'agent', 'aimodel', 'knowledgebase', 'consumedmcpservice', 'datatransformer']; + const fallbackTypes = ['entity', 'microflow', 'nanoflow', 'page', 'enumeration', 'snippet', 'constant', 'javaaction', 'javascriptaction', 'scheduledevent', 'queue', 'buildingblock', 'pagetemplate', 'imagecollection', 'businesseventservice', 'databaseconnection', 'publishedrestservice', 'workflow', 'layout', 'importmapping', 'exportmapping', 'restclient', 'jsonstructure', 'agent', 'aimodel', 'knowledgebase', 'consumedmcpservice', 'datatransformer']; const typesToTry = [type, ...fallbackTypes.filter(t => t !== type)]; for (const tryType of typesToTry) { diff --git a/vscode-mdl/src/projectTreeProvider.ts b/vscode-mdl/src/projectTreeProvider.ts index 57c5d9da7..317b0e33e 100644 --- a/vscode-mdl/src/projectTreeProvider.ts +++ b/vscode-mdl/src/projectTreeProvider.ts @@ -253,6 +253,8 @@ export class MendixProjectTreeProvider implements vscode.TreeDataProvider = { 'JavaAction': 'javaaction', 'Constant': 'constant', 'ScheduledEvent': 'scheduledevent', + 'Queue': 'queue', 'ODataClient': 'odataclient', 'ODataService': 'odataservice', }; From 5d07764a4e5df6435be3d9b81376f80f45585236 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 06:39:40 +0000 Subject: [PATCH 7/7] fix(catalog,linter): catalog scheduled events and queues, and stop calling scheduled microflows dead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Neither document type was in the catalog, and the consequence was worse than a missing table. A scheduled event produced no edge in the reference graph, so a microflow run only by one was reported as unused from three directions at once: show callers of Ops.SE_Cleanup -> (no callers found) CATALOG.GRAPH_DEAD_ASSETS -> Ops.SE_Cleanup mxcli lint -> [QUAL004] Microflow 'SE_Cleanup' is not called from anywhere. -> Remove if unused on a microflow that runs nightly. The table is an absence somebody notices; the missing edge is a wrong answer three tools state confidently, one of which recommends deleting the file. Changes: - CATALOG.SCHEDULED_EVENTS (microflow, repeat, a readable schedule phrase, interval, enabled, timezone, on-overlap) and CATALOG.QUEUES (parallelism as a string — Mendix stores an expression — and cluster-wide), both registered in Tables() so SHOW CATALOG TABLES lists them. - A `schedule` edge from each event to its microflow, added to callerRefKinds, graphRefKinds and the QUAL004 rule — three consumers, none sharing the others' list. - The linter's interval_seconds is now derived from the Schedule child instead of the stored Interval/IntervalType pair. Studio Pro writes that pair and does not keep it in sync (Workflow Commons ships 0/"Minute" beside a daily schedule), so a "fires too often" rule read a nightly job as every 0 seconds. Rules can now also branch on repeat / on_overlap / time_zone, and iterate queues(). Also fixes a regression from the queue feature: QUEUES became a lexer keyword, so `select * from CATALOG.QUEUES` parsed to nothing — no error, no output — until QUEUES was added to catalogTableName, the same trap COMMUNITIES hit before. Both fixes have a control: reverting the interval derivation fails the schedule tests, and removing the ref edge takes QUAL004 back from 1 to 2 on the test project. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LUToAkUx54bNkNjsBpufRH --- .claude/lint-rules/orphaned_elements.star | 7 +- .claude/skills/fix-issue.md | 1 + .../mendix/scheduled-events-and-queues.md | 22 ++ CLAUDE.md | 2 +- docs-site/src/tools/catalog-tables.md | 56 +++++ mdl/catalog/builder.go | 13 + mdl/catalog/builder_graph.go | 4 + mdl/catalog/builder_references.go | 26 ++ mdl/catalog/builder_scheduling.go | 227 ++++++++++++++++++ mdl/catalog/builder_scheduling_test.go | 92 +++++++ mdl/catalog/catalog.go | 2 + mdl/catalog/tables.go | 46 ++++ mdl/executor/cmd_search.go | 7 +- mdl/grammar/domains/MDLCatalog.g4 | 1 + mdl/linter/context.go | 131 ++++++++-- mdl/linter/starlark.go | 37 ++- mdl/linter/starlark_scheduledevents_test.go | 186 +++++++++----- mdl/visitor/visitor_catalog_test.go | 12 + 18 files changed, 784 insertions(+), 88 deletions(-) create mode 100644 mdl/catalog/builder_scheduling.go create mode 100644 mdl/catalog/builder_scheduling_test.go diff --git a/.claude/lint-rules/orphaned_elements.star b/.claude/lint-rules/orphaned_elements.star index e3d16cdf5..1aa790cb2 100644 --- a/.claude/lint-rules/orphaned_elements.star +++ b/.claude/lint-rules/orphaned_elements.star @@ -54,10 +54,13 @@ def check(): # Get references to this microflow refs = refs_to(mf.qualified_name) - # Check if any reference is a call + # A scheduled event is an entry point: it runs the microflow without + # anything "calling" it, so a 'schedule' edge counts as a caller. Without + # this, a microflow that runs nightly in production was reported as + # orphaned — with the suggestion "Remove if unused". has_callers = False for ref in refs: - if ref.ref_kind == "call": + if ref.ref_kind == "call" or ref.ref_kind == "schedule": has_callers = True break diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index db8459c9b..033993449 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -474,5 +474,6 @@ extracting `OffsetExpression`/`LimitExpression`. | An association's line anchors can be preserved but not AUTHORED — a scripted domain model cannot lay out its own connector lines, so `@Position(x, y)` gets you boxes and nothing gets you the lines between them | Feature gap, not a defect. `DomainModels$Association.ParentConnection`/`ChildConnection` had no MDL surface | `mdl/grammar/domains/MDLDomainModel.g4` (`SET ANCHOR`/`anchorPoint` — the ONLY grammar change), `mdl/visitor/visitor_association.go` (`anchorAnnotation`, `annotationParenPoint`, `anchorCoord`), `mdl/ast/ast_association.go` (`FromAnchor`/`ToAnchor` on both create and alter), `mdl/executor/cmd_associations.go` (`applyAnchors`, `describeConnectionPoints`) | **Look for an existing annotation before inventing one** — `@anchor(from:, to:)` already existed for microflow sequence flows, asking the same question (where does the connector attach), and `annotationParamName` already admitted FROM and TO, and `(x, y)` was already `annotationParenValue`: CREATE needed **zero** grammar. The two forms cannot be confused because the microflow one names its inner params (`(from: right, to: left)`) while a coordinate pair is positional. **Let the storage pick the value type**: the measured pair is continuous (x takes 14 distinct values across 88 samples), so named anchors were never an option — see the preservation row above for how that was established. **Silence must mean "preserve", not "default"** — naming one end sets it and omitting one keeps what is stored, which is what stops a `create or modify association` about the delete behaviour from flattening a hand-tuned line; the AST carries POINTERS so "not mentioned" and "mentioned as (0, 0)" stay distinguishable. **Reject what the LOADER rejects, at check time**: a fractional coordinate must error, not be truncated to 0 — Mendix refuses to open such a project, and a silently-wrong value in a file that still loads is the worse failure. **Prove DESCRIBE round-trips by parsing its own output** — asserting on a string literal passes against a formatter emitting something nothing can read. Tests `mdl/visitor/visitor_association_anchor_test.go`, `mdl/executor/cmd_associations_anchor_test.go`, example `mdl-examples/bug-tests/872-association-line-anchors.mdl`. upstream #872 | | A microflow call bound to a task queue in Studio Pro loses that binding on `create or replace microflow` — silently, and `mx check` goes from `[CE1613] "The selected task queue no longer exists"` to **0 errors**, because the configuration the error was about has been deleted | Two defects behind one report. (1) `Microflows$MicroflowCall` has `QueueSettings` in `NullFields` of `codec.RegisterTypeDefaults` (and the legacy writer hardcodes null too) — correct for a newly authored call, destructive for a stored one, and since a CREATE OR REPLACE rebuilds the whole microflow it fires on every rewrite. (2) MDL had **no queue surface at all**, so a script could not restate the binding even in principle | `mdl/executor/validate_queued_calls.go` (new: the refusal), `mdl/executor/cmd_microflows_create.go` (call site), `mdl/types/queue.go` + `mdl/backend/infrastructure.go` (`QueueBackend`), `mdl/backend/modelsdk/queue_write.go` + `sdk/mpr/queues.go` (both engines), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4`, `mdl/executor/cmd_queues.go` | **A green build can be the bug**: the fix made `mx check` report MORE errors than before (CE1613 came back), because the binding it complains about now survives. Any "error count went down" check would have scored the data loss as a fix. **`describe` showing nothing is not evidence of nothing** — the binding was invisible from every angle (describe omitted it, check went quiet, the write reported success), which is why it needed a stored-BSON probe: inject `Queue`+`QueueSettings` into a stored call, rewrite, and dump. **Isolate which property Mendix acts on before guarding on it**: a call carrying only `Queue` (with `QueueSettings` null) draws NO complaint from `mx check` — `QueueSettings` is the load-bearing one, so a guard keyed on `Queue` alone would have both missed the real case and fired on inert ones. **Get the BSON shape from a Mendix-authored model, not from the metamodel**: `mxcli marketplace download 202649` (Business Events 3.12.1) has four real queues, and all four agree — `Config` is a `Queues$BasicQueueConfig` whose `ParallelismExpression` is a **STRING**, with the sibling int32 `Parallelism` absent in every one. Writing the int (which the metamodel makes look equally valid) is inventing a property Mendix does not write. **Refuse, do not half-author** (ADR-0005): `in queue` on a call is deliberately still unimplemented, because the `Retry` shape has no Studio-Pro-authored sample to diff against, and guessing it would put a second unverified shape into user projects. **Adding a keyword costs an identifier** — `QUEUE`/`QUEUES` had to go into the `keyword` rule or `queue` would have stopped working as an attribute name. Tests `mdl/backend/modelsdk/queue_write_test.go`, `mdl/executor/validate_queued_calls_test.go`, `mdl/executor/cmd_queues_mock_test.go`, `mdl/visitor/visitor_queue_test.go`, example `mdl-examples/bug-tests/queue-authoring-and-rewrite-guard.mdl` | | Scheduled events are read-only in MDL — a project's cron cannot be scripted at all, so `mxcli new` + a script leaves you opening Studio Pro to add the one thing that makes a batch job run. Separately, `describe` of an existing one omits its schedule entirely | Two gaps behind one request. (1) No write path: `ScheduledEventBackend` had List/Get and nothing else, and there was no grammar/AST/visitor/executor. (2) The READ was lossy in both engines — the legacy `parseScheduledEvent` never looked at `Schedule`, and the modelsdk reader went through `modelsdk/gen`, whose generated types **disagree with what Studio Pro writes** on two properties | `mdl/scheduledevents/codec.go` (new, shared by both engines), `mdl/backend/modelsdk/scheduledevent_read.go` + `sdk/mpr/scheduledevents.go` + `sdk/mpr/parser_enumeration.go` (both engines through the one codec), `mdl/grammar/domains/MDLDomainModel.g4` + `MDLCatalog.g4` + `MDLParser.g4`, `mdl/ast/ast_scheduledevent.go`, `mdl/visitor/visitor_scheduledevent.go`, `mdl/executor/cmd_scheduledevents.go`, `mdl/executor/validate_scheduled_events.go` (MDL-SCHED01) | **The generated metamodel is not the storage's ground truth** — `modelsdk/gen` declares `Interval`/`HourOfDay`/… as **int32** where Studio Pro writes **int64**, and `StartDateTime` as a **string** where Studio Pro writes a **BSON datetime**. This is the same mismatch as #585 (a reader that asserted int32), so the pattern is now twice-confirmed: when gen and an observed document disagree about a numeric width or a date, the document is right. **A byte-level pin beats field assertions**: three whole documents are hex-embedded in `codec_test.go` and re-serialized element by element in ORDER, so key order, key SET and BSON TYPE are all covered — a field-by-field test passes happily while writing int32 and an empty enum. **`mxcli marketplace download` is the Studio Pro substitute**: Business Events and Community Commons have none, but Workflow Commons 4.11.0, OIDC SSO 4.6.0 and SAML 4.2.1 have four between them (only Day and Hour variants — the other six had to be metamodel-derived, then verified to load). **An empty string is not "unset" for an enum property**: the first draft wrote `IntervalType: ""` and `mx check` reported 0 errors, but the enumeration has no such member. **`Interval`/`IntervalType` are LEGACY siblings of `Schedule` that Studio Pro does not keep in sync** — Workflow Commons stores `0`/`Minute` beside a `DaySchedule` of 01:00 — so they are derived on CREATE and carried through untouched on MODIFY, never re-derived. **A polymorphic child must be dispatched on `$Type`, and its field sets never merged** (the ADR-0005 rule): the eight variants differ in arity, so the executor refuses a field belonging to another Repeat rather than dropping it. **Validate at check time, not only at exec** — the validation is decidable from the statement, so it runs in the no-project pass and `check` and `exec` call the SAME function. **Prove the describe round-trips by PARSING its own output** for every variant, not by asserting on strings. Idempotence came free (ADR-0008 canon elides the regenerated `Schedule` `$ID`) — verified with the `MXCLI_ALWAYS_WRITE=1` control, which changes 9 units where the normal run changes 0. Tests `mdl/scheduledevents/codec_test.go`, `mdl/executor/cmd_scheduledevents_test.go`, `mdl/visitor/visitor_scheduledevent_test.go`, `mdl/backend/modelsdk/scheduledevent_read_test.go`, example `mdl-examples/doctype-tests/scheduled-events.mdl` | +| A microflow run only by a scheduled event is reported as unused from three directions at once: `show callers of Mod.SE_Cleanup` says `(no callers found)`, `CATALOG.GRAPH_DEAD_ASSETS` lists it, and `mxcli lint` emits `[QUAL004] Microflow 'SE_Cleanup' is not called from anywhere.` with the suggestion **"Remove if unused"** — on a microflow that runs nightly in production | The catalog had no scheduled-event table and, more importantly, **no `refs` edge** from an event to the microflow it runs. Every consumer of the reference graph therefore agreed the microflow was dead. Separately, the linter's `ScheduledEvents()` iterator computed `interval_seconds` from the stored `Interval`/`IntervalType` pair | `mdl/catalog/tables.go` + `catalog.go` (new `scheduled_events`/`queues` views, registered in `Tables()`), `mdl/catalog/builder_scheduling.go` (new), `mdl/catalog/builder_references.go` (`RefKindSchedule` + `extractScheduledEventRefs`), `mdl/catalog/builder_graph.go` (`graphRefKinds`), `mdl/executor/cmd_search.go` (`callerRefKinds`), `.claude/lint-rules/orphaned_elements.star`, `mdl/linter/context.go` (`scheduleSeconds`, `Queues()`), `mdl/linter/starlark.go` | **A missing edge is worse than a missing table** — the table is an absence a user notices, the edge is a *wrong answer* three tools state confidently, and one of them recommends deleting the file. When adding a document type that references another, add the `refs` row in the same change. **An entry point is not a caller but must count as one**: `schedule` had to go into `callerRefKinds`, `graphRefKinds` AND the QUAL004 rule — three independent consumers, none of which shares the other's list. **Derive from the live property, not its legacy twin**: `interval_seconds` came from `Interval`/`IntervalType`, which Studio Pro writes and does not keep in sync with `Schedule` (Workflow Commons stores 0/"Minute" beside a DaySchedule of 01:00), so a "fires too often" rule read a nightly job as every 0 seconds; the old tests pinned the wrong source and had to be rewritten, which is the tell. **Adding a plural keyword breaks `CATALOG.`**: `QUEUES` became a lexer token and `select * from CATALOG.QUEUES` then parsed to NOTHING — no error, no output, just silence — until `QUEUES` was added to `catalogTableName` (`COMMUNITIES` had been through this before). `SCHEDULED_EVENTS` escaped only because the underscore makes it one IDENTIFIER. **`cmd/mxcli/lint-rules/` is gitignored and regenerated** from `.claude/lint-rules/` by `make build`, exactly like `cmd/mxcli/skills/` — editing the embed dir silently reverts on the next build. **`TestTables_CoversAllViews` catches a view missing from `Tables()`**, so a new catalog table is invisible to `SHOW CATALOG TABLES` until registered. Tests `mdl/catalog/builder_scheduling_test.go`, `mdl/linter/starlark_scheduledevents_test.go`, `mdl/visitor/visitor_catalog_test.go` | | `mxcli new` into a deep output directory dies with `System.IO.PathTooLongException` and leaves the directory holding ~259 files and no `.mpr` — output that looks like a project until you try to open it | MxToolset refuses any full destination path over **259 characters** (its own Windows-compatibility limit, not the filesystem's) and aborts extraction PART WAY THROUGH. mxcli pointed `mx create-project` straight at the user's output directory, so the abort happened there | `cmd/mxcli/newproject_paths.go` (new: `stagedProjectDirs`, `longestRelativePath`, `warnIfPathTooLongForStudioPro`, `moveProject`), wired into `cmd/mxcli/cmd_new.go` step 2 | **Stage the work somewhere safe and move it in, rather than validating a path you could just avoid** — creating in a short temp dir and renaming makes ANY depth work (POSIX allows 4096) instead of merely failing politely, and the destination is never partially populated because nothing is written there until creation succeeded. Check the relocation is safe first: `grep -rl ` returned **0 files**, so a fresh Mendix project embeds no absolute paths. **Bisect for the real threshold instead of trusting arithmetic** — 77 characters creates a project on 11.13.0, 78 fails leaving 259 files, which pins `len(dest) + 1 + longest ≤ 259` exactly. **Measure the template, don't hardcode it**: the longest relative path is 181 on 11.13.0 and 182 on 11.12.0, so walk the staged tree and use the real number, or the reported budget drifts a character per release. **Warn, don't refuse, when the finished path is over budget** — the project works on POSIX, so refusing would block a machine where it is fine; but Studio Pro on Windows would not open it, so silence would be worse. Cross-device staging needs an `os.CopyFS` fallback: `os.Rename` cannot cross filesystems. Tests `cmd/mxcli/newproject_paths_test.go`. upstream #825 | | `show callers of ` and `show references to ` report "(no callers found)" for a document reached only from a page action button — a false negative that reads as "safe to delete" | TWO independent defects behind one symptom. (1) `scanWidgetOwnRefs` collected `Entity`/`Microflow`/`Nanoflow` from a widget's raw BSON but not **`Form`**, the key a PAGE reference uses, so `widgets_data` had no page column and the refs projection had no page row. (2) `execShowCallers` filtered `RefKind = 'call'` — the kind a microflow CALL ACTIVITY produces — so the button→microflow row, which was already in the refs table, was hidden by the query | `mdl/catalog/builder_pages.go` (`scanWidgetOwnRefs` + `rawWidgetInfo.PageRef`), `mdl/catalog/tables.go` (`widgets_data.PageRef`), `mdl/catalog/builder_references.go` (projection row), `mdl/executor/cmd_search.go` (`callerRefKinds`) | **Find the live code path before fixing anything** — `builder_references.go` has an inviting `extractWidgetRefs` with a per-widget-type switch, and it is DEAD: its only caller is its own recursion. Extending it changes nothing. The live path is a SQL projection out of `widgets_data`, and the standing `NOTE: widget-level datasource/action refs still require a parsed widget tree` comment beside it is the tell. **Separate "the reference is missing" from "the query hides it"** by reading the refs table directly: the button→microflow row was present all along, so fixing only the scanner would have closed half the issue and left the reporter's second scenario broken. **`Form` is `Page`** — the same rename behind `ShowFormAction`/`CloseFormAction` (CLAUDE.md's storage-name table); grepping for `Page` in a BSON scanner finds nothing. **One action can carry two references**: `create object … then open page` holds an entity AND a page, so collecting the entity alone still leaves the page unreferenced. **Do not widen `callers` into `references`** — `datasource`/`parameter`/`generalize` are uses of a TYPE, not invocations, and including them makes the two commands synonyms; the test pins both the included and the excluded set. Tests `builder_pages_test.go` (`TestScanWidgetOwnRefs_PageReference`), `cmd_search_callers_test.go`. upstream #773 | diff --git a/.claude/skills/mendix/scheduled-events-and-queues.md b/.claude/skills/mendix/scheduled-events-and-queues.md index a6ebf19e1..187b06ecf 100644 --- a/.claude/skills/mendix/scheduled-events-and-queues.md +++ b/.claude/skills/mendix/scheduled-events-and-queues.md @@ -179,6 +179,28 @@ mxcli check script.mdl -p app.mpr --references - [ ] Enum values spelled exactly (`Server`, `DelayNext`, `Last`, `Monday`) - [ ] The target microflow exists and takes no parameters +## Querying and Linting + +Both document types are in the catalog after `refresh catalog`: + +```sql +-- Anything that fires more often than once a minute +select QualifiedName, RepeatDescription, Microflow +from CATALOG.SCHEDULED_EVENTS +where Enabled = 1 and IntervalSeconds < 60; + +select QualifiedName, Parallelism, ClusterWide from CATALOG.QUEUES; +``` + +A scheduled event counts as a caller of the microflow it runs, so +`show callers of Ops.SE_Cleanup` lists it and the lint rule for orphaned +microflows (QUAL004) does not flag it. `IntervalSeconds` is derived from the +schedule, not from the legacy `Interval`/`IntervalType` pair Mendix also stores. + +Starlark lint rules can iterate both: `scheduled_events()` yields +`repeat`, `interval_seconds`, `on_overlap`, `time_zone`, `enabled`, and +`microflow_name`; `queues()` yields `parallelism` (a string) and `cluster_wide`. + ## Related - `mxcli syntax scheduled-event`, `mxcli syntax queue` — full syntax reference diff --git a/CLAUDE.md b/CLAUDE.md index 9bcdf0fbe..c08a53743 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -669,7 +669,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - ALTER WORKFLOW (SET properties, INSERT/DROP/REPLACE activities, outcomes, paths, conditions, boundary events) - CALCULATED BY microflow syntax for calculated attributes - Image collections (SHOW/DESCRIBE/CREATE/DROP) -- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. See `.claude/skills/mendix/scheduled-events-and-queues.md` +- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. Both are in the catalog (`CATALOG.SCHEDULED_EVENTS`, `CATALOG.QUEUES`) and a scheduled event emits a `schedule` edge into `CATALOG.REFS` — without it a microflow run only by a scheduled event was reported as dead by `show callers`, `GRAPH_DEAD_ASSETS` and lint rule QUAL004. See `.claude/skills/mendix/scheduled-events-and-queues.md` - Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005): the rebuild used to write `QueueSettings` back as null, which made `mx check` go from CE1613 to 0 errors by deleting the user's configuration - AI agent documents: Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) - OData contract browsing (SHOW/DESCRIBE CONTRACT ENTITIES/ACTIONS FROM cached $metadata) diff --git a/docs-site/src/tools/catalog-tables.md b/docs-site/src/tools/catalog-tables.md index d6bdbfe39..6f5d20825 100644 --- a/docs-site/src/tools/catalog-tables.md +++ b/docs-site/src/tools/catalog-tables.md @@ -130,6 +130,62 @@ JOIN CATALOG.ENTITIES e ON ar.EntityId = e.Id WHERE e.ModuleName = 'Sales'; ``` +### CATALOG.SCHEDULED_EVENTS + +Scheduled events — Mendix's cron. + +| Column | Description | +|--------|-------------| +| `Name`, `QualifiedName`, `ModuleName`, `Folder` | Identity | +| `Microflow` | Qualified name of the microflow the event runs | +| `Repeat` | Schedule variant: `Minute`, `Hour`, `Day`, `Week`, `MonthDate`, `MonthWeekday`, `YearDate`, `YearWeekday` | +| `RepeatDescription` | The schedule as a phrase, e.g. `weekly Mon/Fri at 09:30` | +| `IntervalSeconds` | Gap between runs, derived from the schedule | +| `Enabled` | 1 if the event runs | +| `TimeZone` | `UTC` or `Server` | +| `OnOverlap` | `DelayNext` or `SkipNext` | + +`IntervalSeconds` comes from the schedule, **not** from the stored +`Interval`/`IntervalType` pair. Those are a legacy sibling that Studio Pro writes +and does not keep in sync — a shipped Mendix module stores `0`/`Minute` beside a +daily schedule — so a query keyed on them would read a nightly job as firing +every 0 seconds. Month and year figures are averages (30 and 365 days): the +column is for thresholds and ordering, not calendar arithmetic. + +```sql +-- Anything that fires more often than once a minute +select QualifiedName, RepeatDescription, Microflow +from CATALOG.SCHEDULED_EVENTS +where Enabled = 1 and IntervalSeconds < 60; + +-- Scheduled events whose microflow no longer exists +select se.QualifiedName, se.Microflow +from CATALOG.SCHEDULED_EVENTS se +left join CATALOG.MICROFLOWS m on m.QualifiedName = se.Microflow +where m.Id is null; +``` + +A scheduled event also produces a `schedule` row in `CATALOG.REFS`, so +`show callers of ` and the dead-asset analysis both see it. Without +that edge a microflow run only by a scheduled event looked unreferenced. + +### CATALOG.QUEUES + +Task queues. + +| Column | Description | +|--------|-------------| +| `Name`, `QualifiedName`, `ModuleName`, `Folder` | Identity | +| `Parallelism` | How many tasks run at once — an **expression string**, not a number | +| `ClusterWide` | 1 if the limit applies across the cluster | + +`Parallelism` is stored as text because Mendix stores an expression: a query must +not assume it parses as an integer. + +```sql +select QualifiedName, Parallelism, ClusterWide from CATALOG.QUEUES; +``` + ## Graph-Analysis Tables The dependency graph (`CATALOG.REFS`, full refresh) is analysed by a family of diff --git a/mdl/catalog/builder.go b/mdl/catalog/builder.go index d3aeafc87..0540318a6 100644 --- a/mdl/catalog/builder.go +++ b/mdl/catalog/builder.go @@ -91,6 +91,11 @@ type Builder struct { resolution float64 // Leiden resolution for the graph-analysis pass describeFunc DescribeFunc + // Scheduled event → microflow edges, collected while cataloguing the events + // and emitted by buildReferences (a later pass). Carried on the Builder + // rather than re-queried because CatalogTx has no Query. + scheduledEventRefs []scheduledEventRef + // Built-in widget definitions supplied by the caller — used to populate // the widget_definitions catalog table alongside project widgets/. builtinWidgetMetas []WidgetDefinitionMeta @@ -402,6 +407,14 @@ func (b *Builder) Build(progress ProgressFunc) error { return fmt.Errorf("failed to build data transformers: %w", err) } + if err := b.buildScheduledEvents(); err != nil { + return fmt.Errorf("failed to build scheduled events: %w", err) + } + + if err := b.buildQueues(); err != nil { + return fmt.Errorf("failed to build queues: %w", err) + } + if err := b.buildAgentEditorDocs(); err != nil { return fmt.Errorf("failed to build agent-editor documents: %w", err) } diff --git a/mdl/catalog/builder_graph.go b/mdl/catalog/builder_graph.go index 12cc298c0..eb926f4ea 100644 --- a/mdl/catalog/builder_graph.go +++ b/mdl/catalog/builder_graph.go @@ -15,6 +15,10 @@ import ( var graphRefKinds = []string{ "call", "retrieve", "create", "change", "delete", "associate", "generalize", "parameter", "return", + // A scheduled event is an entry point: the microflow it runs is reachable + // even though nothing calls it. Without this kind, GRAPH_DEAD_ASSETS reports + // every scheduled microflow as dead. + "schedule", } // betweennessNodeCap bounds the O(V*E) betweenness computation. Above it, diff --git a/mdl/catalog/builder_references.go b/mdl/catalog/builder_references.go index b85da8997..4b840db91 100644 --- a/mdl/catalog/builder_references.go +++ b/mdl/catalog/builder_references.go @@ -32,6 +32,7 @@ const ( RefKindDelete = "delete" // Microflow deletes an entity object RefKindCalculate = "calculate" // Calculated attribute uses a microflow RefKindReturn = "return" // Microflow/nanoflow returns an entity type + RefKindSchedule = "schedule" // Scheduled event runs a microflow ) // collectActionActivities returns all ActionActivity objects from an ObjectCollection, @@ -481,10 +482,35 @@ func (b *Builder) buildReferences() error { } } + // Scheduled events run a microflow. Without this edge the microflow looks + // unreferenced: `show callers` reported none, GRAPH_DEAD_ASSETS listed it, + // and QUAL004 said "is not called from anywhere" with the suggestion + // "Remove if unused" — on a microflow that runs nightly in production. + refCount += b.extractScheduledEventRefs(stmt, projectID, snapshotID) + b.report("References", refCount) return nil } +// extractScheduledEventRefs emits one `schedule` edge per scheduled event, from +// the event to the microflow it runs. +// +// The edges are collected by buildScheduledEvents, which runs earlier in the +// same transaction. +func (b *Builder) extractScheduledEventRefs(stmt *sql.Stmt, projectID, snapshotID string) int { + count := 0 + for _, r := range b.scheduledEventRefs { + if _, err := stmt.Exec( + "SCHEDULED_EVENT", "", r.qualifiedName, + "MICROFLOW", "", r.microflow, + RefKindSchedule, r.moduleName, projectID, snapshotID, + ); err == nil { + count++ + } + } + return count +} + // extractMenuItemRefs extracts page and microflow references from menu items recursively. func (b *Builder) extractMenuItemRefs(stmt *sql.Stmt, items []*types.NavMenuItem, sourceName, projectID, snapshotID string) int { refCount := 0 diff --git a/mdl/catalog/builder_scheduling.go b/mdl/catalog/builder_scheduling.go new file mode 100644 index 000000000..c7a7cbd59 --- /dev/null +++ b/mdl/catalog/builder_scheduling.go @@ -0,0 +1,227 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "fmt" + "strings" + + "go.mongodb.org/mongo-driver/bson" + + sched "github.com/mendixlabs/mxcli/mdl/scheduledevents" + "github.com/mendixlabs/mxcli/model" +) + +// Catalog rows for the two scheduling document types. +// +// Both read through the raw-unit surface and decode with the same codec the +// writers use, so the catalog cannot drift from what is stored. Scheduled events +// get their own builder rather than going through buildSimpleNamedDocs because +// the interesting columns — which microflow runs, how often, whether it is +// enabled — are the whole reason to query them. + +// scheduledEventRef is one event → microflow edge, held until buildReferences. +type scheduledEventRef struct{ qualifiedName, moduleName, microflow string } + +// buildScheduledEvents catalogs ScheduledEvents$ScheduledEvent units. +func (b *Builder) buildScheduledEvents() error { + units, err := b.reader.ListRawUnitsByType(sched.TypeName) + if err != nil { + return err + } + + stmt, err := b.tx.Prepare( + `INSERT INTO scheduled_events_data + (Id, Name, QualifiedName, ModuleName, Folder, Description, Microflow, + Repeat, RepeatDescription, IntervalSeconds, Enabled, TimeZone, OnOverlap, + ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + if err != nil { + return err + } + defer stmt.Close() + + projectID, snapshotID := b.snapshotMeta() + + count := 0 + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + continue + } + ev := sched.Parse(doc, model.ID(u.ID), model.ID(u.ContainerID)) + if ev.Name == "" { + continue + } + moduleID := b.hierarchy.findModuleID(u.ContainerID) + moduleName := b.hierarchy.getModuleName(moduleID) + + repeat, desc := "", "" + if ev.Schedule != nil { + repeat = string(ev.Schedule.Kind) + desc = describeSchedule(ev.Schedule) + } + if _, err := stmt.Exec( + string(u.ID), ev.Name, moduleName+"."+ev.Name, moduleName, + b.hierarchy.buildFolderPath(u.ContainerID), ev.Documentation, + string(ev.MicroflowID), repeat, desc, + scheduleIntervalSeconds(ev.Schedule), boolToInt(ev.Enabled), + ev.TimeZone, ev.OnOverlap, projectID, snapshotID, + ); err != nil { + return err + } + if ev.MicroflowID != "" { + b.scheduledEventRefs = append(b.scheduledEventRefs, scheduledEventRef{ + qualifiedName: moduleName + "." + ev.Name, + moduleName: moduleName, + microflow: string(ev.MicroflowID), + }) + } + count++ + } + + b.report("Scheduled Events", count) + return nil +} + +// buildQueues catalogs Queues$Queue units. +func (b *Builder) buildQueues() error { + units, err := b.reader.ListRawUnitsByType("Queues$Queue") + if err != nil { + return err + } + + stmt, err := b.tx.Prepare( + `INSERT INTO queues_data + (Id, Name, QualifiedName, ModuleName, Folder, Description, Parallelism, + ClusterWide, ProjectId, SnapshotId) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + if err != nil { + return err + } + defer stmt.Close() + + projectID, snapshotID := b.snapshotMeta() + + count := 0 + for _, u := range units { + var doc bson.M + if err := bson.Unmarshal(u.Contents, &doc); err != nil { + continue + } + name, _ := doc["Name"].(string) + if name == "" { + continue + } + documentation, _ := doc["Documentation"].(string) + // Parallelism lives on the nested Config node and is an EXPRESSION + // string, so it is stored as text rather than an integer column. + parallelism := "" + clusterWide := false + if cfg, ok := doc["Config"].(bson.M); ok { + parallelism, _ = cfg["ParallelismExpression"].(string) + clusterWide, _ = cfg["ClusterWide"].(bool) + } + moduleID := b.hierarchy.findModuleID(u.ContainerID) + moduleName := b.hierarchy.getModuleName(moduleID) + + if _, err := stmt.Exec( + string(u.ID), name, moduleName+"."+name, moduleName, + b.hierarchy.buildFolderPath(u.ContainerID), documentation, + parallelism, boolToInt(clusterWide), projectID, snapshotID, + ); err != nil { + return err + } + count++ + } + + b.report("Task Queues", count) + return nil +} + +// scheduleIntervalSeconds is how often the event fires, derived from the +// Schedule child. +// +// It deliberately does NOT use the stored Interval/IntervalType pair, which is a +// legacy sibling of Schedule that Studio Pro writes and does not keep in sync — +// Workflow Commons ships an event storing 0/"Minute" beside a DaySchedule of +// 01:00, which would catalog as "fires every 0 seconds". +// +// The month and year figures are averages (30 and 365 days); the column is for +// ordering and thresholds ("anything under a minute"), not for arithmetic on +// calendar dates. +func scheduleIntervalSeconds(s *model.Schedule) int64 { + if s == nil { + return 0 + } + mult := int64(s.Multiplier) + if mult < 1 { + mult = 1 + } + const day = int64(86400) + switch s.Kind { + case model.ScheduleMinute: + return mult * 60 + case model.ScheduleHour: + return mult * 3600 + case model.ScheduleDay: + return day + case model.ScheduleWeek: + // A weekly schedule can name several days, so the gap between runs is + // the week divided by how many are selected. + n := int64(0) + for _, on := range s.Weekdays { + if on { + n++ + } + } + if n == 0 { + return 7 * day + } + return (7 * day) / n + case model.ScheduleMonthDate, model.ScheduleMonthWeekday: + return mult * 30 * day + case model.ScheduleYearDate, model.ScheduleYearWeekday: + return 365 * day + } + return 0 +} + +// describeSchedule renders the repeat rule as a short human phrase, so a catalog +// query is readable without decoding the variant's fields. +func describeSchedule(s *model.Schedule) string { + if s == nil { + return "" + } + at := fmt.Sprintf("%02d:%02d", s.HourOfDay, s.MinuteOfHour) + switch s.Kind { + case model.ScheduleMinute: + return fmt.Sprintf("every %d min", s.Multiplier) + case model.ScheduleHour: + return fmt.Sprintf("every %dh at :%02d", s.Multiplier, s.MinuteOffset) + case model.ScheduleDay: + return "daily at " + at + case model.ScheduleWeek: + var days []string + for i, on := range s.Weekdays { + if on { + days = append(days, sched.WeekdayNames[i][:3]) + } + } + if days == nil { + days = []string{"(no days)"} + } + return "weekly " + strings.Join(days, "/") + " at " + at + case model.ScheduleMonthDate: + return fmt.Sprintf("every %d month(s) on day %d at %s", s.Multiplier, s.DayOfMonth, at) + case model.ScheduleMonthWeekday: + return fmt.Sprintf("every %d month(s) on the %s %s at %s", s.Multiplier, s.DaySelector, s.Weekday, at) + case model.ScheduleYearDate: + return fmt.Sprintf("yearly on %d/%d at %s", s.Month, s.DayOfMonth, at) + case model.ScheduleYearWeekday: + return fmt.Sprintf("yearly on the %s %s of month %d at %s", s.DaySelector, s.Weekday, s.Month, at) + } + return string(s.Kind) +} diff --git a/mdl/catalog/builder_scheduling_test.go b/mdl/catalog/builder_scheduling_test.go new file mode 100644 index 000000000..ad22ec327 --- /dev/null +++ b/mdl/catalog/builder_scheduling_test.go @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 + +package catalog + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/model" +) + +// TestScheduleIntervalSeconds pins the interval a catalog query sees. +// +// It is derived from the Schedule child on purpose. The stored +// Interval/IntervalType pair is a legacy sibling that Studio Pro writes and does +// NOT keep in sync — Workflow Commons 4.11.0 ships an event storing 0/"Minute" +// beside a DaySchedule of 01:00 — so a threshold query keyed on the pair would +// read a nightly job as firing every 0 seconds. +func TestScheduleIntervalSeconds(t *testing.T) { + const day = int64(86400) + tests := []struct { + name string + sched *model.Schedule + want int64 + }{ + {"nil", nil, 0}, + {"every 2 minutes", &model.Schedule{Kind: model.ScheduleMinute, Multiplier: 2}, 120}, + {"every 3 hours", &model.Schedule{Kind: model.ScheduleHour, Multiplier: 3}, 10800}, + {"daily", &model.Schedule{Kind: model.ScheduleDay}, day}, + {"weekly, one day", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, false, false}}, 7 * day}, + // Two selected days means it fires twice a week, not once. + {"weekly, two days", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, true, false}}, (7 * day) / 2}, + {"weekly, no days", &model.Schedule{Kind: model.ScheduleWeek}, 7 * day}, + {"every 3 months", &model.Schedule{Kind: model.ScheduleMonthDate, Multiplier: 3}, 3 * 30 * day}, + {"yearly", &model.Schedule{Kind: model.ScheduleYearWeekday}, 365 * day}, + // An unstated multiplier is 1, not 0 — 0 would read as "never fires". + {"multiplier defaults to 1", &model.Schedule{Kind: model.ScheduleHour}, 3600}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := scheduleIntervalSeconds(tt.sched); got != tt.want { + t.Errorf("scheduleIntervalSeconds = %d, want %d", got, tt.want) + } + }) + } +} + +// TestDescribeSchedule checks the human phrase each variant produces, and that +// only that variant's fields reach it — a MonthWeekday must not print a day of +// the month, or the catalog would describe a schedule the model does not have. +func TestDescribeSchedule(t *testing.T) { + full := model.Schedule{ + Multiplier: 3, MinuteOffset: 23, MonthOffset: 2, + HourOfDay: 18, MinuteOfHour: 30, DayOfMonth: 15, Month: 3, + DaySelector: "Last", Weekday: "Friday", + Weekdays: [7]bool{false, true, false, false, false, true, false}, + } + tests := []struct { + kind model.ScheduleKind + want string + mustNot []string + }{ + {model.ScheduleMinute, "every 3 min", []string{"18:30", "Friday"}}, + {model.ScheduleHour, "every 3h at :23", []string{"18:30", "Friday"}}, + {model.ScheduleDay, "daily at 18:30", []string{"Friday", "3 month"}}, + {model.ScheduleWeek, "weekly Mon/Fri at 18:30", []string{"Last"}}, + {model.ScheduleMonthDate, "every 3 month(s) on day 15 at 18:30", []string{"Friday"}}, + {model.ScheduleMonthWeekday, "every 3 month(s) on the Last Friday at 18:30", []string{"day 15"}}, + {model.ScheduleYearDate, "yearly on 3/15 at 18:30", []string{"Friday"}}, + {model.ScheduleYearWeekday, "yearly on the Last Friday of month 3 at 18:30", []string{"day 15"}}, + } + for _, tt := range tests { + t.Run(string(tt.kind), func(t *testing.T) { + s := full + s.Kind = tt.kind + got := describeSchedule(&s) + if got != tt.want { + t.Errorf("describeSchedule = %q, want %q", got, tt.want) + } + for _, bad := range tt.mustNot { + if strings.Contains(got, bad) { + t.Errorf("describeSchedule = %q — %q belongs to a different variant", got, bad) + } + } + }) + } + if got := describeSchedule(nil); got != "" { + t.Errorf("describeSchedule(nil) = %q, want empty", got) + } +} diff --git a/mdl/catalog/catalog.go b/mdl/catalog/catalog.go index 07383a3d7..83eaf8815 100644 --- a/mdl/catalog/catalog.go +++ b/mdl/catalog/catalog.go @@ -115,6 +115,8 @@ func (c *Catalog) Tables() []string { "CATALOG.JAVA_ACTION_PARAMETERS", "CATALOG.JAVASCRIPT_ACTIONS", "CATALOG.IMAGE_COLLECTIONS", + "CATALOG.SCHEDULED_EVENTS", + "CATALOG.QUEUES", "CATALOG.DATA_TRANSFORMERS", "CATALOG.AGENTS", "CATALOG.AI_MODELS", diff --git a/mdl/catalog/tables.go b/mdl/catalog/tables.go index 583cc993d..c76dc769d 100644 --- a/mdl/catalog/tables.go +++ b/mdl/catalog/tables.go @@ -312,6 +312,44 @@ func (c *Catalog) createTables() error { )`, viewWithFullSnapshot("image_collections"), + // scheduled_events — Mendix's cron. Repeat/RepeatDescription come from the + // Schedule child; IntervalSeconds is derived from it, NOT from the legacy + // Interval/IntervalType pair (Studio Pro does not keep those in sync). + `CREATE TABLE IF NOT EXISTS scheduled_events_data ( + Id TEXT PRIMARY KEY, + Name TEXT, + QualifiedName TEXT, + ModuleName TEXT, + Folder TEXT, + Description TEXT, + Microflow TEXT, + Repeat TEXT, + RepeatDescription TEXT, + IntervalSeconds INTEGER, + Enabled INTEGER, + TimeZone TEXT, + OnOverlap TEXT, + ProjectId TEXT, + SnapshotId TEXT + )`, + viewWithFullSnapshot("scheduled_events"), + + // queues — task queues (Queues$Queue). Parallelism is an expression + // string, not a number. + `CREATE TABLE IF NOT EXISTS queues_data ( + Id TEXT PRIMARY KEY, + Name TEXT, + QualifiedName TEXT, + ModuleName TEXT, + Folder TEXT, + Description TEXT, + Parallelism TEXT, + ClusterWide INTEGER, + ProjectId TEXT, + SnapshotId TEXT + )`, + viewWithFullSnapshot("queues"), + // data_transformers `CREATE TABLE IF NOT EXISTS data_transformers_data ( Id TEXT PRIMARY KEY, @@ -925,6 +963,14 @@ func (c *Catalog) createTables() error { ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource FROM image_collections UNION ALL + SELECT Id, 'SCHEDULED_EVENT' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, + ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource + FROM scheduled_events + UNION ALL + SELECT Id, 'QUEUE' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, + ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource + FROM queues + UNION ALL SELECT Id, 'DATA_TRANSFORMER' as ObjectType, Name, QualifiedName, ModuleName, Folder, Description, ProjectId, ProjectName, SnapshotId, SnapshotDate, SnapshotSource FROM data_transformers diff --git a/mdl/executor/cmd_search.go b/mdl/executor/cmd_search.go index 689528ecd..dec48316f 100644 --- a/mdl/executor/cmd_search.go +++ b/mdl/executor/cmd_search.go @@ -20,7 +20,10 @@ import ( // // A false negative here reads as "nothing uses this", which is the answer // somebody acts on before deleting a document — so the set errs toward -// including a kind rather than omitting it. +// including a kind rather than omitting it. 'schedule' is the same shape: a +// microflow run only by a scheduled event reported "(no callers found)", was +// listed in GRAPH_DEAD_ASSETS, and drew QUAL004 "is not called from anywhere" +// with the suggestion "Remove if unused" — on a microflow that runs nightly. // // Deliberately excluded: 'datasource', 'parameter', 'return', 'retrieve', // 'create', 'change', 'delete', 'associate', 'generalize' and 'layout'. Those @@ -34,6 +37,7 @@ var callerRefKinds = []string{ RefKindCallerHomePage, // navigation RefKindCallerLoginPage, RefKindCallerMenuItem, + RefKindCallerSchedule, // scheduled event: the microflow it runs } // Kind literals, kept next to the set that uses them so the SQL below cannot @@ -46,6 +50,7 @@ const ( RefKindCallerHomePage = "home_page" RefKindCallerLoginPage = "login_page" RefKindCallerMenuItem = "menu_item" + RefKindCallerSchedule = "schedule" ) // callerRefKindsSQL renders callerRefKinds as a SQL IN list. diff --git a/mdl/grammar/domains/MDLCatalog.g4 b/mdl/grammar/domains/MDLCatalog.g4 index 0ed72c210..9581d0061 100644 --- a/mdl/grammar/domains/MDLCatalog.g4 +++ b/mdl/grammar/domains/MDLCatalog.g4 @@ -220,6 +220,7 @@ catalogTableName | CONSTANTS // keyword token — must be listed explicitly | OBJECTS // keyword token — must be listed explicitly | COMMUNITIES // keyword token (SHOW COMMUNITIES) — must be listed explicitly for CATALOG.COMMUNITIES + | QUEUES // keyword token (SHOW QUEUES) — must be listed explicitly for CATALOG.QUEUES | SOURCE_KW // For CATALOG.SOURCE FTS table | ODATA // For CATALOG.ODATA_CLIENTS and CATALOG.ODATA_SERVICES (via IDENTIFIER) | IDENTIFIER // For tables like activities, xpath_expressions, projects, snapshots, refs, strings, odata_clients, odata_services, java_actions diff --git a/mdl/linter/context.go b/mdl/linter/context.go index e38e774fb..d0b08b438 100644 --- a/mdl/linter/context.go +++ b/mdl/linter/context.go @@ -814,32 +814,77 @@ func (ctx *LintContext) Snippets() iter.Seq[Snippet] { // ScheduledEvent represents a scheduled event document. type ScheduledEvent struct { - Name string - QualifiedName string - ModuleName string - MicroflowName string // qualified name of the microflow to execute + Name string + QualifiedName string + ModuleName string + MicroflowName string // qualified name of the microflow to execute + // IntervalSeconds is how often the event fires, derived from the Schedule + // child — NOT from the stored Interval/IntervalType pair, which Studio Pro + // writes and does not keep in sync with Schedule. Workflow Commons ships an + // event storing 0/"Minute" beside a DaySchedule of 01:00, so a rule keyed on + // the legacy pair would read it as "fires every 0 seconds". IntervalSeconds int - Enabled bool -} - -// intervalToSeconds converts a Mendix interval value and type to seconds. -// Returns 0 for unrecognised interval types (treated as "not convertible"). -func intervalToSeconds(interval int, intervalType string) int { - multipliers := map[string]int{ - "Second": 1, - "Minute": 60, - "Hour": 3600, - "Day": 86400, - "Week": 604800, - "Month": 2592000, - "Year": 31536000, + // Repeat is the schedule variant (Minute, Hour, Day, Week, MonthDate, + // MonthWeekday, YearDate, YearWeekday); empty when the event has no + // Schedule child. + Repeat string + // OnOverlap is DelayNext or SkipNext — the event's own concurrency control. + OnOverlap string + TimeZone string + Enabled bool +} + +// scheduleSeconds is the gap between runs implied by a schedule. +// +// The month and year figures are averages (30 and 365 days): the value is for +// thresholds and ordering ("anything that fires more often than a minute"), not +// for calendar arithmetic. +func scheduleSeconds(s *model.Schedule) int { + if s == nil { + return 0 + } + mult := s.Multiplier + if mult < 1 { + mult = 1 } - if mult, ok := multipliers[intervalType]; ok { - return interval * mult + const day = 86400 + switch s.Kind { + case model.ScheduleMinute: + return mult * 60 + case model.ScheduleHour: + return mult * 3600 + case model.ScheduleDay: + return day + case model.ScheduleWeek: + n := 0 + for _, on := range s.Weekdays { + if on { + n++ + } + } + if n == 0 { + return 7 * day + } + return (7 * day) / n + case model.ScheduleMonthDate, model.ScheduleMonthWeekday: + return mult * 30 * day + case model.ScheduleYearDate, model.ScheduleYearWeekday: + return 365 * day } return 0 } +// Queue represents a task queue document. +type Queue struct { + Name string + QualifiedName string + ModuleName string + // Parallelism is an EXPRESSION, not a number — Mendix stores it as a string, + // so a rule must not assume it parses as an integer. + Parallelism string + ClusterWide bool +} + // ScheduledEvents returns an iterator over all scheduled events (excluding system modules). // Returns an empty iterator if no reader is available. func (ctx *LintContext) ScheduledEvents() iter.Seq[ScheduledEvent] { @@ -889,12 +934,19 @@ func (ctx *LintContext) ScheduledEvents() iter.Seq[ScheduledEvent] { if mfName == "" { mfName = string(e.MicroflowID) } + repeat := "" + if e.Schedule != nil { + repeat = string(e.Schedule.Kind) + } se := ScheduledEvent{ Name: e.Name, QualifiedName: moduleName + "." + e.Name, ModuleName: moduleName, MicroflowName: mfName, - IntervalSeconds: intervalToSeconds(e.Interval, e.IntervalType), + IntervalSeconds: scheduleSeconds(e.Schedule), + Repeat: repeat, + OnOverlap: e.OnOverlap, + TimeZone: e.TimeZone, Enabled: e.Enabled, } if !yield(se) { @@ -981,6 +1033,43 @@ type DatabaseConnection struct { } // DatabaseConnections returns an iterator over all database connections (excluding system modules). +// Queues returns an iterator over all task queues (excluding platform modules). +// +// Backed by the catalog rather than the reader, so no LintReader change is +// needed; the catalog is already built whenever rules run. +func (ctx *LintContext) Queues() iter.Seq[Queue] { + return func(yield func(Queue) bool) { + rows, err := ctx.db.Query(fmt.Sprintf(` + SELECT q.Name, q.QualifiedName, q.ModuleName, q.Parallelism, q.ClusterWide + FROM queues q + LEFT JOIN modules m ON q.ModuleName = m.Name + WHERE %s + ORDER BY q.ModuleName, q.Name + `, notPlatformModule("m"))) + if err != nil { + ctx.recordQueryError("Queues", err) + return + } + defer rows.Close() + + for rows.Next() { + var q Queue + var clusterWide int + if err := rows.Scan(&q.Name, &q.QualifiedName, &q.ModuleName, &q.Parallelism, &clusterWide); err != nil { + ctx.recordQueryError("Queues (row scan)", err) + continue + } + q.ClusterWide = clusterWide != 0 + if ctx.IsExcluded(q.ModuleName) { + continue + } + if !yield(q) { + return + } + } + } +} + func (ctx *LintContext) DatabaseConnections() iter.Seq[DatabaseConnection] { return func(yield func(DatabaseConnection) bool) { rows, err := ctx.db.Query(fmt.Sprintf(` diff --git a/mdl/linter/starlark.go b/mdl/linter/starlark.go index 88623fa7f..a1d23e880 100644 --- a/mdl/linter/starlark.go +++ b/mdl/linter/starlark.go @@ -326,6 +326,7 @@ func (r *StarlarkRule) buildPredeclared() starlark.StringDict { "refs_from": starlark.NewBuiltin("refs_from", r.builtinRefsFrom), "attributes_for": starlark.NewBuiltin("attributes_for", r.builtinAttributesFor), "scheduled_events": starlark.NewBuiltin("scheduled_events", r.builtinScheduledEvents), + "queues": starlark.NewBuiltin("queues", r.builtinQueues), // Graph-analysis facts (populated by `refresh catalog communities`). "community_of": starlark.NewBuiltin("community_of", r.builtinCommunityOf), @@ -1016,15 +1017,43 @@ func databaseConnectionToStarlark(dc DatabaseConnection) starlark.Value { func scheduledEventToStarlark(se ScheduledEvent) starlark.Value { return starlarkstruct.FromStringDict(starlark.String("scheduled_event"), starlark.StringDict{ - "name": starlark.String(se.Name), - "qualified_name": starlark.String(se.QualifiedName), - "module_name": starlark.String(se.ModuleName), - "microflow_name": starlark.String(se.MicroflowName), + "name": starlark.String(se.Name), + "qualified_name": starlark.String(se.QualifiedName), + "module_name": starlark.String(se.ModuleName), + "microflow_name": starlark.String(se.MicroflowName), + // Derived from the Schedule child, not the legacy Interval/IntervalType + // pair — see ScheduledEvent.IntervalSeconds. "interval_seconds": starlark.MakeInt(se.IntervalSeconds), + "repeat": starlark.String(se.Repeat), + "on_overlap": starlark.String(se.OnOverlap), + "time_zone": starlark.String(se.TimeZone), "enabled": starlark.Bool(se.Enabled), }) } +// builtinQueues returns all task queues. +func (r *StarlarkRule) builtinQueues(_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { + if r.ctx == nil { + return starlark.NewList(nil), nil + } + var result []starlark.Value + for q := range r.ctx.Queues() { + result = append(result, queueToStarlark(q)) + } + return starlark.NewList(result), nil +} + +func queueToStarlark(q Queue) starlark.Value { + return starlarkstruct.FromStringDict(starlark.String("queue"), starlark.StringDict{ + "name": starlark.String(q.Name), + "qualified_name": starlark.String(q.QualifiedName), + "module_name": starlark.String(q.ModuleName), + // An EXPRESSION string, not a number. + "parallelism": starlark.String(q.Parallelism), + "cluster_wide": starlark.Bool(q.ClusterWide), + }) +} + // builtinXPathExpressions returns all XPath expression entries from the catalog. func (r *StarlarkRule) builtinXPathExpressions(_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { if r.ctx == nil { diff --git a/mdl/linter/starlark_scheduledevents_test.go b/mdl/linter/starlark_scheduledevents_test.go index d27e3e3b8..c4112c5cf 100644 --- a/mdl/linter/starlark_scheduledevents_test.go +++ b/mdl/linter/starlark_scheduledevents_test.go @@ -40,62 +40,127 @@ func (m *minimalReader) ListScheduledEvents() ([]*model.ScheduledEvent, error) { return nil, nil } -// TestIntervalToSeconds is a white-box test; we call it via the exported -// ScheduledEvents iterator rather than calling the unexported helper directly. -// The expected IntervalSeconds values verify all multipliers and the unknown-type fallback. -func TestIntervalToSeconds(t *testing.T) { +// TestScheduleSeconds checks the interval a rule sees, exercised through the +// exported ScheduledEvents iterator. +// +// It is derived from the Schedule child, NOT from the stored +// Interval/IntervalType pair: Studio Pro writes that pair and does not keep it +// in sync with Schedule — Workflow Commons 4.11.0 ships an event storing +// 0/"Minute" beside a DaySchedule of 01:00 — so a rule keyed on the legacy pair +// reads a daily job as firing every 0 seconds. TestScheduleSeconds_IgnoresStaleLegacyPair +// below is that exact document. +func TestScheduleSeconds(t *testing.T) { tests := []struct { - interval int - intervalType string - want int + name string + sched *model.Schedule + want int }{ - {1, "Second", 1}, - {2, "Minute", 120}, - {3, "Hour", 10800}, - {1, "Day", 86400}, - {1, "Week", 604800}, - {1, "Month", 2592000}, - {1, "Year", 31536000}, - {5, "Unknown", 0}, // unrecognised type → 0 - {5, "", 0}, // empty type → 0 - } - - containerID := model.ID("mod-1") + {"minutely x2", &model.Schedule{Kind: model.ScheduleMinute, Multiplier: 2}, 120}, + {"hourly x3", &model.Schedule{Kind: model.ScheduleHour, Multiplier: 3}, 10800}, + {"daily", &model.Schedule{Kind: model.ScheduleDay}, 86400}, + {"weekly, one day", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, false, false}}, 604800}, + // Two selected days means it fires twice a week. + {"weekly, two days", &model.Schedule{Kind: model.ScheduleWeek, + Weekdays: [7]bool{false, true, false, false, false, true, false}}, 302400}, + {"monthly", &model.Schedule{Kind: model.ScheduleMonthDate, Multiplier: 1}, 2592000}, + {"yearly", &model.Schedule{Kind: model.ScheduleYearDate}, 31536000}, + // An unstated multiplier is 1, not 0 — a 0 would read as "never". + {"multiplier defaults to 1", &model.Schedule{Kind: model.ScheduleHour}, 3600}, + {"no schedule", nil, 0}, + } + for _, tt := range tests { - reader := &minimalReader{ - listScheduledEvents: func() ([]*model.ScheduledEvent, error) { - return []*model.ScheduledEvent{{ - ContainerID: containerID, - Name: "SE", - Interval: tt.interval, - IntervalType: tt.intervalType, - Enabled: true, - }}, nil - }, - } + t.Run(tt.name, func(t *testing.T) { + if got := scheduledEventInterval(t, &model.ScheduledEvent{ + ContainerID: model.ID("mod-1"), Name: "SE", Schedule: tt.sched, Enabled: true, + }); got != tt.want { + t.Errorf("IntervalSeconds = %d, want %d", got, tt.want) + } + }) + } +} - cat, err := catalog.NewFromFile(filepath.Join(t.TempDir(), "cat.db")) - if err != nil { - t.Fatalf("NewFromFile: %v", err) - } - db := cat.CatalogDB() - if _, err := db.Exec( - `INSERT INTO modules_data (Id, Name, ProjectId, SnapshotId) VALUES (?,?,?,?)`, - string(containerID), "MyModule", "default", "s1", - ); err != nil { - t.Fatalf("insert module: %v", err) - } - cat.Close() +// TestScheduleSeconds_IgnoresStaleLegacyPair is the regression: this is the +// Workflow Commons document, whose legacy pair disagrees with its Schedule. +// Reading the pair gives 0; reading the Schedule gives a day. +func TestScheduleSeconds_IgnoresStaleLegacyPair(t *testing.T) { + got := scheduledEventInterval(t, &model.ScheduledEvent{ + ContainerID: model.ID("mod-1"), + Name: "SE_WorkflowAuditTrailRecord_CleanUp", + Interval: 0, + IntervalType: "Minute", + Schedule: &model.Schedule{Kind: model.ScheduleDay, HourOfDay: 1}, + Enabled: false, + }) + if got != 86400 { + t.Errorf("IntervalSeconds = %d, want 86400 — the legacy Interval/IntervalType pair must not be used", got) + } +} - ctx := linter.NewLintContext(cat, reader) - var got int - for se := range ctx.ScheduledEvents() { - got = se.IntervalSeconds +// TestScheduledEventExposesSchedule checks the fields a rule can branch on. +func TestScheduledEventExposesSchedule(t *testing.T) { + reader := &minimalReader{ + listScheduledEvents: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{{ + ContainerID: model.ID("mod-1"), Name: "SE", + Schedule: &model.Schedule{Kind: model.ScheduleMonthWeekday, Multiplier: 3}, + OnOverlap: "SkipNext", TimeZone: "Server", Enabled: true, + }}, nil + }, + } + ctx := linter.NewLintContext(newSingleModuleCatalog(t, model.ID("mod-1")), reader) + found := false + for se := range ctx.ScheduledEvents() { + found = true + if se.Repeat != "MonthWeekday" { + t.Errorf("Repeat = %q", se.Repeat) + } + if se.OnOverlap != "SkipNext" { + t.Errorf("OnOverlap = %q", se.OnOverlap) } - if got != tt.want { - t.Errorf("interval=%d type=%q: IntervalSeconds=%d, want %d", tt.interval, tt.intervalType, got, tt.want) + if se.TimeZone != "Server" { + t.Errorf("TimeZone = %q", se.TimeZone) } } + if !found { + t.Fatal("no scheduled event yielded") + } +} + +// scheduledEventInterval runs one event through the iterator and returns the +// interval a rule would see. +func scheduledEventInterval(t *testing.T, ev *model.ScheduledEvent) int { + t.Helper() + reader := &minimalReader{ + listScheduledEvents: func() ([]*model.ScheduledEvent, error) { + return []*model.ScheduledEvent{ev}, nil + }, + } + ctx := linter.NewLintContext(newSingleModuleCatalog(t, ev.ContainerID), reader) + got := 0 + for se := range ctx.ScheduledEvents() { + got = se.IntervalSeconds + } + return got +} + +// newSingleModuleCatalog builds a catalog holding one module, so the iterator +// can resolve the container to a module name. +func newSingleModuleCatalog(t *testing.T, containerID model.ID) *catalog.Catalog { + t.Helper() + cat, err := catalog.NewFromFile(filepath.Join(t.TempDir(), "cat.db")) + if err != nil { + t.Fatalf("NewFromFile: %v", err) + } + if _, err := cat.CatalogDB().Exec( + `INSERT INTO modules_data (Id, Name, ProjectId, SnapshotId) VALUES (?,?,?,?)`, + string(containerID), "MyModule", "default", "s1", + ); err != nil { + t.Fatalf("insert module: %v", err) + } + cat.Close() + return cat } func TestScheduledEvents_MicroflowNameResolution(t *testing.T) { @@ -169,11 +234,10 @@ func TestScheduledEvents_ExcludedModules(t *testing.T) { reader := &minimalReader{ listScheduledEvents: func() ([]*model.ScheduledEvent, error) { return []*model.ScheduledEvent{{ - ContainerID: containerID, - Name: "ExcludedSE", - Interval: 1, - IntervalType: "Day", - Enabled: true, + ContainerID: containerID, + Name: "ExcludedSE", + Schedule: &model.Schedule{Kind: model.ScheduleDay}, + Enabled: true, }}, nil }, } @@ -211,8 +275,8 @@ func TestScheduledEvents_IncludedModules(t *testing.T) { reader := &minimalReader{ listScheduledEvents: func() ([]*model.ScheduledEvent, error) { return []*model.ScheduledEvent{ - {ContainerID: modA, Name: "SE_A", Interval: 1, IntervalType: "Hour", Enabled: true}, - {ContainerID: modB, Name: "SE_B", Interval: 1, IntervalType: "Hour", Enabled: true}, + {ContainerID: modA, Name: "SE_A", Schedule: &model.Schedule{Kind: model.ScheduleHour, Multiplier: 1}, Enabled: true}, + {ContainerID: modB, Name: "SE_B", Schedule: &model.Schedule{Kind: model.ScheduleHour, Multiplier: 1}, Enabled: true}, }, nil }, } @@ -289,11 +353,15 @@ func TestStarlarkScheduledEventsBuiltin(t *testing.T) { reader := &minimalReader{ listScheduledEvents: func() ([]*model.ScheduledEvent, error) { return []*model.ScheduledEvent{{ - ContainerID: containerID, - Name: "DailySE", - MicroflowID: mfID, + ContainerID: containerID, + Name: "DailySE", + MicroflowID: mfID, + // The interval a rule sees comes from Schedule. The legacy pair + // is left deliberately inconsistent here (it claims 2 days) to + // prove the builtin does not read it. Interval: 2, IntervalType: "Day", + Schedule: &model.Schedule{Kind: model.ScheduleHour, Multiplier: 2}, Enabled: true, }}, nil }, @@ -343,7 +411,7 @@ func TestStarlarkScheduledEventsBuiltin(t *testing.T) { for _, want := range []string{ "se Billing.DailySE", "mf Billing.SUB_DailyJob", - "secs 172800", // 2 * 86400 + "secs 7200", // 2 hours, from the Schedule — NOT the 172800 the legacy pair implies "enabled yes", } { if !strings.Contains(joined, want) { diff --git a/mdl/visitor/visitor_catalog_test.go b/mdl/visitor/visitor_catalog_test.go index b8599c969..8eba1f62b 100644 --- a/mdl/visitor/visitor_catalog_test.go +++ b/mdl/visitor/visitor_catalog_test.go @@ -68,6 +68,18 @@ func TestSelectFromCatalog(t *testing.T) { {"communities keyword table", "SELECT * FROM CATALOG.COMMUNITIES;"}, {"communities lowercase", "select * from catalog.communities;"}, {"communities with where", "SELECT AssetName FROM CATALOG.COMMUNITIES WHERE CommunityId = 1;"}, + // Same trap, hit again when QUEUES became a lexer keyword (SHOW QUEUES): + // CATALOG.QUEUES stopped parsing and the SELECT produced no statement and + // no output — no error, just silence. Every new plural keyword that names + // a catalog table has to be added to catalogTableName. + {"queues keyword table", "SELECT * FROM CATALOG.QUEUES;"}, + {"queues lowercase", "select * from catalog.queues;"}, + {"queues with where", "SELECT Name FROM CATALOG.QUEUES WHERE ClusterWide = 1;"}, + // SCHEDULED_EVENTS lexes as one IDENTIFIER (the underscore joins it), so + // the SCHEDULED keyword does not reach it — asserted so a future lexer + // change that splits it is caught here. + {"scheduled events table", "SELECT * FROM CATALOG.SCHEDULED_EVENTS;"}, + {"scheduled events with where", "SELECT Name FROM CATALOG.SCHEDULED_EVENTS WHERE Enabled = 1;"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {