Add Association (<| ... |>) data structure#14
Open
msollami wants to merge 65 commits into
Open
Conversation
Introduce a first-class, hash-backed Association data structure modelled
on the Wolfram Language, together with its family of builtins and full
end-to-end coverage.
Representation
- Associations are ordinary expressions: Association[Rule[k,v], ...] with
unique, insertion-ordered keys (first occurrence fixes position, last
fixes value). This keeps the generic toolchain (Length, Map, ReplaceAll,
===, FullForm) working unchanged.
- Bulk operations are driven by a transient open-addressing hash index
(keyed by expr_hash / expr_eq) for amortised O(n) construction,
de-duplication, grouping and lookup.
Builtins (src/assoc.c), all Protected; AssociateTo is HoldFirst
- Association, AssociationQ, Keys, Values
- Lookup (single key with optional default -> Missing["KeyAbsent", key],
or a list of keys resolved with one index build, O(n+m))
- KeyExistsQ, KeyDrop, KeyTake, KeyValueMap
- AssociationThread, Counts, GroupBy, Merge, AssociateTo
Language integration
- Parser: <| ... |> literal syntax (src/parse.c), including postfix forms
<|...|>[[key]] and <|...|>[args].
- Part: assoc[[key]] / assoc[[Key[k]]] / positional assoc[[i]]; assoc[[0]]
gives the head; a missing key gives Missing["KeyAbsent", key] (src/part.c).
- Normal[assoc] -> list of rules (src/calculus/series.c).
- Printing: <| ... |> StandardForm and TeXForm (src/print.c) and KaTeX
notebook output (src/print_latex.c), falling back to head[args] form for
ill-formed associations.
- Symbols registered in src/sym_names.{c,h}; assoc_init wired into core_init.
Tests, docs, examples
- tests/test_association.c: 43 end-to-end tests (parser, builtins, printing,
Part, in-place AssociateTo, generic-tool interaction); 0 leaks under leaks.
- docs/spec/builtins/data-structures.md (new category) with verified
examples; changelog entry; Mathilda_spec.md index row.
- frontend/src/lib/canvas.ts: an Associations demo notebook in the gallery.
- examples/: association_bench.m, association_bench.py, and
association-benchmarks.md documenting measured O(n) scaling — Counts keeps
pace with CPython's Counter and is ~2,500x faster than naive O(n^2)
accumulation at N=2000.
Extend the Association data structure so it flows through the functional
toolchain the Wolfram way, and round out the key-operation family.
- Map and Select thread over association values, preserving keys:
Map[f, <|k -> v|>] -> <|k -> f[v]|>; Select[assoc, p] filters by value.
New assoc_map_values / assoc_select_values (src/assoc.c) dispatched from
builtin_map / builtin_select (src/funcprog.c) for the default level.
- New key operations (Protected): KeySort, KeySortBy (stable), KeyMap,
KeySelect.
- New aggregation builtins: CountsBy[list, f], PositionIndex[list]
(<|value -> {positions}|>, hash-indexed O(n)), AssociationMap[f, {keys}].
- Symbols registered in src/sym_names.{c,h}; all wired into assoc_init.
Tests/docs: +10 end-to-end tests (53 total in tests/test_association.c),
new sections in docs/spec/builtins/data-structures.md, changelog entry, and
extra demo cells in the frontend Associations notebook. Clean -Wall -Wextra
build, 0 leaks under leaks, no regressions in funcprog/list/core/eval/regression.
Ordering and aggregation now act on an association's values (Wolfram semantics), consistent with the earlier Map/Select value threading. - Sort[assoc] orders entries by value (keys follow); Total/Min/Max reduce over the values. New assoc_sort_by_value and reusable assoc_apply_over_values (src/assoc.c), dispatched from builtin_sort (src/sort.c), builtin_total (src/list/total.c), and builtin_min/builtin_max (src/list/minmax.c). - Join[assoc1, assoc2, ...] merges associations (later value wins) — worked already via key de-duplication; now documented and covered by a test. Tests/docs: +5 e2e tests (58 total), new spec section, changelog entry. Clean -Wall -Wextra build; 0 leaks; no regressions in sort/list/core/eval/regression.
Make associations mutable through Part, and round out value aggregation. - Part assignment on associations (src/part.c, expr_part_assign_rec): a[[key]] = val updates an existing key or appends a new key -> val entry; a[[Key[k]]] = val targets a key explicitly; a[[i]] = val updates the i-th value positionally; multi-index a[[k1, k2]] = val descends into nested associations and lists. Read-modify-write (a[[k]] = a[[k]] + 1) works. - Mean[assoc] averages the values (via assoc_apply_over_values, src/stats.c). Tests/docs: +8 e2e tests (66 total), new spec sections, changelog entry, and an in-place-mutation demo in the frontend Associations notebook. Clean -Wall -Wextra build; 0 leaks; no regressions in list_set/stats/core/eval/ regression/list suites.
Extend the pattern-matching toolchain to associations (Wolfram semantics). - Cases[assoc, patt] and Count[assoc, patt] match against the association's values, delegating through Values[assoc] via the shared assoc_apply_over_values (src/patterns.c). - DeleteCases[assoc, patt] removes entries whose value matches the pattern, returning an association — new assoc_delete_cases (src/assoc.c) tests each value with MatchQ. Tests/docs: +4 e2e tests (70 total), new spec section, changelog entry, and two demo cells in the frontend Associations notebook. Clean -Wall -Wextra build; 0 leaks; no regressions in patterns/core/eval/regression suites.
New general predicate builtins, plus association value-threading. - AllTrue, AnyTrue, NoneTrue (src/funcprog.c, Protected): test a predicate across a list's elements, short-circuiting and left unevaluated when a test result is neither True nor False (Wolfram semantics). Over an association they test the values. - MemberQ[assoc, form] now tests the association's values (src/patterns.c). Tests/docs: +6 e2e tests (82 total), new functional-programming spec section, value-threading note in data-structures, changelog, and a frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in funcprog/patterns/core/eval/regression/list suites.
New general SortBy, filling a gap and rounding out association ordering.
- SortBy[list, f] sorts a list by canonical order of f[element], evaluating
the key once per element via a paired-key qsort (src/sort.c).
- SortBy[assoc, f] sorts an association by f applied to each value; keys
follow their values.
- SortBy[f] is the operator form: SortBy[f][expr] == SortBy[expr, f].
- SYM_SortBy registered in src/sym_names.{c,h}.
Tests/docs: +6 e2e tests (88 total), new spec sections in
functional-programming and data-structures, changelog, frontend demo cell.
Clean -Wall -Wextra build; 0 leaks; no regressions in
sort/list/core/eval/regression suites.
New general extreme-selection builtins, idiomatic with associations.
- MaximalBy[list, f] / MinimalBy[list, f] give the element(s) maximising /
minimising f (all ties, in order); the key is evaluated once per element
(src/sort.c). Over an association they return the entries whose value is
extremal, as an association. MaximalBy[f] / MinimalBy[f] are operator forms.
- SYM_MaximalBy / SYM_MinimalBy registered in src/sym_names.{c,h}.
Tests/docs: +5 e2e tests (93 total), new spec sections in
functional-programming and data-structures, changelog, frontend demo cell.
Clean -Wall -Wextra build; 0 leaks; no regressions in
sort/core/eval/regression suites.
Ranked top-N selection over lists and association values. - TakeLargest[list, n] / TakeSmallest[list, n] give the n ranked-extreme elements (descending / ascending); TakeLargestBy / TakeSmallestBy rank by f[element] (src/sort.c, shared take_extreme helper over SortBy's paired-key machinery). Over an association they rank by value (or f of value) and return an association. n beyond the length returns all elements, ranked. Tests/docs: +7 e2e tests (100 total), new spec sections in functional-programming and data-structures, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in sort/core/eval/regression suites.
Turn GroupBy into a full aggregation primitive and add its list-returning
sibling.
- GroupBy[list, f, g] applies reducer g to each group, giving
<|f[x] -> g[{group}], ...|> (e.g. GroupBy[Range[10], EvenQ, Total] ->
<|False -> 25, True -> 30|>). Extends builtin_groupby (src/assoc.c).
- GatherBy[list, f] gathers elements with equal f[element] into a list of
sublists in first-appearance order (hash-indexed, O(n)).
Tests/docs: +5 e2e tests (105 total), updated GroupBy + new GatherBy spec
sections, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks;
no regressions in core/eval/regression suites.
- ReverseSort[coll] / ReverseSortBy[coll, f]: descending Sort / SortBy (src/sort.c), thin wrappers that reverse the ascending result; over an association they sort the entries by value (or f of value), descending. - examples/association_showcase.m + association-showcase.md: a 100k-record split-apply-combine pipeline (Counts, GroupBy + reducer, ReverseSort, TakeLargest, Mean per group) with measured timings, showing the toolchain end-to-end at scale. Tests/docs: +4 e2e tests (109 total), functional-programming spec section, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in sort/core/regression suites.
First-match search with Missing["NotFound"] fallback, over lists and association values. - SelectFirst[list, pred[, default]] (src/funcprog.c): first element for which pred is True (short-circuits); Missing["NotFound"] or default otherwise. Over an association, tests the values and returns the first matching value. - FirstCase[expr, patt[, default]] (src/patterns.c): first element matching patt, reusing Cases so it inherits the pattern and association-value semantics; Missing["NotFound"] or default otherwise. Tests/docs: +8 e2e tests (117 total), functional-programming spec section, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in patterns/funcprog/core/eval/regression suites.
- DeleteMissing[expr] removes all Missing[...] elements, delegating to DeleteCases[expr, _Missing] (src/patterns.c) so it inherits list and association-value handling. Natural cleanup after a multi-key Lookup; over an association it drops entries whose value is Missing[...]. Tests/docs: +4 e2e tests (121 total), data-structures spec section, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in patterns/core/eval/regression suites.
Hardening iteration (no new builtins).
- Add multi-builtin integration tests exercising realistic pipelines
(word-frequency top-N, group-reduce-rank, Lookup -> DeleteMissing -> Total,
Merge of Counts) plus empty-collection edge tests confirming association
reductions match the underlying list behaviour (Total[<||>] as Total[{}],
etc.). 129 e2e tests total in tests/test_association.c.
- Document the value-threading design principle (element-wise, reductions,
ordering and pattern/predicate ops all thread over values, key-aligned)
in docs/spec/builtins/data-structures.md, and add a composed-pipeline demo
to the frontend Associations notebook.
All association tests pass; no source changes so no build/leak impact.
Make associations destructurable in patterns and rules.
- KeyValuePattern[{k1 -> p1, ...}] (or a single k -> p) matches an association
or list of rules containing the given keys with matching values; value
patterns bind, so associations can be destructured (Replace with v_) and used
to filter records (Cases[records, KeyValuePattern[{"t" -> _}]]). Implemented
as a self-contained branch in the matcher (src/match.c) keyed on a new pattern
head, so existing matching is unaffected; registered Protected with docstring.
- Lock in that Append/Prepend already extend associations (update-or-add,
order-preserving) with tests.
Tests/docs: +11 e2e tests (140 total), data-structures spec sections, changelog,
frontend pattern-matching demo. Clean -Wall -Wextra build; 0 leaks; no
regressions in match/match_extensive/patterns/replace/core/eval/regression.
- Applying an association as a function now looks the key up: <|...|>[key] gives the value or Missing["KeyAbsent", key]; assoc[Key[k]] is the explicit form. Implemented as a compound-head application branch in the evaluator (src/eval.c), alongside pure-function application. Makes Map[#[key] &, records] work over a list of associations. Normal function application is unaffected. Tests/docs: +5 e2e tests (145 total), data-structures spec update, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in eval/purefunc/funcprog/match/core/regression suites.
- Extend the association accessor (src/eval.c) to multi-key nested lookup: assoc[k1, k2, ...] looks up k1 and applies the value to the remaining keys, so <|"a" -> <|"b" -> 5|>|>["a", "b"] is 5 and tab["row", "col"] reads a cell of an association-of-associations. A missing intermediate key propagates Missing["KeyAbsent", ...]. Tests/docs: +4 e2e tests (149 total), data-structures spec update, changelog, frontend demo cell. Clean -Wall -Wextra build; 0 leaks; no regressions in eval/purefunc/core/regression suites.
Associations can now be destructured directly in a function definition:
area[KeyValuePattern[{"w" -> w_, "h" -> h_}]] := w h
area[<|"w" -> 3, "h" -> 4|>] (* -> 12 *)
Root cause: the DownValue dispatch fast-path filter (src/symtab.c,
pattern_arg_head_canon) treated a first-arg pattern head of KeyValuePattern or
Except as a literal head, so it skipped the rule whenever the input's first-arg
head differed (e.g. Association). Both are now treated as wildcards (return
NULL), so the filter never skips them -- strictly more conservative, cannot
cause missed matches. Matching always worked via MatchQ/Cases; only this
fast-path filter was wrong. Fix also repairs Except in a DownValue LHS.
Tests/docs: +4 e2e tests (153 total), data-structures spec update, changelog,
frontend demo. Clean -Wall -Wextra build; 0 leaks; no regressions in
symtab/match/match_extensive/patterns/replace/eval/core/regression suites.
- GroupBy[list, keyfn -> valfn] groups by keyfn[x] but collects valfn[x] in each group; GroupBy[list, keyfn -> valfn, g] then reduces each group by g. Completes the Wolfram GroupBy signature and expresses the whole group / extract-field / reduce pipeline in one call, e.g. GroupBy[txns, First -> Last, Total]. Extends builtin_groupby (src/assoc.c). - Simplified examples/association_showcase.m and the frontend demo to use the cleaner one-call form (output unchanged). Tests/docs: +3 e2e tests (156 total), data-structures spec update, changelog. Clean -Wall -Wextra build; 0 leaks; no regressions in core/eval/regression.
- SortBy[list, {f1, f2, ...}] sorts by f1, breaking ties with f2, and so on
(also over associations, by value). Implemented by building each element's
sort key as the tuple {f1[e], ...}; expr_compare already orders equal-length
lists lexicographically, giving exact multi-criteria ordering (src/sort.c).
Single-criterion and operator forms are unchanged.
Tests/docs: +2 e2e tests (158 total), functional-programming spec update,
changelog, frontend demo. Clean -Wall -Wextra build; 0 leaks; no regressions
in sort/core/regression suites.
- Fold[f, seed, assoc] / FoldList[...] now fold over the association's values in key order (rebuild over Values[assoc] at the top of fold_impl, src/funcprog.c). Fold over a plain list is unchanged. - New Scan[f, expr] (src/funcprog.c, Protected): applies f to each element for side effects and returns Null; over an association it scans the values. Tests/docs: +5 e2e tests (163 total), functional-programming spec sections, changelog, frontend demo. Clean -Wall -Wextra build; 0 leaks; no regressions in fold/foldlist/core/eval/regression suites.
- First[assoc] / Last[assoc] now give the first / last value (matching Wolfram) instead of the whole key -> value rule (src/part.c). Lists and general expressions are unchanged. - Add tests covering First/Last plus Rest/Most/Take/Drop (which already slice entries and return an association, order preserved). Tests/docs: +6 e2e tests (169 total), data-structures spec section, changelog, frontend demo. Clean -Wall -Wextra build; 0 leaks; no regressions in core/eval/regression/list suites.
- KeyMemberQ[assoc, key] (== KeyExistsQ) and KeyFreeQ[assoc, key] (its complement, True when the key is absent), src/assoc.c, both Protected. Round out the key-presence predicates. - Add a test locking in Reverse[assoc] (reverses key order). Tests/docs: +3 e2e tests (174 total), data-structures spec section, changelog, frontend demo. Clean -Wall -Wextra build; 0 leaks; no regressions in core/regression suites.
- Add a "toolchain at a glance" table to the data-structures reference grouping the full association surface (~65 operations) by category, making the now-large page scannable. Verified every listed operation is a registered builtin or matcher pattern head. - Add capstone e2e tests chaining construct -> group+reduce -> order -> access in one expression (176 total). Docs + tests only (no source change); all association tests pass.
- Position[assoc, patt] now reports positions of matching values as {Key[k]}
(Wolfram semantics) instead of raw {ruleIndex, valueSlot} indices; these
positions round-trip through Part/Extract. Contained branch in
builtin_position (src/patterns.c) using the matcher directly; list and
general Position are unchanged.
Tests/docs: +3 e2e tests (179 total), data-structures spec section + overview
table entry, changelog, frontend demo. Clean -Wall -Wextra build; 0 leaks; no
regressions in patterns/core/regression suites.
- examples/associations.lb: a four-notebook tour of the association toolchain (Basics; Aggregation; Transform & rank; Patterns & pipelines; 51 cells) in the frontend "mathilda-library" format, openable in the app. All expressions verified against the binary. Linked from association-showcase.md. Examples/docs only (no source change).
Correctness fixes from a review of the association branch:
- Part assignment on associations with All / Span / a key-or-index list now
assigns into the targeted entry values (a[[All]]=v, a[[i;;j]]=v,
a[[{k...}]]=v) instead of appending a bogus All->v / Span[...]->v / {...}->v
key (src/part.c, new assoc_assign_value helper).
- Select[assoc, pred, n] (three-argument form) filters values and keeps the
first n, instead of testing Rule nodes (src/funcprog.c; assoc_select_values
gained a max-count parameter).
- Sort[assoc] is now stable: equal values keep input order (tie-break by
original position, not key; src/assoc.c).
- Lookup accepts a bare list of rules like Keys/Values; assoc_scan skips
non-rule entries defensively (src/assoc.c).
- Convention: cached SYM_MatchQ replaces a runtime intern_symbol("MatchQ")
(src/sym_names.{c,h}), per CLAUDE.md.
+6 regression tests (185 total). Clean -Wall -Wextra build; 0 leaks; no
regressions in association/funcprog/patterns/list_set/sort/match/core/eval/
regression suites.
- Position[assoc, patt] now returns {Key[k], subpos...} positions, descending
into nested values (Position[<|"a" -> {1,2}|>, 1] -> {{Key["a"], 1}}); the
positions round-trip through Part/Extract. Reimplemented by delegating to
Position over Values[assoc] and remapping the leading value-index to
Key[key] (src/patterns.c), reusing the existing level-descent machinery
rather than a shallow whole-value match.
Tests/docs: +4 e2e tests (189 total), data-structures spec update, changelog.
Clean -Wall -Wextra build; 0 leaks; no regressions in patterns/core/regression.
- An iterator spec {var, assoc} now enumerates the association's values, so
Table[v^2, {v, <|"a" -> 2, "b" -> 3|>}] gives {4, 9} and Do/Sum/Product work
over associations too. Implemented with one change in the shared
iterator-spec parser (src/iter.c) that converts an association bound to its
Values list; lists and numeric ranges are unchanged.
Tests/docs: +3 e2e tests (192 total), data-structures spec section + overview
table, changelog, frontend demo. Clean -Wall -Wextra build; 0 leaks; no
regressions in iter/core/eval/regression suites.
- assoc[[{k1, k2, ...}]] now returns the list of looked-up values (Missing for
absent keys), mirroring list Part and Lookup[assoc, {keys}]; further indices
apply per element (assoc[[{k1,k2}, 2]]). Refactored the expr_part association
branch into a single-index helper (assoc_part_single) mapped over the list
index (src/part.c).
Tests/docs: +3 e2e tests (195 total), data-structures spec update, changelog.
Clean -Wall -Wextra build; 0 leaks; no regressions in
core/eval/regression/list_set suites.
- Delete[assoc, {Key[k]}] removes an entry by key position; a nested position
{Key[k1], Key[k2]} descends into inner associations, and {{Key[k1]},{Key[k2]}}
removes several (src/part.c, delete_path gained an association-key branch).
Completes the position toolset (Extract/Position/MapAt/Delete) for Key[]
positions.
- Fixed a latent expr_delete position-spec ambiguity: a List position is
"multiple paths" only when every element is itself a List, so a Key[k] step is
part of a single path ({Key["a"], Key["x"]} is one nested path, not two).
Integer paths are unaffected.
Tests/docs: +4 e2e tests (213 total), data-structures spec section, changelog.
Clean -Wall -Wextra build; 0 leaks; no regressions in
expr/core/eval/regression/list suites.
- ReplacePart[assoc, {Key[k]} -> v] (or bare Key[k], or positional i) replaces
the value at that key; multiple positions supported. Replace-only, matching
Wolfram (absent keys unchanged) via a self-contained association branch in
builtin_replace_part (src/replace.c). Completes the position toolset for
Key[] positions: Extract / Position / MapAt / Delete / ReplacePart.
Tests/docs: +5 e2e tests (218 total), data-structures spec section, changelog.
Clean -Wall -Wextra build; 0 leaks; no regressions in
expr/replace/core/regression suites.
- examples/associations.lb: add a fifth notebook ("Positions & edits") showing
Position -> Extract/MapAt/Delete/ReplacePart on an inventory association;
add a matching frontend demo cell.
- Fix a MapAt doc example that used Minus (not a Mathilda builtin -> stayed
Minus[9]); use the pure function -# & so the documented -9 output is correct.
Examples/docs/frontend only (no source change). All expressions verified
against the binary; association tests remain green.
- Merge[<|"g1" -> a1, "g2" -> a2|>, f] now merges the values of an association of associations (not just a list), delegating to the list form via Merge[Values[col], f] (src/assoc.c). - Verified every documented In/Out example in the association spec pages against the binary: data-structures.md is 100% correct and all association examples in functional-programming.md pass (remaining mismatches there are pre-existing, unrelated examples). Tests/docs: +1 e2e test (219 total), data-structures spec update, changelog. Clean -Wall -Wextra build; 0 leaks; no regressions in core/regression suites.
- Counts[assoc] tallies the association's values, and CountsBy[assoc, f] by f, consistent with the value-threading principle (src/assoc.c, via assoc_apply_over_values). Lists are unchanged. Tests/docs: +2 e2e tests (221 total), data-structures spec update, changelog. Clean -Wall -Wextra build; 0 leaks; no regressions in core/regression suites.
DeleteDuplicates[assoc] now keeps the first entry for each distinct value and returns an association (Wolfram semantics). The scan is hash-indexed and O(n): assoc_delete_duplicate_values in src/assoc.c indexes seen values in a transient open-addressing table instead of comparing every value pairwise, and builtin_deleteduplicates dispatches to it for the default (no custom test) case. Lists and the custom-equivalence-test path are unchanged. Tests/docs: +1 e2e test group in tests/test_association.c, data-structures spec section, changelog entry, and a DeleteDuplicates cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
Median, Variance, and StandardDeviation now reduce over an association's values (joining Mean/Total/Min/Max), and Tally/Commonest tally the values (siblings of Counts/CountsBy). Each uses the established value-threading delegation through assoc_apply_over_values, so head[assoc, rest...] rewrites to head[Values[assoc], rest...] and re-evaluates. Lists are unchanged. Tests/docs: +2 e2e test groups in tests/test_association.c, data-structures spec sections, changelog entry, and Median/Tally cells in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
KeyUnion[{assoc1, assoc2, ...}] pads every association to the union of all their
keys (first-appearance order), filling a key absent from an association with
Missing["KeyAbsent", key], and returns the list of equalised associations --
the standard preparation step before row-wise or tabular processing.
The implementation is hash-indexed and linear: one O(sum of sizes) pass collects
the distinct keys, then each association is rebuilt against a hash index of its
own keys, avoiding the O(keys * entries) of repeated membership scans. Registered
Protected with a docstring; SYM_KeyUnion added to sym_names.
Tests/docs: +1 e2e test group in tests/test_association.c, data-structures spec
section, changelog entry, and a KeyUnion cell in the associations tour.
Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
Accumulate[assoc] now gives the running totals of the values with every key retained, and Differences[assoc] the successive value differences keyed by the trailing key of each pair (the leading key drops, n -> n-1). Both previously mis-fired by treating the Rule nodes as arithmetic operands. Adds a shared helper, assoc_rekey_over_values: it rewrites head[assoc, rest...] as head[Values[assoc], rest...], evaluates, then re-pairs the resulting values with the trailing keys so a dropped leading value drops the leading key. This is the shape-preserving counterpart to assoc_apply_over_values (which collapses), and Accumulate/Differences both delegate to it. Tests/docs: +2 e2e test groups in tests/test_association.c, data-structures spec section, changelog entry, and an Accumulate cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green (list paths, including Kahan CompensatedSummation, unchanged).
FoldList[f, assoc] now keeps the association shape, pairing each key with its running result (n -> n) -- the general form of Accumulate[assoc]. It previously returned a bare list of running values. Fold[f, assoc] still collapses to a scalar. Refactors the windowed-rekey path: assoc_rekey_from_list (re-pairs a values List with an association's trailing keys) is extracted from assoc_rekey_over_values so it is reusable when the collection is not the first argument. fold_impl uses it for the 2-arg FoldList; Accumulate/Differences keep using assoc_rekey_over_values, now layered on the shared helper. Tests/docs: +1 e2e test group in tests/test_association.c, data-structures spec section, changelog entry, and a FoldList cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green (list FoldList/Fold, including the seeded 3-arg form, unchanged).
TakeWhile[list, crit] gives the longest leading run of elements e for which crit[e] is True (same head as the input); LengthWhile[list, crit] gives the length of that run. Over an association the criterion tests the values: TakeWhile keeps the matching leading entries as an association (keys preserved), LengthWhile counts them. Both share a leading_run_length helper in funcprog.c (one predicate pass, short- circuiting at the first failure) and are registered Protected with docstrings. Tests/docs: +1 e2e test group in tests/test_association.c (list + association + empty + all-pass + first-fail cases), new sections in functional-programming and data-structures specs, changelog entry, and a TakeWhile cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
MinMax[list] gives {Min[list], Max[list]} in one shot -- the value range, handy
for plot bounds -- and over an association uses the values. It delegates to Min
and Max so all numeric handling (bignums, reals, symbolic extrema, empty-list
Infinity) stays in one place. Registered Protected with a docstring.
While adding it, MinMax[{singleton}] surfaced a pre-existing bug: Min/Max left a
single argument unevaluated (Min[5] stayed Min[5]). Fixed at the root -- a lone
element after list-flattening now reduces to itself (Min[5] -> 5, Max[x] -> x),
matching Wolfram. Multi-argument, symbolic-mixed, empty, and list-flattening
behaviour are unchanged.
Tests/docs: +1 e2e test group in tests/test_association.c (list + association +
single-element Min/Max), MinMax and Min/Max sections in structural-manipulation
and data-structures specs, changelog entry, and a MinMax cell in the associations
tour. Clean -Wall -Wextra build; 0 leaks; full suite green apart from the two
pre-existing FLINT-disabled number-theory suites (no Min/Max dependency).
Catenate[{e1, e2, ...}] takes a single list and flattens one level: the ei must
share a head, and their elements are concatenated under it ({{1,2},{3,4}} ->
{1,2,3,4}; {f[1,2],f[3]} -> f[1,2,3]). A list of associations merges into one
association with later keys winning (via assoc_from_rules, which copies and
collapses duplicate keys), so Catenate composes with GroupBy/Merge pipelines that
yield a list of associations. Mixed heads are left unevaluated.
Registered Protected with a docstring; implemented in src/list/join.c alongside
Join.
Tests/docs: +1 e2e test group in tests/test_association.c (lists + custom head +
association merge + empty + GroupBy pipeline), a Catenate section in the
structural-manipulation spec, changelog entry, and a Catenate cell in the
associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression/list
suites green.
MapIndexed[f, list] gives {f[e1, {1}], f[e2, {2}], ...}, passing each element's
position {i} as the second argument to f. Over an association it applies
f[value, {Key[k]}] keeping keys -- the {Key[k]} position is the same shape
Position reports for associations, so MapIndexed and Position compose. The
f[...] applications are left for the evaluator to reduce, matching Map.
Registered Protected with a docstring; implemented in src/funcprog.c beside Map.
Tests/docs: +1 e2e test group in tests/test_association.c (list + association +
#1/#2 slots + empty), a MapIndexed section in the functional-programming spec,
changelog entry, and a MapIndexed cell in the associations tour. Clean -Wall
-Wextra build; 0 leaks; core/regression suites green.
Lookup[{a1, a2, ...}, key] now threads over a list of associations, extracting
the key from each -- the natural way to pull one field out of a column of
records. Each element is delegated to Lookup, so a key-list and a default thread
through unchanged. This is distinct from the rule-list form (whose elements are
Rules, not associations), which is unchanged, and an empty list still behaves as
before (treated as an empty association).
Tests/docs: +1 e2e test group in tests/test_association.c (thread, per-element
default, key-list per record, rule-list unchanged), an updated Lookup section in
the data-structures spec, changelog entry, and a records-column Lookup cell in
the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression
suites green.
Ratios[assoc] now gives the successive value ratios keyed by the trailing key of each pair (the leading key drops, n -> n-1), completing the windowed-transform family: Accumulate, Differences, Ratios, and FoldList all keep the association shape through the shared assoc_rekey_over_values mechanism. Previously Ratios[assoc] mis-fired, treating the Rule nodes as division operands. Lists are unchanged. Tests/docs: +1 e2e test group in tests/test_association.c, updated windowed- transform section in the data-structures spec, changelog entry, and a Ratios cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression/list suites green.
On macOS the Tauri WebView (WKWebView) applied "smart dashes"/"smart quotes" to the CodeMirror editor, turning a typed `->` into `–>` (en-dash) and `"` into curly quotes. The parser correctly rejects those characters, so entering an association like <|"k" -> v|> surfaced a spurious "Parse error". Fixed at two layers: - CodeCell.svelte adds EditorView.contentAttributes with autocorrect, autocapitalize, and spellcheck disabled, so the WebView stops substituting. - ipc.ts adds normalizeInput, applied in evaluateCell before the expression reaches the kernel: curly quotes -> straight, en/em/minus dashes -> '-', and the Unicode arrow -> '->'. This also recovers text pasted from word processors. The C engine is unchanged: en-dash is genuinely not Mathilda syntax, so the kernel is right to reject it; the mangling was purely a WebView input artifact. Verified the current engine parses <|"gold" -> 5, ...|>, inv["gold"], and MinMax[inv] once the input is normalized to ASCII.
The parser now recognises the Unicode rightwards arrow → (U+2192, Wolfram \[Rule]) as a synonym for ->, at the same precedence (120, right-associative). Wolfram-Language rules and associations now paste and parse directly: <|"gold" → 5|>, 1 → 10, x /. 2 → 99. get_operator in src/parse.c matches the three UTF-8 bytes E2 86 92; ASCII -> is unchanged. This is an engine-level complement to the frontend input normalization: it also helps CLI/pipe users and anyone pasting Wolfram-Language code. Tests/docs: +1 e2e test group in tests/test_association.c (bare rule, association literal, accessor, ReplaceAll), a Rule/RuleDelayed row plus a note in the operators spec, and a changelog entry. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
Apply[f, assoc] (f @@ assoc) now uses the association's values as f's arguments:
f @@ <|k1 -> v1, ...|> is f[v1, ...], matching Wolfram and consistent with
Total == Plus @@ assoc and Apply[List, assoc] == Values[assoc]. Previously Apply
replaced the head but kept the Rule nodes as arguments, so Plus @@ <|"a"->1|> gave
("a"->1) rather than 1.
Only the default level is special-cased in builtin_apply; an explicit level spec
(@@@) and ordinary lists fall through to the generic traversal unchanged.
Tests/docs: +1 e2e test group in tests/test_association.c (Plus/Times/Max,
symbolic head, Apply[List], empty association), an Apply section in the
data-structures spec, changelog entry, and a Plus @@ assoc cell in the
associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites
green.
Extended the built-in "Associations" demo notebook (nb9 in canvas.ts) with two sections covering the builtins added over the last ~15 increments: - Windowed & statistical: MinMax, Median, Tally, Accumulate, Ratios, Plus @@ assoc. - Records & alignment: Lookup over a list of records, KeyUnion, Catenate, TakeWhile, MapIndexed, and the Unicode-arrow literal (<|"gold" → 5|>). All 49 code cells in the demo were evaluated end-to-end through the engine and confirmed correct (0 errors). Frontend/examples only; no engine or spec change.
Key[k][assoc] now extracts the value at key k (or Missing["KeyAbsent", k]) -- the curried complement of assoc[Key[k]], handled in the evaluator's function- application path. This is the record-field extractor that makes list-of-record pipelines work: GroupBy[records, Key["field"]], SortBy[records, Key["field"]], Map[Key["field"], records]. Previously Key["g"][<|...|>] stayed unevaluated, so GroupBy[records, Key["g"]] produced meaningless keys. Tests/docs: +1 e2e test group in tests/test_association.c (extract, missing key, integer key, GroupBy/SortBy/Map over records, accessor unchanged), an access- section note in the data-structures spec, changelog entry, a Records pipeline in both the built-in demo (canvas.ts) and the loadable associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
KeyDrop[{a1, a2, ...}, keys] and KeyTake[{a1, ...}, keys] now thread over a
non-empty list of associations, dropping/keeping the keys in each record -- the
column-of-records form, matching Wolfram and consistent with Lookup's threading.
Each element is delegated to the same builtin (shared key_drop_take), so a single
key or a key-list both thread. Single-association forms are unchanged.
Tests/docs: +1 e2e test group in tests/test_association.c (drop/take thread,
key-list per record, single-association unchanged), updated KeyDrop/KeyTake
sections in the data-structures spec, changelog entry, and a KeyDrop-over-records
cell in the associations tour. Clean -Wall -Wextra build; 0 leaks;
core/regression suites green.
DeleteDuplicatesBy[expr, f] keeps the first element for each distinct f[element], preserving order -- the by-key companion to DeleteDuplicates. Over an association f is applied to the values and the surviving entries keep their keys; works on any list head. Implemented in src/list/setops.c and registered Protected with a docstring. Tests/docs: +1 e2e test group in tests/test_association.c (list, StringLength key, association, empty, singleton), a DeleteDuplicatesBy section in the data-structures spec, changelog entry, and a cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression/list suites green.
First[expr, default] and Last[expr, default] now return default when expr has no elements -- an empty association, an empty list, or an atom -- matching Wolfram. The single-argument forms are unchanged (still left unevaluated on an empty or atomic argument). The default form works for lists as well as associations. Tests/docs: +1 e2e test group in tests/test_association.c (association value, empty association/list default, atomic default, list element), an updated First/Last section in the data-structures spec, changelog entry, and a First-with- default cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
GroupBy[assoc, f] now groups an association's entries by f[value] into sub- associations (keys preserved), and GroupBy[assoc, f, g] reduces each sub- association -- so GroupBy[assoc, f, Total] composes with value-threading Total. Implemented by parametrizing the existing builtin_groupby rather than duplicating the hash-grouping loop: the association path groups by f applied to the value, collects the whole entry (so keys survive), and wraps each group as Association[] instead of List[]. The keyfn -> valfn transform form remains list-only; plain lists and the GroupBy[records, Key["field"]] form are unchanged. Tests/docs: +1 e2e test group in tests/test_association.c (parity grouping, reducer, group-by-value, empty), an updated GroupBy section in the data-structures spec, changelog entry, and a cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
GatherBy[assoc, f] now gathers an association's entries by f[value] into sub- associations (keys preserved), returned as a list in first-appearance order -- the sibling of GroupBy[assoc, f] without the outer group keys. Parametrized the existing builtin_gatherby with the same association flag GroupBy uses: group by f applied to the value, collect the whole entry, and wrap each group as an Association. Lists are unchanged. Tests/docs: +1 e2e test group in tests/test_association.c (parity, group-by-value, empty, singleton), an updated GatherBy section in the data-structures spec, changelog entry, and a cell in the associations tour. Clean -Wall -Wextra build; 0 leaks; core/regression suites green.
The LaTeX form of a string now emits a left/right curly-quote pair
(\text{“...”}, U+201C / U+201D) instead of a straight " on both sides. In the
math font a straight " renders as a closing quote, so "apples" displayed as
”apples” with the opening quote reflected the wrong way; it now renders
“apples” correctly.
Also removed three pre-existing dead-code warnings in print_latex.c (the unused
is_one and split_fraction functions and an unused cur variable), which surfaced
when the file recompiled; print_latex.c now builds clean under -Wall -Wextra.
The generic-function fallback in the LaTeX renderer dumped the plain re-parseable
form (expr_to_string), so a function containing a string -- Key["a"],
Missing["KeyAbsent", "z"] -- rendered with straight quotes (Key[”a”], opening
quote reflected the wrong way). It now renders Name[arg, ...] with each argument
recursed through the LaTeX printer, so nested strings pick up the directional
\text{“...”} quotes and nested expressions render as math (e.g. x + y).
expr_to_string (InputForm) is deliberately left unchanged -- it must keep
straight quotes to remain re-parseable; only the LaTeX display path is affected.
tests/bench_assoc.c guards against the Association operations silently degrading over time. The failure mode that matters is an O(n) op quietly becoming O(n^2), so rather than gate on machine-dependent absolute times, it times each hash-backed op at size n and 2n and checks the doubling ratio t(2n)/t(n): ~2.0 for O(n), ~4.0 for O(n^2). The ratio is machine-independent, so it makes a robust gate -- it exits nonzero if any op exceeds 3.3. It also prints absolute ns/element so constant-factor drift can be eyeballed across runs. Covers Association construction, Counts, CountsBy, GroupBy, Merge, KeyUnion, bulk Lookup, Map over values, and KeySort. All currently ratio ~2.0 (clean linear scaling). Wired into tests/CMakeLists.txt (built by make, run as ./bench_assoc); build note added to SPEC.md section 9.
Extends tests/bench_assoc.c with a second gate alongside the scaling-ratio
check: an absolute-cost tripwire that errors if an operation got much slower.
Absolute wall times are machine-dependent, so each op's per-element time is
normalized against a runtime calibration (Total[list] of the same size) -- a
dimensionless "cost in calibration units" that is stable across machines -- and
compared to a checked-in baseline_norm recorded on the reference machine. It
fails (nonzero exit) if any op exceeds 2.5x its baseline: well above the few-
percent run-to-run noise, tight enough to catch a real regression.
Verified both directions: the tripwire fires on a simulated 9.7x slowdown
("MUCH SLOWER", exit 1) and passes on the real timings (every op ~1.0x baseline).
SPEC.md section 9 now documents both gates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a first-class, hash-backed Association (
<| … |>) data structure to Mathilda, modelled on the Wolfram Language, with its full family of builtins, parser/printer/Part integration, tests, docs, a frontend demo, and benchmarks. Associations are ordinaryAssociation[Rule[k,v], …]expressions with unique, insertion-ordered keys, so the generic toolchain (Length,Map,ReplaceAll,===,FullForm) works unchanged. Bulk operations use a transient open-addressing hash index (keyed byexpr_hash/expr_eq) for amortisedO(n)construction, grouping and lookup.Changes
src/assoc.c, Protected;AssociateTois HoldFirst):Association,AssociationQ,Keys,Values,Lookup(single/default/list-of-keys),KeyExistsQ,KeyDrop,KeyTake,KeyValueMap,AssociationThread,Counts,GroupBy,Merge,AssociateTo.src/parse.c):<| … |>literal syntax, including<|…|>[[key]]and<|…|>[args].src/part.c):assoc[[key]],assoc[[Key[k]]], positionalassoc[[i]],assoc[[0]]→ head; missing key →Missing["KeyAbsent", key].src/calculus/series.c):Normal[assoc]→ list of rules.<| … |>StandardForm/TeXForm (src/print.c) and KaTeX notebook output (src/print_latex.c); ill-formed associations fall back tohead[args].src/sym_names.{c,h};assoc_initwired intocore_init.docs/spec/builtins/data-structures.md, changelog entry,Mathilda_spec.mdindex row.frontend/src/lib/canvas.ts.examples/association_bench.m,examples/association_bench.py,examples/association-benchmarks.md.Testing
tests/test_association.c: 43 end-to-end tests (parser, all builtins, printing, Part, in-placeAssociateTo, generic-tool interaction) — all pass.parse_tests,eval_tests,core_tests,list_tests,regression_tests.0 leaks for 0 total leaked bytesunder macOSleakswhile exercising every builtin (including the mutating and bulk paths).-std=c99 -Wall -Wextra(no new warnings).Countsscales linearly and keeps pace with CPython'scollections.Counter; ~2,500× faster than naiveO(n²)accumulation at N=2000.Note: the frontend
npm run checkwas not run (node_modules not installed in this environment); thecanvas.tschange mirrors the existing gallery-card pattern exactly.JIRA Ticket
N/A