Add linear-tail squared activation variants

[klei22]

No description provided.

[Copilot]

Pull request overview

Adds C1-continuous “linear-tail” variants for squared ReLU-style activations (both as a general activation and as a ReLU2Max softmax alternative), wiring them through configuration/CLI, plus a sweep YAML and targeted tests.

Changes:

• Introduces SquaredReLULinear and ReLU2MaxLinear with configurable positive cutoffs and linear tails matching value/slope at the cutoff.
• Registers the new variants in the activation/softmax dictionaries and exposes new cutoff args via train_args.py and defaults via gpt_conf.py.
• Adds pytest coverage and an exploration sweep YAML for the new variants.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
variations/softmax_variations.py	Adds ReLU2MaxLinear and registers it in softmax_dictionary.
variations/activation_variations.py	Adds SquaredReLULinear and registers it in activation_dictionary.
train_args.py	Adds CLI choice strings and new cutoff arguments for both variants.
gpt_conf.py	Adds config defaults for the new cutoff parameters.
tests/test_linear_squared_variants.py	Adds unit tests for values, gradients, and cutoff validation.
explorations/linear_squared_variants_pre_norm.yaml	Adds a sweep spec to explore cutoff values for both variants.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.