VahterBanBot: cap ephemeral spam warning at SPAM_WARNING_MAX_SCORE (default 3.0) - #396
Merged
Conversation
…efault 3.0) Restricts the PR #395 ephemeral spam-deletion warning to low-score (likely false-positive) deletions: MlSpam/LlmSpam/ContentFilterSpam all carry a score, warn only when score < SpamWarningMaxScore (strict, bot_setting-backed since the ML score scale drifts under daily retraining). Prod data: at the 3.0 default, 93.8% of ham-marked deletions still warn while warned-spammer volume drops from 100% to 57%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wi7gKmshkHfVtSB3tA4gmJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The ephemeral spam-deletion warning (#395) currently fires for every MlSpam/LlmSpam/ContentFilterSpam deletion, including blatant, high-confidence spam -- per vahter feedback, warning obvious spammers just teaches them detection exists without protecting anyone. Prod-data research (binding, see PR discussion) shows a 3.0 score cutoff keeps 93.8% of ham-marked (false-positive) deletions warned while cutting warned-spammer volume from 100% to 57% (a 43-point reduction). LlmSpam/ContentFilterSpam scores are < 1.5 by construction (LLM band), so they pass a 3.0 cutoff naturally -- no reason-specific special-casing needed.
What changed
New setting
SPAM_WARNING_MAX_SCORE->BotConfiguration.SpamWarningMaxScore: float, default3.0, wired viagetSettingOrexactly like the otherSPAM_WARNING_*settings (src/VahterBanBot/Types.fs,src/VahterBanBot/Program.fs). Read viaIOptions<BotConfiguration>.Valueat call time (never captured), soPOST /reload-settingspicks it up live.DeleteSpam(src/VahterBanBot/Bot.fs) now extracts the score carried by the warned reasons and only sends the warning whenscore < botConfig.Value.SpamWarningMaxScore(strict). All other VahterBanBot: ephemeral warning on spam auto-deletion (flag off by default) #395 gating (flag on, not just-banned, reason whitelist) is unchanged:Enable / tune (hand-run, NOT a Flyway seed -- see AGENTS.md's "Settings seeds, not migrations")
Then
POST /reload-settingsto pick it up without a redeploy. Missing the row is safe --getSettingOrfalls back to the code default of3.0.Tests (
tests/VahterBanBot.Tests/SpamWarningTests.fs)"2222222"scores1.5686...under the fixture ML model (seeMLScoreDeterminismTests), comfortably below the default 3.0 cutoff -- no change needed, just a doc comment.SPAM_WARNING_MAX_SCORE(cutoff pinned just below a probed score) -> deletion happens, no warning.SPAM_WARNING_MAX_SCORE(cutoff pinned to the exact probed score) -> no warning, proving the comparison is strict<.SpamWarningMlTests.DisposeAsyncnow also resetsSPAM_WARNING_MAX_SCOREto"3.0"between tests.Test run
dotnet build src/VahterBanBot -c Releaseanddotnet build tests/VahterBanBot.Tests -c Releaseboth passed clean (0 warnings/errors) locally. The localdotnet test tests/VahterBanBot.Tests -c Releaserun hit a local container-startup environment failure (fixtureMlEnabledVahterTestContainersnever produced anapp.log, so every HTTP call in the suite gotConnection refusedagainst the bot's own webhook port) rather than a test-logic failure -- deferred to CI per owner direction rather than re-running locally.Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Wi7gKmshkHfVtSB3tA4gmJ