Add wildcard and exclusion support to message filters

docs/commands.md

@@ -16,7 +16,7 @@ ___
 | [/co reload](#co-reload) | Reload the configuration file |
 | [/co status](#co-status) | View the plugin status |
 | [/co consumer](#co-consumer) | Toggle consumer processing |
-| [/co migrate-db](#co-migrate-db) | Migrate between database backends |
+| [/co migrate-db](#co-migrate-db) | Migrate between database backends |
 
 ### Alias Commands
 
@@ -56,6 +56,7 @@ Perform a lookup. Nearly all of the parameters are optional.
 | [`a:<action>`](#aaction) | Restrict the lookup to a certain action. |
 | [`i:<include>`](#iinclude) | Include specific blocks/entities in the lookup. |
 | [`e:<exclude>`](#eexclude) | Exclude blocks/entities from the lookup. |
+| [`f:<filter>`](#ffilter) | Filter chat, command, or sign text. |
 | [`#<hashtag>`](#hashtag) | Add a hashtag to perform additional actions. |
 
 #### Pagination
@@ -109,16 +110,16 @@ For example, `/co purge t:30d r:#world_nether` will delete all data older than o
 You can optionally specify block types in CoreProtect v23+.  
 For example, `/co purge t:30d i:stone,dirt` will delete all stone and dirt data older than one month, without removing other block data.
 
-**Database Optimization**
-
-In CoreProtect v2.15+, adding `#optimize` to the end of the command (for example, `/co purge t:30d #optimize`) will also optimize supported database tables and reclaim unused disk space. How this option is handled depends on the database backend:
-
-* SQLite already rebuilds the database from retained data and reclaims unused file space as part of a manual purge, so `#optimize` is not needed.
-* MySQL normally deletes matching rows. Adding `#optimize` also optimizes its tables to reclaim unused space.
-* DuckDB deletes matching rows in one transaction and checkpoints afterward. `#optimize` has no additional effect.
-* ClickHouse drops fully covered monthly partitions for an unfiltered time purge and synchronously removes rows from partial or filtered partitions. Adding `#optimize` also runs `OPTIMIZE TABLE ... FINAL`.
-
-`#optimize` can significantly slow MySQL and ClickHouse purges and is generally unnecessary.
+**Database Optimization**
+
+In CoreProtect v2.15+, adding `#optimize` to the end of the command (for example, `/co purge t:30d #optimize`) will also optimize supported database tables and reclaim unused disk space. How this option is handled depends on the database backend:
+
+* SQLite already rebuilds the database from retained data and reclaims unused file space as part of a manual purge, so `#optimize` is not needed.
+* MySQL normally deletes matching rows. Adding `#optimize` also optimizes its tables to reclaim unused space.
+* DuckDB deletes matching rows in one transaction and checkpoints afterward. `#optimize` has no additional effect.
+* ClickHouse drops fully covered monthly partitions for an unfiltered time purge and synchronously removes rows from partial or filtered partitions. Adding `#optimize` also runs `OPTIMIZE TABLE ... FINAL`.
+
+`#optimize` can significantly slow MySQL and ClickHouse purges and is generally unnecessary.
 
 ___
 
@@ -135,19 +136,19 @@ Console command to pause or resume consumer queue processing.
 ___
 
 ### /co migrate-db
-Migrate data from the active database backend to a different backend. This is a console-only command.
+Migrate data from the active database backend to a different backend. This is a console-only command.
 
 | Command | Parameters |
 | --- | --- |
-| /co migrate-db | `<sqlite|mysql|duckdb|clickhouse>` |
-
-The target namespace must contain no CoreProtect data; a DuckDB target must use a new database file, and `database-lock` must remain enabled. After a successful migration, CoreProtect automatically updates `database-type` in `config.yml` before queued writes resume.
+| /co migrate-db | `<sqlite|mysql|duckdb|clickhouse>` |
 
-> **Note:** Migrations between SQLite and MySQL require a CoreProtect 23.0+ Patreon build. Any migration involving DuckDB or ClickHouse requires CoreProtect 25.0+.
+The target namespace must contain no CoreProtect data; a DuckDB target must use a new database file, and `database-lock` must remain enabled. After a successful migration, CoreProtect automatically updates `database-type` in `config.yml` before queued writes resume.
 
-For complete migration instructions, safety guidelines, and troubleshooting information, see the [Database Migration documentation](/database-migration/).
-
-___
+> **Note:** Migrations between SQLite and MySQL require a CoreProtect 23.0+ Patreon build. Any migration involving DuckDB or ClickHouse requires CoreProtect 25.0+.
+
+For complete migration instructions, safety guidelines, and troubleshooting information, see the [Database Migration documentation](/database-migration/).
+
+___
 
 
 ## Parameter Details
@@ -208,10 +209,10 @@ ___
 | `a:-inventory` | items removed from player inventories |
 | `a:item` | items dropped, thrown, picked up, deposited, or withdrawn by players |
 | `a:+item` | items picked up or withdrawn by players |
-| `a:-item` | items dropped, thrown, or deposited by players |
-| `a:kill` | mobs/animals killed |
-| `a:spawn` | entities placed or spawned by players |
-| `a:session` | player logins/logouts |
+| `a:-item` | items dropped, thrown, or deposited by players |
+| `a:kill` | mobs/animals killed |
+| `a:spawn` | entities placed or spawned by players |
+| `a:session` | player logins/logouts |
 | `a:+session` | player logins |
 | `a:-session` | player logouts |
 | `a:sign` | messages written on signs |
@@ -239,6 +240,20 @@ ___
 
 ---
 
+### `f:<filter>`
+
+*Filters the text of a lookup, and can only be used with `a:chat`, `a:command`, or `a:sign`.*
+
+* Example: `a:command f:/ban` *(commands starting with "/ban")*
+* Example: `a:command f:*ban*` *(commands containing "ban")*
+* Example: `a:command f:/ban,/kick` *(specify multiple filters)*
+* Example: `a:chat f:-hello` *(exclude messages starting with "hello")*
+
+> A filter matches from the start of a message unless it begins with `*`.  
+> Matching is case insensitive, and each filter requires at least three characters.
+
+---
+
 ### `#<hashtag>`
 
 Add a hashtag to the end of your command to perform additional actions.

lang/en.yml

@@ -45,7 +45,7 @@ HELP_COMMAND: "Display more info for that command."
 HELP_EXCLUDE_1: "Exclude blocks/users."
 HELP_EXCLUDE_2: "Examples: [e:stone], [e:Notch], [e:stone,Notch]"
 HELP_FILTER_1: "Filter chat, command, or sign text."
-HELP_FILTER_2: "Examples: [a:command f:/co], [a:sign f:Shop]"
+HELP_FILTER_2: "Examples: [a:command f:/co], [a:sign f:Shop], [f:*ban*], [f:-/co]"
 HELP_HEADER: "{0} Help"
 HELP_INCLUDE_1: "Include specific blocks/entities."
 HELP_INCLUDE_2: "Examples: [i:stone], [i:zombie], [i:stone,wood,bedrock]"

src/main/java/net/coreprotect/command/parser/MessageFilterParser.java

@@ -8,6 +8,8 @@
 import java.util.Objects;
 import java.util.Set;
 
+import net.coreprotect.model.lookup.MessageFilter;
+
 public final class MessageFilterParser {
     public static final int MINIMUM_FILTER_CODE_POINTS = 3;
 
@@ -157,7 +159,8 @@ public boolean hasInvalidLength() {
                 return true;
             }
             for (String filter : filters) {
-                if (filter.codePointCount(0, filter.length()) < MINIMUM_FILTER_CODE_POINTS) {
+                String term = MessageFilter.getTerm(filter).replace(MessageFilter.WILDCARD, "");
+                if (term.codePointCount(0, term.length()) < MINIMUM_FILTER_CODE_POINTS) {
                     return true;
                 }
             }

src/main/java/net/coreprotect/database/LookupRaw.java

@@ -35,6 +35,7 @@
 import net.coreprotect.model.lookup.EntityLookupContext;
 import net.coreprotect.model.lookup.LookupCursor;
 import net.coreprotect.model.lookup.LookupRollbackState;
+import net.coreprotect.model.lookup.MessageFilter;
 import net.coreprotect.utility.EntitySpawnTracking;
 import net.coreprotect.utility.EntityUtils;
 import net.coreprotect.utility.ErrorReporter;
@@ -1419,37 +1420,39 @@ private static String appendMessageFilters(String baseQuery, List<String> messag
             return baseQuery;
         }
 
+        String query = appendMessageMatch(baseQuery, messageFilters, table, false, bindings);
+        return appendMessageMatch(query, messageFilters, table, true, bindings);
+    }
+
+    private static String appendMessageMatch(String baseQuery, List<String> messageFilters, String table, boolean excluded, List<String> bindings) {
+        List<String> terms = filterTerms(messageFilters, excluded);
+        if (terms.isEmpty()) {
+            return baseQuery;
+        }
+
         if (ConfigHandler.databaseType.isDuckDB()) {
-            StringBuilder query = new StringBuilder(baseQuery).append(" AND (");
-            for (int index = 0; index < messageFilters.size(); index++) {
+            String column = matchColumn("message", excluded);
+            StringBuilder query = new StringBuilder(baseQuery).append(excluded ? " AND NOT (" : " AND (");
+            for (int index = 0; index < terms.size(); index++) {
                 if (index > 0) {
                     query.append(" OR ");
                 }
-                query.append("message ILIKE ? ESCAPE '~'");
-                String filter = messageFilters.get(index) == null ? "" : messageFilters.get(index);
-                bindings.add(escapeLike(filter) + "%");
+                query.append(column).append(" ILIKE ? ESCAPE '~'");
+                bindings.add(messagePattern(terms.get(index)));
             }
             return query.append(')').toString();
         }
 
         String alias = table + "FilterRows";
-        String likeOperator = ConfigHandler.databaseType.isColumnar() ? " ILIKE " : " LIKE ";
-        String escapeClause = ConfigHandler.databaseType.isClickHouse() ? "" : " ESCAPE '~'";
         StringBuilder query = new StringBuilder(baseQuery)
-                .append(" AND rowid IN (SELECT ").append(alias).append(".rowid FROM ")
+                .append(excluded ? " AND rowid NOT IN (SELECT " : " AND rowid IN (SELECT ").append(alias).append(".rowid FROM ")
                 .append(ConfigHandler.prefix).append(table).append(" ").append(alias).append(" WHERE (");
-        for (int index = 0; index < messageFilters.size(); index++) {
+        for (int index = 0; index < terms.size(); index++) {
             if (index > 0) {
                 query.append(" OR ");
             }
 
-            String prefixExpression = messagePrefix(alias + ".message");
-            query.append("(").append(prefixExpression).append(likeOperator).append("?").append(escapeClause).append(" AND ")
-                    .append(alias).append(".message").append(likeOperator).append("?").append(escapeClause).append(")");
-
-            String filter = messageFilters.get(index) == null ? "" : messageFilters.get(index);
-            bindings.add(escapeLike(firstCodePoints(filter, 16)) + "%");
-            bindings.add(escapeLike(filter) + "%");
+            query.append(messageCondition(alias + ".message", terms.get(index), bindings));
         }
         return query.append("))").toString();
     }
@@ -1459,20 +1462,29 @@ private static String appendSignMessageFilters(String baseQuery, List<String> me
             return baseQuery;
         }
 
+        String query = appendSignMessageMatch(baseQuery, messageFilters, false, bindings);
+        return appendSignMessageMatch(query, messageFilters, true, bindings);
+    }
+
+    private static String appendSignMessageMatch(String baseQuery, List<String> messageFilters, boolean excluded, List<String> bindings) {
+        List<String> terms = filterTerms(messageFilters, excluded);
+        if (terms.isEmpty()) {
+            return baseQuery;
+        }
+
         if (ConfigHandler.databaseType.isDuckDB()) {
-            StringBuilder query = new StringBuilder(baseQuery).append(" AND (");
-            for (int filterIndex = 0; filterIndex < messageFilters.size(); filterIndex++) {
+            StringBuilder query = new StringBuilder(baseQuery).append(excluded ? " AND NOT (" : " AND (");
+            for (int filterIndex = 0; filterIndex < terms.size(); filterIndex++) {
                 if (filterIndex > 0) {
                     query.append(" OR ");
                 }
                 query.append("((face=0 AND (");
-                appendDuckDBSignLines(query, 1, 4);
+                appendDuckDBSignLines(query, 1, 4, excluded);
                 query.append(")) OR (face<>0 AND (");
-                appendDuckDBSignLines(query, 5, 8);
+                appendDuckDBSignLines(query, 5, 8, excluded);
                 query.append(")))");
 
-                String filter = messageFilters.get(filterIndex) == null ? "" : messageFilters.get(filterIndex);
-                String message = escapeLike(filter) + "%";
+                String message = messagePattern(terms.get(filterIndex));
                 for (int line = 1; line <= 8; line++) {
                     bindings.add(message);
                 }
@@ -1481,42 +1493,71 @@ private static String appendSignMessageFilters(String baseQuery, List<String> me
         }
 
         String alias = "signFilterRows";
-        String likeOperator = ConfigHandler.databaseType.isColumnar() ? " ILIKE " : " LIKE ";
-        String escapeClause = ConfigHandler.databaseType.isClickHouse() ? "" : " ESCAPE '~'";
-        StringBuilder query = new StringBuilder(baseQuery).append(" AND rowid IN (");
+        StringBuilder query = new StringBuilder(baseQuery).append(excluded ? " AND rowid NOT IN (" : " AND rowid IN (");
         boolean union = false;
-        for (String filter : messageFilters) {
-            String prefix = escapeLike(firstCodePoints(filter, 16)) + "%";
-            String message = escapeLike(filter) + "%";
+        for (String term : terms) {
             for (int line = 1; line <= 8; line++) {
                 if (union) {
                     query.append(" UNION ALL ");
                 }
 
-                String column = "line_" + line;
-                String prefixExpression = messagePrefix(alias + "." + column);
                 query.append("SELECT ").append(alias).append(".rowid FROM ")
                         .append(ConfigHandler.prefix).append("sign ").append(alias)
                         .append(" WHERE ").append(alias).append(line <= 4 ? ".face = 0" : ".face <> 0")
-                        .append(" AND (").append(prefixExpression).append(likeOperator).append("?").append(escapeClause).append(" AND ")
-                        .append(alias).append(".").append(column).append(likeOperator).append("?").append(escapeClause).append(")");
-                bindings.add(prefix);
-                bindings.add(message);
+                        .append(" AND ").append(messageCondition(alias + ".line_" + line, term, bindings));
                 union = true;
             }
         }
         return query.append(")").toString();
     }
 
-    private static void appendDuckDBSignLines(StringBuilder query, int firstLine, int lastLine) {
+    private static void appendDuckDBSignLines(StringBuilder query, int firstLine, int lastLine, boolean excluded) {
         for (int line = firstLine; line <= lastLine; line++) {
             if (line > firstLine) {
                 query.append(" OR ");
             }
-            query.append("line_").append(line).append(" ILIKE ? ESCAPE '~'");
+            query.append(matchColumn("line_" + line, excluded)).append(" ILIKE ? ESCAPE '~'");
         }
     }
 
+    private static List<String> filterTerms(List<String> messageFilters, boolean excluded) {
+        List<String> terms = new ArrayList<>();
+        for (String filter : messageFilters) {
+            if (MessageFilter.isExcluded(filter) == excluded) {
+                terms.add(MessageFilter.getTerm(filter));
+            }
+        }
+        return terms;
+    }
+
+    private static String messageCondition(String column, String term, List<String> bindings) {
+        String likeOperator = ConfigHandler.databaseType.isColumnar() ? " ILIKE " : " LIKE ";
+        String escapeClause = ConfigHandler.databaseType.isClickHouse() ? "" : " ESCAPE '~'";
+        int wildcard = term.indexOf(MessageFilter.WILDCARD);
+        String anchor = wildcard < 0 ? term : term.substring(0, wildcard);
+        StringBuilder condition = new StringBuilder("(");
+        if (!anchor.isEmpty()) { // a term starting with a wildcard can't use the message prefix index
+            condition.append(messagePrefix(column)).append(likeOperator).append("?").append(escapeClause).append(" AND ");
+            bindings.add(escapeLike(firstCodePoints(anchor, 16)) + "%");
+        }
+        condition.append(column).append(likeOperator).append("?").append(escapeClause);
+        bindings.add(messagePattern(term));
+        return condition.append(")").toString();
+    }
+
+    private static String messagePattern(String term) {
+        String pattern = escapeLike(term).replace(MessageFilter.WILDCARD, "%");
+        return pattern.endsWith("%") ? pattern : pattern + "%";
+    }
+
+    /**
+     * A negated match must treat a null column as an empty string, as "NOT (null ILIKE ?)" is null
+     * and would otherwise discard the row.
+     */
+    private static String matchColumn(String column, boolean excluded) {
+        return excluded ? "COALESCE(" + column + ",'')" : column;
+    }
+
     private static String firstCodePoints(String value, int maximum) {
         if (value == null) {
             return "";

src/main/java/net/coreprotect/language/Language.java

@@ -82,7 +82,7 @@ public static void loadPhrases() {
         phrases.put(Phrase.HELP_EXCLUDE_1, "Exclude blocks/users.");
         phrases.put(Phrase.HELP_EXCLUDE_2, "Examples: [e:stone], [e:Notch], [e:stone,Notch]");
         phrases.put(Phrase.HELP_FILTER_1, "Filter chat, command, or sign text.");
-        phrases.put(Phrase.HELP_FILTER_2, "Examples: [a:command f:/co], [a:sign f:Shop]");
+        phrases.put(Phrase.HELP_FILTER_2, "Examples: [a:command f:/co], [a:sign f:Shop], [f:*ban*], [f:-/co]");
         phrases.put(Phrase.HELP_HEADER, "{0} Help");
         phrases.put(Phrase.HELP_INCLUDE_1, "Include specific blocks/entities.");
         phrases.put(Phrase.HELP_INCLUDE_2, "Examples: [i:stone], [i:zombie], [i:stone,wood,bedrock]");

src/main/java/net/coreprotect/model/lookup/MessageFilter.java

@@ -0,0 +1,28 @@
+package net.coreprotect.model.lookup;
+
+import java.util.Objects;
+
+/**
+ * Syntax of a single message filter value, as supplied by the "f:<filter>" lookup parameter.
+ */
+public final class MessageFilter {
+
+    /** Marks a filter as excluded, e.g. "f:-/co". */
+    public static final String EXCLUDE = "-";
+
+    /** Matches any number of characters within a filter, e.g. "f:*ban*". */
+    public static final String WILDCARD = "*";
+
+    private MessageFilter() {
+        throw new IllegalStateException("Model class");
+    }
+
+    public static boolean isExcluded(String filter) {
+        return Objects.toString(filter, "").startsWith(EXCLUDE);
+    }
+
+    public static String getTerm(String filter) {
+        String value = Objects.toString(filter, "");
+        return isExcluded(value) ? value.substring(EXCLUDE.length()) : value;
+    }
+}