[FLINK-40338][table-runtime] ELT() throws ClassCastException when index is not INT - #28931
[FLINK-40338][table-runtime] ELT() throws ClassCastException when index is not INT#28931hulincup wants to merge 1 commit into
Conversation
…ex is not INT EltFunction.eval declares index as java.lang.Number to support TINYINT/SMALLINT/INT/BIGINT, but indexes the varargs array with exprs[(int) index - 1]. Per JLS 5.5, casting a Number reference to int compiles to a checkcast to Integer followed by unboxing, so a Byte, Short, or Long value throws ClassCastException on the success path (1 <= index <= exprs.length). The out-of-range guard above uses index.longValue(), so out-of-range indices of any type still return NULL correctly; the exception only fires in the valid range. Narrow the already-unboxed long idx (computed above for the range check) via primitive narrowing (JLS 5.1.3) instead, which emits no checkcast and works for every INTEGER_NUMERIC type. Adds TINYINT/SMALLINT/BIGINT index regression cases to StringFunctionsITCase.
|
Hi @hulincup — I'm the reporter of FLINK-40338 and had asked to be assigned before this PR was opened, with a patch and regression tests ready. Per the contribute-code guide, PRs on unassigned tickets aren't reviewed or merged, so could we wait for a committer to decide who takes it? |
raminqaf
left a comment
There was a problem hiding this comment.
Changes look good. Left a nit
| DataTypes.VARCHAR(5)) | ||
| .testResult( | ||
| lit(2).elt("a", "b"), "ELT(2, 'a', 'b')", "b", DataTypes.CHAR(1)) | ||
| // FLINK-40338: non-INT INTEGER_NUMERIC index must not throw ClassCastException |
There was a problem hiding this comment.
I would remove this
| // FLINK-40338: non-INT INTEGER_NUMERIC index must not throw ClassCastException |
| .testResult( | ||
| lit(2).cast(DataTypes.TINYINT()).elt("scala", "java"), | ||
| "ELT(CAST(2 AS TINYINT), 'scala', 'java')", | ||
| "java", | ||
| DataTypes.VARCHAR(5)) | ||
| .testResult( | ||
| lit(2).cast(DataTypes.SMALLINT()).elt("scala", "java"), | ||
| "ELT(CAST(2 AS SMALLINT), 'scala', 'java')", | ||
| "java", | ||
| DataTypes.VARCHAR(5)) | ||
| .testResult( | ||
| lit(2).cast(DataTypes.BIGINT()).elt("scala", "java"), | ||
| "ELT(CAST(2 AS BIGINT), 'scala', 'java')", | ||
| "java", | ||
| DataTypes.VARCHAR(5)) |
There was a problem hiding this comment.
Thanks for picking this up, the fix itself looks right. One coverage gap worth closing before merge.
All three new cases pass constant arguments, so ExpressionReducer folds the entire call during optimization and the generated runtime code is never reached. The plans:
-- ELT(CAST(2 AS TINYINT), 'scala', 'java')
== Optimized Execution Plan ==
Calc(select=[CAST('java' AS VARCHAR(5)) AS EXPR$0]) <- ELT folded away
-- ELT(b, 'scala', 'java') where b TINYINT
== Optimized Execution Plan ==
Calc(select=[ELT(b, 'scala', 'java') AS EXPR$0]) <- ELT reaches the operator
To be clear, these cases do fail without the fix, because the reducer executes the function at plan time. But they only cover the constant-folding path, not the codegen'd operator path that a real job hits, so a future regression in the runtime path wouldn't be caught here.
The case just below at line 201 already uses the field-reference pattern, so extending the existing fields covers it:
.onFieldsWithData(null, null, null, new byte[] {1, 2, 3}, (byte) 2, (short) 2, 2L)
.andDataTypes(
DataTypes.INT(), DataTypes.STRING(), DataTypes.BYTES(), DataTypes.BYTES(),
DataTypes.TINYINT(), DataTypes.SMALLINT(), DataTypes.BIGINT())and then $("f4").elt("scala", "java"), $("f5"), $("f6"), keeping one of the constant cases so the reducer path stays covered too.
I'm the reporter of FLINK-40338 and already have these written and verified locally (they fail with ClassCastException on java.lang.Byte/Short/Long without the fix). Happy to hand them over for you to include here, or to open them as a follow-up if you'd rather keep this PR as is, whichever you prefer.
| .testResult( | ||
| lit(2).cast(DataTypes.TINYINT()).elt("scala", "java"), | ||
| "ELT(CAST(2 AS TINYINT), 'scala', 'java')", | ||
| "java", | ||
| DataTypes.VARCHAR(5)) | ||
| .testResult( | ||
| lit(2).cast(DataTypes.SMALLINT()).elt("scala", "java"), | ||
| "ELT(CAST(2 AS SMALLINT), 'scala', 'java')", | ||
| "java", | ||
| DataTypes.VARCHAR(5)) | ||
| .testResult( | ||
| lit(2).cast(DataTypes.BIGINT()).elt("scala", "java"), | ||
| "ELT(CAST(2 AS BIGINT), 'scala', 'java')", | ||
| "java", | ||
| DataTypes.VARCHAR(5)) |
There was a problem hiding this comment.
Just in case write a test with non-literals too please
There was a problem hiding this comment.
@raminqaf agreed. Here's the patch I have locally, already verified, it fails with ClassCastException on java.lang.Byte, Short and Long before the fix and passes after:
.onFieldsWithData(null, null, null, new byte[] {1, 2, 3}, (byte) 2, (short) 2, 2L)
.andDataTypes(
DataTypes.INT(), DataTypes.STRING(), DataTypes.BYTES(), DataTypes.BYTES(),
DataTypes.TINYINT(), DataTypes.SMALLINT(), DataTypes.BIGINT()).testResult(
$("f4").elt("scala", "java"),
"ELT(f4, 'scala', 'java')",
"java",
DataTypes.VARCHAR(5))
.testResult(
$("f5").elt("scala", "java"),
"ELT(f5, 'scala', 'java')",
"java",
DataTypes.VARCHAR(5))
.testResult(
$("f6").elt("scala", "java"),
"ELT(f6, 'scala', 'java')",
"java",
DataTypes.VARCHAR(5))@hulincup feel free to take these directly.
Problem fixed & how
ELT(index, expr, exprs...)accepts anyINTEGER_NUMERICindex (TINYINT/SMALLINT/INT/BIGINT). However,EltFunction#evaldeclaresindexasjava.lang.Numberand indexes the varargs array withexprs[(int) index - 1]. Per JLS 5.5, casting aNumberreference tointcompiles to acheckcasttoIntegerfollowed by unboxing, so aByte,Short, orLongvalue throwsClassCastExceptionon the success path (1 <= index <= exprs.length).Same for
CAST(2 AS TINYINT)andCAST(2 AS SMALLINT).The out-of-range guard above the cast uses
index.longValue(), so out-of-range indices of any type still returnNULLcorrectly; the exception only fires in the valid range. That is why the existing testELT(9223372036854775807, 'ab', 'b')passes (returnsNULLbefore reaching the cast) and every other existing test uses anINTliteral.Present since FLINK-35987 introduced ELT; confirmed absent from release-1.20 and present from release-2.0.
Fix: narrow the already-unboxed
long idx(computed above for the range check) via primitive narrowing(int) idx(JLS 5.1.3, nocheckcast) instead of casting theNumberreference.Behavior modified
ELTwith a non-INTINTEGER_NUMERICindex (TINYINT/SMALLINT/BIGINT) in the valid range1 <= index <= exprs.lengththrewClassCastException.INTindices,NULL, and out-of-range behavior are unchanged.Code refactored
EltFunction.eval:exprs[(int) index - 1]→exprs[(int) idx - 1], with a 3-line comment explaining the JLS rationale.Features added
N/A
Functions optimized
N/A
Test plan
StringFunctionsITCase.eltTestCases()coveringTINYINT,SMALLINT, andBIGINTindices (the three types that previously threwClassCastException). Each expects the correct expression ("java"for index 2).