Skip to content

fix: failed_count is always zero in result analysis#178

Open
WatchTree-19 wants to merge 1 commit into
EleutherAI:mainfrom
WatchTree-19:fix-failed-count-metric
Open

fix: failed_count is always zero in result analysis#178
WatchTree-19 wants to merge 1 commit into
EleutherAI:mainfrom
WatchTree-19:fix-failed-count-metric

Conversation

@WatchTree-19

Copy link
Copy Markdown

problem

in compute_confusion (delphi/log/result_analysis.py), failed_count is computed as len(df_valid) - total, but total is itself len(df_valid), so failed_count is always 0. as a result the "Average fraction of failed examples" that log_results prints is always 0.0, even when the scorer failed to produce a (parseable) prediction for many examples (those rows are dropped by the prediction.notna() filter earlier in the function).

fix

-        failed_count=len(df_valid) - total,
+        failed_count=len(df) - total,

len(df) - total is the number of rows removed by the .notna() filter, i.e. the examples that failed.

test

added tests/test_log/test_result_analysis.py (pandas-only): builds a frame with 2 NaN predictions (what classifier.py stores when generation/parse fails) and asserts failed_count == 2 and fraction 0.5. fails on current main (assert 0 == 2), passes with the fix. ruff clean.

found by reading the code.

Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant