RFC 0008: An Apache Iceberg Lakehouse for RADAR-base

[this-Aditya]

Summary

RADAR's data ends up in two file trees, and neither of them can answer a question. Any research question today means downloading and parsing thousands of files, and the next question starts again from zero.

This RFC proposes writing each Kafka topic into an Apache Iceberg table on the object storage we already run, and querying it with Trino. It reads the same topics as a separate consumer group, so the S3 sink, and radar-output-restructure keep working exactly as they do now.

The chart work is ready and will follow as a separate PR on radar-helm-charts.

Checklist

• Linked pre-RFC discussion issue (none yet)
• Used file name rfcs/NNNN-title.md and updated RFC front matter
• Filled all required sections (Summary, Motivation, Design, Security, Migration, Testing)
• Added to index table in README.md
• Considered alternatives and documented open questions
• Applied appropriate status label
• Applied area/type labels as relevant