HDDS-16206. Start MiniOzoneCluster Datanodes finalized - #11111
Conversation
MiniOzoneCluster wrote a datanode.id file before each Datanode started, so that a Datanode given a synthetic hostname could report an IP address it has no way to resolve. A datanode.id file with no VERSION file beside it means "upgraded from an installation predating the upgrade framework", so every Datanode came up at metadata layout version 0 and reached the current version only once SCM ordered it to finalize. On master SCM finalizes those Datanodes before they can satisfy safemode, which hides the problem. With ZDU, pre-finalized Datanodes are valid for all existing operations, so a test that expects finalized Datanodes can be handed one that has not finalized yet and fails intermittently.
|
@hevinhsu could you help to check this when you are available? We have merged HDDS-16018 so technically the workaround in HDDS-14812 that is causing this issue is not needed anymore? Can we revert the workaround instead? |
Thanks for checking! I actually tried reverting it completely at first, but it caused It turns out HDDS-14812 introduced two workarounds. While HDDS-16018 fixed the mapping issue, we still need the Therefore, this PR only keeps the file for those specific test cases and cleans up the rest. |
|
There are actually two workarounds introduced in HDDS-14812:
HDDS-16018 addressed (1), so we can remove that workaround now(This is now tracked in HDDS-16286). However, (2) is still needed for tests using synthetic hostnames, and this issue is related to a side effect introduced by (2). So I don't think we can completely revert the workaround from HDDS-14812 yet. |
What changes were proposed in this pull request?
Every Datanode in a mini cluster starts pre-finalized.
MiniOzoneClusterpre-creates each Datanode'sdatanode.idfile — HDDS-14812 added that so a Datanode given a synthetic hostname can report an IP address it has no way to resolve,ipAddressbeing required at registration — andDatanodeLayoutStoragereads adatanode.idfile with no VERSION file beside it as an upgrade from an install predating the upgrade framework, defaulting to metadata layout version 0. The Datanode reaches the current version only once SCM orders it to finalize, after it registers.Master hides this, because SCM finalizes those Datanodes before they can satisfy safemode and tests wait on them. Under ZDU a pre-finalized Datanode is valid for existing operations and no longer holds up safemode exit, so a write meant for a finalized Datanode can land on one that has not finalized yet — the intermittent
TestFinalizeBlockfailures.The identity file is now written only for Datanodes that are given a synthetic hostname, with the layout version stamped beside it; every other cluster builds its Datanodes the way a real one does. Tests that want an older layout version set it through
UniformDatanodesFactory, which writes the VERSION file itself and takes precedence.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16206
How was this patch tested?
A new parameterized test in
TestMiniOzoneClusterbuilds a cluster with its Datanodes not started and asserts the metadata layout version defaults to the maximum, with and without configured hostnames; both cases fail without the change (expected: <10> but was: <0>).TestRackAwarePlacement— added with HDDS-14812, covering every rack and hostname combination — andTestDNDataDistributionFinalizationpass, as doescheckstyle:checkon both modules.TestHDDSUpgraderan out of memory on this machine before finishing, so it is left to CI.Generated-by: Claude Code (Claude Opus 5)