feat(diffusion): periodic validation loss during training

[beccohov]

What does this PR do ?

Adds periodic validation loss to the diffusion training recipe, so checking eval loss no longer means saving a checkpoint and running the eval set through the model by hand.

Changelog

• New optional data.validation_dataloader block, resolved by the same resolve_diffusion_dataloader as the training loader (_typed_config.py), so it takes the same four builders and the same field validation. One loader, matching the training side, rather than the dict of named loaders the LLM recipe has.
• The recipe builds it in setup() with the same dp sharding and batch size as the training loader, and raises if it comes out empty. Half-configured setups get a warning instead of silence: a loader without val_every_steps validates on checkpoint steps only, val_every_steps without a loader is skipped.
• _run_validation_epoch scores the held-out set with the training flow-matching objective in eval() under torch.no_grad(), restores train mode in a finally, and reduces [loss_sum, batch_count] in one all-reduce over the dp group, so the mean does not depend on how batches are spread across ranks. CP is excluded from that reduction since every peer computes the same full-sequence loss.
• The pass runs inside ScopedRNG(seed=self.seed). Without it the sampled timesteps dominate the curve and validation would consume the training RNG stream, changing the run. FP8 autocast is deliberately not applied: delayed-scaling amax history is training state and should not be updated from held-out batches.
• The loop calls it on step_scheduler.is_val_step, before the checkpoint branch, and logs val_loss to the console and to W&B.
• Example config (wan2_2_t2v_flow.yaml) and the diffusion fine-tuning guide.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?

Additional Information

• Closes Support periodic validation loss evaluation for diffusion training #3244
• Unit tests cover the averaging in eval mode, train mode restored after a failing batch, repeatable sampling with the training RNG left untouched, the dp reduction, and that the loop validates only when a loader is configured.
• Ran it for real on 2x H200 with FSDP2 (dp_size: 2) against a small Wan transformer: validation fired at the configured steps and the run finished clean. Also ran the same training twice, with and without validation enabled, and the training losses and final weights come out bit-identical, which is the property the fixed seed and RNG restore are there for.
• Not exercised: cp_size > 1, tp_size > 1, multi-node, and the LoRA and FP8 paths.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[beccohov]

Hey @HuiyingLi,
I created PR as we agreed in #3244

[HuiyingLi]

Thank you! Taking a look

[HuiyingLi]

/claude review

[claude]

LGTM

[jgerh]

Completed tech pubs review of docs/guides/diffusion/finetune.mdx and provided a few copyedits and suggested text revisions.

[HuiyingLi]

/ok to test bdf62d5

[github-actions]

🌿 Preview your docs: https://nvidia-preview-preview-01a0d3db9687.docs.buildwithfern.com/nemo/automodel

[beccohov]

Thanks, @HuiyingLi
I'll wait till the tests finish and then probably I should update the docs according to what @jgerh suggested?

[HuiyingLi]

@zyzhou5 could you please take a look too, thanks!

[akoumpa]

/ok to test 01a0d3d