Update for the training data preprocessing - #968
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the training pipeline’s preprocessing and error surfacing so invalid training-folder setups are rejected earlier (before model initialization), while also refining dataset splitting and upsampling behavior and adding coverage for the new/changed logic.
Changes:
- Add preflight validation for training folders that contain no supported audio files, and propagate localized errors with placeholder payloads to the GUI.
- Refactor
random_split,random_multilabel_split, andupsamplinglogic to reduce duplication/inefficiencies and adjust shuffling behavior around validation splitting. - Add/extend tests for training-data loading validation, split correctness, and upsampling behavior; add new localization keys for new validations.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/train/test_train_loading.py | Adds regression test ensuring empty class folders are detected before model loading. |
| tests/train/test_model.py | Adds unit tests covering split and upsampling behavior. |
| birdnet_analyzer/train/utils.py | Adds _check_input_folders and runs it before model setup; fixes validation message typo. |
| birdnet_analyzer/model.py | Refactors split and upsampling implementations; adjusts training-data shuffling and enables shuffle=True in Keras fit. |
| birdnet_analyzer/gui/train.py | Formats localized validation messages with payload arguments for placeholders. |
| birdnet_analyzer/lang/en.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/de.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/fr.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/fi.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/id.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/pt-br.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/ru.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/se.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/tlh.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/zh_TW.json | Adds localization strings for new validations and multi-label upsampling restriction. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+167
to
+172
| has_audio_file = any( | ||
| entry.is_file() | ||
| and not entry.name.startswith(".") | ||
| and entry.name.rsplit(".", 1)[-1].lower() in ALLOWED_FILETYPES | ||
| for entry in os.scandir(folder_path) | ||
| ) |
Comment on lines
+242
to
+243
| train_indices = np.concatenate(train_indices) | ||
| val_indices = np.concatenate(val_indices) |
Comment on lines
+281
to
+285
| source_indices = np.flatnonzero(y == minority_label) | ||
| missing_samples = min_samples - len(source_indices) | ||
|
|
||
| while np.where(y == minority_label)[0].shape[0] + len(y_temp) < min_samples: | ||
| random_index = rng.choice(np.where(y == minority_label)[0], size=size) | ||
| x_app, y_app = apply(x, y, random_index) | ||
| for _ in range(max(0, missing_samples)): | ||
| random_index = rng.choice(source_indices, size=size) |
wcornwell
added a commit
to wcornwell/BirdNET-Analyzer
that referenced
this pull request
Aug 9, 2026
Five additions from a 2026-08-09 maintenance pass: - .git size: 1.2 GB is the floor (upstream model binaries; rewriting history would break upstream merges). Above that is loose garbage from staged-then- unstaged audio. Auto-gc never fires because it counts objects, not bytes, so this recurs silently -- `git gc --prune=now` reclaimed 1.24 GB. - Branch listing: `git branch -a` showed ~24 phantom origin refs for branches deleted on GitHub. Pruned; origin is now main + gh-pages only. Use `git ls-remote --heads origin` for ground truth. - Upstream sync 2026-08-09 parked on upstream-sync-2026-08 (not merged): upstream birdnet-team#968 rewrites the split/upsampling path and subsumes our added_count fix, so the fork no longer carries it. Records the auto-merged `added_count += 1` trap, the (y == 1) count fix, and why this one needs a full-reallybig + soundscape-eval gate rather than subset smoke. - .venv has lost its [tests]/[dev] extras -- no pytest, no ruff -- which is why that sync is unvalidated. - Stray data (~1.55 GB) migrated to call_library and deleted. Two gotchas worth keeping: cc_output was NOT reproducible (its 1STSMM2 source audio exists nowhere), and OneDrive dataless stubs fake checksum mismatches unless you check `stat -f%b` first. Also records the audition verdict on sl1_segments: nothing was worth adding to reallybig (global-model output, species already well covered, confidence caps saturated). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.