Byte fallback Hangul POS Tokenizer - #891
Open
kahye wants to merge 3 commits into
Open
Conversation
…, and resilience experiment report
…ility evaluation suite - Expand Korean POS dataset in get_dataset.sh with OPUS-100 and KLUE task splits (DP, NER, MRC, NLI, RE, STS, YNAT) and update lane_metadata.json - Add --pos_loss_weight argument in train_args.py and handle POS loss weighting and milestone checkpoint saving in train.py - Add --mc_ckpt and --base_ckpt path override support to benchmarks/run_phonetic_slang_eval.py and benchmarks/run_vocab_tail_perplexity.py - Add 4-capability evaluation benchmark suite (benchmarks/run_four_capability_evals.py) covering KLUE-NER, KLUE-DP, noisy text resilience (NSMC/UnSmile), and rare vocabulary/OOV (KorMedMCQA) - Add evaluation runner demos/run_all_epoch_evals.sh, Option 1 sweep runner run_option1_sweep.py, and 10-epoch experiment runner run_opt1_10ep_experiment.py
… POS tagsets, and benchmark support - Implement HangulFullPosFactorizedTokenizer (46 Sejong tags) and HangulCoarsePosFactorizedTokenizer (17 mapped macro tags) in hangul_factorizer.py - Add make_byte_fallback_meta() to support 256-byte companion character stream without OOV drop - Update POS lane metadata and unit tests in test_hangul_factorizer.py - Add 59.5M token milestone checkpoint saves (3ep: 10899, 5ep: 18165, 10ep: 36330) in train.py - Add prepare_pos_and_byte_lanes.py to prepare Full POS and 256-Byte Fallback companion stream - Add run_pos_byte_experiments.py automation runner for training and evaluating Full vs Coarse POS under Weighted and Unweighted loss - Update evaluation benchmarks (run_four_capability_evals.py, run_ko_hellaswag.py, run_vocab_tail_perplexity.py, run_phonetic_slang_eval.py) to support byte fallback and full/coarse POS models
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
• Hangul POS Tokenizer & Byte Fallback:
• hangul_factorizer.py: Added HangulFullPosFactorizedTokenizer (46 Sejong tags) and HangulCoarsePosFactorizedTokenizer (17 mapped macro tags).
• Implemented hangul_factorizer.py:280-290 to support the synchronous 256-byte companion stream without OOV token loss.
• test_hangul_factorizer.py: Added unit tests for Full POS, Coarse POS, and 256-byte fallback metadata.
• Dataset Preparation & Experiment Runners:
• prepare_pos_and_byte_lanes.py: Pipeline script to extract and encode Full POS and 256-byte fallback companion stream from the 59.5M token corpus.
• run_pos_byte_experiments.py: Multi-model training and evaluation orchestrator across all 4 architectures and epoch milestones.
• Training & Benchmarks:
• train.py:2661: Added checkpoint milestone saves at iterations 10899 (3ep), 18165 (5ep), and 36330 (10ep) on 59.5M tokens.
• run_four_capability_evals.py, run_ko_hellaswag.py, run_vocab_tail_perplexity.py, run_phonetic_slang_eval.py: Updated evaluation runners to dynamically
detect and support 256-byte companion streams, Full POS (46 tags), Coarse POS (17 tags), and baseline character models.