Skip to content

feat(korean): introduce coarse POS tag mapping for Hangul multicontext tokenizer - #890

Open
kahye wants to merge 1 commit into
ReaLLMASIC:masterfrom
kahye:coarse-pos-tokenizer
Open

feat(korean): introduce coarse POS tag mapping for Hangul multicontext tokenizer#890
kahye wants to merge 1 commit into
ReaLLMASIC:masterfrom
kahye:coarse-pos-tokenizer

Conversation

@kahye

@kahye kahye commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator
Consolidate fine-grained Sejong/Kiwi POS tags (45 tags) into 17 high-level linguistic categories in `HangulFactorizedTokenizer`:
- `NOUN` (NNG, NNP, NNB)
- `PRON` (NP)
- `NUM` (NR, SN)
- `VERB` (VV, VX)
- `ADJ` (VA)
- `COP` (VCP, VCN)
- `ADV` (MAG, MAJ)
- `DET` (MM)
- `INTJ` (IC)
- `JOSA` (JKS, JKC, JKG, JKO, JKB, JKV, JKQ, JX, JC)
- `EOMI_PRE` (EP)
- `EOMI_FINAL` (EF)
- `EOMI_CONN` (EC)
- `EOMI_MOD` (ETM, ETN)
- `AFFIX` (XPN, XSN, XSV, XSA, XR)
- `PUNCT` (SF, SP, SS, SE, SO, SW, W_URL, W_EMAIL, W_HASHTAG, W_MENTION)
- `FOREIGN` (SL, SH)

### Changes
- Added `POS_TAG_MAP` and updated `POS_TAGS` in `data/template/utils/korean/hangul_factorizer.py`.
- Mapped token tags in `_features()`, `encode_char()`, and `encode_text()`.
- Updated test cases in `data/template/utils/korean/test_hangul_factorizer.py`.
- Updated lane 23 POS values in `data/korean_pos_mc/lane_metadata.json`.

### Testing
- `PYTHONPATH=data/template/utils/korean python3 -m unittest test_hangul_factorizer.py` (all tests passed)

…t tokenizer

- Map 45 fine-grained Kiwi POS tags into 17 coarse linguistic categories (NOUN, PRON, NUM, VERB, ADJ, COP, ADV, DET, INTJ, JOSA, EOMI_PRE, EOMI_FINAL, EOMI_CONN, EOMI_MOD, AFFIX, PUNCT, FOREIGN)
- Update HangulFactorizedTokenizer._features, encode_char, and encode_text to automatically map tags via POS_TAG_MAP
- Update test_hangul_factorizer unit tests to verify coarse tag mapping
- Update lane_metadata.json POS lane values to match the 17 coarse tags plus PAD and UNK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant