Skip to content

fix(tokenizer): import KoreanCleaner so the korean_cleaner text cleaner works - #3915

Open
sohailshk wants to merge 1 commit into
modelscope:mainfrom
sohailshk:fix/korean-cleaner-import
Open

sohailshk wants to merge 1 commit into
modelscope:mainfrom
sohailshk:fix/korean-cleaner-import

Conversation

@sohailshk

Copy link
Copy Markdown
Contributor

Summary

TextCleaner handles the cleaner type korean_cleaner by calling KoreanCleaner.normalize_text, but funasr/tokenizer/cleaner.py never imports KoreanCleaner, so the branch raises NameError. funasr.bin.tokenize_text --cleaner korean_cleaner offers the option and the training preprocessor TextPreprocessSegDict accepts it as text_cleaner, so both fail on first use. This adds the missing import from the existing funasr/tokenizer/korean_cleaner.py.

Type of change

  • Bug fix

Validation

  • python -m compileall funasr examples tests
  • python -m pytest tests/test_text_cleaner_korean.py -q: 2 failed (NameError: name 'KoreanCleaner' is not defined), 1 passed (unknown cleaner types are still rejected) on current main; 3 passed with the fix.
  • CLI, CPU: printf 'abc 12\n' | python -m funasr.bin.tokenize_text --token_type char --input - --output - --cleaner korean_cleaner raises the NameError on main; with the fix it prints 에 이 비 씨 일 이.
  • Existing test_char_tokenizer_seg_dict and test_char_tokenizer_non_linguistic_symbols: 7 passed with the fix.

User impact

Korean text normalization (digits and Latin letters spelled out in Hangul) for tokenizer preparation and training works through the documented cleaner option.

Notes for reviewers

  • One import; no behavior change for the other cleaner types. tacotron is still a no-op, as before.
  • The new test is added to the existing NumPy matrix workflow with a path trigger for cleaner.py.
  • import funasr already imports funasr.tokenizer.korean_cleaner (it only needs re), so the new import adds no module or dependency.

…er works

TextCleaner handles cleaner type "korean_cleaner" (offered by tokenize_text
--cleaner and usable as text_cleaner in the training preprocessor), but
cleaner.py never imported KoreanCleaner, so the call raised NameError.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant