Repository navigation
fix(paraformer): skip blank lines when loading the hotword seg_dict - #3907
Open
MohammadHijjawi97 wants to merge 1 commit into
Open
MohammadHijjawi97 wants to merge 1 commit into
MohammadHijjawi97 wants to merge 1 commit into
Conversation
SeACo-Paraformer and Contextual Paraformer load the model directory's seg_dict with their own copy of load_seg_dict, which read the first field of every line without checking it. A blank or whitespace-only line made generate_hotwords_list raise IndexError whenever hotwords were passed. Skip empty lines, as funasr.tokenizer.char_tokenizer.load_seg_dict does.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SeACo-Paraformer and Contextual Paraformer load the model directory's
seg_dictwith their own copy ofload_seg_dict, which reads the first field of every line without checking it. If theseg_dictcontains a blank or whitespace-only line (for example a trailing empty line after editing it), any hotword request fails ingenerate_hotwords_list:This change skips empty lines, matching
funasr.tokenizer.char_tokenizer.load_seg_dict(#3903).Type of change
Validation
test_blank_lines_in_seg_dict_are_skippedtotests/test_paraformer_hotword_file.pyfor both model classes. It fails withIndexErrorbefore the change and passes after.python -m pytest tests/test_paraformer_hotword_file.py tests/test_char_tokenizer_seg_dict.py: 7 passedpython -m compileall funasr examples testsUser impact
Users of SeACo-Paraformer or Contextual Paraformer hotwords whose model directory
seg_dictcontains blank lines get their hotwords applied instead of anIndexError.Notes for reviewers
Two lines per model file. Non-empty lines are parsed exactly as before. Follow-up to #3903, which fixed the same pattern in
char_tokenizer.py.