Skip to content

fix(paraformer): skip blank lines when loading the hotword seg_dict - #3907

Open
MohammadHijjawi97 wants to merge 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/paraformer-seg-dict-blank-lines
Open

MohammadHijjawi97 wants to merge 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/paraformer-seg-dict-blank-lines

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor

Summary

SeACo-Paraformer and Contextual Paraformer load the model directory's seg_dict with their own copy of load_seg_dict, which reads the first field of every line without checking it. If the seg_dict contains a blank or whitespace-only line (for example a trailing empty line after editing it), any hotword request fails in generate_hotwords_list:

File "funasr/models/seaco_paraformer/model.py", line 731, in load_seg_dict
    key = s[0]
IndexError: list index out of range

This change skips empty lines, matching funasr.tokenizer.char_tokenizer.load_seg_dict (#3903).

# seg_dict: "阿 阿\n\n里 里\n   \n巴 巴\n云 云\n\n", hotwords "阿里巴巴 阿里云"
# before: IndexError
# after:  [[2, 3, 4, 4], [2, 3, 5], [1]]

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • Added test_blank_lines_in_seg_dict_are_skipped to tests/test_paraformer_hotword_file.py for both model classes. It fails with IndexError before the change and passes after.
  • python -m pytest tests/test_paraformer_hotword_file.py tests/test_char_tokenizer_seg_dict.py: 7 passed
  • python -m compileall funasr examples tests
  • Docs or links checked
  • Runtime/deployment command tested

User impact

Users of SeACo-Paraformer or Contextual Paraformer hotwords whose model directory seg_dict contains blank lines get their hotwords applied instead of an IndexError.

Notes for reviewers

Two lines per model file. Non-empty lines are parsed exactly as before. Follow-up to #3903, which fixed the same pattern in char_tokenizer.py.

SeACo-Paraformer and Contextual Paraformer load the model directory's
seg_dict with their own copy of load_seg_dict, which read the first field
of every line without checking it. A blank or whitespace-only line made
generate_hotwords_list raise IndexError whenever hotwords were passed.
Skip empty lines, as funasr.tokenizer.char_tokenizer.load_seg_dict does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant