zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-06-24 01:17:28 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

BUG FIX: Decoding results in garbled text when multiple tokens represent a single character (e.g., Chinese). (#398)

Browse Source 
* Decoding results in garbled text when multiple tokens represent a single character (e.g., Chinese).

* Decoding results in garbled text when multiple tokens represent a single character (e.g., Chinese).
This commit is contained in:
ZHAOKAI WANG 2024-02-01 11:27:29 +08:00 committed by GitHub
parent 94358219cf
commit 0340113e02
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
lora/lora.py
View File

@ -292,8 +292,9 @@ def generate(model, prompt, tokenizer, args):
tokens.append(token.item())
s = tokenizer.decode(tokens)
print(s[skip:], end="", flush=True)
skip = len(s)
if len(s) - skip > 1:
print(s[skip:-1], end="", flush=True)
skip = len(s) - 1
print(tokenizer.decode(tokens)[skip:], flush=True)
print("=" * 10)
if len(tokens) == 0: