zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-09-03 23:14:34 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fix detokenizer space match for quote (#1072)

Browse Source 
* fix + test

* remove transformer flax/torch warning

* format
This commit is contained in:
Awni Hannun
2024-10-27 15:06:07 -07:00
committed by GitHub
parent ab4bf05c6e
commit 8fe9539af7
3 changed files with 9 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
llms/mlx_lm/tokenizer_utils.py
View File

@@ -169,7 +169,7 @@ class BPEStreamingDetokenizer(StreamingDetokenizer):
"""
_byte_decoder = None
_space_matches = (".", "?", "!", ",", "'", "n't", "'m", "'s", "'ve", "'re")
_space_matches = (".", "?", "!", ",", "n't", "'m", "'s", "'ve", "'re")
def __init__(self, tokenizer):