zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-07-19 17:41:11 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch 'adding-support-for-mamba2' of https://github.com/Goekdeniz-Guelmez/mlx-examples into adding-support-for-mamba2

Browse Source
This commit is contained in:
Goekdeniz-Guelmez 2024-10-22 18:26:05 +02:00
commit 5326d9373a
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
llms/mlx_lm/tokenizer_utils.py
View File

@ -193,7 +193,9 @@ class BPEStreamingDetokenizer(StreamingDetokenizer):
self.tokens = []
def _maybe_trim_space(self, current_text):
if current_text[0] != " ":
if len(current_text) == 0:
return current_text
elif current_text[0] != " ":
return current_text
elif not self.text:
return current_text[1:]