zhangyiss/mlx-examples
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx-examples.git synced 2025-09-01 04:14:38 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fix argpartition call in Mixtral and other MOES (#676)

Browse Source 
* Update mixtral.py

* fix all moes

---------

Co-authored-by: yuhai-china <yuhai.china@gmail.com>
This commit is contained in:
Awni Hannun
2024-04-12 11:00:56 -07:00
committed by GitHub
parent 9c5554d8ee
commit d3f8e4aee9
4 changed files with 4 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
llms/mixtral/mixtral.py
View File

@@ -103,7 +103,7 @@ class MOEFeedForward(nn.Module):
x = x.reshape(-1, x.shape[-1])
gates = self.gate(x)
inds = mx.argpartition(-gates, kth=ne, axis=-1)[:, :ne]
inds = mx.argpartition(-gates, kth=ne - 1, axis=-1)[:, :ne]
scores = mx.softmax(
mx.take_along_axis(gates, inds, axis=-1).astype(mx.float32),
axis=-1,