zhangyiss/mlx
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx.git synced 2025-07-27 20:07:59 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix transformer decoder post norm LN (#1637)

Browse Source
This commit is contained in:
Awni Hannun 2024-12-02 07:02:17 -08:00 committed by GitHub
parent 974bb54ab2
commit aa86876813
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
python/mlx/nn/layers/transformer.py
View File

@ -238,7 +238,7 @@ class TransformerDecoderLayer(Module):
y = self.cross_attention(y, memory, memory, memory_mask) y = self.cross_attention(y, memory, memory, memory_mask)
y = self.dropout2(y) y = self.dropout2(y)
x = self.ln1(x + y) x = self.ln2(x + y)
y = self.linear1(x) y = self.linear1(x)
y = self.activation(y) y = self.activation(y)