zhangyiss/mlx
1
0
Fork 0
mirror of https://github.com/ml-explore/mlx.git synced 2025-09-18 01:50:16 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Fix cross-attention (#210)

Browse Source 
* Fix cross-attention

With the current code, ln2 is a no-op. Its output should be passed to the cross-attention layer

* Add name to contributors
This commit is contained in:
Juarez Bochi
2023-12-18 15:27:27 -05:00
committed by GitHub
parent 4d4af12c6f
commit f4f6e17d45
2 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
python/mlx/nn/layers/transformer.py
View File

@@ -157,7 +157,7 @@ class TransformerDecoderLayer(Module):
x = x + y
y = self.ln2(x)
y = self.cross_attention(x, memory, memory, memory_mask)
y = self.cross_attention(y, memory, memory, memory_mask)
x = x + y
y = self.ln3(x)