Transformer fix (#167)

* add transformer with dropout, fix transformer ffm, layernorm order

* precommit changes

* precommit changes

* add docstring, activation, norm_first

* run precommit

* run precommit

* add doctstring

* precommit

* style nits in docs

---------

Co-authored-by: junwoo-yun <junwoo.yun@bagelcode.com>
Co-authored-by: Awni Hannun <awni@apple.com>
This commit is contained in:
YUN, Junwoo
2023-12-28 00:48:36 +08:00
committed by GitHub
parent 79c95b6919
commit 4417e37ede
5 changed files with 220 additions and 77 deletions

View File

@@ -9,7 +9,7 @@ Layers
:toctree: _autosummary
:template: nn-module-template.rst
Embedding
Sequential
ReLU
PReLU
GELU
@@ -17,17 +17,19 @@ Layers
Step
SELU
Mish
Embedding
Linear
QuantizedLinear
Conv1d
Conv2d
BatchNorm
LayerNorm
RMSNorm
GroupNorm
RoPE
MultiHeadAttention
Sequential
QuantizedLinear
Dropout
Dropout2d
Transformer
MultiHeadAttention
ALiBi
RoPE
SinusoidalPositionalEncoding