Nripesh Niketan
e09bf35b28
feat: Add Dropout3d layer to nn.layers ( #313 )
...
* feat: Add Dropout3d layer to nn.layers
* acknowledgement
* Add dropout tests to test_nn.py
* run pre-commit
* Add activation functions and dropout3d ops
* Add dropout tests for bfloat16 and float16
2023-12-31 14:01:21 -08:00
Hazem Essam
e3b8da2a49
Added implementation for Scaled RoPE. ( #261 )
...
* Added scale for RoPE
* Ran pre-commit
* Added RoPE scaling test
* Added docstring for scale parameter
* Modified docstrings
2023-12-31 06:06:01 -08:00
Nripesh Niketan
5ad8fb7268
feat: add softsign, softmax, hardswish, logsoftmax activation function ( #309 )
...
* feat: add softsign activation function
* run pre-commit
* Add Softsign activation function
* Add Softsign activation function
* Add documentation for ReLU6, Softplus, and Softsign activations
* Update activation functions in neural network layers
* Add LogSoftmax and Hardswish activations
* run pre-commit
* Update activations.py
* Added acknowledgements
* Fix activation function comments
* Fix activation functions in neural network layers
2023-12-29 11:49:36 -08:00
Chunyang Wen
2aedf3e791
Minor refactor for tree_map and tree_unflatten ( #311 )
...
* Minor refact for tree_map and tree_unflatten
* Remove the if statement
---------
Co-authored-by: Chunyang Wen <chunyang_wen@apple.com>
2023-12-28 20:55:10 -08:00
Chunyang Wen
473b6b43b4
Use defaultdict ( #307 )
...
Co-authored-by: Chunyang Wen <chunyang_wen@apple.com>
2023-12-28 14:46:13 -08:00
Angelos Katharopoulos
d29770eeaa
Update batchnorm to have the running stats in parameters ( #305 )
2023-12-28 14:31:10 -08:00
Chunyang Wen
040c3bafab
Add missing f str ( #306 )
...
Co-authored-by: Chunyang Wen <chunyang_wen@apple.com>
2023-12-28 06:09:34 -08:00
Chunyang Wen
05767b026f
Add information for dropout probability ( #304 )
...
Co-authored-by: Chunyang Wen <chunyang_wen@apple.com>
2023-12-27 21:51:30 -08:00
YUN, Junwoo
4417e37ede
Transformer fix ( #167 )
...
* add transformer with dropout, fix transformer ffm, layernorm order
* precommit changes
* precommit changes
* add docstring, activation, norm_first
* run precommit
* run precommit
* add doctstring
* precommit
* style nits in docs
---------
Co-authored-by: junwoo-yun <junwoo.yun@bagelcode.com>
Co-authored-by: Awni Hannun <awni@apple.com>
2023-12-27 08:48:36 -08:00
__mo_san__
a123c3c7d2
implement-batch-norm-layer ( #217 )
...
- Add batch normalization layer
---------
Co-authored-by: Robert McCraith <mccraithrobert@gmail.com>
Co-authored-by: Awni Hannun <awni@apple.com>
2023-12-25 07:32:53 -08:00
Zach Schillaci
22fee5a383
Remove redundant assert in losses.py ( #281 )
2023-12-24 08:39:08 -08:00
Vidit Agarwal
acf1721b98
Corrected the example of value_and_grad ( #274 )
...
* Corrected the example for mx.value_and_grad
* Reformat through pre-commit/black
2023-12-23 11:06:38 -08:00
Nicholas Santavas
d35fa1db41
Add Hinge, Huber and LogCosh losses ( #199 )
2023-12-22 10:28:10 -08:00
Justin Deschenaux
e8deca84e0
Add dropout2d ( #250 )
2023-12-22 08:02:29 -08:00
Hazem Essam
0aa65c7a6b
Added ALiBi implementation ( #232 )
2023-12-21 14:36:38 -08:00
Angelos Katharopoulos
b3916cbf2b
Improve names of quantization arguments ( #235 )
...
* Change the default quantization group_size to 64
* Rename groups to group_size and width to bits
2023-12-20 16:53:53 -08:00
Angelos Katharopoulos
57fe918cf8
Adds C++ and nn quantization utilities ( #230 )
...
* Add C++ de-/quantize ops
* Add quantize functions to the docs and tests
* Add a QuantizedLinear module
2023-12-20 14:17:38 -08:00
Justin Deschenaux
4912ff3ec2
Add Lion optimizer ( #209 )
...
* Add Lion optimizer
* Update acknowledgements also with past contributions
2023-12-20 13:54:58 -08:00
Emircan Erol
e549f84532
Triplet Loss ( #211 )
...
* Triplet Loss
* Requested Changes
* Margin to alpha
2023-12-19 12:37:12 -08:00
Juarez Bochi
f4f6e17d45
Fix cross-attention ( #210 )
...
* Fix cross-attention
With the current code, ln2 is a no-op. Its output should be passed to the cross-attention layer
* Add name to contributors
2023-12-18 12:27:27 -08:00
jojopuppet
18cca64c81
Add smoothed L1 loss and enhancements to cross entropy loss ( #166 )
...
* Add smooth_l1_loss
* Add labels moothing for cross entropy loss
---------
Co-authored-by: Awni Hannun <awni@apple.com>
2023-12-18 07:26:21 -08:00
Awni Hannun
ee0c2835c5
Docs updates ( #198 )
...
Reorganize NN docs + a few other tidbits.
2023-12-17 13:20:55 -08:00
YUN, Junwoo
eebd7c275d
Add optimizers (AdaMax, AdaDelta, RMSprop) and ordering optimizer classes ( #142 )
...
* Add AdaMax, AdaDelta, RMSprop
2023-12-16 21:43:15 -08:00
Awni Hannun
2e02acdc83
add base kwarg to rope ( #186 )
2023-12-15 16:47:59 -08:00
Víctor Aguilar
f24200db2c
accross -> across ( #183 )
2023-12-15 13:46:50 -08:00
Awni Hannun
25f70d4ca4
Fix divide types + floor divide (//) ( #138 )
...
* divide types
* fix black + test
2023-12-11 20:20:58 -08:00
Diogo
02de234ef0
Activations LeakyReLU / PReLU / Softplus / Mish ( #109 )
...
* Leaky_relu / prelu / softplus / mish
* added tests
* updated bench
* remove torch refs, add init to PReLU
* added arvix reference to mish
* added missing docs
2023-12-11 19:40:57 -08:00
Nicholas Santavas
f5df47ec6e
Add Step, ELU, SELU, Swish activation functions ( #117 )
...
* Add Step, ELU, SELU, Swish activation functions
This commit adds the Step, ELU, SELU and Swish activations functions
* add to the docs
* review
2023-12-11 17:04:07 -08:00
Awni Hannun
69505b4e9b
fixes ( #131 )
2023-12-11 09:26:49 -08:00
__mo_san__
f4ddd7dc44
Add Binary Cross Entropy loss ( #122 )
...
* update BCE added tests for it ...
* added binary cross entropy loss to docs
* resolving conflicts for merge
2023-12-11 07:55:18 -08:00
Jason
b0cd092b7f
Added activation functions: leaky_relu relu6 softplus elu celu logsigmoid ( #108 )
...
* added leaky_relu relu6 softplus elu celu logsigmoid
* minor fixes for docstring and benchmark imports
* fixed elu implementation and added tests
* added tests for optional param, changed leaky_relu param to fit pytorch documentation
2023-12-10 16:31:38 -08:00
Awni Hannun
71d1fff90a
Bug fix in metal binary kernel dispatch for large arrays ( #125 )
...
* bug fix
* format
2023-12-10 16:12:31 -08:00
Awni Hannun
2d0130f80f
fix loss tests ( #118 )
...
* fix loss tests
* use none as default
2023-12-10 10:08:19 -08:00
__mo_san__
c1e1c1443f
Added Adagrad optimizer ( #102 )
2023-12-10 09:22:39 -08:00
Henry Ansah
68bf1d7867
add nn module for sigmoid activation ( #111 )
...
* add nn module for sigmoid activation
* update .gitignore with .cache folder generated by jetbrains fleet ide
* remove .cache folder
2023-12-10 07:00:39 -08:00
Angelos Katharopoulos
600db7d754
Fix build on Xcode 14 ( #116 )
...
* Fix build on Xcode 14
* Style fixes
2023-12-10 06:58:52 -08:00
__mo_san__
ef7b8756c0
Add tanh activation function ( #115 )
...
* added Adagrad optimizer ...
* added Tanh activation function ...
* reformatted file ...
* remove unrelated stuff ...
* Update activations.py
2023-12-09 19:25:38 -08:00
Enoch Kan
0b28399638
added mse_loss, nll_loss and kl_div_loss ( #98 )
...
* added mse_loss, nll_loss and kl_div_loss
* fixed axis not defined error in nll_loss
* fixed axis not defined in kl_div_loss
* added tests for mse, nll and kl_div
* modified docstrings and added reduce helper func
* updated docstring in kl_div_loss and moved helper func
* added new kl divergence implementation
* added reduction to test
* updated docstring of kl_div_loss with correct spelling
* added losses to nn.rst in docs
2023-12-09 14:25:03 -08:00
Joe Barrow
ac6dc5d3eb
Adding optional bias param to MultiHeadAttention ( #104 )
...
* Adding optional param to
* Run style-checker
2023-12-09 11:04:28 -08:00
Awni Hannun
2520dbcf0a
add losses to the docs, fix black failur ( #92 )
2023-12-09 06:06:52 -08:00
Abe Leininger
430bfb4944
Adds Nesterov momentum to SGD ( #87 )
2023-12-08 23:23:36 -08:00
Kai Ma
cb9e585b8e
Style fix for loss functions ( #91 )
...
* MLE and L1 loss functions
* logsoftmax change and tests
* subtract max logit for numerical stability
* l1 name change
* cross entropy reduction + unit tests
* docstrings
* l1 test name change
* old loss impl + default none
* style
2023-12-08 21:11:56 -08:00
Kai Ma
641d316484
MLE and L1 loss functions ( #88 )
...
* MLE and L1 loss functions
* logsoftmax change and tests
* subtract max logit for numerical stability
* l1 name change
* cross entropy reduction + unit tests
* docstrings
* l1 test name change
* old loss impl + default none
2023-12-08 20:21:37 -08:00
Joe Barrow
69a24e6a1e
AdamW implementation ( #72 )
...
* AdamW implementation without bias correction
* Makes use of the underlying Adam implementation
2023-12-08 14:45:34 -08:00
Zach Schillaci
5b9be57ac3
Add isort pre-commit and run ( #68 )
2023-12-08 11:31:47 -08:00
Zach Schillaci
d11d77e581
Spelling fixes in transformer.py ( #59 )
2023-12-07 13:32:09 -08:00
rushyam
2e126aeb7e
Feature Addition: Encoder-Decoder Transformer Architecture ( #50 )
...
* Implemented decoder-transformer-layer, decoder-transformer and introduce encoder-decoder transformer
* added relu layer
* add src, tgt, memory mask
---------
Co-authored-by: rushyam <rushyam@rushyams-MacBook-Air.local>
2023-12-07 07:37:36 -08:00
Jagrit Digani
2440fe0124
NPY loading segfault bug ( #34 )
...
* Fixed Gil semantics in loading and saving from python file streams
2023-12-06 12:03:47 -08:00
Markus Enzweiler
2ffaee0c0d
Updated default argument for stride to 1 in Conv2d() in the docstring ( #22 )
2023-12-06 07:17:58 -08:00
Awni Hannun
db487e6b1a
format
2023-11-30 11:50:50 -08:00
Awni Hannun
46a39e5b1f
copyright + ack
2023-11-30 11:12:53 -08:00
Jagrit Digani
e6306cfee9
jagrit's commit files
2023-11-29 10:52:08 -08:00
Angelos Katharopoulos
d1f86272a2
angelos's commit files
2023-11-29 10:42:59 -08:00
Awni Hannun
8ca7f9e8e9
awni's commit files
2023-11-29 10:30:41 -08:00
