mlx.nn.GELU

class mlx.nn.GELU(approx='none')

Applies the Gaussian Error Linear Units.

\[\textrm{GELU}(x) = x * \Phi(x)\]

where \(\Phi(x)\) is the Gaussian CDF.

However, if approx is set to ‘precise’ or ‘fast’ it applies

\[\begin{split}\textrm{GELUApprox}(x) &= x * \sigma\left(1.60033 * x \left(1 + 0.0433603 * x^2\right)\right) \\ \textrm{GELUFast}(x) &= x * \sigma\left(1.773 * x\right)\end{split}\]

respectively.

See gelu(), gelu_approx() and gelu_fast_approx() for the functional equivalents and information regarding error bounds.

Parameters:

approx ('none' | 'precise' | 'fast') – Which approximation to gelu to use if any.