# Layers — MLX 0.0.7 documentation

Skip to main content
`Ctrl`+`K`

# Layers

`Sequential`(*modules) | A layer that calls the passed callables in order.
`ReLU`() | Applies the Rectified Linear Unit.
`PReLU`([num_parameters, init]) | Applies the element-wise parametric ReLU.
`GELU`([approx]) | Applies the Gaussian Error Linear Units.
`SiLU`() | Applies the Sigmoid Linear Unit.
`Step`([threshold]) | Applies the Step Activation Function.
`SELU`() | Applies the Scaled Exponential Linear Unit.
`Mish`() | Applies the Mish function, element-wise.
`Embedding`(num_embeddings, dims) | Implements a simple lookup table that maps each input integer to a high-dimensional vector.
`Linear`(input_dims, output_dims[, bias]) | Applies an affine transformation to the input.
`QuantizedLinear`(input_dims, output_dims[, ...]) | Applies an affine transformation to the input using a quantized weight matrix.
`Conv1d`(in_channels, out_channels, kernel_size) | Applies a 1-dimensional convolution over the multi-channel input sequence.
`Conv2d`(in_channels, out_channels, kernel_size) | Applies a 2-dimensional convolution over the multi-channel input image.
`BatchNorm`(num_features[, eps, momentum, ...]) | Applies Batch Normalization over a 2D or 3D input.
`LayerNorm`(dims[, eps, affine]) | Applies layer normalization [1] on the inputs.
`RMSNorm`(dims[, eps]) | Applies Root Mean Square normalization [1] to the inputs.
`GroupNorm`(num_groups, dims[, eps, affine, ...]) | Applies Group Normalization [1] to the inputs.
`InstanceNorm`(dims[, eps, affine]) | Applies instance normalization [1] on the inputs.
`Dropout`([p]) | Randomly zero a portion of the elements during training.
`Dropout2d`([p]) | Apply 2D channel-wise dropout during training.
`Dropout3d`([p]) | Apply 3D channel-wise dropout during training.
`Transformer`(dims, num_heads, ...) | Implements a standard Transformer model.
`MultiHeadAttention`(dims, num_heads[, ...]) | Implements the scaled dot product attention with multiple heads.
`ALiBi`() |
`RoPE`(dims[, traditional, base, scale]) | Implements the rotary positional encoding.
`SinusoidalPositionalEncoding`(dims[, ...]) | Implements sinusoidal positional encoding.