Robot Learning

Activation Function

An activation function is the elementwise nonlinearity applied to a neural network layer's output, enabling the network to represent functions beyond linear maps. ReLU and its variants dominate convolutional and policy networks for their cheap computation and stable gradients, while GELU and SwiGLU are standard in transformer backbones. Saturating functions like tanh are still used to bound continuous action outputs to a normalized range.

Why it matters for physical AI

Bounded output activations keep policy commands within actuator limits, and activation choice affects the smoothness of learned control signals, which translates directly into motor wear and motion quality on hardware.