01 Foundations

A survey of activation functions

By Hiva Mohammadzadeh · Stanford MSCS, UC Berkeley EECS

Activation functions add the nonlinearity that makes deep learning work. Linear activations cannot model complex data. Sigmoid and tanh were the early standards, but they suffer from vanishing gradients. ReLU dominates ConvNets, though it kills neurons on the negative side, and Leaky ReLU patches that. GeLU and Swish are the modern picks. They are smoother, they converge better, and they outperform ReLU on deeper models. When you choose an activation, think about your gradients, because smoother derivatives mean smoother training.

Why activation functions matter

Every neuron needs a decision-maker, and that decision-maker is the activation function. It takes the weighted sum of inputs and decides what signal to pass forward. Without it, the entire network collapses into a single linear transformation, no matter how deep it is. A 100-layer network with no activation function is mathematically equivalent to a single-layer network.

Activation functions do two things. First, they set the output range of each neuron, such as 0 to 1, -1 to 1, or 0 to infinity. Second, they inject nonlinearity into the network. That nonlinearity lets the model learn the complex patterns in data that matter in real problems.

Comparison curves of Mish, Swish, SELU, ELU, ReLU6, Leaky ReLU, ReLU, Softplus, and Tanh — Figure 1 Comparison curves for Mish, Swish, SELU, ELU, ReLU6, Leaky ReLU, ReLU, Softplus, and tanh.

I have worked with most of these functions, and the choice is not a minor detail. It affects whether your model trains at all, how fast it converges, and the final accuracy you get.

Linear and non-linear activations

The simplest activation function is the identity:

f(x) = x

The range is unbounded, (-infinity, +infinity), and it passes the signal straight through. The identity adds zero complexity, so the model cannot learn anything beyond a linear relationship between inputs and outputs. For the data we work with (images, text, audio, tabular data with interactions), linear relationships are almost never sufficient. A stack of linear layers is still a linear function, so we need nonlinearity.

Non-linear activation functions make neural networks universal function approximators. They let the model generalize across varied data and separate outputs in meaningful ways.

Two terms come up often when you compare activation functions:

Differentiable: You can compute the slope of the function at any point. This matters because backpropagation needs gradients.
Monotonic: The function is either always non-decreasing or always non-increasing. This affects optimization stability.

Sigmoid and tanh

Sigmoid

Sigmoid was the default activation function for years, especially in feedforward networks (FFNs). It maps any input into the range (0, 1), which makes it a natural choice when you need to predict a probability.

σ(z) = 1 / (1 + e -z)

Sigmoid is differentiable and monotonic (though its derivative is not monotonic). It is smooth, it is bounded, and it has an intuitive probabilistic interpretation.

Sigmoid has one big problem, vanishing gradients. For large positive or large negative inputs, the output saturates near 0 or 1, and the gradient there is nearly zero. During backpropagation, those near-zero gradients multiply together across layers, and the network stops learning. I have seen models get stuck in training because of this, where the loss plateaus and no learning-rate change fixes it.

Tanh

Tanh is sigmoid's zero-centered cousin. It maps inputs to the range (-1, 1).

tanh(z) = (e z - e -z) / (e z + e -z)

Side-by-side comparison of Sigmoid and Tanh curves — Figure 2 Side-by-side comparison of the sigmoid and tanh curves.

Tanh improves on sigmoid because negative inputs map to strongly negative outputs and zero inputs map near zero. This zero-centering helps gradient flow and makes optimization easier. Tanh is differentiable and monotonic, and like sigmoid its derivative is not monotonic.

I reach for tanh mainly in binary classification. It is still used in FFNs, but deep networks have better options now.

Sigmoid and tanh share the same weakness, gradient saturation at the extremes, which rules them out for the hidden layers of deep networks.

The ReLU family

ReLU

ReLU is the activation function used in almost all convolutional networks and most deep learning architectures today.

R(z) = max(0, z)

It is half-rectified from the bottom, so anything negative becomes zero and anything positive passes through unchanged. The range is [0, +infinity). Both the function and its derivative are monotonic. ReLU is simple to compute, and it does not saturate on the positive side, so gradients flow freely for positive activations.

ReLU function curve showing the characteristic zero-floor shape — Figure 3 The ReLU curve, with its characteristic zero-floor shape.

ReLU has a failure mode, the dying ReLU. Any neuron that receives a negative input outputs zero, and the gradient through it is also zero. Once a neuron falls into this state, it may never recover. It becomes dead, so it contributes nothing to the model and learns nothing from the data. If your learning rate is too high, a large fraction of your neurons can die early in training, and the model's capacity collapses.

Leaky ReLU

Leaky ReLU is the direct fix for the dying ReLU problem. Instead of clamping negative values to zero, it lets them through with a small slope:

f(y) = ay for y < 0 f(y) = y for y \geq 0 (where a = 0.01 typically)

The range is (-infinity, +infinity). Both the function and its derivative are monotonic. The small negative slope, usually 0.01, means neurons never fully die. They always have a non-zero gradient, so they can recover and keep learning.

In practice, Leaky ReLU is a drop-in replacement for ReLU. The performance gain is not always large, but it removes the dying-neuron failure mode, which makes training less likely to stall.

Modern activations: GeLU and Swish

GeLU (Gaussian Error Linear Unit)

GeLU is a smoothed version of ReLU. It outperforms both ELU and ReLU in my experience, and the benchmarks agree.

GeLU compared to ReLU and ELU, plus GeLU derivative curve — Figure 4 GeLU compared with ReLU and ELU, next to the GeLU derivative curve.

GeLU is non-convex and non-monotonic. It has curvature at all points and can output both positive and negative values. The non-monotonicity gives the function a richer landscape for the network to exploit.

A few things I have learned about GeLU in practice:

Use an optimizer with momentum. Because GeLU is non-convex and non-monotonic, plain SGD can struggle. Adam or SGD with momentum handles it well.
It converges faster than sigmoids. In my experiments, switching from sigmoid-based architectures to GeLU consistently reduced the number of epochs needed to reach target accuracy.
The steep slope near x = 0 matters. This region is where many activations sit during the early stages of training. GeLU's steep gradient there makes it effective at learning complex patterns right from the start.

GeLU is the default activation in GPT, BERT, and most modern transformer architectures.

Swish

Swish is another smooth, non-monotonic activation, and it comes from a neural architecture search by Google Brain:

swish(x) = x \cdot σ(βx) = x / (1 + e -βx)

Swish works better than ReLU on deeper models across a range of datasets. It is unbounded above and bounded below. Like GeLU, it is smooth and non-monotonic.

Swish has a non-zero gradient at x = 0. The network can learn in the region around zero, unlike ReLU, where the gradient is undefined at exactly zero and zero for all negative values. That small difference adds up across millions of parameters and thousands of training steps.

Why derivatives matter

During backpropagation, you do not use the activation function directly. You use its derivative. The derivative tells the optimizer two things: which direction to update the weights, and how much to update them.

Derivative curves for Sigmoid, Tanh, ReLU, Softplus, and Gaussian — Figure 5 Derivative curves for sigmoid, tanh, ReLU, Softplus, and Gaussian.

Smoother derivatives produce smoother training. Jagged or discontinuous derivatives, like ReLU's hard transition at zero, can cause instability. Smooth derivatives, like those of GeLU, Swish, or Softplus, give the optimizer a more consistent signal and more stable convergence.

This is one of the main reasons GeLU and Swish have displaced ReLU in transformer architectures. The functions are similar in shape, but the smoothness of their derivatives makes a measurable difference in training dynamics.

Quick reference table

Name	Equation	Derivative	Range
Identity	f(x) = x	f'(x) = 1	(-inf, +inf)
Binary Step	f(x) = 0 for x < 0, 1 for x >= 0	f'(x) = 0 (everywhere except x=0)	{0, 1}
Sigmoid	f(x) = 1/(1+e^(-x))	f'(x) = f(x)(1 - f(x))	(0, 1)
Tanh	f(x) = tanh(x)	f'(x) = 1 - f(x)^2	(-1, 1)
ArcTan	f(x) = arctan(x)	f'(x) = 1/(x^2 + 1)	(-pi/2, pi/2)
ReLU	f(x) = max(0, x)	f'(x) = 0 for x<0, 1 for x>=0	[0, +inf)
PReLU	f(x) = ax for x<0, x for x>=0	f'(x) = a for x<0, 1 for x>=0	(-inf, +inf)
ELU	f(x) = a(e^x - 1) for x<0, x for x>=0	f'(x) = f(x)+a for x<0, 1 for x>=0	(-a, +inf)
Softplus	f(x) = ln(1 + e^x)	f'(x) = 1/(1 + e^(-x))	(0, +inf)

Summary table with function plots for each activation function — Figure 6 Summary table with a function plot for each activation.

Takeaway

My practical guidelines for choosing an activation function:

Default for deep networks and ConvNets: Start with ReLU. It is fast, well-understood, and works.
If you see dying neurons: Switch to Leaky ReLU. Same computational cost, no dead neurons.
For transformers and modern architectures: Use GeLU. It gives smoother gradients, faster convergence, and better final performance.
For deeper models where you want extra performance: Try Swish. It consistently beats ReLU on deep architectures.
For output layers with probability targets: Sigmoid (binary) or Softmax (multi-class) are still the right tools.
Avoid sigmoid and tanh in hidden layers of deep networks. The vanishing gradient problem is real and it will slow you down.

The field is moving toward smoother, non-monotonic activations. GeLU and Swish changed how we think about the nonlinearities in our networks, because the smoothness of their derivatives leads to more stable and efficient training.

Pick the right activation function for your architecture, pay attention to your gradients, and experiment.

Based on the presentations I created in 2023.