LearnViz CompSci
Home / Learn / AI & Machine Learning / Activation Functions

Intermediate Topic 31 of 51 AI & Machine Learning

Activation Functions

Nonlinearities such as ReLU and sigmoid.

How it works

Activation functions introduce nonlinearity into neural networks, without which stacked layers would collapse into a single linear map. Sigmoid squashes values to 0 to 1 and tanh to minus 1 to 1 but saturate and cause vanishing gradients; ReLU outputs max of zero and x, is cheap and the default for hidden layers, with variants such as leaky ReLU, GELU and SiLU; softmax turns scores into class probabilities in the output layer; linear outputs suit regression.

Step-by-step walkthrough

  1. Without nonlinearity many layers collapse to one linear map.
  2. ReLU outputs zero for negatives and x for positives, cheap and common.
  3. Sigmoid squashes to between 0 and 1.
  4. Tanh centres outputs around 0.
  5. Softmax turns scores into probabilities that sum to 1.

Where it is used

Choosing activations affects training speed and stability in convolutional networks, transformers, classifiers and generative models.

Common pitfall

Sigmoid and tanh in deep networks cause vanishing gradients, and ReLU can produce dead neurons that always output zero. Softmax needs numerically stable implementations. Output activation must match the loss, such as sigmoid with binary cross-entropy. Initialization schemes such as He and Xavier are designed for specific activations. Start with ReLU or GELU in hidden layers.

Where it fits in the course

Activation Functions is topic 31 of 51 in AI & Machine Learning, rated Intermediate. It follows Perceptron. The next topic is Backpropagation.

Practice and explore

How to study this topic

  1. Read the explanation above and restate the definition in your own words.
  2. Open the interactive lesson and answer its mini quiz.
  3. Read the common pitfall, then write a short example that avoids it.
  4. Check your understanding in the AI & Machine Learning quiz, and revisit the topic if you miss a question.

Frequently asked questions

What is Activation Functions?
Nonlinearities such as ReLU and sigmoid. It belongs to AI & Machine Learning and is an intermediate-level topic.
Where do people use Activation Functions?
Choosing activations affects training speed and stability in convolutional networks, transformers, classifiers and generative models.
What is the most common mistake with Activation Functions?
Sigmoid and tanh in deep networks cause vanishing gradients, and ReLU can produce dead neurons that always output zero.
What should I learn before this topic?
Start with Perceptron, the topic just before it in AI & Machine Learning.
How can I practice it?
Answer the mini quiz in the interactive lesson, then take the AI & Machine Learning quiz.
Is Activation Functions hard to learn?
It is an Intermediate topic. Be comfortable with the Beginner topics in this area first, then read the common pitfall carefully, since it is where most mistakes happen.
How long does it take to learn Activation Functions?
Most learners can read the explanation and finish the interactive lesson in under an hour. Fluency comes from applying the idea, so plan a few short practice sessions with the quiz.
Which topics are related to Activation Functions?
Perceptron, Backpropagation, Neural Networks are closely connected. The full list is in the related topics section of this page.
What should I study after Activation Functions?
Continue with Backpropagation, the next topic in AI & Machine Learning.

Related topics