---
title: Softmax
date: 2025-11-09T10:16:26Z
modified: 2026-06-29T06:52:26Z
permalink: "https://www.micheledpierri.com/2025/11/09/softmax-machine-learning/"
type: post
status: publish
excerpt: ""
wpid: 1044
categories:
- Data Analysis
- Machine Learning
- Programming
tags:
- Data Analysis
- Machine Learning
- Programming
- Python
featured_image: "https://www.micheledpierri.com/wp-content/uploads/2025/11/softmax_.png"
featured_image_alt: Villagers and children in period clothing gaze upward as dozens of muted red, gold, blue, and cream balloons drift above a cobbled old European street in a warm, painterly historical scene.
timestamp: 2026-06-29T06:52:26Z
---
## Introduction
The softmax function is essential in mathematics and machine learning.
It transforms a vector of real numbers into a probability distribution.
Put simply, it converts a set of numbers into probabilities.
To understand how it works, let’s look at a list of risk values for some patients:
| Patients | Surgical Risk |
| --- | --- |
| Patient A | 2 |
| Patient B | 1 |
| Patient C | |
When we apply the softmax function to this series of numbers \[2,1,0\] we get:
| Patients | Surgical Risk |
| --- | --- |
| Patient A | 66% (0.66) |
| Patient B | 24% (0.24) |
| Patient C | 9% (0.9) |
Note that the resulting probability values from the softmax function always sum to 1 (or 100%).
## How does it work?
The Softmax transformation consists of two key operations: exponentiation and normalization.
During exponentiation, we calculate e^x for each number. This amplifies larger numbers while reducing smaller ones in the series.
In the normalization step, we sum all the numbers and divide each by that total. This produces values between 0 and 1 that always sum to 1.
These two steps together create our final probability distribution.
The graphs illustrate how Softmax transforms numerical values into probabilities, with the resulting probabilities always summing to one.
## Mathematical Formula for Softmax
For a vector z = \[z₁, z₂, z₃…zₙ\], the Softmax formula is:
Where: –  is the 
-th element of the input vector 
, –  is the exponential of 
, –  is the sum of the exponentials of all elements in the vector 
, –  is the total number of elements in the vector 
. This formula ensures that each output value lies between 0 and 1, and the sum of all outputs equals 1.
## Graphical Examples of Softmax
For two classes, the Softmax function simplifies to a sigmoid function, where the first class has probability p1 and the second class has probability 1-p1 (since probabilities must sum to 1). As one class’s probability increases, the other’s must decrease proportionally.
For three classes, we can visualize the function using a three-dimensional graph. When we treat the first two classes as variables and fix the third as a constant, the graph displays the probabilities of the first two classes. The third class’s probability is then calculated as 1 minus the sum of the first two class probabilities.
### Numerical Saturation and Normalization by Maximum
A key challenge when applying the Softmax function occurs with extremely large or small z values.
These extreme values can cause overflow or underflow—situations where numbers become too large or too small for a computer to represent accurately.
Consider a vector z with large numbers:
z=\[1000, 1001, 1002\]
Calculating 
, 
, and  for Softmax would produce enormous numbers that cause overflow.
To solve this, we can normalize the vector. The new vector z’ then has much smaller values:
z’ = \[1000 – 1002, 1001 – 1002, 1002 – 1002\] = \[-2, -1, 0\]
This normalization gives us manageable exponential values:
Finally, we calculate the sum of exponentials and apply the softmax function:
![\text{Softmax} = \left[\frac{0.135}{1.503}, \frac{0.368}{1.503}, \frac{1}{1.503}\right] \approx [0.09, 0.24, 0.67]](https://www.micheledpierri.com/wp-content/ql-cache/quicklatex.com-94c1754c5750e6a6cd4727722d1861a9_l3.svg "Rendered by QuickLaTeX.com")
This same approach works for very small z values.
For both extremely large and small values, we can use a modified formula:
## Softmax with Python
Let’s explore how to implement the Softmax function in Python, covering both single vector applications and matrix operations.
# Example of applying softmax to a NumPy array
import numpy as np
def softmax(z):
# Subtract maximum value to prevent numerical overflow
z = z - np.max(z)
exp_z = np.exp(z)
return exp_z / np.sum(exp_z)
# Example usage
z = np.array([2.0, 1.0, 0.1])
print("Input:", z)
print("Softmax Output:", softmax(z))```
# Example of applying softmax to a NumPy array
import numpy as np
def softmax(z):
# Subtract maximum value to prevent numerical overflow
z = z - np.max(z)
exp_z = np.exp(z)
return exp_z / np.sum(exp_z)
# Example usage
z = np.array([2.0, 1.0, 0.1])
print("Input:", z)
print("Softmax Output:", softmax(z))
```
# Matrix application example
def softmax_batch(z):
# Subtract the maximum along axis 1 (per row)
z = z - np.max(z, axis=1, keepdims=True)
exp_z = np.exp(z)
return exp_z / np.sum(exp_z, axis=1, keepdims=True)
# Usage example
z_batch = np.array([[2.0, 1.0, 0.1], [1.0, 2.0, 3.0]])
print("Input Batch:\n", z_batch)
print("Softmax Output Batch:\n", softmax_batch(z_batch))```
# Matrix application example
def softmax_batch(z):
# Subtract the maximum along axis 1 (per row)
z = z - np.max(z, axis=1, keepdims=True)
exp_z = np.exp(z)
return exp_z / np.sum(exp_z, axis=1, keepdims=True)
# Usage example
z_batch = np.array([[2.0, 1.0, 0.1], [1.0, 2.0, 3.0]])
print("Input Batch:\n", z_batch)
print("Softmax Output Batch:\n", softmax_batch(z_batch))
```
## Applications
The Softmax function has three main applications:
In Multiclass Classification (Machine Learning): It converts raw scores into a probability distribution across possible classes
In Neural Networks: It serves as the activation function in the output layer
In Reinforcement Learning: It transforms action scores into selection probabilities
## Conclusion
The Softmax function plays a vital role in machine learning, especially for multiclass classification tasks. Its mathematical properties and computational efficiency have made it indispensable in neural networks and predictive models. Proper implementation and careful handling of numerical challenges are key to achieving optimal results.