Softmax - micheledpierri.com: statistics, data analysis and coding

Introduction

The softmax function is essential in mathematics and machine learning.

It transforms a vector of real numbers into a probability distribution.

Put simply, it converts a set of numbers into probabilities.

To understand how it works, let’s look at a list of risk values for some patients:

Patients	Surgical Risk
Patient A	2
Patient B	1
Patient C	0

When we apply the softmax function to this series of numbers [2,1,0] we get:

Patients	Surgical Risk
Patient A	66% (0.66)
Patient B	24% (0.24)
Patient C	9% (0.9)

Note that the resulting probability values from the softmax function always sum to 1 (or 100%).

How does it work?

The Softmax transformation consists of two key operations: exponentiation and normalization.

During exponentiation, we calculate e^x for each number. This amplifies larger numbers while reducing smaller ones in the series.

In the normalization step, we sum all the numbers and divide each by that total. This produces values between 0 and 1 that always sum to 1.

These two steps together create our final probability distribution.

The graphs illustrate how Softmax transforms numerical values into probabilities, with the resulting probabilities always summing to one.

From initial scores to final probability bar graphs

Mathematical Formula for Softmax

For a vector z = [z₁, z₂, z₃…zₙ], the Softmax formula is:

$\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}$

Where: – $z_i$ is the $i$ -th element of the input vector $z$ , – $e^{z_i}$ is the exponential of $z_i$ , – $\sum_{j=1}^{n} e^{z_j}$ is the sum of the exponentials of all elements in the vector $z$ , – $n$ is the total number of elements in the vector $z$ . This formula ensures that each output value lies between 0 and 1, and the sum of all outputs equals 1.

Graphical Examples of Softmax

For two classes, the Softmax function simplifies to a sigmoid function, where the first class has probability p1 and the second class has probability 1-p1 (since probabilities must sum to 1). As one class’s probability increases, the other’s must decrease proportionally.

For three classes, we can visualize the function using a three-dimensional graph. When we treat the first two classes as variables and fix the third as a constant, the graph displays the probabilities of the first two classes. The third class’s probability is then calculated as 1 minus the sum of the first two class probabilities.

Numerical Saturation and Normalization by Maximum

A key challenge when applying the Softmax function occurs with extremely large or small z values.

These extreme values can cause overflow or underflow—situations where numbers become too large or too small for a computer to represent accurately.

Consider a vector z with large numbers:

z=[1000, 1001, 1002]

Calculating $e^{1000}$ , $e^{1001}$ , and $e^{1002}$ for Softmax would produce enormous numbers that cause overflow.

To solve this, we can normalize the vector. The new vector z’ then has much smaller values:

z’ = [1000 – 1002, 1001 – 1002, 1002 – 1002] = [-2, -1, 0]

This normalization gives us manageable exponential values:

$e^{-2} \approx 0.135, \quad e^{-1} \approx 0.368, \quad e^{0} = 1$

Finally, we calculate the sum of exponentials and apply the softmax function:

$\text{Sum} = 0.135 + 0.368 + 1 = 1.503$

$\text{Softmax} = \left[\frac{0.135}{1.503}, \frac{0.368}{1.503}, \frac{1}{1.503}\right] \approx [0.09, 0.24, 0.67]$

This same approach works for very small z values.

For both extremely large and small values, we can use a modified formula:

$\text{softmax}(z_i) = \frac{e^{z_i - \max(z)}}{\sum_{j=1}^{n} e^{z_j - \max(z)}}$

Softmax with Python

Let’s explore how to implement the Softmax function in Python, covering both single vector applications and matrix operations.

# Example of applying softmax to a NumPy array

import numpy as np

def softmax(z):
    # Subtract maximum value to prevent numerical overflow
    z = z - np.max(z)
    exp_z = np.exp(z)
    return exp_z / np.sum(exp_z)

# Example usage
z = np.array([2.0, 1.0, 0.1])
print("Input:", z)
print("Softmax Output:", softmax(z))

# Example of applying softmax to a NumPy array

import numpy as np

def softmax(z):
    # Subtract maximum value to prevent numerical overflow
    z = z - np.max(z)
    exp_z = np.exp(z)
    return exp_z / np.sum(exp_z)

# Example usage
z = np.array([2.0, 1.0, 0.1])
print("Input:", z)
print("Softmax Output:", softmax(z))

# Matrix application example

def softmax_batch(z):
    # Subtract the maximum along axis 1 (per row)
    z = z - np.max(z, axis=1, keepdims=True)
    exp_z = np.exp(z)
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

# Usage example
z_batch = np.array([[2.0, 1.0, 0.1], [1.0, 2.0, 3.0]])
print("Input Batch:\n", z_batch)
print("Softmax Output Batch:\n", softmax_batch(z_batch))

# Matrix application example

def softmax_batch(z):
    # Subtract the maximum along axis 1 (per row)
    z = z - np.max(z, axis=1, keepdims=True)
    exp_z = np.exp(z)
    return exp_z / np.sum(exp_z, axis=1, keepdims=True)

# Usage example
z_batch = np.array([[2.0, 1.0, 0.1], [1.0, 2.0, 3.0]])
print("Input Batch:\n", z_batch)
print("Softmax Output Batch:\n", softmax_batch(z_batch))

Applications

The Softmax function has three main applications:

In Multiclass Classification (Machine Learning): It converts raw scores into a probability distribution across possible classes

In Neural Networks: It serves as the activation function in the output layer

In Reinforcement Learning: It transforms action scores into selection probabilities

Conclusion

The Softmax function plays a vital role in machine learning, especially for multiclass classification tasks. Its mathematical properties and computational efficiency have made it indispensable in neural networks and predictive models. Proper implementation and careful handling of numerical challenges are key to achieving optimal results.