October 13, 2024

Python Tutorials

Gradient Descent Algorithm

Gradient Descent is an optimization algorithm used to find the minimum of a function. It is widely used in machine learning and data science to minimize the loss function and optimize the parameters of a model. The algorithm iteratively adjusts parameters in the direction that reduces the function’s value.

1. Concept

The idea behind Gradient Descent is to move towards the minimum of a function by taking steps proportional to the negative of the gradient (or the derivative) of the function with respect to the parameters. The gradient provides the direction of the steepest ascent, so moving in the opposite direction will lead to the minimum.

2. Algorithm Steps

  1. Initialize Parameters: Start with initial values for the parameters.
  2. Compute Gradient: Calculate the gradient of the loss function with respect to the parameters.
  3. Update Parameters: Adjust the parameters in the direction of the negative gradient by a certain step size (learning rate).
  4. Repeat: Continue updating the parameters until convergence or a stopping criterion is met.

3. Mathematical Formulation

For a loss function L(θ), where θ represents the parameters, the update rule for the parameters is given by:

θ = θ - α * ∇L(θ)

where:

  • θ is the vector of parameters.
  • α is the learning rate.
  • ∇L(θ) is the gradient of the loss function with respect to θ.

4. Types of Gradient Descent

  • Batch Gradient Descent: Uses the entire dataset to compute the gradient for each update. It can be computationally expensive for large datasets.
  • Stochastic Gradient Descent (SGD): Uses one training example at a time to compute the gradient. It is more efficient but can be noisy.
  • Mini-Batch Gradient Descent: Uses a small random subset of the dataset (mini-batch) to compute the gradient. It balances the efficiency and stability of the updates.

5. Example in Python

Here is a basic example of implementing Gradient Descent in Python to minimize a simple quadratic function:

import numpy as np

# Define the loss function
def loss_function(x):
    return x**2 + 3*x + 2

# Define the gradient of the loss function
def gradient(x):
    return 2*x + 3

# Gradient Descent function
def gradient_descent(starting_point, learning_rate, num_iterations):
    x = starting_point
    for _ in range(num_iterations):
        grad = gradient(x)
        x = x - learning_rate * grad
    return x

# Parameters
starting_point = 0
learning_rate = 0.1
num_iterations = 100

# Perform Gradient Descent
optimal_x = gradient_descent(starting_point, learning_rate, num_iterations)
print(f"Optimal x: {optimal_x}")

This example demonstrates Gradient Descent to find the minimum of a quadratic function. The function gradient_descent iteratively updates the parameter x to minimize the loss function.

6. Conclusion

Gradient Descent is a fundamental optimization technique used to train machine learning models and optimize various types of functions. By adjusting parameters in the direction of the negative gradient, it iteratively reduces the loss function’s value. Understanding and implementing Gradient Descent is crucial for developing efficient and effective machine learning algorithms.

You May Like