From f706149676b96a51dc58256555e683f087205fa5 Mon Sep 17 00:00:00 2001 From: PragatiVerma18 Date: Mon, 9 Dec 2024 13:45:41 +0530 Subject: [PATCH 1/7] Add concept entry for ai gradient descent --- .../gradient-descent/gradient-descent.md | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md new file mode 100644 index 00000000000..44438630348 --- /dev/null +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -0,0 +1,112 @@ +--- +Title: 'Gradient Descent' +Description: 'Gradient Descent is an optimization algorithm used in machine learning and neural networks to minimize a cost function by iteratively moving towards the minimum using the gradient of the function.' +Subjects: + - 'Machine Learning' + - 'Data Science' + - 'Computer Science' +Tags: + - 'AI' + - 'Machine Learning' + - 'Neural Networks' + - 'Functions' +CatalogContent: + - 'machine-learning' + - 'path/data-science' +--- + +**Gradient Descent** is an optimization algorithm commonly used to minimize a cost function in machine learning and neural networks. The goal of gradient descent is to find the optimal parameters (weights) for a model that minimizes the error or loss function. + +In the context of neural networks, gradient descent adjusts the model’s parameters by computing the gradient (or derivative) of the cost function with respect to each parameter. The algorithm then updates the parameters in the direction that reduces the cost. + +## Types of Gradient Descent + +There are three main types of gradient descent: + +| Type | Description | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Batch Gradient Descent** | Uses the entire dataset to compute the gradient and update the weights. Typically slower but more accurate for large datasets. | +| **Stochastic Gradient Descent (SGD)** | Uses a single sample to compute the gradient and update the weights. Faster, but the updates are noisy and can lead to fluctuations in the convergence path. | +| **Mini-batch Gradient Descent** | A compromise between batch and stochastic gradient descent, using a small batch of samples to compute the gradient. It balances the speed and accuracy of the learning process. | + +## Gradient Descent Update Rule + +The basic update rule for gradient descent is: + +```pseudo +theta = theta - learning_rate * gradient_of_cost_function +``` + +- `theta`: The parameter (weight) of the model. +- `learning_rate`: A hyperparameter that controls the step size. +- `gradient_of_cost_function`: The gradient (derivative) of the cost function with respect to the parameters. + +## Syntax + +Here's a basic syntax for Gradient Descent in the context of machine learning, specifically for updating the model parameters (weights) in order to minimize the cost function: + +```pseudo +# Initialize parameters (weights) and learning rate +theta = initial_value # Parameters (weights) +learning_rate = value # Learning rate +iterations = number_of_iterations # Number of iterations + +# Repeat until convergence +for i in range(iterations): + # Calculate the gradient of the cost function + gradient = compute_gradient(X, y, theta) + + # Update the parameters (weights) + theta = theta - learning_rate * gradient # Update rule + + # Optionally, compute and store the cost (for monitoring convergence) + cost = compute_cost(X, y, theta) + store(cost) +``` + +## Example + +In the following example, we implement simple gradient descent to minimize the cost function of a linear regression problem: + +```py +import numpy as np + +# Sample data (X: inputs, y: actual outputs) +X = np.array([1, 2, 3, 4, 5]) +y = np.array([1, 2, 1.3, 3.75, 2.25]) + +# Parameters initialization +theta = 0.0 # Initial weight +learning_rate = 0.01 # Step size +iterations = 1000 # Number of iterations + +# Cost function (Mean Squared Error) +def compute_cost(X, y, theta): + m = len(y) + cost = (1/(2*m)) * np.sum((X*theta - y)**2) + return cost + +# Gradient Descent function +def gradient_descent(X, y, theta, learning_rate, iterations): + m = len(y) + cost_history = [] + + for i in range(iterations): + gradient = (1/m) * np.sum(X * (X*theta - y)) # Derivative of cost function + theta = theta - learning_rate * gradient # Update theta + cost_history.append(compute_cost(X, y, theta)) # Track cost + return theta, cost_history + +# Run Gradient Descent +theta_optimal, cost_history = gradient_descent(X, y, theta, learning_rate, iterations) + +print(f"Optimal Theta: {theta_optimal}") +``` + +The output for the above code will be something like this: + +```shell +Optimal Theta: 0.6390909090909086 +``` + +> **Note**: The optimal `theta` value will be an approximation, as the gradient descent approach iteratively updates the weight to reduce the cost function. From 25473921687e294c5913a9fe50f0954d32d8517f Mon Sep 17 00:00:00 2001 From: PragatiVerma18 Date: Mon, 16 Dec 2024 20:01:42 +0530 Subject: [PATCH 2/7] Review Fixes --- .../gradient-descent/gradient-descent.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md index 44438630348..e9011508277 100644 --- a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -1,6 +1,6 @@ --- Title: 'Gradient Descent' -Description: 'Gradient Descent is an optimization algorithm used in machine learning and neural networks to minimize a cost function by iteratively moving towards the minimum using the gradient of the function.' +Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively moving towards its minimum using the function's gradient.' Subjects: - 'Machine Learning' - 'Data Science' @@ -15,9 +15,9 @@ CatalogContent: - 'path/data-science' --- -**Gradient Descent** is an optimization algorithm commonly used to minimize a cost function in machine learning and neural networks. The goal of gradient descent is to find the optimal parameters (weights) for a model that minimizes the error or loss function. +**Gradient Descent** is an optimization algorithm commonly used in machine learning and neural networks to minimize a cost function. Its goal is to iteratively find the optimal parameters (weights) that minimize the error or loss. -In the context of neural networks, gradient descent adjusts the model’s parameters by computing the gradient (or derivative) of the cost function with respect to each parameter. The algorithm then updates the parameters in the direction that reduces the cost. +In neural networks, gradient descent computes the gradient (derivative) of the cost function with respect to each parameter. It then updates the parameters in the direction of the negative gradient, effectively reducing the cost with each step. ## Types of Gradient Descent @@ -26,7 +26,7 @@ There are three main types of gradient descent: | Type | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Batch Gradient Descent** | Uses the entire dataset to compute the gradient and update the weights. Typically slower but more accurate for large datasets. | -| **Stochastic Gradient Descent (SGD)** | Uses a single sample to compute the gradient and update the weights. Faster, but the updates are noisy and can lead to fluctuations in the convergence path. | +| **Stochastic Gradient Descent (SGD)** | Uses a single sample to compute the gradient and update the weights. It is faster, but the updates are noisy and can cause fluctuations in the convergence path. | | **Mini-batch Gradient Descent** | A compromise between batch and stochastic gradient descent, using a small batch of samples to compute the gradient. It balances the speed and accuracy of the learning process. | ## Gradient Descent Update Rule @@ -37,9 +37,9 @@ The basic update rule for gradient descent is: theta = theta - learning_rate * gradient_of_cost_function ``` -- `theta`: The parameter (weight) of the model. +- `theta`: The parameter (weight) of the model that is being optimized. - `learning_rate`: A hyperparameter that controls the step size. -- `gradient_of_cost_function`: The gradient (derivative) of the cost function with respect to the parameters. +- `gradient_of_cost_function`: The gradient (derivative) of the cost function with respect to the parameters, indicating the direction and magnitude of the change needed. ## Syntax @@ -47,14 +47,14 @@ Here's a basic syntax for Gradient Descent in the context of machine learning, s ```pseudo # Initialize parameters (weights) and learning rate -theta = initial_value # Parameters (weights) -learning_rate = value # Learning rate +theta = initial_value # Model Parameters (weights) +learning_rate = value # Learning rate (step size) iterations = number_of_iterations # Number of iterations # Repeat until convergence for i in range(iterations): # Calculate the gradient of the cost function - gradient = compute_gradient(X, y, theta) + gradient = compute_gradient(X, y, theta) # Gradient calculation # Update the parameters (weights) theta = theta - learning_rate * gradient # Update rule @@ -83,7 +83,7 @@ iterations = 1000 # Number of iterations # Cost function (Mean Squared Error) def compute_cost(X, y, theta): m = len(y) - cost = (1/(2*m)) * np.sum((X*theta - y)**2) + cost = (1/(2*m)) * np.sum((X*theta - y)**2) # The cost function for linear regression return cost # Gradient Descent function From 712a6950343f12a0b291b6b4b0ed759739080309 Mon Sep 17 00:00:00 2001 From: PragatiVerma18 Date: Mon, 16 Dec 2024 20:14:52 +0530 Subject: [PATCH 3/7] Fix lint errors --- .../neural-networks/terms/gradient-descent/gradient-descent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md index e9011508277..33b18d478b7 100644 --- a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -2,9 +2,9 @@ Title: 'Gradient Descent' Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively moving towards its minimum using the function's gradient.' Subjects: + - 'Computer Science' - 'Machine Learning' - 'Data Science' - - 'Computer Science' Tags: - 'AI' - 'Machine Learning' From 9a74c1baba7f4ac40c685ad6428e6455ca883ca2 Mon Sep 17 00:00:00 2001 From: PragatiVerma18 Date: Mon, 16 Dec 2024 20:20:15 +0530 Subject: [PATCH 4/7] Fix lint errors --- .../neural-networks/terms/gradient-descent/gradient-descent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md index 33b18d478b7..e9011508277 100644 --- a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -2,9 +2,9 @@ Title: 'Gradient Descent' Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively moving towards its minimum using the function's gradient.' Subjects: - - 'Computer Science' - 'Machine Learning' - 'Data Science' + - 'Computer Science' Tags: - 'AI' - 'Machine Learning' From 11122a67a1c7d6b128312387de4f8c27d099da8c Mon Sep 17 00:00:00 2001 From: PragatiVerma18 Date: Tue, 17 Dec 2024 11:45:01 +0530 Subject: [PATCH 5/7] Fix lint errors --- .../terms/gradient-descent/gradient-descent.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md index e9011508277..6c6871815b6 100644 --- a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -11,8 +11,8 @@ Tags: - 'Neural Networks' - 'Functions' CatalogContent: - - 'machine-learning' - - 'path/data-science' + - 'paths/data-science' + - 'paths/machine-learning' --- **Gradient Descent** is an optimization algorithm commonly used in machine learning and neural networks to minimize a cost function. Its goal is to iteratively find the optimal parameters (weights) that minimize the error or loss. From d4dfea643ed521f05b0dfc411f3b40445341aabc Mon Sep 17 00:00:00 2001 From: PragatiVerma18 Date: Tue, 17 Dec 2024 11:47:10 +0530 Subject: [PATCH 6/7] Fix lint errors --- .../neural-networks/terms/gradient-descent/gradient-descent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md index 6c6871815b6..c9fa3b9ff8c 100644 --- a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -1,6 +1,6 @@ --- Title: 'Gradient Descent' -Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively moving towards its minimum using the function's gradient.' +Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively moving towards its minimum using the gradient of the function.' Subjects: - 'Machine Learning' - 'Data Science' From 8eaf907aae98a28ded1538634edf5b38aa9aa4ba Mon Sep 17 00:00:00 2001 From: Mamta Wardhani Date: Wed, 18 Dec 2024 18:28:49 +0530 Subject: [PATCH 7/7] Update gradient-descent.md --- .../neural-networks/terms/gradient-descent/gradient-descent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md index c9fa3b9ff8c..ef4f66b5957 100644 --- a/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md +++ b/content/ai/concepts/neural-networks/terms/gradient-descent/gradient-descent.md @@ -1,6 +1,6 @@ --- Title: 'Gradient Descent' -Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively moving towards its minimum using the gradient of the function.' +Description: 'Gradient Descent is an optimization algorithm that minimizes a cost function by iteratively adjusting parameters in the direction of its gradient.' Subjects: - 'Machine Learning' - 'Data Science'