Welcome to our lesson on basic derivatives! In machine learning, understanding how things change is crucial. Derivatives help measure these changes, which is key for optimizing models. By the end, you'll understand derivatives, know how to calculate them, and see a practical example using Python.

Let's understand the concept of a derivative. Imagine you're driving a car. Your speed indicates how your position changes over time. Similarly, a derivative measures how a function's output changes as its input changes. It's like the speed for functions. The derivative is calculated for every input of the original function. So, the derivative is also a function of the same input.

Let's try to approximate the derivative of the function $f(x) = x^2$ at $x=2$. In other words, we want to see the speed of the function's growth (or decay) at this point. Using a small change $\Delta x$, we will find how much function changed when the x changed:

Here’s a graph to illustrate this:

In the picture, the green dashed line shows our slope approximation using $\Delta x = 1$. We see, that the function changed from 4 to 9, while x changed from 2 to 3. We can make a conclusion that the speed of the function at $x=2$ is roughly $\frac{9-4}{1}$, which equal to 5.

We care about the derivative at point $x=2$, but what we really calculate is the speed of the function on the $[2, 3]$ interval. Let's improve the accuracy of our approximation by choosing a smaller $\Delta x$. Here is a new graph:

Here, the $\Delta x=0.5$. The derivative approximation is $\frac{f(2.5) - f(2)}{0.5}$, which is equal to 4.5. This time, we calculated the function's growth speed on the $[2, 2.5]$ interval, which gives us a better approximation quality. By shrinking this interval (in other words, by making $\Delta x$ lower), we can improve the approximation quality.

For example, making the same calculations on the $[2, 2.00001]$ interval will give you a value, which is very close to the actual derivative at point $x=2$. However, it won't be exact.

Let's demonstrate this with code:

Python
Copy to clipboard
1# Define the function f(x) = x^2
2def f(x):
3    return x**2
4
5delta_x_values = [1, 0.5, 0.1, 0.01, 0.001]
6x = 2
7
8for delta_x in delta_x_values:
9    approx_derivative = (f(x + delta_x) - f(x)) / delta_x
10    print(f'Approx. derivative with delta_x = {delta_x}: {approx_derivative}')

In this code, we make delta_x smaller and smaller and see how the approximation for a derivative changes.. For each delta_x, the approximate derivative is calculated using the difference quotient formula. The result is printed with an f-string, including the current delta_x, and a calculated approximate derivative for it. Here is the output:

Copy to clipboard
1Approx. derivative with delta_x = 1: 5.0
2Approx. derivative with delta_x = 0.5: 4.5
3Approx. derivative with delta_x = 0.1: 4.100000000000001
4Approx. derivative with delta_x = 0.01: 4.009999999999891
5Approx. derivative with delta_x = 0.001: 4.000999999999699

As you can see, the value gets closer and closer to 4.

To get an exact derivative, make $\Delta x$ as small as possible. When $\Delta x$ is closer to 0, the approximation becomes more accurate. So, to find the exact value of the derivative, find a limit of our expression when $\Delta x$ approaches 0.

Evaluating this will give us an exact derivative function, which can be used to calculate the derivative at any given input.

However, this course aims to give you an understanding of the idea behind derivative, not to teach how to calculate it. So, to calculate a derivative, we will use numerical methods, which give us a good approximation and doesn't require heavy math.

Let's see how to calculate it using Python. We'll use the forward difference method to estimate the derivative numerically.

Here’s the function:

Python
Copy to clipboard
1# Numerical derivative using forward difference
2def derivative(f, x, h=1e-5):
3    return (f(x + h) - f(x)) / h
4
5# Sample function: f(x) = x^2
6f = lambda x: x**2
7
8# Compute derivative of f at x=3
9print("Derivative of f(x) at x=3:", derivative(f, 3))  # Derivative of f(x) at x=3: 6.000009999951316

In this code:

• derivative takes a function f, a point x, and a small step h.
• A small h (1e-5) makes the approximation accurate.
• We define f(x) = x^2 using a lambda function.
• We compute and print the derivative of f at x = 3.

To summarize, today we:

• Learned what derivatives are and how they measure change.
• Used pictures and Python code to visualize and approximate derivatives.
• Calculated derivatives using numerical methods.

Next, you'll move to practice, where you'll calculate derivatives for different functions using Python. Happy coding!