Understanding the derivative as a function is essential in calculus and machine learning. It helps us see how inputs affect outputs, which is crucial for optimization and learning algorithms. In this lesson, you'll learn what it means for a function to be differentiable, see examples of functions that aren't differentiable at certain points, and plot both a function and its derivative to explore their relationships. By the end, you'll identify where a function grows, decays, and has extremum points through its derivative.

A function is differentiable at a point if it has a defined derivative there. This means the function should be smooth without sharp turns or corners at that point. For instance, the function $f(x) = |x|$ is not differentiable at $x = 0$ because it has a sharp corner there, making it impossible to define a single tangent line.

To visualize this:

The derivative itself can be seen as a function that varies with $x$. For example, for the function $f(x) = x^2$, we can define the derivative as $f'(x) = 2x$, which is also a function of $x$.

There are rules to obtain this formula. However, we don't cover them in this lesson, as for machine learning its much more important to understand the idea of the derivative rather than learning to calculate it.

Let's visualize this relationship:

From the plot:

1. Derivative is Negative When the Function Decays: For $x < 0$, $f(x) = x^2$ is decreasing, and $f'(x) = 2x$ is negative, indicating a downward slope.

2. Derivative is Zero at a Extremum Point: At $x = 0$, $f(x) = x^2$ has a minimum point, and $f'(x) = 2x$ is zero, showing a extremum point. An extremum point is either a local minimum or a local maximum of the function. Note that there are other cases where $f'(x)$ could be zero, for example it could happen at the inflection point, but we won't consider it in this course.

3. Derivative is Positive When the Function Grows: For $x > 0$, $f(x) = x^2$ is increasing, and $f'(x) = 2x$ is positive, indicating an upward slope.

Note that we are talking about a local minimum/maximum here. For example, take a look at this graph:

Here, $x=0$ is the point of local minimum. The function grows both to the right and to the left of it. However, it is not a global minimum: we can clearly observe the function getting lower to the left.

Consider a more complex function, $f(x) = x^3 - 3x^2 + 2x$. Its derivative is $f'(x) = 3x^2 - 6x + 2$. We'll plot both the function and its derivative to see their relationship.

In this plot, you see:

• Growth and decay at different $x$ intervals.
• Multiple extremum points where the derivative $f'(x)$ equals zero.
• Concave and convex regions, indicating changes in the original function's curvature.

Understanding these relationships helps you grasp the behavior of mathematical models used in machine learning.

Today, we explored what it means for a function to be differentiable and identified functions that aren't differentiable at certain points. We learned that derivatives can be treated as functions and visualized that for both simple and complex functions. You should now recognize when a function grows, decays, and has extremum points based on its derivative.

Most important conclusions:

• If the derivative at some $x$ is positive, the function grows at this point
• If the derivative at some $x$ is negative, the function decays at this point
• If the derivative at some $x$ is zero, the function has a local minimum or maximum at this point

Now that you've understood the theory, it's time to put your knowledge into practice! You'll write and modify similar code to visualize various functions and their derivatives. This will solidify your understanding and prepare you for more advanced topics in calculus and machine learning.