I can see that the definition of local maximum and unconstrained local maximum is written differently, but to me they look like they are defining the same thing. Furthermore, based on Fig 4.1, it looks like both $x^*$ and $y^*$ meet the definition of local maximum and unconstrained local maximum?
How do I distinguish between the 2 definitions?
I'll assume $\mathcal{D}$ refers to the domain of the function.
I'm not familiar with the term "unconstrained local maximum," but the definitions given here are different. Look at the left-most point on the curve; say that it's located at $x = x_0$. Then, taking a small ball $B(x_0, \varepsilon)$, $f(x_0) > f(y)$ for all $y \in B(x_0, \varepsilon) \cap \mathcal{D}$ so $x_0$ is a local maximum by this definition.
On the other hand, no ball $B(x_0, r)$ is contained in $\mathcal{D}$ -- $\mathcal{D}$ contains no points to the left of $x_0$ -- so $x_0$ cannot be an unconstrained local maximum by this definition.
So, in other words, according to these definitions an "unconstrained local maximum" is a "local maximum" which occurs in the middle of the domain, as opposed to at an endpoint.
Just to add on, I believe the textbook from where this definition is taken, which is Sundaram's 'A First Course in Optimization Theory' states that the term "unconstrained optimum" is slightly misleading and that a more descriptive term would be "interior optimum", but that this is less commonly used in optimization theory. Of course, here we are defining an unconstrained local maximum but a minimum would be defined analogously.
The difference in the definitions lies in where the point in consideration, $x$ is found in the domain $D$. The second definition specifically refers to an interior point $x$ of the domain of the function as we require that $\exists \ r >0$ such that $B\ (x,r)\subset D$, meaning that there must be some open ball around the point we are considering that is fully contained in the domain $D$. The first definition applies to all points in $D$ - we place no such restriction on $x$ as in the case of the second defintion. The requirement "$ \forall \ y \in D \ \cap B\ (x,r)$" means we are comparing only to points in a small region about $x$ that are also in the domain in consideration, meaning that $x$ could also be an end point of the domain. We can also clearly see that an unconstrained local maximum would also be a local maximum but the converse is not necessarily true.
