How might we deal with functions that we can't easily compute an analytic derivative for? Do we enter sampling land again?
What if the second derivates don't exist or we can't derive the function?
Building on the last two questions, what if the function is not differentiable at all? Can we make any guarantees about the existence of a minimum in this case?