I don't see why we bring up bias and variance and not mention the bias-variance tradeoff. That is sometimes we can tradeoff a little more bias for less variance. This applies especially for small sample sizes with large variance.

see https://en.wikipedia.org/wiki/Bias_of_an_estimator