Although if there is a drastic change in color or stark lighting, wouldn't we want to keep that variance?

What is the significance of the variance here? Can we think of it as an error on the estimate?