I am getting ahead of things a bit, but I seem to remember computing normals in Calculus in 3D and discovering that the procedure gives two normals antiparrallel to each other. This confused me, since the graphics package I had used would usually keep good track of the direction of normals, and if it ever did lose track it could recover them by explicitly recalculating and/or flipping them. What additional information is generally used to pick the direction of the normal?
It looks like our homework touches on this, but it has us manually choose the direction of the normals. One option would be to only try to ensure that the normals are consistent (i.e. all facing the right direction or the wrong direction) and let the user flip them if you get it wrong, but even keeping the normals consistent seems challenging. I was thinking you could choose the normal that is closest to parallel with the adjacent normals, but it it not hard to construct simple geometry that would break this.
@THINK These are terrific questions. Yes, keeping normals consistent can be a pain, and can cause nasty rendering artifacts (and simulation artifacts, and geometry processing artifacts, ...) if not done right. One way to do it is to make sure that all the faces in a polygon mesh have vertices in a consistent "winding order," i.e., clockwise or counterclockwise. But this is just kicking the can down the road: how do you now ensure this ordering is consistent across the whole mesh?
Fortunately, if you think about it for a bit, you realize it's not too hard to fix either problem. Basically you can start with one face and do a breadth-first (or depth-first or whatever-first) traversal of the mesh, at each step picking an orientation of the new normal that is consistent with the predecessor in the traversal. The question is: what calculation can you do to check whether or not it's consistent?
Rather than give the answer... think about it! ;-)
(Maybe someone else in class can post an answer here too.)