Not A Number
NaN is an insidious bug that often plague games and shader programs during development. If you are lucky, you would probably never encounter it before, but the first time would often be very painful.
It begin with a single unit disappearing from the map, then suddenly whole armies are teleported somewhere, and at last the player is teleported to limbo.
On a shader program, it is less obvious, as the color of NaN is BLACK!!! So for most of the time it would seem just like a few black speck on the screen. But turning on effect like HDR when there is a NaN would cause the whole screen to blacken out.
The worst thing about NaN is, it is more contagious than the plague. Any maths operator with a NaN would give you a NaN
Luckily there are not many ways to cause NaN. The most common ways and solution are
Division by zero
Have a check to ensure your divisor is not zero. In certain situations with a positive divisor, we can add a very small number to the divisor to ensure it is not zero.
Normalize a zero vector
Remember to normalize a vector is to divide every component of a vector with it magnitude. So to normalize a zero vector would cause a divison by zero. Do a check on the vector length or lengthSquare to ensure it is not zero before normalizing it.
Zero vector Normals are also very common in exported model, it could be the normal is not exported or some bug with teh exported, to export a zero normal. I often find a shader to display normals as color to be useful in such situation. If you have control over the content pipeline, Assert when there is a zero normal.
Calling ACos on a number outside -1 to 1
Dot product of two unit vectors would give the cosAngle. if any of the two vectors is not unit length, the result would probably be outside -1 to 1. Ensure both vectors are unit length, and try not to call ACos as cosAngle is very sufficent to solve most problems.