Not necc. the same as that point - only so on certain surfaces. By convention true for the real numbers, since it is true for their extension, the ComplexPlane - but it can often be useful for them to be separate.
In the (mathematically uninteresting but actually very useful) IEEE floating point domain, positive infinity, negative infinity and NaN (which is short for Not A Number, and roughly corresponds to the mathematical concept of "not defined") are separate points, each in its own way representing 'everything else'.
Sadly not really. Lots of things which ought to return NaN actually end up returning positive infinity. But yeah, conceptually it's supposed to be used that way - and the logic of combining it with other numbers supports that.