Attached is a fixed version of OverlayNode.cpp. I fixed CustomPolytope::cut( osg::Plane ) method. Bug was apparent in such scenario:
Let P1 be some random frustum polytope
Let P2 be the polytope that was created from P1 bounding box (P2 contains P1 entirely)
Then ignoring precision errors: P1.cut( P2 ) == P2.cut( P1 ) == P1. But this condition was not always met. Cut failed when some of the polytope reference points happened to lie exactly on some intersecting planes in both P1 & P2 (plane distance was = 0).
I only use CustomPolytope for my shadowing stuff so I did not test how this affects rest of OverlayNode.cpp.
----2----
Also attached is a minor precision improvement for osg::Plane intersect method (double version).
----3----
I have also one observation regarding osg::Plane - There are two intersect vertices methods (float and double flavour):
inline int intersect(const std::vector<Vec3>& vertices) const
inline int intersect(const std::vector<Vec3d>& vertices) const
I guess osg::Plane won't compile when someone changes default vec3 typedef to vec3d. Shouldn't the first method be changed to use vec3f explicitly ? Ie:
inline int intersect(const std::vector<Vec3f>& vertices) const"
55de8b03d9