Well, looks like a pretty straightforward fix.
What I did was to make Linecasts optionally accept NNConstraint. NavMeshBase was modified to use that constraint in neightbour node check, right where it currently checks walkability:
at around line 1247:
// This might be the next node that we enter
var neighbour = nodeConnections[i].node as TriangleMeshNode;
if (neighbour == null || !neighbour.Walkable || !constraint.Suitable (neighbour)) continue;
That seems to be it? I’m not sure if it covers all cases, but at least that’s the only relevant check I found.
After that, it’s just about sub-classing RaycastModifier into ConstrainedRaycastModifier that accepts NNConstrain and passes it into Linecast calls. Seems to work: no curved lines, no pathing through untraversable terrain. More tests to follow.
P.S. This logic should probably be present in the project itself. It’s weird that Pathfinding respects constraints, while Linecasts completely disregard them and check walkability even if it’s not actually constrained. This is made that much more unintuitive by the modifiers (and probably pathfinding itself?) using Linecasts internally.