How to Reproduce Stack Overflow Error

If you attach NavMesh cuts to physics objects the Stackoverflow error is a constant nightmare. It crashes the Unity editor as well as any builds. This bug haunts my project like a phantom, getting hundreds of bug reports from users. I don’t know if this is possible, but I would really like to get this fixed. Here is an exact repo step:

  1. Make a Nav Mesh graph.
  2. Make a cube. Make it a physics object with a rigidbody.
  3. Attach a navmesh cutter to the cube, make sure rotate is turned on.
  4. Have the cube flip onto its side. If the two cast points overlap then…
  5. Crash.

This happens on all version of A* Pathfinding Project. Any fixes to this bug would be greatly appreciated.

Hi

Navmesh cuts are 2D contours. You should ideally never rotate them so that they are tilted 90 degrees like that because this will feed degenerate contours to the underlying algorithms. Unfortunately the computational geometry parts are handled by 3rd party libraries, and it’s a bit tricky for me to make those libraries handle these cases well. I would recommend that you disable the ‘Use rotation and scale’ checkbox. Often for cubes I usually use a hexagonal cut which is not rotated with the object. At some point I really should change or rewrite those libraries though because they have been a constant source of bugs.

Hi Aron thanks for the response. The following pictures show why the solution you proposed is simply not tenable. The object must be a physics object so it has to rotate freely. If the box rectangle constraint followed the shape that would be fine, but it doesn’t as you can see from #1 and #2. If I could lock the “Scale and Rotation” to only the Yaw axis that would be acceptable, but as you know this is not possible.

So the only solution left is to make something very hacky and very inefficient such as in #3. If I have hundreds of these objects you can see why #3 is not a viable solution.

Any ideas?

Hi

That’s a fair point.
What you would want in this case is to project the 2D outline of a 3D box onto the ground. Currently, this is not implemented though. I have some ideas for workarounds, but they all fall apart at some angle.
I don’t think implementing a 3D box outline would be too hard though. Essentially you would project all corners of the box onto a 2D plane and then run a convex hull algorithm (e.g. the gift wrapping algorithm) on those points to get the 2D outline.

Even in the short term, that solution would solve my problem to an acceptable level. Obviously, with a procedurally generated world it is possible for the edge case to still happen and the cutter to create the stack overflow, so solving for this bug would be ideal. However, I understand you’re saying this isn’t the easiest fix so if it the cutter were able to have a better shape to follow the contours of the prop it sits around that really would be fantastic as a small workaround.

I’ve been looking into this and managed to implement a solution. There’s still some cleanup to do in the code, but I think I’ll be able to include it in the next update of the beta version.

You’re the best, thank you so much for doing that. It’s going to really help my project, thank you.