This question has been retracted.
The problem was that the pathfinder was plotting paths ontop and inside of walls, and generally everywhere it wanted to.
(It was preferring either a straight line trough terrain, or plotting a long winding path ontop of the walls)
The solution was to add a real floor made out of vertices, and not have it walk ontop of a plane.