The links system is a bit limited at the moment.
What you can do is to create a point graph which holds the start and end points of the ladder, then link the navmesh node with the start point of the ladder and the ladder ends and then the ladder end with another navmesh node near it. A bit messy, but it works.