Sometimes it really helps to spell out the problem. I created an overloaded version of ABPath that takes start and end node when you construct it (instead of start and end position).
This way, in Prepare, I can leave away the GetNearest and just use the given start and end node. Obviously this only works because I always know the start and end node already.
The increase in performance is significant though:
