In general, node in point graph build link by checking distance.Is it possible to build link by custom instead of distance?
You can subclass the PointGraph and override the IsValidConnection method. That will allow you to control which connections are added and which ones are not.
See also this thread which has a few more details about a similar case http://forum.arongranberg.com/t/tagging-procedural-2d-tiles/4088/2
You can also construct a completely custom graph. Here is a tutorial for that: https://arongranberg.com/astar/docs/writing-graph-generators.php