Hi,
have you seen this?
https://arongranberg.com/astar/documentation/4_0_8_e597295/graph-updates.php
Personally, in my game I try not to use colliders either.
I have code like this:
GraphUpdateObject graphUpdateObject = new GraphUpdateObject(bounds);
graphUpdateObject.updatePhysics = true;
graphUpdateObject.modifyWalkability = true;
graphUpdateObject.setWalkability = false;
AstarPath.active.UpdateGraphs(graphUpdateObject);
and it works for me. The only problem is LayeredGridGraph and paths that are above or under another path/blockade. For that I need to use basic colliders.