The only thing I can think about which could cause that to happen is if the raycast modifier is configured to treat the agent itself as an obstacle. Check the layer masks for that component.
Graph raycasting is usually more stable though, so I recommend it.