Creating a responsive Diablo-esque movement script

I’ve been struggling a bit combining pathfinding with some responsive movement controls in action RPGs like Diablo. The way I understand those controls work is that the agent walks directly to the click location if it is visible, and if not they perform some simple pathfinding avoiding some obstacles, but mostly still walk directly towards the mouse.

Using a Grid Graph I thought adding a Raycast Modifier would solve the problem, and it does seem to help sometimes. But somehow the paths it sometimes computes look really weird, e.g. as in the screenshot

I don’t have any other modifier on the player character, only AIPath, RaycastModifier and Seeker (and a Character Controller, but in this case there are no obstacles and the floor is one flat mesh).

The other agent on the graph have basically the same setup, except they use the AlternativePath modifier instead of the RaycastModifier.

Is it even a good idea to have both of these on the same graph? It feels that I need some way of making the player paths higher priority than anything else in case it starts taking nontrivial amount of time to compute the paths, but I’m not sure if that’s something that’s possible?

edit: One more question: since I want the character to follow the mouse when the button is kept pressed, I’m doing something like this

if (Input.GetMouseButton(1)) {
    _movementScript.destination = mousePoint;
    _movementScript.SearchPath();
}

But intuitively since the mouse doesn’t move much between two frames it feels quite inefficient to do it like this, as the path will barely change, but I haven’t found a good alternative.