AIPath with optional WASD Keys (2D)

I have 2D setup with AIPath currently, and it works great for the clicking part. It goes right where I want it to.

However, I want to be able to use the WASD to move in their respected directions.

Currently I am using this method:

float horizontal = Input.GetAxis("Horizontal");
float vertical = Input.GetAxis("Vertical");

Vector2 direction = new Vector2(horizontal, vertical);

Vector2 modPos = agent.position;
agent.destination = modPos + (direction * 6f);

And the player does move in the right direction but once the player releases the key instead of stopping, the player pathfinds a couple steps back.

Is there any way to fix this?



Could you share a small video (with debug enabled so we can see the path).