Local Avoidance with a Character Controller

Is there any way to do local avoidance with a Character Controller? I have seen several topics on it but they haven’t had full answers as to how you would do so. I have my movement script which adds an agent, and sets its properties. Then I have this script in my move function:

agent.Locked = false;
agent.DesiredVelocity = new Vector3(direction.x * speed * amount, 0f, direction.z * speed * amount);
characterController.Move(new Vector3(agent.InterpolatedPosition.x - transform.position.x, -100f, agent.InterpolatedPosition.z - transform.position.z));
agent.Locked = true;

The -100f for the y axis is just to keep him glued to the ground.

This causes the character to stand completely still, I assume because it take the simulator some time to calculate the interpolated position. The hardest part of this is actually keeping the agent’s location in sync with the transform’s. Say you hit a wall. The agent will keep moving, while the character will not.

Any insight into how to do this?