Having an agent go to an obstacle is always using the same path

Hello!

When I set the destination of an agent to a building in my RTS game, the agent will always try to go the the exact same node whatever his initial position.

At the moment, I simply do a Raycast. If the raycast collides with a building, I set the destination of my unit to go to the building position.

Basically, if the unit is closer to the left of the building, I would like the unit to stop to the closest node of the left of the building. If the unit is closer to the right of the building, I would like the unit to stop to the closest node to the right of the building. The issue is that at the moment, whatever the unit destination, he may walk all around the building to go to a specific node, why?

I’m using a Grid Graph and my building are obstacles (they create holes in the grid graph).

/// <summary>
/// Give units orders based on the returned hit object
/// </summary>
private void GiveOrders()
{
    RaycastHit hit;

    if (Physics.Raycast(Camera.main.ScreenPointToRay(Input.mousePosition), out hit, 100))
    {
        for (var i = 0; i < SelectedUnits.Count; i++)
        {
            if (hit.transform.gameObject.TryGetComponent(out BuildingConstruction buildingConstruction))
            {
                SetUnitConstructOrder(SelectedUnits[i], buildingConstruction);
            }
           
        }
    }
}

/// <summary>
/// Make units construct the selected building
/// </summary>
/// <param name="unit"></param>
/// <param name="buildingConstruction"></param>
private void SetUnitConstructOrder(Unit unit, BuildingConstruction buildingConstruction)
{
    if (unit.transform.TryGetComponent(out Villager villager))
    {
        unit.AIPath.destination = buildingConstruction.transform.position;
        villager.SetBuildingConstruction(buildingConstruction);
        villager.Unit.UnitStateMachine.ChangeState(villager.UnitMoveConstructionState);
    }
}

What am I doing wrong? Thanks