Agents get extremely slow movement when theres a lot of them

So, in my tower Defense type game, the enemies are path agents and they move just fine. But when they have a high count (around 70-100), it seems like their movements get really slow, almost to a halt essentially.

I checked the aiPath component’s values and they are all correct and don’t change from when theyre walking fine and when theyre not. My FPS is also fine, its just the agents themself seem to transverse the paths really slowly.

Is that something that sounds familiar or a know issue that i’d be running into with that many entities?

I noticed that removing rigidbodies suddenly made all of the enemy agents behave completely normally. I am using AIPath component for my agents. Is there a way to keep my movement consistent much like how they are without rigidbody component, while keeping that component?

Reading more, it turns out that having a rigidbody on the gameObject forces the AiPath to use different movement calculations. Is there a reason why they would be much slower for me when theyre using that method?