Upon investigating further, I found it relates to me using two scenes instead of one. So I have a LevelCommon Scene that has the main camera, canvas, astar, etc. And then a Level Scene that has the tilemap and level design (enemies, pickups, triggers, etc.).
The in-game transition code loads the LevelCommon Scene first, then the Level Scene, preventing any weird issues.
But in the Unity Editor, I open both scene's to test. I noticed that for about a second, I get this error:
No RVOSimulator component found in the scene. Please add one.
I assume the the Unity Editor loaded the levels asynchronously and the Level Scene finishes first causing the error to occur for a split second. Even though it goes away and the AI works properly, this scenario causes any AI in the scene prior to entering Play Mode (aka not spawned) to have a velocity of zero despite pathfinding just fine.
That being said, not sure if this is actually a bug lol.
EDIT: Andddddd right clicking LevelCommon -> Set Active Scene appears to make this load first, fixing the errors and velocity issues. So looks like everything's good now. Thanks!