Sorry for the late answer. The reason for the late answer is that I have during the last week been experimenting with using the Unity Burst compiler to improve the performance of the local avoidance system and so far it is looking pretty good. I wanted to see what the performance was before I answered. It can now in the most optimal scenarios (i.e. pretty much no overhead, all agents are just rendered as quads in a single mesh, they don’t even have their own GameObjects or pathfinding) handle 20000 agents on my computer at 20 fps (or 120 fps if I restrict the RVO simulation fps and use double buffering, though that has some consequences for the local avoidance quality).
Note that in the RTS beta there is the SC2Avoidance script which currently does some relatively slow stuff and tends to be the slowest part of the simulation. It loops over many nearby agents in order to determine if it should stop or not. This is possible to optimize significantly, but that has not been done yet. This is one of the reasons it is still a beta.
Here is a screenshot of 20000 agents
Note that you should not expect to have anything near 20000 units in the game with RVO as the above scenario was pretty much the ideal case. Even the code for generating the mesh to render the agents had to use the Burst compiler.