Increasing the number of threads will primarily reduce the latency between an agent requesting a path and it getting that path back when the system has a high load. If you only have one or two agents, it is completely unnecessary because there will pretty much never be more than one path that is being calculated at any one time. So no parallelism can take place.
The downsides of more threads is primarily a higher memory usage. Each thread has to store some data for every node. So if you have a large graph and many threads, the memory usage can add up.
Are you using RVO? One potential issue is that RVO adds one frame of latency to the agents reaction time, because it has to tell the RVO system where it wants to move, and then the RVO system has to calculate that, and finally the movement script has to read that info from the RVO system.
If you are using the ‘double buffering’ option on the RVO Simulator I would recommend you to disable it, because that adds another frame of latency as the RVO calculations then run over several frames.