Too many command builders active - again?!

General
1
  • A* version: 5.3.7
  • Unity version: 6000.1.4f

Hi there,

We seem to be facing the same issue that was reported here, unless we are using a Recast Graph:
Too many command builders active. Are some command builders not being disposed?
(sorry, but posting links directly is not possible for me)

We are running our game server as a dedicated server build with no cameras, and recently all AI-controlled creatures stopped moving, with this error message appearing in our logs.
Before this, the server had been running without any issues for about 25 days, so it does not happen very often and cannot be reproduced on our side.

Exception: Too many command builders active. Are some command builders not being disposed?
  at Pathfinding.Drawing.DrawingData+BuilderData.Reserve (System.Int32 dataIndex, System.Boolean isBuiltInCommandBuilder) [0x00000] in <00000000000000000000000000000000>:0
  at Pathfinding.Drawing.DrawingData+BuilderDataContainer.Reserve (System.Boolean isBuiltInCommandBuilder) [0x00000] in <00000000000000000000000000000000>:0
  at Pathfinding.Drawing.CommandBuilder..ctor (Pathfinding.Drawing.DrawingData gizmos, Pathfinding.Drawing.DrawingData+Hasher hasher, Pathfinding.Drawing.RedrawScope frameRedrawScope, Pathfinding.Drawing.RedrawScope customRedrawScope, System.Boolean isGizmos, System.Boolean isBuiltInCommandBuilder, System.Int32 sceneModeVersion) [0x00000] in <00000000000000000000000000000000>:0
  at Pathfinding.Drawing.DrawingData.GetBuilder (Pathfinding.Drawing.RedrawScope redrawScope, System.Boolean renderInGame) [0x00000] in <00000000000000000000000000000000>:0
  at Pathfinding.Drawing.DrawingManager.GetBuilder (Pathfinding.Drawing.RedrawScope redrawScope, System.Boolean renderInGame) [0x00000] in <00000000000000000000000000000000>:0
  at Pathfinding.RVO.SimulatorBurst.UpdateInternal[T] (Unity.Jobs.JobHandle dependency, System.Single deltaTime, System.Boolean drawGizmos, Unity.Collections.Allocator allocator) [0x00000] in <00000000000000000000000000000000>:0
  at Pathfinding.RVO.RVOSimulator.Update () [0x00000] in <00000000000000000000000000000000>:0

On shutdown, we saw this message as well:

Drawing data is being destroyed, but a drawing instance is still active. Are you sure you have called Dispose on all drawing instances? This will cause a memory leak!

Are we missing something on our side, as the bug should have been fixed long ago?

2

It’s possible this is a regression or something :smiley: I’m seeing the thread you’re talking about (Too many command builders active. Are some command builders not being disposed?) and it looks like it was quite a minute ago. Thanks for the report, I’ll tag @aron_granberg on it :+1:

3

Thank you.

To clarify, is this a normal unity build which just happens to have no cameras being rendered, or does it use special unity flags to run headless?

4

It is a Linux Server IL2CPP build

5

I haven’t been able to replicate this in a window server build :confused:
I monitored the number of command builders active, and saw no increase over time.
Though, I didn’t run it for 25 days, of course.

If you are able to find some more details or way to replicate it, I’d be interested.

6

Thanks for looking into the issue.
Would it be possible for you to add some debug logging to AStar that could help track down the root cause if the problem happens again?
If you can provide the code, we can integrate it on our side and run it, as long as the amount of logging data remains reasonable, since excessive logging could impact our productive system.

From the call stack, it looks like the issue might be related to code responsible for gizmos or debug output. From my understanding, this should be inactive in a dedicated server build.
I don’t fully understand the code myself, so I’d appreciate hearing your thoughts on this!

7

Hi

That’s correct.
The relevant code reserves a builder for debug drawing, but always just discard it later.

8

Thanks for the clarification.
Could you please remove the debug code from dedicated server builds in a future AStar release?
These builds never use debug output, but the debug code can cause hidden errors (as reported here) and increase GC pressure.

9

I’ll remove it in the next beta update.