Multiple GraphUpdates issue on XboxOne

General Recast graph
#1

Hello Aron,

We are having an issue with Astar specifically on Xbox One and we need your help. We perform a graph update (using the TileHandlerHelper) only once in a while (never more than once per frame) and we check if Astar.active.IsAnyGraphUpdatesQueued is true to avoid overlapping updates. This works fine on PS4 and PC but on Xbox One we sometimes get this exception :

Exception: Processing work items recursively. Please do not wait for other work items to be completed inside work items. If you think this is not caused by any of your scripts, this might be a bug.

This suggests that ProcessWorkItems was called again before the work from the previous call was completed.

Is checking Astar.active.IsAnyGraphUpdatesQueued the right way to check if an update is in progress? Is there another way to get notified when the graph update work is done. Is this a bug that specifically affect the Xbox One?

Any help would be appreciated.

Thank you.

#2

Hi

Do you happen to have the stacktrace for that exception? That would be very helpful.

IsAnyGraphUpdatesQueued should be a good way to prevent overlapping updates.
Also see http://arongranberg.com/astar/docs/class_astar_path.php#aae2403939ee3da2029ad5e80042dc8c3

#3

Hello Aron,

Indeed I should have posted it, in the first place.
Here it is:

Exception: Processing work items recursively. Please do not wait for other work items to be completed inside work items. If you think this is not caused by any of your scripts, this might be a bug.
at AstarPath.ProcessWorkItems (Boolean force) [0x0001b] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:991

at AstarPath.PerformBlockingActions (Boolean force, Boolean unblockOnComplete) [0x00033] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:907

at AstarPath.Update () [0x00000] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:865

In the meantime we are going to try to increase the maxGraphUpdateFreq.

Thanks,

L.

#4

Hi

Are you sure there is no exception or something before that?
It looks like some earlier work item has failed due to some exception. At least I see no other way for that to happen other than an earlier exception.

#5

Hello Aron,

There is indeed another exception that I missed, here is the stack trace :

NullReferenceException: A null value was found where an object instance was required.
at Pathfinding.MeshNode.FloodFill (System.Collections.Generic.Stack`1 stack, UInt32 region) [0x0001c] in ***\dev\Assets\AstarPathfindingProject\Core\Nodes\GraphNode.cs:329

at AstarPath+c__AnonStorey8B.<>m__23 (Pathfinding.GraphNode node) [0x0012f] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:1851

at Pathfinding.RecastGraph.GetNodes (Pathfinding.GraphNodeDelegateCancelable del) [0x0006d] in ***\dev\Assets\AstarPathfindingProject\Generators\RecastGenerator.cs:537

at AstarPath.FloodFill () [0x000ef] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:1861

at AstarPath.EnsureValidFloodFill () [0x0000b] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:946

at AstarPath.ProcessWorkItems (Boolean force) [0x000da] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:1024

at AstarPath.PerformBlockingActions (Boolean force, Boolean unblockOnComplete) [0x00033] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:907

at AstarPath.Update () [0x00000] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:865

Also, I tried setting the maxGraphUpdateFreq to a larger value (1.5) and it definitely helps. I get another exception though :

IndexOutOfRangeException: Array index is out of range.
at Pathfinding.RecastGraph.ReplaceTile (Int32 x, Int32 z, Int32 w, Int32 d, Pathfinding.Int3[] verts, System.Int32[] tris, Boolean worldSpace) [0x00465] in ***\dev\Assets\AstarPathfindingProject\Generators\RecastGenerator.cs:1932

at Pathfinding.Util.TileHandler+c__AnonStoreyA9.<>m__48 (Boolean force) [0x0017c] in***\dev\Assets\AstarPathfindingProject\Generators\Utilities\TileHandler.cs:1253

at AstarPath.ProcessWorkItems (Boolean force) [0x0005e] in ***\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:1006

No other exception comes before this one. This happens after we call NavmeshCut.ForceContourRefresh() and NavmeshCut.ForceUpdate()

Thanks,

L.

#6

Hm… What version of the system are you using?
Try the latest version as that looks suspiciously like a race condition and I have changed some code related to locking recently.

#7

At least Xbox One uses x86-64. So this shouldn’t be caused by architectural differences.

#8

Ok!
We will try it and keep you posted!

Thanks,

S.

#9

Hello Aron,

Me again. We had to put X1 on the ice for some time but I’m now back working on porting our game on it. In the meantime we updated the version to 3.7.5 since we wanted to benefits from the “coroutined” scan.
That being said we are still experiencing the same issue with exactly the same errors.

Did you happen to find any new leads on this one ?

Thanks,

L.

#10

Hi

Do you happen to have a relatively small example project which you could share with me?
Then I could debug it more easily.

#11

Hello Aron,

After a more detailed investigation here is what I found:
It appears that in the function GraphNode.FlooFill, the variable “other” can be null. In my case the connections array contained 2 elements, both set to null. Will a simple null check be enough or is a more deep issue ?
If I continue the execution after receiving this error then the thread exception arises.

For information we are using NavCutters for our units.

Do you have any idea of what can cause this issue or what I can do to give you more details ?

Thanks again for your precious help.

Laurent

#12

Hi All.

I’m going to chime in here since this error has new started surfacing on our PC build as well.

here are the errors we are getting:

Exception while updating graphs:
System.NullReferenceException: Object reference not set to an instance of an object
at Pathfinding.MeshNode.FloodFill (System.Collections.Generic.Stack`1 stack, UInt32 region) [0x0001c] in C:\Dev\DivideUnity5\Assets\AstarPathfindingProject\Core\Nodes\GraphNode.cs:347
at AstarPath+c__AnonStorey48.<>m__7 (Pathfinding.GraphNode node) [0x0012f] in C:\Dev\DivideUnity5\Assets\AstarPathfindingProject\Core\AstarPath.cs:1882
at Pathfinding.RecastGraph.GetNodes (Pathfinding.GraphNodeDelegateCancelable del) [0x0006d] in C:\Dev\DivideUnity5\Assets\AstarPathfindingProject\Generators\RecastGenerator.cs:548
at AstarPath.FloodFill () [0x000ef] in C:\Dev\DivideUnity5\Assets\AstarPathfindingProject\Core\AstarPath.cs:1892
at AstarPath.ProcessGraphUpdatesAsync (System.Object _astar) [0x00099] in C:\Dev\DivideUnity5\Assets\AstarPathfindingProject\Core\AstarPath.cs:1375

Next up:

NullReferenceException: Object reference not set to an instance of an object
Pathfinding.MeshNode.ClearConnections (Boolean alsoReverse) (at Assets/AstarPathfindingProject/Core/Nodes/GraphNode.cs:326)
Pathfinding.GraphNode.Destroy () (at Assets/AstarPathfindingProject/Core/Nodes/GraphNode.cs:55)
Pathfinding.RecastGraph.ReplaceTile (Int32 x, Int32 z, Int32 w, Int32 d, Pathfinding.Int3[] verts, System.Int32[] tris, Boolean worldSpace) (at Assets/AstarPathfindingProject/Generators/RecastGenerator.cs:1713)
Pathfinding.Util.TileHandler+c__AnonStorey64.<>m__2E (Boolean force) (at Assets/AstarPathfindingProject/Generators/Utilities/TileHandler.cs:1231)
AstarPath.ProcessWorkItems (Boolean force) (at Assets/AstarPathfindingProject/Core/AstarPath.cs:1039)

followed by:

IndexOutOfRangeException: Array index is out of range.
Pathfinding.PathHandler.GetPathNode (Pathfinding.GraphNode node) (at Assets/AstarPathfindingProject/Core/PathHandler.cs:272)
Pathfinding.TriangleMeshNode.Open (Pathfinding.Path path, Pathfinding.PathNode pathNode, Pathfinding.PathHandler handler) (at Assets/AstarPathfindingProject/Generators/NodeClasses/TriangleMeshNode.cs:191)
Pathfinding.ABPath.CalculateStep (Int64 targetTick) (at Assets/AstarPathfindingProject/Pathfinders/ABPath.cs:355)
AstarPath.CalculatePathsThreaded (System.Object _threadInfo) (at Assets/AstarPathfindingProject/Core/AstarPath.cs:2502)
UnityEngine.Debug:LogException(Exception)
AstarPath:CalculatePathsThreaded(Object) (at Assets/AstarPathfindingProject/Core/AstarPath.cs:2577)

then :
Unhandled exception during pathfinding. Terminating.

and:

Error : This part should never be reached.

this is happening with 3.8.2 & unity 5.3.3f1. We are also using a cached graph.

Adding Null checks at lines 347 & 326 of GraphNode.cs eliminates the errors, I’m just not sure if there might be other issues that it will cause that I’m unaware of.

#13

So it looks like the issue is related to the data from 3.8.1

When I made a new graph with 3.8.2, all the errors went away, and NPCs are traversing the scene properly again.

#14

Updated to 3.8.2 and still having the same issue (on xboxone).
Also the wonderful ScanAsync is gone.
And I did completely rebuild the Astar object so I don’t have any “data migration” issue.

Still needing help!
Thanks.

L.

#15

Hi

Ok, so this definitely looks like some multithreading issue.
How are you loading the graphs? Are you using cached startup directly or are you loading a graph by code from a file?

Try disabling multithreading and see if that helps.

Can this issue be replicated consistently?

@lsalaun
Yeah, ScanAsync was written in another branch which I have not fully merged in to the main branch yet. It will be back in a future update. I released it to the beta channel because I thought I would have merged in that branch by the time I released 3.8.

When does this occur? Right at the start of the game or at an arbitrary time?

#16

Hello Aron,

We do not load any graph we recast the whole level at runtime during loading (explain the needs of the async since it takes quite a long time)

I disabled the multithreading by setting the threadcount to none. Yet the issue still arises.

I get the issue 100% of the time, yet it’s kind of random and I’m still not able to nail down the exact moment where it happens.

It never happens during initial scan of the graph. Always during gameplay.

What we do looks a little bit like this:
We have a turn by turn game so the flow of a unit turns looks a little bit like this pathfinding wise:

  • Deactivate units navmeshcut.
  • reduce/increase size of allies/enemies respectively navmeshcut
  • Launch a graphupdate using TileHandlerHelper.ForceUpdate the next frame.
  • Then let the player move around
    or
    If it’s ai. Then deactivate the navmeshcut of the target, update the graph, wait for it to finish. calculate a path to the target. Then reactivate it’s navmeshcut and deactivate the next targets one. update the graph once again and calculate the path ot the target. as many time we need it. And then move the AI around.

Hope it helps.

L.

#17

Here is another error I just had

NullReferenceException: A null value was found where an object instance was required.
at Pathfinding.PointNode.ClearConnections (Boolean alsoReverse) [0x00018] in P:\mordheim\dev\Assets\AstarPathfindingProject\Generators\NodeClasses\PointNode.cs:34
at Pathfinding.NodeLink2.RemoveConnections (Pathfinding.GraphNode node) [0x00000] in P:\mordheim\dev\Assets\AstarPathfindingProject\Core\Misc\NodeLink2.cs:175
at Pathfinding.NodeLink2.Apply (Boolean forceNewCheck) [0x00061] in P:\mordheim\dev\Assets\AstarPathfindingProject\Core\Misc\NodeLink2.cs:201
at Pathfinding.NodeLink2.OnGraphsPostUpdate () [0x00069] in P:\mordheim\dev\Assets\AstarPathfindingProject\Core\Misc\NodeLink2.cs:129
at Pathfinding.GraphModifier.TriggerEvent (EventType type) [0x000e5] in P:\mordheim\dev\Assets\AstarPathfindingProject\Core\Misc\GraphModifier.cs:58
at Pathfinding.Util.TileHandler+c__AnonStoreyC2.<>m__3D (Boolean force) [0x001a1] in P:\mordheim\dev\Assets\AstarPathfindingProject\Generators\Utilities\TileHandler.cs:1235
at AstarPath.ProcessWorkItems (Boolean force) [0x0005e] in P:\mordheim\dev\Assets\AstarPathfindingProject\Core\AstarPath.cs:1039

and looking at the callstack I must also add that we use NodeLink2 of which we set the StartNode/EndNode.walkable to false if they are to close to a unit. Just in case it raise flag on your side!

#18

Hi

Ok. Odd, it really seemed like a multithreading issue.
Do you happen to have a relatively small example project (or the whole thing) which you could share with me? Then I could debug it more easily.

#19

All those errors are mostly just symptoms but not the actual cause.
Nothing in the system should add connections which are null, but apparently it happens.

#20

Hello Aron,

I finally had the time to make a test project.
You can download the project here:

I gave you the full project. Just build and run on X1. Once you attached VS you should catch the error.

All the ugly code stressing the game is in the Class BallMover.

Keep me posted and thanks again for your time.

Laurent.