Android multi-threading bug

Hi!

We’re having issues with the library when enabling multi-threading on some Android devices: the Grid graph path pooling fails every once in a while breaking the whole system. Based on our tests, roughly every fifth device exhibits the erroneous behaviour and the frequency increases as you increase the number of threads. The problem goes away if you disable multi-threading. In addition, everything works fine with iOS devices (with or without multi-threading).

Any hints/suggestions/bug fix ideas are welcome!

DETAILS
Grid type: Grid graph
Graph size: ~250x250 nodes with node size of 0.6
Pathfinding usage: approx. 20 active seekers querying a path once every 0.5-1.0 seconds
Astar version: 3.8.2 Pro (the bug was also present in the 3.7.x versions)
Unity version: 5.1.4f1
Android OS version: varies (the bug is present on 5.x and 6.x)

— EXAMPLE #1

STACK TRACE
ArgumentException: You are releasing a path which is not claimed at all (most likely it has been pooled already). Are you releasing the path with the same object (Astar (AstarPath)) twice?
Check out the documentation on path pooling for help.
Pathfinding.Path.Release (System.Object o, Boolean silent)
AstarPath.ReturnPaths (Boolean timeSlice)
AstarPath.Update ()

SYSTEM INFO
operatingSystem: Android OS 5.1.1 / API-22 (LMY48Y/cee4e8702d)
deviceModel: OnePlus A0001
deviceType: Handheld
graphicsDeviceID: 0
graphicsDeviceName: Adreno ™ 330
graphicsDeviceType: OpenGLES2
graphicsDeviceVendor: Qualcomm
graphicsDeviceVendorID: 0
graphicsDeviceVersion: OpenGL ES 3.0 V@84.0 AU@ (CL@)
graphicsMemorySize: 1024
graphicsMultiThreaded: True
graphicsShaderLevel: 30
maxTextureSize: 4096
processorCount: 4
processorType: ARMv7 VFPv3 NEON
supportedRenderTargetCount: 1
supportsStencil: 1
systemMemorySize: 2877

— EXAMPLE #2

STACK TRACE
ArgumentException: You have already claimed the path with that object (CharacterBody_Enemy_130 (Seeker)). Are you claiming the path with the same object twice?
Pathfinding.Path.Claim (System.Object o)
Seeker.OnPathComplete (Pathfinding.Path p, Boolean runModifiers, Boolean sendCallbacks)
Seeker.OnPathComplete (Pathfinding.Path p)
Pathfinding.Path.ReturnPath ()
AstarPath.ReturnPaths (Boolean timeSlice)
AstarPath.Update ()

SYSTEM INFO
operatingSystem: Android OS 6.0 / API-23 (MRA58K/160421432533f)
deviceModel: LGE LG-D855
deviceType: Handheld
graphicsDeviceID: 0
graphicsDeviceName: Adreno ™ 330
graphicsDeviceType: OpenGLES2
graphicsDeviceVendor: Qualcomm
graphicsDeviceVendorID: 0
graphicsDeviceVersion: OpenGL ES 3.0 V@140.0 AU@ (GIT@I49f5588d88)
graphicsMemorySize: 176
graphicsMultiThreaded: True
graphicsShaderLevel: 30
maxTextureSize: 4096
processorCount: 4
processorType: ARMv7 VFPv3 NEON
supportedRenderTargetCount: 1
supportsStencil: 1
systemMemorySize: 1872

Hi

Interesting…
What movement script are you using?
Is this with IL2CPP or without?
You can enable the define ‘ASTAR_POOL_DEBUG’ in the Optimizations tab to get some extra information, however that is mostly for making sure things are pooled at all, not being pooled too much as seems to be the case for you.

So this only happens on a few devices? And it happens consistently on only those devices?

We’re using IL2CPP and the movement is done by the Seeker component.

We’ve been playtesting our game with ~20 Android devices and maybe half of them exhibit this behaviour at some point (but some more often than others). And it occurs pretty consistently after playing the game actively for ~10-20mins (the game uses the pathfinding system continuously all the time).

Vilppu

Hi

Ok.
However the Seeker component does not do any movement, it merely calculates paths.

Ah yes sorry, I meant to say that we use the CharacterController component together with the Seeker.

Hi

Ok, but there must be some other script that is using the character controller to move things.

Yes, so basically:

  • each character has a Velocity property (=Vector3) that is updated every once in a while in FixedUpdate() by the active AI behaviour
  • each character is moved in Update() by:
  1. CharacterController.Move( velStep ), where velStep is (Velocity*Time.deltaTime)
  2. apply gravity by direclty modifying the Transform.position

Hi

Ok, so you are using some custom movement script that you have written?
Is this script using pooling at all (calling path.Release and path.Claim)? (this is not required, but I just want to get a sense for what is happening here).

Yes, we do call Path.Claim() and Path.Release(), which, as you said, isn’t probably necessary. We tried to remove those calls but that didn’t affect the Android bug.

Ok.
Are there any other exceptions that are being thrown (earlier) during the execution that could possibly have caused the pathfinding scripts to reach a bad internal state?

None that we are catching (but we haven’t checked if there might be some internal exceptions caught and handled by the A* library, but no errors or warnings are coming from there).

Here is our custom path planning component that wraps the Seeker component: https://www.dropbox.com/s/orvobolthokzn37/PathPlanner.cs

I have sent you a PM to a bleeding edge package. I have recently done some changes to the path queue that is used internally, possibly this can have fixed the issue.

I think it should be a clean upgrade, but make a backup just in case.
There are some upgrade notes, you can check them by opening the changelog.cs file. There is also the list of other fixes and improvements.

Thanks! We’ll update that and see if the problem persists.

So far so good, the pooling issue seems to be solved in the bleeding edge version!

However, we did get this error once (with the Jump Point search disabled, because we don’t use the “cut corners” feature):

Trying to use Jump Point Search, but support for it is not enabled. Please enable it in the inspector (Grid Graph settings).
UnityEngine.Debug:LogError(Object)
Pathfinding.<ScanInternal>c__Iterator14:MoveNext() (at Assets/External/AstarPathfindingProject/Generators/GridGenerator.cs:888)
<ScanGraph>c__IteratorA:MoveNext() (at Assets/External/AstarPathfindingProject/Core/AstarPath.cs:1559)
<ScanAsync>c__Iterator9:MoveNext() (at Assets/External/AstarPathfindingProject/Core/AstarPath.cs:1501)
AstarPath:Scan() (at Assets/External/AstarPathfindingProject/Core/AstarPath.cs:1395)
GameLogic.<executeRoutine>c__Iterator5E:MoveNext() (at Assets/Scripts/GameLogic/Gameplay/CmdLoadRoom.cs:235)
GameLogic.<executeRoutine>c__Iterator4A:MoveNext() (at Assets/Scripts/GameLogic/CommandProcessor.cs:53)

--- SYSTEM INFO ---

operatingSystem: Android OS 6.0.1 / API-23 (MOB30H/2751534)
deviceModel: LGE Nexus 5
deviceType: Handheld
graphicsDeviceID: 0
graphicsDeviceName: Adreno (TM) 330
graphicsDeviceType: OpenGLES2
graphicsDeviceVendor: Qualcomm
graphicsDeviceVendorID: 0
graphicsDeviceVersion: OpenGL ES 3.0 V@127.0 AU@  (GIT@I96aee987eb)
graphicsMemorySize: 194
graphicsMultiThreaded: True
graphicsShaderLevel: 30
maxTextureSize: 4096
processorCount: 4
processorType: ARMv7 VFPv3 NEON
supportedRenderTargetCount: 1
supportsStencil: 1
systemMemorySize: 1854

Hi

Awesome :slight_smile:

That’s odd. The code which logs that is here

#if !ASTAR_JPS
		if (this.useJumpPointSearch) {
			Debug.LogError("Trying to use Jump Point Search, but support for it is not enabled. Please enable it in the inspector (Grid Graph settings).");
		}
#endif 

So I am not sure how that could have been called without useJumpPointSearch being enabled…

Yeah it is weird, we haven’t touched on that toggle ever (even for testing)…

Actually, the bug still persists (although considerably less frequent now). Here’s the latest stack trace which we’ve encountered a couple of times:

ArgumentException: You have already claimed the path with that object (CharacterBody_Enemy_922 (Seeker)). Are you claiming the path with the same object twice?
Pathfinding.Path.Claim (System.Object o) (at /Users/buildservermac2/teamcity/work/d695c02e4fcf07f2/Assets/External/AstarPathfindingProject/Core/Path.cs:550)
Seeker.OnPathComplete (Pathfinding.Path p, Boolean runModifiers, Boolean sendCallbacks) (at /Users/buildservermac2/teamcity/work/d695c02e4fcf07f2/Assets/External/AstarPathfindingProject/Core/AI/Seeker.cs:250)
Seeker.OnPathComplete (Pathfinding.Path p) (at /Users/buildservermac2/teamcity/work/d695c02e4fcf07f2/Assets/External/AstarPathfindingProject/Core/AI/Seeker.cs:230)
Pathfinding.Path.ReturnPath () (at /Users/buildservermac2/teamcity/work/d695c02e4fcf07f2/Assets/External/AstarPathfindingProject/Core/Path.cs:712)
Pathfinding.PathReturnQueue.ReturnPaths (Boolean timeSlice) (at /Users/buildservermac2/teamcity/work/d695c02e4fcf07f2/Assets/External/AstarPathfindingProject/Core/Misc/PathReturnQueue.cs:73)
AstarPath.Update () (at /Users/buildservermac2/teamcity/work/d695c02e4fcf07f2/Assets/External/AstarPathfindingProject/Core/AstarPath.cs:812)

--- SYSTEM INFO ---

operatingSystem: Android OS 5.0.1 / API-21 (LRX22C/1602158)
deviceModel: motorola Nexus 6
deviceType: Handheld
graphicsDeviceID: 0
graphicsDeviceName: Adreno (TM) 420
graphicsDeviceType: OpenGLES2
graphicsDeviceVendor: Qualcomm
graphicsDeviceVendorID: 0
graphicsDeviceVersion: OpenGL ES 3.0V@95.0 (GIT@I86da836d38)
graphicsMemorySize: 1024
graphicsMultiThreaded: True
graphicsShaderLevel: 30
maxTextureSize: 16384
processorCount: 4
processorType: ARMv7 VFPv3 NEON
supportedRenderTargetCount: 1
supportsStencil: 1
systemMemorySize: 2970

Hm. That’s annoying…
I am not really sure what to do about it however…