JobCopyRectangle Crash

Hi,

we just tried upgrading our game with the latest A* version, but we are still experiencing strange issues and crashes. The most frequent one ist:

0x00007FFB586F9CCE (UnityPlayer) UnityMain
0x000001BFEECCD346 (Mono JIT Code) (wrapper managed-to-native) Unity.Collections.LowLevel.Unsafe.UnsafeUtility:MemCpy (void*,void*,long)
0x000001C03B9AC05B (Mono JIT Code) Unity.Collections.NativeArray`1<Unity.Mathematics.float4>:Copy (Unity.Collections.NativeArray`1<Unity.Mathematics.float4>,int,Unity.Collections.NativeArray`1<Unity.Mathematics.float4>,int,int)
0x000001C03B9ABFBB (Mono JIT Code) Unity.Collections.NativeArray`1<Unity.Mathematics.float4>:Copy (Unity.Collections.NativeArray`1<Unity.Mathematics.float4>,Unity.Collections.NativeArray`1<Unity.Mathematics.float4>)
0x000001C03B9ABEFB (Mono JIT Code) Unity.Collections.NativeArray`1<Unity.Mathematics.float4>:CopyTo (Unity.Collections.NativeArray`1<Unity.Mathematics.float4>)
0x000001C03B9AAF13 (Mono JIT Code) Pathfinding.Jobs.JobCopyRectangle`1<Unity.Mathematics.float4>:Copy (Unity.Collections.NativeArray`1<Unity.Mathematics.float4>,Unity.Collections.NativeArray`1<Unity.Mathematics.float4>,Unity.Mathematics.int3,Unity.Mathematics.int3,Pathfinding.IntBounds,Pathfinding.IntBounds)
0x000001C03B9A88D3 (Mono JIT Code) Pathfinding.Jobs.JobCopyRectangle`1<Unity.Mathematics.float4>:Execute ()
0x000001C03B9A850B (Mono JIT Code) Unity.Jobs.IJobExtensions/JobStruct`1<Pathfinding.Jobs.JobCopyRectangle`1<Unity.Mathematics.float4>>:Execute (Pathfinding.Jobs.JobCopyRectangle`1<Unity.Mathematics.float4>&,intptr,intptr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,int)
0x000001C03B9A860A (Mono JIT Code) (wrapper delegate-invoke) Unity.Jobs.IJobExtensions/JobStruct`1/ExecuteJobFunction<Pathfinding.Jobs.JobCopyRectangle`1<Unity.Mathematics.float4>>:invoke_void_T&_intptr_intptr_JobRanges&_int (Pathfinding.Jobs.JobCopyRectangle`1<Unity.Mathematics.float4>&,intptr,intptr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,int)
0x000001C0398CB6FE (Mono JIT Code) (wrapper runtime-invoke) <Module>:runtime_invoke_void__this___intptr&_intptr_intptr_intptr&_int (object,intptr,intptr,intptr)
0x00007FFB5D18CBA0 (mono-2.0-bdwgc) mono_get_runtime_build_info
0x00007FFB5D112112 (mono-2.0-bdwgc) mono_perfcounters_init
0x00007FFB5D11B10F (mono-2.0-bdwgc) mono_runtime_invoke
0x00007FFB57BE97FD (UnityPlayer) UnityMain
0x00007FFB57BE6BD3 (UnityPlayer) UnityMain
0x00007FFB57A6926B (UnityPlayer) UnityMain
0x00007FFB57A697FF (UnityPlayer) UnityMain
0x00007FFB57A6775C (UnityPlayer) UnityMain
0x00007FFB57A687BC (UnityPlayer) UnityMain
0x00007FFB57A67887 (UnityPlayer) UnityMain
0x00007FFB57A67B97 (UnityPlayer) UnityMain
0x00007FFB57A68CC0 (UnityPlayer) UnityMain
0x00007FFB57B537B8 (UnityPlayer) UnityMain
0x00007FFBE7457BD4 (KERNEL32) BaseThreadInitThunk
0x00007FFBE778CE51 (ntdll) RtlUserThreadStart


========== END OF STACKTRACE ===========

A crash has been intercepted by the crash handler. For call stack and other details, see the latest crash report generated in:
 * .../Crashes
Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak 
(Filename:  Line: 554)

With IL2CPP activated it is even worse :frowning:

Any idea what this could be or how i could help debugging this?

Best regards
Bennet

Hi

Do you think you could make a build with burst safety checks enabled?

Which exact version are you using?
And what are you doing when it crashes?

Safety checks are enabled, I think i even tried without Burst being activated and it sometimes crashed. It happens when our random map was generated and the graph is scanned. We scan the graph with the async method and additionally call UpdateGraph with bounds sometimes.

Maybe a few more interesting things:

  1. It just happens in Build
  2. If run in editor or with development build, then we also get an empty exception which is triggered by some Asserts in JobUtils (UtilityJobs, line 89). So even your Asserts seem to be off in our cases, but not sure why (i’ll immediately debug it now!)

Which exact version are you using?

Unity 2019.3.11f1 and the latest .21 A* version via package manager.

Would it be possible for you to upload an example project in which you can reliably replicate the issue?

Those asserts will catch the errors in the parameter values. When running in a release build any errors may cause the program to crash instead (which is what you are seeing).

Yeah i know that about the asserts, but isn’t it strange that it gets wrong parameters anyway?

Regarding a replication project i’m not sure. Would it help you if I send you a binary of the game? Otherwise i’ll see if i can build a repro project, but it’ll take some time i think.

Yes, it definitely shouldn’t get wrong parameters. It’s a bug for sure.

Not really unfortunately. It’s very hard to debug that.

Just debugged and the assert that fails is:

Assert(input.Length == inputSize.x*inputSize.y*inputSize.z);

Where:

  • input.Length = 39060
  • inputSize = int3(173, 1, 115)
  • so inputSize.x*inputSize.y*inputSize.z = 19.895

What’s the stack trace?
You can probably get better stack traces if you disable the unity worker threads for the job system.

The stack trace is:

JobCopyRectangle<float4>.Assert() at C:\...\Library\PackageCache\com.arongranberg.astar@4.3.21\Generators\Utilities\UtilityJobs.cs:line 87
JobCopyRectangle<float4>.Copy() at C:...\Library\PackageCache\com.arongranberg.astar@4.3.21\Generators\Utilities\UtilityJobs.cs:line 116
JobCopyRectangle<float4>.Execute() at C:\...\Library\PackageCache\com.arongranberg.astar@4.3.21\Generators\Utilities\UtilityJobs.cs:line 83
IJobExtensions.JobStruct<JobCopyRectangle<float4>>.Execute() at C:\...\JetBrains\Rider2020.1\resharper-host\DecompilerCache\decompiler\48050808-8AEE-40CA-B708-18FF164A29D5\3f\f754b760\IJobExtensions.cs:line 50

If you disable worker threads I think the stack trace should be deeper.

Can you give me a hint on how to that? Just found Unity.Jobs.LowLevel.Unsafe.JobsUtility.JobCompilerEnabled, but this doesn’t seem to change the stacktrace.

You can find it in the menubar. Burst -> Enable Jobs or Enable Worker Threads or something like that.

Alternatively you could just edit the GridGraph.cs code. There is a line where it calls dependencyTracker.ForceLinearDependencies. Make sure it passes ‘true’ as the argument.
That will force it to disable any threading.

Ok, got it working with the lineareDependencies thingy. Here the full trace:

Exception: Exception of type 'System.Exception' was thrown.
Pathfinding.Jobs.JobCopyRectangle`1[T].Assert (System.Boolean value) (at Library/PackageCache/com.arongranberg.astar@4.3.21/Generators/Utilities/UtilityJobs.cs:89)
Pathfinding.Jobs.JobCopyRectangle`1[T].Copy (Unity.Collections.NativeArray`1[T] input, Unity.Collections.NativeArray`1[T] output, Unity.Mathematics.int3 inputSize, Unity.Mathematics.int3 outputSize, Pathfinding.IntBounds inputBounds, Pathfinding.IntBounds outputBounds) (at Library/PackageCache/com.arongranberg.astar@4.3.21/Generators/Utilities/UtilityJobs.cs:116)
Pathfinding.Jobs.JobCopyRectangle`1[T].Execute () (at Library/PackageCache/com.arongranberg.astar@4.3.21/Generators/Utilities/UtilityJobs.cs:83)
Unity.Jobs.IJobExtensions+JobStruct`1[T].Execute (T& data, System.IntPtr additionalPtr, System.IntPtr bufferRangePatchData, Unity.Jobs.LowLevel.Unsafe.JobRanges& ranges, System.Int32 jobIndex) (at <480508088aee40cab70818ff164a29d5>:0)
Unity.Jobs.LowLevel.Unsafe.JobsUtility:Schedule_Injected(JobScheduleParameters&, JobHandle&)
Unity.Jobs.LowLevel.Unsafe.JobsUtility:Schedule(JobScheduleParameters&)
Unity.Jobs.IJobExtensions:Run(JobCopyRectangle`1)
Pathfinding.Jobs.IJobExtensions:Schedule(JobCopyRectangle`1, JobDependencyTracker) (at Library/PackageCache/com.arongranberg.astar@4.3.21/Utilities/JobDependencyTracker.cs:484)
Pathfinding.<UpdateAreaBurstCoroutine>d__103:MoveNext() (at Library/PackageCache/com.arongranberg.astar@4.3.21/Generators/GridGenerator.cs:1805)
Pathfinding.Jobs.<CompleteTimeSliced>d__6:MoveNext() (at Library/PackageCache/com.arongranberg.astar@4.3.21/Utilities/JobDependencyTracker.cs:95)
Pathfinding.<ScanInternalBurst>d__98:MoveNext() (at Library/PackageCache/com.arongranberg.astar@4.3.21/Generators/GridGenerator.cs:1257)
Pathfinding.<ScanInternal>d__106:MoveNext() (at Library/PackageCache/com.arongranberg.astar@4.3.21/Generators/GridGenerator.cs:1849)
<ScanGraph>d__143:MoveNext() (at Library/PackageCache/com.arongranberg.astar@4.3.21/Core/AstarPath.cs:1846)
<ScanInternal>d__142:MoveNext() (at Library/PackageCache/com.arongranberg.astar@4.3.21/Core/AstarPath.cs:1768)
<ScanGraphs>d__7:MoveNext() (at Assets/Scripts/Game/GraphUpdateManager.cs:28)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

Ok, I think i found the cause. Still somehow an error, because in the non-jobbified versions of A* it worked.

When the map is generated we adjust the GridGraph size and center to fit our newly generated map.

            var nodeSize = gridGraph.nodeSize;
            var width = Mathf.RoundToInt((bounds.size.x) / gridGraph.nodeSize);
            var depth = Mathf.RoundToInt((bounds.size.y) / gridGraph.nodeSize);

            gridGraph.SetDimensions(width, depth, nodeSize);

            gridGraph.center = bounds.center;

If i comment this out, i don’t get the error anymore (but this is not a solution for us). So i’d say it is some race condition between our size changes and the pathfinding. Any idea? Like I said in the earlier versions without jobs it worked - but maybe we just absued the system :sweat_smile:

Yup. Already found it.
I was trying to copy some earlier data when I resized an array, but if the grid graph had been resized the copying bounds would be incorrect (it shouldn’t even have tried to copy the data if the graph had been resized).

I’m uploading version 4.3.22 right now which fixes this bug. You can access in the package manager in a few minutes.

2 Likes