[ECS] FollowerEntity performance issues

[Will_44]

• A* version: 5.2.5
• Unity version: 6000.0.25f1
• Render Pipeline: URP

TLDR:
[ECS] Issues in AIMovementSystemGroup and JobRepairPath drastically impacting performance

Summary:
Hey so my team and I have been running into issues with the ECS performance. The issue seems to be related to the Repair Path functionality. We created a sample project that reproduces this issue linked here GitHub - willmorris44/Astar-Pathfinding.

Breakdown of the sample project:
There are two scenes. A scene with a complex mesh and a scene with a simple mesh. In each of these scenes there is a cube (the target). There is also a TargetManager in each scene. This manager gets the position of the cube in the scene. The PathTargetSystem then sets the DestinationPoint for each Follower entity using the position from the TargetManager.

When the game starts, the EnemySpawner in the scene will spawn the set number of Follower entities. The rest is left to the AStar Pathfinding package.

Issues:
The performance is fine until around 1000 entities are spawned. Once it reaches this number, performance starts to exponentially drop (frames drop to around 5fps when 3000 are spawned). This seems to be happening for 2 reasons.

1. AIMovementSystemGroup is triggering the RepairPathSystem to repair paths multiple times each frame. In the ShouldGroupUpdate method, there seems to be logic for triggering a group update if there is something taking too much processing time. However, it doesn’t seem to take into account if the Group itself is taking up too much processing time. That means, if the RepairPathSystem hangs or idles for too long, the AIMovementSystemGroup will trigger another RepairPathSystem.OnUpdate call, resulting in multiple calls per frame.

2. The RepairPathSystem is idling for too long. For some reason (was not able to figure this one out) when the JobRepairPath.Execute tries to repair the path, the PathTracer.GetNextCornerIndices (or sometimes GetNearest?) is taking many milliseconds per frame.

These two issues are the main contributors to high CPU usage times per frame. When spawning 3000 entities we are seeing Current Frame Accumulated Time of around 400ms. This is happening on both the complex and simple mesh (though the simple mesh does a bit better). Our main questions are:

1. Is this just an issue with too many triangles? The complex mesh is not very optimized and has funky geometry, but the simple mesh is simply a large plane.

2. Are we setting something up wrong?

3. Is this an issue with the package?

Any help would be greatly appreciated!

[aron_granberg]

Hi

For stability, the FollowerEntity tries to run its simulation loop at a fixed fps. However, it will reduce this desired fps if it cannot keep up.

It sounds like your system is overloaded.

What system are you running this on? Are you sure burst compilation is enabled?

[Will_44]

Burst compilation is enabled. If I am understanding your system question correctly, I am running this on an Intel i7 13700k and a Nvidia 3080.

We know its used for stability but it does not seem to be working. If we hard code numUpdatesThisFrame = 1, we see a decent boost in fps / performance. We believe this is because the AIMovementSystemGroup is unaware that its own systems are causing the large delta time. This also does not explain why Repair Paths is taking so long to finish.

If this is because the system (CPU) is overloaded, that means the real issue for us is the mesh? It can’t handle the number of triangles for that many entities? The simple mesh has around 300,000 triangles.

[aron_granberg]

Hi

The repair path job is the primary path calculation job of the FollowerEntity. It is expected that this is taking the most amount of time.

However, it does sound like it’s taking a really long time for you.

This is what I get when using 1200 agents in a simple scene, including local avoidance. I’m running on a i9-9900k, but according to benchmarks, your CPU should be faster than mine.

@Will_44

The simple mesh has around 300,000 triangles.

What mesh are you talking about? The navmesh?

The RepairPathSystem will take longer to run if the destination has changed. So a static destination will be faster (or a destination that only moves say once every 10 frames).

Do you have a screenshot of your profiler in the timeline view (with jobs expanded)?

[Will_44]

Yes the navmesh. In the sample project linked there are two example navmesh’s, a simple one (flat plane) and a complex one (mountains).

Here is a screenshot of the profiler (this one is actually on a MacBook M2 Pro and is performing better) with 3000 entities on the simple navmesh.

Note: In the Job setting the DestinationPoint, it only sets every 100 Execute calls

[BurstCompile]
[UpdateBefore(typeof(Pathfinding.ECS.AIMovementSystemGroup))]
public partial struct PathTargetJob : IJobEntity {
    [ReadOnly]
    public NativeArray<float3> targetPositions;

    void Execute(ref DestinationPoint destinationPoint, ref PathTargetComponent pathTargetComponent, in LocalTransform transform) {
        if (pathTargetComponent.checkCount > 100) {
            pathTargetComponent.checkCount = 0;
        } else {
            pathTargetComponent.checkCount += 1;
            return;
        }

        destinationPoint.destination = targetPositions[0];
    }
}

[Will_44]

Here are some more screenshots of the Jobs. It looks like all those little tasks are sub-tasks of GetNextCorners

[aron_granberg]

That’s a lot of time spent in GetNextCorners… Do you have any screenshots of your paths?

[Will_44]

Not entirely sure what you mean by ‘paths’ but I took these:

[Will_44]

As well as these:

[aron_granberg]

Hi

That’s a very high resolution navmesh graph. It will make the follower entity spend a lot of time simplifying its path.

A navmesh graph should contain large triangles covering as much of the world as possible.
Your test world could probably be covered by less than 10 triangles, instead of the thousands that you are using.

Is there any reason you are not using the recast graph?

[Will_44]

Yes the number of triangles is mostly correct. The actual world we will be using will be similar in the number of triangles. Here is an example image of the terrain:

We are hoping to optimize the mesh so that it has fewer triangles, but are still searching on how to do this. However, I imagine it will still be many.

What would be the difference of navmesh vs recast graph in this situation? Is the recast graph more suited for this?

[aron_granberg]

Hi

Where on that terrain are the characters actually supposed to walk? Your mesh looks like it’s just covering the whole surface. A navmesh represents where the characters should be able to walk.

[Will_44]

They should be able to walk anywhere on the terrain so covering the whole surface is correct

[aron_granberg]

Well, you can definitely make it much lower resolution.
The navmesh only needs to loosely track the agents’ y-coordinates.

But why do you need pathfinding at all, if they can just walk straight to the targets?

[Will_44]

We figured we would need pathfinding because we plan to have irregular structures like arches, bridges, caves, etc.

What do you mean by “loosely track the agents’ y-coordinates”? Would this create a visual discrepancy of the characters path and the visual terrain? If the graph is a lower resolution than the terrain, there would be discrepancies with normals as well, correct? Is the recommended approach to just raycast and correct based on a criteria?

[aron_granberg]

The agent uses a physics raycast to position itself on the ground. So the path doesn’t need to line up with the ground perfectly.

[Will_44]

I was able to reduce the number of triangles by 99% (from 300,000 to about 3,000). However, I am still getting subpar performance. With 2000 characters, I am only getting about 30fps. This is with Burst compile enabled.

[aron_granberg]

Hi

Would you mind trying one more thing. If you change the Movement Plane Source on the FollowerEntity to “Graph” instead of “Raycast”, does that affect the performance?

[Will_44]

Unfortunately no, the performance stays about the same. The project is in the GitHub link provided in the original post if you’d like to take a closer look.