Multiple stacked Recast Update Issue

I have 16 recast graphs (300x25x300@1.5/3) stacked for updating floor areas in a building, they are generated and the initial scans done via code. They generate fine on start and I have 30-50 blue stacked layers. I am then making changes (wall, chair, shelf) and trying to use UpdateGraphs two ways but getting bad results each time.

  1. Passing just the world bounds of the area changed. This works, but triggers an update on all the graphs so I get multiple errors that a graph couldn’t find any MeshFilters as expected (Haven’t built anything yet). I’m guessing the updategraphs(worldBounds) isn’t properly determining which graphs to update and ends up calling them all? If all the floors in the building request a graph update (like on spawn, x30-50) then Unity will wait about 30 seconds, the graph thread goes full on, then unity balloons to 35GB of ram and dies. I’d expect the requests to be queued?

  2. I create a GraphUpdateObject using the same bounds as above. This doesn’t work at all, I get at least one no meshfilters error for each graph. I might as well have passed coords for the moon. No crash though. Graph thread never jumps.

Version: 4.1.22 - I was using a single 300x500x300 recast graph but some updates were taking forever, not against going back to this if better? Also, I can click “Scan” in the editor forever without impact except delay with scan graph :-\

Thanks

I may have resolved this using the first scenario and some code modification to GraphUpdateProcessor.cs:129

I added a dedupe check to prevent the introductory graphUpdateQueue from receiving duplicate requests based on its existing spatial data vs request. I also added the same check to QueueGraphUpdatesInternal but it doesn’t appear to cause any problems, though I am still seeing graphs without any buy in to the graph update request turning down offers to update but I think I read that’s normal for some graph types (not sure about recast)?

Thanks

Hi

Ah. The ‘couldn’t find any mesh filters’ is really just a warning. I should probably do something about that being fired during graph updates. The only effect is that the tiles will be completely empty, no navmesh will be generated because no geometry was found, as expected.

First though. Why do you need so many graphs? A recast graph can already handle multiple floors in a building, why do you need more than one?

If you are updating the graph during runtime I would recommend that you use a tiled recast graph (set the ‘Use Tiles’ setting), best performance for these cases is usually so that a tile is about the same size or a bit larger than the size of the volume you usually update the graph with.

35GB does seem like a lot. It would allocate a bunch of memory for each graph to be able to update it, but that is a lot of memory…

Also. If you can, take a look at navmesh cutting. That is a lot faster usually, so if your obstacles can be represented as navmesh cuts, you could see large performance improvements.
See https://arongranberg.com/astar/docs/navmeshcutting.php

I broke up the recast while trying to figure out the freezing related to the repeat queue entries. I haven’t retried the single recast with tiles since figuring it out. I’m running into an issue now where all my AI freeze after reaching a point. Then sit still until I do a graph update, then they resume for a bit before freezing again. I have just started looking at this one so it may be in the forums somewhere.

Thanks

One thing that might be the cause of that is that it cannot calculate a path because the graphs are busy being updated in a separate thread (recast graph updates are mostly handled in a separate thread).

I have debug logs on ScanGraph() and ScanAsync() to notify of scans and I’m not seeing anything. I’ve also got the profiler up and the threads are all calculating paths continuously, though not a peep on graph update thread itself. Then I run a manual ‘global’ update (emulate scan button) and then they start moving after it completes. While sitting still the calculating path shows on all the threads pretty active. It’s as if the bot just never gets the result.

Thanks

Okay. You are not seeing warning messages from the seeker that the previous path was canceled because a new path was requested then?

Not a peep. If I turn debugging to ingame, the path number climbs like the national debt. Any other just shows a console full of Path Completed with computation times ~5-25ms Nodes:0 Path Length 1.

Hm… Could you perhaps show a screenshot of the graph when you AI is stuck?
From that log message, I get the feeling that you have recalculated the graph and it has included the AI in the scan, so a small piece of navmesh has been generated on top of the character’s head, and so it thinks it is stuck right on that spot.

Showing individual graphs (all identical) mapping out floors 5f apart vertically.
Showing 5 bots spawning and moving off in their own ways.
image Same bots now sitting still after successfully repeating move/find a few times.
Profile while bots are idle
IdleBot for IdleByte :frowning:

Oh wow, that’s a lot of floors.
You should be able to handle that with a single recast graph though. Downside is that recalculating a tile will still have to recalculate that tile on every single floor, but at least it’s better than now because it is doing way more work than that right now.

Can’t see anything wrong with your idle agents though.
You can show their current paths by checking ‘Draw Gizmos’ on the Seeker component.

You do seem to have a lot of simultaneous path requests though, I count 13 parallel path requests, which you really shouldn’t have unless you have at least 13 different characters, but I only count 5 in that picture. Are you using a custom movement script or recalculating their paths manually somehow?

I created the multiple recasts to prevent the update from locking up all floors while it refreshes a group (current Y:25f range per ‘slice’) of them. edit: does this work/help? Should the slices be turned to go up/down if so? /edit
Seems to update very fast across all 50 floors. I removed the layers for the player and NPC from the graphs but the freezing still occurs.

I’m using the standard Bot with included scripts with the wandering script from the site with radius. I’m testing with 5-10 per floor for testing. With 50 floors = 250 agents, freezing occurs with 5 agents, just happens faster with 250. It works fine until they all go to get the next path and stop. The gizmos show nothing other than the collider outline around the spider once static. Refreshing the graphs globally causes the spiders to immediately draw lines and move around (if they have gizmos on). Then they eventually stop (not all at once but very close) until I refresh graphs again. If i keep clicking refresh graphs they will move forever o_O but that’s choppy and not a good fix.

The lastRepath variable aipath just counts up. The Prev Frame on AIPath and Last PathID on seeker also increment continuously. Also, reached end of path true, is stopped false (but it’s sitting still), Waiting for path calculation true, can move/search true. Hope any of this helps.

Definitely a disconnect somewhere :frowning: heading to bed for now.

Thanks
imageIt’s very pretty to watch up until they all stop :slight_smile:
image Spiral wall introduced hoping some complexity might free them up, no change.

10 Minutes after spawn, some are still working their way around the spiral. Once they get close to the destination it’s like the the last step somehow freezes.

My cpu never goes above 54%, the pathfinding threads 1-14 are a small percentage of the graph. With 250 agents active I’m seeing


while some sit idle. I’m wondering if the pathing threads should be more loaded/active or the dispatch to the pathing threads needs to be more beefy? Maybe put ai control on a thread of it’s own since the numbers are getting up there?

I’m thrashing at this point, I’v got ai wandering off the spiral and walking in circles as if they forgot they started the path and needed to reset :frowning:

Thanks again for any help.

Been reading http://dspace.lib.ntua.gr/dspace2/bitstream/handle/123456789/41363/Georgios_Karafotias_Masters_Thesis.pdf?sequence=4 [Thesis: Crowd Evacuation Simulation of a Passenger Ship in Unity3D]
Where it talks about a single thread being able to manage ~250 agents (pg 64).

Basic slide script just using seeker to ask for path and using it until reaching target, since this is mostly architectural visualization the rapid update isn’t necessary. 10000 cylinder agents on 50 floors using just Update. This will probably work for me for now.

Hm… That is odd. I would have expected the pathfinding system to be overloaded if you are seeing them freeze like that, but apparently that is not the case, you seem to have plenty of CPU power to spare…
I’m not sure what could be going on… :confused: