NavMesh cutting during bake

Hey, I feel like I may be missing something simple, so if anyone knows how to go about this, please let me know.

My goal is to simply mark a layer and/or object as one that will cut out the nav mesh, but I want it to happen when the recast graph is baked, not at runtime.

I understand that I can include layers into the bake, but I don’t want the object itself to be included as part of the nav mesh.

The simplest example I can think of would involve a cube: Let’s say I simply want to place a cube in the world, and I want that cube to cut a hole in the mesh (much like a building would). If I do that by including its layer in the bake layer, it will cut out the mesh, but it will also place nav geometry on the top of the cube itself.

We’re dealing with a very large map, so we are trying to avoid any unnecessary nav geometry.

Plus, if there was a way to simply have an object act as nav mesh blocking geometry, we’d be able to stretch it farther for our purpose.

Our ultimate goal would be to place navigation occlusion geometry around the world in areas where we know for sure would not need any nav mesh geometry. (such as the sides of a bottle-necked canyon in a campaign mission). Then when we bake the recast graph, those areas simply would be culled out of the graph completely.

Exporting as an obj and modifying the graph is something we are trying to avoid. I’ve also investigated the “Relevant Graph Surface Mode” and that is not proving to be a good fit for our purpose.

Any help on this would be greatly appreciated. Thanks!

Hi

Try try the RecastMeshObj component.
If you add it to your mesh and set the area field to -1, it will remove it from the navmesh.
See http://arongranberg.com/astar/docs/class_pathfinding_1_1_recast_mesh_obj.php
Sorry that the documentation for it is not very good atm, I have improved it in my dev version.

Hey, thanks for reply!

I tried the RecastMeshObj but it appears it is only half-working. Perhaps there is a setting I’m missing.

Here’s a simple example of what’s happening.

In this image, I have a Unity terrain and a cube that is to act as the nav mesh occluder.

The cube is not on a layer that the nav mesh uses and contains a recastMeshObj component with an area set to -1.

When I bake the graph, it does indeed appear to cut out the mesh now, which is great, but it only cuts out the edges. To be clear, the pink nav geometry is NOT on the cube itself, it is on the terrain below it. (It’s a flat piece of terrain for simplicity sake)

I tested this with a chunk of mountains as well, and the large encompassing cube causes the edges around it to be cut out, but the mountains themselves inside of the cube still have nav mesh geometry on them.

As you can see from the image, the base of this large cube has a cut around its edges, but the mountains inside of it still contain a nav mesh.

Thanks for the help!

Hi

That is unfortunately due to how the rasterizer works, only surfaces are considered and it does not know about volumes in any way.

Could you possibly use this together with the Relevant Graph Surface mode?

If you are sure that you will not need overhangs or tunnels, you can set the “character height” field to something really huge (so huge that it would not fit inside that cube), then I think you will get the behaviour that you want.

Hi Aron,

As Joe mentions, we are trying to build a nav mesh with as little waste as possible on a large 1K square terrain with enough fidelity to handle AI navigation around buildings as well as wide open terrain. As such, we’d like to control where the nav mesh is built and where it simply does not need to be built - thus saving valuable iteration time, memory and perf.

However, the tools available for culling out navigable areas are not working like we’d expect. This seems like it should be an important aspect for anyone who really cares about building an efficient high quality nav mesh.

Is there any assistance you could provide in helping us find a viable solution to this problem? We would very much appreciate it.

Thanks!

Hey Aron,

We will likely have tunnels, so the character height suggestions is not a feasible option for us unfortunately.

I’m curious, since the act of cutting the nav mesh at runtime does consider volumes and subtracts the nav mesh accordingly. How easy would it be to run the cutting algorithm through the API as soon as a bake completes?

Since our main issue is the final cached size of the terrain, if we could simply run the cut once after it constructs the mesh, it would definitely solve our problem.

If you have any suggestions, such as what file contains the baking code, and what functions are called when cutting the mesh at runtime, I don’t mind adding that functionality to our build if it’s not on your feature list. Granted, we may need a tad bit of guidance, but I don’t imagine it would be a complicated task.

Hi

Well, if you just add navmesh cut components along with a TileHandlerHelper in the scene it would cut out everything directly at start and not cause any more processing overhead.

It is currently not possible to do the cutting in the editor (it is a good idea though, I will consider adding it) however what you could do is to add navmesh cuts to the scene, run the game and then use the editor Save and Load tab to save the calculated data (including navmesh cuts) to a file, then when you release the final game you would have removed the navmesh cut components to avoid processing the cuts again during start.
See http://arongranberg.com/astar/docs/save-load-graphs.php

If you really need to run it in the editor you should have a look at the TileHandlerHelper script. Basically what you need to do is to run the Start method in the editor and also modify it slightly so that it really finds all NavmeshCuts even in the editor (it currently relies on that the navmesh cuts have run their OnEnable method). And then at last run the ForceUpdate method.
All this could probably be run from the AstarPath.OnPostScan event.

I was looking for a solution for this issue, since this is an old post I was wondering if this is still the only way to cut out areas of a Navmesh during the editor scan.

Thanks

I discovered the RecastMeshObj and it looks like that’s exactly what I needed.

1 Like

Actually RecastMeshObj did not work as I intended. Baking the NavmeshCut seems to be the best solution for the reasons stated above and to visually see the impact of the NavmeshCut without being in play mode. I attempted the above suggestion by aron but did not see a change.

I’m using a RecastGraph and added a TileHandlerHelper to the scene plus some NavMeshCut objects.

Created a script that listens for the OnPostScan event and the event gets called successfully.

Call Start on the tileHandlerHelper in the scene, then Call ForceUpdate.

In the ForceUpdate method I’m grabbing all the NavMeshCuts in the scene like so:

#if UNITY_EDITOR
			if (!Application.isPlaying) {
				if (cuts.Count == 0) {
					cuts = new List<NavmeshCut>(GameObject.FindObjectsOfType<NavmeshCut>());
				}
			}
#endif

I then Call ForceUpdate on the NavMeshCuts when they are iterated:

if (!Application.isPlaying)
     cuts[i].ForceUpdate();
 if (cuts[i].RequiresUpdate()) {
     any++;
     break;
}

Any guidance would be greatly appreciated.

Hi

You might have to call AstarPath.FlushWorkItems after you have done the other things to make sure the cutting code is executed.

Sorry for the necro, but I solved this issue, so if somebody needs NavmeshCuts to work at edit time, this does it. Place this script anywhere, and have a TileHandlerHelper in the scene. You’ll have to edit TileHandlerHelper in order to make the private stuff referenced here public, and make NavmeshCut get hold of it’s cached transform somehow, as that’s set up during Awake (I made it lazy)

#if UNITY_EDITOR
using Pathfinding;
using UnityEngine;
using UnityEditor;

[InitializeOnLoad]
public static class TileHandlerEditTime {
    static TileHandlerEditTime() {
        AstarPath.OnPostScan -= OnPostScan;
        AstarPath.OnPostScan += OnPostScan;
    }

    private static void OnPostScan(AstarPath astarPath) {
        var tileHandlerHelper = Object.FindObjectOfType<TileHandlerHelper>();
        if (tileHandlerHelper == null)
            return;

        // Clear the handler, so it's regenerated
        tileHandlerHelper.handler = null;
        tileHandlerHelper.Start();

        // This usually happens by assinging a callback to NavmeshClipper's OnEnable.
        foreach (var clipper in Object.FindObjectsOfType<NavmeshClipper>()) {
            tileHandlerHelper.HandleOnEnableCallback(clipper);
        }

        tileHandlerHelper.ForceUpdate();
        astarPath.FlushGraphUpdates();
    }
}
#endif

It would be pretty nice to have this available as an official feature, and in that case with a component that’s seperate from NavmeshCut.

3 Likes

Nice work @Baste!
I’ll see if I can add it officially.

Thanks! I still haven’t loaded 4.1 since asset store hasn’t approved it, but in case it’s not there, I am getting reference to private members through Reflection:

#if UNITY_EDITOR

using Pathfinding;
using UnityEngine;
using UnityEditor;
using System.Reflection;

[InitializeOnLoad]
public static class TileHandlerEditTime {
static MethodInfo _tileHandler_startMeth;
static MethodInfo _tileHandler_enableCallback_Meth;
static FieldInfo _tileHandler_handlerVar;

static TileHandlerEditTime() {
    AstarPath.OnPostScan -= OnPostScan;
    AstarPath.OnPostScan += OnPostScan;

    get_Reflected_infos();
}


private static void OnPostScan(AstarPath astarPath) {
    var tileHandlerHelper = Object.FindObjectOfType<TileHandlerHelper>();
    if (tileHandlerHelper == null)
        return;

    // Clear the handler, so it's regenerated
    _tileHandler_handlerVar.SetValue(tileHandlerHelper, null);//set ".handle" to 'null'

    _tileHandler_startMeth.Invoke(tileHandlerHelper, null);//null - no parmeters

    // This usually happens by assinging a callback to NavmeshClipper's OnEnable.
    foreach (var clipper in Object.FindObjectsOfType<NavmeshClipper>()) {
        _tileHandler_enableCallback_Meth.Invoke(tileHandlerHelper, new object[] { clipper});
    }

    tileHandlerHelper.ForceUpdate();
    astarPath.FlushGraphUpdates();
}//end()



static void get_Reflected_infos() {
    //var:
    _tileHandler_handlerVar = typeof(TileHandlerHelper).GetField("handler", 
                                                                  BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.FlattenHierarchy );
    //method:
    _tileHandler_startMeth = typeof(TileHandlerHelper).GetMethod("Start",
                                                                 BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.FlattenHierarchy );
    //method:
    _tileHandler_enableCallback_Meth = typeof(TileHandlerHelper).GetMethod("HandleOnEnableCallback",
                                                                             BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.FlattenHierarchy );
}

}
#endif

However, with this script, I must ensure the Layer is Default for the NavmeshCut component (empty gameObject with a NavMeshCut). If I set it to “MyCustomLayerForNavGeneration”, I am getting error.

AstarPath.cs is configured to have LayerMask as “MyCustomLayerForNavGeneration” and nothing else (in Recast Graph).

Instantly Returning from TileHandlerEditTime.OnPostScan() seems to avoid the error during Scanning in Editor, so it must be due to the script.


The error:
(popup that says “There was an error when generating graphs, check the console for more info”)
Stacktrace:

There was an error generating the graphs:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.NullReferenceException: Object reference not set to an instance of an object
at Pathfinding.TileHandlerHelper.HandleOnEnableCallback (Pathfinding.NavmeshClipper obj) [0x00001] in C:\MyDrive\Vigor\Assets\AstarPathfindingProject\Navmesh\TileHandlerHelper.cs:107
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:305
— End of inner exception stack trace —
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0004d] in /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:313
at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in /Users/builduser/buildslave/mono/build/mcs/class/referencesource/mscorlib/system/reflection/methodbase.cs:229
at TileHandlerEditTime.OnPostScan (AstarPath astarPath) [0x00054] in C:\MyDrive\Vigor\Assets\AstarPathfindingProject\Core\TileHandlerEditTime.cs:34
at AstarPath+c__Iterator2.MoveNext () [0x00376] in C:\MyDrive\Vigor\Assets\AstarPathfindingProject\Core\AstarPath.cs:1606
at Pathfinding.AstarPathEditor.MenuScan () [0x000f2] in C:\MyDrive\Vigor\Assets\AstarPathfindingProject\Editor\AstarPathEditor.cs:1469

If you think this is a bug, please contact me on forum.arongranberg.com (post a new thread)

Followed by:

NullReferenceException: Object reference not set to an instance of an object

Pathfinding.TileHandlerHelper.HandleOnEnableCallback (Pathfinding.NavmeshClipper obj) (at Assets/AstarPathfindingProject/Navmesh/TileHandlerHelper.cs:107)
System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:305)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
Pathfinding.AstarPathEditor.MenuScan () (at Assets/AstarPathfindingProject/Editor/AstarPathEditor.cs:1480)
Pathfinding.AstarPathEditor.OnInspectorGUI () (at Assets/AstarPathfindingProject/Editor/AstarPathEditor.cs:285)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor[] editors, System.Int32 editorIndex, System.Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1240)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)