Unity 5 RC1 and il2cpp build

DziD · March 16, 2015, 4:17pm

Thanks Aron, I’ll try it right now

Here my enabled options in Otimization section http://take.ms/afws4

And exception in xCode, after that A* not work

 Caught exception while deserializing data.
System.Exception: No metadata found in serialized data.
  at AIPath+<RepeatTrySearchPath>c__Iterator0.Reset () [0x00000] in <filename unknown>:0 
  at Pathfinding.Serialization.AstarSerializer.DeserializeMeta (Pathfinding.Serialization.Zip.ZipEntry entry) [0x00000] in <filename unknown>:0 
  at Pathfinding.Serialization.AstarSerializer.OpenDeserialize (System.Byte[] bytes) [0x00000] in <filename unknown>:0 
  at Pathfinding.AstarData.DeserializeGraphs (System.Byte[] bytes) [0x00000] in <filename unknown>:0 
  at Pathfinding.AstarData.LoadFromCache () [0x00000] in <filename unknown>:0 
  at Pathfinding.AstarData.Awake () [0x00000] in <filename unknown>:0 
  at AstarPath.Initialize () [0x00000] in <filename unknown>:0 
  at AstarPath.Awake () [0x00000] in <filename unknown>:0 
  at uScript_Lerper..ctor () [0x00000] in <filename unknown>:0 
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:LogWarning(Object)
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()

DziD · March 16, 2015, 4:55pm

So! I insert Debug.Log in SimpleZipReplacement.cs http://take.ms/y1R9f

And next you can see logs from xCode:

*

Unloading 7 Unused Serialized files (Serialized files now loaded: 0)
UnloadTime: 5.066791 ms
name graph0.binary length 229 bytes 229
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
name graph0_nodes.binary length 0 bytes 0
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
name graph0_conns.binary length 0 bytes 0
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
name graph_references.binary length 20 bytes 20
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
name  length 141 bytes 141
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
name  length 0 bytes 0
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
name  length 218 bytes 218
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
Caught exception while deserializing data.
System.Exception: No metadata found in serialized data.
  at AIPath+<RepeatTrySearchPath>c__Iterator0.Reset () [0x00000] in <filename unknown>:0 
  at Pathfinding.Serialization.AstarSerializer.DeserializeMeta (Pathfinding.Serialization.Zip.ZipEntry entry) [0x00000] in <filename unknown>:0 
  at Pathfinding.Serialization.AstarSerializer.OpenDeserialize (System.Byte[] bytes) [0x00000] in <filename unknown>:0 
  at Pathfinding.AstarData.DeserializeGraphs (System.Byte[] bytes) [0x00000] in <filename unknown>:0 
  at Pathfinding.AstarData.LoadFromCache () [0x00000] in <filename unknown>:0 
  at Pathfinding.AstarData.Awake () [0x00000] in <filename unknown>:0 
  at AstarPath.Initialize () [0x00000] in <filename unknown>:0 
  at AstarPath.Awake () [0x00000] in <filename unknown>:0 
  at uScript_Lerper..ctor () [0x00000] in <filename unknown>:0 
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:LogWarning(Object)
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()

As you can see in logs, not any “meta” word, but my exception connect with “meta” ! Also you can see in logs, sometime name is empty.

Any ideas?

aron_granberg · March 16, 2015, 5:34pm

There are a bunch of sections which seem to be deserialized to have empty names. That’s definitely something wrong. No section should have an empty name. Do you think you could save this file (Save & Load tab) and send it to me and I will try to figure out what it is doing.

DziD · March 16, 2015, 5:38pm

Aron here that file from Editor: http://take.ms/QyzzK
(I zipped it)

P.S: It fine works in editor, but not work on iOS device with il2cpp (I test only in Unity5)

aron_granberg · March 16, 2015, 9:02pm

Ok, I verified the file in a hex editor. And it is definitely valid. So IL2CPP is interpreting something incorrectly.
More specifically, when the ReadInt32 method is called, it seems to use only the first byte instead of all 4 bytes. This works for the first 3 sections because their lengths are 229, 0 and 0 respectively (all < 255), but the fourth section has a length of 144404 bytes which does not fit into the first byte.
I verified that the first byte of the size field of the fourth section (graph_references.binary) is indeed equal to 20 when interpreted as an unsigned byte, but if you read all 4 bytes for the field it gives 144404 which is the correct value.

Do you think it would be possible for you to send a bug report to Unity with this info (maybe link to this thread)?
I have also attached an image which shows the position where IL2CPP seems to interpret the data incorrectly.

DziD · March 17, 2015, 8:05am

Hello Aron, after some research I’am write a little extension (for parse int32 from binary stream)
here it is:

public static class BinaryReaderExtension
{
	public static bool IsEndOfStream(this System.IO.BinaryReader reader)
	{
		return reader.BaseStream.Position == reader.BaseStream.Length;
	}

	public static int ReadInt32(this System.IO.BinaryReader reader)
	{
		int result = 0;

		if (reader.IsEndOfStream())
			return result;

		byte[] intBytes = null;

		intBytes = reader.ReadBytes(4);
		if(intBytes.Length > 0)			
			result = System.BitConverter.ToInt32(intBytes, 0);

		return result;

	}
}

And change small code: http://take.ms/dNaEV
P.S.: Now I see correct values

After it I don’t see exception which I saw yesterday, but current my broblem in that:

"Node data did not match with bounds data. Probably a change to the bounds/width/depth data was made after scanning the graph just prior to saving it. Nodes will be discarded"

Here my new log from xCode:

    Unloading 7 Unused Serialized files (Serialized files now loaded: 0)
UnloadTime: 3.962083 ms
7
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name graph0.binary length 229 bytes 229
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name graph0_nodes.binary length 0 bytes 0
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name graph0_conns.binary length 0 bytes 0
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name graph_references.binary length 90004 bytes 90004
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name graph0_extra.binary length 495004 bytes 495004
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name graph0_references.binary length 0 bytes 0
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

name meta.binary length 95 bytes 95
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.Serialization.Zip.ZipFile:Read(Stream)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

Trying to load data from a newer version of the A* Pathfinding Project
Current version: 3.6 Data version: 3.6.255.255
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:LogWarning(Object)
Pathfinding.Serialization.AstarSerializer:OpenDeserialize(Byte[])
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

PostDeserialization228 width 150 depth 150
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:Log(Object)
Pathfinding.GridGraph:PostDeserialization()
VortexEffect:OnRenderImage(RenderTexture, RenderTexture)
Pathfinding.Serialization.AstarSerializer:PostDeserialization()
Pathfinding.AstarData:DeserializeGraphsPart(AstarSerializer)
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

Node data did not match with bounds data. Probably a change to the bounds/width/depth data was made after scanning the graph just prior to saving it. Nodes will be discarded
uScript_Lerper:.ctor()
UnityEngine.Debug:Internal_Log(Int32, String, Object)
UnityEngine.Debug:LogWarning(Object)
Pathfinding.GridGraph:PostDeserialization()
VortexEffect:OnRenderImage(RenderTexture, RenderTexture)
Pathfinding.Serialization.AstarSerializer:PostDeserialization()
Pathfinding.AstarData:DeserializeGraphsPart(AstarSerializer)
Pathfinding.AstarData:DeserializeGraphs(Byte[])
Pathfinding.AstarData:LoadFromCache()
Pathfinding.AstarData:Awake()
AstarPath:Initialize()
AstarPath:Awake()
uScript_Lerper:.ctor()
 
(Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)

System memory in use before: 28.5 MB.
System memory in use after: 29.0 MB.

Unloading 23 unused Assets to reduce memory usage. Loaded Objects now: 17861.
Total: 22.612499 ms (FindLiveObjects: 5.821666 ms CreateObjectMapping: 0.844708 ms MarkObjects: 13.439040 ms  DeleteObjects: 2.500083 ms)

After 20 minutes I found a problem with wrong nodes count: http://take.ms/cL8Zf

It main il2cpp bug wrong ReadInt32 method implementation (instead 4 bytes it read only one!)

DziD · March 17, 2015, 11:32am

Soo, after a few hours BinaryReaderExtension looks this:

public static class BinaryReaderExtension

{
public static bool IsEndOfStream(this System.IO.BinaryReader reader)
{
return reader.BaseStream.Position == reader.BaseStream.Length;
}

public static int ReadInt32(this System.IO.BinaryReader reader)
{
	int result = 0;

	if (reader.IsEndOfStream())
		return result;

	byte[] intBytes = null;

	intBytes = reader.ReadBytes(4);
	if (intBytes.Length > 0)
		result = System.BitConverter.ToInt32(intBytes, 0);

	return result;

}

public static uint ReadUInt32(this System.IO.BinaryReader reader)
{
	uint result = 0;

	if (reader.IsEndOfStream())
		return result;

	byte[] intBytes = null;

	intBytes = reader.ReadBytes(4);
	if (intBytes.Length > 0)
		result = System.BitConverter.ToUInt32(intBytes, 0);

	return result;

}

public static ushort ReadUInt16(this System.IO.BinaryReader reader)
{
	ushort result = 0;

	if (reader.IsEndOfStream())
		return result;

	byte[] intBytes = null;

	intBytes = reader.ReadBytes(2);
	if (intBytes.Length > 0)
		result = System.BitConverter.ToUInt16(intBytes, 0);

	return result;
}

}

In A* PathFinding I replace any ReadUInt32 and ReadInt32 and ReadUint16 to appropriate method from BinaryReaderExtension class, and now it work (But I don’t know how correct or not, but I don’t see any exceptions in xCode). Also I think, while Unity not fix this bug, need write a wrapper for parsing BinaryReader stream like a BinaryReaderExtension.

Bug Description (Unity5.0.0f4 iOS 64Bit il2cpp): ReadInt32 and other bugged methods get read only ONE BYTE instead 4 byte for 32 bit methods, and 2 bit for 16 bit methods, from input sbyte stream (Aron correct me please, my English is very Bad for describe bug reason)

BuggedMethods (methods that return incorrect value): ReadInt32, ReadUInt32, ReadInt16, ReadUInt16

P.S.: methods ReadSingle work good (just in case I would have to override it in BinaryReaderExtension too)

Today I write bugReport to Unity (if Aron not create bugreport yet)!

Hope this helps to anyone

DziD · April 6, 2015, 8:44am

In the 5.0.1 this bug Resolved!

aron_granberg · April 6, 2015, 12:22pm

Yep, I also just uploaded version 3.6.1 (beta) which has some additional fixes for WebGL.