You may want to post what your mesh looks like in Marmoset with the normal map applied as a normal map. Gradients and other changes in the map itself might stick out as looking wrong to our eyes, but its really just data that works along side the vertex-normals and tangent basis to create the end result. If there is still a problem when applied as a normal map:
- Try to rule out any issues with the UVs. Baking a gradient to them might help display any faces that could have gotten detached and flipped somehow
- If you don’t find any problem with the UVs, trying using an Object Space normal map instead.
- If that works out ok, then you might want to reconsider your stance on baking your normal map in an external program like xnormal.
The problem with sticking to zbrush for baking is that (tangent based) normal maps just contain data that is designed to work using the same data that helped create it. The renderer and the baker need to be synced (it’s sort of like how moon letters in the Hobbit can only be read by the moon of the shape and season as on the day they were written). And from what I can tell, zbrush doesn’t seem to care about vertex normals. It doesn’t use them in its materials, it doesn’t read custom normals from an OBJ, and it doesn’t write the OBJ with them included (until recently this caused problems with Marmoset, which required objs to have some normal data present). Additionally it’s tangent basis calculations are likely to be completely different (there isn’t an industry standard so different programs calculate them differently). An issue as simple as the engine triangulating your model differently could potentially be enough to throw off the result. If you were to bake in a program like xnormal then you can have your vertex normals locked in, mesh triangulated, and you can even specify the tangent bases used through plugins. I believe the newer Toolbag2 also allows you to switch the renderer’s tangent basis between Marmoset, Xnormal/Mkk, 3DsMax, and Maya).