Tangent VDM in Mental Ray for Max (or Maya) - Getting the settings right

[Trist]

Hey ZBC,

I thought it might be a good idea to open a thread regarding the correct settings for ZB Tangent Vector Displacement in Mental Ray for 3DS Max and Maya (or any other app that uses this engine). I’m assuming by this stage that eveyone has consulted the diagnostic chart render for the right “flip and switch” values (#11 in my case).

While I had initial success with Tangent Vector Displacement in Mental Ray I am having trouble recreating an accurate effect.

If anyone can share their knowledge there are a few things I would like to clear up.

-Beginning in ZBrush, what kind of results have people had with using automatic UV’s such as AUV tiles as opposed to the maps generated in UV Master or from other 3D apps? So far I know that they do “work” but the rendered effect is not ideal.

-If I understand correctly, not only do VDM’s give a more accurate result (undercuts etc) but you also no longer need a very high res map to get great detail. Has anyone noticed a big difference between a render from a 1k map and an 8k map?

-The next step, setting up your VDM in Max or Maya. The first thing I did when I got my hands on Z4R3 was test the MR “SSS Fast Skin + Displacement” shader in Max. And it does work pretty well, especially when you set it up properly with multiple maps for the skin layers. I have also tried setting up the VDM in a standard shader’s displacement slot so I know it works there too.

-The details of how you configure the map here is important. Firstly, how different practically are VDM’s from regular height maps? Previously we had to set the RGB offset to -0.5 and the Blur to 0.001. Is this still necessary? If you are using 32bit displacement maps the mid-value is irrelevant so this should still be the case here (leave it at 1).

-The next step is configuring Mental Ray. Is it better to have each object with it’s own displacement settings or to use global settings? I suppose it would depend largely on the project but I thought I would mention it anyway.

-Down to the actual settings. I have found that like height maps, VDM’s should have the “edge length” set to no more than 1 for production to get an accurate effect. In Max, “Max. Displace” defaults to 20 but I have had errors at render time that my displacement was being “clipped” so I suppose this value is relative to your map. Also, there is the “Smoothing” option. This may be relative to whether you have set your VDM to SUV back in ZBrush, but i’m not sure. I suppose the “Max. Subdiv.”
setting in MR serves pretty much the same function as with a height map.

So without making this post unbearably long, I just wanted to gather some information from 3D artists and technicians on the subject. Vector displacement is now the standard for reproducing your brilliant work from ZBrush in other 3D apps for animation and rendering and we really should know how to do it right.

If you have any tips, tricks or insight into successfully fitting ZBrush VDM’s into your workflow please speak up and share your knowledge.

Thanks in advance for any contribution to this thread. I know that many in the community will appreciate it!

[Zeddicus]

I’ve had some success with MR “SSS Fast Skin + Displacement” shader in Max too, but nothing great as far as quality goes. These settings are weird and not really applicable to other users, but here they are anyways for the sake of discussion:

Exported 16-bit World Map from ZBrush
Flip & Switch = 1
Tangent Flip & Switch = 1
In 3ds Max + MR, VectorDisplacement strength = -1
Green & Blue channels didn’t need to be swapped.

Exported 16-bit Tangent Map from ZBrush
Flip & Switch = 1
Tangent Flip & Switch = 7
In 3ds Max + MR, VectorDisplacement strength = -4.2
Green & Blue channels needed to be swapped.

The world map rendered completely accurate and the tangent one less so. Neither rendered overhangs correctly though and I don’t know which application is the culprit. Until I figure that out, VDM is useless since overlapping geometry is it’s primary benefit. Another reason I need to do more experiments is because tangent is more preferable, especially for those who animate/deform. Can’t do that with world maps which limits it to static renders only.

Other stuff I’ve been tracking around the net:

Vray for 3ds Max can’t render VDM’s at all, only bump & tangent normal (using bump material slot of VRayMtl), and regular dispalcement using grayscale bitmaps (via the VrayDisplacementMod). Vlado is looking into it.

C4D R13 handles world maps just fine, but not tangent maps (yet). Users are still working on figuring it out on the CGSociety forum.

If my memory is accurate, Maya handles world maps pretty well when using Mental Ray. Not sure about tangent maps or Vray for Maya though.

Modo 601 handles vector displacement as well, though not sure which type was used in the example I saw.

[Trist]

Thanks for the your input Zeddicus,

I have also noticed that world vdm’s work better than tangent. This is a let down for me too, I have been seeking a better way to get my detail into Max for character animation etc.

In my experience so far, the most obvious fail of ZBrush vdm’s in Max is the black artifacts on the surface of the mesh, like tiny pixel sized holes. I have found this can be remedied for the most part by using a texture map - then it becomes less obvious.

The problem may just be Max specific because I seem to recall a while back a Max user complaining of the same black artifacts using a vdm from Mudbox. But, as far as I know vdm’s are meant to be virtually artifact free in effect. And then there is the claim that they can produce undercuts etc due to the height and direction values in each map. Sounds brilliant in theory, right?

In practice, well I can say that I could get decent results but that was with a 32bit 8k vdm smoothed to 4 iterations in Max and the render time in mental ray was just too slow to do any extensive testing. It’s possible to animate at the base mesh level and render out with detail, but it’s certainly not ideal.

I just can’t seem to be able to nail down the right settings and process, it’s like i’m getting varied results every time I try to get it to work.

Anyone else have some experience with ZBrush vdm’s? Are you experiencing any render artifacts similar to what I have described above? It would be helpful even if your render engine is not mental ray, I just want to figure this out if I can.

Thanks again!

[Zeddicus]

In my experience so far, the most obvious fail of ZBrush vdm’s in Max is the black artifacts on the surface of the mesh, like tiny pixel sized holes. I have found this can be remedied for the most part by using a texture map - then it becomes less obvious.

I never encountered this in any of my tests with 3ds Max 2012 SP2 x64. Do you have a link showing these artifacts? Mental Ray has never been very good at rendering displacement so I’m not all too surprised TBH. Until Vlado adds support for VDM’s, I can’t do any comparisons either.

In practice, well I can say that I could get decent results but that was with a 32bit 8k vdm smoothed to 4 iterations in Max and the render time in mental ray was just too slow to do any extensive testing. It’s possible to animate at the base mesh level and render out with detail, but it’s certainly not ideal.

The same was true for grayscale displacement maps back in the day. Like I said, MR isn’t very impressive in this area and when put up against VRay there was a noticeable difference in both quality and time it takes to render. I’m hopeful Vlado will be able to do the same for VDM’s. The video on Youtube by Wayne Robson shows what can be accomplished so far as subdivisions are concerned. It’s all up to the developers to get everything working together both smoothly and efficiently. Could be waiting a long time for that lol.

I just can’t seem to be able to nail down the right settings and process, it’s like i’m getting varied results every time I try to get it to work.

I wasn’t able to either, though I did come close overhangs and all. In the end I exported a 16-bit tangent map for nearly every single flip and switch setting (gave up about half way through, got tired of swapping the green and blue channels). A setting of 7 was still the best, and the object setting instead of a tangent one in the vector displacement material seemed to work best (if one required a negative value, the other would need a positive one). It seems anything close to the center axis (where the brush comes together into one when symmetry is turned on) is what doesn’t want to accurately render. Anything further away from that center axis renders relatively fine, albeit not with as much detail overall as can be seen in ZBrush.

[Trist]

Hi again,

Here is an image demonstrating the artifacts I am getting as described above. The irony here is the fact that I cannot re-create the exact same error, lol.

FYI: This is a tangent vdm, I used sd level 2 as the base and one iteration of turbosmooth in Max. The MR SSS + Displacement shader is used but I experienced the same sort of artifacts with the standard material.

From your description Zeddicus, VRay looks pretty good but I don’t think I will jump ship just yet as Max 2013 will have the latest Mental Ray as standard. Fingers crossed we will find improvements with this coming release.

[Trist]

You will find the subtle black artifacts on the top arch of the foot.

[Zeddicus]

You will find the subtle black artifacts on the top arch of the foot.

I honestly don’t know what would be causing that. The only thing I can think to ask is whether everything is properly welded, UV’s included. I saw something similar recently by a 3DCoat/Max user and it ended up being overlapping UV’s. Looked fine in the former, but not in the latter when rendered.

I think I may have figured out where I was going wrong. I forgot I had been playing with my gamma/lut settings and may have been loading my vector displacement maps with a gamma of 2.2, silly me. I got decent results when I retested today.

In ZBrush 4R3:

1. Imported a custom sphere (384 quads, no poles).
2. Subdivided to level 7 and sculpted, including overhangs.
3. Reimported custom sphere OBJ to level 1.
4. Set Tangent FlipAndSwitch to 7.
5. vd Tangent set to on.
6. vd 32Bit set to off.
7. vd SNormals set to on.
8. vd SUV set to on.
9. Exported VDMap (said no to exporting mesh).

I took my exported map and swapped the green and blue channels using Photoshop. Make sure you load/save it without applying any color profiles. You might need to flip the map vertically too, depending on whether you told ZBrush to do it or not in the preferences (the NormalMapFlipVert button). Using 32 bit might be a problem because Photoshop won’t let me load an EXR without applying a color profile to it. I still need to do more testing, but 16-bit worked fine sooooo…

In 3ds Max 2012:

1. Imported my custom sphere.
2. Applied SSS Fast Skin+Displacement.
3. Clicked it’s 3D Displacement button.
4. Chose Vector Displacement for it’s Extrusion Map.
5. Pointed Vector: to bitmap, loaded the VDM with gamma 1.0
6. Set it’s value to +2.5 (which ZB said I should use).
7. Left High Dynamic Range on (off is inaccurate).
8. Changed it from Tangent to Object mode.
9. Rendered using Mental Ray, all other settings untouched.

It rendered 99% perfect. It’s still a tiny bit soft compared to how my sculpted mesh looks in ZBrush, plus there are tiny jaggies here and there that are barely visible (edit: changing the render option Edge Length from 2.0 to 1.0 fixed this) but otherwise it looks ok. Letting Max load the map with a gamma of 2.2 was what messed things up before. As for subdividing, adding a turbosmooth wasn’t needed with this mesh and I didn’t see much of a difference when I added one, other than quickly increasing render times with each iteration. It rendered fine (and super fast) without the turbosmooth. Hoping VRay will be able to do an even quicker/better job.

[Trist]

I honestly don’t know what would be causing that. The only thing I can think to ask is whether everything is properly welded, UV’s included.

I’m pretty sure that’s not the issue but thanks for the suggestion. I’ll have a better look later (could be a problem in the mouth cavity or something on my model).

Unfortunately I have not made much progress at all with 16bit vdm’s in Max. Once again I have consulted the diagnostic chart and narrowed the flip/switch down to a value of 9 or 25. In this preliminary render these examples both looked promising. Applying these settings to my model not so good.

I have tried swapping the green and blue channels in PS (using the channel mixer) and I’m sure my gamma has not been changed from 1.0 at any time in the process to Max.

In my recent testing I have otherwise reproduced your settings. Originally I had been just replacing the “3D Displacement” with the “Vector Displacement” shader as I didn’t want to complicate the network but it turns out this doesn’t make a difference either way.

The displace value I was given by ZBrush (40 on this occasion) just doesn’t seem right to me, it produced a bloated result.

Tried using “Object” type instead of “Tangent” but this produced no accurate effect.

So at the moment I’m stuck. Unless I can get 16bit vdm’s to work as well as you have described I must go back to 32bit. Either way, I have not been able to produce any real overhang effect at all.

It seems ZBrush just makes no sense from a technical point of view. This makes it very hard to figure it all out by process of elimination as the results of my testing are not consistent with the settings from one moment to the next.

Would be really nice if someone from Pixo could step in and sort this out for us. Anyone?

[Zeddicus]

I threw together a slightly more complex test and you’re right, there is still something wrong. The correct gamma value did fix the problem I was having in my previous test though fwiw.

This is the closest I can get using a 16-bit tangent:

And this is the closest I can get using a 32-bit tangent:

ZBrush renders are the ones on the right side. It could just be that I still don’t have the right flip & switch values right, but my instinct is telling me the problem lays with ZBrush and it’s odd VDM implementation. The only things I know for absolute certain right now is that the green (y) and blue (Z) channels need to be swapped in Photoshop and the NormalMapFlipVert button in ZBrush needs to be on prior to exporting.

Edit: I was surfing the web this morning and came across this:

Well as it turned out the trial of ZBrush doesn’t support vector displacement, so I didn’t have any way to check this out. We are talking to some guys at Pixologic to get a version for testing.

Best regards,
Vlado

Good to see he is still on it.

[Trist]

I’ve made a bit of a discovery…

So we know by now that passing the base mesh as it is with it’s Zbrush VDM to Mental Ray (in Max for example) will not yield good results because the polygons just aren’t being subdivided enough at render time. To get a more accurate effect you must smooth the mesh before hand, and this has been the case also with regular height maps in the past. We also know that this process does not necessarily result in a perfect displacement with MR.

Just now I was testing my character and his tangent VDM in Max 2013 to see if improvements had been made to vector displacement with the new version of MR. Unfortunately nothing new here as far as I can tell. What I did realize however is that applying a TurboSmooth modifier to the base mesh (as you do), actually distorts the map as it is applied to the geometry.

While it may not be that obvious on a larger scale, this somewhat necessary step is messing up the way that the geometry is being displaced at render time. I would guess that because of the more complex way that vector displacement is calculated this could result in the information from the map not translating to a correct deformation of the surface.

This is just speculation but it could be a factor in why we have not been able to get accurate under-cuts in our renders…

Here is a screenshot in Max 2013 of a base mesh set to show it’s VDM in the viewport:

And here is the same mesh with turbosmooth on:

Notice the detail on the ear and how it is distorted when the turbosmooth is applied.

This could be a real problem. Looks like with or without the smooth applied, there is no simple way to achieve an accurate effect. I suppose you could apply the VDM to a higher sub-division of your mesh but that’s not so practical for rigging and animation.

Can anyone think of a work-around for this problem?

Thanks, and sorry for rambling on but I feel like I’m making progress so I thought I should share.

Please don’t hesitate to contribute your experiences on the matter. Maybe I’m missing something here - I will continue my testing and hopefully come to a solution.

Cheers.

[hotknife]

Hi the problem with applying Turbosmooth is that it ‘distorts’ the uv’s of the mesh in a way that’s not consistent to other programs.

http://www.neilblevins.com/cg_tools/cg_wishlist/3dsmax/wishlist_linear_uvs_in_turbosmooth.htm

This I believe alters that.
http://www.mariussilaghi.com/products/turbosmooth-pro***8203;

[hotknife]

Max base mesh, Max render and the model in ZB