1.First correct your uv layout, looks like you have some uvs outside 0-1 and overlapping. Also the actual layout looks totally wrong, spend more time optimizing the space and the uv to poly ratio.
2.I would then suggest totally checking your mesh twice for any nasty geo.
3.Now export that clean mesh out of maya, then bring it back in and re-apply your shader and create a new approx node. (This is a maya related step, I have personally had meshes not render properly simply because there was a problem with the nodes in maya, exporting out then in again will solve this.)
- Apply the disp as a bump, does it work? (of course re-create with better uvs first).
Use a higher mesh to render with
here is a list of values i use for the min and max of the approx node
2,3
3,5
4,6
make sure you are using a subdivision apporx not disp.
Change the approx to spatial at least, parametric is too simple.
Make sure you understand that undercuts will not render out.
Use 32bit, but do not edit it, if you use 16bit increase the texture size.
The artifacting (spikes) in your render is from the bitdepth banding and values of the map getting crunched from too high settings in your file node.
Some things to consider
When using Adaptive mode to create displacement or normal maps in 3.5, please first set the Tool : SubTool : PA Blur value to 8. This setting works in tandem with Adaptive mode to create the best quality maps.
When creating a 32-bit displacement map, please first assign any alpha to the Tool : Displacement Map thumbnail to enable the Mid slider. Once enabled, set this slider to 0. This will ensure that your 32-bit displacement map is created with the correct grayscale values.
When creating a displacement map for a model that has hard edges (such as a cube), do the following after sculpting your details: Go to level 1, restore your base mesh as usual but then Shift+Click on the Tool : Geometry : Crease button. Now create your displacement map. This will ensure that the hard edges are calculated correctly without any gradiation in the displacement map.