hey all!
Is this mesh unwrapped, and you apply texture from which you make masking ?
If so, maybe try extracting mask on higher sub-d level.
But i think you neeed to work on the topology ?! Add some more detail to the base mesh before importing to ZB.
Actually i never had such a problem with extracting masking from texture - if your model is uwrapped correctly.