OK, now I understand your problem with the masking brush. You need to break it down in sections and not try to make an enclosed shape. I’m including a picture here showing stages of building the key part you are trying to mask. Note the round portions are made by switching to the round masking brush.
-
Mask two triangles separately,side by side. Drag down to the lower right, click alt twice for a sharp corner and go up to make a “v” shape
-
Drag around the shapes to fill in
-
Switch to the circle mask brush to make the little circles
-
use the straight clipping brush with the ALT key to trim away the unwanted parts.
You can use more then one shadowbox (you might already know this), as subtools, and work in parts. You might be better off with a cylinder for the barrel of the key in any event.
I hope this works for you. Again, I’m also a beginner and struggling along with ZB4, but I enjoy a challenge and I learned a lot myself from this. And please excuse the sloppiness of the work here, I did it fairly quickly, but with a little effort you should get a very nice shape 