Dynamesh subtraction that results in edges that protrude at less than a 90 degree angle will frequently result in excessively sharp edges with artifacts. Try to avoid this where possible.
The most reliable way is to model that extrusion in with the Zmodeler tool, if you have a sphere with precise enough low poly topology to allow it.
If you’re working with a high poly object, or one with unsuitable topology, try slicing the shape to be extruded instead, and use transpose-extrusion. See the quick and dirty example I’ve made below:
Slice the square shape in with the Slice Rect brush or slice curve brush (I actually recommend the latter, but I’ve used the former for speed). Shift-click on the resulting square polygroup to hide all but. If you sliced without hiding the backfacing side of the sphere first, you’ll have another matching square on the rear side. Either hide that or polygroup it separately. Once your target square polygroup is the only thing on the screen, ctrl-click the canvas to mask it. Shift-ctrl click the empty canvas to unhide everything. Ctrl+click empty canvas space again to invert the mask. Now you’re ready to work.
In Transpose Scale mode, center theaction lineon the target area, and ctrl-drag on the end circle to extrude the shape inward a bit. Now, In Transpose-Move mode, make sure the action line is perpendicular to the target area, and Shift-drag on the middle circle to move the target area inward just a bit while constrained to the action line direction. This will result in a slight bevel to the edge of the extrusion that will blunt those impossibly sharp edges, and make it easier to dynamesh. Now, still in Tranpose-Move mode, ctrl-shift-drag on the middle circle to extrude the target group inward, again constrained to the action line axis. Finally Ctrl-Drag on the forward most circle to flatten the target polygroup (you may need to move the action line forward a bit so the circle is located on the plane you wish to flatten.
And there you go. A much cleaner result. You’ll notice there are a few “grooves” in the side of the extrusion. That was because I was hurrying, and didnt clean up any extraneous polys on the edge of my target polygroup before extruding. You’ll have to clean those up if they bother you, or use the slice curve brush to slice one side of the square in at a time, because it results in cleaner cuts than the marquee slice tools.
