The hard material edge that you are referring to is caused whenever to strongly different materials come into contact with each other. Because each material handles the rendering of light in a different way, you get these artifacts.
There are two ways to overcome them. One option is to use the Draw>Channels>Mat Overwrite slider. This slider determines how strongly the new material overwrites existing materials, using the grayscale values of your current alpha. At the default value of 2, almost every shade of gray in your alpha receives the new material. As you increase the slider setting, the new material is only applied where the alpha is whitest. This can help to blend two materials together successfully.
A second approach is to create transitional materials. First, save the starting material. Next, select a material that you wonāt be using in your project (one of the many Fast Shaders is normally a good choice). Now load the material that you just saved. This overwrites the currently selected material, creating a copy of the original. You can adjust the modifiers so that it is a little more like the target material. Go ahead and paint it on the canvas in what will be the material blend zone. You can then modify the settings a little more until you start to see artifacts at the material edges, then back off and save the new material. Now repeat these steps to create another transitional material, bringing you even closer to the target material.
Regarding the matter of masks, you can paint any shape mask that you would like at any time. Simply create a new layer and turn off the other layers. Now paint what you would like, then use the MRGBZGrabber tool to capture it. You will get both a texture (based on the colors) and an alpha (based on the depth values). If the texture is more suitable for your purposes, you can convert it to an alpha before finally making it into a stencil. In fact, there is a tutorial on this in the Primers. You can easily use a Plane3D and angle it however you like to create an instant gradient. Or you can use a Terrain3D to make a gradient based on a curve.
Create a new layer