If you are writing a plugin then you can create a new sub-palette in any of the existing palettes except the ZScript palette (which you can’t add a plugin to at all). All you need to do is include the palette name like this:
[ISubPalette,"Texture:Image Plane"]//creates a new sub-palette in the Texture palette
[IButton, "Texture:Image Plane:Load Image","pop-up info",
//commands
,,1]//end of button
You can add plugin buttons to existing sub-palettes but they must be there when ZBrush launches (so that means the Tool palette will not work, as they only appear when a 3D tool is selected).
I’ve never tried to add a sub-palette to a custom menu using zscript. As custom menus are created as part of a custom interface configuration then there might be problems with loading order. If you want to do that I suggest you create your plugin as part of the ZPlug palette and then separately create your custom interface, moving the plugin buttons to where you want them. This is essential the method I used for the PaintStop plugin. All the buttons are in the PaintStop sub-palette (in the Document palette) but are hidden until PaintStop is launched.