I also just bought a miniMAD and am working on a project requiring masks as well.
I have yet to try it, but what if you just created a new quad, drag it over to the black edge outside of your video (on input). On output, have it overlap your video where you need it, create as many subdivisions as you need, and adjust the mask. You may be able to use Bezier curves too.
The process is described toward the end of this Youtube video: https://www.youtube.com/watch?v=UOcDM8yQZnE
Another option, which I wrote in another thread yesterday (if you have After Effects):
In After Effects' Preferences, under the Video Preview tab, select "Enable Mercury Transmit." Select your secondary display (the projector) as the video preview device. Handle the keystone with After Effects' Corner Pin effect. Handle any specific mapping with the Mesh Warp Effect. For each mask, create a new Shape Layer by navigating to the Layer Menu > New > Shape Layer. Set the Fill to black, and set the Stroke to none (click the text that reads "Stroke:" (not the rectangle next to it) and select the leftmost thumbnail in the popup window. (The term "None" will appear.) Then use the Pen tool to draw the path outlining your shape. Repeat this step for every mask. After Effects also allows Bezier curves with the Pen tool. Then re-export your video, bring it into MadMapper, and export to miniMAD as you had done before.
It's less complicated than it sounds...hope you find a solution!