To create a new polygroup of any masked area, press CTRL + W. Any masked area will be turned into a new polygroup.
Take the following masked object, for example.
Pressing SHIFT + F puts us in polyframe mode and we can see only one color (one polygroup) with the darker masked area still represented.
After pressing CTRL + W, we can see that what was formerly the masked area has now become a new polygroup. This is evidenced by the fact that the new polygroup is now its own new color in the polyframe mode.
Note: Pressing CTRL + W is the same as using Tool > Polygroups > Group Masked Clear Mask
There is another technique I’ve witnessed, which is to mask an area and then select Geometry > EdgeLoop > Edgeloop Masked Border, or press CTRL + SHIFT + E. I think this method attempts to create an edge around the mask perimeter based on the EdgeLoop setting.