When the mesh is thin that smooth brush backfacemask is not working right 
and smooth directional or smooth perpendicular is not working right with symmetry too.
hi @Yune !
in your case as you have different polygroups on both side, you can set the mask by polygroup to 100 so it gonna affect only the front side.
I noticed that backfacemask on smooth brush do not work too i am on the latest 2021.6.6.
I can only recommend you to contact the support center and start a conversation that describe the issue you have with the smooth brush when you turn on BackFaceMask and smooth on thin meshes.
Hope it helps!
Nicolas
Same here, and I’m working on a thicker mesh. I don’t recall this being a problem before, it seems backface masking is completely broken with smooth brushes