Have you checked your system properties to see how much RAM Windows is saying you have available? It does no good to modify your ini files if the changes that you make are not being recognized by the operating system. (Case in point: Windows ME refused to recognize any RAM over 512 MB. You could put in a GB if you wanted and see it recognized by your BIOS during boot, but Windows was still only going to make 512 of it available to you. I know that you’re not running ME, but this situation may be similar.)
Also, have you adjusted your Mem>Compact Mem setting? This setting tells ZBrush how much of your available memory to use. You could have 8 GB of RAM in your system, but if this setting is not changed then ZBrush will only use the default amount of 256 KB.
One last thing that I recommend doing: Rename your UICustom.cfg file to UICustom.bak. Restart ZBrush. What does it say for the MaxPolyPerMesh value? That setting will be thre recommended max based upon how much RAM Windows is making available to ZBrush. You should not set this value to more than twice the recommended amount.
Regarding the maximum RAM that ZBrush can use, the answer is 4 GB.