The column in the process tab you need to check is the "VM Size". The label is misleading and actually refers to the private bytes usage of the process. "Mem Usage" refers to physical memory usage and will be highly dynamic, depending on system load and other factors.
Process Explorer is really a better tool for this than Task Manager. It will provide more information, and without the misleading and inconsistent terminology. In Process Explorer you would look at the "Private Bytes" column.
