I worked it out-I was using cached credentials to connect to the terminal server.
I had earlier RDPed to the server as the administrator, and it saved these credentials. So each time I opened the remote app, it was using the admin credentials
My GPO isn't applied to the admin account, so the C: drive was visible.
To clear the cached credentials, I went to RDP. There's an option to 'delete' saved credentials here.