I'm not sure you can remove that. I found a couple of posts, and was even able to implement one Ribbon scheme (as you apparently did) which removed all items from the Office button ... except the Exit Access button.
As far as I know, you cannot hide the Office button. This seems to be well documented in various locations, from various MS-type sources.
If you can hide the Exit Access button, I've never seen any way to do it.
It could be possible to change the Text, but as Access handles these things differently from every other Windows programs, I'd not know where to begin.
I've posted a message in the private Access MVP newsgroups. This is monitored by the Access team contacts at Microsoft as well, so perhaps we'll get some word from the horse's mouth (so to speak).