Login to the server using domain admin account on the server,where you want to give user access for remote login.
Add the domain user to the remote desktop user group on that server only.
It will work.
The domain admin has access to give any user right to any server & if domain user can't login to the server & define the server which user want to access then check permission.