If you are sending a meeting request, then I am afraid that there is no way you can limit the number of responses. All of them who get the meeting will have an option to accept/decline. I guess you will have to mention in the body of the meeting and sort the responses in the inbox to find who all responded.