I just got booted from a multiplayer game for pressing the small ''x" beside a players name, when they invited me. I did not hit the players name to get to the accept/decline window, I just used the x in the top right when the invite showed up. Next time I will try the actual decline button and let you know what happens.
I seem to remember reading an earlier post about this same topic... but I can't find it now.
AvatarLee