Don't allow anonymous responses for Exchange/known users
It appears that non-Microsoft-account users who click the response options are allowed to impersonate anyone else on the list. They get a list of all invitees, and can just click whoever they want to pretend to be.
This is not great.
Invitees with Microsoft accounts should be required to sign into the account that was invited. Invitees without Microsoft accounts should be given a unique link that only allows them to respond for themselves.
The current implementation is far too trusting.