To me - and yes, I get that we're reading this scene differently - Matt is more uncomfortable with the fact that she's right than he is about her touching his face.

Yeah, this. Watching that scene I don't get the impression that he was uncomfortable with her touching his face. He was uncomfortable with the conversation.