Hmmm, not sure about that, part of the point here is that Matt's pretending to be a helpless blind man remember? Maybe they start thinking that Frank's got eyes in the back of his head, Matt tells him who's leering when he lets him know that he needs to stop to get info.