I can just imagine them going "You know he never asks anyone else for any of that stuff." And Foggy snarling back "Yeah, that's because he doesn't trust you assholes to respect him. Not that it seems to have made a difference."

Oooh, and make it worse - they don't realise just how sensitive Matt's hearing is, so they do this where he can hear it, and Foggy knows it, knows that Matt can hear his 'allies' casting aspersions on his character which is why his answer's so vicious. (Especially if it's one of the 'nice' Avengers taking him aside - e.g. Captain America)