There's a reason why Matt's relationships never last that long - he's too ashamed of himself to ever take it further than kissing.

(Taking a page from the Unsullied and Game of Thrones) Stick disfigures his genitals when he's a kid so women will no longer be a distraction.

Eventually, Foggy finds out what happened.

He's there for Matt in every way possible.