So in the comics Frank and Matt have an antagonistic friendship. Matt understands why Frank does what he does but holds out continuous hope that Frank will change and heal. Frank has no intention of doing so but he respects Matt a great deal, worries about him when he's in trouble, and frequently hits Matt up for help with his projects. They argue a great deal about Frank's methods but never truly give up on winning the other one over. Other heroes, especially the Avengers, think Matt is a bit on the crazy side for keeping in contact and frequently depending on Frank Castle when they dismiss Frank as a mentally disturbed murderer. It's put a crimp in Matt's relationships with non-Frank people a fair number of times but Matt being Matt, he's happy to keep on keeping on and well, he was never much of a team player to begin with.

Now knowing what you know now, someone please take this quote: "His expression was serious, but I liked how it softened slightly when he looked at me. Like the way he looked at me was different from how he looked at everyone else." and run with it.

Maybe Frank and Matt were childhood friends. Maybe Matt simply understands the jagged edges of Frank's loss in a way no one else ever has. Maybe Frank finds it comforting to know that there's someone out there that isn't dismissing him as a crazy person even if he doesn't like what Frank does.

Regardless, it should be noted and remarked upon by bewildered heroes that the relationship between Matt and Frank has an undercurrent of mutual respect and affection, and that it effects how Frank treats Matt, even when they disagree.