Yeah, I think he keeps calling Inara a whore as an unconscious way to remind himself that he shouldn't be into her, because even if the rest of the 'verse thinks Companions are totally respectable, Mal clearly doesn't agree.

And Inara is torn between "He's so cute!" and "But it's so confusing whether he really likes me" and "Do I care if he really likes me? He may be cute, but he's also a jerk!"

I think the makers of Castle borrowed some of Mal's more portable characteristics when they were designing Castle's character: probably they realized they were going to get an influx of Firefly fans.