I mean, Cassius totally does want to tap that. He recognizes that Brutus is a higher and a nobler spirit than himself, and it draws him to Brutus even as it makes him want to tear Brutus down to his level - just as his recognition of Caesar's superiority makes Cassius want to destroy him.