In this work, we aimed to tackle the challenge of fusing information from multiple echocardiographic views, mimicking cardiologists making diagnoses with an integrative approach. We proposed intra-subject and inter-subject contrastive losses with varying margin to encode heterogeneous input views to a shared view-invariant and objective-relevant feature space, where feature fusion can be facilitated. The result demonstrated that the contrastive losses successfully improved the integration of complementary information from the input views.