“On the contrary” in Chinese is 反而 (fǎn'ér).
你不但不和我一起吃饭，反而去和另一个女生吃饭！
Not only did you not eat with me, you went to have dinner with another girl!
Example sentence from Tatoeba (CC BY 2.0).