Yes, there have been many. Taronga once mixed Chimpanzees with Black-backed Jackals, which as you would imagine did not end well. They then thought it was a good idea to mix chimps with Patas Monkeys, but thankfully this did not last long.
More recently, Taronga had to remove the White-lipped Tree Frogs from their Green Tree Python exhibit, as they kept trying to swallow the python. They also had to remove Northern Snake-necked Turtles from the Freshwater Crocodile exhibit due to aggression from the crocs.
Bronx Zoo tried mixing Ebony Langurs with Asian Small-clawed Otters. The langurs kept annoying the otters until the otters finally ganged up on the langurs and drowned one.
Burgers Zoo tried to have hornbills free-roaming in their tropical hall, only to find that they ate every amphibian, reptile, small bird and small mammal they could find.
Squirrel monkeys and macaws were a failed mix at both Adelaide and Melbourne, as the macaws would bully the monkeys.
I can't remember which zoo it was, but there was a failed attempt at keeping lions with meerkats.
I am sure other people have a lot more information.
