there are very few mammals that cannot be kept in captivity. Off the top of my head you've only got the large ceataceans (due to size and feeding requirements - although even grey whale has been kept temporarily); indri (unknown reasons, perhaps dietary - even Madagascar's Tsimbizaza Zoo has failed); colugos (have never survived in captivity); most pangolins (limited success at a couple of facilities). Three-toed sloths are always cited as being impossible to keep in captivity, but there are some zoos keeping them.
There are mammals that have proved difficult or impossible in the past (the mountain gorilla being the most-quoted species) but it is quite likely that most of those "impossible" species could be kept quite successfully with modern knowledge. Even species like insectivorous bats or numbats aren't impossible at all, simply needing dedication to keep adequately fed.
In terms of what mammals aren't kept at zoos, there are literally thousands of species. I mean, there are over 1200 species of bats in the world, and only a tiny number are kept in zoos. Likewise for rodents, with well over 2200 species.
There are mammals that have proved difficult or impossible in the past (the mountain gorilla being the most-quoted species) but it is quite likely that most of those "impossible" species could be kept quite successfully with modern knowledge. Even species like insectivorous bats or numbats aren't impossible at all, simply needing dedication to keep adequately fed.
In terms of what mammals aren't kept at zoos, there are literally thousands of species. I mean, there are over 1200 species of bats in the world, and only a tiny number are kept in zoos. Likewise for rodents, with well over 2200 species.