@Jxckson Ridiculously expensive to acquire/keep (especially in terms of their diet!), require large filtration systems and bodies of water, and are herd animals to boot. SeaWorld Orlando and SeaWorld San Diego are the only holders in the US of walrus herds, with Point Defiance and Indianapolis only holding pairs.
In addition to what @StoppableSan mentioned, walruses also have a unique desire to swallow everything that gets into their enclosure which makes them tricker to keep healthy and necessitates that their enclosures must be netted over for their safety.