why does it seem to be so hard to keep walruses in zoos and aquariums?
I think the biggest issue is reproduction rather than simply keeping them, walruses simply don't breed sustainably in captivity at this time
Only about 35-40% or less of cows who survive to maturity will ever reproduce; 31.5% of known pregnancies are lost or otherwise unsuccessful (*this number is very likely higher considering unconfirmed and unannounced pregnancies), and then about 58% of liveborn calves die in their first year
Beyond this point, life expectancy for captive-born animals is 33.5 years, with a maximum lifespan of 43+ years, which matches estimates made for wild populations of walrus. An ongoing struggle with captive walruses is infections stemming from their tusks which has been the cause of death for a large number of individuals, hence why slightly less than half of the population is tuskless; tusk caps are often used in an effort to prevent and control damage, and in some cases habitat modifications have been attempted with pretty limited success due to walruses being very, very destructive haha.
There's currently only one single country with births outnumbering deaths, and that's Japan -- 12 of the at least 23 calves conceived over the last six years were in Japan. So across the rest of the world the population is declining, often very steeply so.
Edit: As far as why reproduction doesn't work out, I don't have an answer for that. It's been mentioned in this thread before the case of a bull and his two cows being out of sync reproductively, but that was one single case and not necessarily so with the remainder of the population.
The vast majority of the captive population seems to be extremely poorly managed (many facilities only have single animals, same-sex herds, and/or will only have one of multiple cows at the parks housed with bulls during breeding season), and of the facilities actively attempting reproduction, very few practice seasonal separation of bulls and cows + house multiple bulls to stimulate their natural competitive nature, both of which could bring an increase in mating responses.
Walruses are lekking animals, but this natural system is very rarely allowed to be expressed in captivity; the best example is the highly successful Marineland of the Pacific, which had two separate habitats both housing breeding pairs, allowing bulls stimulating auditory and olfactory access to each other.
Chimelong Ocean Kingdom, Pairi Daiza, Tierpark Hagenbeck, and to some extent Toba Aquarium seem to be the most adherent to these more natural arrangements currently. There are plenty of bulls and parks that have been successful without them (after all, at the end of the day all you have to do is put two and two together

), but I figure why not allow the benefit of an increase in mating responses each season; worst case scenario, the effect is minimal but the animals are still allowed a more natural life history in captivity and allowed to express more natural behaviors.