I think you are exaggerating the sensitivity of red pandas. A species with the physiology to live on bamboo has not got much nervous energy to waste. They have very few natural predators and they don't even interact very vigorously with each other. Lemurs, even bamboo lemurs, are much more highly strung.
Have you seen the red panda enclosure at Blackpool? It's quite small and the nestboxes are right up against the public viewing area, but the pandas don't seem to be stressed. So long as they have enough climbing opportunities - preferably getting well above the children, I'm sure they will be fine.
Alan