Server outage

[Simon Hampel]

Apologies for the extended outage - we've had some ongoing networking and performance issues which escalated last night to the point where I was unable to recover.

I've moved ZooChat to a new server, so hopefully things will improve now!

There may be some quirks and things not quite working as expected yet - so please do let me know if you come across anything which is not quite right.

I will need a few days to properly tune the database server, so there may be some minor performance issues in the short term.

Also, the search engine is not yet functional - so site search will be offline for a while yet.

On a more positive note - the new server is located on the east coast of the US (previously in Singapore), so this should (once things are fully functional again) see improvements for people located in the US and Europe.

 

[jwer]

Thanks for the hard work!

Quite a shock when Zoochat is down, what do I do with my spare time? I almost spent time with my family last night....

Quite a few people on this site seem not quite right, but I don’t think you can fix that.

 

[Hipporex]

First of all thank you for restoring things! Second I don't know if this is just me but whenever I try to make/delete a post or try searching in the search bar it takes significantly longer than it used to or doesn't work at all and I get the Error page again. I'm not trying to make a big deal or anything just thought I should share.

 

[Hipporex]

First of all thank you for restoring things! Second I don't know if this is just me but whenever I try to make/delete a post or try searching in the search bar it takes significantly longer than it used to or doesn't work at all and I get the error page again. Not trying to make a big deal about it but just thought I should share.

 

[Simon Hampel]

Yeah, still investigating the "write" performance (writing data to the database when creating posts) - not sure what is causing that.

The search issue is known - I haven't had time to work on building a new search server yet, that's next on my todo list.

The old search server is in Singapore - connecting to it from New Jersey isn't really going to work

 

[Simon Hampel]

testing 1-2-3

 

[Simon Hampel]

testing 4-5-6

 

[MRJ]

testing 4-5-6

 

[Simon Hampel]

testing 7-8-9

 

[Simon Hampel]

testing 10-11-12

 

[Simon Hampel]

much better - I think the problems with duplicate posts and the time it was taking to make posts has now been fixed.

Ironically, it was actually related to the search engine - I've built a new search server for ZooChat and it's currently re-indexing all of the data on the site. Fixing that solved both problems (no search and duplicate posts / taking a long time for posts to work).

Hopefully things will be back to normal now!

I also fixed some of the caching issues I was having, so I expect pretty good performance now.

 

[KevinB]

Thank you very much for all your hard work, your efforts and your time put into all this, Simon Hampel. It is very much appreciated.

 

[aardvark250]

Thanks. Really appreciate all that.

 

[Simon Hampel]

How is performance for everyone? Site working well?

I'm surprised at how well the site has been performing for me given that it is now technically further away from me here in Australia .

Interestingly, I'm generally finding the site faster even though it is now technically further away from me (east coast US vs Singapore).

From Sydney Australia I get 172ms ping times (the time it takes a packet of data to reach the server) from Singapore but 275ms ping times from Newark. That's 60% slower.

Data hops through 11 network points to get from me to Singapore and 15 from here to Newark.

I'm thinking that the improvements in the server itself has been far greater than the penalty of it being further away (ie it's pretty easy to improve server response time by 500ms, so losing 100ms or so in network performance isn't a problem).

 

[Daktari JG]

Was out a bit yesterday. For me its about the same. But I wasn't really having much problem that I noticed.

 

[Simon Hampel]

Just FYI - I've made some changes to the server and the way we store images, which gives us virtually unlimited storage space (no more chance of another "great purge" - yay )

Please do let me know if you have any difficulty uploading or viewing photos in the gallery - or attachments in posts (they use the same storage system).

 

[Simon Hampel]

I know most people here are not technical, so probably don't understand or care about these things - but I thought I'd share some details and stats about the recent server move from Singapore to the US.

I've been hosting most of my websites in Linode's Singapore Data Centre (DC) for quite a few years now - primarily because it was the closest (network-wise) to Australia where most of my audience is for PropertyChat.

However, ZooChat now has a very international audience, with around 40% of users coming from the US and 25% from the UK, with Australia coming in only 3rd with 5% of the users (overall, North America is 43%, Europe 36%, Asia 10%, Oceania 7%).

So Singapore is a long way geographically from the majority of our users.

I had been planning on relocating the server for quite some time - but there were higher priorities, so it never got done. However, the server outage last week forced me to build a new server and I figured that was as good a time as any to move the site. I chose Linode's Newark DC on the east coast of the US on the basis that it should be close enough to most US users and much closer to Europe than the US west coast.

Yesterday I discovered some interesting data when reconfiguring StatusCake to monitor the new servers - I had forgotten I set up StatusCake pagespeed monitoring a while back and you can see the impact the server move had!

StatusCake is a website monitoring tool which I use to alert me about any problems with my websites or servers.

Their pagespeed monitoring measures how long it takes to download and render a web page from various points around the world so you can compare real world performance impacted by network paths and other geographical factors.

Here are the charts generated showing download time for ZooChat, and how it has changed after we relocated the server ... times are in milliseconds (1000 milliseconds = 1 second), and lower is faster/better.

From the UK:



... I estimate roughly 2x page speed boost for my UK audience.

From the US:



... I estimate as much as a 3x page speed boost for my US audience!


Naturally there was always going to be a penalty from moving the server further away from Australia - but given the US-centric nature of the internet, most countries should have pretty decent connectivity to there, so I'm hoping the impact is not as bad as the gains for most other users.

From Australia:



I estimate approximately 40-50% page speed penalty from Australia - not as bad as I had feared it would be!

It's interesting to see how much more stable the page speed performance is on the new server! The site is now on its own server - so doesn't have to contend for resources with PropertyChat like it did previously.

So, I'm going to call that mission accomplished - the performance boost (for the majority of ZooChat users) from relocating the server to the east coast of the US is significant and worthwhile.