Did the Website go Down?

[Simon Hampel]

Why does this happen every day at 3PM?

Backups get taken daily - and the database is now so large that the site stops responding for a few minutes until it completes.

The time was chosen to be approximately the low point of daily traffic on the site.

The following chart shows the number of sessions (visits) for each hour of the day, with times in GMT:



You'll see that the quietest time of the day is approximately 5AM GMT, which is 15:00 (aka 3PM) on the east coast of Australia.

So as annoying as it is for it to be unavailable in the middle of the day here in Australia - I figured it was the least disruptive time of the day overall when we consider the international audience of the site.

 

[CGSwans]

Cool. I wasn't bothered by it, was just curious. It obviously just correlates with the stage of the day when I'm bored and not paying attention at work.

 

[Simon Hampel]

Cool. I wasn't bothered by it, was just curious. It obviously just correlates with the stage of the day when I'm bored and not paying attention at work.

I wonder if it would help if it was something like 2:45PM or 3:15PM?

I presume not - since you were likely already on the site when it went down?

 

[CGSwans]

Let's just say if it's within east coast business hours, I'm probably going to bump into it at some point.

 

[lintworm]

The website has been agonizingly slow in the past days and seems to go offline multiple times a day... Any idea on the cause?

 

[Simon Hampel]

The website has been agonizingly slow in the past days and seems to go offline multiple times a day... Any idea on the cause?

Curiously, it has been faster than ever for me of late! ZooChat is hosted on the east coast of the US and I'm in Australia - so there's a lot of network between me at the site, but it's still quite snappy.

It could be local network congestion at your end or a problem with your internet? I assume other sites have been functioning as normal?

 

[lintworm]

It could be local network congestion at your end or a problem with your internet? I assume other sites have been functioning as normal?

Other websites function as normal, I am having the problem both on my smartphone and my laptop. The website seemed down about 10 minutes ago for 1 minute and is now back to normal and this happens quite frequently....

 

[pachyderm pro]

Curiously, it has been faster than ever for me of late! ZooChat is hosted on the east coast of the US and I'm in Australia - so there's a lot of network between me at the site, but it's still quite snappy.

It could be local network congestion at your end or a problem with your internet? I assume other sites have been functioning as normal?

Other websites function as normal, I am having the problem both on my smartphone and my laptop. The website seemed down about 10 minutes ago for 1 minute and is now back to normal and this happens quite frequently....

I’ve been having the exact same issue over the last few weeks. Both on my phone and laptop, there have been many instances where the site will go briefly go offline for no longer than a minute before coming back. It happened at least twice yesterday. The screen that comes up explicitly states that’s it’s an error on the hosts side as well and not an issue on my end.

 

[Simon Hampel]

I looked at some of the logs and analytics for the site and server and I did note some anomalies where there seemed to be some timeouts occurring - with one happening around 4pm my time, which is around when lintworm reported the issue above.

I'm not sure exactly what is causing the problems at this point and it's not typically an easy thing to identify the root cause.

I'll continue to investigate - but it may require a server rebuild and upgrade to reduce the number of timeouts occurring.

 

[Chlidonias]

The website has been agonizingly slow in the past days and seems to go offline multiple times a day... Any idea on the cause?

I’ve been having the exact same issue over the last few weeks. Both on my phone and laptop, there have been many instances where the site will go briefly go offline for no longer than a minute before coming back. It happened at least twice yesterday. The screen that comes up explicitly states that’s it’s an error on the hosts side as well and not an issue on my end.

Happening to me as well over the last few days (in New Zealand). For short periods through the day pages will be really slow to load or there will be host error messages.

 

[birdsandbats]

It's been happening to me as well, several other reported the same thing in the chatroom.

 

[DesertRhino150]

A bit ironically, my connection to the site went to a host error message at around 10:30 today when I clicked on this thread. I've noticed it happen a few times over the past few weeks as well, but the speed of the site itself remains fine.

 

[Nile Hippo Expert]

I looked at some of the logs and analytics for the site and server and I did note some anomalies where there seemed to be some timeouts occurring - with one happening around 4pm my time, which is around when lintworm reported the issue above.

I'm not sure exactly what is causing the problems at this point and it's not typically an easy thing to identify the root cause.

I'll continue to investigate - but it may require a server rebuild and upgrade to reduce the number of timeouts occurring.

I have also gotten at least twenty notifications by just clicking on threads on this website that say "Bad Gateway, Error Code 502" in the past week. The problem might just be that the website in general is having some problems. I am 100% certain it isn't just my computer, iPhone, or network.

 

[birdsandbats]

This problem is significantly worse for me today. It seems like the website is going down every hour or two.

 

[Bactrian Deer]

This problem is significantly worse for me today. It seems like the website is going down every hour or two.

I seem to be having the same issue, I had recently had it around 20 minutes ago.

 

[Simon Hampel]

This problem is significantly worse for me today. It seems like the website is going down every hour or two.

I can see the errors - but I haven't yet worked out what is causing them.





Those times are Sydney time (UTC+10), and the first of the two peaks corresponds to the time you mentioned the issue.

 

[TinoPup]

I wasn't having problems with the issue was first reported at the end of June, but the last few weeks I've been having the site be down several times a day.

 

[Simon Hampel]

I wasn't having problems with the issue was first reported at the end of June, but the last few weeks I've been having the site be down several times a day.

It continues to be a problem.

The purple lines below represent the number of errors visitors have received over the past 24 hours (Sydney time).



I've been trying to analyse the root cause of the issue - I suspect there's too much bot traffic on the site, so I have tried implementing some firewall rules to block the worst of them to see if that helps.

I need to upgrade the site to new infrastructure - but I can't do that until I get the software upgraded, and unfortunately that's a few months away, so I'm going to have to ask everyone to be a bit patient for now.