13h21 Server is back online.
12h57 Faulty hardware have been replaced, services should be back shortly.
12h45 We are in touch with the datacenter, seem to be an hardware problem.
12h34 An alert was received that that one of our server is down. It's impacting other service like the console as the number of failed request have increased.
17h30 EST Everything is back to normal. We no longer detect any issue. We will mark this as solved for now. If there is anything, as always, you can contact us (help at cloudrad.io) and we will be happy to help.
14h00 EST Seems to be back, still monitoring the situation.
13h35 EST Just confirmed with the data center, they are aware of the situation. Since it's pretty major, I'm expecting (and hoping) for a quick fix.
13h10 EST Network issue again between data center. Monitoring, but during that time the console and player will be hard to reach.
8h55 EST System are now back. It was a network issue between our backend system, we are monitoring the situation.
8h40 EST we are seeing error again on the backend. More to follow.
8h30 EST we have identified the problem being a server on the CDN that wasn't responding properly to the call of our backend.
8h00 EST today the player was inaccessible to all radio and the console was responding slowly.
11h42 Services are now back online. We can confirm that it was a network issue at the datacenter. We will continue to monitor the situation.
11h30 We are seeing other service degradation, our console and web player are slow to respond. We are seeing a huge increase in failed request over our API. We continue to monitor the situation.
11h15 There is an network incident in Europe right now. Most of our services in this area are unreachable. The provider is already aware of the situation, we expect a quick resolution since of the scale of this issue.
The latest update of Chrome (version 80.0) now blocks mixed content, which causes the player to fail loading audio on a https page.
Here the official Google statement: "In Chrome 80, mixed audio and video resources will be autoupgraded to https://, and Chrome will block them by default if they fail to load over https://. Chrome 80 will be released to early release channels in January 2020. Users can unblock affected audio and video resources with the setting described above."
No worries, we have you covered, you can use our free https proxy. To do so, simply go to the console, and navigate to Listen > Player and enable the https option. For more details: https://help.cloudrad.io/s/article/player-https-proxy
This option is even available for our customers using our free player without hosting their radio with us. Note that it's provided on a best effort basis and that we might restrict its usage at any time.
22h10 EST - Monitoring have been re-enabled on the auto DJ.
22h00 EST - The issue has been resolved. It was a router issue at our provider that was affecting the traffic on certain IP range and the IP of our master server was affected.
19h00 EST - This issue is causing a lot of false down notification from our monitoring system. We have disabled the monitoring for now.
17h30 EST - We are still investigating this issue.
16h30 EST - There is some intermittent issue with the CDN. While the end point is working for most radio, the master fails to answer to some request.
06h45 EST - Server is back online. There was a miss configured service on the server which caused a kernel panic. The server was rebooted and is now back online. We will investigate further.
06h30 EST - A server in Salt Lake City is currently down, investigating.