06h45 EST - Server is back online. There was a miss configured service on the server which caused a kernel panic. The server was rebooted and is now back online. We will investigate further.
06h30 EST - A server in Salt Lake City is currently down, investigating.
- Radio hosting has now been integrated directly with the auto DJ. So a separate radio hosting is no longer needed with the auto DJ. (Standard radio hosting is still available). - New CDN radio hosting available with the auto DJ. Here the main features: 1. It supports an unlimited amount of listeners and unlimited bandwidth, no need to worry about the number of slots anymore. 2. It supports 2 streams (one high quality up to 320kbps and one low bandwidth up to 64kbps). 3. The CDN provides 3 locations worldwide at this time (more to come). Your listeners will automatically connect to the server closest to them. The Auto DJ itself can be deployed in 2 locations, you should choose the closest location to your studio (where you are doing Live show from). 4. The CDN provides native support on port 80 and https stream.
You can order this feature by going into Create > Auto DJ. It's available at $39/month including 10gb disk space and the CDN radio hosting.
- Fixed charset with the new radio CDN to support accents.
- New scheduled show now default to 1-hour duration and start/end time now increment by 5 minutes.
- A show can no longer overlap another show in the schedule. - Fixed a bug where the delete button was not showing after a show have been canceled. - Fixed the kick source for the Input. - Fixed various crossfade between different source. - Changed the way the radio monitoring process is running to increase reliability. - Various bug fix.
5h43 EST Feb 8 - Console and Player is currently unavailable. Will update with more information.
5h48 EST Feb 8 - Everything is back online.
- Added support to the web player for the older version of Icecast. Album and title info should now update correctly. - Added short notification when a track is added to a playlist. - Added crossfade when Input (Live show) connect. - Input now support SHOUTcast v1 encoder format and Icecast to make it compatible with all encoder.
- Crossfade was not working as specified in Auto DJ settings.
- At the end of a scheduled show, there is no transition. It should be fixed in the next release.
To get all the fix in your auto DJ, go to Settings and click on "Save and Restart".
[14 January 7h00 EST]There is an issue with the console at this and the service on 1 service location (Roubaix, Europe) that we are investigating.
[7h30 EST] There was an issue with one of our radio location (Roubaix). All the secondary IP stopped responding on the server. This caused a lot of error on our console which degraded the service for the radio player and console access. We restarted the network interface on Roubaix which resolved the issue with all the secondary IP. Services are going back to normal now. We are still monitoring the situation. More information to come.
[8h03 EST] We are adding monitoring on every secondary IP to have full visibility on this. All radio should be back online, if your radio is still offline, contact email@example.com
Incident details: Last night at around 1h40 EST, our servers in Roubaix location were doing a critical security update. One of the update caused the network interface to restart. Upon restart, the secondary network interface failed to load its configuration. Because of this, some IP were not accessible until we restarted the interface around 7h15 EST. It impacted multiple radios who became unavailable. This had for impact to cause higher load than usual on our console due to all the failed call. It explains why the console was not accessible (or very slow) for our customer. There are multiple steps we will take to address this issue. First, we added monitoring on secondary IP on all our server so we have visibility in the future if such a problem arise again. We will also improve the way we receive those critical notifications so we are alerted day and night. We are reviewing and improving our internal procedure as well so everyone in our team knows how to react and who to contact in such an event. We will also add more resource to our console so it can handle the load better in those type of situation.
We apologize for the issue again, we take your radio uptime very seriously. If you have any question, do not hesitate to contact us at firstname.lastname@example.org
[16 January 13h10 EST] Update: The issue happened again last night. While we responded quickly to avoid extended downtime, we are still investigating the root cause of this. I will update shortly on this.
[16h00 EST] After a complete review, it seems like automatic updates were causing the issue. Both day (14 and 16) there was a systemd update. We have disabled all automatic update at this time and we will perform them manually. We have also updated the firmware of the network interface on affected servers.