An open letter about the worlds

0 Dec. 10, 2012, 11:39 a.m.

After the most frustrating week in the 7 year existence of LiveRC, the event is finally over. There is no doubt though that I need to publicly address many of the issues that we encountered during the broadcast. I kindly ask that you take the time to read through this letter. Given the level of issues that were encountered, it is only fair that I be 100% transparent in this letter and tell all of our viewers the entire truth about everything. But first, I would like to congratulate Robert Batlle on a well deserved world championship win. He put on a textbook example of how to drive one of the most difficult racing surfaces in recent worlds memories.

With that said, I am absolutely disgusted by the level of coverage that we were able to provide for the biggest event in the R/C industry. I can honestly say that we came into this event more prepared than we have ever been for any event. We spent weeks testing...every night, and every weekend. Many of you probably noticed the "LiveRC Testing" event on LiveRC each night for about a month before the worlds. We had our setup down to a science, with redundancy built in at all levels. Unfortunately, there is one key element that is the foundation of our entire service; an Internet connection. We made numerous calls to arrange a solid Internet connection at the worlds, and we paid an abundance of money for this service. We were promised a dedicated 10/10mbps uplink/downlink, and we were assured that it would be solid. To say that this connection was an epic failure would be an understatement. I know many people in the LiveRC chat have many ideas, and while I appreciate the input, let me assure you that we exhausted all options. I spent the entire week trying to arrange bonded cellular connections, a satellite uplink, various microwave antennas, etc, etc. Unfortunately, the infrastructure is just not in place here to get something like that done on that short of notice. The type of connection we had at this event was the exact same type of connection that we used at the 2010 worlds in Pattaya, Thailand, and also at the recent on-road worlds in Amsterdam. At both of these events, we had great service without any of the issues we encountered here.

While I am still frustrated beyond belief at the quality of Internet service that we received, I do want to say that there are a few people who truly busted their butts to try and help us. First, I would like to thank Adrian Castro from Argentina. While he was busy racing the worlds, he was truly our go-to guy to talk and translate with the ISP. Adrian spent countless hours communicating with and fighting on our behalf with the ISP, all out of the kindness of his heart. So to Adrian, I want to extend my sincerest thank you. He truly is a class act racer and an excellent representative of his home country. Second, I can honestly say that the event organizer, Flavio, did try to do everything in his power to get our connection fixed. He made numerous calls to get things fixed...literally dozens of calls per day. Unfortunately, the ISP seems to have absolutely zero interest in customer service. Every day, they promised us new things and that they would be on-site to fix it before 8am. Not once did they arrive before 1pm. The company is a complete joke and never came through on any of their promises.

With all of that said, I really hope that people do not hold a grudge against any of the people involved. Everyone out there needs to know that they put in an honest effort to get things fixed. Unfortunately, it just wasn't meant to be. There is no doubt that things need to change for future worlds, but the event organizer did give it their all and put on a solid event. I am truly amazed that they even had the track in shape to run after the Thursday night storms.

Now back to our broadcast. First, I want to explain as simply as possible how our broadcast operates. Typically, we have our entire crew at the event. However, the fear of customs and getting our equipment stolen was so bad that we restructured our entire setup. We only brought cameras and a small video mixer to Argentina. This allowed us to have every piece of our equipment as a carry-on on the plane. Our entire show was produced back in the USA in a custom built studio in my home basement, which we had setup with dedicated and redundant high-speed Internet connections. In short, we switched cameras in Argentina between camera 1, camera 2, and the mobile camera. That feed was then sent directly to our studio in the USA over a private VPN connection. The graphical overlays, A-Main Hobbies Instant Replays, the TLR Broadcast Studio, and commentary was all produced in the USA and then streamed out to the net, only a few seconds behind real-time. In hind-sight, this was absolutely the best thing we could have done, and I am extremely happy we chose this route. It allowed us to broadcast non-stop for the entire event, even when the Internet was down in Argentina. Had we not gone this route, the coverage would have been non-existent anytime our Internet at the track cut out.

When we tested for the weeks before the race, we streamed a TV-quality video feed at 7.5mbps into our studio, then back out to the net at 2mbps with absolutely amazing quality. Unfortunately, the best we were ever able to stream from Argentina back to the USA was just over 1mbps (on our "10mbps" connection). This was best case scenario. In most cases, it was more like 0.3 to 0.7mbps. This is the reason for the sometimes blocky or laggy video. To make matters worse, the Internet was not just slow, but it cut out every 60-90 seconds. To help fix this, we increased the buffer we were sending to the USA to 12 seconds. This means that we were delaying the "live" video up to 12 seconds to accommodate for any Internet hiccups. This helped immensely...we could lose Internet for 5-6 seconds and nobody ever knew. This was going on the entire event. Quite often though, the Internet would just cut out for 2-3 minutes, 30 minutes, or 8 hours. At this point, we were literally helpless and there was nothing that we could do.

This issue got progressively worse on the day of the mains. The Internet company tried to install a new antenna, but all they did was mess up the existing one worse. At that point, it was noon on Saturday, and the guy simply said, "I am done working now," and he left without fixing anything. Now, instead of dropping connections for 5-6 seconds, it would drop for 30 seconds. This directly caused our servers to crash out. This might be a bit difficult to explain, but I want to be 100% transparent with all of our viewers and let you know the truth of what was going on when our live broadcast servers went offline at times. For you tech guys out there, this will be easy to follow. For everyone else, I will do my best to explain as simply as possible.

The LiveRC network of servers consists of 8 dedicated web servers that we operate, and about 2,500 servers to serve video through a content delivery network that we hire to stream video through. The first server hosts the main LiveRC.com website. Second, we have a very high powered database server. Third, there is an ad server that serves banners and videos. Forth, we have a video transcoding server (this server takes our master video stream and converts it instantly into multiple versions (LD, SD, HD, iPhone version, etc), then sends it off to the 2500 servers to serve the end user). None of these servers had any issues all week long.

The fifth server in our network runs Adobe Interactive Media Server. This server is used to serve up the live scoring and chat. We have stress tested this server numerous times with 10,000+ connections successfully. This server, however, had issues directly related to our Internet connection at the track. The way this server works is quite unique. Typically, a live scoring interface simply reloads a text document every X seconds. This method sucks though. It's slow, its boring, and it uses a lot of bandwidth. The way our server works is that it sends out ONLY the data that changes, instantly when it changes. This saves bandwidth for the end user and provides a real-time scoring interface that shows you exactly (within half a second) as each driver crosses the line instead of updating every 10-20 seconds.

Unfortunately, when we lost Internet at the track, it would empty these variables. Then when our Internet came back, it would reset the variables, thus sending out ALL the chunks of data to each of the 3,500+ users who were online at the time. Once or twice, this was not a problem, however when our connection at the track started going in and out every 5-6 seconds, it slam blasted our server with over 1 million data requests in under 30 seconds. Despite the fact that we stress test for 10,000 users, our system was not designed for 10,000 users to load all the data at the same time, 4 times over a 30 second span. The end result is that this caused our Flash Interactive Media Server to crash out. This was the first in a bad chain of events.

The final 3 servers in our network are 3 identical web servers that host the live events. Typically, we only need one server, but we increased this to 3 servers for the worlds to make sure we could handle the load with no problems. Again, we stress tested on 10,000 connections with absolutely no problems. These servers serve the core foundation of the broadcast screen, the results, etc, etc. These servers also serve the "firewall friendly" mode of the live scoring. The firewall friendly mode is an invisible feature that lets users who are stuck behind a firewall at work to view live scoring. Our interactive data server (chat/scoring) runs on a specific port which many large companies block. If the user is unable to connect to the interactive server, then it automatically switches to the simple live scoring method of updating a text file every few seconds. Typically, we only see about 1% of our users using this mode. However, when the interactive server crashed out due to the lack of Internet at the track (as mentioned above), all 3,500 people instantly got rerouted to the firewall friendly mode. This threw 85,000 file requests at these 3 servers in less than all at the same time, which literally brought them to their knees. As some people experienced, they could not launch the live broadcast, or it took forever to load. The servers never actually went offline, but they got so backed up with requests that it slowed them to a snails pace.

With the above issues, we had no choice but to lock down the live broadcast to paying members only. This was an absolute last resort...it was either lock it down and have it work for a few people, or have the broadcast accessible to nobody. This was ABSOLUTELY NOT a ploy to sell more subscriptions. In fact, this choice is instead going to cost us a ton of money. We have commitments to our sponsors to get as many people watching as possible, and by cutting off free viewers, we cut off a huge number of viewers for the day. There is no doubt that we were on pace to break our viewership record, but after blocking free viewers, I am very bummed that we didn't do that. Now, I need to call each sponsor on our site, explain the situation, and deal with the financial consequences. During the event though, our option was to either lose the servers for everyone, or limit it to only the people who paid so we could keep the servers up as our Internet went in and out. Although this method was a last resort, it did work successfully. To all those who could not view the coverage because they did not have a paid account, please accept my sincerest apologies, and I hope you can see why we did what we did.

If you made it this far into this letter, I thank you for reading through it. I know it is a lot to read, but I truly feel that the best way to handle this is to be 100% transparent and let everyone know exactly what happened. With that said though, I do understand people's frustrations, and ultimately responsibility lies on us to provide a quality service. As I look back at this event, we really did "leave it all out there," and I am very happy with how prepared we were. Unfortunately, I just never saw the scenario that we were dealt with coming. I seriously can't even fake an Internet connection as bad as we had to try and prepare for it. We will live and learn...and should there be a next time, we will be prepared for this scenario.

Finally, I want everyone to know that we are 100% dedicated to our customers. We will make this right, but I kindly ask that everyone gives us a couple days to get home first. I cannot possibly answer the abundance of e-mails that are in our inbox before then, nor am I in the state-of-mind to dive into this task yet. Our entire staff is completely and utterly burned out right now, and we need a day or two to recover. We have done everything possible to make things work, and in the end, we did have a working broadcast for the semi's and final, along with the majority of qualifying. I do hope people take that into consideration.

However, we still have a ton of very upset customers and we will deal with this and make things right, I assure you all. I don't know how we will do this yet though. As it sits right now, this entire trip is a huge financial disaster for us. The "smart" way for us to broadcast an event would be to minimize our costs and put on the bare minimum show with one camera, one announcer, and put it out there. However, this is not how LiveRC is built. Myself and our entire staff strives to put on the absolute best show each and every event, and when it comes to the worlds, we pull out all the stops. On a personal level, I do not take any sort of paycheck from LiveRC. I work a full-time job, 45 hours a week. LiveRC is a hobby; its my baby and my passion, and I absolutely love doing it. Every cent that comes into LiveRC is put right back into it. I honestly think anyone who has seen our operation in person could attest to that. If this were about profit, we would not be attempting to produce a broadcast that costs tens of thousands of dollars per event. We could easily put in 25% of the effort and turn a profit. However, to us, LiveRC is all about progressing our beloved hobby to new levels. We are operating an extremely expansive broadcast in a very niche industry, and in doing that, we are riding the "break-even" line at each and every race. With that said, I need to take responsibility for doing this and acknowledge that it is a very risky way to operate a company. This has never been a problem in the past because we have always been able to quickly overcome any issues and put on a great show. At this event though, we just could not overcome the problems that were out of our hands. While our viewership was at an all-time high, the conversion rate of our sales were extremely low, which is understandable because we had issues all week long and people were afraid to upgrade. On top of that, we have countless people demanding refunds. When all is said and done, I am going to do whatever is necessary to make sure things are right with our customers. What that means at this point for the future of LiveRC, I honestly can not say. I won't know that for a few days yet until the emails slow down and I can assess the financial damage. I will be asking the track to refund the money we paid for the Internet connection, and that will go straight back to our customers. I doubt they will do this, but I hope they will. This certainly will not relieve us of the position we put ourselves in, but its a start.

Irregardless of what the future holds, I look back at the past 7 years with absolutely no regrets. I've made some smart choices and I've made some poor choices, but no matter what, I feel like what we have done has helped show that R/C racing IS a spectator sport. I sincerely hope that we can continue what we have started, and I will put in whatever personal financial contributions I can to keep things rolling, but this event has really pushed us backwards to a near unrecoverable point. I'm sure that somehow we will find a way to get through this, but I won't know until we get home on Tuesday night to assess the damage more thoroughly.

Last but not least, I want everyone to know that I am EXTREMELY proud of the effort our staff put in. We have the absolute hardest working crew in the industry, and I cannot thank them enough.

Sincerely,
Brandon Rohde - LiveRC.com

Share: