The science and tech behind live streaming; a deep-dive with Prime Video
On his recent visit to India, BA Winston, Vice President, Technology at Prime Video, spoke about the tech that goes into providing a seamless live video streaming experience, the challenges and solutions implemented to overcome the same. Here are the excerpts from the conversation:
Q: Last year, Prime Video live-streamed India v/s New Zealand men’s cricket series, where you introduced a host of new features. From a tech perspective, how different is the Indian market as compared to other Prime Video locales for live streaming, especially in the context of cricket?
Winston: The last couple of years have been quite exciting, especially with our preparation and launch of New Zealand Cricket here in India. Obviously, we had to build several technologies that are unique to India. We are obsessed with the customer experience, and we wanted to deliver a great, great experience.
We offered the live streaming experience in five different languages, we built technologies for customers to be able to select the stream from within the game itself, making it easier for them to switch. We built a technology called Rapid Recap and launched it for the marquee India men’s tour of New Zealand – a unique feature to allow customers to quickly catch up if they joined the stream after the game or if the match had already started.
Obviously, the Indian market is different. The device spread has different technologies, the network technologies are different. We had to build video acquisition and video processing infrastructure unique for India. We had to build processing facilities and production facilities for different languages and had to build several unique things to be able to optimize that experience from a customer perspective to account for the device strategies, different network conditions, and bandwidth conditions. We wanted to optimize for the bitrate that is very unique for India and for cricket. So, we had to do a lot of unique things to ensure we get the experience right for our Indian customers.
Q: How challenging was it to bring Live streaming to India? How did you overcome these challenges?
Winston: I think as a technologist, challenges are always exciting, as well as interesting. I would say broadly, there are four challenges that I want to highlight and while some of these are not unique to India, they are still relevant.
The first is Availability. Availability requirements from a customer perspective mean it should be 100 per cent interruption free. An interruption is a massive defect and even a few seconds of interruption is actually unacceptable from our perspective. So, availability requirements for Live, especially in sports, are actually quite different.
The second is scaling. We just do not know what and when our peaks are going to be. Depending on how the match goes, customers could join and rejoin around key moments. So, from a scaling perspective, we had to build a lot of ML-based models to be able to predict and also to be able to optimize our scaling.
The third is reach. We need to be able to reach all customers. This again, in the context of India, is quite complex because of the network infrastructure. It is a big country and is spread out, you have rural India and urban centers - so, you have to be able to take care of the customers in all parts of the country - and ensuring they have a good experience, despite the connectivity, is actually challenging.
And the fourth, I would say is the quality of the stream itself. There are challenges but still, customers’ expectation is that the quality needs to be good. So, how do we do that despite these network challenges, despite the challenges with the device spread, how do we ensure that the actual picture quality when they are watching on a mobile device is actually good?
These are some of the broad challenges we deal with, across the board in all Live events. But when it comes to India, I would say the device mix plays a key role. You cannot just pick up all Android phones and test them. It is just impossible. The spread of Android devices, the variances in chipset and capability, and the memory is very different. Not all Android phones are made the same way, you have different price points, and therefore the spread is very different. We had to optimize and be able to provide a better experience across all those devices.
Also, the network, as I mentioned, has to be a local optimization because the network in India is different from the network in the US or in European countries. So, we actually built local teams here, as well as local processes and operations to be able to work with the network operators here. We have to model what the capacity need is in a certain rural part of the country would have, then we need to be able to understand what are the networks and ISPs that are serving that rural part. And then we have to figure out how we get to that ISP that is serving that rural part from our cloud locations. And so, you need to literally work out that entire flow, we can leave nothing to chance, in our quest to offer customers a good experience. We have to have the capacity need predicting model right and need to know exactly how the network traversal is going to happen. We have to be very deliberate and intentional about this whole capacity and workflow, especially for India. That was a pretty challenging effort, as well as time-consuming. But if you are obsessed with the customer experience, you have to do it.
The other challenge I mentioned is the networks here are different from the networks in other countries. Therefore, if the available bandwidth is lower, you have to optimize your bitrate and you have to optimize the picture quality unique for that bandwidth condition. That is a lot of science and there is an element of art because more bits aren’t equivalent to better, it is the perceived picture quality, which you need to optimize. So, we actually have scientists who have that kind of experience in this field, to be able to figure out the perceived picture quality on different devices, and that has allowed us to optimize the different bitrates to provide. The goal is to reduce the bitrate and improve the picture quality or reduce the bitrate and provide the maximum picture quality at that bitrate, keeping the network condition in mind because you need to have different band conditions. We use AI and ML frequently in the space because the variations are quite a bit.
Once you have optimized the bitrate, and once you have the connectivity, the third problem, I would say is - how do we deliver during the live event? That is also not an easy problem. People might try to download a game or people might be streaming something else or a network gear goes down in a local ISP, you lose half the capacity. You need to be able to resiliently traverse through that, despite these network conditions. And this is where again AI and ML come into play. In our case, we use ML-based technologies to adaptively change bitrate based on network conditions. We have to do a level of prediction based on what is on the device at that point in time when I am streaming, and what is happening to the network. We cannot give you the best experience while showing everything in the field. The shot needs to be optimized for the scorecard, optimized for just the ball, or optimized for the players, but not everything. Those are all hard problems to solve. And you cannot solve it once for football in the US and assume it is going to work for cricket, as it is a different sport. So, the pictures are different, right? You need to be able to optimize for cricket. And, again, you can do that if you have dedicated people and dedicated scientists and engineering talent, who will do that. And that's what we brought to the table. And we are quite proud of what we were able to deliver to our customers here.
Q: How different is live streaming cricket as compared to other sports? Especially as we consider the fact that some of the other sports may be for 45 minutes or 90 minutes but a T20 game can go on for 3+ hours and then you have the 50 overs matches too. Also, in cricket, you have natural ad insertions after every over. You also spoke about bitrate; then how does it impact the latency?
Winston: I talked about some of the uniqueness that we had to do for India. We had to optimize for cricket as it is a different sport. We had to optimize for Indian network conditions, which are different and we had to build capacity and build ML-based models to optimize. Then we had to optimize for Android because we had to optimize for the Android phone resolution and the form factor. And then even the video infrastructure as to how we get the content is different. Because I mean, this was playing cricket in New Zealand, we had processing facilities here in this country, we had to do audio production for different languages, we had to do the local caching.
On latency issue
Latency is an important aspect. With sports, you don't want to be left behind and be the last one to know about key moments. We want to be able to reduce latency as much as possible. At Amazon, we always work backward from the customers, we know that there is a customer need, and we know what our customers want. So, we had been working on it. We have dedicated teams focused on reducing latency. We will announce more soon. But what I can tell you is we are not hesitant to even explore other network protocols. We have been experimenting and actually using a different protocol called UDP which is User Datagram Protocol, different from HTTP which has allowed us to reduce latency quite a bit. We will share more in the coming months but we believe we have the technology in our storehouse for us to be able to reduce latency and continue to improve the experience for our customers.
Ads are a very complex infrastructure. We did deliver ads for our customers here for cricket. And I would say that changes our video processing flow. When it comes to ads, we have to take care of ads with different formats. It has to be seamless for our customers. We build technologies to be able to do that, for India, for cricket. And I would say, ads are a space, especially in the context of Live, where there is a lot of innovation out there, a lot of things that we are working on while reducing latency and delivering a good ad experience that is personalized, dynamic, more real-time and interactive, that is quite different from broadcasts.
Q: What is the acceptable level of latency for you when it comes to live sports?
Winston: I would say that one, it is different for different sports; and two, the goal is not to get to real-time, because if we get closer to real-time, the picture quality is not going to be good. It is about reducing latency to a point where it is not really perceived to be lagging by customers. We look at the overall perceived experience. And having the tech be able to reduce the latency is how we are looking at it. I don’t want to throw specific numbers because it is different with different sports and also with different devices. When you think about live streaming and where the event is being played, and you need to stream the video from there to where you are delivering, you can do that with extremely low latency but you also need to do that with high availability. So, a lot of factors come into play to be able to get that. What I can assure you is that we look at every single part of our ecosystem and ensure we provide the highest quality, lowest latency, and high availability; you need to look at all those three together.
Q: How does Prime Video fare when compared to the tech stacks of its competitors like Netflix and Disney Hotstar at the global and local levels?
Winston: I would say that we are not very focused on competitors from a tech perspective. We are, however, very focused on our customers. We believe focusing on our customers will drive us to the right tech stack and help deliver the right experience for them, all the time. And that has worked for us. So, we will continue to be focused on our customers.
Disclaimer: This article has been produced on behalf of Prime Video by HT Brand Studio.