Proposal
Summary
Conduit is a project to reduce bandwidth requirements for streaming live 4K panoramic video to a HMD (head-mounted display, e.g. Oculus Rift). To do this, we will take advantage of the fact that the human eye has a much higher resolution in the center than in the outside, and transcode video on the outer part of your vision at a lower quality. Our goal is to do this with minimal latency, and maximum stream-per-system performance.
Background
Definitions
Source video: The original raw 4K panoramic video stream we want to make available through HMDs.
View-optimized video: The source video, cropped to show just the parts the user is looking at (plus some padding if they move). This is optimized to display lower quality video to lower resolution parts of the human eye.
HMD video: The video cropped to just the part the HMD is looking at, re-projected for both eyes, and formatted via barrel distortion for the Oculus.
Client: A personal computer or phone receiving the video stream, and performing some processing on the stream.
Project
This project revolves around the idea of a bandwidth-latency tradeoff, a challenge in parallelizing programs. We spend extra time processing the video to make it smaller, in order to reduce bandwidth requirements.
The naive approach to getting streaming VR video would be to stream the entire 4K source video, and process it on the client. However, Netflix recommends an Internet connection speed of 25 Mbps for streaming a 4K video. According to Akamai, the average American internet speed is about 10 Mbps. In addition, while 4G internet can get almost to the 4K streaming point, at 20 Mbps down, on average it's a lot lower. Mobile networks aren't built to handle everyone using that much data all the time.
If this technique is successful (and fully realizing it goes well beyond the scope of this project), it may make it practical to attend events live performances and events like concerts and sports games in VR, even from a mobile device, by reducing the bandwidth requirement.
Challenge
There are many challenges to this project. First, neither of us have worked with developing low-level code for VR, or video streaming and video codecs. So we'll have to learn how to do that, and how to process streaming video in parallel. Additionally, at its core, we're exploring the challenge of a bandwidth-latency tradeoff, which is a real issue in developing parallel programs. The challenge is highly scalable, in that if we are able to finish all of our goals early, there are many different ways to optimize the video.
Workload
In terms of dependencies, within a single frame, pixels in the rendering. However, in a blur filter, a pixel's new value depends on the surrounding ones. We're mostly depending on getting HMD data, and also depending on the streaming video to come in.
We'll use both a lot of bandwidth, and a lot of compute, as 4K video contains a lot of data, and video compression does a lot of work with that data. On the client side, it is fundamentally a rendering problem, which can be done well in parallel.
Constraints
During the video processing phase, we don't split the video into nice rectangular partitions, but irregular shapes that overlap, and possibly cause false sharing. The largest constraint is the performance requirement that we need to have very low latency to be comfortable for people to use.
Resources
We'll start from scratch, but ultimately probably need to use video compression libraries. We'll use the Oculus SDK. Oculus also has C++ example code for a scene in case we get stuck.
Goals & Deliverables
Plans
For the initial product, we won't use a separate client and server. All computation will be done on the same machine for simplicity.
We plan to:
- Render the panoramic video to the Oculus in real time. It has to be real time because any latency makes viewing uncomfortable. This should be achievable because we are only rendering one cylinder and we can hardcode the math.
- View optimize the stream by cropping and blurring to reduce resolution and bitrate in unfocused regions for the eye. This should be possible because a simple blur filter is straightforward.
We hope to:
- Implement more advanced view optimizations.
- Separate the program into client and server.
Demo
We plan to:
- Show a demo of the video being rendered in real time on the Oculus, along with log statements showing how much bandwidth would be used.
- We will also give results of the bandwidth we were able to achieve, as well as speedup we got rendering.
Platform Choice
For initial development and debugging, we'll be running the code on our laptops with discrete mobile graphics cards. For performance testing and benchmarking, we can use the graphics processors on the Gates machines for better results. We plan to use CUDA for running our code on the graphics cards.
Schedule
Each week ends at the end of Saturday. For example, week 1 is due on April 11, 23:59.
Week 1: Find suitable video decoder, and get it to work. Set up Oculus SDK on a simple scene.
Week 2: Finish video renderer and cylinder warp program. Start writing the video renderer and cylinder warp. Draw a procedurally generated cylinder using sequential code. Write the project checkpoint writeup, due Thursday, April 16.
Week 3: Implement basic blurring and view optimization.
Week 4: Improve view optimization for lower bandwidth usage. Implement server-client dichotomy for streaming video.
Week 5: Prepare for presentation and final improvements, bug fixes, and tuning. Prepare final writeup.