The technical principle of XR+LED virtual shooting

LED virtual shooting technology integrates many cutting-edge film technologies such as real-time digital images, real-time light and shadow rendering, real-time camera motion tracking and real-time computing. Through XR+LED, you can shoot blockbuster movies without leaving home, which can significantly reduce the time and economic cost of film and television production.

1. What is XR ?

XR (Extended Reality) is a term that describes the fusion or interaction of virtual and real elements. It is generally considered to be a general term for various technologies such as AR (Augmented Reality)/VR (Virtual Reality)/MR (Mixed Reality). Through camera tracking and real-time image rendering technology, the display medium (LED/projection bowl screen) and the virtual scene outside the display medium can track the camera angle of view in real time, and synthesize with the real shooting picture in front of the camera lens, thus creating an infinite sense of space technology.

"Fashion Lab" uses XR extension technology to create a virtual immersive concert

2. The principle basis of XR

Understanding perspective is an important prerequisite for understanding XR technology. XR technology is a bridge between virtual and reality, which is based on the principle of perspective.

In daily life, some composite photos of people P in front of tourist attractions can be faked, while others can be seen as fake at a glance. One of the conditions to make the composite photo look real is to properly handle the spatial relationship between the characters and the background, that is, the perspective relationship. All objects in the composite photo must have a unified perspective point in space. If the background in the photo is taken from a top-down angle, then the person who needs to be composited must not use the image of the person looking up. Otherwise, the picture will look uncoordinated and unreal.

All objects in the photo must have a uniform point of view in space, otherwise the image will appear unreal

This is a way to confuse virtuality and reality on a two-dimensional plane.

So how to realistically confuse virtual and reality in three-dimensional space?

In 2021, Hong Kong artist Andy Lau, who is thousands of miles away, will perform on the same stage with CCTV Spring Festival Gala actors through XR technology

If you want to synthesize a video, the actor is in the room, but there is a whole forest behind him under the lens, is there any other way besides using the traditional green screen + matting method?

If the camera lens is fixed and the position is appropriate, you can even hang a huge forest pattern canvas directly behind the actor. As long as the actor and the background plane show the correct perspective relationship under the lens, some of the most basic needs can be met.

But once the camera moves, the fact that the background forest is actually a flat canvas cannot be concealed anyway.

In fact, the realism of the background of the scene is shaped and enhanced by the movement of the camera. In every frame where the camera is pushed, pulled and panned, such as the large panorama when looking down, and the middle shot when following the shot-there should be a background scene that satisfies the real perspective relationship at this position and angle behind the actor.

It would be nice if the background behind the actor could match the camera lens when the camera rotated or changed angles.

And this is the problem that XR technology is going to solve.

The realism of the background of the video scene is shaped and enhanced by the movement of the camera.

3. Technical equipment conditions for XR implementation

After understanding the concept and principle of XR, how to realize it in practical application?

At present, the technical equipment required for the realization of mainstream XR technology in the industry mainly includes: LED display screen, LED control system, virtual rendering system, camera and tracking and positioning system, synchronization system and other auxiliary systems. With the above technical equipment. Through the real-time workflow, performers can be integrated into the virtual world without green screen and post-production, and the final image can be output in real time.

The overall technical framework is shown in the figure below:

The XR scene technology framework includes several major elements such as a shooting site composed of LED displays, a camera, object position tracking and positioning technology, a real-time graphics rendering engine (AR technology), and a high-speed and low-latency network.

● LED control system

It is mainly responsible for receiving virtual rendering images and outputting them to the China LED display screen. An excellent control system needs to have superb video processing capabilities and be able to perform precise color management and brightness adjustment on the LED screen body to maximize the display of the LED screen body. Display effect, and provide technical support for special application requirements of XR (such as multi-camera, high frame rate, etc.) on the hardware side.

● Virtual rendering system

It mainly includes a media server and a real-time rendering engine. The master media server, specifically designed for advertising display screen led, is responsible for controlling the rendering engine, processing camera data, and outputting LED video source images. The real-time rendering engine provides the integration and production of virtual digital assets and builds a complete virtual expansion space.

● Camera system

To capture real images and send them back to the virtual rendering system, a professional-grade film and television camera is required to achieve clear and lossless real image acquisition. Generally, cameras that support acquisition frame rates above 120Hz are used.

● Tracking and positioning system

It is mainly used for positioning to realize real camera displacement and lens rotation data. The key indicators of the positioning system are accuracy, degree of freedom and low delay. More commonly used are mechanical tracking (pan/tilt, robotic arm, etc.), optical positioning, and other infrared tracking , visual camera tracking, etc.

● The synchronization system is mainly used for the reference time and picture frame of the overall XR workflow. An independent and accurate synchronization system is a necessary condition for XR projects. The synchronization system can effectively reduce the scanning lines and black fields that appear when the camera shoots the LED screen. And other problems, you can also achieve multi-camera shooting effects by controlling the synchronization time and matching the LED control system.

XR technology is an emerging technology that has developed rapidly in recent years. Compared with the traditional green screen + matting virtual shooting method, XR+LED display has significant advantages. In the future, driven by the rapid iteration of technology and the internal drive of demand innovation, the XR technology in the ascendant will surely usher in a vast world of its own.