Facebook is experimenting with new technologies to turn flat photos into 3D photos

Facebook is experimenting with new technologies to turn flat photos into 3D photos

With the development of technology, people can now take pictures of their favorite moments with mobile phones and other devices. Many people may have thought, if there is a black technology that will make the flat 2D photos we take into three-dimensional 3D photos …

Facebook has long thought about this, and to improve the user experience, in 2018, Facebook introduced the 3D photo feature. This is a new immersive format that you can use to share photos with friends and family. However, this feature relies on the dual-lens “portrait mode” function found only on high-end smartphones, and cannot be used on ordinary mobile devices.

To allow more people to experience this new visual format, Facebook has developed a system using machine learning. This system can infer the 3D structure of any image, and any device and any time the image taken can be converted into 3D form, which can make people easily use 3D photo technology.

Not only that, it can also process family photos and other precious images from decades ago. Anyone with an iPhone 7 and above, or a mid-range or higher Android device can now try this feature in the Facebook app.

Building such enhanced 3D pictures requires overcoming many technical challenges, such as training a model that can correctly infer the 3D positions of various topics, and optimizing the system to execute on a typical mobile processor device in 1 second. To overcome these challenges, Facebook trained convolutional neural networks (CNNs) on millions of public 3D images and their accompanying depth maps, and leveraged various action optimization technologies previously developed by Facebook AI, such as FBNet and ChamNet. The team also recently discussed related research on 3D understanding .

This feature is now available to anyone using Facebook, so how exactly is it built? Let’s take a look at the technical details.

Delivering efficient performance on mobile devices

Given a standard RGB image, 3D Photos CNN (3D Photo Convolutional Neural Network) can estimate the distance of each pixel from the camera. Researchers achieve this goal in four ways:

  • Build a network architecture with a set of parameterizable, action-optimizable neural building blocks.
  • Automate architecture searches to find effective configurations of these modules, enabling the system to perform tasks on a variety of devices in less than a second.
  • Quantitative perception training, using high-performance INT8 quantization on mobile devices, while minimizing performance degradation during quantization.
  • Get a lot of training data from public 3D photos.

Neural Construction Module

Facebook’s architecture use is inspired by the building blocks of FBNet. FBNet is a framework for optimizing the ConvNet architecture for resource-constrained devices such as mobile devices. A building block consists of pointwise convolution, optional upsampling, kxk depth convolution, and additional point-by-point convolution. Facebook implemented a U-net-style architecture that has been modified to place FBNet building blocks along skip connections. The U-net encoder and decoder each contain 5 stages, each of which corresponds to a different spatial resolution.

Automated architecture search

In order to find an effective architecture configuration, the ChamNet algorithm developed by Facebook AI automates the search process. The ChamNet algorithm continuously extracts points from the search space to train precision predictors. The accuracy predictor is used to accelerate genetic search to find a model that maximizes prediction accuracy under the condition of meeting specific resource constraints.

A search space is used in this setup, which can change the channel expansion factor and the number of output channels of each module, thereby generating 3.4 × 1,022 possible architectures. Facebook then used the 800 Tesla V100 GPU to complete the search in approximately 3 days, setting and adjusting FLOP constraints on the model architecture to achieve different operating points.

Quantitative Perception Training

By default, its model is trained using single-precision floating-point weights and triggers, but researchers have found that quantizing weights and triggers to 8 bits has significant advantages. In particular, the int8 weight requires only a quarter of the storage required for the float32 weight, which reduces the number of bits that must be transferred to the device when first used.

The throughput of Int8-based operators is also much higher compared to float32-based operators, thanks to an optimized database such as QNNPACK of Facebook AI, which has been integrated into PyTorch. We use Quantitative Sensing Training (QAT) to avoid quality degradation caused by quantization. QAT is now part of PyTorch, which simulates quantization and supports back-propagation during training, thereby closing the gap between training and production performance.

Finding new ways to create 3D experiences

In addition to improving depth estimation algorithms, researchers are also working to provide high-quality depth estimates for images taken by mobile devices.

Since the depth of each frame must be consistent with the next frame, image processing technology is challenging, but it is also an opportunity to improve performance. Observing the same object multiple times can provide additional signals for highly accurate depth estimation. As the performance of Facebook’s neural network continues to improve, the team will also explore the use of technologies such as depth estimation, surface normal estimation, and spatial inference in real-time applications such as augmented reality.

In addition to these potential new experiences, this work will help researchers better understand the content of 2D images. A better understanding of 3D scenes can also help robots navigate and interact with the physical world. Facebook hopes to help the artificial intelligence community make progress in these areas by sharing the details of the 3D picture system and create new advanced 3D experiences.

More article: Beginner photography tips for better photos

Old photos from the 19th century. Why do the characters in the photos look like having poker face?

Old photos from the 19th century. Why do the characters in the photos look like having poker face?

If you look closely at the photos of people in the 19th century, you will find a “fun” phenomenon: the expressions of the characters in many photos are like the portrait expressions of having poker face  J, Q, and K. The expressions are almost the same, without smiles. Facial stiffness. The pictures below are one by one serious and feel like they are being coerced into taking pictures.

Why is this happening?

Although there are various reasons for this phenomenon, the following factors are the most likely:

At that time, the photographing technology was limited, the photographing cost was very high, and the wealthy people were able to take photographs. Therefore, when taking photos, photographers do not waste film as much as possible and save the film as much as possible, but they need to put as many pose as possible, but the problem comes: the more you want to pose, the more it is counterproductive and makes people feel bad. The expression is unnatural.

Because there are not many opportunities to take pictures. At the moment, the subject has almost no natural and relaxed experience in taking pictures. The more I want to take pictures, the more pretentious.

The most important thing is that the teeth of the people at that time were generally not good-looking. Therefore, try not to expose your teeth when taking pictures.

Beginner photography tips for better photos

Beginner photography tips for better photos

About off-camera lighting, or hence the name, flash. I just mentioned him to him, Ken Rockwell mentioned him. He is almost a Nikon fanatic and he talks about jokes on many different topics. He also has a lot of great information about Nikon camera equipment. Joshua Hoffine is probably one of my favorite photographers because the camera he sets is like a movie. He also has a great blog, so I suggest you follow him so you can see his latest amazing work and how he created it. Now there is no brain…

Continue to press the shutter!

The practice is perfect, and taking pictures is no exception. If you want to be the best photographer in town or in the world (oh, this is possible!) You have to leave and start shooting more. For those who register for more than 60 hours a week and still strive to be a great photographer, you must take time out to shoot. I know that it is very difficult. I work only 50 hours a week in my retail job in Berkeley, Michigan, and I know firsthand how hard it is to take photos. If you want to be better, you have to take the time to take a photo. When I first entered photography, my camera took no more than a few hours per month. It shows; my photos look the same as I did at the beginning. What I need to do makes me develop the habit of spending a few hours a week studying technology. Every other day, I finally develop into a few hours to learn how to take pictures. Soon I found myself taking pictures and learning a few hours a day, and I felt that my head got a few centimeters from all the information I collected!

Last words

If I try to be a better photographer last year, I found one thing, that is, attitude is related to photos. If you like, most of this article has nothing to do with the actual photography “technology”, because I feel that if you are told to take a picture in some way, you will be limited in a set of stupid rules, and you will never be able to Do your best to explore photography with your own feelings. Anyway, I hope that I can make some good points, and you will take action soon, because if you do, I know that you will soon find that you have a great photographer in your heart. You must have the motivation to continue and maintain the right attitude. Do you want to know how to take better pictures? How to create great images with the help of Photoshop? I know that I have. Ok, I will tell you something; I will tell you some secrets to create amazing and memorable images, and how to make the most of your camera through these photography techniques, these techniques will definitely improve your photography. .