Hyderabad: Cinema has undergone a complete transformation, thanks to technology and its rampant use in shooting movies. These days, most of the movies are dubbed into multiple languages to cater to a larger global audience. While the movie is shot in one language, it is then dubbed into many with the actors' voices being overlapped by dubbing artists'. While it has worked well so far, two researchers at IIIT Hyderabad aim to take this to the next level.
Ph.D. scholars Rudrabha Mukhopadhyay and KR Prajwa of the Centre for Visual Information Technology at IIIT-H have managed to make actors lip-sync in several languages with near perfection. So while a movie may originally be shot in English, with the new technology, a viewer would see and hear the actor speaking in, for example, Hindi or Telugu.
According to the scholar duo, the key to their software is accurately masking lip movements in regional languages over the original dialogues. For instance, until now actor Rober Downey Jr. spoke in English as Iron Man and his dialogues were dubbed in Hindi and other languages. With the new tech, however, viewers can enjoy Iron Man dialogues in clear Hindi and have Robert speaking the language as if he knows it.
The two researchers at IIIT-H have been successful in creating such technology with the lip-synch software capable of handling 100 languages. With the help of Professor CV Jawahar, Professor Vinay Nambudri, and CEO Pawan Reddy, the technology came to fruition within two years of research. It works entirely with the help of artificial intelligence. For this, the Wav2Lip model was created by analyzing hundreds of hours of videos.
Also read:UN Secretary-General appoints India's Amandeep Singh Gill as his envoy on technology
Rudrabha and Prajwal founded a company called Neural Sync in IIIT-H. The first project from that company was the Wav2Lip. This technology was originally made for students. Many Telugu students are lagging behind in English. Many are unable to read as more information is available in English. This Wav2Lip was created using artificial intelligence so that such students could easily learn in the language of their choice.
Many film directors are happy with the launch of Wav2Lip technology. They hope that this technology will contribute greatly to the film dubbing industry. Many producers are already in talks to sign contracts with Wav2Lip. This technology is available in markets in India as well as the US and Europe. Representatives from the company recently applied for a patent in the US. Rudrabha and Pawan Reddy are trying to take this technology to Hollywood. They are confident that Wav2Lip technology is set to revolutionize not only education but also cinema.
