THE RESURRECTION OF KOBZAR: Alexandr Smirnov about the reconstruction of the voice of Taras Shevchenko and the living voices of Ukrainian culture
The main creative event of the year! The innovative experiment that will be discussed deserves such a definition. On the basis of anthropometric data using a neural network, a team of enthusiasts managed to reconstruct the voice of Taras Shevchenko! The peculiarity of the project, on which 150 people have been working for almost a year, is that no recordings of the poet’s voice have survived.
Shevchenko died in 1861 – 16 years before Thomas Edison presented the world with the first working prototype of a phonograph. Read about this unique project, modern technologies, Shevchenko’s gene and real magic in an interview with Aleksandr Smirnov, the partner and the co-owner of the creative agency TABASCO – the ideological inspirer of Kobzar’s “resurrection” in modern Ukrainian history.
MORE THAN ALIVE
We work in the creative business, so we constantly monitor and test technological innovations that open up new opportunities for the implementation of our ideas. It applies to both commercial and social projects – we always try to give them some kind of technological value. Naturally, we could not pass by the Deepfake technology. It provides tremendous opportunities for very accurate reproduction of the appearance and voice of any person.
It is clear that, first of all, it concerns some outstanding people, “stars” of the present and past. One of the most striking examples is the exhibition Dali Alive at the Salvador Dali Museum in Florida, where a wonderful performance was born at the intersection of art and artificial intelligence. The great surrealist and hoaxer was revived, based on the analysis of a large number of recordings of his voice and video filming.
It has long ceased to be something out of the ordinary – to perform on the same stage with real, live artists of holographic clones of Elvis Presley, Michael Jackson, Marilyn Monroe, Frank Sinatra, Amy Winehouse and Whitney Houston.
Thanks to the projection technology of Deepfake and the synthesizer of speech, David Beckham spoke in commercials and documentaries in eight languages that he does not speak. And Ronald Reagan or Malcolm X can still turn to their adherents.
RESURRECTION OF KOBZAR
All these examples are united by the fact that the reenactors originally had recordings of real voices. We asked ourselves: is it possible with the help of modern technologies to restore a voice, the recordings of which have not been preserved? Having set ourselves such an ambitious task, we asked the following question: who in this case would we like to “resurrect” in the first place? The dispute was short-lived.
Everyone agreed on how great it would be if “Ukrainian №1” – our Great Kobzar, Taras Shevchenko spoke to us! After that, an active, months-long search for all available materials started, on the basis of which it was possible to proceed with the reconstruction.
First of all, our attention was attracted by the death mask of Kobzar. Thanks to it, we managed to make a 3D model of Shevchenko’s head. Further, we took into account the data of his photographs, as well as descriptions by his contemporaries of his appearance, a timbre and a manner of voice, features of everyday behavior. Invaluable assistance in it was provided by the staff of the National Museum of Taras Shevchenko.
CASTING TWINS
As a result, we have accumulated a fairly large amount of information, on the basis of which we have built a multivariate model together with our partner – the WANTENT company. It must be said that if we take only one anthropometric parameters, then up to 70% of a person’s voice can be restored from them.
But we supplemented anthropometry with other data: we conducted a national casting of Shevchenko’s twins, the correspondence of which to Kobzar was determined through comparison with the obtained 3D model, physiological and psychological characteristics, a specific voice portrait, which the specialists of the National Museum of Taras Shevchenko helped to determine. Thus, 108 candidates were selected.
Later another great technology came into play – an artificial neural network. It narrowed the circle of Kobzar’s doubles to 26 Ukrainians, who most closely matched the psycho-anthropometric portrait of his voice and the descriptions of eyewitnesses. If a person outwardly coincided with Shevchenko by 98%, he had similar everyday habits, but at the same time he did not have a baritone, like Taras Shevchenko, he was excluded from the sample.
NEURAL NETWORK AND MAGIC
At the final stage, the neural network analyzed the median characteristic features of all 26 people and settled on 3 votes that were the closest to the “optimal value”. After that, the voices were synthesized with the help of special computer programs into one – a voice that could be spoken by Taras Shevchenko. The text that our “resurrected” Kobzar utters has been read by the operator, and the neural network is already transforming it into this resulting voice.
At the same time, it was not without mysticism. When we looked at who the neural network ultimately chose, it turned out that the 3 final voices belong to the artist, the poet and the military man – that is, those hypostases in which Shevchenko realized himself during his lifetime.
There were other “signals” that could be interpreted as luck, predetermined from above. For example, if there was no death mask, then it would be necessary to carry out reconstruction from photographs, and it would give much less accuracy.
BETWEEN CREATIVITY AND SCIENCE
Undoubtedly, the bottleneck of the reconstruction is the lack of information about the state of the larynx. The soft tissues can change during a person’s life. In particular, under the influence of diseases or bad habits. Naturally, we did not have such information and it is not possible to obtain it. Therefore, we define our project very carefully – it is more experimental, creative hypothesis than a scientific reconstruction.
Although science and technology do not stand still. Who knows, maybe later our hypothesis will be 100% confirmed. In principle, it is possible to check the validity, strictly scientifically prove or disprove the effectiveness of the methodology we have chosen today. To do it, you need to spend four years and a huge budget for the reverse reconstruction.
That is, to record the voice of a person living now, make an impression of his face, build a 3D model on its basis, take a description of his voice, made by friends, relatives, etc. If we do everything in the reverse order and get identical voices, then the method works and can be considered scientific. We will be happy to share our developments with a company that decides to implement this project as a scientific one.
VOICES OF ARTISTS AND KINGS
It must be said that it is the first experiment of this kind in Ukraine. There have been isolated attempts in the world to reconstruct the voices of people who have been dead for a very long time. But scholars regard them as some kind of curiosity, ironic performances. For example, there was an attempt to restore the larynx of the mummy of the herald of the Egyptian pharaoh and simulate the sound of an ancient spell. Do you remember the Dutch project to resurrect the voice of Rembrandt?
It was recreated using the computer programs of the special services, which are used to combat telephone terrorism. They help to identify the voice, as well as ethnicity, education, social status, region of origin of its speaker. Resurrecting the voice of Rembrandt, the neural network did the whole procedure in reverse order.
It was provided with information about the area where the great artist was born and lived, about his social status, etc. The main reproaches to the authors of the project were that Dutch in the 17th century sounded differently, and the neural network was “trained” by the special services precisely for the modern version of the language.
Another attempt in 2018 was made by the French channel Plus: as an advertising move for the new season of Versailles. The marketers came up with the idea of reconstructing the voice of Louis XIV. We also studied this experience in Ukraine, but, unfortunately, we did not have a holistic understanding of the methodology used by the French.
SUCH AN IMPORTANT “LITTLE”
It is difficult to say what is more in such projects – PR or scientific method. Channel Plus has to take its word for it – it is impossible to find out how things really are. Therefore, we tried to make all the nuances of our methodology as public as possible.
Of course, we initially understood that we would face some criticism. But for us it was more important that people did not only hear how Shevchenko’s voice could sound, but also imagined what technologies were behind it.
What a great research was done by us and the staff of the National Museum. This experience is worth of a separate study. After all, no one has ever made such attempts – to collect in a single information bank all the descriptions of how the voice of Taras Shevchenko sounded.
A LIVING GENIUS AGAINST MYTH
In our opinion, from the point of view of school education, as a tool for the development of creative thinking of schoolchildren and teachers, this project is very important. Kobzar’s voice is a symbolic resurrection of his presence in modern times. In such a presentation, Shevchenko’s work becomes clearer and closer to the modern Ukrainian, and he himself turns from an abstract academic character into a living and voluminous personality.
We talked a lot on this topic with the director of the National Museum. Shevchenko by Dmitry Stus, arguing that the demythologized artist will do much more for the modernization and development of Ukrainian culture than the formal image of the poet as “the father of the nation”. The living Kobzar is our gateway to a new, viable cultural discourse.
Therefore, reproaches that the reconstructed voice of Kobzar sounds too simple, like the voice of an ordinary Ukrainian living in the neighborhood, I take as a compliment. After all, it is indeed the case. Shevchenko’s voice is, in a sense, the voice of each of us. You can’t imagine what delight the heroes of our casting were overwhelmed when they found out that they were possible Shevchenko’s doubles. Some even admitted that they themselves felt it before.
SHEVCHENKO’S GENE
Shevchenko’s gene is so deeply present in the DNA of Ukrainian people that, probably, there is no other such moral, cultural, political – or whatever – authority for the nation. It is a very strong gene, a prototype of whom a Ukrainian should be, regardless of the era and place in which he lives. Our experiment on the reconstruction of Kobzar’s voice from the point of view of the media effect has become a very successful case.
The news spread throughout the media. But I would like it to become also a cultural phenomenon, whose life in the Ukrainian cultural space will be long and bright. A simple example – even the voiceover of announcements at the Shevchenko metro station in a reconstructed voice could greatly increase the possibilities of implementing the media component of the project in the realities of Ukrainian life.
WILL ANCIENT PRINCES SPEAK?
Our experience with Shevchenko’s voice can have a very bright and deeply meaningful continuation. Thanks to the outstanding sculptor-anthropologist Mikhail Gerasimov, we have images of many ancient Russian princes. Perhaps among the most famous works of Gerasimov to the general public is the sculptural image of Yaroslav the Wise.
Nowadays, a huge experience has been accumulated in recreating a documentary portrait of a historical person from his skull. Modern digital technology makes these techniques even more effective. It could be a very ambitious and significant cultural and historical project – if Ukrainian history spoke with the voices of its grand dukes.
Photos courtesy of Tabasco Group