The digital platform of Mikhail Shemyakin's Imaginary Museum

‍

Imagine the fusion of art and technology. Artist and researcher Mikhail Shemyakin has dedicated 60 years to creating the Imaginary Museum, and our team has brought this vision online. The outcome? A dynamic digital platform that not only showcases artworks but also serves as a comprehensive research facility, enhanced by neural network technology.

This project aims to trace the evolution of specific images, ideas, and artistic methods throughout history, offering insights into their variations, symbolism, and philosophical underpinnings. 

Where it all started

In 2021, together with the Non-profit Foundation for the Preservation and Digital Transformation of the World Cultural Heritage, we started work on the creation of the digital platform of Mikhail Shemyakin's Imaginary Museum.

Mikhail Shemyakin is an artist, sculptor, theater practitioner, and researcher. Over 60 years, he has collected over 3 million reproductions of paintings in the Château de Champs-sur-Marne in France. All of this was done to demonstrate how the same image, symbol, or form can transform art across different eras and cultures.

It is a very large-scale research — 3 million reproductions and thousands of themes: "Steps, ladders, stairs", "Wrapped figures", "Hands", "Scream", "Blurred images" and others.

"Imaginary Museum" is a project with amazing scientific and educational potential. Art historians and artists can use the research materials in scientific and artistic works, students of creative universities and schools — as educational material, and representatives of other industries — to develop skills of analysis and synthesis, to pump up their vision and sense of harmony.

Mikhail Shemyakin once said that there are actually no real serious professional books on visual art in the world.

"You can buy a monograph on Leonardo Da Vinci in any store. But you will never find books like "The Ball in Art", "The Hand in Fine Art", "The Image of Death in Graphics and Painting"... There is a lack of generalization, and philosophical understanding of art", — Mihail Chemiakin, artist and sculptor, cultural and historical researcher

In 2002, in one of the episodes of a series of movie-lectures for the TV channel "Culture", Mikhail Shemyakin uttered a prophetic phrase: "All this must be digitized! We must do it and transfer it into digital form".

However, the idea of digitizing the Imaginary Museum could only be realized 20 years later.

Material Archive and The Digital Platform

All these years the research was conducted exclusively in the analog format: Mikhail Shemyakin cut out illustrations from books, newspapers, and art magazines, analyzed them, identified similar motifs, developed themes, and, following the author's method, pasted them on sheets of thick cardboard, which were sent to thematic folders. This particular collection is kept in the Château de Chamusseau in France.

"For us, this project is an opportunity to touch the unique exploration of visual art culture that Mikhail Shemyakin has made. Moreover, we got an amazing chance to open this heritage to the world. There are practically no similar art projects in the world, especially of this scale. This is truly a cultural project with global significance", —  Ilia Samofeev, co-CEO red_mad_robot

The colossal significance of the research was clear from the beginning. Back in 2019, the red_mad_robot team traveled to Chateau Chamusso to see Shemyakin's heritage live, learn the process of working with the research, and discuss the possibilities of creating a digital platform. Then a pandemic hit and the work had to be postponed. 

At the end of 2021, we returned to the project and started to realize it. The French company ARKHENUM, which had previously digitized the collections of the Louvre and other major museums, was hired to digitize the archive. In six months, more than 700 folders of the Imaginary Museum were scanned and a cloud archive of 200 thousand images was created.

"Interestingly, part of the project team was in Russia, part was scattered around the world, the product owner was in the heart of France in Chamusseau Castle, and the French company ARKHENUM was in Bordeaux. In addition, the Internet in Chamusseau Castle was as fast as it could be in an ancient castle: to put it mildly, not the fastest", — Ilay Galtsin, Project Manager red_mad_robot

After digitizing the archive, the team set out to create a digital platform that would help preserve Shemyakin's research, augment it with digital tools, and open it up to the first users.

Platform development

"The main difficulty in developing the platform was the fact that there were a lot of images in the archive — 200 thousand. Some files weighed 300 megabytes", — Ilia Trusov, Lead Backend Developer red_mad_robot

Smart search and neural networks

We wanted to optimize work with a large number of images and tried to "digitize" Mikhail Shemyakin's method. We decided to create a convenient archive search for this purpose and started working out the idea using the latest artificial intelligence technologies.However the neural networks turned out to be not that smart — although they recognize objects in images perfectly, the algorithm is not yet able to perform the kind of analysis that the artist himself does. This is what makes the "Imaginary Museum" Shemyakin's museum, not anyone else's. 

Therefore, it was decided to make the neural network an "invisible" assistant inside the search engine. The task of the neural network is to analyze images from one more point of view in addition to Shemyakin's reference view, to "reward" them with tags indicating objects in the image in the background, and to expand the search possibilities due to this. For this purpose, we chose the already existing CLIP neural network. 

CLIP (Contrastive Language-Image Pre-training) is an OpenAI neural network capable of processing text and images simultaneously. It is learning to associate text descriptions with corresponding images without explicitly teaching “image + text” pairs.

CLIP uses two built-in vocabularies: text and visual — 32 thousand unique words and 4 thousand unique objects and concepts. It analyzes the input data and "understands" how the image placed in it relates to the concepts from its own dictionary.

"For example, when you upload a picture of an Australian Shepherd puppy to CLIP, the neural network gets a fixed-length vector for it. Since it is trained on images and text simultaneously, it can get a vector for the text as well. These vectors can be compared to see how well they match each other. CLIP goes through the terms from the dictionary, picks the most appropriate ones, and "describes" what is in the image. In our case, it is the same "Australian Shepherd puppy", — Ivan Timofeev, Head of Development Department rdl by red_mad_robot

CLIP accuracy can vary, so queries had to be categorized into three categories: 

• simple vocabulary, 
• art history vocabulary, 
• Mikhail Shemyakin's specific vocabulary. 

We also used ruCLIP — the same model but with a Russian-language dictionary at its core. 

"We considered that ruCLIP would improve the quality of the search because the topics that Shemyakin categorized images by may not always be correctly translated into English. We pre-trained the neural networks on Shemyakin's archives and online art encyclopedia", — Liliya Kamalieva, Business Analyst rdl by red_mad_robot

After choosing a neural network, we began to think about how to approach Mikhail Shemyakin's analytical method. The neural network identifies the features of an image. Will it be able to identify their themes? To test the hypothesis, we created a sample of images with 32 themes: "Shrouded figure", "Gryphons", "Cubes", "Shadows", "Blurred images" and others. For each of them, the neural network generated a set of tags.

At the same time for clustering, we compiled a sample of 4 thousand images from the artist's collection — and this is what we got:

In all cases, only visually similar images were selected. And none of the applied algorithms could reproduce Shemyakin's own grouping by themes because of the difficulty of determining the attributes by which the artist combines images into themes.

How the platform operates

An important element of the Digital Museum is searching among the vast archive of visual research. For the user's convenience, the platform has two types of search: attributive and machine search (based on a neural network). The first one finds images by metadata, and the second one, in addition to searching by author and date of creation, can search for paintings based on their content and meaning. 

For example, you need to find a "spiral red staircase". Attribute search will fail if there is no such description in the metadata. The machine search will find images not just with stairs, but with red spiral stairs.

As part of the search, the neural network gives the user a list of tags that can prompt researchers to search for new semantic connections in the context of Mikhail Shemyakin's method. This list can be taken as a basis for the search and:

• find images with the same tags and isolate them into a separate subject;
• focus on key features and ideas that can be related to a particular theme; 
• develop a recommendation system: it narrows down the range of topics of interest, producing relevant results based on analysis.

For example, in this image, the neural network saw the tags "border guard", "sapper", "carpenter", "defender", "raft", "gangway", "lift", "railroad", "ladder" and "stairs". This gives research project staff space for new interpretations and experimentation with 'cues' from the neural network while demonstrating that the neural network has not yet been able to replicate the artist's analytical method.

How it works: design

The appearance of the digital museum reflects the essence of the project – images are at the head of everything because the work of researchers depends on them alone. For this purpose, we put images in the center of attention, and all other design elements serve as a shell, emphasizing the value and importance of the research subject. 

The images are placed in the grid in their entirety, not cropped or framed. All because every detail and every inscription is valuable to the research. This variety of proportions added dynamics to the whole page grid.

Inspired by Shemyakin's physical museum – the inscriptions on the boxes, the color of the sheets on which the artist glues the images, and his works – as well as looking at hundreds of museum and art gallery websites, we chose a minimalistic and academic design in beige and brown shades. 

We also used artifacts from the museum as a visual component to make the platform match Shemyakin's image. Even the Antiqua font with serifs was chosen for a reason – it not only catches the eye but also overlaps with the inscriptions on the author's sheets where Mikhail Shemyakin collected his research.

The result

"Mikhail Shemyakin's Imaginary Museum" is an inexhaustible resource for creating new exhibitions, books, and educational programs. It is interesting for researchers, practitioners (artists and designers), and a wide audience — everyone who wants to "train the eye" and develop their analytical skills.

"With the help of the digital platform and embedded neural network, visitors will be able to explore over 200,000 images from the "Imaginary Museum" and find inspiration in them. Art experts, on the other hand, can deepen their knowledge and apply this material to their intellectual and artistic endeavors", — Alexey Abakumov, Head of the Digital Imaginary Museum project, art historian