Using Intelligent Workflows to Analyze Artistic Style

Fitchburg, MA, 19 May 2017

Alumnus Ricky J. Sethi and his colleagues at Fitchburg State University in Massachusetts are using ISI's WINGS workflow system for art history in the WAIVS (Workflows for Analysis of Images and Visual Stylometry) project. WAIVS workflows were demonstrated at a workshop held at the Fitchburg Art Museum on May 19, 2017.

The WAIVS project is funded by a grant from the National Endowment for the Arts. The principal investigators are Sethi, assistant professor in the Computer Science at Fitchburg State, and colleagues Catherine A. Buell, assistant professor in the Mathematics Department, and William P. Seeley, a lecturer in the Philosophy Department at Bates College. Other project members include RaghuRam Rangaraju and Jake Lee, both computer science students at Fitchburg State. The project is in collaboration with Dr. Mary M. Tinti of the Fitchburg Art Museum, Dr. Yolanda Gil of the USC Information Sciences Institute, and Dr. Charlene Villaseñor Black of the department of art history at UCLA.

WAIVS project members, from left to right: RaghuRam Rangaraju a graduate student at Fitchburg State University; Ricky Sethi, assistant professor of computer science at Fitchburg State University; Catherine Buell, assistant professor of mathematics at Fitchburg State University; William Seeley, lecturer in philosophy at Bates College; and Jake Lee, undergraduate student at Fitchburg State University. On the far right is WAIVS collaborator Charlene Villaseñor-Black, a professor of art history at UCLA.

The focus of the WAIVS project is in visual stylometry, an emerging field that applies image analysis and machine learning tools to digital artwork for art analysis and investigation. Although there have been tremendous advances in the field of image processing that are relevant to visual stylometry, they are not very accessible to art historians. To address this, WAIVS is using workflows to provide an accessible visual programming interface that simply shows how the data is generated and used by different computational steps. Workflows effectively capture complex multi-step data analysis methods in a simple dataflow graph. WAIVS is using the WINGS workflow system, developed by Gil's group at ISI, because it adds intelligent reasoning to workflows.

WINGS is a unique workflow system that uses artificial intelligence planning techniques and semantic web languages to capture expert knowledge about setting up the parameters that control the image analysis algorithms, so that users can get recommendations of parameter settings to create valid workflows that work best with their data.

Sethi, who is an expert in video processing, developed workflows that include state-of-the-art methods such as deep learning and convolutional neural networks to analyze images. Sethi's postdoctoral research at ISI was under a prestigious NSF Computing Innovation Fellows award. During that time, he collaborated with Gil on combining text and image analysis workflows to detect human trafficking by analyzing personal ads in Web sites. They just published a paper about the use of deep learning techniques in workflows to capture artistic style, which will appear later this Spring in the Future Generation Computer Systems journal.

Using WINGS, WAIVS image processing experts create workflows that capture state-of-the-art image processing techniques. Current workflows created by the WAIVS group include entropy calculation, discrete tonal analysis, and convolutional neural networks. Art historians learned to use these workflows during the workshop.

ISI alum Ricky Sethi gives a presentation of WINGS workflows to analyze artistic style. Sethi, who was a postdoctoral researcher at ISI under a prestigious CRA Fellows scholarship, is an expert in video and image processing. The workshop was held with a backdrop of the exhibit of Lionel Reinford, a well-known local painter.

The 2017 WAIVS workshop was attended by more than twenty art historians, mostly in the New England area, and was supported by the American Society for Aesthetics and the New England Museum Association. The workshop was held in the room that hosts the exhibit of Lionel Reinford, a well-known local painter. The discussions centered on possible approaches to quantifying artistic style. As Sethi, Buell, and other WAIVS project members demonstrated workflows to compute the entropy of a painting and other quantitative ways to represent a painting, art historians discussed the possibilities of using such measures to design more formal descriptions of artistic style.

The first talk was by Daniel Graham of Hobart and William Smith College, who discussed the neurobiological aspects of artistic style. Next, Gil gave an introduction to workflows and to the WINGS intelligent workflow system.

Seeley discussed the origins of the WAIVS project as a collaborative teaching exercise with Buell at Bates College. The goal of the initial project was to foster interdisciplinary collaboration among undergraduates in the humanities, mathematics, and computer science. Seeley mentioned that the initial choice of focus on Hudson River School and Impressionist landscape paintings was strategic. The particular Hudson River school landscape images in the set were chosen because they share a similar general composition and palette that can be traced to earlier seventeenth, eighteenth, and nineteenth century Dutch and English landscapes. This ties the work to E. H. Gombrich's research on the development of artistic style. Further, all of the works chosen are in the public domain and available via online archives like WikiArt. These works represent styles that are familiar and well represented in art museums. This makes WAIVS accessible as a teaching exercise for students, researchers and the broader public. Finally, the choice of paintings with similar palettes and composition, as well as the choice to contrast Hudson River School and Impressionist paintings, was designed to test an initial hypothesis that texture information, which is indicative of differences in brushstroke styles, would be sufficient to classify artworks by school and individual artists.

Catherine Buell of the mathematics department at Fitchburg State University shows workshop participants how a convolutional neural network learns from examples of Van Gogh's artwork about his artistic style (left of her slide), and can then render any image (middle of her slide) using the distinctive strokes and colors of the Dutch painter (right of her slide).

Sethi, Buell, and their students gave a demonstration of the WAIVS system, and guided participants through several practical exercises to use WAIVS workflows to analyze a variety of paintings, some of them from a current exhibit a the host museum.

Some WAIVS workflows capture interesting quantitative measures of an image's characteristics. For example, one of the workflows generates an entropy value and an entropy image, allowing art historians to compare different paintings in terms of their entropy levels.

A workflow that generates an entropy value and an entropy image Church's The Heart of the Andes (1859).

Another workflow uses a convolutional neural network, and is trained with examples of a painter's artwork (the style images) to then render any image (the content image) using the distinctive strokes and colors of that painter. This is based on a technique developed by Leon Gatys, Alexander Ecker, and Matthias Bethge from Tübingen in Germany in 2015. The WINGS workflow was implemented using the Torch open source software for deep learning. The components of these workflows can be linked together to create different analyses.

The workflow GenerateStylizedImages uses convolutional neural networks to process two separate paintings (the style images) and renders an image (the content image) in the style of those paintings. Comparing the two resulting synthetic stylized images helps art historians contrast the styles of the paintings.
An illustration of how convolutional neural networks in the GenerateStylizedImages workflow are used to transform Monet's The Railroad bridge in Argenteuil (1873) (bottom middle) into the painterly styles of Munch's The Scream (right) and Van Gogh's The Starry Night (left). The resulting synthetic stylized images are shown at the top.
Workshop participants used the GenerateStylizedImages workflow to compare a painting from Shelley Reed (A) with a painting by Edwin Landseer (B) that she appropriated in her painting, both used as input style images. The content image from Frederick Church (bottom) was used to generate two synthetic stylized images (top), which expose stylistic differences in the strokes of the tiger's stripes were painted in the Reed and the original Landseer paintings. The starker tonal contrasts of the Reed painting are also evident in the way the waterfall and the sky have been depicted in the two stylized images.

Workshop participants worked with images by contemporary painter Shelley Reed, the subject of a current exhibit at the museum. Reed appropriates imagery from seventeenth, eighteenth, and nineteenth century Northern European painters in her works. Workshop participants to learn how to use the WAIVS software to evaluate differences in artistic style between Reed's paintings and the earlier paintings. Participants used the GenerateStylizedImages workflow with the Cropped grayscale versions of A) Edwin Landseer's Portrait of Mr. Van Amburgh, as He Appeared with His Animals at the London Theatres (1847) and B) Shelley Reed's Tiger (after Landseer and Thiele) (2007) as the style images. Frederick Church's Heart of the Andes (1859) was used as the content image. The resulting synthetic images, A) Stylized Landseer and B) Stylized Reed, reflect several interesting stylistic differences between the original Landseer and Reed paintings. The most striking can be seen in the grove of trees in the foreground right of the paintings. The trees are rendered in more tightly packed and sharply articulated stripes in the Stylized Reed than the Stylized Landseer. This difference recapitulates differences in the way that the tiger's stripes were painted in the Reed and the original Landseer paintings. The starker tonal contrasts of the Reed painting are also evident in the way the waterfall and the sky have been depicted in the two stylized images.

Fitchburg State University student Jake Lee shows the "Making WAIVS exhibit" at the Fitchburg Art Museum in Fitchburg, Massachusetts. Museum goers interact with workflows to analyze paintings.

Workshop attendees also had the opportunity to examine Reed's artistic style in the exhibit Curious Nature, running at the Fitchburg Art Museum (February 12 - June 4). A demonstration version of WAIVS is currently available for use by the general public in association with the Reed exhibition. The exhibit materials are also offered in Spanish to appeal to the local latino population.

Workshop participant John Kulvicki, from the Department of Philosophy at Dartmouth, analyzed paintings with WINGS workflows on a mobile phone. He is interested in understanding the subjectivity of artistic style.
Workshop participants John Garton of Clark University, Amy Schlegel, and Valerie Kinkade of Museum and Collector Resource, run workflows to analyze the artistic style of Albert Bierstadt's 1895 painting "The Morteratsch Glacier, Upper Engadine Valley, Pontresina," as WAIVS student Jake Lee looks on.

Workshop participant John Garton of Clark University proposed using workflows to understand the 3D effect on color when paintings have texture that changes how the color is reflected on the 3D structure. He explained how El Greco used lapis in the mixes he did for blues, giving his paintings unique color effects. Workshop participants Valerie Kinkade of the Museum and Collector Resource and Amy Schlegel discussed how art historians collect mass spectrometry to understand the chemical composition of the pigments, as well as stratigraphy data about the paint thickness and its 3D structure. This kind of data opens the door to new research to analyze that kind of data and the 3D effects on the perception of color in paintings. Kinkade also saw applications in legal aspects of copyright infringement of paintings.

Charlene Villaseñor-Black, a professor of art history at UCLA, presented examples of early uses of technology as a tool by painters, and discussed the potential of workflows and computer vision tools to help art historians think differently about style, and to open the doors for students to learn about visual style in a more analytical way.

Charlene Villaseñor-Black, a professor in the department of art history at the University of California Los Angeles, discussed early uses of technology as a tool by painters, exemplified by the use of camera obscura by Vermeer and Caravaggio, and the different levels of detail designed to reflect the eye's perception in the forefront figures of Las Meninas from Velazquez. She discussed the potential of workflows and computer vision tools to help art historians think differently about style, and to open the doors for students to learn about visual style in a more analytical way.

Sethi was particularly proud to see this workshop come together. "My wife is a historian, and I see first hand how challenging it is for people in the humanities to access the powerful technologies for data science that are available today. Ever since I started to use WINGS at ISI, I could see that workflows can be a game changer for historians. For art historians in particular, workflows can bring very sophisticated tools from image processing into their hands, and allow them to experiment with different mathematical measures of the properties of an image that they could then ascribe to artistic style."

Gil, who uses WINGS workflows to teach data science to non-computer science students at USC, was not surprised that the art historians were able to run sophisticated quantitative analyses on paintings. "What is unique about the WAIVS project is the use of methods from computer vision in order to give quantitative definitions of technical terms in art history," she said. "This project is visionary in bringing recent revolutionary deep learning AI techniques to quantify the study of art, and putting them squarely in the hands of humanities researchers."

"I am impressed by WAIVS and its potential to revolutionize the way we look, the way we think, the way we see images" Villaseñor-Black underscored. "The WAIVS tool is able to do things with images that art historians cannot do, such as measure entropy or remove the chromatic value from the foreground, or transfer what it calls 'style' from one image to another. These are not skills that art historians are trained in, or things we can currently do, and they have the potential to radically change how we look and think about style."