Alumnus Ricky J. Sethi and his colleagues at Fitchburg State University in Massachusetts are using ISI's WINGS workflow system for art history in the WAIVS (Workflows for Analysis of Images and Visual Stylometry) project. WAIVS workflows were demonstrated at a workshop held at the Fitchburg Art Museum on May 19, 2017.
The WAIVS project is funded by a grant from the National Endowment for the Arts. The principal investigators are Sethi, assistant professor in the Computer Science at Fitchburg State, and colleagues Catherine A. Buell, assistant professor in the Mathematics Department, and William P. Seeley, a lecturer in the Philosophy Department at Bates College. Other project members include RaghuRam Rangaraju and Jake Lee, both computer science students at Fitchburg State. The project is in collaboration with Dr. Mary M. Tinti of the Fitchburg Art Museum, Dr. Yolanda Gil of the USC Information Sciences Institute, and Dr. Charlene Villaseñor Black of the department of art history at UCLA.
The focus of the WAIVS project is in visual stylometry, an emerging field that applies image analysis and machine learning tools to digital artwork for art analysis and investigation. Although there have been tremendous advances in the field of image processing that are relevant to visual stylometry, they are not very accessible to art historians. To address this, WAIVS is using workflows to provide an accessible visual programming interface that simply shows how the data is generated and used by different computational steps. Workflows effectively capture complex multi-step data analysis methods in a simple dataflow graph. WAIVS is using the WINGS workflow system, developed by Gil's group at ISI, because it adds intelligent reasoning to workflows.
WINGS is a unique workflow system that uses artificial intelligence planning techniques and semantic web languages to capture expert knowledge about setting up the parameters that control the image analysis algorithms, so that users can get recommendations of parameter settings to create valid workflows that work best with their data.
Sethi, who is an expert in video processing, developed workflows that include state-of-the-art methods such as deep learning and convolutional neural networks to analyze images. Sethi's postdoctoral research at ISI was under a prestigious NSF Computing Innovation Fellows award. During that time, he collaborated with Gil on combining text and image analysis workflows to detect human trafficking by analyzing personal ads in Web sites. They just published a paper about the use of deep learning techniques in workflows to capture artistic style, which will appear later this Spring in the Future Generation Computer Systems journal.
Using WINGS, WAIVS image processing experts create workflows that capture state-of-the-art image processing techniques. Current workflows created by the WAIVS group include entropy calculation, discrete tonal analysis, and convolutional neural networks. Art historians learned to use these workflows during the workshop.
The 2017 WAIVS workshop was attended by more than twenty art historians, mostly in the New England area, and was supported by the American Society for Aesthetics and the New England Museum Association. The workshop was held in the room that hosts the exhibit of Lionel Reinford, a well-known local painter. The discussions centered on possible approaches to quantifying artistic style. As Sethi, Buell, and other WAIVS project members demonstrated workflows to compute the entropy of a painting and other quantitative ways to represent a painting, art historians discussed the possibilities of using such measures to design more formal descriptions of artistic style.
The first talk was by Daniel Graham of Hobart and William Smith College, who discussed the neurobiological aspects of artistic style. Next, Gil gave an introduction to workflows and to the WINGS intelligent workflow system.
Seeley discussed the origins of the WAIVS project as a collaborative teaching exercise with Buell at Bates College. The goal of the initial project was to foster interdisciplinary collaboration among undergraduates in the humanities, mathematics, and computer science. Seeley mentioned that the initial choice of focus on Hudson River School and Impressionist landscape paintings was strategic. The particular Hudson River school landscape images in the set were chosen because they share a similar general composition and palette that can be traced to earlier seventeenth, eighteenth, and nineteenth century Dutch and English landscapes. This ties the work to E. H. Gombrich's research on the development of artistic style. Further, all of the works chosen are in the public domain and available via online archives like WikiArt. These works represent styles that are familiar and well represented in art museums. This makes WAIVS accessible as a teaching exercise for students, researchers and the broader public. Finally, the choice of paintings with similar palettes and composition, as well as the choice to contrast Hudson River School and Impressionist paintings, was designed to test an initial hypothesis that texture information, which is indicative of differences in brushstroke styles, would be sufficient to classify artworks by school and individual artists.
Sethi, Buell, and their students gave a demonstration of the WAIVS system, and guided participants through several practical exercises to use WAIVS workflows to analyze a variety of paintings, some of them from a current exhibit a the host museum.
Some WAIVS workflows capture interesting quantitative measures of an image's characteristics. For example, one of the workflows generates an entropy value and an entropy image, allowing art historians to compare different paintings in terms of their entropy levels.
Another workflow uses a convolutional neural network, and is trained with examples of a painter's artwork (the style images) to then render any image (the content image) using the distinctive strokes and colors of that painter. This is based on a technique developed by Leon Gatys, Alexander Ecker, and Matthias Bethge from Tübingen in Germany in 2015. The WINGS workflow was implemented using the Torch open source software for deep learning. The components of these workflows can be linked together to create different analyses.
Workshop participants worked with images by contemporary painter Shelley Reed, the subject of a current exhibit at the museum. Reed appropriates imagery from seventeenth, eighteenth, and nineteenth century Northern European painters in her works. Workshop participants to learn how to use the WAIVS software to evaluate differences in artistic style between Reed's paintings and the earlier paintings. Participants used the GenerateStylizedImages workflow with the Cropped grayscale versions of A) Edwin Landseer's Portrait of Mr. Van Amburgh, as He Appeared with His Animals at the London Theatres (1847) and B) Shelley Reed's Tiger (after Landseer and Thiele) (2007) as the style images. Frederick Church's Heart of the Andes (1859) was used as the content image. The resulting synthetic images, A) Stylized Landseer and B) Stylized Reed, reflect several interesting stylistic differences between the original Landseer and Reed paintings. The most striking can be seen in the grove of trees in the foreground right of the paintings. The trees are rendered in more tightly packed and sharply articulated stripes in the Stylized Reed than the Stylized Landseer. This difference recapitulates differences in the way that the tiger's stripes were painted in the Reed and the original Landseer paintings. The starker tonal contrasts of the Reed painting are also evident in the way the waterfall and the sky have been depicted in the two stylized images.
Workshop attendees also had the opportunity to examine Reed's artistic style in the exhibit Curious Nature, running at the Fitchburg Art Museum (February 12 - June 4). A demonstration version of WAIVS is currently available for use by the general public in association with the Reed exhibition. The exhibit materials are also offered in Spanish to appeal to the local latino population.
Workshop participant John Garton of Clark University proposed using workflows to understand the 3D effect on color when paintings have texture that changes how the color is reflected on the 3D structure. He explained how El Greco used lapis in the mixes he did for blues, giving his paintings unique color effects. Workshop participants Valerie Kinkade of the Museum and Collector Resource and Amy Schlegel discussed how art historians collect mass spectrometry to understand the chemical composition of the pigments, as well as stratigraphy data about the paint thickness and its 3D structure. This kind of data opens the door to new research to analyze that kind of data and the 3D effects on the perception of color in paintings. Kinkade also saw applications in legal aspects of copyright infringement of paintings.
Charlene Villaseñor-Black, a professor in the department of art history at the University of California Los Angeles, discussed early uses of technology as a tool by painters, exemplified by the use of camera obscura by Vermeer and Caravaggio, and the different levels of detail designed to reflect the eye's perception in the forefront figures of Las Meninas from Velazquez. She discussed the potential of workflows and computer vision tools to help art historians think differently about style, and to open the doors for students to learn about visual style in a more analytical way.
Sethi was particularly proud to see this workshop come together. "My wife is a historian, and I see first hand how challenging it is for people in the humanities to access the powerful technologies for data science that are available today. Ever since I started to use WINGS at ISI, I could see that workflows can be a game changer for historians. For art historians in particular, workflows can bring very sophisticated tools from image processing into their hands, and allow them to experiment with different mathematical measures of the properties of an image that they could then ascribe to artistic style."
Gil, who uses WINGS workflows to teach data science to non-computer science students at USC, was not surprised that the art historians were able to run sophisticated quantitative analyses on paintings. "What is unique about the WAIVS project is the use of methods from computer vision in order to give quantitative definitions of technical terms in art history," she said. "This project is visionary in bringing recent revolutionary deep learning AI techniques to quantify the study of art, and putting them squarely in the hands of humanities researchers."
"I am impressed by WAIVS and its potential to revolutionize the way we look, the way we think, the way we see images" Villaseñor-Black underscored. "The WAIVS tool is able to do things with images that art historians cannot do, such as measure entropy or remove the chromatic value from the foreground, or transfer what it calls 'style' from one image to another. These are not skills that art historians are trained in, or things we can currently do, and they have the potential to radically change how we look and think about style."