Computer History Museum Shared How they are Using Machine Learning to Enhance Digital Media Collections

June 14, 2022

IMLS Supervisory Grants Management Specialist Helen Wechsler interviewed Computer History Museum Chief Information Officer and Vice President of Technology, Dave Evans, and Senior Director, Collections & Archives, Paula Jabloner, to discuss their use of an IMLS National Leadership Grant award to assist the museum field. The interview has been edited for length and clarity.

Dave Evans 

Helen: Can you summarize your National Leadership for Museums Rapid Prototyping project?

Dave: The Computer History Museum (CHM) is engaged in a long-term strategy to strengthen its technology infrastructure to reach new audiences and better serve existing ones.

This project utilized a new commercial machine learning product, Microsoft Cognitive Services (MCS), to assign metadata cheaply and quickly to a subset of the Museum’s digital media collection, as well as perform transcription and language translation, and create a prototype proof-of-concept (POC) search portal for user testing. The aim of the POC search portal was to illuminate the extent to which machine learning-created metadata can make digital collection materials more accessible and discover connections between artifacts. CHM worked with Microsoft and Mobile Programming, a digital prototype development company, to design and build the portal. User feedback indicated that certain machine-learning-enabled capabilities, such as transcription and language translation, provided significant benefits to users.

Helen: What problem would be solved for museums by using machine learning to assign metadata (descriptive tags) to their digitized collections? What is the dream future?

Paula: Most libraries, archives & museums have a backlog in providing multiple access points or any access to their collections. The most significant problem solved by using machine learning is providing enhanced access and therefore use. This is especially relevant for AV collections, which can often be a sort of black box in terms of understanding. Unless an organization has the people resources to watch and catalog every video in its entirety, important information about the video’s content, speakers, etc. won’t be captured. For digitized video, machine learning creates an automatic transcript of the recording, which our system treats as a metadata field. It further extracts metadata from the transcript: keywords, topics, and people. This opens a whole new world of access: the ability to search through 100s if not 1,000s of moving images or audio files at once. An added benefit of automatic transcription is the ability to translate transcriptions on the fly to open access to individuals worldwide.

A dream future allows for whole new ways to discover and use collections, especially in the AV realm, which is so important in the internet age. I’m very excited about the ability to find hidden materials that might not have been the main focus of a video as expressed in the title only. This technology could help bring to the fore more relevant material on women and people of color or uncover the beginnings of an innovative line of thinking within the computer industry earlier than currently known based on the machine-generated tags and transcript.

Video search results for the search term "Alto". The search index was machine-generated by transcribing the audio track to text. (Photo courtesy of the Computer History Museum).
 

Helen: You tested this idea using some off the shelf machine learning software. What were the biggest successes and the biggest failures of this testing?

Dave: The biggest benefit of using off the shelf software is the low barrier to trying out new concepts without the overhead of setting up infrastructure and the minimization of custom development. Machine learning is a great example of that. We used Microsoft Azure Cognitive Services, which has a rich set of APIs to perform machine learning: things like object recognition, face recognition, video transcription, language translation and much more.

Among many other items in our collection, we have over a hundred terabytes of video and being able to process the video using machine learning was a big success. Not only could we create written transcripts of the content with relatively high accuracy rates, but we could also translate the content into multiple other languages. This is significant for us as we aim to reach a much broader, global audience. Being able to share content in the native language of the guest makes our content more valuable and the experience richer.

One of the biggest failures was our inability to create a visual knowledge graph of relationships in our content. We envisioned a visual map that showed relationships mined from the video, displaying connections between people, companies, products, etc. Due to the vast amount of data and, as well as budget and timeline constraints, we were not able to incorporate this feature into the project. However, it was still a very valuable exercise as we learned more about the capabilities of the technology.

Helen: What needs to happen to make machine learning a real benefit to museums as they seek to add the kind of information to databases of digital collections that would make them more accessible for the public?

Dave: Machine learning is still daunting for a lot of non-technical organizations. Museum professionals focus on things such as the collecting, curating, and preservation of artifacts. Many Galleries, Libraries, Archives, and Museums (GLAM) organizations do not have large (in some cases any) IT resources. Tools, in general, need to be accessible, easy to use, and intuitive. Simple but powerful user interfaces are needed to obfuscate the complexity of the backend.

Many out-of-the-box collection databases are not built both for maintaining the machine learning-created data and providing a useful online search interface. Over time the algorithms will improve, and this will require the institution to constantly reinvest in running their collections through the machine learning algorithm.

Helen: How did the IMLS NLG grant help this project?

Paula: The IMLS grant was extremely instructive for the collections department to understand how machine learning might impact collections and especially our workflows and procedures. Going through the steps of running the machine learning on a subset of our collection and conceptualizing a new online access tool really got us thinking in much broader terms as we move forward with making priority decisions about cataloging, access, and acquisitions. We are more likely to acquire and therefore preserve AV collections knowing that we can make them accessible much quicker and with better means of understanding their content. We can streamline some cataloging tasks, though definitely not all, by working hand-in-glove with machine learning.

In broad terms, as we implement a more advanced collections management system in which we plan to embed online storytelling and browse features, the learnings from this small grant will reverberate across the decisions we make about access and how to embed machine learning into our catalog.

You can read more about the Computer History Museum’s machine learning project in their blog "A Museum’s Experience with AI."