“No one person can look at all these data. But if we can teach computers to filter out the chaff, analysts might be able to zero in on key indicators faster and more thoroughly.”
– Barry Chen
Images, video, and text abound in this multimodal, data-rich world, but finding interesting and relevant samples from unannotated data remains a challenge. For example, how could one quickly find all the instances of “toddlers learning to catch a ball” among millions of hours of untagged video clips? The stakes are higher when the goal is preventing rogue states or nefarious actors from building a nuclear weapon.
LLNL computer scientist Barry Chen leads a team developing new deep learning algorithms that map images, video, and text into a joint semantic feature space where conceptually related items are proximal. Dubbed the “Semantic Wheel,” this approach allows analysts to quickly find multimodal data conceptually related to a query. The project involves developing self-supervised unimodal (i.e., individual imagery, video, or text modality) feature-learning algorithms that enable learning of high-quality transferable representations from the virtually limitless supply of unlabeled data.
Chen’s team is also developing the multimodal learning algorithms that merge the unimodal representations into a shared multimodal semantic feature space. To make this vision a reality, the project tackles new scalable training algorithms that take advantage of LLNL’s world-class supercomputers for rapidly training large neural networks on massive data sets.
Team Acknowledgments
Pictured, left to right: Back row: Doug Poland, Maggie Arno, Yana Feldman, Sam Jacobs, Brenda Ng, Andy Yoo, and Juanita Ordonez. Front row: Brian Van Essen, Lance Kim, David Hysom, Jae-Seung Yeom, Tim Moon, Barry Chen, T. Nathan Mundhenk, Carmen Carrano, George Anzelon, and David Buttler.
Not pictured: Ryan Ball (intern), Nikoli Dryden (subcontractor), Jaeyoung Choi (subcontractor), Stefan Hau-Riege, Cynthia Lai (intern), Sahitya Mantravadi (intern), Paul L. Rawson, and Michael Zelinski.
