Communications in Information and Systems

Volume 23 (2023)

Number 3

Special issue dedicated Professor Avner Friedman in celebration of his 90th birthday

Guest Editors: Jong-Shenq Guo, Bei Hu, Robert Jensen, and Stephen S.-T. Yau

Prediction of human looking behavior using interest-based image representations

Pages: 245 – 262

DOI: https://dx.doi.org/10.4310/CIS.2023.v23.n3.a2

Authors

Jong-Shenq Guo (Department of Mathematics, Tamkang University, Tamsui, New Taipei City, Taiwan)

Karen Guo (Department of Data Science and Big Data Analytics, Providence University, Taichung, Taiwan)

Paul Schrater (Department of Computer Science, University of Minnesota, Minneapolis, Mn., U.S.A.)

Abstract

Looking behavior allows human to understand and interact with an enormous amount of information, a capacity challenging to replicate in AI systems. One of the core elements of this work is an effort to predict scan-paths from a combination of image information and past looking behavior. The success of this scan-path predication relies heavily on whether this image information can provide a sufficiently rich representation for prediction. In this paper, we show that changing representations dramatically simplifies and improves predictions of looking behavior. We introduce a representation of looking behavior that centers around interest-regions in images, defined by natural and collective looking behavior. These regions (called interest-based regions) can be used to partition images for semantic labeling and to provide a basis for shared representation across observers. Without any additional label or image information, we achieve highly accurate sequence prediction using this interest-based image representation.

Full Text (PDF format)

This work was supported in part by the Ministry of Science and Technology of Taiwan under the grants 111-2115-M-032-005 (JSG) and 111-2115-M-126-001 (KG). We would like to thank the anonymous referee for some valuable comments. Also, appreciation for the help from Ching-Yao Chen, Yu-Fang Chou, and Chiao-Ying Cheng from Providence University, who provided help on running and evaluation of MultiMatch on SALICON and MIT1003.

Received 6 December 2022

Published 12 October 2023