Dr. Yunhe Feng a fellow Infoseeking lab member and his team attended the Discovering AI@UW event co-hosted by the eScience Institute last week and presented his gender bias in image search work at the poster session.
This inspirational and educational event is a great networking opportunity for faculty, researchers, and students who are interested in using AI, finding synergies with major campus initiatives, and strategizing about the future of AI research at UW.
ExpScore: Learning Metrics for Recommendation Explanation
ABSTRACT Many information access and machine learning systems, including recommender systems, lack transparency and accountability. Highquality recommendation explanations are of great significance to enhance the transparency and interpretability of such systems. However, evaluating the quality of recommendation explanations is still challenging due to the lack of human-annotated data and benchmarks. In this paper, we present a large explanation dataset named RecoExp, which contains thousands of crowdsourced ratings of perceived quality in explaining ecommendations. To measure explainability in a comprehensive and interpretable manner, we propose ExpScore, a novel machine learning-based metric that incorporates the definition of explainability from various perspectives (e.g., relevance, readability, subjectivity, and sentiment polarity). Experiments demonstrate that ExpScore not only vastly outperforms existing metrics and but also keeps itself explainable. Both the RecoExp dataset and open-source implementation of ExpScore will be released for the whole community. These resources and our findings can serve as forces of public good for scholars as well as recommender systems users.
Bingbing Wen | University of Washington, Seattle, WA, US | bingbw@uw.edu
Yunhe Feng | University of Washington, Seattle, WA, US| yunhe@uw.edu
Yongfeng Zhang | Rutgers University, New Brunswick, NJ, US | yongfeng.zhang@rutgers.edu Chirag Shah | University of Washington, Seattle, WA, US, chirags@uw.edu |
Shruti Phadke is one of our lab members who is currently a Ph.D. candidate studying in Information School. Her presentation focuses on natural language processing computational methods and combining them with theories from social psychology to understand various Community practices in communities of problematic information.
In information seeking and retrieval research, researchers often collect data about users’ behaviors to predict task characteristics and personalize information for users. The reliability of user behavior may be directly influenced by data collection methods. This article reports on a mixed-methods study examining the impact of study setting (laboratory setting vs. remote setting) and task authenticity (authentic task vs. simulated task) on users’ online browsing and searching behaviors. Thirty-six undergraduate participants finished one lab session and one remote session in which they completed one authentic and one simulated task. Using log data collected from 144 task sessions, this study demonstrates that the synthetic lab study setting and simulated tasks had significant influences mostly on behaviors related to content pages (e.g., page dwell time, number of pages visited per task). Meanwhile, first-query behaviors were less affected by study settings or task authenticity than whole-session behaviors, indicating the reliability of using first-query behaviors in task prediction. Qualitative interviews reveal why users were influenced. This study addresses methodological limitations in existing research and provides new insights and implications for researchers who collect online user search behavioral data.
Technologies like search engines that allow people to quickly identify and interpret the most relevant and reliable facts, figures or files are essential today, given the proliferation of misinformation on the internet and the treasure trove of digital data accessible online.
The influence and prevalence of these kinds of tools gives the academic discipline of information systems many broad applications. There are multiple compelling reasons to pursue an advanced degree in this discipline, according to experts, who suggest that it’s important to compare the cost of such a degree with its potential long-term benefits.
“Coming into this, of course, it’s important that you like technology and know technology, but also you have to like and know it as a designer, as a creator, as a manager (and) as a developer,” Dr Chirag Shah mentioned.
A recent paper that got accepted at IP&M by Dr. Yunhe Feng & Prof. Chirag Shah
Open data is becoming ubiquitous as governments, companies, and even individuals have the option to offer more or less unrestricted access to their non-sensitive data. The benefits of open data, such as accessibility and transparency, have motivated and enabled a large number of research studies and applications in both academia and industry. However, each open data only offers a single perspective, and its potential inherent limitations (e.g., demographic biases) may lead to poor decisions and misjudgments. This paper discusses how to create and use multiple digital lenses empowered by open data, including census data (macro lens), search logs (meso lens), and social data (micro lens), to investigate general real-world events. To reveal the unique angles and perspectives brought by each open lens, we summarize and compare the underpinning open data from eleven dimensions, such as utility, data volume, dynamic variability, and demographic fairness. Then, we propose an easy-to-use and generalized open data-driven framework, which automatically retrieves multi-source data, extracts features, and trains machine learning models for the event specified by answering what, when, and where questions. With low labor efforts, the framework’s generalization and automation capabilities guarantee an instant investigation of general events and phenomena, such as disasters, sports events, and political activities. We also conduct two case studies, i.e., the COVID-19 pandemic and Great American Eclipse (see Appendix), to demonstrate its feasibility and effectiveness at different time granularities.
Led by our lab director Dr. Chirag Shah, editor in chief.
Information Matters (IM) is a digital-only communication translational forum for information science, bringing relevant and current research evidence and industry developments, news, and opinion to a global public audience free of charge. The platform is sponsored by the Association for Information Science & Technology (ASIS&T).
IM is committed to producing translational reports, news, reviews, columns, and features of the highest standards. Our publication serves to inform the public, industry professionals, educational practitioners, and policymakers on information science issues of critical importance to society and civil life. Our readers rely on our content, to be informed and aware, and also to support decision-making. It is therefore essential that every published piece meets our commitment to quality, accuracy, and integrity.
Want to contribute to Information Matters and publish your own online research article?
Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and “collaborations” among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies.
Authors:Jiqun Liu, Rutgers University,Chirag Shah, Rutgers University
Get yours online at: http://www.morganclaypoolpublishers.com/catalog_Orig/product_info.php?products_id=1418
An Integrated Model of Task, Information Needs, Sources and Uncertainty to Design Task-Aware Search Systems paper presented by Shawon Sarkar at the ICTIR conference
The varieties of information seeking behavior encompass a range of practices and constructs such as the realization of an information need, selecting the nature of information, as well as information sources. Most of the past works have studied various constructs of the information seeking process, i.e., information, information need, and information sources individually. However, a person forms and re-forms his or her information seeking strategy based on continually shifting values of these dimensions associated with information seeking. This preliminary study conducted a survey with 15 search scenarios and multiple-choice characteristics completed by 114 Amazon’s Mechanical Turk workers to find out more about how these constructs play a role in people’s preferences regarding information seeking strategies. The study took an exploratory and inferential research approach to investigate how different forms of information and information needs might lead to different information sources by building binary classification models. The results show that the choice of sources can be predicted (with 80% accuracy) if the information need, representation, and form of information are apparent.
An Integrated Model of Task, Information Needs, Sources and Uncertainty to Design Task-Aware Search Systems Shawon Sarkar, Chirag Shah, ICTIR 2021
Shawon Sarkar successfully defended her dissertation proposal!
Congratulations to our PhD student, Shawon Sarkar for successfully defending her dissertation titled “An Integrated Model of Tasks and Uncertainties to Design Task-aware Intelligent Search Assistance”
Abstract
Search behaviors are generally motivated by some tasks that prompt users in search processes. Complex tasks often initiate lengthy, intermittently changing, interactive search processes with shifting goals at various search stages. At these different stages of the search, users’ search strategies are influenced by their search intentions, encountered problems, as well as knowledge states. However, search systems are primarily designed to optimize one request at a time, disregarding the underlying overarching task, shifting states of the task, or even the holistic nature of a search session. Although a set of descriptive and theoretical models of the search process can be found in the literature that characterizes tasks, there is a gap in research focused on leveraging dynamic task features in search ranking and recommendation processes. More importantly, there is a lack of support for users to complete their tasks in an adaptive, dynamic way across multiple devices and modalities. To address this issue, the proposed dissertation aims to develop new methods for constructing unified task representation using implicit search behavioral data and applying the task representation to improve existing search and recommendation systems and address emerging problems of conversational and interactive search. Specifically, this study creates a task-information need-strategy-problem map that can be leveraged to provide task-based support in various information formats (e.g., suggesting query, document, or people) to overcome problems and lead toward tasks completion. The main focus of this dissertation work is to develop task-aware search systems capable of understanding and extracting tasks and supporting user’s complex search task completion. Therefore, this research revolves around three broad objectives. First, developing a conceptual model for understanding how different types of tasks trigger particular information needs that may lead to different methods and strategies of seeking different forms of information, information sources, and various problems. Second, drawing knowledge from the first stage, developing computational models for extracting task states from users’ search behaviors. Third, leveraging the acquired task knowledge in designing scalable and efficient task-based proactive search systems to meet users’ task goals and provide relevant information in various formats (i.e., query, document, people).
