Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and “collaborations” among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies.
Authors:Jiqun Liu, Rutgers University,Chirag Shah, Rutgers University
Get yours online at: http://www.morganclaypoolpublishers.com/catalog_Orig/product_info.php?products_id=1418
An Integrated Model of Task, Information Needs, Sources and Uncertainty to Design Task-Aware Search Systems paper presented by Shawon Sarkar at the ICTIR conference
The varieties of information seeking behavior encompass a range of practices and constructs such as the realization of an information need, selecting the nature of information, as well as information sources. Most of the past works have studied various constructs of the information seeking process, i.e., information, information need, and information sources individually. However, a person forms and re-forms his or her information seeking strategy based on continually shifting values of these dimensions associated with information seeking. This preliminary study conducted a survey with 15 search scenarios and multiple-choice characteristics completed by 114 Amazon’s Mechanical Turk workers to find out more about how these constructs play a role in people’s preferences regarding information seeking strategies. The study took an exploratory and inferential research approach to investigate how different forms of information and information needs might lead to different information sources by building binary classification models. The results show that the choice of sources can be predicted (with 80% accuracy) if the information need, representation, and form of information are apparent.
An Integrated Model of Task, Information Needs, Sources and Uncertainty to Design Task-Aware Search Systems Shawon Sarkar, Chirag Shah, ICTIR 2021
Shawon Sarkar successfully defended her dissertation proposal!
Congratulations to our PhD student, Shawon Sarkar for successfully defending her dissertation titled “An Integrated Model of Tasks and Uncertainties to Design Task-aware Intelligent Search Assistance”
Search behaviors are generally motivated by some tasks that prompt users in search processes. Complex tasks often initiate lengthy, intermittently changing, interactive search processes with shifting goals at various search stages. At these different stages of the search, users’ search strategies are influenced by their search intentions, encountered problems, as well as knowledge states. However, search systems are primarily designed to optimize one request at a time, disregarding the underlying overarching task, shifting states of the task, or even the holistic nature of a search session. Although a set of descriptive and theoretical models of the search process can be found in the literature that characterizes tasks, there is a gap in research focused on leveraging dynamic task features in search ranking and recommendation processes. More importantly, there is a lack of support for users to complete their tasks in an adaptive, dynamic way across multiple devices and modalities. To address this issue, the proposed dissertation aims to develop new methods for constructing unified task representation using implicit search behavioral data and applying the task representation to improve existing search and recommendation systems and address emerging problems of conversational and interactive search. Specifically, this study creates a task-information need-strategy-problem map that can be leveraged to provide task-based support in various information formats (e.g., suggesting query, document, or people) to overcome problems and lead toward tasks completion. The main focus of this dissertation work is to develop task-aware search systems capable of understanding and extracting tasks and supporting user’s complex search task completion. Therefore, this research revolves around three broad objectives. First, developing a conceptual model for understanding how different types of tasks trigger particular information needs that may lead to different methods and strategies of seeking different forms of information, information sources, and various problems. Second, drawing knowledge from the first stage, developing computational models for extracting task states from users’ search behaviors. Third, leveraging the acquired task knowledge in designing scalable and efficient task-based proactive search systems to meet users’ task goals and provide relevant information in various formats (i.e., query, document, people).
Analyzing users’ perceptions of search engine biases and their satisfaction when the biases are regulated
In our survey study, we paired a real page from search engine Bing and a synthesized page with more diversities in the results (i.e. less biased). Both pages show the top-10 search items given search queries. We asked participants which one they prefer and why do they prefer the selected one. Statistical analyses revealed that overall, participants prefer the original Bing pages. Additionally, the location where the diversities are introduced is significantly associated with users’ preferences.
We found out that users prefer results that are more consistent and relevant to the search queries. Introducing diversities undermines the relevance of the search results and impairs users’ satisfaction to some degree. It was interesting to see that users tend to pay more attention to the top portion of the results rather than the bottom ones, which is consistent with some previous findings.
Han, B., Shah, C., & Saelid, D. (2021). Users’ Perception of Search-Engine Biases and Satisfaction. Second International workshop on algorithmic bias in search and recommendation (Bias 2021). April 1, 2021.
At the BIAS @ ECIR 2021 Workshop, our lab members continue to investigate the importance of fairness in search and recommendation that is increasingly drawing attention in recent years.
The paper explores how to define latent groups, which cannot be determined by self-contained features but must be inferred from external data sources, for fairness-aware ranking. In particular, taking the Semantic Scholar dataset released in TREC 2020 Fairness Ranking Track as a case study, we infer and extract multiple fairness-related dimensions of author identity including gender and location to construct groups.
We propose a fairness-aware re-ranking algorithm incorporating both weighted relevance and diversity of returned items for given queries. Our experimental results demonstrate that different combinations of relative weights assigned to relevance, gender, and location groups perform as expected.
Due to inaccurate group classifications, for our future work, we propose to explore public personal locations, such as using Twitter profile locations.
Several of the lab members participated in the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR) 2021 last week. CHIIR focuses on elements such as human involvement in search activities, and information seeking and use in context. Our lab director Dr. Shah is the Chair of the CHIIR Steering Committee. He presented a paper that was in collaboration with Microsoft Research (MSR) AI.
Several InfoSeeking students were also working as volunteers for the conference. “It was helpful to get feedback and questions from expert mentors in the field. As always, it was fun to be at CHIIR and seize opportunities to meet friends and colleagues. I had some productive discussions in front of the aquarium in the lobby at Gather Town”. Shawon Sarker PhD Student @ InfoSeeking Lab presented her dissertation “A context-independent representation of task” that aims to explicate task information from user behaviors and apply task knowledge to search and recommendations in order to support users to complete their tasks, especially complex tasks, across multiple devices.
When people talk about data science programs, what do you think of? Artificial intelligence, machine learning, or coding is probably the most popular answer for those outside the discipline. What if we told you there is more to that than what meets the eye. At iSchool, we continue to empower students in understanding the implications of using such a powerful tool. With our unique approach, we have designed a program that incorporates data science through a human-centered lens, promoting solutions that are socially responsible and understanding where the potential solutions get implemented. We strive to embody students to focus on human values such as privacy, human rights, and ethics while working on data problems. This means asking not just what technology could do, but also what it should do. And this means acknowledging and addressing the individuals, organizations, and communities behind the production and consumption of data and technology. At iSchool, a small group of iSchool faculty- the iSchool Data Science Curriculum Committee (iDSCC) continues to create a more cohesive and comprehensive program. In doing so, the iSchools are paving a path for DS that can create informative, insightful, and impactful solutions for the whole of humanity for generations to come. We believe foregrounding human needs and business understanding in this way not only will lead to more ethical data science practices but also a more successful (and profitable) outcome.
Our lab director Dr. Shah wrote an article with his international collaborators on what it is to do and teach data science in an iSchool.
Shah, C., Anderson, T., Hagen, L., & Zhang, Y. An iSchool approach to data science: Human-centered, socially responsible, and context-driven. Journal of the Association for Information Science and Technology (JASIST).
RecSys is the premier international forum for the presentation of new research results, systems, and techniques in the broad field of recommender systems.
We are thrilled to be involved in one of the most important annual conferences for the presentation and discussion of recommender systems research. This year, our Lab Director, Chirag Shah, collaborated with Spotify paper – Investigating Listeners’ Responses to Divergent Recommendations – is being presented at the conference.
Search and recommender systems have unprecedented influence on how and what information people access. These gateways to information on the one hand create an easy and universal access to online information, and on the other hand create biases that have shown to cause knowledge disparity and ill-decisions for information seekers. Most of the algorithms for indexing, retrieval, ranking, and recommendation are heavily driven by the underlying data that itself is biased. In addition, ordering of the search and recommendation results create position bias and exposure bias due to their considerable focus on relevance and user satisfaction. These and other forms of biases that are implicitly and some times explicitly woven in search and recommender systems are becoming increasing threats to information seeking and sense-making processes. In this tutorial, we will introduce the issues of biases in search and recommendation and show how we could think about and create systems that are fairer, with increasing diversity and transparency. Specifically, the tutorial will present several fundamental concepts such as relevance, novelty, diversity, bias, and fairness using socio-technical terminologies taken from various communities, and dive deeper into metrics and frameworks that allow us to understand, extract, and materialize them. The tutorial will cover some of the most recent works in this area and show how this interdisciplinary research has opened up new challenges and opportunities for communities such as RecSys.
Heartiest congratulations go to Dr. Dongho Choi and Dr. Long Le for completing their PhDs in Information Science and Computer Science, respectively. And equally ecstatic congratulations to Ms. Shawon Sarkar and Mr. Jiho An for completing their Masters in Information degrees.