Browsed by
Month: June 2019

Bias, Fairness, Diversity and Novelty

Bias, Fairness, Diversity and Novelty

When dealing with bias in IR systems, we are often faced with the question of what are the differences and connections between bias, fairness, diversity and novelty. We have briefly talked about the relationship between bias and fairness in the previous article. Let us now look at diversity and novelty.  

DIVERSITY aims to increase the topical coverage of the search results. Because the information need varies by individual users, predicting the intention based on the query alone is a difficult and sometimes impossible task for the IR system. Even with the presence of additional information such as personal profiles, social identity and networking, geolocations when issuing the search request and browsing history, it is still hard for the system to precisely accommodate each individual’s information need. One perspective is on the ambiguity associated with search query. For example, “apple” may mean the fruit, or the company named Apple. Apart from the ambiguity inherent in the language, a query may also be ambiguous due to the search intent. For instance, a user searching for “rutgers” may be looking for the official website of Rutgers University, the location, description, ranking, or recent news about the university, etc. The IR system must consider all the possibilities of user search intent and return the most relevant results. Another perspective is on the subtopics or topic aspects regarding the searched topic. For instance, different opinion polarities about “gun control” should be included in the search results so that the information presented provides a comprehensive view of the topic. Increasing diversity means to include as many topical subgroups as possible. As a result, diversity can alleviate some bias by enriching the results with more perspective, avoiding results that are all from the same group. Meanwhile, diversity can increase the fairness because it accounts for all subgroups/aspects of the topic.

NOVELTY aims to reduce the redundancy in the retrieved information. For instance, given the search query “gun control”, if two consecutive results are from the same website, or one is a forwarded or cited article from another, then the users may find the second one to be redundant to the first one. In other words, novelty tries to bring as much “new” information as possible in the set of retrieved results. From this perspective, we can see that diversity and novelty can benefit each other to some extent, but none of which guarantees the other. On the one hand, diversity brings new information by introducing different subtopics/aspects. But it does not address in-group information, i.e., it does not care how many results are in the same topical group, as long as the results cover as many groups as possible. So diversity does not guarantee novelty. Novelty, on the other hand, can surface different subtopics/aspects by reducing redundant information. But novelty does not care about the topical groups, as long as the result introduces new information compared to the previous results. In terms of bias, skewed view may be avoided by increasing novelty, but since there is no guarantee on the “group” distribution, there is no guarantee on removing the bias.

FAIRNESS aims to bring balance to the retrieved results according to a subjective design need. If the goal is to enforce topical fairness, then fairness requires all subtopics/aspects to be covered, hence the maximum diversity. But fairness does not necessarily have to be concerned with topical groups. Fairness can be imposed on other groups such as gender, racial and religion. So achieving fairness and diversity can be different goals. In addition, the key points of diversity fairness is to balance the number of results from each topical group, while diversity only aims to maximize the total number of groups covered. For example, if there are two subtopic groups for a given query in the search results, then diversity can be achieved by including one result from one group, and take the rest of the results from another group. But fairness may need the same number of results from each group, depending on the notion of fairness by system design need.

To sum up, while diversity and novelty can potentially reduce bias and fairness, their goals are essentially different from the concepts and goals of unbiasedness and fairness.

InfoSeekers Publish a Book!

InfoSeekers Publish a Book!

We’re excited to share the news that our InfoSeekers, Jiqun Liu and Chirag Shah, have published a new book! The title of the book is Interactive IR User Study Design, Evaluation, and Reporting.

Abstract

Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and “collaborations” among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies.

Table of Contents: Preface / Acknowledgments / Introduction / Interactive Information Retrieval / Methodology: Paper Selection and Coding Scheme / Faceted Framework of IIR User Studies / Evaluating IIR User Studies of Different Types / Implications and Limitations of the Faceted Framework / Conclusion and Future Directions / Appendix / Bibliography / Authors’ Biographies

Jiqun Liu shared his thoughts on the release of his new book: “Synthesis Lectures on Information Concepts, Retrieval, and Services includes a variety of interesting topics that are highly relevant to my research. I am thrilled and honored to have my own book published as part of the Synthesis Lectures book series.”

Interactive IR User Study Design, Evaluation, and Reporting is available via the following link: https://www.morganclaypool.com/doi/pdf/10.2200/S00923ED1V01Y201905ICR067