What is your take on intrinsic vs. extrinsic evaluation?
Should gender bias evaluation be task specific?
What is currently missing the most in this space?
Relation between evaluation of gender bias and related harmful biases/phenomena (e.g. hate speech)
(+ questions from Twitter or panelists)
Panelists:
Kellie Webster, Google Research
Kai-Wei Chang, University of California Los Angeles
Seraphina Goldfarb-Tarrant, University of Edinburgh
Mark Yatskar, University of Pennsylvania
Submissions will be accepted as short papers (4-6 pages) and as long papers (8-10 pages), plus additional pages for references, following the NAACL 2022 guidelines. Supplementary material can be uploaded separately. Blind submission is required.
All submissions have the requirement to include a statement which explicitly defines
(a) what system behaviours are considered as bias in the work, and
(b) why those behaviours are harmful, in what ways, and to whom (cf. Blodgett et al. (2020)). We encourage authors to engage with definitions of bias and other relevant concepts such as prejudice, harm, discrimination from outside NLP, especially from social sciences and normative ethics, in this statement and in their work in general.
Please, find help on how to write a bias statment in here.
Non-archival option
The authors have the option of submitting previously unpublished research as non-archival, meaning that only the abstract will be published in the conference proceedings. We expect these submissions to describe the same quality of work and format as archival submissions.
Final submissions: Both long and short papers can be extended by 1 page in the camera-ready version.
Kellie Webster & Kevin Robinson, Research Scientists, Google Research
Abstract: Language modeling improvements and emerging zero-shot capabilities have led to an explosion of interest in potential applications. With this power, many have cautioned about potential shortcomings and harms of the current technology. In this talk, we make a deliberate distinction between shortcomings intrinsic to a model, notably what is discussed in the bias literature, and the potential for a model used in a system to cause harm to real people. This distinction enables us to discuss what factors in pre-training influence the encoding of social stereotypes, and observe that improving language modeling may improve bias measurements even in the presence of data skew. On the other hand, measuring potential harms remains an outstanding challenge and requires new forms of cross-disciplinary understanding. We discuss our experience with exploring how to formulate scalable, relevant, and actionable measures of potential harm, with a case study on machine translation.
Kai-Wei Chang, Assistant Professor, University of California
Abstract: Natural Language Generation (NLG) technologies have advanced drastically in recent years, and they have empowered various real-world applications that touch our daily lives. Despite their remarkable performance, recent studies have shown that NLG models run the risk of aggravating the societal biases present in the data. Without properly quantifying and reducing the reliance on such correlations, the broad adoption of these models might have the undesirable effect of magnifying prejudice or harmful implicit biases that rely on sensitive demographic attributes. In this talk, I will discuss metrics and datasets for evaluating gender bias in language generation models. I will review existing bias measurements and demonstrate how intricate bias metrics are inconsistent with the extrinsic ones. I will further discuss the harms of gender exclusivity and challenges in representing non-binary gender in NLP.
4th Workshop on Gender Bias in Natural Language Processing (in conjunction with NAACL 2022)
Gender bias, among other demographic biases (e.g. race, nationality, religion), in machine-learned models is of increasing interest to the scientific community and industry. Models of natural language are highly affected by such biases, which are present in widely used products and can lead to poor user experiences. There is a growing body of research into improved representations of gender in NLP models. Key example approaches are to build and use balanced training and evaluation datasets (e.g. Webster et al., 2018), and to change the learning algorithms themselves (e.g. Bolukbasi et al., 2016). While these approaches show promising results, there is more to do to solve identified and future bias issues. In order to make progress as a field, we need to create widespread awareness of bias and a consensus on how to work against it, for instance by developing standard tasks and metrics. Our workshop provides a forum to achieve this goal.
Our workshop follows up three successful previous editions of the Workshop collocated with ACL 2019, COLING 2020, and ACL-IJCNLP 2021, respectively. As in the two previous years (2020 and 2021), special efforts will be made this year to encourage a careful and reflective approach to gender bias by the means of separately reviewed bias statements (Blodgett et al., 2020; Hardmeier et al., 2021). This helps to make clear (a) what system behaviors are considered as bias in the work, and (b) why those behaviors are harmful, in what ways, and to whom. We encourage authors to engage with definitions of bias and other relevant concepts such as prejudice, harm, discrimination from outside NLP, especially from social sciences and normative ethics, in this statement and in their work in general. We will keep pushing the integration of several communities such as social sciences as well as a wider representation of approaches dealing with bias.
4th Workshop on Gender Bias in Natural Language Processing
At NAACL in Seattle, USA, during July 15, 2022
Gender bias, among other demographic biases (e.g. race, nationality, religion), in machine-learned models is of increasing interest to the scientific community and industry. Models of natural language are highly affected by such biases, which are present in widely used products and can lead to poor user experiences. There is a growing body of research into improved representations of gender in NLP models. Key example approaches are to build and use balanced training and evaluation datasets (e.g. Webster et al., 2018), and to change the learning algorithms themselves (e.g. Bolukbasi et al., 2016). While these approaches show promising results, there is more to do to solve identified and future bias issues. In order to make progress as a field, we need to create widespread awareness of bias and a consensus on how to work against it, for instance by developing standard tasks and metrics. Our workshop provides a forum to achieve this goal.
Our workshop follows up three successful previous editions of the Workshop collocated with ACL 2019, COLING 2020, and ACL-IJCNLP 2021, respectively. As in the two previous years (2020 and 2021), special efforts will be made this year to encourage a careful and reflective approach to gender bias by the means of separately reviewed bias statements (Blodgett et al., 2020; Hardmeier et al., 2021). This helps to make clear (a) what system behaviors are considered as bias in the work, and (b) why those behaviors are harmful, in what ways, and to whom. We encourage authors to engage with definitions of bias and other relevant concepts such as prejudice, harm, discrimination from outside NLP, especially from social sciences and normative ethics, in this statement and in their work in general. We will keep pushing the integration of several communities such as social sciences as well as a wider representation of approaches dealing with bias.
Organizers
Marta R. Costa-jussà, Meta AI, Paris
Christian Hardmeier, Uppsala University
Hila Gonen, FAIR and University of Washington
Christine Basta, Universitat Politècnica de Catalunya, Barcelona
Gabriel Stanovsky, Hebrew University of Jerusalem
Contact persons
Marta R. Costa-jussà: costajussa (at) fb (dot) com
