Engineering | Press releases | Search | Technology
June 21, 2024
UW researchers found that ChatGPT consistently ranked resumes with disability-related honors and degrees — such as the “Tom Wilson Disability Leadership Award” — lower than the same resumes without those honors and degrees. But when the researchers customized the tool with written instructions instructing it not to be ableist, the tool reduced this bias for all but one of the disabilities tested.Solen Feyissa/Unsplash
While looking for research internships last year, University of Washington graduate student Kate Glazko noticed recruiters posting online that they had used OpenAI’s ChatGPT and other research tools. artificial intelligence to summarize CVs and rank candidates. Automated selection has been commonplace in recruiting for decades. Yet Glazko, a doctoral student in the UW’s Paul G. Allen School of Computer Science & Engineering, is studying how generative AI can reproduce and amplify real-world biases, such as those against people with disabilities. How could such a system, she wondered, classify resumes that implied a person had a disability?
In a new study, UW researchers found that ChatGPT consistently ranked resumes with disability-related honors and degrees — such as the “Tom Wilson Disability Leadership Award” — lower than the same resumes without those honors. and diplomas. When asked to explain the rankings, the system spewed out biased perceptions of people with disabilities. For example, he claimed that a resume with an autism leadership award placed “less emphasis on leadership roles,” implying the stereotype that autistic people are not good leaders.
But when the researchers customized the tool with written instructions instructing it not to be ableist, the tool reduced this bias for all but one of the disabilities tested. Five of the six implied disabilities – deafness, blindness, cerebral palsy, autism and the general term “disability” – improved, but only three ranked higher than resumes that did not mention the disability.
The team presented its findings on June 5 at the ACM 2024 Conference on Equity, Accountability and Transparency in Rio de Janeiro.
“Rankings using AI are starting to proliferate, but there is not much research to determine whether it is safe and effective,” said Glazko, the lead author of the study. “For a job seeker with a disability, when submitting a resume, the question always arises as to whether you should include disability references. I think disabled people consider this even when the evaluators are humans.
The researchers used the publicly available curriculum vitae (CV) of one of the study authors, which was approximately 10 pages long. The team then created six enhanced CVs, each involving a different disability by including four disability-related degrees: a scholarship; a price; a seat on the diversity, equity and inclusion (DEI) panel; and membership in a student organization.
The researchers then used ChatGPT’s GPT-4 model to rank these enhanced resumes against the original version for a real “student researcher” job posting at a large US-based software company. They performed each comparison 10 times; In 60 trials, the system ranked improved CVs first, which were identical except for the implied handicap, only in a quarter of the cases.
“In a just world, the enhanced resume should be ranked first every time,” said lead author Jennifer Mankoff, a UW professor in the Allen School. “I can’t think of a job in which someone recognized for their leadership skills, for example, shouldn’t be ranked ahead of someone with the same background who isn’t.”
When researchers asked GPT-4 to explain the ranking, his responses demonstrated explicit and implicit ableism. For example, he noted that a candidate with depression had “an added focus on DEI and personal challenges,” which “detracted from the core technical and research-oriented aspects of the role.”
“Some descriptions of GPT would color a person’s entire resume based on their disability and claim that involvement in DEI or disability could potentially harm other parts of the resume,” Glazko said. “For example, he hallucinated the concept of “challenges” in the depression CV comparison, even though “challenges” were not mentioned at all. We could therefore see certain stereotypes emerging.
Given this, the researchers wondered if the system could be trained to be less biased. They turned to the GPTs Editor tool, which allowed them to customize GPT-4 with written instructions (no code required). They asked this chatbot to not display ability bias and instead work with disability justice and DEI principles.
They started the experiment again, this time using the newly trained chatbot. Overall, this system ranked the improved CVs higher than the control CV 37 times out of 60. However, for some disabilities the improvements were minimal or absent: the autism CV ranked first only three times out of 10, and the CV of depression only twice. (unchanged from the original GPT-4 results).
“People need to be aware of system biases when using AI for these real-world tasks,” Glazko said. “Otherwise, a recruiter using ChatGPT cannot make these corrections, nor be aware that, even with instructions, biases may persist. »
The researchers note that some organizations, such as ourability.com and inclusively.com, are working to improve outcomes for disabled job seekers, who face bias whether or not AI is used for the job. hiring. They also point out that more research is needed to document and address AI bias. These include testing other systems, such as Google’s Gemini and Meta’s Llama; including other disabilities; study the intersections of system bias against disabilities with other attributes such as gender and race; explore whether further personalization could reduce bias more consistently across disabilities; and see if the base version of GPT-4 can be made less biased.
“It’s very important that we study and document these biases,” Mankoff said. “We have learned a lot and hope to contribute to a broader conversation – not only regarding disability, but also other minority identities – aimed at ensuring that technology is implemented and deployed equitably and fairly.
Additional co-authors were Yusuf Mohammed, a UW undergraduate student in the Allen School; Venkatesh Potluri, UW doctoral student at the Allen School; and Ben Kosa, who conducted this research as a UW undergraduate in the Allen School and is a new doctoral student at the University of Wisconsin-Madison. This research was funded by the National Science Foundation; by donors to the UW Center for Research and Education on Accessible Technologies and Experiences (CREATE); and by Microsoft.
For more information, contact Glazko at glazko@cs.washington.edu and Mankoff at jmankoff@cs.washington.edu.
Tag(s): Center for Research and Education on Accessible Technologies and Experiences • College of Engineering • Jennifer Mankoff • Kate Glazko • Paul G. Allen School of Computer Science and Engineering