We are all witnessing a revolution in the progress of artificial intelligence (AI) and its adoption. AI’s latest content creation capabilities have created an enormous interest in the media and the public.
ChatGPT most recently demonstrated an astonishing ability to generate coherent text when you ask it to write an essay or answer a question. Along with excitement, many ethical questions immediately popped up. Are these systems able to reason on par with humans? Are they aware of the content they generate? Are their answers fair and unbiased?
But AI is not just about chatbots anymore. Even if you’ve never used a tool like ChatGPT, AI is already affecting your life. AI may have been used in your doctor’s office or bank without you even knowing.
Here too, we need to worry about the same issues, including:
- fairness,
- bias,
- security, and
- robustness and resilience.
These are all part of what’s known as trustworthy AI.
As the power of AI grows, businesses, governments and the public will have to manage AI’s impact on society.
The key will be to allow the industry to innovate while managing risk. Although it will be challenging, we will have to find the right balance among self-interest, business interest and societal interest.
Our Human Brains Have Their Limits
I’ve long been interested in the human brain. Despite the great progress in science, many questions remain unresolved: How do we comprehend information? How do we store and retrieve information in our brains? What is intelligence?
I’ve learned through the work of Nobel-prize winning psychologist Daniel Kahneman and others in behavioral economics that humans are terrible at being consistent in reasoning and coming up with the best solutions. I’m fascinated by how we as people can be both very limited and so creative and capable of deep thinking.
As AI advances, I’m curious how it can be good for all people. How can it serve the needs of everyone equally well? It’s an intriguing question.
How Do You Define Fairness in AI?
Last year, we developed a comprehensive report on AI bias. This report established that fairness and bias are not just abstract or universal statistical problems.
Fairness and bias issues in AI have complex aspects to them that cannot be easily defined by mathematics.
The definition of fairness in lending, for example, has evolved over time and will continue to do so. Interest-based loans started thousands of years ago in Mesopotamia, and the idea of fairness in lending has certainly changed since then.
Fairness is also a social concept. Many people may feel something is unfair, even if the data shows it’s unbiased.
We need to ensure fairness across the board, but the problem of finding the best approaches to managing bias and fairness vary based on the context and application. For example, the process to get a mortgage on a home is much more rigorous than the process to get a car loan because you’re being loaned much more money for a home than for a car.
We need to consider both the technological and human factors in AI and have an approach to fairness and bias that has realistic definitions for different contexts, such as finances, health care or hiring. We also need task-specific datasets for machine learning model development and evaluation.
AI Decides if You Get a Loan. Is It Fair?
As a next step to the seminal work started in last year’s report, we are continuing to develop guidance and testing infrastructure for managing AI bias in context, one context at a time.
We are starting with a project on consumer credit underwriting in financial services because it touches many lives.
Whether it’s applying for a credit card, a mortgage or a car loan, nearly everyone has interacted with the credit system. You may not realize it, but when you apply for credit, AI is nearly always a part of that process. We need to make sure these systems are fair. Bias in the system could lead to an unfairly high interest rate or someone being denied credit entirely.
We humans have numerous cognitive biases. Studies show these biases result in inconsistent decision making, regardless of the level of experience or training of the personnel.
Machines tend to be more consistent than humans, but that does not make them fair. One of the problems with machine learning systems is we train them on historical data to make future decisions. Well, that historical data reflects the societal biases at the time, which our research shows are many.
If we’re not careful to interrupt those biases, AI will replicate those historical mistakes. We need to fully understand exactly what data is going into AI decision making, so we can make sure to adjust for biases. This is one of the limitations of machine learning.
People + AI = More or Less Bias?
Another interesting, but not well understood, aspect of AI-assisted decision making is when a human is assisted by an AI system to make credit decisions.
Both AI and people have their own biases. How does that play out in decision making? Do the biases compound each other? Do they cancel each other out? How do we address both biases? This is something that’s not well understood, so we’re planning to study it and answer some of these questions.
At NIST, we’re working with partners in the financial and technology industries, from small companies and startups to large banks and technology companies. We’re working to see what we can learn from their experiences, their data and the tools that detect and manage bias.
We’re assembling a representative sample of the industry in both financial services and tech to help us come up with recommendations on how we can responsibly use this technology in consumer credit underwriting. We’re also talking with consumer groups and other organizations for their input.
In August 2022, we hosted a workshop and invited participants to help us frame the research questions. What’s the problem? What are the tools we have? How best can we deploy them to have the most effect? What guidance is needed to assist the financial services industry in adopting good business practices in deploying AI technology?
There are many steps we must follow, as part of a long-established process, to begin work on this study. We completed and published the project description document, which describes the specific scenarios we will be working on. It reflects feedback from the workshop. While we are working through the approvals and public notices, we are working to recruit potential collaborators from the financial and technology sectors.
We want to work with companies that are willing to work in good faith to solve these hard problems. Doing the right thing also makes good business sense. It doesn’t have to be one or the other; you can do both things.
Eliminating Bias in AI Is a Hard Problem. We Will Keep Studying It.
Once our study on bias in credit underwriting is complete, we will map the findings in our reports to the high-level principles established in our comprehensive report from last year. This will ensure consistency and harmony between the general principles and sector-specific recommendations. We will continue to study bias in AI and how we can eliminate it in other sectors of the economy where AI is used.
AI is going to continue to be a part of everyone’s lives for years to come, so we want to make sure it’s operating fairly.