ChatGPT works — but do we really need to understand why? This is a question asked by AI practitioners and consumers alike. Explainability is seen as a prerequisite for using AI, and a model which produces outputs without clear explanations is often met with mistrust. This is especially so in professional fields such as medicine, where evidence based reasoning is crucial and professionals have a right to know why automated systems produce certain decisions. Thanks to the General Data Protection Regulation (GDPR), it is legally obligatory in the EU to explain why systems have made certain decisions.
In 2021, Stanford professor of Medicine and Biomedical Data Science, Nigam Shah, was interviewed by Katherine Miller of Stanford HAI (Human-Centered Artificial Intelligence). Shah argued that medical professionals routinely prescribe treatments without understanding how or why they work. Although this may sometimes work as a useful practice, Shah explained how explainability in AI takes three main forms. These forms are the engineer version, the causal version and the trust-inducing version.
The engineer version of explainability is reflected in the underlying working of a model. The causal version is related to the input of a model and determines the outputs. The trust-inducing version is the one needed by people who have to put confident trust in a model. This is especially relevant in situations where an algorithm is rejecting loan applications.
Fast-forward two years to 2021 and large language models (LLMs) such as GPT-4 can do multipart tasks with heightened accuracy. These models can be complex and it is not easy to understand the choices they are making. Despite this, they are being rapidly adopted due to their efficacy.
For example, GitHub and MIT are looking to LLCs, such as ChatGPT, to increase the productivity of coders. By using ChatGPT practitioners are said to find a greater sense of “fun” coding, as well as come to tasks with an increase in speed. Similarly, it has been noted that LLCs substitute for worker effort rather than complementing worker skills, so that people can focus energies on creative tasks over busy works.
Despite the lack of engineer, causal or trust-inducing interpretability inherent in such language models, people are still comfortable using them. Is this because such tasks or seen as menial labor and people don’t mind not understanding? Or is there an ever increasing boundary between work that requires emotional intelligence and work that does not?
As LLCs come to hold increasingly more decisive roles, the questioning regarding their use and necessity will certainly grow. To ensure that such machines remain non-discriminatory, it may be necessary to upgrade the explainability frameworks outlined by Nigam Shah and make sure they fulfil the needs of the future.
