Transparency around large language models (LLMs) currently is lacking and this problem should be addressed, if artificial intelligence (AI) is to become trustworthy.
AI needs to be understandable and explainable, and users must be able to trust AI. In particular, using “ordinary” AI in critical systems does not work, as wrong decisions can have significant impact, said Marko Erman, Thales’ senior vice president and chief scientific officer.
It is especially pertinent that AI delivers transparency, reliability, and trust that are beyond commercial systems in mission-critical environments, where milliseconds matter and stakes are high, Erman said.
And building this level of trustworthiness has to be addressed from the component level up and across the entire system, he said. It encompasses, amongst others, secured cloud platforms, embedded cybersecurity, data validity and accuracy, and digital protection of algorithms.
Asked what was most challenging in helping AI systems gain trustworthiness, he pointed to transparency in relation to the data that trains the machines.
“[That] transparency is kind of limited [and] never complete,” Erman said, in an interview with FutureCIO.
He likened it to a public library, where he would like to see what is inside. “That will give me trust. That will give me a guarantee that there is no IP infringement,” he said.
He noted that some market players had promised to provide transparency “at the beginning when they were young”, but “forgot” this pledge when they gained more maturity in the market.
According to the Foundation Model Transparency Index, which assessed the transparency of 10 major foundation models, the mean score clocked at 58%, with StarCoder earning the highest score at 85%.
Developed by the BigCode project, which is a partnership between Hugging Face and ServiceNow, StarCoder is an LLM designed specifically for tasks related to code development. The foundation model is trained on data from GitHub, including programming languages and GitHub issues.
The transparency index was first released by researchers from Stanford University, MIT, and Princeton, in October 2023 and updated in May 2024.
OpenAI’s ChatGPT-4 has the fourth-lowest score at 49%, just below Anthropic’s Claude 3 at 51% and above Google’s Gemini 1.0 Ultra at 47%.
Transparency necessary to build trustworthy AI systems
Elaborating on the importance of transparency, Erman noted that the lack of it can result in malicious information being distributed, unintentionally or otherwise, particularly by organisations and users that are not equipped to identify them.
He explained that when a box contains a bottle of food and a bottle of poison next to it, without information to clearly differentiate either, it is easy for anyone to mistakenly take out the poison or for the poison to be mixed with the food that is meant for consumption.
This scenario reflects the current state of some LLMs, he said.
While Thales has a team capable of scrutinising commercial LLMs and extracting malicious data and activities from these models, he noted that the general public and most organisations may not necessarily have specialised skillsets to do the same.
As it is, less than half of organisations, at 40%, are investing in governance, explainability, and ethical safeguards to ensure AI systems are trustworthy, revealed an IDC study commissioned by SAS, which was released this week. The research polled 2,375 respondents globally, including North America, Europe, and Asia-Pacific.
Asked about their top business priorities, just 2% highlighted the need to develop an AI governance framework, while under 10% cited developing a responsible AI policy, the study found.
Interestingly, those that had the least investment in trustworthy AI systems viewed generative AI (GenAI) as 200% more trustworthy than traditional AI, the study found.
Some 48% of respondents said they had “complete trust” in GenAI, while 33% said likewise for agentic AI.
The least trusted form of AI was traditional AI, in which just 18% said they had complete trust.
“Forms of AI with humanlike interactivity and social familiarity seem to encourage the greatest trust, regardless of actual reliability or accuracy," said Kathy Lange, IDC’s research director of AI and automation practice.
“As AI providers, professionals, and personal users, we must ask: GenAI is trusted, but is it always trustworthy? And are leaders applying the necessary guardrails and AI governance practices to this emerging technology?” Lange posed.
Nonetheless, 62% of respondents did express concerns about data privacy, while 57% pointed to transparency and explainability, and 56% said they had concerns about ethical AI use.
The report noted that enterprise that prioritise trustworthy AI are 60% more likely to double their returns on investment from AI projects.
“For the good of society, businesses, and employees -- trust in AI is imperative,” said SAS CTO Bryan Harris. “In order to achieve this, the AI industry must increase the success rate of implementations, humans must critically review AI results, and leadership must empower the workforce with AI.”
Trust and humans vital in mission-critical environments
Reiterating the importance of building AI that is robust enough for critical systems, Erman said: “Failures in areas such as defence, aerospace, or cybersecurity can undermine public trust, disrupt economic stability, or even affect diplomatic balance.”
And in situations where lives are at stake, humans must remain the centre of decision-making, he noted, adding that Thales believes AI should complement -- not replace -- human expertise.
He pointed to the company’s Avionics Asia-Pacific Industrial Centre in Singapore, where it is trialling an AI-powered system that reviews safety tests for flight equipment. Such tasks are typically carried out manually by technicians.
“Built with strict human oversight, the system has already improved fault detection and productivity, and we plan to replicate this success across our global avionics network,” Erman said. “In the area of air traffic management, AI has a role to play in the safe and timely sequencing of flights, helping reduce travel delays and optimising airlines’ carbon footprints for a more sustainable aviation future.”
In dense urban environments such as Singapore, AI can enhance the performance of a radar to differentiate drones from birds. This is a critical feature with the proliferation of unmanned objects in the airspace, he noted.
He stressed the need to further ensure AI recommendations remain valid, explainable, and actionable in real-time.
Erman spoke with FutureCIO last week during his visit to Singapore, where Thales works with local organisations that include the public sector, such as defence.
In May, the French multinational tech company announced several initiatives with various Singapore government agencies, including one with the Economic Development Board to launch cortAIx SG, which aims to push the “ethical and effective” adoption of AI in the Asian nation. The facility also will support the local civil and defence ecosystems.
With more than 100 of its products are integrated with AI, Thales described cortAIx as an “important accelerator” for the tech vendor to identify challenges and develop “high-impact” AI products.
Thales also widened its partnership with Singapore’s Home Team Science & Technology Agency to set up a lab focusing on AI-enabled technologies, developing local capabilities, and establishing best practices for trustworthy AI within the Singapore context. The two partners also will develop a research and technology roadmap encompassing five key areas, including AI, brain-computer interfaces, cybersecurity, and quantum.
In April, Thales said it would work with Singapore’s Defence Science and Technology Agency on a lab to work on AI-powered technologies that can enhance combat systems currently used by the Singapore Armed Forces to address drone threats.
