Generative AI contributes to rising technical debt but may also be the key to solving it.
Today’s tech leaders are in a race to deploy next-generation products that can deliver major returns. This is further amplified by the emergence of generative artificial intelligence (genAI). However, decision-makers must keep a keen eye out for the hidden cost: technical debt.
Emerging in risk management at the turn of the millennium, when Y2K brought it into the public consciousness, technical debt is the expected and unexpected expense that organisations suffer when implementing new technology. This includes the cost of fixing bugs that were missed during the product launch, adding patches for new vulnerabilities detected, and updating old technologies.
If left to manifest, these issues can complicate digital transformation with a Forrester report finding that for 15% of respondents, technical debt was diverting resources away from innovation.
Factors driving technical debt
Some reasons why technical debt is on the rise include growing corporate investment in digital transformation, migration to the cloud, an increasingly complex technical environment, and now, the use of AI to produce lines of code. Research by IDC found that 35% of companies surveyed prioritised AI infrastructure spending, while 38% expected to overspend to make up for accumulated technical debt.
Though each new deployment of genAI tools brings in new risks as well as future and potential debts, the technology could be the key to tackling these problems.
With the speed of AI’s advancement and lack of standards across different genAI models, making the most of emerging genAI capabilities hinges on proactively addressing compounding technical debt. In fact, Freshworks’ 2024 Global AI Workplace Report found that 71% of IT professionals reported that the time freed up by AI boosts operational efficiencies, indicating that with strategic implementation, AI's benefits can outweigh the technical debt it may create.
Key considerations when managing technical debt
While a majority of organisations allocate about 12.8% of their IT budget to reducing technical debt, over 79% do not have formal processes for tracking and reporting this debt, according to an IDC report. The complex technological environment of today as well as all unknown future risks related to the use of genAI adds to the challenge in managing technical debt.
In particular, large language models work on six types of media: text, software code, images, audio, video, and 3D content. Each type of media brings an added layer of complexity, and therefore, risks in tech debt management.
However, there are ways tech leaders can address these issues to get a handle on their organisation’s technical debt without losing ground on innovation:
Integrate LLMs with legacy systems: It is important to ensure that their genAI models fit into the organisation’s existing IT network. It can be easy for tech leaders to jumpstart their genAI initiatives, but without proper investment in data quality, integration, and governance, they will eventually come up against some major challenges. After all, the two main areas of technical debt are governance and legacy system integration.
Establish "LLMOps" for operational and data management: Efficient regulation of genAI is possible only when there is a separate governance framework for it. This approach is an extension of DevOps and AIOps and what is commonly known as LLMOps or XOps. This kind of framework is critical to streamlining data and monitoring operations.
Use API “wrappers” to mitigate risks: LLMs are vulnerable to occasional breakdowns when they are adopted and implemented within an existing system, which can lead to a build-up of more technical debt. To mitigate this risk, tech leaders should consider using API wrappers that can assist in protecting code from interferences and alterations, allowing it to complete outputs on time, error-free, and protected from external risks. Consequently, this means developer teams would not have to slow their pace whenever something breaks.
Address critical risks when fine-tuning LLMs: When fine-tuning an LLM, it’s often about trying to ensure the models can deliver higher quality outputs. However, it can also create new vulnerabilities. Though training a model on a domain-specific data set may seem useful and benign, it creates a potential for exploitation. To avoid that, CIOs should be wary of this risk when fine-tuning and addressing those risks as they crop up.
Does GenAI help or hurt?
The key here is for tech leaders to remain proactive in identifying new AI risk factors that could contribute to future technical debt. Simultaneously, tech leaders should explore the application of AI in addressing technical debt. Some potential strategies include:
The key here is for tech leaders to remain proactive in identifying new AI risk factors that could contribute to future technical debt.
Simon Ma
Code analysis: Using genAI to efficiently analyse and understand the code base of legacy technologies, which can be immensely helpful to engineers who are often keeping tabs on these systems. Understanding the code can help IT teams better manage technical debt.
Updating old code: Using genAI to update old codes, also called refactoring. This frees up developers for more strategic and value-added projects.
Code documentation: Many organisations still rely on manually updating their IT documentation – a major factor of technical debt. With genAI, organisations can streamline and automate the process to create and update documentation for both legacy and new applications.
The benefits of digital transformation
Tech leaders who can successfully balance AI innovation with technical debt management are better positioned to survive the digital transformation without incurring further technical debt. The world of LLMs is in a precarious and constantly evolving state right now and is growing rapidly. Tech leaders need to be aware of and invest in additional strategies to address technical
Simon Ma, Managing Director, Asia, Freshworks - A resolute and goal-focused management leader with 12 years of success in driving growth in sales and revenue by effectively managing customer relations and acquisition of new accounts.
Prior to the appointment, Simon was the Regional Sales Director for ServiceNow where he was responsible for hiring, coaching, and overseeing business growth across Southeast Asia.
Possessing exceptional organisational and leadership abilities, Ma excelled at structuring effective sales teams to close negotiations for large, multi-year contracts. His tenure was marked by impressive growth in new large enterprise business in the region.
