The AI field has evolved dramatically. For over a decade, structured data fueled AI value. Supervised and deep learning excelled at labeling, but this approach overlooks a crucial reality: most organizational data is unstructured. This blog post explores the shift towards leveraging unstructured data with Large Language Models (LLMs), addresses the hype around Artificial General Intelligence (AGI), and examines the challenges and technical debt associated with LLM deployment.

The Untapped Potential of Unstructured Data

Estimates suggest over 80% of organizational data is unstructured. A significant 71% of organizations struggle to manage and secure this data. Previously, building automated systems to leverage this data proved difficult. LLMs now offer a powerful solution. They can use this unstructured data as extended memory, enabling contextualized and customized responses based on domain-specific information. This opens doors to powerful new applications within organizations.

Augmenting Intelligence, Not Replacing Jobs: Addressing the AGI Hype

Much of the current discourse around generative AI focuses on fears of job displacement and a looming “Doomsday” scenario. However, a more pragmatic approach is needed. As the Nature article “Stop talking about tomorrow’s AI doomsday when AI poses risks today” argues, we should address current AI risks instead of speculating about future threats. Focusing on present concerns allows regulators to create effective policies for existing AI systems and proactively adapt to new algorithmic approaches.

At Mastercard, we view LLMs and generative AI as tools to augment human productivity. We acknowledge the hype but remain focused on responsible development and deployment.

The Algorithmic Foundation of LLMs: A Path to AGI?

While LLMs have transformed many aspects of our lives, their core algorithmic foundation, based on autoregressive next-token prediction, isn’t the path to AGI. This approach, while powerful, is inherently flawed. Mistakes amplify over time due to the dependence of subsequent token generation on previously generated tokens. As Ada Lovelace, a pioneer of computing, observed in 1843, analytical engines (or modern-day machine learning) cannot originate anything; they only perform what we instruct them to do. This statement remains relevant despite the significant advancements in AI algorithms.

The success of models like ChatGPT isn’t solely due to novel algorithms, but rather a revolutionized user interface. ChatGPT’s natural language prompting allows seamless interaction with the LLM. This accessibility has driven widespread adoption, but it’s crucial to understand that the underlying technology is built upon decades-old concepts. The real breakthrough lies in the improved user experience.

The Mastercard Approach: LLMs for Enhanced Fraud Detection

MasterCard actively explores LLMs for various applications. We’ve successfully used generative AI to boost fraud detection by up to 300%, as highlighted in a recent press release. This demonstrates the practical value of LLMs when applied responsibly.

Building Successful Generative AI Applications: Essential Components and Challenges

Developing successful generative AI applications requires several key components:

• Access to Diverse Foundation Models: Balancing cost and model size is critical. While access to various models isn’t a major hurdle, informed decision-making is necessary.
• Customization Environment: Existing enterprise AI environments often lack the capabilities for handling large language models. Adapting these environments presents a challenge.
• User-Friendly Tools: The lack of pre-existing tools tailored for the generative AI landscape poses a significant challenge. Most available tools are as new as LLMs themselves, requiring continuous development and refinement.
• Scalable ML Infrastructure: Scaling inference for LLMs demands significant GPU compute and RAM, often exceeding training requirements. Building and maintaining this infrastructure can be complex.

These essentials are intertwined with the challenges of LLM deployment. Access to foundation models, while available, involves cost-benefit analysis. Customization environments require significant adaptation. The lack of user-friendly tools remains a major hurdle. And finally, scalable infrastructure demands ongoing investment and optimization.

The LLM Development Pipeline: Beyond Just Code

Developing LLM applications is far more complex than simply connecting APIs. Much like traditional machine learning, the ML code itself represents a small fraction (less than 5%) of the overall end-to-end pipeline. The bulk of the work lies in building and maintaining the surrounding infrastructure, data pipelines, and safeguards. This often overlooked aspect is crucial for successful LLM deployment.

Two Approaches to LLM Deployment: Closed Book vs. RAG

Two primary approaches dominate LLM deployment:

1. Closed Book: Using a foundation model as is (zero-shot, few-shot), or fine-tuning it with domain-specific data. This approach faces challenges like hallucination, explainability issues, maintaining up-to-date information, model editing difficulties, and customization limitations.

2. Retrieval Augmented Generation (RAG): Coupling the foundation model to an external memory. This approach addresses many of the closed book limitations. It improves factual recall, reduces hallucination, allows for easier updates, enables attribution, and facilitates customization. However, RAG introduces its own set of complexities, particularly in optimizing the interaction between the retriever and generator. While common practice treats them as separate entities, the original RAG concept involves training them in parallel. This requires access to model parameters, now increasingly possible thanks to the open-source community.

The Challenges of Building End-to-End LLM Applications

Building end-to-end LLM applications presents unique challenges beyond the core algorithmic aspects. Data collection, preserving access controls within enterprise settings, addressing current AI risks, and building safeguards are crucial considerations. These factors often account for over 90% of the development effort.

At Mastercard, responsible AI development is paramount. Our seven principles of responsible AI guide our approach, focusing on privacy, security, reliability, and more. A dedicated governing body and clear strategy ensure these principles are embedded in our LLM applications. This allows us to leverage the transformative potential of LLMs while mitigating risks.

Embracing the Challenges of LLM Innovation

The numerous challenges and technical debt associated with LLM deployment can be daunting. However, embracing transformative technology often requires navigating uncharted territory. The potential benefits of LLMs outweigh the challenges, driving innovation and reshaping the technological landscape. By focusing on responsible development, addressing current risks, and continuously refining our approaches, we can harness the full power of LLMs while building a more secure and beneficial future.