LLMs' 'Lying' Problem: From AI Slop to Security Threats

The L in 'LLM' Stands for Lying

The relentless hype around Large Language Models suggests a revolution in productivity and creativity. Yet, years into their deployment, the tangible results often feel underwhelming and fraught with new problems. The core issue, as argued by developer Steven Wittens, is that LLMs are fundamentally engines of imitation and forgery, not authentic creation.

This act of forgery—producing imitations faster than genuine output—is creating systemic issues. In software development, it manifests as a flood of low-quality, 'slop' code that degrades open-source projects and burdens maintainers. The craft of coding is being threatened by cheap, AI-generated imitations that lack understanding and authenticity.

Craft vs. Kraft: The Forgery Framework

Wittens frames the LLM's output through the lens of forgery. Whether it's a painting, a legal document, or a study, a forgery's nature is intrinsic to its method of creation. LLMs allow individuals to forge their own potential output or that of others at scale. The problems arise when these forgeries are passed off as authentic work.

This parallels protections for artisanal goods like French Brie de Meaux, where geographic origin and traditional methods are safeguarded to preserve value and expertise. Society draws lines to prevent a race to the bottom, from authentic cheese to 'fake eggs made from industrial chemicals.' Software, however, lacks such protections, leaving it vulnerable to degradation.

The 'Slop' Flood: Open Source and Engineering Under Siege

The impact is acutely felt in open source. Maintainers, already struggling for contributors, are inundated with AI-generated pull requests from 'vibe-coders' seeking to pad their resumes. This has led projects to close public contributions or drop bug bounties. The feedback loop is absurd: suggestions are merely re-pasted into the AI.

In corporate settings, new hires using AI may seem productive quickly, but they inject generic mediocrity instead of bringing a fresh perspective. Worse, as Wittens notes, even senior engineers can produce 'highly embarrassing goofs' via AI, treating code as a disposable output rather than a liability. The promise of 10x or 100x productivity ignores the fundamental truth that every line of code is a liability that requires understanding.

Distrust and Verify: The Marketer's AI Nightmare

The forgery problem extends beyond code into the very information LLMs consume and regurgitate. Recent data highlights a critical shift: YouTube has overtaken Reddit as the top source for LLM training data, with Reddit remaining a close second, according to Bluefish data cited by MediaPost.

This poses a severe problem for marketers. LLMs favor 'community-driven platforms like Reddit, where heavy moderation makes it harder for brands to directly influence authentic conversations.' In other words, brands have lost control of their narratives to AI systems that amplify organic, often negative, user sentiment.

Analysis by Profound and Semrush confirms the dominance of these sources. Reddit accounts for 40% of citations generated by major AI search tools (Perplexity, ChatGPT Search, Google AI Overviews), well ahead of Wikipedia at 26%. It is the most-cited domain for Perplexity and Google AI Overviews and a top-three source for others.

The content cited is often low-engagement posts or Q&A threads, not authoritative articles. This creates a 'sea of negativity' for marketers, as consumers are far more likely to share bad experiences. A Zendesk study found 54% share bad experiences with five+ people, versus 33% for good ones. An American Express survey noted that while 46% always share good service, 60% always share bad, telling three times more people.

Security: Treat LLM Output as Untrusted

The reliability issue becomes a direct security threat. As Ankit Gupta writes for CSOonline, LLMs have arrived in security as productivity tools, embedded components, and as targets themselves. The core rule must be: treat LLM output as untrusted.

Gupta outlines how attackers can leverage LLMs: as persuasion engines for better scams, as productivity engines for faster malware development, and as targets for prompt theft or data poisoning. The OWASP Top 10 for LLM applications catalogs real risks like prompt injection, insecure output handling, and sensitive information disclosure.

The defensive implication is that existing controls for verification, identity proofing, and process hardening are more critical than ever. If an approval workflow can be bypassed by a convincing message, an LLM will help attackers find and exploit that weakness faster.

Consumer Pushback and the Attribution Imperative

Not all industries have capitulated. The video game market shows effective consumer pushback. Numerous titles have apologized for and removed unlabeled AI-generated content. Platforms like Steam have clear disclosure policies, though they recently updated them to exclude AI tools used only for developer 'efficiency gains.'

This highlights a key distinction: art requires authenticity and a creator's unique vision. Gamers, as direct consumers, demand transparency. This stands in stark contrast to software, where reuse is often beneficial, and artistic design in open source is instantly pilfered.

The root of the problem, as Wittens argues, is the lack of sourcing. LLMs are trained on 'shadow libraries' of pirated books and code, creating a 'plausible deniability' that makes it impossible to distinguish citation, hallucination, or novelty. Labeling or watermarking AI content is 'largely an exercise in ass-covering.'

The Path Forward: Sourcing as a Requirement

The proposed solution is technically daunting but conceptually simple: LLMs must perform correct source attribution alongside inference. This would reveal how much 'vibe code' is merely copy-pasted, omitting original authors and licenses. It would also validate information.

Currently, citation in LLMs is an emergent property of the training data and prompt context—essentially 'citation role-play.' There's no guarantee of accuracy or provenance. Implementing true attribution would require a fundamental rethinking of model architecture, making backpropagation and forward passes auditable.

This is what 'AI detection tools' are trying to solve backwards. It's a profound irony that the technology heralded as the successor to the search engine is fundamentally incapable of disclosing its sources. Until this changes, the output of LLMs should be treated as a forgery—inauthentic until proven otherwise.

The conundrum leaves engineers who rely on AI in an awkward position: to justify its use, they must consider their own output disposable, uncreative, and unworthy of credit. Meanwhile, as LLMs increasingly mediate information and automate tasks, the industries that fail to understand their inherent flaws—from software development to marketing to cybersecurity—risk being overwhelmed by a rising tide of slop.