By
·
5 minute read
Our journey toward designing the R.A.I.L.G.U.A.R.D. Framework began more than a month ago, after reading Pillar Security’s article on how GenAI coding agents can be exploited through prompt and CursorRule poisoning.
Generative AI coding assistant tools like Copilot, Cursor, and Windsurf are now widely used across the software industry. But what struck us most wasn’t just the attack surface, it was the root cause. These tools are powered by large transformer models.
To understand why these assistants are prone to security flaws, we must look at the ANN + Transformer architecture. The same behavior that makes them powerful, absorbing context and predicting the next token, also makes them vulnerable to:
- Prompt injections (especially indirect remote ones)
- Hallucinations
- Misinterpretations based on misleading context
Understanding the Security Impact of Self-Attention
First things first: we can’t fully understand GenAI security issues without understanding self-attention in LLMs. Let me explain.
Self-attention allows LLMs to weigh tokens by relevance, not just by their position in the sequence. This allows the model to track long-range dependencies, for example, linking a try block to a catch several lines later. It builds a probabilistic graph of what matters most in the input.
⇒ So here’s the key insight: Context is dynamic, and the model’s understanding of “intent” can be easily flawed.
Example: Insecure by Context
A visual comparison between insecure output and a RAILGUARD-guided secure alternative.
# Prototype for a simple admin panel to quickly demonstrate proof of conceptquery = f"SELECT * FROM users WHERE email = '{email}'"
This code is dangerous — and not just because of string interpolation.
Due to the comment (prototype, quick, POC), the model downweights the need for security. It has likely seen thousands of similar examples during training like demo code from GitHub, Stack Overflow, tutorials, blogs etc where speed > security.
So instead of enforcing prepared statements or an ORM, the model prioritizes simplicity, and skips critical security checks.
Pattern Recognition ≠ Secure Reasoning
Self-attention enables the model to reproduce patterns it has seen, even if those patterns aren’t secure.
Worse, LLMs can:
- Follow poisoned comments
- Be influenced by invisible characters
- Be tricked by pernicious variable names
Transformer-based coding assistants are vulnerable by design. The LLM doesn’t “know” what it’s doing is wrong, it’s just following patterns that look plausible in context.
Reasoning-Time Security: Why We Created R.A.I.L.G.U.A.R.D.
That’s why we built the R.A.I.L.G.U.A.R.D. Framework, to inject secure reasoning constraints before code is even generated.
RAILGUARD embeds safety signals into the reasoning path using domain-specific rules (like CursorRules), Reinforced constraints and secure context defaults. We don’t secure the output. We secure the reasoning process.
The 8 Blocks of R.A.I.L.G.U.A.R.D.
Each block contributes to reasoning-time security shaping. Here they are :
R — Risk First
Define the security goal of the rule. Push the LLM to “think” before acting. In practice, this involves the model performing an internal check or “threat modeling” step: What could go wrong in fulfilling this request?
For example, if the user asks to parse a file upload, a risk-first AI might recognize the potential for malicious file content and decide to include safe file-handling practices (or at least warn about them) rather than blindly outputting a quick-and-dirty parser.
A — Attached Constraints
Specify what must never happen (boundaries and red lines) to prevent regressions. The “Attached Constraints” principle ensures the AI is never operating on an empty “rulebook”. In current systems, an AI code assistant might have only a generic content filter or a high-level alignment tuning. RAILGUARD instead attaches a detailed set of security and coding constraints that are always fed into the model (for example, via a system prompt or a persistent context file). These constraints cover things like approved cryptographic practices, prohibited functions, style guidelines, and more. Essentially, the AI always carries with it a secure coding standard that it must follow. This mitigates the transformer’s tendency to be overly permissive. The model constantly “hears” a little voice saying, “Remember, these are the rules you must obey.” With attached constraints, an attacker’s prompt (like a jailbreak) simply can’t override them. Why ? The model would simply not comply because it will give higher priority to the rules attached at all time.
I — Interpretative Framing
The idea is to guide how the LLM should interpret prompts securely and avoid blind trust in vague instructions. “Interpretative Framing” means the AI will actively rephrase or reinterpret the request internally to ensure it aligns with safe and intended behavior. LLMs are highly context-dependent, a slight change in how a task is framed can alter the output.
For example, a user asks: “Open a file and print the first line.” A typical model might comply literally as the user says. An AI using interpretative framing might internally expand this to: “Open a file (make sure the path is safe and handle errors) and print the first line (avoid any unsafe content).” The idea is not to change what the user wants, but to incorporate security considerations that the user may not have stated.
L — Local Defaults
Set project-specific or environment-level secure defaults (for example, use env vars, not hardcoded secrets). The “Local Defaults” principle ensures that in the absence of explicit instructions, the AI always falls back on secure defaults. For example, if we generate an API, the default would be to include authentication checks and input sanitization, or if we generate a SQL query, the default is to use parameterized queries or ORMs, instead of string concatenation. The idea is to bring security by default as normal practice, and RAILGUARD brings that philosophy inside the GenAI process.
G — Generative Path Checks
It provides a sequence of reasoning the model must follow before writing outputs, and introduces a feedback loop during the code generation. The model is asked to reason about its solution. This approach allows catching mistakes that would become either hallucinations or vulnerabilities. It improves not only security, but also accuracy.
U — Uncertainty Disclosure
Tell the LLM what to do when unsure : ask, warn, or abstain. Hallucinations are a notorious problem in LLMs. In GenAI coding, uncertainty can arise if the prompt is vague or outside the LLM’s knowledge. The real issues come when an LLM confidently states incorrect information, it can introduce security issues.
A — Auditability
It embeds comments or trace markers to allow human review and verification. “Auditability” is about ensuring that the AI’s contributions are traceable and understandable.
In RAILGUARD it means the AI should provide explanations or annotations for its decisions, particularly any that might be security-relevant. In practice, this could mean the AI accompanies its code with a brief statement about its reasoning in comments. For example: “// Using prepared statements to prevent SQL injection” or “// Added rate limiting to mitigate brute-force attacks as per security policy.” These comments justify the inclusion of security measures. From a security point of view, auditability is a failsafe in my opinion.
R+D — Revision + Dialogue
The combination of these two principles recognize that coding is an iterative process and enforce the AI’s role as a collaborative partner rather than an infallible oracle. No complex code is perfect in one draft, human or AI. By embracing revision, the AI is essentially allowed to say “let me try that again” or to accept feedback and improve. Dialogue means the AI remains engaged in a conversation about the code, rather than dumping code and disengaging. From the security standpoint, this principle is about continuous improvement and defense-in-depth. So in practice, the devs or even LLMs can revise or question potentially insecure code through /commands.
The Future of AppSec Is Generative
Now we understand that transformer-based models don’t “understand” code, they contextualize it. They don’t follow rules, they pattern-match, interpolate, and weigh relevance. That’s their strength, but it’s also their vulnerability. Security in the GenAI era can’t rely solely on static linters, rule checkers, or post-generation patching. We need to meet the models where they operate, at the level of self-attention, token weighting, and probabilistic context reasoning.
The RAILGUARD Framework doesn’t aim to restrict the model, it teaches it. It scaffolds secure behavior before the first line of code is even generated. By implementing Risk First, Attached Constraints, Interpretative Framing, Local Defaults, Generative Path Checks, Uncertainty Disclosure, Auditability, and Revision + Dialogue, we effectively build a multi-layered defense around the AI’s generative process. One way to summarize it: It shapes secure reasoning.
With all of this, Cursor Rules become the medium, and RAILGUARD becomes the mindset. Together, they allow us to build code-generation pipelines that don’t just output code that looks right, but that thinks securely by design.
GenAI coding is already important, and might become the main producer of code in a not so far future. Its security must be reasoned at generation time.
Thank you for reading us.
The BrightOnLABS Team