The system prompt as an attack point: tracing a dangerous data flow

A change that looks entirely reasonable

A colleague asks: “Put the ticket description directly into the instruction so the answer is more accurate.” The developer updates the function:

const BASE_SYSTEM = "Summarize support tickets for an operator.";

function addContext(base: string, value: string) {
  return `${base}\nTicket context:\n${value.trim()}`;
}

export async function summarize(req: Request) {
  const ticket = req.body.ticket;
  const system = addContext(BASE_SYSTEM, ticket);

  return llm.generate({
    system,
    user: "Prepare a short summary."
  });
}

The code does not resemble a conventional exploit. However, the user controls ticket, and its text becomes part of the highest-level instruction. Trimming whitespace does not change its trust level.

Map the source → propagation → sink route

The route in this example is short:

1. Mark req.body.ticket as the source.
2. The addContext call, template literal, and trim() operation form the propagation path.
3. The system field passed to the language-model client is the sink.

This route matters more than a text search for system or prompt. Variable names can change, while the value may travel through several modules, objects, and helper functions. The actual data flow determines the risk.

Use a manual review runbook

Start by inventorying calls to OpenAI, Anthropic, Google GenAI, and other language-model clients. For each call, record where the system messages are constructed.

Then inspect every inserted value:

• Does it originate in an HTTP request, form, file, or external API?
• Can a user, supplier, or another system modify it?
• Which functions transform or forward it?
• Which message role receives it at the end?

The trust boundary is not limited to a web form. An uploaded document or a partner API response can also contain untrusted content.

What SAST adds and how to triage a result

CodeQL 2.26.0 introduced the js/system-prompt-injection query for JavaScript and TypeScript. It is a current example of how SAST can trace data into known system-message sinks. The release also expanded sink modeling for OpenAI, Anthropic, and Google GenAI calls.

Before scanning, verify the analyzer version and query suite. For every result, open its path and confirm the source, propagation, and final message role.

Do not dismiss a result as a false positive merely because the code calls trim(), removes HTML, or limits the input length. A justified exception could be a value selected on the server from a fixed allowlist without including submitted text. The code and tests should make that guarantee visible.

Rebuild the trust boundary

The basic remediation is to keep the system instruction static and send third-party text as user content:

function buildMessages(ticket: string) {
  return [
    { role: "system", content: "Summarize support tickets for an operator." },
    { role: "user", content: `Ticket to summarize:\n${ticket}` }
  ];
}

If the application needs a dynamic style, map a short identifier to a server-owned value through an allowlist. Do not insert the raw style field into the instruction. Anti-patterns include concatenation, hidden role mixing inside a helper, and assuming that escaping makes natural-language input trusted.

Add a negative test

The test should enforce an architectural property rather than attempt to prove that every model response is safe:

it("keeps untrusted ticket text out of the system message", () => {
  const marker = "IGNORE PRIOR INSTRUCTIONS";
  const messages = buildMessages(marker);

  expect(messages.find(m => m.role === "system")?.content)
    .not.toContain(marker);
  expect(messages.find(m => m.role === "user")?.content)
    .toContain(marker);
});

This test preserves the repaired structure during later refactoring. Separate behavioral evaluations are still required for actual model responses and tool calls.

Roll out the check gradually

Begin with an inventory and create a baseline. Manually triage its results, fix confirmed flows, and document narrow exceptions. Next, add a non-blocking pull-request check that reports new findings. Enable blocking only after the rules and ownership of triage have stabilized.

SAST cannot observe every message assembled at runtime or evaluate model behavior. The system also needs threat modeling, least-privilege tools, authorization for every action, output validation, logging, and runtime monitoring. Separating message roles removes a dangerous flow, but it is not complete protection against prompt injection.

Sources

• CodeQL 2.26.0: AI prompt injection detection
• About data flow analysis in CodeQL
• About code scanning with CodeQL