cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks › 2 · Before the model › Content Safety

Agent Harness Tutorial · End to end · Zoom level 3

How we take care of it: Content Safety

Content Safety catches jailbreaks (Prompt Shields) and harm.

Open: 11before loopopen ›Open: 22before modelopen ›Open: 33after modelopen ›Open: 44before toolopen ›Open: 55the toolopen ›Open: 66after toolopen ›Open: 77after loopopen ›Open: 88before replyopen ›It readsthe newest user message (last9,000 chars) before the model, andContent Safetyclassic ML · Azure APIMpassCarries onfailIf it fails403 content_safety → "I am havingWHERE IN THE CODE: infra/policies/groq-api.xml:41-92
The strip at the top shows every check point where this check runs, in dark.

How we took care of it.

  1. APIM sends the text to Azure AI Content Safety: Prompt Shields (a trained jailbreak classifier) and a harm check.
  2. Harm is scored 0–6 for hate, sexual content, violence and self-harm; 4 or more blocks.
  3. The verdict comes back in the x-content-safety header and is written on the model span.
  4. It fails open: if the service is down or the 5,000-a-month quota is used, the call passes as "unavailable".
It readsthe newest user message (last 9,000 chars) before the model, and the answer after it
Kindclassic ML
Provided byAzure APIM
If it fails403 content_safety → "I am having trouble answering right now."
Where in the codeinfra/policies/groq-api.xml:41-92
Runs at2 · Before the model
Real exampleFlow 18: a jailbreak prompt got 403 in 366 ms. Flow 1b: 5 checks, all passed. Gap F-124: tool results are not screened.

More on this: The APIM gateway.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

    <!-- The newest user message: the text Content Safety screens. -->
    <set-variable name="prompt" value="@{
        var body = context.Request.Body.As<JObject>(preserveContent: true);
        var messages = body["messages"] as JArray;
        if (messages == null) { return ""; }
        for (int i = messages.Count - 1; i >= 0; i--) {
            if ((string)messages[i]["role"] == "user") {
                var content = messages[i]["content"];
                var text = content.Type == JTokenType.String ? (string)content : content.ToString();
                return text.Length > 9000 ? text.Substring(text.Length - 9000) : text;
            }
        }
        return "";
    }" />
    <choose>
      <when condition="@(((string)context.Variables["prompt"]).Length > 0)">
        <send-request mode="new" response-variable-name="shield" timeout="10" ignore-error="true">
          <set-url>{{content-safety-endpoint}}contentsafety/text:shieldPrompt?api-version=2024-09-01</set-url>
          <set-method>POST</set-method>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
          <set-body>@(new JObject(new JProperty("userPrompt", (string)context.Variables["prompt"]), new JProperty("documents", new JArray())).ToString())</set-body>
          <authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
        </send-request>
        <send-request mode="new" response-variable-name="promptHarm" timeout="10" ignore-error="true">
          <set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
          <set-method>POST</set-method>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
          <set-body>@(new JObject(new JProperty("text", (string)context.Variables["prompt"]), new JProperty("outputType", "FourSeverityLevels")).ToString())</set-body>
          <authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
        </send-request>
        <set-variable name="promptVerdict" value="@{
            var shield = context.Variables.GetValueOrDefault<IResponse>("shield");
            var harm = context.Variables.GetValueOrDefault<IResponse>("promptHarm");
            if (shield == null || harm == null || shield.StatusCode != 200 || harm.StatusCode != 200) { return "unavailable"; }
    <set-variable name="completion" value="@{
        if (context.Response.StatusCode != 200) { return ""; }
        var body = context.Response.Body.As<JObject>(preserveContent: true);
        var text = (string)body["choices"]?[0]?["message"]?["content"] ?? "";
        return text.Length > 9000 ? text.Substring(0, 9000) : text;
    }" />
    <choose>
      <when condition="@(((string)context.Variables["completion"]).Length > 0)">
        <send-request mode="new" response-variable-name="completionHarm" timeout="10" ignore-error="true">
          <set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
          <set-method>POST</set-method>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>