cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Checks › 3 · After the model › Content Safety on the answer

Agent Harness Tutorial · End to end · Zoom level 3

How we take care of it: Content Safety on the answer

Content Safety on the answer catches harmful text.

Open: 11before loopopen ›Open: 22before modelopen ›Open: 33after modelopen ›Open: 44before toolopen ›Open: 55the toolopen ›Open: 66after toolopen ›Open: 77after loopopen ›Open: 88before replyopen ›It readsthe model's answer, on the wayback through APIMContent Safety on the answerclassic ML · Azure APIMpassCarries onfailIf it fails403 → "I am having trouble answeringWHERE IN THE CODE: infra/policies/groq-api.xml
The strip at the top shows every check point where this check runs, in dark.

How we took care of it.

  1. The same harm check as on the prompt, run on the completion.
  2. A harmful answer becomes a 403 before it reaches the harness.
It readsthe model's answer, on the way back through APIM
Kindclassic ML
Provided byAzure APIM
If it fails403 → "I am having trouble answering right now."
Where in the codeinfra/policies/groq-api.xml
Runs at3 · After the model
Real exampleNo real answer has been blocked so far.

More on this: The APIM gateway.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

    <set-variable name="completion" value="@{
        if (context.Response.StatusCode != 200) { return ""; }
        var body = context.Response.Body.As<JObject>(preserveContent: true);
        var text = (string)body["choices"]?[0]?["message"]?["content"] ?? "";
        return text.Length > 9000 ? text.Substring(0, 9000) : text;
    }" />
    <choose>
      <when condition="@(((string)context.Variables["completion"]).Length > 0)">
        <send-request mode="new" response-variable-name="completionHarm" timeout="10" ignore-error="true">
          <set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
          <set-method>POST</set-method>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
          <set-body>@(new JObject(new JProperty("text", (string)context.Variables["completion"]), new JProperty("outputType", "FourSeverityLevels")).ToString())</set-body>
          <authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
        </send-request>
        <set-variable name="completionVerdict" value="@{
            var harm = context.Variables.GetValueOrDefault<IResponse>("completionHarm");
            if (harm == null || harm.StatusCode != 200) { return "unavailable"; }
            foreach (var c in (JArray)harm.Body.As<JObject>(preserveContent: true)["categoriesAnalysis"]) {
                if ((int)c["severity"] >= {{content-safety-threshold}}) { return "block:" + (string)c["category"]; }
            }
            return "pass";
        }" />
        <choose>
          <when condition="@(((string)context.Variables["completionVerdict"]).StartsWith("block"))">
            <return-response>
              <set-status code="403" reason="Blocked by Content Safety" />
              <set-header name="x-content-safety" exists-action="override"><value>@((string)context.Variables["completionVerdict"] + "; prompt=" + context.Variables.GetValueOrDefault<string>("promptVerdict", "skipped") + "; completion=" + (string)context.Variables["completionVerdict"])</value></set-header>
              <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
              <set-body>{"error":{"code":"content_safety","message":"the completion was blocked by Content Safety"}}</set-body>