Agent Harness Tutorial · End to end · Zoom level 3
How we take care of it: Content Safety
Content Safety catches jailbreaks (Prompt Shields) and harm.
How we took care of it.
- APIM sends the text to Azure AI Content Safety: Prompt Shields (a trained jailbreak classifier) and a harm check.
- Harm is scored 0–6 for hate, sexual content, violence and self-harm; 4 or more blocks.
- The verdict comes back in the x-content-safety header and is written on the model span.
- It fails open: if the service is down or the 5,000-a-month quota is used, the call passes as "unavailable".
| It reads | the newest user message (last 9,000 chars) before the model, and the answer after it |
| Kind | classic ML |
| Provided by | Azure APIM |
| If it fails | 403 content_safety → "I am having trouble answering right now." |
| Where in the code | infra/policies/groq-api.xml:41-92 |
| Runs at | 2 · Before the model |
Real exampleFlow 18: a jailbreak prompt got 403 in 366 ms. Flow 1b: 5 checks, all passed. Gap F-124: tool results are not screened.
More on this: The APIM gateway.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
infra/policies/groq-api.xml:41On the prompt <!-- The newest user message: the text Content Safety screens. -->
<set-variable name="prompt" value="@{
var body = context.Request.Body.As<JObject>(preserveContent: true);
var messages = body["messages"] as JArray;
if (messages == null) { return ""; }
for (int i = messages.Count - 1; i >= 0; i--) {
if ((string)messages[i]["role"] == "user") {
var content = messages[i]["content"];
var text = content.Type == JTokenType.String ? (string)content : content.ToString();
return text.Length > 9000 ? text.Substring(text.Length - 9000) : text;
}
}
return "";
}" />
<choose>
<when condition="@(((string)context.Variables["prompt"]).Length > 0)">
<send-request mode="new" response-variable-name="shield" timeout="10" ignore-error="true">
<set-url>{{content-safety-endpoint}}contentsafety/text:shieldPrompt?api-version=2024-09-01</set-url>
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
<set-body>@(new JObject(new JProperty("userPrompt", (string)context.Variables["prompt"]), new JProperty("documents", new JArray())).ToString())</set-body>
<authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
</send-request>
<send-request mode="new" response-variable-name="promptHarm" timeout="10" ignore-error="true">
<set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
<set-body>@(new JObject(new JProperty("text", (string)context.Variables["prompt"]), new JProperty("outputType", "FourSeverityLevels")).ToString())</set-body>
<authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
</send-request>
<set-variable name="promptVerdict" value="@{
var shield = context.Variables.GetValueOrDefault<IResponse>("shield");
var harm = context.Variables.GetValueOrDefault<IResponse>("promptHarm");
if (shield == null || harm == null || shield.StatusCode != 200 || harm.StatusCode != 200) { return "unavailable"; }infra/policies/groq-api.xml:104On the answer <set-variable name="completion" value="@{
if (context.Response.StatusCode != 200) { return ""; }
var body = context.Response.Body.As<JObject>(preserveContent: true);
var text = (string)body["choices"]?[0]?["message"]?["content"] ?? "";
return text.Length > 9000 ? text.Substring(0, 9000) : text;
}" />
<choose>
<when condition="@(((string)context.Variables["completion"]).Length > 0)">
<send-request mode="new" response-variable-name="completionHarm" timeout="10" ignore-error="true">
<set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>