Agent Harness Tutorial · End to end · Zoom level 3
The APIM gateway
Between our harness and Groq sits Azure API Management (APIM), used as an AI gateway. It is one place where Azure can check every model call, whichever agent made it.
Content Safety runs two checks on the newest user message: Prompt Shields, a trained classifier for jailbreak attempts, and a harm check for hate, sexual content, violence and self-harm on a 0–6 scale (blocked at 4 or more). The answer gets the harm check too.
It fails open. If Content Safety itself is down, or the free 5,000-a-month quota is used up, the call is let through and the verdict is recorded as "unavailable". That is why our own checks still matter: see the pre-model check.
The Groq key never lives in our code; APIM fetches it from Key Vault with a managed identity.
Want every detail? The long version is in the Series: APIM, with the live policy file.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
infra/policies/groq-api.xml:171. Only the allowed model gets through (403 model_not_allowed) <!-- Tier 4a A8: only an allowed model gets through. Checked first, so a refused
request costs no Content Safety call and no tokens. The agent checks the
same list (approved_models); this is the second wall, at the door. -->
<set-variable name="model" value="@{
var body = context.Request.Body.As<JObject>(preserveContent: true);
return body == null ? "" : ((string)body["model"] ?? "");
}" />
<choose>
<when condition="@(!"{{allowed-models}}".Split(',').Select(m => m.Trim()).Contains((string)context.Variables["model"]))">
<return-response>
<set-status code="403" reason="Model not allowed" />
<set-header name="x-content-safety" exists-action="override"><value>skipped; prompt=skipped; completion=skipped</value></set-header>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
<set-body>@(new JObject(new JProperty("error", new JObject(new JProperty("code", "model_not_allowed"), new JProperty("message", "This agent may not use model '" + (string)context.Variables["model"] + "'.")))).ToString())</set-body>
</return-response>
</when>
</choose>infra/policies/groq-api.xml:342. 30 calls a minute, and token counts into App Insights<!-- Request rate per subscription key (the agent's): 30 calls a minute. --> <rate-limit calls="30" renewal-period="60" remaining-calls-header-name="x-ratelimit-remaining" /> <!-- Token counts per call into Application Insights (customMetrics). --> <llm-emit-token-metric namespace="claims-fnol"> <dimension name="API ID" /> <dimension name="Subscription ID" /> </llm-emit-token-metric>
infra/policies/groq-api.xml:413. Content Safety on the newest user message: Prompt Shields and the harm check <!-- The newest user message: the text Content Safety screens. -->
<set-variable name="prompt" value="@{
var body = context.Request.Body.As<JObject>(preserveContent: true);
var messages = body["messages"] as JArray;
if (messages == null) { return ""; }
for (int i = messages.Count - 1; i >= 0; i--) {
if ((string)messages[i]["role"] == "user") {
var content = messages[i]["content"];
var text = content.Type == JTokenType.String ? (string)content : content.ToString();
return text.Length > 9000 ? text.Substring(text.Length - 9000) : text;
}
}
return "";
}" />
<choose>
<when condition="@(((string)context.Variables["prompt"]).Length > 0)">
<send-request mode="new" response-variable-name="shield" timeout="10" ignore-error="true">
<set-url>{{content-safety-endpoint}}contentsafety/text:shieldPrompt?api-version=2024-09-01</set-url>
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
<set-body>@(new JObject(new JProperty("userPrompt", (string)context.Variables["prompt"]), new JProperty("documents", new JArray())).ToString())</set-body>
<authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
</send-request>
<send-request mode="new" response-variable-name="promptHarm" timeout="10" ignore-error="true">
<set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
<set-method>POST</set-method>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
<set-body>@(new JObject(new JProperty("text", (string)context.Variables["prompt"]), new JProperty("outputType", "FourSeverityLevels")).ToString())</set-body>
<authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
</send-request>
<set-variable name="promptVerdict" value="@{infra/policies/groq-api.xml:714. The verdict: block on an attack or severity ≥ threshold; "unavailable" passes (fails open) <set-variable name="promptVerdict" value="@{
var shield = context.Variables.GetValueOrDefault<IResponse>("shield");
var harm = context.Variables.GetValueOrDefault<IResponse>("promptHarm");
if (shield == null || harm == null || shield.StatusCode != 200 || harm.StatusCode != 200) { return "unavailable"; }
var attack = shield.Body.As<JObject>(preserveContent: true)["userPromptAnalysis"]?["attackDetected"];
if (attack != null && (bool)attack) { return "block:prompt-attack"; }
foreach (var c in (JArray)harm.Body.As<JObject>(preserveContent: true)["categoriesAnalysis"]) {
if ((int)c["severity"] >= {{content-safety-threshold}}) { return "block:" + (string)c["category"]; }
}
return "pass";
}" />
<choose>
<when condition="@(((string)context.Variables["promptVerdict"]).StartsWith("block"))">
<return-response>
<set-status code="403" reason="Blocked by Content Safety" />
<set-header name="x-content-safety" exists-action="override"><value>@((string)context.Variables["promptVerdict"] + "; prompt=" + (string)context.Variables["promptVerdict"] + "; completion=skipped")</value></set-header>
<set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
<set-body>{"error":{"code":"content_safety","message":"the prompt was blocked by Content Safety"}}</set-body>
</return-response>
</when>
</choose>infra/policies/groq-api.xml:945. Forward to Groq with the key from Key Vault; the agent's own key never leaves APIM <!-- To Groq, with the key from Key Vault; the agent's subscription key stays here. -->
<set-backend-service backend-id="groq" />
<set-header name="Authorization" exists-action="override"><value>Bearer {{groq-api-key}}</value></set-header>
<set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />