cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Core loop › Model call › The APIM gateway

Agent Harness Tutorial · End to end · Zoom level 3

The APIM gateway

Between our harness and Groq sits Azure API Management (APIM), used as an AI gateway. It is one place where Azure can check every model call, whichever agent made it.

Allowed model?only gpt-oss-120bstop: 403Rate limit30 calls a minutestop: 429Content Safety on the promptjailbreak + harm checkstop: 403Fetch Groq's keyfrom Key VaultForward to GroqContent Safety on the answerharm checkstop: 403
Grey boxes can stop the call. The verdict comes back in a header called x-content-safety and is written to the trace.

Content Safety runs two checks on the newest user message: Prompt Shields, a trained classifier for jailbreak attempts, and a harm check for hate, sexual content, violence and self-harm on a 0–6 scale (blocked at 4 or more). The answer gets the harm check too.

It fails open. If Content Safety itself is down, or the free 5,000-a-month quota is used up, the call is let through and the verdict is recorded as "unavailable". That is why our own checks still matter: see the pre-model check.

The Groq key never lives in our code; APIM fetches it from Key Vault with a managed identity.

Want every detail? The long version is in the Series: APIM, with the live policy file.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

infra/policies/groq-api.xml:171. Only the allowed model gets through (403 model_not_allowed)
    <!-- Tier 4a A8: only an allowed model gets through. Checked first, so a refused
         request costs no Content Safety call and no tokens. The agent checks the
         same list (approved_models); this is the second wall, at the door. -->
    <set-variable name="model" value="@{
        var body = context.Request.Body.As<JObject>(preserveContent: true);
        return body == null ? "" : ((string)body["model"] ?? "");
    }" />
    <choose>
      <when condition="@(!"{{allowed-models}}".Split(',').Select(m => m.Trim()).Contains((string)context.Variables["model"]))">
        <return-response>
          <set-status code="403" reason="Model not allowed" />
          <set-header name="x-content-safety" exists-action="override"><value>skipped; prompt=skipped; completion=skipped</value></set-header>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
          <set-body>@(new JObject(new JProperty("error", new JObject(new JProperty("code", "model_not_allowed"), new JProperty("message", "This agent may not use model '" + (string)context.Variables["model"] + "'.")))).ToString())</set-body>
        </return-response>
      </when>
    </choose>
infra/policies/groq-api.xml:342. 30 calls a minute, and token counts into App Insights
    <!-- Request rate per subscription key (the agent's): 30 calls a minute. -->
    <rate-limit calls="30" renewal-period="60" remaining-calls-header-name="x-ratelimit-remaining" />
    <!-- Token counts per call into Application Insights (customMetrics). -->
    <llm-emit-token-metric namespace="claims-fnol">
      <dimension name="API ID" />
      <dimension name="Subscription ID" />
    </llm-emit-token-metric>
infra/policies/groq-api.xml:413. Content Safety on the newest user message: Prompt Shields and the harm check
    <!-- The newest user message: the text Content Safety screens. -->
    <set-variable name="prompt" value="@{
        var body = context.Request.Body.As<JObject>(preserveContent: true);
        var messages = body["messages"] as JArray;
        if (messages == null) { return ""; }
        for (int i = messages.Count - 1; i >= 0; i--) {
            if ((string)messages[i]["role"] == "user") {
                var content = messages[i]["content"];
                var text = content.Type == JTokenType.String ? (string)content : content.ToString();
                return text.Length > 9000 ? text.Substring(text.Length - 9000) : text;
            }
        }
        return "";
    }" />
    <choose>
      <when condition="@(((string)context.Variables["prompt"]).Length > 0)">
        <send-request mode="new" response-variable-name="shield" timeout="10" ignore-error="true">
          <set-url>{{content-safety-endpoint}}contentsafety/text:shieldPrompt?api-version=2024-09-01</set-url>
          <set-method>POST</set-method>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
          <set-body>@(new JObject(new JProperty("userPrompt", (string)context.Variables["prompt"]), new JProperty("documents", new JArray())).ToString())</set-body>
          <authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
        </send-request>
        <send-request mode="new" response-variable-name="promptHarm" timeout="10" ignore-error="true">
          <set-url>{{content-safety-endpoint}}contentsafety/text:analyze?api-version=2024-09-01</set-url>
          <set-method>POST</set-method>
          <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
          <set-body>@(new JObject(new JProperty("text", (string)context.Variables["prompt"]), new JProperty("outputType", "FourSeverityLevels")).ToString())</set-body>
          <authentication-managed-identity resource="https://cognitiveservices.azure.com" client-id="{{apim-identity-client-id}}" />
        </send-request>
        <set-variable name="promptVerdict" value="@{
infra/policies/groq-api.xml:714. The verdict: block on an attack or severity ≥ threshold; "unavailable" passes (fails open)
        <set-variable name="promptVerdict" value="@{
            var shield = context.Variables.GetValueOrDefault<IResponse>("shield");
            var harm = context.Variables.GetValueOrDefault<IResponse>("promptHarm");
            if (shield == null || harm == null || shield.StatusCode != 200 || harm.StatusCode != 200) { return "unavailable"; }
            var attack = shield.Body.As<JObject>(preserveContent: true)["userPromptAnalysis"]?["attackDetected"];
            if (attack != null && (bool)attack) { return "block:prompt-attack"; }
            foreach (var c in (JArray)harm.Body.As<JObject>(preserveContent: true)["categoriesAnalysis"]) {
                if ((int)c["severity"] >= {{content-safety-threshold}}) { return "block:" + (string)c["category"]; }
            }
            return "pass";
        }" />
        <choose>
          <when condition="@(((string)context.Variables["promptVerdict"]).StartsWith("block"))">
            <return-response>
              <set-status code="403" reason="Blocked by Content Safety" />
              <set-header name="x-content-safety" exists-action="override"><value>@((string)context.Variables["promptVerdict"] + "; prompt=" + (string)context.Variables["promptVerdict"] + "; completion=skipped")</value></set-header>
              <set-header name="Content-Type" exists-action="override"><value>application/json</value></set-header>
              <set-body>{"error":{"code":"content_safety","message":"the prompt was blocked by Content Safety"}}</set-body>
            </return-response>
          </when>
        </choose>
infra/policies/groq-api.xml:945. Forward to Groq with the key from Key Vault; the agent's own key never leaves APIM
    <!-- To Groq, with the key from Key Vault; the agent's subscription key stays here. -->
    <set-backend-service backend-id="groq" />
    <set-header name="Authorization" exists-action="override"><value>Bearer {{groq-api-key}}</value></set-header>
    <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />