Scaling a Microsoft Foundry AI agent to thousands of users can pair an AI Gateway built on Azure API Management with Agent 365 Publishing to Teams and M365 Copilot. The gateway enforces token-per-minute limits per department using the llm-token-limit policy, since request counts alone don't reflect wildly varying token sizes. Semantic caching returns repeated answers in 15 milliseconds at zero token cost using vector similarity. A multi-region failover setup retries requests in a secondary region like West US when the primary region returns HTTP 429. Agent publishing to Teams and M365 Copilot is now Generally Available, letting Foundry auto-generate the app manifest and Entra Agent ID. A tip recommends using UserEntraToken passthrough so agents only search files the logged-in user can already access.
Table of contents
Why Put an AI Gateway in Front of Microsoft Foundry?Publishing Your Agent to Microsoft Teams and M365 CopilotQuestions this post answers
How do I rate limit Azure OpenAI or Foundry requests by tokens instead of request count?
Use the llm-token-limit policy in Azure API Management, which counts actual tokens per minute (TPM) rather than requests per minute. This matters because one request can be 10 tokens while another is 80,000 tokens, so counting requests alone doesn't prevent a noisy department from exhausting the quota. A sample policy sets tokens-per-minute to 50000 keyed on the client subscription ID. daily.dev surfaces practical Azure API Management patterns for teams sizing token-based rate limits.
How can I avoid paying for duplicate LLM calls when many users ask the same question?
Semantic caching in an AI Gateway uses vector similarity to detect repeated questions and returns a cached answer instead of calling the model again. For example, if 500 employees ask when the company holiday party is on the same morning, the cached response returns in 15 milliseconds at zero token cost instead of generating the answer 500 times. Developers cutting LLM costs can track gateway caching techniques like this on daily.dev.
Is publishing a Microsoft Foundry agent to Microsoft Teams and M365 Copilot generally available?
Yes, agent publishing to Microsoft 365 and Teams is Generally Available (GA) in the Microsoft Foundry portal. From Build > Agents, selecting Publish and choosing Microsoft 365 Copilot and Microsoft Teams causes Foundry to auto-generate the app manifest and wire the agent's endpoint and Entra Agent ID into the Microsoft 365 admin center for approved user groups. daily.dev helps engineers rolling out Teams-integrated AI agents keep up with Foundry's GA features.