Microsoft and NVIDIA Map Open-Weight Model Risk to Enterprise Deployments

A model evaluation can look sound while the application using that model still has access to sensitive data and tools. That is the gap Microsoft and NVIDIA’s new open-weight model guidance asks enterprise teams to examine. Published October 16, the joint work gives teams planning an open-weight deployment five risk categories and a way to connect selected risks to evaluations. The company running the deployment still has to assess the application around the model.

Moving from an internal drafting assistant to a customer-facing agent changes the inputs the model sees and the tools its application can invoke. As Microsoft puts it, “weight openness alone does not determine an AI system’s risk.” Its post identifies the model, use case, deployment and safeguards as the relevant combination. Open weights let a team inspect, evaluate, modify and run a model itself; the team may also consume an open-weight model through a managed platform.

Model access meets application authority

Microsoft and NVIDIA group the model risks under Cybersecurity, Data Protection, Content, Human-AI Interaction and Bias. That taxonomy is a starting vocabulary for engineering, security, legal and business owners, rather than a ready-made deployment checklist. The paper focuses mainly on the model layer and maps evaluations to a selection of risks in those categories.

The application changes the security question. A standalone inference endpoint returns model output; an agent with connected tools can act through whatever permissions its application grants. If the app can retrieve internal records or call an external service, an evaluation of the model’s responses alone cannot tell the team whether those application permissions are appropriate. Microsoft explicitly calls for assessment of the agent’s permissions, data access, tools, runtime and operational controls alongside the model.

This is also a responsibility question. A team operating its own model deployment may need to maintain safeguards that a managed service otherwise provides. Even on a managed platform, the service and configuration decide which controls the enterprise owns. Use the provider contract and deployed configuration to assign the owner of each control at the model, inference, application and operational layers.

Evaluations need deployment conditions

NIST describes its AI Risk Management Framework as voluntary guidance for managing risks to people, organizations and society across the design, development, use and evaluation of AI systems. Microsoft says the joint taxonomy draws on that framework and other standards. The five categories help a team name a risk; a test must still reproduce the conditions in which its own model and application operate.

Microsoft says Foundry offers built-in and custom evaluations, automated red teaming and configurable content-safety controls. That is a vendor describing its own product. Its advice is to check control availability for the chosen model and deployment, including preview status, then combine evaluations with application-specific testing, monitoring and human oversight. An evaluation of harmful output, for example, addresses a different question from whether an agent is allowed to invoke a sensitive tool.

Record attempted tool calls and changes to application access, then verify that monitoring captures both in an authorized test. Set evaluation acceptance criteria against the use case and the permissions the application actually holds.

Bind each risk to an owner

Start with an inventory of each proposed deployment’s model provenance, intended use, users and hosting arrangement. For applications using that model, record the data stores and connected tools, the permissions granted to the application, and who can change them. Those are the conditions under which a model-level risk may reach a customer or an internal record. Assign one owner for each evaluation and each safeguard at the model, inference, application or operational layer.

Then change the system where the risk sits. Restrict a tool’s data access to the application’s actual job, apply input or output filtering where the use case warrants it, and require human review for actions that deserve an approval step. Microsoft names those controls as options; the team has to choose them against its own deployment. When the model, tools or permissions change, rerun the relevant evaluation and review the control owner.

A practical verification is to test the deployed application with a permitted test identity and synthetic records. Check that its approved task still works, a task outside that identity’s tool permission is denied, and the monitor records the attempted tool use. Compare those results with the team’s model evaluation for the same use case. The denied call and its recorded event verify the application’s tool-permission boundary.