Microsoft tests Copilot on synthetic companies instead of customer data

Microsoft Research has built a platform that creates entire synthetic companies for evaluating Copilot. They contain hundreds of employees, organizational structures, emails, meetings, documents and histories that evolve over time. Three environments model pharmaceuticals, capital markets and a software company without ingesting customer data.

Three artificial companies replace sensitive customer environments

The platform grounds its structure in public sources including the O*NET occupation database, online job postings and filings with the US Securities and Exchange Commission. These sources shape roles, hierarchies, activities and industry context. Messages, documents and meeting records are generated synthetically.

The system then creates Copilot questions and checks their grounding, naturalness and uniqueness. This supports time-aware queries such as what a team decided last Thursday or what changed after the latest quarterly review. Microsoft says no real customer data enters any stage of the pipeline.

Evals move from a handful of tests to repeatable company infrastructure

A generic benchmark cannot show whether Copilot correctly follows a project, relationships between teams and decisions over time. A synthetic company gives product teams a stable scenario for regression testing and lets them repeat the same evaluation after changing a model or retrieval layer.

Microsoft estimates savings of about $100,000 per evaluation dataset and $300,000 across the project. Automated runs replaced a manual process that took several days. For enterprise teams, the more important benefit is the ability to share test data internally without the usual restrictions around personal and customer information.

A 0.9 score measures simulation fidelity, not user satisfaction

Microsoft reports a 39% improvement in its composite quality metric compared with its initial generation approach. Some industry environments scored above 0.9 out of 1.0. This is an internal measure of structural, contextual and statistical fidelity, not an independent measure of Copilot accuracy for customers.

Synthetic data can also inherit its creators' assumptions about how an organization works. Job postings and SEC filings describe roles and scale reasonably well, but they are weaker at capturing informal decisions, incomplete records, local language and politics inside a particular team.

Correlation with production failures will establish the platform's value

Microsoft plans to add industries and model workflows in greater depth. The decisive question is whether failures found inside synthetic companies predict problems later visible through safely aggregated telemetry, user feedback and production incidents.

A useful eval catches a regression before the customer does. If the platform merely produces tidy environments that measure what Microsoft already encoded, it can deliver precise scores while missing real work.

Lilith's verdict

A synthetic company is an excellent crash-test dummy for Copilot. It should not be mistaken for a live office, where half the decisions happen in passing and the rest may never be written down.