Test Auth skill

Test Agent Authentication

Authenticate with the Power Platform API for evaluation and SDK chat workflows. This caches a token in the "test-agent" slot that is shared by run-eval and chat-sdk.

Step 1: Find the workspace

Locate the agent workspace (directory containing .mcs/conn.json):

Glob: **/agent.mcs.yml

The conn.json provides the tenant ID automatically — no need to ask the user for it.

Step 2: Get the client ID

CRITICAL: You MUST present the FULL configuration checklist below when asking for the client ID. Do NOT omit any items — users frequently miss the redirect URI and public client settings, which causes auth failures.

Ask the user for their App Registration Client ID and show them all of the following requirements in your message:

What is your App Registration Client ID for the Evaluation API?

Before sharing it, please verify your App Registration has all of these settings:

1. Redirect URI: Under Authentication > Mobile and desktop applications > redirect URI set to http://localhost
2. Public client: Under Authentication > Advanced settings > Allow public client flows set to Yes
3. API permissions: Under API permissions, these delegated permissions on the Power Platform API:
   • CopilotStudio.MakerOperations.Read (list test sets, get results)
   • CopilotStudio.MakerOperations.ReadWrite (start evaluation runs)
   • CopilotStudio.Copilots.Invoke (SDK chat — send utterances to the agent)
4. Admin consent: Granted for the tenant (green checkmarks in the API permissions list)

Missing any of these will cause authentication to fail.

If the user doesn't have one, guide them:

How to create an App Registration:

1. Go to Azure Portal > App Registrations
2. Click New registration, name it (e.g., "Copilot Studio Eval")
3. Under API permissions > Add a permission > APIs my organization uses > search Power Platform API
4. Select Delegated permissions and add:
   • CopilotStudio.MakerOperations.Read
   • CopilotStudio.MakerOperations.ReadWrite
   • CopilotStudio.Copilots.Invoke
5. Click Grant admin consent for the tenant
6. Under Authentication > Add a platform > Mobile and desktop applications > set redirect URI to http://localhost
7. Still in Authentication, scroll to Advanced settings and set Allow public client flows to Yes (required — without this the interactive browser login will fail with an AADSTS error)
8. Copy the Application (client) ID from the Overview page

Step 3: Authenticate

Run the auth command (opens a browser — no device code needed):

node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js auth --workspace <path> --client-id <id>

If successful, you'll see:

{"status": "ok", "message": "Authentication successful. Token cached for subsequent commands."}

Return the client ID and workspace path to the caller — they need these for subsequent run-eval and chat-sdk calls.

Troubleshooting

• HTTP 403 / InsufficientDelegatedPermissions: The app registration is missing required permissions. Guide the user through adding them (Step 2).
• AADSTS700016 / app not found in directory: The client ID doesn't exist in the agent's tenant. The app must be registered in the same tenant as the Copilot Studio environment (check conn.json > AccountInfo.TenantId).
• AADSTS7000218 / request body must contain client_secret or client_assertion: The app registration is not configured as a public client. Go to Authentication > Advanced settings > set Allow public client flows to Yes.
• SDK chat hangs after auth: The cached token may have stale permissions. Clear the cache and re-authenticate:
  • Delete ~/.copilot-studio-cli/test-agent.cache.json
  • Run security delete-generic-password -s "copilot-studio-cli" -a "test-agent" to clear the macOS Keychain entry
  • Re-run the auth command