# 17. Governing and Measuring Your Security Service Providers

Topic 11 covered how to buy vCISO and managed security services. This topic covers what happens after you buy, because that's where most programs quietly fail.

Context: The security services market has a dirty secret: the relationship that gets sold is almost never the relationship that gets delivered. The senior consultant who led the pitch is managing ten other accounts. The SOC team your MSSP showed you is the A-team from the facility tour; your account gets the B-team on overnight shifts. The quarterly business reviews are polished decks that say very little about whether you're actually more secure than last year.

This isn't unique to security services, it's the fundamental challenge of any professional services relationship. But in security, the stakes of a poorly performing provider are higher than most: you may not know it's failing until you're in the middle of an incident.

The discipline of governing security service relationships is genuinely underinvested. Most organizations spend significant effort selecting providers and almost no structured effort measuring whether those providers are delivering. A few basic questions asked quarterly would catch most relationship failures before they become security failures.

This applies equally to:

• MSSPs and MDR providers managing your detection and response
• vCISOs leading or advising your security program
• Managed SIEM, vulnerability management, or compliance providers
• Pen test and red team firms running your offensive testing program
• Any security service provider receiving money in exchange for security outcomes

The core question for governing these relationships: "Are we getting what we paid for, and how would we know if we weren't?"

13 Questions to Ask Your Security Service Providers (Regularly, Not Just at Renewal)

1. "Show me the last 90 days of your work, specifically what was detected, investigated, escalated, and resolved. Not a dashboard summary. The actual activity."
Why: Managed security services often provide dashboards that show impressive numbers, alerts processed, threats blocked, incidents handled. But dashboards can be gamed and are often calibrated for optics rather than insight. Requesting the underlying activity log for a specific period forces accountability for specific decisions, not aggregate statistics.
Good answer: Detailed incident log with timestamps, analyst notes, escalation decisions, resolution actions, and any misses acknowledged. Red flag: Dashboard screenshots and "we processed X alerts this quarter" without underlying detail.

2. "Tell me about something you missed, a detection that should have fired but didn't, an escalation that was delayed, an incident you wish you'd handled differently."
Why: Every security service provider misses things. The question is whether they know about it, acknowledge it, and learn from it. Providers who claim a clean record either aren't measuring carefully or aren't being honest. The willingness to discuss misses and near-misses is the most reliable indicator of operational maturity.
Good answer: A specific example with root cause, remediation, and what changed afterward. Red flag: "We haven't had any misses" or deflection.

3. "Who specifically is working on my account, names, certifications, tenure, and shifts? Have any of those people changed since we started working together?"
Why: The person in the pitch deck and the person watching your alerts at 3am are often different people. Staff turnover in managed security is significant, and institutional knowledge of your environment leaves when people leave. Quarterly verification of who's actually on your account catches quiet degradation in service quality.
Good answer: Named analysts and engineers with specific account responsibility, disclosure of any changes since last review, confirmation of training and certification status. Red flag: "Our team is always available" without naming the people on your account.

4. "Are you meeting your SLAs, every one of them, with the raw data?"
Why: SLA reporting is often selective. Providers report the metrics they're meeting and find reasons to exclude the ones they're missing. Requesting the raw data behind SLA calculations, not just the summary, reveals whether the numbers are real.
Good answer: Full SLA performance data, measurement methodology explained, exceptions disclosed and root-caused. Red flag: Summary SLA compliance percentages without underlying data, or SLAs that conveniently changed just before they would have been missed.

5. "What has changed in the threat landscape relevant to my environment in the last 90 days, and what specifically did you do about it?"
Why: A security service provider should be proactively adapting to the evolving threat environment, not just running the same playbooks indefinitely. New ransomware variants, vulnerabilities affecting your stack, sector-specific campaigns, your provider should know about these and should have a specific answer for how they adapted your coverage.
Good answer: Specific threat intelligence relevant to your sector, documented changes to detection rules, playbooks updated, or new monitoring added in response. Red flag: Generic "the threat landscape is evolving" without specifics to your environment.

6. "How many incidents did you escalate to my team this quarter, and how many turned out to be false positives? What's the trend over time?"
Why: Escalation quality is one of the most important metrics for a managed security service. Too few escalations may mean misses; too many may mean your team is being flooded with noise and tuning out. The false positive rate among escalations, and the trend line, tells you whether the service is getting smarter about your environment or staying static.
Good answer: Escalation count, FP rate, trend over the engagement lifecycle, root cause for persistent FP sources. Red flag: High escalation volume with no FP tracking, or escalation volume so low that it raises questions about missed detections.

7. "How is the service maturing, what's better now than when we started, and what's the roadmap for the next 12 months?"
Why: A security service that delivers the same thing in year three as it did in year one isn't improving as your environment evolves. Coverage should expand, detection quality should improve, and the provider's knowledge of your environment should deepen over time. If you can't see that trajectory, you may be paying for stasis.
Good answer: Measurable improvement metrics over the engagement lifetime, roadmap for capability expansion, evidence of environment-specific tuning. Red flag: The same slides every QBR with updated dates.

8. "What access do you have to my environment right now, credentials, agents, API keys, network connections? Show me the inventory."
Why: Over time, security service providers accumulate access that outlives its purpose. A quarterly access review should confirm that the provider's current access is limited to what's required for current services, nothing more. Stale credentials and forgotten remote access paths are a supply chain risk (see Topic 16).
Good answer: A current, complete access inventory, reviewed jointly, with any excess access identified for removal. Red flag: "We only use what's necessary" without an inventory to verify against.

9. "When was the last time you tested what you would actually do in an incident involving my environment, with my team, not just your team?"
Why: Incident response is a team sport. The MSSP knows their playbooks; your internal team knows your environment. The interface between them, escalation paths, communication protocols, decision authorities, only gets tested in a real incident unless you proactively exercise it. Tabletop exercises that include your MSSP should be annual at minimum.
Good answer: Recent joint tabletop exercise, documented escalation contacts on both sides, communication protocol tested, post-exercise findings addressed. Red flag: "We're ready for anything" with no joint exercise history.

10. "What would you tell me to do differently, things my team should be doing that we're not, gaps in my environment that make your job harder?"
Why: A mature security service provider should be a candid advisor, not just a service executor. If they've been watching your environment for 12 months, they have opinions about what's working and what isn't. The willingness to offer constructive criticism reveals whether they're invested in your actual security or just in their contract.
Good answer: Specific, actionable recommendations with supporting evidence from their monitoring experience. Red flag: "You're doing everything right" (almost never true) or generic recommendations not grounded in observations of your environment.

11. "What are your competitors doing better than you right now, and how do you plan to close that gap?"
Why: This question is deliberately uncomfortable. Mature providers have self-awareness about market position and capability gaps. Asking it signals that you're engaged in the market and evaluating alternatives, which is healthy for the relationship and keeps providers from getting complacent.
Good answer: Honest competitive landscape commentary, acknowledged gaps with roadmap to address, genuine confidence about their strengths. Red flag: "We're the best at everything" or complete deflection.

12. "Walk me through how you've changed your service model in response to AI-augmented attacks, what's different about how you detect and respond compared to 18 months ago?"
Why: The threat landscape is changing faster than it ever has. AI-generated phishing, automated vulnerability discovery, faster attack chains, these require detection and response capabilities to evolve. A provider whose answer is unchanged from 18 months ago either isn't evolving or isn't being honest about the changes.
Good answer: Specific capability changes, new detection models, updated playbooks for AI-generated content, faster response automation, new threat intel sources. Red flag: "Our platform handles all threat types" with no specifics.

13. "If this relationship ended tomorrow, for any reason, what would it take to transition to another provider? What data, integrations, and knowledge would we need to transfer?"
Why: This question should be asked annually, not just at contract renewal time. Understanding the exit cost and complexity before you're motivated to exit is how you avoid being trapped. Providers who make exit difficult know they're not delivering enough to retain you on merit.
Good answer: Clear transition documentation, portable data and configurations, honest time estimate for transition, cooperative attitude toward the question. Red flag: Defensiveness, vague answers, or "you won't want to leave."

The Meta-Test

• Q1, Q2, and Q4 are the evidence test. If a provider can't show you specific, detailed evidence of recent work and honest performance data, including misses, they're managing a relationship, not a service.
• Q3 and Q8 are the access and trust test. Knowing who's working on your account and what access they have should be non-negotiable quarterly disciplines, not occasional curiosities.
• Q10 and Q13 are the relationship health test. A provider who's candid about what you should be doing differently and transparent about exit conditions is invested in your security. One who isn't is invested in your contract.

Honest Observations

1. The QBR is the most underused tool for holding providers accountable, and the most abused tool for avoiding accountability.

A quarterly business review run by the provider will almost always be a deck designed to look impressive. Numbers that are up (alerts processed, coverage expanded), numbers that are down (MTTD, false positive rate), and graphs that trend in the right direction. What it rarely contains: the specific incidents that were missed, the detection rules that haven't been updated, the analysts who turned over.

The antidote: own your own QBR agenda. Show up with your questions, not their slides. If they don't know what you're going to ask, the answers will be more honest.

2. Staff turnover is the silent service degrader.

The analysts who know your environment, your specific architecture, your noise patterns, what's normal for your environment, are extraordinarily valuable. When they leave (and in managed security, they leave often), that institutional knowledge goes with them and their replacement starts from near-zero.

Ask explicitly about turnover on your account. Ask whether analysts rotate between accounts or stay assigned. Ask what knowledge management processes exist to preserve institutional knowledge when people leave. The answers are often uncomfortable.

3. The most common MSSP relationship failure is ungoverned scope creep, in both directions.

The service delivers less than what was sold because "that's not in scope", but it's also frequently true that the provider is doing things that weren't in scope and charging for them, or providing services that overlap with what you've now built internally.

An annual scope review, what are we actually delivering vs. what was contracted, what have you built internally that we could stop managing, what gaps exist that we're not covering, is worth its weight in avoided waste and disappointment.

4. Security service providers should be part of your incident response plan, not a surprise participant.

The number of organizations that have an MSSP and a separate IR retainer with no documented interface between them would be alarming if it weren't so common. Who calls whom, what decisions does the MSSP make autonomously, what requires your approval, who talks to legal and communications, these questions should be answered in writing before an incident, not improvised during one.

5. Measure outcomes, not activity.

The metrics that actually matter for security service relationships:

• Mean time to detect (MTTD) for real incidents, trending in the right direction over the engagement?
• Mean time to respond (MTTR), how long from detection to containment?
• Escalation quality, what percentage of escalations were genuine incidents vs. noise?
• Coverage gaps, what in your environment is not covered by monitoring, and is that list shrinking?
• Detected vs. missed, from exercises and post-incident analysis, what did they catch vs. what got through?

The metrics that almost never matter: number of alerts processed, number of signatures updated, number of reports delivered. These measure activity, not outcome. Providers know this and lean on them precisely because they're opaque.