How to evaluate an EDR: 10 questions to ask every vendor

Choosing an endpoint detection and response (EDR) platform is one of the more consequential procurement decisions a SOC team makes, and it’s made harder by the fact that every vendor claims to do everything. “AI-powered”, “real-time”, “comprehensive coverage” - the words blur together fast.

The antidote is specificity. When you move from capability claims to concrete operational questions, vendors differentiate themselves quickly. Below are ten questions worth asking in every vendor evaluation. Some have objectively better answers; others reveal trade-offs you’ll need to weigh against your environment.

Questions about detection and coverage

1. What is your MITRE ATT&CK coverage, and can you show me the matrix - not just a percentage?

A vendor quoting “90% ATT&CK coverage” is largely meaningless. Ask them to show you the MITRE Navigator layer. Which techniques are detected behaviourally? Which rely on signature matching? Which sub-techniques are missing entirely? A vendor willing to show gaps is more trustworthy than one who deflects with a headline number.

2. Does your detection engine run on-sensor or cloud-only?

Cloud-only detection means your agents send telemetry to the vendor, events are evaluated remotely, and alerts come back. On-sensor detection means the agent can flag threats locally, without a round trip. The difference matters for air-gapped environments, high-latency networks, and containment speed. Ask what happens when the agent loses connectivity - does detection pause?

3. What’s the agent’s disk and CPU footprint?

Any vendor will say “lightweight”. Ask for actual numbers: binary size, memory RSS at idle, peak CPU during a scan, and how the footprint changes on older hardware. A sub-40 MB static binary that uses inotify and epoll is meaningfully different from a 300 MB installer that kicks off a kernel module and a background scan service.

4. How does your alert explanation work - and does AI analysis ever block ingestion?

If the vendor has AI-assisted triage, ask whether the LLM call is synchronous or asynchronous. A synchronous call means every alert waits for the AI to respond before being delivered - that’s a reliability risk. The correct design is fire-and-forget: ingest the alert, queue the LLM analysis separately, deliver the explanation when it’s ready. Ask also which LLM providers are supported; vendor lock-in to a single provider is a risk if your organisation has data residency requirements.

5. What telemetry is retained, and for how long?

Incident investigations routinely require looking back 30, 60, or 90 days. Ask what raw telemetry (file events, process events, network connections) is stored, for how long, and whether retention is included in the base price or a paid add-on. Some vendors charge separately for longer retention windows.

Questions about operations and trust

6. How are agents authenticated, and can a compromised machine be instantly revoked?

Username/password-authenticated agents are a significant attack surface. Certificate-based mutual TLS (mTLS) is meaningfully stronger: each agent holds a private key that never transits the network, and revocation can happen instantly. Ask specifically: what’s the revocation latency? If it’s “next check-in” (e.g., 60 seconds), that’s a window an attacker can exploit.

7. What does containment actually do - and what doesn’t it isolate?

“One-click containment” sounds decisive. Ask exactly what network traffic is blocked. Does DNS still resolve? Can the agent still communicate with your control plane? What about management tools like SSH or WinRM? Some platforms leave these open (deliberately), others block everything. Neither is wrong, but you need to know which one you’re buying.

8. How is multi-tenancy implemented for MSPPs and multi-team environments?

If you manage multiple clients or business units, isolation matters. Ask whether clients share a database schema, or whether there’s true tenant isolation. Ask who can see what - can a junior analyst accidentally query another client’s telemetry? Ask whether role-based access control (RBAC) is granular enough for your use case.

9. What does the pricing model look like at scale - and what are the hidden costs?

Get a written price per agent per month for your expected volume. Then ask about ingestion fees, retention add-ons, AI feature tiers, API access charges, and professional services requirements for onboarding. The sticker price on enterprise EDR rarely reflects what you’ll actually pay. If a vendor won’t give you a number in writing, that’s information.

Common hidden cost	What to ask
Telemetry retention beyond 30 days	Included or metered?
AI/LLM triage features	Base tier or add-on?
API / SIEM integration	Included or per-connector?
Onboarding / professional services	Mandatory or optional?
Overage on agent count	Soft cap or hard billing event?

10. What’s the support model, and can I talk to an engineer?

Ticket queues staffed by first-line support are fine for password resets. For a live incident at 2am, you want to reach someone who knows the product code. Ask how escalation works, what the SLA is for critical incidents, and whether there’s a direct channel to engineering during an active breach. Smaller vendors often have faster escalation paths precisely because there are fewer layers between you and the person who built the thing.

Using these questions in practice

Send these questions in advance rather than springing them in a live demo. A vendor who can’t answer them in writing before the demo is unlikely to answer them satisfactorily during it.

The best evaluation is a proof of concept (PoC) in your own environment - even a short one. Watch how the agent behaves on your OS mix, test containment on a non-critical host, fire a known-safe test payload (EICAR or an ATT&CK simulation tool) and see what triggers an alert.

If you’d like to run a PoC against LightEDR or want to discuss how we answer these questions, get in touch.