Hosting Automation Alert Triage: Blast Radius and Customer Impact

You wake up at 3am to an alert: a provisioning job failed. Is it urgent? It depends on how many customers are affected and how badly. Without a consistent way to judge, you either ignore critical issues or burn out on noise. This framework helps you classify failures by blast radius and customer impact, so you can set alert thresholds that match real-world severity.

What is blast radius and why does it matter for hosting automation?

Blast radius is the scope of a failure: how many customers, services, or systems it touches. A single failed provisioning job for one client has a small blast radius; a database outage that stops all automation has a huge one. Customer impact measures how severely those affected customers are hurt. A delayed invoice email is low impact; a server down for hours is high impact. Combining these two dimensions lets you prioritize alerts rationally.

How do you measure blast radius in hosting automation?

Blast radius depends on the component that fails and its dependencies. Consider these factors:

• Number of affected customers: One client, a reseller's entire customer base, or all clients?
• Number of services: A single VPS, all VPSs on a node, or all shared hosting accounts?
• Criticality of the service: Is it a production website, a test environment, or a backup?
• Redundancy: Is there a failover? If yes, blast radius shrinks.

For example, a failed domain registration API call affects one customer (small radius). A billing system outage prevents all new orders (large radius).

How do you assess customer impact?

Customer impact is about the severity of the effect on the customer's business. Ask:

• Is the customer's service down? Downtime directly harms revenue and trust.
• Is data at risk? Data loss or corruption is severe.
• Is the customer blocked from performing a task? E.g., cannot log in, cannot pay, cannot deploy.
• Is there a workaround? If yes, impact is lower.

High impact means the customer cannot operate normally and has no easy fix.

What are the four severity classes for hosting automation alerts?

Combine blast radius and impact into a simple matrix:

• Critical (page immediately): Large blast radius AND high impact. Example: provisioning system down for all new orders, or a database corruption affecting many customers.
• High (page during business hours or escalate): Large blast radius but low impact, OR small blast radius but high impact. Example: a single customer's server down (high impact, small radius) or a mass email delivery delay (large radius, low impact).
• Medium (ticket for next business day): Small blast radius AND low impact. Example: one customer's invoice failed to generate, but they can still use services.
• Low (dashboard only): Minimal or no customer impact, often internal. Example: a failed backup for a test environment, or a non-critical log warning.

Use this matrix to set alert routing: critical goes to PagerDuty or SMS, high goes to email or Slack, medium creates a ticket, low logs to a dashboard.

How to implement this framework in your monitoring stack?

Start by tagging every alert with metadata: affected customer count, service criticality, and estimated impact. Then write rules:

• If blast radius > 10 customers and impact = high, page on-call.
• If blast radius = 1 and impact = high, page on-call but with lower urgency.
• If blast radius > 10 and impact = low, send to Slack during business hours.
• If blast radius = 1 and impact = low, create a ticket.

Tools like Prometheus Alertmanager, Grafana, or your hosting automation platform's webhook system can route alerts based on these rules. For example, Teculiar provides an API and webhooks that let you integrate alert classification into your existing workflows.

What are common pitfalls and how to avoid them?

Even with a framework, mistakes happen. Watch for:

• Alert fatigue: Too many low-severity alerts desensitize you. Tune thresholds regularly.
• Misclassified blast radius: A single customer might represent a large reseller. Always check if the customer is a reseller.
• Ignoring cascading failures: One failure can trigger others. Monitor dependencies.
• Static rules: Your business changes; revisit classifications quarterly.

Also, ensure your alerting system has a feedback loop: after each incident, review whether the severity was appropriate and adjust.

What to do next

• Audit your current alerts and tag each with blast radius and impact.
• Define routing rules for each severity class and test them.
• Integrate with your hosting automation platform's API or webhooks for automated classification.
• Review and adjust quarterly.

Start by classifying your next alert using this framework, and refine from there.