high_variance_latencyQuery latency is inconsistent across executions
High variance in execution time suggests unpredictable performance — often caused by cache misses, resource contention, or cold starts.
Common Causes
- —Inconsistent cache hit rates
- —Resource contention at peak load
- —GC pressure affecting query timing
- —Index selectivity issues
How to Fix
- 1.Check for cache misses and warm caches
- 2.Investigate index selectivity
- 3.Analyze data distribution
- 4.Look for resource contention
Example
typescript
// Monitor latency distribution
const times = [];
const monitor = createMonitor({
exporter: (event) => {
if (event.model === 'User' && event.operation === 'findOne') {
times.push(event.durationMs);
const variance = calculateVariance(times);
if (variance > 1000) sendAlert('high_variance', { variance, times });
}
}
});P99 matters more than average
High variance means your average latency looks fine but some users experience 10× slower responses. Always track p95 and p99, not just mean.
Common causes by pattern
typescript
// Variance from cold cache — warm on startup
async function warmCache() {
const hotUsers = await User.find({ active: true })
.sort({ lastLogin: -1 })
.limit(1000)
.lean();
await Promise.all(
hotUsers.map(u => redis.setex(`user:${u._id}`, 3600, JSON.stringify(u)))
);
}
// Variance from lock contention — use optimistic concurrency
const session = await Session.findById(id);
const result = await Session.findOneAndUpdate(
{ _id: id, version: session.version }, // guard against concurrent update
{ $set: { data }, $inc: { version: 1 } },
{ new: true }
);
if (!result) throw new Error('Concurrent update detected — retry');