‹ All interview questions

Performance

Your production ServiceNow instance is experiencing slow transaction times during peak hours.

The question

Your production ServiceNow instance is experiencing slow transaction times during peak hours. Users report that certain form submissions take 8-12 seconds to complete, but your transaction logs show server processing time is only 1.2 seconds. The delays appear random across different modules. Walk me through your diagnostic approach: What are the most likely culprits, how would you isolate the root cause, and what are the trade-offs between different solutions?

The Real Question

This scenario tests whether a candidate can distinguish between server-side processing delays and client-side/network issues. The key insight is that transaction logs show fast server response (1.2s) but users experience 8-12s delays, indicating the bottleneck exists elsewhere. A junior engineer might blame the database or business rules; a senior architect recognizes this points toward external integrations, session synchronization, ACL processing, or network latency.

Diagnostic Approach

Step 1: Analyze Transaction Log Metrics - Review the transaction logs focusing on client_network_time, total_wait_time, and session_wait_time, not just transaction_processing_time. If client_network_time is 6-10 seconds while server processing is 1.2s, you've isolated the problem to network or client-side execution.

Step 2: Identify the Culprit Categories - The delay could stem from: (1) REST API integrations without circuit breaker patterns causing retries, (2) Session synchronization delays when users trigger multiple transactions, (3) ACL evaluation on large permission sets, (4) Client-side script execution or UI actions, or (5) Network latency between user and instance.

Step 3: Reproduce with Session Debug Enabled - Enable Session Debug on the affected user and reproduce the transaction. Look for EXCESSIVE markers in the debug output to pinpoint which component consumes time.

The Solution

// PROBLEM: Uncontrolled retry loop hammering an external API
var rm = new sn_ws.RESTMessageV2('VendorAPI', 'GET');
var response = rm.execute();
var retryCount = 0;
while (response.getStatusCode() !== 200 && retryCount < 10) {
  retryCount++; // no backoff, no cap on total wall-clock time
  response = rm.execute(); // blocks the user's transaction each time
}
// SOLUTION: Single attempt with timeout + circuit-breaker fallback
var rm = new sn_ws.RESTMessageV2('VendorAPI', 'GET');
rm.setHttpTimeout(3000); // ms — fail fast instead of hanging the form
try {
  var response = rm.execute();
  if (response.getStatusCode() === 200) {
    return response.getBody();
  }
  // Trip the breaker: record the failure and serve cached data
  gs.warn('VendorAPI unhealthy (status ' + response.getStatusCode() + ')');
  return useCachedData(); // Fallback strategy
} catch (e) {
  gs.warn('VendorAPI timeout - using cache');
  return useCachedData();
}

Senior Considerations

  • Root Cause vs. Symptom - Distinguish between what appears slow (user experience) and what actually causes slowness (server metrics). This question separates architects from developers.
  • Circuit Breaker Pattern - External integrations without timeout/fallback mechanisms can cascade failures. A production-grade solution requires circuit breaker implementation, not just longer timeouts.
  • Session Synchronization - If multiple transactions from the same user overlap, session_wait_time will spike. Understanding when to use async processing vs. synchronous calls is critical.
  • ACL Performance at Scale - With thousands of users, ACL evaluation can consume significant time. Solutions include ACL caching strategies, role-based optimization, or offloading to external identity services.
  • Measurement Discipline - A great answer includes how to measure improvement: baseline current metrics, implement solution, validate 95th percentile response time improves, then monitor for regression.
  • Trade-offs Awareness - Caching improves speed but risks stale data. Async processing improves responsiveness but complicates error handling. Circuit breakers improve resilience but require fallback strategies. Senior architects articulate these explicitly.
  • Production Impact Planning - Solutions should be testable in non-prod, deployable without instance downtime, and include rollback procedures. Mentioning change management and monitoring thresholds demonstrates production maturity.