‹ All interview questions

Performance

Your ServiceNow production instance handles 500,000 daily transactions across Incident, Change, and CMDB…

The question

Your ServiceNow production instance handles 500,000 daily transactions across Incident, Change, and CMDB tables. Over the past week, users report 2-5 second delays in form loads and submissions, spiking during peak hours (8-10 AM). Transaction logs show average response times jumped from 200ms to 3.5s. Business impact: 15% SLA breaches on incident resolution. You notice an outbound REST integration to VendorX's API (via Integration Hub Flow Designer) running every 5 minutes, syncing 10,000 CMDB CIs. It's configured with scripted REST steps for data transformation. No errors in integration logs, but VendorX reports receiving requests. Walk us through your diagnosis approach (tools/metrics), root cause hypotheses, 3+ solution options with trade-offs, and production rollout plan. Include code showing the likely problematic pattern and your fix. What separates a good answer (basic fix) from great (architectural resilience)?

Why Interviewers Ask This

In production scenarios like this, delays often stem from synchronous blocking calls, inefficient queries, or unhandled async buildup in integrations. Diagnosis starts with Performance Analytics dashboards for transaction trends, then Transaction Logs (filter by slow URLs), Node Logs for integration errors, and Integration Hub Activity logs. Check Slow Queries report and Stats.do for GlideRecord bottlenecks. Hypothesize: Scripted REST step with heavy server-side queries (e.g., N+1 problem) blocking the flow, or missing timeouts causing queue buildup. VendorX latency compounds during peaks.

The Solution

Problem Code (Likely Culprit in Flow Designer Script Step): Synchronous heavy query inside loop, no timeouts, blocking instance threads.

var ciGr = new GlideRecord('cmdb_ci');
ciGr.addQuery('vendor', 'VendorX');
ciGr.query();
var payload = [];
while(ciGr.next()) {
  var relGr = new GlideRecord('cmdb_rel_ci'); // N+1 query!
  relGr.addQuery('parent', ciGr.sys_id);
  relGr.query();
  if(relGr.next()) payload.push({name: ciGr.name, rel: relGr.child.name});
}
var request = new sn_ws.RESTMessageV2('vendorx/sync', 'POST');
request.setRequestBody(JSON.stringify(payload));
var response = request.execute(); // Synchronous, blocks 10k records!
if(response.getStatusCode() != 200) gs.error('Sync failed');

Solution Code (Async Transform + Retry with Timeout): Decouple query/transform from API call using Event and Script Include for resilience.

// Script Include: VendorXTransformer (with query optimization)
var VendorXTransformer = Class.create();
VendorXTransformer.prototype = {
  transformCIs: function() {
    var cis = [];
    var ciGr = new GlideRecord('cmdb_ci');
    ciGr.addQuery('vendor', 'VendorX');
    ciGr.addQuery('sys_updated_on', '>', gs.minutesAgo(5)); // Indexed field
    ciGr.setLimit(10000);
    ciGr.query();
    while(ciGr.next()) {
      cis.push({sys_id: ciGr.sys_id, name: ciGr.name});
    }
    gs.eventQueue('vendorx.sync.ready', ciGr, JSON.stringify(cis), ''); // Async fire-and-forget; parm1 must be a string
  },
  type: 'VendorXTransformer'
};

// Flow Designer: Event-triggered subflow with timeout
// Event: vendorx.sync.ready --> Script Step calls Transformer.transformCIs()
// Then: REST Step with timeout=30s, retry=3, exponential backoff

Senior Considerations

  • Diagnosis Depth: Log review is the baseline; the stronger move is correlating node statistics (Stats.do), AIOps anomaly detection, and vendor-side traces via a correlation ID.
  • Solution Options & Trade-offs: - Option1: Add timeouts/retries (quick, but masks symptoms). - Option2: Async via Events/Message Queue (scalable, adds complexity). - Option3: MID Server offload (secure for on-prem, network latency risk). - Option4: Integration Hub Bulk API (native, but license cost).
  • Production Rollout: Canary in sub-prod, A/B test with perf baseline, post-deploy monitoring (KPI: <500ms p95), rollback via flow disable. Capacity plan for 2x volume.
  • Edge Cases: Handles VendorX downtime (fallback to queue), token expiry (centralized vault), data skew (idempotent upserts).
  • Why Great Answers Shine: Discusses ITIL alignment (Problem record), CMDB impact, future-proofing (scoped app), and ROI (quantify what SLA recovery is worth to the business).

The core point: synchronous bulk calls exhaust worker threads at scale, while event-driven async decouples the transform from the API call.