Module 01 · Lesson 14 of 14Free

Lab: load testing a single server

Level

Intermediate

Topic

System design fundamentals

Reading time

20 min

Kartly needs to handle 1,000 requests a second when its festival sale starts. One developer opens a product page, sees a response in about 95 ms, and declares the server ready.

But one fast request does not establish capacity. How can you measure when additional customers stop getting more completed work and start getting longer waits?

Isolate one limit before testing

You will build a local fixture: a small server with deliberately controlled behavior. It has one application slot. Every product request occupies that slot for a configured delay.

Kartly’s real product request includes CPU work, a database call and a pricing call. Here, a timer replaces the entire 95 ms request. There is no database or pricing service.

This is not a benchmark of Kartly’s eight-core server. It isolates one question from Server concurrency: what happens when requests share a limited number of occupied slots?

The load generator will act as several clients. Each client sends another request only after its previous request finishes. This is a closed load model; slower responses also slow the generator’s next arrivals.^{3Open and closed modelsGrafana Labs, Grafana k6 documentation, 2026}

That makes this experiment bounded and easy to interpret. It does not reproduce an independently arriving festival crowd.

Set up a bounded fixture

Use a desktop terminal with either Python 3.12.15 or Node.js 22.18.0 installed. These are pinned lab versions, not recommendations to deploy these versions.

Choose one language throughout. Run the server and generator in separate terminals. Both versions use only their runtime’s standard library.

python3 --version
node --version
mkdir kartly-load-lab
cd kartly-load-lab

Only your chosen runtime must be available. On Windows, use py -3.12 wherever the commands show python3.

The TypeScript files use .mts, which Node treats as ES modules. Node 22.18.0 runs their erasable type annotations directly, without a compiler step. It does not type-check them.^{2Modules: TypeScriptNode.js contributors, Node.js v22.18.0 documentation, 2025}

Save the matching server as server.py or server.mts.

import sys
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Semaphore

delay_ms: float = float(sys.argv[1]) if len(sys.argv) > 1 else 95.0
if not 0 <= delay_ms <= 1000:
    raise SystemExit("delay must be between 0 and 1000 ms")
slot = Semaphore(1)

class Handler(BaseHTTPRequestHandler):
    def do_GET(self) -> None:
        if self.path != "/product":
            self.send_error(404)
            return
        with slot:
            time.sleep(delay_ms / 1000)
            body = b'{"product_id":"p_42"}'
            try:
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.send_header("Content-Length", str(len(body)))
                self.send_header("Connection", "close")
                self.end_headers()
                self.wfile.write(body)
            except OSError:
                pass  # A disconnected client must not stop the fixture.

    def log_message(self, format: str, *args: object) -> None:
        pass

class Server(ThreadingHTTPServer):
    request_queue_size = 128

print(f"127.0.0.1:8000; one slot; delay={delay_ms} ms", flush=True)
Server(("127.0.0.1", 8000), Handler).serve_forever()

import { createServer } from "node:http";
import { setTimeout as sleep } from "node:timers/promises";

const delayMs: number = Number(process.argv[2] ?? "95");
if (!Number.isFinite(delayMs) || delayMs < 0 || delayMs > 1000) {
  throw new Error("delay must be between 0 and 1000 ms");
}
let tail: Promise<void> = Promise.resolve();

const server = createServer((req, res) => {
  if (req.url !== "/product" || req.method !== "GET") {
    res.writeHead(404);
    res.end();
    return;
  }
  tail = tail.then(async (): Promise<void> => {
    await sleep(delayMs);
    if (res.destroyed) return;
    const body: string = '{"product_id":"p_42"}';
    res.writeHead(200, {
      "Content-Type": "application/json",
      "Content-Length": Buffer.byteLength(body),
      "Connection": "close",
    });
    res.end(body);
  });
  res.on("error", () => res.destroy());
});

server.listen(8000, "127.0.0.1", 128, () => {
  console.log(`127.0.0.1:8000; one slot; delay=${delayMs} ms`);
});

The Python server can receive requests on separate threads, but its semaphore permits only one handler into the delayed section. ThreadingHTTPServer supplies the threaded request handling. It is a teaching server, not a production server.^{1http.server — HTTP serversPython Software Foundation, Python 3.12 documentation, 2026}

The TypeScript server chains requests onto one promise. Each delayed section starts after the previous section ends. That chain is its application queue.

Both timers leave the CPU mostly idle. Low CPU use therefore does not imply spare application slots.

Try it yourself

Start one server:

python3 server.py 95
# Or:
node server.mts 95

In the second terminal, make a single request:

curl --noproxy "*" --max-time 5 -sS -o /dev/null -w "status=%{http_code} total=%{time_total}s\n" http://127.0.0.1:8000/product

On Windows, replace /dev/null with NUL; use curl.exe if your shell aliases curl.

Illustrative output:

status=200 total=0.097s

This measures the request from the client, including connection setup and reading the response. It is not an isolated timer for application work.

The request succeeds alone. Next, make requests overlap.

Count completed work and waiting

Save the matching generator as load.py or load.mts.

Each run attempts exactly 160 requests, without retries. The argument selects concurrent clients and must be between one and sixteen. The target is fixed to localhost.

import math
import sys
from concurrent.futures import ThreadPoolExecutor
from time import perf_counter
from urllib.request import ProxyHandler, Request, build_opener

clients: int = int(sys.argv[1]) if len(sys.argv) > 1 else 1
if not 1 <= clients <= 16:
    raise SystemExit("clients must be between 1 and 16")

def request(_: int) -> tuple[bool, float]:
    start = perf_counter()
    try:
        opener = build_opener(ProxyHandler({}))
        req = Request("http://127.0.0.1:8000/product",
                      headers={"Connection": "close"})
        with opener.open(req, timeout=5) as response:
            body = response.read()
            ok = response.status == 200 and body == b'{"product_id":"p_42"}'
    except Exception:
        ok = False
    return ok, (perf_counter() - start) * 1000

start = perf_counter()
with ThreadPoolExecutor(max_workers=clients) as pool:
    results = list(pool.map(request, range(160)))
elapsed = perf_counter() - start
latencies = sorted(ms for ok, ms in results if ok)
successes = len(latencies)
p95 = f"{latencies[math.ceil(0.95 * successes) - 1]:.0f}" if successes else "n/a"
print(f"success={successes} errors={160 - successes} "
      f"rps={successes / elapsed:.1f} p95_ms={p95}")

import { performance } from "node:perf_hooks";

const clients: number = Number(process.argv[2] ?? "1");
if (!Number.isInteger(clients) || clients < 1 || clients > 16) {
  throw new Error("clients must be between 1 and 16");
}
let next: number = 0;
let errors: number = 0;
const latencies: number[] = [];

async function worker(): Promise<void> {
  while (next++ < 160) {
    const start: number = performance.now();
    try {
      const response = await fetch("http://127.0.0.1:8000/product", {
        headers: { Connection: "close" },
        signal: AbortSignal.timeout(5000),
      });
      const body: string = await response.text();
      if (response.status !== 200 || body !== '{"product_id":"p_42"}') {
        throw new Error("unexpected response");
      }
      latencies.push(performance.now() - start);
    } catch {
      errors++;
    }
  }
}

const start: number = performance.now();
await Promise.all(Array.from({ length: clients }, () => worker()));
const elapsed: number = (performance.now() - start) / 1000;
latencies.sort((a, b) => a - b);
const successes: number = latencies.length;
const p95: string = successes
  ? latencies[Math.ceil(0.95 * successes) - 1]!.toFixed(0) : "n/a";
console.log(`success=${successes} errors=${errors} ` +
  `rps=${(successes / elapsed).toFixed(1)} p95_ms=${p95}`);

The generator reports three different observations:

Latency starts when a client begins its request. It includes server-side waiting, but not time awaiting a free generator client.

Errors do not enter the latency percentile. Always read the error count beside p95: fast successes can hide failed customers.

Use the definitions from Latency and throughput to predict the results before running.

These cycle-time estimates describe steady behavior, not exact percentile predictions. Starting and draining the run create shorter waits.

Add clients without adding capacity

Run the commands for your chosen language:

python3 load.py 1
python3 load.py 4
python3 load.py 16
# Or:
node load.mts 1
node load.mts 4
node load.mts 16

Each baseline run should take roughly sixteen seconds. The following is illustrative output, not a recorded benchmark:

success=160 errors=0 rps=10.3 p95_ms=99
success=160 errors=0 rps=10.4 p95_ms=390
success=160 errors=0 rps=10.4 p95_ms=1540

Record your results in a small table. Repeat each run once. Do not compare tiny differences between repeats as evidence of a faster runtime.

The important pattern is stable throughput with rising latency. Customers pay for the waiting even when every request eventually succeeds.

For this lab, assume Kartly wants p95 below 300 ms, with no errors. The four-client run can fail that target without failing a single HTTP request. Capacity must include an acceptable waiting time, not just eventual completion.

A flat rate identifies a limit, but not automatically the production bottleneck. Here, you know the limit because you built it.

Change one cause, then remeasure

Stop the server with Ctrl+C. Restart it with a 45 ms delay:

python3 server.py 45
# Or:
node server.mts 45

Repeat the same generator commands. The prediction follows directly from the fixture:

This edit simulates shorter occupancy; it does not demonstrate a real database optimization. In a real application, first measure where the request spends time.

Adding independent slots is another option. It helps only if dependencies and resources can support the extra work. Vertical and horizontal scaling examines that decision.

Your experiment now supports a causal claim: changing occupied time changed throughput. It still does not establish production readiness.

Know what this run cannot prove

This generator waits before sending each client’s next request. At saturation, it cannot maintain an independent arrival rate. A festival crowd can keep arriving even when earlier customers are still waiting. An open load model separates arrival scheduling from completion.^{3Open and closed modelsGrafana Labs, Grafana k6 documentation, 2026}

The fixture also omits authentication, real product data, database contention and pricing failures. It is intentionally different from the preceding single-server URL shortener: this lab isolates queueing rather than benchmarking stored links.

Before testing a real service, write down the request mix and data size. Set a latency target and an error threshold. Arrange a stop condition before creating load.

If this lab reports errors, stop the generator first. Confirm the server is running, check its terminal, then retry with one client. Restarting clears the fixture’s pending work; there is no stored business data.

Never point this generator at production or a public website. For larger experiments, use a separate load-generating machine and measure the generator itself.

The next lesson follows a dependency this localhost experiment largely hides: TCP and UDP and how data travels between machines.

Common mistakes

• Calling the largest rate “capacity”. Name the latency and error limits that made the rate acceptable.
• Changing several variables together. Keep client count fixed when testing a service-time change.
• Reporting p95 without errors. Failed requests are absent from this generator’s percentile.
• Treating low CPU as proof of spare capacity. The fixture waits inside its only application slot.

Check your understanding

Summary