Case notes

Eight pieces of work in the same order: problem, analysis, fixes, verification. Customers, internal product components and absolute figures are not named — only the role, the method, the tooling and the result.

Outcome only: problem and fixes

API-security platform · 2025–2026

A memory leak in a Go component

Problem

A component of a distributed Kubernetes installation grew without bound under load and was OOM-killed on reaching its memory limit. It could only be reproduced in the cluster: stand up a multi-component bench, apply load, wait. The check cycle ran in hours, so development's hypotheses were tested slowly and one at a time.

Analysis

I collapsed the cluster installation into a local docker-compose, keeping whatever affects memory behaviour: container limits and the environment variables that drive the allocator. The reproduction criterion was set up front and measurably — the limit exceeded within minutes under load, not "it seems to be growing". Load came from the large-body profile, which is where the leak showed. Then heap profiles through pprof, taken over time and compared against each other: that is what separates "the component allocates a lot" from "the component does not give memory back".

Fixes

I reduced the profile to a specific place in the code and handed the component owner a report with pointed recommendations for that place, not "go look for a leak". The owner of the code made the change — that is the right split: reproduction, localisation and confirmation are mine.

Verification

A retest on the same bench with the same profile, compared before and after against the same criterion. The hypothesis-check cycle went from hours on a cluster to minutes on a laptop, and the reproduction bench stayed as an asset: a regression in that place can be checked at any time.

API-security platform · 2025–2026

Generating large bodies with attacks inside

Problem

The profile needed was one no off-the-shelf tool produces: large request bodies in several formats — JSON, XML, multipart, form-urlencoded — with attack payloads embedded inside the body, generated on the fly. Static fixtures do not work on principle: they get cached on the generator side and on the system side alike. Out of the box k6 could not carry the profile: generator memory grew without bound, the bench went into swap, and the test never reached the target RPS.

Analysis

The cause was not the volume of data but where the bodies were assembled: assembly happened inside the iteration, and in k6 every VU is a separate JS runtime with its own heap, so the cost multiplied by the number of VUs.

Fixes

I moved assembly into a pool of pre-built bodies held in a structure shared across VUs, with the iteration taking the next element in turn, and gave each combination of format, attack count and body size its own pool. Allocations came out of the hot path: pools of ready strings, pre-allocated arrays, single-pass concatenation, a fast identifier generator of my own. The body is padded to the requested size while staying valid for its format — otherwise the system rejects the request at parsing and the test measures the wrong thing.

Verification

Generator memory stopped growing over the course of a test — a flat shelf instead of unbounded growth — and the profile holds the target RPS within the bench's normal resources. A set of scenarios for different body sizes and attack shares was built on top of it.

API-security platform · 2025–2026

Picking instances for a class of load

Problem

The machine type for an installation was chosen by tradition: whatever had worked before. Yet cloud instance families differ severalfold in their ratio of price, cores, memory and network, and for one specific load profile the difference in cost of ownership is money.

Analysis

Instance types cannot be swept by hand: building one bench manually eats a day. I automated the whole build — environment, system under test, backend stubs and load generator in one command. One load profile was fixed across all configurations, otherwise the comparison means nothing, and I ran the matrix of instance type against profile, capturing CPU, memory, network and failure metrics at every point.

Fixes

It came down to one efficiency metric: not peak RPS but RPS per dollar while latency requirements hold and no requests are lost — the machine with the higher RPS often loses on cost of ownership. The output was instance-type recommendations per class of load, backed by measurements.

Verification

For some traffic types ARM instances turned out twice as efficient on CPU at the same load — a difference the by-tradition choice had been missing. The procedure itself stayed repeatable: re-running the matrix is a pipeline run, so it can be recomputed for a new product version or a new instance family.

API-security platform · 2025–2026

Capacity planning instead of expert guesswork

Problem

"How much resource should we budget for our traffic" is a standing question from presales and customers, and the answer was given from expertise, by analogy with a similar installation. Being wrong is expensive in both directions: under-provision and it fails at the peak, over-provision and the customer pays for air and argues about it.

Analysis

I built a dataset out of the accumulated runs: total and attack RPS, inbound and outbound bandwidth, core count, memory, and the observed CPU utilisation as mean and percentiles. Features were normalised per core: without that the model learns "bigger machine, fewer per cent" and does not transfer between configurations.

Fixes

Ridge regression with cross-validation: few data points, few features, a near-linear relationship — a heavier model would have overfitted the noise. For the parameters a customer usually does not know, defaults are derived from the sample's median ratios, so they supply only what they do know. The model predicts not just mean utilisation but percentiles: you plan against the tail, or the system falls over at the peak. On top of it a web tool for CPU and RAM sizing, so presales and deployment compute it themselves instead of asking the perf team.

Verification

Where a request landed on a point already in the run history, the prediction matched the measurement; outside the dataset it is an approximation, and I describe it as exactly that. Any assembled profile could also be run for real and give actual data within half an hour — so a forecast always has something to check it against, and every such run sharpens the model.

API-security platform · 2025–2026

Load testing as a service

Problem

Tests ran through CI: to execute a scenario an engineer edited pipeline variables and kept their combinations in their head. For everyone outside two or three people it was a black box, and results lived in job artefacts.

Analysis

The hard part is not the web interface. First, an honest conversion between bandwidth and RPS through the actual sizes of bodies and headers: the customer thinks in gigabits, the tool thinks in requests per second, and the conversion has to be exact or the test measures a different profile. Second, proof that a generated scenario really delivers the requested load rather than merely being syntactically valid.

Fixes

A scenario generator: from the given parameters — method distribution, headers, body formats and sizes, load shapes (plateau, steps, linear ramp, spikes) and a catalogue of attack payloads embedded into a body of the right format — it assembles a self-contained k6 script, ready to run and to edit by hand; VUs and iterations are derived from RPS. Validation in three levels: static checks, a real script start on the backend, and a full check that drives an emulator and compares actual RPS against the requested figure within ±1%, plus a background re-check while idle so stale scenarios do not surface at the moment they are needed. Alongside it a run service in Go and React: a task queue, deployment of the configuration into Kubernetes through Helm, log streaming over WebSocket, and run history in PostgreSQL.

Verification

The service runs in production as an internal team tool. Scenarios are versioned, reused and cannot go stale unnoticed, and run results became structured data — the very data the model in case 04 was built on.

Industrial platform on Kubernetes · 2022–2025

Delete pod: a readiness probe that checked nothing

Problem

The platform ships into the customer's Kubernetes and pods restart on every update — we needed to know what the client sees in that window.

Analysis

I deleted service and worker pods during a run and watched errors and latency across the restart window. One service kept returning 5xx after the cluster already considered the pod ready: the readiness probe checked only that the port was open, not that dependencies were reachable and the cache warm, so traffic arrived too early. A second hole sat next to it: drain without a PodDisruptionBudget removed replicas at the same time, so during an update the service could be left with no live instances at all.

Fixes

The probe was pointed at a real health endpoint. A PodDisruptionBudget with minAvailable set to 1 went into the Helm chart.

Verification

A repeat run with the same injections: the 5xx window after a restart closed, and drain no longer left the service without live replicas. Both scenarios stayed in the release run, so restarts and cluster updates are checked under load rather than during a customer's upgrade.

Industrial platform on Kubernetes · 2022–2025

PostgreSQL failover under load

Problem

The customer needed an RTO for the database — measured under load, not quoted from documentation.

Analysis

A bench with a PostgreSQL cluster and streaming replication. Under load I killed the primary and counted not only the cluster's switchover time but also how many operations the client lost in that period. The finding was the gap between two numbers: the cluster had already promoted a new primary while the application still held stale connections and answered with errors — the real outage window was noticeably wider than the cluster's.

Fixes

The recommendations went to the application side: connection-pool handling and reconnection after a primary change, so the application does not hold dead connections for longer than the failover itself lasts. The customer's test protocol carried the real window and the lost operations rather than the cluster's switchover time — an RTO should be stated as the client experiences it.

Verification

A repeat failover under the same load, measured against the same criterion so the effect of the changes was visible. The scenario stayed in the test methodology: the database RTO is now a measurement that can be repeated on any installation, not a declaration.

Industrial platform on Kubernetes · 2022–2025

The acceptance act would not close while the scene stalled

Problem

Two jobs at once. The formal one: load testing was part of the key customer's requirements, and without a signed test protocol the acceptance act would not close. The substantive one: the 3D scene stalled badly at times on the customer's real data. Access to their infrastructure was restricted and test windows had to be agreed.

Analysis

I reproduced the problem on a bench with the customer's profile and data volumes. It came down to heavy database queries that indexes could not cure: the issue was not one query's plan but the way the data was stored. That called for a full audit of the data and the schema rather than query-by-query tuning.

Fixes

The storage schema and tables themselves were reworked: normalisation, with part of the logic moved into functions inside the database. A lot of access went through an ORM, so to keep the system from falling apart during the move some queries were rewritten by hand and the old structures were covered with views over the new tables — the application kept seeing a familiar shape while storage changed underneath it.

Verification

A run on the customer's profile after the storage rework, against the same criteria that had been agreed before the test. The test protocol satisfied the requirement for the acceptance act. The recurring run was not a formality here: this functionality was migrating from the ORM to hand-written queries step by step, and every step had to be compared against the previous measurement — otherwise one rewritten query could quietly bring the stalls back in the next release.