<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[EniyiSunucum]]></title><description><![CDATA[EniyiSunucum]]></description><link>https://eniyisunucum.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/69928e7f1ec3d43c05c0efc1/d36dc991-2497-49d3-b776-84a1b42562b8.png</url><title>EniyiSunucum</title><link>https://eniyisunucum.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Mon, 21 Sep 2026 03:22:05 GMT</lastBuildDate><atom:link href="https://eniyisunucum.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Your VDS Benchmark Is Probably Measuring the Wrong Thing]]></title><description><![CDATA[A benchmark screenshot is easy to produce and hard to interpret. A fast dd result may be page cache, a CPU score may be a short turbo burst, and a Speedtest result may say more about the selected serv]]></description><link>https://eniyisunucum.hashnode.dev/your-vds-benchmark-is-probably-measuring-the-wrong-thing</link><guid isPermaLink="true">https://eniyisunucum.hashnode.dev/your-vds-benchmark-is-probably-measuring-the-wrong-thing</guid><category><![CDATA[Linux]]></category><category><![CDATA[networking]]></category><category><![CDATA[performance]]></category><category><![CDATA[Devops]]></category><category><![CDATA[virtualization]]></category><dc:creator><![CDATA[EniyiSunucum]]></dc:creator><pubDate>Thu, 17 Sep 2026 00:52:30 GMT</pubDate><content:encoded><![CDATA[<p>A benchmark screenshot is easy to produce and hard to interpret. A fast <code>dd</code> result may be page cache, a CPU score may be a short turbo burst, and a Speedtest result may say more about the selected server than the virtual machine. None of those numbers is necessarily false. They are simply incomplete.</p>
<p>When evaluating a <a href="https://eniyisunucum.com/en/vds-server">VDS</a>, the useful question is not “What is the highest number I can obtain?” It is “Does the system deliver predictable service under a documented workload, and can another operator reproduce the observation?” This guide builds a small Linux baseline around three signals that expose different failure modes:</p>
<ul>
<li>CPU steal time, which can reveal scheduling contention outside the guest</li>
<li>disk p95 and p99 completion latency, which exposes slow tail operations hidden by averages</li>
<li>network delay variation and loss, measured without saturating an unknown path</li>
</ul>
<p>The method is intentionally vendor-neutral. It produces evidence, not a universal pass/fail score.</p>
<h2>Why common benchmark results mislead</h2>
<p>Virtual infrastructure has layers. The guest kernel sees virtual CPUs, virtual block devices, and virtual network interfaces. Below them may be a hypervisor scheduler, shared storage, caching, rate limits, and a physical network used by other tenants. A single benchmark run collapses all of those changing conditions into one number.</p>
<p>Several effects can make that number look better or worse than normal service:</p>
<ol>
<li><strong>Warm caches:</strong> repeated reads may come from RAM or a storage controller cache rather than the underlying media.</li>
<li><strong>Burst allowances:</strong> CPU, disk, or network limits may permit a short burst before enforcing a sustained rate.</li>
<li><strong>Queue depth:</strong> an unrealistic queue can maximize throughput while producing latency that would hurt a database.</li>
<li><strong>Remote endpoint choice:</strong> a public test server can be overloaded or reached through a different path each time.</li>
<li><strong>Noisy timing:</strong> five quiet minutes at noon do not describe the same host during the evening peak.</li>
</ol>
<p>That is why the baseline below records context, runs for minutes rather than seconds, and is repeated in separate time windows.</p>
<h2>1. Freeze the test conditions</h2>
<p>Do not benchmark a production disk or run stress tools while latency-sensitive customer work is active. Use an expendable instance or a maintenance window, keep at least 20% of the filesystem free, and remove the temporary test file afterward.</p>
<p>Install the required tools using your distribution packages: <code>sysstat</code>, <code>stress-ng</code>, <code>fio</code>, <code>jq</code>, <code>iperf3</code>, and <code>mtr</code>. Before every run, capture the environment:</p>
<pre><code class="language-bash">mkdir -p "$HOME/vds-baseline"
cd "$HOME/vds-baseline"

{
  date --iso-8601=seconds
  uname -a
  lscpu
  free -h
  lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,ROTA,DISC-MAX
  df -hT
  fio --version
  iperf3 --version
} &gt; environment.txt 2&gt;&amp;1
</code></pre>
<p>Also record the instance size, region, image, filesystem, advertised limits, and whether other workloads were running. Keep the same image and test parameters when comparing providers or dates. Change one variable at a time.</p>
<p>Run the complete baseline at least three times in each of three windows: quiet, normal, and expected peak. One sample is an anecdote; repeated distributions are a baseline.</p>
<h2>2. Observe CPU steal instead of trusting a CPU score</h2>
<p>Linux reports <code>%steal</code> as time in which a virtual CPU was ready to run but the hypervisor did not schedule it. On a KVM-style guest, recurring steal during CPU demand can indicate host contention. Zero steal does not prove an uncontended host: some virtualization platforms do not expose it, and frequency throttling or host-level limits can appear elsewhere.</p>
<p>First observe an idle period:</p>
<pre><code class="language-bash">LC_ALL=C mpstat -P ALL -o JSON 1 300 &gt; cpu-idle-mpstat.json
</code></pre>
<p>Then create a controlled five-minute load while collecting the same metric in a second terminal:</p>
<pre><code class="language-bash"># Terminal 1: monitor every vCPU
LC_ALL=C mpstat -P ALL -o JSON 1 330 &gt; cpu-load-mpstat.json

# Terminal 2: use all visible vCPUs for five minutes
stress-ng --cpu "$(nproc)" --cpu-method matrixprod \
  --timeout 300s --metrics-brief |&amp; tee cpu-stress.txt
</code></pre>
<p>Do not report only the final <code>Average:</code> line. Look for the shape of steal time: its median, upper percentiles, longest burst, and whether one vCPU is consistently worse. A brief scheduling event and repeated ten-second clusters have different operational consequences even if their averages match.</p>
<p>A simple extraction for the aggregate <code>all</code> row is:</p>
<pre><code class="language-bash">jq -r '.sysstat.hosts[0].statistics[]."cpu-load"[]
  | select(.cpu=="all") | .steal' cpu-load-mpstat.json \
  | sort -n &gt; cpu-steal-percent.txt

awk '{v[NR]=$1} END {
  p50=int((NR-1)*0.50)+1
  p95=int((NR-1)*0.95)+1
  p99=int((NR-1)*0.99)+1
  printf "steal p50=%.2f%% p95=%.2f%% p99=%.2f%%\n", v[p50],v[p95],v[p99]
}' cpu-steal-percent.txt
</code></pre>
<p>The JSON layout can vary between major <code>sysstat</code> releases, so inspect one sample before automating the parser. Preserve the raw file; it is the source of truth.</p>
<p>Avoid declaring that every value above a fixed percentage is “bad.” The meaningful threshold depends on the workload. For a latency-sensitive service, even short steal bursts may matter. For a batch worker, completion time and sustained throughput may matter more. Correlate steal bursts with application latency before blaming the host.</p>
<h2>3. Measure disk tail latency with fio</h2>
<p>Storage throughput and storage responsiveness are different properties. A device can deliver many megabytes per second while a small portion of operations waits hundreds of milliseconds. Databases, package managers, and control panels often feel that tail.</p>
<p>Use a regular file on the filesystem you actually want to test. Do not point <code>fio</code> at a raw device containing data. Create a dataset larger than trivial caches, using direct I/O:</p>
<pre><code class="language-bash">TESTFILE=/var/tmp/vds-fio.bin

fio --name=prepare --filename="$TESTFILE" --size=4G \
  --rw=write --bs=1M --ioengine=libaio --iodepth=16 \
  --direct=1 --end_fsync=1
</code></pre>
<p>Start with a latency-oriented random-read workload at queue depth 1. This resembles a single thread waiting for each small read rather than a synthetic throughput contest:</p>
<pre><code class="language-bash">fio --name=randread-qd1 --filename="$TESTFILE" --size=4G \
  --rw=randread --bs=4k --ioengine=libaio --iodepth=1 --numjobs=1 \
  --direct=1 --time_based=1 --runtime=300 --ramp_time=30 \
  --percentile_list=50:95:99:99.9 --output-format=json \
  --output=fio-randread-qd1.json
</code></pre>
<p>Next run a modest mixed workload. A 70/30 read/write ratio is not “the standard”; it is simply a documented second probe. Replace it with a trace-informed ratio if you know the application:</p>
<pre><code class="language-bash">fio --name=randrw-qd8 --filename="$TESTFILE" --size=4G \
  --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio \
  --iodepth=8 --numjobs=1 --direct=1 --time_based=1 \
  --runtime=300 --ramp_time=30 --percentile_list=50:95:99:99.9 \
  --output-format=json --output=fio-randrw-qd8.json
</code></pre>
<p>Extract completion-latency percentiles in nanoseconds from the JSON rather than copying the bandwidth headline:</p>
<pre><code class="language-bash">jq '.jobs[0] | {
  read_iops: .read.iops,
  read_p95_ns: .read.clat_ns.percentile["95.000000"],
  read_p99_ns: .read.clat_ns.percentile["99.000000"],
  write_iops: .write.iops,
  write_p95_ns: .write.clat_ns.percentile["95.000000"],
  write_p99_ns: .write.clat_ns.percentile["99.000000"]
}' fio-randrw-qd8.json
</code></pre>
<p>Compare p50, p95, and p99 together. A growing p99-to-p50 ratio is often more informative than a small change in average latency. Run <code>iostat -xz 1</code> alongside <code>fio</code> and preserve it too; queue size, utilization, and device latency help explain a poor result.</p>
<p>Delete only the known test file when finished:</p>
<pre><code class="language-bash">rm -- "$TESTFILE"
</code></pre>
<h2>4. Separate network jitter from bandwidth</h2>
<p>Jitter is delay variation, not low bandwidth. Testing at line rate can create the very queueing delay you then report as a defect. Use an <code>iperf3</code> server you control, pin its location, and test at a rate safely below the expected link limit. Ideally use two targets: one nearby to expose the access path and one in another network to expose transit behavior.</p>
<p>Collect five minutes of RTT samples without DNS lookup:</p>
<pre><code class="language-bash">TARGET=203.0.113.10   # replace with your controlled endpoint

ping -n -i 0.2 -c 1500 "$TARGET" \
  | awk -F'time=' '/time=/{split($2,a," "); print a[1]}' &gt; rtt-ms.txt

awk 'NR&gt;1 {d=$1-prev; if(d&lt;0)d=-d; print d} {prev=$1}' \
  rtt-ms.txt &gt; ipdv-ms.txt
</code></pre>
<p>The second file contains absolute successive RTT differences, a practical form of inter-packet delay variation. Calculate percentiles for both files:</p>
<pre><code class="language-bash">for file in rtt-ms.txt ipdv-ms.txt; do
  sort -n "$file" | awk -v name="$file" '{v[NR]=$1} END {
    p50=int((NR-1)*0.50)+1
    p95=int((NR-1)*0.95)+1
    p99=int((NR-1)*0.99)+1
    printf "%s p50=%.3fms p95=%.3fms p99=%.3fms samples=%d\n", \
      name,v[p50],v[p95],v[p99],NR
  }'
done
</code></pre>
<p>Then use UDP <code>iperf3</code> at a declared offered rate, for example 50 Mbit/s, and capture its jitter and loss in both directions:</p>
<pre><code class="language-bash">iperf3 -c "$TARGET" -u -b 50M -t 120 -O 10 --json &gt; udp-forward.json
iperf3 -c "$TARGET" -u -b 50M -t 120 -O 10 -R --json &gt; udp-reverse.json

jq '.end.sum | {jitter_ms, lost_packets, packets, lost_percent}' \
  udp-forward.json udp-reverse.json
</code></pre>
<p>Treat ICMP results as supporting evidence, not a verdict; routers may deprioritize echo traffic. If a run degrades, use <code>mtr -ezbw -c 300 "$TARGET"</code> to compare paths, but do not assign blame to the first hop showing loss unless that loss continues to later hops.</p>
<h2>Turn measurements into a defensible comparison</h2>
<p>Create one row per run, not one row per provider. Include timestamp, host region, workload state, CPU-steal p50/p95/p99, disk read and write p50/p95/p99, RTT percentiles, IPDV percentiles, UDP loss, and tool versions. Report the median run plus the worst observed window. Keep raw outputs so someone else can audit the summary.</p>
<p>The final interpretation should distinguish capacity from consistency:</p>
<ul>
<li>CPU throughput with low steal can still vary because of clock policy.</li>
<li>High disk IOPS with unstable p99 latency may be poor for synchronous work.</li>
<li>High network throughput with loss or high IPDV may be poor for voice, games, or clustered systems.</li>
<li>One degraded remote route does not prove a weak virtual machine.</li>
</ul>
<p>A useful benchmark is a controlled experiment with enough context to be repeated. Once the method is stable, the numbers become comparable. Until then, a large score is only a screenshot.</p>
<hr />
<p><strong>Disclosure:</strong> This article was prepared by the EniyiSunucum team, operator of AS215068. It presents a vendor-neutral test method and contains no benchmark result or performance claim for the company. Company information: <a href="https://eniyisunucum.com/">EniyiSunucum</a>.</p>
]]></content:encoded></item></channel></rss>