Skip to content

Commit 0a40dfd

Browse files
alexeyrclaude
alexeyr
and
claude
committed
Add github-action-benchmark integration for benchmark tracking
- Add convert_to_benchmark_json.rb script to convert benchmark results to JSON format compatible with github-action-benchmark - Track RPS (customBiggerIsBetter) and latency/failure % (customSmallerIsBetter) - Exclude max latencies (not stable enough for regression detection) - Alert threshold set to 150% (50% regression) - Store benchmark data in docs/benchmarks on master branch - Enable job summary for all runs (PRs and pushes) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 71bb129 commit 0a40dfd

File tree

2 files changed

+245
-1
lines changed

2 files changed

+245
-1
lines changed

‎.github/workflows/benchmark.yml‎

Lines changed: 87 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,45 @@ jobs:
297297
echo "Generated files:"
298298
ls -lh bench_results/
299299
300+
- name: Convert Core benchmark results to JSON
301+
if: env.RUN_CORE
302+
run: |
303+
ruby benchmarks/convert_to_benchmark_json.rb "Core: "
304+
305+
- name: Store Core RPS benchmark results
306+
if: env.RUN_CORE
307+
uses: benchmark-action/github-action-benchmark@v1
308+
with:
309+
name: Core Benchmark - RPS
310+
tool: customBiggerIsBetter
311+
output-file-path: bench_results/benchmark_rps.json
312+
gh-pages-branch: benchmark-data
313+
benchmark-data-dir-path: docs/benchmarks
314+
alert-threshold: '150%'
315+
github-token: ${{ secrets.GITHUB_TOKEN }}
316+
comment-on-alert: true
317+
alert-comment-cc-users: '@alexeyr-ci2'
318+
fail-on-alert: true
319+
summary-always: true
320+
auto-push: false
321+
322+
- name: Store Core latency benchmark results
323+
if: env.RUN_CORE
324+
uses: benchmark-action/github-action-benchmark@v1
325+
with:
326+
name: Core Benchmark - Latency
327+
tool: customSmallerIsBetter
328+
output-file-path: bench_results/benchmark_latency.json
329+
gh-pages-branch: benchmark-data
330+
benchmark-data-dir-path: docs/benchmarks
331+
alert-threshold: '150%'
332+
github-token: ${{ secrets.GITHUB_TOKEN }}
333+
comment-on-alert: true
334+
alert-comment-cc-users: '@alexeyr-ci2'
335+
fail-on-alert: true
336+
summary-always: true
337+
auto-push: false
338+
300339
- name: Upload Core benchmark results
301340
uses: actions/upload-artifact@v4
302341
if: env.RUN_CORE && always()
@@ -470,6 +509,45 @@ jobs:
470509
echo "Generated files:"
471510
ls -lh bench_results/
472511
512+
- name: Convert Pro benchmark results to JSON
513+
if: env.RUN_PRO
514+
run: |
515+
ruby benchmarks/convert_to_benchmark_json.rb "Pro: "
516+
517+
- name: Store Pro RPS benchmark results
518+
if: env.RUN_PRO
519+
uses: benchmark-action/github-action-benchmark@v1
520+
with:
521+
name: Pro Benchmark - RPS
522+
tool: customBiggerIsBetter
523+
output-file-path: bench_results/benchmark_rps.json
524+
gh-pages-branch: benchmark-data
525+
benchmark-data-dir-path: docs/benchmarks
526+
alert-threshold: '150%'
527+
github-token: ${{ secrets.GITHUB_TOKEN }}
528+
comment-on-alert: true
529+
alert-comment-cc-users: '@alexeyr-ci2'
530+
fail-on-alert: true
531+
summary-always: true
532+
auto-push: false
533+
534+
- name: Store Pro latency benchmark results
535+
if: env.RUN_PRO
536+
uses: benchmark-action/github-action-benchmark@v1
537+
with:
538+
name: Pro Benchmark - Latency
539+
tool: customSmallerIsBetter
540+
output-file-path: bench_results/benchmark_latency.json
541+
gh-pages-branch: benchmark-data
542+
benchmark-data-dir-path: docs/benchmarks
543+
alert-threshold: '150%'
544+
github-token: ${{ secrets.GITHUB_TOKEN }}
545+
comment-on-alert: true
546+
alert-comment-cc-users: '@alexeyr-ci2'
547+
fail-on-alert: true
548+
summary-always: true
549+
auto-push: false
550+
473551
- name: Upload Pro benchmark results
474552
uses: actions/upload-artifact@v4
475553
if: env.RUN_PRO && always()
@@ -488,7 +566,15 @@ jobs:
488566
echo "✅ Server stopped"
489567
490568
# ============================================
491-
# STEP 7: WORKFLOW COMPLETION
569+
# STEP 7: PUSH BENCHMARK DATA
570+
# ============================================
571+
- name: Push benchmark data
572+
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
573+
run: |
574+
git push 'https://github-actions:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git' benchmark-data:benchmark-data
575+
576+
# ============================================
577+
# STEP 8: WORKFLOW COMPLETION
492578
# ============================================
493579
- name: Workflow summary
494580
if: always()

‎benchmarks/convert_to_benchmark_json.rb‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
#!/usr/bin/env ruby
2+
# frozen_string_literal: true
3+
4+
# Converts benchmark summary files to JSON format for github-action-benchmark
5+
# Outputs two files:
6+
# - benchmark_rps.json (customBiggerIsBetter)
7+
# - benchmark_latency.json (customSmallerIsBetter)
8+
#
9+
# Usage: ruby convert_to_benchmark_json.rb [prefix]
10+
# prefix: Optional prefix for benchmark names (e.g., "Core: " or "Pro: ")
11+
12+
require "json"
13+
14+
BENCH_RESULTS_DIR = "bench_results"
15+
PREFIX = ARGV[0] || ""
16+
17+
# rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
18+
19+
# Parse a summary file and return array of hashes with metrics
20+
# Expected format (tab-separated):
21+
# Route RPS p50(ms) p90(ms) p99(ms) max(ms) Status
22+
# or for node renderer:
23+
# Test Bundle RPS p50(ms) p90(ms) p99(ms) max(ms) Status
24+
def parse_summary_file(file_path, prefix: "")
25+
return [] unless File.exist?(file_path)
26+
27+
lines = File.readlines(file_path).map(&:strip).reject(&:empty?)
28+
return [] if lines.length < 2
29+
30+
header = lines.first.split("\t")
31+
results = []
32+
33+
lines[1..].each do |line|
34+
cols = line.split("\t")
35+
row = header.zip(cols).to_h
36+
37+
# Determine the name based on available columns
38+
name = row["Route"] || row["Test"] || "unknown"
39+
bundle_suffix = row["Bundle"] ? " (#{row['Bundle']})" : ""
40+
full_name = "#{prefix}#{name}#{bundle_suffix}"
41+
42+
# Skip if we got FAILED values
43+
next if row["RPS"] == "FAILED"
44+
45+
# Parse numeric values
46+
rps = row["RPS"]&.to_f
47+
p50 = row["p50(ms)"]&.to_f
48+
p90 = row["p90(ms)"]&.to_f
49+
p99 = row["p99(ms)"]&.to_f
50+
51+
# Calculate failed percentage from Status column
52+
failed_pct = calculate_failed_percentage(row["Status"])
53+
54+
results << {
55+
name: full_name,
56+
rps: rps,
57+
p50: p50,
58+
p90: p90,
59+
p99: p99,
60+
failed_pct: failed_pct
61+
}
62+
end
63+
64+
results
65+
end
66+
67+
# rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
68+
69+
# Calculate failed request percentage from status string
70+
# Status format: "200=7508,302=100,5xx=10" etc.
71+
def calculate_failed_percentage(status_str)
72+
return 0.0 if status_str.nil? || status_str == "missing"
73+
74+
total = 0
75+
failed = 0
76+
77+
status_str.split(",").each do |part|
78+
code, count = part.split("=")
79+
count = count.to_i
80+
total += count
81+
82+
# Consider 0 (for Vegeta), 4xx and 5xx as failures, also "other"
83+
failed += count if code.match?(/^[045]/) || code == "other"
84+
end
85+
86+
return 0.0 if total.zero?
87+
88+
(failed.to_f / total * 100).round(2)
89+
end
90+
91+
# Convert results to customBiggerIsBetter format (for RPS)
92+
def to_rps_json(results)
93+
results.map do |r|
94+
{
95+
name: "#{r[:name]} - RPS",
96+
unit: "requests/sec",
97+
value: r[:rps]
98+
}
99+
end
100+
end
101+
102+
# Convert results to customSmallerIsBetter format (for latencies and failure rate)
103+
def to_latency_json(results)
104+
output = []
105+
106+
results.each do |r|
107+
output << {
108+
name: "#{r[:name]} - p50 latency",
109+
unit: "ms",
110+
value: r[:p50]
111+
}
112+
output << {
113+
name: "#{r[:name]} - p90 latency",
114+
unit: "ms",
115+
value: r[:p90]
116+
}
117+
output << {
118+
name: "#{r[:name]} - p99 latency",
119+
unit: "ms",
120+
value: r[:p99]
121+
}
122+
output << {
123+
name: "#{r[:name]} - failed requests",
124+
unit: "%",
125+
value: r[:failed_pct]
126+
}
127+
end
128+
129+
output
130+
end
131+
132+
# Main execution
133+
all_results = []
134+
135+
# Parse Rails benchmark
136+
rails_summary = File.join(BENCH_RESULTS_DIR, "summary.txt")
137+
all_results.concat(parse_summary_file(rails_summary, prefix: PREFIX)) if File.exist?(rails_summary)
138+
139+
# Parse Node Renderer benchmark
140+
node_renderer_summary = File.join(BENCH_RESULTS_DIR, "node_renderer_summary.txt")
141+
if File.exist?(node_renderer_summary)
142+
all_results.concat(parse_summary_file(node_renderer_summary, prefix: "#{PREFIX}NodeRenderer: "))
143+
end
144+
145+
if all_results.empty?
146+
puts "No benchmark results found to convert"
147+
exit 0
148+
end
149+
150+
# Write RPS JSON (bigger is better)
151+
rps_json = to_rps_json(all_results)
152+
File.write(File.join(BENCH_RESULTS_DIR, "benchmark_rps.json"), JSON.pretty_generate(rps_json))
153+
puts "Wrote #{rps_json.length} RPS metrics to benchmark_rps.json"
154+
155+
# Write latency/failure JSON (smaller is better)
156+
latency_json = to_latency_json(all_results)
157+
File.write(File.join(BENCH_RESULTS_DIR, "benchmark_latency.json"), JSON.pretty_generate(latency_json))
158+
puts "Wrote #{latency_json.length} latency/failure metrics to benchmark_latency.json"

0 commit comments

Comments
 (0)