benchmarking languages and framework
Working with customers and partners on application modernization projects sparks discussions about governance. This system of rules and practices should influence the choice of language and related framework. However, from a governance and a bias perspective, I highlight the risks of randomly selecting these without performing relevant test case benchmarking.
Regarding bias, Application teams have their favorite language and framework in mind and will research proof that this was the best option. The Internet is resourceful in supporting opinionated views with a few details about the methodology used.
Let's take this article as a reference. While the overall content provides a sufficient overview for an introduction, it does not help to perform a similar benchmarking within our environment, which will most likely significantly differ from the author. Let's address this missing part as a companion to the article.
the article benchmark
From a setup perspective, the author defines the following:
- The hardware is based on a MacBook Pro M2 with 16GB of RAM
- Bombardier; a HTTP(S) bencharmking tool written in Go(lang)
- Go version 1.21.3; follow this guide to install Go on your system
- Quarkus 3.5.1 with Java v21; follow this guide to install Quarkus on your system
These are a good start; however, the Go code calls from Gin, a Go HTTP framework for which we don't have any version reference. The same goes for Bombardier.
code
The below code is the as-is output from the article. These are the traditional hello world examples to showcase languages and easily compare them. Even with years of development experience, anyone needed more to run the code from an unknown language and benchmark it.
The code is also available within this repository within docs/sources/hello-world-article in the following subdirectories:
hello-world-gothe ready to use Go codehello-world-quarkusthe ready to use Quarkus code
go/gin
The Go code from the article
package main
import (
"net/http"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.New()
r.GET("/", func(c *gin.Context) {
c.String(http.StatusOK, "Hello world!")
})
r.Run(":3000")
}
Here the steps to run this code:
- Open two terminal consoles
- Create a directory like
hello-world-goandcdin the directory - Create a file called
main.goand copy the above code in - Initialize the module(s) for the project:
go.mod with the following content:
module github.com/romdalf/ocp-projects/docs/sources/hello-world-article/go
go 1.21.5
require github.com/gin-gonic/gin v1.9.1
require (
github.com/bytedance/sonic v1.9.1 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.14.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.2.4 // indirect
github.com/leodido/go-urn v1.2.4 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
golang.org/x/arch v0.3.0 // indirect
golang.org/x/crypto v0.9.0 // indirect
golang.org/x/net v0.10.0 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/text v0.9.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
- Check for missing modules:
go.sum with the following content:
github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s=
github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams=
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/gabriel-vasile/mimetype v1.4.2 h1:w5qFW6JKBz9Y393Y4q372O9A7cUSequkh1Q7OhCmWKU=
github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.9.1 h1:4idEAncQnU5cB7BeOkPtxjfCSye0AAm1R0RVIqJ+Jmg=
github.com/gin-gonic/gin v1.9.1/go.mod h1:hPrL7YrpYKXt5YId3A/Tnip5kqbEAP+KLuI3SUcPTeU=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.14.0 h1:vgvQWe3XCz3gIeFDm/HnTIbj6UGmg/+t63MyGU2n5js=
github.com/go-playground/validator/v10 v10.14.0/go.mod h1:9iXMNT7sEkjXb0I+enO7QXmzG6QCsPWY4zveKFVRSyU=
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk=
github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
github.com/leodido/go-urn v1.2.4 h1:XlAE/cm/ms7TE/VMVoduSpNBoyc2dOxHs5MZSwAN63Q=
github.com/leodido/go-urn v1.2.4/go.mod h1:7ZrI8mTSeBSHl/UaRyKQW1qZeMgak41ANeCNaVckg+4=
github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ=
github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY=
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU=
github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
golang.org/x/arch v0.3.0 h1:02VY4/ZcO/gBOH6PUaoiptASxtXU10jazRCP865E97k=
golang.org/x/arch v0.3.0/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g=
golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0=
golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
- In one of the console, run the code:
Resulting in the following output:
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET / --> main.main.func1 (1 handlers)
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
[GIN-debug] Listening and serving HTTP on :3000
- In a second console, use
curlto check the service:
Resulting in the following output:
Note that there is no output logged into the first console where the service is running.
- The code seems to be working, we can build a binary:
Resulting in the creating a binary called hello-world with the following size:
- Finally, to run the binary:
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET / --> main.main.func1 (1 handlers)
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
[GIN-debug] Listening and serving HTTP on :3000
quarkus
The Java/Quarkus code from the article
package org.acme;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
import io.smallrye.common.annotation.NonBlocking;
@Path("/")
public class HelloWorldApplication {
@GET
@NonBlocking
@Produces(MediaType.TEXT_PLAIN)
public String hello() {
return "Hello World!";
}
}
Here are the steps to run this code:
- Open two terminal consoles
- Create a Quarkus project and
cdin the directory:
Resulting in the following output:
-----------
applying codestarts...
📚 java
🔨 maven
📦 quarkus
📝 config-properties
🔧 tooling-dockerfiles
🔧 tooling-maven-wrapper
🚀 resteasy-reactive-codestart
-----------
[SUCCESS] ✅ quarkus project has been successfully generated in:
--> /Users/romdalf/dev/hello-world-quarkus
-----------
Navigate into this directory and get started: quarkus dev
- Edit the file
src/main/java/org/acme/GreetingResource.javato replace the code with the article one (see above) - Rename the file
src/main/java/org/acme/GreetingResource.javatosrc/main/java/org/acme/HelloWorldApplication.java:
Note: the classname needs to match the filename.
- Remove the default
resourcesdirectory:
- In onf of the console, run the code:
Resulting in the following output:
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< org.acme:hello-world-quarkus >--------------------
[INFO] Building hello-world-quarkus 1.0.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- quarkus:3.6.4:dev (default-cli) @ hello-world-quarkus ---
[INFO] Invoking resources:3.3.1:resources (default-resources) @ hello-world-quarkus
[INFO] skip non existing resourceDirectory /Users/romdalf/dev/hello-world-quarkus/src/main/resources
[INFO] Invoking quarkus:3.6.4:generate-code (default) @ hello-world-quarkus
[INFO] Invoking compiler:3.11.0:compile (default-compile) @ hello-world-quarkus
[INFO] Changes detected - recompiling the module! :source
[INFO] Compiling 1 source file with javac [debug release 21] to target/classes
[INFO] Annotation processing is enabled because one or more processors were found
on the class path. A future release of javac may disable annotation processing
unless at least one processor is specified by name (-processor), or a search
path is specified (--processor-path, --processor-module-path), or annotation
processing is enabled explicitly (-proc:only, -proc:full).
Use -Xlint:-options to suppress this message.
Use -proc:none to disable annotation processing.
[INFO] Invoking resources:3.3.1:testResources (default-testResources) @ hello-world-quarkus
[INFO] skip non existing resourceDirectory /Users/romdalf/dev/hello-world-quarkus/src/test/resources
[INFO] Invoking quarkus:3.6.4:generate-code-tests (default) @ hello-world-quarkus
[INFO] Invoking compiler:3.11.0:testCompile (default-testCompile) @ hello-world-quarkus
[INFO] Changes detected - recompiling the module! :dependency
[INFO] Compiling 2 source files with javac [debug release 21] to target/test-classes
[INFO] Annotation processing is enabled because one or more processors were found
on the class path. A future release of javac may disable annotation processing
unless at least one processor is specified by name (-processor), or a search
path is specified (--processor-path, --processor-module-path), or annotation
processing is enabled explicitly (-proc:only, -proc:full).
Use -Xlint:-options to suppress this message.
Use -proc:none to disable annotation processing.
Listening for transport dt_socket at address: 5005
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2023-12-25 11:05:17,735 INFO [io.quarkus] (Quarkus Main Thread) hello-world-quarkus 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.6.4) started in 1.076s. Listening on: http://localhost:8080
2023-12-25 11:05:17,737 INFO [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2023-12-25 11:05:17,738 INFO [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, resteasy-reactive, smallrye-context-propagation, vertx]
- In a second console, use
curlto check the service:
Resulting in the following output:
Note that there is no output logged into the first console where the service is running.
- The code seems to be working, we can build a binary:
Resulting in a error linked to the test suites:
2023-12-25 11:10:08,561 INFO [io.quarkus] (main) hello-world-quarkus stopped in 0.014s
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] GreetingResourceTest.testHelloEndpoint:16 1 expectation failed.
Expected status code <200> but was <404>.
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.915 s
[INFO] Finished at: 2023-12-25T11:10:08+01:00
[INFO] ------------------------------------------------------------------------
- Rename the test files:
mv src/test/java/org/acme/GreetingResourceIT.java src/test/java/org/acme/HelloWorldApplicationIT.java
mv src/test/java/org/acme/GreetingResourceTest.java src/test/java/org/acme/HelloWorldApplicationTest.java
- Adapt the content of each test files like:
src/test/java/org/acme/HelloWorldApplicationIT.java
package org.acme;
import io.quarkus.test.junit.QuarkusIntegrationTest;
@QuarkusIntegrationTest
class HelloWorldApplicationIT extends HelloWorldApplicationTest {
// Execute the same tests but in packaged mode.
}
src/test/java/org/acme/HelloWorldApplicationTest.java
package org.acme;
import io.quarkus.test.junit.QuarkusTest;
import org.junit.jupiter.api.Test;
import static io.restassured.RestAssured.given;
import static org.hamcrest.CoreMatchers.is;
@QuarkusTest
class HelloWorldApplicationTest {
@Test
void testHelloEndpoint() {
given()
.when().get("/")
.then()
.statusCode(200)
.body(is("Hello World!"));
}
}
- Delete the
targetdirectory and build agan:
Resulting in the following output:
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< org.acme:hello-world-quarkus >--------------------
[INFO] Building hello-world-quarkus 1.0.0-SNAPSHOT
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ hello-world-quarkus ---
[INFO] skip non existing resourceDirectory /Users/romdalf/dev/hello-world-quarkus/src/main/resources
[INFO]
[INFO] --- quarkus:3.6.4:generate-code (default) @ hello-world-quarkus ---
[INFO]
[INFO] --- compiler:3.11.0:compile (default-compile) @ hello-world-quarkus ---
[INFO] Changes detected - recompiling the module! :source
[INFO] Compiling 1 source file with javac [debug release 21] to target/classes
[INFO]
[INFO] --- quarkus:3.6.4:generate-code-tests (default) @ hello-world-quarkus ---
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @ hello-world-quarkus ---
[INFO] skip non existing resourceDirectory /Users/romdalf/dev/hello-world-quarkus/src/test/resources
[INFO]
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ hello-world-quarkus ---
[INFO] Changes detected - recompiling the module! :dependency
[INFO] Compiling 2 source files with javac [debug release 21] to target/test-classes
[INFO] Annotation processing is enabled because one or more processors were found
on the class path. A future release of javac may disable annotation processing
unless at least one processor is specified by name (-processor), or a search
path is specified (--processor-path, --processor-module-path), or annotation
processing is enabled explicitly (-proc:only, -proc:full).
Use -Xlint:-options to suppress this message.
Use -proc:none to disable annotation processing.
[INFO]
[INFO] --- surefire:3.1.2:test (default-test) @ hello-world-quarkus ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.acme.HelloWorldApplicationTest
2023-12-25 11:19:08,017 INFO [io.quarkus] (main) hello-world-quarkus 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.6.4) started in 0.994s. Listening on: http://localhost:8081
2023-12-25 11:19:08,017 INFO [io.quarkus] (main) Profile test activated.
2023-12-25 11:19:08,018 INFO [io.quarkus] (main) Installed features: [cdi, resteasy-reactive, smallrye-context-propagation, vertx]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.769 s -- in org.acme.HelloWorldApplicationTest
2023-12-25 11:19:08,588 INFO [io.quarkus] (main) hello-world-quarkus stopped in 0.012s
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- jar:3.3.0:jar (default-jar) @ hello-world-quarkus ---
[INFO] Building jar: /Users/romdalf/dev/hello-world-quarkus/target/hello-world-quarkus-1.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- quarkus:3.6.4:build (default) @ hello-world-quarkus ---
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 750ms
[INFO]
[INFO] --- failsafe:3.1.2:integration-test (default) @ hello-world-quarkus ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- failsafe:3.1.2:verify (default) @ hello-world-quarkus ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- install:3.1.1:install (default-install) @ hello-world-quarkus ---
[INFO] Installing /Users/romdalf/dev/hello-world-quarkus/pom.xml to /Users/romdalf/.m2/repository/org/acme/hello-world-quarkus/1.0.0-SNAPSHOT/hello-world-quarkus-1.0.0-SNAPSHOT.pom
[INFO] Installing /Users/romdalf/dev/hello-world-quarkus/target/hello-world-quarkus-1.0.0-SNAPSHOT.jar to /Users/romdalf/.m2/repository/org/acme/hello-world-quarkus/1.0.0-SNAPSHOT/hello-world-quarkus-1.0.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.686 s
[INFO] Finished at: 2023-12-25T11:19:09+01:00
[INFO] ------------------------------------------------------------------------
- Finally, to run the JAR:
Resulting in the following output:
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2023-12-25 11:22:35,496 INFO [io.quarkus] (main) hello-world-quarkus 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.6.4) started in 0.636s. Listening on: http://0.0.0.0:8080
2023-12-25 11:22:35,501 INFO [io.quarkus] (main) Profile prod activated.
2023-12-25 11:22:35,502 INFO [io.quarkus] (main) Installed features: [cdi, resteasy-reactive, smallrye-context-propagation, vertx]
benchmarks
Based on the article, we don't have the overall process to perform the benchmarks. However, there is some specific parameters that are shared:
- Bombardier is the tool; in my case the binary is
bombardier-darwin-arm64and needs to be adapted to your specific environment - About 10 million requests were executed
- Different concurrency scenarios were used; 50, 100, and 300
According to Bombardier repository, the following commands would represent the above parameters:
Benchmarks for Go/Gin
./bombardier-darwin-arm64 -c 50 -n 10000000 http://localhost:3000
./bombardier-darwin-arm64 -c 100 -n 10000000 http://localhost:3000
./bombardier-darwin-arm64 -c 300 -n 10000000 http://localhost:3000
Benchmarks for Quarkus
./bombardier-darwin-arm64 -c 50 -n 10000000 http://localhost:8080
./bombardier-darwin-arm64 -c 100 -n 10000000 http://localhost:8080
./bombardier-darwin-arm64 -c 300 -n 10000000 http://localhost:8080
xychart-beta
title "Time Taken"
x-axis "Concurrent Sessions (G=Go and Q=Quarkus)" [50G, 50Q, 100G, 100Q, 300G, 300Q]
y-axis "Time in seconds"
bar [84, 106, 94, 100, 107, 103]xychart-beta
title "Requests per second"
x-axis "Concurrent Sessions (G=Go and Q=Quarkus)" [50G, 50Q, 100G, 100Q, 300G, 300Q]
y-axis "Requests in thousands"
bar [118.974, 94.220, 105.666, 99.732, 92.972, 96.810]xychart-beta
title "Minimum Latency"
x-axis "Concurrent Sessions (G=Go and Q=Quarkus)" [50G, 50Q, 100G, 100Q, 300G, 300Q]
y-axis "Latency in ms"
bar [0.417, 0.528, 0.94, 1, 3.22, 3.1]Here are the output of each Go/Gin benchmark sessions for reference:
Benchmarks for Go/Gin with 50 concurrent sessions
./bombardier-darwin-arm64 -c 50 -n 10000000 http://localhost:3000
Bombarding http://localhost:3000 with 10000000 request(s) using 50 connection(s)
10000000 / 10000000 [========================================] 100.00% 118974/s 1m24s
Done!
Statistics Avg Stdev Max
Reqs/sec 119217.54 12349.05 142939.36
Latency 417.95us 235.34us 30.12ms
HTTP codes:
1xx - 0, 2xx - 10000000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 21.72MB/s
Benchmarks for Go/Gin with 100 concurrent sessions
./bombardier-darwin-arm64 -c 100 -n 10000000 http://localhost:3000
Bombarding http://localhost:3000 with 10000000 request(s) using 100 connection(s)
10000000 / 10000000 [========================================] 100.00% 105666/s 1m34s
Done!
Statistics Avg Stdev Max
Reqs/sec 105705.98 16564.25 145158.63
Latency 0.94ms 502.03us 115.73ms
HTTP codes:
1xx - 0, 2xx - 10000000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 19.26MB/s
Benchmarks for Go/Gin with 300 concurrent sessions
./bombardier-darwin-arm64 -c 300 -n 10000000 http://localhost:3000
Bombarding http://localhost:3000 with 10000000 request(s) using 300 connection(s)
10000000 / 10000000 [=========================================] 100.00% 92972/s 1m47s
Done!
Statistics Avg Stdev Max
Reqs/sec 92812.94 9474.50 152756.79
Latency 3.22ms 0.96ms 69.58ms
HTTP codes:
1xx - 0, 2xx - 10000000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 16.94MB/s
Here are the output of each Quarkus benchmark sessions for reference:
Benchmarks for Quarkus with 50 concurrent sessions
./bombardier-darwin-arm64 -c 50 -n 10000000 http://localhost:8080
Bombarding http://localhost:8080 with 10000000 request(s) using 50 connection(s)
10000000 / 10000000 [=========================================] 100.00% 94220/s 1m46s
Done!
Statistics Avg Stdev Max
Reqs/sec 94411.32 15183.30 111797.34
Latency 528.69us 408.60us 118.56ms
HTTP codes:
1xx - 0, 2xx - 10000000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 13.77MB/s
Benchmarks for Quarkus with 100 concurrent sessions
./bombardier-darwin-arm64 -c 100 -n 10000000 http://localhost:8080
Bombarding http://localhost:8080 with 10000000 request(s) using 100 connection(s)
10000000 / 10000000 [=========================================] 100.00% 99732/s 1m40s
Done!
Statistics Avg Stdev Max
Reqs/sec 99857.30 7043.92 112384.92
Latency 1.00ms 139.38us 23.76ms
HTTP codes:
1xx - 0, 2xx - 10000000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 14.57MB/s
Benchmarks for Quarkus with 300 concurrent sessions
./bombardier-darwin-arm64 -c 300 -n 10000000 http://localhost:8080
Bombarding http://localhost:8080 with 10000000 request(s) using 300 connection(s)
10000000 / 10000000 [=========================================] 100.00% 96810/s 1m43s
Done!
Statistics Avg Stdev Max
Reqs/sec 96891.15 12888.43 114944.16
Latency 3.10ms 0.97ms 113.66ms
HTTP codes:
1xx - 0, 2xx - 10000000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 14.13MB/s
conclusion
Putting aside the code, even with identical parameters, the benchmark's results are significantly influenced by the uniqueness of each environment. The only constant is the methodology to perform these benchmarks.
And yet, the above is still missing crucial details:
- Did the web service run with the language interpreter or a binary?
- Is running on a laptop representing a production-grade environment?
- How would running the web service and benchmark on different machines impact the result?
- How would running the web service running natively or as a container impact the results?
- Is the code representing a minimal viable product?