Jenkins has been a core component of DevOps toolchain for over a decade, while the status of pipelines hosted on Jenkins servers is one key thing to monitor each and every day. Several status views are provided as part of the system to show the healthiness of the jobs, but it is far from enough to provide more insights into the data.
It is common to see pipelines fail, so the goal is never to ensure a 100% success or to reach to a certain number of nines. Instead, people would focus on detecting and fixing the failures at earliest, and monitors should be set against when the pipelines fail and recover.
As part of the homebrew DevOps solution/platform, we build a simple data scraping service since day 1 with a modularized design to collect all sorts of transaction data from the toolchains. The diagram below shows the high-level design.
There are push sources actively reporting data to the service, and data sources that do not support events or webhooks are scraped in configured interval. Raw data is always persisted, so that whenever certain data fields are added/removed to/from the models used in analytics, we are capable to rebuild ingested data from the raw documents, which really ensures the flexibility in respond to ever-changing business requirements.
Choosing the database
Choosing the database for persisting raw data is easy. MongoDB was selected for the convenience of persisting JSON payloads without any special treatment, thanks to the adoptions of JSON-based RESTful HTTP APIs. However we did experience a little bit struggle while deciding the database for analytics.
In the very first version of the service implementation, we chose TiDB as the backend database as it is said to support both batch data processing and streaming well. After a few months, we realized this was not the answer due to:
- Operational cost. A production deployment TiDB is huge and requires quite a lot of computing resources.
- Considering the amount of our data which is under 500k rows (from all the tools), TiDB is kind of overkill.
Then in phase 2 we migrated to Elasticsearch + Kibana. Frankly speaking, I could not recall why the decision was made (maybe Kibana is simply too good-looking at first glance), but it turned out Elasticsearch was not designed for processing non-log data, especially nested JSON data:
- Each nested JSON value would be treated as a new index. For instance, considering the following JSON data
{
"parameters": {
"version": "1.2.3",
"build": "1234"
}
}2 indexes are created, parameters.version.1.2.3 and parameters.build.1234 . Then, if a new payload with version equals 1.2.4 is sent to Elastic, parameters.version.1.2.4 will be created. Along with more different parameter values are persisted, the total number of indexes of a certain collection in Elastic would easy go over the default 1000 limit. This can be easily fixed by enlarging the limit, but it is not sustainable, obviously.
- Aggregations over nested JSON data elements are not doable, only the first element would be honored. Considering the following data
{
"stages": [
{
"name": "stage 1",
"duration": 1,
},
{
"name": "stage 2",
"duration": 10,
}
]
}Running an average aggregation over this data would produce 1 , a sum aggregation would produce 2 , this is certainly not acceptable. Meanwhile, the limitations with Kibana visualization made me keep explaining to the dashboard viewers that a certain visual element is not supported. Finally we dropped this ECK stack after 3 months of PoC.
And now the ingested data is persisted into PostgreSQL which perfectly matches:
- Affordable operational cost. Actually we see higher query performance with PostgreSQL with less computing resources consumed than ECK.
- Its native support on JSON is a huge helper to data aggregation when advanced/complex queries are demanded.
The good thing is, the migration of the analytics database is almost painless, by implementing the data access interface that talks to PostgreSQL and updating the service configuration to point to this new implementation.
Extending the data scraping service
Extending the service to handle Jenkins data analytics is no more than implementing a new module to accept Jenkins payloads and translate them into new models.
This is the core data model to support both plain and multibranch pipelines:
type ingestedJenkinsPipeline struct {
ID string `bson:"_id" gorm:"column:id"`
Status string `gorm:"column:status"`
Created time.Time `gorm:"column:created;type:time"`
Started time.Time `gorm:"column:started;type:time"`
Updated time.Time `gorm:"column:updated;type:time"`
Finished time.Time `gorm:"column:finished;type:time"`
Lifespan time.Duration `gorm:"column:lifespan;type:bigint"`
Name string `gorm:"column:name"`
Number int `gorm:"column:number;type:bigint"`
Event string `gorm:"column:event"`
Trigger string `gorm:"column:trigger"` // pr / push
Complexity int `gorm:"column:complexity;type:bigint"` // number of stages
ConcurrencyRate float32 `gorm:"column:concurrency_rate"`
QueuedDuration time.Duration `gorm:"column:queued_duration;type:bigint"`
Type string `gorm:"column:type"` // jenkins / drone
Mode string `gorm:"column:mode"` // for Jenkins, could be multibranch_pipeline, pipeline, dsl, etc.
HTMLURL string `gorm:"column:htmlurl"` // URL of the pipeline
RequestorID string `gorm:"column:requestor_id"`
RequestorName string `gorm:"column:requestor_name"`
Stages IngestedPipelineStages `gorm:"column:stages;type:jsonb"`
Parameters StringMap `gorm:"column:parameters;type:jsonb"`
ChangeNumber int `gorm:"column:change_number;type:bigint"`
ChangeBaseRef string `gorm:"column:change_base_ref"`
ChangeOwner string `gorm:"column:change_owner"`
ChangeRepo string `gorm:"column:change_repo"`
ChangeHTMLURL string `gorm:"column:change_htmlurl"`
}There are 2 customized data model, the first ingestedPipelineStage is to hold the pipeline stage data. Since gorm.io does not natively support the encoding/decoding of slices, we further wrap the slice into a new type and implement Scan() and Value() according to the official documentation.
type IngestedPipelineStages []IngestedPipelineStage
func (d *IngestedPipelineStages) Scan(value interface{}) error {
b, ok := value.([]byte)
if !ok {
return errors.New("type assertion to []byte failed")
}
return json.Unmarshal(b, &d)
}
func (d IngestedPipelineStages) Value() (driver.Value, error) {
return json.Marshal(d)
}
type IngestedPipelineStage struct {
Name string `json:"Name"`
Status string `json:"Status"`
Created time.Time `json:"Created"`
QueuedDuration time.Duration `json:"QueuedDuration"`
ExecutionDuration time.Duration `json:"ExecutionDuration"`
}
Same technique for map[string]string :
type StringMap map[string]string
func (d *StringMap) Scan(value interface{}) error {
b, ok := value.([]byte)
if !ok {
return errors.New("type assertion to []byte failed")
}
return json.Unmarshal(b, &d)
}
func (d StringMap) Value() (driver.Value, error) {
return json.Marshal(d)
}
Then, ingesting functions are implemented to translate the original Jenkins job payload into the new model and persist into the analytics database.
Implement a new method in Jenkins shared library
The snippet below shows the core code piece to build and send the payload to the webhook exposed by the data service. One thing worth mentioning here is that we also implemented our own methods to collect every piece of data of the jobs from the original Jenkins job and workflow data models and APIs, since this is complicated and we don’t want the data service to handle it. The Pipelines.constructBuildData() is where the heavy lift happens, that I will cover it in future posts.
import Pipelines
import groovy.json.*
import groovyx.net.http.HTTPBuilder
import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*
@Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7.1')
def call() {
script {
// parse the env vars here so to overcome the cps-mismatch issue
// while calling the "sh" step inside a class
def envs = sh(script: 'printenv', returnStdout: true)
if (envs == null || envs == "") {
println "Failed to get the environment variables."
return
}
def envVars = [:]
envs.split("\n").each { line ->
def splits = line.split("=")
// To avoid the exception when the line is empty
if (splits != null && splits.length == 2) {
envVars[splits[0]] = splits[1]
}
}
def data = Pipelines.constructBuildData(this, envVars)
httpRequest httpMode: 'POST',
requestBody: JsonOutput.toJson(data),
responseHandle: 'NONE',
url: 'http://<data service url>/hooks/jenkins',
wrapAsMultipart: false,
timeout: 10
}
}
Visualization in Grafana
This is my favorite part, seeing data diagrams is always of huge fun. After connecting to the PostgreSQL instance all you need to do is writing SQLs. One thing to mention is that please DO setup a read replica of the database for Grafana to connect, for the sake of query performance.
Finally it is the time to go back to our goal —— setting up timeline views for pipelines. Let’s get down to it.
The “State timeline” view panel is designed for this (available in recent Grafana releases). Write a SQL query as below and format as “Time Series”:
SELECT $__time(started), name,
CASE
WHEN status = 'success' THEN 0
WHEN status = 'failure' THEN 1
WHEN status = 'aborted' THEN 2
END AS " "
FROM ingested_jenkins_pipelines
WHERE mode = 'pipeline'
AND $__timeFilter(started)
ORDER BY started
Then create value mappings for the panel:
The final result:
With the same solution we build timeline views for integration test cases, which is quite similar to Kubernetes testgrid which shows failing and flaky tests, but with zero frontend development effort.
SELECT $__time(execution_timestamp), d->>'Name',
CASE
WHEN d->>'Status' = 'pass' then 1
WHEN d->>'Status' = 'fail' then 0
END AS " "
FROM ingested_integration_test_reports t, jsonb_array_elements(test_case_details) d
WHERE $__timeFilter(execution_timestamp)
ORDER BY execution_timestamp
Summary
The Jenkins pipeline data has been collected for around 2 years but we have not been using it until several months ago when the build pipelines went quite unstable due to various outages or errors of dependent facilities, and the engineering teams started to complain. This drove us to build better views which deliver the messages we concern, like the status changes, the end-to-end durations, slowest / most unstable stages and so on. The timeline views not only display the stabilities of the pipelines but also how quickly the team can respond to the failures and recover everything. If an error is fixed before being percepted, then we are all GOOD.
Hope this post could provide some simple ideas. Thank you for reading.