Custom Objectives
An SLO is rarely all things to all people all the time. The reality is that some systems are complex and bespoke, and only masters of that domain are able to define meaningful objectives.
A Custom Objective is one that allows you to define the source of its metrics using HTTP Requests. You are able to dictate where and what information is used to determine the state of your objective. Any metric aggregation system with an HTTP API is supported.
The reliably populate ...
command supports only a finite number of objectives per provider, however, with a Custom Objective you are able to define your own metric providers.
Prerequisites
- The
reliably
CLI installed and authenticated against Reliably
Populate Reliably using Custom
Unlike other reliably populate
commands, custom relies heavily on the user to define the objective and metric source.
Running reliably populate custom
trigger an interactive process, requesting the various values required for creating your Objective. Below we’ll describe each value that is required and its purpose.
Objective Structure
When defining a Custom Objective there are 5 fields under the objective.spec.indicatorSelector
to consider:
- category
- metric_url
- metric_jsonpath
- metric_url_query
- metric_http_headers
category [required]
The category field must be set to
custom
, this tells the
reliably agent
that this Objective must be evaluated as a
Custom Objectives
.
metric_url [required]
This field represents the URL endpoint used to retrieve metric data. The field expects a URL and path without any queries. (there is a separate field for that).
Eg.
indicatorSelector:
category: custom
metric_url: https://my.metrics.com/path/to/api
metric_url_query [optional]
This field allows us to define URL queries/parameters. When defining this field, each key/value pair must be separated by a newline or newline character.
Eg.
Without url encoding template function:
indicatorSelector:
metric_url_query: |
window.start=16000213123123
window.end=16000213123121
limit=100
With url encoding template function:
indicatorSelector:
metric_url_query: |
window.start=16000213123123
window.end=16000213123121
limit=100
query={{ urlencode `clamp(sum(kube_service_info, vector(0), 0, 1)*100` }}
If pairs are not separated by newlines, this may result in some pairs not being parsed correctly by the reliably agent
.
URL Queries are automatically encoded when making HTTP Requests, however, this functionality can be overriden using the {{ urlencode }}
template function. When the template function is used, automatic encoding is disabled so only the parts of the query that {{ urlencode }}
is applied to will be encoded.
With the 1st example above, the entire url query will be encoded when the HTTP Request is generated, however, with the 2nd example, only the string applied to {{ urlencode }}
will be encoded, i.e clamp(sum(kube_service_info, vector(0), 0, 1)*100
See more on Template Functions Function) below.
metric_http_headers [optional]
This field allows for any HTTP Headers required for the HTTP Request to be defined. Similar to the query field, each key/value pair must be separated by a newline or newline character.
Eg.
indicatorSelector:
metric_http_headers: |
Accept: application/json
Authorization: Bearer {{ env `TOKEN` }}
Note that template functions can also be applied to this field.
metric_jsonpath [required]
This field allows a JSONPath to be defined which tells the reliably agent
where to find your metrics in the JSON HTTP Response once HTTP call has been made successfully.
Eg.
indicatorSelector:
metric_jsonpath: $.path.to.values
More on jsonpaths below.
Template Functions
At times we may need to dynamically retrieve or utilise values defined only within our environment or the objective itself when making metric http requests via the reliably agent
. This can be accomplished using Templating. The reliably agent
uses Go Templating to allow values to be evaluatied at runtime.
The reliably agent
supports all functionality of the Go Templating library as well as a few custom functions.
function | desc | eg. |
---|---|---|
urlencode | This function applies URL Encoding to any given string | {{ urlencode `value to encode` }} {{ `value to encode` | urlencode }} |
env | Allows Environment variables to to be evaluated at runtime | {{ env `SOME_ENV_VAR` }} {{ `SOME_ENV_VAR` | urlencode }} |
from to | These functions return the start (from) and end (to) timestamp of the objective window. For example, if the Objective window is 5m, then from would return now-5m and to would return now where now is a timestamp of the current time. The timestamps returned are in RFC3339 time format. Note that these functions take no arguments | {{ from }} {{ to }} |
You are also able to access metadata.labels
defined for an objective by using {{ .label_name }}
. For example, given:
apiVersion: reliably.com/v1
kind: Objective
metadata:
labels:
env: staging
name: gcp_cloudrun_log_size_in_bytes
provider: custom
project: my_gcp_project
You would be able to generate a url dynamically using some of the values defined in labels:
indicatorSelector:
metric_url: https://monitoring.googleapis.com/{{ .env }}/projects/{{ .project }}/timeSeries
The above would evaluate to:
indicatorSelector:
metric_url: https://monitoring.googleapis.com/staging/projects/my_gcp_project/timeSeries
Note that template values like labels are retrieved using a .
infront of the label name, however, template functions are called directly, without the .
For eg. {{ .env }}
represents the value at metadata.labels.env
while {{ env "TOKEN" }}
is a call to the env
template function to read the value of TOKEN
from the environment.
At runtime, template functions are applied to the following fields, allowing them to be generated dynamically:
- metric_url
- metric_url_query
- metric_http_headers
JSONPaths
While JSONPath is only utilised for a single field, getting it right, is important to insuring the reliably agent
will calculate the correct values from the response JSON.
The agent supports the JSONPaths standard completely along with additional custom JSON functions to help with aggregation.
function | desc | eg. |
---|---|---|
agg_avg | returns an average of the value(s) evaluated from the JSONPath | agg_avg($.path.to.value) |
agg_max | returns the maximum of the value(s) evaluated from the JSONPath | agg_max($.path.to.value) |
agg_min | returns the minimum of the value(s) evaluated from the JSONPath | agg_min($.path.to.value) |
agg_sum | returns the sum of the value(s) evaluated from the JSONPath | agg_sum($.path.to.value) |
Note: all agg functions can evaluate an array of strings or numbers, if strings are received, they are converted to floats if possible. If this cannot be done, the evaluation will fail.
For accuracy we recommend validating your JSONPaths using tools such as jsonpath.com
Example
Below we have a complex example utilising all the functionality described above. The Objective describes an SLO whos metric data source is Google Cloud metrics.
apiVersion: reliably.com/v1
kind: Objective
metadata:
labels:
env: staging
name: gcp_cloudrun_log_size_in_bytes
provider: custom
project: my_gcp_project
spec:
objectivePercent: 95.0
window: 24h
indicatorSelector:
# category must be set to custom
category: custom
# gcp requires authentication via a Bearer Token,
# here we read the TOKEN from the environment at runtime
# reliably itself, has no idea what this value is, and does not
# store it
metric_http_headers: |
Accept: application/json
Authorization: Bearer {{ `TOKEN` | env }}
# JSONPath using the agg_max custom JSON path function
metric_jsonpath: agg_max($.timeSeries..int64Value)
# the metric_url field can only use dynamic values, here
# the url is using the .project value defined in labels
metric_url: https://monitoring.googleapis.com/v3/projects/{{ .project }}/timeSeries
# define the metric_url_query, note that each pair
# is separated by a newline
# Note here that in the output of `from` and `to` are used as
# parameters for `urlencode`, essentially encoding the timestamps
metric_url_query: |
aggregation.alignmentPeriod=86400s
aggregation.crossSeriesReducer=REDUCE_COUNT
aggregation.groupByFields=resource.label.service_name
aggregation.perSeriesAligner=ALIGN_RATE
interval.endTime={{ to | urlencode }}
interval.startTime={{ from | urlencode }}
filter={{ `metric.type="logging.googleapis.com/log_entry_count" resource.type="cloud_run_revision" resource.label.service_name="entity-server-staging-europe-west1"` | urlencode }}
When the reliably agent
receives this custom objective it:
-
evaluates the HTTP Headers using the metric_http_headers field. Using the
env
template function, the agent reads theTOKEN
environment variable -
evaluates the metric_url_query field calling the
urlencode
,from
andto
template fucntions. Since theurlencode
function is used here, then only those parts of the query will be encoded. -
builds and executes an HTTP Request by combining the metric_url, metric_http_headers & the metric_url_query
-
evaluates the response using the metric_jsonpath to identify and potentially aggregate the metric value.
-
finally, an SLO Indicator is created from the evaluated value and sent to Reliably.