Custom Objectives

An SLO is rarely all things to all people all the time. The reality is that some systems are complex and bespoke, and only masters of that domain are able to define meaningful objectives.

A Custom Objective is one that allows you to define the source of its metrics using HTTP Requests. You are able to dictate where and what information is used to determine the state of your objective. Any metric aggregation system with an HTTP API is supported.

The reliably populate ... command supports only a finite number of objectives per provider, however, with a Custom Objective you are able to define your own metric providers.

Prerequisites

1. The reliably CLI installed and authenticated against Reliably

Populate Reliably using Custom

Unlike other reliably populate commands, custom relies heavily on the user to define the objective and metric source.

Running reliably populate custom trigger an interactive process, requesting the various values required for creating your Objective. Below we’ll describe each value that is required and its purpose.

Objective Structure

When defining a Custom Objective there are 5 fields under the objective.spec.indicatorSelector to consider:

• category
• metric_url
• metric_jsonpath
• metric_url_query
• metric_http_headers

category [required]

metric_url [required]

This field represents the URL endpoint used to retrieve metric data. The field expects a URL and path without any queries. (there is a separate field for that).

Eg.

indicatorSelector:
  category: custom
  metric_url: https://my.metrics.com/path/to/api

metric_url_query [optional]

This field allows us to define URL queries/parameters. When defining this field, each key/value pair must be separated by a newline or newline character.

Eg.

Without url encoding template function:

indicatorSelector:
  metric_url_query: |
    window.start=16000213123123
    window.end=16000213123121
    limit=100

With url encoding template function:

indicatorSelector:
  metric_url_query: |
    window.start=16000213123123
    window.end=16000213123121
    limit=100
    query={{ urlencode `clamp(sum(kube_service_info, vector(0), 0, 1)*100` }}

If pairs are not separated by newlines, this may result in some pairs not being parsed correctly by the reliably agent.

URL Queries are automatically encoded when making HTTP Requests, however, this functionality can be overriden using the {{ urlencode }} template function. When the template function is used, automatic encoding is disabled so only the parts of the query that {{ urlencode }} is applied to will be encoded.

With the 1st example above, the entire url query will be encoded when the HTTP Request is generated, however, with the 2nd example, only the string applied to {{ urlencode }} will be encoded, i.e clamp(sum(kube_service_info, vector(0), 0, 1)*100

See more on Template Functions Function) below.

metric_http_headers [optional]

This field allows for any HTTP Headers required for the HTTP Request to be defined. Similar to the query field, each key/value pair must be separated by a newline or newline character.

Eg.

indicatorSelector:
  metric_http_headers: |
    Accept: application/json
    Authorization: Bearer {{ env `TOKEN` }}

Note that template functions can also be applied to this field.

metric_jsonpath [required]

This field allows a JSONPath to be defined which tells the reliably agent where to find your metrics in the JSON HTTP Response once HTTP call has been made successfully.

Eg.

indicatorSelector:
  metric_jsonpath: $.path.to.values

More on jsonpaths below.

Template Functions

At times we may need to dynamically retrieve or utilise values defined only within our environment or the objective itself when making metric http requests via the reliably agent. This can be accomplished using Templating. The reliably agent uses Go Templating to allow values to be evaluatied at runtime.

The reliably agent supports all functionality of the Go Templating library as well as a few custom functions.

You are also able to access metadata.labels defined for an objective by using {{ .label_name }}. For example, given:

apiVersion: reliably.com/v1
kind: Objective
metadata:
  labels:
    env: staging
    name: gcp_cloudrun_log_size_in_bytes
    provider: custom
    project: my_gcp_project

You would be able to generate a url dynamically using some of the values defined in labels:

indicatorSelector:
  metric_url: https://monitoring.googleapis.com/{{ .env }}/projects/{{ .project }}/timeSeries

The above would evaluate to:

indicatorSelector:
  metric_url: https://monitoring.googleapis.com/staging/projects/my_gcp_project/timeSeries

Note that template values like labels are retrieved using a . infront of the label name, however, template functions are called directly, without the . For eg. {{ .env }} represents the value at metadata.labels.env while {{ env "TOKEN" }} is a call to the env template function to read the value of TOKEN from the environment.

At runtime, template functions are applied to the following fields, allowing them to be generated dynamically:

• metric_url
• metric_url_query
• metric_http_headers

JSONPaths

While JSONPath is only utilised for a single field, getting it right, is important to insuring the reliably agent will calculate the correct values from the response JSON.

The agent supports the JSONPaths standard completely along with additional custom JSON functions to help with aggregation.

Note: all agg functions can evaluate an array of strings or numbers, if strings are received, they are converted to floats if possible. If this cannot be done, the evaluation will fail.

For accuracy we recommend validating your JSONPaths using tools such as jsonpath.com

Example

Below we have a complex example utilising all the functionality described above. The Objective describes an SLO whos metric data source is Google Cloud metrics.

apiVersion: reliably.com/v1
kind: Objective
metadata:
  labels:
    env: staging
    name: gcp_cloudrun_log_size_in_bytes
    provider: custom
    project: my_gcp_project
spec:
  objectivePercent: 95.0
  window: 24h
  indicatorSelector:

    # category must be set to custom
    category: custom

    # gcp requires authentication via a Bearer Token,
    # here we read the TOKEN from the environment at runtime
    # reliably itself, has no idea what this value is, and does not
    # store it
    metric_http_headers: | 
      Accept: application/json
      Authorization: Bearer {{ `TOKEN` | env }}

    # JSONPath using the agg_max custom JSON path function
    metric_jsonpath: agg_max($.timeSeries..int64Value)

    # the metric_url field can only use dynamic values, here
    # the url is using the .project value defined in labels
    metric_url: https://monitoring.googleapis.com/v3/projects/{{ .project }}/timeSeries

    # define the metric_url_query, note that each pair
    # is separated by a newline
    # Note here that in the output of  `from` and `to` are used as 
    # parameters for `urlencode`, essentially encoding the timestamps
    metric_url_query: |
      aggregation.alignmentPeriod=86400s
      aggregation.crossSeriesReducer=REDUCE_COUNT
      aggregation.groupByFields=resource.label.service_name
      aggregation.perSeriesAligner=ALIGN_RATE
      interval.endTime={{ to | urlencode }}
      interval.startTime={{ from | urlencode }}
      filter={{ `metric.type="logging.googleapis.com/log_entry_count" resource.type="cloud_run_revision" resource.label.service_name="entity-server-staging-europe-west1"` | urlencode }}

When the reliably agent receives this custom objective it:

1. evaluates the HTTP Headers using the metric_http_headers field. Using the env template function, the agent reads the TOKEN environment variable

2. evaluates the metric_url_query field calling the urlencode, from and to template fucntions. Since the urlencode function is used here, then only those parts of the query will be encoded.

3. builds and executes an HTTP Request by combining the metric_url, metric_http_headers & the metric_url_query

4. evaluates the response using the metric_jsonpath to identify and potentially aggregate the metric value.

5. finally, an SLO Indicator is created from the evaluated value and sent to Reliably.