Computing Results and Scores

Overview

The purpose of this document is to illustrate the process behind how an objective definition alongside indicators results in objective results, objective scores, and score cards. There will also be an overview of the calculations involved in this process. The example given focuses on what the flow would look like when using the pre-built data provider integrations alongside Reliably’s CLI and Web UI.

Entities and Relevant Properties

Below are the entities that are relevant for this flow and their relevant properties for each:

  • Objective: this entity allows us to define the target value for a given indicator of a data provider. Data providers are usually a cloud platform that provides observability data about APIs and other services hosted, but they can take other forms.

    • Indicator Selector: a selector defined within an objective that uses a particular set of labels to find an indicator. Selector functionality here allows for individual indicators to potentially be targeted by multiple objectives, although this is not a typical scenario. Typically, there is one indicator for each objective.
    • Target: a float64 value, which is essentially the objective for a given indicator.
  • Indicator: an indicator is a high-level, windowed metric. It’s essentially a single float64 value, accompanied by window information (start and end time). The float64 value intends to represent a percentage of good outcomes / total outcomes, with the remaining outcomes conceptually understood to be errors. When using our CLI agent, depending on the data provider you specify (e.g. AWS, GCP, Dynatrace), an indicator will retrieve information from a data provider, pre-process if necessary, and then store an indicator for the time frame. Once this indicator is PUT to Reliably’s API, the granular data used to calculate the indicator is no longer available. It becomes a single number representing a window in time essentially.

    • Metadata.Labels: the labels of an indicator allow for objectives to select that indicator as the metric they will compare against their target value
    • Percent: this is the value of the indicator, expressed as a percentage, that is usually calculated as (good outcomes / total outcomes) * 100 by the CLI agent or a custom indicator’s processor.
    • From/To: these values represent the start and end time of the indicator, otherwise known as the window.
  • Objective Result: this is an entity that is created automatically every time an indicator is PUT. When an indicator is received, matching objectives are found, and an objective result is created for each objective that matches. The indicator’s value, the objective target’s value, and the delta are recorded. This allows for the recording of how an indicator fared against an objective at that moment in time. So, if an objective’s target changes, the previous objective results serve as an archive for the results based on the previous objective target.

    • Objective Percent: the objective’s target value.
    • Actual Percent: the indicator’s actual value.
    • Remaining Percent: the difference between the objective and actual values. This is analogous to a service’s error budget.
  • Objective Score: this entity is created after a new objective result is created. It uses objective results of the last 60 days to compute a few metrics as seen below.

    • Historical Reliability Score: a measure of an objective’s health over a longer period that takes into account all relevant objective results within the last 60 days. Each objective result within this timeframe is assigned a relevance (based on its age) which is then multiplied by its result to obtain a Weighted Result. These Weighted Results are then averaged to produce the Historical Reliability Score.
    A graph showing the 'freshness' function depending on the age in days of the result. At 0 days, the multiplier is 1. At 30 days, the multiplier is 0.625. At 60 days, the multiplier is 0.2. Results older than 60 days are not used.
    • Target Delta Percentage: a percentage difference between the latest result and the target (aka Objective Percent) and is calculated as such: Latest Result / Target * 100. This value is then capped at 100 - for the case where the latest result exceeds the target.
    • Reliability Score: an average of the Target Delta Percentage and the Historical Reliability. It values an objective’s history as much as its current state to provide insight into an objective’s long-standing health.
  • Score Card: an entity for grouping objectives into one score, allowing a higher-level abstraction. This allows teams to group objectives into single topics that encompass a particular interest area. A score card is generated whenever an objective score entity is PUT, or when a new score card is PUT for the first time. A single score is calculated for a score card called a ReliabilityScore.

    • Reliability Score: the average of all Objective Scores (the most recent for each objective), for all the objectives selected by the score card.

    • Objective Selector: A selector denoting all the objectives that a score card will encompass.

Entity Creation Flow & Calculations

The sequence of events and entity creations of a typical scenario from objective definition to score card is as follows:

  1. Create Objective
  2. Create a Score Card with the objective as the target
  3. Push an Indicator
  4. That Indicator triggers an Objective Result new entity creation (as long as there is a matching Objective)
  5. Objective Result creation triggers an Objective Score calculation and new entity creation
  6. The Objective Score creation triggers a Score Card calculation (as long as there is a Score Card that targets this Objective), and PUTs the Score Card entity, replacing any previous one with the same label