Rules
Notifications
As we mentioned earlier, assertions are not alerts. If you want to get notified, you can add notification rules. Asserts uses Prometheus Alert Manager for notification internally, so we support all the receivers listed on its official page (Email, Pagerduty, Slack, etc.)
Notifications are organized into different sections, just like the thresholds, and the same hierarchical granularity levels apply. You can set notifications for an assertion whenever it fires or when it fires on a specific job
or a job
and request type
. Free-form label conditions are also supported here.
There is also a quick shortcut from the Workbench to bring you to the notification settings with all the labels pre-populated.
Suppressions
Similar to notifications, you can also choose a subset of assertions to suppress, so they don’t fire. Suppressed assertions are ignored in Asserts processing. They will not be available in Top Insights, Graph, or the Workbench. Note this is very different from an assertion that fires but does not notify.
Bring your own
If you have your own rules, you can also bring them into the Asserts system by defining custom assertion rules. You can add them individually on the UI or upload a rule file.
Relabelling
Relabelling rules are used for manipulating labels in the metrics ingestion. They can also be used to rename or drop metrics.
Adding new label
New label can be added with the following relabeling rule:
- target_label: 'foo'
replacement: 'bar'
This relabeling rule adds {foo=“bar”} label to all the incoming metrics. For example, metric{job=“aa”} will be converted to metric{job=“aa”,foo=“bar”}.
Updating existing label
Existing label can be updated with the relabeling rule mentioned above:
- target_label: 'foo'
replacement: 'bar'
This rule rewrites metric{foo=“aaaa”} with metric{foo=“bar”}.
Rewriting existing label
The following relabeling rule can be used for removing port part from instance label:
- source_labels: [instance]
regex: '([^:]+):.+'
target_label: 'instance'
replacement: '$1'
This rule translates foo{instance=“bar:123”} to foo{instance=“bar”}.
How does it work: it extracts instance label value (see source_labels list above), applies the given regex to it, then generates replacement string ($1 is substituted by the part of instance label that is matched by regex part in the first parenthesis), and then puts the replacement string into target_label.
Updating metric name
Metric name can be updated with the following relabeling rule:
- source_labels: [__name__]
regex: '(.+)_suffix'
target_label: '__name__'
replacement: 'prefix_$1'
This rule removes _suffix from metric name and adds prefix_ to it. For example, foo_suffix{bar=“aaa”} metric would be substituted with prefix_foo{bar=“aaa”} with this relabeling rule.
VictoriaMetrics provides additional action — replace_all, which can be used for replacing all the occurrences of the given pattern with something else. For example, the following relabeling rule substitutes all the . chars with _ chars in metric names:
- action: replace_all
source_labels: [__name__]
target_label: '__name__'
regex: "\\."
replacement: '_'
Graphite-like metric foo.bar.baz would be substituted with foo_bar_baz after applying this relabeling rule.
Removing unneeded labels
Sometimes it is needed to remove certain labels before storing metrics in Prometheus or VictoriaMetrics. This can be done with action: labeldrop. The name of the label to drop must be specified as a regular expression in regex . For example, the following relabeling rule drops all the label names starting with foo:
- action: labeldrop
regex: 'foo.+'
This relabeling rule transforms the following metric: metric{job=“a”,instance=“xyz”,foobar=“baz”,foox=“aaa”} 123 into: metric{job=“a”,instance=“xyz”} 123
Sometimes it is needed to drop all the labels except of a few label names matching the given regexp. This can be done with action: labelkeep. The following relabeling rule drops all the labels except of **name** and labels ending with keepme:
- action: labelkeep
regex: '__name__|.*keepme'
This means that metric{job=“aa”,foo=“bar”,letskeepme=“aaa”} would be translated to metric{letskeepme=“aaa”} with the given relabeling rule.
Removing the specific label value
The following rule removes foo=“bar” label value:
- source_labels: [foo]
regex: 'bar'
target_label: foo
replacement: ''
For example, metric{foo=“bar”,baz=“x”} becomes metric{baz=“x”} after applying this relabeling rule, while metric{foo=“xxx”} remains the same.
Removing unneeded metrics
Metrics can be dropped with action: drop. For example, the following relabeling rule drops metric if it contains instance label starting from foobar:
- action: drop
source_labels: [instance]
regex: 'foobar.+'
This relabeling rule drops the following metrics:
foo{instance=“foobar1”}
foo{instance=“foobar2”,job=“xxx”,aaa=“bb”}
It doesn’t drop the following metrics:
foo{instance=“xxx”}
foo{instance=“abc”,job=“xyz”}
Sometimes it is easier to specify metrics that needs to be preserved instead of metrics that needs to be dropped. In this case action: keep must be used:
- action: keep
source_labels: [job]
regex: 'foobar'
This relabeling rule preserves metrics with job label equal to foobar, while other metrics will be dropped.
How to drop metrics if they contain certain values for multiple labels? Just enumerate these labels in source_labels and then specify the desired regex. Then source_labels contains multiple labels, they are concatenated with ; char before matching the provided regex. For example, the following relabeling rule would drop metric with {job=“foo”,instance=“bar”} labels:
- action: drop
source_labels: [job, instance]
regex: 'foo;bar'
Dropping metrics on certain condition
Sometimes it is necessary to drop a metric if it contains two labels with identical values. This can be done with drop_if_equal action, which is supported by VictoriaMetrics and vmagent. For example, the following relabeling rule would drop metric if it contains identical label values for real_port and needed_port:
- action: drop_if_equal
source_labels: [real_port, needed_port]
The rule would drop the following metric: foo{real_port=“123”,needed_port=“123”}
but would keep the following metric: foo{real_port=“123”,needed_port=“456”}
VictoriaMetrics also provides* keep_if_equal* action, which drops metric if its source_labels aren’t equal. This may be useful for filtering out superfluous scrape targets detected by kubernetes_sd_config — see these docs for details.
Modifying label names
Sometimes it is necessary to modify label names. Then action: labelmap can be used for this. For example, the following relabeling rule substitutes foo_ prefix in all the label names with bar_ prefix:
- action: labelmap
regex: 'foo_(.+)'
replacement: 'bar_$1'
Note that the rule leaves the original label untouched in the metric. I.e. the following metric — aa{foo_xx=“bb”,job=“qq”} — is translated to aa{foo_xx=“bb”,bar_xx=“bb”,job=“qq”} by this relabeling rule.
VictoriaMetrics supports additional action — labelmap_all — which allows substituting specified patterns in label names. For example, the following relabeling rule would substitute all the — chars with _ chars in all the label names:
- action: labelmap_all
regex: '-'
replacement: '_'
This rule translates foo{a-b=“x”,qwe-x-zz=“aa”} with foo{a_b=“x”,qwe_x_zz=“aa”}.
Constructing a label from multiple existing labels
Suppose a metric has host and port labels, while you need constructing address label with host:port contents. This can be done with the following relabeling rule:
- source_labels: [host, port]
separator: ':'
target_label: 'address'
This relabeling rule joins host and port label values with : separator and stores the result at target_label.
Sometimes it is necessary to construct a label from parts of existing labels. For example, the url label with http://hostname/path value must be constructed from address and path labels, where address contains hostname:port. This can be done with the following relabeling rule:
- source_labels: [address, path]
regex: '([^:]+):[^;]+;(.*)'
replacement: 'http://$1/$2'
target_label: 'url'
By default source_labels are joined with ; separator before applying regex to the resulting string. So the resulting string will contain address:port;path value. The given regex matches this string and extracts address and path parts into $1 and $2. Then these values are used for constructing replacement string in the form http:///
Chaining relabeling rules
Relabeling rules can be chained. For example, the following rules add {foo=“bar”} label and remove port from instance label:
- replacement: 'bar'
target_label: 'foo'
- source_labels: [instance]
regex: '([^:]+):.*'
replacement: '$1'
target_label: 'instance'
Arbitrary number of relabeling rules can be chained.