Alternative to configuration files in Prometheus? The Next CEO of Stack OverflowAlert Manager Alert rules are not functioning properlyComplex rules/filters for Prometheus-Alertmanager AlertsCan't load prometheus.yml config file with docker (prom/prometheus)How can I add alerts in Prometheus + AlertManager to certain Prometheus Job Names?How to configure alerts in Prometheus for diskspaceHow to configure Prometheus in a multi-location scenario?How to divide after grouping two different metrics in Prometheus?prometheus aggregate table data from offset; ie pull historical data from 2 weeks ago to presentReceiver called with already notified resolved alertsConfigure alertmanager alerting in prometheus helm chart values.yaml
Compilation of a 2d array and a 1d array
Is a distribution that is normal, but highly skewed, considered Gaussian?
My ex-girlfriend uses my Apple ID to login to her iPad, do I have to give her my Apple ID password to reset it?
What day is it again?
Mathematica command that allows it to read my intentions
How dangerous is XSS
The sum of any ten consecutive numbers from a fibonacci sequence is divisible by 11
How can the PCs determine if an item is a phylactery?
How exploitable/balanced is this homebrew spell: Spell Permanency?
What are the unusually-enlarged wing sections on this P-38 Lightning?
Find the majority element, which appears more than half the time
Does the Idaho Potato Commission associate potato skins with healthy eating?
What does it mean 'exit 1' for a job status after rclone sync
Direct Implications Between USA and UK in Event of No-Deal Brexit
How to find if SQL server backup is encrypted with TDE without restoring the backup
Planeswalker Ability and Death Timing
Is it correct to say moon starry nights?
Can I cast Thunderwave and be at the center of its bottom face, but not be affected by it?
What steps are necessary to read a Modern SSD in Medieval Europe?
How do I keep Mac Emacs from trapping M-`?
Small nick on power cord from an electric alarm clock, and copper wiring exposed but intact
Calculating discount not working
Is there a rule of thumb for determining the amount one should accept for of a settlement offer?
Is it reasonable to ask other researchers to send me their previous grant applications?
Alternative to configuration files in Prometheus?
The Next CEO of Stack OverflowAlert Manager Alert rules are not functioning properlyComplex rules/filters for Prometheus-Alertmanager AlertsCan't load prometheus.yml config file with docker (prom/prometheus)How can I add alerts in Prometheus + AlertManager to certain Prometheus Job Names?How to configure alerts in Prometheus for diskspaceHow to configure Prometheus in a multi-location scenario?How to divide after grouping two different metrics in Prometheus?prometheus aggregate table data from offset; ie pull historical data from 2 weeks ago to presentReceiver called with already notified resolved alertsConfigure alertmanager alerting in prometheus helm chart values.yaml
I want to implement a monitoring system in a small organization.
I have experience with Zabbix, but I'm considering Prometheus as it seems there's a big community behind it, and it's pretty much the standard today (please correct me if I'm wrong).
However, I wonder how it is convenient to handle all the configuration of the scrapes and alerting rules in configuration files, rather in a web UI like in Zabbix. I mean, it may be ok for 10-20 alerts, but how can it be manageable with 20 different services and 1000 different alerts, for example? Maybe there's a solution for it that I just missed? In Zabbix it's very convenient to manage everything, since it's UI and also everything is going into groups (host groups, templates, etc.)
I'd appreciate your insights.
prometheus prometheus-alertmanager asked Mar 21 at 19:27
roeezabroeezab
42
add a comment |
I want to implement a monitoring system in a small organization.
I have experience with Zabbix, but I'm considering Prometheus as it seems there's a big community behind it, and it's pretty much the standard today (please correct me if I'm wrong).
However, I wonder how it is convenient to handle all the configuration of the scrapes and alerting rules in configuration files, rather in a web UI like in Zabbix. I mean, it may be ok for 10-20 alerts, but how can it be manageable with 20 different services and 1000 different alerts, for example? Maybe there's a solution for it that I just missed? In Zabbix it's very convenient to manage everything, since it's UI and also everything is going into groups (host groups, templates, etc.)
I'd appreciate your insights.
prometheus prometheus-alertmanager asked Mar 21 at 19:27
roeezabroeezab
42
add a comment |
I want to implement a monitoring system in a small organization.
I have experience with Zabbix, but I'm considering Prometheus as it seems there's a big community behind it, and it's pretty much the standard today (please correct me if I'm wrong).
However, I wonder how it is convenient to handle all the configuration of the scrapes and alerting rules in configuration files, rather in a web UI like in Zabbix. I mean, it may be ok for 10-20 alerts, but how can it be manageable with 20 different services and 1000 different alerts, for example? Maybe there's a solution for it that I just missed? In Zabbix it's very convenient to manage everything, since it's UI and also everything is going into groups (host groups, templates, etc.)
I'd appreciate your insights.
prometheus prometheus-alertmanager asked Mar 21 at 19:27
roeezabroeezab
42
I want to implement a monitoring system in a small organization.
I have experience with Zabbix, but I'm considering Prometheus as it seems there's a big community behind it, and it's pretty much the standard today (please correct me if I'm wrong).
However, I wonder how it is convenient to handle all the configuration of the scrapes and alerting rules in configuration files, rather in a web UI like in Zabbix. I mean, it may be ok for 10-20 alerts, but how can it be manageable with 20 different services and 1000 different alerts, for example? Maybe there's a solution for it that I just missed? In Zabbix it's very convenient to manage everything, since it's UI and also everything is going into groups (host groups, templates, etc.)
I'd appreciate your insights.
prometheus prometheus-alertmanager prometheus prometheus-alertmanager asked Mar 21 at 19:27
roeezabroeezab
42
asked Mar 21 at 19:27
roeezabroeezab
42
asked Mar 21 at 19:27
roeezabroeezab
42
asked Mar 21 at 19:27
roeezabroeezab
42
asked Mar 21 at 19:27
roeezabroeezab
42
42
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Prometheus is a compelling solution that -- deservedly -- has considerable momentum. For completeness, I urge you to consider other changes that may required by Prometheus as you determine whether it's appropriate for you to switch from your existing solution. This will be one of many changes for which you should be prepared.
I don't know why Prometheus' developers took the path that they did with configuration but I'll give you one benefit of the model. By requiring that configuration is written in (YAML) configuration files, you are able to commit these configurations to your source control and manage changes to these files just as you would with other sources. This is a tenet of "Infrastructure as code" and there are many benefits to this.
Now, it's also true that a UI Console tool could write changes to configuration files for you and you could then manage these files using source control as before. But, this functionality isn't available in the core solution.
One facility that may interest you is that you needn't restart prometheus nor alertmananger when you make changes to their configuration. You can change the configuration and then, POST to the services' /-/reload endpoint and the service will refresh configuration.
See:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://prometheus.io/docs/alerting/configuration/
Also see this useful article that explains how to have Prometheus automatically refresh its configuration when changes are made to a list of targets:
https://www.robustperception.io/using-json-file-service-discovery-with-prometheus
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
add a comment |
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55287964%2falternative-to-configuration-files-in-prometheus%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Prometheus is a compelling solution that -- deservedly -- has considerable momentum. For completeness, I urge you to consider other changes that may required by Prometheus as you determine whether it's appropriate for you to switch from your existing solution. This will be one of many changes for which you should be prepared.
I don't know why Prometheus' developers took the path that they did with configuration but I'll give you one benefit of the model. By requiring that configuration is written in (YAML) configuration files, you are able to commit these configurations to your source control and manage changes to these files just as you would with other sources. This is a tenet of "Infrastructure as code" and there are many benefits to this.
Now, it's also true that a UI Console tool could write changes to configuration files for you and you could then manage these files using source control as before. But, this functionality isn't available in the core solution.
One facility that may interest you is that you needn't restart prometheus nor alertmananger when you make changes to their configuration. You can change the configuration and then, POST to the services' /-/reload endpoint and the service will refresh configuration.
See:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://prometheus.io/docs/alerting/configuration/
Also see this useful article that explains how to have Prometheus automatically refresh its configuration when changes are made to a list of targets:
https://www.robustperception.io/using-json-file-service-discovery-with-prometheus
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
add a comment |
Prometheus is a compelling solution that -- deservedly -- has considerable momentum. For completeness, I urge you to consider other changes that may required by Prometheus as you determine whether it's appropriate for you to switch from your existing solution. This will be one of many changes for which you should be prepared.
I don't know why Prometheus' developers took the path that they did with configuration but I'll give you one benefit of the model. By requiring that configuration is written in (YAML) configuration files, you are able to commit these configurations to your source control and manage changes to these files just as you would with other sources. This is a tenet of "Infrastructure as code" and there are many benefits to this.
Now, it's also true that a UI Console tool could write changes to configuration files for you and you could then manage these files using source control as before. But, this functionality isn't available in the core solution.
One facility that may interest you is that you needn't restart prometheus nor alertmananger when you make changes to their configuration. You can change the configuration and then, POST to the services' /-/reload endpoint and the service will refresh configuration.
See:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://prometheus.io/docs/alerting/configuration/
Also see this useful article that explains how to have Prometheus automatically refresh its configuration when changes are made to a list of targets:
https://www.robustperception.io/using-json-file-service-discovery-with-prometheus
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
add a comment |
Prometheus is a compelling solution that -- deservedly -- has considerable momentum. For completeness, I urge you to consider other changes that may required by Prometheus as you determine whether it's appropriate for you to switch from your existing solution. This will be one of many changes for which you should be prepared.
I don't know why Prometheus' developers took the path that they did with configuration but I'll give you one benefit of the model. By requiring that configuration is written in (YAML) configuration files, you are able to commit these configurations to your source control and manage changes to these files just as you would with other sources. This is a tenet of "Infrastructure as code" and there are many benefits to this.
Now, it's also true that a UI Console tool could write changes to configuration files for you and you could then manage these files using source control as before. But, this functionality isn't available in the core solution.
One facility that may interest you is that you needn't restart prometheus nor alertmananger when you make changes to their configuration. You can change the configuration and then, POST to the services' /-/reload endpoint and the service will refresh configuration.
See:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://prometheus.io/docs/alerting/configuration/
Also see this useful article that explains how to have Prometheus automatically refresh its configuration when changes are made to a list of targets:
https://www.robustperception.io/using-json-file-service-discovery-with-prometheus
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
Prometheus is a compelling solution that -- deservedly -- has considerable momentum. For completeness, I urge you to consider other changes that may required by Prometheus as you determine whether it's appropriate for you to switch from your existing solution. This will be one of many changes for which you should be prepared.
I don't know why Prometheus' developers took the path that they did with configuration but I'll give you one benefit of the model. By requiring that configuration is written in (YAML) configuration files, you are able to commit these configurations to your source control and manage changes to these files just as you would with other sources. This is a tenet of "Infrastructure as code" and there are many benefits to this.
Now, it's also true that a UI Console tool could write changes to configuration files for you and you could then manage these files using source control as before. But, this functionality isn't available in the core solution.
One facility that may interest you is that you needn't restart prometheus nor alertmananger when you make changes to their configuration. You can change the configuration and then, POST to the services' /-/reload endpoint and the service will refresh configuration.
See:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://prometheus.io/docs/alerting/configuration/
Also see this useful article that explains how to have Prometheus automatically refresh its configuration when changes are made to a list of targets:
https://www.robustperception.io/using-json-file-service-discovery-with-prometheus
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
answered Mar 22 at 2:52
DazWilkinDazWilkin
2,23421928
2,23421928
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
add a comment |
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
Thanks for your answer. "Infrastructure as code" is great and I'm going this path in other tools like Ansible and Terraform, but there are some tools that I think it's ok to consider as "blackbox" and just backup their configuration routinely in traditional manners, rather than source control. Anyway, going this path of configuration files, I think that one of the basic features should be the ability to separate the configuration into multiple files with imports and includes, to avoid one huge configuration file.
– roeezab
Mar 22 at 7:36
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
It just seems that some features in Prometheus were designed for monitoring a very small system. Another thing that I find hard to manage is searching in all the metrics. In Graphite, for example, you have a tree for that. In AWS CloudWatch, you have namespaces and it's very easy to start looking for metrics for your service. It's a shame, because Prometheus could be a great monitoring tool, but lacks very basic features to support large systems monitoring.
– roeezab
Mar 22 at 7:39
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55287964%2falternative-to-configuration-files-in-prometheus%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
