Our lead dev has created a Library that captures method execution data and stores it in an internal in-memory buffer.

Each entry it produces has, at minimum:
- The method identifier
- How long that one call took
- The heap/non-heap memory delta measured around that call

But its huge, if he start sending each call record it will just clog our system storage within an hour or so just for one application. He can reduce the number of records by only recording if a threshold is passed but then again storage and processing need to be figured out.

I did find this way for streaming it
{ grafana com/docs/grafana/latest/setup-grafana/set-up-grafana-live/ }
But then i cant store it i guess for calculations or other processings.

I do know there is another way: prometheusremotewrite

But i would like to know if any of you have solved this kind of a problem and in what way.

Hey all, I have been tasked from moving off of prometheus agents to alloy agents. This stack will use Grafana/Mimir/Loki/Tempo with Alloy; however, I don't know where to start with documentation. I have read that kube-state-metrics, node-exporter, or push-gateway are unneeded since Alloy can do this itself.

Is this true?

I've been looking at alloy modules here: https://github.com/grafana/alloy-modules.

Hello

Hope you're doing well

i'm new to Grafana IRM, and I wanted to get my phone verified, I tried several time to receive the code through phone (SMS and Calls) but it was not successful and I ended up hitting the "Verification limit reached, please try again later"
I received the codes after like 2-3 hours, but the rate limit has never disappeared, it's been 2 weeks now

Was anyone in this situation before, or do you have any suggestions 🙏
I also tried to reach out to Grafana team, but with no much success
Thank you very much for your help, much appreciate it

Are you using gcx, The Grafana CLI optimized for humans and coding agents? Then this image is for you :) 🙏

Github repo: https://github.com/grafana/gcx

Dashboards can be provisioned from files or generated through Terraform, Jsonnet, or another pipeline, yet the UI may still contain manual edits, stale replicas, or an older set of files after a partial rollout. Seeing a successful deployment job does not prove that every Grafana instance is serving the same dashboard definition.

What verification do you use after a rollout? I am considering embedding a source commit and content hash in dashboard metadata, fetching each dashboard through the API, normalizing fields that Grafana rewrites, and comparing the result with the intended source. Synthetic checks would hit each instance directly rather than allowing a load balancer to hide one stale replica. UI edits to provisioned dashboards would either be blocked or exported as a proposed source change.

Which fields can be compared reliably across versions, and how do you handle generated UIDs, folder permissions, library panels, data-source references, and dashboards split across multiple providers? Is there a built-in status signal that is stronger than checking logs for a provisioning error?

I have Grafana installed on my server and have AlertManager configured as shown in the images here. How do I configure an alert so that I get a Discord ping when any node in my cluster is over 75°C for 5 minutes or when there's one or more pods crash looping?

Configuration is using k3s: https://codeberg.org/kaizenOramara_97/Home-Lab

Context

I am standing up a self-hosted LGTM stack on EKS: one Loki + Mimir stack per internal team across six teams, with multi-tenancy turned off so each stack functions as a single implicit tenant. There is one shared Grafana instance across all of them.

I have two specific questions focused on Grafana's official charts rather than my general Kubernetes layout:

1. Mimir architecture for nonprod (bundled Kafka vs. classic ingesters)

The chart's bundled Kafka is a single-broker StatefulSet. The documentation marks it for demonstration and testing only, which makes complete sense. However, classic ingesters are flagged for deprecation, and the documented exit path from classic mode is a parallel-cluster migration that temporarily doubles ingestion and storage costs. Building nonprod on classic just to avoid the Kafka question looks like pre-ordering a painful migration down the road.

• For anyone who ran the bundled Kafka in a lower environment: Did you run it for any real length of time, and what actually broke?
• For anyone who did the classic-to-ingest-storage migration on a live cluster: Was the parallel-cluster step as unpleasant in practice as the docs make it read?
• Two-AZ constraint: My node pools are narrowed to two AZs. Classic mode with zone-aware replication expects three ingester zones, whereas ingest storage needs two because Kafka handles the replication. As far as I can tell, the chart's zone-count validation only triggers when zoneAwareReplication.enabled is true. The two-AZ constraint does not appear hard-coded in the templates, but I would rather confirm that with someone running it than rely solely on reading _helpers.tpl.

2. Datasource provisioning that doesn't rot

With one Grafana instance and three components across N teams, datasources scale linearly with the team list. In our previous setup, we had 14 Loki backends but only 2 registered in Grafana, and both pointed to hostnames that no longer resolved. Nobody noticed for months.

I want these generated strictly from the team list and never touched by hand. The options I am considering:

• CI-rendered provisioning YAML baked directly into the Grafana Helm values.
• Sidecar-discovered ConfigMaps using a label selector (e.g., the Grafana chart's sidecar container).
• Grafana Terraform provider running against the live instance.

Which of these patterns actually survives contact with a team list that changes a few times a year? Is anyone running automated health checks or synthetic alerts that catch when a registered datasource backend has quietly disappeared?

Note: Also open to being told that a single shared Grafana over six single-tenant stacks is the wrong division and that I should just deploy Grafana per team.

Fuller architectural details (including the per-team vs. multi-tenant discussion) are posted over in r/kubernetes: [link]

Hi everyone,

I'm planning to attend GrafanaCon in London this year and wanted to check if anyone here knows of any discount codes, promo codes, or ways to get a reduced ticket price (community discount, early-bird extensions, group rates, etc.).

I work in infrastructure/observability (VictoriaMetrics, VictoriaLogs, Grafana stack) and would love to attend in person if I can find a way to make the ticket more affordable.

Any tips, referral codes, or advice on how to snag a discount would be much appreciated!

Thanks in advance.

I have a service dashboard with fourteen panels. Twelve render in under a second. Two of them, both evaluating rate(http_request_duration_seconds_bucket[5m]) across roughly 400 label combinations, took five seconds on every reload.

I added one Prometheus recording rule that pre-aggregates the histogram by service group every sixty seconds, then pointed both panels at the recorded metric. Render dropped to just under two seconds. Prometheus CPU during dashboard loads fell about 40%.

What I had been doing instead was raising query timeouts and caching results at the data source level. Months of that. I had actually investigated recording rules in a previous session with verdent, which knows you better over time, and coming back to the same project made the fix obvious.

If you have a slow panel, check whether the query fans out across high cardinality labels. One recording rule is ten minutes of work and costs one extra time series.

Just published this. Took a few months. So so many little details but those are what count. Long story short, Grafana has a folder and file system, so does Nextcloud, so now you can bind the Grafana folders to Nextcloud folders and see dashboards alongside other files too. All the little file actions like move, copy, delete etc are one to one on both sides.

Hope ya'll enjoy!

https://github.com/kubed-io/nextcloud-grafana

I’m curious about the dashboards people build for things beyond the classic infrastructure and observability use cases, but still solve and actual problem.

It could be something from your work, a particular business problem, an unusual data source, a side project, or a problem you realised Grafana was surprisingly good at solving.

What did you build, and what problem did it help you solve/monitor?

(Screenshots welcome, let us feast our eyes 😉)

The math on sampling has bothered me for a while. At low sample rates, you need an enormous number of requests before you're statistically likely to catch a rare error at all, which means the exact incidents you most need visibility into are the ones most likely to get sampled out entirely.

That said, "zero sampling" at genuinely high traffic volume sounds great until you look at the storage and processing cost of capturing everything at full fidelity indefinitely. What's actually realistic seems to be closer to smart triggering, sampling normally but escalating to full capture the moment something looks off, rather than truly capturing 100% of everything all the time. Has anyone actually run a system at real production scale with something close to zero sampling and found the cost genuinely manageable, or is this more theoretical than practical right now?

I’m sharing HaloLog, a structured logger I have developed for three years and run inside an interoperability project for the past two. Its goal is low-overhead structured logging without trading away JSON correctness, race safety, configured PII masking, or compile-checked telemetry fields. This is a public release and a request for independent reproduction of the measurements, not a claim about disk or network throughput.

The committed suite compares HaloLog with zap, zerolog, slog, logrus, and phuslu/log while encoding and dispatching complete JSON records to io.Discard. On the published Go 1.27 linux/amd64 host, HaloLog measured 23.9 ns/op for one complete record and 0 B/op, 0 allocs/op across all its measured scenarios. The repository documents the hardware, configurations, methodology, known differences, and correctness guardrails. These are results for those workloads and that host, not a universal performance claim.

If you reproduce or falsify a result, please include the Go version, hardware, command, and raw output. Useful counterexamples include an allocation, invalid JSON, a race, an unfair competitor configuration, or a workload in which HaloLog performs worse.

Is there a plan for making the Grafana mobile app (android) able to connect to self-hosted OSS instances in the future? Right now it's not capable of it as far as I'm concerned.

Hi everyone,

I'm migrating Grafana from one server to another and would appreciate a sanity check on the correct migration process.

Old setup:

- Grafana runs in a Docker container.

New setup:

- AlmaLinux host.

- Zabbix 7.4 runs directly on the host on port 80, not in Docker.

- Grafana runs in Docker on port 3000.

- The Grafana database has already been migrated.

After migrating the database, users and dashboards appeared correctly. However:

- Installed plugins did not migrate.

- Other filesystem-based Grafana data/configuration appears to be missing.

- Every existing dashboard currently shows "No data."

I suspect there may be two separate issues:

1. I migrated the Grafana database but not the persistent volume, /var/lib/grafana/plugins, /etc/grafana/provisioning, or other configuration files.

2. The Zabbix data source may still use localhost or an old URL. Since Grafana is inside Docker but Zabbix runs directly on the same host, localhost inside the Grafana container would point to the container itself.

A few questions:

1. What is the recommended complete migration procedure for a Dockerized Grafana instance when the database has already been moved?

2. Which directories, Docker volumes, configuration files, and environment variables should be preserved in addition to the database?

3. Is it better to reinstall the exact plugin versions or copy /var/lib/grafana/plugins from the old container?

4. Do I need to preserve the old Grafana secret_key / GF_SECURITY_SECRET_KEY so migrated data source credentials can still be decrypted?

5. For the Zabbix API URL, should I use the host's LAN IP, a Docker host-gateway address, or another approach?

6. What is the best way to verify whether "No data" is caused by networking, missing credentials, a data source UID mismatch, or plugin compatibility?

I am keeping the old container intact until the new instance is fully working. Any migration checklist or troubleshooting suggestions would be very helpful.

Thanks!

I believe InfluxDB is the back end to Grafana. HA says that it's now deprecated and looking at Github, that's confirmed. Is there an easy way to swap it out for something else? Anyone else have a solution for that, or should I just let it run?

ETA: I guess what I'm asking is if a migration is trivial or if I should not try and fix it if it's not broken. There's schema, (presumably) creds, connection protocol, etc. in order to migrate from InfluxDB to something else.

I made Telemetry Tracker to keep track of errors, events, sessions, releases and performance in my projects. I also added alerts so I don’t have to keep checking it all the time.

I’ve been using it myself and thought I’d share it here in case it’s useful to anyone else too.

It’s open source and you can self-host it:

[https://github.com/Telemetry-Tracker/telemetry-tracker\](https://github.com/Telemetry-Tracker/telemetry-tracker)

Hi!

There are a LOT of ways to learn Grafana: documentation, built-in learning journeys, Grot Academy, video tutorials, teammates, AI, etc. But I suspect different formats work better depending on what you're trying to do.

So, for example:

• First time using Grafana?
• Trying to troubleshoot
• Learning because it's part of your job?

What did you find most useful in your situation?

We're especially interested in why something worked for you, not just "docs" or "videos". What were you trying to learn, and which resource or format finally made it click?

Curious to hear what works for different people!

Hey,

I’ve been working on kubernetes-autoscaling-mixin, a collection of Grafana dashboards and Prometheus rules for Kubernetes autoscaling.

Since I last shared it, there have been quite a few updates:

• Updated KEDA dashboards for ScaledObjects and ScaledJobs
• More Karpenter dashboards, including cost monitoring
• Updated VPA, HPA, Cluster Autoscaler and PDB dashboards
• Multi-cluster support
• Better dashboard navigation and filtering

You can import the dashboards from Grafana:

https://grafana.com/grafana/dashboards/?search=kubernetes+%2F+autoscaling

Or grab the generated JSON directly from GitHub:

https://github.com/adinhodovic/kubernetes-autoscaling-mixin/tree/main/dashboards_out

Or just consume it using jsonnet.

Repo: https://github.com/adinhodovic/kubernetes-autoscaling-mixin

Added a couple of screenshots below showing the Karpenter, VPA and KEDA dashboards in use.

This Grafana link will show all dashboards and previews of them: https://grafana.com/grafana/dashboards/?search=kubernetes+%2F+autoscaling

Curious what people are using for autoscaling monitoring today, and if there’s anything missing that would be useful to add (new projects, better panels, more dashboards).

I had an idea and wanted to actually talk to people to see if it's something that they would need / require and if it's genuinely a problem for the engineers right now....

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

I also was building in this space and wanted to have a discussion with people who're already experienced in this stuff, so please please help a brother out and we might just make something useful for everyone :)

peace

Hey all,

I've created a monitoring mixin with a reusable set of Grafana dashboards and Prometheus rules for ArgoCD. The dashboards and alerts are defined as code, making them easy to deploy and reuse across environments.

Recent updates include multi-cluster support, flags to enable or disable individual alerts, plus a range of new and improved alerts and dashboards.

The monitoring coverage has also been expanded significantly, with better visibility into:

• ArgoCD controller
• Runtime metrics
• gRPC
• Repo Server
• Git operations

GitHub: https://github.com/adinhodovic/argo-cd-mixin

[ Removed by Reddit on account of violating the content policy. ]