Interactive dashboards: using in-panel links to control Grafana variables

One of my colleagues recently had a question from a customer: is it possible to have one panel in Grafana show a set of all metrics, but the other panels on a dashboard filter to a single metric value from a click in the first panel?

It's an interesting way of working with data, and something that many people find intuitive.  If I have a menu of options displayed, I probably want to see only things relevant to one of those options if I click it.  Unfortunately, doing that's not something that is immediately obvious in Grafana.  For better or worse (and there are good reasons you might not want that behavior!) there's no automatic filtering inside of panels on a dashboard.

Usually people will use dashboard variables for this.  Variables provide powerful functionality for filtering queries in Grafana.  But they have a drawback: they aren't integrated directly into a dashboard, always appearing at the very top of the screen.

After playing around for a bit, it turns out that there is a way to use variables inside of a panel and have them act the same when you click a value as if you had selected that value from a dropdown at the top of the dashboard.  To do this, we can slightly abuse the data links functionality provided by panels.

I'll provide an example of this using Prometheus as a data source, but it should work with any data source you like.  We're going off the beaten path in Grafana here.  Buckle up, because this one gets a little bumpy!

Text panel tricks: drawing dividers in Grafana dashboards

Grafana has a lot of fantastic visualizations built in.  You can draw graphs of nearly every imaginable type, create beautiful maps, even build interactive flowcharts with multiple data elements that update in real time.

But my favorite (and definitely the most overlooked) panel is the simple text panel.

The text panel contains far more power than most people realize.  The fact that it can act as a container for almost any HTML means that you can do a lot of really cool things with it.

Today let's look at how to use the text panel to draw horizontal and vertical separators on a dashboard.

I've started by designing a simple dashboard showing some status information for three different manufacturing lines.  In a plant like this, the plant manager wants to see the status of all the lines at a glance but still easily differentiate them for troubleshooting.  With a small group of items, like the three here, it's useful to use a columnar layout.  But Grafana uses a row orientation for dashboards, meaning it is easy to accidentally move a panel around slightly and have it be unclear which column it belonged to.  And even if alignment isn't an issue, how can you let viewers know that you're lining up your data in columns rather than the default rows?

In the screenshot below, you'll see a possible solution: dividers between columns that make it perfectly clear that each graph "belongs" to a vertical group.  (In this case I've used a very bright violet to make the dividers stand out, but in a real dashboard you'd probably want something a bit more subtle!)

While there's no "line" option in the text panel, you can use some simple HTML and CSS tricks to create one.

Vertical dividers

To make a vertical line, start by setting a few options on the text panel:

• No panel title (just delete the text that's there)
• Transparent background turned on
  
• Panel mode set to "HTML"

This will make your text panel as unobtrusive as possible.  Getting rid of the title means that the full area of the panel will be available for your line with no space reserved for title text.  Enabling a transparent background means that you won't have a bright border around the panel itself, and we'll need to write HTML for the trick to work.

Once your panel is configured, add the following text:

<div

    style="margin: auto; border-left: 3px solid #f0f; width: 3px; height: 100%">

</div>

Once you're done, your panel configuration should look something like this:

Let's break this down a bit.

The <div> HTML tag lets us create an empty HTML container.  Usually this is used to hold text, images, or other content, but for our purposes we really just want a "thing" that sits on the page to draw a border around.

The style atrribute lets us attach CSS to this empty container.  We set the margin to auto which means that our div will be centered within its parent container, in this case the Grafana panel.

We want either the left or right border on this empty container to be turned on; the div will only be as wide as the full border so it doesn't really matter which one, and "left" is just slightly shorter to type than "right" so being lazy, I used that.  The properties of the border-left attribute control how the border will look.  Changing "3px" to "5px" will make the border two pixels wider, and changing "#f0f" to "black" will change the color from violet to black.  There are a number of ways to change this up, so check out the CSS border properties for more information.

Finally, we set the height of this container to 100%.  This makes it adapt as the panel is resized.  You can then easily make your divider longer or shorter just by resizing the Grafana panel.

Once this is done, you can shrink the panel down to the thinnest width and drag it out to be as tall as you need it to be.

Horizontal dividers

If you want a horizontal divider in Grafana, there are a couple of options available.  You could use a similar technique as above but with a border-top instead of border-left.  But HTML gives us a slightly easier way to make horizontal borders: the <hr> tag.

The <hr> tag creates a horizontal rule (i.e. line) in a page.  And like most other HTML elements, it can be styled with CSS.

To create a styled horizontal divider in Grafana, start by setting up a transparent HTML text panel as described above.  This time, add the following HTML:

<hr style="height: 3px; background-color: #f0f">

This will create a horizontal line on the panel with a height of 3 pixels and the same violet color we used before.  Feel free to change these values to match your dashboard style.

Note that this won't center the line vertically in your panel, but that's usually okay -- you'll be resizing the panel to be as short as possible anyway.  (If you really need some blank space above your line, remember that you can create a completely empty transparent text panel and resize it to whatever spacing you want.  That's often easier than trying to center things vertically in CSS!)

I hope this helps you make beautiful and logical divisions in your dashboards.  Go line everything up!

Working with noisy data in Prometheus

 I haven't written much here lately as I've been busy elsewhere.  But some of that includes working extensively with Prometheus for monitoring various services.  Along the way, I've written up a blog post on the Grafana Labs blog about working with noisy data.  The beginning of it is here but the full post is up on the Grafana blog.

Most of us have learned the hard way that it’s usually cheaper to fix something before it breaks and needs an expensive emergency repair. Because of that, I like to keep track of what’s happening in my house so I know as early as possible if something is wrong.

As part of that effort, I have a temperature sensor in my attic attached to a Raspberry Pi, which Prometheus scrapes every 15 seconds so I can view the data in Grafana. This way, I know how things look over time, and I can get alerts if my house is getting too hot or too cold.

Unfortunately, my temperature sensor is a bit flaky. It works most of the time, but occasionally it gives me wildly inaccurate readings. Here’s an example that looks at a few hours of data:

Even though the weather can be unpredictable here in New England, it’s pretty unlikely that the room temperature dropped from 20°C to -10° for 15 seconds before returning to normal!

Still, other than these occasional single readings that are obviously wrong, most of the data looks good. So my first thought when I saw this glitch was to replace the sensor with one that works more consistently. After all, having good data helps with everything.

But I started to think about the problem more . . .

What would I do if I had a sensor like this that couldn’t easily be replaced? If my sensor were on top of a mountain or deep under the ocean, it would be difficult and expensive to fix. And if it were on a satellite or in a rover on Mars, it would be impossible.

I couldn’t help but wonder: If most of the data is good, is there a way to keep the good bits and throw out the bad?

After speaking with some data scientist friends, it turns out that the answer is yes. And even better, Prometheus has a function to do exactly that!

Read the rest here!