Caching Grafana graphs with the open source version

If you’ve found this then I’m going to assume you are in the situation I was in a week or so ago. Using the open source /free Grafana to produce graphs, but running up against them being slow.  Figuring there must be a way to at least just cache the responses you take to Google only to find that whilst Grafana does have a caching feature, it’s only in the Enterprise version and that’s just a whole different thing.

Well, this is how I solved that problem: NGINX caching proxy

The tricky part is that of course it’s not really designed for something else to cache it, and you need to understand a bit about the underlying calls to get a solution that works properly.

So when you load a grafana dashboard, it generates calls to all the charts individually (the same as just taking the chart embed url and calling it directly)

Those calls all fetch a bunch of JavaScript resources, and somewhere in the set of responses it gets the SQL query your chart is running, and it uses those J’s libraries to fill in the parameters and importantly the ‘time period’ based on the client browser time. It then sticks all this together into a POST call to a data source API which responds with all the datapoints that get passed to the JavaScript visualisation piece that actually renders the graph.

Obviously it’s easy to cache all those JavaScript loads, but that’s not where you’re losing time. it’s that POST request that is doing your db query, and it’s generating a unique payload everytime because it’s turning the current time into a timestamp down to milliseconds for ‘now’ and calculating the appropriate ‘from’ time based on your selected time horizon.

For my case, I’m exposing graphs about a 90 day rolling window, via some chart embeds, and I don’t need each user to see down to the Ms up to date data. basically for me it’s just fine if everyone sees the same data as any one else in a given day.

For me then, where I’m already running grafana in a container in a kubernetes cluster, I can just drop in an nginx pod, with it’s own service exposed via an ingress and point a dns to that instead of going direct to Grafana.

nginx caching config is pretty simple and I won’t repeat their docs here, however the important details are that you need to enable caching on POST (the default sensibly is just GET HEAD)

You also need to ignore the cache-control header that grafana is sending, it’s setting no-cache on that api. otherwise nginx will just obey the no cache instruction.

The other tricky bit is the cache key. because on the POST request the only difference between the calls for different charts is the payload (and sometimes the referrer).

Ideally you’d be able to regex out some piece of the payload to form a unique cache key, but that’s a feature that requires nginx plus, or some extra plugins. Instead we do something hacky and just use the referrer url and content_length in the cache key and assume that for a given dashboard you are probably using queries that don’t accidentally turn out to be identical lengths.

1. to explicitly add the POST method to those to be cached. proxy_cache_methods GET HEAD POST;
2. To construct a cache key:
   set $cache_key $scheme$request_method$host$request_uri$http_referer$content_length;
3. tell nginx to ignore the no-cache directive coming from Grafana itself, and suppress those headers going to the browser
   proxy_ignore_headers Cache-Control;
proxy_hide_header Cache-Control;
proxy_hide_header Expires;

And that’s basically it. set a cache time appropriate to your use case, point your embedded charts via the cache DNS name and away you go. For my use case where we had some pretty slow data queries (they needed to parse into some JSON for a custom field) it meant what used to take about 20s to load now loads in <2s.