tevents: event logger and job monitor for tailnets

on
Photo of Robin Verton
This article is contributed by a member of the Tailscale community, not a Tailscale employee. If you have an interesting story to share about how you use Tailscale, reach out to devrel@tailscale.com.

Since I found Tailscale some years ago, it's this magic little unobtrusive tool for me, always running in the background. I don't use it daily, but that's the nice thing: It does not get in my way and is crucial for me when I need it.

Some weeks ago, I read a blog post about golink, a tool written by the Tailscale team for sharing links in a Tailscale network (called tailnet). Link sharing services are nothing new, but having a services in your own network, accessible anytime you are connected to your own tailnet (and only to your network) did make sense. Not for me, to be honest, because there is not a lot of value in sharing links with myself (other than the nice templating feature). But something else here was new to me: Having the possibility to share a single service in my tailnet, only by using a token and without giving this complete machine access to my tailnet.

Idea for tevents

Thinking of interesting services I can put in my tailnet, I came up with something I usually solve with some hacky scripts. For small services, I don't have a central place to gather events/logs. I don't want to use some online service for a handful of events I'm going to watch. I also don't want to maintain some logging service with all dependencies. There was also a second usecase I thought of: I wanted to have a simple job monitor, visualizing executions of cronjobs and other scheduled tasks to keep an eye on them. I called it tevents.

Just Go for it

Let's start with some implementation detail here. tevents is written in Go, compiled to a single executable which embeds all required (web) assets. As it is essential for a Tailscale related project, I am using a JSON file as a database here. Just kidding, I skipped this ritual and spent my innovation token recklessly on some currently hyped database technology: sqlite!

I like to keep direct dependencies to a minimum as long as I can, so the only external deps I need to import are the sqlite driver and tsnet. Go's stdlib is awesome, giving nearly all utilities which were needed: Web handling, database stuff and templates. I love how Go makes it possible to embed static resources in the executable file. It allows to bundle assets as the DB schema or web resouces:


//go:embed assets/*
var assetsFS embed.FS
func parseTpl(funcs template.FuncMap, file string) *template.Template {
return template.Must(
template.New("layout.html").Funcs(funcs).ParseFS(assetsFS, "assets/layout.html", file))
}

The web page is styled via TailwindCSS, which allows for fast prototyping web layouts.

For convenience I made use of a Makefile which holds some useful commands for development: Building tevents, building assets (Tailwind has a built step to minimize the resulting CSS file), and watching for file changes during development.

Integrating tsnet

The last step was to integrate tsnet, the tailscale-as-a-library package which allows to put services on a tailnet. I nearly did a 1:1 copy from the tsnet blog post and was done:


ts := &tsnet.Server{
Hostname: *hostname,
}
defer ts.Close()
ln, err := ts.Listen("tcp", ":80")
if err != nil {
log.Fatal(err)
}
defer ln.Close()
lc, err := ts.LocalClient()
if err != nil {
log.Fatal(err)
}
// pass listener and local client to the tevents web server setup
s := tevents.NewServer(":8080", db, ln, lc)
s.EventService = tdb.NewEventService(db)
if err := s.Start(); err != nil {
log.Fatal("http server failed:", err)
}

The hostname is passed to the tsnet.Server and allows to set a custom hostname in tailnets which make use of MagicDNS. This way we can just call http://tevents for event logging and to access the dashboard.

And that was all it takes for making a service in Go available in a tailnet! It's nice how a simple abstraction like this allows to deploy a private and secure service, directly embedded in a Go application.

Server-side events for realtime

I added some realtime capabilities for the event page to see events immediately show when they occured. I used htmx to subscribe to a server-side event endpoint. There is a nice blog post describing how to do it which I took some inspiration from. It allows to add the event handling very easy in just a few lines. I usually use some (high interactivity) frontend like Next.js when writing user interface in Go, but for a lot of usecases like this one, there really is no need for this. Its enough to partially render on the server and send it over to the frontend. This keeps the overhead, languages and dependencies in use low.

So let's add the htmx SSE handling:


<div hx-sse="connect:/.sse" class="mt-1 flex flex-col font-mono text-sm">
<div hx-sse="swap:message" hx-swap="afterbegin"></div>
{{ range .Events }} {{template "row.html" . }} {{ else }}
<div class="py-2">There are currently no events in your database.</div>
{{ end }}
</div>

The hx- attributes will results in a connection to the path /.sse endpoint and listen for new server data. As soon as an SSE event is retrieved, it is prepended before the already rendered elements. Because this will only add a DOM element, you can also keep your selected text in the other rows.

On the server side, there is a custom notification/pubsub handler holding a list of all connected listeners. When a new event is inserted, the notifier will send this event to all listeners, which will then render it:


func (s *Server) handleSSE() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/event-stream")
w.Header().Set("Cache-Control", "no-cache")
w.Header().Set("Connection", "keep-alive")
w.Header().Set("Access-Control-Allow-Origin", "*")
flusher, _ := w.(http.Flusher)
// ticker for keepalive
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
// subscribe to event publisher
notifyClient := s.Notifier.AddListener()
defer s.Notifier.RemoveListener(notifyClient)
for {
select {
// if a new event is received, flush it to the sse-connected client
case event := <-notifyClient.c:
// print in the format 'data:<html>' and remove newslines
var b bytes.Buffer
fmt.Fprintf(w, "data:")
rowTmpl.Execute(&b, event)
fmt.Fprintf(w, "%s\n\n", strings.ReplaceAll(b.String(), "\n", ""))
flusher.Flush()
// keepalive stuff
case <-ticker.C:
fmt.Fprintf(w, "keepalive: \n\n")
flusher.Flush()
case <-r.Context().Done():
return
}
}
}
}

Using htmx allowed to leave out all manual javascript handling. This usecase I have here is so simple, it would be even worth removing the htmx dependency and writing the few lines of Javascript for listening by myself, but thats for another time.

Using tevents

tevents can now be used with a simple HTTP request. For example, let's assume there is some script which you want to log its start, end and exit code:


#!/bin/bash
curl http://tevents/.log?origin=worknode -d "started cleanup"
./do_cleanup.sh
curl http://tevents/.log?origin=worknode -d "finished cleanup: $?"

The other usecase I'm currently using tevents for is to watch cron jobs. The monitor event is only sent to tevents if the backup.sh script did not fail (return with a non-zero exit code):


0 1 * * * /usr/local/bin/backup.sh && curl -X POST http://tevents/.monitor?origin=cron:backup

This makes some nice visualizations which show the last execution of the backup script:

Screeshot of tevents event logger and job monitor for tailnets

tevents is very early in development. Feedback is always appreciated, and of course, contributions too!

As always, make sure to review any code before running it on your computer.