The ELK Stack: What Each Piece Actually Does

In my last post I wrote about structured logging — turning Error: something went wrong into a JSON event with real fields. That's step one. Step two is putting those events somewhere you can actually search, filter, and graph them. For a lot of teams, that somewhere is ELK.

ELK gets thrown around as one word, but it's four separate tools that happen to ship together. Knowing what each one does — and doesn't do — saves you from the most common mistake I see: standing up the whole stack because "that's what ELK is," when half of it isn't doing anything useful for your setup.

Elasticsearch: the part that actually stores and searches

Elasticsearch is a document store built on Lucene, with an HTTP API in front of it. You send it JSON documents, it indexes them, and you get fast search and aggregation back. For logs specifically, this means: send a log line as a JSON document, and later ask "how many 500s did the payment service return between 2 and 3pm, broken down by region" and get an answer in milliseconds instead of grepping across a dozen servers.

The mechanism underneath is an inverted index — for every value in every field, Elasticsearch keeps a list of which documents contain it. That's why full-text search over millions of log lines is fast: it's not scanning, it's a lookup.

Elasticsearch is also the only piece of the four that's stateful. It's the thing you have to size, shard, back up, and lose sleep over. The rest of this series spends more time here than anywhere else for that reason.

Logstash: the pipeline that shapes data on the way in

Logstash is an ingest pipeline with three stages: input, filter, output.

input {
  beats { port => 5044 }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "web-logs-%{+YYYY.MM.dd}"
  }
}

It reads from somewhere (Beats, Kafka, a file, syslog), transforms the data (parse unstructured text into fields, rename things, drop noise, enrich with a GeoIP lookup), and writes it somewhere (usually Elasticsearch, but not always).

The thing worth knowing early: Logstash is a JVM process, and it's not lightweight. Running it on every server you want to monitor is a common early mistake — it's meant to run as a small number of central pipeline nodes, not as an agent on every box.

Beats: the lightweight agents that actually run everywhere

Beats are the answer to "then what runs on every server?" Filebeat tails log files and ships them. Metricbeat collects system and service metrics. Packetbeat, Heartbeat, and others cover more specific cases. They're small, single-purpose Go binaries with a low resource footprint, built to sit on thousands of hosts without anyone noticing they're there.

A common and perfectly reasonable pipeline is Filebeat on every server, shipping straight to Elasticsearch, with Logstash only in the mix where you actually need transformation logic that Beats' lighter processors can't do. Not every setup needs Logstash at all.

Kibana: the window into everything else

Kibana is the UI. Discover for ad hoc searching through raw documents, Visualize and Dashboards for charts and aggregations, Alerting for rules that fire when a condition is met. It talks to Elasticsearch's API and doesn't store data of its own beyond its own configuration.

It's also the piece most likely to turn into an unmaintained pile of forgotten dashboards if nobody owns it — more on that in the Kibana post later in this series.

How it actually fits together

A typical flow, log line to dashboard:

Application → Filebeat (tails the log file)
            → Logstash (optional: parse, enrich, filter)
            → Elasticsearch (index, store)
            → Kibana (search, visualize, alert)

Any of these can be swapped. Filebeat can write straight to Elasticsearch. Logstash can read from Kafka instead of Beats — which, if you read my last post, is exactly the "Kafka as the event backbone" pattern: Kafka holds the log stream, Logstash (or a Kafka-native ingest pipeline) consumes it and writes to Elasticsearch downstream.

When ELK is the right tool

Reach for it when you need full-text search over log content, ad hoc exploration across many fields, and a UI a non-engineer can use to answer their own questions during an incident. It's genuinely good at all three.

When it's the wrong tool

If you mostly need metrics and a handful of well-known queries, a time-series database with lower operational cost — Prometheus plus Grafana, for instance — will get you there with less to run. Elasticsearch's flexibility comes with real weight: JVM heap tuning, shard management, disk watermarks. I'll get into exactly what that costs you in production two posts from now. If you don't need the flexibility, you don't need the weight.

Where this series goes next

Next up: indexing and mappings — the part of Elasticsearch that will quietly wreck your cluster if you let it run on defaults, and how to not let that happen.