Sooner or later you need to answer a question that grep cannot: which service is slow, how often a given error happens, what changed at 14:05. The Elastic Stack answers those questions well, but a shared cluster is the wrong place to learn it, and a stack that only runs after an afternoon of certificates and manual password resets is one you will never rebuild.

This guide builds a single Compose project that runs Elasticsearch, Logstash and Kibana on your machine, with authentication switched on, memory limits that hold, and data that survives restarts. Every file is shown in full, and all of it was tested against Elastic Stack 9.5.3.

By the end you will have:

  • Elasticsearch with authentication, a persistent volume and a memory budget it cannot exceed.
  • A one-shot setup container that creates the users each component needs, so a fresh start is a single command.
  • Logstash accepting JSON log events over HTTP and writing them to a data stream.
  • Kibana to search, filter and chart those events.

Prerequisites

  • Docker Desktop, or Docker Engine with the Compose plugin.
  • About 6.5 GB of disk for the three images.
  • At least 6 GB of memory assigned to Docker. The limits in this guide add up to 4.5 GB, and the rest is headroom for Docker itself. On Docker Desktop this lives under Settings → Resources.

On a Linux host, also raise the kernel's memory map limit, which Elasticsearch relies on for its index files. Docker Desktop's VM already ships with this value, so macOS and Windows users can skip it.

terminal
sudo sysctl -w vm.max_map_count=262144
# Persist it across reboots
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-elasticsearch.conf

The project layout

elk/
elk/
├── .env
├── docker-compose.yml
├── setup/
│   └── setup.sh
└── logstash/
    └── pipeline/
        └── logstash.conf

Four services run from it, and the order in which they start is the whole trick:

  1. elasticsearch stores and indexes everything. Nothing else starts until it reports healthy.
  2. setup is a one-shot container. It sets passwords and creates a user for Logstash, then exits.
  3. kibana and logstash start only after setup has exited successfully.

1. Pin the version and the secrets

Compose reads .env automatically, which makes it the one place to change the version or a password.

.env
STACK_VERSION=9.5.3

# Local-only credentials. Change them if anyone else can reach the stack.
ELASTIC_PASSWORD=elastic-dev-password
KIBANA_PASSWORD=kibana-dev-password
LOGSTASH_PASSWORD=logstash-dev-password

# Any random string of 32+ characters, e.g. from `openssl rand -hex 32`
KIBANA_ENCRYPTION_KEY=c4f1b0e9a27d4e8f93b6a5d2e17c08f4b9a3d6e2f1c5a8b7

The version is a single variable because Elasticsearch, Kibana and Logstash are released together, and running them at the same version is the only combination Elastic supports. Moving to a newer release is a one-line change. Going back is not: a data directory written by a newer Elasticsearch cannot be opened by an older one.

2. The Compose file

Here is the whole file. The sections below explain each service and why the settings are what they are.

docker-compose.yml
name: elk

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    environment:
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      # Heap at half the container limit, the rest is for Lucene and the OS
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    mem_limit: 2g
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - 127.0.0.1:9200:9200
    healthcheck:
      test: ["CMD-SHELL", "curl -fs -u elastic:${ELASTIC_PASSWORD} 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s'"]
      interval: 10s
      timeout: 10s
      retries: 30

  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_PASSWORD=${KIBANA_PASSWORD}
      - LOGSTASH_PASSWORD=${LOGSTASH_PASSWORD}
    volumes:
      - ./setup/setup.sh:/setup.sh:ro
    command: ["bash", "/setup.sh"]
    restart: "no"

  kibana:
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    depends_on:
      setup:
        condition: service_completed_successfully
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${KIBANA_ENCRYPTION_KEY}
      - XPACK_SECURITY_ENCRYPTIONKEY=${KIBANA_ENCRYPTION_KEY}
    mem_limit: 1536m
    ports:
      - 127.0.0.1:5601:5601
    healthcheck:
      test: ["CMD-SHELL", "curl -fs http://localhost:5601/api/status"]
      interval: 10s
      timeout: 10s
      retries: 30

  logstash:
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    depends_on:
      setup:
        condition: service_completed_successfully
    environment:
      - LS_JAVA_OPTS=-Xms256m -Xmx256m
      - LOGSTASH_PASSWORD=${LOGSTASH_PASSWORD}
      - CONFIG_RELOAD_AUTOMATIC=true
    mem_limit: 1g
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - 127.0.0.1:8080:8080

volumes:
  esdata:

Elasticsearch

discovery.type=single-node makes the node elect itself master and stop looking for peers, which is exactly what a laptop cluster of one needs.

Security has been on by default since Elasticsearch 8, and it stays on here. Every request needs a username and password, just like on any real cluster, so nothing you build locally breaks when it meets production. What is turned off is TLS on the HTTP layer, so that curl and the other containers can speak plain HTTP. For a stack that only listens on localhost that trade-off is reasonable. For anything else it is not.

The data directory is a named volume, esdata, which Docker manages for you. Indices, users and Kibana's saved objects all live in it and survive docker compose down. The pitfalls section explains why this beats a bind mount.

Each port is published on 127.0.0.1 rather than on all interfaces. On Linux, Docker opens published ports with its own firewall rules, which go around tools like ufw, so 9200:9200 on a laptop at a conference means your cluster is on the conference Wi-Fi.

The healthcheck asks for the cluster health and waits for at least yellow. With -f, curl fails on any HTTP error, including the 408 that Elasticsearch returns when the status is not reached in time. The doubled $ stops Compose from filling in the variable itself, so the shell inside the container expands it from the container's own environment.

The setup container

Built-in users such as kibana_system start without a password, and something has to set one before Kibana can connect. Doing it by hand with docker exec works once and is forgotten by the next fresh volume. A one-shot container reuses the Elasticsearch image, since it already has curl, runs a script against the healthy cluster and exits.

setup/setup.sh
#!/usr/bin/env bash
# One-shot bootstrap: runs once Elasticsearch is healthy, then exits.
set -euo pipefail

# Usage: es METHOD PATH BODY, authenticated as the elastic superuser
es() {
  curl -fsS -u "elastic:${ELASTIC_PASSWORD}" -H 'Content-Type: application/json' \
    -X "$1" "http://elasticsearch:9200$2" -d "$3"
  echo
}

# Kibana connects as the built-in kibana_system user
es POST /_security/user/kibana_system/_password "{\"password\":\"${KIBANA_PASSWORD}\"}"

# Logstash gets a user that can only write to the logs-app-* data streams
es PUT /_security/role/logstash_writer '{
  "cluster": ["monitor"],
  "indices": [{ "names": ["logs-app-*"], "privileges": ["create_doc", "auto_configure"] }]
}'
es PUT /_security/user/logstash_internal "{
  \"password\": \"${LOGSTASH_PASSWORD}\",
  \"roles\": [\"logstash_writer\"]
}"

# A single node cannot host replicas; without this every logs-* data stream stays yellow
es PUT /_component_template/logs@custom '{
  "template": { "settings": { "index.number_of_replicas": 0 } }
}'

echo "Setup complete"

Every call is idempotent, so the script runs again on each docker compose up without harm, and it reapplies the passwords from .env each time. If a call fails, set -e and curl -f make the container exit with an error, and Kibana and Logstash never start. A loud failure at startup beats a Kibana that loops on authentication errors.

The last call deserves a word. Data streams named logs-*-* use a built-in template that asks for one replica per shard, and a one-node cluster has nowhere to put it, so the cluster reports yellow forever. logs@custom is the hook that the built-in template reserves for exactly this kind of override.

Kibana

Kibana authenticates as kibana_system with the password that setup just set. The two encryption keys are optional on paper, but without them Kibana disables alerting and connectors, Fleet retries its setup in a loop, and every Kibana restart logs you out, because it generates a new session key each time it boots. One key reused for both is fine on a laptop.

There is no volume. Kibana keeps its data views, dashboards and saved searches as documents in Elasticsearch, so they already live in esdata.

The limit is 1.5 GB because Kibana 9 idles at around 1.2 GB. With a 1 GB limit it ran within a few megabytes of the ceiling in testing, which ends in an out-of-memory kill the first time you open a heavy dashboard.

Logstash

Logstash's default heap is 1 GB. A pipeline that forwards a few events per second is happy with 256 MB, so LS_JAVA_OPTS shrinks it. The image maps environment variables onto logstash.yml settings, so CONFIG_RELOAD_AUTOMATIC=true turns on config.reload.automatic: edit the pipeline file and Logstash reloads it within seconds, with no restart.

The pipeline itself is short:

logstash/pipeline/logstash.conf
input {
  # Accept JSON log events over HTTP: POST http://localhost:8080
  http {
    port => 8080
  }
}

filter {
  # Drop the request metadata added by the http input and the raw copy of the body
  mutate {
    remove_field => ["http", "url", "user_agent", "host", "event"]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    user => "logstash_internal"
    password => "${LOGSTASH_PASSWORD}"
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "app"
  }
}

The http input decodes JSON bodies on its own, so every field you send becomes a field in the event. ${LOGSTASH_PASSWORD} is read from the container's environment, which keeps the secret in .env rather than in the pipeline.

The output writes to a data stream called logs-app-default rather than to hand-named daily indices. Data streams that match logs-*-* pick up a built-in template with an @timestamp mapping, sensible defaults for log fields and a lifecycle policy that rolls the stream over to a fresh backing index as it grows. That is work you would otherwise do yourself, and it is the layout Elastic's own agents use, so your events end up next to theirs if you add them later.

3. Start the stack

terminal
docker compose up -d --wait

--wait blocks until every service with a healthcheck is healthy and exits with an error if one of them fails, which makes the command safe to script. The first run downloads the images. After that, a cold start takes about a minute.

Check that setup ran and everything else is up:

terminal
docker compose ps -a --format 'table {{.Service}}\t{{.Status}}'
output
SERVICE         STATUS
elasticsearch   Up About a minute (healthy)
kibana          Up 37 seconds (healthy)
logstash        Up 37 seconds
setup           Exited (0) 37 seconds ago

Exited (0) for setup is the success state. Then ask Elasticsearch about itself:

terminal
curl -s -u elastic:elastic-dev-password 'http://localhost:9200/_cluster/health?pretty'

Look for "status" : "green". Quote the URL as shown, since zsh treats the ? as a wildcard and refuses to run the command otherwise.

4. Send some logs

Post a JSON event to Logstash. It answers ok as soon as the event is queued.

terminal
curl -X POST http://localhost:8080 \
  -H 'Content-Type: application/json' \
  -d '{"level":"error","service":"checkout","message":"Payment provider timed out","duration_ms":5012}'

A few more, so that there is something to compare:

terminal
for event in \
  '{"level":"info","service":"checkout","message":"Order placed","duration_ms":182}' \
  '{"level":"warn","service":"auth","message":"Password retry limit reached","duration_ms":45}' \
  '{"level":"info","service":"auth","message":"User signed in","duration_ms":61}'
do
  curl -s -X POST http://localhost:8080 -H 'Content-Type: application/json' -d "$event"
done

Now search for the error:

terminal
curl -s -u elastic:elastic-dev-password \
  'http://localhost:9200/logs-app-default/_search?q=level:error&pretty'
output (trimmed to the document)
"_source" : {
  "duration_ms" : 5012,
  "@timestamp" : "2026-09-11T14:35:11.516424545Z",
  "level" : "error",
  "service" : "checkout",
  "data_stream" : {
    "namespace" : "default",
    "type" : "logs",
    "dataset" : "app"
  },
  "@version" : "1",
  "message" : "Payment provider timed out"
}

Logstash added the @timestamp and the data_stream fields. duration_ms was mapped as a number, which means you can aggregate on it. ES|QL, Elasticsearch's piped query language, turns those log lines into latency figures per service:

terminal
curl -s -u elastic:elastic-dev-password 'http://localhost:9200/_query?format=txt' \
  -H 'Content-Type: application/json' \
  -d '{"query": "FROM logs-app-* | STATS requests = COUNT(*), avg_ms = AVG(duration_ms), p95_ms = PERCENTILE(duration_ms, 95) BY service"}'
output
   requests    |    avg_ms     |    p95_ms     |    service
---------------+---------------+---------------+---------------
2              |53.0           |60.2           |auth
2              |2597.0         |4770.5         |checkout

5. Explore in Kibana

Open http://localhost:5601 and log in as elastic with the password from .env.

  1. Open Discover. With no data view yet, Kibana offers to create one. Use logs-app-* as the index pattern and @timestamp as the timestamp field.
  2. Filter with KQL, for example level : "error" and service : "checkout". Click a field in the sidebar to see its top values at a glance.
  3. Switch Discover to ES|QL and paste the FROM logs-app-* | STATS ... query from above. The same query that ran in the terminal renders as a table and a chart.
  4. From any field, Visualize opens Lens. Average duration_ms broken down by service is the classic first chart.

Pitfalls worth knowing in advance

Everything above works on the first try. The problems below are the ones that show up a week later, and most of them look unrelated to their cause.

Memory limits

Exit code 137 means the kernel killed the container. When the JVM heap plus everything outside it grows past mem_limit, the container dies without a Java stack trace, and the logs simply stop. Confirm it with:

terminal
docker compose ps -a
docker inspect -f '{{.State.OOMKilled}}' elk-elasticsearch-1

The fix is almost never "more heap". Elasticsearch needs memory outside the heap for Lucene's memory-mapped files, network buffers and thread stacks, which is why the heap is set to half the container limit. Raising -Xmx to fill the limit is the fastest way to trigger the kill you are trying to avoid. Raise both numbers together, and keep the ratio.

Without a limit, Elasticsearch sizes itself to the machine. If you drop ES_JAVA_OPTS, Elasticsearch picks a heap from the memory it can see. That is the container limit when there is one, and the whole Docker VM when there is not. The same laptop then behaves differently depending on what else happens to be running.

Limits do not help if the VM is too small. If the limits add up to more than the memory given to Docker Desktop, the VM itself runs out, and containers get killed seemingly at random. Keep the sum of mem_limit values below the VM's memory with a gigabyte or two to spare.

Volumes

Prefer a named volume over a bind mount. Elasticsearch runs as user 1000 and group 0 inside the container. On Linux, a bind-mounted directory keeps its ownership from the host, so unless it happens to belong to uid 1000, the node fails at startup with an AccessDeniedException on its data path. If you do want the data in a folder you can see, give group 0 access to it first:

terminal
mkdir -p esdata
chmod g+rwx esdata
sudo chgrp 0 esdata

On macOS the permissions usually work, but every read and write crosses Docker Desktop's file-sharing layer, which is noticeably slower than a named volume living inside the VM.

docker compose down -v deletes everything. down removes the containers and keeps esdata. down -v removes the volume too, and with it every index, user, role and Kibana saved object. The setup container recreates the users on the next start. Your dashboards are not recreated, so export the ones you care about first.

A full disk makes indices read-only. Elasticsearch watches the free space of the disk it runs on, which in Docker Desktop is the VM's virtual disk, shared with every image, container and volume. Past the flood-stage watermark, 95% by default, it blocks writes, and Logstash starts logging cluster_block_exception errors. Free space, with docker system df to see where it went, and the block lifts by itself once usage drops. For a throwaway local cluster you can also turn the check off by adding cluster.routing.allocation.disk.threshold_enabled=false to the Elasticsearch environment.

Passwords

ELASTIC_PASSWORD is a bootstrap password. Elasticsearch reads it on every start, and it keeps working until you change the elastic password through the API or Kibana's user management. From that moment the stored password wins and the .env value is rejected. The symptom is confusing: the healthcheck gets a 401, Elasticsearch never becomes healthy, and Kibana and Logstash wait forever. Put the new password in .env and run docker compose up -d again.

The other two passwords have no such trap. Change KIBANA_PASSWORD or LOGSTASH_PASSWORD in .env, run docker compose up -d, and setup applies them before the recreated containers connect.

Where to go next

  • Metrics: add Metricbeat or Elastic Agent with the Docker integration, and container CPU and memory land in the same cluster as your logs.
  • Plain-text logs: not every application writes JSON. The grok and dissect filters in Logstash turn log lines into fields.
  • TLS: before the stack leaves localhost, generate a CA and certificates with elasticsearch-certutil and turn xpack.security.http.ssl back on.
  • A clean slate: docker compose down -v && docker compose up -d --wait rebuilds the whole stack from nothing in about a minute. That is the real benefit of putting the setup in a script.