Skip to content

Data

Elasticsearch

Fast, relevant search and centralised logging, built by engineers who run the clusters they ship.

Overview

Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene. At its core sits the inverted index — a structure that maps every term to the documents that contain it, which is what makes full-text search return relevant results across millions of documents in milliseconds rather than scanning row by row. Around that core Elasticsearch adds a JSON document model, a REST API, and the machinery to spread data across many machines and keep serving when some of them fail. It is the engine that powers search in a great many products you use, and the store behind a great many logging and observability platforms.

There are really two jobs people reach for Elasticsearch to do, and it is worth being clear which one you are buying. The first is search inside a product: an e-commerce catalogue, a documentation site, a support tool, anywhere users type a query and expect fast, forgiving, relevance-ranked results with typo tolerance, faceting and highlighting. The second is centralised logging and observability — the Elastic Stack, often called ELK, pairing Elasticsearch with Logstash and Beats to collect logs and metrics and Kibana to search and visualise them. The engine is the same; the data model, the tuning and the operational profile are quite different, and conflating the two is a common way to design a cluster that fights you.

What Elasticsearch is not is a system of record. It is eventually consistent, its durability and transactional guarantees are weaker than a database built for the job, and its real strength is reading and ranking rather than being the authoritative home for data you cannot lose. We treat it as an index you feed from a durable primary store — usually PostgreSQL — rather than the place your data lives. Used that way, with a source of truth elsewhere and Elasticsearch kept in sync, it is a superb search and analytics layer. Used as your only copy of the data, it becomes a liability the first time a shard goes missing or a reindex goes wrong.

Best for — Products that need fast, relevant full-text search, and organisations centralising logs and events for observability and analytics — where an index fed from a durable primary store earns the cost of running a cluster.

Why teams choose Elasticsearch

  • Search that feels instant and forgiving

    The inverted index turns full-text queries across millions of documents into millisecond responses, and the relevance model, typo tolerance and faceting mean users find things even when they misspell or half-remember them. That is the difference between a search box people trust and one they give up on and email support instead.

  • One place to see what production is doing

    Pulling logs, metrics and events from every service into a single searchable store — with Kibana on top — turns debugging a live incident from grepping across machines into one query. When something breaks at 2am, having the answer in one searchable place rather than scattered across servers is worth a great deal.

  • Analytics in near real time

    Aggregations let you bucket, count and summarise huge document sets fast enough to power interactive dashboards, so questions your team currently answers with an overnight batch job can be answered live. The data is searchable within a second or so of being indexed, which keeps those dashboards genuinely current.

Why businesses choose Elasticsearch

  • Your product’s search has outgrown what your database can do, and slow or irrelevant results are costing you users who cannot find what they came for.
  • You are running enough services that logs scattered across machines have become a liability, and you want one searchable, visualised place to understand production.
  • You need analytics dashboards over large, semi-structured data that must feel live rather than being computed overnight.
  • You want a team that will tell you honestly whether you actually need a cluster — and, if you do, will run it properly rather than leaving you a fragile black box.

What we build with Elasticsearch

The capabilities this technology is genuinely strong at — and what we most often build with it.

  • Inverted index and relevance ranking

    The inverted index is what makes Elasticsearch fast, and its scoring — term frequency weighed against how common a term is, tuned with boosting and custom analysers — is what makes results relevant. We tune analysers, synonyms and scoring to your domain, because default relevance is rarely the relevance your users actually want.

  • Aggregations for analytics

    Aggregations bucket, count, average and nest over huge document sets in near real time, turning Elasticsearch into an analytics engine as much as a search one. We use them to power dashboards and facets that stay interactive where a relational query would time out.

  • The Elastic Stack for logging

    For observability we assemble the full stack deliberately: Beats and Logstash to collect and shape logs and metrics, Elasticsearch to store and search them, and Kibana to explore and alert. We design index lifecycle and retention up front, because unbounded log indices are the classic way a cluster fills its disks.

  • Near-real-time indexing

    Documents become searchable within roughly a second of being indexed, which is what lets search and dashboards reflect current data. We design the indexing pipeline — from a primary store, often via a queue such as Kafka — so writes flow in reliably without hammering the cluster.

  • Sharding and replication

    Each index is split into shards distributed across the cluster, with replica shards for redundancy and read throughput. Getting shard count and size right is one of the most consequential decisions in an Elasticsearch design, and one of the most commonly botched, so we size it to your data and growth rather than to a default.

  • Vector search and kNN

    Recent versions add dense vector fields and kNN search, so Elasticsearch can do semantic search and serve as the retrieval layer for RAG alongside its traditional keyword search. Where it fits, we combine the two — keyword precision with semantic recall — rather than treating vector search as a wholesale replacement.

Use cases

  • Product and catalogue search

    Fast, relevance-ranked, typo-tolerant search over an e-commerce catalogue, documentation set or support knowledge base, with facets, filters and autocomplete that let users narrow to what they want in a few keystrokes.

  • Centralised logging and observability

    Logs, metrics and events from every service pulled into one searchable store with Kibana on top, so incidents are diagnosed with a query rather than by grepping across machines, and retention is bounded by design.

  • Live analytics dashboards

    Aggregations over large, semi-structured datasets driving dashboards that slice and summarise millions of documents in near real time — business metrics, security events, usage analytics — without an overnight batch.

  • Semantic search and RAG retrieval

    Using dense vector fields and kNN, often combined with keyword search, to power semantic search and to retrieve relevant context for a language model, with the authoritative documents kept in a durable primary store.

When Elasticsearch is the right choice

  • You need genuine full-text search inside a product — relevance ranking, typo tolerance, faceting, autocomplete, highlighting across large volumes of text — and you have outgrown what a relational database can do. This is Elasticsearch’s home ground and where it clearly earns its keep.
  • You are aggregating logs, metrics or events from many services into one place to search, alert on and visualise them. The Elastic Stack is a mature, well-trodden answer to centralised logging and observability, and Kibana gives your team a real window into what production is doing.
  • You need analytics over large, semi-structured datasets that Elasticsearch’s aggregations are built for — dashboards that slice, count, bucket and summarise millions of documents in near real time, where a batch query against a relational store would be too slow to feel interactive.
  • Wrong for: your primary system of record. Elasticsearch is eventually consistent and not a transactional database. Keep the authoritative copy in PostgreSQL and index into Elasticsearch — never let the cluster be the only place your data exists.
  • Wrong for: simple search needs. If you want to match a few fields on a modest dataset, PostgreSQL full-text search or a hosted search service will very likely do the job with none of the operational burden. Do not stand up a cluster you must then operate unless the search or logging need genuinely justifies it.

Elasticsearch: pros and cons

Strengths

  • Full-text search that is fast and relevant at scale — the inverted index, a mature relevance model, typo tolerance, faceting and highlighting are exactly what a relational database is bad at and Elasticsearch is built for.
  • Powerful aggregations turn the same engine into an analytics tool, summarising and bucketing millions of documents in near real time to drive interactive dashboards.
  • It scales horizontally by design: data is split into shards spread across a cluster, with replicas for redundancy, so you add capacity and resilience by adding nodes rather than rewriting the application.
  • The Elastic Stack is a complete, well-supported ecosystem for logging and observability — Logstash and Beats to collect, Elasticsearch to store and search, Kibana to visualise — with a huge community and deep documentation behind it.

Trade-offs

  • It is not a primary system of record. Elasticsearch is eventually consistent with weaker durability and no real transactions, so you keep the source of truth elsewhere and index into it — which means owning the synchronisation between the two.
  • Operating a cluster well is genuinely hard. Shard sizing, heap and memory management, cluster health, hot nodes and reindexing are all real skills, and a cluster left on defaults tends to be fine until it suddenly is not.
  • It is resource-hungry. Elasticsearch wants a lot of RAM — much of its speed depends on data sitting in memory and the filesystem cache — so the hosting bill is meaningfully higher than a modest database, and undersizing shows up as instability.
  • For simple needs it is overkill. Standing up and running a cluster to power a basic search box is a lot of operational cost for a problem PostgreSQL full-text search or a hosted service would solve, and reaching for it reflexively is a common mistake.

Architecture

The first architectural decision is that Elasticsearch is an index, not a database. The authoritative data lives in a durable primary store — typically PostgreSQL — and is projected into Elasticsearch in a shape optimised for search or analytics rather than for transactions. That means owning the synchronisation path: how a change in the primary store becomes a change in the index, how you reindex without downtime, and how you recover when the two drift apart. We design that path explicitly, frequently routing writes through a queue such as Kafka so the cluster is fed at a rate it can absorb and a burst of writes never destabilises search.

Inside the cluster, the load-bearing choices are the index mappings and the shard layout. Mappings decide how each field is analysed and therefore how it can be searched and aggregated, and they are painful to change after the fact, so we settle them deliberately. Shard count and size govern how work spreads across nodes and how the cluster behaves as data grows; too many small shards waste memory, too few large ones create hot nodes and slow recovery. For logging we add index lifecycle management so indices roll over by age or size and old data ages out to cheaper storage or is deleted, rather than growing until the disks fill.

Performance

Elasticsearch’s speed rests on data being resident in memory — both the JVM heap and, crucially, the operating system’s filesystem cache that Lucene relies on. Most performance problems trace back to that: heap sized wrong, too many shards each carrying fixed overhead, or a working set that no longer fits in RAM so queries start hitting disk. We size heap and memory to the real data, keep shard counts sane, and measure query latency and cache hit rates rather than guessing, because on a cluster the difference between healthy and struggling is often a configuration detail rather than raw hardware.

On the query side, performance is about asking the engine the right questions. Filters that can be cached, aggregations bounded to what a dashboard actually needs, and mappings that avoid analysing fields that only ever need exact matching all make a large difference. For search relevance we tune rather than accept defaults, and for logging we tune ingestion so indexing throughput keeps up without starving search. Where Elasticsearch is being asked to do something it is poor at, the honest answer is to move that work — heavy relational joins belong in PostgreSQL, not in a search index.

Security

The historical trap with Elasticsearch is the exposed cluster. For years default installations shipped without authentication, and unsecured clusters open to the internet have been the source of some very large data leaks. We never run a cluster without transport encryption, authentication and role-based access control enabled, and we keep it off the public internet, reachable only from the services that need it. This is table stakes, and the number of production clusters still running open is the reason we say it plainly.

Beyond the perimeter, we use role-based access control to limit what each service and person can read and write, so a logging pipeline cannot touch product indices and a read-only dashboard user cannot delete data. Because Elasticsearch is an index fed from a primary store, we also treat it as reproducible: the authoritative data and the secrets live in the durable store and the secret manager, and the cluster can be rebuilt and reindexed rather than being an irreplaceable single copy. That posture limits the blast radius if something does go wrong.

Scalability

Elasticsearch scales horizontally by spreading shards across nodes and adding replicas for read throughput and redundancy, so in principle you grow by adding machines. In practice that scaling is only as good as the shard design underneath it: because a shard’s data is fixed at index-creation time, capacity planning is something you do up front and revisit through reindexing, not something the cluster fully sorts out for you. We plan shard count and size against expected data volume and growth, and we separate concerns — dedicated master nodes for stability, data nodes for storage and query — as a cluster grows large enough to warrant it.

The two workloads scale differently. Logging is write-heavy and time-based, so it scales well with lifecycle management rolling indices over and tiering old data down to cheaper hardware. Product search is read-heavy and latency-sensitive, so it scales through replicas and careful query tuning. We size for each rather than treating them identically, and we are candid that scaling a cluster smoothly requires headroom and monitoring — a cluster run at the edge of its memory is one traffic spike away from turning red.

Elasticsearch integrations & ecosystem

The technologies we most often pair with it — each links to how we work with it.

How we work

We start by asking whether you need Elasticsearch at all. For a modest search box or a small dataset, PostgreSQL full-text search or a hosted search service will do the job without the cost of a cluster to operate, and we will say so rather than sell you infrastructure. When the search, logging or analytics need genuinely justifies it, we design the whole path — the primary store, the synchronisation, the mappings, the shard layout and the retention — before indexing a single document, because the expensive mistakes in Elasticsearch are the ones baked into mappings and shards early on.

When we build, the engineers who design the cluster are the ones who run it. That is what operating what we build means here: sensible shard sizing, monitored cluster health, bounded log indices, secured and authenticated nodes, and a reindexing story that works without downtime. We would rather hand you a cluster your own team can understand and keep healthy than a fast demo that becomes a fragile black box the first time it needs to grow or recover.

The service behind it

Delivered throughData Engineering

What we build with Elasticsearch

The disciplines this technology most often shows up in — from a first build to taking over and stabilising an existing one.

How we deliver

  1. 01

    Discover

    We map the system, the constraints and the business it serves — including the parts nobody documented.

    Architecture brief

  2. 02

    Architect

    Decisions get made, written down and defended before a line of production code exists.

    Decision records

  3. 03

    Build

    Short cycles against working software. You see progress in the product, not in a status deck.

    Shipping increments

  4. 04

    Operate

    Monitoring, incident response and iteration. The system is alive, so the engagement is too.

    Runbooks & SLOs

Industries we use Elasticsearch in

Domain knowledge changes what gets built. A few of the sectors we know before the first meeting.

Also in Data

Why teams choose us for Elasticsearch

  • Senior engineers only

    Elasticsearch is mostly judgement — shard sizing, mappings, relevance tuning, when a cluster is even the right answer. That is earned on previous systems, not learned on yours. No juniors treating a cluster as a search box you can leave on defaults and forget.

  • We operate what we build

    We run the clusters we ship, so we design for the day a shard goes missing, the week the log indices try to fill the disks, and the traffic spike that pushes memory to the edge — not just the demo. Monitored health, bounded indices, secured nodes and a working reindex path.

  • Honest about whether you need it

    We will tell you plainly when PostgreSQL full-text search or a hosted service would serve you better than a cluster you have to operate, and when Elasticsearch is being asked to be a database it should not be. We would rather leave the cluster out than saddle you with operational cost you did not need.

  • Clear on licensing and the OpenSearch fork

    The Elastic licence change and AWS’s OpenSearch fork are a real decision, not a footnote. We will walk you through which is right for your situation — features, licence terms, managed offerings and lock-in — rather than defaulting to whichever we happen to know best.

Typical timeline

  1. 01

    Discovery and design

    One to two weeks establishing whether a cluster is warranted, then designing the primary store, synchronisation path, mappings, shard layout and retention before any indexing begins.

  2. 02

    First index and pipeline

    One to two weeks building the indexing pipeline from the primary store, standing up the first index with real mappings, and getting either search or logging working end to end against real data.

  3. 03

    Tuning and rollout

    Iterating on relevance, aggregations and ingestion in short cycles — tuning analysers and scoring for search, or lifecycle and dashboards for logging — each change measured against real queries.

  4. 04

    Hardening and handover

    Securing and authenticating the cluster, sizing shards and memory for growth, setting up health and latency monitoring, proving a downtime-free reindex, and documenting so your team can own it.

How pricing works

  • Fixed-scope engagements for a defined piece of work — adding product search, standing up centralised logging, or building an analytics dashboard — quoted once we understand the data volumes and the search or observability requirement.
  • Monthly senior engagement for ongoing work where Elasticsearch is one part of a larger system and the scope evolves, covering indexing pipelines, relevance tuning and cluster operations over time.
  • Focused audits and rescues for existing clusters — a red or unstable cluster, runaway shard counts, log indices filling the disks, poor search relevance, or an exposed and insecure deployment — priced by assessment.

Hire Elasticsearch engineers

Need Elasticsearch capacity on your own team? We embed named senior engineers into your existing team — reporting to your leads, working in your rituals — so you add capacity without a hiring cycle.

Hire Elasticsearch engineers

Common questions

Can Elasticsearch be our main database?

For almost every application, no. Elasticsearch is eventually consistent, has weaker durability guarantees than a purpose-built database, and offers no real transactions, so it is not designed to be the authoritative home for data you cannot afford to lose. The right pattern is to keep the source of truth in a durable store such as PostgreSQL and index into Elasticsearch for search and analytics. Treating the cluster as your only copy of the data is the single most common way teams get Elasticsearch wrong, and it tends to reveal itself at the worst possible moment — during a reindex or a node failure.

Do we actually need Elasticsearch, or is PostgreSQL enough?

Often PostgreSQL is enough. Its full-text search handles matching across a few fields on a modest dataset perfectly well, and it comes with none of the burden of running a cluster. You genuinely need Elasticsearch when search has become a core feature — large text volumes, relevance ranking, typo tolerance, faceting, autocomplete — or when you are aggregating logs and events from many services. Our starting question is always whether the need justifies the operational cost, and we will recommend PostgreSQL or a hosted search service when it does, rather than selling you infrastructure you then have to keep alive.

What is the difference between Elasticsearch and OpenSearch?

In 2021 Elastic changed Elasticsearch’s licence away from permissive open source to source-available terms that restrict how managed providers can offer it. AWS responded by forking the last open-source version as OpenSearch, which stays under the permissive Apache 2.0 licence. The two share a common heritage and are broadly similar, but they have diverged over time in features, tooling and managed offerings, and Elastic later added an open-source licence option back for Elasticsearch itself. Which one fits depends on your licence requirements, the managed service you want and how much you care about avoiding lock-in — we will walk you through it honestly rather than defaulting to one.

Why does our cluster keep going yellow or red, or running out of memory?

Almost always one of a handful of causes: too many shards, each carrying fixed memory overhead, so the heap is exhausted; a working set that no longer fits in RAM, so queries start hitting disk and slow down; unbounded log indices growing until the disks fill; or heap sized wrong for the data. A red or yellow status usually means shards are unassigned — often after a node dropped out or during a strained recovery. Each has a concrete fix, from consolidating shards and adding lifecycle management to resizing memory, and an audit finds which one you actually have. Unstable clusters are one of the most common rescues we do.

Can Elasticsearch do semantic search and support RAG?

Yes. Recent versions add dense vector fields and kNN search, so Elasticsearch can find documents by semantic similarity as well as by keyword, and serve as the retrieval layer feeding context to a language model in a RAG system. In practice the strongest results usually come from combining the two — keyword search for precision and vector search for recall — rather than replacing one with the other. As always, the authoritative documents stay in a durable primary store and are indexed into Elasticsearch, so the vector index is a fast retrieval layer rather than the only copy of your knowledge base.

Building on Elasticsearch?

A technical conversation with the engineers who would do the work. If we are not the right fit, we will say so on the call.

Prefer to book a call?Or email [email protected]

Two fields required. We reply to real enquiries — no list, no sequence.