LearnViz CompSci
Home / Learn / Data Structures / HyperLogLog

Advanced Topic 49 of 52 Data Structures

HyperLogLog

Probabilistic counter estimating distinct elements.

How it works

HyperLogLog estimates the number of distinct elements in a very large stream using tiny memory. It hashes each item, uses the first bits to pick a register and records the maximum number of leading zeros seen there. A harmonic mean over registers gives the estimate, with a standard error of about 1.04 divided by the square root of the register count, so a few kilobytes handle billions of items.

Step-by-step walkthrough

  1. Each item is hashed to a random bit string.
  2. Leading zeros are counted, a long run is rare.
  3. Registers keep the maximum run seen.
  4. Many distinct items make large maxima likely.
  5. A harmonic average of registers gives the distinct count within a few percent using kilobytes.

Where it is used

Counting unique visitors, distinct queries in analytics, database cardinality estimation and monitoring systems use HyperLogLog, built into Redis and BigQuery. Sketches can be merged by taking register-wise maximums.

Common pitfall

Estimates are approximate, with errors of one or two percent, and small cardinalities need correction. A poor hash function destroys accuracy. It cannot enumerate members or support deletion. Do not use it when exact counts are legally or financially required.

Where it fits in the course

HyperLogLog is topic 49 of 52 in Data Structures, rated Advanced. It follows Count-Min Sketch. The next topic is Merkle Trees.

Practice and explore

How to study this topic

  1. Read the explanation above and restate the definition in your own words.
  2. Open the interactive lesson and answer its mini quiz.
  3. Read the common pitfall, then write a short example that avoids it.
  4. Check your understanding in the Data Structures quiz, and revisit the topic if you miss a question.

Frequently asked questions

What is HyperLogLog?
Probabilistic counter estimating distinct elements. It belongs to Data Structures and is an advanced-level topic.
Where do people use HyperLogLog?
Counting unique visitors, distinct queries in analytics, database cardinality estimation and monitoring systems use HyperLogLog, built into Redis and BigQuery.
What is the most common mistake with HyperLogLog?
Estimates are approximate, with errors of one or two percent, and small cardinalities need correction.
What should I learn before this topic?
Start with Count-Min Sketch, the topic just before it in Data Structures.
How can I practice it?
Answer the mini quiz in the interactive lesson, then take the Data Structures quiz.
Is HyperLogLog hard to learn?
It is an Advanced topic. Expect to read the explanation more than once and to connect it with earlier topics. Working through a small example by hand helps most.
How long does it take to learn HyperLogLog?
Most learners can read the explanation and finish the interactive lesson in under an hour. Fluency comes from applying the idea, so plan a few short practice sessions with the quiz.
Which topics are related to HyperLogLog?
Count-Min Sketch, Merkle Trees, Cuckoo Hashing are closely connected. The full list is in the related topics section of this page.
What should I study after HyperLogLog?
Continue with Merkle Trees, the next topic in Data Structures.

Related topics