LearnViz CompSci
Home / Learn / Math for Computing / Floating Point Error

Advanced Topic 41 of 51 Math for Computing

Floating Point Error

Rounding and precision limits in machine arithmetic.

How it works

Floating-point numbers store a limited number of significant digits, so most real numbers are rounded, and arithmetic results are rounded again. Errors include rounding error bounded by machine epsilon, about 1e-16 for doubles, cancellation when subtracting nearly equal numbers loses significant digits, absorption when adding a tiny number to a huge one has no effect, overflow and underflow and non-associativity: (a + b) + c may differ from a + (b + c). Special values include infinity and NaN.

Where it is used

Scientific computing, financial calculations, graphics, game physics and machine learning must manage floating-point error, using techniques like Kahan summation, compensated algorithms and higher precision.

Common pitfall

Comparing floats for equality, accumulating many small values in single precision and using floats for currency cause bugs. Results can differ across compilers, hardware and optimization settings. Use relative and absolute tolerances for comparisons, sort or pairwise sum data, rearrange formulas to avoid cancellation and use decimal types for money.

Where it fits in the course

Floating Point Error is topic 41 of 51 in Math for Computing, rated Advanced. It follows Numerical Methods. The next topic is Root Finding.

Practice and explore

How to study this topic

  1. Read the explanation above and restate the definition in your own words.
  2. Open the interactive lesson and answer its mini quiz.
  3. Read the common pitfall, then write a short example that avoids it.
  4. Check your understanding in the Math for Computing quiz, and revisit the topic if you miss a question.

Frequently asked questions

What is Floating Point Error?
Rounding and precision limits in machine arithmetic. It belongs to Math for Computing and is an advanced-level topic.
Where do people use Floating Point Error?
Scientific computing, financial calculations, graphics, game physics and machine learning must manage floating-point error, using techniques like Kahan summation, compensated algorithms and higher precision.
What is the most common mistake with Floating Point Error?
Comparing floats for equality, accumulating many small values in single precision and using floats for currency cause bugs.
What should I learn before this topic?
Start with Numerical Methods, the topic just before it in Math for Computing.
How can I practice it?
Answer the mini quiz in the interactive lesson, then take the Math for Computing quiz. You can also explore Binary & Two's Complement in the Visual Lab.
Is Floating Point Error hard to learn?
It is an Advanced topic. Expect to read the explanation more than once and to connect it with earlier topics. Working through a small example by hand helps most.
How long does it take to learn Floating Point Error?
Most learners can read the explanation and finish the interactive lesson in under an hour. Fluency comes from applying the idea, so plan a few short practice sessions with the quiz.
Which topics are related to Floating Point Error?
Numerical Methods, Root Finding, Simplex Method are closely connected. The full list is in the related topics section of this page.
What should I study after Floating Point Error?
Continue with Root Finding, the next topic in Math for Computing.

Related topics