Projects

Ticket #521 (closed defect: duplicate)

Opened 2 years ago

Last modified 14 months ago

Float problem with 8.1 and 9.1

Reported by: macruby@… Owned by: lsansonetti@…
Priority: major Milestone:
Component: MacRuby Keywords:
Cc:

Description

MacRuby 0.5b2 seems to have a problem with the floats 8.1 and 9.1. Here is what 'macirb' does:

irb(main):005:0> 9.1 => 9.09999999999999 irb(main):006:0> 8.1 => 8.09999999999999

I get the same results from 'macruby'. The strange thing is that if I do the following:

irb(main):007:0> 9.1 + 1 => 10.1 irb(main):008:0> 10.0 - 0.9 => 9.09999999999999

very strange - AYAL

Change History

Changed 2 years ago by emoy@…

Actually, this should be fixed with the float changes in #159. This problem stems from the last two bits of the floating point number being used as flags, so there is a lose in precision. Worse is that in 32-bit mode, floating point is done with 32-bit floats, so the two bit lose in precision is even worse:

% macruby -e 'puts 8.1'
8.09999999999999
% arch -i386 macruby -e 'puts 8.1'
8.09999847412109

With the changes in #159:

% miniruby -e 'puts 8.1'
8.1
% arch -i386 miniruby -e 'puts 8.1'
8.1

Changed 14 months ago by lsansonetti@…

  • status changed from new to closed
  • resolution set to duplicate
  • milestone MacRuby 0.5 deleted

Dup of #611.

Note: See TracTickets for help on using tickets.