Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 576
Default Numerical accuracy in Form Text-boxes

Ron,

Thanks for this info. It helps explain why I find strange values in my axis
limits for my charts. (I see the same effect there) Have been using round
functions and stuff to avoid it when changing the axis settings by code.

--
sb
"Ron Rosenfeld" wrote in message
...
On Thu, 21 Aug 2003 22:47:25 +1000, IM wrote:

I use a text-box in a form to enter numerical data. When I enter a
number, say, 14.6, in the text-box and then have the VBA code transfer
the data to my spreadsheet. Sometimes, but not always, the number shown
on the spreadsheet is not exactly 14.6, but 14.59999987..

Of course, I could add a few lines to the code to round the number off
to 2 or 3 decimal places. But there must a better way. What is it ? Why
does this happen ?


It happens because VBA of precision limits in VBA and/or Excel; and due to

the
fact that certain numbers cannot be represented exactly in binary. An

example
in decimal arithmetic would be the attempt to represent 1/3 in decimal

format.

There are several solutions depending on your goals:

Use the DECIMAL data type.
Round.
Use an appropriate number format in the Excel cell (this would leave the

result
as 14.5999999... but display it as 14.60).


--ron



  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 5,651
Default Numerical accuracy in Form Text-boxes

On Sat, 23 Aug 2003 11:04:33 +1000, IM wrote:

Ron

Thanks for the reply. I do know exactly how numbers are stored and the
cause of round-off errors. But what irks me is the problem of
consistency. What I mean by that is if I enter 14.6 in a worksheet cell,
the problem I mentioned will never happen. But if the number is
extracted from a textbox via VBA, it does.

Apparently, Excel by itself works fine in this respect. Only its
interfacing with VBA that will bring up the problem. I just wonder if
that's an inherent problem with Visual Basic. If yes, Microsoft could
handle that easily, since there is no problem with that within Excel.

IM


Could there be an issue in your code?

If I enter 14.6 into an input box, assign it to a variable which has been dim'd
as double, and then print it, I get 14.6

Very simply:

=============
Sub test()
Dim c As Double
c = InputBox("Enter a number")
Debug.Print c
End Sub
============

It seems as if VBA is handling 14.6 OK in this simple routine.


--ron
Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Link Text Boxes on Form? Joyce Excel Discussion (Misc queries) 8 September 1st 09 06:20 AM
User Form Text Boxes - Copy format of text boxes NDBC Excel Discussion (Misc queries) 3 July 2nd 09 02:02 AM
text boxes retaining their last value on form close and restart NDBC Excel Discussion (Misc queries) 3 July 1st 09 12:55 PM
convert numerical value to a text form Converting number to text in Excel Excel Discussion (Misc queries) 4 April 9th 07 07:55 PM
Form Text Boxes moving md2503 Excel Worksheet Functions 2 May 16th 05 07:57 PM


All times are GMT +1. The time now is 02:17 PM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"