Wednesday, March 28, 2012

PRINT 1/(1+26)=0?

Why does this T-SQL round incorrectly?

PRINT 1/(1+26)

when it should be 0.037037037?

Thanks,
MichaelI should have been more specific:

DECLARE @.A int
DECLARE @.B int
DECLARE @.C float

SET @.A = 1
SET @.B = 26
SET @.C = @.A / (@.A + @.B)
PRINT @.C

mpa...@.htxml.com wrote:

Quote:

Originally Posted by

Why does this T-SQL round incorrectly?
>
PRINT 1/(1+26)
>
when it should be 0.037037037?
>
Thanks,
Michael

|||Ok, I just figured it out:

DECLARE @.A int
DECLARE @.B int
DECLARE @.C float

SET @.A = 1
SET @.B = 26
SET @.C = CAST(@.A AS float) / (CAST(@.A AS float) + CAST(@.B AS float))
PRINT @.C

This is unlike any other programming language that I have used before
-- so strange.

mpaine@.htxml.com wrote:

Quote:

Originally Posted by

Why does this T-SQL round incorrectly?
>
PRINT 1/(1+26)
>
when it should be 0.037037037?
>
Thanks,
Michael

|||If any one of the operators had a decimal point then the results would
have had a decimal point. Performaing pure integer arithmatic, as in
the original example, returns an integer result.

Roy Harvey
Beacon Falls, CT

On 20 Nov 2006 12:57:04 -0800, mpaine@.htxml.com wrote:

Quote:

Originally Posted by

>Ok, I just figured it out:
>
>
>
>DECLARE @.A int
>DECLARE @.B int
>DECLARE @.C float
>
>SET @.A = 1
>SET @.B = 26
>SET @.C = CAST(@.A AS float) / (CAST(@.A AS float) + CAST(@.B AS float))
>PRINT @.C
>
>This is unlike any other programming language that I have used before
>-- so strange.
>
>mpaine@.htxml.com wrote:

Quote:

Originally Posted by

>Why does this T-SQL round incorrectly?
>>
>PRINT 1/(1+26)
>>
>when it should be 0.037037037?
>>
>Thanks,
>Michael

|||So is PRINT CAST(1 AS float)/(CAST(1 AS float)+CAST(26 AS float)) the
best way to get the correct answer for PRINT 1/(1+26)?

- Michael

Roy Harvey wrote:

Quote:

Originally Posted by

If any one of the operators had a decimal point then the results would
have had a decimal point. Performaing pure integer arithmatic, as in
the original example, returns an integer result.
>
Roy Harvey
Beacon Falls, CT
>
On 20 Nov 2006 12:57:04 -0800, mpaine@.htxml.com wrote:
>

Quote:

Originally Posted by

Ok, I just figured it out:

DECLARE @.A int
DECLARE @.B int
DECLARE @.C float

SET @.A = 1
SET @.B = 26
SET @.C = CAST(@.A AS float) / (CAST(@.A AS float) + CAST(@.B AS float))
PRINT @.C

This is unlike any other programming language that I have used before
-- so strange.

mpaine@.htxml.com wrote:

Quote:

Originally Posted by

Why does this T-SQL round incorrectly?
>
PRINT 1/(1+26)
>
when it should be 0.037037037?
>
Thanks,
Michael

|||On 20 Nov 2006 14:53:34 -0800, mpaine@.htxml.com wrote:

Quote:

Originally Posted by

>So is PRINT CAST(1 AS float)/(CAST(1 AS float)+CAST(26 AS float)) the
>best way to get the correct answer for PRINT 1/(1+26)?
>
>- Michael


If a value is a constant, simply adding a decimal point to it is
sufficient, and only one of the operators needs the decimal point:

PRINT 1.0/(1+26)

Alternately, if none of the values are constants, converting any one
of them is sufficient. However, I would not choose to convert to
FLOAT, but to an appropriately sized DECIMAL.

PRINT CONVERT(DECIMAL(15,6),1)/(1+26)

Roy Harvey
Beacon Falls, CT

No comments:

Post a Comment