PRINT 1/(1+26)
when it should be 0.037037037?
Thanks,
MichaelI should have been more specific:
DECLARE @.A int
DECLARE @.B int
DECLARE @.C float
SET @.A = 1
SET @.B = 26
SET @.C = @.A / (@.A + @.B)
PRINT @.C
mpa...@.htxml.com wrote:
Quote:
Originally Posted by
Why does this T-SQL round incorrectly?
>
PRINT 1/(1+26)
>
when it should be 0.037037037?
>
Thanks,
Michael
DECLARE @.A int
DECLARE @.B int
DECLARE @.C float
SET @.A = 1
SET @.B = 26
SET @.C = CAST(@.A AS float) / (CAST(@.A AS float) + CAST(@.B AS float))
PRINT @.C
This is unlike any other programming language that I have used before
-- so strange.
mpaine@.htxml.com wrote:
Quote:
Originally Posted by
Why does this T-SQL round incorrectly?
>
PRINT 1/(1+26)
>
when it should be 0.037037037?
>
Thanks,
Michael
have had a decimal point. Performaing pure integer arithmatic, as in
the original example, returns an integer result.
Roy Harvey
Beacon Falls, CT
On 20 Nov 2006 12:57:04 -0800, mpaine@.htxml.com wrote:
Quote:
Originally Posted by
>Ok, I just figured it out:
>
>
>
>DECLARE @.A int
>DECLARE @.B int
>DECLARE @.C float
>
>SET @.A = 1
>SET @.B = 26
>SET @.C = CAST(@.A AS float) / (CAST(@.A AS float) + CAST(@.B AS float))
>PRINT @.C
>
>This is unlike any other programming language that I have used before
>-- so strange.
>
>mpaine@.htxml.com wrote:
Quote:
Originally Posted by
>Why does this T-SQL round incorrectly?
>>
>PRINT 1/(1+26)
>>
>when it should be 0.037037037?
>>
>Thanks,
>Michael
best way to get the correct answer for PRINT 1/(1+26)?
- Michael
Roy Harvey wrote:
Quote:
Originally Posted by
If any one of the operators had a decimal point then the results would
have had a decimal point. Performaing pure integer arithmatic, as in
the original example, returns an integer result.
>
Roy Harvey
Beacon Falls, CT
>
On 20 Nov 2006 12:57:04 -0800, mpaine@.htxml.com wrote:
>
Quote:
Originally Posted by
Ok, I just figured it out:
DECLARE @.A int
DECLARE @.B int
DECLARE @.C float
SET @.A = 1
SET @.B = 26
SET @.C = CAST(@.A AS float) / (CAST(@.A AS float) + CAST(@.B AS float))
PRINT @.C
This is unlike any other programming language that I have used before
-- so strange.
mpaine@.htxml.com wrote:
Quote:
Originally Posted by
Why does this T-SQL round incorrectly?
>
PRINT 1/(1+26)
>
when it should be 0.037037037?
>
Thanks,
Michael
Quote:
Originally Posted by
>So is PRINT CAST(1 AS float)/(CAST(1 AS float)+CAST(26 AS float)) the
>best way to get the correct answer for PRINT 1/(1+26)?
>
>- Michael
If a value is a constant, simply adding a decimal point to it is
sufficient, and only one of the operators needs the decimal point:
PRINT 1.0/(1+26)
Alternately, if none of the values are constants, converting any one
of them is sufficient. However, I would not choose to convert to
FLOAT, but to an appropriately sized DECIMAL.
PRINT CONVERT(DECIMAL(15,6),1)/(1+26)
Roy Harvey
Beacon Falls, CT
No comments:
Post a Comment