Filter Spark DF to column having only 2 decimals without using round/floorHow to truncate a floating point number after a certain number of decimal places (no rounding)?How to round a number to n decimal places in JavaHow to nicely format floating numbers to String without unnecessary decimal 0?Round a double to 2 decimal placesHow to always round off upto 2 decimal places in javaLoad CSV file with SparkSpark Dataframe distinguish columns with duplicated nameRounding off values in a column - SparkRCheck column datatype and execute SQL only on Integer and Decimal in Spark SQLApache Spark DecimalType uses BigDecimal setScale that rounds upjoin two dataframe without having common column spark, scala
What happened to the HDEV ISS Experiment? Is it over?
How to check whether a sublist exist in a huge database lists in a fast way?
Filling a listlineplot with a texture
How can I download a file through 2 SSH connections?
Talk interpreter
Can $! cause race conditions when used in scripts running in parallel?
What should come first—characters or plot?
How does encoder decoder network works?
"There were either twelve sexes or none."
Boot Windows from SAN
How to gently end involvement with an online community?
Does Yeshayahu 43:10b / 43:13a imply HaShem was created?
Was the Boeing 2707 design flawed?
Round towards zero
How do proponents of Sola Scriptura address the ministry of those Apostles who authored no parts of Scripture?
How to maximize the drop odds of the Essences in Diablo II?
Papers on arXiv solving the same problem at the same time
Discussing work with supervisor in an invited dinner with his family
Where does learning new skills fit into Agile?
How to obtain a polynomial with these conditions?
Book with the Latin quote 'nihil superbus' meaning 'nothing above us'
What do these commands specifically do?
Does this VCO produce a sine wave or square wave
Can an ISO file damage—or infect—the machine it's being burned on?
Filter Spark DF to column having only 2 decimals without using round/floor
How to truncate a floating point number after a certain number of decimal places (no rounding)?How to round a number to n decimal places in JavaHow to nicely format floating numbers to String without unnecessary decimal 0?Round a double to 2 decimal placesHow to always round off upto 2 decimal places in javaLoad CSV file with SparkSpark Dataframe distinguish columns with duplicated nameRounding off values in a column - SparkRCheck column datatype and execute SQL only on Integer and Decimal in Spark SQLApache Spark DecimalType uses BigDecimal setScale that rounds upjoin two dataframe without having common column spark, scala
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I have spark df as below
+-----------+-------+-----+----+----+-------+----+----+-----+
|_c1 |_c2 |_c3 |_c4 |_c5 |_c6 |_c7 |_c8 |_c9 |
+-----------+-------+-----+----+----+-------+----+----+-----+
|10000000001|7982015|0.00 |0 |0.00|12.9912|0.00|1 |0.00 |
+-----------+-------+-----+----+----+-------+----+----+-----+
I am trying to trim the column _c6 value to 2 decimals and getting it accurate for which I cannot use round/floor as they give closed values.
Is there a way to pick first two decimals get 12.99 instead of rounding it off?
java apache-spark
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
add a comment |
I have spark df as below
+-----------+-------+-----+----+----+-------+----+----+-----+
|_c1 |_c2 |_c3 |_c4 |_c5 |_c6 |_c7 |_c8 |_c9 |
+-----------+-------+-----+----+----+-------+----+----+-----+
|10000000001|7982015|0.00 |0 |0.00|12.9912|0.00|1 |0.00 |
+-----------+-------+-----+----+----+-------+----+----+-----+
I am trying to trim the column _c6 value to 2 decimals and getting it accurate for which I cannot use round/floor as they give closed values.
Is there a way to pick first two decimals get 12.99 instead of rounding it off?
java apache-spark
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
What have you tried so far?
– Andronicus
Mar 27 at 19:35
1
multiply by 100, convert to int, then divide byfloat(100). Possible dupe: How to truncate a floating point number after a certain number of decimal places (no rounding)?. Also rounding 12.9912 to 2 digits should give you 12.99 - there shouldn't be rounding in this example.
– pault
Mar 27 at 19:36
add a comment |
I have spark df as below
+-----------+-------+-----+----+----+-------+----+----+-----+
|_c1 |_c2 |_c3 |_c4 |_c5 |_c6 |_c7 |_c8 |_c9 |
+-----------+-------+-----+----+----+-------+----+----+-----+
|10000000001|7982015|0.00 |0 |0.00|12.9912|0.00|1 |0.00 |
+-----------+-------+-----+----+----+-------+----+----+-----+
I am trying to trim the column _c6 value to 2 decimals and getting it accurate for which I cannot use round/floor as they give closed values.
Is there a way to pick first two decimals get 12.99 instead of rounding it off?
java apache-spark
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
I have spark df as below
+-----------+-------+-----+----+----+-------+----+----+-----+
|_c1 |_c2 |_c3 |_c4 |_c5 |_c6 |_c7 |_c8 |_c9 |
+-----------+-------+-----+----+----+-------+----+----+-----+
|10000000001|7982015|0.00 |0 |0.00|12.9912|0.00|1 |0.00 |
+-----------+-------+-----+----+----+-------+----+----+-----+
I am trying to trim the column _c6 value to 2 decimals and getting it accurate for which I cannot use round/floor as they give closed values.
Is there a way to pick first two decimals get 12.99 instead of rounding it off?
java apache-spark
java apache-spark
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
edited Mar 27 at 19:38
pault
21k4 gold badges36 silver badges60 bronze badges
21k4 gold badges36 silver badges60 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
asked Mar 27 at 19:34
ThatComputerGuyThatComputerGuy
184 bronze badges
184 bronze badges
What have you tried so far?
– Andronicus
Mar 27 at 19:35
1
multiply by 100, convert to int, then divide byfloat(100). Possible dupe: How to truncate a floating point number after a certain number of decimal places (no rounding)?. Also rounding 12.9912 to 2 digits should give you 12.99 - there shouldn't be rounding in this example.
– pault
Mar 27 at 19:36
add a comment |
What have you tried so far?
– Andronicus
Mar 27 at 19:35
1
multiply by 100, convert to int, then divide byfloat(100). Possible dupe: How to truncate a floating point number after a certain number of decimal places (no rounding)?. Also rounding 12.9912 to 2 digits should give you 12.99 - there shouldn't be rounding in this example.
– pault
Mar 27 at 19:36
What have you tried so far?
– Andronicus
Mar 27 at 19:35
What have you tried so far?
– Andronicus
Mar 27 at 19:35
1
1
multiply by 100, convert to int, then divide by
float(100). Possible dupe: How to truncate a floating point number after a certain number of decimal places (no rounding)?. Also rounding 12.9912 to 2 digits should give you 12.99 - there shouldn't be rounding in this example.– pault
Mar 27 at 19:36
multiply by 100, convert to int, then divide by
float(100). Possible dupe: How to truncate a floating point number after a certain number of decimal places (no rounding)?. Also rounding 12.9912 to 2 digits should give you 12.99 - there shouldn't be rounding in this example.– pault
Mar 27 at 19:36
add a comment |
1 Answer
1
active
oldest
votes
A funny way to do it is to use regexp_extract so as to only keep at most 2 digits after the decimal:
Seq(8989.09888, 22.1, 88, 345.111)
.toDF("x")
.select(regexp_extract('x, "[0-9]+(\.[0-9]1,2)?", 0) cast "double" as "x")
.show
+-------+
| x|
+-------+
|8989.09|
| 22.1|
| 88.0|
| 345.11|
+-------+
Note that we handle all the different cases (no decimal, only one...).
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
3
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55385181%2ffilter-spark-df-to-column-having-only-2-decimals-without-using-round-floor%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
A funny way to do it is to use regexp_extract so as to only keep at most 2 digits after the decimal:
Seq(8989.09888, 22.1, 88, 345.111)
.toDF("x")
.select(regexp_extract('x, "[0-9]+(\.[0-9]1,2)?", 0) cast "double" as "x")
.show
+-------+
| x|
+-------+
|8989.09|
| 22.1|
| 88.0|
| 345.11|
+-------+
Note that we handle all the different cases (no decimal, only one...).
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
3
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
add a comment |
A funny way to do it is to use regexp_extract so as to only keep at most 2 digits after the decimal:
Seq(8989.09888, 22.1, 88, 345.111)
.toDF("x")
.select(regexp_extract('x, "[0-9]+(\.[0-9]1,2)?", 0) cast "double" as "x")
.show
+-------+
| x|
+-------+
|8989.09|
| 22.1|
| 88.0|
| 345.11|
+-------+
Note that we handle all the different cases (no decimal, only one...).
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
3
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
add a comment |
A funny way to do it is to use regexp_extract so as to only keep at most 2 digits after the decimal:
Seq(8989.09888, 22.1, 88, 345.111)
.toDF("x")
.select(regexp_extract('x, "[0-9]+(\.[0-9]1,2)?", 0) cast "double" as "x")
.show
+-------+
| x|
+-------+
|8989.09|
| 22.1|
| 88.0|
| 345.11|
+-------+
Note that we handle all the different cases (no decimal, only one...).
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
A funny way to do it is to use regexp_extract so as to only keep at most 2 digits after the decimal:
Seq(8989.09888, 22.1, 88, 345.111)
.toDF("x")
.select(regexp_extract('x, "[0-9]+(\.[0-9]1,2)?", 0) cast "double" as "x")
.show
+-------+
| x|
+-------+
|8989.09|
| 22.1|
| 88.0|
| 345.11|
+-------+
Note that we handle all the different cases (no decimal, only one...).
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
answered Mar 27 at 20:54
OliOli
3,3632 gold badges5 silver badges27 bronze badges
3,3632 gold badges5 silver badges27 bronze badges
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
3
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
add a comment |
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
3
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
This will not actually round off. If I am not wrong, for both, 12.9912 and 12.9956 , this will return 12.99.
– Apurba Pandey
Mar 28 at 6:02
3
3
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
You are completely right. Yet I am not sure I fully grasped the question, but from what I understand, it is about truncating, not rounding.
– Oli
Mar 28 at 6:33
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
This worked perfectly, I tried with converting to decimal types and select only 2 decimals that also works
– ThatComputerGuy
Apr 2 at 16:29
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55385181%2ffilter-spark-df-to-column-having-only-2-decimals-without-using-round-floor%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What have you tried so far?
– Andronicus
Mar 27 at 19:35
1
multiply by 100, convert to int, then divide by
float(100). Possible dupe: How to truncate a floating point number after a certain number of decimal places (no rounding)?. Also rounding 12.9912 to 2 digits should give you 12.99 - there shouldn't be rounding in this example.– pault
Mar 27 at 19:36