Spark/Scala update the value of a variable in another map?Apache spark: map csv file to key: value formatHow to define a global scala variable in Spark which will be shared by all workers?Spark performance for Scala vs PythonNoSuchMethodError on a Map (Spark Scala class)How to sort an RDD of tuples with 5 elements in Spark Scala?word count(frequency) spark rdd scalaspark map partitions to fill nan valuesSpark combineByKey on values that contains tuplesHow to update a global variable inside RDD map operationspark: value histogram is not a member of org.apache.spark.rdd.RDD[Option[Any]]
Are spot colors limited and why CMYK mix is not treated same as spot color mix?
In Endgame, wouldn't Stark have remembered Hulk busting out of the stairwell?
Do application leftovers have any impact on performance?
Convert shapefille to KML
Why does the weaker C–H bond have a higher wavenumber than the C=O bond?
Should I use the words "pyromancy" and "necromancy" even if they don't mean what people think they do?
Count the number of triangles
Why does Sauron not permit his followers to use his name?
Is there a way to tell what frequency I need a PWM to be?
Printing a list as "a, b, c." using Python
What checks exist against overuse of presidential pardons in the USA?
Inspiration for failed idea?
Journal published a paper, ignoring my objections as a referee
Can two aircraft stay on the same runway at the same time?
How to differentiate between two people with the same name in a story?
How can I fix cracks between the bathtub and the wall surround?
Is it recommended to point out a professor's mistake during their lecture?
Why do presidential pardons exist in a country having a clear separation of powers?
What should be done with the carbon when using magic to get oxygen from carbon dioxide?
Notice period 60 days but I need to join in 45 days
Why is 3/4 a simple meter while 6/8 is a compound meter?
How do Barton (Hawkeye/Ronin) and Romanov (Black Widow) end up on the Benatar on Morag in 2014?
In what language did Túrin converse with Mím?
Are sweatpants frowned upon on flights?
Spark/Scala update the value of a variable in another map?
Apache spark: map csv file to key: value formatHow to define a global scala variable in Spark which will be shared by all workers?Spark performance for Scala vs PythonNoSuchMethodError on a Map (Spark Scala class)How to sort an RDD of tuples with 5 elements in Spark Scala?word count(frequency) spark rdd scalaspark map partitions to fill nan valuesSpark combineByKey on values that contains tuplesHow to update a global variable inside RDD map operationspark: value histogram is not a member of org.apache.spark.rdd.RDD[Option[Any]]
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
In Spark, I have a
closest: org.apache.spark.rdd.RDD[(Int, (breeze.linalg.Vector[Double], Int))] = MapPartitionsRDD[476] at map at command-1043253026161724:1
I want to calculate some total distance:
var tempDist=0.0
closest.foreach(x=> tempDist=tempDist+squaredDistance(x._2._1, kPoints(x._1)))
But this doesn't change tempDist's value at all. I suspect Spark doesn't do anything. So how can I calculate the distance?
scala apache-spark rdd
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
add a comment |
In Spark, I have a
closest: org.apache.spark.rdd.RDD[(Int, (breeze.linalg.Vector[Double], Int))] = MapPartitionsRDD[476] at map at command-1043253026161724:1
I want to calculate some total distance:
var tempDist=0.0
closest.foreach(x=> tempDist=tempDist+squaredDistance(x._2._1, kPoints(x._1)))
But this doesn't change tempDist's value at all. I suspect Spark doesn't do anything. So how can I calculate the distance?
scala apache-spark rdd
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
add a comment |
In Spark, I have a
closest: org.apache.spark.rdd.RDD[(Int, (breeze.linalg.Vector[Double], Int))] = MapPartitionsRDD[476] at map at command-1043253026161724:1
I want to calculate some total distance:
var tempDist=0.0
closest.foreach(x=> tempDist=tempDist+squaredDistance(x._2._1, kPoints(x._1)))
But this doesn't change tempDist's value at all. I suspect Spark doesn't do anything. So how can I calculate the distance?
scala apache-spark rdd
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
In Spark, I have a
closest: org.apache.spark.rdd.RDD[(Int, (breeze.linalg.Vector[Double], Int))] = MapPartitionsRDD[476] at map at command-1043253026161724:1
I want to calculate some total distance:
var tempDist=0.0
closest.foreach(x=> tempDist=tempDist+squaredDistance(x._2._1, kPoints(x._1)))
But this doesn't change tempDist's value at all. I suspect Spark doesn't do anything. So how can I calculate the distance?
scala apache-spark rdd
scala apache-spark rdd
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
asked Mar 27 at 22:05
Jiapeng ZhangJiapeng Zhang
2992 silver badges22 bronze badges
2992 silver badges22 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Don't do mutable vars. It's a bad idea in general, and doesn't work at all with spark, at least, not the way you are doing it, because it's a distributed system.
Different partitions of the sequence are located on different computers, and are being processed independently in parallel and in different JVMs, each of which has its own copy of the var.
val tempDist = closest
.map x => squaredDistance(x._2._1, kPoints(x._1)
.fold(0) _ + _
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55387196%2fspark-scala-update-the-value-of-a-variable-in-another-map%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Don't do mutable vars. It's a bad idea in general, and doesn't work at all with spark, at least, not the way you are doing it, because it's a distributed system.
Different partitions of the sequence are located on different computers, and are being processed independently in parallel and in different JVMs, each of which has its own copy of the var.
val tempDist = closest
.map x => squaredDistance(x._2._1, kPoints(x._1)
.fold(0) _ + _
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
add a comment |
Don't do mutable vars. It's a bad idea in general, and doesn't work at all with spark, at least, not the way you are doing it, because it's a distributed system.
Different partitions of the sequence are located on different computers, and are being processed independently in parallel and in different JVMs, each of which has its own copy of the var.
val tempDist = closest
.map x => squaredDistance(x._2._1, kPoints(x._1)
.fold(0) _ + _
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
add a comment |
Don't do mutable vars. It's a bad idea in general, and doesn't work at all with spark, at least, not the way you are doing it, because it's a distributed system.
Different partitions of the sequence are located on different computers, and are being processed independently in parallel and in different JVMs, each of which has its own copy of the var.
val tempDist = closest
.map x => squaredDistance(x._2._1, kPoints(x._1)
.fold(0) _ + _
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
Don't do mutable vars. It's a bad idea in general, and doesn't work at all with spark, at least, not the way you are doing it, because it's a distributed system.
Different partitions of the sequence are located on different computers, and are being processed independently in parallel and in different JVMs, each of which has its own copy of the var.
val tempDist = closest
.map x => squaredDistance(x._2._1, kPoints(x._1)
.fold(0) _ + _
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
edited Mar 28 at 0:29
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
answered Mar 27 at 23:06
DimaDima
27.8k3 gold badges27 silver badges44 bronze badges
27.8k3 gold badges27 silver badges44 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55387196%2fspark-scala-update-the-value-of-a-variable-in-another-map%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
