Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update GradientDescentSuite.scala #588

Closed
wants to merge 3 commits into from
Closed

Update GradientDescentSuite.scala #588

wants to merge 3 commits into from

Conversation

baishuo
Copy link
Contributor

@baishuo baishuo commented Apr 29, 2014

use more faster way to construct an array

@baishuo
Update GradientDescentSuite.scala
b666d27 
use more faster way to construct an array
@baishuo baishuo mentioned this pull request Apr 29, 2014
Closed
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@techaddict
Copy link
Contributor

IMHO this will not compile if used like this 1 +: features, Integers with DenseVector[Double]. Are you sure we this is very useful ? @mengxr

@mengxr
Copy link
Contributor

mengxr commented Apr 29, 2014

Should use Vectors.dense(1.0 +: features), which may be faster than Vectors.dense(1.0, features: _*), depending on how scala compiles the code.

techaddict
techaddict reviewed Apr 29, 2014
View reviewed changes
mllib/src/test/scala/org/apache/spark/mllib/optimization/GradientDescentSuite.scala
@@ -81,11 +81,11 @@ class GradientDescentSuite extends FunSuite with LocalSparkContext with ShouldMa
// Add a extra variable consisting of all 1.0's for the intercept.
val testData = GradientDescentSuite.generateGDInput(A, B, nPoints, 42)
val data = testData.map { case LabeledPoint(label, features) =>
label -> Vectors.dense(1.0, features.toArray: _*)
label -> (1.0 +: features)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't compile either, we need to write features +: 1.0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for other changes too.

@techaddict
Copy link
Contributor

@mengxr yupp that would be better.

@baishuo
Update GradientDescentSuite.scala
c03b61c 
modify to Vectors.dense(1.0 +: features)
@baishuo
Copy link
Contributor Author

baishuo commented Apr 29, 2014

I had modify the code to Vectors.dense(1.0 +: features) .
can some setting can make this code compile successful? @mengxr

@techaddict
Copy link
Contributor

@baishuo change it to Vectors.dense(1.0 +: features.toArray)

@baishuo
Copy link
Contributor Author

baishuo commented Apr 30, 2014

had modified, thanks to @techaddict @mengxr

@mengxr
Copy link
Contributor

mengxr commented Apr 30, 2014

Jenkins, test this please.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14581/

@baishuo
Copy link
Contributor Author

baishuo commented May 5, 2014

Hi guys @mengxr @techaddict , Is there something need to do before merge? :)

@srowen
Copy link
Member

srowen commented May 5, 2014

(I'd be curious to know why this might or might not be faster, if someone has the time to sketch an explanation. I don't doubt it might be.)

@mengxr
Copy link
Contributor

mengxr commented May 5, 2014

@srowen They are the same because in Vectors.dense(first: Double, others: Double*) defined, we use first +: others.toArray and toArray does not copy the data. But it is safer to create an Array before calling Vectors.dense. For example, Array(1.0, values: _*) is slower than 1.0 +: values if values is an Array. See https://github.com/scala/scala/blob/master/src/library/scala/Array.scala#L199

@mengxr
Copy link
Contributor

mengxr commented May 7, 2014

LGTM.

@pwendell
Copy link
Contributor

pwendell commented May 7, 2014

Merged, thanks.

@asfgit asfgit closed this in 0c19bb1 May 7, 2014
asfgit pushed a commit that referenced this pull request May 7, 2014
@baishuo @pwendell
Update GradientDescentSuite.scala
0972b62 
use more faster way to construct an array

Author: baishuo(白硕) <[email protected]>

Closes #588 from baishuo/master and squashes the following commits:

45b95fb [baishuo(白硕)] Update GradientDescentSuite.scala
c03b61c [baishuo(白硕)] Update GradientDescentSuite.scala
b666d27 [baishuo(白硕)] Update GradientDescentSuite.scala
(cherry picked from commit 0c19bb1)

Signed-off-by: Patrick Wendell <[email protected]>
@baishuo baishuo mentioned this pull request May 18, 2014
Closed
asfgit pushed a commit that referenced this pull request May 23, 2014
@baishuo @rxin
Update LBFGSSuite.scala
a08262d 
the same reason as #588

Author: baishuo(白硕) <[email protected]>

Closes #815 from baishuo/master and squashes the following commits:

6876c1e [baishuo(白硕)] Update LBFGSSuite.scala
asfgit pushed a commit that referenced this pull request May 23, 2014
@baishuo @rxin
Update LBFGSSuite.scala
9be103a 
the same reason as #588

Author: baishuo(白硕) <[email protected]>

Closes #815 from baishuo/master and squashes the following commits:

6876c1e [baishuo(白硕)] Update LBFGSSuite.scala

(cherry picked from commit a08262d)
Signed-off-by: Reynold Xin <[email protected]>
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@baishuo @pdeyhim
Update GradientDescentSuite.scala
5213a7e 
use more faster way to construct an array

Author: baishuo(白硕) <[email protected]>

Closes apache#588 from baishuo/master and squashes the following commits:

45b95fb [baishuo(白硕)] Update GradientDescentSuite.scala
c03b61c [baishuo(白硕)] Update GradientDescentSuite.scala
b666d27 [baishuo(白硕)] Update GradientDescentSuite.scala
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@baishuo @pdeyhim
Update LBFGSSuite.scala
245535d 
the same reason as apache#588

Author: baishuo(白硕) <[email protected]>

Closes apache#815 from baishuo/master and squashes the following commits:

6876c1e [baishuo(白硕)] Update LBFGSSuite.scala
chuckchen pushed a commit to chuckchen/spark that referenced this pull request Jun 25, 2015
@rxin
Merge pull request apache#588 from pwendell/branch-0.7
8115127 
Spark-742: TaskMetrics should not employ per-record timing.
helenyugithub pushed a commit to helenyugithub/spark that referenced this pull request Aug 20, 2019
@azarum @bulldozer-bot
log conda output (apache#588)
cd97fb1 
## Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain)
When running verbose conda commands, outputs via `log.info` are not being logged, but outputs with `logInfo` are. We also want to log the conda command in the case the verbosity flag is on (>0).
## What changes were proposed in this pull request?

Change the logging of the conda command to `logInfo` and log the command when the verbosity flag is on.

## How was this patch tested?

N/A.
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
@liusheng @kiwik
Change ARM jobs according to job naming notitions (apache#588)
23713c7 
* Change ARM jobs according to job naming notitions

- hbase-arm-build -> hbase-build-arm64
- hive-build-arm64 -> hive-build-arm64
- update hbase job description
- use "arm64" instead of "aarch64"

Related-Bug: theopenlab/openlab#318
arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
@mgorbov
MapR [MS-925] After upgrade to MEP 6.2 (Spark 2.4.0) can no longer (a…
c5cbbcc 
…pache#588)

consume Kafka / MapR Streams.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@baishuo @AmplabJenkins @techaddict @mengxr @srowen @pwendell