Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-24777][SQL] Add write benchmark for AVRO #22451

Closed
wants to merge 2 commits into from
Closed

[SPARK-24777][SQL] Add write benchmark for AVRO #22451

wants to merge 2 commits into from

Conversation

gengliangwang
Copy link
Member

What changes were proposed in this pull request?

Refactor DataSourceWriteBenchmark and add write benchmark for AVRO.

How was this patch tested?

Build and run the benchmark.

@SparkQA
Copy link

SparkQA commented Sep 18, 2018

Test build #96182 has finished for PR 22451 at commit d410a36.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • trait DataSourceWriteBenchmark

@dongjoon-hyun
Copy link
Member

HI, @gengliangwang . Since this aims to refactor, could you check #22443 to be consistent?

Refactor DataSourceWriteBenchmark and add write benchmark for AVRO.

dongjoon-hyun
dongjoon-hyun reviewed Sep 18, 2018
View reviewed changes
external/avro/src/test/scala/org/apache/spark/sql/execution/benchmark/AvroWriteBenchmark.scala Outdated
/**
* Benchmark to measure Avro data sources write performance.
* Usage:
* spark-submit --class <this class> <spark sql test jar>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#22443 is introducing separate benchmark output file and sbt. It would be great if we can follow the style for a newly updated benchmark.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, no problem 👍

@SparkQA
Copy link

SparkQA commented Sep 18, 2018

Test build #96191 has finished for PR 22451 at commit 34af59d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gengliangwang
Copy link
Member Author

retest this please.

@SparkQA
Copy link

SparkQA commented Sep 19, 2018

Test build #96217 has finished for PR 22451 at commit 34af59d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu
Copy link
Member

maropu commented Sep 19, 2018 •
edited
Loading

btw, why this pr doesn't automatically link to the corresponding jira...?: https://issues.apache.org/jira/browse/SPARK-24777

@gengliangwang
Copy link
Member Author

gengliangwang commented Sep 19, 2018 •
edited
Loading

@maropu When I created this PR, I typed to "[[" at the beginning of title, thus it failed to link to the JIRA.

@maropu
Copy link
Member

maropu commented Sep 19, 2018

oh, I see. can you put it in the jira manually?

@dongjoon-hyun dongjoon-hyun mentioned this pull request Sep 20, 2018
Closed
gatorsmile
gatorsmile reviewed Sep 21, 2018
View reviewed changes
Copy link
Member

@gatorsmile gatorsmile left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Merged to master.

Thanks! This is test only changes. I also merged it to 2.4.

asfgit pushed a commit that referenced this pull request Sep 21, 2018
@gengliangwang @gatorsmile
[SPARK-24777][SQL] Add write benchmark for AVRO
51f3659 
## What changes were proposed in this pull request?

Refactor `DataSourceWriteBenchmark` and add write benchmark for AVRO.

## How was this patch tested?

Build and run the benchmark.

Closes #22451 from gengliangwang/avroWriteBenchmark.

Authored-by: Gengliang Wang <[email protected]>
Signed-off-by: gatorsmile <[email protected]>
(cherry picked from commit 950ab79)
Signed-off-by: gatorsmile <[email protected]>
@asfgit asfgit closed this in 950ab79 Sep 21, 2018
@gengliangwang
Copy link
Member Author

Oh I was creating a PR for refactoring BenchmarkBase, I planned to merge this one after that one.
Since this is merged, I will create one to refactor both.

otterc pushed a commit to linkedin/spark that referenced this pull request Mar 22, 2023
@gengliangwang @kunkun-tang
[SPARK-24777][SQL] Add write benchmark for AVRO
de6775f 
Refactor `DataSourceWriteBenchmark` and add write benchmark for AVRO.

Build and run the benchmark.

Closes apache#22451 from gengliangwang/avroWriteBenchmark.

Authored-by: Gengliang Wang <[email protected]>
Signed-off-by: gatorsmile <[email protected]>
(cherry picked from commit 950ab79)

RB=2129470
G=spark-reviewers
R=ekrogen
A=ekrogen
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gatorsmile gatorsmile gatorsmile left review comments

@dongjoon-hyun dongjoon-hyun dongjoon-hyun left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gengliangwang @SparkQA @dongjoon-hyun @maropu @gatorsmile