Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-6807] [SparkR] Merge recent SparkR-pkg changes #5436

Closed
wants to merge 35 commits into from
Closed

[SPARK-6807] [SparkR] Merge recent SparkR-pkg changes #5436

wants to merge 35 commits into from

Commits on Apr 14, 2015

  1. Define functions for schema and fields 

    Instead of using a list[list[list[]]], use specific constructors for schema and field objects.
    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    0e2a94f View commit details
    Browse the repository at this point in the history

  2. Documentation

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    7dd81b7 View commit details
    Browse the repository at this point in the history

  3. Merge pull request apache#233 from redbaron/fail-early-on-missing-dep 

    Fail worker early if dependency is missing
    @shivaram
    shivaram authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    8b76e81 View commit details
    Browse the repository at this point in the history

  4. new line at EOF

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    cd66603 View commit details
    Browse the repository at this point in the history

  5. Merge pull request apache#242 from hqzizania/stats 

    [SPARKR-92] Phase 2: implement sum(rdd)
    @concretevitamin
    concretevitamin authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    136a07e View commit details
    Browse the repository at this point in the history

  6. Merge pull request apache#243 from hqzizania/master 

    [SPARKR-199] Change takeOrdered, top to fetch one partition at a time
    @concretevitamin
    concretevitamin authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    b317aa7 View commit details
    Browse the repository at this point in the history

  7. Merge pull request apache#208 from lythesia/master 

    [SPARKR-188] Add profiling of R execution on worker side
Conflicts:
	pkg/inst/worker/worker.R
    @shivaram
    shivaram authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    c9497a3 View commit details
    Browse the repository at this point in the history

  8. Merge pull request apache#238 from sun-rui/SPARKR-154_4 

    [SPARKR-154] Phase 3: implement intersection().
    @shivaram
    shivaram authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    ba54e34 View commit details
    Browse the repository at this point in the history

  9. Merge pull request apache#234 from hqzizania/assist 

    [SPARKR-163] Support sampleByKey()
Conflicts:
	pkg/R/pairRDD.R
    @concretevitamin
    concretevitamin authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    20b97a6 View commit details
    Browse the repository at this point in the history

  10. Merge pull request apache#244 from sun-rui/SPARKR-154_5 

    [SPARKR-154] Phase 4: implement subtract() and subtractByKey().
    @concretevitamin
    concretevitamin authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    40338a4 View commit details
    Browse the repository at this point in the history

  11. refactor schema functions 

    Refactored `structType` and `structField` so that they can be used to create schemas from R for use with `createDataFrame`.

Moved everything to `schema.R`

Added new methods to `SQLUtils.scala` for handling `StructType` and `StructField` on the JVM side
    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    be5d5c1 View commit details
    Browse the repository at this point in the history

  12. Update createDataFrame and toDF 

    Refactored to use the new `structType` and `structField` functions.
    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    836c4bf View commit details
    Browse the repository at this point in the history

  13. new version of CreateDF 

    New version uses takes a `StructType` from R and creates a DataFrame.

Commented out the `tojson` version since we don't currently use it.
    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    1a3b63d View commit details
    Browse the repository at this point in the history

  14. Update NAMESPACE and tests 

    Updated `NAMESPACE`, `DESCRIPTION`, and unit tests for new schema functions.

Deleted `SQLTypes.R` since everything has been moved to `schema.R`.
    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    275deb4 View commit details
    Browse the repository at this point in the history

  15. Fixed duplicate export

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    f3ba785 View commit details
    Browse the repository at this point in the history

  16. Update subtract to work with generics.R

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    ed66c81 View commit details
    Browse the repository at this point in the history

  17. Merge pull request apache#246 from hlin09/fixCombineByKey 

    Fixes combineByKey
    @shivaram
    shivaram authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    7e8caa3 View commit details
    Browse the repository at this point in the history

  18. [SPARKR-244] Fix test failure after integration of subtract() and sub… 

    …tractByKey() for RDD.
    Sun Rui authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    07d0dbc View commit details
    Browse the repository at this point in the history

  19. Move except into sorted position

    @shivaram
    shivaram authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    40199eb View commit details
    Browse the repository at this point in the history

  20. fix style

    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    9387402 View commit details
    Browse the repository at this point in the history

  21. Merge pull request apache#245 from hqzizania/upstream 

    Add Rd files for sampleByKey() of [SPARKR-163] and sumRDD() of [SPARKR-92]
    @shivaram
    shivaram authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    141efd8 View commit details
    Browse the repository at this point in the history

  22. Rename the SQL DataType function

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    7741d66 View commit details
    Browse the repository at this point in the history

  23. Fix spacing

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    6ef5f2d View commit details
    Browse the repository at this point in the history

  24. Remove tojson functions

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    8526d2e View commit details
    Browse the repository at this point in the history

  25. Update docs and examples

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    71372d9 View commit details
    Browse the repository at this point in the history

  26. Use object attribute instead of argument

    cafreeman authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    5a553e7 View commit details
    Browse the repository at this point in the history

  27. Updates to README.md.

    @concretevitamin
    concretevitamin authored and Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    1bdcb63 View commit details
    Browse the repository at this point in the history

  28. Merge pull request apache#237 from sun-rui/SPARKR-154_3 

    [SPARKR-154] Phase 2: implement cartesian().
    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    ae78312 View commit details
    Browse the repository at this point in the history

  29. rm man

    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    41f8184 View commit details
    Browse the repository at this point in the history

  30. Merge branch 'master' of github.com:apache/spark into R5

    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    4f5ac09 View commit details
    Browse the repository at this point in the history

  31. fix schema.R

    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    e74c04e View commit details
    Browse the repository at this point in the history

  32. fix conflict in README.md

    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    b1fe460 View commit details
    Browse the repository at this point in the history

  33. sort generics

    Davies Liu committed Apr 14, 2015
    Configuration menu
    Copy the full SHA
    168b7fe View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2015

  1. Merge branch 'master' of github.com:apache/spark into R3

    Davies Liu committed Apr 16, 2015
    Configuration menu
    Copy the full SHA
    a5a02f2 View commit details
    Browse the repository at this point in the history

  2. SQLTypes -> schema

    Davies Liu committed Apr 16, 2015
    Configuration menu
    Copy the full SHA
    c2b09be View commit details
    Browse the repository at this point in the history