Skip to content

Commit

Permalink
Merge pull request apache#431 from ankurdave/graphx-caching-doc
Browse files Browse the repository at this point in the history
Describe caching and uncaching in GraphX programming guide

(cherry picked from commit ad294db)
Signed-off-by: Reynold Xin <[email protected]>
  • Loading branch information
rxin committed Jan 15, 2014
1 parent 2f930d5 commit 6fa4e02
Showing 1 changed file with 10 additions and 1 deletion.
11 changes: 10 additions & 1 deletion docs/graphx-programming-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -611,11 +611,20 @@ class GraphOps[VD, ED] {
> substantial communication. If possible try expressing the same computation using the
> `mapReduceTriplets` operator directly.
## Caching and Uncaching

In Spark, RDDs are not persisted in memory by default. To avoid recomputation, they must be explicitly cached when using them multiple times (see the [Spark Programming Guide][RDD Persistence]). Graphs in GraphX behave the same way. **When using a graph multiple times, make sure to call [`Graph.cache()`][Graph.cache] on it first.**

[RDD Persistence]: scala-programming-guide.html#rdd-persistence
[Graph.cache]: api/graphx/index.html#org.apache.spark.graphx.Graph@cache():Graph[VD,ED]

In iterative computations, *uncaching* may also be necessary for best performance. By default, cached RDDs and graphs will remain in memory until memory pressure forces them to be evicted in LRU order. For iterative computation, intermediate results from previous iterations will fill up the cache. Though they will eventually be evicted, the unnecessary data stored in memory will slow down garbage collection. It would be more efficient to uncache intermediate results as soon as they are no longer necessary. This involves materializing (caching and forcing) a graph or RDD every iteration, uncaching all other datasets, and only using the materialized dataset in future iterations. However, because graphs are composed of multiple RDDs, it can be difficult to unpersist them correctly. **For iterative computation we recommend using the Pregel API, which correctly unpersists intermediate results.**

# Pregel API
<a name="pregel"></a>

Graphs are inherently recursive data-structures as properties of vertices depend on properties of
their neighbors which intern depend on properties of *their* neighbors. As a
their neighbors which in turn depend on properties of *their* neighbors. As a
consequence many important graph algorithms iteratively recompute the properties of each vertex
until a fixed-point condition is reached. A range of graph-parallel abstractions have been proposed
to express these iterative algorithms. GraphX exposes a Pregel-like operator which is a fusion of
Expand Down

0 comments on commit 6fa4e02

Please sign in to comment.