Share Data from one Spark application to another

[sjahongir]

I run Apache Spark as a standalone.

I have 2 Spark applications and these connected to Apache Spark with master URL:

 var config = new SparkConf()
                .Set("spark.driver.host", "localhost").
                SetMaster("spark://localhost:7077");

The first application reads data from CSV and shares this data frame in some way
DataFrame dataFrame = spark.Read().Text("input.txt");

The second application needs to access this shared data frame.

How can I do this? Is it possible?

[Niharikadutta]

It is not possible to share distributed objects like DataFrame between 2 different Spark applications, since an RDD/DataFrame is tied to the context that was used to create it. The only way to do this is to have the first application write your dataframe to a disk/storage account and then load it from the second application.

[sjahongir]

Can I use the Apache Arrow or Parquet in this scenario?

[Niharikadutta]

Yes you can use parquet to write the DataFrame using the DataFrameWriter API.

[sjahongir]

Working with parquet is understandable, Can I use Apache Arrow (in memory) with Arrow Flight instead of Parquet?