[SPARK-21893][SPARK-22142][TESTS][FOLLOWUP] Enables PySpark tests for Flume and Kafka in Jenkins

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to enable PySpark tests for Flume and Kafka in Jenkins by explicitly setting the environment variables in modules.py.

Seems we are not taking the dependencies into account when calculating environment variables:

spark/dev/run-tests.py

Lines 554 to 561 in 3a07eff

	# setup environment variables
	# note - the 'root' module doesn't collect environment variables for all modules. Because the
	# environment variables should not be set if a module is not changed, even if running the 'root'
	# module. So here we should use changed_modules rather than test_modules.
	test_environ = {}
	for m in changed_modules:
	test_environ.update(m.environ)
	setup_test_environ(test_environ)

How was this patch tested?

Manual tests with Jenkins in #20126.

Before - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85559/consoleFull

[info] Setup the following environment variables for tests: 
...

After - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85560/consoleFull

[info] Setup the following environment variables for tests: 
ENABLE_KAFKA_0_8_TESTS=1
ENABLE_FLUME_TESTS=1
...

[HyukjinKwon]

cc @srowen, @vanzin, @yhuai, @ueshin and @zsxwing who I believe are primarily related with this change.

[srowen]

Ah I probably missed that when moving these behind a profile. I wasn't fully aware of the dependency mechanism. If it resolves the problem, yes go for it.

[SparkQA]

Test build #85561 has finished for PR 20128 at commit 3b1533a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ueshin]

LGTM.

[HyukjinKwon]

Merged to master.

Thank you @srowen, @felixcheung and @ueshin for reviewing this.