Updating log configuration from command line #30
Comments
|
It was very hard to get to the bottom of this. I just solved this in zipkin-dependencies (spark, but not streaming). The bottom line is that spark really wants you to use log4j :) I will make something similar to this here, if interested. |
Before, we could only control logging by rebuilding. This special-cases the "zipkin" category such that it can be controlled without distributing files to the cluster or changing dependencies. Note: this only special-cases the zipkin log category. Third party code that logs outside the zipkin category will need to do something more involved, possibly creating and distributing a log4j.properties file and enabling it on the executors. This is a port of work done in openzipkin/zipkin-dependencies#62 Fixes #30
|
@adriancole Thanks for the fix. It looks good. Do you think this can be extended to spring framework logs as well? I'm interesting in logs like below, to get some insight into the spring magic. |
|
personally, I'm not terribly interested in whack-a-mole for each category,
especially as spring logs are mostly interesting when you are debugging
(which can be done locally without distributing to a spark cluster).
Finally, you can always add a log statement in the auto-config (which would
be in the zipkin category perhaps debug) to say which adjusters are in the
list, right?
|
|
one alternative to whack-a-mole would be to assign the root log level. then when you are debugging something, you set everything to DEBUG (which would work regardless of debugging kafka, zookeeper or spring). That would be more volume than necessary, but would not require in whack-a-mole (ex cherry-picking categories and adding tech debt for each one) |
|
Since we are using Spring framework to build our spark job and its plugins, we should support insights into the framework as first class citizen. Exposing the root log level can be a good compromise. |
|
Since we are using Spring framework to build our spark job and its
plugins, we should support insights into the framework as first class
citizen.
Exposing the root log level can be a good compromise.
we are also using several other libraries, the most first class being spark
(which is the reason why logging is a hassle). For example, there's zero
reason we can't ask people to simply distribute log configuration like
normal spark does. We aren't a spark extension library, eventhough we did
add one helper for the zipkin category.
Maintenance is very important in projects like this, especially as not many
are involved or active and this code will last a long time. That's why
taking small steps (least custom code first). I hope we can all start
adopting a "ease in" approach as sprinkling knobs and custom code is
otherwise a road to perdition. For example, there will be very few who know
how any of spark logging works (google search for details), so anyway
please consider things like this as one day I'd love for you to be arguing
for long term maintenance approach vs short term gains.
|
|
NEXT STEP: look into resetting the root logger (possibly via SparkConf if we are lucky enough for that to work). We are probably getting to a point where we'll need to backfill and add integration tests so that if this doesn't work the next one who touches the code isn't in for a hard time. |
Can you share sample command for this? I tried passing the log config file using the parameters below, but it didn't work. The same parameters work for non-spring spark jobs. |
|
Each task in spark is me learning how to do something then helping make the
change. I havent tried the below and if you cant troubleshoot and figure it
out I can help, but it will be a little while as I am displaced.
On 15 Mar 2017 03:30, "Naoman Abbas" <[email protected]> wrote:
For example, there's zero
reason we can't ask people to simply distribute log configuration like
normal spark does.
Can you share sample command for this? I tried passing the log config file
using the parameters below, but it didn't work. The same parameters work
for non-spring spark jobs.
…--zipkin.sparkstreaming.conf.spark.executor.extraJavaOptions="-Dlog4j.configuration=file:/tmp/log4j.properties"
\
--zipkin.sparkstreaming.conf.spark.driver.extraJavaOptions="-Dlog4j.configuration=file:/tmp/log4j.properties"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAD613OYM2na3LPv0rBLx6a7NcPuJL56ks5rlz9PgaJpZM4ML1d6>
.
|
Welcome to my life :) (just switch spark with Spring) BTW really appreciate your help! Didn't want to burden you with more work. Your earlier messages gave the impression that you have done this already thats why I asked. |
|
I did use regular log distribution until i figured out how to get dynamic
to work. Most of this is testing.. i am hoping you will try some to figure
out some of the things you dont know yet.
For example we have no spark expert and that hurts. We have a similar
problem with ES where many ask for things but few make anything happen.
Spring is far more commodity knowledge and several of us on gitter know it
(even if i only learned it less than 2years ago). Bottom line is rolling up
sleeves is an eventuality and digging into not yet learned or hard problems
is an unescapable part of being on a project team. Until more pickup
ownership of all code (not just the parts we feel comfortable with), the
project will remain troublesome.
…On 15 Mar 2017 08:32, "Naoman Abbas" ***@***.***> wrote:
Each task in spark is me learning how to do something then helping make the
change.
Welcome to my life :) (just switch spark with Spring)
BTW really appreciate your help! Didn't want to burden you with more work.
Your earlier messages gave the impression that you have done this already
thats why I asked.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAD61zsNxt1GcnwimRqjKxS2_Repz4IGks5rl4YZgaJpZM4ML1d6>
.
|
|
I'm able to get the log config working in local mode (non-cluster). I'll debug more and try to get it working for the cluster mode as well. |
|
glad to hear progress!
as a hint, it probably involves affecting the property affecting files
distributed to the cluster, where by default we only send the main jar and
any adjusters. you'd need to override to send these and also the custom
log4j file iirc.
|
To enable debug logs, we have to update logback.xml and rebuilding the code. We should be able to configure log levels through command line parameters.
The text was updated successfully, but these errors were encountered: