Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Review] Print debug message if SVM convergence is poor #3562

Merged
merged 1 commit into from
Mar 2, 2021
Merged

[Review] Print debug message if SVM convergence is poor #3562

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions cpp/src/svm/smosolver.cuh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,12 @@ class SmoSolver {
cache_size, svmType);
// Init counters
max_outer_iter = GetDefaultMaxIter(n_train, max_outer_iter);
int n_iter = 0;
n_iter = 0;
int n_inner_iter = 0;
diff_prev = 0;
n_small_diff = 0;
n_increased_diff = 0;
report_increased_diff = true;
bool keep_going = true;

while (n_iter < max_outer_iter && keep_going) {
Expand Down Expand Up @@ -376,9 +378,31 @@ class SmoSolver {
math_t diff_prev;
int n_small_diff;
int nochange_steps;
int n_increased_diff;
int n_iter;
bool report_increased_diff;

bool CheckStoppingCondition(math_t diff) {
// TODO improve stopping condition to detect oscillations, see Issue #947
if (diff > diff_prev * 1.5 && n_iter > 0) {
// Ideally, diff should decrease monotonically. In practice we can have
// small fluctuations (10% increase is not uncommon). Here we consider a
// 50% increase in the diff value large enough to indicate a problem.
// The 50% value is an educated guess that triggers the convergence debug
// message for problematic use cases while avoids false alarms in many
// other cases.
n_increased_diff++;
}
if (report_increased_diff && n_iter > 100 &&
n_increased_diff > n_iter * 0.1) {
CUML_LOG_DEBUG(
"Solver is not converging monotonically. This might be caused by "
"insufficient normalization of the feature columns. In that case "
"MinMaxScaler((0,1)) could help. Alternatively, for nonlinear kernels, "
"you can try to increase the gamma parameter. To limit execution time, "
"you can also adjust the number of iterations using the max_iter "
"parameter.");
report_increased_diff = false;
}
bool keep_going = true;
if (abs(diff - diff_prev) < 0.001 * tol) {
n_small_diff++;
Expand Down