Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit maximum func_name length #2863

Merged
merged 1 commit into from
Mar 20, 2019
Merged

Conversation

ajtulloch
Copy link
Contributor

Consider a GRU cell with approximate sigmoid/tanh nodes, i.e.

dtype = "float32"

def C(x):
    return relay.expr.const(x, dtype)

def approx_sigmoid(v):
    x = relay.abs(v)
    x2 = v * v
    e = C(1.0) + x + x2 * C(0.5658) + C(0.143) * x2 * x2
    e_pos = e / (C(1) + e)
    e_neg = C(1) / (C(1) + e)
    return relay.where(relay.greater_equal(v, C(0.0)), e_pos, e_neg)

def approx_tanh(v):
    x = relay.abs(v)
    x2 = v * v
    e = C(1.0) + x + x2 * C(0.5658) + C(0.143) * x2 * x2
    return relay.sign(v) * (e - C(1) / e) / (e + C(1) / e)

def gru(X, H, W_X, W_H, B, **kwargs):
    XT = relay.nn.bias_add(relay.nn.sparse_dense(X, W_X), B)
    HT = relay.nn.sparse_dense(H, W_H)
    XT_gates = relay.split(XT, indices_or_sections=3, axis=1)
    HT_gates = relay.split(HT, indices_or_sections=3, axis=1)
    u_t = approx_sigmoid(XT_gates[0] + HT_gates[0])
    r_t = approx_sigmoid(XT_gates[1] + HT_gates[1])
    e_t = approx_tanh(r_t * HT_gates[2] + XT_gates[2])
    return u_t * HT_gates[0] + (C(1.0) - u_t) * e_t

The u_t, r_t, e_t, and return value of the GRU cell are currently fused together into a single function called:
fused_add_abs_add_multiply_multiply_add_add_divide_multiply_subtract_add_abs_add_multiply_multiply_add_add_divide_multiply_add_sign_abs_add_multiply_multiply_add_divide_subtract_multiply_add_divide_multiply_add_add. This is pretty ungainly when it shows up in the debug_runtime profiler, perf, etc.

What I'd propose is modifying https://github.com/dmlc/tvm/blob/89acfeb2582e19e20ba031e896f6b1e64ecc2fb1/src/relay/backend/compile_engine.cc#L82-L84 to detect "long" function names, and replacing them with a truncated version plus a hash of the full name. This allows to disambiguate long fusions (due to taking the hash of the original candidate name), while keeping the size bounded.

Does this sound reasonable? It's arguably an interface change, so it's worth double checking this is OK.

cc @tqchen, @jroesch, @yidawang.

@icemelon icemelon merged commit 259aa46 into apache:master Mar 20, 2019
@icemelon
Copy link
Member

LGTM. This is now merged. Thanks @ajtulloch @tqchen .

Laurawly pushed a commit to Laurawly/tvm-1 that referenced this pull request Mar 22, 2019
wweic pushed a commit to wweic/tvm that referenced this pull request Mar 24, 2019
wweic pushed a commit to neo-ai/tvm that referenced this pull request Mar 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants