Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP][transformer] bring llm component #2363

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

[WIP][transformer] bring llm component #2363

wants to merge 9 commits into from

Commits on Feb 22, 2024

  1. [transformer] add gated mlp

    @Mddct
    Mddct committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    bbedafb View commit details
    Browse the repository at this point in the history

  2. gated mlp works

    @Mddct
    Mddct committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    a17fc45 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2024

  1. support linear bias

    @Mddct
    Mddct committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    4d20b69 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. support rms norm

    @Mddct
    Mddct committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    1afd9de View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. fix conformer rms+bias+mlp

    @Mddct
    Mddct committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    abc1c48 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. support multi query attention

    @Mddct
    Mddct committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    412e51b View commit details
    Browse the repository at this point in the history

  2. support rope

    @Mddct
    Mddct committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    0dc48f1 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. fix init rope attention and rope

    @Mddct
    Mddct committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    6dec5bd View commit details
    Browse the repository at this point in the history

  2. fix rope attention decoding

    @Mddct
    Mddct committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    ddf648b View commit details
    Browse the repository at this point in the history