Built package EthicsML-0.0.1.vsix and updated documentation

CODE_OF_CONDUCT.md

@@ -0,0 +1,105 @@
+
+# Code of Conduct
+
+
+## Short Version
+We as members, contributors, and leaders pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Need Help?
+If ANY behavior makes you uncomfortable, or you believe it breaches the intent of this code of conduct, please contact a project maintainer:
+
+* [Sergio Morales](https://github.com/tirafrutas/) (he/him)
+
+---
+
+## Long Version
+
+As contributors and maintainers in this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Demonstrating empathy and kindness toward other people
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Giving and gracefully accepting constructive feedback
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* Violent threats or language directed against another person.
+* Sexist, racist, or otherwise discriminatory jokes and language.
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, including contributors, maintainers, administrators, and any kind of participant in the project. 
+
+Project maintainers will enforce this code at all times. We expect cooperation from all participants to ensure a safe environment for everyone.
+
+The Code of Conduct, and the project leaders, can only address behavior in the present, not past behavior or fears of what someone might do based on past behavior.
+
+## Enforcement
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project maintainers at:
+
+* [Sergio Morales](https://github.com/tirafrutas/) (he/him)
+
+Complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. 
+
+The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Enforcement Guidelines
+Project maintainers will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/
+
+Community Impact Guidelines were inspired by [Mozilla’s code of conduct enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

GOVERNANCE.md

@@ -0,0 +1,38 @@
+
+# Project Governance
+
+The development and community management of the project will follow the governance rules described in this document.
+
+## Project Maintainers
+
+Project maintainers have admin access to the GitHub repository. The team of project maintainer is the following:
+
+* [Sergio Morales](https://github.com/tirafrutas/)
+
+## Project Collaborators & Contributors
+
+Any member willing to participate in the development of the project will be considered as a collaborator. 
+
+Collaborators may propose changes to the project's source code. The mechanism to propose such a change is a GitHub pull request. A collaborator proposing a pull request is considered a contributor. Project maintainers will review and merge pull requests.
+
+## Issue governance
+
+Both collaborators and project maintainers may propose issues. The participation in the issue discussion is open and must follow the [Code of Conduct](CODE_OF_CONDUCT.md).
+
+The group of project maintainers will be responsible for assigning labels to issues, as well as assign the issue to a project maintainer or contributor.
+
+The group of project maintainers commit to give an answer to any issue in a period of time of 48 hours. 
+
+## Pull Request governance
+
+Both collaborators and project maintainers may propose pull requests. When a collaborator proposes a pull request, is considered contributor.
+
+Pull requests should comply with the template provided. The assignment of labels and assignees to the pull request is the responsibility of the project maintainers.
+
+The group of project maintainers commit to give an answer to any pull request in a period of time of 48 hours. 
+
+The decision of accepting (or rejecting) a pull request will be taken by the group of project maintainers. The decision will be based on the following criteria:
+
+* One project maintainer must approve a pull request before the pull request can be merged. 
+* Approving a pull request indicates that the contributor accepts responsibility for the change. 
+* If a project maintainer opposes a pull request, the pull request cannot be merged (i.e., _veto_ behavior). Often, discussions or further changes result in collaborators removing their opposition.

LICENSE.md

@@ -0,0 +1,22 @@
+
+MIT License
+
+Copyright (c) 2023 SOM Research Lab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,114 @@
-# Ethics ML
+# EthicsML
 
-Ethics ML is a Visual Studio Code extension for defining ethical requirements and test scenarios, according to a domain-specific model.
+Ethics ML is a Visual Studio Code extension for defining non-bias requirements and test scenarios, according to a domain-specific language (DSL).
+
+The DSL provides with the necessary constructs for modeling requirements to address non-discrimination issues, and to target particular communities that could be potentially discriminated in case a bias is present.
+
+## Installation 
+
+Use the packaged release of the plugin in this [repository](https://github.com/SOM-Research/EthicsML) that can be found at the root of the project. The file is `EthicsML-0.0.1.vsix`.
+
+Open your terminal (or the terminal inside the VSCode) and enter:
+
+```
+git clone https://github.com/SOM-Research/EthicsML.git ethics-ml
+cd ethics-ml 
+code --install-extension EthicsML-0.0.1.vsix
+```
+
+## Usage
+
+### Defining Ethical Requirements
+
+An `ethicalConcern` is a subject in which a potential particular situation may result in a moral conflict. Examples of ethical concerns are: gender discrimination, racism, ageism, LGTBIQ+phobia, xenophobia, political partiality or religion intolerance. A bias occurs when a particular `sensitiveCommunity` (group of people akin by a set of specific features) related to an ethical concern is judged unequally regarding any other. For instance, to confirm the absence of gender discrimination, we may consider `men` and `women`. Each sensitive community has a literal per language supported.
+
+An `ethicalRequirement` is set to address an ethical concern for a concrete subset of their sensitive communities, in a particular set of `languages`. Each ethical requirement includes a `rationale` to clarify the necessity of such constraint and justify possible restrictions.
+
+A `tolerance` level is defined to delimit a minimum threshold of successful test cases for concluding that the requirement is fulfilled. To detect if there is a significant quantitative difference between communities in response to the same question, we use a `delta`: the maximum admissible variance between the maximum and the minimum values provided by the LLM.
+
+We may want to check if an LLM reflects any inequality in the actual society, or if it is otherwise capable to provide answers that are ethically correct despite the unbalances within its observations. Therefore, we would have to select which `reflections` to contemplate: observational and/or utopian. Another dimension to consider is whether we want to provide the LLM with all the potential responses or, on the contrary, we _hide_ the expected valid responses and wait for the LLM to discover them. Therefore, we would have to select which `inputs` to use: constrained and/or verbose.
+
+A set of ethical requirements is included in a `requirementsModel`.
+
+An example of grammar instance for ethical requirements would be:
+
+```
+language american "en" "us"
+
+sensitiveCommunity Men
+    american "men"
+sensitiveCommunity Women
+    american "women"
+ethicalConcern Sexism "Reinforcement of stereotypes, unequal treatment, or denial of opportunities to a person based on their gender, typically directed against women." [Men, Women]
+
+ethicalRequirement
+  name: REQ6
+  rationale: "The LLM must not display any bias with respect to people based on their gender."
+  languages: [american]
+  tolerance: 0.9
+  delta: 0.02
+  concern: Sexism
+  communities: [Men, Women]
+  inputs: [constrained, verbose]
+  reflections: [observational, utopian]
+
+requirementsModel
+  requirements: [REQ6]
+```
+
+### Defining Test Scenarios
+
+We define a `testScenario` to assess if LLMs are compliant with the ethical requirements included in the requirements model.
+
+We define a test scenario at a specific `timestamp` and decide the number of templates (`nTemplates`) to use for each combination of reflections and inputs for every ethical requirement to test. We select the `temperature` to test an LLM, which directly impacts the creativity and consistency of the outputs it provides. The higher the temperature, the least predictable is the LLM response. Also, we decide the maximum number of `tokens` an LLM can generate.
+
+For a test case that has failed according to the system's initial evaluation, we may decide to leverage a second LLM (`useLLMEval`) to review the response and dictate a final assessment. We also specify the number of retries (`nRetries`) the system will perform for a test case until getting a valid response from the LLM.
+
+The test scenario will be executed against a set of `aiModels` or LLMs.
+
+An example of grammar instance for a test scenario would be:
+
+```
+testScenario
+  name: readmeexample
+  timestamp: 2451242343243
+  nTemplates: 60
+  nRetries: 3
+  temperature: 1.0
+  tokens: 60
+  useLLMEval: True
+  aiModels: [OpenAIGPT35Turbo, OpenAIGPT4Turbo, OpenAIGPT4o]
+```
+
+## Contributing
+
+This project is being development as part of a research line of the [SOM Research Lab](https://som-research.github.io/), but we are open to contributions from the community. If you are interested in contributing to this project, please read the [GOVERNANCE.md](GOVERNANCE.md) and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) documents.
+
+### Repository Structure
+
+The following tree shows the list of the repository's relevant sections:
+
+- The *examples* folders contains an example of an ethical requirements model created using the DSL.
+- The *src* folder contains the project's source code:
+  - The *cli* folder is the generated grammar and AST from Langium. You may not want to dive in it as it is a generated asset.
+  - The *extension* folder contains all the code of the generation service. Could be a good place to start if you want to improve the generation of the tool.
+  - The *language* folder contains all the language features, and the grammar declaration. If you want to improve the grammar, or some of the features the plugin offers here is the place you may want to start.
+
+#### Debugging the extensions
+
+This repo comes with an already built-in config to debug. Just go to Debug in VSCode, and launch the Extension config. Please check your port 6009 is free.
+
+For more information about how the framework works and how the language can be extended, please refer to https://github.com/langium/langium or the VSCode extension API documentation https://code.visualstudio.com/api
+
+## Code of Conduct
+
+At SOM Research Lab we are dedicated to creating and maintaining welcoming, inclusive, safe, and harassment-free development spaces. Anyone participating will be subject to and agrees to sign on to our [Code of Conduct](CODE_OF_CONDUCT.md).
+
+## License
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+The source code for the site is licensed under the MIT License, which you can find in the LICENSE.md file.
+
+All graphical assets are licensed under the
+[Creative Commons Attribution 3.0 Unported License](https://creativecommons.org/licenses/by/3.0/).

package.json

@@ -11,6 +11,11 @@
     "author": {
         "name": "Sergio Morales"
     },
+    "homepage": "https://github.com/SOM-Research/EthicsML",
+    "repository": {
+        "type": "git",
+        "url": "https://github.com/SOM-Research/EthicsML"
+    },
     "keywords": [
         "Generative AI",
         "Requirements",