Skip to content

Commit

Permalink
Add New ET and Appendix pages
Browse files Browse the repository at this point in the history
  • Loading branch information
@michaelfrising
michaelfrising committed Jun 10, 2024
1 parent c8ccfa3 commit 3e61ed2
Show file tree
Hide file tree
Showing 12 changed files with 808 additions and 7 deletions.
Binary file modified .DS_Store
View file
Binary file not shown.
2 changes: 0 additions & 2 deletions ...l--Converters--Markdown/24/47a7decfa5d35f25aef3941b0111c382731e60c408224e7d95bad539d8ffe5
View file

This file was deleted.

2 changes: 0 additions & 2 deletions ...l--Converters--Markdown/2a/1c643656e3a6e1bec4b78fed528c33e6f9acdc7c520b7749e1fd6e3893f3fe
View file

This file was deleted.

7 changes: 7 additions & 0 deletions _data/navbar.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,13 @@ assigned:
show_in_menu: true
show_in_footer: false

-
text: ET Framework
href: pages/et-framework.md
permalink: /et-framework/
show_in_menu: true
show_in_footer: false

-
text: Blog
href: pages/blog.md
Expand Down
209 changes: 209 additions & 0 deletions _layouts/artificial-intelligence.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
---
layout: full-width
---

<!-- Main In Page Navigation -->
<section class="fedramp-page-container position-sticky resources-nav padding-top-2">
<div class="full-row grid-row">
<div class="full-col tablet:grid-col-4 tablet:padding-y-2 tablet:padding-right-6">
<div id="main-header" class="lavalamp-menu">
<nav>
<ul class="in-page-nav-menu" id="top-menu">
<li><a href="#ai">Artificial Intelligence</a></li>
<li><a href="#definitions">Definitions</a></li>
<li><a href="#criteria-for-all-capabilities">Criteria for All Capabilities</a></li>
<li><a href="#et-criteria-for-prioritization">ET Criteria for Prioritization- Artificial Intelligence</a></li>
<li><a href="#et-cso">ET CSO - FedRAMP Model Card guidance</a></li>
</ul>
<div class="lavalamp"></div>
</nav>
</div>
</div>
</div>
</section>

<!-- Mobile Navigation
<section class="fedramp-page-container position-relative resources-nav-mobile padding-bottom-2 padding-top-4">
<div class="full-row grid-row">
<div class="full-col">
<div>
<nav>
<h4 class="mobile-nav-title">You can also browse the topics below.</h4>
<ul class="in-page-nav-menu" id="top-menu-mobile">
<li><a href="#faq-general">General</a></li>
<li><a href="#faq-federal-agencies">Federal Agencies</a></li>
<li><a href="#faq-cloud-service-providers">Cloud Service Providers</a></li>
<li><a href="#faq-third-party">Third Party Assessors</a></li>
<li><a href="#faq-authorization">Authorization</a></li>
<li><a href="#faq-continous-monitoring">Continuous Monitoring</a></li>
<li><a href="#faq-acquisition">Acquisition</a></li>
<li><a href="#faq-security">Security</a></li>0
<li><a href="#faq-rev-5">Rev. 5</a></li>
</ul>
<div class="lavalamp"></div>
</nav>
</div>
</div>
</div>
</section>
-->

<section class="fedramp-page-container in-page-nav-section">
<div class="grid-container">
<div class="full-row grid-row grid-gap">
<div class="full-col tablet:grid-col-8 tablet:grid-offset-4 nav-main-section">

<div id="ai" class="margin-top-6">
<h2>Artificial Intelligence</h2>
<p>FedRAMP is prioritizing Generative AI, specifically when used for <strong>chat interfaces and code generation and debugging tools</strong> that use Large Language Models (LLMs), and <strong>prompt-based image generation</strong>. This applies both to use cases where a human is directly interfacing with the service, and when the service is accessed using associated application programming interfaces (APIs).</p>

<p>To ensure that FedRAMP is appropriately focused on CSOs that offer these services, FedRAMP will request that CSPs include service-related documentation in the form of a model card that provides agencies easily understandable information about the models provided by a vendor, as well as potential uses and limitations. The template model card that FedRAMP requests is included in this <a href="https://github.com/GSA/fedramp-ai" target="_blank" rel="nofollow">GitHub repository</a> for Artificial Intelligence. The FedRAMP PMO will perform a qualitative review of this CSP provided model card(s) to ensure necessary fields are completed, responses are relevant to the field, and clearly presented. </p>
</div>

<div id="definitions" class="margin-top-6">
<h2>Definitions</h2>
<p><strong>Artificial intelligence (AI)</strong> (as defined in 15 U.S.C. 9401(3)): A machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. Artificial intelligence systems use machine- and human-based inputs to perceive real and virtual environments; abstract such perceptions into models through analysis in an automated manner; and use model inference to formulate options for information or action.</p>

<p><strong>AI model:</strong> A component of an information system that implements AI technology and uses computational, statistical, or machine-learning techniques to produce outputs from a given set of inputs.</p>

<p><strong>AI model card:</strong> Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information.</p>

<p><strong>Foundation model:</strong> An AI model that is trained on broad data, generally uses self-supervision, contains at least tens of billions of parameters, and is applicable across a wide range of contexts.</p>

<p><strong>Generative AI:</strong> The class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital content.</p>
</div>

<div id="criteria-for-all-capabilities" class="margin-top-6">
<h2>Criteria for All Capabilities</h2>
<p><strong>Primary Purpose:</strong> The core features of the CSP’s offering are to allow end-users or developers to interact with generative AI capabilities. CSPs with generative AI capabilities embedded within a broader product may not receive prioritization.</p>

<p><strong>Use of Algorithms:</strong> The offering utilizes a foundation model as an underlying technology for its generative AI capabilities. These models will generally have been released after September 2022. Use of a particular algorithm or technical approach is not as important as the performance of the generative AI to produce accurate, valid, and reliable outputs.</p>

<p><strong>Features align to capabilities:</strong> The CSO meets one or more of the following capabilities: 1) large language model-based chat interfaces, 2) code-generation and debugging tools, and 3) prompt-based image generators. Specifically, as stated in Executive Order 14110, “…generative AI offerings that have the primary purpose of providing large language model-based chat interfaces, code-generation and debugging tools, and associated application programming interfaces, as well as prompt-based image generators…”. </p>

<p><strong>CSO Prioritization Target:</strong> The number of CSOs that will be targeted for prioritized review process for each emerging technology capability.</p>
</div>

<div id="et-criteria-for-prioritization" class="margin-top-6">
<h2>A.1 ET Criteria for Prioritization- Artificial Intelligence</h2>

<h3>A.1.1 Chat interfaces </h3>

<p><strong>Description:</strong> Products that deliver conversational chat interfaces to provide text prompts to the user and then use information consumed by the large language model (LLM) to produce aggregate text-based content on behalf of the user.</p>

<p></strong>Technical Characteristics:</strong> The offering should be capable of discerning meaning from open-ended user inputs and provide an appropriate response. The interface may provide an externally accessible API facilitating integration and use by other systems.</p>

<p><strong>Capability Prioritization Target:</strong> Three cloud service offerings</p>

<h3>A.1.2 Code generation and debugging tools </h3>

<p><strong>Description:</strong> A tool used by software developers to help them with creating and debugging software. </p>

<p><strong>Technical Characteristics:</strong> The offering should be capable of auto-detecting relevant programming language based off of submitted code snippets or direct prompts. The offering should be able to generate complete methods, subroutines or functions based on input prompts. The offering can refine responses if requested by the user and can debug code inputs. The offering may be able to provide links to additional documentation or directly explain output with appropriate attribution or citation. The offering may be able to translate code into a newer version or into another programming language. The offering may be able to generate code off of describing a potential problem or use-case and generate suggested code. The interface may provide an externally accessible API facilitating integration and use by other systems.</p>

<p><strong>Capability Prioritization Target:</strong> Three cloud service offerings</p>

<h3>A.1.3 Prompt-based image generators</h3>

<p><strong>Description:</strong> A product that takes text or photographic input and generates new images or videos based on those inputs. </p>

<p><strong>Technical Characteristics:</strong> The offering should generate new visual content by recapitulating pixels based on patterns in the training data. The offering may be able to modify an input image based on accompanying text. The offering may be able to generate the output’s subject in multiple graphical or artistic styles. The interface may provide an externally accessible API facilitating integration and use by other systems.</p>

<p><strong>Capability Prioritization Target:</strong> Three cloud service offerings</p>

<h3>A.1.4 General Purpose API offerings that facilitate the integration of chat interface, code generation and debugging tools, or prompt-based image generation capabilities into new or existing systems.</h3>

<p><strong>Description:</strong> A general purpose API-based solution available to developers and system integrators to incorporate chat, code generation, or image generation capabilities into their new or existing products and services.</p>

<p><strong>Technical Characteristics:</strong> The offering should meet the requirements for chat interfaces, code generation, or prompt-based image generation above. The solution must provide a general purpose model(s). APIs must allow external integration and use by both government and commercial systems. The offering must include well-formed documentation with examples for system integrations. The leveraged APIs must support either REST (representational state transfer) or RPC (Remote Procedure Call). The API-based solution may allow for further refinement, training, or fine tuning by the consuming system. </p>

<p>Note: Offerings that only do not make the API externally available, and only leverage or utilize the API from within their offering are not considered for this prioritization category.</p>

<p><strong>Capability Prioritization Target:</strong> Three cloud service offerings</p>
</div>

<div id="et-cso" class="margin-top-6">
<h2>A.2 ET CSO - FedRAMP Model Card guidance</h2>

<h3>FedRAMP model card template</h3>

<p>FedRAMP is adopting the Hugging Face model card template as a standard way to provide agencies with easily consumable information about the models provided by a vendor, potential uses, limitations, as well as relevant benchmarks.</p>

<p>The provided model card template can be located here in <a href="https://github.com/GSA/fedramp-ai" target="_blank" rel="nofollow">GitHub</a>. FedRAMP PMO will be doing a qualitative review of the provided model card and will use this assessment in contributing to the prioritization decision. The PMO will verify completeness of data provided to the Mandatory fields. Optional Fields will not be considered as part of the review.</p>

<p>The following areas and fields are required to be completed:</p>
<ul>
<li>Model Details
<ul>
<li>Model Description & non-optional subfields</li>
</ul>
</li>
<li>Uses
<ul>
<li>Direct Use</li>
<li>Out-of-Scope Use</li>
</ul>
</li>
<li>Bias, Risks, and Limitations
<ul>
<li>Recommendations</li>
</ul>
</li>
<li>How to get started with the model</li>
<li>Training Details
<ul>
<li>Training Data</li>
<li>Training Procedure</li>
<li>Training Hyperparameters</li>
</ul>
</li>
<li>Evaluation
<ul>
<li>Testing Data, Factors, & Metrics</li>
<li>Testing Data</li>
<li>Factors</li>
<li>Metrics</li>
<li>Results</li>
<li>Summary</li>
</ul>
</li>
<li>Model Architecture and Objective</li>
<li>Compute Infrastructure
<ul>
<li>Hardware</li>
<li>Software</li>
</ul>
</li>
<li>Model Card contact</li>
</ul>

<h3>A.2.2 Publicly available example model cards (WebSite FAQ)</h3>

<p>These are a sampling of reasonable, public or open source model cards that includes information that reflects expected content in a model card. FedRAMP has elected to use the Hugging Face model card as its base. The examples provided below may not follow the Hugging Face model card standard and are provided as illustrative examples. Offerers are expected to use the FedRAMP model card template and must provide a response to all non-optional fields.</p>
<ul>
<li><a href="https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md" target="_blank" rel="nofollow">Hugging Face model card template</a></li>
<li><a href="https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md" target="_blank" rel="nofollow">Meta Llama3 family of LLMs model card</a></li>
<li><a href="https://github.com/meta-llama/codellama/blob/main/MODEL_CARD.md" target="_blank" rel="nofollow">Meta Code Llama model card</a></li>
<li><a href="https://ai.google.dev/gemma/docs/model_card" target="_blank" rel="nofollow">Google Gemma model card</a></li>
<li><a href="https://www-cdn.anthropic.com/5c49cc247484cecf107c699baf29250302e5da70/claude-2-model-card.pdf" target="_blank" rel="nofollow">Anthropic Claude model card with focus on Claude2</a></li>
<li><a href="https://huggingface.co/lmsys/vicuna-7b-v1.5" target="_blank" rel="nofollow">LMSYS Vicuna model card</a></li>
<li><a href="https://huggingface.co/openai-community/gpt2" target="_blank" rel="nofollow">openai gpt2 model cardxx</a></li>
<li><a href="https://huggingface.co/hpcai-tech/Colossal-LLaMA-2-7b-base/blob/main/README.md" target="_blank" rel="nofollow">HPCAI Colossal Llama model card</a></li>
</ul>

<h3>A.2.3 Model card writing utilities (WebSite FAQ)</h3>

<p>Below are some sample model card writing guides or utilities that may be useful to CSPs in developing their model card. This is not an endorsement of any included tools or guides. CSPs are encouraged to follow the FedRAMP model card template located in <a href="https://github.com/GSA/fedramp-ai" target="_blank" rel="nofollow"></a>GitHub.</p>

<p><a href="https://huggingface.co/docs/hub/en/model-card-annotated" target="_blank" rel="nofollow">https://huggingface.co/docs/hub/en/model-card-annotated</a></p>

<p>This model card writing tool can be used by a CSP to generate a basic model card and then use an editor which supports markdown, such as visual studio code or <a href="https://hackmd.io/" target="_blank" rel="nofollow">https://hackmd.io/</a> to modify the generated model card.</p>

<p><a href="https://huggingface.co/spaces/huggingface/Model_Cards_Writing_Tool" target="_blank" rel="nofollow">https://huggingface.co/spaces/huggingface/Model_Cards_Writing_Tool</a></p>

</div>

</div>
</div>
</div>
</section>
Loading

0 comments on commit 3e61ed2

Please sign in to comment.