Skip to content

Commit

Permalink
feat: add About screen (#723)
Browse files Browse the repository at this point in the history
* Add About screen

* Disable markdown-link-check for a link

Error: Exceeded maxRedirects. Probably stuck in a redirect loop https://developer.android.com/ndk/guides/neuralnetworks
  • Loading branch information
anhappdev authored Jun 6, 2023
1 parent 44d930d commit da4dd40
Show file tree
Hide file tree
Showing 6 changed files with 217 additions and 86 deletions.
66 changes: 66 additions & 0 deletions flutter/assets/text/about.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# MLPerf Mobile App

The MLPerf Mobile App supports the following runtime backends to run AI tasks on different mobile devices.

## TFLite Backend

TensorFlow Lite is Google’s cross-platform framework for deploying machine learning (ML) models on mobile and embedded systems. The runtime uses a set of custom operators that are optimized for efficiency, offering lower latency and smaller binary size compared to the full TensorFlow runtime. It can perform inference on CPUs using optimized kernels that take advantage of ARM’s Neon vector instructions on any platform. The runtime can also execute on a variety of accelerator delegates to take advantage of specialized APIs and hardware. For example, it can target GPUs for iOS and Android, Core ML for newer iOS devices, and the Android NNAPI delegate which can target GPUs, DSPs, NPUs, and custom back-ends.
When invoked on the CPU it will use 4 threads.

## Google Pixel

Google Pixel smartphones use the Android NNAPI run time. The Android Neural Networks API (NNAPI) is an Android C API designed for running computationally intensive operations for machine learning on Android devices. NNAPI is designed to provide a base layer of functionality for higher-level machine learning frameworks, such as TensorFlow Lite and Caffe2, that build and train neural networks.

The MLPerf app uses NNAPI on the following families:

* Pixel 6 and Pixel 6 PRO using Tensor G1 SOC
* Pixel 7 and Pixel 7 PRO using Tensor G2 SOC

<!-- markdown-link-check-disable-next-line -->
For More Info: [https://developer.android.com/ndk/guides/neuralnetworks](https://developer.android.com/ndk/guides/neuralnetworks)

## MediaTek Neuron Delegate

The MediaTek Neuron Delegate is a TensorFlow Lite delegate designed to work with the MediaTek Neuron runtime, providing access to more underlying capabilities and features for user applications. The Neuron Delegate is modeled after the TensorFlow Lite NNAPI delegate, but it communicates directly with MediaTek's Neuron runtime to deliver better performance. MediaTek processors include support for INT8, INT16, and FP16 calculations, and can work on dedicated AI tasks or hybrid tasks such as AI-ISP, AI-GPU, and AI-Video in conjunction with other dedicated compute resources.

The MLPerf app uses the Neuron delegate on the following families:

* Dimensity 9000 series (9000/9000+/9200/9200+)
* Dimensity 8000 series (8000/8020/8050/8100/8200)
* Dimensity 7000 series (7020/7050/7200)

Link to MediaTek Neuron Delegate: [https://github.com/MediaTek-NeuroPilot/tflite-neuron-delegate](https://github.com/MediaTek-NeuroPilot/tflite-neuron-delegate)

Link to MediaTek Dimensity 9000/8000/7000 Series: [https://www.mediatek.com/products/smartphones/dimensity-5g](https://www.mediatek.com/products/smartphones/dimensity-5g)

## Qualcomm

The Qualcomm Neural Processing SDK (SNPE) is a software accelerated runtime and toolkit for optimizing and executing neural networks on devices with Snapdragon processors. It can take advantage of many different execution resources in Snapdragon processors including the Qualcomm Kryo™ CPU, Qualcomm Adreno™ GPU, Qualcomm Hexagon™ Processor. The Qualcomm backend provides the flexibility to users in choosing the best approach for their applications.

The MLPerf Application uses the SNPE Hexagon Processor runtimes on the following families:

* Snapdragon 8 Gen 2
* Snapdragon 7+ Gen 2
* Snapdragon 8+ Gen 1
* Snapdragon 8 Gen 1
* Snapdragon 7 Gen 1
* Snapdragon 888
* Snapdragon 778

The Qualcomm backend utilizes the Snapdragon platform's AI hardware to accelerate neural network execution.

Link to Qualcomm Neural Processing SDK: [https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk)

Link to Snapdragon Platform: [https://www.qualcomm.com/snapdragon](https://www.qualcomm.com/snapdragon)

## Samsung Exynos Neural Network (ENN)

Exynos Neural Network (ENN) is Samsung's Exynos runtime, toolkit, and set of libraries for optimizing and executing pre-trained neural networks on Samsung devices. The ENN SDK is designed to take maximum advantage of the underlying hardware components for AI tasks including CPU, GPU, DSP, and NPU.

Official website URL: [https://developer.samsung.com/neural/overview.html](https://developer.samsung.com/neural/overview.html)

The MLPerf Mobile Benchmarking App uses the Exynos Neural Network SDK on the following families of Exynos processors [https://semiconductor.samsung.com/processor/](https://semiconductor.samsung.com/processor/):

* Exynos 990
* Exynos 2100
* Exynos 2200
11 changes: 6 additions & 5 deletions flutter/lib/l10n/app_en.arb
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
"settingsCooldownSubtitle": "Pause <cooldownPause> minutes before running each benchmark to avoid thermal throttling.",
"settingsPrivacyPolicy": "Privacy policy",
"settingsEula": "End User License Agreement",
"settingsAbout": "About",
"settingsTaskConfigTitle": "Task configuration",
"settingsTaskConfigInternetResource": "downloadable",
"settingsTaskConfigLocalResource": "local",
Expand Down Expand Up @@ -94,11 +95,11 @@
"benchInfoImageSegmentation": "Image Segmentation",
"benchInfoLanguageProcessing": "Language Processing",
"benchInfoSuperResolution": "Super Resolution",
"benchInfoImageClassificationDesc": "Image classification picks the best label to describe an input image and is commonly used for photo search and text extraction. The MobileNetEdgeTPU reference model is evaluated on the ImageNet 2012 validation dataset and requires 74.66% (98% of FP32 accuracy) Top-1 accuracy (app uses a different dataset).\n\nThe MobileNetEdgeTPU network is a descendent of the MobileNet-v2 family that is optimized for low-latency and mobile accelerators. The MobileNetEdgeTPU model architecture is based on convolutional layers with inverted residuals and linear bottlenecks, similar to MobileNet v2, but is optimized by introducing fused inverted bottleneck convolutions to improve hardware utilization, and removing hard-swish and squeeze-and-excite blocks.",
"benchInfoObjectDetectionDesc": "Object detection draws bounding boxes around recognized objects in an input image, assigning each one a label. This is a common approach for identifying objects in photos, and automotive safety. The reference model is a Single Shot Detector based MobileDet model operating on the COCO 2017 validation dataset with a mean Average Precision (mAP) of 27.1 (95% of FP32 accuracy).\n\nMobileDet-SSD provides substantial improvements in the latency-accuracy trade-off by incorporating regular convolutions along with depthwise-separable convolutions in the search space for the object detection task, and effectively placing them in the network via neural architecture search (from: https://arxiv.org/pdf/2004.14525.pdf)",
"benchInfoImageSegmentationDesc": "Semantic image segmentation divides an input image into labeled regions at pixel granularity (e.g., labeling all pixels associated with a car and labeling every pixel in the image), and is used for complex image scene parsing to power downstream image analysis and processing.\n\nThe reference model is the MOSAIC network, which was designed with current mobile processors and accelerators in mind, and uses a 512x512 input image resolution. The backbone feature extractor uses a modified version of MobileNet-v3.5-MultiHardware which was developed with Network Architecture Search (NAS) to optimize inference speed on various mobile processors and accelerators. Performance and accuracy are evaluated on the ADE20K validation dataset. The benchmark requires a mean Intersection Over Union (mIoU) value of 57.36% (96% of FP32 accuracy).\n\nMOSAIC uses an efficient multi-scale context encoder and a light-weight hybrid decoder to recover spatial details from aggregated information. MOSAIC produces state-of-the-art results balancing between model accuracy and speed across CPUs, GPUs, DSPs and NPUs.",
"benchInfoLanguageProcessingDesc": "Question Answering finds the best answer to an input question based on a body of text. The reference model, MobileBERT, is evaluated on the Stanford Question Answering Dataset (SQUAD) v1.1 Dev and achieves an F1-score of 90.0.\n\nMobileBERT is a thin, mobile-optimized version of the larger BERT (BERT_LARGE) network, equipped with bottleneck structures and a carefully designed balance between self-attention and feed-forward networks. Like BERT, it is task-agnostic, applicable for a variety of downstream NLP tasks, but for MLPerf we use the variant fine-tuned for Q&A.",
"benchInfoSuperResolutionDesc": "Image Super resolution (SR) aims to retrieve a high resolution (HR) image from a given low resolution (LR) image. Image SR is a common ISP (Image Signal Processing) task, and adopted in many mobile application like digital zoom. This SR task uses a model from the EDSR family introduced in Enhanced Deep Residual Networks for Single Image Super Resolution”, 2017 NTIRE CVPR-workshops. The EDSR block is a simple residual block constituting of a residual connection on one branch, a conv-relu-conv on the other branch.\n\nThis SR task adopts a light EDSR model, called EDSR F32B5, which has 5 EDSR blocks each with 32 Feature Maps, and the final upsampling layer is a depth-to-space operator. EDSR F32B5 is trained for x2 super resolution, on the DIV2k dataset with bicubic downsampling. It is tested on the OpenSR test-set which constitutes of 25 carefully selected FHD (1920x1080) HR images. The minimum accuracy threshold for a valid submission is 33 dB PSNR (Peak Signal to Noise Ratio).",
"benchInfoImageClassificationDesc": "Image classification picks the best label to describe an input image and is commonly used for photo search and text extraction. The MobileNetEdgeTPU reference model is evaluated on the ImageNet 2012 validation dataset and requires a minimum accuracy of 74.66% (98% of FP32 accuracy of 76.19%) Top-1 accuracy (For Performance measurements, App uses a different dataset).\n\nThe MobileNetEdgeTPU network is a descendent of the MobileNet-v2 family that is optimized for low-latency and mobile accelerators. The MobileNetEdgeTPU model architecture is based on convolutional layers with inverted residuals and linear bottlenecks, similar to MobileNet v2, but is optimized by introducing fused inverted bottleneck convolutions to improve hardware utilization, and removing hard-swish and squeeze-and-excite blocks.\n\nThe offline variant of image classification has no latency constraints and typically uses batched inference and has higher throughput.",
"benchInfoObjectDetectionDesc": "Object detection draws bounding boxes around recognized objects in an input image, assigning each one a label. This is a common approach for identifying objects in photos, and automotive safety. Since v1.0, our reference model has been updated to MobileDets (from v0.7 model, Single Shot Detector with a MobileNet-v2 feature extractor operating). MobileDets are trained on the COCO 2017 validation dataset. The MobileDets Object Detection task is evaluated on the COCO 2017 dataset with an input image resolution of 320x320. It requires a minimum mean Average Precision (mAP) of 27.075 (95% of FP32 mAP of 28.5%), which is significantly higher than that of the previous model.\n\nMobileDets are searched for object detection. A key feature of MobileDets is that the search space includes both inverted bottleneck blocks and regular convolution operations to help improve the accuracy-latency trade-off on several hardware accelerators.",
"benchInfoImageSegmentationDesc": "Semantic image segmentation partitions an input image into labeled objects at pixel granularity, and is used for complex image manipulation such as red-eye reduction as well as automotive and medical applications. The reference model is the MOSAIC network paired with a tailored feature extraction backbone. It operates on 512x512 resolution input images from the ADE20K validation set and requires a minimum mean Intersection Over Union (mIoU) value of 57.36% (96% of FP32 mIoU of 59.75%), significantly higher than the previous segmentation model (MobileNetv2-Deeplabv3+).\n\nMOSAIC employs a simple asymmetric encoder-decoder structure which consists of an efficient multi-scale context encoder and a light-weight hybrid decoder to recover spatial details from aggregated information with multiple lateral connections between the two. The feature extractor is a variant of MobileNet Multi-Hardware, which is a network built and optimized with neural architecture search. It is further enhanced for image segmentation by reducing the output stride, adding dilated convolutions at the end stage, and halving the feature channels.",
"benchInfoLanguageProcessingDesc": "Question Answering finds the best answer to an input question based on a body of text, and is commonly employed in applications such as virtual assistants and chatbots. The reference model, MobileBERT, is evaluated on the Stanford Question Answering Dataset (SQUAD) v1.1 Dev-mini. The task requires a minimum F1-score of 87.4% (93% of FP32 F1-score of 93.08%).\n\nMobileBERT is a streamlined, mobile-optimized version of the larger BERT_LARGE network. It features bottleneck structures and a carefully designed balance between self-attention and feed-forward networks. While BERT is task-agnostic and can be applied to various downstream natural language processing tasks, the MobileBERT variant used in MLPerf is specifically fine-tuned for question answering.",
"benchInfoSuperResolutionDesc": "Image Super Resolution (SR) upscales a lower resolution input into a higher resolution output image, enhancing the quality and detail. It is a common task in many mobile applications such as digital zoom. The reference model, EDSR F32B5, is a lightweight member of the Enhanced Deep Super Resolution (EDSR) family that is trained for 2X super resolution on the DIV2K dataset with bicubic downsampling and tested on the OpenSR test-set which comprises 25 selected 1920x1080 HDR images. The benchmark requires a minimum accuracy of 33 dB Peak Signal to Noise Ratio (PSNR) relative to a 33.58 dB accuracy with FP32.\n\nThe EDSR family of models demonstrated excellent performance by winning a super resolution challenge at CVPR 2017. The EDSR F32B5 reference model features five EDSR blocks, each with 32 feature maps. The EDSR block is a simple residual block consisting of a residual connection on one branch and a convolution-ReLU-convolution on the other branch. The final upsampling layer is a depth-to-space operator, which facilitates the x2 super resolution process.",

"progressScreenNamePerformance": "<taskName> (performance)",
"progressScreenNameAccuracy": "<taskName> (accuracy)",
Expand Down
13 changes: 13 additions & 0 deletions flutter/lib/ui/config/config_screen.dart
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import 'package:provider/provider.dart';

import 'package:mlperfbench/benchmark/state.dart';
import 'package:mlperfbench/localizations/app_localizations.dart';
import 'package:mlperfbench/ui/settings/about_screen.dart';

class ConfigScreen extends StatefulWidget {
const ConfigScreen({Key? key}) : super(key: key);
Expand Down Expand Up @@ -52,6 +53,18 @@ class _ConfigScreen extends State<ConfigScreen> {
));
}

childrenList.addAll([
const Divider(),
ListTile(
title: Text(stringResources.settingsAbout),
trailing: const Icon(Icons.chevron_right),
onTap: () async {
await Navigator.of(context).push(MaterialPageRoute(
builder: (context) => const AboutScreen(),
));
}),
]);

return Scaffold(
appBar: AppBar(
title: Text(
Expand Down
49 changes: 49 additions & 0 deletions flutter/lib/ui/settings/about_screen.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import 'package:flutter/material.dart';
import 'package:flutter/services.dart' show rootBundle;

import 'package:flutter_markdown/flutter_markdown.dart';

import 'package:mlperfbench/localizations/app_localizations.dart';

class AboutScreen extends StatelessWidget {
const AboutScreen({Key? key}) : super(key: key);

@override
Widget build(BuildContext context) {
final l10n = AppLocalizations.of(context);
return Scaffold(
appBar: AppBar(title: Text(l10n.settingsAbout)),
body: const Center(child: AboutText()),
);
}
}

class AboutText extends StatefulWidget {
const AboutText({Key? key}) : super(key: key);

@override
State<AboutText> createState() => _AboutText();
}

class _AboutText extends State<AboutText> {
final Future<String> _markdownData =
rootBundle.loadString('assets/text/about.md');

@override
Widget build(BuildContext context) {
return FutureBuilder<String>(
future: _markdownData,
builder: (BuildContext context, AsyncSnapshot<String> snapshot) {
Widget child;
if (snapshot.hasData) {
child = Markdown(data: snapshot.data!);
} else if (snapshot.hasError) {
child = Text('${snapshot.error}');
} else {
child = const CircularProgressIndicator();
}
return Center(child: child);
},
);
}
}
Loading

0 comments on commit da4dd40

Please sign in to comment.