Add Flutter DevTools tests to test registry. #267

kenzieschmoll · 2023-06-09T17:47:40Z

Original attempt here: #258.

kenzieschmoll · 2023-09-28T20:41:28Z

I'm seeing a discrepancy between the results on the CI and the analysis results I get locally when running this script.

analyze.sh

#!/bin/bash -e
# Script to analyze the devtools repo for the flutter/tests registry
# https://github.com/flutter/tests
# This is executed as a pre-submit check for every PR in flutter/flutter

which dart
dart --version
which flutter
flutter --version

cd tool
flutter pub get
dart bin/devtools_tool.dart pub-get
dart bin/devtools_tool.dart analyze
cd ..

Test results: https://github.com/flutter/tests/actions/runs/6342602352/job/17228711531?pr=267 (starting at line 1231)

>> ./tool/flutter_customer_tests/analyze.sh
# (which dart)
| /home/runner/work/tests/tests/flutter/bin/dart
# (dart --version)
| Dart SDK version: 3.2.0-199.0.dev (dev) (Mon Sep 25 17:02:52 2023 -0700) on "linux_x64"
# (which flutter)
| /home/runner/work/tests/tests/flutter/bin/flutter
# (flutter --version)
| Flutter 3.15.0-11.0.pre.27 • channel master • https://github.com/flutter/flutter.git
| Framework • revision 6cde5da5ea (26 minutes ago) • 2023-09-28 13:44:46 -0400
| Engine • revision d9eaebd058
| Tools • Dart 3.2.0 (build 3.2.0-199.0.dev) • DevTools 2.28.0-dev.12
| Resolving dependencies...
| + args 2.4.2
| + async 2.11.0
| + cli_util 0.3.5 (0.4.0 available)
| + collection 1.18.0
| + crypto 3.0.3
| ! devtools_shared 4.2.0 from path ../packages/devtools_shared (overridden)
| + extension_discovery 2.0.0
| + http 0.13.6 (1.1.0 available)
| + http_parser 4.0.2
| + io 1.0.4
| + js 0.6.7
| + lints 2.1.1
| + logging 1.2.0
| + meta 1.10.0
| + path 1.8.3
| + pool 1.5.1
| + shelf 1.4.1
| + source_span 1.10.0
| + sse 4.1.2
| + stack_trace 1.11.1
| + stream_channel 2.1.2
| + string_scanner 1.2.0
| + term_glyph 1.2.1
| + typed_data 1.3.2
| + usage 4.1.1
| + vm_service 11.10.0
| + web_socket_channel 2.4.0
| + webkit_inspection_protocol 1.2.1
| + yaml 3.1.2
| + yaml_edit 2.1.1
| Changed 29 dependencies!
| 2 packages have newer versions incompatible with dependency constraints.
| Try `flutter pub outdated` for more information.
| Running flutter pub get...
|   case_study/code_size/optimized/code_size_images...
|   case_study/code_size/optimized/code_size_package...
|   case_study/code_size/unoptimized/code_size_images...
|   case_study/code_size/unoptimized/code_size_package...
|   case_study/memory_leaks/images_1...
|   case_study/memory_leaks/leaking_counter_1...
|   case_study/memory_leaks/memory_leak_app...
|   case_study/platform_channel...
|   packages...
|   packages/devtools_app...
|   packages/devtools_app/test/test_infra/fixtures/flutter_app...
|   packages/devtools_app/test/test_infra/fixtures/flutter_error_app...
|   packages/devtools_app/test/test_infra/fixtures/memory_app...
|   packages/devtools_app/test/test_infra/fixtures/provider_app...
|   packages/devtools_app/test/test_infra/fixtures/riverpod_app...
|   packages/devtools_app_shared...
|   packages/devtools_extensions...
|   packages/devtools_extensions/example/app_that_uses_foo...
|   packages/devtools_extensions/example/foo/packages/foo...
|   packages/devtools_extensions/example/foo/packages/foo_devtools_extension...
|   packages/devtools_shared...
|   packages/devtools_test...
|   third_party/packages/ansi_up...
|   third_party/packages/codicon...
|   third_party/packages/perfetto_ui_compiled...
|   third_party/packages/widget_icons...
|   third_party/packages/widget_icons/example...
|   tool...
| Running flutter analyze...
|   case_study/code_size/optimized/code_size_images...
|   case_study/code_size/optimized/code_size_package...
|   case_study/code_size/unoptimized/code_size_images...
|   case_study/code_size/unoptimized/code_size_package...
|   case_study/memory_leaks/images_1...
|   case_study/memory_leaks/leaking_counter_1...
|   case_study/memory_leaks/memory_leak_app...
|   case_study/platform_channel...
|   packages...
|     Analyzing packages...
|        info - devtools_app/test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:71:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|        info - devtools_app/test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:80:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|        info - devtools_app/test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:92:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|        info - devtools_app/test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:110:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|     4 issues found.
|
|   packages/devtools_app...
|     Analyzing devtools_app...
|        info - test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:71:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|        info - test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:80:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|        info - test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:92:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|        info - test/vm_developer/object_inspector/vm_developer_common_widgets_test.dart:110:7 - Don't return 'null' from a function with a return type of 'void'. Try removing the 'null'. - avoid_returning_null_for_void
|     4 issues found.

Local results:

Succeeds with no errors:

 ~/develop/devtools/ [test-registry] ./tool/flutter_customer_tests/analyze.sh 
/Users/kenzieschmoll/develop/flutter/bin/dart
Dart SDK version: 3.2.0-199.0.dev (dev) (Mon Sep 25 17:02:52 2023 -0700) on "macos_x64"
/Users/kenzieschmoll/develop/flutter/bin/flutter
Flutter 3.15.0-11.0.pre.27 • channel [user-branch] • unknown source
Framework • revision 6cde5da5ea (3 hours ago) • 2023-09-28 13:44:46 -0400
Engine • revision d9eaebd058
Tools • Dart 3.2.0 (build 3.2.0-199.0.dev) • DevTools 2.28.0-dev.12
Downloading package sky_engine...                                  446ms
Downloading flutter_patched_sdk tools...                           402ms
Downloading flutter_patched_sdk_product tools...                   314ms
Downloading darwin-x64 tools...                                  1,838ms
Downloading darwin-x64/font-subset tools...                        140ms
Resolving dependencies... 
  cli_util 0.3.5 (0.4.0 available)
! devtools_shared 4.2.0 from path ../packages/devtools_shared (overridden)
  http 0.13.6 (1.1.0 available)
  meta 1.9.1 (1.10.0 available)
Got dependencies!
3 packages have newer versions incompatible with dependency constraints.
Try `flutter pub outdated` for more information.
Running flutter pub get...
  case_study/code_size/optimized/code_size_images... 1.6s
  case_study/code_size/optimized/code_size_package... 1.6s
  case_study/code_size/unoptimized/code_size_images... 1.5s
  case_study/code_size/unoptimized/code_size_package... 1.6s
  case_study/memory_leaks/images_1...  1.5s
  case_study/memory_leaks/leaking_counter_1... 1.5s
  case_study/memory_leaks/memory_leak_app... 1.5s
  case_study/platform_channel...       1.5s
  packages...                          0.9s
  packages/devtools_app...             3.5s
  packages/devtools_app/test/test_infra/fixtures/flutter_app... 1.3s
  packages/devtools_app/test/test_infra/fixtures/flutter_error_app... 1.5s
  packages/devtools_app/test/test_infra/fixtures/memory_app... 1.3s
  packages/devtools_app/test/test_infra/fixtures/provider_app... 1.6s
  packages/devtools_app/test/test_infra/fixtures/riverpod_app... 1.4s
  packages/devtools_app_shared...      1.9s
  packages/devtools_extensions...      1.8s
  packages/devtools_extensions/example/app_that_uses_foo... 1.5s
  packages/devtools_extensions/example/foo/packages/foo... 1.4s
  packages/devtools_extensions/example/foo/packages/foo_devtools_extension... 1.8s
  packages/devtools_shared...          1.7s
  packages/devtools_test...            2.7s
  third_party/packages/ansi_up...      1.1s
  third_party/packages/codicon...      1.3s
  third_party/packages/perfetto_ui_compiled... 0.9s
  third_party/packages/widget_icons... 1.8s
  third_party/packages/widget_icons/example... 1.6s
  tool...                              1.5s
Running flutter analyze...
  case_study/code_size/optimized/code_size_images... 13.8s
  case_study/code_size/optimized/code_size_package... 4.8s
  case_study/code_size/unoptimized/code_size_images... 3.0s
  case_study/code_size/unoptimized/code_size_package... 3.1s
  case_study/memory_leaks/images_1...  2.6s
  case_study/memory_leaks/leaking_counter_1... 2.8s
  case_study/memory_leaks/memory_leak_app... 3.2s
  case_study/platform_channel...       2.9s
  packages...                          31.8s
  packages/devtools_app...             9.8s
  packages/devtools_app/test/test_infra/fixtures/flutter_app... 2.0s
  packages/devtools_app/test/test_infra/fixtures/flutter_error_app... 1.8s
  packages/devtools_app/test/test_infra/fixtures/memory_app... 1.8s
  packages/devtools_app/test/test_infra/fixtures/provider_app... 2.0s
  packages/devtools_app/test/test_infra/fixtures/riverpod_app... 2.0s
  packages/devtools_app_shared...      2.2s
  packages/devtools_extensions...      2.4s
  packages/devtools_extensions/example/app_that_uses_foo... 3.9s
  packages/devtools_extensions/example/foo/packages/foo... 2.6s
  packages/devtools_extensions/example/foo/packages/foo_devtools_extension... 3.3s
  packages/devtools_shared...          1.5s
  packages/devtools_test...            2.8s
  third_party/packages/ansi_up...      1.0s
  third_party/packages/codicon...      2.3s
  third_party/packages/widget_icons... 3.1s
  third_party/packages/widget_icons/example... 2.1s
  tool...                              1.5s

kenzieschmoll · 2023-09-28T23:48:34Z

The 'setup' step added here should unblock this PR flutter/flutter#135726

registry/template.test

Piinks

kenzieschmoll force-pushed the devtools-tests branch from 1e25b5e to 0b816d3 Compare June 9, 2023 19:17

kenzieschmoll mentioned this pull request Sep 28, 2023

Add DevTools tests to Flutter Customer Test Registry flutter/devtools#4433

Closed

kenzieschmoll mentioned this pull request Oct 2, 2023

Add support for setup steps in flutter customer tests flutter/flutter#135726

Merged

goderbauer reviewed Oct 2, 2023

View reviewed changes

registry/template.test Outdated Show resolved Hide resolved

kenzieschmoll requested a review from Piinks October 5, 2023 23:14

kenzieschmoll mentioned this pull request Oct 5, 2023

Prepare DevTools tests for running in the Flutter customer test registry flutter/devtools#6505

Merged

kenzieschmoll added 22 commits October 5, 2023 17:55

Add Flutter DevTools tests to test registry.

38c3ce8

new line

6ca3671

use local branch

01ff8f7

fix

6b2f20a

change hash

3a667d7

fix hash

ca64457

new hash

d15dbdf

new hash

42480e4

new

b252a26

test integration tests

8484dd5

fix hash and add update for new packages

85490f5

gen

1d44db8

run analyze

141ee77

fix analyze

c923a1c

analyze w script

9b6d711

chmod

49784b3

pub get

e3adeb1

swap order

ff2ed6e

hash with -e

3599a93

delete bad test

d9848e5

add versions

992166e

next

97d674e

kenzieschmoll added 11 commits October 5, 2023 17:55

ordering

ac63056

Use setup step

94726cb

2 iterations

4d3a304

fix comment

ed5978a

try to run all tests

00f6818

reduce to 1

354e8c4

only run integration tests

5d7ef17

new hash

b241bf1

new hash and add todo

b27d865

rename file

e7cf35a

use flutter devtools master hash

7ada7d4

kenzieschmoll force-pushed the devtools-tests branch from 5d9d386 to 7ada7d4 Compare October 5, 2023 23:55

fix script name

46cace6

kenzieschmoll marked this pull request as ready for review October 9, 2023 17:34

Piinks approved these changes Oct 10, 2023

View reviewed changes

Piinks merged commit efbe6dc into flutter:main Oct 10, 2023
13 checks passed

kenzieschmoll deleted the devtools-tests branch October 10, 2023 17:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Flutter DevTools tests to test registry. #267

Add Flutter DevTools tests to test registry. #267

kenzieschmoll commented Jun 9, 2023

kenzieschmoll commented Sep 28, 2023

kenzieschmoll commented Sep 28, 2023

Piinks left a comment

Add Flutter DevTools tests to test registry. #267

Add Flutter DevTools tests to test registry. #267

Conversation

kenzieschmoll commented Jun 9, 2023

kenzieschmoll commented Sep 28, 2023

kenzieschmoll commented Sep 28, 2023

Piinks left a comment

Choose a reason for hiding this comment