Add android sdk module #9236

shamilovstas · 2024-12-10T22:54:01Z

SUMMARY

This module allows to manage Android SDK packages

ISSUE TYPE

New Module/Plugin Pull Request

COMPONENT NAME

android_sdk

ADDITIONAL INFORMATION

This module uses sdkmanager binary, which is used to install Android SDK packages. Usually, Android SDK packages are installed by Android developers using Android Studio IDE, but in case when a server needs to be set up to be used as a CI/CD runner to build Android applications, sdkmanager command line tool is used instead.
In order to use this module, sdkmanager binary must be present on the target machine. The official way of installing sdkmanager is described here: https://developer.android.com/tools/sdkmanager
There is one caveat I'm not entirely sure how/whether to fix. For some of the packages installed with sdkmanager a license must be accepted beforehand. The accepted licenses are placed inside the directory containing Android SDK, in a subdirectory licenses as a file containing a single string:

[vagrant@alma licenses]$ pwd
/usr/local/android/sdk/licenses
[vagrant@alma licenses]$ ls
android-googletv-license     android-sdk-preview-license
android-sdk-arm-dbt-license  google-gdk-license
android-sdk-license          mips-android-sysimage-license
[vagrant@alma licenses]$ cat android-sdk-license 

24333f8a63b6825ea9c5514f83c2829b004d1fee

There are three ways how a license may be accepted

if, after requesting of installation of a new package, sdkmanager sees that the license for the package has not been accepted, a command line prompt will be displayed that the user is supposed to answer interactively.
sdkmanager --licenses command may be used. In this case sdkmanager prints every license that has not been accepted sequentially and prompts the user after each one.
a file, containing the license as a string (I suppose, it is some sort of a hash), must be placed inside licenses subdirectory for any license that must be accepted (but IMO this is error-prone as there might be new licenses, which can't be known in advance).

The easiest way to accept licenses from an Ansible playbook is to use the following task:

- name: Accept all licenses
  shell: 'yes | sdkmanager --licenses'

I haven't figured a straightforward way of how to accept licenses from inside the module. What I tried was to use data parameter of module.run_command function to send y to stdin, but the problem is that in case if we try to use the 2nd approach from the list above (sdkmanager --license), the sdkmanager tool prints multiple prompts for each license that is not accepted. The data parameter sends y only to the first license prompt, but the remaining prompts hang the module as they are waiting for the user input. Another idea was to use data='y' every time a package is installed (1st approach), but this may also lead to the module hanging as there might be several packages requested for installation, and if they belong to different licenses, this ends up the same way.
So, I decided that the module assumes that all licenses are accepted in advance (this is reflected in the "Notes" section of the documentation).

…fy version for a package while using sdkmanager

…ng 'latest' state

shamilovstas · 2024-12-16T14:51:19Z

@felixfontein @russoz I also have a question regarding BOTMETA.yml. I see that some modules have labels there, but I didn't manage to find the docs describing what these labels are and whether I should add one.

felixfontein · 2024-12-16T20:23:41Z

It might be documented somewhere in the bot's repo / code (https://github.com/ansible-community/collection_bot/), but I'm not sure. Generally I would simply ignore that field. The bot might add corresponding labels to PRs/issues for the module, but that's not something we really use.

russoz · 2024-12-16T21:32:59Z

plugins/module_utils/sdkmanager.py

+    _RE_INSTALLED_PACKAGE = (
+        re.compile(r'^\s*(?P<name>\S+)\s*\|\s*\S+\s*\|\s*.+\s*\|\s*(\S+)\s*$')
+    )


Breaking into multiple lines not really needed here. Moreover, if you search the second column for version numbers instead of just "not spaces", then you could possibly simplify the loop in lines 144-162 because you would not need to skip the header, it would be skipped by the fact that it does not match the regexp (with digits):

Suggested change

_RE_INSTALLED_PACKAGE = (

re.compile(r'^\s*(?P<name>\S+)\s*\|\s*\S+\s*\|\s*.+\s*\|\s*(\S+)\s*$')

)

_RE_INSTALLED_PACKAGE = re.compile(r'^\s*(?P<name>\S+)\s*\|\s*[\d\.]+\s*\|\s*.+\s*\|\s*(\S+)\s*$')

Might not work because sometimes the version column is not just numbers, for instance:
build-tools;32.1.0-rc1 | 32.1.0 rc1 | Android SDK Build-Tools 32.1-rc1 | build-tools/32.1.0-rc1

Here the version is 32.1.0 rc1

Again, this is not something that is documented anywhere, but looking at the output of sdkmanager --list, I see the next possible options for version:

24.0.3

32.1.0 rc1

3.6.4111459

2

27.0.11902837 rc2
Does it make sense to make a regexp covering all these version forms, considering the fact that the module does not use version at all?

russoz · 2024-12-16T21:34:16Z

plugins/module_utils/sdkmanager.py

+    )
+
+    # Example: '   platform-tools | 27.0.0    | 35.0.2'
+    _RE_UPDATABLE_PACKAGE = re.compile(r'^\s*(?P<name>\S+)\s*\|\s*\S+\s*\|\s*\S+\s*$')


You would need to change this regexp to search for numbers as well.

See comment above

plugins/modules/android_sdk.py

plugins/module_utils/sdkmanager.py

russoz · 2024-12-17T00:16:45Z

plugins/module_utils/sdkmanager.py

+    _RE_UPDATABLE_PACKAGES_HEADER = re.compile(r'^Available Updates:$')
+
+    # Example: '  platform-tools     | 27.0.0  | Android SDK Platform-Tools 27 | platform-tools  '
+    _RE_INSTALLED_PACKAGE = re.compile(r'^\s*(?P<name>\S+)\s*\|\s*\S+\s*\|\s*.+\s*\|\s*(\S+)\s*$')


Well, from these examples, the only two possibilities that are not covered by the suggested regexp (the ones with rc1 and rc2) are also not covered by the existing regexp: |\s*\S+\s*\| will not match | 32.1.0 rc1 | (because the \S+ will not match the whitespace before rc1).

The one thing that is the delimiter here is the | character, so maybe it would be better of as:

_RE_INSTALLED_PACKAGE = re.compile(r'^\s*(?P<name>\S+)\s*\|\s*[0-9][^|]*\b\s*\|\s*.+\s*\|\s*(\S+)\s*$')

That forces the version to begin with a digit, then anything-but-the-vertical-bar until the space and vertical bar. The \b is used to make sure the "anything-but-the-vertical-bar" part does not include the whitespaces before the bar.

As to your question of whether it makes sense to cover all the cases with the regexp, I would answer that IMHO it makes more sense to cover all the search cases with a search tool (regexp) then to cover only part of the search with it and create another search mechanism later on. Just my $0.02 ;-)

Good catch, thanks

…0 rc1". Also, this allows to simplify the parsing logic as there is no need to skip table headers anymore.

russoz

LGTM

plugins/module_utils/sdkmanager.py

plugins/modules/android_sdk.py

Co-authored-by: Felix Fontein <[email protected]>

.github/BOTMETA.yml

felixfontein · 2024-12-17T20:19:22Z

Looks good to me as well! If nobody objects, I'll merge by the end of the week.

patchback · 2024-12-20T21:55:16Z

Backport to stable-10: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-10/2b2872f0efe7f2fcf2c8c0e2b40764d940c79dff/pr-9236

Backported as #9293

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

* adds simple implementation of adding and removing android sdk packages * adds package update * adds simple installed packages parsing * moves parsing logic to a separate class * adds absent state for sdkmanager packages and setup for tests * adds output for installing and removing packages * removes version from Package object since it is not possible to specify version for a package while using sdkmanager * adds 'latest' state * adds tests * fixes crash when sdkmanager is invoked from python with LC_ALL=C * fixes latest state * adds sdk_root parameter * adds channel parameter * simplifies regexps, removes unused named groups * minor refactoring of sdkmanager parsing * adds java dependency variable for different distributions * adds RETURN documentation * adds check for nonexisting package * adds check for non-accepted licenses * removes excessive methods from sdkmanager * removes unused 'update' module parameter, packages may be updated using 'latest' state * minor refactoring * adds EXAMPLES doc section * adds DOCUMENTATION section and license headers * fixes formatting issues * removes diff_params * adds maintainer * fixes sanity check issues in sdkmanager * adds java dependency for macos and moves some tests to a separate FreeBSD configuration * fixes dependencies setup for OSX * fixes dependencies setup for OSX (2) * fixes dependencies setup for OSX (3) * Apply minor suggestions from code review Co-authored-by: Alexei Znamensky <[email protected]> * applies code review suggestions * changes force_lang from C.UTF-8 to auto in sdkmanager (as per discussion #9236 (comment)) * Revert "changes force_lang from C.UTF-8 to auto in sdkmanager (as per discussion #9236 (comment))" This reverts commit 619f28d. * fixes some more comments from review * minor sanity issue fix * uses the 'changed' test instead of checking the 'changed' attribute * adds 'accept_licenses' parameter. Installation is now performed independently for each package specified. * removes "Accept licenses" task from examples * fixes docs sanity issues * applies minor suggestions from code review * fixes regexps. The previous version didn't match versions like "32.1.0 rc1". Also, this allows to simplify the parsing logic as there is no need to skip table headers anymore. * renamed sdkmanager.py to android_sdkmanager.py * applies minor suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * updates BOTMETA * reordered BOTMETA --------- Co-authored-by: Alexei Znamensky <[email protected]> Co-authored-by: Felix Fontein <[email protected]> (cherry picked from commit 2b2872f)

felixfontein · 2024-12-20T21:55:25Z

@shamilovstas thanks for your contribution!
@russoz thanks for reviewing!

Add android sdk module (#9236) * adds simple implementation of adding and removing android sdk packages * adds package update * adds simple installed packages parsing * moves parsing logic to a separate class * adds absent state for sdkmanager packages and setup for tests * adds output for installing and removing packages * removes version from Package object since it is not possible to specify version for a package while using sdkmanager * adds 'latest' state * adds tests * fixes crash when sdkmanager is invoked from python with LC_ALL=C * fixes latest state * adds sdk_root parameter * adds channel parameter * simplifies regexps, removes unused named groups * minor refactoring of sdkmanager parsing * adds java dependency variable for different distributions * adds RETURN documentation * adds check for nonexisting package * adds check for non-accepted licenses * removes excessive methods from sdkmanager * removes unused 'update' module parameter, packages may be updated using 'latest' state * minor refactoring * adds EXAMPLES doc section * adds DOCUMENTATION section and license headers * fixes formatting issues * removes diff_params * adds maintainer * fixes sanity check issues in sdkmanager * adds java dependency for macos and moves some tests to a separate FreeBSD configuration * fixes dependencies setup for OSX * fixes dependencies setup for OSX (2) * fixes dependencies setup for OSX (3) * Apply minor suggestions from code review Co-authored-by: Alexei Znamensky <[email protected]> * applies code review suggestions * changes force_lang from C.UTF-8 to auto in sdkmanager (as per discussion #9236 (comment)) * Revert "changes force_lang from C.UTF-8 to auto in sdkmanager (as per discussion #9236 (comment))" This reverts commit 619f28d. * fixes some more comments from review * minor sanity issue fix * uses the 'changed' test instead of checking the 'changed' attribute * adds 'accept_licenses' parameter. Installation is now performed independently for each package specified. * removes "Accept licenses" task from examples * fixes docs sanity issues * applies minor suggestions from code review * fixes regexps. The previous version didn't match versions like "32.1.0 rc1". Also, this allows to simplify the parsing logic as there is no need to skip table headers anymore. * renamed sdkmanager.py to android_sdkmanager.py * applies minor suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * updates BOTMETA * reordered BOTMETA --------- Co-authored-by: Alexei Znamensky <[email protected]> Co-authored-by: Felix Fontein <[email protected]> (cherry picked from commit 2b2872f) Co-authored-by: Stanislav Shamilov <[email protected]>

* adds simple implementation of adding and removing android sdk packages * adds package update * adds simple installed packages parsing * moves parsing logic to a separate class * adds absent state for sdkmanager packages and setup for tests * adds output for installing and removing packages * removes version from Package object since it is not possible to specify version for a package while using sdkmanager * adds 'latest' state * adds tests * fixes crash when sdkmanager is invoked from python with LC_ALL=C * fixes latest state * adds sdk_root parameter * adds channel parameter * simplifies regexps, removes unused named groups * minor refactoring of sdkmanager parsing * adds java dependency variable for different distributions * adds RETURN documentation * adds check for nonexisting package * adds check for non-accepted licenses * removes excessive methods from sdkmanager * removes unused 'update' module parameter, packages may be updated using 'latest' state * minor refactoring * adds EXAMPLES doc section * adds DOCUMENTATION section and license headers * fixes formatting issues * removes diff_params * adds maintainer * fixes sanity check issues in sdkmanager * adds java dependency for macos and moves some tests to a separate FreeBSD configuration * fixes dependencies setup for OSX * fixes dependencies setup for OSX (2) * fixes dependencies setup for OSX (3) * Apply minor suggestions from code review Co-authored-by: Alexei Znamensky <[email protected]> * applies code review suggestions * changes force_lang from C.UTF-8 to auto in sdkmanager (as per discussion ansible-collections#9236 (comment)) * Revert "changes force_lang from C.UTF-8 to auto in sdkmanager (as per discussion ansible-collections#9236 (comment))" This reverts commit 619f28d. * fixes some more comments from review * minor sanity issue fix * uses the 'changed' test instead of checking the 'changed' attribute * adds 'accept_licenses' parameter. Installation is now performed independently for each package specified. * removes "Accept licenses" task from examples * fixes docs sanity issues * applies minor suggestions from code review * fixes regexps. The previous version didn't match versions like "32.1.0 rc1". Also, this allows to simplify the parsing logic as there is no need to skip table headers anymore. * renamed sdkmanager.py to android_sdkmanager.py * applies minor suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * updates BOTMETA * reordered BOTMETA --------- Co-authored-by: Alexei Znamensky <[email protected]> Co-authored-by: Felix Fontein <[email protected]>

shamilovstas added 28 commits December 7, 2024 18:00

adds simple implementation of adding and removing android sdk packages

215467f

adds package update

2526420

adds simple installed packages parsing

ec674cd

moves parsing logic to a separate class

ca3d11a

adds absent state for sdkmanager packages and setup for tests

7349127

adds output for installing and removing packages

01c3674

removes version from Package object since it is not possible to speci…

d7f6451

…fy version for a package while using sdkmanager

adds 'latest' state

0434249

adds tests

3edfac2

fixes crash when sdkmanager is invoked from python with LC_ALL=C

ba7f1bf

fixes latest state

de6f740

adds sdk_root parameter

97b07ee

adds channel parameter

b41eb57

simplifies regexps, removes unused named groups

6d756e2

minor refactoring of sdkmanager parsing

4441be7

adds java dependency variable for different distributions

100a91f

adds RETURN documentation

fdef9c4

adds check for nonexisting package

9b3d444

adds check for non-accepted licenses

ece03d1

removes excessive methods from sdkmanager

e8ef594

removes unused 'update' module parameter, packages may be updated usi…

1ac1ad5

…ng 'latest' state

minor refactoring

22b317f

adds EXAMPLES doc section

e499391

adds DOCUMENTATION section and license headers

7e53827

fixes formatting issues

e5a6d02

removes diff_params

003bd1e

adds maintainer

20eb1aa

fixes sanity check issues in sdkmanager

3358ac9

ansibullbot added integration tests/integration module module labels Dec 10, 2024

shamilovstas added 2 commits December 16, 2024 16:24

removes "Accept licenses" task from examples

83dd52e

fixes docs sanity issues

ffd170d

shamilovstas requested review from felixfontein and russoz December 16, 2024 14:49

russoz reviewed Dec 16, 2024

View reviewed changes

plugins/modules/android_sdk.py Outdated Show resolved Hide resolved

russoz reviewed Dec 16, 2024

View reviewed changes

plugins/modules/android_sdk.py Outdated Show resolved Hide resolved

russoz reviewed Dec 16, 2024

View reviewed changes

plugins/module_utils/sdkmanager.py Outdated Show resolved Hide resolved

applies minor suggestions from code review

bf10fe0

russoz reviewed Dec 17, 2024

View reviewed changes

fixes regexps. The previous version didn't match versions like "32.1.…

1878cd0

…0 rc1". Also, this allows to simplify the parsing logic as there is no need to skip table headers anymore.

russoz approved these changes Dec 17, 2024

View reviewed changes

felixfontein reviewed Dec 17, 2024

View reviewed changes

plugins/module_utils/sdkmanager.py Outdated Show resolved Hide resolved

plugins/modules/android_sdk.py Outdated Show resolved Hide resolved

plugins/modules/android_sdk.py Outdated Show resolved Hide resolved

shamilovstas and others added 3 commits December 17, 2024 21:48

renamed sdkmanager.py to android_sdkmanager.py

043fc16

applies minor suggestions from code review

7be3b3c

Co-authored-by: Felix Fontein <[email protected]>

updates BOTMETA

9033695

felixfontein reviewed Dec 17, 2024

View reviewed changes

.github/BOTMETA.yml Outdated Show resolved Hide resolved

reordered BOTMETA

2159c01

felixfontein removed the check-before-release PR will be looked at again shortly before release and merged if possible. label Dec 20, 2024

felixfontein merged commit 2b2872f into ansible-collections:main Dec 20, 2024
129 checks passed

patchback bot mentioned this pull request Dec 20, 2024

[PR #9236/2b2872f0 backport][stable-10] Add android sdk module #9293

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add android sdk module #9236

Add android sdk module #9236

shamilovstas commented Dec 10, 2024

shamilovstas commented Dec 16, 2024 •

edited

Loading

felixfontein commented Dec 16, 2024

russoz Dec 16, 2024

shamilovstas Dec 16, 2024 •

edited

Loading

shamilovstas Dec 16, 2024

russoz Dec 16, 2024

shamilovstas Dec 16, 2024

russoz Dec 17, 2024

russoz Dec 17, 2024

shamilovstas Dec 17, 2024

russoz left a comment

felixfontein commented Dec 17, 2024

patchback bot commented Dec 20, 2024 •

edited

Loading

felixfontein commented Dec 20, 2024

Add android sdk module #9236

Add android sdk module #9236

Conversation

shamilovstas commented Dec 10, 2024

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

shamilovstas commented Dec 16, 2024 • edited Loading

felixfontein commented Dec 16, 2024

russoz Dec 16, 2024

Choose a reason for hiding this comment

shamilovstas Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

shamilovstas Dec 16, 2024

Choose a reason for hiding this comment

russoz Dec 16, 2024

Choose a reason for hiding this comment

shamilovstas Dec 16, 2024

Choose a reason for hiding this comment

russoz Dec 17, 2024

Choose a reason for hiding this comment

russoz Dec 17, 2024

Choose a reason for hiding this comment

shamilovstas Dec 17, 2024

Choose a reason for hiding this comment

russoz left a comment

Choose a reason for hiding this comment

felixfontein commented Dec 17, 2024

patchback bot commented Dec 20, 2024 • edited Loading

Backport to stable-10: 💚 backport PR created

felixfontein commented Dec 20, 2024

shamilovstas commented Dec 16, 2024 •

edited

Loading

shamilovstas Dec 16, 2024 •

edited

Loading

patchback bot commented Dec 20, 2024 •

edited

Loading