From 4fc13e01c44e19bb53f2070c4346273dd45270b5 Mon Sep 17 00:00:00 2001 From: dcm0 Date: Thu, 2 Jun 2022 16:34:07 +0200 Subject: [PATCH] Squashed commit of the following: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit commit 6af431375d30fab6fead2df747c67e325dcf2fa1 Author: Bart Ribbers Date: Thu Jun 2 02:11:03 2022 +0200 Don't install pulseaudio-dev on Alpine Linux systems (#3109) * Prefix the Alpine Linux virtual package name with a dot This way the package is easy to recognize when inspecting /etc/apk/world, and it's actually a standard to do so for virtual packages * Put all Alpine Linux deps on seperate lines This way you get cleaner git diffs, showing more clearly what is changing when adding or removing something * Don't install pulseaudio-dev on Alpine Linux systems It doesn't actually seem to be required by any of the PyPi deps commit 56ceb80179a79dfb294f0b3eecdc6612642964c1 Author: Bart Ribbers Date: Sun Sep 6 12:00:11 2020 +0200 {start,stop}-mycroft.sh: port to POSIX sh This makes the start and stop scripts compatible with POSIX shells. Overview of the changes: - "function" statements removed, not necessary and incompatible - dashes in function and variable names for lower ones (- to _) - source statements changed for . - double square brackets replaced for single ones - double equal statements replaced for single ones - &> (piping stdout and stderr to the same file) replaced for 2>&1 > - sourcing of mycroft-skill-testrunner replaced with direct execution with Bash - replaced BASH_SOURCE with $0, these scripts are never sourced anyway - replaced "echo -n" statements with "printf" - merged the "" and "all" cases to a single one commit dadbd23976079694b7b6870546ac1589943b31d7 Author: Bart Ribbers Date: Sun Sep 6 11:33:04 2020 +0200 {start,stop}-mycroft.sh: fix shellcheck issues ShellCheck is a static analysis tool for shell scripts with the goal to: - point out and clarify typical beginner's syntax issues that cause a shell to give cryptic error messages - point out and clarify typical intermediate level semantic problems that cause a shell to behave strangely and counter-intuitively - point out subtle caveats, corner cases and pitfalls that may cause an advanced user's otherwise working script to fail under future circumstances I've ran this tool over both start-mycroft.sh and stop-mycroft.sh and fixed any issue that popped up commit a909fc8f197aeb069999135dfe6ad1579ff69772 Author: Åke Date: Thu Apr 21 23:47:56 2022 +0200 Requirements: remove pychromecast (#3098) commit f030b7e165f767844f8c29b249a9a3200a227b1d Author: Kris Gesling Date: Wed Apr 20 10:23:48 2022 +0930 Silence file does not exist error when clearing VK files (#3093) When clearing VK test files - if some files didn't exist (because they weren't created) the bare rm command will report an error that the files didn't exist. We don't care, only that they are removed if they did exist. commit bf85e5c9c5b0e2eefd22e73449533b4faa65275d Author: Åke Date: Tue Apr 19 23:31:30 2022 +0200 Remove the chromecast audio backend (#3097) The pychromecast module is quite outdated and I think the backend would fit better as a plugin now. commit 3d64aa5940a8046894d563146bfc88015104ed94 Author: Åke Date: Fri Apr 8 02:58:56 2022 +0200 dev_setup: Give packages in separate args to pacman (#3094) This makes sure the packages aren't clumped together into a single argument when using pacman to install packages for arch linux. commit 28d512e54c80ef574394bbda6cf66911dc0d9a10 Author: Kris Gesling Date: Thu Apr 7 15:45:10 2022 +0930 Fix shellcheck error on Arch install command (#3092) Double quote to prevent globbing commit 2d15fca5816c754a3210a6850bf676fcc413b0fd Author: Khionu Sybiern Date: Wed Apr 6 22:15:06 2022 -0700 Add pipewire check for Arch installs (#3091) Check if pipewire-pulse is installed, installs pulse otherwise ==== Fixed Issues ==== Partial fix: #2980 ==== Tech Notes ==== pipewire-pulse is a compat layer, providing a pulseaudio API commit c6f7c1ad0a74cdeaa6883ae8be0847bf9824c1de Author: Åke Date: Thu Apr 7 06:51:26 2022 +0200 Shellcheck scripts in root folder (#3090) * Fix Shellcheck: Add -r flag to read read will mangle backslashes without it. These are not expected characters so not explicitly necessary, but also won't hurt. * Fix Shellcheck SC2086: Double quote variables * Fix Shellcheck SC2004: dollar sign unnecessary on arithmetic vars * Fix Shellcheck SC2129: use curly braces to >> file * Disable shellcheck on sourced venv/bin/activate * Fix Shellcheck SC1004 Simplify sed-expression to be a single line only removing the need for the offending / and linefeed * Fix Shellcheck SC2046 when finding repo root Splits the check into to files to be able to quote the expressions in a good way. * Fix Shellcheck SC2086 Unquoted apt packagaes This converts the variable APT_PACKAGE_LIST from a string to an array. This is a safer way to handle arguments according to BashFAQ (http://mywiki.wooledge.org/BashFAQ/050) * Fix Shellcheck SC2230 by changing which to command -v * CICD: Activate shellcheck job for dev_setup.sh * Shellcheck start-mycroft.sh This fixes the following shellcheck issues: - https://www.shellcheck.net/wiki/SC2164 -- Use 'cd ... || exit' or 'cd ... |... - https://www.shellcheck.net/wiki/SC2206 -- Quote to prevent word splitting/g... - https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ... - https://www.shellcheck.net/wiki/SC2124 -- Assigning an array to a string! A... The command is added to the CICD run of shellcheck * Shellcheck stop-mycroft.sh Fixes the following issues: - https://www.shellcheck.net/wiki/SC2164 -- Use 'cd ... || exit' or 'cd ... |... - https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ... The file is now automatically checked by the CICD job * Shellcheck venv-activate.sh Fix the following shellcheck issues: - https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ... - https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ... - https://www.shellcheck.net/wiki/SC2128 -- Expanding an array without an ind... - https://www.shellcheck.net/wiki/SC2139 -- This expands when defined, not wh... - https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ... The help for invalid arg now works when the faulty argument isn't the first argument Co-authored-by: Kris Gesling commit ccd577bbf4583fce228826818cff76aa65268ae4 Author: fsa317 <33432460+fsa317@users.noreply.github.com> Date: Tue Mar 22 21:54:38 2022 -0400 Issue-3006 replaced hot_words with hotwords to correctly lookup configuration (#3088) commit 536ea7e9b5db5f6b8d879ede7c18e47a963b3278 Author: Kris Gesling Date: Tue Mar 22 15:47:43 2022 +0930 Remove skills dir symlink on --clean (#3086) * Remove skills dir symlink on --clean * Add note on --clean about files not removed commit 5f4c68e5831e668f11dddc9a2d00bf11698f8c5b Author: Åke Date: Mon Mar 7 01:33:10 2022 +0100 Don't overwrite invalid json in config (#2881) * Separate tests of LocalConf into new test class * Do not overwrite configs with malformed json If a config file is loaded and is invalid saves to that file will not be possible until it's manually restored. The store accepts a force flag to override this protection. The method returns True if the store succeeded so call sites can verbally report the issue as well. commit 660b7b9bd32c4b2cd366a2599f2f20ed5142f9ab Merge: e7ddd51256 b583cbbf7e Author: Åke Date: Wed Mar 2 08:33:42 2022 +0100 Merge pull request #3082 from MycroftAI/refactor/python-detect Send python version detection output to /dev/null commit b583cbbf7e693d009998b0a6e0a0f3b4cff7b0aa Author: Kris Gesling Date: Wed Mar 2 12:14:53 2022 +0930 Send python version detection output to /dev/null commit e7ddd5125650ff9dfcee849a396fe5ebb3eeebb5 Author: Åke Date: Wed Mar 2 01:29:57 2022 +0100 TTS playback queue singleton (#3055) * fix issues when remote excepts out * Remove explicit clear_cache from MimicTTS * Updates for using singleton TTS playback thread - Cache is called on all tts's registered as using the thread - Begin audio and end audio is handled by the playback thread - Further changes from self.playback to TTS.playback for consistency * Remove redundant try/except * Consolidate general and TTS-specific sentence splitting This performs all sentence-splitting at the same stage. This fixes a subtle issue where a TTS splits a sentence into chunks and throws an error on only one of those chunks. The fallback would generate a sentence for the original un-chunked sentence. possibly saying the same parts twice. This also pre-compiles the regexes used to speed things up a bit. Co-authored-by: Ken commit 36620af703dd0fcc3e75dd285ff42c4e20b9ea7c Author: Kris Gesling Date: Tue Mar 1 07:46:21 2022 +0930 Remove inflection dependency (#3031) The dependency was being used for a single string operation that is easily replaced. commit f0eecc93e1381c255b2718a60cdc0f73e10f1ed5 Author: Kris Gesling Date: Tue Mar 1 07:44:54 2022 +0930 Bump requests to remove GPL from dependency tree (#3081) Versions prior to 2.26.0 used a GPL licensed package Chardet. This has been replaced with charset_normalizer (MIT license). commit 012d5a517c15c186c759f78ba13d5d0a35c5e954 Merge: eb97bb339e 909d89e28e Author: Åke Date: Mon Feb 28 21:54:32 2022 +0100 Merge pull request #3080 from MycroftAI/bugfix/pip-install Fix pip url for Python > 3.6 commit eb97bb339e506143ba40c162d09c66139c651315 Author: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Sun Feb 27 22:06:48 2022 -0800 Add `supported_languages` parameter to STT and TTS base classes (#3059) * Add `supported_languages` parameter to STT and TTS base classes * Refactor `supported_languages` to `available_languages` and updated docstring to be more precise commit dd7f7abba6fe9e8f88519c2f07eded7d5645fbde Author: Åke Date: Mon Feb 28 06:14:15 2022 +0100 Shellcheck scripts folder (#3063) * Remove unused scripts install-pocketsphinx and install-pygtk is no longer used by the dev-setup. * Fix shellcheck issues in scripts * Remove space in cores argument * Update my-info.sh script - Make it use the new method to activate venv if needed - Improve requirements.txt parsing - Update process detection from old "screen" setup - Update log-files paths - Fix finding mycroft-core folder * Fix shellcheck issues in mycroft-use.sh Mainly quoting but also unpacking of arguments and improving some if statements * Update shellcheck test to include scripts folder commit df0b8fe212b8508155edb4f32e57e1819575a709 Author: Kris Gesling Date: Wed Feb 23 11:41:24 2022 +0930 Remove mplayer audioservice It is no longer supported by the projects maintainer. commit 909d89e28e3bcfb92437f0662ed04ce10e375ba5 Author: Kris Gesling Date: Mon Feb 28 14:28:54 2022 +0930 Fix pip url for Python > 3.6 commit 8ee3e8d64018d87aee87983c6a1f64acff581666 Author: Åke Forslund Date: Sun Dec 19 14:20:44 2021 +0100 Add noise_words.list for Swedish commit e1731f53ec3c8ff842989ede0ead2cdb753ccb7a Author: Åke Forslund Date: Fri Dec 17 16:56:02 2021 +0100 Remove unused imports from common_query_skill.py commit 3fa99f288fbe2916ec36670e04a239964d3841c5 Author: Åke Forslund Date: Fri Dec 17 16:53:42 2021 +0100 Handle missing noise words file in CQS This treats a None result from resolve_resource_file() as a FileNotFound exception. commit 47911146095e8ebae29bd4694029b283a3c8133c Author: luca-vercelli Date: Thu Feb 24 01:51:17 2022 +0100 missing initial value (#3061) Condition if [ $disable_precise_later == true ]; gives a syntax error if variable is not initialized commit 953cc53e13cb35a6e1012b980ad3ceadb09dbeb0 Merge: 99ac12cbf8 6d66b800dc Author: Kris Gesling Date: Thu Feb 24 10:17:11 2022 +0930 Merge pull request #3060 from kleo/dev Use double bracket for shell conditional commit 99ac12cbf84ef8609a52851e21c1bfada8f74c0c Merge: 34f66d234c f8aa77c8e7 Author: Kris Gesling Date: Wed Feb 23 16:44:33 2022 +0930 Merge pull request #3068 from forslund/bugfix/vk-dialogfile-from-sentence Voight kampff: Bugfix dialogfile from sentence commit 34f66d234c73715422d8fa392f366700a8760ff9 Author: TheRealDGD <99516116+TheRealDGD@users.noreply.github.com> Date: Wed Feb 23 07:38:46 2022 +0100 Added duration_to_bytes to FileMockMicrophone (#3071) * Added duration_to_bytes to FileMockMicrophone Added duration_to_bytes to fix AttributeError: 'FileMockMicrophone' object has no attribute 'duration_to_bytes' in call from mycroft-core/mycroft/client/speech/mic.py _wait_until_wake_word * Update wake_word_test.py Fixed formatting issues * Update wake_word_test.py Fixed formatting * Remove whitespaces for PEP8 Co-authored-by: Kris Gesling commit f556f844120ac30701536b37f3e747624db123ab Author: Kris Gesling Date: Wed Feb 23 15:42:16 2022 +0930 Add phonetic spellings for IP and Wikipedia (#3023) Co-authored-by: jarbasal commit 1ae6900b26e8efb4c014b04cd26572adf10b07b2 Author: Kris Gesling Date: Wed Feb 23 15:40:53 2022 +0930 Remove ResponsiveVoice TTS module from core (#3049) This module in its current format does not work due to API changes. There is also a community built plugin that is now described in our documentation. commit 5d88eb3a33c17271d04364b610cd204517b05181 Author: Aditya Mehra Date: Wed Feb 23 16:31:19 2022 +1030 Fix: show text delegate as per autofit label refactor (#3065) * fix show text delegate as per autofit label refactor * Make rectangle containing text transparent For different background colors, the background on the card should be set. Co-authored-by: Kris Gesling commit 0dadc9ba0750aaf6680dd1b91fa1c01fa398b2ec Author: Kris Gesling Date: Wed Feb 23 14:47:03 2022 +0930 Switch CLA Checker to new credentials (#3077) commit a487f66958b759342d055e3f61add6dd594be485 Author: Åke Date: Mon Feb 14 06:40:24 2022 +0100 Select the proper get-pip url for Python 3.6 (#3073) The bootstrap script no longer supports Python 3.6, instead it recommends python 3.6 users to use an alternative url. commit f8aa77c8e7ccdb39bafab9bdf12e86b07dbb2310 Author: Åke Forslund Date: Wed Jan 26 21:01:46 2022 +0100 VK simplify dialog matching This uses the existing dialog renderer and the standard format library to in two steps create a regex where the {elements} in a dialog is replaced with ".*" to match the given sentence. commit a3fd830cb0ae3deef01d0e4d3cfc4512fd55f269 Author: Åke Forslund Date: Wed Jan 26 12:55:36 2022 +0100 VK: Expand the parentheses from dialog files commit 825b3879a51c15513e069e61872266bfa55ecd99 Author: Åke Forslund Date: Tue Jan 18 13:13:50 2022 +0100 VK: Fix regex used in _match_dialog_patterns A redundant step caused issue when performing tests in the tv-remove-control-skill commit 03ef7f4b219533507f016c64a45487bc827ec78c Author: Åke Forslund Date: Tue Jan 18 13:12:30 2022 +0100 VK: Include locale folders in dialog_from_sentence Only dialog folder was used previously this adds globbing through the locale/lang/ folder and it's subfolders commit 6d66b800dc9d86235ab0139d146eb89494c482ca Author: Kleo Bercero Date: Sat Jan 1 19:49:36 2022 +0800 Use double bracket commit 3d963cee402e232174850f36918313e87313fb13 Merge: 2411f5d317 4dd3dd3027 Author: Kris Gesling Date: Mon Dec 13 10:07:11 2021 +0930 Merge pull request #3047 from forslund/feature/do-not-create-identity-folder Do not create identity folder when trying to read the identity file commit 2411f5d317c0299e4f42bd459fdc7dd19299f571 Merge: 947b7f3fc7 bdfb7c3981 Author: Kris Gesling Date: Mon Dec 13 10:06:30 2021 +0930 Merge pull request #3043 from MycroftAI/feature/pyxdg-update Update pyxdg to ensure Python3.8 compatibility commit 947b7f3fc72f475b90f9f1b516bea61c66fb7542 Merge: 4061a04bb4 b0f9a10e71 Author: Kris Gesling Date: Mon Dec 13 10:06:15 2021 +0930 Merge pull request #3030 from forslund/cicd/shellcheck-directives Use shellcheck directive instead of excludes commit 4061a04bb473726183419620a0e6517be944486d Merge: 322a4c5599 9468c20558 Author: Kris Gesling Date: Mon Dec 13 10:05:51 2021 +0930 Merge pull request #3014 from putnik/dev Update Russian words and dialogues commit 322a4c55993605bff1fbf88a3cfbb109dc144c74 Author: devs-mycroft Date: Fri Dec 10 04:51:54 2021 +0000 Version bump from 21.2.1 to 21.2.2 commit 4dd3dd30274386644856a9c1b1a567431777a0ce Author: Åke Forslund Date: Sun Dec 5 16:47:00 2021 +0100 Add log if an exception loading identity occurs commit b342ab70c35641db69f01b51eb19cfc6e95cabf3 Author: Åke Forslund Date: Sat Dec 4 17:18:33 2021 +0100 Only try to open the identity2.json if it exists commit ae9997439866cd89d6a698e589d13dcefc9102b1 Merge: d479a79b7e 6b5d45e507 Author: Kris Gesling Date: Wed Dec 1 09:17:41 2021 +0930 Merge pull request #3045 from forslund/bugfix/config-creation-at-import Do not create configs folders until writing commit d479a79b7e132ee7022039bae2613832890b133b Merge: 8e319b49ef 70df575d43 Author: Kris Gesling Date: Wed Dec 1 09:13:28 2021 +0930 Merge pull request #3033 from forslund/bugfix/config Fix config priority commit 6b5d45e50780aef98e706313f652fcf58cf3579b Author: Åke Forslund Date: Tue Nov 30 21:34:46 2021 +0100 Do not create configs folders until writing This replaces save_*_path with usage of the xdg_*_home when handling config files. This means the config folders will not be created unless actually written to. The check for whether a directory needs to be created is handled behind a lock to avoid race conditions commit bdfb7c3981917275fd0f90a79fb3521f19faf46f Author: Kris Gesling Date: Tue Nov 30 22:20:12 2021 +0930 Update pyxdg for Python3.8 compatibility commit 8e319b49ef98b2f7f5d8c365e8a1d64be0fc58dd Merge: c3bd1f2c5f d9c280afd4 Author: Kris Gesling Date: Tue Nov 30 07:09:26 2021 +0930 Merge pull request #3042 from HFabi/feature/issue-3041 Issue-3041 - Install pip for python version specified by argument in dev_setup commit d9c280afd4ece00f91628ba5d543ca5162cb23e3 Author: Fabian Heck Date: Mon Nov 29 16:57:34 2021 +0100 Issue-3041 - Install pip for python version specified by argument in dev_setup commit c3bd1f2c5f4d2c3b0a7c0196898c0119c33b2240 Merge: 89cfad7943 bd85c94cdb Author: Kris Gesling Date: Mon Nov 29 12:12:58 2021 +0930 Merge pull request #3039 from rooky-c3bo/bugfix/issue-3026 Issue-3026 - Update Gentoo libffi dependency reference commit bd85c94cdbb4af8b2cdd346d6abae6dcc54ea8e2 Author: Bahadır Yaren Date: Thu Nov 25 12:21:00 2021 +0100 Issue-3026 - Fixing Gentoo dependency commit 89cfad794315198f5b1d99e4c11537425ae74ae5 Author: Kris Gesling Date: Wed Nov 24 14:51:10 2021 +0930 Check if GUI is connected rather than maintain list of platforms (#3025) * Check if GUI is connected rather than maintain list of platforms There are already many Mycroft platforms that have GUIs and this will only grow. We want to know if the device has a GUI connected rather than if it is in a pre-defined list of platforms. * Create a mock GUI with a settable connected attribute commit b0f9a10e719c6fb50a253dcbba9cbd4813b1a7ea Author: Åke Forslund Date: Tue Nov 23 11:56:54 2021 +0100 Disable source lookup errors commit 77549d01e2d99440d39a762f6d5fd97587c1864c Author: Kris Gesling Date: Mon Nov 22 15:00:20 2021 +0930 Remove incorrectly added args in GUIWebsocketHandler methods (#3036) PR #2879 updated the websocket-client and changed function signatures to account for API changes in that package. These were falsely changed as the GUI bus does not use the websocket-client. Consistency would be good however modifying the protocols would require significant work. For now we have two slightly different bus interfaces. commit e1dcebff8cb19729f2c1cdbaa168244eefabce54 Merge: ef56d71462 958d4fa05e Author: Kris Gesling Date: Thu Nov 11 14:04:12 2021 +0930 Merge pull request #3028 from MycroftAI/bugfix/link-update Update link for Hub to point to official Github version commit 70df575d4377f2fa2ecdbaec85b931d893821e2b Author: Åke Forslund Date: Tue Nov 9 21:04:58 2021 +0100 Make log settings not cache local only config After startup the cached config would be without remote config. This makes sure the config without remote isn't cached. commit 5ff2cb099f2e5fb9f3b80ec0cd6efd80a6885ebc Author: Åke Forslund Date: Tue Nov 9 20:51:38 2021 +0100 Fix order of configs The old SYSTEM config was prioritized lower than the remote config commit ef56d714621728682f7376277df273b583e75722 Author: Genei180 Date: Tue Nov 9 06:37:33 2021 +0100 Added Possibility for ESpeak Config (#3020) Added Possibility for ESpeak Config commit e6fe1bbc8affd2f7b22455dc21539ee6725fb45b Author: Åke Forslund Date: Mon Nov 8 07:46:52 2021 +0100 Use shellcheck directive instead of excludes This uses shellcheck directives instead of excludes only available in CICD. This leaves SC1090 as a general exclude commit 958d4fa05e0e7d76baefde98daf454168f67c039 Author: Kris Gesling Date: Sat Nov 6 06:13:58 2021 +0930 Update link for Hub to point to official Github version commit ea157598d70581dcaca719e19a2de76e880f76ed Merge: fd12c88da5 f8f640e3e5 Author: Kris Gesling Date: Fri Nov 5 06:12:08 2021 +0930 Merge pull request #3022 from MycroftAI/bugfix/gui-namespace-race-cond Fix race condition in GUI namespace insertion commit fd12c88da532fe5f1f37c8f73690ff4501ac6886 Author: Åke Date: Thu Nov 4 08:31:17 2021 +0100 Refactor/shellcheck bin (#3019) * Shellcheck and update mycroft-config * Shellcheck and update mycroft-pip * Shellcheck and update mycroft-cli-client * Shellcheck and update mycroft-help * Shellcheck and update mycroft-listen * Shellcheck and update mycroft-mic-test * Shellcheck and update mycroft-msk * Shellcheck and update mycroft-msm * Shellcheck and update mycroft-say-to * Shellcheck and update mycroft-skill-testrunner * Shellcheck and update mycroft-speak * Shellcheck and update mycroft-start * Shellcheck and update mycroft-stop * Add shellcheck step to github actions This runs most of the shellcheck tests. The excludes are: - SC1091: Avoids errors when shellcheck can't find sourced file - SC2034: Unused variables, for example colors that aren't used yet - SC2012: use of ls, from what I can see in our case this is fine (wc -l) The version is locked to latest master as of November 2 2021 commit 0ad093a8e33e245a87621047eac9c8e5c6d1a283 Merge: dd710a6a53 dca1184bd5 Author: Kris Gesling Date: Thu Nov 4 17:00:24 2021 +0930 Merge pull request #3021 from Saymantech-org/dev Azerbaijani language support commit f8f640e3e5260da33794fee55417cbeeb6cb198a Author: jarbasal Date: Fri Oct 22 00:04:00 2021 +0100 fix/race condition dictionary changed iteration bus events could cause the dict to change while being iterated commit dca1184bd5a54f950df947d132174bac9ad7bded Author: Siavash Mollayi Date: Wed Nov 3 15:57:34 2021 +0330 Azerbaijani language support commit 9468c205582663e040825ebbca5f2baa1a1b973c Author: Sergey Leschina Date: Tue Nov 2 03:59:38 2021 +0300 Add more Russian phonetic spellings commit 81de3d4c446f1eb3559c0d59152807e61e7ebdf9 Author: Sergey Leschina Date: Fri Oct 22 02:57:21 2021 +0300 Update Russian words and dialogs commit dd710a6a53383f257384758c6d1fa2833ffb98a6 Merge: 0247b3a4b5 2b05f27fea Author: Kris Gesling Date: Mon Nov 1 11:17:18 2021 +0930 Merge pull request #2995 from MycroftAI/dependabot/pip/pillow-8.3.2 Bump pillow from 8.2.0 to 8.3.2 commit 0247b3a4b5c3389347bfa05aaa293ac24ee54842 Author: Gaëtan Trellu Date: Fri Oct 29 01:29:22 2021 -0400 [log_format] Add an option to change the log format (#3016) * [log_format] Add an option to change the log format Hacing an option to change the log format could be useful when logs are shipped into an aggregator such as Elasticsearch. The current format is very hard to parse. * [log_format] Set default Formatter Because mycroft.configuration.Configuration() class import LOG class, a default Formatter have to be defined before import the mycroft.configuration.Configuration() class. commit 34ee3a9a847dea9007357114fb9dc4b23142d903 Author: Bart Ribbers Date: Thu Oct 28 07:12:23 2021 +0200 Upgrade websocket-client to 1.2.1 (#2879) * Upgrade websocket-client to 1.2.1 core equivalent of https://github.com/MycroftAI/mycroft-messagebus-client/pull/21 There was an incompability with the latest websocket-client and the messagebus which needed fixing for Linux distro compatibility. Since messagebus-client was going to depend on websocket-client 1.2.1, let's do the same in core * Bump message bus client and tornado versions New version of the messagebus-client released. Upgraded Tornado to match new function signature. * Revert removal of websocket-client It is still used in the `mycroft.client.text.gui_server` Should investigate removal of this so that websocket client versions do not need to be kept in sync between mycroft-core and mycroft-messagebus-client. Co-authored-by: Kris Gesling commit a8ef467f4ef7c2741662554f8f60e436b1f40d8c Merge: 3495acf9ab aa067419b0 Author: Kris Gesling Date: Thu Oct 28 11:11:01 2021 +0930 Merge pull request #3015 from goldyfruit/dev [mycroft-config] Fix issue #2991 - use new XDG compliant mycroft.conf location commit aa067419b0119e8c312d9416625f5f17136f76e6 Author: Gaëtan Trellu Date: Mon Oct 25 14:20:25 2021 -0400 [mycroft-config] Fix missing $... commit 782c174668cf57e1c6b08eda517c40b488251d9c Author: Gaëtan Trellu Date: Sat Oct 23 21:43:06 2021 -0400 [mycroft-config] Fix issue #2991 commit 3495acf9abb183a7b36e5ebc0a8f5c25159993f8 Merge: e6ad7faf80 0f98c566b9 Author: Kris Gesling Date: Thu Oct 21 12:25:34 2021 +0930 Merge pull request #3003 from Joanguitar/dev Add reusable PadatiousMatcher to speed up intent matching commit e6ad7faf8054311f136c18e5f2c0825126d26fe6 Merge: 6daccc4171 f50b27f839 Author: Kris Gesling Date: Wed Oct 20 16:02:46 2021 +0930 Merge pull request #3012 from in03/bugfix/issue-3011 Fix disabling of Precise on systems without AVX or similar commit f50b27f8392737649f4b107e3a233f37e9f670ec Author: Caleb Trevatt Date: Mon Oct 18 23:37:16 2021 +1000 Issue-3011 - Fixed AVX / Precise unavailable check commit faebb13e1029a86ec6ecaccba8276c696cdf22f7 Author: Caleb Trevatt Date: Mon Oct 18 23:32:43 2021 +1000 Issue-3011 - Fixing AVX / Precise unavailable check commit 6daccc41711a8c0b51c152ef23fe2d6fefbece9e Merge: df78af15cb 5229f61bbf Author: Åke Date: Sat Oct 16 13:06:47 2021 +0200 Merge pull request #3010 from MycroftAI/bugfix/skill-loader-log-msg Fix formatting error in skill loader log message commit 5229f61bbf37f91efe7e63c3bd95b43b85fe18d1 Author: Chris Veilleux Date: Fri Oct 15 14:51:05 2021 -0500 Fixed formatting error in skill loader log message commit df78af15cb8e736e849b96bcd9a0c36e6e4315c9 Merge: 039c84ee81 4b729dc978 Author: Chris Veilleux Date: Thu Oct 7 10:59:50 2021 -0500 Merge pull request #3002 from MycroftAI/test/extend-criteria-matcher Extended the idea of the VK CriteriaWaiter commit 4b729dc978d9dd13efaf9356231bbcb49d4e015f Author: Chris Veilleux Date: Wed Sep 29 17:57:31 2021 -0500 Make matcher class signatures more consistent. commit acd6d4065adfdac3e9dd91503d912f2f7681171b Author: Chris Veilleux Date: Tue Sep 28 12:41:36 2021 -0500 Apply code review changes. commit 039c84ee818529a7a37c1b0f3f3cb27e00c54b78 Merge: 2a6bb90c9f 32812f605b Author: Kris Gesling Date: Tue Sep 28 17:09:59 2021 +0930 Merge pull request #3000 from el-tocino/bugfix/2999 alter avx test to accomodate aarch64 commit 2a6bb90c9f673b9ff569945e885bb42cd7814354 Merge: 5e81e2747b f709bb9a1e Author: Kris Gesling Date: Mon Sep 27 10:30:12 2021 +0930 Merge pull request #2963 from forslund/feature/update-adapt-keyword-registration-message Update key names in adapt keyword registration message commit f709bb9a1e4c7315be9ffa433aeabfb24c185891 Author: Åke Forslund Date: Mon Aug 2 10:52:10 2021 +0200 Use IntentServiceInterface in MycroftSkill.register_vocabulary() This moves the message logic for adapt keyword registration into a single location. commit 189267b6f66c0569f25a5538051b05369e101d05 Author: Åke Forslund Date: Thu Jul 29 13:17:56 2021 +0200 Minor cleanup of test case for keyword registration commit 693100e8c61566ffb140b53ebb76a1303bada238 Author: Åke Forslund Date: Thu Jul 29 13:10:54 2021 +0200 Match keyword entity terms in Mycroft with Adapt This changes the internally used names for entities and entity values when sent on the messagebus and used interanally in the intent service from start / end to entity_value and entity_type. This makes the terminology easier to understand and follow across into Adapt. The old terms are still included and usable for compatibility but should be removed in an upcoming major release (22.02). commit 5e81e2747b94376c374b196fb3e11a436b72d694 Author: ken-mycroft <67077327+ken-mycroft@users.noreply.github.com> Date: Mon Sep 20 03:03:28 2021 -0400 Improve confidence calculation for Common Query (#2986) * Improve confidence calculation * Add actual noise words file * Update expected test confidence levels Co-authored-by: Kris Gesling commit 0f98c566b94369aa8b34a9e936d250aa211790e0 Author: Joan Palacios Date: Thu Sep 16 13:31:22 2021 -0400 Removed lru_cache commit 863e7844d1486c3ebc987f18ee67e6dbad2d9e70 Author: Joan Date: Thu Sep 16 12:48:23 2021 -0400 Padatious doesn't need to run 3 times commit 3c76177e75c1859a4a23b639438379455033d063 Merge: 9e9f2f74e5 0896c3ceda Author: Chris Veilleux Date: Wed Sep 15 08:27:12 2021 -0500 Merge pull request #3001 from MycroftAI/bugfix/vk-bus-remove-exception fixed an error with message formatting in a ValueError message commit 7f7460a6cd195a4931c0c11fb42d9671db0aac06 Author: Chris Veilleux Date: Tue Sep 14 14:49:39 2021 -0500 Fix a PEP8 speaks issue commit 1856296ea7282ece7457a559757f96a8c40b0013 Author: Chris Veilleux Date: Tue Sep 14 14:40:27 2021 -0500 Extended the idea of the criteria matcher to handle other types of matching commit 0896c3cedae91661828584b6bc7e0454bf4556e1 Author: Chris Veilleux Date: Tue Sep 14 14:31:35 2021 -0500 fixed an error with message formatting in a ValueError message. commit 32812f605b2a4b2048e7a0fefbdf64a97348c7dc Author: el-tocino Date: Tue Sep 14 01:27:06 2021 -0500 alter avx test to accomodate aarch64 commit 9e9f2f74e551efebacf1bd26fc3ed7fde35fae48 Merge: 600aa76206 8d11349b65 Author: Kris Gesling Date: Mon Sep 13 16:37:49 2021 +0930 Merge pull request #2975 from MycroftAI/feature/vk-only-shutdown-on-ci VK: Only stop Mycroft services if running in CI commit 600aa76206a7e7ffa2fbba9dfc62f51080420a44 Author: Daniel McKnight <34697904+NeonDaniel@users.noreply.github.com> Date: Sun Sep 12 18:45:57 2021 -0700 Add 'utterances' to message emitted to skill intent handler (#2997) * Add 'utterances' to message emitted to skill intent handler * Reformat comment to resolve PEP warnings * Update comments per PR feedback * Cut line-length for style compliance Co-authored-by: Kris Gesling commit 06253906fc191c59bded69232f699ff427f58175 Merge: 5a1e4ed8b9 d26201d978 Author: Chris Veilleux Date: Fri Sep 10 16:30:02 2021 -0500 Merge pull request #2979 from MycroftAI/bugfix/release-gui-on-shutdown Release GUI on Skill shutdown or reload commit 5a1e4ed8b9b77f5d83a89eed875c78301aaf2cf1 Merge: aa4a25f29f df94e2192a Author: Chris Veilleux Date: Fri Sep 10 16:29:14 2021 -0500 Merge pull request #2990 from MycroftAI/bugfix/vk-bus-clear-messages Fix messages clearing in the InterceptAllBusClient commit aa4a25f29f1f615a209b598c787ab8aa0a20d4cf Merge: 31b2979bf7 59c473cac8 Author: Kris Gesling Date: Fri Sep 10 11:30:43 2021 +0930 Merge pull request #2992 from MycroftAI/bugfix/ci-pulse-failing Fix pulseaudio daemon failing to start in CI commit 31b2979bf72e91a18b5144a49340b116d1c9331b Merge: 4bce5345cb 96e719fe2d Author: Kris Gesling Date: Thu Sep 9 14:21:38 2021 +0930 Merge pull request #2996 from MycroftAI/bugfix/behave-install Patch dependencies that use the deprecated use_2to3 commit 96e719fe2d8e46a0f8686f8e31fd3f9ec500b3b2 Author: Kris Gesling Date: Wed Sep 8 11:57:46 2021 +0930 remove use_2to3 from dependencies Setuptools deprecated use of `use_2to3` from v58 See changelog: https://setuptools.readthedocs.io/en/latest/history.html#v58-0-0 This has been reported upstream to both packages. The simplest fix seems to be removing Python2 support. It's more difficult to pin setuptools as this is installed via system packages in dev_setup.sh commit 2b05f27feaf6ede7cbd613266bc0c9eed3ed5e7f Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Sep 7 23:45:13 2021 +0000 Bump pillow from 8.2.0 to 8.3.2 Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.2.0 to 8.3.2. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/8.2.0...8.3.2) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] commit 4bce5345cbad1bbe39db4f7546f64edcf299361d Merge: 13539d3397 8a68c88bcf Author: Kris Gesling Date: Tue Sep 7 16:09:05 2021 +0930 Merge pull request #2985 from MycroftAI/feature/gTTS-warning Add warning for anyone loading GoogleTTS commit 59c473cac88a62c9420eab9b5a5aba9bf4b7435b Author: Kris Gesling Date: Fri Sep 3 14:54:09 2021 +0930 Fix pulseaudio daemon failing to start in CI commit df94e2192a40e0b092900e2b84389935ac5d3d32 Author: Chris Veilleux Date: Wed Sep 1 13:21:50 2021 -0500 fixed a few spelling errors. commit b6066f90c665b192f7532c22718d9ab07c3ecf85 Author: Chris Veilleux Date: Wed Sep 1 13:15:53 2021 -0500 fixed an issue with how messages are cleared in the InterceptAllBusClient commit 13539d3397d4577b6f1e1cfbf02c5146f5123bd6 Author: devs-mycroft Date: Fri Aug 27 03:02:54 2021 +0000 Version bump from 21.2.0 to 21.2.1 commit 8a68c88bcf6aeca3bd18270f1d2ca3c8a714a8ec Author: Kris Gesling Date: Fri Aug 27 09:49:40 2021 +0930 Add warning for anyone loading GoogleTTS commit c039aebc273fb87f32d287d11b6b5622ad3d7ed4 Merge: 61cbaa71cd 0f8222eef4 Author: Kris Gesling Date: Wed Aug 18 15:22:53 2021 +0930 Merge pull request #2966 from CrossStream/sandbox/rzr/review/master scripts: Support busybox chown commit d26201d978d9288488cedd822894d8544e1147b4 Author: Kris Gesling Date: Tue Aug 17 16:40:05 2021 +0930 Release GUI on Skill shutdown or reload commit 61cbaa71cddba2547682a3eb84bb715abf24e517 Merge: 8fcfa901c3 ac17d7899a Author: Kris Gesling Date: Tue Aug 17 09:53:25 2021 +0930 Merge pull request #2971 from strugee/patch-1 Install pulseaudio-utils on Fedora commit 8fcfa901c3ff9c0151e42773f54c185be6f18af1 Merge: 196e9750e3 eadb5c9985 Author: Chris Veilleux Date: Fri Aug 13 14:53:46 2021 -0500 Merge pull request #2970 from MycroftAI/feature/mycroft-skills-adaptintent Provide AdaptIntent from mycroft.skills commit 8d11349b6515f45afaea9c4bcd547011e18a57d3 Author: Kris Gesling Date: Thu Aug 12 10:24:38 2021 +0930 Only stop Mycroft services if running in CI commit ac17d7899a357cc21876520b95ccc6e29935c6c1 Author: AJ Jordan Date: Mon Aug 9 06:29:20 2021 -0400 Install pulseaudio-utils on Fedora Needed for `paplay`. commit 196e9750e341ec7a0a8931fcf625b35d0a8eae6b Merge: 5f72294b61 8be5e9ba72 Author: Kris Gesling Date: Wed Aug 11 09:44:05 2021 +0930 Merge pull request #2967 from AIIX/fix/webview_fullscreen add fullscreen fix for webviews commit 5f72294b6193a1759df916d2827f4ab0e27e652f Merge: 1256723c90 a65a3f7770 Author: Kris Gesling Date: Tue Aug 10 15:39:44 2021 +0930 Merge pull request #2968 from forslund/test/vk-no-skill-updates-at-runtime Voight Kampff: No skill updates at runtime commit 1256723c90831fa1d66b98fc8802ee2bc5cdffba Merge: 170ebc44ec 4258e2f2ed Author: Kris Gesling Date: Tue Aug 10 13:47:33 2021 +0930 Merge pull request #2794 from forslund/pure-try-out/xgd-config Pure-try-out's XDG config PR with changed Jenkinsfile commit 0f8222eef4b19374d9dfd717d1ea8eacbd859820 Author: Philippe Coval Date: Thu Aug 5 18:12:11 2021 +0200 scripts: Support busybox chown Those extra options are not enabled in busybox (at least not the configured version in poky yocto distribution) This will also help to support systems without coreutils, Forwarded: https://github.com/MycroftAI/mycroft-core/pull/2966 Relate-to: https://github.com/MycroftAI/mycroft-core/pull/2686 Signed-off-by: Philippe Coval commit eadb5c9985b174290fae8a258f9bf9670317b663 Author: Kris Gesling Date: Mon Aug 9 15:55:04 2021 +0930 Provide AdaptIntent from mycroft.skills This is convenience for Skill developers who can now import all standard items from a single level. Eg: from mycroft.skills import MycroftSkill, intent_handler, AdaptIntent commit a65a3f7770e8ec092cedf582d963c0cb58c7e350 Author: Åke Forslund Date: Thu Aug 5 17:03:34 2021 +0200 Disable auto updates of skills during VK test run commit 8be5e9ba722771334b38c95d0b88be013af88678 Author: Aditya Mehra Date: Fri Aug 6 14:32:52 2021 +0930 add fullscreen fix commit 4258e2f2ed7fbe8691a62ae4ad916a75999c3c63 Author: Åke Forslund Date: Sun Jul 18 21:13:28 2021 +0200 Reorder imports of mycroft.configuration commit 87d22d30a49891056b5659a488c37da121865f2c Author: jarbasal Date: Wed Jul 7 15:38:40 2021 +0100 fix cyclic imports with LOG The mycroft.util.log module called LOG.init() which needs to read the config, the configuration module imports the log system to log causing cyclic import issues. This moves the LOG.init() call out of the log module making it possible to use the normal config system to init the logs and does a slight re-arrangement so that the uninited log can be used. commit 3fd96cf71b25aed79310cb4e9a0edf4fb185609d Author: Åke Forslund Date: Sat Jul 10 17:40:34 2021 +0200 WIP Review comments - Add TODO for 22.02 to remove the compatibility code - Make Warning a single Log statement - mycroft-config script now uses XDG-environment variable - Remove redundant code - Replace hard coded references to ~/.config - Explicitly remove new path before move of "filesystem" (if needed) commit 9029dc1f41d89d86c629ba45c750330b39755b61 Author: Åke Forslund Date: Mon Jul 5 07:22:52 2021 +0200 Slight cleanup - Make XDG usage more visible by using the xdg module prefix - fix overloaded python keywords - remove unused imports commit 8e69d4616d7052e512ff5f90fe72fe62c62f3361 Author: Åke Forslund Date: Wed May 5 20:32:19 2021 +0200 Correct resolution order for resolving log configs commit 28017993c5fae3fedb0d8b8bd08d7db3de76faff Author: Åke Forslund Date: Sun Feb 21 18:08:46 2021 +0100 Restore system locations to mycroft.configuration Fixes issue with Timer skill commit faf101bfcd0ab5784a88e9a62bb633ff8b72b3ae Author: Åke Forslund Date: Sun Jan 31 08:56:10 2021 +0100 Increase Jenkins timeout commit e20443b82458a1185743fe82ced5bf1788b8d3f0 Author: Bart Ribbers Date: Fri May 8 21:32:33 2020 +0200 Use XDG Base directories for settings, cache and runtime data Improve deprecation warning message commit ff9f8e898dc11fe38a8731f9240e037b5cae1129 Author: Åke Forslund Date: Fri Jan 8 21:40:22 2021 +0100 Update identity location for VK test Moved from .mycroft to XDG folder commit 170ebc44ecf5cf3ceeaf44f282c992754b1bbc1b Merge: e40530a8ac b40fcf0e93 Author: Kris Gesling Date: Thu Aug 5 15:40:04 2021 +0930 Merge pull request #2951 from forslund/feature/then-wait-with-event-handler Use event handler to capture messages in then_wait() commit e40530a8ac57087ca59d4d14beb3a73b87c1988d Merge: 313f4e8759 4b66fb1dd1 Author: Kris Gesling Date: Thu Aug 5 15:39:12 2021 +0930 Merge pull request #2948 from forslund/bugfix/vk-message-race Fix Possible vk message race commit 313f4e87592222d40fdfb7d605ef4533b3f4267d Merge: c996008478 8f41d176d0 Author: Kris Gesling Date: Thu Aug 5 09:29:17 2021 +0930 Merge pull request #2946 from MycroftAI/refactor/vk-test-helpers Improve the speed of waiting for dialogs helper function commit c996008478917e7460a995b678d2239a50b24de8 Merge: 7c413de32c d14e6b65d7 Author: Kris Gesling Date: Wed Aug 4 12:05:13 2021 +0930 Merge pull request #2944 from MycroftAI/bugfix/ci-unique-temp-paths Use unique paths for temp file storage commit 7c413de32c6916531b568faef8affa18d0475c5d Merge: c21e74d4d4 45d5d9e478 Author: Kris Gesling Date: Wed Aug 4 11:42:45 2021 +0930 Merge pull request #2956 from AIIX/web_enable_request_feature Add feature request functionality to webviews commit c21e74d4d460f80a0f69571de3a426ddfe25f7df Merge: 480c604dfa b7a0853f3c Author: Kris Gesling Date: Tue Aug 3 15:25:57 2021 +0930 Merge pull request #2960 from MycroftAI/feature/upgrade-adapt Upgrade Adapt to v0.5.1 commit 480c604dfa53b87326116d81cb9e30d83f452f46 Merge: 450a092887 a794db0c9a Author: Kris Gesling Date: Tue Aug 3 07:55:47 2021 +0930 Merge pull request #2962 from MycroftAI/feature/configurable-network-tests Make network tests configurable commit a794db0c9a4b54c134a84cc209c7c9ff42134039 Author: Kris Gesling Date: Mon Aug 2 15:13:06 2021 +0930 Make network tests configurable Previously test URIs were hardcoded. They can now be configured in mycroft.conf commit b7a0853f3cb66ae12fe2df51be7cdfc2e9983594 Author: Kris Gesling Date: Tue Jul 27 23:06:43 2021 +0930 Upgrade Adapt to v0.5.1 - Fix removal of regex entities - Update trie dosctrings - Guarantee sorted results from IntentDeterminationEngine - Enumerate all possible parse results if context or regex entities are in play. - Explicit test to assert results are sorted - Fix name of LICENSE file in Adapt package commit 450a0928874b30eb3e697514594140caa816f624 Merge: 1d03eadb59 24c4ba4a05 Author: Kris Gesling Date: Mon Jul 26 20:51:47 2021 +0930 Merge pull request #2957 from forslund/bugfix/load-audioservice-plugin Fix loading of audioservice plugins commit b40fcf0e938f726c9e5b427c6295ca029052468a Author: Åke Forslund Date: Thu Jul 22 21:07:02 2021 +0200 Update VK then step for checking messagetype Simplify the function and use the standard then_wait() for the heavy lifting, this makes it utilize the new event driven functionality. commit 171d3840f3ba2e696280d5d00dcb7908dc9c5b94 Author: Åke Forslund Date: Thu Jul 22 17:08:50 2021 +0200 Remove sleeps from end of scenarios and features commit 578a3ec439accb15bf70752e8302223ea57d2137 Author: Åke Forslund Date: Sun Jul 11 19:40:28 2021 +0200 Use event to capture messages in then_wait() Instead of busily polling the bus for new messages an event handler is registered (and teared down after check is complete) commit 24c4ba4a057c335045d70a2c734f93375ee8f034 Author: jarbasal Date: Thu Jul 22 20:31:23 2021 +0100 Fix loading of audioservice plugins commit 45d5d9e4787e86519d06ec6054843422ff1250f6 Author: Aditya Mehra Date: Fri Jul 23 15:14:06 2021 +0930 Add feature request functionality to webviews commit 1d03eadb5988258ec6d5b478f974b69c8e0bf0e2 Merge: d1816f5933 cbd17a8dc3 Author: Chris Veilleux Date: Thu Jul 22 14:32:15 2021 -0500 Merge pull request #2954 from forslund/bugfix/vk-missing-stock-skill VK: Remove mycroft-stock from install list commit cbd17a8dc34f4bcbbed61dbca8275df7a9d73b8d Author: Åke Forslund Date: Thu Jul 22 17:15:12 2021 +0200 Remove mycroft-stock from install list commit d1816f593376c551d7b98a6a563696f861900435 Merge: 31e1e2d857 e0489a8488 Author: Kris Gesling Date: Wed Jul 14 09:43:49 2021 +0930 Merge pull request #2807 from forslund/test/jenkins-update-comment Add update comment logic to Jenkins commit 31e1e2d857d2804032dc65e9e18c0340d8dac880 Merge: b3a0b3b1e4 a211441acc Author: Kris Gesling Date: Mon Jul 12 21:46:17 2021 +0930 Merge pull request #2927 from ChanceNCounter/fix/lang-none stop passing lang=None to Lingua Franca commit 8f41d176d069b751adbf5a3d61951963dda4d237 Author: Chris Veilleux Date: Fri Jul 9 16:39:49 2021 -0500 remove print statement used for testing commit 6fd97b4e85b0cb2057b166f930c68e51ca85138d Author: Chris Veilleux Date: Fri Jul 9 15:45:07 2021 -0500 fixed PEP8 issue commit fbb1d06adc601a35e8d1d0ff2008b24f731c5cdb Merge: 4d7ed0f25e 105a5b4be4 Author: Chris Veilleux Date: Fri Jul 9 15:43:25 2021 -0500 Merge remote-tracking branch 'origin/refactor/vk-test-helpers' into refactor/vk-test-helpers commit 4b66fb1dd1c8407eb3996561136dbea131334f1c Author: Åke Forslund Date: Fri Jul 9 16:47:30 2021 +0200 Add docstrings to InterceptAllBusClient commit cf355360addb7aa496fe6bfbe29a1177461533bb Author: Åke Forslund Date: Fri Jul 9 16:08:05 2021 +0200 Fix message race when using clear_messages() This handles a scenario that a message arrives between a call to get_messages() and clear_messages(). clear_messages() will only clear the messages that has been evaluated atleast once. A new method clear_all_messages() has been added to clear the entire message stack and is used between scenarios to reset the list. commit b3a0b3b1e4ed379ca2c242bca91ff30add176644 Merge: 10d1dcfe04 0ae02b5939 Author: Kris Gesling Date: Thu Jul 8 11:56:32 2021 +0930 Merge pull request #2938 from forslund/bugfix/tts-returned-cache-path Fix TTS using the returned path commit 10d1dcfe0495be5a02eefced543fa2bdcc511087 Merge: 2deab67021 18cb28088d Author: Chris Veilleux Date: Wed Jul 7 02:17:59 2021 -0500 Merge pull request #2945 from MycroftAI/refactor/race-condition-comment Added comments to document race condition commit 105a5b4be4b9d0b9518e780559bda9cdda3c0fab Author: Chris Veilleux Date: Tue Jul 6 14:26:23 2021 -0500 Improve the speed of waiting for dialogs by exiting the loop after a match is spoken. Also provide error handling for when a match is not found. commit 18cb28088d5da217eecd0bc39a0d83db88cc53b9 Author: Chris Veilleux Date: Tue Jul 6 13:48:52 2021 -0500 Added comments to document race condition commit 2deab67021185ba72878e90573afe30b2fbb3b86 Merge: 80f9eb053c 28c52fef75 Author: Kris Gesling Date: Tue Jul 6 22:16:44 2021 +0930 Merge pull request #2943 from MycroftAI/feature/localization-fa-ir fa-ir localization initialized commit d14e6b65d73010fd7b383d15c41c7930d13af64f Author: Kris Gesling Date: Tue Jul 6 12:41:28 2021 +0930 Use unique paths for temp file storage When transferring the Allure report and the Mycroft logs to the report host, the zip files were being written to a common directory. In the event that multiple jobs were writing to or reading from this directory at the same time, conflicts could occur. This ensures that both zip files are written to unique paths and cleaned up afterward. commit 28c52fef756eea25f886a249820e20c7cff6464f Author: HKalbasi <45197576+HKalbasi@users.noreply.github.com> Date: Mon Jul 5 05:32:15 2021 +0430 fa-ir localization initialized (#2778) ==== Localization Notes ==== fa-ir: 35 strings added commit 80f9eb053c693f2fb23777e9c6630892ffb05aa0 Merge: 6099da18b3 3959267726 Author: Chris Veilleux Date: Fri Jul 2 15:03:03 2021 -0500 Merge pull request #2937 from MycroftAI/bugfix/vk-race-condition Fix VK race condition in MycroftSkill.get_response() commit 0ae02b5939bd7ed8d110561f76e528e8d8c4006c Author: Åke Forslund Date: Fri Jul 2 08:08:30 2021 +0200 Fix TTS using the returned path commit 395926772699888a1a3a2f68442d7f89962385b4 Author: Chris Veilleux Date: Thu Jul 1 22:31:22 2021 -0500 Reordered wait_while_speaking and sleep to fix a race condition that was occurring with the converse logic in MycroftSkill.get_response() commit a211441acc1799023d33c2787ac8ae879148fcfa Author: ChanceNCounter Date: Wed Jun 23 21:56:19 2021 -0700 stop passing lang=None to Lingua Franca LF's only breaking change over the past two versions has been the deprecation of `lang=None` as a valid parameter. This is because the new language loading paradigm wants to load certain functions on the fly, which it cannot do when it is explicitly told to look for a null lang. I've addressed this by passing `lingua_franca.get_default_lang()` where the `lang=None` call remained. Bonus: Gets rid of over 200 DeprecationWarnings in unit tests! commit 4d7ed0f25ecd468838b3fabbf21e1e3880c6a1c8 Author: Chris Veilleux Date: Sat Jun 5 13:39:57 2021 -0500 Added to docstring to explain why the method took a list of utterances instead of a single utterance. commit 2a9f63c1730f95c10fe2bf85bb2d5bc5afc4c91a Author: Chris Veilleux Date: Sat Jun 5 12:15:02 2021 -0500 Fixed a bug where the highest confidence from the Adapt parser is different than the highest confidence from the Adapt intent matcher. commit e0489a8488f4412a4d081d233470797cec187b7d Author: Åke Forslund Date: Sun Jan 17 21:52:17 2021 +0100 Add update comment logic to Jenkins --- .github/CONTRIBUTING.md | 2 +- .github/workflows/unit_test.yml | 12 + .shellcheckrc | 3 + Jenkinsfile | 77 ++- README.md | 8 +- bin/mycroft-cli-client | 2 +- bin/mycroft-config | 56 ++- bin/mycroft-help | 4 +- bin/mycroft-listen | 2 +- bin/mycroft-mic-test | 4 +- bin/mycroft-msk | 2 +- bin/mycroft-msm | 2 +- bin/mycroft-pip | 2 +- bin/mycroft-say-to | 6 +- bin/mycroft-skill-testrunner | 31 +- bin/mycroft-speak | 6 +- bin/mycroft-start | 4 +- bin/mycroft-stop | 4 +- dev_setup.sh | 139 ++++-- mycroft/__init__.py | 3 + mycroft/api/__init__.py | 17 +- mycroft/audio/audioservice.py | 8 +- mycroft/audio/services/chromecast/__init__.py | 168 ------- mycroft/audio/services/mplayer/__init__.py | 130 ----- mycroft/audio/speech.py | 13 +- mycroft/client/enclosure/__main__.py | 12 +- mycroft/client/enclosure/base.py | 2 +- mycroft/client/enclosure/mark1/__init__.py | 11 +- mycroft/client/speech/hotword_factory.py | 37 +- mycroft/client/text/text_client.py | 36 +- mycroft/configuration/config.py | 118 ++++- mycroft/configuration/locations.py | 29 +- mycroft/configuration/mycroft.conf | 27 +- mycroft/enclosure/gui.py | 2 +- mycroft/filesystem/__init__.py | 13 +- mycroft/identity/__init__.py | 11 +- mycroft/messagebus/send_func.py | 7 +- mycroft/res/text/az-az/and.word | 1 + mycroft/res/text/az-az/backend.down.dialog | 1 + mycroft/res/text/az-az/cancel.voc | 3 + .../text/az-az/checking for updates.dialog | 2 + mycroft/res/text/az-az/day.word | 1 + mycroft/res/text/az-az/hour.word | 1 + .../res/text/az-az/i didn't catch that.dialog | 3 + mycroft/res/text/az-az/last.voc | 2 + .../text/az-az/message_loading.skills.dialog | 1 + .../res/text/az-az/message_rebooting.dialog | 1 + .../text/az-az/message_synching.clock.dialog | 1 + .../res/text/az-az/message_updating.dialog | 1 + mycroft/res/text/az-az/minute.word | 1 + mycroft/res/text/az-az/mycroft.intro.dialog | 1 + mycroft/res/text/az-az/no.voc | 3 + .../not connected to the internet.dialog | 4 + mycroft/res/text/az-az/not.loaded.dialog | 1 + mycroft/res/text/az-az/or.word | 1 + mycroft/res/text/az-az/phonetic_spellings.txt | 3 + .../az-az/reset to factory defaults.dialog | 1 + mycroft/res/text/az-az/second.word | 1 + mycroft/res/text/az-az/skill.error.dialog | 1 + mycroft/res/text/az-az/skills updated.dialog | 1 + ...y I couldn't install default skills.dialog | 1 + .../res/text/az-az/time.changed.reboot.dialog | 1 + mycroft/res/text/az-az/yes.voc | 4 + mycroft/res/text/en-us/noise_words.list | 30 ++ mycroft/res/text/en-us/phonetic_spellings.txt | 2 + mycroft/res/text/fa-ir/and.word | 1 + mycroft/res/text/fa-ir/backend.down.dialog | 4 + mycroft/res/text/fa-ir/cancel.voc | 3 + .../text/fa-ir/checking for updates.dialog | 2 + mycroft/res/text/fa-ir/day.word | 1 + mycroft/res/text/fa-ir/days.word | 1 + mycroft/res/text/fa-ir/hour.word | 1 + mycroft/res/text/fa-ir/hours.word | 1 + .../res/text/fa-ir/i didn't catch that.dialog | 4 + mycroft/res/text/fa-ir/last.voc | 3 + .../res/text/fa-ir/learning disabled.dialog | 1 + .../res/text/fa-ir/learning enabled.dialog | 1 + .../text/fa-ir/message_loading.skills.dialog | 1 + .../res/text/fa-ir/message_rebooting.dialog | 1 + .../text/fa-ir/message_synching.clock.dialog | 1 + .../res/text/fa-ir/message_updating.dialog | 1 + mycroft/res/text/fa-ir/minute.word | 1 + mycroft/res/text/fa-ir/minutes.word | 1 + mycroft/res/text/fa-ir/mycroft.intro.dialog | 1 + mycroft/res/text/fa-ir/no.voc | 4 + .../not connected to the internet.dialog | 4 + mycroft/res/text/fa-ir/not.loaded.dialog | 1 + mycroft/res/text/fa-ir/or.word | 1 + mycroft/res/text/fa-ir/phonetic_spellings.txt | 5 + .../fa-ir/reset to factory defaults.dialog | 1 + mycroft/res/text/fa-ir/second.word | 1 + mycroft/res/text/fa-ir/seconds.word | 1 + mycroft/res/text/fa-ir/skill.error.dialog | 1 + mycroft/res/text/fa-ir/skills updated.dialog | 2 + ...y I couldn't install default skills.dialog | 1 + mycroft/res/text/fa-ir/ssh disabled.dialog | 1 + mycroft/res/text/fa-ir/ssh enabled.dialog | 1 + .../res/text/fa-ir/time.changed.reboot.dialog | 1 + mycroft/res/text/fa-ir/yes.voc | 5 + mycroft/res/text/ru-ru/and.word | 1 + mycroft/res/text/ru-ru/backend.down.dialog | 4 + mycroft/res/text/ru-ru/cancel.voc | 3 + .../text/ru-ru/checking for updates.dialog | 2 + mycroft/res/text/ru-ru/days.word | 2 +- mycroft/res/text/ru-ru/hours.word | 2 +- .../res/text/ru-ru/i didn't catch that.dialog | 5 + mycroft/res/text/ru-ru/last.voc | 4 + .../res/text/ru-ru/learning disabled.dialog | 1 + .../res/text/ru-ru/learning enabled.dialog | 1 + .../text/ru-ru/message_loading.skills.dialog | 1 + .../res/text/ru-ru/message_rebooting.dialog | 1 + .../text/ru-ru/message_synching.clock.dialog | 1 + .../res/text/ru-ru/message_updating.dialog | 1 + mycroft/res/text/ru-ru/minutes.word | 2 +- mycroft/res/text/ru-ru/mycroft.intro.dialog | 1 + mycroft/res/text/ru-ru/no.voc | 4 +- .../not connected to the internet.dialog | 5 + mycroft/res/text/ru-ru/not.loaded.dialog | 1 + mycroft/res/text/ru-ru/or.word | 1 + mycroft/res/text/ru-ru/phonetic_spellings.txt | 19 + .../ru-ru/reset to factory defaults.dialog | 1 + mycroft/res/text/ru-ru/second.word | 2 +- mycroft/res/text/ru-ru/seconds.word | 2 +- mycroft/res/text/ru-ru/skill.error.dialog | 1 + mycroft/res/text/ru-ru/skills updated.dialog | 2 + ...y I couldn't install default skills.dialog | 1 + mycroft/res/text/ru-ru/ssh disabled.dialog | 1 + mycroft/res/text/ru-ru/ssh enabled.dialog | 1 + .../res/text/ru-ru/time.changed.reboot.dialog | 1 + mycroft/res/text/ru-ru/yes.voc | 1 - mycroft/res/text/sv-se/noise_words.list | 29 ++ mycroft/res/ui/FeatureRequest.qml | 123 +++++ mycroft/res/ui/RequestHandler.qml | 35 ++ mycroft/res/ui/SYSTEM_TextFrame.qml | 55 ++- mycroft/res/ui/WebViewHtmlFrame.qml | 17 + mycroft/res/ui/WebViewUrlFrame.qml | 17 + mycroft/skills/__init__.py | 1 + mycroft/skills/common_query_skill.py | 99 +++- mycroft/skills/event_scheduler.py | 12 +- mycroft/skills/intent_service.py | 67 ++- mycroft/skills/intent_service_interface.py | 24 +- mycroft/skills/intent_services/__init__.py | 2 +- .../skills/intent_services/adapt_service.py | 38 +- .../intent_services/padatious_service.py | 127 ++--- mycroft/skills/mycroft_skill/mycroft_skill.py | 16 +- mycroft/skills/skill_loader.py | 2 +- mycroft/skills/skill_updater.py | 7 +- mycroft/stt/__init__.py | 12 + mycroft/tts/espeak_tts.py | 28 +- mycroft/tts/google_tts.py | 7 + mycroft/tts/mimic_tts.py | 2 - mycroft/tts/responsive_voice_tts.py | 61 --- mycroft/tts/tts.py | 179 +++++-- mycroft/util/file_utils.py | 20 +- mycroft/util/format.py | 2 + mycroft/util/log.py | 34 +- mycroft/util/network_utils.py | 34 +- mycroft/util/parse.py | 6 +- mycroft/version/__init__.py | 2 +- requirements/extra-audiobackend.txt | 1 - requirements/requirements.txt | 16 +- requirements/tests.txt | 3 +- scripts/install-mimic.sh | 4 +- scripts/install-pocketsphinx.sh | 83 ---- scripts/install-pygtk.sh | 89 ---- scripts/my-info.sh | 68 ++- scripts/mycroft-use.sh | 90 ++-- scripts/prepare-msm.sh | 8 +- start-mycroft.sh | 126 ++--- stop-mycroft.sh | 61 +-- test/Dockerfile | 2 +- .../voight_kampff/__init__.py | 19 +- .../voight_kampff/default.yml | 1 - .../voight_kampff/features/environment.py | 45 +- .../features/steps/utterance_responses.py | 96 ++-- .../voight_kampff/run_test_suite.sh | 10 +- test/integrationtests/voight_kampff/tools.py | 447 ++++++++++++++++-- test/unittests/audio/test_speech.py | 3 + .../configuration/test_configuration.py | 76 ++- .../skills/test_common_query_skill.py | 24 +- test/unittests/skills/test_intent_service.py | 29 +- .../skills/test_intent_service_interface.py | 52 +- test/unittests/skills/test_mycroft_skill.py | 9 +- test/unittests/skills/test_skill_updater.py | 6 +- test/unittests/tts/test_tts.py | 57 ++- test/unittests/util/commented.json | 3 +- test/unittests/util/plain.json | 3 +- test/unittests/util/test_network_utils.py | 52 ++ test/wake_word/wake_word_test.py | 3 + venv-activate.sh | 12 +- 190 files changed, 2461 insertions(+), 1314 deletions(-) create mode 100644 .shellcheckrc delete mode 100644 mycroft/audio/services/chromecast/__init__.py delete mode 100644 mycroft/audio/services/mplayer/__init__.py create mode 100644 mycroft/res/text/az-az/and.word create mode 100644 mycroft/res/text/az-az/backend.down.dialog create mode 100644 mycroft/res/text/az-az/cancel.voc create mode 100644 mycroft/res/text/az-az/checking for updates.dialog create mode 100644 mycroft/res/text/az-az/day.word create mode 100644 mycroft/res/text/az-az/hour.word create mode 100644 mycroft/res/text/az-az/i didn't catch that.dialog create mode 100644 mycroft/res/text/az-az/last.voc create mode 100644 mycroft/res/text/az-az/message_loading.skills.dialog create mode 100644 mycroft/res/text/az-az/message_rebooting.dialog create mode 100644 mycroft/res/text/az-az/message_synching.clock.dialog create mode 100644 mycroft/res/text/az-az/message_updating.dialog create mode 100644 mycroft/res/text/az-az/minute.word create mode 100644 mycroft/res/text/az-az/mycroft.intro.dialog create mode 100644 mycroft/res/text/az-az/no.voc create mode 100644 mycroft/res/text/az-az/not connected to the internet.dialog create mode 100644 mycroft/res/text/az-az/not.loaded.dialog create mode 100644 mycroft/res/text/az-az/or.word create mode 100644 mycroft/res/text/az-az/phonetic_spellings.txt create mode 100644 mycroft/res/text/az-az/reset to factory defaults.dialog create mode 100644 mycroft/res/text/az-az/second.word create mode 100644 mycroft/res/text/az-az/skill.error.dialog create mode 100644 mycroft/res/text/az-az/skills updated.dialog create mode 100644 mycroft/res/text/az-az/sorry I couldn't install default skills.dialog create mode 100644 mycroft/res/text/az-az/time.changed.reboot.dialog create mode 100644 mycroft/res/text/az-az/yes.voc create mode 100644 mycroft/res/text/en-us/noise_words.list create mode 100644 mycroft/res/text/fa-ir/and.word create mode 100644 mycroft/res/text/fa-ir/backend.down.dialog create mode 100644 mycroft/res/text/fa-ir/cancel.voc create mode 100644 mycroft/res/text/fa-ir/checking for updates.dialog create mode 100644 mycroft/res/text/fa-ir/day.word create mode 100644 mycroft/res/text/fa-ir/days.word create mode 100644 mycroft/res/text/fa-ir/hour.word create mode 100644 mycroft/res/text/fa-ir/hours.word create mode 100644 mycroft/res/text/fa-ir/i didn't catch that.dialog create mode 100644 mycroft/res/text/fa-ir/last.voc create mode 100644 mycroft/res/text/fa-ir/learning disabled.dialog create mode 100644 mycroft/res/text/fa-ir/learning enabled.dialog create mode 100644 mycroft/res/text/fa-ir/message_loading.skills.dialog create mode 100644 mycroft/res/text/fa-ir/message_rebooting.dialog create mode 100644 mycroft/res/text/fa-ir/message_synching.clock.dialog create mode 100644 mycroft/res/text/fa-ir/message_updating.dialog create mode 100644 mycroft/res/text/fa-ir/minute.word create mode 100644 mycroft/res/text/fa-ir/minutes.word create mode 100644 mycroft/res/text/fa-ir/mycroft.intro.dialog create mode 100644 mycroft/res/text/fa-ir/no.voc create mode 100644 mycroft/res/text/fa-ir/not connected to the internet.dialog create mode 100644 mycroft/res/text/fa-ir/not.loaded.dialog create mode 100644 mycroft/res/text/fa-ir/or.word create mode 100644 mycroft/res/text/fa-ir/phonetic_spellings.txt create mode 100644 mycroft/res/text/fa-ir/reset to factory defaults.dialog create mode 100644 mycroft/res/text/fa-ir/second.word create mode 100644 mycroft/res/text/fa-ir/seconds.word create mode 100644 mycroft/res/text/fa-ir/skill.error.dialog create mode 100644 mycroft/res/text/fa-ir/skills updated.dialog create mode 100644 mycroft/res/text/fa-ir/sorry I couldn't install default skills.dialog create mode 100644 mycroft/res/text/fa-ir/ssh disabled.dialog create mode 100644 mycroft/res/text/fa-ir/ssh enabled.dialog create mode 100644 mycroft/res/text/fa-ir/time.changed.reboot.dialog create mode 100644 mycroft/res/text/fa-ir/yes.voc create mode 100644 mycroft/res/text/ru-ru/and.word create mode 100644 mycroft/res/text/ru-ru/backend.down.dialog create mode 100644 mycroft/res/text/ru-ru/cancel.voc create mode 100644 mycroft/res/text/ru-ru/checking for updates.dialog create mode 100644 mycroft/res/text/ru-ru/i didn't catch that.dialog create mode 100644 mycroft/res/text/ru-ru/last.voc create mode 100644 mycroft/res/text/ru-ru/learning disabled.dialog create mode 100644 mycroft/res/text/ru-ru/learning enabled.dialog create mode 100644 mycroft/res/text/ru-ru/message_loading.skills.dialog create mode 100644 mycroft/res/text/ru-ru/message_rebooting.dialog create mode 100644 mycroft/res/text/ru-ru/message_synching.clock.dialog create mode 100644 mycroft/res/text/ru-ru/message_updating.dialog create mode 100644 mycroft/res/text/ru-ru/mycroft.intro.dialog create mode 100644 mycroft/res/text/ru-ru/not connected to the internet.dialog create mode 100644 mycroft/res/text/ru-ru/not.loaded.dialog create mode 100644 mycroft/res/text/ru-ru/or.word create mode 100644 mycroft/res/text/ru-ru/phonetic_spellings.txt create mode 100644 mycroft/res/text/ru-ru/reset to factory defaults.dialog create mode 100644 mycroft/res/text/ru-ru/skill.error.dialog create mode 100644 mycroft/res/text/ru-ru/skills updated.dialog create mode 100644 mycroft/res/text/ru-ru/sorry I couldn't install default skills.dialog create mode 100644 mycroft/res/text/ru-ru/ssh disabled.dialog create mode 100644 mycroft/res/text/ru-ru/ssh enabled.dialog create mode 100644 mycroft/res/text/ru-ru/time.changed.reboot.dialog create mode 100644 mycroft/res/text/sv-se/noise_words.list create mode 100644 mycroft/res/ui/FeatureRequest.qml create mode 100644 mycroft/res/ui/RequestHandler.qml delete mode 100644 mycroft/tts/responsive_voice_tts.py delete mode 100755 scripts/install-pocketsphinx.sh delete mode 100755 scripts/install-pygtk.sh mode change 100644 => 100755 scripts/my-info.sh create mode 100644 test/unittests/util/test_network_utils.py diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 1d3b019348a2..c7de137164a6 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -56,7 +56,7 @@ git push -f * Push your changes to a topic branch in your fork of the repository. * Open a pull request to the original repository and choose the right original branch you want to patch. - _Advanced users may install the `hub` gem and use the [`hub pull-request` command](https://github.com/defunkt/hub#git-pull-request)._ + _Advanced users may install the `hub` gem and use the [`hub pull-request` command](https://hub.github.com/#developer)._ * If not done in commit messages (which you really should do) please reference and update your issue with the code changes. But _please do not close the issue yourself_. * Even if you have write access to the repository, do not directly push or merge pull-requests. Let another team member review your pull request and approve. diff --git a/.github/workflows/unit_test.yml b/.github/workflows/unit_test.yml index 858f23c0f89b..fb73db95b01d 100644 --- a/.github/workflows/unit_test.yml +++ b/.github/workflows/unit_test.yml @@ -66,3 +66,15 @@ jobs: run: if [[ ${{ matrix.python-version }} == 3.9 ]]; then bash <(curl -s https://codecov.io/bash); fi env: CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + shellcheck: + name: Shellcheck + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Shell check mycroft tools + uses: ludeeus/action-shellcheck@203a3fd018dfe73f8ae7e3aa8da2c149a5f41c33 + env: + SHELLCHECK_OPTS: -x + with: + ignore_names: 'run_test_suite.sh' +# ignore_paths: doc mycroft test requirements diff --git a/.shellcheckrc b/.shellcheckrc new file mode 100644 index 000000000000..b96990b42e9d --- /dev/null +++ b/.shellcheckrc @@ -0,0 +1,3 @@ +# Disable sourcing errors +disable=SC1090 +disable=SC1091 diff --git a/Jenkinsfile b/Jenkinsfile index be7ea9d8fa08..7074760979c7 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -14,7 +14,7 @@ pipeline { } environment { //spawns GITHUB_USR and GITHUB_PSW environment variables - GITHUB=credentials('38b2e4a6-167a-40b2-be6f-d69be42c8190') + GITHUB=credentials('DevOps-CLA-Checker-Github-Key') } steps { // Using an install of Github repo CLA tagger @@ -52,12 +52,12 @@ pipeline { --label build=${JOB_NAME} \ -t voight-kampff-mark-1:${BRANCH_ALIAS} .' echo 'Running Mark I Voight-Kampff Test Suite' - timeout(time: 60, unit: 'MINUTES') + timeout(time: 90, unit: 'MINUTES') { sh 'mkdir -p $HOME/core/$BRANCH_ALIAS/allure' sh 'mkdir -p $HOME/core/$BRANCH_ALIAS/mycroft-logs' sh 'docker run \ - -v "$HOME/voight-kampff/identity:/root/.mycroft/identity" \ + -v "$HOME/voight-kampff/identity:/root/.config/mycroft/identity" \ -v "$HOME/core/$BRANCH_ALIAS/allure:/root/allure" \ -v "$HOME/core/$BRANCH_ALIAS/mycroft-logs:/var/log/mycroft" \ --label build=${JOB_NAME} \ @@ -107,29 +107,53 @@ pipeline { sh 'rmdir $HOME/core/$BRANCH_ALIAS' sh ( label: 'Publish Report to Web Server', - script: '''scp allure-report.zip root@157.245.127.234:~; - ssh root@157.245.127.234 "unzip -o ~/allure-report.zip"; + script: ''' + ssh root@157.245.127.234 "mkdir -p ~/allure-reports/core/${BRANCH_ALIAS}"; + scp allure-report.zip root@157.245.127.234:~/allure-reports/core/${BRANCH_ALIAS}; + ssh root@157.245.127.234 "unzip -o ~/allure-reports/core/${BRANCH_ALIAS}/allure-report.zip -d ~/allure-reports/core/${BRANCH_ALIAS}/"; ssh root@157.245.127.234 "rm -rf /var/www/voight-kampff/core/${BRANCH_ALIAS}"; - ssh root@157.245.127.234 "mv allure-report /var/www/voight-kampff/core/${BRANCH_ALIAS}" - scp mycroft-logs.zip root@157.245.127.234:~; + ssh root@157.245.127.234 "mv ~/allure-reports/core/${BRANCH_ALIAS}/allure-report /var/www/voight-kampff/core/${BRANCH_ALIAS}" + ssh root@157.245.127.234 "rm ~/allure-reports/core/${BRANCH_ALIAS}/allure-report.zip"; + ssh root@157.245.127.234 "rmdir ~/allure-reports/core/${BRANCH_ALIAS}"; + ssh root@157.245.127.234 "mkdir -p ~/mycroft-logs/core/${BRANCH_ALIAS}"; + scp mycroft-logs.zip root@157.245.127.234:~/mycroft-logs/core/${BRANCH_ALIAS}/; ssh root@157.245.127.234 "mkdir -p /var/www/voight-kampff/core/${BRANCH_ALIAS}/logs" - ssh root@157.245.127.234 "unzip -oj ~/mycroft-logs.zip -d /var/www/voight-kampff/core/${BRANCH_ALIAS}/logs/"; + ssh root@157.245.127.234 "unzip -oj ~/mycroft-logs/core/${BRANCH_ALIAS}/mycroft-logs.zip -d /var/www/voight-kampff/core/${BRANCH_ALIAS}/logs/"; + ssh root@157.245.127.234 "rm ~/mycroft-logs/core/${BRANCH_ALIAS}/mycroft-logs.zip"; + ssh root@157.245.127.234 "rmdir ~/mycroft-logs/core/${BRANCH_ALIAS}"; ''' ) echo 'Report Published' } failure { script { + def comment_text = 'Voight Kampff Integration Test Failed ([Results](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + ')). ' + + '\nMycroft logs are also available: ' + + '[skills.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/skills.log), ' + + '[audio.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/audio.log), ' + + '[voice.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/voice.log), ' + + '[bus.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/bus.log), ' + + '[enclosure.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/enclosure.log)' + // Create comment for Pull Requests if (env.CHANGE_ID) { - echo 'Sending PR comment' - pullRequest.comment('Voight Kampff Integration Test Failed ([Results](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + ')). ' + - '\nMycroft logs are also available: ' + - '[skills.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/skills.log), ' + - '[audio.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/audio.log), ' + - '[voice.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/voice.log), ' + - '[bus.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/bus.log), ' + - '[enclosure.log](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '/logs/enclosure.log)') + def found_comment = false + for (comment in pullRequest.comments) { + echo "Author: ${comment.user}" + if (comment.user == "devops-mycroft" && + comment.body.contains("Voight Kampff")) { + echo "Updating comment..." + found_comment = true + pullRequest.editComment( + comment.id, + comment_text + ) + } + } + if (!found_comment) { + echo 'Sending PR comment' + pullRequest.comment(comment_text) + } } } // Send failure email containing a link to the Jenkins build @@ -184,8 +208,25 @@ pipeline { success { script { if (env.CHANGE_ID) { - echo 'Sending PR comment' - pullRequest.comment('Voight Kampff Integration Test Succeeded ([Results](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '))') + def comment_text = 'Voight Kampff Integration Test Succeeded ([Results](https://reports.mycroft.ai/core/' + env.BRANCH_ALIAS + '))' + def found_comment = false + for (comment in pullRequest.comments) { + echo "Author: ${comment.user}" + if (comment.user == "devops-mycroft" && + comment.body.contains("Voight Kampff")) { + echo "Updating comment!" + found_comment = true + pullRequest.editComment( + comment.id, + comment_text + ) + break + } + } + if (!found_comment) { + echo 'Sending PR comment' + pullRequest.comment(comment_text) + } } } // Send success email containing a link to the Jenkins build diff --git a/README.md b/README.md index 01958ef2c736..82fcca33e7bf 100644 --- a/README.md +++ b/README.md @@ -73,20 +73,20 @@ Mycroft is nothing without skills. There are a handful of default skills that a ### Pairing Information Pairing information generated by registering with Home is stored in: -`~/.mycroft/identity/identity2.json` <-- DO NOT SHARE THIS WITH OTHERS! +`~/.config/mycroft/identity/identity2.json` <-- DO NOT SHARE THIS WITH OTHERS! ### Configuration Mycroft's configuration consists of 4 possible locations: - `mycroft-core/mycroft/configuration/mycroft.conf`(Defaults) - [Mycroft Home](https://home.mycroft.ai) (Remote) -- `/etc/mycroft/mycroft.conf`(Machine) -- `$HOME/.mycroft/mycroft.conf`(User) +- `/etc/mycroft/mycroft.conf` (Machine) +- `$XDG_CONFIG_DIR/mycroft/mycroft.conf` (which is by default `$HOME/.config/mycroft/mycroft.conf`) (USER) When the configuration loader starts, it looks in these locations in this order, and loads ALL configurations. Keys that exist in multiple configuration files will be overridden by the last file to contain the value. This process results in a minimal amount being written for a specific device and user, without modifying default distribution files. ### Using Mycroft Without Home -If you do not wish to use the Mycroft Home service, before starting Mycroft for the first time, create `$HOME/.mycroft/mycroft.conf` with the following contents: +If you do not wish to use the Mycroft Home service, before starting Mycroft for the first time, create `$HOME/.config/mycroft/mycroft.conf` with the following contents: ``` { diff --git a/bin/mycroft-cli-client b/bin/mycroft-cli-client index f40a316e3f61..a06e0351eb71 100755 --- a/bin/mycroft-cli-client +++ b/bin/mycroft-cli-client @@ -21,4 +21,4 @@ DIR="$( dirname "$SOURCE" )" source "$DIR/../venv-activate.sh" -q # Invoke the Command Line Interface -python -m mycroft.client.text $@ +python -m mycroft.client.text "$@" diff --git a/bin/mycroft-config b/bin/mycroft-config index a1271b328dfc..d2743c2317bf 100755 --- a/bin/mycroft-config +++ b/bin/mycroft-config @@ -15,7 +15,7 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit DIR="$( pwd )" script=${0} script=${script##*/} @@ -49,7 +49,7 @@ function help() { VIEWER="nano --syntax=json --view" if [ -z "$EDITOR" ] ; then - if [ $( which sensible-editor ) ] ; then + if which sensible-editor > /dev/null ; then EDITOR="sensible-editor" else EDITOR="nano --syntax=json --tempfile" @@ -65,11 +65,17 @@ function found_exe() { } if found_exe tput ; then + # shellcheck disable=SC2034 GREEN="$(tput setaf 2)" + # shellcheck disable=SC2034 BLUE="$(tput setaf 4)" + # shellcheck disable=SC2034 CYAN="$(tput setaf 6)" + # shellcheck disable=SC2034 YELLOW="$(tput setaf 3)" + # shellcheck disable=SC2034 RESET="$(tput sgr0)" + # shellcheck disable=SC2034 HIGHLIGHT=${YELLOW} fi @@ -82,26 +88,26 @@ function validate_config_file() { return 0 fi - echo -n ${BLUE} + echo -n "${BLUE}" # Remove any comments (lines starting with # or //) found in the file and # Use jq to validate and output errors sed 's/^\s*[#\/].*$//g' "$1" | sed '/^$/d' | jq -e "." > /dev/null result=$? - echo -n ${RESET} + echo -n "${RESET}" #xxx echo "RESULT=$result for $1" return $result } -_conf_file="~/.mycroft/mycroft.conf" +_conf_file="${XDG_CONFIG_HOME:-$HOME/.config}/mycroft/mycroft.conf" function name_to_path() { case ${1} in "system") _conf_file="/etc/mycroft/mycroft.conf" ;; - "user") _conf_file=$(readlink -f ~/.mycroft/mycroft.conf) ;; + "user") _conf_file=$(readlink -f "${XDG_CONFIG_HOME:-$HOME/.config}/mycroft/mycroft.conf") ;; "default") _conf_file="$DIR/../mycroft/configuration/mycroft.conf" ;; - "remote") _conf_file="/var/tmp/mycroft_web_cache.json" ;; + "remote") _conf_file="$HOME/.cache/mycroft/web_cache.json" ;; *) echo "ERROR: Unknown name '${1}'." @@ -113,12 +119,12 @@ function name_to_path() { ################################################################ function edit_config() { - name_to_path $1 - validate_config_file $_conf_file + name_to_path "$1" + validate_config_file "$_conf_file" rc=$? if [ $rc -ne 0 ] ; then echo "${YELLOW}WARNING: ${RESET}Configuration file did not pass validation before edits." - read -p "Review errors above and press ENTER to continue with editing." + read -r -p "Review errors above and press ENTER to continue with editing." fi if [ -f "${_conf_file}" ] ; then @@ -128,7 +134,7 @@ function edit_config() { echo "}" >> "${TEMP}/mycroft.json" fi - while [ 1 ] ; do + while true ; do case $1 in system | user) # Allow user to edit @@ -150,19 +156,18 @@ function edit_config() { fi # file was changed, validate changes - validate_config_file $TEMP/mycroft.json - if [ $? -ne 0 ] ; then + if validate_config_file $TEMP/mycroft.json > /dev/null ; then + key="S" + else echo "${YELLOW}WARNING: ${RESET}Configuration file does not pass validation, see errors above." echo "Press X to abandon changes, S to force save, any other key to edit again." - read -N1 -s key - else - key="S" + read -r -N1 -s key fi case $key in [Ss]) echo "Saving..." - mv $TEMP/mycroft.json $_conf_file + mv $TEMP/mycroft.json "$_conf_file" signal_reload_config break ;; @@ -180,11 +185,11 @@ function signal_reload_config() { source "$DIR/../venv-activate.sh" -q # Post a messagebus notification to reload the config file - output=$(python -m mycroft.messagebus.send "configuration.updated" "{}") + python -m mycroft.messagebus.send "configuration.updated" "{}" > /dev/null } function show_config() { - name_to_path $1 + name_to_path "$1" # Use jq to display formatted nicely (after stripping out comments) sed 's/^\s*[#\/].*$//g' "${_conf_file}" | sed '/^$/d' | jq "." @@ -201,7 +206,7 @@ function get_config() { json_config=$( source "$DIR/../venv-activate.sh" -q && python -c "import json; from mycroft.configuration import Configuration; print(json.dumps(Configuration.get()))" ) # Read the given variable from the mix - echo ${json_config} | jq -r "${value}" + echo "${json_config}" | jq -r "${value}" } function set_config() { @@ -212,10 +217,9 @@ function set_config() { value=".${value}" fi - jq "${value} = \"$2\"" ~/.mycroft/mycroft.conf > "${TEMP}/~mycroft.conf" - if [ $? -eq 0 ] ; then + if jq "${value} = \"$2\"" "$_conf_file" > "${TEMP}/~mycroft.conf" ; then # Successful update, replace the config file - mv "${TEMP}/~mycroft.conf" ~/.mycroft/mycroft.conf + mv "${TEMP}/~mycroft.conf" "$_conf_file" signal_reload_config fi } @@ -223,16 +227,16 @@ function set_config() { _opt=$1 case ${_opt} in "edit") - edit_config $2 + edit_config "$2" ;; "reload") signal_reload_config ;; "show") - show_config $2 + show_config "$2" ;; "get") - get_config $2 + get_config "$2" ;; "set") set_config "$2" "$3" diff --git a/bin/mycroft-help b/bin/mycroft-help index 822e078a2a60..dc1716be6fad 100755 --- a/bin/mycroft-help +++ b/bin/mycroft-help @@ -15,10 +15,10 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )"/.. +cd -P "$( dirname "$SOURCE" )"/.. || exit DIR="$( pwd )" -echo -e "\e[36mMycroft\e[0m is your open source voice assistant. Full source" +echo -e "\\e[36mMycroft\\e[0m is your open source voice assistant. Full source" echo "can be found at: ${DIR}" echo echo "Mycroft-specific commands you can use from the Linux command prompt:" diff --git a/bin/mycroft-listen b/bin/mycroft-listen index 95a93b642556..d4533eb602eb 100755 --- a/bin/mycroft-listen +++ b/bin/mycroft-listen @@ -15,7 +15,7 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit DIR="$( pwd )" # Enter the Mycroft venv diff --git a/bin/mycroft-mic-test b/bin/mycroft-mic-test index 584d1380940e..676ee712d58f 100755 --- a/bin/mycroft-mic-test +++ b/bin/mycroft-mic-test @@ -15,7 +15,7 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit DIR="$( pwd )" restart=0 @@ -32,7 +32,7 @@ fi # Launch the standard audiotest -"$DIR/../start-mycroft.sh" audiotest $@ +"$DIR/../start-mycroft.sh" audiotest "$@" if [ $restart -eq 1 ] diff --git a/bin/mycroft-msk b/bin/mycroft-msk index be0a719f1e56..99c1e2d8a1c0 100755 --- a/bin/mycroft-msk +++ b/bin/mycroft-msk @@ -21,4 +21,4 @@ DIR="$( dirname "$SOURCE" )" source "$DIR/../venv-activate.sh" -q # Invoke the Mycroft Skills Kit from within the venv -msk $@ +msk "$@" diff --git a/bin/mycroft-msm b/bin/mycroft-msm index ab580da800ba..4edd2c1b0efa 100755 --- a/bin/mycroft-msm +++ b/bin/mycroft-msm @@ -21,4 +21,4 @@ DIR="$( dirname "$SOURCE" )" source "$DIR/../venv-activate.sh" -q # Invoke the Mycroft Skills Manager (msm) within the venv -msm $@ +msm "$@" diff --git a/bin/mycroft-pip b/bin/mycroft-pip index a42b16b847a3..dffe8b9399c8 100755 --- a/bin/mycroft-pip +++ b/bin/mycroft-pip @@ -21,4 +21,4 @@ DIR="$( dirname "$SOURCE" )" source "$DIR/../venv-activate.sh" -q # Install pip packages within the Mycroft venv -pip $@ \ No newline at end of file +pip "$@" diff --git a/bin/mycroft-say-to b/bin/mycroft-say-to index 4ae597f3062e..f5abbfe21a3d 100755 --- a/bin/mycroft-say-to +++ b/bin/mycroft-say-to @@ -15,7 +15,7 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit DIR="$( pwd )" # Enter the Mycroft venv @@ -25,5 +25,5 @@ source "$DIR/../venv-activate.sh" -q set -- "${1:-$(> /dev/null diff --git a/bin/mycroft-skill-testrunner b/bin/mycroft-skill-testrunner index 0e9a7297b450..e0e35491a8bf 100755 --- a/bin/mycroft-skill-testrunner +++ b/bin/mycroft-skill-testrunner @@ -20,26 +20,31 @@ DIR="$( dirname "$SOURCE" )" # Enter the Mycroft venv source "$DIR/../venv-activate.sh" -q +function count-files() { + # shellcheck disable=SC2012 + ls "$1" | wc -l +} + function vktest-clear() { FEATURES_DIR="$DIR/../test/integrationtests/voight_kampff/features" - num_feature_files=$(ls $FEATURES_DIR | wc -l) + num_feature_files=$(count-files "$FEATURES_DIR") # A clean directory will have `steps/` and `environment.py` - if [ $num_feature_files -gt "2" ] ; then + if [ "$num_feature_files" -gt "2" ] ; then echo "Removing Feature files..." - rm ${DIR}/../test/integrationtests/voight_kampff/features/*.feature - rm ${DIR}/../test/integrationtests/voight_kampff/features/*.config.json + rm -f "${DIR}"/../test/integrationtests/voight_kampff/features/*.feature + rm -f "${DIR}"/../test/integrationtests/voight_kampff/features/*.config.json fi STEPS_DIR="$FEATURES_DIR/steps" - num_steps_files=$(ls $STEPS_DIR | wc -l) - if [ $num_steps_files -gt "2" ] ; then + num_steps_files=$(count-files "$STEPS_DIR") + if [ "$num_steps_files" -gt "2" ] ; then echo "Removing Custom Step files..." TMP_DIR="$STEPS_DIR/tmp" - mkdir $TMP_DIR - mv "$STEPS_DIR/configuration.py" $TMP_DIR - mv "$STEPS_DIR/utterance_responses.py" $TMP_DIR - rm ${STEPS_DIR}/*.py - mv ${TMP_DIR}/* $STEPS_DIR - rmdir $TMP_DIR + mkdir "$TMP_DIR" + mv "$STEPS_DIR/configuration.py" "$TMP_DIR" + mv "$STEPS_DIR/utterance_responses.py" "$TMP_DIR" + rm -f "${STEPS_DIR}"/*.py + mv "${TMP_DIR}"/* "$STEPS_DIR" + rmdir "$TMP_DIR" fi echo "Voight Kampff tests clear." } @@ -55,5 +60,5 @@ elif [ "$1" = "vktest" ] ; then python -m test.integrationtests.voight_kampff "$@" fi else - python -m test.integrationtests.skills.runner $@ + python -m test.integrationtests.skills.runner "$@" fi diff --git a/bin/mycroft-speak b/bin/mycroft-speak index d80f0e75b13a..a4eb4d533119 100755 --- a/bin/mycroft-speak +++ b/bin/mycroft-speak @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit DIR="$( pwd )" # Sets var 1 to stdin if no args were given @@ -23,5 +23,5 @@ set -- "${1:-$(> /dev/null diff --git a/bin/mycroft-start b/bin/mycroft-start index 65723c2c3734..cec59efaec86 100755 --- a/bin/mycroft-start +++ b/bin/mycroft-start @@ -15,7 +15,7 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )"/.. +cd -P "$( dirname "$SOURCE" )"/.. || exit DIR="$( pwd )" -. "$DIR/start-mycroft.sh" $@ +. "$DIR/start-mycroft.sh" "$@" diff --git a/bin/mycroft-stop b/bin/mycroft-stop index b86b0ea65577..dc79ca34a8bb 100755 --- a/bin/mycroft-stop +++ b/bin/mycroft-stop @@ -15,7 +15,7 @@ # limitations under the License. SOURCE="${BASH_SOURCE[0]}" -cd -P "$( dirname "$SOURCE" )"/.. +cd -P "$( dirname "$SOURCE" )"/.. || exit DIR="$( pwd )" -. "$DIR/stop-mycroft.sh" $@ +. "$DIR/stop-mycroft.sh" "$@" diff --git a/dev_setup.sh b/dev_setup.sh index 3bbfabc2470c..be51c7ec54f9 100755 --- a/dev_setup.sh +++ b/dev_setup.sh @@ -22,22 +22,28 @@ export LANGUAGE=en # exit on any error set -Ee -cd $(dirname $0) +ROOT_DIRNAME=$(dirname "$0") +cd "$ROOT_DIRNAME" TOP=$(pwd -L) function clean_mycroft_files() { echo ' This will completely remove any files installed by mycroft (including pairing -information). +information). + +NOTE: This will not remove Mimic (if you chose to compile it), or other files +generated within the mycroft-core directory. + Do you wish to continue? (y/n)' while true; do - read -N1 -s key + read -rN1 -s key case $key in [Yy]) sudo rm -rf /var/log/mycroft rm -f /var/tmp/mycroft_web_cache.json rm -rf "${TMPDIR:-/tmp}/mycroft" rm -rf "$HOME/.mycroft" + rm -f "skills" # The Skills directory symlink sudo rm -rf "/opt/mycroft" exit 0 ;; @@ -70,6 +76,7 @@ opt_forcemimicbuild=false opt_allowroot=false opt_skipmimicbuild=false opt_python=python3 +disable_precise_later=false param='' for var in "$@" ; do @@ -140,7 +147,7 @@ function get_YN() { # Loop until the user hits the Y or the N key echo -e -n "Choice [${CYAN}Y${RESET}/${CYAN}N${RESET}]: " while true; do - read -N1 -s key + read -rN1 -s key case $key in [Yy]) return 0 @@ -176,8 +183,8 @@ your environment.' sleep 0.5 # The AVX instruction set is an x86 construct # ARM has a range of equivalents, unsure which are (un)supported by TF. - if ! grep -q avx /proc/cpuinfo && [[ ! $(uname -m) == 'arm'* ]]; then - echo " + if ! grep -q avx /proc/cpuinfo && ! [[ $(uname -m) == 'arm'* || $(uname -m) == 'aarch64' ]]; then + echo " The Precise Wake Word Engine requires the AVX instruction set, which is not supported on your CPU. Do you want to fall back to the PocketSphinx engine? Advanced users can build the precise engine with an older @@ -185,20 +192,20 @@ version of TensorFlow (v1.13) if desired and change use_precise to true in mycroft.conf. Y)es, I want to use the PocketSphinx engine or my own. N)o, stop the installation." - if get_YN ; then - if [[ ! -f /etc/mycroft/mycroft.conf ]]; then - $SUDO mkdir -p /etc/mycroft - $SUDO touch /etc/mycroft/mycroft.conf - $SUDO bash -c 'echo "{ \"use_precise\": true }" > /etc/mycroft/mycroft.conf' + if get_YN ; then + if [[ ! -f /etc/mycroft/mycroft.conf ]]; then + $SUDO mkdir -p /etc/mycroft + $SUDO touch /etc/mycroft/mycroft.conf + $SUDO bash -c 'echo "{ \"use_precise\": false }" > /etc/mycroft/mycroft.conf' + else + # Ensure dependency installed to merge configs + disable_precise_later=true + fi else - $SUDO bash -c 'jq ". + { \"use_precise\": true }" /etc/mycroft/mycroft.conf > tmp.mycroft.conf' - $SUDO mv -f tmp.mycroft.conf /etc/mycroft/mycroft.conf + echo -e "$HIGHLIGHT N - quit the installation $RESET" + exit 1 fi - else - echo -e "$HIGHLIGHT N - quit the installation $RESET" - exit 1 - fi - echo + echo fi echo " Do you want to run on 'master' or against a dev branch? Unless you are @@ -265,9 +272,11 @@ Would you like this to be added to your PATH in the .profile?' if [[ ! -f ~/.profile_mycroft ]] ; then # Only add the following to the .profile if .profile_mycroft # doesn't exist, indicating this script has not been run before - echo '' >> ~/.profile - echo '# include Mycroft commands' >> ~/.profile - echo 'source ~/.profile_mycroft' >> ~/.profile + { + echo '' + echo '# include Mycroft commands' + echo 'source ~/.profile_mycroft' + } >> ~/.profile fi echo " @@ -289,9 +298,9 @@ fi" > ~/.profile_mycroft echo 'This script will create that folder for you. This requires sudo' echo 'permission and might ask you for a password...' setup_user=$USER - setup_group=$(id -gn $USER) + setup_group=$(id -gn "$USER") $SUDO mkdir -p /opt/mycroft/skills - $SUDO chown -R ${setup_user}:${setup_group} /opt/mycroft + $SUDO chown -R "${setup_user}":"${setup_group}" /opt/mycroft echo 'Created!' fi if [[ ! -d skills ]] ; then @@ -319,7 +328,7 @@ If unsure answer yes. fi function os_is() { - [[ $(grep "^ID=" /etc/os-release | awk -F'=' '/^ID/ {print $2}' | sed 's/\"//g') == $1 ]] + [[ $(grep "^ID=" /etc/os-release | awk -F'=' '/^ID/ {print $2}' | sed 's/\"//g') == "$1" ]] } function os_is_like() { @@ -339,11 +348,11 @@ function redhat_common_install() { } function debian_install() { - APT_PACKAGE_LIST="git python3 python3-dev python3-setuptools libtool \ + APT_PACKAGE_LIST=(git python3 python3-dev python3-setuptools libtool \ libffi-dev libssl-dev autoconf automake bison swig libglib2.0-dev \ portaudio19-dev mpg123 screen flac curl libicu-dev pkg-config \ libjpeg-dev libfann-dev build-essential jq pulseaudio \ - pulseaudio-utils" + pulseaudio-utils) if dpkg -V libjack-jackd2-0 > /dev/null 2>&1 && [[ -z ${CI} ]] ; then echo " @@ -351,10 +360,10 @@ We have detected that your computer has the libjack-jackd2-0 package installed. Mycroft requires a conflicting package, and will likely uninstall this package. On some systems, this can cause other programs to be marked for removal. Please review the following package changes carefully." - read -p "Press enter to continue" - $SUDO apt-get install $APT_PACKAGE_LIST + read -rp "Press enter to continue" + $SUDO apt-get install "${APT_PACKAGE_LIST[@]}" else - $SUDO apt-get install -y $APT_PACKAGE_LIST + $SUDO apt-get install -y "${APT_PACKAGE_LIST[@]}" fi } @@ -366,12 +375,20 @@ function open_suse_install() { function fedora_install() { - $SUDO dnf install -y git python3 python3-devel python3-pip python3-setuptools python3-virtualenv pygobject3-devel libtool libffi-devel openssl-devel autoconf bison swig glib2-devel portaudio-devel mpg123 mpg123-plugins-pulseaudio screen curl pkgconfig libicu-devel automake libjpeg-turbo-devel fann-devel gcc-c++ redhat-rpm-config jq make + $SUDO dnf install -y git python3 python3-devel python3-pip python3-setuptools python3-virtualenv pygobject3-devel libtool libffi-devel openssl-devel autoconf bison swig glib2-devel portaudio-devel mpg123 mpg123-plugins-pulseaudio screen curl pkgconfig libicu-devel automake libjpeg-turbo-devel fann-devel gcc-c++ redhat-rpm-config jq make pulseaudio-utils } function arch_install() { - $SUDO pacman -S --needed --noconfirm git python python-pip python-setuptools python-virtualenv python-gobject libffi swig portaudio mpg123 screen flac curl icu libjpeg-turbo base-devel jq pulseaudio pulseaudio-alsa + pkgs=( git python python-pip python-setuptools python-virtualenv python-gobject libffi swig portaudio mpg123 screen flac curl icu libjpeg-turbo base-devel jq ) + + if ! pacman -Qs pipewire-pulse > /dev/null + then + pulse_pkgs=( pulseaudio pulseaudio-alsa ) + pkgs=( "${pkgs[@]}" "${pulse_pkgs[@]}" ) + fi + + $SUDO pacman -S --needed --noconfirm "${pkgs[@]}" pacman -Qs '^fann$' &> /dev/null || ( git clone https://aur.archlinux.org/fann.git @@ -398,11 +415,30 @@ function redhat_install() { } function gentoo_install() { - $SUDO emerge --noreplace dev-vcs/git dev-lang/python dev-python/setuptools dev-python/pygobject dev-python/requests sys-devel/libtool virtual/libffi virtual/jpeg dev-libs/openssl sys-devel/autoconf sys-devel/bison dev-lang/swig dev-libs/glib media-libs/portaudio media-sound/mpg123 media-libs/flac net-misc/curl sci-mathematics/fann sys-devel/gcc app-misc/jq media-libs/alsa-lib dev-libs/icu + $SUDO emerge --noreplace dev-vcs/git dev-lang/python dev-python/setuptools dev-python/pygobject dev-python/requests sys-devel/libtool dev-libs/libffi virtual/jpeg dev-libs/openssl sys-devel/autoconf sys-devel/bison dev-lang/swig dev-libs/glib media-libs/portaudio media-sound/mpg123 media-libs/flac net-misc/curl sci-mathematics/fann sys-devel/gcc app-misc/jq media-libs/alsa-lib dev-libs/icu } function alpine_install() { - $SUDO apk add --virtual makedeps-mycroft-core alpine-sdk git python3 py3-pip py3-setuptools py3-virtualenv mpg123 vorbis-tools pulseaudio-utils fann-dev automake autoconf libtool pcre2-dev pulseaudio-dev alsa-lib-dev swig python3-dev portaudio-dev libjpeg-turbo-dev + $SUDO apk add --virtual .makedeps-mycroft-core \ + alpine-sdk \ + alsa-lib-dev \ + autoconf \ + automake \ + fann-dev \ + git \ + libjpeg-turbo-dev \ + libtool \ + mpg123 \ + pcre2-dev \ + portaudio-dev \ + pulseaudio-utils \ + py3-pip \ + py3-setuptools \ + py3-virtualenv \ + python3 \ + python3-dev \ + swig \ + vorbis-tools } function install_deps() { @@ -445,7 +481,7 @@ function install_deps() { ${YELLOW}Make sure to manually install:$BLUE git python3 python-setuptools python-venv pygobject libtool libffi libjpg openssl autoconf bison swig glib2.0 portaudio19 mpg123 flac curl fann g++ jq\n$RESET" echo 'Warning: Failed to install all dependencies. Continue? y/N' - read -n1 continue + read -rn1 continue if [[ $continue != 'y' ]] ; then exit 1 fi @@ -457,16 +493,30 @@ VIRTUALENV_ROOT=${VIRTUALENV_ROOT:-"${TOP}/.venv"} function install_venv() { $opt_python -m venv "${VIRTUALENV_ROOT}/" --without-pip + + # Check if old script for python 3.6 is needed + if "${VIRTUALENV_ROOT}/bin/${opt_python}" --version | grep " 3.6" > /dev/null; then + GET_PIP_URL="https://bootstrap.pypa.io/pip/3.6/get-pip.py" + else + GET_PIP_URL="https://bootstrap.pypa.io/get-pip.py" + fi + # Force version of pip for reproducability, but there is nothing special # about this version. Update whenever a new version is released and # verified functional. - curl https://bootstrap.pypa.io/get-pip.py | "${VIRTUALENV_ROOT}/bin/python" - 'pip==20.0.2' + curl "${GET_PIP_URL}" | "${VIRTUALENV_ROOT}/bin/${opt_python}" - 'pip==20.0.2' # Function status depending on if pip exists [[ -x ${VIRTUALENV_ROOT}/bin/pip ]] } install_deps +# It's later. Update existing config with jq. +if [[ $disable_precise_later == true ]]; then + $SUDO bash -c 'jq ". + { \"use_precise\": false }" /etc/mycroft/mycroft.conf > tmp.mycroft.conf' + $SUDO mv -f tmp.mycroft.conf /etc/mycroft/mycroft.conf +fi + # Configure to use the standard commit template for # this repo only. git config commit.template .gitmessage @@ -479,7 +529,7 @@ else # first, look for a build of mimic in the folder has_mimic='' if [[ -f ${TOP}/mimic/bin/mimic ]] ; then - has_mimic=$(${TOP}/mimic/bin/mimic -lv | grep Voice) || true + has_mimic=$("${TOP}"/mimic/bin/mimic -lv | grep Voice) || true fi # in not, check the system path @@ -506,6 +556,7 @@ if [[ ! -x ${VIRTUALENV_ROOT}/bin/activate ]] ; then fi # Start the virtual environment +# shellcheck source=/dev/null source "${VIRTUALENV_ROOT}/bin/activate" cd "$TOP" @@ -514,7 +565,7 @@ HOOK_FILE='./.git/hooks/pre-commit' if [[ -n $INSTALL_PRECOMMIT_HOOK ]] || grep -q 'MYCROFT DEV SETUP' $HOOK_FILE; then if [[ ! -f $HOOK_FILE ]] || grep -q 'MYCROFT DEV SETUP' $HOOK_FILE; then echo 'Installing PEP8 check as precommit-hook' - echo "#! $(which python)" > $HOOK_FILE + echo "#! $(command -v python)" > $HOOK_FILE echo '# MYCROFT DEV SETUP' >> $HOOK_FILE cat ./scripts/pre-commit >> $HOOK_FILE chmod +x $HOOK_FILE @@ -532,17 +583,15 @@ if [[ ! -f $VENV_PATH_FILE ]] ; then echo "import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)" >> "$VENV_PATH_FILE" || return 1 fi -if ! grep -q "$TOP" $VENV_PATH_FILE ; then +if ! grep -q "$TOP" "$VENV_PATH_FILE" ; then echo 'Adding mycroft-core to virtualenv path' - sed -i.tmp '1 a\ -'"$TOP"' -' "$VENV_PATH_FILE" + sed -i.tmp "1 a$TOP" "$VENV_PATH_FILE" fi # install required python modules if ! pip install -r requirements/requirements.txt ; then echo 'Warning: Failed to install required dependencies. Continue? y/N' - read -n1 continue + read -rn1 continue if [[ $continue != 'y' ]] ; then exit 1 fi @@ -553,7 +602,7 @@ if [[ ! $(pip install -r requirements/extra-audiobackend.txt) || ! $(pip install -r requirements/extra-stt.txt) || ! $(pip install -r requirements/extra-mark1.txt) ]] ; then echo 'Warning: Failed to install some optional dependencies. Continue? y/N' - read -n1 continue + read -rn1 continue if [[ $continue != 'y' ]] ; then exit 1 fi @@ -565,7 +614,7 @@ if ! pip install -r requirements/tests.txt ; then fi SYSMEM=$(free | awk '/^Mem:/ { print $2 }') -MAXCORES=$(($SYSMEM / 2202010)) +MAXCORES=$((SYSMEM / 2202010)) MINCORES=1 CORES=$(nproc) @@ -590,7 +639,7 @@ cd "$TOP" if [[ $build_mimic == 'y' || $build_mimic == 'Y' ]] ; then echo 'WARNING: The following can take a long time to run!' - "${TOP}/scripts/install-mimic.sh" " $CORES" + "${TOP}/scripts/install-mimic.sh" "$CORES" else echo 'Skipping mimic build.' fi diff --git a/mycroft/__init__.py b/mycroft/__init__.py index 90e3494c3e03..9a6e4581fedc 100644 --- a/mycroft/__init__.py +++ b/mycroft/__init__.py @@ -20,6 +20,7 @@ from mycroft.skills import (MycroftSkill, FallbackSkill, intent_handler, intent_file_handler) from mycroft.skills.intent_service import AdaptIntent +from mycroft.util.log import LOG MYCROFT_ROOT_PATH = abspath(join(dirname(__file__), '..')) @@ -33,3 +34,5 @@ 'intent_handler', 'intent_file_handler', 'AdaptIntent'] + +LOG.init() # read log level from config diff --git a/mycroft/api/__init__.py b/mycroft/api/__init__.py index 2e74268bebc1..9da1ca0b5803 100644 --- a/mycroft/api/__init__.py +++ b/mycroft/api/__init__.py @@ -20,8 +20,6 @@ from requests import HTTPError, RequestException from mycroft.configuration import Configuration -from mycroft.configuration.config import DEFAULT_CONFIG, SYSTEM_CONFIG, \ - USER_CONFIG from mycroft.identity import IdentityManager, identity_lock from mycroft.version import VersionManager from mycroft.util import get_arch, connected, LOG @@ -49,12 +47,9 @@ class Api: def __init__(self, path): self.path = path - # Load the config, skipping the REMOTE_CONFIG since we are + # Load the config, skipping the remote config since we are # getting the info needed to get to it! - config = Configuration.get([DEFAULT_CONFIG, - SYSTEM_CONFIG, - USER_CONFIG], - cache=False) + config = Configuration.get(cache=False, remote=False) config_server = config.get("server") self.url = config_server.get("url") self.version = config_server.get("version") @@ -239,9 +234,7 @@ def activate(self, state, token): platform_build = "" # load just the local configs to get platform info - config = Configuration.get([SYSTEM_CONFIG, - USER_CONFIG], - cache=False) + config = Configuration.get(cache=False, remote=False) if "enclosure" in config: platform = config.get("enclosure").get("platform", "unknown") platform_build = config.get("enclosure").get("platform_build", "") @@ -263,9 +256,7 @@ def update_version(self): platform_build = "" # load just the local configs to get platform info - config = Configuration.get([SYSTEM_CONFIG, - USER_CONFIG], - cache=False) + config = Configuration.get(cache=False, remote=False) if "enclosure" in config: platform = config.get("enclosure").get("platform", "unknown") platform_build = config.get("enclosure").get("platform_build", "") diff --git a/mycroft/audio/audioservice.py b/mycroft/audio/audioservice.py index 09ff37d5c5fb..2ad9d20556dc 100644 --- a/mycroft/audio/audioservice.py +++ b/mycroft/audio/audioservice.py @@ -111,6 +111,7 @@ def setup_service(service_module, config, bus): except Exception as e: LOG.error('Failed to load service. ' + repr(e)) else: + LOG.error('Failed to load service. loading function not found') return None @@ -160,9 +161,10 @@ def load_plugins(config, bus): List of started services """ plugin_services = [] - plugins = find_plugins('mycroft.plugin.audioservice') - for plug in plugins: - service = setup_service(plug, config, bus) + found_plugins = find_plugins('mycroft.plugin.audioservice') + for plugin_name, plugin_module in found_plugins.items(): + LOG.info(f'Loading audio service plugin: {plugin_name}') + service = setup_service(plugin_module, config, bus) if service: plugin_services += service return plugin_services diff --git a/mycroft/audio/services/chromecast/__init__.py b/mycroft/audio/services/chromecast/__init__.py deleted file mode 100644 index aa5818035166..000000000000 --- a/mycroft/audio/services/chromecast/__init__.py +++ /dev/null @@ -1,168 +0,0 @@ -# Copyright 2017 Mycroft AI Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -import time -from mimetypes import guess_type - -import pychromecast - -from mycroft.audio.services import RemoteAudioBackend -from mycroft.messagebus.message import Message -from mycroft.util.log import LOG - - -class ChromecastService(RemoteAudioBackend): - """ - Audio backend for playback on chromecast. Using the default media - playback controller included in pychromecast. - """ - def _connect(self, message): - LOG.info('Trying to connect to chromecast') - casts = pychromecast.get_chromecasts() - if self.config is None or 'identifier' not in self.config: - LOG.error("Chromecast identifier not found!") - return # Can't connect since no id is specified - else: - identifier = self.config['identifier'] - for c in casts: - if c.name == identifier: - self.cast = c - break - else: - LOG.info('Couldn\'t find chromecast ' + identifier) - self.connection_attempts += 1 - time.sleep(10) - self.bus.emit(Message('ChromecastServiceConnect')) - return - - def __init__(self, config, bus, name='chromecast', cast=None): - super(ChromecastService, self).__init__(config, bus) - self.connection_attempts = 0 - self.bus = bus - self.config = config - self.name = name - - self.tracklist = [] - - if cast is not None: - self.cast = cast - else: - self.cast = None - self.bus.on('ChromecastServiceConnect', self._connect) - self.bus.emit(Message('ChromecastServiceConnect')) - - def supported_uris(self): - """ Return supported uris of chromecast. """ - LOG.info("Chromecasts found: " + str(self.cast)) - if self.cast: - return ['http', 'https'] - else: - return [] - - def clear_list(self): - """ Clear tracklist. """ - self.tracklist = [] - - def add_list(self, tracks): - """ - Add list of tracks to chromecast playlist. - - Args: - tracks (list): list media to add to playlist. - """ - self.tracklist = tracks - pass - - def play(self, repeat=False): - """ Start playback. - - TODO: add playlist support and repeat - """ - self.cast.wait() # Make sure the device is ready to receive command - self.cast.quit_app() - while self.cast.status.status_text != '': - time.sleep(1) - - track = self.tracklist[0] - # Report start of playback to audioservice - if self._track_start_callback: - self._track_start_callback(track) - LOG.debug('track: {}, type: {}'.format(track, guess_type(track)[0])) - mime = guess_type(track)[0] or 'audio/mp3' - self.cast.wait() # Make sure the device is ready to receive command - self.cast.play_media(track, mime) - - def stop(self): - """ Stop playback and quit app. """ - if self.cast.media_controller.is_playing: - self.cast.media_controller.stop() - self.cast.quit_app() - return True - else: - return False - - def pause(self): - """ Pause current playback. """ - if not self.cast.media_controller.is_paused: - self.cast.media_controller.pause() - - def resume(self): - if self.cast.media_controller.is_paused: - self.cast.media_controller.play() - - def next(self): - """ Skip current track. (Not implemented) """ - pass - - def previous(self): - """ Return to previous track. (Not implemented) """ - pass - - def lower_volume(self): - # self.cast.volume_down() - pass - - def restore_volume(self): - # self.cast.volume_up() - pass - - def track_info(self): - """ Return info about currently playing track. """ - info = {} - ret = {} - ret['name'] = info.get('name', '') - if 'album' in info: - ret['artist'] = info['album']['artists'][0]['name'] - ret['album'] = info['album'].get('name', '') - else: - ret['artist'] = '' - ret['album'] = '' - return ret - - def shutdown(self): - """ Disconnect from the device. """ - self.cast.disconnect() - - -def autodetect(config, bus): - """ - Autodetect chromecasts on the network and create backends for each - """ - casts = pychromecast.get_chromecasts(timeout=5, tries=2, retry_wait=2) - ret = [] - for c in casts: - LOG.info(c.name + " found.") - ret.append(ChromecastService(config, bus, c.name.lower(), c)) - - return ret diff --git a/mycroft/audio/services/mplayer/__init__.py b/mycroft/audio/services/mplayer/__init__.py deleted file mode 100644 index 1da8a902f677..000000000000 --- a/mycroft/audio/services/mplayer/__init__.py +++ /dev/null @@ -1,130 +0,0 @@ -# Copyright 2017 Mycroft AI Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -from mycroft.audio.services import AudioBackend -from mycroft.util.log import LOG -try: - from py_mplayer import MplayerCtrl -except ImportError: - LOG.error("install py_mplayer with " - "pip install git+https://github.com/JarbasAl/py_mplayer") - raise - - -class MPlayerService(AudioBackend): - """ - Audio backend for mplayer. - """ - - def __init__(self, config, bus, name='mplayer'): - super(MPlayerService, self).__init__(config, bus) - self.config = config - self.bus = bus - self.name = name - self.index = 0 - self.normal_volume = None - self.tracks = [] - self.mpc = MplayerCtrl() - - def supported_uris(self): - return ['file', 'http', 'https'] - - def clear_list(self): - self.tracks = [] - - def add_list(self, tracks): - self.tracks += tracks - LOG.info("Track list is " + str(tracks)) - - def play(self, repeat=False): - """ Start playback of playlist. - - TODO: Add support for repeat - """ - self.stop() - if len(self.tracks): - # play first track - self.mpc.loadfile(self.tracks[0]) - # add other tracks - for track in self.tracks[1:]: - self.mpc.loadfile(track, 1) - - def stop(self): - self.mpc.stop() - return True # TODO: Return False if not playing - - def pause(self): - if not self.mpc.paused: - self.mpc.pause() - - def resume(self): - if self.mpc.paused: - self.mpc.pause() - - def next(self): - self.index = self.index + 1 - if self.index > len(self.tracks): - self.index = 0 - self.play() - - def previous(self): - self.index = self.index - 1 - if self.index < 0: - self.index = 0 - self.play() - - def lower_volume(self): - if self.normal_volume is None: - self.normal_volume = self.mpc.volume - self.mpc.volume = self.mpc.volume / 3 - - def restore_volume(self): - if self.normal_volume: - self.mpc.volume = self.normal_volume - else: - self.mpc.volume = 50 - self.normal_volume = None - - def track_info(self): - """ - Fetch info about current playing track. - - Returns: - Dict with track info. - """ - ret = {} - ret['title'] = self.mpc.get_meta_title() - ret['artist'] = self.mpc.get_meta_artist() - ret['album'] = self.mpc.get_meta_album() - ret['genre'] = self.mpc.get_meta_genre() - ret['year'] = self.mpc.get_meta_year() - ret['track'] = self.mpc.get_meta_track() - ret['comment'] = self.mpc.get_meta_comment() - return ret - - def shutdown(self): - """ - Shutdown mplayer - - """ - self.mpc.destroy() - - -def load_service(base_config, emitter): - backends = base_config.get('backends', []) - services = [(b, backends[b]) for b in backends - if backends[b]['type'] == 'mplayer' and - backends[b].get("active", True)] - instances = [MPlayerService(s[1], emitter, s[0]) for s in services] - return instances diff --git a/mycroft/audio/speech.py b/mycroft/audio/speech.py index 9f40dda859d0..a5f6a645ec3d 100644 --- a/mycroft/audio/speech.py +++ b/mycroft/audio/speech.py @@ -75,13 +75,7 @@ def handle_speak(event): # so we likely will want to get rid of this when not running on Mimic if (config.get('enclosure', {}).get('platform') != "picroft" and len(re.findall('<[^>]*>', utterance)) == 0): - # Remove any whitespace present after the period, - # if a character (only alpha) ends with a period - # ex: A. Lincoln -> A.Lincoln - # so that we don't split at the period - utterance = re.sub(r'\b([A-za-z][\.])(\s+)', r'\g<1>', utterance) - chunks = re.split(r'(? 0 ? true : false - Component.onCompleted: { - console.log(hasTitle) - } + contentItem: Rectangle { + color: "transparent" - contentItem: ColumnLayout { - Label { - id: systemTextFrameTitle - Layout.fillWidth: true - font.pixelSize: Math.min(systemTextFrame.height/4, Math.max(systemTextFrame.height/10, systemTextFrameMainBody.fontInfo.pixelSize * 1.4)) - wrapMode: Text.Wrap - horizontalAlignment: Text.AlignHCenter - visible: hasTitle - enabled: hasTitle - font.family: "Noto Sans" - font.weight: Font.Bold - text: sessionData.title - } - - Mycroft.AutoFitLabel { - id: systemTextFrameMainBody - Layout.fillWidth: true - Layout.fillHeight: true - wrapMode: Text.Wrap - font.family: "Noto Sans" - font.weight: Font.Bold - text: sessionData.text + ColumnLayout { + anchors.fill: parent + + Mycroft.AutoFitLabel { + id: systemTextFrameTitle + wrapMode: Text.Wrap + visible: hasTitle + enabled: hasTitle + Layout.fillWidth: true + Layout.fillHeight: true + font.family: "Noto Sans" + font.weight: Font.Bold + text: sessionData.title + } + + Mycroft.AutoFitLabel { + id: systemTextFrameMainBody + wrapMode: Text.Wrap + font.family: "Noto Sans" + Layout.fillWidth: true + Layout.fillHeight: true + font.weight: Font.Bold + text: sessionData.text + } } } } diff --git a/mycroft/res/ui/WebViewHtmlFrame.qml b/mycroft/res/ui/WebViewHtmlFrame.qml index ec080afb2de5..25a57d24023d 100644 --- a/mycroft/res/ui/WebViewHtmlFrame.qml +++ b/mycroft/res/ui/WebViewHtmlFrame.qml @@ -19,6 +19,12 @@ Item { } } + RequestHandler { + id: interactionBar + anchors.top: parent.top + z: 1001 + } + WebEngineView { id: webview anchors.fill: parent @@ -47,6 +53,17 @@ Item { onJavaScriptDialogRequested: function(request) { request.accepted = true; } + + onFeaturePermissionRequested: { + interactionBar.setSource("FeatureRequest.qml") + interactionBar.interactionItem.securityOrigin = securityOrigin; + interactionBar.interactionItem.requestedFeature = feature; + interactionBar.isRequested = true; + } + + onFullScreenRequested: { + request.accept() + } } Popup { diff --git a/mycroft/res/ui/WebViewUrlFrame.qml b/mycroft/res/ui/WebViewUrlFrame.qml index c87d33f4add4..db60ae12b330 100644 --- a/mycroft/res/ui/WebViewUrlFrame.qml +++ b/mycroft/res/ui/WebViewUrlFrame.qml @@ -14,6 +14,12 @@ Item { } } + RequestHandler { + id: interactionBar + anchors.top: parent.top + z: 1001 + } + WebEngineView { id: webview anchors.fill: parent @@ -42,6 +48,17 @@ Item { onJavaScriptDialogRequested: function(request) { request.accepted = true; } + + onFeaturePermissionRequested: { + interactionBar.setSource("FeatureRequest.qml") + interactionBar.interactionItem.securityOrigin = securityOrigin; + interactionBar.interactionItem.requestedFeature = feature; + interactionBar.isRequested = true; + } + + onFullScreenRequested: { + request.accept() + } } Popup { diff --git a/mycroft/skills/__init__.py b/mycroft/skills/__init__.py index 2deb9d2e4b3a..c6617da7b92e 100644 --- a/mycroft/skills/__init__.py +++ b/mycroft/skills/__init__.py @@ -20,6 +20,7 @@ from .mycroft_skill import (MycroftSkill, intent_handler, intent_file_handler, resting_screen_handler, skill_api_method) +from .intent_service import AdaptIntent from .fallback_skill import FallbackSkill from .common_iot_skill import CommonIoTSkill from .common_play_skill import CommonPlaySkill, CPSMatchLevel diff --git a/mycroft/skills/common_query_skill.py b/mycroft/skills/common_query_skill.py index 4bb92d6519d0..3e77d6fb60a9 100644 --- a/mycroft/skills/common_query_skill.py +++ b/mycroft/skills/common_query_skill.py @@ -11,11 +11,12 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - from enum import IntEnum from abc import ABC, abstractmethod from .mycroft_skill import MycroftSkill +from mycroft.util.file_utils import resolve_resource_file + class CQSMatchLevel(IntEnum): EXACT = 1 # Skill could find a specific answer for the question @@ -32,11 +33,19 @@ def is_CQSVisualMatchLevel(match_level): return isinstance(match_level, type(CQSVisualMatchLevel.EXACT)) -VISUAL_DEVICES = ['mycroft_mark_2'] +"""these are for the confidence calculation""" +# how much each topic word is worth +# when found in the answer +TOPIC_MATCH_RELEVANCE = 5 +# elevate relevance above all else +RELEVANCE_MULTIPLIER = 2 -def handles_visuals(platform): - return platform in VISUAL_DEVICES +# we like longer articles but only so much +MAX_ANSWER_LEN_FOR_CONFIDENCE = 50 + +# higher number - less bias for word length +WORD_COUNT_DIVISOR = 100 class CommonQuerySkill(MycroftSkill, ABC): @@ -49,8 +58,29 @@ class CommonQuerySkill(MycroftSkill, ABC): This class works in conjunction with skill-query which collects answers from several skills presenting the best one available. """ + def __init__(self, name=None, bus=None): super().__init__(name, bus) + noise_words_filepath = "text/%s/noise_words.list" % (self.lang,) + noise_words_filename = resolve_resource_file(noise_words_filepath) + self.translated_noise_words = [] + try: + if noise_words_filename: + with open(noise_words_filename) as f: + read_noise_words = f.read().strip() + self.translated_noise_words = read_noise_words.split() + else: + raise FileNotFoundError + except FileNotFoundError: + self.log.warning("Missing noise_words.list file in " + f"res/text/{self.lang}") + + # these should probably be configurable + self.level_confidence = { + CQSMatchLevel.EXACT: 0.9, + CQSMatchLevel.CATEGORY: 0.6, + CQSMatchLevel.GENERAL: 0.5 + } def bind(self, bus): """Overrides the default bind method of MycroftSkill. @@ -80,7 +110,8 @@ def __handle_question_query(self, message): level = result[1] answer = result[2] callback = result[3] if len(result) > 3 else None - confidence = self.__calc_confidence(match, search_phrase, level) + confidence = self.__calc_confidence( + match, search_phrase, level, answer) self.bus.emit(message.response({"phrase": search_phrase, "skill_id": self.skill_id, "answer": answer, @@ -92,27 +123,57 @@ def __handle_question_query(self, message): "skill_id": self.skill_id, "searching": False})) - def __calc_confidence(self, match, phrase, level): + def remove_noise(self, phrase): + """remove noise to produce essence of question""" + phrase = ' ' + phrase + ' ' + for word in self.translated_noise_words: + mtch = ' ' + word + ' ' + if phrase.find(mtch) > -1: + phrase = phrase.replace(mtch, " ") + phrase = ' '.join(phrase.split()) + return phrase.strip() + + def __calc_confidence(self, match, phrase, level, answer): # Assume the more of the words that get consumed, the better the match consumed_pct = len(match.split()) / len(phrase.split()) if consumed_pct > 1.0: consumed_pct = 1.0 + consumed_pct /= 10 + + # bonus for more sentences + num_sentences = float(float(len(answer.split("."))) / float(10)) # Add bonus if match has visuals and the device supports them. - platform = self.config_core.get('enclosure', {}).get('platform') - if is_CQSVisualMatchLevel(level) and handles_visuals(platform): + bonus = 0.0 + if is_CQSVisualMatchLevel(level) and self.gui.connected: bonus = 0.1 - else: - bonus = 0 - - if int(level) == int(CQSMatchLevel.EXACT): - return 0.9 + (consumed_pct / 10) + bonus - elif int(level) == int(CQSMatchLevel.CATEGORY): - return 0.6 + (consumed_pct / 10) + bonus - elif int(level) == int(CQSMatchLevel.GENERAL): - return 0.5 + (consumed_pct / 10) + bonus - else: - return 0.0 # should never happen + + # extract topic + topic = self.remove_noise(match) + + # calculate relevance + answer = answer.lower() + matches = 0 + for word in topic.split(' '): + if answer.find(word) > -1: + matches += TOPIC_MATCH_RELEVANCE + + answer_size = len(answer.split(" ")) + answer_size = min(MAX_ANSWER_LEN_FOR_CONFIDENCE, answer_size) + + relevance = 0.0 + if answer_size > 0: + relevance = float(float(matches) / float(answer_size)) + + relevance = relevance * RELEVANCE_MULTIPLIER + + # extra credit for more words up to a point + wc_mod = float(float(answer_size) / float(WORD_COUNT_DIVISOR)) * 2 + + confidence = self.level_confidence[level] + \ + consumed_pct + bonus + num_sentences + relevance + wc_mod + + return confidence def __handle_query_action(self, message): """Message handler for question:action. diff --git a/mycroft/skills/event_scheduler.py b/mycroft/skills/event_scheduler.py index cc91a76c43ad..79f29de24ead 100644 --- a/mycroft/skills/event_scheduler.py +++ b/mycroft/skills/event_scheduler.py @@ -16,10 +16,12 @@ times. """ import json +import shutil import time from datetime import datetime, timedelta from threading import Thread, Lock from os.path import isfile, join, expanduser +import xdg.BaseDirectory from mycroft.configuration import Configuration from mycroft.messagebus.message import Message @@ -54,14 +56,20 @@ class EventScheduler(Thread): """ def __init__(self, bus, schedule_file='schedule.json'): super().__init__() - data_dir = expanduser(Configuration.get()['data_dir']) self.events = {} self.event_lock = Lock() self.bus = bus self.is_running = True - self.schedule_file = join(data_dir, schedule_file) + old_schedule_path = join(expanduser(Configuration.get()['data_dir']), + schedule_file) + new_schedule_path = join( + xdg.BaseDirectory.load_first_config('mycroft'), schedule_file + ) + if isfile(old_schedule_path): + shutil.move(old_schedule_path, new_schedule_path) + self.schedule_file = new_schedule_path if self.schedule_file: self.load() diff --git a/mycroft/skills/intent_service.py b/mycroft/skills/intent_service.py index d7d4c210f9cf..9239f3b8fb25 100644 --- a/mycroft/skills/intent_service.py +++ b/mycroft/skills/intent_service.py @@ -21,7 +21,10 @@ from mycroft.util.parse import normalize from mycroft.metrics import report_timing, Stopwatch from .intent_services import ( - AdaptService, AdaptIntent, FallbackService, PadatiousService, IntentMatch + AdaptService, AdaptIntent, + FallbackService, + PadatiousService, PadatiousMatcher, + IntentMatch ) from .intent_service_interface import open_intent_envelope @@ -280,13 +283,16 @@ def handle_utterance(self, message): stopwatch = Stopwatch() + # Create matchers + padatious_matcher = PadatiousMatcher(self.padatious_service) + # List of functions to use to match the utterance with intent. # These are listed in priority order. match_funcs = [ - self._converse, self.padatious_service.match_high, + self._converse, padatious_matcher.match_high, self.adapt_service.match_intent, self.fallback.high_prio, - self.padatious_service.match_medium, self.fallback.medium_prio, - self.padatious_service.match_low, self.fallback.low_prio + padatious_matcher.match_medium, self.fallback.medium_prio, + padatious_matcher.match_low, self.fallback.low_prio ] match = None @@ -305,6 +311,10 @@ def handle_utterance(self, message): # Launch skill if not handled by the match function if match.intent_type: reply = message.reply(match.intent_type, match.intent_data) + # Add back original list of utterances for intent handlers + # match.intent_data only includes the utterance with the + # highest confidence. + reply.data["utterances"] = utterances self.bus.emit(reply) else: @@ -354,12 +364,18 @@ def handle_register_vocab(self, message): Args: message (Message): message containing vocab info """ - start_concept = message.data.get('start') - end_concept = message.data.get('end') + # TODO: 22.02 Remove backwards compatibility + if _is_old_style_keyword_message(message): + LOG.warning('Deprecated: Registering keywords with old message. ' + 'This will be removed in v22.02.') + _update_keyword_message(message) + + entity_value = message.data.get('entity_value') + entity_type = message.data.get('entity_type') regex_str = message.data.get('regex') alias_of = message.data.get('alias_of') - self.adapt_service.register_vocab(start_concept, end_concept, - alias_of, regex_str) + self.adapt_service.register_vocabulary(entity_value, entity_type, + alias_of, regex_str) self.registered_vocab.append(message.data) def handle_register_intent(self, message): @@ -434,17 +450,20 @@ def handle_get_intent(self, message): lang = message.data.get("lang", "en-us") combined = _normalize_all_utterances([utterance]) + # Create matchers + padatious_matcher = PadatiousMatcher(self.padatious_service) + # List of functions to use to match the utterance with intent. # These are listed in priority order. # TODO once we have a mechanism for checking if a fallback will # trigger without actually triggering it, those should be added here match_funcs = [ - self.padatious_service.match_high, + padatious_matcher.match_high, self.adapt_service.match_intent, # self.fallback.high_prio, - self.padatious_service.match_medium, + padatious_matcher.match_medium, # self.fallback.medium_prio, - self.padatious_service.match_low, + padatious_matcher.match_low, # self.fallback.low_prio ] # Loop through the matching functions until a match is found. @@ -550,3 +569,29 @@ def handle_entity_manifest(self, message): self.bus.emit(message.reply( "intent.service.padatious.entities.manifest", {"entities": self.padatious_service.registered_entities})) + + +def _is_old_style_keyword_message(message): + """Simple check that the message is not using the updated format. + + TODO: Remove in v22.02 + + Args: + message (Message): Message object to check + + Returns: + (bool) True if this is an old messagem, else False + """ + return ('entity_value' not in message.data and 'start' in message.data) + + +def _update_keyword_message(message): + """Make old style keyword registration message compatible. + + Copies old keys in message data to new names. + + Args: + message (Message): Message to update + """ + message.data['entity_value'] = message.data['start'] + message.data['entity_type'] = message.data['end'] diff --git a/mycroft/skills/intent_service_interface.py b/mycroft/skills/intent_service_interface.py index 6fe69aadcf60..89a2f292be93 100644 --- a/mycroft/skills/intent_service_interface.py +++ b/mycroft/skills/intent_service_interface.py @@ -44,15 +44,27 @@ def register_adapt_keyword(self, vocab_type, entity, aliases=None): vocab_type(str): Keyword reference entity (str): Primary keyword - aliases (list): List of alternative kewords + aliases (list): List of alternative keywords """ + # TODO 22.02: Remove compatibility data aliases = aliases or [] - self.bus.emit(Message("register_vocab", - {'start': entity, 'end': vocab_type})) + entity_data = {'entity_value': entity, 'entity_type': vocab_type} + compatibility_data = {'start': entity, 'end': vocab_type} + + self.bus.emit( + Message("register_vocab", + {**entity_data, **compatibility_data}) + ) for alias in aliases: - self.bus.emit(Message("register_vocab", { - 'start': alias, 'end': vocab_type, 'alias_of': entity - })) + alias_data = { + 'entity_value': alias, + 'entity_type': vocab_type, + 'alias_of': entity} + compatibility_data = {'start': alias, 'end': vocab_type} + self.bus.emit( + Message("register_vocab", + {**alias_data, **compatibility_data}) + ) def register_adapt_regex(self, regex): """Register a regex with the intent service. diff --git a/mycroft/skills/intent_services/__init__.py b/mycroft/skills/intent_services/__init__.py index 46463f5c19a6..8603aad314d4 100644 --- a/mycroft/skills/intent_services/__init__.py +++ b/mycroft/skills/intent_services/__init__.py @@ -1,4 +1,4 @@ from .adapt_service import AdaptService, AdaptIntent from .base import IntentMatch from .fallback_service import FallbackService -from .padatious_service import PadatiousService +from .padatious_service import PadatiousService, PadatiousMatcher diff --git a/mycroft/skills/intent_services/adapt_service.py b/mycroft/skills/intent_services/adapt_service.py index 4148efbb21ec..a1e46115833d 100644 --- a/mycroft/skills/intent_services/adapt_service.py +++ b/mycroft/skills/intent_services/adapt_service.py @@ -202,8 +202,12 @@ def match_intent(self, utterances, _=None, __=None): """Run the Adapt engine to search for an matching intent. Args: - utterances (iterable): iterable of utterances, expected order - [raw, normalized, other] + utterances (iterable): utterances for consideration in intent + matching. As a practical matter, a single utterance will be + passed in most cases. But there are instances, such as + streaming STT that could pass multiple. Each utterance + is represented as a tuple containing the raw, normalized, and + possibly other variations of the utterance. Returns: Intent structure, or None if no match was found. @@ -227,7 +231,10 @@ def take_best(intent, utt): include_tags=True, context_manager=self.context_manager)] if intents: - take_best(intents[0], utt_tup[0]) + utt_best = max( + intents, key=lambda x: x.get('confidence', 0.0) + ) + take_best(utt_best, utt_tup[0]) except Exception as err: LOG.exception(err) @@ -242,14 +249,35 @@ def take_best(intent, utt): ret = None return ret + # TODO 22.02: Remove this deprecated method def register_vocab(self, start_concept, end_concept, alias_of, regex_str): - """Register vocabulary.""" + """Register Vocabulary. DEPRECATED + + This method should not be used, it has been replaced by + register_vocabulary(). + """ + self.register_vocabulary(start_concept, end_concept, + alias_of, regex_str) + + def register_vocabulary(self, entity_value, entity_type, + alias_of, regex_str): + """Register skill vocabulary as adapt entity. + + This will handle both regex registration and registration of normal + keywords. if the "regex_str" argument is set all other arguments will + be ignored. + + Argument: + entity_value: the natural langauge word + entity_type: the type/tag of an entity instance + alias_of: entity this is an alternative for + """ with self.lock: if regex_str: self.engine.register_regex_entity(regex_str) else: self.engine.register_entity( - start_concept, end_concept, alias_of=alias_of) + entity_value, entity_type, alias_of=alias_of) def register_intent(self, intent): """Register new intent with adapt engine. diff --git a/mycroft/skills/intent_services/padatious_service.py b/mycroft/skills/intent_services/padatious_service.py index 36c527f361ed..e45ce677b70e 100644 --- a/mycroft/skills/intent_services/padatious_service.py +++ b/mycroft/skills/intent_services/padatious_service.py @@ -26,6 +26,76 @@ from .base import IntentMatch +class PadatiousMatcher: + """Matcher class to avoid redundancy in padatious intent matching.""" + def __init__(self, service): + self.service = service + self.has_result = False + self.ret = None + self.conf = None + + def _match_level(self, utterances, limit): + """Match intent and make sure a certain level of confidence is reached. + + Args: + utterances (list of tuples): Utterances to parse, originals paired + with optional normalized version. + limit (float): required confidence level. + """ + if not self.has_result: + padatious_intent = None + LOG.debug('Padatious Matching confidence > {}'.format(limit)) + for utt in utterances: + for variant in utt: + intent = self.service.calc_intent(variant) + if intent: + best = padatious_intent.conf \ + if padatious_intent else 0.0 + if best < intent.conf: + padatious_intent = intent + padatious_intent.matches['utterance'] = utt[0] + + if padatious_intent: + skill_id = padatious_intent.name.split(':')[0] + self.ret = IntentMatch( + 'Padatious', padatious_intent.name, + padatious_intent.matches, skill_id + ) + self.conf = padatious_intent.conf + self.has_result = True + + if self.conf and self.conf > limit: + return self.ret + return None + + def match_high(self, utterances, _=None, __=None): + """Intent matcher for high confidence. + + Args: + utterances (list of tuples): Utterances to parse, originals paired + with optional normalized version. + """ + return self._match_level(utterances, 0.95) + + def match_medium(self, utterances, _=None, __=None): + """Intent matcher for medium confidence. + + Args: + utterances (list of tuples): Utterances to parse, originals paired + with optional normalized version. + """ + return self._match_level(utterances, 0.8) + + def match_low(self, utterances, _=None, __=None): + """Intent matcher for low confidence. + + Args: + utterances (list of tuples): Utterances to parse, originals paired + with optional normalized version. + """ + return self._match_level(utterances, 0.5) + + class PadatiousService: """Service class for padatious intent matching.""" def __init__(self, bus, config): @@ -167,63 +237,6 @@ def register_entity(self, message): self.registered_entities.append(message.data) self._register_object(message, 'entity', self.container.load_entity) - def _match_level(self, utterances, limit): - """Match intent and make sure a certain level of confidence is reached. - - Args: - utterances (list of tuples): Utterances to parse, originals paired - with optional normalized version. - limit (float): required confidence level. - """ - padatious_intent = None - LOG.debug('Padatious Matching confidence > {}'.format(limit)) - for utt in utterances: - for variant in utt: - intent = self.calc_intent(variant) - if intent: - best = padatious_intent.conf if padatious_intent else 0.0 - if best < intent.conf: - padatious_intent = intent - padatious_intent.matches['utterance'] = utt[0] - - if padatious_intent and padatious_intent.conf > limit: - skill_id = padatious_intent.name.split(':')[0] - ret = IntentMatch( - 'Padatious', padatious_intent.name, padatious_intent.matches, - skill_id - ) - else: - ret = None - return ret - - def match_high(self, utterances, _=None, __=None): - """Intent matcher for high confidence. - - Args: - utterances (list of tuples): Utterances to parse, originals paired - with optional normalized version. - """ - return self._match_level(utterances, 0.95) - - def match_medium(self, utterances, _=None, __=None): - """Intent matcher for medium confidence. - - Args: - utterances (list of tuples): Utterances to parse, originals paired - with optional normalized version. - """ - return self._match_level(utterances, 0.8) - - def match_low(self, utterances, _=None, __=None): - """Intent matcher for low confidence. - - Args: - utterances (list of tuples): Utterances to parse, originals paired - with optional normalized version. - """ - return self._match_level(utterances, 0.5) - - @lru_cache(maxsize=2) # 2 catches both raw and normalized utts in cache def calc_intent(self, utt): """Cached version of container calc_intent. diff --git a/mycroft/skills/mycroft_skill/mycroft_skill.py b/mycroft/skills/mycroft_skill/mycroft_skill.py index 4302b49b4ce4..4f331e27b6a5 100644 --- a/mycroft/skills/mycroft_skill/mycroft_skill.py +++ b/mycroft/skills/mycroft_skill/mycroft_skill.py @@ -411,7 +411,16 @@ def converse(self, message=None): return False def __get_response(self): - """Helper to get a reponse from the user + """Helper to get a response from the user + + NOTE: There is a race condition here. There is a small amount of + time between the end of the device speaking and the converse method + being overridden in this method. If an utterance is injected during + this time, the wrong converse method is executed. The condition is + hidden during normal use due to the amount of time it takes a user + to speak a response. The condition is revealed when an automated + process injects an utterance quicker than this method can flip the + converse methods. Returns: str: user's response or None on a timeout @@ -1159,9 +1168,8 @@ def register_vocabulary(self, entity, entity_type): entity: word to register entity_type: Intent handler entity to tie the word to """ - self.bus.emit(Message('register_vocab', { - 'start': entity, 'end': to_alnum(self.skill_id) + entity_type - })) + keyword_type = to_alnum(self.skill_id) + entity_type + self.intent_service.register_adapt_keyword(keyword_type, entity) def register_regex(self, regex_str): """Register a new regex. diff --git a/mycroft/skills/skill_loader.py b/mycroft/skills/skill_loader.py index bcc4517bb12c..f2301b55a7d9 100644 --- a/mycroft/skills/skill_loader.py +++ b/mycroft/skills/skill_loader.py @@ -41,7 +41,7 @@ def remove_submodule_refs(module_name): module_name: name of skill module. """ submodules = [] - LOG.debug('Skill module'.format(module_name)) + LOG.debug('Skill module: {}'.format(module_name)) # Collect found submodules for m in sys.modules: if m.startswith(module_name + '.'): diff --git a/mycroft/skills/skill_updater.py b/mycroft/skills/skill_updater.py index 18852775e432..bdc1fa3d02dc 100644 --- a/mycroft/skills/skill_updater.py +++ b/mycroft/skills/skill_updater.py @@ -17,6 +17,7 @@ import sys from datetime import datetime from time import time +import xdg.BaseDirectory from msm import MsmException @@ -97,9 +98,9 @@ def installed_skills_file_path(self): '.mycroft-skills' ) else: - self._installed_skills_file_path = os.path.expanduser( - '~/.mycroft/.mycroft-skills' - ) + self._installed_skills_file_path = os.path.join( + xdg.BaseDirectory.save_data_path('mycroft'), + '.mycroft-skills') return self._installed_skills_file_path diff --git a/mycroft/stt/__init__.py b/mycroft/stt/__init__.py index 3f2c8404b19b..9f8eea8ffd52 100644 --- a/mycroft/stt/__init__.py +++ b/mycroft/stt/__init__.py @@ -37,6 +37,18 @@ def __init__(self): self.recognizer = Recognizer() self.can_stream = False + @property + def available_languages(self) -> set: + """Return languages supported by this STT implementation in this state + + This property should be overridden by the derived class to advertise + what languages that engine supports. + + Returns: + set: supported languages + """ + return set() + @staticmethod def init_language(config_core): """Helper method to get language code from Mycroft config.""" diff --git a/mycroft/tts/espeak_tts.py b/mycroft/tts/espeak_tts.py index 51bf83e4bc6d..e09fb7620f99 100644 --- a/mycroft/tts/espeak_tts.py +++ b/mycroft/tts/espeak_tts.py @@ -32,8 +32,32 @@ def get_tts(self, sentence, wav_file): Returns: tuple ((str) file location, None) """ - subprocess.call(['espeak', '-v', self.lang + '+' + self.voice, - '-w', wav_file, sentence]) + + # Create Argument String for Espeak + arguments = ['espeak', '-v', self.lang + '+' + self.voice] + amplitude = self.config.get('amplitude') + if amplitude: + arguments.append('-a '+amplitude) + + gap = self.config.get('gap') + if gap: + arguments.append('-g '+gap) + + capital = self.config.get('capital') + if capital: + arguments.append('-k '+capital) + + pitch = self.config.get('pitch') + if pitch: + arguments.append('-p '+pitch) + + speed = self.config.get('speed') + if speed: + arguments.append('-s '+speed) + + arguments.extend(['-w', wav_file, sentence]) + + subprocess.call(arguments) return wav_file, None diff --git a/mycroft/tts/google_tts.py b/mycroft/tts/google_tts.py index be755f8786f1..c47c510313b9 100755 --- a/mycroft/tts/google_tts.py +++ b/mycroft/tts/google_tts.py @@ -70,6 +70,13 @@ def __init__(self, lang, config): self._google_lang = None super(GoogleTTS, self).__init__(lang, config, GoogleTTSValidator( self), 'mp3') + LOG.warning( + "The Google TTS module uses the gTTS Python package which itself " + "interfaces with the Google Translate text-to-speech API. This is " + "not intended for commercial or production usage. The service " + "may break at any time, and you are subject to their Terms of " + "Service that can be found at https://policies.google.com/terms" + ) @property def google_lang(self): diff --git a/mycroft/tts/mimic_tts.py b/mycroft/tts/mimic_tts.py index 68d5ff9bccd4..32e7cd82d286 100644 --- a/mycroft/tts/mimic_tts.py +++ b/mycroft/tts/mimic_tts.py @@ -130,8 +130,6 @@ def __init__(self, lang, config): ) self.default_binary = get_mimic_binary() - self.clear_cache() - # Download subscriber voices if needed self.subscriber_voices = get_subscriber_voices() self.is_subscriber = DeviceApi().is_subscriber diff --git a/mycroft/tts/responsive_voice_tts.py b/mycroft/tts/responsive_voice_tts.py deleted file mode 100644 index 15e3d610bc0a..000000000000 --- a/mycroft/tts/responsive_voice_tts.py +++ /dev/null @@ -1,61 +0,0 @@ -# Copyright 2017 Mycroft AI Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -import requests -from .tts import TTS, TTSValidator - - -class ResponsiveVoice(TTS): - def __init__(self, lang, config): - super(ResponsiveVoice, self).__init__( - lang, config, ResponsiveVoiceValidator(self), 'mp3', - ssml_tags=[] - ) - self.clear_cache() - self.pitch = config.get("pitch", 0.5) - self.rate = config.get("rate", 0.5) - self.vol = config.get("vol", 1) - if "f" not in config.get("gender", "male"): - self.sv = "g1" - self.vn = "rjs" - else: - self.vn = self.sv = "" - - def get_tts(self, sentence, wav_file): - params = {"t": sentence, "tl": self.lang, - "pitch": self.pitch, "rate": self.rate, - "vol": self.vol, "sv": self.sv, "vn": self.vn} - base_url = "http://responsivevoice.org/responsivevoice/getvoice.php" - r = requests.get(base_url, params) - with open(wav_file, "w") as f: - f.write(r.content) - return wav_file, None - - -class ResponsiveVoiceValidator(TTSValidator): - def __init__(self, tts): - super(ResponsiveVoiceValidator, self).__init__(tts) - - def validate_lang(self): - # TODO: Verify responsive voice can handle the requested language - pass - - def validate_connection(self): - r = requests.get("http://responsivevoice.org") - if r.status_code == 200: - return True - raise AssertionError("Could not reach http://responsivevoice.org") - - def get_tts_class(self): - return ResponsiveVoice diff --git a/mycroft/tts/tts.py b/mycroft/tts/tts.py index 8bbe71b73650..2e4934fdc989 100644 --- a/mycroft/tts/tts.py +++ b/mycroft/tts/tts.py @@ -12,14 +12,15 @@ # See the License for the specific language governing permissions and # limitations under the License. # -from cmath import log from copy import deepcopy import os import random import re from abc import ABCMeta, abstractmethod +from pathlib import Path from threading import Thread from time import time +from warnings import warn import os.path from os.path import dirname, exists, isdir, join @@ -38,22 +39,43 @@ from queue import Queue, Empty from .cache import hash_sentence, TextToSpeechCache - _TTS_ENV = deepcopy(os.environ) _TTS_ENV['PULSE_PROP'] = 'media.role=phone' - EMPTY_PLAYBACK_QUEUE_TUPLE = (None, None, None, None, None) +SSML_TAGS = re.compile(r'<[^>]*>') +WHITESPACE_AFTER_PERIOD = re.compile(r'\b([A-za-z][\.])(\s+)') +SENTENCE_DELIMITERS = re.compile( + r'(?', utterance) + chunks = SENTENCE_DELIMITERS.split(utterance) + return chunks + class PlaybackThread(Thread): """Thread class for playing back tts audio and sending viseme data to enclosure. """ - def __init__(self, queue): super(PlaybackThread, self).__init__() self.queue = queue + self.tts = [] + self.bus = None + self._terminated = False self._processing_queue = False self.enclosure = None @@ -65,7 +87,28 @@ def __init__(self, queue): self.pulse_env = None def init(self, tts): - self.tts = tts + """DEPRECATED! Init the TTS Playback thread. + + TODO: 22.02 Remove this + """ + self.attach_tts(tts) + self.set_bus(tts.bus) + + def set_bus(self, bus): + """Provide bus instance to the TTS Playback thread. + + Args: + bus (MycroftBusClient): bus client + """ + self.bus = bus + + def attach_tts(self, tts): + """Add TTS to be cache checked.""" + self.tts.append(tts) + + def detach_tts(self, tts): + """Remove TTS from cache check.""" + self.tts.remove(tts) def clear_queue(self): """Remove all pending playbacks.""" @@ -76,12 +119,6 @@ def clear_queue(self): except Exception: pass - - def sendSpace(self): - LOG.info("Sending the command to the process") - self.p.stdin.write("s".encode("utf-8")) - self.p.stdin.flush() - def run(self): """Thread main loop. Get audio and extra data from queue and play. @@ -95,7 +132,7 @@ def run(self): the loop then wait for the playback process to finish before starting checking the next position in queue. - If the queue is empty the tts.end_audio() is called possibly triggering + If the queue is empty the end_audio() is called possibly triggering listening. """ while not self._terminated: @@ -105,7 +142,7 @@ def run(self): self.blink(0.5) if not self._processing_queue: self._processing_queue = True - self.tts.begin_audio() + self.begin_audio() stopwatch = Stopwatch() with stopwatch: @@ -121,7 +158,7 @@ def run(self): report_timing(ident, 'speech_playback', stopwatch) if self.queue.empty(): - self.tts.end_audio(listen) + self.end_audio(listen) self._processing_queue = False self.blink(0.2) except Empty: @@ -129,9 +166,42 @@ def run(self): except Exception as e: LOG.exception(e) if self._processing_queue: - self.tts.end_audio(listen) + self.end_audio(listen) self._processing_queue = False + def begin_audio(self): + """Perform befining of speech actions.""" + # Create signals informing start of speech + if self.bus: + self.bus.emit(Message("recognizer_loop:audio_output_start")) + else: + LOG.warning("Speech started before bus was attached.") + + def end_audio(self, listen): + """Perform end of speech output actions. + + Will inform the system that speech has ended and trigger the TTS's + cache checks. Listening will be triggered if requested. + + Args: + listen (bool): True if listening event should be emitted + """ + if self.bus: + # Send end of speech signals to the system + self.bus.emit(Message("recognizer_loop:audio_output_end")) + if listen: + self.bus.emit(Message('mycroft.mic.listen')) + + # Clear cache for all attached tts objects + # This is basically the only safe time + for tts in self.tts: + tts.cache.curate() + + # This check will clear the filesystem IPC "signal" + check_for_signal("isSpeaking") + else: + LOG.warning("Speech started before bus was attached.") + def show_visemes(self, pairs): """Send viseme data to enclosure @@ -172,6 +242,8 @@ class TTS(metaclass=ABCMeta): phonetic_spelling (bool): Whether to spell certain words phonetically ssml_tags (list): Supported ssml properties. Ex. ['speak', 'prosody'] """ + queue = None + playback = None def __init__(self, lang, config, validator, audio_ext='wav', phonetic_spelling=True, ssml_tags=None): @@ -188,9 +260,12 @@ def __init__(self, lang, config, validator, audio_ext='wav', self.filename = get_temp_path('tts.wav') self.enclosure = None random.seed() - self.queue = Queue() - self.playback = PlaybackThread(self.queue) - self.playback.start() + + if TTS.queue is None: + TTS.queue = Queue() + TTS.playback = PlaybackThread(TTS.queue) + TTS.playback.start() + self.spellings = self.load_spellings() self.tts_name = type(self).__name__ self.cache = TextToSpeechCache( @@ -198,6 +273,18 @@ def __init__(self, lang, config, validator, audio_ext='wav', ) self.cache.clear() + @property + def available_languages(self) -> set: + """Return languages supported by this TTS implementation in this state + + This property should be overridden by the derived class to advertise + what languages that engine supports. + + Returns: + set: supported languages + """ + return set() + def load_spellings(self): """Load phonetic spellings of words as dictionary.""" path = join('text', self.lang.lower(), 'phonetic_spellings.txt') @@ -245,9 +332,10 @@ def init(self, bus): bus: Mycroft messagebus connection """ self.bus = bus - self.playback.init(self) + TTS.playback.set_bus(bus) + TTS.playback.attach_tts(self) self.enclosure = EnclosureAPI(self.bus) - self.playback.enclosure = self.enclosure + TTS.playback.enclosure = self.enclosure def get_tts(self, sentence, wav_file): """Abstract method that a tts implementation needs to implement. @@ -299,7 +387,7 @@ def validate_ssml(self, utterance): return self.remove_ssml(utterance) # find ssml tags in string - tags = re.findall('<[^>]*>', utterance) + tags = SSML_TAGS.findall(utterance) for tag in tags: if any(supported in tag for supported in self.ssml_tags): @@ -311,6 +399,21 @@ def validate_ssml(self, utterance): # return text with supported ssml tags only return utterance.replace(" ", " ") + def preprocess_utterance(self, utterance): + """Preprocess utterance into list of chunks suitable for the TTS. + + Perform general chunking and TTS specific chunking. + """ + # Remove any whitespace present after the period, + # if a character (only alpha) ends with a period + # ex: A. Lincoln -> A.Lincoln + # so that we don't split at the period + chunks = default_preprocess_utterance(utterance) + result = [] + for chunk in chunks: + result += self._preprocess_sentence(chunk) + return result + def _preprocess_sentence(self, sentence): """Default preprocessing is no preprocessing. @@ -340,13 +443,7 @@ def execute(self, sentence, ident=None, listen=False): sentence = self.validate_ssml(sentence) create_signal("isSpeaking") - try: - self._execute(sentence, ident, listen) - except Exception: - # If an error occurs end the audio sequence through an empty entry - self.queue.put(EMPTY_PLAYBACK_QUEUE_TUPLE) - # Re-raise to allow the Exception to be handled externally as well. - raise + self._execute(sentence, ident, listen) def _execute(self, sentence, ident, listen): if self.phonetic_spelling: @@ -355,6 +452,8 @@ def _execute(self, sentence, ident, listen): sentence = sentence.replace(word, self.spellings[word.lower()]) + # TODO: 22.02 This is no longer needed and can be removed + # Just kept for compatibility for now chunks = self._preprocess_sentence(sentence) # Apply the listen flag to the last chunk, set the rest to False chunks = [(chunks[i], listen if i == len(chunks) - 1 else False) @@ -377,7 +476,20 @@ def _execute(self, sentence, ident, listen): # of the TTS cache. But this requires changing the public # API of the get_tts method in each engine. audio_file = self.cache.define_audio_file(sentence_hash) - _, phonemes = self.get_tts(sentence, str(audio_file.path)) + # TODO 21.08: remove mutation of audio_file.path. + returned_file, phonemes = self.get_tts( + sentence, str(audio_file.path)) + # Convert to Path as needed + returned_file = Path(returned_file) + if returned_file != audio_file.path: + warn( + DeprecationWarning( + f"{self.tts_name} is saving files " + "to a different path than requested. If you are " + "the maintainer of this plugin, please adhere to " + "the file path argument provided. Modified paths " + "will be ignored in a future release.")) + audio_file.path = returned_file if phonemes: phoneme_file = self.cache.define_phoneme_file( sentence_hash @@ -389,7 +501,7 @@ def _execute(self, sentence, ident, listen): audio_file, phoneme_file ) viseme = self.viseme(phonemes) if phonemes else None - self.queue.put( + TTS.queue.put( (self.audio_ext, str(audio_file.path), viseme, ident, l) ) @@ -469,10 +581,6 @@ def load_phonemes(self, key): LOG.debug("Failed to read .PHO from cache") return None - def __del__(self): - self.playback.stop() - self.playback.join() - class TTSValidator(metaclass=ABCMeta): """TTS Validator abstract class to be implemented by all TTS engines. @@ -480,7 +588,6 @@ class TTSValidator(metaclass=ABCMeta): It exposes and implements ``validate(tts)`` function as a template to validate the TTS engines. """ - def __init__(self, tts): self.tts = tts @@ -552,7 +659,6 @@ class TTSFactory: from mycroft.tts.spdsay_tts import SpdSay from mycroft.tts.bing_tts import BingTTS from mycroft.tts.ibm_tts import WatsonTTS - from mycroft.tts.responsive_voice_tts import ResponsiveVoice from mycroft.tts.mimic2_tts import Mimic2 from mycroft.tts.yandex_tts import YandexTTS from mycroft.tts.dummy_tts import DummyTTS @@ -570,7 +676,6 @@ class TTSFactory: "spdsay": SpdSay, "watson": WatsonTTS, "bing": BingTTS, - "responsive_voice": ResponsiveVoice, "yandex": YandexTTS, "polly": PollyTTS, "mozilla": MozillaTTS, diff --git a/mycroft/util/file_utils.py b/mycroft/util/file_utils.py index 279c7d16a3e6..ee441286f5a8 100644 --- a/mycroft/util/file_utils.py +++ b/mycroft/util/file_utils.py @@ -22,6 +22,7 @@ import psutil from stat import S_ISREG, ST_MTIME, ST_MODE, ST_SIZE import tempfile +import xdg.BaseDirectory import mycroft.configuration from .log import LOG @@ -33,15 +34,16 @@ def resolve_resource_file(res_name): Resource names are in the form: 'filename.ext' or 'path/filename.ext' - The system wil look for ~/.mycroft/res_name first, and - if not found will look at /opt/mycroft/res_name, - then finally it will look for res_name in the 'mycroft/res' - folder of the source code package. + The system wil look for $XDG_DATA_DIRS/mycroft/res_name first + (defaults to ~/.local/share/mycroft/res_name), and if not found will + look at /opt/mycroft/res_name, then finally it will look for res_name + in the 'mycroft/res' folder of the source code package. Example: With mycroft running as the user 'bob', if you called ``resolve_resource_file('snd/beep.wav')`` it would return either: + '$XDG_DATA_DIRS/mycroft/beep.wav', '/home/bob/.mycroft/snd/beep.wav' or '/opt/mycroft/snd/beep.wav' or '.../mycroft/res/snd/beep.wav' @@ -60,8 +62,14 @@ def resolve_resource_file(res_name): if os.path.isfile(res_name): return res_name - # Now look for ~/.mycroft/res_name (in user folder) - filename = os.path.expanduser("~/.mycroft/" + res_name) + # Now look for XDG_DATA_DIRS + for conf_dir in xdg.BaseDirectory.load_data_paths('mycroft'): + filename = os.path.join(conf_dir, res_name) + if os.path.isfile(filename): + return filename + + # Now look in the old user location + filename = os.path.join(os.path.expanduser('~'), '.mycroft', res_name) if os.path.isfile(filename): return filename diff --git a/mycroft/util/format.py b/mycroft/util/format.py index b37bf8fa453c..de2e52073a7d 100644 --- a/mycroft/util/format.py +++ b/mycroft/util/format.py @@ -31,6 +31,7 @@ from enum import Enum # These are the main functions we are using lingua franca to provide +from lingua_franca import get_default_loc # TODO 21.08 - move nice_duration methods to Lingua Franca. from lingua_franca.format import ( join_list, @@ -104,6 +105,7 @@ def _duration_handler(time1, lang=None, speech=True, *, time2=None, Returns: str: timespan as a string """ + lang = lang or get_default_loc() _leapdays = 0 _input_resolution = resolution milliseconds = 0 diff --git a/mycroft/util/log.py b/mycroft/util/log.py index 0ea391f8f79c..5f7c9f92b5be 100644 --- a/mycroft/util/log.py +++ b/mycroft/util/log.py @@ -21,7 +21,7 @@ for use. The default log level of the logger created here can ONLY be set in -/etc/mycroft/mycroft.conf or ~/.mycroft/mycroft.conf +/etc/mycroft/mycroft.conf or ~/.config/mycroft/mycroft.conf The default log level can also be programatically be changed by setting the LOG.level parameter. @@ -31,10 +31,7 @@ import logging import sys -from os.path import isfile - -from mycroft.util.json_helper import load_commented_json, merge_dict -from mycroft.configuration.locations import SYSTEM_CONFIG, USER_CONFIG +import mycroft def getLogger(name="MYCROFT"): @@ -65,7 +62,7 @@ class LOG: _custom_name = None handler = None - level = None + level = logging.getLevelName('INFO') # Copy actual logging methods from logging.Logger # Usage: LOG.debug(message) @@ -80,20 +77,6 @@ def init(cls): """ Initializes the class, sets the default log level and creates the required handlers. """ - - # Check configs manually, the Mycroft configuration system can't be - # used since it uses the LOG system and would cause horrible cyclic - # dependencies. - confs = [SYSTEM_CONFIG, USER_CONFIG] - config = {} - for conf in confs: - try: - merge_dict(config, - load_commented_json(conf) if isfile(conf) else {}) - except Exception as e: - print('couldn\'t load {}: {}'.format(conf, str(e))) - - cls.level = logging.getLevelName(config.get('log_level', 'INFO')) log_message_format = ( '{asctime} | {levelname:8} | {process:5} | {name} | {message}' ) @@ -103,6 +86,14 @@ def init(cls): cls.handler = logging.StreamHandler(sys.stdout) cls.handler.setFormatter(formatter) + config = mycroft.configuration.Configuration.get(cache=False, + remote=False) + if config.get('log_format'): + formatter = logging.Formatter(config.get('log_format'), style='{') + cls.handler.setFormatter(formatter) + + cls.level = logging.getLevelName(config.get('log_level', 'INFO')) + # Enable logging in external modules cls.create_logger('').setLevel(cls.level) @@ -144,6 +135,3 @@ def _log(cls, func, *args, **kwargs): name = 'Mycroft' func(cls.create_logger(name), *args, **kwargs) - - -LOG.init() diff --git a/mycroft/util/network_utils.py b/mycroft/util/network_utils.py index 9a06ac4bae7a..d4607aa34f2a 100644 --- a/mycroft/util/network_utils.py +++ b/mycroft/util/network_utils.py @@ -6,6 +6,14 @@ from .log import LOG +def _get_network_tests_config(): + """Get network_tests object from mycroft.configuration.""" + # Wrapped to avoid circular import errors. + from mycroft.configuration import Configuration + config = Configuration.get() + return config.get('network_tests', {}) + + def connected(): """Check connection by connecting to 8.8.8.8 and if google.com is reachable if this fails, Check Microsoft NCSI is used as a backup. @@ -27,16 +35,19 @@ def _connected_ncsi(): Returns: True if internet connection can be detected """ + config = _get_network_tests_config() + ncsi_endpoint = config.get('ncsi_endpoint') + expected_text = config.get('ncsi_expected_text') try: - r = requests.get('http://www.msftncsi.com/ncsi.txt') - if r.text == 'Microsoft NCSI': + r = requests.get(ncsi_endpoint) + if r.text == expected_text: return True except Exception: - pass + LOG.error("Unable to verify connection via NCSI endpoint.") return False -def _connected_dns(host="8.8.8.8", port=53, timeout=3): +def _connected_dns(host=None, port=53, timeout=3): """Check internet connection by connecting to DNS servers Returns: @@ -46,18 +57,25 @@ def _connected_dns(host="8.8.8.8", port=53, timeout=3): # Host: 8.8.8.8 (google-public-dns-a.google.com) # OpenPort: 53/tcp # Service: domain (DNS/TCP) + config = _get_network_tests_config() + if host is None: + host = config.get('dns_primary') try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(timeout) s.connect((host, port)) return True except IOError: + LOG.error("Unable to connect to primary DNS server, " + "trying secondary...") try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(timeout) - s.connect(("8.8.4.4", port)) + dns_secondary = config.get('dns_secondary') + s.connect((dns_secondary, port)) return True except IOError: + LOG.error("Unable to connect to secondary DNS server.") return False @@ -67,10 +85,12 @@ def _connected_google(): True if connection attempt succeeded """ connect_success = False + config = _get_network_tests_config() + url = config.get('web_url') try: - urlopen('https://www.google.com', timeout=3) + urlopen(url, timeout=3) except URLError as ue: - LOG.debug('Attempt to connect to internet failed: ' + str(ue.reason)) + LOG.error('Attempt to connect to internet failed: ' + str(ue.reason)) else: connect_success = True diff --git a/mycroft/util/parse.py b/mycroft/util/parse.py index cbc20c27bcd7..8feaeff513ed 100644 --- a/mycroft/util/parse.py +++ b/mycroft/util/parse.py @@ -30,6 +30,7 @@ from difflib import SequenceMatcher from warnings import warn +from lingua_franca import get_default_loc from lingua_franca.parse import ( extract_duration, extract_number, @@ -115,4 +116,7 @@ def extract_datetime(text, anchorDate="DEFAULT", lang=None, "deprecated. This parameter can be omitted.")) if anchorDate is None or anchorDate == "DEFAULT": anchorDate = now_local() - return _extract_datetime(text, anchorDate, lang, default_time) + return _extract_datetime(text, + anchorDate, + lang or get_default_loc(), + default_time) diff --git a/mycroft/version/__init__.py b/mycroft/version/__init__.py index 4a452e86456e..8c81638eab80 100644 --- a/mycroft/version/__init__.py +++ b/mycroft/version/__init__.py @@ -25,7 +25,7 @@ # START_VERSION_BLOCK CORE_VERSION_MAJOR = 21 CORE_VERSION_MINOR = 2 -CORE_VERSION_BUILD = 0 +CORE_VERSION_BUILD = 2 # END_VERSION_BLOCK CORE_VERSION_TUPLE = (CORE_VERSION_MAJOR, diff --git a/requirements/extra-audiobackend.txt b/requirements/extra-audiobackend.txt index f3d45ae66261..15d298873293 100644 --- a/requirements/extra-audiobackend.txt +++ b/requirements/extra-audiobackend.txt @@ -1,2 +1 @@ -pychromecast==3.2.2 python-vlc==1.1.2 diff --git a/requirements/requirements.txt b/requirements/requirements.txt index bc916114f430..3ec0df4c9263 100644 --- a/requirements/requirements.txt +++ b/requirements/requirements.txt @@ -1,16 +1,15 @@ -requests>=2.20.0,<2.26.0 +requests~=2.26.0 gTTS>=2.2.2,<2.3.0 PyAudio==0.2.11 pyee==8.1.0 SpeechRecognition==3.8.1 -tornado==6.0.3 -websocket-client==0.54.0 +tornado~=6.1 +websocket-client~=1.2.1 requests-futures==0.9.5 pyserial==3.0 psutil==5.6.6 pocketsphinx==0.1.0 -inflection==0.3.1 -pillow==8.2.0 +pillow==8.3.2 python-dateutil==2.6.0 fasteners==0.14.1 PyYAML==5.4 @@ -18,12 +17,11 @@ PyYAML==5.4 lingua-franca==0.4.2 msm==0.8.9 msk==0.3.16 -mycroft-messagebus-client==0.9.1 -adapt-parser==0.4.1 +mycroft-messagebus-client==0.9.4 +adapt-parser==0.5.1 padatious==0.4.8 fann2==1.0.7 padaos==0.1.9 precise-runner==0.2.1 petact==0.1.2 -pyxdg==0.26 -hidapi==0.10.1 \ No newline at end of file +pyxdg==0.27 diff --git a/requirements/tests.txt b/requirements/tests.txt index 9e038123e51a..ca6addcc0d66 100644 --- a/requirements/tests.txt +++ b/requirements/tests.txt @@ -5,7 +5,8 @@ pytest-cov==2.8.1 cov-core==1.15.0 sphinx==2.2.1 sphinx-rtd-theme==0.4.3 -git+https://github.com/behave/behave@v1.2.7.dev1 +https://github.com/krisgesling/tag-expressions-python/tarball/master/ +https://github.com/krisgesling/behave/tarball/master/ allure-behave==2.8.10 python-vlc==1.1.2 diff --git a/scripts/install-mimic.sh b/scripts/install-mimic.sh index 1d13545a28b1..7766064dffef 100755 --- a/scripts/install-mimic.sh +++ b/scripts/install-mimic.sh @@ -31,7 +31,7 @@ if [ ! -d ${MIMIC_DIR} ] ; then cd ${MIMIC_DIR} ./autogen.sh ./configure --with-audio=alsa --enable-shared --prefix="$(pwd)" - make -j${CORES} + make -j"${CORES}" make install else # ensure mimic is up to date @@ -43,6 +43,6 @@ else ./autogen.sh ./configure --with-audio=alsa --enable-shared --prefix="$(pwd)" make clean - make -j${CORES} + make -j"${CORES}" make install fi diff --git a/scripts/install-pocketsphinx.sh b/scripts/install-pocketsphinx.sh deleted file mode 100755 index 44d329b79854..000000000000 --- a/scripts/install-pocketsphinx.sh +++ /dev/null @@ -1,83 +0,0 @@ -#!/usr/bin/env bash - -# Copyright 2017 Mycroft AI Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# exit on any error -set -Ee - -#TOP="." - - -function enable_local() { - sed -i 's/from pocketsphinx.pocketsphinx import Decoder/from pocketsphinx import Decoder/g' mycroft/client/speech/local_recognizer.py -} - -function disable_local() { - sed -i 's/from pocketsphinx import Decoder/from pocketsphinx.pocketsphinx import Decoder/g' mycroft/client/speech/local_recognizer.py -} - -function install_pocketsphinx() { - # clone pocketsphinx-python at HEAD (fix to a constant version later) - if [ ! -d ${TOP}/pocketsphinx-python ] ; then - # build sphinxbase and pocketsphinx if we haven't already - git clone --recursive https://github.com/cmusphinx/pocketsphinx-python - pushd ./pocketsphinx-python/sphinxbase - ./autogen.sh - ./configure - make -j$CORES - popd - pushd ./pocketsphinx-python/pocketsphinx - ./autogen.sh - ./configure - make -j$CORES - popd - fi - - # build and install pocketsphinx python bindings - cd ${TOP}/pocketsphinx-python - python setup.py install -} - -if [ "$1" = "-q" ] ; then - enable_local - install_pocketsphinx - exit 0 -fi - -echo "This script will checkout, compile, and install pocketsphinx locally if the debian package python-pocketsphinx is not available" - -PS3='Please enter your choice: ' -options=("Enable local checkout, compile and install" "Disable local checkout and exit" "Do nothing and quit") -select opt in "${options[@]}" ; do - case $opt in - "Enable local checkout, compile and install") - echo "you chose choice 1" - enable_local - install_pocketsphinx - ;; - "Disable local checkout and exit") - echo "you chose choice 2" - disable_local - exit 0 - ;; - "Do nothing and quit") - echo "you chose choice 3" - exit 0 - ;; - *) - echo invalid option - ;; - esac -done diff --git a/scripts/install-pygtk.sh b/scripts/install-pygtk.sh deleted file mode 100755 index ad6aa301859d..000000000000 --- a/scripts/install-pygtk.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/usr/bin/env bash - -# Copyright 2017 Mycroft AI Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Ensure we're in a virtualenv. -if [ "$VIRTUAL_ENV" == "" ] ; then - echo "ERROR: not in a virtual environment." - exit -1 -fi - -# Setup variables. -CACHE="/tmp/install-pygtk-$$" -CORES=$1 - -# Make temp directory. -mkdir -p $CACHE - -# Test for py2cairo. -echo -e "\E[1m * Checking for cairo...\E[0m" -python -c " -try: import cairo; raise SystemExit(0) -except ImportError: raise SystemExit(-1)" - -if [ $? == 255 ] ; then - echo -e "\E[1m * Installing cairo...\E[0m" - # Fetch, build, and install py2cairo. - ( cd $CACHE - curl 'https://www.cairographics.org/releases/py2cairo-1.10.0.tar.bz2' > "py2cairo.tar.bz2" - tar -xvf py2cairo.tar.bz2 - ( cd py2cairo-* - autoreconf -ivf - ./configure --prefix=$VIRTUAL_ENV --disable-dependency-tracking - make -j${CORES} - make install - ) - ) -fi - -# Test for gobject. -echo -e "\E[1m * Checking for gobject...\E[0m" -python -c " -try: import gobject; raise SystemExit(0) -except ImportError: raise SystemExit(-1)" - -if [ $? == 255 ] ; then - echo -e "\E[1m * Installing gobject...\E[0m" - # Fetch, build, and install gobject. - ( cd $CACHE - curl 'http://ftp.gnome.org/pub/GNOME/sources/pygobject/2.28/pygobject-2.28.6.tar.bz2' > 'pygobject.tar.bz2' - tar -xvf pygobject.tar.bz2 - ( cd pygobject-* - ./configure --prefix=$VIRTUAL_ENV --disable-introspection - make -j${CORES} - make install - ) - ) -fi - -# Test for gtk. -echo -e "\E[1m * Checking for gtk...\E[0m" -python -c " -try: import gtk; raise SystemExit(0) -except ImportError: raise SystemExit(-1)" 2&> /dev/null - -if [ $? == 255 ] ; then - echo -e "\E[1m * Installing gtk...\E[0m" - # Fetch, build, and install gtk. - ( cd $CACHE - curl -L 'https://files.pythonhosted.org/packages/source/P/PyGTK/pygtk-2.24.0.tar.bz2' > 'pygtk.tar.bz2' - tar -xvf pygtk.tar.bz2 - ( cd pygtk-* - ./configure --prefix=$VIRTUAL_ENV PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$VIRTUAL_ENV/lib/pkgconfig - make -j${CORES} - make install - ) - ) -fi diff --git a/scripts/my-info.sh b/scripts/my-info.sh old mode 100644 new mode 100755 index e39541112017..7597556140ae --- a/scripts/my-info.sh +++ b/scripts/my-info.sh @@ -41,7 +41,6 @@ fi MYCROFT_HOME="" RUN_AS_ROOT=1 -source $( locate virtualenvwrapper.sh ) # log stuff and things LOG_FILE=/tmp/my-info.$$.out @@ -56,7 +55,8 @@ fi # it's big, it's heavy, it's wood! function mlog() { - local timestamp="[$( date +"%Y-%m-%d %H:%M:%S" )]" + local timestamp + timestamp="[$( date +"%Y-%m-%d %H:%M:%S" )]" message="$*" echo "${timestamp} ${message}" |tee -a ${LOG_FILE} } @@ -87,18 +87,26 @@ fi echo "If you can read this, we may need glasses." } + +function list_log_files() { + for logfile in "/var/log/mycroft/"*.log; do + echo "$logfile" + done +} + + # Check before we wreck: function checkfiles() { mlog "Permission checks..." cat << EOF > /tmp/my-list.$$ ${MYCROFT_HOME} -${MYCROFT_HOME}/scripts/logs +$(list_log_files) /tmp/mycroft/ /opt/mycroft/skills EOF if [[ ${RUN_AS_ROOT} -eq 1 ]] ; then - while read CHECKFN ; do + while read -r CHECKFN ; do checkperms "${CHECKFN}" case $? in "0") mlog " - ${CHECKFN} has viable permissions." ;; @@ -127,7 +135,7 @@ function checksysinfo() { # -v function checkversion() { - mlog "Mycroft version is $( grep -B3 'END_VERSION_BLOCK' ${MYCROFT_HOME}/mycroft/version/__init__.py | cut -d' ' -f3 | tr -s '\012' '\056' )" + mlog "Mycroft version is $( grep -B3 'END_VERSION_BLOCK' "${MYCROFT_HOME}/mycroft/version/__init__.py" | cut -d' ' -f3 | tr -s '\012' '\056' )" } # do you want to do repeat? @@ -143,20 +151,38 @@ function checkmimic() { # pythoning! function checkPIP() { + REQUIREMENTS_FILE="${MYCROFT_HOME}/requirements/requirements.txt" + VENV_ACTIVATE_SCRIPT="${MYCROFT_HOME}/venv-activate.sh" mlog "Python checks" - mlog " - Verifying ${MYCROFT_HOME}/requirements/requirements.txt:" - if workon mycroft ; then + mlog " - Verifying $REQUIREMENTS_FILE:" + if [[ -f "$VENV_ACTIVATE_SCRIPT" ]] ; then + # shellcheck source=/dev/null + source "$VENV_ACTIVATE_SCRIPT" pip list > /tmp/mycroft-piplist.$$ - while read reqline ; do - IFS='==' read -r -a PIPREQ <<< "$reqline" - PIPREQVER=$( grep -i ^"${PIPREQ[0]} " /tmp/mycroft-piplist.$$ | cut -d'(' -f2 | tr -d '\051' ) - if [[ "${PIPREQVER}" == "${PIPREQ[2]}" ]] ; then - mlog " -- pip ${PIPREQ[0]} version ${PIPREQ[2]}" + while read -r reqline ; do + if (echo "$reqline" | grep -q '>='); then + delim='>=' + elif echo "$reqline" | grep -q '~='; then + delim='~=' + elif echo "$reqline" | grep -q '=='; then + delim='==' else - mlog " ~~ Warn: can't find ${PIPREQ[0]} ${PIPREQ[2]} in pip. (found ${PIPREQVER})" + delim='' + fi + if [[ $delim != '' ]]; then + IFS=${delim} read -r -a PIPREQ <<< "$reqline" + PIPREQVER=$( grep -i ^"${PIPREQ[0]} " /tmp/mycroft-piplist.$$ | cut -d'(' -f2 | tr -d '\051' | sed 's/ */ /g' | sed 's/ $//g') + + PKG_NAME="${PIPREQ[0],,}" + PKG_VERSION="${PIPREQ[2]}" + if [[ "${PIPREQVER,,}" == "${PKG_NAME} ${PKG_VERSION}" ]] ; then + mlog " -- pip ${PIPREQ[0]} version ${PIPREQ[2]}" + else + mlog " ~~ Warn: can't find ${PIPREQ[0]} ${PIPREQ[2]} in pip. (found ${PIPREQVER})" + fi fi - done < "${MYCROFT_HOME}/requirements.txt" + done < "${REQUIREMENTS_FILE}" deactivate mlog " - PIP list can be found at /tmp/mycroft-piplist.$$ to verify any issues." else @@ -176,13 +202,11 @@ function checktubes() { # I prefer biking myself. function checkrunning() { - while read SCREEN_SESS ; do - SESS_NAME=$( echo "${SCREEN_SESS}" | cut -d'(' -f1 | cut -d'.' -f2 ) - SESS_ID=$( echo "${SCREEN_SESS}" | cut -d'.' -f1 ) - if [[ $( ps flax| grep "$SESS_ID" | awk ' { print $4 } ' | grep -c "$SESS_ID" ) -eq 1 ]]; then - mlog " - ${SESS_NAME} appears to be currently running." - fi - done < <(screen -list | grep mycroft) + for pid in $(pgrep -ax "python." | grep '\-m mycroft' | awk -F ' ' '{print $1}') + do + cmdline=$(tr '\000' ' ' < "/proc/${pid}/cmdline") + mlog " - ${cmdline}(${pid}) appears to be currently running." + done } # He's dead, Jim. @@ -217,7 +241,7 @@ fi RUNDIR=$( readlink -f "${0}" | tr -s '\057' '\012' | sed \$d | tr -s '\012' '\057' ) # Where is mycroft installed? -if [[ -f "${RUNDIR}/mycroft-service.screen" && -f "${RUNDIR}/../mycroft/__init__.py" ]] ; then +if [[ -f "${RUNDIR}/../mycroft/__init__.py" ]] ; then MYCROFT_HOME=$(cd "${RUNDIR}" && cd .. && pwd ) else if [[ -f "/opt/mycroft/mycroft/__init__.py" ]] ; then diff --git a/scripts/mycroft-use.sh b/scripts/mycroft-use.sh index a00d2bc5461d..f5a2aa56b824 100755 --- a/scripts/mycroft-use.sh +++ b/scripts/mycroft-use.sh @@ -26,8 +26,8 @@ current_pkg=$( cat /etc/apt/sources.list.d/repo.mycroft.ai.list ) stable_pkg="deb http://repo.mycroft.ai/repos/apt/debian debian main" unstable_pkg="deb http://repo.mycroft.ai/repos/apt/debian debian-unstable main" -mark_1_package_list="mycroft-mark-1 mycroft-core mycroft-wifi-setup" -picroft_package_list="mycroft-picroft mycroft-core mycroft-wifi-setup" +mark_1_package_list=(mycroft-mark-1 mycroft-core mycroft-wifi-setup) +picroft_package_list=(mycroft-picroft mycroft-core mycroft-wifi-setup) # Determine the platform mycroft_platform="null" @@ -44,7 +44,7 @@ fi function service_ctl() { service=${1} action=${2} - sudo /etc/init.d/${service} ${action} + sudo "/etc/init.d/${service}" "${action}" } function stop_mycroft() { @@ -88,10 +88,10 @@ function restore_init_scripts() { sudo sh -c 'cat /etc/init.d/mycroft-admin-service.original > /etc/init.d/mycroft-admin-service' sudo rm /etc/init.d/*.original chown mycroft:mycroft /home/mycroft/.mycroft/identity/identity2.json - sudo chown -Rvf mycroft:mycroft /var/log/mycroft* - sudo chown -Rvf mycroft:mycroft /tmp/mycroft - sudo chown -Rvf mycroft:mycroft /var/run/mycroft* - sudo chown -Rvf mycroft:mycroft /opt/mycroft + sudo chown -R mycroft:mycroft /var/log/mycroft* + sudo chown -R mycroft:mycroft /tmp/mycroft + sudo chown -R mycroft:mycroft /var/run/mycroft* + sudo chown -R mycroft:mycroft /opt/mycroft sudo chown mycroft:mycroft /var/tmp/mycroft_web_cache.json # reload daemon scripts @@ -115,39 +115,39 @@ function github_init_scripts() { # switch to point a github install and run as the current user # TODO Verify all of these - sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'${path}'/start-mycroft.sh audio"_g' /etc/init.d/mycroft-audio - sudo sed -i 's_.*RUNAS=.*_RUNAS='${user}'_g' /etc/init.d/mycroft-audio + sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'"${path}"'/start-mycroft.sh audio"_g' /etc/init.d/mycroft-audio + sudo sed -i 's_.*RUNAS=.*_RUNAS='"${user}"'_g' /etc/init.d/mycroft-audio sudo sed -i 's_stop() {_stop() {\nPID=$(ps ax | grep mycroft/audio/ | awk '"'NR==1{print \$1; exit}'"')\necho "${PID}" > "$PIDFILE"_g' /etc/init.d/mycroft-audio - sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'${path}'/start-mycroft.sh enclosure"_g' /etc/init.d/mycroft-enclosure-client - sudo sed -i 's_.*RUNAS=.*_RUNAS='${user}'_g' /etc/init.d/mycroft-enclosure-client + sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'"${path}"'/start-mycroft.sh enclosure"_g' /etc/init.d/mycroft-enclosure-client + sudo sed -i 's_.*RUNAS=.*_RUNAS='"${user}"'_g' /etc/init.d/mycroft-enclosure-client sudo sed -i 's_stop() {_stop() {\nPID=$(ps ax | grep mycroft/client/enclosure/ | awk '"'NR==1{print \$1; exit}'"')\necho "${PID}" > "$PIDFILE"_g' /etc/init.d/mycroft-enclosure-client - sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'${path}'/start-mycroft.sh bus"_g' /etc/init.d/mycroft-messagebus - sudo sed -i 's_.*RUNAS=.*_RUNAS='${user}'_g' /etc/init.d/mycroft-messagebus + sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'"${path}"'/start-mycroft.sh bus"_g' /etc/init.d/mycroft-messagebus + sudo sed -i 's_.*RUNAS=.*_RUNAS='"${user}"'_g' /etc/init.d/mycroft-messagebus sudo sed -i 's_stop() {_stop() {\nPID=$(ps ax | grep mycroft/messagebus/ | awk '"'NR==1{print \$1; exit}'"')\necho "${PID}" > "$PIDFILE"_g' /etc/init.d/mycroft-messagebus - sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'${path}'/start-mycroft.sh skills"_g' /etc/init.d/mycroft-skills - sudo sed -i 's_.*RUNAS=.*_RUNAS='${user}'_g' /etc/init.d/mycroft-skills + sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'"${path}"'/start-mycroft.sh skills"_g' /etc/init.d/mycroft-skills + sudo sed -i 's_.*RUNAS=.*_RUNAS='"${user}"'_g' /etc/init.d/mycroft-skills sudo sed -i 's_stop() {_stop() {\nPID=$(ps ax | grep mycroft/skills/ | awk '"'NR==1{print \$1; exit}'"')\necho "${PID}" > "$PIDFILE"_g' /etc/init.d/mycroft-skills - sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'${path}'/start-mycroft.sh voice"_g' /etc/init.d/mycroft-speech-client - sudo sed -i 's_.*RUNAS=.*_RUNAS='${user}'_g' /etc/init.d/mycroft-speech-client + sudo sed -i 's_.*SCRIPT=.*_SCRIPT="'"${path}"'/start-mycroft.sh voice"_g' /etc/init.d/mycroft-speech-client + sudo sed -i 's_.*RUNAS=.*_RUNAS='"${user}"'_g' /etc/init.d/mycroft-speech-client sudo sed -i 's_stop() {_stop() {\nPID=$(ps ax | grep mycroft/client/speech | awk '"'NR==1{print \$1; exit}'"')\necho "${PID}" > "$PIDFILE"_g' /etc/init.d/mycroft-speech-client # soft link the current user to the mycroft user's identity folder - chown ${user}:${user} /home/mycroft/.mycroft/identity/identity2.json - if [ ! -e ${HOME}/.mycroft ] ; then - mkdir ${HOME}/.mycroft + chown "${user}:${user}" /home/mycroft/.mycroft/identity/identity2.json + if [ ! -e "${HOME}/.mycroft" ] ; then + mkdir "${HOME}/.mycroft" fi - if [ ! -e ${HOME}/.mycroft/identity ] ; then - sudo ln -s /home/mycroft/.mycroft/identity ${HOME}/.mycroft/ + if [ ! -e "${HOME}/.mycroft/identity" ] ; then + sudo ln -s /home/mycroft/.mycroft/identity "${HOME}/.mycroft/" fi - sudo chown -Rvf ${user}:${user} /var/log/mycroft* - sudo chown -Rvf ${user}:${user} /var/run/mycroft* - sudo chown -Rvf ${user}:${user} /tmp/mycroft - sudo chown -Rvf ${user}:${user} /var/tmp/mycroft_web_cache.json + sudo chown -R "${user}:${user}" /var/log/mycroft* + sudo chown -R "${user}:${user}" /var/run/mycroft* + sudo chown -R "${user}:${user}" /tmp/mycroft + sudo chown -R "${user}:${user}" /var/tmp/mycroft_web_cache.json # reload daemon scripts sudo systemctl daemon-reload @@ -159,24 +159,24 @@ function github_init_scripts() { function invoke_apt() { if [ ${mycroft_platform} == "mycroft_mark_1" ] ; then echo "${1}ing the mycroft-mark-1 metapackage..." - sudo apt-get ${1} mycroft-mark-1 -y + sudo apt-get "${1}" mycroft-mark-1 -y elif [ ${mycroft_platform} == "picroft" ] ; then echo "${1}ing the mycroft-picroft metapackage..." - sudo apt-get ${1} mycroft-picroft -y + sudo apt-get "${1}" mycroft-picroft -y else # for unknown, just update the generic package echo "${1}ing the generic mycroft-core package..." - sudo apt-get ${1} mycroft-core -y + sudo apt-get "${1}" mycroft-core -y fi } function remove_all() { if [ ${mycroft_platform} == "mycroft_mark_1" ] ; then echo "Removing the mycroft mark-1 packages..." - sudo apt-get remove ${mark_1_package_list} -y + sudo apt-get remove "${mark_1_package_list[@]}" -y elif [ ${mycroft_platform} == "picroft" ] ; then echo "Removing the picroft packages..." - sudo apt-get remove ${picroft_package_list} -y + sudo apt-get remove "${picroft_package_list[@]}" -y else # for unknown, just update the generic package echo "Removing the generic mycroft-core package..." @@ -197,8 +197,7 @@ function stable_to_unstable_server() { conf_path=/home/mycroft/.mycroft/ # check if on stable (home-test.mycroft.ai) already - cmp --silent ${conf_path}/mycroft.conf ${conf_path}/mycroft.conf.unstable - if [ $? -eq 0 ] ; then + if ! cmp --silent "${conf_path}/mycroft.conf" "${conf_path}/mycroft.conf.unstable" ; then echo "Already set to use the home-test.mycroft.ai server" return fi @@ -206,15 +205,16 @@ function stable_to_unstable_server() { # point to test server echo "Changing mycroft.conf to point to test server api-test.mycroft.ai" if [ -f ${conf_path}mycroft.conf ] ; then - cp ${conf_path}mycroft.conf ${conf_path}mycroft.conf.stable + cp "${conf_path}mycroft.conf" "${conf_path}mycroft.conf.stable" else echo "could not find mycroft.conf, was it deleted?" fi if [ -f ${conf_path}mycroft.conf.unstable ] ; then - cp ${conf_path}mycroft.conf.unstable ${conf_path}mycroft.conf + cp ${conf_path}mycroft.conf.unstable "${conf_path}mycroft.conf" else rm -r ${conf_path}mycroft.conf - echo '{"server": {"url":"https://api-test.mycroft.ai", "version":"v1", "update":true, "metrics":false }}' $( cat ${conf_path}mycroft.conf.stable ) | jq -s add > ${conf_path}mycroft.conf + conf_data=$( cat "${conf_path}mycroft.conf.stable") + echo '{"server": {"url":"https://api-test.mycroft.ai", "version":"v1", "update":true, "metrics":false }}' "${conf_data}" | jq -s add > "${conf_path}mycroft.conf" fi # saving identity2.json to stable state @@ -238,8 +238,7 @@ function unstable_to_stable_server() { conf_path=/home/mycroft/.mycroft/ # check if on stable (home.mycroft.ai) already - cmp --silent ${conf_path}/mycroft.conf ${conf_path}/mycroft.conf.stable - if [ $? -eq 0 ] ; then + if cmp --silent ${conf_path}/mycroft.conf ${conf_path}/mycroft.conf.stable; then echo "Already set to use the home.mycroft.ai server" return fi @@ -247,7 +246,9 @@ function unstable_to_stable_server() { # point api to production server echo "Changing mycroft.conf to point to production server api.mycroft.ai" if [ -f ${conf_path}mycroft.conf ] ; then - echo '{"server": {"url":"https://api-test.mycroft.ai", "version":"v1", "update":true, "metrics":false }}' $( cat ${conf_path}mycroft.conf ) | jq -s add > ${conf_path}mycroft.conf.unstable + base_config='{"server": {"url":"https://api-test.mycroft.ai", "version":"v1", "update":true, "metrics":false }}' + current_config=$( cat "${conf_path}mycroft.conf" ) + echo "$base_config" "$current_config" | jq -s add > "${conf_path}mycroft.conf.unstable" else echo "could not find mycroft.conf, was it deleted?" fi @@ -319,16 +320,15 @@ elif [ "${change_to}" == "stable" ] ; then elif [ "${change_to}" == "github" ] ; then echo "Switching to github..." - if [ ! -d ${path} ] ; then + if [ ! -d "${path}" ] ; then mkdir --parents "${path}" - cd "${path}" - cd .. + cd "${path}/.." || exit git clone https://github.com/MycroftAI/mycroft-core.git "${path}" fi sudo chmod -x /etc/cron.hourly/mycroft-core # Disable updates - if [ -d ${path} ] ; then + if [ -d "${path}" ] ; then if [ -f /usr/local/bin/mimic ] ; then echo "Mimic file exists" mimic_flag="-sm" @@ -336,9 +336,9 @@ elif [ "${change_to}" == "github" ] ; then echo "file doesn't exist" mimic_flag="" fi - cd ${path} + cd "${path}" || exit # Build the dev environment - ${path}/dev_setup.sh --allow-root ${mimic_flag} + "${path}"/dev_setup.sh --allow-root "${mimic_flag}" # Switch init scripts to start the github version github_init_scripts diff --git a/scripts/prepare-msm.sh b/scripts/prepare-msm.sh index aae08b96f94c..469fcf9290f9 100755 --- a/scripts/prepare-msm.sh +++ b/scripts/prepare-msm.sh @@ -29,7 +29,7 @@ DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" # Determine which user is running this script setup_user=$USER -setup_group=$( id -gn $USER ) +setup_group=$( id -gn "$USER" ) function found_exe() { hash "$1" 2>/dev/null @@ -42,8 +42,8 @@ fi # change ownership of ${mycroft_root_dir} to ${setup_user } recursively function change_ownership { - echo "Changing ownership of" ${mycroft_root_dir} "to user:" ${setup_user} "with group:" ${setup_group} - $SUDO chown -Rf ${setup_user}:${setup_group} ${mycroft_root_dir} + echo "Changing ownership of" ${mycroft_root_dir} "to user:" "${setup_user}" "with group:" "${setup_group}" + $SUDO chown -Rf "${setup_user}:${setup_group}" ${mycroft_root_dir} } @@ -54,7 +54,7 @@ if [[ ${CI} != true ]] ; then change_ownership fi - if [ ! -w ${SKILLS_DIR} ] ; then + if [ ! -w "${skills_dir}" ] ; then change_ownership fi fi diff --git a/start-mycroft.sh b/start-mycroft.sh index 4c6e453a3a50..0812cfa3872e 100755 --- a/start-mycroft.sh +++ b/start-mycroft.sh @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/bin/sh # Copyright 2017 Mycroft AI Inc. # @@ -14,15 +14,15 @@ # See the License for the specific language governing permissions and # limitations under the License. -SOURCE="${BASH_SOURCE[0]}" +SOURCE="$0" script=${0} script=${script##*/} -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit 1 # Enter scripts folder or fail! DIR="$( pwd )" VIRTUALENV_ROOT=${VIRTUALENV_ROOT:-"${DIR}/.venv"} -function help() { +help() { echo "${script}: Mycroft command/service launcher" echo "usage: ${script} [COMMAND] [restart] [params]" echo @@ -60,7 +60,7 @@ function help() { } _module="" -function name-to-script-path() { +name_to_script_path() { case ${1} in "bus") _module="mycroft.messagebus.service" ;; "skills") _module="mycroft.skills" ;; @@ -77,51 +77,51 @@ function name-to-script-path() { esac } -function source-venv() { +source_venv() { # Enter Python virtual environment, unless under Docker if [ ! -f "/.dockerenv" ] ; then - source ${VIRTUALENV_ROOT}/bin/activate + . "${VIRTUALENV_ROOT}/bin/activate" fi } first_time=true -function init-once() { +init_once() { if ($first_time) ; then echo "Initializing..." "${DIR}/scripts/prepare-msm.sh" - source-venv + source_venv first_time=false fi } -function launch-process() { - init-once +launch_process() { + init_once - name-to-script-path ${1} + name_to_script_path "${1}" # Launch process in foreground echo "Starting $1" - python3 -m ${_module} $_params + python3 -m ${_module} "$@" } -function require-process() { +require_process() { # Launch process if not found - name-to-script-path ${1} + name_to_script_path "${1}" if ! pgrep -f "python3 (.*)-m ${_module}" > /dev/null ; then # Start required process - launch-background ${1} + launch_background "${1}" fi } -function launch-background() { - init-once +launch_background() { + init_once # Check if given module is running and start (or restart if running) - name-to-script-path ${1} + name_to_script_path "${1}" if pgrep -f "python3 (.*)-m ${_module}" > /dev/null ; then if ($_force_restart) ; then echo "Restarting: ${1}" - "${DIR}/stop-mycroft.sh" ${1} + "${DIR}/stop-mycroft.sh" "${1}" else # Already running, no need to restart return @@ -131,39 +131,39 @@ function launch-background() { fi # Security warning/reminder for the user - if [[ "${1}" == "bus" ]] ; then + if [ "${1}" = "bus" ] ; then echo "CAUTION: The Mycroft bus is an open websocket with no built-in security" echo " measures. You are responsible for protecting the local port" echo " 8181 with a firewall as appropriate." fi # Launch process in background, sending logs to standard location - python3 -m ${_module} $_params >> /var/log/mycroft/${1}.log 2>&1 & + python3 -m ${_module} "$@" >> "/var/log/mycroft/${1}.log" 2>&1 & } -function launch-all() { +launch_all() { echo "Starting all mycroft-core services" - launch-background bus - launch-background skills - launch-background audio - launch-background voice - launch-background enclosure + launch_background bus + launch_background skills + launch_background audio + launch_background voice + launch_background enclosure } -function check-dependencies() { +check_dependencies() { if [ -f .dev_opts.json ] ; then auto_update=$( jq -r ".auto_update" < .dev_opts.json 2> /dev/null) else auto_update="false" fi - if [ "$auto_update" == "true" ] ; then + if [ "$auto_update" = "true" ] ; then # Check github repo for updates (e.g. a new release) git pull fi - if [ ! -f .installed ] || ! md5sum -c &> /dev/null < .installed ; then + if [ ! -f .installed ] || ! md5sum -c > /dev/null 2>&1 < .installed ; then # Critical files have changed, dev_setup.sh should be run again - if [ "$auto_update" == "true" ] ; then + if [ "$auto_update" = "true" ] ; then echo "Updating dependencies..." bash dev_setup.sh else @@ -179,83 +179,91 @@ function check-dependencies() { _opt=$1 _force_restart=false + +if [ $# -eq 0 ]; then + help + return +fi + shift -if [[ "${1}" == "restart" ]] || [[ "${_opt}" == "restart" ]] ; then +if [ "${1}" = "restart" ] || [ "${_opt}" = "restart" ] ; then _force_restart=true - if [[ "${_opt}" == "restart" ]] ; then + if [ "${_opt}" = "restart" ] ; then # Support "start-mycroft.sh restart all" as well as "start-mycroft.sh all restart" _opt=$1 fi - shift + + if [ $# -gt 0 ]; then + shift + fi fi -_params=$@ -if [[ ! "${_opt}" == "cli" ]] ; then - check-dependencies +if [ ! "${_opt}" = "cli" ] ; then + check_dependencies fi case ${_opt} in "all") - launch-all + launch_all ;; "bus") - launch-background ${_opt} + launch_background "${_opt}" ;; "audio") - launch-background ${_opt} + launch_background "${_opt}" ;; "skills") - launch-background ${_opt} + launch_background "${_opt}" ;; "voice") - launch-background ${_opt} + launch_background "${_opt}" ;; "debug") - launch-all - launch-process cli + launch_all + launch_process cli ;; "cli") - require-process bus - require-process skills - launch-process ${_opt} + require_process bus + require_process skills + launch_process "${_opt}" ;; # TODO: Restore support for Wifi Setup on a Picroft, etc. # "wifi") - # launch-background ${_opt} + # launch_background ${_opt} # ;; "unittest") - source-venv + source_venv pytest test/unittests/ --cov=mycroft "$@" ;; "singleunittest") - source-venv + source_venv pytest "$@" ;; "skillstest") - source-venv + source_venv pytest test/integrationtests/skills/discover_tests.py "$@" ;; "vktest") - source "$DIR/bin/mycroft-skill-testrunner" vktest "$@" + "$DIR/bin/mycroft-skill-testrunner" vktest "$@" ;; "audiotest") - launch-process ${_opt} + launch_process "${_opt}" ;; "wakewordtest") - launch-process ${_opt} + launch_process "${_opt}" ;; "sdkdoc") - source-venv - cd doc - make ${_params} + source_venv + cd doc || exit 1 # Exit if doc directory doesn't exist + make "$@" cd .. ;; "enclosure") - launch-background ${_opt} + launch_background "${_opt}" ;; *) diff --git a/stop-mycroft.sh b/stop-mycroft.sh index da74fbecd886..546e3e5c1bac 100755 --- a/stop-mycroft.sh +++ b/stop-mycroft.sh @@ -1,4 +1,4 @@ -#!/usr/bin/env bash +#!/bin/sh # Copyright 2017 Mycroft AI Inc. # @@ -14,13 +14,14 @@ # See the License for the specific language governing permissions and # limitations under the License. -SOURCE="${BASH_SOURCE[0]}" +# This script is never sourced but always directly executed, so this is safe to do +SOURCE="$0" script=${0} script=${script##*/} -cd -P "$( dirname "$SOURCE" )" +cd -P "$( dirname "$SOURCE" )" || exit 1 # quit if change of folder fails -function help() { +help() { echo "${script}: Mycroft service stopper" echo "usage: ${script} [service]" echo @@ -40,37 +41,37 @@ function help() { exit 0 } -function process-running() { - if [[ $( pgrep -f "python3 (.*)-m mycroft.*${1}" ) ]] ; then +process_running() { + if [ "$( pgrep -f "python3 (.*)-m mycroft.*${1}" )" ] ; then return 0 else return 1 fi } -function end-process() { - if process-running $1 ; then +end_process() { + if process_running "$1" ; then # Find the process by name, only returning the oldest if it has children pid=$( pgrep -o -f "python3 (.*)-m mycroft.*${1}" ) - echo -n "Stopping $1 (${pid})..." - kill -SIGINT ${pid} + printf "Stopping %s (%s)..." "$1" "${pid}" + kill -s INT "${pid}" # Wait up to 5 seconds (50 * 0.1) for process to stop c=1 while [ $c -le 50 ] ; do - if process-running $1 ; then + if process_running "$1" ; then sleep 0.1 - (( c++ )) + c=$((c + 1)) else c=999 # end loop fi done - if process-running $1 ; then + if process_running "$1" ; then echo "failed to stop." pid=$( pgrep -o -f "python3 (.*)-m mycroft.*${1}" ) - echo -n " Killing $1 (${pid})..." - kill -9 ${pid} + printf " Killing %s (%s)...\n" "$1" "${pid}" + kill -9 "${pid}" echo "killed." result=120 else @@ -87,33 +88,33 @@ result=0 # default, no change OPT=$1 -shift +if [ $# -gt 0 ]; then + shift +fi case ${OPT} in - "all") - ;& - "") + ""|"all") echo "Stopping all mycroft-core services" - end-process skills - end-process audio - end-process speech - end-process enclosure - end-process messagebus.service + end_process skills + end_process audio + end_process speech + end_process enclosure + end_process messagebus.service ;; "bus") - end-process messagebus.service + end_process messagebus.service ;; "audio") - end-process audio + end_process audio ;; "skills") - end-process skills + end_process skills ;; "voice") - end-process speech + end_process speech ;; "enclosure") - end-process enclosure + end_process enclosure ;; *) @@ -125,4 +126,4 @@ esac # 0 if nothing changed (e.g. --help or no process was running) # 100 at least one process was stopped # 120 if any process had to be killed -exit $result \ No newline at end of file +exit $result diff --git a/test/Dockerfile b/test/Dockerfile index 0ae41a48485a..c76c02b68eb7 100644 --- a/test/Dockerfile +++ b/test/Dockerfile @@ -96,7 +96,7 @@ FROM core_builder as voight_kampff_builder ARG platform # Setup a dummy TTS backend for the audio process RUN mkdir /etc/mycroft -RUN echo '{"tts": {"module": "dummy"}}' > /etc/mycroft/mycroft.conf +RUN echo '{"tts": {"module": "dummy"}, "skills": {"auto_update": false}}' > /etc/mycroft/mycroft.conf RUN mkdir ~/.mycroft/allure-result # The behave feature files for a skill are defined within the skill's diff --git a/test/integrationtests/voight_kampff/__init__.py b/test/integrationtests/voight_kampff/__init__.py index a14b910eef82..ec4e5594e4e2 100644 --- a/test/integrationtests/voight_kampff/__init__.py +++ b/test/integrationtests/voight_kampff/__init__.py @@ -12,7 +12,18 @@ # See the License for the specific language governing permissions and # limitations under the License. # - -from .tools import (emit_utterance, wait_for_dialog, then_wait, - then_wait_fail, mycroft_responses, - print_mycroft_responses, wait_for_audio_service) +"""Public API into the voight_kampff package.""" +from .tools import ( + emit_utterance, + format_dialog_match_error, + mycroft_responses, + print_mycroft_responses, + then_wait, + then_wait_fail, + wait_for_audio_service, + wait_for_dialog, + wait_for_dialog_match, + VoightKampffCriteriaMatcher, + VoightKampffDialogMatcher, + VoightKampffMessageMatcher +) diff --git a/test/integrationtests/voight_kampff/default.yml b/test/integrationtests/voight_kampff/default.yml index 7ffe2f5ee176..1b3a9f901dc4 100644 --- a/test/integrationtests/voight_kampff/default.yml +++ b/test/integrationtests/voight_kampff/default.yml @@ -12,7 +12,6 @@ test_skills: - mycroft-npr-news - mycroft-installer - mycroft-singing -- mycroft-stock - mycroft-mark-1 - fallback-unknown - fallback-query diff --git a/test/integrationtests/voight_kampff/features/environment.py b/test/integrationtests/voight_kampff/features/environment.py index 6debde17653c..1b25fd380302 100644 --- a/test/integrationtests/voight_kampff/features/environment.py +++ b/test/integrationtests/voight_kampff/features/environment.py @@ -38,32 +38,69 @@ def create_voight_kampff_logger(): class InterceptAllBusClient(MessageBusClient): + """Bus Client storing all messages received. + + This allows read back of older messages and non-event-driven operation. + """ def __init__(self): super().__init__() self.messages = [] self.message_lock = Lock() self.new_message_available = Event() + self._processed_messages = 0 + + def on_message(self, _, message): + """Extends normal operation by storing the received message. - def on_message(self, message): + Args: + message (Message): message from the Mycroft bus + """ with self.message_lock: self.messages.append(Message.deserialize(message)) self.new_message_available.set() - super().on_message(message) + super().on_message(_, message) def get_messages(self, msg_type): + """Get messages from received list of messages. + + Args: + msg_type (None,str): string filter for the message type to extract. + if None all messages will be returned. + """ with self.message_lock: + self._processed_messages = len(self.messages) if msg_type is None: return [m for m in self.messages] else: return [m for m in self.messages if m.msg_type == msg_type] def remove_message(self, msg): + """Remove a specific message from the list of messages. + + Args: + msg (Message): message to remove from the list + """ with self.message_lock: + if msg not in self.messages: + raise ValueError(f'{msg.msg_type} was not found in ' + 'the list of messages.') + # Update processed message count if a read message was removed + if self.messages.index(msg) < self._processed_messages: + self._processed_messages -= 1 + self.messages.remove(msg) def clear_messages(self): + """Clear all messages that has been fetched at least once.""" + with self.message_lock: + self.messages = self.messages[self._processed_messages:] + self._processed_messages = 0 + + def clear_all_messages(self): + """Clear all messages.""" with self.message_lock: self.messages = [] + self._processed_messages = 0 def before_all(context): @@ -111,14 +148,12 @@ def after_all(context): def after_feature(context, feature): context.log.info('Result: {} ({:.2f}s)'.format(str(feature.status.name), feature.duration)) - sleep(1) def after_scenario(context, scenario): """Wait for mycroft completion and reset any changed state.""" # TODO wait for skill handler complete - sleep(0.5) wait_while_speaking() - context.bus.clear_messages() + context.bus.clear_all_messages() context.matched_message = None context.step_timeout = 10 # Reset the step_timeout to 10 seconds diff --git a/test/integrationtests/voight_kampff/features/steps/utterance_responses.py b/test/integrationtests/voight_kampff/features/steps/utterance_responses.py index de4197e1741b..6277d9922bf3 100644 --- a/test/integrationtests/voight_kampff/features/steps/utterance_responses.py +++ b/test/integrationtests/voight_kampff/features/steps/utterance_responses.py @@ -17,23 +17,22 @@ use with behave. """ from os.path import join, exists, basename -from glob import glob +from pathlib import Path import re +from string import Formatter import time from behave import given, when, then +from mycroft.dialog import MustacheDialogRenderer from mycroft.messagebus import Message from mycroft.audio import wait_while_speaking +from mycroft.util.format import expand_options from test.integrationtests.voight_kampff import (mycroft_responses, then_wait, then_wait_fail) -TIMEOUT = 10 -SLEEP_LENGTH = 0.25 - - def find_dialog(skill_path, dialog, lang): """Check the usual location for dialogs. @@ -47,10 +46,15 @@ def find_dialog(skill_path, dialog, lang): def load_dialog_file(dialog_path): """Load dialog files and get the contents.""" - with open(dialog_path) as f: - lines = f.readlines() - return [l.strip().lower() for l in lines - if l.strip() != '' and l.strip()[0] != '#'] + renderer = MustacheDialogRenderer() + renderer.load_template_file('template', dialog_path) + expanded_lines = [] + for template in renderer.templates: + # Expand parentheses in lines + for line in renderer.templates[template]: + expanded_lines += expand_options(line) + return [line.strip().lower() for line in expanded_lines + if line.strip() != '' and line.strip()[0] != '#'] def load_dialog_list(skill_path, dialog): @@ -69,6 +73,26 @@ def load_dialog_list(skill_path, dialog): return load_dialog_file(dialog_path), debug +def _get_dialog_files(skill_path, lang): + """Generator expression returning all dialog files. + + This includes both the 'locale' and the older style 'dialog' folder. + + Args: + skill_path (str): skill root folder + lang (str): language code to check + + yields: + (Path) path of each found dialog file + """ + in_dialog_dir = Path(skill_path, 'dialog', lang).rglob('*.dialog') + for dialog_path in in_dialog_dir: + yield dialog_path + in_locale_dir = Path(skill_path, 'locale', lang).rglob('*.dialog') + for dialog_path in in_locale_dir: + yield dialog_path + + def dialog_from_sentence(sentence, skill_path, lang): """Find dialog file from example sentence. @@ -79,9 +103,8 @@ def dialog_from_sentence(sentence, skill_path, lang): Returns (str): Dialog file best matching the sentence. """ - dialog_paths = join(skill_path, 'dialog', lang, '*.dialog') best = (None, 0) - for path in glob(dialog_paths): + for path in _get_dialog_files(skill_path, lang): patterns = load_dialog_file(path) match, _ = _match_dialog_patterns(patterns, sentence.lower()) if match is not False: @@ -96,19 +119,25 @@ def dialog_from_sentence(sentence, skill_path, lang): def _match_dialog_patterns(dialogs, sentence): """Match sentence against a list of dialog patterns. - Returns index of found match. + dialogs (list of str): dialog file entries to match against + sentence (str): string to match. + + Returns: + (tup) index of found match, debug text """ # Allow custom fields to be anything - dialogs = [re.sub(r'{.*?\}', r'.*', dia) for dia in dialogs] - # Remove left over '}' - dialogs = [re.sub(r'\}', r'', dia) for dia in dialogs] - dialogs = [re.sub(r' .* ', r' .*', dia) for dia in dialogs] - # Merge consequtive .*'s into a single .* - dialogs = [re.sub(r'\.\*( \.\*)+', r'.*', dia) for dia in dialogs] - # Remove double whitespaces - dialogs = ['^' + ' '.join(dia.split()) for dia in dialogs] + # i.e {field} gets turned into ".*" + regexes = [] + for dialog in dialogs: + data = {element[1]: '.*' + for element in Formatter().parse(dialog)} + regexes.append(dialog.format(**data)) + + # Remove double whitespaces and ensure that it matches from + # the beginning of the line. + regexes = ['^' + ' '.join(reg.split()) for reg in regexes] debug = 'MATCHING: {}\n'.format(sentence) - for index, regex in enumerate(dialogs): + for index, regex in enumerate(regexes): match = re.match(regex, sentence) debug += '---------------\n' debug += '{} {}\n'.format(regex, match is not None) @@ -235,8 +264,14 @@ def check_contains(message): @then('the user replies "{text}"') @then('the user says "{text}"') def then_user_follow_up(context, text): - time.sleep(2) + """Send a user response after being prompted by device. + + The sleep after the device is finished speaking is to address a race + condition in the MycroftSkill base class conversational code. It can + be removed when the race condition is addressed. + """ wait_while_speaking() + time.sleep(2) context.bus.emit(Message('recognizer_loop:utterance', data={'utterances': [text], 'lang': context.lang, @@ -247,13 +282,10 @@ def then_user_follow_up(context, text): @then('mycroft should send the message "{message_type}"') def then_messagebus_message(context, message_type): - """Set a timeout for the current Scenario.""" - cnt = 0 - while context.bus.get_messages(message_type) == []: - if cnt > int(TIMEOUT * (1.0 / SLEEP_LENGTH)): - assert False, "Message not found" - break - else: - cnt += 1 - - time.sleep(SLEEP_LENGTH) + """Verify a specific message is sent.""" + def check_dummy(message): + """We are just interested in the message data, just the type.""" + return True, "" + + message_found, _ = then_wait(message_type, check_dummy, context) + assert message_found, "No matching message received." diff --git a/test/integrationtests/voight_kampff/run_test_suite.sh b/test/integrationtests/voight_kampff/run_test_suite.sh index 7286fb9c5c25..6fe9be0c9b28 100755 --- a/test/integrationtests/voight_kampff/run_test_suite.sh +++ b/test/integrationtests/voight_kampff/run_test_suite.sh @@ -9,6 +9,10 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" # Start pulseaudio if running in CI environment if [[ -v CI ]]; then + # Ensure pulseaudio is stateless on start up + # This stops the daemon from randomly failing on startup + # See https://superuser.com/a/1545361 for more info + rm -rf /root/.config/pulse pulseaudio -D fi # Start all mycroft core services. @@ -18,8 +22,10 @@ ${SCRIPT_DIR}/../../../start-mycroft.sh all echo "Running behave with the arguments \"$@\"" behave $@ RESULT=$? -# Stop all mycroft core services. -${SCRIPT_DIR}/../../../stop-mycroft.sh all +if [[ -v CI ]]; then + # Stop all mycroft core services if running in CI environment. + ${SCRIPT_DIR}/../../../stop-mycroft.sh all +fi # Reort the result of the behave test as exit status exit $RESULT diff --git a/test/integrationtests/voight_kampff/tools.py b/test/integrationtests/voight_kampff/tools.py index 8a26b044550d..b0e00a1fad88 100644 --- a/test/integrationtests/voight_kampff/tools.py +++ b/test/integrationtests/voight_kampff/tools.py @@ -12,19 +12,312 @@ # See the License for the specific language governing permissions and # limitations under the License. # - """Common tools to use when creating step files for behave tests.""" +from threading import Event +from typing import Any, Callable, List, Tuple import time +from mycroft.audio.utils import wait_while_speaking from mycroft.messagebus import Message +DEFAULT_TIMEOUT = 10 + -TIMEOUT = 10 +class VoightKampffMessageMatcher: + """Matches a specified message type to messages emitted on the bus. + Usage: + Intended for use in a single test condition. -def then_wait(msg_type, criteria_func, context, timeout=None): - """Wait for a specified time for criteria to be fulfilled. + matcher = VoightKampffMessageMatcher(message_type, context) + match_found, error_message = matcher.match() + assert match_found, error_message + + Attributes: + message_type: identifier of the message to search for on the bus + context: the Behave context from the test utilizing this class + match_event: mechanism for knowing when a match is found + error_message: message that can be used by the test to communicate + the reason for a failed match to the tester. + """ + def __init__(self, context: Any, message_type: str): + self.message_type = message_type + self.context = context + self.match_event = Event() + self.error_message = "" + + @property + def match_found(self): + return self.match_event.is_set() + + def match(self, timeout: int = None): + """Attempts to match the requested message type to emitted bus events. + + Use a message bus event handler to capture any message emitted on the + bus that matches the message type specified by the caller. Also + checks any messages emitted prior to the handler being defined to + protect against a race condition. + + Args: + timeout: number of seconds to attempt matching before giving up + """ + timeout = timeout or self.context.step_timeout + self.context.bus.on(self.message_type, self.handle_message) + self._check_historical_messages() + if not self.match_event.is_set(): + self.match_event.wait(timeout=timeout) + self.context.bus.remove(self.message_type, self.handle_message) + if not self.match_found: + self._build_error_message() + + return self.match_found, self.error_message + + def _check_historical_messages(self): + """Searches messages emitted before the event handler was defined.""" + for message in self.context.bus.get_messages(self.message_type): + self.handle_message(message) + if self.match_found: + break + self.context.bus.clear_messages() + + def handle_message(self, message: Message): + """Applies matching criteria to the emitted event. + + Args: + message: message emitted by bus with the requested message type + """ + self.context.matched_message = message + self.match_event.set() + + def _build_error_message(self): + """Builds a message that communicates the failure to the test.""" + self.error_message = ( + f"Expected message type {self.message_type} was not emitted." + ) + + +class VoightKampffDialogMatcher(VoightKampffMessageMatcher): + """Variation of VoightKampffEventMatcher for matching dialogs. + + Usage: + Intended for use in a single test condition. + + matcher = VoightKampffDialogMatcher(context, dialogs) + match_found, error_message = matcher.match() + assert match_found, error_message + + Attributes: + dialogs: one or more dialog names that will constitute a match + speak_messages: bus messages with message type of "speak" captured + in the matching process + """ + def __init__(self, context: Any, dialogs: List[str]): + super().__init__(context, message_type="speak") + self.dialogs = dialogs + self.speak_messages = list() + + def handle_message(self, message: Message): + """Applies matching criteria to the emitted event. + + Args: + message: message emitted by bus with the requested message type + """ + self.speak_messages.append(message) + dialog = message.data.get('meta', {}).get('dialog') + if dialog in self.dialogs: + wait_while_speaking() + self.context.matched_message = message + self.match_event.set() + + def _build_error_message(self): + """Builds a message that communicates the failure to the test.""" + self.error_message = ( + 'Expected Mycroft to respond with one of:\n' + f"\t{', '.join(self.dialogs)}\n" + "Actual response(s):\n" + ) + if self.speak_messages: + for message in self.speak_messages: + meta = message.data.get("meta") + if meta is not None: + if 'dialog' in meta: + self.error_message += f"\tDialog: {meta['dialog']}" + if 'skill' in meta: + self.error_message += ( + f" (from {meta['skill']} skill)\n" + ) + else: + self.error_message += "\tMycroft didn't respond" + + +class VoightKampffCriteriaMatcher(VoightKampffMessageMatcher): + """Variation of VoightKampffEventMatcher for matching event data. + + In some cases, matching the message type is not enough. The test + requires data in the message payload to match a specified criteria + to pass. + + Usage: + Intended for use in a single test condition. + + matcher = VoightKampffCriteriaMatcher( + message_type, context, criteria_matcher + ) + match_found, error_message = matcher.match() + assert match_found, error_message + + Attributes: + criteria_matcher: Function to determine if a message contains + the data necessary for the test case to pass + """ + def __init__(self, context: Any, message_type: str, + criteria_matcher: Callable): + super().__init__(context, message_type) + self.criteria_matcher = criteria_matcher + self.error_message = "" + + def handle_message(self, message: Message): + """Applies matching criteria to the emitted event. + + Args: + message: message emitted by bus with the requested message type + """ + status, error_message = self.criteria_matcher(message) + self.error_message += error_message + if status: + self.context.matched_message = message + self.match_event.set() + + def _build_error_message(self): + """Builds a message that communicates the failure to the test.""" + # The error message is built from the return value of the criteria + # matcher so this method is not needed. + pass + + +# TODO: Remove in 21.08 +class CriteriaWaiter: + """Wait for a message to meet a certain criteria. + + Args: + msg_type: message type to watch + criteria_func: Function to determine if a message fulfilling the + test case has been found. + context: behave context + """ + def __init__(self, msg_type, criteria_func, context): + self.msg_type = msg_type + self.criteria_func = criteria_func + self.context = context + self.result = Event() + + def reset(self): + """Reset the wait state.""" + self.result.clear() + + def wait_unspecific(self, timeout): + """ + Wait for a specified time for criteria to be fulfilled by any message. + + This use case is deprecated and only for backward compatibility + + Args: + timeout: Time allowance for a message fulfilling the criteria, if + provided will override the normal normal step timeout. + + Returns: + tuple (bool, str) test status and debug output + """ + timeout = timeout or self.context.step_timeout + start_time = time.monotonic() + debug = '' + while time.monotonic() < start_time + timeout: + for message in self.context.bus.get_messages(None): + status, test_dbg = self.criteria_func(message) + debug += test_dbg + if status: + self.context.matched_message = message + self.context.bus.remove_message(message) + return True, debug + self.context.bus.new_message_available.wait(0.5) + # Timed out return debug from test + return False, debug + + def _check_historical_messages(self): + """Search through the already received messages for a match. + + Returns: + tuple (bool, str) test status and debug output + + """ + debug = '' + for message in self.context.bus.get_messages(self.msg_type): + status, test_dbg = self.criteria_func(message) + debug += test_dbg + if status: + self.context.matched_message = message + self.context.bus.remove_message(message) + self.result.set() + break + return debug + + def wait_specific(self, timeout=None): + """Wait for a specific message type to fullfil a criteria. + + Uses an event-handler to not repeatedly loop. + + Args: + timeout: Time allowance for a message fulfilling the criteria, if + provided will override the normal normal step timeout. + + Returns: + tuple (bool, str) test status and debug output + """ + timeout = timeout or self.context.step_timeout + + debug = '' + + def on_message(message): + nonlocal debug + status, test_dbg = self.criteria_func(message) + debug += test_dbg + if status: + self.context.matched_message = message + self.result.set() + + self.context.bus.on(self.msg_type, on_message) + # Check historical messages + historical_debug = self._check_historical_messages() + + # If no matching message was already caught, wait for it + if not self.result.is_set(): + self.result.wait(timeout=timeout) + self.context.bus.remove(self.msg_type, on_message) + return self.result.is_set(), historical_debug + debug + + def wait(self, timeout=None): + """Wait for a specific message type to fullfil a criteria. + + Uses an event-handler to not repeatedly loop. + + Args: + timeout: Time allowance for a message fulfilling the criteria, if + provided will override the normal normal step timeout. + + Returns: + (result (bool), debug (str)) Result containing status and debug + message. + """ + if self.msg_type is None: + return self.wait_unspecific(timeout) + else: + return self.wait_specific(timeout) + + +def then_wait(msg_type: str, criteria_func: Callable, context: Any, + timeout: int = None) -> Tuple[bool, str]: + """Wait for a specific message type to fulfill a criteria. Args: msg_type: message type to watch @@ -35,25 +328,16 @@ def then_wait(msg_type, criteria_func, context, timeout=None): provided will override the normal normal step timeout. Returns: - tuple (bool, str) test status and debug output + The success of the match attempt and an error message. """ - timeout = timeout or context.step_timeout - start_time = time.monotonic() - debug = '' - while time.monotonic() < start_time + timeout: - for message in context.bus.get_messages(msg_type): - status, test_dbg = criteria_func(message) - debug += test_dbg - if status: - context.matched_message = message - context.bus.remove_message(message) - return True, debug - context.bus.new_message_available.wait(0.5) - # Timed out return debug from test - return False, debug + matcher = VoightKampffCriteriaMatcher(context, msg_type, criteria_func) + match_found, error_message = matcher.match(timeout) + return match_found, error_message -def then_wait_fail(msg_type, criteria_func, context, timeout=None): + +def then_wait_fail(msg_type: str, criteria_func: Callable, context: Any, + timeout: int = None) -> Tuple[bool, str]: """Wait for a specified time, failing if criteria is fulfilled. Args: @@ -66,10 +350,12 @@ def then_wait_fail(msg_type, criteria_func, context, timeout=None): Returns: tuple (bool, str) test status and debug output """ - status, debug = then_wait(msg_type, criteria_func, context, timeout) - return (not status, debug) + match_found, error_message = then_wait(msg_type, criteria_func, + context, timeout) + return not match_found, error_message +# TODO: remove in 21.08 def mycroft_responses(context): """Collect and format mycroft responses from context. @@ -91,25 +377,63 @@ def mycroft_responses(context): return responses +# TODO: remove in 21.08 def print_mycroft_responses(context): print(mycroft_responses(context)) -def emit_utterance(bus, utt): - """Emit an utterance on the bus. +# TODO: remove in 21.08 +def format_dialog_match_error(potential_matches, speak_messages): + """Format error message to be displayed when an expected + + This is similar to the mycroft_responses function above. The difference + is that here the expected responses are passed in instead of making + a second loop through message bus messages. + + Args: + potential_matches (list): one of the dialog files in this list were + expected to be spoken + speak_messages (list): "speak" event messages from the message bus + that don't match the list of potential matches. + + Returns: (str) Message detailing the error to the user + """ + error_message = ( + 'Expected Mycroft to respond with one of:\n' + f"\t{', '.join(potential_matches)}\n" + "Actual response(s):\n" + ) + if speak_messages: + for message in speak_messages: + meta = message.data.get("meta") + if meta is not None: + if 'dialog' in meta: + error_message += f"\tDialog: {meta['dialog']}" + if 'skill' in meta: + error_message += f" (from {meta['skill']} skill)\n" + error_message += f"\t\tUtterance: {message.data['utterance']}\n" + else: + error_message += "\tMycroft didn't respond" + + return error_message + + +def emit_utterance(bus, utterance): + """Emit an utterance event on the message bus. Args: bus (InterceptAllBusClient): Bus instance to listen on - dialogs (list): list of acceptable dialogs + utterance (str): list of acceptable dialogs """ bus.emit(Message('recognizer_loop:utterance', - data={'utterances': [utt], + data={'utterances': [utterance], 'lang': 'en-us', 'session': '', 'ident': time.time()}, context={'client_name': 'mycroft_listener'})) +# TODO: remove in 21.08 def wait_for_dialog(bus, dialogs, context=None, timeout=None): """Wait for one of the dialogs given as argument. @@ -121,42 +445,63 @@ def wait_for_dialog(bus, dialogs, context=None, timeout=None): provided by context or 10 seconds """ if context: - timeout = timeout or context.step_timeout + timeout_duration = timeout or context.step_timeout else: - timeout = timeout or TIMEOUT - start_time = time.monotonic() - while time.monotonic() < start_time + timeout: + timeout_duration = timeout or DEFAULT_TIMEOUT + wait_for_dialog_match(bus, dialogs, timeout_duration) + + +# TODO: remove in 21.08 +def wait_for_dialog_match(bus, dialogs, timeout=DEFAULT_TIMEOUT): + """Match dialogs spoken to the specified list of expected dialogs. + + Only one of the dialogs in the provided list need to match for this + check to be successful. + + Args: + bus (InterceptAllBusClient): Bus instance to listen on + dialogs (list): list of acceptable dialogs + timeout (int): how long to wait for the message, defaults to timeout + provided by context or 10 seconds + + Returns: + A boolean indicating if a match was found and the list of "speak" + events found on the message bus during the matching process. + """ + match_found = False + speak_messages = list() + timeout_time = time.monotonic() + timeout + while time.monotonic() < timeout_time: for message in bus.get_messages('speak'): + speak_messages.append(message) dialog = message.data.get('meta', {}).get('dialog') if dialog in dialogs: - bus.clear_messages() - return - bus.new_message_available.wait(0.5) - bus.clear_messages() + wait_while_speaking() + match_found = True + break + bus.clear_messages() + if match_found: + break + time.sleep(1) + + return match_found, speak_messages -def wait_for_audio_service(context, message_type): +def wait_for_audio_service(context: Any, message_type: str): """Wait for audio.service message that matches type provided. May be play, stop, or pause messages Args: - context (behave Context): optional context providing scenario timeout - message_type (string): final component of bus message in form - `mycroft.audio.service.{type} + context: optional context providing scenario timeout + message_type: final component of bus message in form + mycroft.audio.service.{type} + + Raises: + AssertionError if no match is found. """ msg_type = 'mycroft.audio.service.{}'.format(message_type) + event_matcher = VoightKampffMessageMatcher(context, msg_type) + match_found, error_message = event_matcher.match() - def check_for_msg(message): - return (message.msg_type == msg_type, '') - - passed, debug = then_wait(msg_type, check_for_msg, context) - - if not passed: - debug += mycroft_responses(context) - if not debug: - if message_type == 'play': - message_type = 'start' - debug = "Mycroft didn't {} playback".format(message_type) - - assert passed, debug + assert match_found, error_message diff --git a/test/unittests/audio/test_speech.py b/test/unittests/audio/test_speech.py index 4236c46d4794..7354c7eca24e 100644 --- a/test/unittests/audio/test_speech.py +++ b/test/unittests/audio/test_speech.py @@ -23,6 +23,7 @@ import mycroft.audio.speech as speech from mycroft.messagebus import Message +from mycroft.tts.tts import default_preprocess_utterance from mycroft.tts.remote_tts import RemoteTTSTimeoutException """Tests for speech dispatch service.""" @@ -36,6 +37,8 @@ def setup_mocks(config_mock, tts_factory_mock): config_mock.get.return_value = {} tts_factory_mock.create.return_value = tts_mock + + tts_mock.preprocess_utterance.side_effect = default_preprocess_utterance config_mock.reset_mock() tts_factory_mock.reset_mock() tts_mock.reset_mock() diff --git a/test/unittests/configuration/test_configuration.py b/test/unittests/configuration/test_configuration.py index 77cb376f186a..dfe71271eb0c 100644 --- a/test/unittests/configuration/test_configuration.py +++ b/test/unittests/configuration/test_configuration.py @@ -32,12 +32,29 @@ def test_remote(self, mock_api): self.assertTrue(rc['test_config']) self.assertEqual(rc['location']['city']['name'], 'Stockholm') - @patch('json.dump') - @patch('mycroft.configuration.config.exists') - @patch('mycroft.configuration.config.isfile') - @patch('mycroft.configuration.config.load_commented_json') - def test_local(self, mock_json_loader, mock_isfile, mock_exists, - mock_json_dump): + @patch('mycroft.configuration.config.RemoteConf') + @patch('mycroft.configuration.config.LocalConf') + def test_update(self, mock_remote, mock_local): + mock_remote.return_value = {} + mock_local.return_value = {'a': 1} + c = mycroft.configuration.Configuration.get() + self.assertEqual(c, {'a': 1}) + + mock_local.return_value = {'a': 2} + mycroft.configuration.Configuration.updated('message') + self.assertEqual(c, {'a': 2}) + + def tearDown(self): + mycroft.configuration.Configuration.load_config_stack([{}], True) + + +@patch('mycroft.configuration.config.exists') +@patch('mycroft.configuration.config.isfile') +@patch('mycroft.configuration.config.load_commented_json') +class TestLocalConf(TestCase): + """Test cases for LocalConf class.""" + def test_create(self, mock_json_loader, mock_isfile, mock_exists): + """Test that initialization and creation works as expected.""" local_conf = {'answer': 42, 'falling_objects': ['flower pot', 'whale']} mock_exists.return_value = True mock_isfile.return_value = True @@ -45,13 +62,30 @@ def test_local(self, mock_json_loader, mock_isfile, mock_exists, lc = mycroft.configuration.LocalConf('test') self.assertEqual(lc, local_conf) - # Test merge method + def test_merge(self, mock_json_loader, mock_isfile, mock_exists): + """Check that configurations are merged correctly.""" + local_conf = {'answer': 42, 'falling_objects': ['flower pot', 'whale']} + mock_exists.return_value = True + mock_isfile.return_value = True + mock_json_loader.return_value = local_conf + lc = mycroft.configuration.LocalConf('test') + merge_conf = {'falling_objects': None, 'has_towel': True} + lc.merge(merge_conf) self.assertEqual(lc['falling_objects'], None) self.assertEqual(lc['has_towel'], True) - # test store + @patch('json.dump') + def test_store(self, mock_json_dump, mock_json_loader, mock_isfile, + mock_exists): + """Check that the config is stored correctly.""" + local_conf = {'answer': 42, 'falling_objects': ['flower pot', 'whale']} + mock_exists.return_value = True + mock_isfile.return_value = True + mock_json_loader.return_value = local_conf + lc = mycroft.configuration.LocalConf('test') + lc.store('test_conf.json') self.assertEqual(mock_json_dump.call_args[0][0], lc) # exists but is not file @@ -64,17 +98,19 @@ def test_local(self, mock_json_loader, mock_isfile, mock_exists, lc = mycroft.configuration.LocalConf('test') self.assertEqual(lc, {}) - @patch('mycroft.configuration.config.RemoteConf') - @patch('mycroft.configuration.config.LocalConf') - def test_update(self, mock_remote, mock_local): - mock_remote.return_value = {} - mock_local.return_value = {'a': 1} - c = mycroft.configuration.Configuration.get() - self.assertEqual(c, {'a': 1}) + @patch('json.dump') + def test_store_invalid(self, mock_json_dump, mock_json_loader, mock_isfile, + mock_exists): + """Storing shouldn't happen when config content is invalid.""" + mock_exists.return_value = True + mock_isfile.return_value = True - mock_local.return_value = {'a': 2} - mycroft.configuration.Configuration.updated('message') - self.assertEqual(c, {'a': 2}) + def raise_error(*arg, **kwarg): + raise Exception('Test exception') - def tearDown(self): - mycroft.configuration.Configuration.load_config_stack([{}], True) + mock_json_loader.side_effect = raise_error + + lc = mycroft.configuration.LocalConf('invalid') + self.assertFalse(lc.is_valid) + self.assertFalse(lc.store()) + mock_json_dump.assert_not_called() diff --git a/test/unittests/skills/test_common_query_skill.py b/test/unittests/skills/test_common_query_skill.py index 8c138ef33345..455154d0e9b5 100644 --- a/test/unittests/skills/test_common_query_skill.py +++ b/test/unittests/skills/test_common_query_skill.py @@ -2,8 +2,7 @@ from mycroft.messagebus import Message from mycroft.skills.common_query_skill import (CommonQuerySkill, CQSMatchLevel, - CQSVisualMatchLevel, - handles_visuals) + CQSVisualMatchLevel) from test.unittests.mocks import AnyCallable @@ -23,11 +22,6 @@ def test_lifecycle(self): bus.on.assert_any_call('question:action', AnyCallable()) skill.shutdown() - def test_handles_visuals(self): - """Test the helper method to determine if the skill handles visuals.""" - self.assertTrue(handles_visuals('mycroft_mark_2')) - self.assertFalse(handles_visuals('mycroft_mark_1')) - def test_common_test_skill_action(self): """Test that the optional action is triggered.""" query_action = self.bus.on.call_args_list[-1][0][1] @@ -44,6 +38,7 @@ def test_common_test_skill_action(self): class TestCommonQueryMatching(TestCase): """Tests for CQS_match_query_phrase.""" + def setUp(self): self.skill = CQSTest() self.bus = mock.Mock(name='bus') @@ -96,14 +91,13 @@ def test_successful_match_query_phrase(self): 'What\'s the meaning of life') self.assertEqual(response.data['skill_id'], self.skill.skill_id) self.assertEqual(response.data['answer'], '42') - self.assertEqual(response.data['conf'], 1.0) + self.assertEqual(response.data['conf'], 1.12) def test_successful_visual_match_query_phrase(self): - self.skill.config_core['enclosure']['platform'] = 'mycroft_mark_2' + self.skill.gui.connected = True query_phrase = self.bus.on.call_args_list[-2][0][1] self.skill.CQS_match_query_phrase.return_value = ( 'What\'s the meaning of life', CQSVisualMatchLevel.EXACT, '42') - query_phrase(Message('question:query', data={'phrase': 'What\'s the meaning of life'})) @@ -120,19 +114,27 @@ def test_successful_visual_match_query_phrase(self): 'What\'s the meaning of life') self.assertEqual(response.data['skill_id'], self.skill.skill_id) self.assertEqual(response.data['answer'], '42') - self.assertEqual(response.data['conf'], 1.1) + self.assertEqual(response.data['conf'], 1.2200000000000002) class CQSTest(CommonQuerySkill): """Simple skill for testing the CommonQuerySkill""" + def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.CQS_match_query_phrase = mock.Mock(name='match_phrase') self.CQS_action = mock.Mock(name='selected_action') self.skill_id = 'CQSTest' + self.gui = MockGUI() def CQS_match_query_phrase(self, phrase): pass def CQS_action(self, phrase, data): pass + + +class MockGUI(): + def __init__(self): + self.connected = False + self.setup_default_handlers = AnyCallable diff --git a/test/unittests/skills/test_intent_service.py b/test/unittests/skills/test_intent_service.py index 2363561f48ee..776c442926f4 100644 --- a/test/unittests/skills/test_intent_service.py +++ b/test/unittests/skills/test_intent_service.py @@ -214,12 +214,18 @@ def test_lang_exists(self): self.assertEqual(_get_message_lang(msg), 'sv-se') -def create_vocab_msg(keyword, value): +def create_old_style_vocab_msg(keyword, value): """Create a message for registering an adapt keyword.""" return Message('register_vocab', {'start': value, 'end': keyword}) +def create_vocab_msg(keyword, value): + """Create a message for registering an adapt keyword.""" + return Message('register_vocab', + {'entity_value': value, 'entity_type': keyword}) + + def get_last_message(bus): """Get last sent message on mock bus.""" last = bus.emit.call_args @@ -230,14 +236,27 @@ class TestIntentServiceApi(TestCase): def setUp(self): self.intent_service = IntentService(mock.Mock()) - def setup_simple_adapt_intent(self): - msg = create_vocab_msg('testKeyword', 'test') + def setup_simple_adapt_intent(self, + msg=create_vocab_msg('testKeyword', 'test')): self.intent_service.handle_register_vocab(msg) intent = IntentBuilder('skill:testIntent').require('testKeyword') msg = Message('register_intent', intent.__dict__) self.intent_service.handle_register_intent(msg) + def test_keyword_backwards_compatibility(self): + self.setup_simple_adapt_intent( + create_old_style_vocab_msg('testKeyword', 'test') + ) + + # Check that the intent is returned + msg = Message('intent.service.adapt.get', data={'utterance': 'test'}) + self.intent_service.handle_get_adapt(msg) + + reply = get_last_message(self.intent_service.bus) + self.assertEqual(reply.data['intent']['intent_type'], + 'skill:testIntent') + def test_get_adapt_intent(self): self.setup_simple_adapt_intent() # Check that the intent is returned @@ -300,8 +319,8 @@ def test_get_adapt_vocab_manifest(self): msg = Message('intent.service.adapt.vocab.manifest.get') self.intent_service.handle_vocab_manifest(msg) reply = get_last_message(self.intent_service.bus) - value = reply.data['vocab'][0]['start'] - keyword = reply.data['vocab'][0]['end'] + value = reply.data['vocab'][0]['entity_value'] + keyword = reply.data['vocab'][0]['entity_type'] self.assertEqual(keyword, 'testKeyword') self.assertEqual(value, 'test') diff --git a/test/unittests/skills/test_intent_service_interface.py b/test/unittests/skills/test_intent_service_interface.py index f5d60c9024b2..0645b86857f2 100644 --- a/test/unittests/skills/test_intent_service_interface.py +++ b/test/unittests/skills/test_intent_service_interface.py @@ -25,13 +25,15 @@ def reset(self): self.results = [] -class FunctionTest(unittest.TestCase): - def check_emitter(self, result_list): - for type in self.emitter.get_types(): - self.assertEqual(type, 'register_vocab') - self.assertEqual(sorted(self.emitter.get_results(), - key=lambda d: sorted(d.items())), - sorted(result_list, key=lambda d: sorted(d.items()))) +class KeywordRegistrationTest(unittest.TestCase): + def check_emitter(self, expected_message_data): + """Verify that the registration messages matches the expected.""" + for msg_type in self.emitter.get_types(): + self.assertEqual(msg_type, 'register_vocab') + self.assertEqual( + sorted(self.emitter.get_results(), + key=lambda d: sorted(d.items())), + sorted(expected_message_data, key=lambda d: sorted(d.items()))) self.emitter.reset() def setUp(self): @@ -40,18 +42,40 @@ def setUp(self): def test_register_keyword(self): intent_service = IntentServiceInterface(self.emitter) intent_service.register_adapt_keyword('test_intent', 'test') - self.check_emitter([{'start': 'test', 'end': 'test_intent'}]) + entity_data = {'entity_value': 'test', 'entity_type': 'test_intent'} + compatibility_data = {'start': 'test', 'end': 'test_intent'} + expected_data = {**entity_data, **compatibility_data} + self.check_emitter([expected_data]) def test_register_keyword_with_aliases(self): + # TODO 22.02: Remove compatibility data intent_service = IntentServiceInterface(self.emitter) intent_service.register_adapt_keyword('test_intent', 'test', ['test2', 'test3']) - self.check_emitter([{'start': 'test', 'end': 'test_intent'}, - {'start': 'test2', 'end': 'test_intent', - 'alias_of': 'test'}, - {'start': 'test3', 'end': 'test_intent', - 'alias_of': 'test'}, - ]) + + entity_data = {'entity_value': 'test', 'entity_type': 'test_intent'} + compatibility_data = {'start': 'test', 'end': 'test_intent'} + expected_initial_vocab = {**entity_data, **compatibility_data} + + alias_data = { + 'entity_value': 'test2', + 'entity_type': 'test_intent', + 'alias_of': 'test' + } + alias_compatibility = {'start': 'test2', 'end': 'test_intent'} + expected_alias1 = {**alias_data, **alias_compatibility} + + alias_data2 = { + 'entity_value': 'test3', + 'entity_type': 'test_intent', + 'alias_of': 'test' + } + alias_compatibility2 = {'start': 'test3', 'end': 'test_intent'} + expected_alias2 = {**alias_data2, **alias_compatibility2} + + self.check_emitter([expected_initial_vocab, + expected_alias1, + expected_alias2]) def test_register_regex(self): intent_service = IntentServiceInterface(self.emitter) diff --git a/test/unittests/skills/test_mycroft_skill.py b/test/unittests/skills/test_mycroft_skill.py index 0af99abf2899..bc60ca5c90a7 100644 --- a/test/unittests/skills/test_mycroft_skill.py +++ b/test/unittests/skills/test_mycroft_skill.py @@ -303,7 +303,14 @@ def test_register_vocab(self): # Normal vocaubulary self.emitter.reset() - expected = [{'start': 'hello', 'end': 'AHelloKeyword'}] + expected = [ + { + 'start': 'hello', + 'end': 'AHelloKeyword', + 'entity_value': 'hello', + 'entity_type': 'AHelloKeyword' + } + ] s.register_vocabulary('hello', 'HelloKeyword') self.check_register_vocabulary(expected) # Regex diff --git a/test/unittests/skills/test_skill_updater.py b/test/unittests/skills/test_skill_updater.py index 38d4083d7f2e..d7cc72e3e675 100644 --- a/test/unittests/skills/test_skill_updater.py +++ b/test/unittests/skills/test_skill_updater.py @@ -13,8 +13,9 @@ # limitations under the License. # """Unit tests for the SkillUpdater class.""" -from os import path +import os from time import sleep +from xdg import BaseDirectory from unittest.mock import Mock, patch, PropertyMock from mycroft.skills.skill_updater import SkillUpdater @@ -143,7 +144,8 @@ def test_installed_skills_path_not_virtual_env(self): os_patch.return_value = False updater = SkillUpdater(self.message_bus_mock) self.assertEqual( - path.expanduser('~/.mycroft/.mycroft-skills'), + os.path.join(BaseDirectory.save_data_path('mycroft'), + '.mycroft-skills'), updater.installed_skills_file_path ) diff --git a/test/unittests/tts/test_tts.py b/test/unittests/tts/test_tts.py index 2f4b562ccab9..5923234c0b5b 100644 --- a/test/unittests/tts/test_tts.py +++ b/test/unittests/tts/test_tts.py @@ -8,10 +8,18 @@ import mycroft.tts mock_phoneme = mock.Mock(name='phoneme') -mock_audio = mock.Mock(name='audio') +mock_audio = "/tmp/mock_path" mock_viseme = mock.Mock(name='viseme') +class MsgTypeCheck: + def __init__(self, msg_type): + self.msg_type = msg_type + + def __eq__(self, other): + return self.msg_type == other.msg_type + + class MockTTS(mycroft.tts.TTS): def __init__(self, lang, config, validator, audio_ext='wav', phonetic_spelling=True, ssml_tags=None): @@ -58,17 +66,20 @@ def test_process_queue(self, mock_play_mp3, mock_play_wav, mock_time): # Test wav data wav_mock = mock.Mock(name='wav_data') queue.put(('wav', wav_mock, None, 0, False)) - time.sleep(0.2) - mock_tts.begin_audio.called_with() + time.sleep(0.3) mock_play_wav.assert_called_with(wav_mock, environment=None) - mock_tts.end_audio.assert_called_with(False) + mock_tts.bus.emit.assert_called_with( + MsgTypeCheck('recognizer_loop:audio_output_end') + ) # Test mp3 data and trigger listening True mp3_mock = mock.Mock(name='mp3_data') queue.put(('mp3', mp3_mock, None, 0, True)) time.sleep(0.2) mock_play_mp3.assert_called_with(mp3_mock, environment=None) - mock_tts.end_audio.assert_called_with(True) + mock_tts.bus.emit.assert_called_with( + MsgTypeCheck('mycroft.mic.listen') + ) self.assertFalse(playback.enclosure.get.called) # Test sending visemes @@ -92,15 +103,43 @@ def test_execute(self, mock_playback_thread): tts.init(bus_mock) self.assertTrue(tts.bus is bus_mock) - tts.queue = mock.Mock() + mycroft.tts.TTS.queue = mock.Mock() + with mock.patch('mycroft.tts.tts.open') as mock_open: + tts.cache.temporary_cache_dir = Path('/tmp/dummy') + tts.execute('Oh no, not again', 42) + tts.get_tts.assert_called_with( + 'Oh no, not again', + '/tmp/dummy/8da7f22aeb16bc3846ad07b644d59359.wav' + ) + mycroft.tts.TTS.queue.put.assert_called_with( + ( + 'wav', + mock_audio, + mock_viseme, + 42, + False + ) + ) + + def test_execute_path_returned(self, mock_playback_thread): + tts = MockTTS("en-US", {}, MockTTSValidator(None)) + tts.get_tts.return_value = (Path(mock_audio), mock_viseme) + bus_mock = mock.Mock() + tts.init(bus_mock) + self.assertTrue(tts.bus is bus_mock) + + mycroft.tts.TTS.queue = mock.Mock() with mock.patch('mycroft.tts.tts.open') as mock_open: tts.cache.temporary_cache_dir = Path('/tmp/dummy') tts.execute('Oh no, not again', 42) - self.assertTrue(tts.get_tts.called) - tts.queue.put.assert_called_with( + tts.get_tts.assert_called_with( + 'Oh no, not again', + '/tmp/dummy/8da7f22aeb16bc3846ad07b644d59359.wav' + ) + mycroft.tts.TTS.queue.put.assert_called_with( ( 'wav', - '/tmp/dummy/8da7f22aeb16bc3846ad07b644d59359.wav', + mock_audio, mock_viseme, 42, False diff --git a/test/unittests/util/commented.json b/test/unittests/util/commented.json index 87c7d3e2c2db..b7fed8a81391 100644 --- a/test/unittests/util/commented.json +++ b/test/unittests/util/commented.json @@ -54,7 +54,7 @@ } }, "skills": { - "directory": "~/.mycroft/skills" + "directory": "~/.local/share/mycroft/skills" }, "server": { "url": "https://api.mycroft.ai", @@ -85,6 +85,7 @@ "test": false }, "log_level": "DEBUG", + "log_format": "{asctime} {levelname} {process} {name} {message}", "ignore_logs": ["enclosure.mouth.viseme"], "session": { "ttl": 180 diff --git a/test/unittests/util/plain.json b/test/unittests/util/plain.json index f8c474139ce8..746fc7bb40c9 100644 --- a/test/unittests/util/plain.json +++ b/test/unittests/util/plain.json @@ -36,7 +36,7 @@ } }, "skills": { - "directory": "~/.mycroft/skills" + "directory": "~/.local/share/mycroft/skills" }, "server": { "url": "https://api.mycroft.ai", @@ -67,6 +67,7 @@ "test": false }, "log_level": "DEBUG", + "log_format": "{asctime} {levelname} {process} {name} {message}", "ignore_logs": ["enclosure.mouth.viseme"], "session": { "ttl": 180 diff --git a/test/unittests/util/test_network_utils.py b/test/unittests/util/test_network_utils.py new file mode 100644 index 000000000000..76e86fe76289 --- /dev/null +++ b/test/unittests/util/test_network_utils.py @@ -0,0 +1,52 @@ +from unittest import TestCase, mock + +from mycroft.util.network_utils import connected + + +class TestNetworkConnected(TestCase): + def test_default_config_succeeds(self): + """Check that happy path succeeds""" + self.assertTrue(connected()) + + +@mock.patch('mycroft.configuration.Configuration') +class TestNetworkFailure(TestCase): + + def test_dns_and_ncsi_fail(self, mock_conf): + """Check that DNS and NCSI failure results in False response""" + mock_conf.get.return_value = { + "network_tests": { + "dns_primary": "127.0.0.1", + "dns_secondary": "127.0.0.1", + "web_url": "https://www.google.com", + "ncsi_endpoint": "http://www.msftncsi.com/ncsi.txt", + "ncsi_expected_text": "Unexpected text" + } + } + self.assertFalse(connected()) + + def test_secondary_dns_succeeds(self, mock_conf): + """Check that only primary DNS failing still succeeds""" + mock_conf.get.return_value = { + "network_tests": { + "dns_primary": "127.0.0.1", + "dns_secondary": "8.8.4.4", + "web_url": "https://www.google.com", + "ncsi_endpoint": "http://www.msftncsi.com/ncsi.txt", + "ncsi_expected_text": "Microsoft NCSI" + } + } + self.assertTrue(connected()) + + def test_dns_success_url_fail(self, mock_conf): + """Check that URL connection failure results in False response""" + mock_conf.get.return_value = { + "network_tests": { + "dns_primary": "8.8.8.8", + "dns_secondary": "8.8.4.4", + "web_url": "https://test.invalid", + "ncsi_endpoint": "http://www.msftncsi.com/ncsi.txt", + "ncsi_expected_text": "Microsoft NCSI" + } + } + self.assertFalse(connected()) diff --git a/test/wake_word/wake_word_test.py b/test/wake_word/wake_word_test.py index b91bd04aa2b1..2111c8e55167 100644 --- a/test/wake_word/wake_word_test.py +++ b/test/wake_word/wake_word_test.py @@ -89,6 +89,9 @@ def unmute(self): def close(self): self.stream.close() + def duration_to_bytes(self, ww_duration): + return int(ww_duration * self.SAMPLE_RATE * self.SAMPLE_WIDTH) + class AudioTester: def __init__(self, samp_rate): diff --git a/venv-activate.sh b/venv-activate.sh index 24d381696a47..a7f81e8398f3 100644 --- a/venv-activate.sh +++ b/venv-activate.sh @@ -45,21 +45,23 @@ function main() { ;; *) - echo "ERROR: Unrecognized option: $@" + echo "ERROR: Unrecognized option: $arg" return 1 ;; esac done - if [[ "$0" == "$BASH_SOURCE" ]] ; then + if [[ "$0" == "${BASH_SOURCE[0]}" ]] ; then # Prevent running in script then exiting immediately echo "ERROR: Invoke with 'source venv-activate.sh' or '. venv-activate.sh'" else - local SRC_DIR="$( builtin cd "$( dirname "${BASH_SOURCE}" )" ; pwd -P )" - source ${SRC_DIR}/.venv/bin/activate + local SRC_DIR + SRC_DIR="$( builtin cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1; pwd -P )" + source "${SRC_DIR}/.venv/bin/activate" # Provide an easier to find "mycroft-" prefixed command. unalias mycroft-venv-activate 2>/dev/null + # shellcheck disable=SC2139 # The intention _is_ to resolve the variable at define time alias mycroft-venv-deactivate="deactivate && unalias mycroft-venv-deactivate 2>/dev/null && alias mycroft-venv-activate=\"source '${SRC_DIR}/venv-activate.sh'\"" if [ $quiet -eq 0 ] ; then echo "Entering Mycroft virtual environment. Run 'mycroft-venv-deactivate' to exit" @@ -67,4 +69,4 @@ function main() { fi } -main $@ +main "$@"