We're not using avocado anymore, so while the TODO item is still
relevant, suggesting use of avocado.utils is not.

Reviewed-by: Alex Bennée <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

These tags are not honoured under the new functional test harness.

Reviewed-by: Alex Bennée <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Historical bugs in avocado related to zstd support are not relevant to
the code now that it uses QEMU's native test harness.

Reviewed-by: Alex Bennée <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The first comment is still relevant but should talk about our own test
harness instead. The second comment adds no value over reading the code
and can be removed.

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Pierrick Bouvier <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The ACPI bits test sets up its own private temporary directory into it
creates scratch files. This is justified by a suggestion that we need
to be able to preserve the scratch files. We have the ability to
preserve the scratch dir with our functional harness, so there's no
reason to diverge from standard practice in file placement.

Reviewed-by: Alex Bennée <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

We are not passing the 'log_dir' parameter to QEMUMachine, so the
QEMU stdout/err logs are being placed in a temp directory and thus
deleted after execution. This makes them inaccessible as gitlab
CI artifacts.

Pass the testcase log directory path into QEMUMachine to make the
logs persistent.

Reviewed-by: Alex Bennée <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The functional test case class is going to the trouble of passing
around a machine name, but then fails to give this QEMUMachine. As
a result, QEMUMachine will create a completely random name. Since
log file names match the machine name, this results in log files
accumulating over time.

Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Set the 'qemu.machine' logger to 'DEBUG' level, to ensure we see log
messages related to the QEMUMachine class. Most importantly this
ensures we capture the full QEMU command line args for instances we
spawn.

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Tested-by: Thomas Huth <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Message-Id: <[email protected]>

When functional tests go wrong, it will often be related to the console
interaction wait state. By logging the messages that we're looking for,
and data we're about to be sending, it'll be easier to diagnose where
tests are getting stuck.

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Telling exec_command_wand_wait_for_pattern to wait for the empty
string does not make any conceptual sense, as a check for empty
string will always succeed. It makes even less sense when followed
by a call to wait_for_console_pattern() with a real match.

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

When waiting for expected output, the 'success_message' is a mandatory
parameter, with 'failure_message' defaulting to None.

The code has logic which indicates it was trying to cope with
'success_message' being None and 'failure_message' being non-None but
it does not appear able to actually do anything useful. The check for
'success_message is None' will break out of the loop before any check
for 'failure_message' has been performed.

IOW, for practcal purposes 'success_message' must be non-None unless
'send_string' is set. Assert this expectation and simplify the loop
logic.

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The console interaction that waits for predicted strings uses
readline(), and thus is only capable of waiting for strings
that are followed by a newline.

This is inconvenient when needing to match on some things,
particularly login prompts, or shell prompts, causing tests
to use time.sleep(...) instead, which is unreliable.

Switch to reading the console 1 byte at a time, comparing
against the success/failure messages until we see a match,
regardless of whether a newline is encountered.

The success/failure comparisons are done with the python bytes
type, rather than strings, to avoid the problem of needing to
decode partially received multibyte utf8 characters.

Heavily inspired by a patch proposed by Cédric, but written
again to work in bytes, rather than strings.

Co-developed-by: Cédric Le Goater <[email protected]>
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The tuxrun tests send a series of strings to the guest to login
and then run commands. Since we have been unable to match on
console output that isn't followed by a newline, the test used
many time.sleep() statements to pretend to synchronize with
the guest.

This has proved to be unreliable for the aarch64be instance of
the tuxrun tests, with the test often hanging. The hang is a
very subtle timing problem, and it is suspected that some
(otherwise apparently harmless) I/O error messages could be
resulting in full FIFO buffers, stalling interaction with
the guest.

With the newly rewritten console interaction able to match
strings that don't have a following newline, the tux run
tests can now match directly on the login prompt, and/or
shell PS1 prompt.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2689
Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:

 $ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
   QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
   PYTHONPATH=./python \
   ./tests/functional/test_arm_tuxrun.py

And then, when the test stalls, in a second shell run:

 $ ./scripts/qmp/qmp-shell backdoor.sock

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

If a failure occurs early in the QemuBaseTest constructor, the
'log_filename' object atttribute may not exist yet. This happens
most notably if the QEMU_TEST_QEMU_BINARY is not set. We can't
initialize 'log_filename' earlier as we use the binary to identify
the architecture which is then used to build the path in which the
logs are stored.

Signed-off-by: Daniel P. Berrangé <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Move the tests to a new file so that they can be run via
qemu-system-aarch64 in the functional framework.

Since these were the last tests in tests/avocado/tuxrun_baselines.py,
we can now remove that file, too.

Signed-off-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Now there are new up to date images available we should update to them.
With the new rootfs the blk I/O errors also go away on arm64be.

Cc: Anders Roxell <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Fixes: 4a448b1 ("plugins: add qemu_plugin_num_vcpus function")
Reviewed-by: Philippe Mathieu-Daudé <[email protected]>
Signed-off-by: Pierrick Bouvier <[email protected]>
Message-Id: <[email protected]>
Reviewed-by: Richard Henderson <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Instead of using a static file (error prone and hard to keep in sync),
we generate it using a script.

Note: if a symbol is not exported, we'll now notice it when linking for
Windows/MacOS platforms.

Signed-off-by: Pierrick Bouvier <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Signed-off-by: Pierrick Bouvier <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The peripheral and PrimeCell identification registers of pl011 are located at
offset 0xFE0 - 0xFFC. To check if a read falls to such registers, the C
implementation checks if the offset-shifted-by-2 (not the offset itself) is in
the range 0x3F8 - 0x3FF.

Use the same check in the Rust implementation.

This fixes the timeout of the following avocado tests:

  * tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_virt
  * tests/avocado/replay_kernel.py:ReplayKernelNormal.test_arm_virt
  * tests/avocado/replay_kernel.py:ReplayKernelNormal.test_arm_vexpressa9

Reported-by: Peter Maydell <[email protected]>
Signed-off-by: Junjie Mao <[email protected]>
Tested-by: Alex Bennée <[email protected]>
Reviewed-by: Alex Bennée <[email protected]>
Message-Id: <SY0P300MB102644C4AC34A3AAD75DC4D5955C2@SY0P300MB1026.AUSP300.PROD.OUTLOOK.COM>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

The docs for submitting a patch describe using your "Real Name" with
the Signed-off-by line. Although somewhat ambiguous, this has often
been interpreted to mean someone's legal name.

In recent times, there's been a general push back[1] against the notion
that use of Signed-off-by in a project automatically requires / implies
the use of legal ("real") names and greater awareness of the downsides.

Full discussion of the problems of such policies is beyond the scope of
this commit message, but at a high level they are liable to marginalize,
disadvantage, and potentially result in harm, to contributors.

TL;DR: there are compelling reasons for a person to choose distinct
identities in different contexts & a decision to override that choice
should not be taken lightly.

A number of key projects have responded to the issues raised by making
it clear that a contributor is free to determine the identity used in
SoB lines:

 * Linux has clarified[2] that they merely expect use of the
   contributor's "known identity", removing the previous explicit
   rejection of pseudonyms.

 * CNCF has clarified[3] that the real name is simply the identity
   the contributor chooses to use in the context of the community
   and does not have to be a legal name, nor birth name, nor appear
   on any government ID.

Since we have no intention of ever routinely checking any form of ID
documents for contributors[4], realistically we have no way of knowing
anything about the name they are using, except through chance, or
through the contributor volunteering the information. IOW, we almost
certainly already have people using pseudonyms for contributions.

This proposes to accept that reality and eliminate unnecessary friction,
by following Linux & the CNCF in merely asking that a contributors'
commonly known identity, of their choosing, be used with the SoB line.

[1] Raised in many contexts at many times, but a decent overall summary
    can be read at https://drewdevault.com/2023/10/31/On-real-names.html
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d4563201f33a022fc0353033d9dfeb1606a88330
[3] https://github.com/cncf/foundation/blob/659fd32c86dc/dco-guidelines.md
[4] Excluding the rare GPG key signing parties for regular maintainers

Signed-off-by: Daniel P. Berrangé <[email protected]>
Acked-by: Stefan Hajnoczi <[email protected]>
Acked-by: Peter Maydell <[email protected]>
Reviewed-by: Philippe Mathieu-Daudé <[email protected]>
Reviewed-by: Alex Bennée <[email protected]>
Acked-by: Richard Henderson <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>
Message-Id: <[email protected]>

Drop the SSH connection which was introduced in the avocado tests to
workaround read issues when interacting with console.

Signed-off-by: Cédric Le Goater <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>

Drop the SSH connection which was introduced in the avocado tests to
workaround read issues when interacting with console.

EXTRA_BOOTARGS was introduced to reduce the console output at Linux
boot time. This didn't have the desired effect as we still had issues
when trying to match patterns on the console and we had to use the ssh
connection as a workaround.

While at it, remove the U-Boot EXTRA_BOOTARGS variable which has
become useless.

Signed-off-by: Cédric Le Goater <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>

These were introduced in the avocado tests to workaround read issues
when interacting with console. They are no longer necessary and we can
use the expected "login:" string or the command prompt now. Drop the
last use of exec_command.

Signed-off-by: Cédric Le Goater <[email protected]>
Reviewed-by: Thomas Huth <[email protected]>
Message-Id: <[email protected]>
Signed-off-by: Alex Bennée <[email protected]>

… staging

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCAAdFiEEIV1G9IJGaJ7HfzVi7wSWWzmNYhEFAmdEEtgACgkQ7wSWWzmN
# YhH5qAgAlKdcx/gFt4EBXtjVq/qbPluEGOQxvcRYLlN90rPLHPgCjAoT5ly3fIv1
# 4kCgcVZyG8SdGu1n0TzTTS9kg5tL7weQ9xEWwF0oyyuZABgAB7w/wpC8MHSkJFOn
# 2Tv+2Iab0dJ+e1pw71OMpE/YR5X2xq5vopsSHRtnyGWfRPGswJFwka+f8FS5DSiq
# 2CeNxADgTkPxJgDmOrNSsAPz8Rns77FAZdvDMqFjx1Lrqm8kPv9jzwOMO+a/2LpC
# t6OkpFzGjiiskPjSnSn/tzo4TfWYoABjJaI7b3vEqmNEJSTAaxltZNtSXZucctEt
# 1ihnFdjr/wPwGK/5Wu+qGnfDbFNxBw==
# =W4y1
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 25 Nov 2024 06:02:00 GMT
# gpg:                using RSA key 215D46F48246689EC77F3562EF04965B398D6211
# gpg: Good signature from "Jason Wang (Jason Wang on RedHat) <[email protected]>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 215D 46F4 8246 689E C77F  3562 EF04 965B 398D 6211

* tag 'net-pull-request' of https://github.com/jasowang/qemu:
  virtio-net: Copy received header to buffer
  virtio-net: Initialize hash reporting values
  virtio-net: Fix hash reporting when the queue changes
  virtio-net: Do not check for the queue before RSS
  virtio-net: Fix size check in dhclient workaround
  net: checksum: Convert data to void *

Signed-off-by: Peter Maydell <[email protected]>

Block layer patches

- Fix qmp_device_add() to not throw non-scalar options away (fixes
  iothread-vq-mapping being silently ignored in device_add)
- Fix qdev property crash with integer PCI addresses and JSON -device
- iotests: Fix mypy failure
- parallels: Avoid potential integer overflow
- ssh: libssh broke with non-blocking sessions, use a blocking one for now
- Fix crash in migration_is_running()

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAmdES74RHGt3b2xmQHJl
# ZGhhdC5jb20ACgkQfwmycsiPL9abFQ//fGmBl3Jp99GWB1R0y49/hPhfj0166UGj
# zeEmhdy+k6gKywyhVy0Fj0xLztDTb/2bGflrwtDDxYA0PBLel461QSeJUzwDsn9h
# ZGLyFrosXgIPADP55RF1wJ6c+m13MX4jVy80Neh2jemhinMazjj3ADb4RdCf0B4M
# XoYOy96goDFPlzZNvr08dlaDvJaD5QmPYX8nK7TaZqZOSYvdSRWMuB+QQCPj+qEf
# UfpBo3beNsxedNu/1wKS1Nc6FVX7VHKoMzhDLAvxkYMBKcCg9l5lEAGrgp61O+79
# nYZmPtEG5RHsMNBCZtk8zZMIHPg2Ydxpj3jOV3eA0rF4Twk/fPrOOfBEUHT6PapX
# tCS1UJtgyQA2GTULiax3vKV4yBSpmUzbhjddNwBkW7uG1md67d17nqbjkEhHVxZL
# yMuauFRCx5onzE0TSgTYEMAmAgD9oawuGUqBiNCOqJlTbGZwJ9l7jtwP4Bl1gskk
# pWzL/PLP8MkVf50dcP0QBPNHn85/oZOwv5yNr2Z893qNQhh/0xqCEFwqSq2SJOkg
# vKd/bAusgmicoh1XD0o0+mv2ewZor/JghrU83YDPKWM1MmOwePZ8wRTx9pJtZWvq
# Pnc71397zppHIw7aIWKYDoyQ3aeaoTM/oY2Q5Y7et6c/FvGW5JtFjsPCGbgm9mw+
# +6JA51ujtYU=
# =oLdE
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 25 Nov 2024 10:04:46 GMT
# gpg:                using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg:                issuer "[email protected]"
# gpg: Good signature from "Kevin Wolf <[email protected]>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* tag 'for-upstream' of https://repo.or.cz/qemu/kevin:
  ssh: Do not switch session to non-blocking mode
  vl: use qmp_device_add() in qemu_create_cli_devices()
  qdev-monitor: avoid QemuOpts in QMP device_add
  tests/avocado/hotplug_blk: Fix addr in device_add command
  qdev: Fix set_pci_devfn() to visit option only once
  python: silence pylint raising-non-exception error
  python: disable too-many-positional-arguments warning
  iotests: correct resultclass type in ReproducibleTestRunner
  iotests: reflow ReproducibleTestRunner arguments
  parallels: fix possible int overflow

Signed-off-by: Peter Maydell <[email protected]>

Libvirt may still use pipes for old file migrations in fd: URI form,
especially when loading old images dumped from Libvirt's compression
algorithms.

In that case, Libvirt needs to compress / uncompress the images on its own
over the migration binary stream, and pipes are passed over to QEMU for
outgoing / incoming migrations in "fd:" URIs.

For future such use case, it should be suggested to use mapped-ram when
saving such VM image.  However there can still be old images that was
compressed in such way, so libvirt needs to be able to load those images,
uncompress them and use the same pipe mechanism to pass that over to QEMU.

It means, even if new file migrations can be gradually moved over to
mapped-ram (after Libvirt start supporting it), Libvirt still needs the
uncompressor for the old images to be able to load like before.

Meanwhile since Libvirt currently exposes the compression capability to
guest images, it may needs its own lifecycle management to move that over
to mapped-ram, maybe can be done after mapped-ram saved the image, however
Dan and PeterK raised concern on temporary double disk space consumption.
I suppose for now the easiest is to enable pipes for both sides of "fd:"
migrations, until all things figured out from Libvirt side on how to move
on.

And for "channels" QMP interface support on "migrate" / "migrate-incoming"
commands, we'll also need to move away from pipe.  But let's leave that for
later too.

So far, still allow pipes to happen like before on both save/load sides,
just like we would allow sockets to pass.

Cc: qemu-stable <[email protected]>
Cc: Fabiano Rosas <[email protected]>
Cc: Peter Krempa <[email protected]>
Cc: Daniel P. Berrangé <[email protected]>
Fixes: c55deb8 ("migration: Deprecate fd: for file migration")
Reviewed-by: Fabiano Rosas <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Peter Xu <[email protected]>

After fixing the loadvm cleanup race the qemu_loadvm_state_cleanup()
is now being called twice in the postcopy listen thread.

Fixes: 4ce5622 ("migration/multifd: Fix rb->receivedmap cleanup race")
Signed-off-by: Fabiano Rosas <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
Signed-off-by: Peter Xu <[email protected]>

Call virtio_net_set_multiqueue() to add queues before loading their
states. Otherwise the loaded queues will not have handlers and elements
in them will not be processed.

Cc: [email protected]
Fixes: 8c49756 ("virtio-net: Add only one queue pair when realizing")
Reported-by: Laurent Vivier <[email protected]>
Signed-off-by: Akihiko Odaki <[email protected]>
Acked-by: Michael S. Tsirkin <[email protected]>
Signed-off-by: Jason Wang <[email protected]>

This is NOT an iCloud bypass. This is utilising code that ALREADY exists
in the activation daemon. This is essentially telling iOS, it's a
development kernel/device, NOT the real product sold on market. IF you
decide to use this knowledge to BYPASS technological countermeasures
or any other intellectual theft or crime, YOU are responsible in full,
AND SHOULD BE PROSECUTED TO THE FULL EXTENT OF THE LAW.
We do NOT endorse nor approve the theft of property.

translation is not 100% accurate but at least you can actually interact now.

…o me: don't try to fix things that aren't broken)