Stable Release Update Certification Coverage for Ubuntu Server

Introduction

This document lists the coverage of regression tests performed for validating a Stable Release Update (SRU) on Ubuntu Server Certified systems. The test plan is not recommended for, nor will it be accepted for self-testing purposes.

For each test job, one of the following certification statuses is specified:

Blocking

Features that are required for certification. If any of the blocking tests fails, the certification will fail.

Non-blocking

Features that are tested but not mandatory for certification. Failure in non-blocking tests will not prevent certification. However, a note will be added to the certificate to inform potential customers or users.

Note

Only categories of hardware are tested and not specific types of hardware. For example, tests are run to verify USB controllers work, but the type of peripheral(s) used during those tests are not specified.

Coverage is flexible based on customer requirements (for example, if a device’s use cases don’t require LEDs, then LEDs can be non-blocking)

Certain test jobs are designed to validate specific hardware capabilities, such as camera and audio playback functionality. To ensure that the required hardware capabilities are present and properly recognised on the machine under test, these features are explicitly defined in manifest entries and linked to the relevant test jobs. This prevents test jobs from being skipped due to system deficiencies in automated detection.

Full test descriptions can be found in Canonical certification site for partners:

https://certification.canonical.com

sru-server

Note

The certification tests presented in this document are validated by Checkbox version 4.3.0.dev36.

Blocking

Bluetooth tests

The following test units are covered in this category:

bluetooth/bluetooth_obex_send

Bluetooth OBEX send

Category ID:

bluetooth
Status:

Blocking
Purpose:

This is an automated Bluetooth file transfer test. It sends an image to the device specified by the BTDEVADDR environment variable
After-suspend:

True
Environment variable:

BTDEVADDR, PLAINBOX_PROVIDER_DATA
Plugin:

shell
Requires:

package.name == ‘bluez’ or snap.name == ‘bluez’ executable.name == ‘obexftp’ and executable.name == ‘hcitool’ device.category == ‘BLUETOOTH’ manifest.has_bt_obex_support == ‘True’
For details about the required manifest entry, see has_bt_obex_support.
bluetooth/detect-output

Store Bluetooth device information for reports.

Category ID:

bluetooth
Status:

Blocking
Purpose:

Automated test to store Bluetooth device information in the Checkbox report
After-suspend:

True
Plugin:

shell
Requires:

package.name == ‘bluez’ or snap.name == ‘bluez’ device.category == ‘BLUETOOTH’
bluetooth4/beacon_eddystone_url_interface

Test system can get beacon EddyStone URL advertisements on the {interface} adapter

Unit type:

template
Category ID:

bluetooth
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
After-suspend:

True
From template:

bluetooth4/beacon_eddystone_url_interface
Template resource:

device
Template filter:

device.category == ‘BLUETOOTH’

Camera tests

The following test units are covered in this category:

camera/detect

This Automated test attempts to detect a camera.

Category ID:

camera
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
After-suspend:

True
User:

root
Plugin:

shell
Requires:

manifest.has_camera == ‘True’
For details about the required manifest entry, see has_camera.
camera/multiple-resolution-images_name

Webcam multiple resolution capture test for {product_slug}

Unit type:

template
Category ID:

camera
Status:

Blocking
Purpose:

Takes multiple pictures based on the resolutions supported by the camera and validates their size and that they are of a valid format.
After-suspend:

True
From template:

camera/multiple-resolution-images_name
Template resource:

device
Template filter:

device.category == ‘CAPTURE’ and device.name != ‘’

CPU tests

The following test units are covered in this category:

cpu/cstates

Run C-States tests

Category ID:

cpu
Status:

Blocking
Purpose:

<missing purpose>
Description:

Uses the Firmware Test Suite (fwts) to test the power saving states of the CPU.
Environment variable:

LD_LIBRARY_PATH, PLAINBOX_SESSION_SHARE, SNAP
User:

root
Plugin:

shell
Requires:

executable.name == ‘fwts’ cpuinfo.platform not in (“aarch64”, “armv7l”, “s390x”)
cpu/maxfreq_test

Test that the CPU can run at its max frequency

Category ID:

cpu
Status:

Blocking
Purpose:

<missing purpose>
Description:

Use the Firmware Test Suite (fwts cpufreq) to ensure that the CPU can run at its maximum frequency.
Environment variable:

LD_LIBRARY_PATH, SNAP
User:

root
Plugin:

shell
Requires:

executable.name == ‘fwts’ cpuinfo.platform in (“i386”, “x86_64”)
cpu/offlining_test

Test offlining of each CPU core

Category ID:

cpu
Status:

Blocking
Purpose:

Attempts to offline each core in a multicore system.
After-suspend:

True
User:

root
Plugin:

shell
Requires:

cpu_offlining.state == ‘supported’
cpu/scaling_test

Test the CPU scaling capabilities

Category ID:

cpu
Status:

Blocking
Purpose:

<missing purpose>
Description:

Use Firmware Test Suite (fwts cpufreq) to test the scaling capabilities of the CPU.
Environment variable:

LD_LIBRARY_PATH, PLAINBOX_SESSION_SHARE, SNAP
User:

root
Plugin:

shell
Requires:

executable.name == ‘fwts’ ‘userspace’ in cpuinfo.governors cpuinfo.platform not in (“ppc64el”, “s390x”)
cpu/topology

Check CPU topology for accuracy between proc and sysfs

Category ID:

cpu
Status:

Blocking
Purpose:

<missing purpose>
Description:

Parses information about CPU topology provided by proc and sysfs and checks that they are consistent.
After-suspend:

True
Plugin:

shell
Requires:

int(cpuinfo.count) > 1 and (cpuinfo.platform == ‘i386’ or cpuinfo.platform == ‘x86_64’)

Disk tests

The following test units are covered in this category:

disk/detect

Gathers information about each disk detected

Category ID:

disk
Status:

Blocking
Purpose:

Uses lsblk to gather information about each disk detected on the system under test.
After-suspend:

True
Plugin:

shell
Requires:

executable.name == ‘lsblk’
disk/read_performance_name

Disk performance test for {product_slug}

Unit type:

template
Category ID:

disk
Status:

Blocking
Purpose:

Verify that disk storage performs at or above baseline performance
After-suspend:

True
From template:

disk/read_performance_name
Template resource:

device
Template filter:

device.category == ‘DISK’

Ethernet Device tests

The following test units are covered in this category:

ethernet/detect

Detect if at least one ethernet device is detected

Category ID:

ethernet
Status:

Blocking
Purpose:

Test to detect and return information about available network controllers on the system under test.
After-suspend:

True
Plugin:

shell
Requires:

manifest.has_ethernet_adapter == ‘True’
For details about the required manifest entry, see has_ethernet_adapter.

Memory tests

The following test units are covered in this category:

memory/info

Check the amount of memory reported by meminfo against DMI

Category ID:

memory
Status:

Blocking
Purpose:

This test checks the amount of memory which is reported in meminfo against the size of the memory modules detected by DMI.
User:

root
Plugin:

shell

Miscellaneous tests

The following test units are covered in this category:

miscellanea/grub_file_check

Check if the file core.efi exists to make sure shim and grub can be upgraded

Category ID:

miscellanea
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
Requires:

“Ubuntu Core” not in lsb.description cpuinfo.platform in (“x86_64”, “aarch64”, “armhf”) bootloader.name == “grub”
miscellanea/oops

Run FWTS OOPS check

Category ID:

miscellanea
Status:

Blocking
Purpose:

<missing purpose>
Description:

Run Firmware Test Suite (FWTS) oops tests.
Environment variable:

PLAINBOX_SESSION_SHARE
User:

root
Plugin:

shell
Requires:

executable.name == ‘fwts’

Non-device specific networking tests

The following test units are covered in this category:

networking/gateway_ping

Verifies if the system can connect to the Internet.

Category ID:

networking
Status:

Blocking
Purpose:

Tests whether the system has a working Internet connection.
After-suspend:

True
User:

root
Plugin:

shell
networking/ntp

Test NTP server synchronization capability.

Category ID:

networking
Status:

Blocking
Purpose:

Test to see if we can sync local clock to an NTP server
After-suspend:

True
User:

root
Plugin:

shell
Requires:

package.name == ‘ntpdate’

Optical Drive tests

The following test units are covered in this category:

optical/detect

Displays discovered optical drives

Category ID:

optical
Status:

Blocking
Purpose:

Detects optical drives (CD/DVD) attached to the system.
Plugin:

shell
Requires:

device.category == ‘CDROM’
optical/read-automated_name

Tests read functionality of optical drive {name}

Unit type:

template
Category ID:

optical
Status:

Blocking
Purpose:

This is an automated version of optical/read. It assumes you have already inserted a data CD or Blu-Ray disc into your optical drive prior to running Checkbox.
After-suspend:

True
From template:

optical/read-automated_name
Template resource:

device
Template filter:

device.category == ‘CDROM’

Power Management tests

The following test units are covered in this category:

power-management/cold-reboot

Cold reboot

Category ID:

power-management
Status:

Blocking
Purpose:

This test powers off the system and then powers it on using RTC
Environment variable:

COLD_REBOOT_DELAY, RTC_DEVICE_FILE
User:

root
Plugin:

shell
Requires:

rtc.state == ‘supported’ rtc.wakealarm == ‘supported’
power-management/fwts_wakealarm

Executes ACPI Wakealarm test to validate functionality.

Category ID:

power-management
Status:

Blocking
Purpose:

Test ACPI Wakealarm (fwts wakealarm)
Environment variable:

PLAINBOX_SESSION_SHARE
User:

root
Plugin:

shell
Requires:

executable.name == ‘fwts’
power-management/post-cold-reboot

Post cold reboot service check

Category ID:

power-management
Status:

Blocking
Purpose:

Check there are no failed services after the cold reboot
Plugin:

shell
power-management/post-warm-reboot

Post warm reboot service check

Category ID:

power-management
Status:

Blocking
Purpose:

Check there are no failed services after the warm reboot
Plugin:

shell
power-management/rtc

Test that RTC functions properly (if present)

Category ID:

power-management
Status:

Blocking
Purpose:

Verify that the Real-time clock (RTC) device functions properly, if present.
After-suspend:

True
Environment variable:

RTC_DEVICE_FILE
User:

root
Plugin:

shell
Requires:

rtc.state == ‘supported’ cpuinfo.other != ‘emulated by qemu’
power-management/tickless_idle

Verify the tickless idle feature configuration in the kernel.

Category ID:

power-management
Status:

Blocking
Purpose:

Check to see if CONFIG_NO_HZ is set in the kernel (this is just a simple regression check)
After-suspend:

True
Plugin:

shell
Requires:

cpuinfo.platform in (‘i386’, ‘x86_64’, ‘ppc64el’, ‘pSeries’)
power-management/warm-reboot

Warm reboot

Category ID:

power-management
Status:

Blocking
Purpose:

This tests reboots the system using the reboot command
User:

root
Plugin:

shell

Suspend tests

The following test units are covered in this category:

suspend/audio_after_suspend_auto

Verify mixer settings consistency after system suspend.

Category ID:

suspend
Status:

Blocking
Purpose:

Verify that mixer settings after suspend are the same as before suspend.
Plugin:

shell
Requires:

manifest.has_audio_playback == ‘True’ package.name == ‘alsa-base’ package.name in [‘pulseaudio-utils’, ‘pipewire’]
For details about the required manifest entry, see has_audio_playback.
suspend/audio_before_suspend

Record mixer settings before suspending.

Category ID:

suspend
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
Requires:

manifest.has_audio_playback == ‘True’ package.name == ‘alsa-base’ package.name in [‘pulseaudio-utils’, ‘pipewire’]
For details about the required manifest entry, see has_audio_playback.
suspend/cpu_after_suspend_auto

Ensure all CPUs are online post-resumption.

Category ID:

suspend
Status:

Blocking
Purpose:

Verify that all CPUs are online after resuming.
Plugin:

shell
suspend/cpu_before_suspend

Verify that all the CPUs are online before suspending

Category ID:

suspend
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
suspend/memory_after_suspend_auto

Ensure all memory is accessible after waking from suspend mode.

Category ID:

suspend
Status:

Blocking
Purpose:

Verify that all memory is available after resuming from suspend.
Plugin:

shell
suspend/memory_before_suspend

Dumps memory info to a file for comparison after suspend

Category ID:

suspend
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
suspend/network_after_suspend_auto

Verify the network functionality after system resume.

Category ID:

suspend
Status:

Blocking
Purpose:

Test the network after resuming.
User:

root
Plugin:

shell
suspend/network_before_suspend

Record the current network before suspending.

Category ID:

suspend
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
User:

root
Plugin:

shell

Touchpad tests

The following test units are covered in this category:

touchpad/detected-as-mouse

Check if the touchpad is recognized as a mouse by the system.

Category ID:

touchpad
Status:

Blocking
Purpose:

This test will check if your touchpad was detected as a mouse.
After-suspend:

True
Plugin:

shell
Requires:

manifest.has_touchpad == ‘True’
For details about the required manifest entry, see has_touchpad.

TPM 2.0 (Trusted Platform Module)

The following test units are covered in this category:

clevis-encrypt-tpm2/detect-ecc-capabilities

Ensure the TPM has required capabilities for clevis ECC test

Category ID:

tpm2
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
User:

root
Plugin:

shell
clevis-encrypt-tpm2/detect-rsa-capabilities

Ensure the TPM has required capabilities for clevis RSA test

Category ID:

tpm2
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
User:

root
Plugin:

shell
clevis-encrypt-tpm2/ecc

clevis encrypt/decrypt key ecc

Category ID:

tpm2
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
User:

root
Plugin:

shell
clevis-encrypt-tpm2/precheck

clevis encrypt/decrypt precheck

Category ID:

tpm2
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
Requires:

manifest.has_tpm2_chip == ‘True’ executable.name == ‘clevis-encrypt-tpm2’
For details about the required manifest entry, see has_tpm2_chip.
clevis-encrypt-tpm2/rsa

clevis encrypt/decrypt key rsa

Category ID:

tpm2
Status:

Blocking
Purpose:

<missing purpose>
Description:

<missing description>
User:

root
Plugin:

shell

USB tests

The following test units are covered in this category:

usb/detect

Display USB devices attached to SUT

Category ID:

usb
Status:

Blocking
Purpose:

Detects and shows USB devices attached to this system.
After-suspend:

True
Plugin:

shell
Requires:

cpuinfo.platform != ‘s390x’

Wireless networking tests

The following test units are covered in this category:

wireless/wireless_connection_open_ac_nm_interface

Connect to unencrypted 802.11ac Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to insecure 802.11ac AP
After-suspend:

True
From template:

wireless/wireless_connection_open_ac_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_open_ax_nm_interface

Connect to unencrypted 802.11ax Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to insecure 802.11ax AP
After-suspend:

True
From template:

wireless/wireless_connection_open_ax_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_open_be_nm_interface

Connect to unencrypted 802.11be Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to insecure 802.11be AP
After-suspend:

True
From template:

wireless/wireless_connection_open_be_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_open_bg_nm_interface

Connect to an unencrypted 802.11b/g Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check the system can connect to an insecure 802.11b/g AP
After-suspend:

True
From template:

wireless/wireless_connection_open_bg_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_open_n_nm_interface

Connect to an unencrypted 802.11n Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check if the system can connect to an unsecured 802.11n access point.
After-suspend:

True
From template:

wireless/wireless_connection_open_n_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa3_ax_nm_interface

Connect to WPA3-encrypted 802.11ax Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to 802.11ax AP with wpa3 security
After-suspend:

True
From template:

wireless/wireless_connection_wpa3_ax_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa3_be_nm_interface

Connect to WPA3-encrypted 802.11be Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to 802.11be AP with wpa3 security
After-suspend:

True
From template:

wireless/wireless_connection_wpa3_be_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_ac_nm_interface

Connect to WPA-encrypted 802.11ac Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to 802.11ac AP with wpa security
After-suspend:

True
From template:

wireless/wireless_connection_wpa_ac_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_ax_nm_interface

Connect to WPA-encrypted 802.11ax Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to 802.11ax AP with wpa security
After-suspend:

True
From template:

wireless/wireless_connection_wpa_ax_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_be_nm_interface

Connect to WPA-encrypted 802.11be Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to 802.11be AP with wpa security
After-suspend:

True
From template:

wireless/wireless_connection_wpa_be_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_bg_nm_interface

Connect to WPA-encrypted 802.11b/g Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can connect to 802.11b/g AP with wpa security
After-suspend:

True
From template:

wireless/wireless_connection_wpa_bg_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_n_nm_interface

Connect to a WPA-encrypted 802.11n Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check if the system can connect to an 802.11n AP with WPA security.
After-suspend:

True
From template:

wireless/wireless_connection_wpa_n_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_scanning_interface

Test system can discover Wi-Fi networks on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Blocking
Purpose:

Check system can find a wireless network AP nearby
After-suspend:

True
From template:

wireless/wireless_scanning_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’

Wireless Wide Area Network

The following test units are covered in this category:

wwan/detect

Identify if WWAN module is missing

Category ID:

wwan
Status:

Blocking
Purpose:

Tests that there is a WWAN module present and indicates that testing of it should follow.
After-suspend:

True
User:

root
Plugin:

shell
Requires:

manifest.has_wwan_module == ‘True’ snap.name == ‘modem-manager’ or package.name == ‘modemmanager’
For details about the required manifest entry, see has_wwan_module.
wwan/gsm-connection-manufacturer-model-hw_id-auto

Verify a GSM broadband modem can create a data connection

Unit type:

template
Category ID:

wwan
Status:

Blocking
Purpose:

Any modems discovered by the resource job that list GSM support will be tested to ensure a data connection can be made.
After-suspend:

True
From template:

wwan/gsm-connection-manufacturer-model-hw_id-auto
Template resource:

wwan_resource

Non-blocking

Audio tests

The following test units are covered in this category:

audio/alsa_info_attachment

Attach audio hardware data collection log to the results.

Category ID:

audio
Status:

Non-blocking
Purpose:

Attaches the audio hardware data collection log to the results.
Plugin:

attachment
audio/alsa_info_collect

Collect audio-related system information for simulation and detailed testing.

Category ID:

audio
Status:

Non-blocking
Purpose:

Collect audio-related system information. This data can be used to simulate this computer’s audio subsystem and perform more detailed tests under a controlled environment.
User:

root
Plugin:

shell
audio/alsa_record_playback_automated

Test playback and recording functionality on the default audio input and output.

Category ID:

audio
Status:

Non-blocking
Purpose:

Play back a sound on the default output and listen for it on the default input. This test is intended mostly for laptops, where the default output will be the internal speakers and the default input will be the internal microphone.
After-suspend:

True
Plugin:

shell
Requires:

package.name == ‘python3-gi’ package.name == ‘gir1.2-gstreamer-1.0’ package.name == ‘libgstreamer1.0-0’ package.name == ‘gstreamer1.0-plugins-good’ package.name in [‘gstreamer1.0-pulseaudio’, ‘gstreamer1.0-pipewire’] package.name == ‘alsa-base’ package.name in [‘pulseaudio-utils’, ‘pipewire’] manifest.has_audio_playback == ‘True’
For details about the required manifest entry, see has_audio_playback.
audio/detect_sinks

Ensure audio sinks are available for detection.

Category ID:

audio
Status:

Non-blocking
Purpose:

Test to detect if there are available sinks
After-suspend:

True
Plugin:

shell
Requires:

package.name in [‘pulseaudio-utils’, ‘pipewire’] manifest.has_audio_playback == ‘True’
For details about the required manifest entry, see has_audio_playback.
audio/list_devices

Check if audio devices can be detected.

Category ID:

audio
Status:

Non-blocking
Purpose:

Test to detect audio devices
Plugin:

shell
Requires:

manifest.has_audio_playback == ‘True’ package.name == ‘alsa-base’
For details about the required manifest entry, see has_audio_playback.

Benchmarks tests

The following test units are covered in this category:

benchmarks/disk/hdparm-cache-read_name

Cached read timing benchmark of {name} using hdparm

Unit type:

template
Category ID:

benchmarks
Status:

Non-blocking
Purpose:

This test runs hdparm timing of cache reads as a benchmark for {name}
After-suspend:

True
From template:

benchmarks/disk/hdparm-cache-read_name
Template resource:

device
Template filter:

device.category == ‘DISK’
benchmarks/disk/hdparm-read_name

Raw read timing benchmark of {name} using hdparm

Unit type:

template
Category ID:

benchmarks
Status:

Non-blocking
Purpose:

This test runs hdparm timing of device reads as a benchmark for {name}
After-suspend:

True
From template:

benchmarks/disk/hdparm-read_name
Template resource:

device
Template filter:

device.category == ‘DISK’

Camera tests

The following test units are covered in this category:

camera/camera-quality-image_name

Attach the image used for the BRISQUE score for {product_slug}

Unit type:

template
Category ID:

camera
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

This test will attach the image used for the BRISQUE score.
After-suspend:

True
From template:

camera/camera-quality-image_name
Template resource:

device
Template filter:

device.category == ‘CAPTURE’ and device.name != ‘’
camera/camera-quality_name

Webcam BRISQUE score for {product_slug}

Unit type:

template
Category ID:

camera
Status:

Non-blocking
Purpose:

Takes a picture and computes the quality based on a No-Reference image quality assessment algorithm called BRISQUE. This test will timeout and fail if the quality has not been computed within 120 seconds.
After-suspend:

True
From template:

camera/camera-quality_name
Template resource:

device
Template filter:

device.category == ‘CAPTURE’ and device.name != ‘’
camera/multiple-resolution-images-attachment_name

Attach an image from the multiple resolution images test for {product_slug}

Unit type:

template
Category ID:

camera
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

This test will attach one of the images used for the multiple resolution images test.
After-suspend:

True
From template:

camera/multiple-resolution-images-attachment_name
Template resource:

device
Template filter:

device.category == ‘CAPTURE’ and device.name != ‘’

CPU tests

The following test units are covered in this category:

cpu/cstates_results.log

Attach C-States test log

Category ID:

cpu
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the FWTS desktop diagnosis results log to the submission.
Plugin:

attachment
Requires:

cpuinfo.platform not in (“aarch64”, “armv7l”, “s390x”)
cpu/maxfreq_test-log-attach

Attach CPU max frequency log

Category ID:

cpu
Status:

Non-blocking
Purpose:

Attaches the log generated by cpu/maxfreq_test to the results submission.
Plugin:

attachment
cpu/scaling_test-log-attach

Attach CPU scaling capabilities log

Category ID:

cpu
Status:

Non-blocking
Purpose:

Attaches the log generated by cpu/scaling_test to the results submission.
Plugin:

attachment

Disk tests

The following test units are covered in this category:

disk/apste_support_on_name

Check support for Autonomous Power State Transition on {name}

Unit type:

template
Category ID:

disk
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
After-suspend:

True
From template:

disk/apste_support_on_name
Template resource:

device
Template filter:

device.driver == ‘nvme’ and device.category == ‘OTHER’
disk/check-software-raid

Validate the configuration of software RAID devices are expected

Category ID:

disk
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Examine the system to detect Software RAID devices are created and the RAID mode are expected the SOFTWARE_RAID_LEVEL variable is needed for this tests. e.g. SOFTWARE_RAID_LEVEL=”raid0 raid1 raid5”
Environment variable:

SOFTWARE_RAID_LEVEL
User:

root
Plugin:

shell
Requires:

executable.name == ‘mdadm’ manifest.has_md_raid == ‘True’
For details about the required manifest entry, see has_md_raid.

Firmware tests

The following test units are covered in this category:

firmware/fwts_desktop_diagnosis

Run FWTS QA-concerned desktop-specific diagnosis tests.

Category ID:

firmware
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Run Firmware Test Suite (fwts) QA-concerned desktop-specific diagnosis tests.
Environment variable:

PLAINBOX_SESSION_SHARE
User:

root
Plugin:

shell
Requires:

executable.name == ‘fwts’
firmware/fwts_desktop_diagnosis_results.log.gz

Attach FWTS desktop diagnosis log to submission

Category ID:

firmware
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the FWTS desktop diagnosis results log to the submission
Plugin:

attachment
firmware/fwts_dump_acpi_attachment.gz

Collect the ACPI tables dump from FWTS

Category ID:

firmware
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

attachment

Informational tests

The following test units are covered in this category:

acpi_sleep_attachment

Attach the contents of /proc/acpi/sleep for further analysis.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches the contents of /proc/acpi/sleep if it exists.
Plugin:

attachment
codecs_attachment

Attach a report of installed codecs for Intel HDA.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a report of installed codecs for Intel HDA
Plugin:

attachment
cpuinfo_attachment

Attach a copy of /proc/cpuinfo

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a report of CPU information
Plugin:

attachment
dkms_info_attachment

Attaches json dumps of installed dkms package information.

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches json dumps of installed dkms package information.
Plugin:

attachment
Requires:

package.name == ‘dkms’
dmesg_attachment

Attach a copy of dmesg or the current dmesg buffer to the test results.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a copy of /var/log/dmesg or the current dmesg buffer to the test results
User:

root
Plugin:

attachment
dmi_attachment

Attach a copy of /sys/class/dmi/id/*

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches info on DMI
Plugin:

attachment
Requires:

dmi_present.state == ‘supported’
dmidecode_attachment

Attach output of dmidecode

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches dmidecode output
User:

root
Plugin:

attachment
Requires:

executable.name == ‘dmidecode’ dmi_present.state == ‘supported’
efi_attachment

Attaches firmware version info

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches the firmware version
User:

root
Plugin:

attachment
info/buildstamp

Attaches the buildstamp identifier for the OS

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the buildstamp identifier for the OS
Plugin:

attachment
info/disk_partitions

Attaches info about disk partitions

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches information about disk partitions
User:

root
Plugin:

attachment
info/hdparm_name.txt

Attaches info from hdparm about {name}

Unit type:

template
Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
From template:

info/hdparm_name.txt
Template resource:

device
Template filter:

device.category == ‘DISK’
info/secure-boot-check

Check secure boot state

Category ID:

info
Status:

Non-blocking
Purpose:

Output whether secure boot is enabled or disabled
Plugin:

attachment
Requires:

package.name == ‘mokutil’
info/systemd-analyze

System boot-up performance statistics

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
info/systemd-analyze-critical-chain

Print the tree of the time-critical chain of SystemD

Category ID:

info
Status:

Non-blocking
Purpose:

This job prints a tree of the time-critical chain of SystemD units.
Plugin:

shell
info/touchpad_driver

Gather touchpad name, driver name, and driver version information.

Category ID:

info
Status:

Non-blocking
Purpose:

Returns the name, driver name and driver version of any touchpad discovered on the system.
Plugin:

attachment
Requires:

device.category == ‘TOUCHPAD’
kernel_cmdline_attachment

Attach a copy of /proc/cmdline

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches the kernel command line used to boot
Plugin:

attachment
lsmod_attachment

Attach a list of currently running kernel modules

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a list of the currently running kernel modules.
Plugin:

attachment
lspci_attachment

Attach a list of PCI devices

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches very verbose lspci output.
Plugin:

attachment
lspci_standard_config_attachment

Attach PCI configuration space hex dump

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a hex dump of the standard part of the PCI configuration space for all PCI devices.
Plugin:

attachment
lstopo_visual_attachment

Attach the output of lstopo command to present system topology.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches the system topology as presented by the lstopo command
Plugin:

attachment
Requires:

executable.name == ‘lstopo’
lsusb_attachment

Attach output of lsusb

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a list of detected USB devices.
After-suspend:

True
User:

root
Plugin:

attachment
manifest

Hardware Manifest

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

This job loads the hardware manifest and exposes it as a resource.
Plugin:

resource
meminfo_attachment

Attach copy of /proc/meminfo

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches info on system memory as seen in /proc/meminfo.
Plugin:

attachment
modinfo_attachment

Attach modinfo information

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches modinfo information for all currently loaded modules
Plugin:

attachment
modprobe_attachment

Attach the contents of /etc/modprobe.*

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the contents of the various modprobe conf files.
User:

root
Plugin:

attachment
modules_attachment

Attach the contents of /etc/modules

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the contents of the /etc/modules file.
Plugin:

attachment
net_if_management_attachment

Collect logging from the net_if_management job

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Allows logging of debug info and errors by the associated resource job
Plugin:

attachment
recovery_info_attachment

Attach the recovery partition versions

Category ID:

info
Status:

Non-blocking
Purpose:

Attach the recovery partition version image_version is the preinstalled OS image version bto_version is only for dell_recovery Example: image_version: somerville-trusty-amd64-20140620-0 bto_version: A00_dell-bto-trusty-houston-15-A11-iso-20141203-0.iso
User:

root
Plugin:

attachment
Requires:

package.name == ‘dell-recovery’ or package.name == ‘ubuntu-recovery’
sysctl_attachment

Attach sysctl configuration files.

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the contents of various sysctl config files.
User:

root
Plugin:

attachment
sysfs_attachment

Attach detailed sysfs property output from udev

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches a report of sysfs attributes.
Plugin:

attachment
Requires:

cpuinfo.platform not in (“aarch64”)
udev_attachment

Attach dump of udev database

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches a dump of the udev database showing system hardware information.
Plugin:

attachment

Input Devices tests

The following test units are covered in this category:

input/fixed_screen_orientation_on_product___index__

Check whether screen orientation is fixed on {product}

Unit type:

template
Category ID:

input
Status:

Non-blocking
Purpose:

This tests whether the display orientation of {product} is fixed, and cannot be altered by tilting {product}.
After-suspend:

True
From template:

input/fixed_screen_orientation_on_product___index__
Template resource:

dmi
Template filter:

dmi.category == ‘CHASSIS’ and dmi.product not in (‘Tablet’, ‘Convertible’, ‘Detachable’)

Media Card tests

The following test units are covered in this category:

Miscellaneous tests

The following test units are covered in this category:

install/apt-get-gets-updates

Ensure apt can access repositories and get updates without installing them, to aid in recovery from broken updates.

Category ID:

miscellanea
Status:

Non-blocking
Purpose:

Tests to see that apt can access repositories and get updates (does not install updates). This is done to confirm that you could recover from an incomplete or broken update.
User:

root
Plugin:

shell
Requires:

package.name == ‘apt’
miscellanea/dkms_build_validation

Validate the build status of DKMS modules, automatically

Category ID:

miscellanea
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Firstly, check the built number of DKMS modules. Secondly, scan APT system update log to see any build error messages. This test is mainly used for pre-installed images which use DKMS modules.
Plugin:

shell
Requires:

package.name == ‘dkms’
miscellanea/oops_results.log

Attach the FWTS oops results for submission.

Category ID:

miscellanea
Status:

Non-blocking
Purpose:

Attaches the FWTS oops results log to the submission
Plugin:

attachment
miscellanea/submission-resources

Check that data for a complete result are present

Category ID:

miscellanea
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

A meta-job that verifies the data necessary for a complete result submission are present. Failure indicates that the results are incomplete and may be rejected.
Plugin:

shell

Non-device specific networking tests

The following test units are covered in this category:

networking/http

Ensure downloading files through HTTP works correctly.

Category ID:

networking
Status:

Non-blocking
Purpose:

Automated test case to make sure that it’s possible to download files through HTTP
Environment variable:

TRANSFER_SERVER
Plugin:

shell

Power Management tests

The following test units are covered in this category:

power-management/fwts_wakealarm-log-attach

Attach and display fwts wakealarm test log.

Category ID:

power-management
Status:

Non-blocking
Purpose:

Attach log from fwts wakealarm test
Plugin:

attachment

Touchpad tests

The following test units are covered in this category:

touchpad/palm-rejection-firmware-labeling_product_slug

Check palm rejection firmware/labeling for touchpads

Unit type:

template
Category ID:

touchpad
Status:

Non-blocking
Purpose:

Libinput firmware/labeling palm detection rely on touchpad ABS_MT_TOOL_TYPE capability. This test checks touchpad’s EV_ABS capability to make sure that firmware/labeling bit is set in touchpad firmware.
After-suspend:

True
From template:

touchpad/palm-rejection-firmware-labeling_product_slug
Template resource:

device
Template filter:

device.category == ‘TOUCHPAD’ and device.driver == ‘hid-multitouch’

TPM 2.0 (Trusted Platform Module)

The following test units are covered in this category:

tpm2/fwts-event-log-dump

Dump the contents of the TPM Event Log

Category ID:

tpm2
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

The information in the TPM Event Log can be useful in debugging problems with TPM command support and adherance to standards. This can be of particular help when checking whether a device can support Ubuntu Core Full Disk Encryption.
User:

root
Plugin:

attachment
Requires:

manifest.has_tpm2_chip == ‘True’ executable.name == ‘fwts’
For details about the required manifest entry, see has_tpm2_chip.

USB tests

The following test units are covered in this category:

usb/storage-detect

Detect storage partitions on a device on the USB bus

Category ID:

usb
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
After-suspend:

True
Plugin:

shell
Requires:

manifest.has_usb_storage == ‘True’
For details about the required manifest entry, see has_usb_storage.

Wireless networking tests

The following test units are covered in this category:

wireless/check_iwlwifi_microcode_crash_interface

Check there have been no iwlwifi crashes

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
Purpose:

Ensure no crashes have occurred in the iwlwifi microcode.
After-suspend:

True
From template:

wireless/check_iwlwifi_microcode_crash_interface
Template resource:

device
Template filter:

device.driver == ‘iwlwifi’
wireless/detect

Detect if at least one Wireless LAN device is detected

Category ID:

wireless
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
After-suspend:

True
Plugin:

shell
Requires:

manifest.has_wlan_adapter == ‘True’
For details about the required manifest entry, see has_wlan_adapter.

Wireless Wide Area Network

The following test units are covered in this category:

wwan/3gpp-scan-manufacturer-model-hw_id-auto

Scan for available 3GPP networks with the {model} modem

Unit type:

template
Category ID:

wwan
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Scan for available 3GPP networks with the target modem
After-suspend:

True
From template:

wwan/3gpp-scan-manufacturer-model-hw_id-auto
Template resource:

wwan_resource
wwan/check-sim-present-manufacturer-model-hw_id-auto

Check if a SIM card is present in a slot connected to the modem

Unit type:

template
Category ID:

wwan
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Check if a SIM card is present in a slot connected to the modem
After-suspend:

True
From template:

wwan/check-sim-present-manufacturer-model-hw_id-auto
Template resource:

wwan_resource

Manifest Entries

The following manifest entries are required for certification:

has_audio_playback

Audio playback: Machine can emit sound. (For example, a desktop PC probably requires speakers connected to it)

Value type:

bool
Prompt:

Does this machine have the following audio features?
has_bt_obex_support

A Bluetooth Module with OBject EXchange (OBEX) Support

Value type:

bool
has_camera

Camera/Capture Device

Value type:

bool
has_card_reader

Media Card Reader

Value type:

bool
has_dvd_bluray_inserted

DVD or Bluray disc inserted

Value type:

bool
has_ethernet_adapter

An Ethernet Port

Value type:

bool
has_md_raid

Software RAID

Value type:

bool
has_sim_card

A working SIM card inserted

Value type:

bool
has_touchpad

Touchpad

Value type:

bool
has_tpm2_chip

TPM 2.0 Support

Value type:

bool
has_usb_storage

USB Storage Device Connected

Value type:

bool
has_wlan_adapter

A Wi-Fi Module

Value type:

bool
has_wwan_module

A WWAN Module

Value type:

bool

Appendix A. FWTS tests

As part of the certification process, we run a series of firmware tests that are part of the Canonical Firmware Test Suite. In general, any HIGH or CRITICAL error found in the fwts log can cause potential errors in the system and should be looked at by OEMs/ODMs.

Category

Test Item

Description

Information

acpidump

Check ACPI table acpidump.

Information

version

Gather kernel system information.

ACPI

acpitables

ACPI table settings confidence checks.

ACPI

apicinstance

Check for single instance of APIC/MADT table.

ACPI

hpet_check

High Precision Event Timer configuration test.

ACPI

mcfg

MCFG PCI Express* memory mapped config space.

ACPI

method

ACPI DSDT Method Semantic Tests.

CPU

mpcheck

Check Multi Processor tables.

CPU

msr

CPU MSR consistency check.

CPU

mtrr

MTRR validation.

System

apicedge

APIC Edge/Level Check.

System

klog

Scan kernel log for errors and warnings.