TCF

TCF is a system that simplifies the creation and execution of test cases (automation, for that matter) with minimal setup effort by engineers (SW, QA and release) and autobuilders/CI alike across a wide amount of hardware platforms. It is distributed under the terms of the Apache 2.0 license.

The test framework provides means to:

• Discover and run one or many unit, integration, end-to-end automated test cases or samples with a single command line; these test cases may need no target (run in the local host) or one or more targets on which to operate.
• Locate, manage and share target hardware to maximize resource efficiency

A developer will create a feature and as part of that will create unit test cases which will be executed while developing the features until the feature is complete. Other engineers (eg: QA) might create more unit tests, integration and end-to-end tests to validate different features working together. Testcase metadata added to the test cases indicates how to build it, where can it be run and how to determine if it is succesful or not, or how to extract significant data (like resource consumption, performance, etc) for postprocessing. These test cases can then be commited as part of the code so that other people/agents can run them. The testcase can request targets, power them up/down, connect/disconnect things to/from it, etc

When is time to run the testcases, a developer, QA engineer, or CI/automation system launches tcf, which locates them, locates remote targets where to execute them them, builds and evaluate them, parallelizing as much as possible and generating reports about the execution.

The system consists of two parts:

• tcf: the client and test runner; this command line utility it is used to manage the test targets exported by the test target brokers (servers) and to execute test cases on said targets.
• ttbd: the server; this manages test targets connected to them, serving as a proxy to the test cases being run by tcf on behalf of users.

TCF focuses only on execution, leaving reporting, coverage analysis, etc, to other tools, providing means to feed data into them. It is designed with the goal of having a small footprint and little dependencies.

Report issues and contact the authors by filling issues in https://github.com/intel/tcf/issues.

Installation

Visit the quickstart instructions.

Documentation

Available at http://intel.github.io/tcf

Best practices for contributing available at the guide.

• 1. Quickstart
  • 1.1. Playing around with a server
  • 1.2. Running Zephyr OS testcases
    • 1.2.1. Running in multiple targets
    • 1.2.2. Running many testcases
  • 1.3. Automating test case building and execution
• 2. Guides
  • 2.1. Configuring the TCF client
    • 2.1.1. Configuring access to ttbd servers
    • 2.1.2. Configuring for Zephyr OS development
    • 2.1.3. Configuring for Arduino Sketch development
    • 2.1.4. Other configuration settings
  • 2.2. Running test cases with tcf run
    • 2.2.1. Running Zephyr OS testcases and samples
    • 2.2.2. Options to tcf run
    • 2.2.3. Target and tag filter specifications
      • 2.2.3.1. Examples of specifications
    • 2.2.4. Interpreting the output of tcf run
  • 2.3. Creating testcases
    • 2.3.1. Zephyr OS’s Hello World!
    • 2.3.2. A test case with multiple targets
    • 2.3.3. Connecting to network ports
    • 2.3.4. Network test cases
    • 2.3.5. Capturing tcpdumps of network traffic
    • 2.3.6. Saving data and files to a location
    • 2.3.7. Connecting things
  • 2.4. Testcase training
    • 2.4.1. A basic testcase
    • 2.4.2. What’s in a testcase
    • 2.4.3. Breaking it up
    • 2.4.4. Give me a target
    • 2.4.5. Give me a target that can run Zephyr
    • 2.4.6. Give me a target that can run Zephyr and is x86
    • 2.4.7. But put some Zephyr into it?
    • 2.4.8. Now we are building all the time, some time savers
    • 2.4.9. So, is Zephyr booting?
    • 2.4.10. And is the code doing what I want? Hello World?
    • 2.4.11. What happens when it fails? Let’s make it fail
    • 2.4.12. Running a Zephyr sanity check testcase
    • 2.4.13. Double down
    • 2.4.14. Double down more efficiently
    • 2.4.15. Double down catchas
    • 2.4.16. Two interconnected targets?
    • 2.4.17. Networking and the Zephyr echo server
    • 2.4.18. Add on the Zephyr echo client
    • 2.4.19. Cover more bases on the Zephyr echo server/client
    • 2.4.20. But let’s test Zephyr’s echo server/client better
    • 2.4.21. Developing Zephyr apps with the TCF’s help
      • 2.4.21.1. Evolving into a TC test case
      • 2.4.21.2. Evolving into a ztest test case
    • 2.4.22. Change of pace, input to a Zephyr test case
    • 2.4.23. FIXME: Missing
  • 2.5. Report Drivers
    • 2.5.1. Example 1: reporting completion of testcase execution
    • 2.5.2. Example 2: reporting failures
  • 2.6. App Builders
    • 2.6.1. Overriding actions
  • 2.7. Contributing
    • 2.7.1. Support & reporting issues
    • 2.7.2. Running the TCF client from the source tree
    • 2.7.3. Running the TCF server (ttbd) from the source tree
    • 2.7.4. Workflow for contributions
• 3. Server deployment guide
  • 3.1. Bill of materials
  • 3.2. Conventions
  • 3.3. Server installation
  • 3.4. Install and setup the TCF software
    • 3.4.1. Target networking
    • 3.4.2. Configure QEMU Zephyr OS targets
    • 3.4.3. Configure QEMU targets
    • 3.4.4. Configure physical test targets and power switches
    • 3.4.5. Configure physical Linux (or other) targets
    • 3.4.6. Configuring things that get plugged to the targets
      • 3.4.6.1. Best practices for server setup
        • 3.4.6.1.1. Naming targets
        • 3.4.6.1.2. Naming Networks
      • 3.4.6.2. Configuration Example 1
    • 3.4.7. Configuring Provisioning OS support
      • 3.4.7.1. POS: Server setup
      • 3.4.7.2. POS: deploy PXE boot image to HTTP and NFS server locations
      • 3.4.7.3. POS: Deploying other images
      • 3.4.7.4. POS: Configuring networks
      • 3.4.7.5. POS: Configuring targets
• 4. Examples and HOWTOs
  • 4.1. Automation/testcase script examples
    • 4.1.1. Returning results
    • 4.1.2. Tag a testcase
    • 4.1.3. Deploying OS images and files to targets over the network
      • 4.1.3.1. Quick way to deploy an OS image to a target and get a prompt
      • 4.1.3.2. Quick way to deploy an OS image to an specific target and get a prompt
      • 4.1.3.3. Deploy an OS image to a target
      • 4.1.3.4. Deploy an OS image to two targets simultaneously
      • 4.1.3.5. Boot a target in Provisioning mode
      • 4.1.3.6. Send a file or directory tree during deployment to the target
    • 4.1.4. Capturing data, doing SSH
      • 4.1.4.1. Reproducing audio and capturing the output
      • 4.1.4.2. Enabling target’s SSH server and executing an SSH command
    • 4.1.5. Keywords that are available to this testcase while running on a target
      • 4.1.5.1. Static testcases (no targets, run local)
      • 4.1.5.2. Testcase using one target
      • 4.1.5.3. Testcase using two targets
  • 4.2. TCF client tricks
    • 4.2.1. Where is the TCF client configuration taken from?
    • 4.2.2. How do I list which targets I own?
    • 4.2.3. How do I release a target I don’t own?
    • 4.2.4. How do I release all the targets I own?
    • 4.2.5. How do I keep a target(s) reserved and powered-on while I fuzz with them?
    • 4.2.6. How can I debug a target?
      • 4.2.6.1. Zephyr debugging with TCF run
    • 4.2.7. How can I quickly flash a Linux target
    • 4.2.8. How can I quickly build and deploy an image to a Zephyr target?
    • 4.2.9. TCF’s run says something failed, can I get more detail?
    • 4.2.10. Linux targets: Common tricks
      • 4.2.10.1. Linux targets: sending Ctrl-C to a target
      • 4.2.10.2. Linux targets: running a Linux shell command
      • 4.2.10.3. Linux targets: ssh login from a testcase / client
      • 4.2.10.4. How do I boot a target to PXE boot mode manually?
      • 4.2.10.5. POS: booting from HTTP or TFTP
      • 4.2.10.6. Linux targets: POS: setting default linux kernel options
      • 4.2.10.7. Linux targets: using proxies
      • 4.2.10.8. Linux targets: removing the root password
      • 4.2.10.9. Linux targets: allowing SSH as root with no passwords
    • 4.2.11. How do I change the default timeout in my test scripts
    • 4.2.12. Making the client always generate report files
    • 4.2.13. Splitting report files by domain
    • 4.2.14. Capturing network traffic
    • 4.2.15. Continuous Integration
      • 4.2.15.1. Generate a unique ID for each run and feed it to TCF
      • 4.2.15.2. Splitting in multiple shards
      • 4.2.15.3. Controlling output location
  • 4.3. General catchas
    • 4.3.1. Hidden characters in console output
    • 4.3.2. Newlines when reading output of a command
  • 4.4. General Linux system
    • 4.4.1. Setup proxies
    • 4.4.2. How do I update my TCF installation?
    • 4.4.3. Configuring a USB or other network adapter for an infrastructure network
    • 4.4.4. Configuring network interfaces with NetworkManager and nmcli
      • 4.4.4.1. Disabling NetworkManager from controlling an interface
      • 4.4.4.2. Configuring a static interface Via NetworkManager’s nmcli
    • 4.4.5. Generating an SSL certificate
    • 4.4.6. Configuring and reloading UDEV
    • 4.4.7. Finding USB device information
      • 4.4.7.1. Finding USB device information with dmesg
      • 4.4.7.2. Finding USB device information with lsusb.py
      • 4.4.7.3. Finding USB device information with udevadm
    • 4.4.8. udev configuration of serial port
      • 4.4.8.1. udev configuration of serial port based on serial number
      • 4.4.8.2. udev configuration of serial port without serial number based on path
      • 4.4.8.3. udev configuration of serial port based on sibling’s serial number
    • 4.4.9. Finding the serial number of a Devantech USBRLY08B relay controller
    • 4.4.10. Finding the serial number of an YKUSH hub
  • 4.5. ttbd: TCF server configuration and tricks
    • 4.5.1. Starting more that one instance
    • 4.5.2. Configure authentication for local users (optional)
    • 4.5.3. Configure simple authentication / for Jenkins jobs (optional)
    • 4.5.4. Configure authentication against LDAP
    • 4.5.5. Configuring a target for just controlling power to something
    • 4.5.6. Configuring a Linux target to power on/off with serial console
    • 4.5.7. Configure physical Linux targets with a fixed Live filesystem
    • 4.5.8. Adding tags to a target
    • 4.5.9. How do I disable a target by default?
    • 4.5.10. Allowing the server to be used remotely
    • 4.5.11. Increasing the verbosity of the server
  • 4.6. Manual installation of TCF from source
    • 4.6.1. Creation and setup of user ttbd
    • 4.6.2. Install required software
    • 4.6.3. Remove conflicting packages
    • 4.6.4. Install TCF
  • 4.7. Manual installation of support packages
    • 4.7.1. Bossac: Arduino Due flasher
    • 4.7.2. Zephyr SDK
    • 4.7.3. Arduino Builder 1.6.13
    • 4.7.4. tunslip6
    • 4.7.5. Xtensa ESP32
    • 4.7.6. libcoap
    • 4.7.7. FPM, fast package manager
  • 4.8. Platform firmware / BIOS update procedures
    • 4.8.1. Updating the serial number and / or description of a FTDI serial device
    • 4.8.2. Updating the serial number and / or description of a CP210x serial device
    • 4.8.3. Updating the firmware in an Arduino101 to factory settings
    • 4.8.4. Updating the firmware in an Quark C1000 reference boards
      • 4.8.4.1. Remediation steps
    • 4.8.5. Updating the firmware in an Quark D2000 reference boards
    • 4.8.6. Updating the FPGA image in Synopsys EMSK boards
• 5. Rationales and Frequently Asked Questions
  • 5.1. TCF is OS and test target agnostic
  • 5.2. Separated networks for infrastructure and test target hardware
  • 5.3. MCU test fixture for highest reliabiltiy
  • 5.4. High cost of fixtures
  • 5.5. Zephyr
    • 5.5.1. tcf run -s will not filter a Zephyr subcase
• 6. Support & Troubleshooting
  • 6.1. TCF client
    • 6.1.1. tcf dependencies installation failures
      • 6.1.1.1. Missing redhat-hardened-cc1
    • 6.1.2. tcf just hangs there, nothing happens
    • 6.1.3. Client operations report EOF occurred in violation of protocol
    • 6.1.4. tcf run fails compiling in Zephyr…
    • 6.1.5. target power on fails with timed out waiting for udev to set permissions
    • 6.1.6. PXE POS provisioning target fails to PXE boot: Unable to locate configuration file
    • 6.1.7. PXE POS provisioning target fails to boot: NFS: an incorrect mount option was specified
  • 6.2. ttbd server
    • 6.2.1. Restart ttbd, watch the output for errors
    • 6.2.2. Log shows can’t find hub, retrying
    • 6.2.3. log shows errors Cannot find YKUSH serial ‘YKNNNNN’
    • 6.2.4. Tips for diagnosing systemd startup
    • 6.2.5. Tips for configuring systemd startup
    • 6.2.6. USB device powers up, but serial port does not come up
    • 6.2.7. Multiple failures caused by SELinux
    • 6.2.8. Different operations fail, daemon log reports Connection reset by peer
    • 6.2.9. Uploading files to the daemon fails with /var/cache/ttbd-production/USERNAME: cannot create path
    • 6.2.10. LDAP login fails with invalid credentials, but the password is right, I know
    • 6.2.11. Freedom Board k64f
      • 6.2.11.1. Freedom Board k64f, openocd fails to power up (unable to open CMSIS-DAP device 0xd28:0x204)
      • 6.2.11.2. FRDM k64f boards fail to power up, OpenOCD prints message about RESET/WDOG loop
        • 6.2.11.2.1. Manual execution of the procedure
      • 6.2.11.3. Set the up the target in debug mode
      • 6.2.11.4. Issue OpenOCD commands to stop the loop
      • 6.2.11.5. Copy a known firmware image to its flash drive
      • 6.2.11.6. Reset the board to normal configuration
    • 6.2.12. Arduino101
      • 6.2.12.1. Arduino101: openocd fails to power up (targets report tap-disabled)
• 7. Architecture
  • 7.1. The TCF client
    • 7.1.1. The testcase finder and runner
  • 7.2. The Test Target Broker
    • 7.2.1. Daemon internals
    • 7.2.2. Maintaining state
    • 7.2.3. Interfaces
      • 7.2.3.1. Power control
      • 7.2.3.2. Console management
      • 7.2.3.3. File deployment
      • 7.2.3.4. Image deployment
      • 7.2.3.5. Debugging
      • 7.2.3.6. Things
    • 7.2.4. Authentication
  • 7.3. Provisioning
  • 7.4. Security considerations
    • 7.4.1. General
    • 7.4.2. Client
    • 7.4.3. Daemon
      • 7.4.3.1. Daemon access control
      • 7.4.3.2. File permissions
      • 7.4.3.3. Other processes started by the daemon
        • 7.4.3.3.1. bossac
        • 7.4.3.3.2. dfu-util
        • 7.4.3.3.3. genisoimage
        • 7.4.3.3.4. ip
        • 7.4.3.3.5. QEMU
        • 7.4.3.3.6. qemu-img
        • 7.4.3.3.7. OpenOCD
        • 7.4.3.3.8. socat
        • 7.4.3.3.9. tunslip6
      • 7.4.3.4. Networking
  • 7.5. Taxonomy of test cases
• 8. APIs
  • 8.1. TCF run: testcase API and target manipulation during testcases
    • 8.1.1. TCF’s backbone test case finder and runner
      • 8.1.1.1. Testcase run identification
    • 8.1.2. Test library (utilities for testcases)
    • 8.1.3. Provisioning/deploying/flashing PC-class devices with a Provisioning OS
    • 8.1.4. Other target interfaces
      • 8.1.4.1. Copy files from and to the server’s user storage area
      • 8.1.4.2. Press and release buttons in the target
      • 8.1.4.3. Capture screenshots or video/audio stream from the target
      • 8.1.4.4. Raw access to the target’s serial consoles
      • 8.1.4.5. Access target’s debugging capabilities
      • 8.1.4.6. Flash the target with fastboot
      • 8.1.4.7. Flash the target with JTAGs and other mechanism
      • 8.1.4.8. Flash the target with ioc_flash_server_app
      • 8.1.4.9. Power the target on or off
      • 8.1.4.10. Run commands a shell available on a target’s serial console
        • 8.1.4.10.1. Shell prompts
      • 8.1.4.11. Run commands to the target and copy files back and forth using SSH
      • 8.1.4.12. Create and remove network tunnels to the target via the server
    • 8.1.5. TCF run Application builders
    • 8.1.6. TCF run report drivers
      • 8.1.6.1. Report infrastructure
  • 8.2. TCF client configuration
    • 8.2.1. Configuration API for tcf
  • 8.3. TCF client internals
    • 8.3.1. Expecting things that have to happen
    • 8.3.2. Client API for accessing ttbd’s REST API
    • 8.3.3. Zephyr’s SanityCheck testcase.ini driver for testcase integration
  • 8.4. Target metadata
    • 8.4.1. Common metadata
    • 8.4.2. Interface specific metadata
    • 8.4.3. Driver / targe type specific metadata
    • 8.4.4. Provisioning OS specific metadata
  • 8.5. ttbd Configuration API for targets
  • 8.6. ttbd Configuration API
  • 8.7. ttbd internals
    • 8.7.1. Target types drivers
    • 8.7.2. User access control and authentication
    • 8.7.3. Console Management Interface
    • 8.7.4. Debugging Interface
    • 8.7.5. Power Control Interface
      • 8.7.5.1. Power control module to start DHCP daemon when a network is powered on
      • 8.7.5.2. Power control module to start a rsync daemon when a network is powered-on
      • 8.7.5.3. Power control module to start a socat daemon when a network is powered-on
    • 8.7.6. Other interfaces
      • 8.7.6.1. Interface to press buttons in a target
      • 8.7.6.2. Stream and snapshot capture interace
      • 8.7.6.3. Interface to provide flash the target using fastboot
      • 8.7.6.4. Interface to flash the target using ioc_flash_server_app
    • 8.7.7. Common helper library
      • 8.7.7.1. TCOB Fixture setup instructions
• 9. Glossary
• 10. Current status and plans
  • 10.1. Challenges
  • 10.2. The code
• 11. TCF release v0.11

Indexes

• Index
• Module Index
• Search Page