6. Support & Troubleshooting¶

For support, see the contributing section.

6.1. TCF client¶

6.1.1. tcf dependencies installation failures¶

6.1.1.1. Missing redhat-hardened-cc1¶

During installation with pip2:

$ pip2 install --user -r requirements.txt
...
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/Levenshtein
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c Levenshtein/_levenshtein.c -o build/temp.linux-x86_64-2.7/Levenshtein/_levenshtein.o
gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory
error: command 'gcc' failed with exit status 1

this usually happens in Fedora based distros; fix by installing missing package:

# dnf install -y redhat-rpm-config

6.1.2. tcf just hangs there, nothing happens¶

Add -vs right after tcf (e.g.: tcf -vvv ...; usually this means TCF is trying to contact a host that is either:

nonresponsive (the server might be up, the port open but the daemon is not responding or the server has crashed)
blocked by a firewall (check your proxy settings are ok)

6.1.3. Client operations report EOF occurred in violation of protocol¶

Any client operation (such as tcf list):

$ ~/z/tcf.git/tcf list
Traceback (most recent call last):
  File "~/z/tcf.git/tcf", line 350, in <module>
    retval = args.func(args)
  ...
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)

If your network needs proxy support and you are seeing messages from the client as EOF occurred in violation of protocol, the proxy may be cutting you off; ensure your proxies are properly setup as per these instructions).

In a pinch, you can unset your proxy environment variables are unset:

$ unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY

if your servers are locally accesible.

6.1.4. tcf run fails compiling in Zephyr…¶

FIXME: update

6.1.5. target power on fails with timed out waiting for udev to set permissions¶

Targets implemented with QEMU virtual machines do networking using Linux’s TAP interfaces, which need proper permissions being set.

ttbd relies on file ttbd/80-udev.rules being installed in /usr/lib/udev/rules.d to perform this function. When a power on fails with a message looking like:

qlf04a: /dev/tapNNN: timed out waiting for udev to set permissions in /usr/lib/udev/rules.d/80-ttbd.rules

this usually means said file is not installed; RPM installation should have put it in /usr/lib/udev/rules.d; if running from source, make sure to install it manually and ask udev to reload:

# install tcf.git/ttbd/80-udev.rules /usr/lib/udev/rules.d
# udevadm control --reload-rules

Sometimes instead this might be SELinux issue:

$ tcf power-on ql06a
Traceback (most recent call last):
  File "/usr/bin/tcf", line 517, in <module>
    retval = args.func(args)
  File "/usr/lib/python2.7/site-packages/tcfl/ttb_client.py", line 799, in rest_target_power_on
    rtb.rest_tb_target_power_on(rt)
  File "/usr/lib/python2.7/site-packages/tcfl/ttb_client.py", line 319, in rest_tb_target_power_on
    data = { 'ticket': ticket })
  File "/usr/lib/python2.7/site-packages/tcfl/ttb_client.py", line 196, in send_request
    commonl.request_response_maybe_raise(r)
  File "/usr/lib/python2.7/site-packages/commonl/__init__.py", line 468, in request_response_maybe_raise
    raise e
requests.exceptions.HTTPError: 400: ql06a: [Errno 13] Permission denied: '/dev/tap1023'

is usually caused by a SELinux check disallowing the daemon access to the /dev/tapNUMBER device.

The system’s journal might show:

audit[11345]: AVC avc:  denied  { open } for  pid=11345 comm="ttbd" path="/dev/tap1023" dev="devtmpfs" ino=35798295 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:tun_tap_device_t:s0 tclass=chr_file permissive=0

there is no current proper fix, other than disabling SELinux:

# setenforce 0

however, this seems to only happen when running the daemon from a $HOME directory, instead of from a system installation.

6.1.6. PXE POS provisioning target fails to PXE boot: Unable to locate configuration file¶

When a system is configured to boot targets to Provisioning OS over PXE, a target fails to boot with a message on the BIOS console such as:

>>Start PXE over IPv4.
Station IP address is 192.168.97.101
Server IP address is 192.168.97.1
NBP filename is ttbd-production/efi-x86_64/syslinux.efi
NBP filesize is 176328 Bytes
Downloading NBP file...
NBP file downloaded successfully.
Getting cached packet
My IP is 192.168.97.101
Unable to locate configuration file

There might be multiple causes for this, but here are the most common:

there is a typo in the target’s MAC address

The target is given, via TFTP, the syslinux PXE bootloader, who will look for configuration files with multiple names (see https://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration_filename) on the TFTP server, but relies in one named after the target’s MAC address.

The TTBD server’s POS environment generates the configuration file when the network to which the target is attached is powered up.

Thus, the target’s MAC address listed in the configuration has to match the one the target actually uses on the network:
```
>>> ttbl.config.targets['TARGETNAME'].add_to_interconnect(
>>>   NETWORKNAME, dict(
>>>       mac_addr = "00:07:32:4b:36:87",
>>>       ipv4_addr = '192.168.98.160', ipv4_prefix_len = 24,
>>>       ipv6_addr = 'fc00::62:a0', ipv6_prefix_len = 112)
>>>   )
```
ensure the mac_addr value is the MAC address of the network interface the target is using to PXE boot from (_addr fields will vary from the example to your configuration).
mismatch in IP addresses

It is frequent to create typos in the network configuration and target configuration so that the target’s IP address won’t be in the network’s address range. Ensure this is correct.

6.1.7. PXE POS provisioning target fails to boot: NFS: an incorrect mount option was specified¶

When the Provisoning OS kernel tries to NFS mount the root, it fails with invalid mount option specified:

Check the MAC address is correct and the system is given the right IP address (vs any from the pool).

This usually happens when the system doesn’t recognize the MAC address as matching the right host. It is very common when changing names. Also, before reinitializing the server, clean up the cache:

$ rm -rf /var/lib/tftpbootlib/ttbd-production/pxelinux.cfg/*

to ensure there are no left-over configuration files with old addresses

6.2. ttbd server¶

6.2.1. Restart ttbd, watch the output for errors¶

Run:

$ journalctl -feu ttbd@NAME

-f follows the output, -e goes to the end of the file as of now, -u means unit ttbd@NAME.

Ensure the targets are visible with tcf list

6.2.2. Log shows can’t find hub, retrying¶

This message:

... pc_ykush._command():139 target-TARGET: YKXXXXX[1]: can't find hub, retrying

this means that for whichever reason, when the YKUSH is powered on, the system is not able to find it in the list of connected devices.

Most common cases:

the configuration is not correct and it is not being powered up

Accessing the infrastructure server, power-on the YKXXXX-base target and visually verify the YKush’s hub led is turning on.

If not:

check the power-control configuration for the YKXXXX, is it the correct one?

check the power brick

check the micro-USB power cable

Note sometimes it is useful to pluck out the components and test against a power outlet or USB power port we know to work.
connect the hub against a wall outlet to a laptop and see if it enumerates, verify with lsusb.py -iu | grep YKXXXX
Use dmesg -w in a laptop or another machine to verify what happens when you plug the hub.
if dmesg shows messages like:
kernel: usb 1-1.2.5: device descriptor read/64, error -32
the micro USB cable for upstream connectivity might be bad; replace it.
If nothing shows but the led is lit, the hub might be toast and is time to replace it

6.2.3. log shows errors Cannot find YKUSH serial ‘YKNNNNN’¶

Plenty of operations randomly fail with messages such as:

... @SERVER/TARGET:BSP: error resetting: 400: TARGET: Cannot find YKUSH serial 'YKNNNNN'

this is usually caused by overloaded USB buses (also other symptoms). Please spread your devices amongst different USB buses that are connected to different USB cards (root ports) in the host machine.

6.2.4. Tips for diagnosing systemd startup¶

if systemd fails to start the unit with a GROUP message, one or more of the supplemental groups in /etc/systemd/system/ttbd@.service might not exist; verify it.
journalctl -u ttbd@NAME: to get all the log messages from the instance
journalctl -feu ttbd@NAME: monitor log messages from the instance instance
systemd fails to start the daemon with a code=203/EXEC:

this is most likely a permissons issue; verify your Unix permissions and ACLs are right. SELinux might be in the way too (check your system output with journalctl -xe and look for ttbd.

systemctl fails with service not found

Besides the unit not existing, this might happen when you run the daemon from a source directory and SELinux is not allowing access:

$ sudo systemctl restart ttbd@afapli.service
Failed to restart ttbd@afapli.service: Unit ttbd@afapli.service not found.

Look in the journalctl for lines like:

$ sudo systemctl restart ttbd@production.service
Failed to restart ttbd@production.service: Unit ttbd@production.service not found.
$ journalctl -S -2min | grep -B4

...audit[1]: AVC avc:  denied  { read } for  pid=1 comm="systemd" name="ttbd@.service" dev="dm-2" ino=48627844 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
...systemd[1]: Cannot access "(null)": Permission denied
...systemd[1]: ttbd@production.service: Failed to load configuration: No such file or directory

A quick way to fix this is to disable SELinux with:

# setenforce

6.2.5. Tips for configuring systemd startup¶

The default integration with systemd in unit file /etc/systemd/system/ttbd@.service can be extended and or modified; for example:

copy /etc/systemd/system/ttbd@.service to ttbd@NAME.service
edit ttbd@NAME.service:
- for example change the username
- add dependencies to the [Unit] section, for example, if there is another daemon that should be started first, like:
```
Requires = ttbd@NAME0.service
```
  this would make the NAME0 daemon start before the NAME daemon.
tell systemd to reload configuration:
```
# systemctl daemon-reload
```

Note you can use the following systemd commands (for the default one, ommit NAME):

systemctl enable ttbd@NAME: enable the running of the

NAME ttbd instance automatically
systemctl start ttbd@NAME: start ttbd NAME instance
systemctl stop ttbd@NAME: stop ttbd NAME instance
systemctl status ttbd@NAME: query status of the instance; add -l to get longer messages (without ellipsis)

6.2.6. USB device powers up, but serial port does not come up¶

Messages such as this pop from the server log:

I[7742] pc.power_on_do():284: target-TARGETNAME[local:5fng]:\
  <ttbl.pc.delay_til_usb_device object at 0x7fce3611edd0>: \
  delaying power-on 5.00s until USB device with serial SOMESERIALNUMBER appears
...
I[7743] ttbd.flask_logi_abort():104: TARGETNAME: \
  [Errno 2] could not open port /dev/tty-TARGETNAME: \
  [Errno 2] No such file or directory: '/dev/tty-TARGETNAME'

And tcf run and others report things such as:

HTTPError: 400: TARGETNAME: timeout (39.67s) on power-on delay waiting \
for USB device with serial SOMESERIALNUMBER to appear

There is a long list of reasons why this can happen, but these are the most common:

The device has no firmware that allows it to expose a serial port
The device exposes more than one USB interface and the one which exposes the serial port hasn’t been connected.

The device does not have enough power to expose it’s functionality. Inspect the kernel log with dmesg and verify the kernel is able to recognize the device and report it is powering up properly (for example, for an FRDM k64f):

usb 2-2.3.1: new full-speed USB device number 127 using xhci_hcd
usb 2-2.3.1: New USB device found, idVendor=0d28, idProduct=0204
usb 2-2.3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-2.3.1: Product: MBED CMSIS-DAP
usb 2-2.3.1: Manufacturer: MBED
usb 2-2.3.1: SerialNumber: 024002011E741E6DE38AE3D5
usb-storage 2-2.3.1:1.0: USB Mass Storage device detected
scsi host3: usb-storage 2-2.3.1:1.0
cdc_acm 2-2.3.1:1.1: ttyACM1: USB ACM device

the USB bus is out of bandwidth and will refuse to configure the device in the USB configuration that provides the serial device. When trying to power up the target, look out for messages in the kernel log similar to (with dmesg):
```
usb 2-2.1.4.2: new full-speed USB device number 27 using xhci_hcd
usb 2-2.1.4.2: New USB device found, idVendor=0d28, idProduct=0204
usb 2-2.1.4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 2-2.1.4.1: Product: MBED CMSIS-DAP
usb 2-2.1.4.1: Manufacturer: MBED                                                                                                            [3199173.308138] usb 2-2.1.4.1: SerialNumber: 024002011E481E5DE3B6E3E5
usb 2-2.1.4.1: Not enough bandwidth for new device state.
```
when this happens, you need to reconfigure your setup, as you have probably maxed out the bandwidth of the USB bus. Posible ways to resolve it:
- Move devices to another USB bus (not port, but bus). You can add more buses by adding new PCI-to-USB cards.
  
  Note some controllers show up as multiple buses to support different USB versions, but they are actually the same.
- Spread your configuration across multiple physical servers operating in active/active mode, which also has the benefit of increasing the redundancy in your setup.
udev is not properly configured and thus it does not see the event that will allow it to create whichever device link it has to create (see udev configuration of serial port).

Double check the configuration, use journalctl -au systemd-udevd to inspect the udev ouput. You can also increase its log level with:
```
# udevadm control -l debug
```
power cycle the device and inspect the output from udev to see why the device link might not be created.

Remember to restore the log level back when done to err to avoid polluting your system logs too much.

6.2.7. Multiple failures caused by SELinux¶

When the daemon is not installed system wide, SELinux will cause multiple issues, even if started via systemd. This is common when testing development versions that run from the git tree.

In these cases, the system journal will report SELinux denials or interventions. In this case, sometimes the best thing is to disable SELinux for the development work–however, it is important to later test in a system wide deployment:

# setenforce 0

will disable SELinux.

6.2.8. Different operations fail, daemon log reports Connection reset by peer¶

This is caused in most cases by OpenOCD crashing, making all kinds of operations fail. Mostly seen during retrying, where it retries and retries but it keeps failing, with logs in the daemon such as:

W[15024] flasher._expect_mgr():522: target-emsk-27[USER:lak2]: OpenOCD/snps_em_sk[251634500808]: target reset/halt for image flashing: Error shutting down socket: [Errno 107] Transport endpoint is not connected

there is no easy solution for this: OpenOCD is crashing, it is a problem in OpenOCD or how it is being used is triggering it. The OpenOCD log files in in /var/run/ttbd-production/TARGETNAME/ will give a hint of what was being done, but will contain no information about the crash itself.

You can run the whole setup with core file dumping enabled.

Note however, there are valid cases where ttbd kills OpenOCD (when it doesn’t start properly, for example), so it might be easy to confuse them both. Run the daemon with increased verbosity to be able to tell those cases apart.

6.2.9. Uploading files to the daemon fails with /var/cache/ttbd-production/USERNAME: cannot create path¶

A client operation (normally uploading files to the server), ends with:

requests.exceptions.HTTPError: 400: /var/cache/ttbd-production/USERNAME: cannot create path: [Errno 13] Permission denied: '/var/cache/ttbd-production/USERNAME'

if the path /var/cache/ttbd-production exists, it is owned by user ttbd:ttbd and the user has permissions to write, create, this might be a SELinux issue.

If in the system’s journal, it can be found:

$ journal -x
...

python3[23087]: SELinux is preventing ttbd from create access on the directory USERNAME.

                *****  Plugin catchall_labels (83.8 confidence) suggests   *******************

                If you want to allow ttbd to have create access on the local directory
                Then you need to change the label on local
                Do
                # semanage fcontext -a -t FILE_TYPE 'USERNAME'
                where FILE_TYPE is one of the following: NetworkManager_unit_file_t, NetworkManager_var_run_t, abrt_unit_file_t, abrt_var_run_t, accountsd_unit_file_t, aiccu_var_run_t, ajaxterm_var_run_t, alsa_lock_t, alsa_unit_file_t, alsa_var_run_t, amanda_unit_file_t, antivirus_unit_file_t, antivirus_var_run_t, apcupsd_lock_t, apcupsd_unit_file_t, apcupsd_var_run_t, apmd_lock_t, apmd_unit_file_t, apmd_var_run_t, arpwatch_unit_file_t, arpwatch_var_run_t, asterisk_var_run_t, audisp_var_run_t, auditd_unit_file_t, auditd_var_run_t, automount_lock_t, automount_unit_file_t, automount_var_run_t, avahi_unit_file_t, avahi_var_run_t, bacula_var_run_t, bcfg2_unit_file_t, bcfg2_var_run_t, bitlbee_var_run_t, blkmapd_var_run_t, blktap_var_run_t, blueman_var_run_t, bluetooth_lock_t, bluetooth_unit_file_t, bluetooth_var_run_t, boinc_unit_file_t, bootloader_var_run_t, brltty_unit_file_t, brltty_var_run_t, bumblebee_unit_file_t, bumblebee_var_run_t, cache_home_t, cachefilesd_var_run_t, callweaver_var_run_t, canna_var_run_t, cardmgr_var_run_t, ccs_var_run_t, certmaster_var_run_t, certmonger_var_run_t, cgred_var_run_t, cgroup_t, chronyd_unit_file_t, chronyd_var_run_t, cinder_api_unit_file_t, cinder_backup_unit_file_t, cinder_scheduler_unit_file_t, cinder_var_run_t, cinder_volume_unit_file_t, clogd_var_run_t, cloud_init_unit_file_t, cluster_unit_file_t, cluster_var_run_t, clvmd_var_run_t, cmirrord_var_run_t, cockpit_unit_file_t, cockpit_var_run_t, collectd_unit_file_t, collectd_var_run_t, colord_unit_file_t, comsat_var_run_t, condor_unit_file_t, condor_var_lock_t, condor_var_run_t, config_home_t, conman_unit_file_t, conman_var_run_t, consolekit_unit_file_t, consolekit_var_run_t, container_lock_t, container_plugin_var_run_t, container_unit_file_t, container_var_run_t, couchdb_unit_file_t, couchdb_var_run_t, courier_var_run_t, cpuplug_lock_t, cpuplug_var_run_t, cpuspeed_var_run_t, cron_var_run_t, crond_unit_file_t, crond_var_run_t, ctdbd_var_run_t, cupsd_config_var_run_t, cupsd_lock_t, cupsd_lpd_var_run_t, cupsd_unit_file_t, cupsd_var_run_t, cvs_var_run_t, cyphesis_var_run_t, cyrus_var_run_t, data_home_t, dbskkd_var_run_t, dbus_home_t, dcc_var_run_t, dccd_var_run_t, dccifd_var_run_t, dccm_var_run_t, dcerpcd_var_run_t, ddclient_var_run_t, deltacloudd_var_run_t, denyhosts_var_lock_t, device_t, devicekit_var_run_t, devpts_t, dhcpc_var_run_t, dhcpd_unit_file_t, dhcpd_var_run_t, dictd_var_run_t, dirsrv_snmp_var_run_t, dirsrv_var_lock_t, dirsrv_var_run_t, dirsrvadmin_lock_t, dirsrvadmin_unit_file_t, dkim_milter_data_t, dlm_controld_var_run_t, dnsmasq_unit_file_t, dnsmasq_var_run_t, dnssec_trigger_unit_file_t, dnssec_trigger_var_run_t, dovecot_var_run_t, drbd_lock_t, drbd_var_run_t, dspam_var_run_t, entropyd_var_run_t, etc_t, eventlogd_var_run_t, evtchnd_var_run_t, exim_var_run_t, fail2ban_var_run_t, fcoemon_var_run_t, fenced_lock_t, fenced_var_run_t, fetchmail_var_run_t, fingerd_var_run_t, firewalld_unit_file_t, firewalld_var_run_t, foghorn_var_run_t, freeipmi_bmc_watchdog_unit_file_t, freeipmi_bmc_watchdog_var_run_t, freeipmi_ipmidetectd_unit_file_t, freeipmi_ipmidetectd_var_run_t, freeipmi_ipmiseld_unit_file_t, freeipmi_ipmiseld_var_run_t, fsadm_var_run_t, fsdaemon_var_run_t, ftpd_lock_t, ftpd_unit_file_t, ftpd_var_run_t, fwupd_unit_file_t, games_srv_var_run_t, gconf_home_t, gdomap_var_run_t, getty_lock_t, getty_unit_file_t, getty_var_run_t, gfs_controld_var_run_t, gkeyringd_gnome_home_t, glance_api_unit_file_t, glance_registry_unit_file_t, glance_scrubber_unit_file_t, glance_var_run_t, glusterd_var_run_t, gnome_home_t, gpm_var_run_t, gpsd_var_run_t, greylist_milter_data_t, groupd_var_run_t, gssproxy_unit_file_t, gssproxy_var_run_t, gstreamer_home_t, haproxy_unit_file_t, haproxy_var_run_t, hostapd_unit_file_t, hostapd_var_run_t, hsqldb_unit_file_t, httpd_lock_t, httpd_unit_file_t, httpd_var_run_t, hugetlbfs_t, hwloc_dhwd_unit_t, hwloc_var_run_t, hypervkvp_unit_file_t, hypervvssd_unit_file_t, icc_data_home_t, icecast_var_run_t, iceccd_var_run_t, ifconfig_var_run_t, inetd_child_var_run_t, inetd_var_run_t, init_tmp_t, init_var_lib_t, init_var_run_t, initrc_var_run_t, innd_unit_file_t, innd_var_run_t, insmod_var_run_t, iodined_unit_file_t, ipa_dnskey_unit_file_t, ipa_ods_exporter_unit_file_t, ipa_otpd_unit_file_t, ipa_var_run_t, ipmievd_lock_t, ipmievd_unit_file_t, ipmievd_var_run_t, ipsec_mgmt_lock_t, ipsec_mgmt_unit_file_t, ipsec_mgmt_var_run_t, ipsec_var_run_t, iptables_lock_t, iptables_unit_file_t, iptables_var_lib_t, iptables_var_run_t, irqbalance_var_run_t, iscsi_lock_t, iscsi_unit_file_t, iscsi_var_run_t, isnsd_var_run_t, iwhd_var_run_t, jetty_unit_file_t, jetty_var_run_t, kadmind_var_run_t, kdump_lock_t, kdump_unit_file_t, keepalived_unit_file_t, keepalived_var_run_t, keystone_unit_file_t, keystone_var_run_t, kismet_var_run_t, klogd_var_run_t, kmscon_unit_file_t, krb5kdc_lock_t, krb5kdc_var_run_t, ksmtuned_unit_file_t, ksmtuned_var_run_t, ktalkd_unit_file_t, l2tpd_var_run_t, likewise_pstore_lock_t, lircd_var_run_t, lldpad_var_run_t, local_login_lock_t, locale_t, locate_var_run_t, lockdev_lock_t, logrotate_lock_t, logwatch_lock_t, logwatch_var_run_t, lpd_var_run_t, lsassd_var_run_t, lsmd_unit_file_t, lsmd_var_run_t, lttng_sessiond_unit_file_t, lttng_sessiond_var_run_t, lvm_lock_t, lvm_unit_file_t, lvm_var_run_t, lwiod_var_run_t, lwregd_var_run_t, lwsmd_var_run_t, mailman_lock_t, mailman_var_run_t, mandb_lock_t, mcelog_var_run_t, mdadm_unit_file_t, mdadm_var_run_t, memcached_var_run_t, minidlna_var_run_t, minissdpd_var_run_t, mip6d_unit_file_t, mirrormanager_var_run_t, mock_var_run_t, modemmanager_unit_file_t, mon_statd_var_run_t, mongod_unit_file_t, mongod_var_run_t, motion_unit_file_t, motion_var_run_t, mount_var_run_t, mpd_var_run_t, mrtg_lock_t, mrtg_var_run_t, mscan_var_run_t, munin_var_run_t, mysqld_unit_file_t, mysqld_var_run_t, mysqlmanagerd_var_run_t, naemon_var_run_t, nagios_var_run_t, named_conf_t, named_unit_file_t, named_var_run_t, netlabel_mgmt_unit_file_t, netlogond_var_run_t, neutron_unit_file_t, neutron_var_run_t, nfsd_unit_file_t, ninfod_run_t, ninfod_unit_file_t, nis_unit_file_t, nmbd_var_run_t, nova_unit_file_t, nova_var_run_t, nrpe_var_run_t, nscd_unit_file_t, nscd_var_run_t, nsd_var_run_t, nslcd_var_run_t, ntop_var_run_t, ntpd_unit_file_t, ntpd_var_run_t, numad_unit_file_t, numad_var_run_t, nut_unit_file_t, nut_var_run_t, nx_server_var_run_t, oddjob_unit_file_t, oddjob_var_run_t, openct_var_run_t, opendnssec_unit_file_t, opendnssec_var_run_t, openhpid_var_run_t, openshift_var_run_t, opensm_unit_file_t, openvpn_var_run_t, openvswitch_unit_file_t, openvswitch_var_run_t, openwsman_run_t, openwsman_unit_file_t, osad_var_run_t, pads_var_run_t, pam_var_console_t, pam_var_run_t, passenger_var_run_t, pcp_var_run_t, pcscd_var_run_t, pdns_unit_file_t, pdns_var_run_t, pegasus_openlmi_storage_var_run_t, pegasus_var_run_t, pesign_unit_file_t, pesign_var_run_t, phc2sys_unit_file_t, piranha_fos_var_run_t, piranha_lvs_var_run_t, piranha_pulse_var_run_t, piranha_web_var_run_t, pkcs11proxyd_unit_file_t, pkcs11proxyd_var_run_t, pkcs_slotd_lock_t, pkcs_slotd_var_run_t, pki_ra_lock_t, pki_ra_var_run_t, pki_tomcat_lock_t, pki_tomcat_unit_file_t, pki_tomcat_var_run_t, pki_tps_lock_t, pki_tps_var_run_t, plymouthd_var_run_t, policykit_var_run_t, polipo_pid_t, polipo_unit_file_t, portmap_var_run_t, portreserve_var_run_t, postfix_var_run_t, postgresql_lock_t, postgresql_unit_file_t, postgresql_var_run_t, postgrey_var_run_t, power_unit_file_t, pppd_lock_t, pppd_unit_file_t, pppd_var_run_t, pptp_var_run_t, prelude_audisp_var_run_t, prelude_lml_var_run_t, prelude_var_run_t, print_spool_t, privoxy_var_run_t, prosody_unit_file_t, prosody_var_run_t, psad_var_run_t, ptal_var_run_t, ptp4l_unit_file_t, pulseaudio_var_run_t, puppet_var_run_t, pwauth_var_run_t, pyicqt_var_run_t, qdiskd_var_run_t, qemu_var_run_t, qpidd_var_run_t, quota_nld_var_run_t, rabbitmq_unit_file_t, rabbitmq_var_lock_t, rabbitmq_var_run_t, radiusd_unit_file_t, radiusd_var_run_t, radvd_var_run_t, rasdaemon_unit_file_t, rdisc_unit_file_t, readahead_var_run_t, redis_unit_file_t, redis_var_run_t, regex_milter_data_t, restorecond_var_run_t, rhev_agentd_unit_file_t, rhev_agentd_var_run_t, rhnsd_unit_file_t, rhnsd_var_run_t, rhsmcertd_lock_t, rhsmcertd_var_run_t, ricci_modcluster_var_run_t, ricci_modstorage_lock_t, ricci_var_run_t, rkt_unit_file_t, rlogind_var_run_t, rngd_unit_file_t, rngd_var_run_t, rolekit_unit_file_t, roundup_var_run_t, rpcbind_unit_file_t, rpcbind_var_run_t, rpcd_lock_t, rpcd_unit_file_t, rpcd_var_run_t, rpm_var_run_t, rsync_var_run_t, rtas_errd_unit_file_t, rtas_errd_var_lock_t, rtas_errd_var_run_t, samba_unit_file_t, sanlk_resetd_unit_file_t, sanlock_unit_file_t, sanlock_var_run_t, saslauthd_var_run_t, sbd_unit_file_t, sbd_var_run_t, sblim_var_run_t, screen_var_run_t, semanage_read_lock_t, semanage_trans_lock_t, sendmail_var_run_t, sensord_unit_file_t, sensord_var_run_t, setrans_var_run_t, setroubleshoot_var_run_t, shorewall_lock_t, slapd_lock_t, slapd_unit_file_t, slapd_var_run_t, slpd_var_run_t, smbd_var_run_t, smokeping_var_run_t, smsd_var_run_t, snmpd_var_run_t, snort_var_run_t, sosreport_var_run_t, soundd_var_run_t, spamass_milter_data_t, spamd_var_run_t, spc_var_run_t, speech-dispatcher_unit_file_t, squid_var_run_t, srvsvcd_var_run_t, sshd_keygen_unit_file_t, sshd_unit_file_t, sshd_var_run_t, sslh_unit_file_t, sslh_var_run_t, sssd_public_t, sssd_unit_file_t, sssd_var_run_t, stapserver_var_run_t, stunnel_var_run_t, svnserve_unit_file_t, svnserve_var_run_t, swat_var_run_t, swift_lock_t, swift_unit_file_t, swift_var_run_t, sysfs_t, syslogd_unit_file_t, syslogd_var_run_t, system_cronjob_lock_t, system_cronjob_var_run_t, system_dbusd_var_run_t, systemd_gpt_generator_unit_file_t, systemd_home_t, systemd_hwdb_unit_file_t, systemd_logind_inhibit_var_run_t, systemd_logind_sessions_t, systemd_logind_var_run_t, systemd_machined_unit_file_t, systemd_machined_var_run_t, systemd_modules_load_unit_file_t, systemd_networkd_unit_file_t, systemd_networkd_var_run_t, systemd_passwd_var_run_t, systemd_resolved_unit_file_t, systemd_resolved_var_run_t, systemd_rfkill_unit_file_t, systemd_runtime_unit_file_t, systemd_timedated_unit_file_t, systemd_unit_file_t, systemd_vconsole_unit_file_t, targetd_unit_file_t, telnetd_var_run_t, tftpd_var_run_t, tgtd_var_run_t, thin_aeolus_configserver_var_run_t, thin_var_run_t, timemaster_unit_file_t, timemaster_var_run_t, tlp_unit_file_t, tlp_var_run_t, tmp_t, tmpfs_t, tomcat_unit_file_t, tomcat_var_run_t, tor_unit_file_t, tor_var_run_t, tuned_var_run_t, udev_var_run_t, uml_switch_var_run_t, usbmuxd_unit_file_t, usbmuxd_var_run_t, user_tmp_t, useradd_var_run_t, uucpd_lock_t, uucpd_var_run_t, uuidd_var_run_t, var_lock_t, var_run_t, varnishd_var_run_t, varnishlog_var_run_t, vdagent_var_run_t, vhostmd_var_run_t, virt_lock_t, virt_lxc_var_run_t, virt_qemu_ga_var_run_t, virt_var_run_t, virtd_unit_file_t, virtlogd_unit_file_t, virtlogd_var_run_t, vmtools_unit_file_t, vmware_host_pid_t, vmware_pid_t, vnstatd_var_run_t, vpnc_var_run_t, watchdog_var_run_t, wdmd_var_run_t, winbind_var_run_t, xdm_lock_t, xdm_var_run_t, xenconsoled_var_run_t, xend_var_run_t, xenstored_var_run_t, xserver_var_run_t, ypbind_unit_file_t, ypbind_var_run_t, yppasswdd_var_run_t, ypserv_var_run_t, ypxfr_var_run_t, zabbix_var_run_t, zarafa_deliver_var_run_t, zarafa_gateway_var_run_t, zarafa_ical_var_run_t, zarafa_indexer_var_run_t, zarafa_monitor_var_run_t, zarafa_server_var_run_t, zarafa_spooler_var_run_t, zebra_unit_file_t, zebra_var_run_t, zoneminder_unit_file_t, zoneminder_var_run_t.
                Then execute:
                restorecon -v 'USERNAME'


                *****  Plugin catchall (17.1 confidence) suggests   **************************

                If you believe that ttbd should be allowed create access on the local directory by default.
                Then you should report this as a bug.
                You can generate a local policy module to allow this access.
                Do
                allow this access for now by executing:
                # ausearch -c 'ttbd' --raw | audit2allow -M my-ttbd
                # semodule -X 300 -i my-ttbd.pp

where USERNAME is any user with access to ttbd.

This usually happens when the daemon is being ran from an installation that is not installed in the system; for remediation:

# semanage fcontext -a -t var_run_t /var/cache/ttbd-staging
# restorecon -v /var/cache/ttbd-staging

6.2.10. LDAP login fails with invalid credentials, but the password is right, I know¶

It’s been seeing that sometimes LDAP fails when the password is well known; seen when journalctl -eu ttbd@production | grep -i ldap reports:

Mar 21 02:08:30 HOSTNAME ttbd[10041]: I[10217] ttbd.login():249: \
  user USERNAME: invalid credentials from ldap://LDAPSERVER:PORT: \
  USERNAME: invalid credentials for LDAP ldap://LDAPSERVER:PORT: \
  {'info': '80090308: LdapErr: DSID-0C0903D9, \
    comment: AcceptSecurityContext error, data 52e, v2580', \
    'desc': 'Invalid credentials'}

Root causing help welcome, as it’s still unknown.

Restarting the daemon usually fixes it:

# systemctl restart ttbd@production

6.2.11. Freedom Board k64f¶

6.2.11.1. Freedom Board k64f, openocd fails to power up (unable to open CMSIS-DAP device 0xd28:0x204)¶

When an frdm target fails to power up and ttbd logs in journalctl -eu ttbd@production show:

Info : auto-selecting first available session transport "swd". To override use 'transport select <transport>'.
Info : add flash_bank kinetis k60.flash
adapter speed: 1000 kHz
none separate
cortex_m reset_config sysresetreq
Error: unable to open CMSIS-DAP device 0xd28:0x204

this usually means openocd is trying to open the /dev/hidrawN file that represents this device, but it can’t. Usually it happens because the way udev has configured it with uaccess, and it looks like:

# getfacl /dev/hidraw0
getfacl: Removing leading '/' from absolute path names
# file: dev/hidraw0
# owner: root
# group: root
user::rw-
user:gdm:rw-
group::---
mask::rw-
other::---

there is no group:: permission and because of ACL rules, even if there is g+rw in the Unix bits. The solution has been implemented with udev rule that forces any CMSIS devices to be owned by group ttbd.

Make sure your ttbd-zephyr version is at least >= 0.10_6_N-M with:

# rpm -qa | grep ttbd
ttbd-0.10_8_g69bcfa3-1.noarch
ttbd-zephyr-0.10_8_g69bcfa3-1.noarch

which can be quickly updated with:

# dnf update --best --allowerasing ttbd ttbd-zephyr

(update instructions).

6.2.11.2. FRDM k64f boards fail to power up, OpenOCD prints message about RESET/WDOG loop¶

An FRDM board fails to power up because OpenOCD cannot initialize, printing in the journal a message similar to:

log: Info : SWD DPIDR 0x2ba01477
log: Warn : **** Your Kinetis MCU is probably locked-up in RESET/WDOG loop. ****
log: Warn : **** Common reason is a blank flash (at least a reset vector).  ****
log: Warn : **** Issue 'kinetis mdm halt' command or if SRST is connected   ****
log: Warn : **** and configured, use 'reset halt'                           ****
log: Warn : **** If MCU cannot be halted, it is likely secured and running  ****
log: Warn : **** in RESET/WDOG loop. Issue 'kinetis mdm mass_erase'         ****
log: Info : SWD DPIDR 0x2ba01477
log: Error: Failed to read memory at 0xe000ed00
log: Info : accepting 'tcl' connection on tcp/38743

(which can be found with journalctl -au ttbd@production | grep BOARDNAME | less -S)

this means that the board cannot boot properly because it is stuck in a Reset/Watchdog loop. The watchdog resets the board before it can load the firmware. OpenOCD is not able to get in there. The green led next to the SDAUSB is on and next to it, the RST red light blinks.

The procedure to fix, involves, in the following order:

set the up the target in debug mode
issue OpenOCD commands to stop the loop
copy a known firmware image to its flash drive
reset the board to normal configuration

this is accomplished by as script in the examples directory:

$ tcf run -vvt frdm-NN /usr/share/tcf/examples/test_frdm_recover.py

6.2.11.2.1. Manual execution of the procedure¶

6.2.11.3. Set the up the target in debug mode¶

Disable the board so none tries to use it:

$ tcf disable frdm-NN; tcf release -f frdm-NN; tcf acquire frdm-NN

Note

this might have to be done a few times, as other jobs might acquire the target in between; it helps to do it all in a single line.

Set OpenOCD to operate in relaxed mode, so it doesn’t require the target to be in standing order to succesfully start:

$ tcf power-off frdm-NN
$ tcf property-set frdm-NN openocd-relaxed True

Power cycle the target (will take a while as it will retry):

$ tcf power-cycle frdm-NN

it might take a while as OpenOCD tries to start. The green red light next to green power light is on, blinking

6.2.11.4. Issue OpenOCD commands to stop the loop¶

Issue the OpenOCD command kinetis mdm halt:

$ tcf debug-openocd frdm-NN "kinetis mdm halt"

Repeat this command until any the following things happen:

the following output is printed:

Halt timed out, wake up GDB.
Polling target k60.cpu failed, trying to reexamine
MDM: Chip is unsecured. Continuing.
k60.cpu: hardware has 6 breakpoints, 4 watchpoints
k60.cpu: target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc

nothing is printed
the red led stops blinking

Verify with the following command reporting the CPU is halted:

$ tcf debug-openocd frdm-28y "targets"
TargetName         Type       Endian TapName            State
--  ------------------ ---------- ------ ------------------ ------------
0* k60.cpu            cortex_m   little k60.cpu            halted

Retry otherwise, with any of:

$ tcf debug-openocd frdm-NN "kinetis mdm halt"
$ tcf debug-openocd frdm-NN "reset halt"

Sometimes you might have to issue a kinetis mdm mass_erase command and press the RESET button on the board. Follow the messages.

6.2.11.5. Copy a known firmware image to its flash drive¶

Flash using openocd commands:

$ tcf debug-openocd frdm-NN "flash write_image erase /var/lib/ttbd/frdm_k64f_recovery.bin 0"
auto erase enabled
Flash Configuration Field written.
Reset or power off the device to make settings effective.
wrote 12288 bytes from file /var/lib/ttbd/frdm_k64f_recovery.bin in 0.998723s (12.015 KiB/s)

might need to retry a couple times.

6.2.11.6. Reset the board to normal configuration¶

Remove the relaxed setting for openocd:

$ tcf property-set frdm-NN openocd-relaxed

power off the target:
```
$ tcf power-off frdm-NN
```

run a healtcheck on it:

$ tcf healthcheck frdm-NN
...
frdm-NN: healthcheck completed

finally, enable the target for normal operation:
```
$ tcf enable frdm-NN
```

6.2.12. Arduino101¶

6.2.12.1. Arduino101: openocd fails to power up (targets report tap-disabled)¶

When the target fails to power up and in the daemon log, journalctl -eu ttbd@production shows something like:

init verification JTAG (1/8): Error condition: n/a
init verification JTAG (1/8): output[before]: TargetName         Type       Endian TapName            State
init verification JTAG (1/8): output[before]: --  ------------------ ---------- ------ ------------------ ------------
init verification JTAG (1/8): output[before]: 0  quark_se.quark     quark_se   little quark_se.quark     tap-disabled
init verification JTAG (1/8): output[before]: 1* quark_se.arc-em    arc32      little quark_se.arc-em    tap-disabled
init verification JTAG (1/8): output[before]: ^Z

Note the tap-disabled part–this says that OpenOCD can talk to the JTAG, but they JTAG can’t talk to the CPU. There are usually the following reasons for it:

the JTAG cable is not properly connected; ensure the cable is the right one and it is properly connected, paying close attention to the pinout (for example, as described in the fixture documented in conf_00_lib.arduino101_add()).
the hardware itself might have a hardware block against JTAGs. Disable it (if possible).