Devops

AWS Control Tower Enrollment Gotchas

by
person looking down from on top of a control tower at an airport.

I have been working on moving a collection of about 20-30 AWS accounts from two different AWS Organizations into a new AWS Organization with Control Tower enabled. During the process I have run into a number of different blockers and issues which have not always had the most obvious solutions due to cryptic errors that the AWS Control Tower enrollment process shows.

This blog post lists out some of the issues I have run into during account migrations, and what the underlying reasons and resolutions ended up being.

You can’t use the AWS IAM Identity Center provided user or account root user when enrolling accounts

If you login to the management account and try to enroll member accounts into AWS Control Tower, you’ll get a cryptic error in the AWS Console like:

An unknown error occurred. Try again later, or contact AWS Support. No launch paths found for resource: prod-xxxxxxxxxxxx

In my case, I was logged in initially with the AWS IAM Identity Center provisioned user account for the management account. This account is identified by the default Display Name of: AWS Control Tower Admin. The solution was to create a single IAM user with AdministratorAccess managed policy attached, then use the AWS Service Catalog console to add the user to the Portfolio Access on the AWS Control Tower Account Factory Portfolio item.

Once that was done, I could use the standard IAM user to login and successfully enroll member accounts into Control Tower.

In theory it should be possible to use the AWS IAM Identity Center provided user to enroll accounts into AWS Control Tower by ensuring it is added to the relevant Groups relating to provisioning and enrollment, but even after this I was unable to. Using a standard IAM user added to Portfolio access as above worked for me.

Forgetting to create the AWSControlTowerExecution IAM role in a member account before enrolling

Don’t forget to create the AWSControlTowerExecution IAM role with Principal ID access for the main Control Tower management account to assume during account enrollment.

This IAM role needs to be created before you can enroll accounts into AWS Control Tower. At a high level, it should be named AWSControlTowerExecution, it should have the AWS managed policy AdministratorAccess attached, and it should have a trust policy attached like the following:

{
   "Version":"2012-10-17",
   "Statement":[
      {
      "Effect":"Allow",
      "Principal":{
         "AWS": "arn:aws:iam::Management Account ID:root"
         },
         "Action": "sts:AssumeRole",
         "Condition": {} 
      }
   ]
  }

The process and role requirements are detailed in this AWS page.

AWS Config in existing accounts before enrolling them with AWS Control Tower

Here’s another issue that caught me out on one account. The account had it’s own AWS Config service Recorder and Delivery Channel setup (but it wasn’t actually being actively used).

When AWS Control Tower enrolls an account, it configures AWS Config with various best practices and configuration settings to record config changes. If there is existing AWS Config in the account, the enrollment process fails.

In my case I got the lovely error message:


I was scratching my head on this one until I decided to use the Update feature in the Control Tower console to try enrollment once more (where the account enrollment status showed Failed). The second time around, a clearer message was displayed:

AWS Control Tower could not enroll your account for the following reason: AWS Control Tower cannot create an AWS Config delivery channel because one already exists. To continue, delete the existing delivery channel and try again.

Trying to delete the AWS Config delivery channel was impossible though. AWS Control Tower applies guard rails via SCP, and because this account was partly enrolled, and now moved to an Organizational Unit (OU) that had SCPs attached to prevent AWS Config changes, it was impossible to remove the AWS Config settings that were blocking the enrollment process.

The fix was to ‘unmanage’ or un-enroll the AWS account (even though it wasn’t fully enrolled) using the AWS Control Tower console. Once that was done, and the account was moved back into the Organization root OU, I was able to use the AWS CLI to remove the offending AWS Config settings.

# find any delivery channels
aws configservice describe-delivery-channels

# describe delivery channel status
aws configservice describe-delivery-channel-status

# stop configuration recorder named 'default' (seen in describe above)
aws configservice stop-configuration-recorder --configuration-recorder-name default

# delete configuration recorder named 'default'
aws configservice delete-configuration-recorder --configuration-recorder-name default

With the AWS Config preferences removed, the AWS Config console should show the initial ‘Set up AWS Config’ option – i.e. nothing is configured. At this stage it’s possible to enroll the account into Control Tower successfully.

Feature Photo by id23: https://www.pexels.com/photo/person-on-air-traffic-control-tower-10893728/

DTrace/dtruss – an alternative to strace on macOS

by
dtruss/DTrace output

If you’re on macOS and wanting to find out what system calls (syscalls) are being made by a process or application, you may have found that strace is not an option.

This is a quick post to demonstrate how to use DTrace to monitor system calls made by a process on macOS.

System Integrity Protection

First things first, you may find that newer mac systems have SIP (System Integrity Protection) enabled and that running dtruss will give you a warning about this. You’ll also likely not see much output at all.

It’s possible to disable this, but first be mindful of the security implications of disabling SIP, and second, be sure to set things back again once you’re done.

To disable SIP for DTrace, reboot your system into recovery mode (on my Apple M2 system I needed to shutdown completely, then power on, holding the button in until the recovery and startup options appeared).

Then I booted into the recovery tools, entered my usual user login, and used the menu bar at the top of the screen to open Utilities -> Terminal

Then ran:

csrutil enable --without dtrace

Tracing with DTrace/dtruss

dtruss is a DTrace version of truss, which is a Unix-specific command that prints out system calls made by a program.

We’ll use dtruss to see the syscalls made by a simple node application.

After this, a reboot back into the normal OS had me back at the terminal and ready to run dtruss.

Here’s a simple nodeJS program that we’ll trace.

const fs = require("fs");

const fd = fs.openSync("foo.txt", "r+");
fs.writeSync(fd, "foo", "utf8");
const test = fs.readFileSync("foo.txt");
console.log(test);

We load the fs module (used to make file operations), create a file descriptor pointing to existing file foo.txt in write mode, write the text “foo” to it, read it back by opening the file, and then write the result to stdout.

Let’s trace this program now.

sudo dtruss node example.js &> dtruss.log

Once that completes, you’ll see a lot of output detailing various system calls in the dtruss.log file. Much of this is made by the node environment itself, and a small part will be the actual example.js program. Scroll through until you find the part where the example.js script is loaded. It should start with

read(0x14, "const fs = require

Shortly after that you’ll see the syscalls made by the node program.

  • open (to open foo.txt)
  • write (to write the string ‘foo’ into the foo.txt file)
  • open (to again open foo.txt and read the content)
  • fstat64 which gets file status
  • write (with Buffer containing hexadecimal value for foo 66 6f 6f) to write the value to stdout.
read(0x14, "const fs = require(\"fs\");\n\nconst fd = fs.openSync(\"foo.txt\", \"r+\");\nfs.writeSync(fd, \"foo\", \"utf8\");\nconst test = fs.readFileSync(\"foo.txt\");\nconsole.log(test);\n\0", 0xA1)                = 161 0
close_nocancel(0x14)             = 0 0
open("foo.txt\0", 0x1000002, 0x0)                = 20 0
write(0x14, "foo\0", 0x3)                = 3 0
open("foo.txt\0", 0x1000000, 0x0)                = 21 0
fstat64(0x15, 0x16B814420, 0x0)          = 0 0
read(0x15, "foo\0", 0x3)                 = 3 0
close_nocancel(0x15)             = 0 0
ioctl(0x1, 0x4004667A, 0x16B813D4C)              = -1 Err#25
ioctl(0x1, 0x40487413, 0x16B813D50)              = -1 Err#25
fstat64(0x1, 0x16B813DC8, 0x0)           = 0 0
mprotect(0x109348000, 0x34000, 0x3)              = 0 0
madvise(0x109348000, 0x34000, 0x8)               = 0 0
mprotect(0x109348000, 0x34000, 0x5)              = 0 0
madvise(0x109348000, 0x34000, 0x8)               = 0 0
write(0x1, "<Buffer 66 6f 6f>\n\0", 0x12)                = 18 0
fstat64(0x0, 0x16B817520, 0x0)           = 0 0
fcntl(0x0, 0x3, 0x3C044618)              = 65538 0
__pthread_sigmask(0x1, 0x16B81751C, 0x0)                 = 0 0
ioctl(0x0, 0x80487414, 0x107D6D640)              = 0 0
__pthread_sigmask(0x2, 0x16B81751C, 0x0)                 = 0 0
fstat64(0x1, 0x16B817520, 0x0)           = 0 0
fcntl(0x1, 0x3, 0x3C044618)              = 65537 0
fstat64(0x2, 0x16B817520, 0x0)           = 0 0
fcntl(0x2, 0x3, 0x3C044618)              = 65537 0

Finally, don’t forget to reboot into recovery mode and set SIP back to the default.

Vim Cheatsheet

by
vim cheatsheet feature

A quick vim cheatsheet for those of us who enjoy using vim, but don’t use it often enough to have the command sequences committed to memory.

Starting with the simplest operations, and then moving on to a few more complex and difficult to remember ones.

Remember that you always start in normal/visual mode. To enter insert mode to start entering text, you can use i or I.

  • Quit without saving: :q!
  • Quit and write changes: :wq
  • Enter insert mode at beginning or end of current line: I
  • Enter insert mode at the current position of the cursor: i
  • Escape current mode: Esc
  • Search forward for text pattern: /text
  • Search backward for text pattern: ?text
  • Go to bottom of page: G
  • Go to top of page: gg

Copy/paste style vim operations:

  • Copy and line (yank): yy
  • Paste a ‘yanked’ line: p (after) or P (before) cursor.
  • Delete character before cursor: X and after cursor: x
  • Delete current line: dd
  • Delete current line and start insert mode: cc

Deleting

  • Delete all lines in file: ddgD

Sorting

  • Sort all lines (no range): :sort
  • Sort all lines (no range, reversed): :sort!
  • Add options to the sort command:
    • i – ignore case
    • n – sort based on the first decimal on the line
    • f – sort based on the first float on the line

Some useful, yet more arcane vim operations:

  • Clear all lines in a file (1 is the first line, $ is the last line, and d is delete): :1,$d
  • Insert the result of Vimscript expressions into your file:
    • Enter Vim’s command line: Enter INSERT mode, and type CTRL + R =
    • Type a Vimscript expression, for example system("ls") and press ENTER
    • The output of the command ls will be inserted into the buffer.

Example of the above vimscript expression register result insert:

This of course only scratches the surface of the surface of what can be done. It’s just a quick little vim cheatsheet that I’ll refer back to when I get a little rusty.

The advanced or more arcane commands are the useful ones that I tend to forget when I’m not using them on a daily basis. I’ll certainly be recalling this post and updating those with new ones that I find useful in the future.

Packing Executable Files to Reduce Distribution Size with UPX

by

Recently I’ve been playing around with Ultimate Packer for Executables (UPX) to reduce a distributable CLI application’s size.

The application is built and stored as an asset for multiple target platforms as a GitHub Release.

I started using UPX as a build step to pack the executable release binaries and it made a big difference in final output size. Important, as the GitHub Release assets cost money to store.

UPX has some great advantages. It supports many different executable formats, multiple types of compression (and a strong compression ratio), it’s performant when compressing and decompressing, and it supports runtime decompression. You can even plugin your own compression algorithm if you like. (Probably a reason that malware authors tend to leverage UPX for packing too).

In my case I had a Node.js application that was being bundled into an executable binary file using nexe. It is possible to compress / pack the Node.js executable before nexe combines it with your Node.js code using UPX. I saw a 30% improvement in size after using UPX.

UPX Packing Example

Let’s demonstrate UPX in action with a simple example.

Create a simple C application called hello.c that will print the string “Hello there.”:

#include "stdio.h"

int main() {
  printf("Hello there.\n");
  return 0;
}

Compile the application using static linking with gcc:

gcc -static -o hello hello.c

Note the static linked binary size of your new hello executable (around 876 KB):

sean@DESKTOP-BAO9C6F:~/hello$ gcc -static -o hello hello.c
sean@DESKTOP-BAO9C6F:~/hello$ ls -la
total 908
drwxr-xr-x  2 sean sean   4096 Oct 24 21:27 .
drwxr-xr-x 26 sean sean   4096 Oct 24 21:27 ..
-rwxr-xr-x  1 sean sean 896336 Oct 24 21:27 hello
-rw-r--r--  1 sean sean  23487 Oct 21 21:33 hello.c
sean@DESKTOP-BAO9C6F:~/hello$

This may be a paltry example, but we’ll take a look at the compression ratio achieved. This can of course, generally be extrapolated for larger file sizes.

Analysing our Executable Before Packing

Before we pack this 876 KB executable, let’s analyse it’s entropy using binwalk. The entropy will be higher in parts where the bytes of the file are more random.

Generate an entropy graph of hello with binwalk:

binwalk --entropy --save hello
entropy analysis with binwalk before running upx to pack the executable.

The lower points of entropy should compress fairly well when upx packs the binary file.

UPX Packing

Finally, let’s pack the hello executable with UPX. We’ll choose standard lzma compression – it should be a ‘friendlier’ compression option for anti-virus packages to more widely support.

upx --best --lzma -o hello-upx hello

Look at that, a 31.49% compression ratio! Not bad considering the code itself is really small and most of the original hello executable size is a result of static linking.

sean@DESKTOP-BAO9C6F:~/hello$ upx --best --lzma -o hello-upx hello
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar &amp;amp; John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    871760 ->    274516   31.49%   linux/amd64   hello-upx

Packed 1 file.
sean@DESKTOP-BAO9C6F:~/hello$

Running the packed binary still works perfectly fine. UPX cleverly re-arranges the binary file to place the compressed contents in a specific location, adds a new entrypoint and a bit of logic to decompress the data when the file is executed.

sean@DESKTOP-BAO9C6F:~/hello$ ./hello-upx
Hello there.

UPX is a great option to pack / compress your files for distribution. It’s performant and supports many different executable formats, including Windows and 64-bit executables.

A great use case, as demonstrated in this post is to reduce executable size for binary distributions, especially when (for example) cloud storage costs, or download sizes are a concern.

Using JSONPath Queries on JSON Data

by
JSON data for querying with JSONPath

JSONPath does for JSON processing what XPath (defined as a W3C standard) does for XML. JSONPath queries can be super useful, and are a great addition to any developer or ops person’s toolbox.

You may want to do a quick data query, test, or run through some JSON parsing scenarios for your code. If you have your data easily available in JSON format, then using JSONPath queries or expressions can be a great way to filter your data quickly and efficiently.

JSONPath 101

JSONPath expressions use $ to refer to the outer level object. If for example you have an array at the root, $ would refer to that array.

When writing JSONPath expressions, you can use dot notation or bracket notation. For example:

  • $.animals.land[0].weight
  • $['animals']['land'][0]['weight']

You can use filter expressions to filter out specific items in your queries. For example: ?(<bool expression>)

Here is an example that would filter our collection of land animals to show only those heavier than 50.0, returning their names:

$.animals.land[?(@.weight > 50.0)].name

The wildcard character * is used to select all objects or elements.

Note the @ symbol that is used to select the ‘current’ item being iterated in the boolean expression.

There are more JSONPath syntax elements to learn about, but the above are what I find most useful and commonly required.

JSONPath Query Example

Here is a chunk of JSON data, and some basic queries that show how you can easily filter down the dataset and select what you need.

JSONPath Queries – Example 1

Find all “Report runs” where root.id is equal to a specific value:

$.runs[?(@.root.id=="af1bcd6b-406f-43f9-86b3-9f01ee211ddc")]

JSONPath Queries – Example 2 (AND operator)

Find all “Report runs” where root.id is equal to a specific value, and shell.id is equal to a specific value:

$..runs[?(@.root.id=="af1bcd6b-406f-43f9-86b3-9f01ee211ddc" && @.shell.id=='d743537e393d')]

Useful JSONPath Resources

Use this webapp to write and test JSONPath expressions live in your browser.