Like a Duck to Merchant Banking

<All title credit goes to Neil Gaiman and Terry Pratchett.>

Well, it has been a while since I made my last post. Honestly, I have been inundated with a massive amount of work over the last few months so this self-improvement project has taken a bit of a hiatus. I did, however, work on improving my social media skills which is no small feat for a guy who prefers discussions in a bar than taking photos of food.

Moving on to today’s topic, though. I had an opportunity over the weekend to attend Texas Linux Fest in Austin, TX. While there, I listened to a presentation by Tammy Bütow on Chaos Engineering. Being someone who is constantly dealing with broken environments, it greatly intrigued me that there are people who do this on purpose. So I looked into Gremlin, the company Tammy is the Principal SRE for, and was given the opportunity to take their demo product out for a spin.

From that experience, I have created a small write-up of my experience and decided to share it with all of you.


The  Plan

I decided to keep everything as simple as possible so I could spend more time playing with Gremlin and less time troubleshooting some complex environment. Thus, I began with a basic cloud server and created the introductory build every Linux tech goes through: LAMP stack with a simple static webpage.

After getting this up and running, the idea was to install Gremlin and put my little cloud-server-that-could through its paces. The plan was in place, now for the execution.


The Execution

What follows are the commands I ran after spinning a cloud server. I used a Rackspace cloud server because it was free, but any server where you have root access should work just fine.

Server Specs: 2GB RAM, 40GB vHDD, 2 vCPU, Ubuntu 18.04

Note: Gremlin does not list Ubuntu 18.04 as a supported version, however I decided to try and see if works. The directions for installation were exactly the same as 16.04. Any issues you experience following my build are most likely not supported by Gremlin and I do not mean to endorse doing the dumb things I do in any way, shape or form. I just like making life a little more like Dark Souls whenever I can.

After you have a server ready to work with, here are the tasks I did to complete the underlying “web server”. It includes the packages I installed, as well as the WordPress and Gremlin installation.


  • apt-get install apache2
    • This will be the web server.
  • apt-get install mysql-server
    • The database for the psuedo-web site.
      • Note: While not necessary for my project as I blew up the server shortly after, it is ALWAYS recommended you run mysql_secure_installation after installing MySQL. Failure to do so may result in a very bad day.
    • The following are to create a WordPress database and user.
        • mysql
        • CREATE DATABASE wordpress;
        • GRANT ALL ON wordpress.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'password';
          • Be sure to change 'password' to reflect whatever password works for you.
        • EXIT;
  • apt-get install php
    • The “P” in LAMP stack.
  • apt-get install php-curl php-gd php-mbstring php-mcrypt php-xml php-xmlrpc php-mysql
    • These modules are intended to help WordPress and add some security to the web site. Not installing them will cause WordPress to panic.
  • cd /var/www/
    • This is where our web site will live.
  • wget
    • Time to fetch WordPress
  • tar -xzvf latest.tar.gz
    • Unpack that stuff!
    • This will create a directory called “wordpress”.
  • chown -R www-data:www-data /var/www/wordpress/
    • Make sure Apache has ownership of the site location.
  • cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/wordpress.conf
    • This will take the default configuration and make a copy to use for the WordPress site.
  • vim /etc/apache2/sites-available/wordpress.conf
    • I like vim. I don’t care who knows it. I <3 vim!
      • Any rate, you are in this file to change a few options. Be sure to remove any # (Octothorpe) that may precede these 4 lines.
        • ServerName
          • If you have an actual website name ( that you own and want to use, put that here. Otherwise, localhost works. For the purposes of testing Gremlin, this is perfectly fine since there is no concern than anyone ever see this website.
        • ServerAlias
          • This is where you would put the name of the site, with the preceding “www” (
        • ServerAdmin webadmin@localhost
          • Just leave this as is. It has little bearing on what we are doing, but not having can cause some issues with Apache and WordPress.
        • DocumentRoot /var/www/wordpress
          • This is where the site lives. ‘Nuff said.
  • a2ensite wordpress.conf
    • This makes Apache read the newly modified configuration.
  • systemctl reload apache2
    • And now to make sure Apache is presenting the new information.
  • At this point work has to be done to finish the WordPress installation. This is accomplished by going to the web site’s page. So, whatever you set ServerName to in the wordpress.conf is the page you need to visit now. Once there, you will be presented with a menu selection and setup options. They are all pretty straightforward. Just remember that when it asks for the database information, the details are found in the steps when we installed MySQL.

Now for Gremlin. If you are not using their demo, you will only be able to install the tools and then use them locally. You will not have access to the dashboard or be able to use some of the more in-depth functions of Gremlin (Like creating teams and monitoring results.), but for testing purposes it should suffice.

These directions come from their own docs located here. I am only including the portions I used for my own testing, but if you visit their docs page, there are a multitude of installation methods.

  • echo "deb release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list
    • Add the Gremlin repo to your source list so that you can install from apt.
  • sudo apt-key adv --keyserver --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6
    • Import the GPG keys.
  • sudo apt-get update && sudo apt-get install -y gremlin gremlind
    • Finally, install Gremlin and its daemon.
  • gremlin syscheck
    • This will verify that the checks are working as intended.
  • gremlin attack <TYPE>
    • Fires off a desired attack type.  Here is the help info for the command:
      • Usage: gremlin attack TYPE [type-specific-options]
        Type "gremlin help attack TYPE" for more details:
        blackhole # An attack which drops all matching 
        network traffic
        cpu # An attack which consumes CPU resources
        io # An attack which consumes IO resources
        latency # An attack which adds latency to all 
        matching network traffic
        memory # An attack which consumes memory
        packet_loss # An attack which introduces packet 
        loss to all matching network traffic
        shutdown # An attack which forces the target to 
        dns # An attack which blocks access to DNS servers
        time_travel # An attack which changes the system 
        disk # An attack which consumes disk resources
        process_killer # An attack which kills the specified 

If you have requested a demo version, I highly recommend following the steps on the Gremlin docs pages for registering the server as a client and playing with their dashboard. I have had a blast pushing my test servers to the limits just to see what could be done and I can’t wait to see what is coming next!



Share the journey

Leave a Reply

Your email address will not be published. Required fields are marked *