A Myriad of Colors

We live in a rainbow of chaos. -Paul Cézanne

 

This post was intended to go up sooner, but life and work being what they are I fell a bit behind. It is here now, though, and continues through my experiences with Gremlin and Chaos Engineering.

If you have not visited their site and requested a demo of Gremlin, this next bit will be incredibly difficult. You do have to be able to run the commands which require a Team ID and a corresponding Team Secret.

Alright, there are few basics to cover real quick before we dive into the actual attacks. First is how you will monitor the attacks. If you are on a Linux or Mac machine locally, this is an easy task because you can simply open a second terminal session and connect to your server for monitoring purposes. Windows folks, we have a bit of explaining to do.

There are two simple options at this point, as PuTTY does not play well with multiple terminals. They are tmux and screen. Since both do relatively the same thing, for the purpose of this discussion I am going to show a limited amount of commands to be able to use tmux effectively and quickly.

TMUX: QUICK & DIRTY

First, create a new tmux session named “gremlin”:

tmux new -s gremlin

Now, this will launch into a tmux screen. You should see the name of your tmux session in the lower left corner. In order to detach from the session, but leave the work in that session run, do the following:

Press <CTRL> and “b”, release and then press “d”.

You may see commands similar to this represented as:

^b d

The “^” before the “b” means to press the <CTRL> key along with the letter. Just some fancy shorthand.

In order to reattach to the session:

tmux attach-session -t gremlin

This will now give you the ability to have a second session to run tests inside of and then jump back to our main session for monitoring. Now to catch back up with everyone else.

GETTING SETUP

If you have not already, please go ahead and follow the short walk-through over here to get the base install created. Building on that foundation, we need to add one more package to help with monitoring and then initialize our Gremlin application.

The package we need is called “iotop” and it shows disk read/write usage. This will be important when we run the disk attacks later. To install it onto the Ubuntu 18.04 server it is just a matter of:

apt install iotop

After that is done, we need to run the init process with Gremlin. When you do this, it will ask for your Team ID and Team Secret. If you have not requested a demo, then you will need to do so at this point.

gremlin init
<TEAM_ID>
<TEAM_SECRET>

Now that all the planning and prepping is done, on to the fun stuff!

GREMLIN ATTACKS

All of the attacks have their own help file that can be accessed with:

gremlin help attack <TYPE>

If you can’t remember the attack name, try the following to see a list:

gremlin help attack

In this post I am just going to cover the 4 Resource attacks: CPU, Disk, I/O and Memory. Each one will include its help file at the start, then any notes I noticed while playing with them, an example command and finally a way to monitor the command in action.

CPU

gremlin help attack cpu
Usage: gremlin attack cpu [-l LENGTH] [-c CORES]

An attack which consumes CPU resources

Options:
-l, --length LENGTH The length of the attack (seconds)
-c, --cores CORES The number of cores to try to utilize

 

If you do not know how many cores your server has, you can use top to see them. Once the system information is pulled up, hit the number 1 and the cores will be listed at the top of the screen under the section “%Cpu”. The count starts with 0.

Here is a simple attack that lasts sixty seconds and stresses a single core:

gremlin attack cpu -l 60 -c1

To monitor this, in your alternate session, go back to top and pull up the core list again. You should see one of the cores gaining a fair amount of usage over the course of sixty seconds.

Disk

gremlin help attack disk
Usage: gremlin attack disk [-l LENGTH] [-d DIR] 
[-w WORKERS] [-b SIZE] [-p PERCENTAGE]

An attack which consumes disk resources

Options:
-l, --length LENGTH The length of the attack (seconds)
-d, --dir DIR The root directory to run the disk attack
-w, --workers WORKERS
The number of disk-write workers to execute
-b, --block-size SIZE
Number of Kilobytes (KB) that are read/written at a
time
-p, --percent PERCENTAGE
Percent of Volume to fill (0-100)

Some recommendations from my mistakes, if you do not have a specific use-case planned out ahead of time:
1) Use /tmp as the root directory. That way, if something does go wrong and the server crashes, it will empty the directory automatically on reboot.
2) Unless you have partitioned set up so that your entire disk drive does not get filled, do not set the percentage to 100.

Example:

gremlin attack disk -l 60 -d /tmp -w 1 -b 4 -p 75

Use df -h to keep an eye on disk space and to avoid having to type it over and over, work in the watch command:

watch -n5 df -h

This will keep it refreshing the command every five seconds until you break (^c) the command.

IO

gremlin help attack io
Usage: gremlin attack io [-l LENGTH] [-w WORKERS] 
[-d DIR] [-m MODE] [-s SIZE] [-c COUNT]

An attack which consumes IO resources

Options:
-l, --length LENGTH The length of the attack (seconds)
-w, --workers WORKERS
The number of io workers to execute
-d, --dir DIR The root directory to run the io attack
-m, --mode MODE The io mode to execute [r,w,rw]
-s, --block-size SIZE
Number of Kilobytes (KB) that are read/written at a
time
-c, --block-count COUNT
The number of blocks read/written by workers

Just like with the Disk attack, it is best to have this target the /tmp directory. Also, with a smaller, cloud server, it is best to keep the block-size around 4KBs and simply increase the worker count if you are wanting to hyper-stress the server.

Example:

gremlin attack io -l 60 -w 1 -d /tmp -m rw -s 4 -c 1

Now, this is the part where iotop comes into play. We will use it to monitor the read and write levels on the disk in our alternate session.

Memory

gremlin help attack memory
Usage: gremlin attack memory [-l LENGTH] [-g GBS] [-m MBS]

An attack which consumes memory

Options:
-l, --length LENGTH The length of the attack (seconds)
-g, --gigabytes GBS The number of gigabytes to allocate
-m, --megabytes MBS The number of megabytes to allocate

The only thing to really pay attention to here is that both -g and -m cannot be used in the same command. It will default to MBs, regardless of the order the two are placed in the command.

gremlin attack memory -60 -g 1

Similar to the disk monitoring, we will use the watch command again:

watch -n5 free -m

 

Wrap-Up

That covers the basics of the first four attacks. Of course, all of these can be launched from inside the dashboard, but personally, I like tearing apart the commands in the terminal first. I believe that once I have a grasp of how the process works, then I can use a GUI or dashboard, if I really feel it will save me time. Until next time!

Like a Duck to Merchant Banking

<All title credit goes to Neil Gaiman and Terry Pratchett.>

Well, it has been a while since I made my last post. Honestly, I have been inundated with a massive amount of work over the last few months so this self-improvement project has taken a bit of a hiatus. I did, however, work on improving my social media skills which is no small feat for a guy who prefers discussions in a bar than taking photos of food.

Moving on to today’s topic, though. I had an opportunity over the weekend to attend Texas Linux Fest in Austin, TX. While there, I listened to a presentation by Tammy Bütow on Chaos Engineering. Being someone who is constantly dealing with broken environments, it greatly intrigued me that there are people who do this on purpose. So I looked into Gremlin, the company Tammy is the Principal SRE for, and was given the opportunity to take their demo product out for a spin.

From that experience, I have created a small write-up of my experience and decided to share it with all of you.

 

The  Plan

I decided to keep everything as simple as possible so I could spend more time playing with Gremlin and less time troubleshooting some complex environment. Thus, I began with a basic cloud server and created the introductory build every Linux tech goes through: LAMP stack with a simple static webpage.

After getting this up and running, the idea was to install Gremlin and put my little cloud-server-that-could through its paces. The plan was in place, now for the execution.

 

The Execution

What follows are the commands I ran after spinning a cloud server. I used a Rackspace cloud server because it was free, but any server where you have root access should work just fine.

Server Specs: 2GB RAM, 40GB vHDD, 2 vCPU, Ubuntu 18.04

Note: Gremlin does not list Ubuntu 18.04 as a supported version, however I decided to try and see if works. The directions for installation were exactly the same as 16.04. Any issues you experience following my build are most likely not supported by Gremlin and I do not mean to endorse doing the dumb things I do in any way, shape or form. I just like making life a little more like Dark Souls whenever I can.

After you have a server ready to work with, here are the tasks I did to complete the underlying “web server”. It includes the packages I installed, as well as the WordPress and Gremlin installation.

 

  • apt-get install apache2
    • This will be the web server.
  • apt-get install mysql-server
    • The database for the psuedo-web site.
      • Note: While not necessary for my project as I blew up the server shortly after, it is ALWAYS recommended you run mysql_secure_installation after installing MySQL. Failure to do so may result in a very bad day.
    • The following are to create a WordPress database and user.
        • mysql
        • CREATE DATABASE wordpress;
        • GRANT ALL ON wordpress.* TO 'wordpressuser'@'localhost' IDENTIFIED BY 'password';
          • Be sure to change 'password' to reflect whatever password works for you.
        • FLUSH PRIVILEGES;
        • EXIT;
  • apt-get install php
    • The “P” in LAMP stack.
  • apt-get install php-curl php-gd php-mbstring php-mcrypt php-xml php-xmlrpc php-mysql
    • These modules are intended to help WordPress and add some security to the web site. Not installing them will cause WordPress to panic.
  • cd /var/www/
    • This is where our web site will live.
  • wget https://wordpress.org/latest.tar.gz
    • Time to fetch WordPress
  • tar -xzvf latest.tar.gz
    • Unpack that stuff!
    • This will create a directory called “wordpress”.
  • chown -R www-data:www-data /var/www/wordpress/
    • Make sure Apache has ownership of the site location.
  • cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/wordpress.conf
    • This will take the default configuration and make a copy to use for the WordPress site.
  • vim /etc/apache2/sites-available/wordpress.conf
    • I like vim. I don’t care who knows it. I <3 vim!
      • Any rate, you are in this file to change a few options. Be sure to remove any # (Octothorpe) that may precede these 4 lines.
        • ServerName
          • If you have an actual website name (example.com) that you own and want to use, put that here. Otherwise, localhost works. For the purposes of testing Gremlin, this is perfectly fine since there is no concern than anyone ever see this website.
        • ServerAlias
          • This is where you would put the name of the site, with the preceding “www” (www.example.com).
        • ServerAdmin webadmin@localhost
          • Just leave this as is. It has little bearing on what we are doing, but not having can cause some issues with Apache and WordPress.
        • DocumentRoot /var/www/wordpress
          • This is where the site lives. ‘Nuff said.
  • a2ensite wordpress.conf
    • This makes Apache read the newly modified configuration.
  • systemctl reload apache2
    • And now to make sure Apache is presenting the new information.
  • At this point work has to be done to finish the WordPress installation. This is accomplished by going to the web site’s page. So, whatever you set ServerName to in the wordpress.conf is the page you need to visit now. Once there, you will be presented with a menu selection and setup options. They are all pretty straightforward. Just remember that when it asks for the database information, the details are found in the steps when we installed MySQL.

Now for Gremlin. If you are not using their demo, you will only be able to install the tools and then use them locally. You will not have access to the dashboard or be able to use some of the more in-depth functions of Gremlin (Like creating teams and monitoring results.), but for testing purposes it should suffice.

These directions come from their own docs located here. I am only including the portions I used for my own testing, but if you visit their docs page, there are a multitude of installation methods.

  • echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list
    • Add the Gremlin repo to your source list so that you can install from apt.
  • sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6
    • Import the GPG keys.
  • sudo apt-get update && sudo apt-get install -y gremlin gremlind
    • Finally, install Gremlin and its daemon.
  • gremlin syscheck
    • This will verify that the checks are working as intended.
  • gremlin attack <TYPE>
    • Fires off a desired attack type.  Here is the help info for the command:
      • Usage: gremlin attack TYPE [type-specific-options]
        Type "gremlin help attack TYPE" for more details:
        blackhole # An attack which drops all matching 
        network traffic
        cpu # An attack which consumes CPU resources
        io # An attack which consumes IO resources
        latency # An attack which adds latency to all 
        matching network traffic
        memory # An attack which consumes memory
        packet_loss # An attack which introduces packet 
        loss to all matching network traffic
        shutdown # An attack which forces the target to 
        shutdown
        dns # An attack which blocks access to DNS servers
        time_travel # An attack which changes the system 
        time.
        disk # An attack which consumes disk resources
        process_killer # An attack which kills the specified 
        process

If you have requested a demo version, I highly recommend following the steps on the Gremlin docs pages for registering the server as a client and playing with their dashboard. I have had a blast pushing my test servers to the limits just to see what could be done and I can’t wait to see what is coming next!

 

 

A Script in Progress

Currently, I am working on writing my first serious Python script. It is ugly right now, but that is to be expected. The plan is to write small parts and then build more around them, cleaning up the code as I go. It’s slow and slightly tedious at times, but can be exhilarating when stuff starts working.

I want to start documenting my progress so that I can look back and see where I have grown, where I have failed and where I have succeeded. I welcome any input on how to correct bad code, fix errors, or generally clean up the script.

https://github.com/nihil777/Python/tree/testing

Also, I am just getting started with github, so be patient if my Markdown and version control is not the best.

The purpose of the program is to be able to scan items in my kitchen and create an inventory list for myself. Right now, the barcode has to be entered manually because I am more interested in it working than I am about scanner-gun functionality. We will cross that bridge later.

I have it mostly working in a simplistic manner, but I am trying to account for errors when a barcode is invalid and gives a 404 error.  That is pretty much where my head will be all day. Also, if anyone knows a robust, free UPC database with a decent API, I am open to suggestions.

What I am currently run into is that the error code for a 404 happens and urllib.error shows it as an error, however it also shows about 6 other errors, all related to the fact that I received a 404. If anyone out there knows how this happens or a good way to correct, I am all ears.

 

[bran8219@fedora-workstation python]$ python3 kitchen_inventory.py 
Enter item
9781491901434
Traceback (most recent call last):
 File "kitchen_inventory.py", line 53, in <module>
 request(combined)
 File "kitchen_inventory.py", line 28, in request
 f = opener.open(combined)
 File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
 response = meth(req, response)
 File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
 'http', request, response, code, msg, hdrs)
 File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
 return self._call_chain(*args)
 File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
 result = func(*args)
 File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
 raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

The Creation of a Problem

Over a few beers last night, I had a discussion with a friend about my issue coming up with a project. I needed some way to learn a programming language, but I am terrible at sticking with the exercises. While it is the best way for many people to learn, I am not one of them; I learn best when I have a final product in mind. Then he said something that resonated with me:

“It needs to be purely selfish if you are going to stick with it.”

This concept hit me like a ton of bricks to the face. Being of the mindset that I needed to make something that everyone could use, the thought of being selfish never occurred to me.

As he explained it, if your goal is to learn by doing, then the end result needs to be for yourself. Otherwise, you will never finish it because you have no investment in it.

With this in mind, I am starting a project and would like input from anyone who has done something similar. The idea is to use a barcode scanner (Those guns from the grocery store) to scan the items in my kitchen and create a database of what I have. I will use this to generate an inventory sheet and be able to record how much of each item I have. I know things like this already exist and that inventory software is abundant, but this is for me to learn from, not for me to have a finished product handed to me.

If anyone knows of an existing database of UPC codes or an API that is publicly available, I would greatly appreciate some direction. I am going to start this project in Python because it is the language I have the most familiarity with, but once it is complete, I intend to convert it over to Go so I can see the differences between the two languages.

Work In Progress

After making my last post all of 15 minutes ago, I realized what I needed to do in order to energize myself about blogging regularly: chronicle what I am doing.

I know. Everyone does that. And honestly, this is no different than what is already being done, but we all have to start somewhere and I am starting with my own epiphany.

So what is the plan for today? Well, here is the laundry list:

  1. Submit talk(s) to B-Sides San Antonio (http://www.bsidessatx.com/).
  2. Update my website.
  3. Create a personal GitHub repo for storing docs and code.
  4. Create 3 unique tweets.

So, the first two are done. Sort of. I have submitted two talks for B-Sides. The first is Sacred Beavers which discusses those processes that people believe increase productivity but actually slow it down considerably. The second is called Get Angry and is similar but from a different perspective. It revolves around the idea that so many people in the tech industry accept hindrances as normal and never speak up about how it affects their work. The Call For Papers (CFP) ends today and hopefully I will hear something by May 1st.

As for the website, well, that is what I am doing right now. There are some under-the-hood projects that I need to start outlining, but for the most part I have got this site back up and running. Eventually, I would like to learn enough HTML and CSS to move off of WordPress and manage everything myself, but for now this is probably the best way to go.

Number 3 is not that hard to accomplish, but it is part of a larger project. I enjoy documenting processes. I have no idea why, but I like writing detailed methods for getting tasks done. Currently, I am trying to find the motivation to learn how to write in a language (Python, Go, Ruby, Pig Latin, anything) and then document the entire process in a method that makes it easy for others to learn. Yeah, rewriting the wheel, but I am hoping to get some practice AND learn how to code/program at the same time.

The fourth is probably the hardest for me. I know it shouldn’t be, but I feel I should have something important to say before I tweet it. I just don’t have that internal desire to tell everyone that I am going to store or I stubbed my toe. Finding that “mojo” that makes one want to use Twitter is what I have to dig down and search for within my own mind. Who knows? Maybe there is some deep thoughts I can impart with the world. Or just shout about random things.

I am starting to look into a community called Write The Docs ( http://www.writethedocs.org/) and if anyone knows more about them or is/has been involved with them, I am interested in talking to you. My email and Twitter handle are at the bottom of the page. Reach out to me!

Yelling Into The Abyss

Alright, so I am bad at this website thing. I am also bad at social media. Now that I have gotten that out of the way, let me explain why I am bad at these things.

I hate talking just to be talking. I like conversation. I have a genuine need to receive input back on what I say. So when I make a blog post or send a tweet, I feel as if I am just speaking into the wind, hoping someone hears the message. I don’t know how to handle this and so I simply stop doing anything.

This is a horrible approach to an issue, or to life in general. Therefore, I am back, making this short post asking for input. Does anyone else feel this nagging talon of give and take ripping into your side and if so, how to you deal with it? I have enabled comments on this site, but only get the expected spam. Obviously, I should tweet more if I want more people to talk to, but what do you say that inspires conversation?

I look forward to hearing your opinions and thoughts!

 

In the beginning…

So I have had the desire to create a website for some time now, but never really had a direction for it. Then, over a great dinner at Murphy’s Kitchen and Tap, I had a discussion with my girlfriend about doing a site discussing food, beer, bars, diners, and all the other objects of my attention.

Therefore, I have to say right off the top, Murphy’s was astounding. The food was solid. We had the Irish Nachos, Irish Stew and the Fish ‘n Chips all of which were devoured. The tartar sauce, which I normally can’t stand anywhere else, was amazing. I even went and had a discussion with the Executive Chef about what he does differently to create it. Definitely, if you are ever in Charlotte, go here at least once.

 

Fish ‘n Chips
Pesto Chicken Pasta

Now, for my first post I wanted to start in my own kitchen with a very humble and simple dinner. It was a Pesto Chicken Pasta and a beer. The pesto was made by hand from basil grown in my girlfriend’s garden using a mortar and pestle. I would say it was a decent 6 out of 10, losing points for the presentation because, well, I just dumped it into a bowl and topped it with cheese and diced tomatoes.

 

Here is the process:

First, I started with Mark Bittman’s  Pesto recipe from How to Cook Everything:

  1. ½ cup pine nuts, lightly toasted.
  2. ¾ cup extra-virgin olive oil.
  3. 2 garlic cloves.
  4. 2 cups packed basil leaves (2 1/2 ounces/75 grams, from 1 big bunch or 2 small bunches)
  5. 1 cup finely grated Parmesan cheese (3 ounces/85 grams)
  6. ½ teaspoon kosher salt, more to taste.

I took the pine nuts, garlic, basil and salt and crushed them into a pulp inside the mortar. A little at a time, I added the oil, blending it in with the pestle. Once all of the oil was added and the leaves had been pretty much reduced to a gooey, yummy spread, I moved the sauce into a small bowl and chilled it while I worked on the other portions of dinner. After roughly an hour, I whisked in the cheese. His recipe calls for Parmesan, which is traditional, but all I had was some Asiago so that is what I used.

Now, during the time that the sauce was chilling in the fridge, I chopped a handful of rosemary and thyme (also from the garden) and tossed it with some olive oil, salt and pepper. Then, I coated chicken thighs with the concoction and placed them on a sheet pan, cooking them for about 20 minutes at 450 degrees.  Afterwards, the thighs were sliced into 1/2 inch strips.

Gourmet Texas Pasta

While that was in the oven, I boiled the pasta (Gourmet Texas Pasta) and prepared some mushrooms and tomatoes that I had on hand. I started with putting the mushrooms in a small skillet with some olive oil and cooked out most of their moisture. Then I added diced tomatoes and cooked those down until everything started to thicken.

After removing the tomatoes and mushrooms from the heat, I added some freshly chopped rosemary and basil, along with some salt and pepper. I let this cool down to room temperature and then mixed with the finished pesto.

After the chicken and pasta were finished, both were tossed with the tomato/pesto sauce. Then, just because I love their taste, I tossed in a few pinenuts and topped it all off with some shredded Asiago cheese and diced tomatoes.

The beer in the pictures is Magic Hat’s Elder Betty. While not something I would normally pick for myself, preferring stouts and porters, it was really quite good. So, if you enjoy ales, this is definitely worth giving a try.

Magic Hat Elder Betty

You may have noticed that throughout this entire post I have provided links to what I have used. Understand that no one is paying for this site except me. These companies (and in the case of Mark Bittman, people) have not paid for any form of promotion. The views I have expressed are 100% my own. I want to share my experiences with you all and want you to know when I find something worth checking out for yourself.

That being said, if you have a bar you think I should check out or a product you want reviewed, please let me know. I am always open to suggestions. While I am in the Texas region (specifically, San Antonio), I do periodically venture outside my comfort-zone. Let me know what you would like to see on this site.