Floyd Hilton

Intro to Application Monitoring

Fri, 13 Jul 2018 18:01:00 +0000

Which of these situations do you identify with more?

A customer reports an issue and your team starts trying to reproduce it. Eventually it’s found that an error has been happening for a while, causing bad data, and it’s going to take a great deal to clean up. The customer is frustrated and other customers need to be notified, if you had only known sooner.
You receive an alert from your synthetic (live) testing that a new error is happening. A new feature was turned on recently and logging suggests that the error stems from this new feature. Custom metrics show that only the test user has used the new feature. You turn off the feature and clean the test data, crises averted.

Chances are you live somewhere in between the two extremes above. If so read on, perhaps some of the following will be useful.

Overview

While at Mingle Analytics I helped put together an application monitoring system which helped us improve system performance, find unreported issues, and greatly speed up troubleshooting. The system also proactively tested the production system to help us quickly find issues before users encountered them.

The image above shows the different types of monitoring and the tools that were used.

Lets take a look at each of the layers above and why you would use them.
I’ll be starting at bottom with the most common type and work my way to the top.

Here are some quick links if you want to skip to a section:

Overview
Server Monitoring - CPU, Memory, Disk
Application Level Logging
Log Aggregation
Application Performance Monitoring (APM)
Real User Monitoring (RUM)
Custom Metrics
Synthetic (Live) Testing
Alerting
Want to learn more?

Server Monitoring - CPU, Memory, Disk

Server monitoring is just what it sounds like: keeping an eye on your servers to make sure they are healthy. Is the CPU spiking? Do we have a memory leak, a bad disk? These are the types of questions you answer with server monitoring.

Server monitoring is typically run by the IT/Operations side. However, thanks to the DevOps movement, developers are becoming more aware of this type of monitoring.

At Mingle the IT group put LogicMonitor in place to do the server level monitoring.

Some Server Monitoring Tools

Resources

The Art of Monitoring Book

Application Level Logging

Application level logging is the first step to seeing what’s happening inside your running application.

Logging is accomplished by adding code to your application that will tell you what parts of the application are being used and if there are any errors.

Logging can be as simple as writing the info text to a file, though using a logging framework such as log4net or NLog provides additional functionality out of the box.

At Mingle we used log4net for application logging. We used the rolling file appender and kept a log file for each day of the month.

When Logging an Application:

Use log levels to control the amount of logging.
If running the app on multiple servers, include the server name in the log file to prevent file locking issues.
Consider structuring your logs using JSON to help with later parsing.
Make sure you log any handled as well as unhandled exceptions.

Some Application Logging Tools

log4net
NLog

Resources

Log Aggregation

Log aggregation pulls together all of your log files into a central location. Many tools allow the searching and filtering of log data to help zero in on the info needed.

At Mingle we used Stackify Retrace for log aggregation. We found it very easy to setup and immediately began to see some errors that we had missed in the log files.

Useful features of Retrace log aggregation:

Select a window of time and see the logs and other monitoring data all within the window.
Receive alerts of errors and their frequency.
Receive alerts when a search term is found in the log file.

Some Log Aggregation Tools

Resources

Log Aggregation 101

Application Performance Monitoring (APM)

APM tools typically help you see a breakdown of how each part of your application is performing.

At Mingle we used Stackify Retrace for APM.

We were really impressed with the amount of information we received after installing Retrace:

Graphs showing
- Number of requests
- Average timing of web requests
- A breakdown of where web requested time is spent (App code, database, Redis, etc)
Performance by
- Request
- SQL query
- Web Service
Code traces for slow or erroring code

Each of the above areas can be monitored and alerts can be sent when a threshold is breached.

Some APM Tools

Real User Monitoring (RUM)

RUM shows how a web application is performing from a users point of view:

Who is logged in
Number of users
Location of users
Page load performance
Errors and crashes
Feature usage

While APM is focused from the webserver down, RUM is focused on the web browser.
RUM is particularly important for applications that use the Single Page Application (SPA) model. In SPA applications most of the code runs in the web browser. It’s quite possible for the SPA to be completely down and have the back APM reporting nothing but a lack of traffic.

One of our main apps at Mingle was a SPA written in AngularJs. We decided to use Raygun to monitor our SPA application.

Raygun proved useful for:

Seeing the page load times by page or by user
Seeing performance issues from the user point of view
Knowing the number of active users
Identifying specific users that were having performance issues
Catching front end errors
Tracking user info such as browser type and location

We created a custom plugin for Raygun that would push errors and performance information to Stackify Retrace via a back-end API. This allowed us to have all of our error logging and alerting in one place.

Note that Raygun continues to evolve and now has many plugins to allow the export of alerts and data. Raygun is also starting to work in the APM space.

Some RUM Tools

Resources

What is real user monitoring?

Custom Metrics

While APM tools can give you quite a bit of metric information out of the box, it’s often necessary to complete the pictures using your own custom metrics.

Custom metrics are used for insight into both the technical and business sides of the application.

Typical metrics might be:

Business
- Number of users per day
- Number of new users
- Number of sales
- Sales amounts
- Page views
- Features used
Technical
- Errors
- Failed logins
- Performance of specific requests or actions
- Latency and response time of requests
- Volumes of data transferred
- Requests and responses

We used Retraces custom metrics to log page load times from Raygun as well as the number of logins and the performance of different parts of the system.

Custom metrics in Retrace are good for basic items but we ran up against the following limitations:

All metrics are by the minute, no custom time frame.
No support for using percentiles instead of average.
Limited access to the metric data to perform our own analysis.
- There is a basic API for metric data but it allows you to access the same info as the dashboard, averaged by the minute.

Some tools with custom metrics support

Resources

Why Averages Suck and Percentiles are Great

Synthetic (Live) Testing

Synthetic testing means running tests in production.

We used custom metrics to monitor the performance of our synthetic tests and give us an early warning for any front end performance related issues.

We setup TeamCity to run Selenium tests which logged into the application as a test user and performed tasks. We would then track and alert on metrics based on the test user ID.

Alerting

Server level alerting was handled by LogicMonitor.

All other alerting was handled by Stackify Retrace and routed to Slack.

In Slack we had a dedicated “Production Issues” channel that was highly guarded and set up to always alert subscribers.

Alterting tip: Make sure alerts are real and infrequent. Too many alerts quickly turn into noise that is ignored.

Want to learn more?

Please reach out to me on Twitter, LinkedIn or contact me at TacomaSoftware if you want more details about application monitoring.

Azure Bootcamp Prerequisites - Docker on Azure

Wed, 12 Apr 2017 18:01:00 +0000

Below are the prerequisites for the Global Azure Bootcamp hands on lab “Docker on Azure”.

Exercise Files
PuTTY
Docker for Windows
Visual Studio Tools for Docker
Docker Hub Account

Exercise Files

The exercise files are located in this GitHub repository.

Note: The GitHub repo is HUGE. I create a smaller zip file of the Docker files for the lab here if you don’t want to clone the whole repository.

PuTTY

Install PuTTY from here: http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

Docker for Windows

Install Docker for Windows from the Stable channel here: https://docs.docker.com/docker-for-windows/install/

Docker for Windows installs the Docker Client and Docker Daemon which allows you to run both Linux and Windows based containers.

Visual Studio Tools for Docker

Docker support needs to be installed as an extension in VS 2015, and a workload in VS 2017, the examples for each are described below.

For Visual Studio 2015
- Install .NET Core 1.0.1 VS 2015 Tooling Preview 2SDK
  - You need this specific version or the extension will not install
- Tools for Docker Extension
  - In Visual Studio 2015, go to Tools->Extensions and Updates
  - Search for docker tools and click download
  - Open the downloaded file to install
  - Create a new .NET Core application, right click on the project, select Add and verify that “Docker Support” is listed.
For Visual Studio 2017
- Go to Start->Visual Studio Installer
- Click Modify
- Select the “Azure Development” Workload
- Select the “.NET Core cross-platform development” Workload
- Click modify to install
- Create a new .NET Core application, right click on the project, select Add and verify that “Docker Support” is listed.

Docker Hub Account

Docker Hub is a repository for both Linux and Windows Docker images.
We will use Docker Hub to host an image that will then be deployed to Azure.

Viewing .NET Framework Source Code - Open Source or Closed with ILSpy and Visual Studio

Thu, 06 Apr 2017 18:01:00 +0000

Ever want (or need) to know how the .NET Framework is working under the covers?

Thanks to the new “Open Source” Microsoft, viewing the framework code is easier than ever.

But what if you are not using an open source version of .NET? Use a decompiler such as ILSpy or dotPeek.

Some useful Links

Decompile using ILSpy

If you can’t find the source code above, or just want to quickly look at the version you are using:

Install the ILSpy extension for Visual Studio (2012-2017)
Right click on an assembly and click “Open in ILSpy”

Understanding Docker "Container Host" vs. "Container OS" for Linux and Windows Containers

Fri, 31 Mar 2017 18:01:00 +0000

Lets explore the relationship between the “Container Host” and the “Container OS” and how they differ between Linux and Windows containers.

Some Definitions:

Container Host: Also called the Host OS. The Host OS is the operating system on which the Docker client and Docker daemon run. In the case of Linux and non-Hyper-V containers, the Host OS shares its kernel with running Docker containers. For Hyper-V each container has its own Hyper-V kernel.
Container OS: Also called the Base OS. The base OS refers to an image that contains an operating system such as Ubuntu, CentOS, or windowsservercore. Typically, you would build your own image on top of a Base OS image so that you can take utilize parts of the OS. Note that windows containers require a Base OS, while Linux containers do not.
Operating System Kernel: The Kernel manages lower level functions such as memory management, file system, network and process scheduling.

Now for some pictures:

In the above example

The Host OS is Ubuntu.
The Docker Client and the Docker Daemon (together called the Docker Engine) are running on the Host OS.
Each container shares the Host OS kernel.
CentOS and BusyBox are Linux Base OS images.
The “No OS” container demonstrates that you do not NEED a base OS to run a container in Linux. You can create a Docker file that has a base image of scratch and then runs a binary that uses the kernel directly.
Check out this article for a comparison of Base OS sizes.

In the above example

The Host OS is Windows 10 or Windows Server.
Each container shares the Host OS kernel.
All windows containers require a Base OS of either nanoserver or windowsservercore.

In the above example

The Host OS is Windows 10 or Windows Server.
Each container is hosted in its own light weight Hyper-V VM.
Each container uses the kernel inside the Hyper-V VM which provides an extra layer of separation between containers.
All windows containers require a Base OS of either nanoserver or windowsservercore.

A couple of good links:

Running Docker Containers in Microsoft Azure

Fri, 24 Mar 2017 18:01:00 +0000

There are multiple options for running Docker containers in Microsoft Azure. Below the options are broken out by the need to run an app in a single container or an app composed of multiple containers.

Monolithic (Single Container) Apps

An application does not need to be built out of microservices to benefit from the advantages of containers.
Developing a monolithic application in a container gives the advantages of dependency management, portability and ease of deployment.

Options for running single containers in Azure:

Azure App Service

Azure App Service now supports running Docker containers using Linux. You can configure containers to be run using the Azure Portal or create a containerized application and deploy the container directly to App Service using Visual Studio.

You can setup a continuous delivery pipeline for your App Service in Visual Studio Team Services using the Continuous Delivery Tools for Visual Studio extension.
Azure VM - IAAS using ARM Templates

You can create your own VM using Azure Resource Manager (ARM) Templates and install Docker. From there you can use Azure VM Scale Sets to scale your service as needed.

Note that “Visual Studio Tools for Docker” is part of Visual Studio 2017 and available as an extension for Visual Studio 2015.

Service-oriented (Multiple Container) Apps

Containers really shine when it comes to building Service-oriented architectures. Using services gives the ability to scale only the required parts of the application and allows developers to improve a single part of the system at a time.

While the above sounds great, the hard part is keeping track of all the containers.
Tools that help manage the provisioning, scheduling and monitoring of containers are called orchestrators.

You can use just about any orchestrator with Azure because you can manually create VMs and install whatever you like on them.
Installing and configuring orchestrators is a complicated process. Microsoft has gone to great lengths to make configuring Orchestrators easier with the Azure Container Service (ACS)

Options for orchestrating containers in Azure:

Azure Container Service (ACS)

Azure Container Service makes it easier to setup orchestration by pre-configuring the hosting environment and deploying the orchestrator for you. Select the orchestrator, the size and number of VMs and ACS takes care of the rest.

ACS currently supports (as of March 2017):
- Docker Swarm (not to be confused with the new Docker Swarm Mode)
- Kubernetes
- DC/OS
There are some great tutorials in the Technical Community Content repository on GitHub if you want to try out ACS.

You can setup a continuous delivery pipeline for your Azure Container Service in Visual Studio Team Services using the Continuous Delivery Tools for Visual Studio extension.
Azure Service Fabric

Azure Service Fabric is Microsoft’s orchestrator that was originally used internally for services such as Azure DocumentDB, Azure SQL Databases and Skype for Business. Service Fabric can be installed on premises or in a cloud provider such as Azure or AWS. Service Fabric can run Linux and Windows containers.
Azure VM - IAAS using ARM Templates

As with the monolithic containers above, you can provision your own VMs and install your own orchestrator.

There is a great example here on how to use the Azure CLI to setup and run Docker Swarm Mode in Azure.

A couple more good links:

The Mastermind Group, worth more than the sum of its parts.

Tue, 03 Jan 2017 18:01:00 +0000

Back in 2013, James Sturtevant and I started a Mastermind Group that met every week for approximatly two years and grew to a total of 4 people.
During those two years we both experienced explosive growth in our careers and more balanced personal lives.

I’ve had a few people ask for more details about our group so I thought a blog post would be a good place to share the information.

What is a Mastermind Group?

My definition of a Mastermind Group is a group of people helping eachother accomplish a common goal.

Examples of common goals are to succeed in business or to finish a graduate thesis.
In our case the common goals were to:

Live a balanced life.
Continue to improve ourselves in our careers and personal lives.

Key Benefits

I believe the key benefits of a Mastermind are:

Keep each other accountable for the goals we are setting.
Provide a sounding board for difficult decisions.

Mastermind Agenda

Each of our meetings had the following agenda:

Meet and greet.
Discuss difficult issues (if any)
Discuss the book we are reading.
Review completed goals and set goals for the next week.

If at any time one of us need help with something, we would susspend the agenda and spend the hour focusing on what the member needed.

The key to a succcessful mastermind group is committement. Everyone needs to meet EVERY week. The whole group is based on the trust that members will show up and be dedicated.

Setting Individual Goals

None of us wanted to work really hard climbing a ladder only to find that the ladder is leaning against the wrong wall.

To make sure we were heading in the right direction, we developed a goal setting process that starts with brainstorming 101 goals and ends with actionable quarterly goals to which we would then keep eachother accountable.

Want more Information?

If people would find it usefel, I’d like to put together some more information about the process and tools we use for setting and accomplshing our goals and some other information about the Mastermind Group.

If you would like to know more, please enter your email address here and I’ll let you know when more information is available.

Running Jekyll GitHub Pages on Windows using Docker

Wed, 14 Dec 2016 18:01:00 +0000

When I first started this blog, I went through the process of setting up Jekyll locally on my Mac so that I could preview posts.

Recently I’ve been doing more work in Windows and wanted to be able to work on my blog without booting up my Mac.

My friend James pointed out on his blog jamessturtevant.com that it’s really easy to get Jekyll running on Windows using Docker so I thought I’d give it a try.

Here are the steps I took based on James advice:

Install Docker for Windows

See instructions here:
Docker for Windows

Checkout Blog Source

Check out the blog source code to C:\Code

PS C:\> git clone https://github.com/fhilton/fhilton.github.io.git C:\Code\

Run Jekyll

Run the following command

PS C:\> docker run -it -p 4000:4000 -v C:/Code/fhilton.github.io:/site itzg/jekyll-github-pages

Which produces the following output

Configuration file: /site/_config.yml
Source: /site
Destination: /site/_site
Generating…
done.
Auto-regeneration: disabled. Use –watch
Configuration file: /site/_config.yml
Server address: http://0.0.0.0:4000/
Server running… press ctrl-c to stop.

The above command tells docker to

run the itzg/jekyll-github-pages image
bind /site in the image to C:\Code\fhilton.github.io
bind port 4000 of our docker host to port 4000 of the container

Browse Away!

Next browse to http://localhost:4000 and the blog is served!

Using Mac Keyboard Shortcuts for Home/End in Visual Studio

Sat, 26 Mar 2016 18:01:00 +0000

Overview

Using Visual Studio (in Parallels) on my Mac works pretty well, except when I try to use Command+Arrow to move to the beginning or end of a line. When you press Command+Arrow in Visual Studio on a Mac, the window moves around the screen, which is quite surprising and not at all helpful.

I did some research and found some useful information about mapping Mac Keyboard Shortcuts to Visual Studio (and Windows) in the following areas:

http://hiltmon.com/blog/2013/10/10/using-mac-navigation-keys-in-visual-studio/
Comments from the post in #1 above
https://autohotkey.com/board/topic/60675-osx-style-command-keys-in-windows/

The information above basically involves using AutoHotKey to remap the keys and then modifying the keyboard shortcuts in Visual Studio to get the desired result.
Below is a summary of the steps required to use Mac keyboard shortcuts such as Command+Left Arrow from within a Parallels or similar Windows VM and Visual Studio.

Note this tutorial is for Windows 10 and Visual Studio 2015.

Steps

Install AutoHotKey
Download the MacKeyboard.ahk file (note this modifies more than just Visual Studio keys and incorporates changes from the links above.The file is a fork from here).
Setup the .ahk file to run at startup
1. Type Command+R to open the run window, then type shell:startup
2. Copy the MacKeyboard.ahk file into the startup folder.
Open Visual Studio
Press Ctrl+Q and type Keyboard to open keyboard settings.
Type “Edit.LineStart” in “Show commands containing” to select the LineStart command
Select “Text Editor” in the “Use new shortcut in:” box
Type Shift+Control+Option+Left Arrow into “Press shortcut keys:” and click Assign
Click OK to close the Options window.
Double click on the MacKeyboard.ahk script to run it.
Open a project in Visual Studio and Command+Left Arrow should bring your cursor to the home position. Alt+Left Arrow should move a word at a time.

Programming Arduino with Atom

Sat, 10 Oct 2015 18:01:00 +0000

Working with Arduino’s is a lot of fun. Getting started is super easy thanks to the provided IDE and lots of great example code:

After getting past the “Hello World” phase I started to miss the editor experience that other editors give me, simple things like the ability to copy a line without highlighting or the way files are managed.

I found a whole list of development tools on the Arduino website. Two that stood out to me are:

The Viper IDE - Allows you to program the Arduino in Python on a web based IDE, neat!
Visual Studio Arduino Plugin - This looks really cool and thanks to Visual Studio Community is free to get started with.

Since for now I want to stick with C++ and the kids I’m teaching use Linux laptops, both of the above tools I found were not going to work.

We were already successfully programming Python and Javascript with Atom on the Linux laptops, so I decided to look for a way to program the Arduino with Atom.
The most promising extension I came across for Atom is platformio. Here is a blog post from the creator that explains what platformio is. See here for a list of the other Atom Extensions I use.
Platformio has a nice some pretty easy setup instructions:

I only ran into two issues trying to install the extension on my Mac.

I got an error “option –single-version-externally-managed not recognized” when trying to install Platformio.
There is an issue report here with a ton of info.
I fixed the issue by adding “–egg” to the install command (which someone in the above issue report said was a bad idea, but it worked for me…)
I use virtualenv to isolate all of my Python environments. When I tried to build from Atom I got an error about not being able to find Platformio because I installed Platformio in a virtualenv.
Luckily the platformio extension allows you to enter a custom path in the settings.
To fix the issue I did the following:
- Open a terminal and enter your virtualenv using workon
- Type Echo $PATH.
- Copy the path string and paste into the settings of the platformio extension:

Once the above issues were completed I was able to build and then upload a sample program to my Arduino!

7 Ways to Future Proof your Programming Career

Wed, 30 Sep 2015 18:01:00 +0000

The rate of change in the software development world is increasing every day. If you don’t pay attention you may wake up one day and find your skills are no longer marketable.

The good news is that by just doing one or two of the items below, you can move yourself into a position where instead of fearing what might be coming next you can be excited about the next opportunity that will be coming your way.

Listen to Podcasts

Learn from the best minds in the industry while you do the dishes
I’m surprised to hear how many people don’t listen to Podcasts. The most common reason I hear is that time is not available, to which I respond, “Do you do the dishes?”. Many of life’s repetitive tasks can be made more enjoyable by turning on your iPod or your smart phone and having a listen to the latest dotNetRocks, Hello World, or MS Dev Show. See my podcasts list for some ideas of where to start. Don’t knock it till you try it.

Attend a User Group

It’s not what you learn, but who you teach
Many of us less-than-outgoing software types tend to try and learn all on our own. With all the blogs, podcasts and videos out there, where is the need to go and learn at a user group?

I’ve found that the best reason to go to a user group is not to learn, but to contribute and lift others up. Once you experience the excitement of helping someone else grasp a concept or complete a project, you will see that it truly is better to give than receive.
What does this have to do with your career? Who would you want to hire: someone you have never heard of, or someone that’s known for teaching others about the type of work you need done?

Have a Side Project

Don’t be limited by what you do at work
One of the best ways to be motivated to improve your skills is to start a side project that interests you. Pick a technology you are drawn to or a domain that you enjoy (or both) and build something. Even if you would never use the specific technology at work, chances are there will be transferable concepts that you can use to improve the way you solve problems.
Whatever you decide to do, make it achievable in a reasonable amount of time, and FINISH IT! It is better to have one smaller finished project than 15 grand ideas that are 10% done.
As you work on your project commit the source code to GitHub, it helps others learn and helps build your “clout” as a software developer.

Pair Programming

Two heads are better than one
If you spend just 10 minutes watching another developer code you will undoubtably pick up something (an IDE trick, a command shortcut, or a concept) that you wouldn’t have come by on your own.
Pair programing doesn’t mean you have to be in the same room; in fact I think it is better for each to use your own computers and use some screen or editor sharing tools to facilitate the pair programming.

Attend a Conference

Or better yet, speak at one
I used to think conferences were expensive events held in places that required flights and hotels to attend.
A couple years ago my friend James Sturtevant told me about “Code Camps”. A Code Camp is a free event that encourages anyone to submit a talk (more information here). James encouraged me to submit a talk to the Boston Code Camp and the talk got selected. I was pretty nervous but the whole experience turned out to be great. I liked the experience so much that I gave another talk at the Vermont Code Camp a few months later. I know of a person that landed a great new job because the employer attended one of their talks, it can happen.

Join a Mastermind Group

Accountabiliy is key
A Mastermind is simply a group of people that are all working toward a common goal. For some more info on Mastermind Groups see this post from SmartPassiveIncome.com.

A friend and I formed our Mastermind Group around the goals of Continuous Improvement and Life Balance. Our group has met weekly for over two years, during which time members have completed an impressive amount of direct goals and some unexpected side benefits such as new jobs, pay raises and promotions.

Start a Blog

Share the solution
There are many reasons to start a blog. Since we are talking about career, the reasons that jump to mind are showing you are knowledgeable and showing you know how to communicate that knowledge.
My suggestions for getting started:

Don’t stress about the platform, just pick one and go. It’s really easy to get bogged down by setting up your blog and never actually get to writing a post.
Every time you solve a tricky problem, write a post. If Google (or Bing) didn’t know the answer for you, it won’t for others as well. Document how you solved the problem, and at they very least when you have the problem again in 6 months you will know how to solve it.

Do It!

You won’t move without taking the first step
None of the above items will do you any good if you don’t give them a try. Pick one that sounds interesting and commit to trying it out.
For Example:

Listen to a podcast whenever you do the dishes
Find an interesting open source project and do a pull request
Find a Code Camp near you and sign up to attend the next event
Find another coder that wants to improve and meet twice a month to talk tech and encourage each-other

Let me know on Twitter if do any of the above and find it helpful, I’d love to hear all about it.