tech – AWS, Puppet and Docker

In my talk – Can Puppet help you run Docker on a T2.Micro? – You will learn the technologies we use to create environments from scratch every day.

A walk through a number of the key concepts of puppet; stages, Role and profile, hieradata and puppet forge, as well as a brief introduction to Docker.
Using these to explain a solution of running a puppet manifest to configure Amazon‘s smallest (Yes I’ve run this on a t2.nano too) server to run a docker containerised web service.
You will learn why puppet stages can be used to help in this solution, how roles and profiles are defined and used, and finally use of the puppet Forge with Hieradata to install and run docker containers.
This talk will contain links to code that can be used afterwards and I’ll touch on what docker is and how to configure the puppet module to automatically run containers.

Come see me talk at Puppet Camp – 8th November 11:15 in London.

Posted in Infrastructure Tagged with: , ,

Development environments

After many years of working with developers, one of the most common problems faced by the ops and support team is code that doesn’t work on production systems.

This is often caused by subtle differences in the developers machine. Software versions of components, different install locations and different shared libraries. All these can have an impact on code that works fine on one machine or server, but won’t install or run on another.

Following our structure and automation, we can provide a machine that looks exactly the same as test and production. Allowing the developers access to the tools needed to build their own machine, via Jenkins for example, everyone is empowered to build and develop knowing that everyone on the team is on the same page and has the same environment or environments to work with.

Some customers have a number of environments to develop and test systems in. Often these are:
development – often shortened to DEV. This can be virtual machines on the developers own computer.
testing – TEST, to pass the initial automated test, (yes those can be automated too, allowing your testers more time to diagnose)
integration – This is usually a bigger environment and has links to other systems in the business to test integration between components and systems.
UAT – User acceptance testing. Where user journeys are tested end to end.
Performance – Load testing occurs at this level, ensuring that the servers can cope with high load, stress testing all the components and can test the scaling code. This helps defend against the slashdot effect.
Preprod or staging – the final environment before production. allows for dry runs of installation or roll out of new code.
Production – Live, The final stage and where your customers really see the efforts of the team.

With so many environments it is important to have version control every step of the way and a release process to enable everyone involved to understand where they stand and how their efforts will effect the overall position of the company.

Please contact us about this and any projects you are thinking about.

Posted in Development, Infrastructure Tagged with: ,

Why you need elastic applications

Ahead of Tuesday’s £150M+ Lottery draw, an email was sent to thousands of users of the UK National Lottery website.

“This is a service message to let you know that we expect to see high volumes of visitors to The National Lottery website and app in the hours leading up to the close of ticket sales for the EuroMillions draw, at 7.30pm, on Tuesday 11th October.
If you plan to buy a ticket for any of our games this week and you need to add funds to your account to play, we would recommend that you do this as early as possible in order to avoid disappointment. ”

We suspect that something in the infrastructure doesn’t scale automatically. No doubt the web or front end servers scale, however they are expecting a performance bottle neck somewhere. This is where performance testing of the whole infrastructure is important, to identify and work on those bottle necks. Once identified, automation of the service can be planned and provided. Often it is the database layer, but these too can be scaled with read replicas or sharded, allowing for greater number of active connections.

For more information or a quick chat, contact us.

Posted in Infrastructure

Automation (Puppet, code and schedules)

Once you know what your servers should be doing every day, and when, you are in a position to automate the processes that manage and build the servers.

There are three levels to consider when building servers automatically.

  1. What does the infrastructure look like? Network, firewalls, load balancers, subnets etc
  2. What is the role of the server? Software to be installed, configuration etc
  3. Scheduling and code deployments.

The phrase infrastructure as code is used to describe the process of creating, storing (including version control) and running of code that builds the environment in which the server(s) live. Of course you don’t have to automate everything at once and when you are ready there are tools to help setup the infrastructure for you.
Terraform (from the same people that brought you vagrant) and cloudformation to name but two. These tools allow you to define exactly what you want your servers and the associated environments to look like, so you can build in a completely predictable way, every time you choose to run the code. These tools allow you to modify the code and then rerun to apply the code. Keeping everything self documenting (if you can read the code) too.

Regardless of how your infrastructure is deployed (manually or automated) you can define the server build process as code too. This is the code the server will process to build itself from the moment it is switched on. Again there are a few tools to assist us; In Linux/Unix land – Puppet, Chef, Ansible and more specific to Microsoft is System Center Configuration Manager and Landesk. All of these tools allow you to define how you want your server too look, either by following a series of scripts or by defining the end configuration result, allowing your team to provide a known good result, every time they build a new server.

Now that our servers are built and working in an environment just for them, things change. Most clients we work with are developing software, so that does change a lot. Deployments of new code can also be automated. Either as part of the initial server build process, a) updated software code, b) rebuild server(s) or as a specific pipeline. The next set of tools available are often referred to as Continuous Integration/Continuous Delivery (CICD) or build tools. These allow a sequence of events to be defined, that the tool will process to build, test and deploy the new code. This can be as simple as building a server, or as complex as building a complete software package bundle. The most common tools are; Jenkins, TeamCity, Travis CI and Team Foundation Server. These work in conjunction with Source control repositories like Git, SVN.

Should you want any assistance with automation or just want to discuss your options, contact us.

Posted in Infrastructure

elastic and scalable

Following on from my article about pets vs cattle, we shall go into more detail about what it means have an elastic and scalable application or service.

Is my application scalable?

The first test to answer this question is, does it run on more than one server – and work. This is not a trivial question. One of our clients has a website sitting on two servers behind a load balancer ( a network server that shares the load between servers based on a set of rules). The load balancer was set to a ‘sticky’ configuration. This means that the customer visiting the website would always be served by the server they originally talked to. However, in this case, should that server not function, die, develop a fault or otherwise become out of service, the customer would find themselves directed to the other server (fail over). Sounds good, until we tried it. When the fail over happened, the server didn’t ‘know’ the customer and asked them to log in again. Not very seamless!

We worked with the developer team to alter the code of the website so that the servers would play nice together. In this case it was a case of making sure shared session data ( who the customer logged in as) was available to both (or all) servers delivering this application. After making the necessary changes and testing, our customer was ready to deploy to production with the side benefit of being able to switch off ‘sticky sessions’. These servers however were fixed. There was always two and you could say they were pets.

Is my application elastic?

Elastic applications go one step further. Imagine the marketing manager has just told you they are running an advert for this application on TV tomorrow. With so little warning, there isn’t much that can be done but hope that two servers will be enough to cope with demand. If they are virtual servers, you could arrange for them to be upgraded with more cpu and memory (Vertical Scaling).

With some automation however the answer is very different. Another of our clients automate the build of their servers (with our help). This means that on demand or a schedule, a new server is created and built from scratch by a computer following a set of instructions. This means the application can be elastic and autoscale. With the same information above, everyone can rest easy knowing that should the monitoring detect a higher demand, more servers (usually one at a time) will be built and added to the load balancer automatically or it hits a predetermined limit (so the internet doesn’t put a hole in your budget). The monitoring also checks when a server has been idle for a programmed time limit and will remove and destroy a server (virtual servers don’t feel a thing), until the configured minimum number of servers is reached.

I hope this article helps show the benefits of having your applications elastic and scalable. Please contact us should you want to learn more.

Posted in Infrastructure Tagged with: , , , ,

pets vs cattle

You’ve got a great application and users are finding it useful. Word is spreading, you need to scale to expand to keep up with demand.

I wrote about this choice in the article Scale UP or Scale OUT?

A metaphor attributed to Randy@Microsoft (link below) known as Cattle vs Pets, helps clarify the choice.

PET computer

PET computer



When you name your server like troy, stella, kryten, etc.
They are unique, loved and cared for.
When they become ill, they are nursed back to health and you hope they will stay well for a long time.
You couldn’t possibly imagine getting rid of or decommissioning the server.


Numbered cattle

A numbered cow


They are name more like a number – S02343
They are pretty much identical to the others
When they get ill, you just get another one.
As the server is so easily replace, you wouldn’t think twice about removing any you don’t need, at night for instance.


Our solution is to give you a way of creating servers, automatically and on demand. How ever many you need. With the confidence of being able to create a server whenever you want, it’s easy to decommission they when not needed. This allows your business to follow the demand curve with the supply of servers meeting that demand. This allows you to pay for the servers, when they are needed, and save money when they are not. In the example below, traditionally 2 servers would be on all day. Imagine the cost savings if you have one server on all day, and only pay for the extra server during the peak working hours. Saving 2/3 cost of a server, every single day.

Elastic Capacity meets demand

Elastic Capacity meets demand

For an informal chat with one of our cloud specialists, contact us.
Randy’s slideshow

Posted in Infrastructure Tagged with: ,

what is cloud computing?

Cloud computing is a metaphor to describe the way shared services are accessed over the internet.

From the user perspective, the specifics of each component of a service are not visible as if obscured by a cloud.


The term however has origins in the 1960s where the server you performed work on would be remote from your terminal and you shared the resources and time with other users. The first multi user systems.

A leap to the 1990s, when computer owners were used to storing their information on their own computer, starting on floppy disks and then later hard drives.

By the time we get to 2000s, the internet has become the network to link all the things, which gives global choices to share central resources again. Amazon launched EC2 (Elastic Compute Cloud) in 2006 which allowed access to powerful hardware to anyone with a credit card.

Gartner defines cloud computing as a style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service using Internet technologies.

Wikipedia breaks this down into the following capabilities:

  • Agile – provide user flexibility with re-provisioning, adding or expanding infrastructure resources
  • Cost – public cloud model converts capital expenditure (on off and up front) to operational expenditure. This lowers barrier to entry as well as allowing an on-demand billing model.
  • Device and location independence – As the infrastructure is off-site and access via the Internet, users can connect to it from anywhere.
  • Maintenance – Cloud computing applications are centrally maintained.
  • Multitenancy – Enables sharing of resources among multiple clients, further sharing costs.
  • Performance – monitoring by IT experts and using the model of loosely coupled architecture can improve workload balance and performance to the user.
  • Reliability – Multiple redundant sites give the choice without the cost, suitable for business continuity and disaster recovery.
  • Scalability and elasticity – on demand workloads without needing to order parts and waiting for engineers to fit them.
  • Security – can improve due to centralisation of data, but concerns can persist about loss of control over sensitive data. Private clouds can overcome this.

In the cloud computing space, 5 service models have appeared.

  • Infrastructure as a Service (IaaS) – providing virtual machines and the infrastructure required such as disk-image libraries, block storage, firewalls, load balancers and networks.
  • Platform as a Service (PaaS) – These provide further abstraction from the servers themselves and often allow deployment of specific applications, allowing you to focus on application delivery without the maintenance of servers and other infrastructure.
  • Software as a Service (SaaS) – Top level of abstraction where as a user you gain access to an application. SaaS is sometime referred to as “on-demand software” and is usually priced on a pay-per-use basis.
  • Mobile “backend” as a service (MBaaS) – Specific services and platforms useful for mobile application developers. Provides application programming interfaces (APIs) where services include user management, push notifications and integration with social network services.
  • Serverless computing or Function as a Service – an execution only model for code. This despite the name, does have servers, however the provider looks after the maintenance, starting and stopping to enable requests to be served. Billing is usually based on number and time for each request.

Many enterprise organisations are adopting public cloud services, however due to migration and/or security are running a ‘hybrid’ cloud, where some of the services are hosting publicly and some are hosted On premise, Internal or even Private Cloud.

What is right for your business depends very much on where you are and what you want to achieve. Should you want to discuss further, please contact us and one of our cloud specialists will be happy to help you.

Posted in Infrastructure

Will automation cost me my job?

One of the biggest concerns I hear from staff at the companies we’re helping is, Will I have a job if everything is automated?

This question was also asked on stackexchange.

The biggest assumption and probably the cause of the question, is the short term nature that many people see the world.
I have found that experience and time grows the horizon which you see and as your viewpoint lifts, you can see further into the future. Starting with days, then weeks and months.

In the answers referenced above are references to assumptions about what is happening now and what could be in the pipeline of work to do. If you are aware of the vision for the organisation, then it is unlikely this question will show up.

Growing organisations are always moving forward, improving processes and this includes automation. If you have the skills to grow and improve the business, then you are exactly the sort of person the business will want to hang on to.

Posted in Development Tagged with: ,

rs_tag for RightScale RL10

TLDR; rs_tag that works on Rightlink 10 agents.

As a DevOps guy that has to deal with AMIs (Amazon Machine Images) not under my control, this creates a few issues where we might be tempted to bake in tools.

One of our clients uses rightscale for infrastructure management and auditing and so one of these tools we use is the rightlink 6 agent. No problem, the provisioning workflow installs it when it creates a server, not an issue unless you loose a server (autoscaling or it just dies) and there is no workflow to install the agent.

Fortunately there is a solution, I’ve been testing the install ‘on boot’ of Rightlink 10, however this has some breaking changes.
The biggest is the missing tool RS_TAG, as a lot of the provisioning and automation scripts rely on this to get configuration information about the server.

The rightlink 10 (RL10) agent does come with a RightScale Client (RSC) that can help, but would require rewriting all the scripts. Some of the scripts need to be updated in any case, as the agent no longer executes as root.

I have written a RL10 RS_TAG that works on servers with the rightlink 10 agent installed as a drop in replacement.The only feature not implemented as of v0.1 is the –query function.

I chose to write this in GO as the RSC comes with a library in Golang. It also allows the binary to be statically linked, like the Rightscale Agent. This means no software dependencies are required on the server (other than the RightScale agent of course). Very handy when you’re about to build and configure the server in your favourite tool such as Puppet or ansible.

Let me know if it is of use to you.

Posted in Infrastructure

AWS User Group North Sept 2016

I attended the AWS User Group North meet yesterday at Zen Internet. What a fabulous day. A change to the usual evening format, we met up during the day.

This gave more time for talks and we were treated to two AWS guys and a few lightening talks.

Network Innovation at Scale – Colin Whittaker (Principal Network Engineer, AWS)

Colin showed us how the network that supports the Amazon Web Services structure has grown exponentially and described a few of the solutions to stay ahead of the demand. The hardware unit in the Datacenters is a prepopulated rack, with the hardware guys deploying a working rack, from arrival at the door to becoming available for customers in 2 hours. Colin also mentioned some of the challenges of moving a massive amount of network data at scale, with miles of fiber optic cables and Enhanced networking requiring custom network cards.

Lightning Talks

  Lambda, Good for the little things – Carl Simpson (Technical Architect, Zen Internet)

In the 1st talk about Lambda, Carl helped us understand how this service from AWS can help with small tasks, such as EBS volume backups and other housekeeping tasks.

  Having a Bash at AWS Management – Michael Pearce (Co-operatives UK)

Michael shared his journey learning about the AWS command line tools and showed us a working example of managing EBS snapshots in a rotation using these tools.

  Five AWS features that shouldn’t exist – Sam Bashton (Director, Bashton / Claranet)

Sam shared his opinion with us about some of the anti best practices that shouldn’t be there but are available to AWS customers. He used the pets and cattle metaphor to show that you shouldn’t use the console (script everything) for stopping instances, creating images or enabling auto recovery. In short don’t use the console, use code, such as CloudFormation or Terraform.

“Mummy, where are the servers?” – Ian Harris (Cloud Consultant, BJSS)

Continuing the theme of Lambda, Ian showed us how a competition entry workflow was hosted mostly on Lambda with Js scripts and even running ClamAV in a Lambda container. A short description of how there are actually servers, however they are containerized and patched by AWS. This solution used POST to S3, Lambda events and API gateway to enable the submission of files and a form. Some security considerations were also mentioned so check your IAM policies.

AWS Update – Danilo Poccia (Technical Evangelist, AWS)

Danilo rounded off the day sharing the latest announcements from AWS, including HTTP/2 support, Application Load Balancing, Elastic File System and Aurora read ahead.


Posted in Infrastructure Tagged with: