Fork me on GitHub

Puppi: A Puppet module for Deployment Automation

Puppi is a Puppet module that lets users manage and automate the deployment of applications or generally every kind of batch activity that involves the execution of a sequence of scripts and commands.
Its structure provides complete flexibility on the actions required for virtually any kind of application deployment but, in order to make usable out of the box, some example defines and scripts are provided to manage common scenarios and needs.
The module provides:
- The puppi command and its whole working environment
- A set of common usage scripts that can be used in chain
- Sample defines that can be used for many common application deployment scenarios.

The whole picture in an example

Before diving into details let's see a brief example.
On your node or role you can use a similar define:

puppi::project::war { "myapp":
    source       => "http://repository.example42.com/myapp/myapp.war",
    user         => "myappuser",
    deploy_root  => "/srv/tomcat/myapp/webapps",
    report_email => "release@example42.com",
    enable       => "true",
}

This creates a set of scripts in /etc/puppi/projects/myapp that make it possibile to write the simple command:

puppi deploy myapp

to run the whole deployment procedure, that, in this case, involves retrieving the war file from the http://repository.example42.com/myapp/myapp.war, backing up /srv/tomcat/myapp/webapps, deploying the new war in the same directory as myappuser and notifying via email release@example42.com.

This case is not particularly complex but consider that the same puppi::project::war define user before has many more options, that let you run custom pre or post commands (in customizable order and with defineable user), stop and start, during the process any custom service, block access from a specific IP (such as the one of your load balancer) and so on.

If something wrong happens you can simply type:

puppi rollback myapp

and choose to what version you want to rollback (latest or any other previous (dated) backup).

Let's see a real life example of a deploy:

root@metaportali-mpc:~# puppi deploy configurator
Puppi setup: 00-configurator-RuntimeConfig-Initialization  [  OK  ]

Deploy: 10-configurator-Run_PRE-Checks                     [  OK  ]
PROCS OK: 4 processes with command name 'apache2'
TCP OK - 0.001 second response time on port 80|time=0.000651s;;;0.000000;10.000000
[ ... ]
Deploy: 20-configurator-Retrieve_WAR                       [  OK  ]

Deploy: 30-configurator-Backup_existing_WAR                [  OK  ]

Deploy: 36-configurator-Disable_extra_services             [  OK  ]

Deploy: 37-configurator-Check_undeploy                     [  OK  ]

Deploy: 38-configurator-Service_stop                       [  OK  ]
 * Stopping tomcat-mpc instance
Using CATALINA_BASE:   /store/tomcat/mpc
[ ... ]
Deploy: 39-configurator-Run_Custom_PreDeploy_Script        [  OK  ]

Deploy: 40-configurator-Deploy_WAR                         [  OK  ]

Deploy: 42-configurator-Service_start                      [  OK  ]
 * Starting tomcat-mpc instance
Using CATALINA_BASE:   /store/tomcat/mpc
[ ... ]
Deploy: 43-configurator-Check_deploy                       [  OK  ]

Deploy: 44-configurator-Enable_extra_services              [  OK  ]

Deploy: 80-configurator-Run_POST-Checks                    [  OK  ]
PROCS OK: 4 processes with command name 'apache2'
TCP OK - 0.004 second response time on port 80|time=0.003824s;;;0.000000;10.000000
[ ... ]

Reporting: 20-configurator-Mail_Notification               [  OK  ]

REPORT FOR PUPPI - STATUS OK
Summary of operations is: /var/log/puppi/configurator/20110224-114729/summary 
Details are in: /var/log/puppi/configurator/20110224-114729/
Temporary workdir has been: /tmp/puppi/configurator/ (Will be rewritten at the next puppi run)
Runtime config file is: /tmp/puppi/configurator/config
Files have been archived in: /var/lib/puppi/archive/configurator/20110224-114729

If you use Example42 modules, enable their automatic monitoring features ($monitor=yes) and use Puppi, among others, as monitoring tool (ie: $monitor_tool=["nagios","munin","puppi"] ) you have out of the box the possibility of running local checks of the applications managed by Puppet with the command puppi check. Let's see an example:

root@metaportali-mpc:/# puppi check
Host check: 50-apache_process                              [  OK  ]
PROCS OK: 4 processes with command name 'apache2'

Host check: 50-apache_tcp_80                               [  OK  ]
TCP OK - 0.001 second response time on port 80|time=0.000668s;;;0.000000;10.000000

Host check: 50-mcollective_process                         [  OK  ]
PROCS OK: 1 process with command name 'ruby', args 'mcollectived'

Host check: 50-munin_process                               [  OK  ]
PROCS OK: 1 process with command name 'munin-node'

Host check: 50-munin_tcp_4949                              [  OK  ]
TCP OK - 0.003 second response time on port 4949|time=0.003184s;;;0.000000;10.000000

Host check: 50-nrpe_process                                [  OK  ]
PROCS OK: 1 process with command name 'nrpe'

Host check: 50-nrpe_tcp_5666                               [  OK  ]
TCP OK - 0.005 second response time on port 5666|time=0.005355s;;;0.000000;10.000000

Host check: 50-openssh_process                             [  OK  ]
PROCS OK: 13 processes with command name 'sshd'

Host check: 50-openssh_tcp_22                              [  OK  ]
TCP OK - 0.001 second response time on port 22|time=0.000857s;;;0.000000;10.000000

Host check: 50-puppet_process                              [  OK  ]
PROCS OK: 1 process with command name 'puppetd'

Host check: 50-rsyslog_process                             [  OK  ]
PROCS OK: 1 process with command name 'rsyslogd'

Host check: 50-snmpd_process                               [  OK  ]
PROCS OK: 1 process with command name 'snmpd'

Host check: 50-tomcat-mpc                                  [  OK  ]
PROCS OK: 2 processes with command name 'java', args 'mpc'

Host check: 50-tomcat-verticali                            [  OK  ]
PROCS OK: 1 process with command name 'java', args 'verticali'

Host check: 50-tomcat_tcp_8200                             [  OK  ]
TCP OK - 0.001 second response time on port 8200|time=0.000697s;;;0.000000;10.000000

Host check: 50-tomcat_tcp_8201                             [  OK  ]
TCP OK - 0.001 second response time on port 8201|time=0.001045s;;;0.000000;10.000000

From these simple examples it should be clear that with Puppi you have:
- A common command to manage deploys and rollbacks on any host.
- A simple to use, coherent and still flexible and customizable set of defines to describe in Puppet manifests all the elements you need for your application deployments.
- A quick way to check if everything is ok on your local node (if you use Example42 module set).

Think about it: with just a Puppet define you build the whole deploy logic
- Reporting for each deploy/rollback is built-in and extensible
- Automatic checks can be built in the deploy procedure
- You have a common, growing, set of general-use scripts for typical actions
- You can define custom deployments workflows (puppi::project::myprocedure) for specific cases not covered by the provided sample defines.

Using the puppi command

Puppi provides various actions and options, let's see the main ones:

puppi <action> <project_name> [ -options ]
These are, currently, the available actions:
- puppi init <project_name> : First time initialization of the defined project
- puppi deploy <project_name> : Deploys the defined project
- puppi rollback <project_name> : Rollback to a previous deploy state
- puppi check [project_name] : Runs project specific and host wide checks
For each of these actions puppi runs the commands in /etc/puppi/projects/$project_name/$action, logs their status and then runs the commands in /etc/puppi/projects/$project_name/report to provide reporting, in whatever, pluggable, way.
You can also provide some options:
-f : Force puppi commands execution also on CRITICAL errors
-i : Interactively ask confirmation for every command
-t : Test mode. Just show the commands that should be executed without doing anything
-d <yes|full>: Debug mode. Show debugging info during execution
-o "parameter=value parameter2=value2" : Set manual options to override defaults. The options must be in parameter=value syntax, separated by spaces and inside double quotes.

Here are some command line samples:

puppi check : Run host-wide checks.
puppi init myapp : Make the first deploy of "myapp". Can be optional.
puppi deploy myapp : Deploys myapp with the standard logic/parameters defined in Puppet
puppi deploy myapp -f : Deploys myapp and doesn't stop in case of Critical errors
puppi deploy myapp -i : Deploys myapp in interactive mode. Confirmation is asked for each step
puppi deploy myapp -t : Test mode. Just show the commands that would be executed
puppi deploy myapp -d full : Deploys myapp with full debugging output
puppi deploy myapp -i -o "version=1.1 source_url=http://dev.example42.com/code/my_app/": Deploys myapp in interactive mode and sets some custom options that override the standard Puppet params.
Note that these parameters change according to the script you use (and the scripts must honour this override).
puppi rollback myapp : Rollbacks myapp to a previous archived state. User is prompted to choose which deploy to override.

File Paths

All the paths used by the puppi scripts are provided, and can be configured, in the puppi Puppet module:
/usr/sbin/puppi - Where the puppi command is placed. Currently is a bash script.
/etc/puppi/puppi.conf - Puppi main config file. Here various puppi wide paths are defined
/etc/puppi/checks/ ($checksdir) - Here can be placed all the host wide checks. If you use the Example42 monitor module and have "puppi" as $monitor_tool, this directory is automatically filled with checks based on Nagios plugins.
/etc/puppi/projects/ ($projectsdir) - In this directory you can have one or more projects subdirs, with the commands to be run for deploy, rollback and check actions. They are completely built (and purged) by the Puppet module.
/etc/puppi/scripts/ ($scriptsdir) - The general-use scripts directory, these are used by the above commands and may require one or more arguments.
/var/lib/puppi/archive/ ($archivedir) - Where all data to rollback is placed.
/var/log/puppi/ ($logdir) - Where logs and reports of the different commands are placed.
/tmp/puppi/ ($workdir) - Temporary, scratchable, directory where Puppi places temporary files. It's wiped at every puppi run.
/tmp/puppi/$project/config - A runtime configuration file, where are dinamically placed variables usable by all the scripts invoked by puppi. This is necessary to mantain "state" information that changes on every puppi run (such as the deploy datetime, used for backups).

Understanding the Puppi module

The puppi module provides few basic defines to manage puppi's setup elements and some example templates that use these defines to build deployment workflows.
The basic defines are:
puppi::project - Creates the main project structure. One or more different deployment projects can exist on a node.
puppi::initialize - Creates a single command to be placed in the init sequence. It's not required for every project.
puppi::deploy - Creates a single command to be placed in the deploy sequence. More than one is generally needed for each project.
puppi::rollback - Creates a single command to be placed in the rollback sequence. More than one is generally needed for each project.
puppi::check - Creates a single check (based on Nagios plugins) for a project or for the whole host (host wide checks are auto generated by Example42 monitor module)
puppi::report - Creates a reporting command to be placed in the report sequence.

The above init, deploy, rollback, check and report defines have generally a standard structure and similar arguments. Every one is reversable (enable => false) but you can wipe out the whole /etc/puppi directory to have it rebuilt from scratch.
Here is an example for a deploy command:

puppi::deploy { "Retrieve files":       # The $name of the define
    command  => "get_curl.sh",          # General-use script to use 
    argument => "file:///storage/file", # Argument(s) for the script
    priority => "10",                   # Execution order
    user     => "root",                 # Execution user
    project  => "myapp",                # The name of the project
}

This define creates a file named /etc/puppi/projects/${project}/deploy/${priority}-${name}
Its content is, simply:
su - ${user} -c "export project=${project} && /etc/puppi/scripts/${command} ${arguments}"

You can glue together, with the desired order according to the priority argument, different basic defines to create a complex project template and use this in your manifests. Various sample defines that can be used to many common deployment scenarios are present in puppi/manifests/project/ You can use puppi::project::war to manage the deployment of a simple war file, as seen before, or other defines to manage deployments of a tarball, an arbitrary list of files, or the maven artifacts published on a Nexus repository.

The puppi/files/scripts directory in the module contains some general usage scripts that can be used in custom deployments.
They are generally made to work together according to a specific logic, which is at the base of the sample defines in puppi/manifests/project/ but you're free to write your own scripts, in whatever language, according to your needs, and integrate them with custom Puppet defines.

The default scripts are engineered to follow this procedure for a deployment:
- Remote files are downloaded in /tmp/puppi/$project/store or directly in the predeploy directory: /tmp/puppi/$project/deploy
- If  necessary the downloaded files are expanded in one or more predeploy directories (default:/tmp/puppi/$project/deploy)- Runtime configuration entries might be saved in /tmp/puppi/$project/config- Files are eventually backed from the deploy directory (Apache' docroot, Tomcat webapps or whatever) to the archive directory (/var/lib/puppi/archive/$project)
- Files are copied from the predeploy directory to the deploy dir.
- Relevant services are eventually sopped and started

The most used common scripts are (they might have different arguments, some of them are quite simple):
get_file.sh - Retrieves a remove file via ssh/http/rsync/svn and places it in a temporary directory (store or predeploy)
deploy.sh - Copies the files in the predeploy dir to deploy dir
archive.sh - Backups and restores files in deploy dir
service.sh - Stops or starts one or more services
wait.sh - Waits for the presence or absence of a file, for the presence of a string in a file or a defined number or seconds.
get_metadata.sh - Extracts metadata from various sources in order to provide info to other scripts
report_mail.sh - Sends a mail with the report of the operations done

How to customize

It should be clear that with puppi you have full flexibility in the definition of a deployment procedure, since the puppi command is basically a wrapper that executes arbitrary scripts with a given sequence, in pure KISS logic.

There are different parts where you can customize the behaviour of puppi:

- The set of general-use scripts in /etc/puppi/scripts/ ( this dir is filled with the content of puppi/files/scripts/ ) can/should be enhanced. As been, these can be arbitrary scripts in whatever language. If you want to follow puppi's logic, though, consider that they should import the common and runtime configuration files and have an exit code logic similar to the one of Nagios plugins: 0 is OK, 1 is WARNING, 2 is CRITICAL.
Note that by default a script that exits with WARNING doesn't block the deploy procedure, on the other hand, if a script exits with CRITICAL (exit 2) by default it blocks the procedure.
Take a second, also, to explore the runtime config file created by the puppi command that contains variables that can be set and used by the scripts invoked by puppi.

- The custom project defines that describe deployments procedures. These are placed in puppi/manifests/project/ and can request all the arguments you want to feed your scripts with.
Generally is a good idea to design a standard enough template that can be used for all the cases where the deployment procedure involves similar steps. Consider also that you can handle exceptions with variables (see the $loadbalancer_ip usage in puppi/manifests/project/maven.pp)

What's next

The Puppi module is a work in progress that is already used in production. It has been designed to automate deployments in a specific environment but a lot of effort has been done to standardize as much as possible the procedures and the options for differents needs and logic and to make it modular by design.
The same puppi command can be re-written from scratch. It's has been made in bash and not in fancier languanges for simplicity and portability.
What it does at the moment, after all, is something quite simple (basically execute commands in a sequence).
The whole point of the Puppi module is to define quickly and standardize in Puppet manifests any kind of deployment procedure (at least, at the moment, any kind of procedure that can be executed from the same node, orchestration is planned but not yet functional).

Future plans involve:
- dedicated agents for orchestration tools like Mcollective or ControlTier
- more reporting scripts (for example reporting on Google Calendar and Docs)
- a web interface to manage the deployment procedure and see reports

18 comments

by ouyangsa on Mon, 03/14/2011 - 11:34

<P>where do I find the "puppi" command? </P>
<P>thanks! </P>

by al on Fri, 03/18/2011 - 16:33

The puppi command is placed by the same puppi module in /usr/sbin/puppi, just include puppi in a node and you find the command, its configuration file and the whole working environment.

by ouyangsa on Sun, 03/27/2011 - 17:19

hi al     I install the puppet package on my RHEL5,but I can't find the "/usr/sbin/puppi" command     my puppet packages  as follow,     --puppet-0.25.4-1.el5.rpm     --puppet-server-0.25.4-1.el5.rpm     could you help me?     Thank you again!

by al on Mon, 03/28/2011 - 14:39

Sorry, I wasn't clear enough. /usr/sbin/puppi is not provided by any package, is provided by the puppi module presebnt in the Example42 modules set. So, in order to use it, you have to download the example42 modules and "include puppi".You can also get only the puppi module out of the whole bunch of example42 modules, just copy it in your modules path.

by ouyangsa on Fri, 04/15/2011 - 09:35

hi al      I have already deployed puppi module to /etc/puppet/modules/, but I still can't find /usr/sbin/puppi command .      I can send you my puppi modules if you tell me your email.      thanks a lot! ouyangnb@qq.com

by al on Wed, 05/25/2011 - 08:56

Once placed in /etc/puppet/modules you have to explicitely include it in your nodes(s) or in classes that your node include.

For example:

node myserver {
     include puppi
}

 If you use Example42 modules, you can also set the $puppi variable to "yes" to automatically include all the puppi subclassed present in most modules (they add some specific puppi extensions related to the application you provide with the module).

For example:

node basenode {
    $puppi = "yes"
}node mynode inherits basenode {
     include puppi
     include apache
     include postfix
}

This installs the masic puppi module on the node, but also the apache and postfix extensions.

by narkisr on Thu, 05/26/2011 - 23:34

Hey,Id like to use puppi in a commercial enviroment, are there any limitations on that considering that the license is gplv3? Thank you

by al on Mon, 06/20/2011 - 10:05

You can freely use Puppi in a commercial environment, as you probably already do with other GPL licenced software.IF you want to use Puppi inside a product that is going to be redistributed, you've to keed the GPL licence.Consider anyway that I'm going to follow Puppet's licence and so Example42 modules are going to be relased under the Apache licence too, which is far less restrictive for software that has to be included is commercial products.

by Vishal on Mon, 02/27/2012 - 10:54

I have tried to download and configure the puppi modile but still i am not able to found puppi command. thanks

by Amit on Mon, 02/27/2012 - 13:18

I have downloaded and configure the puppi module and restarted and puppetmaster service. But Still i am not able findout the command puppi.

by al on Wed, 02/29/2012 - 00:50

You must write "include puppi" in your node(s) or in a class that your nodes include.

by Joanne on Wed, 02/29/2012 - 22:03

I had the same issue on finding puppi command. It turned out that the puppi command was installed when you ran the puppet agent at the first time. of course, you must put down "include puppi" in the node as Al said. Not sure if there is other way to do it.

by Tom on Thu, 03/01/2012 - 01:57

I have put include puppi for my puppet node but what's next?Where exactly do I define e.g. puppi::project::war{...}I've tried it in the init.pp and also under the include puppi but it doesn't create the /etc/puppy/projects/ project files Syntax error at 'myapp'; expected '}  define puppi::project::war { "myapp":        source_url   => "http://XXXX/myapp.war",        user         => "tom",        deploy_root  => "/var/lib/tomcat6/webapps/ROOT",        enable       => "true", }

by al on Thu, 03/01/2012 - 16:24

Here is a sample usage for a puppi project

puppi::project::war { "myapp":
    source           => "http://repo.example42.com/deploy/prod/myapp.war",
    deploy_root      => "/store/tomcat/myapp/webapps",
    report_email     => "sysadmins@example42.com",    
}

by Kim Neunert on Mon, 04/02/2012 - 20:29

Hi,thank you very much for the puppi-module. I could really easily create a deployment-model for my project. But now i'm restricted to run puppi as non-root? Is that possible at all? I asume that puppi is designed to more run as root (or via MCollective). Does anyone have experiences with running it as non-root? Does it make sense to even try that? thanks Kim

by al on Tue, 04/03/2012 - 17:52

Hei Kim, sorry for late reply.

Puppi runs as root because it needs to do many things (deploy to whatever directory with whatever user permissions, manage services, place iptables rules and so on).

If you need to be able to run it by non privileged users you can manage it via sudo. There are some precautions made to avoid bash commands injecting so it should be safe to let users sudo  puppi (or some specific subcommands).

by Pavan on Mon, 07/16/2012 - 19:35

I work on release management and looks like thi stool is cool. I am almost a novice in Linux and am still exploring it. So, can someone let me know the steps to download and installed puppet modules. Do I need to do it on each Linux box I have or can I place it on any centralized location which can be accessed by all the servers.

by al on Thu, 08/16/2012 - 20:47

Best source for puppet documentation is the official one: docs.puppetlabs.com

In brief: you need to download modules only on the puppetmaster and there you configure whatever you want to place on your servers (puppet clients).

On all your servers you just need to install puppet (the client) and eventually configure /etc/puppet/puppet.conf

Copyright © 2011 Lab42. Drupal theme by Kiwi Themes.