Planet Puppet

Lately there have been a lot of organisations trying to hire a devops engineer.
I myselve have been asked to fill in devops roles ..

There's a number of issues with that.

The biggest problem is that I always have to ask what exactly the organisation is looking for.

So you want a devops engineer with experience in Linux, MongoDB, MySQL and Java , does that mean you want a Java developer who is familiar with MySQL and Linux and breaths a devops Culture.
Or a Linux expert who understands Java developers and knows how to tune Mongo and MySQL ?

It's absolutely unclear what you want when you are hiring "A devops engineer"

The second problem is that you are trying to hire people who are knowledgeable about devops,

Yet a lot of those people know that you can't do devops on your own , devops is not a jobtitle. devops is not a new devops team you create.

To some of them you are even making a fool out of yourselve, as to them you show that you don't understand devops

On top .. the ones that do apply for this fancy new devops role, are the ones that might not get the fact that the problem isn't about tooling but about people working together and helping eachother , so you end up hiring the wrong people.

Even in todays devops culture a system engineer is still a system engineer, and a developer is still a developer.
You might have developers supporting the build tool chain, or system engineers focussing on infrastructure automation.

But as John said almost 3 years ago they are good at their job.

Devops is not a word you slap onto a tool, a team or a person and expect magic to happen

Let's face it .. devops is hard, you can't do this on your own .. you need to find the right people ..

PuppetConf offers the perfect opportunity to meet, connect, and learn the latest on IT automation, DevOps and more. You’ll learn plenty at the keynotes and presentations on offer, and we’re also offering some official Puppet Labs training and education classes before and after the show.

Developer Day

Free with your PuppetConf ticket, Developer Day is your opportunity to meet developers from Puppet Labs and the wider Puppet community. You’ll get to spend the day working on your choice of open source projects, documentation, Puppet modules and more.

If you need to get a Puppet development environment up and running, we have a workshop to help you with that, first thing Wednesday morning. Once you’re set up, you’ll be ready to dive into the code or head for the open spaces and unconference sessions.

Developer Day is on Wednesday, August 21. Register now.

What If I’ve Already Registered for PuppetConf?

If you registered for PuppetConf before registration for Developer Day was open, it’s no problem: To add Developer Day to your existing PuppetConf ticket for free, visit this link and select “Developer Day Add-On.”

Puppet Labs Training Classes

If you’ve been meaning to up your Puppet game with some Puppet Labs certified training, we’re offering a full run of courses before and after PuppetConf. This is your opportunity to improve on existing skills before taking in all the keynotes and workshops, or to get some learning under your belt before diving in to the show. Enrollment in each of these classes includes a FREE ticket.

If you’ve already registered for the conference, want to attend one of these classes, and claim your free PuppetConf ticket, you’ll need to:

1. Request a refund for your existing ticket by sending an email to Lisa Gregory.
2. Register for the course you’d like to take. You’ll automatically be registered for PuppetConf once you sign up for the course.

Intro to Puppet

With Intro to Puppet, you’ll learn how to solve IT automation problems with Puppet Enterprise. The course includes a hands-on exercise so you can practice your new skills right away.

Running the day before the show, on Wednesday, August 21, the class costs $1,245 and includes a FREE PuppetConf ticket. Register now.

Puppet Fundamentals

Puppet Fundamentals offers you a chance to learn infrastructure best practices in a hands-on lab environment. By the time this three-day course is over, you’ll know how to write Puppet code and create Puppet modules to accomplish your own IT automation challenges.

Running before the show, from Monday, August 19 to Wednesday, August 21, this three-day class costs $2,395 and includes a FREE PuppetConf ticket. Register now.

Advanced Puppet

The Advanced Puppet Course will train you up in the advanced, enterprise-level skills you need to implement Puppet Enterprise in complex environments. Once you’re done, you’ll know how to tackle the big problems, and optimize Puppet Enterprise for large-scale deployments.

Advanced Puppet runs from Monday, August 19 to Wednesday, August 21 and costs $2,395, with a FREE ticket to PuppetConf included. Register now.

Extending Puppet Using Ruby

If you’ve mastered Puppet technologies and you’re eager to code, this class will give you the advanced skills you need to improve your Puppet Enterprise deployment.

The Extending Puppet course runs for four days after the show, from Monday, August 26 to Thursday, August 29. The class costs $2,995 and includes a FREE ticket to PuppetConf. Register now.

Education and Certification Workshop

If you’re a Certified Puppet Professional, the Puppet Labs Education and Certification team invites you to attend the Education and Certification Workshop. This is your opportunity to influence Puppet certifications by writing exam questions and evaluating the upcoming Puppet Classroom online learning courses.

This session is scheduled for Wednesday, August 21 — the day before the conferences — and the $500 registration fee includes a FREE conference ticket. Register now, and quickly: This session is limited to 25 attendees.

Puppet 3.2.1 landed today. Though it’s a “patch” release, it’s the first public release of the Puppet 3.2 series, and it includes a taste of the Puppet DSL’s future in the form of an experimental parser that introduces some new features you’d expect to find in traditional programming languages.

I spoke to Puppet product owner Eric Sorenson about the Puppet 3.2 series, and he called out the new parser as the headline item:

“It’s called ‘Future Parser,’” he said, “because that’s the command line argument you have to pass to puppet in order to turn it on. It’s really a ground-up reimplementation of the Puppet language, using an expression- instead of statement-based grammar, which allows both a lot more power and flexibility with what you can do inside the language.”

Eric said the new parser was a response to the tough decision made with Puppet 3.1 to deprecate the Puppet Ruby DSL, a pure Ruby implementation of the Puppet DSL.

“What we did instead was double down on the Puppet DSL by implementing the things that people who were using the Ruby DSL were actually trying to do. We asked, ‘what is that capability? What is it that made you feel the Puppet DSL wasn’t meeting your needs and caused you to turn to the Ruby DSL?’”

According to Eric, “people wanted the ability to iterate. They wanted loops. Puppet looks enough like a programming language that people who are used to programming languages say, ‘I should be able to do all the things I do in Ruby or another language.’”

In the experimental parser, Puppet has picked Ruby-like “each” loops, along with many of the same methods one could call on an enumerable object in Ruby: inject, collect, select and reject.

Eric noted that the new parser isn’t enabled by default, “because there are a couple of things we want to work out, and we really want to hear feedback on.”

For instance, “there are two implementations of loop structures. There’s one where each looks like a function similar to existing Puppet parser functions like include or template. The other one is a dot-suffix operator, like that found in Ruby.”

The goal of including two ways to express the same thing is all about community feedback:

“We’re interested in learning which people think is more readable and more in line with their expectations. We’re going to do some user testing from the UX side and solicit feedback from the community.”

You can give the new parser a shot by updating to Puppet 3.2 and enabling it in one of two ways:

• Setting parser = future in your puppet.conf file
• Adding the command line switch --parser=future

Eric said the experimental parser will remain so until the Puppet developer and UX team can gather community feedback, and learn how it interacts with all the Puppet manifests already out in the wild:

“We’re curious not just about how the parser works specifically with new constructs, but how it works with the existing body of Puppet code.”

You can read more about the new parser, including how to use the new iteration functions, in some documentation prepared by Puppet developer Henrik Lindberg on the Puppet docs site.

External CA Support

Puppet 3.2 also includes better support for external certificate authorities, contributed by an unusual visit from Mozilla’s Dustin Mitchell, who flew out to the Puppet Labs offices to work directly with the Puppet development team.

“We’ve never had anybody from outside the organization come in and sit with us and actually do community contributor stuff in real-time,” said Eric.

Dustin’s contribution involved providing Mozilla’s own Puppet use as a blueprint to development efforts around external CA support:

“Because SSL is hard and PKI is hard, we didn’t want to go in shotgun and make everything under the sun work. We wanted to really specifically focus on what Mozilla’s use case was and use it as a spec to write to.”

The result of that approach is a working reference for other Puppet users who want to use an external certificate authority:

“If they follow the Mozilla model, then we can guarantee that will be supported in Puppet. [Puppet Labs developer] Jeff McCune did some great documentation work around that, describing what’s supported and not supported.”

Wi-Fi Puppet With OpenWRT

Also new in Puppet 3.2 is a collection of updates that address compatibility with OpenWRT, a Linux distribution popular on embedded systems like Wi-Fi routers. OpenWRT makes it possible to turn simple consumer routers into full-fledged Linux systems, and now Puppet can manage them.

Some of the support is predicated on the recent Facter 1.7 release:

“There were some assumptions in Facter that were not correct for non-x86 architecture,” Eric said. Facter addresses that problem by working better with ARM chipsets.

In Puppet, “the support is specifically for OpenWRT. There’s a new package provider that supports the OpenWRT OPKG packaging package manager,” and there’s also support for managing services on OpenWRT.

“Facter 1.7 enables the detection of OpenWRT, and Puppet 3.2 guarantees we can do something useful,” said Eric.

And Everything Else

Besides a new, experimental parser, better support for external CAs, and support for a whole new class of device, Puppet 3.2 includes a few other new features:

• Puppet’s “splay” setting, which was meant to mitigate the problems caused by a thundering herd of agents hitting their puppet master at the same time, has been improved.
• Puppet officially supports Ruby 2.
• Puppet module tool now works on Windows

… and even more. As always, the complete list of bug fixes and new features can be found in the official release notes for Puppet 3.2.

Learn More

• Facter 1.7 Introduces External Facts
• Download the Learning Puppet VM for free and master Puppet in a safe and convenient virtual environment

Just now, VMware announced vCloud Hybrid Service, an infrastructure-as-a-service (IaaS) enterprise public cloud. We’re really excited that, right at launch, Puppet Enterprise is being used for the provisioning, configuration and management of application running on top of vCloud Hybrid Service.

Mathew Lodge, VMware’s VP of Cloud Services, took some time out during launch-day preparations to check in with us at Puppet Labs. While we announced the $30 million investment in January, our vCloud Hybrid Service relationship is a new one between the companies. We’re still learning a lot about each other, and we wanted to know more about VMware’s perception of the cloud and where Puppet Enterprise fits in.

He told us that a common theme in customer conversations and surveys was the demand for a VMware-built cloud service that could “function as an extension of existing data centers.”

Perhaps more telling in the market outside VMware’s customer base, he said, was “the limited adoption of public cloud services, usually in a couple of applications; but there wasn’t broad adoption in enterprises.”

According to Lodge, mobile applications and websites are driving a lot of demand:

“Customers are reacting to the rapid growth in mobile devices, but what’s challenging for them is they have to integrate these with their existing systems. They want to build cloud-native applications, but at the same time, those are usually adjuncts or customizations on ERP or CRM platforms. They need the ability to blend what they already have with new cloud applications.”

Offering a PaaS Roadmap

Moreover, they need a path that doesn’t require them to go all-in on the cloud today:

“They see platform-as-a-service [PaaS] as something they want to do in the future. Getting to PaaS is much easier with a brand new application. The challenge to them is how to get there from where they are today.”

So organizations are looking for incremental steps, because they also have to blend native PaaS applications with existing VM-based applications.

vCloud Hybrid Service is also arriving in a market where the primary focus of IT staff is very much in transition. In a lot of medium-sized organizations, Lodge said, infrastructure is necessary but not a differentiator on its own, “so they’re very interested in ways to run their existing applications and have somebody else take care of the infrastructure piece.”

In exchange, he said, “that frees up IT staff to focus on bringing more value to their businesses, usually leveraging the data and applications they already have.”

With its strength in automation, that’s where Puppet Enterprise begins to come into the vCloud Hybrid Service picture:

“Moving toward much greater automation and the use of APIs is part of the value of going to cloud. These applications can be managed much more effectively programmatically. It’s important to us to support platforms like Puppet Enterprise with vCloud Hybrid Service, because our customers are looking to gain those operational benefits that are possible with cloud infrastructure.”

DevOps and the Shadow IT Challenge

Organizations that have decided to embrace DevOps culture have their own challenges to deal with, trying to bridge the gap between IT operations and application development teams. Part of that gap comes from long-standing differences in how each set of concerns is managed and evaluated.

“IT is typically managed on things like cost and uptime,” said Lodge, “application teams are typically managed on speed or agility. With vCloud Hybrid Service, what we’re trying to do is deliver a service that enables the business to get the speed and agility it’s looking for, but at the same time satisfy the needs of IT around management, visibility, performance and reliability.”

And IT operations are also facing a challenge from “shadow IT,” which confers a lot of the benefits of the cloud to development teams, but leaves ops teams on the outside looking in:

“IT organizations are hungry to satisfy the requirements of their constituencies, but they’ve been challenged. Typically, they’re completely overloaded with what they have. If they’re finding shadow IT, it’s not because developers want to manage infrastructure, it’s because they’re looking for more flexibility.”

vCloud Hybrid Service, said Lodge, looks to serve both of those constituencies and facilitate the silo-busting benefits of DevOps culture. Puppet Enterprise provides a natural fit there, making it much simpler to automate the provisioning and configuration of vCloud Hybrid Service-hosted applications and infrastructure from a single console over thousands of nodes.

The Value for Puppet Enterprise Customers

That single console is already valued by a lot of existing Puppet Enterprise users today, so we were curious about what’s in vCloud Hybrid Service for them.

“The great advantage to them,” said Lodge, “is that they get a single management platform both on premises and in the cloud.”

The cloud is just one part of most organizations’ infrastructures, and in many cases it will never completely represent the balance of IT concerns. Moreover, “It represents a totally different way of managing things. Solving that management challenge means customers can take advantage of cloud more completely, more broadly across the organization.”

“We see some organizations that ultimately want to get out of the business of owning infrastructure,” he said, noting that those organizations will use the hybrid cloud as part of a broader migration strategy.

On the other hand, some organizations will be living both on-premises and in the cloud for a long time to come, leveraging cloud infrastructure for more dynamic and customer-facing applications.

With vCloud Hybrid Service and Puppet Enterprise, he said, “you can essentially rent a software-defined data center from VMware as a public cloud service,” yet still have the same familiar management interface you’ve used for on-premises or cloud deployments.

Learn More

• Download Puppet Enterprise for free to automate your on-premise VMware infrastructure today, and use the same configurations on VMware’s vCHS.

• Sign up for future updates on our partnership with VMware

When you’re responsible for keeping other people’s enterprise websites up and running, you never want to say you’re sorry they’re down.

That’s why Justin Seabrook-Rocha and Patrick Adair both use Puppet technology in their work for Hurricane Electric, an internet services company whose transit backbone connects to more than 2,100 IP networks. Hurricane Electric is located in Fremont, California, and Justin & Patrick both work in the same building that was once the manufacturing facility for NeXT Computer, Steve Jobs’ gig before his 1997 return to the helm of Apple.

Both Justin, a network engineer, and Patrick, a network technician, are registered for PuppetConf in August. They’re expecting to get tips and advice from other attendees and speakers on ways to make their own Puppet infrastructure better, and the latest updates on what’s new with Puppet.

Justin has been working for almost three years on maintenance and configuration for Hurricane’s colocation and transit customers. “I’ve always enjoyed getting my hands dirty with network stuff,” he says. “I live on the command line.”

At last year’s PuppetConf, Justin was hoping to learn more about Foreman. He came away with more than he’d expected, including a rewarding conversation with Ohad Levy, the author of Foreman.

“I got a lot more from Ohad than you get out of documentation,” Justin says. “I was able to grill him for upcoming features.”

Last year’s PuppetConf was also where Justin learned about Hiera and Razor. This year, he’s looking forward to more exposure to Puppet 3 , and to having the chance to pick up fresh information. “I’ve been out of the loop in recent months. I’ll have a chance to come up for air, and hear what’s new in the Puppet community,” Justin says.

Patrick also enjoyed learning about Puppet 3 last year, so much so that he decided to upgrade as soon as he got back to work. “I went to the Hiera and Razor talks, and saw that this was where provisioning was going to be,” Patrick says. “If I wanted to make my life easier, that’s where I needed to be.”

Over the past year, Patrick has extended his use of Puppet. One example: setting up Jenkins to help with continuous deployment.

“We will run through a suite of tests for software we’ve written, so we can say for the branch we’re in, ‘these are the things that are broken,’” Patrick says. “That’s something we couldn’t do as easily or as completely before Puppet came into our lives.”

Vagrant is also in frequent use at Hurricane. “One of the things Puppet has enabled us to do is take a blank Ubuntu 12.04 Vagrant box, and with a single command provision an entire development environment on someone’s laptop or desktop,” Patrick says.

“Basically you clone the repository for the project and hit vagrant up to spin up the virtual machine. It contacts the puppet master and lets you test everything you are writing in the real environment for production.”

While Patrick notes he had to short-circuit the SSL node verifications to get that to work, “it has increased our productivity about twofold. Plus we can create repeatable environments we can test inside of.”

Like Justin, Patrick enjoyed meeting the people behind the technology — particularly Mitchell Hashimoto, the creator of Vagrant.

“He gave me some really good tips to make our processes come up faster,” Patrick says. “I also almost, almost sold him service at one of our data centers.”

As well as stimulating discussion, both Patrick and Justin enjoyed other conference amenities — t-shirts and other great swag, and surprisingly good food.

“The food was actually a highlight,” Patrick said. “It wasn’t like normal conference food. You guys actually splurged and got some nice stuff. It gave me lots of energy.”

You don’t want to miss PuppetConf. Sign up now for the 25 percent Early Bird discount!

Patrick and Justin use the same conference room Steve Jobs used when he was running NeXT Computer. Date of left photo unknown.

PuppetCamp SF is in the books. We had a great day of presentations and community at the Google campus, covering everything from cloud automation to running Puppet at scale.

Automating CloudStack Infrastructure

Dave Nalley presented on using Puppet to express your CloudStack infrastructure as a service platform in code. This concept is experimental but it’s pretty awesome and works quite well. The end-result looks something like this.

cloudstack_instance { 'fooserv': ensure   => present, image    => 'CentOS 6.3', flavor   => 'm1.medium', network => 'my_net', }

Being able to codify all components of your infrastructure, including the compute, disk and network resources can be really liberating. The real gravy comes with Puppet’s ability to inspect the current state of these resources and make corrections based on the actual state of reality. Other efforts in this space include resource-based management of Google Compute and Juniper networking devices.  Check out Puppet Resource Types & Providers for CloudStack.

Community Takeaway: You Love Git, You Love the Forge

“Have you used the Forge?”

<ALL HANDS UP>

“Anyone familiar with Puppet Resource Shell?”

<one hand>

“Who has contributed modules to Puppet Forge?”

<1-2 hands max?>

“Git users?”

<Nearly the entire room>

<Almost all of them using GitHub>

“Svn users?”

<half dozen>

“Perforce?”

<one hand>

Just some of the things we learned when Puppet Forge product owner Ryan Coleman delivered a talk on the Forge and some good practices to follow when building Puppet modules.

You can check out his presentation on the modules and the Puppet Forge, and check out this list of  awesome modules he recommended:

• http://forge.puppetlabs.com/domcleal/augeasproviders
• http://forge.puppetlabs.com/cprice404/inifile
• http://forge.puppetlabs.com/maestrodev/wget
• http://forge.puppetlabs.com/jhaals/warranty

Manage Your Macs With Puppet and Box

GitHub has a lot of developers with Macs who want to quickly get up to speed with the development environment and ship code. The team at GitHub built a tool around Puppet called Boxen, which aims to ease the automation of your development environment. It’s got all the power of Puppet coupled together with some handy tricks and pre-made modules for managing macs. Will Farrington gave an awesome presentation on what Boxen is all about and how to get started.   Will did an awesome job of laying out the problem facing operations folk when faced with an army of developers trying to write and ship code from their laptops. They’ve built a great tool for helping their company succeed with low friction,  and the fully-memed story really resonated with the audience. Check it out.

StubHub Scales Puppet to 5,000 Agents

Following Will was Russ Johnson from StubHub. StubHub is growing fast and Russ built Puppet up in a grassroots effort to help the business keep up with demand. He delivered a talk on his lessons learned in scaling to 5000 Puppet agents. Lessons range from how he’s deploying modules to his production Puppet masters to his favorite VIM plugins to make module development quick and painless.

Dropbox On Functional Hostnames

Finally, Andrew Fong & Gary Josack from Dropbox gave a talk on their use of functional hostnames and lessons learned using them with Puppet. Basing your node classification logic on predictable hostnames can be really powerful but it’s not without challenges.

The day ended with a drink and snack social provided by Google. This was hands-down the best spread I’ve seen at a PuppetCamp.  Google set us up with tons of hot and cold snacks, every drink you could ask for and all the hospitality we could handle. Everyone seemed to have a great time talking about infrastructure while soaking in that amazing view.

Thanks to everyone who came out to Puppet Camp San Francisco – and to all those who weren’t able to make it, we hope to see you in lovely San Francisco for PuppetConf in August!

Learn more

• To view presentation materials from this event or from other past Puppet Camps, visit the Previous Puppet Camps section of the Puppet Camp page.
• We have a Puppet Camp coming up in Raleigh, NC, and other camps will be planned soon. Visit the Puppet Camp page to register, attend, or learn more about submitting talks.
• Attend PuppetConf, our annual operations conference at the Fairmont in San Francisco on August 22 and 23. You can register now or submit a session proposal!
• Find a local Puppet User Group or start one in your city!
• Download the Learning Puppet VM, and get started learning Puppet in a safe environment following our introductory tutorial http://info.puppetlabs.com/download-learning-puppet-VM.html
  Already using Puppet, and have questions? Visit our community Ask site, and get answers and advice on how to take Puppet further.
• Join the Puppet community.

In virtual data centers both on-premises and in the cloud, lifecycle management is particularly challenging for administrators due to the dynamic nature and volume of nodes that need to be deployed and managed. Puppet Enterprise can ease a lot of these challenges by allowing  IT teams to automate the deployment and management of their VMware virtual infrastructure.

VMware and Puppet Labs have co-developed modules for managing VMware virtual environments. Much of this work is discussed in a personal blog entry written by Nick Weaver, an automation architect at VMware.

In the entry, he talks about the ways this cutting-edge integration work opens new horizons to IT teams managing VMware environments:

Management for the vCenter Appliance

• Complete initialization – Just point at a virtual machine deployed from the Open Virtualization Format (OVF), and it will stand up vCenter.
• Data Center and Cluster configuration – High Availability, Distributed Resource Scheduler, and more.
• ESXi Host – Grab and add ESXi hosts to clusters and manage them through Puppet Enterprise

Management for vShield Manager

• Complete initialization – Just like the vCenter Server Appliance, point Puppet Enterprise at your deployed vShield Manager appliance, watch it complete the setup, and stand it up.
• Association to vCenter – Bind the vShield Manager to a vCenter node automatically.
• vShield Edge Deployment(also called vCloud Networking and Security) – Point it at a resource pool, and easily deploy vShield Edges(vCloud Networking and Security) to vCenter.
• vShield Edge Configuration(vCloud Networking and Security) – Control the firewall, NAT, load-balancing, and more using Puppet Enterprise configuration management.

With the features above you can deploy vCenter Server Appliance and vShield Manager appliances, and automatically configure them into vSphere clusters with firewall, NAT, load-balancing and more. You can carve up the ESXi nodes into any number of clusters or configuration. And this is just the beginning.

All of the VMware/Puppet Enterprise integration modules are available under the Apache 2.0 license, making them free to download and modify, and are currently on the Puppet Forge. If you’re a VMware administrator who hasn’t tried Puppet Enterprise, the modules provide a great place to get started. Even more importantly, there’s a great number of brilliant people who can add to this work and make it even better for the IT community as a whole.

Learn More

• If you want an awesome technical walkthrough, read the blog entry on this release from one of the main authors, Nan Liu.
• If you’re already working with Puppet Enterprise, download the modules from the Puppet Forge.
• Find more information on managing VMware virtual environments with Puppet Enterprise on the Puppet Labs VMware solutions page.
• You can download Puppet Enterprise and manage 10 nodes for free. Once you download, you’ll hear from Puppet Labs with links to help get you started

For people who might never have considered a career in accounting, a lot of sysadmins seem to spend their days living out of a spreadsheet. They’re not balancing the books, they’re cataloging their infrastructure.

It’s important work that has to be done. You have to know what you’re running, because your ability to maintain a reliable, responsive infrastructure depends on it. It’s also tedious work, and Excel probably isn’t the best tool for the job. Rather than spending all your time in a spreadsheet, you should be spending it on more interesting things.

One key recommendation from the 2013 State of DevOps Report was how important it is to pick a single source of truth that accurately reflects the configuration of all the systems in your infrastructure, then stick with it. Knowing the current state of everything you’re managing, whether it’s inside the firewall or out in the cloud, provides a way to allocate resources, plan for maintenance, or respond quickly to emerging issues.

If you’re using a spreadsheet to help with this, you already know what the problems are:

• You have to update the information by hand
• If you don’t keep things updated, your inventory quickly becomes useless
• There’s always one more thing it might be nice to know, making that spreadsheet dozens of columns wide and overwhelming to review

It’s just not very efficient, it eats time you could be spending doing anything but accounting, and when it comes time to use the information you’ve been painstakingly recording, it’s awkward to search and you can’t always be sure it’s up-to-the-minute accurate.

The unifying theme among high-performing organizations who responded to the DevOps survey was the importance of automation. The value of automation applies to more than configuration management, though: It can also help when you’re just trying to get a handle on what you own.

A Quick Tour of Puppet Enterprise Inventory Tools

Puppet Enterprise provides some inventory tools that make the process of capturing configuration information in one place much simpler.

In most Puppet Enterprise configurations, puppet agents are checking in with their puppet master every half hour. Though we often think about the work the puppet master is doing — directing the configuration of resources it manages — those agents are also reporting in with the exact kinds of configuration information you may have been recording in that spreadsheet.

Each agent uses Facter, Puppet’s cross-platform system profiler, to relate dozens of facts, from the operating system and kernel versions — the kinds of things you can track by hand — all the way down to operational details you can’t know without automated reporting, like the system’s current uptime and how much free swap it has.

Listing Nodes in Your Infrastructure

At the highest level, the Puppet Enterprise console offers a view of all the nodes in your infrastructure via the node manager view. Though this view is fairly general, it does provide a quick, at-a-glance summary of the general state of your puppet agents. It also lists all the nodes in your Puppet Enterprise-managed infrastructure.

You can get a lot more detail about each node in the list by clicking on it and scrolling down to the inventory data section of the individual node page. That’s where you can see all the information Facter has provided to the puppet master about that node:

If you’re curious about just how comprehensive that list is, the Puppet Labs docs site has a report of all the core facts provided by Facter, and it’s worth noting that with the latest release of Facter, you can easily write your own facts for a given node to relate information that’s important to you but not covered by the core Facter library.

All this information is updated with each puppet run, so you know you’re always getting current data without having to do any manual data entry.

Searching on Configuration Data

You can use the inventory search tool to search across your infrastructure for nodes that match a given fact. For instance, if you want to learn which nodes are running CentOS, you’d search operatingsystem for CentOS:

Another useful fact for sysadmins is system uptime. Performing an inventory search of uptime_days > 21 will show just the nodes that have been up for longer than three weeks.

You can also search on multiple conditions, such as a combination of virtual (to determine the virtualization platform under which a guest node is running) and environment (to find just your production systems).

You could do this kind of filtering with a spreadsheet, but the data gathering alone would be incredibly time consuming, and without any of the additional ever-changing operational data puppet agents routinely relate to their puppet master.

Watch a Video and Try It Out

Sometimes it’s better to see for yourself what a piece of software can do, so I asked Puppet Labs engineer Zach Leslie to record a video demonstrating how you can use Puppet Enterprise as a single source of truth for your infrastructure. Once you’ve watched the video and seen it all in action, you can download Puppet Enterprise and try it out for yourself on 10 nodes, free.

For Max Koehler, a senior developer at a startup in Munich, Germany, culture is one of the most important aspects of working life.

That’s why he’s looking forward to PuppetConf in San Francisco this August. It’s a chance to hang out with developers and other technology people, hear how they resolve the same kinds of issues he deals with – and oh yes, drink some good beer.

“I’m interested in DevOps and how people work together,” Max says. “I can learn how other people manage deployment and infrastructure, how they reduce errors in Puppet runs, how they do their everyday tasks, and how they build teams. There are a lot of guys out there who have solved these kind of problems, and I’m looking forward to meeting and learning from them.”

Wywy, where Max works, has a small team — about 20 people, with some located in Israel. Max and his colleagues try to automate as much as possible, so they can concentrate most of their energies on developing wywy’s core technology for synchronizing TV commercials onto the second screen – smartphones, tablets and more – creating cross-media advertising campaigns.

“Once you do something for the third time, you should think about how to automate it,” Max says.

As in most startups, everyone does more than one job. While it can be chaotic, Max enjoys the freedom to experiment, to discuss best practices with his teammates, and to integrate technologies in ways they think make most sense for the business. It’s very different from how IT worked in the past.

“I’ve seen people setting up thousands of servers, and doing minor changes by hand, like firewall changes or log changes — completely trivial stuff. With Puppet, we add or change just one or two lines, wait a few minutes, and it works.”

The time wasted by lack of automation frustrated Max. He knew the upfront investment in Puppet would save his clients time and money later on, but it wasn’t his call to make.

Max says it’s his engineering background that makes Puppet so appealing to him. “I can describe classes and functions, and how my infrastructure should look,” he says. “I don’t have to think about how to set up a server — I can just describe how I want the system to look.”

Consistent repeatability is another benefit. “When I set up a server with Puppet, I can set it up seven times, and every time, I will have the same setup,” Max says.

The wider world of DevOps that Puppet belongs to draws Max as well. “I hate working in silos,” he says. His team works to improve communication across their different functions. When there’s disagreement, “we try to figure out the problem in how we’re communicating, and we talk about it so we don’t continue to have these problems.”

Other aspects of DevOps culture appeal, too. “We have Hack Days at wywy. Then we go to lunch and drink a beer together.”

It’s that kind of companionship Max looks forward to at PuppetConf, especially after meeting Puppet CTO Nigel Kersten at an April meeting of Munich Puppet users.

“He was very open about the community and about Puppet Labs,” Max said. “And he told me all about the awesome hotel and conference venue.”

Another big draw for Max is the two-week holiday he and a colleague plan to enjoy after PuppetConf. One thing we’re sure of: They’ll have plenty to talk about as they drive the California highways.

Get your 25 percent early-bird discount for PuppetConf before it’s gone!

While Puppet may get all the glory, Facter, the hard working information gathering library that can, seldom gets much exciting new functionality. However with the release of Facter 1.7 Puppetlabs have standardised and included a couple of useful facter enhancements that make it easier than ever to add custom facts to your puppet runs.

These two improvements come under the banner of 'External Facts'. The first allows you to surface your own facts from a static file, either plain text key value pairs or a specific YAML / JSON format. These static files should be placed under /etc/facter/facts.d

$ sudo mkdir -p /etc/facter/facts.d # note - the .txt file extension $ echo 'external_fact=yes' | sudo tee /etc/facter/facts.d/external_test.txt external_fact=worked $ facter external_fact worked

At its simplest this is a way to surface basic, static, details from system provisioning and other similar large events but it's also an easy way to include details from other daemon and cronjobs. One of my first use cases for this was to create 'last_backup_time' and 'last_backup_status' facts that are written at the conclusion of my backup cronjob. Having the values inserted from out of band is a much nicer prospect that writing a custom fact that parses the cron logs.

If that's a little too static for you then the second usage might be what you're looking for. Any executable scripts dropped in the same directory that produce the same output formats as allowed above will be executed by facter when it's invoked.

# scripts must be executable! $ sudo chmod a+rx /etc/facter/facts.d/process_count $ cat /etc/facter/facts.d/process_count #!/bin/bash count=$(ps -efwww | wc -l | tr -s ' ') echo "process_count=$count" $ facter process_count 209

The ability to run scripts that provide facts and values makes customisation easier in situations where ruby isn't the best language for the job. It's also a nice way to reuse existing tools or for including information from further afield - such as the current binary log in use by MySQL or Postgres or the hosts current state in the load balancer.

While there have been third party extensions that provided this functionality for a while it's great to see these enhancements get included in core facter.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

If you’re using virtualization in your data center, you’re familiar with the challenges as well as the benefits: VM sprawl, maintaining consistent configuration, provisioning and tearing down VMs at the speed your business demands. Puppet Enterprise is an exceptional tool for dealing with these challenges. Here are 13 reasons that, if you’re not already using Puppet Enterprise to manage your VMware virtual environment, you should be.

1. vSphere Integration Developed in Partnership with VMware

Puppet Labs and VMware have worked closely on Puppet Enterprise. That collaboration enables system administrators to automatically and dynamically provision, configure, and manage virtual infrastructure. Similarly, VMware vFabric Application Director customers can leverage more than 1,000 freely downloadable modules on the Puppet Forge, enabling them to automate the installation and configuration of many applications.

2. Reduced Outages

Up to 90 percent of IT service outages are caused by human error. Puppet Enterprise can significantly reduce the likelihood of error by allowing server changes to be thoroughly tested in a virtual environment that completely represents the servers’ current and intended configuration and performance. With more thorough testing, uncertainty about a production deployment is dramatically reduced.

3. Visibility and Control of VMware Environments

Puppet Enterprise and VMware vSphere together minimize downtime risk by not only monitoring VMware virtual machine system state, but also the status of applications running in the VMware virtual environment, giving administrators unchallenged visibility and control.

4. One-Stop Automation and Lifecycle Management for Multiple Operating Systems

Puppet Enterprise reduces operational complexity and TCO by eliminating the need for different operating system–based lifecycle and automation products.

5. Reduce Training Costs

Puppet Enterprise gives administrators the ability to recover a desired business-critical system state without specific training on a given application.

6. Automatically Discover Resources and Start Managing in Minutes

With its intuitive graphical user interface, Puppet Enterprise automatically discovers resources – packages, hosts, groups, and users – allowing system administrators to establish a single source of truth about their systems across both physical and virtual nodes. With this knowledge, Puppet Enterprise enables administrators to quickly make changes that would otherwise take hours.

7. Quickly Deploy Critical Updates With a Single Command

Critical updates, such as patching security vulnerabilities, require fast, coordinated action. Puppet Enterprise’s orchestration capability allows you to query your physical or virtual infrastructure to discover the vulnerable nodes, and then, in a single command, schedule the required updates, without logging into a single machine. When updates are complete, Puppet Enterprise delivers a single, aggregated report of the results.

8. Rapidly Add Virtual and Cloud Workloads, and Maintain Change Management

The ability to rapidly scale to virtual and cloud environments while maintaining change management practices is vital to a growing organization. With Puppet Enterprise’s provisioning capability, teams can quickly create, configure, and manage VMware instances using the same Puppet manifests used in their physical infrastructure, ensuring consistency across these heterogeneous environments.

9. Prevent Virtual Machine Sprawl

Puppet Enterprise can easily deploy and terminate virtual node instances, preventing unmanaged sprawl.

10. Puppet’s Vast Community

Using vSphere, vCenter, and vFabric Application Director with Puppet Enterprise, you can leverage the Puppet development community for additional deployment and management solutions.

11. More Streamlined Management

Puppet Enterprise allows for the combined management of servers and endpoints, from Linux servers to Windows machines, physical or virtual, all from a single pane of glass.

12. The Puppet Forge

Access to 1000+ pre-built configuration modules are free on the Puppet Forge. VMware related modules from Puppet Forge can help automate away repetitive tasks associated with the lifecycle management of VMware virtual machine infrastructure.

13. Industry Leadership

InfoWorld’s “Top Ten Emerging Enterprise Technologies” says “private cloud orchestration” is #1, and notes Puppet Labs’ leadership in this space.

Learn More

• Find more resources about using VMware with Puppet Enterprise
• Download Puppet Enterprise and use it to manage 10 virtual nodes for free.

This post is about a particularly elegant (and crucial) feature in puppet exported resources: attribute overriding. If you’re not already familiar with exported resources, you should start there, as they are the killer feature that makes configuration management with puppet awesome. (I haven’t found any explicit docs about this feature either, so feel free to comment if you know where they’re hidden.)

Setup: I’ve got a virtual machine which exports a resource to N different nodes. I’d like to define the resource with just one exported (@@) definition on my virtual machine.

Problem: One (or more) of the attributes needs to be changed based on which node it gets collected on. To make things more complicated, I’m using the same class definition on each of those N nodes to collect the resource. I don’t want to have to write N separate node definitions:

@@some::resource { 'the_name':     foo => 'bar',     #abc => 'different_on_each_node',     tag => 'magic', }

Solution: It turns out that for exported (or virtual) resources, you can specify attributes that get set upon collection. Naturally they can depend on a variable such as $name, which is unique to where they get collected:

Some::Resource <<| tag == 'magic' |>> {     abc => "node-${name}",    # override! }

Bonus: You can obviously use other variables throughout including in the collection (tag == ‘magic’) area, on both the source and the destination. Instead of a simple equality like I’ve used, you can actually specify a more complex expression, including other variables such as title (the $name).

Hope this takes your puppet coding to another level,

Happy hacking,

James

 

We spent several days in Austin last week attending Puppet Camp Austin and DevOpsDays. A big thank you to MomentumSI and Geek Austin for sponsoring Puppet Camp, and thanks to the crew at DevOpsDays for a great event!

We kicked off the week at Puppet Camp Austin on Monday with Andy Parker giving the State of Puppet presentation. He talked about the overall state of IT automation and how Puppet fits into the ecosystem, along with updates about our many projects and initiatives.

We had quite a few great presentations including:

• David Nalley talking about how to use Puppet with Apache CloudStack.
• Adrien Thebo on writing & sharing great modules on the Puppet Forge.
• Dennis Rowe with a session about testing Puppet with RSpec.
• Dave Barcelo telling us all about how they use Puppet at Bazaarvoice.
• David Porter on Drupal development, deployment, and automation with Puppet.
• several last minute lightning talks!

We wrapped up the day with Jeremy Adams, who led us in a round of push ups while we waiting for some issues with the projector to be fixed before transitioning into the demos. He showed off some of the features of Puppet Enterprise, support for administering Windows and much more.

Thank you to everyone who joined us at Puppet Camp Austin! If you missed it, you can get links to the presentations or view materials from other past Puppet Camps by visiting the Previous Puppet Camps section of the Puppet Camp page.

We spent Tuesday and Wednesday at DevOpsDays Austin. We had a great time at the Puppet Labs table where we gave away t-shirts, stickers, copies of the 2013 DevOps Report and more! The best part of being there was talking to people, answering questions about Puppet Enterprise and doing demos.

There were so many great sessions at DevOpsDays Austin, but I had a few favorites!

• John Willis talked about the importance of culture in DevOps.
• Pete Cheslock gave a great talk about retaining employees when everyone is hiring (we’re hiring, too)!
• Patrick DeBois gave us a lesson about genes and memes in the Future of DevOps.
• Gene Kim presented on how we can better sell the concept of DevOps.

Since we’re not an all work and no play kind of shop, we also found time after DevOpsDays to visit the Congress Bridge, where over a million bats come streaming out almost every evening at dusk:

Thank you to everyone who stopped by our table or tracked us down at the event to talk about Puppet!

Learn More:

• Visit the Puppet Camp page to get information about upcoming camps and see materials from past Puppet Camps.
• Attend PuppetConf, our annual operations conference at the Fairmont in San Francisco on August 22 and 23. You can register now or submit a session proposal!
• Find a local Puppet User Group, or start one in your city!
• Download the Learning Puppet Enterprise VM, and get started learning Puppet Enterprise in a safe environment following our introductory tutorial.
• Already using Puppet, and have questions? Visit our community Ask site, and get answers and advice for using Puppet.
• Join the Puppet community.

The monthly Puppet community metrics report from April is now ready to sate your curiosity about the booming Puppet community. Here is a quick summary of the April 2013 edition of the metrics for your reading pleasure, and you can also download the full April Puppet metrics report (PDF link) for your monthly dose of data.

April 2013 Metrics Summary

• 5685 members and 711 messages in Puppet-Users
• 1017 members and 145 messages in Puppet-Dev
• 992 nicks on #puppet IRC channel
• 2468 Puppet Forge accounts and 1080 modules
• 4173 Redmine accounts
• 544 forks / 1255 watchers of Puppet
• 247 Ask questions, 1168 posts, 393 members

Six Month Comparison (November 2012)

• 4988 members and 1264 messages in Puppet-Users
• 921 members and 103 messages in Puppet-Dev
• 919 nicks on #puppet IRC channel
• 1849 Puppet Forge accounts and 720 modules
• 3647 Redmine accounts
• 424 Forks / 1055 watchers of Puppet

The Ask Puppet Labs site, where you can post and answer Puppet-related questions, has continued to grow. Congrats to the top participants so far…

Can you rocket yourself into the top tier by sharing puppet knowledge in May? It’s likely! The Ask Site awaits.

Top 5 participants (by number of messages / replies posted) in the Puppet-Users list:

• jcbollinger
• Felix.Frank
• Ygor
• p….@cmcmarkets.com
• Larry Fast

Here are the top 5 participants (by number of messages / replies posted) in the Puppet-Dev list, topped this month by CEO Luke Kanies:

• Luke Kanies
• Andy Parker
• Erik Dalén
• Henrik Lindberg
• Matthaus Litteken

You can see the full report for the top 10 participants in the Puppet-Users, Puppet-Dev, Puppet-Razor and MCollective mailing lists.

Top 5 participants (by number of lines posted) in the #Puppet IRC channel:

• _rc
• zipkid
• rustx
• brendan-
• binford2k

You can see the list of the top 20 people in the full report.

The full metrics report for April (PDF link) has charts and additional details if you want to see more.

Learn More:

• View more Puppet community metrics
• Ask Puppet Labs Question and Answer site
• Watch the James Turnbull’s “State of the Community” video from PuppetConf
• Join the community in our Google Groups and more, #puppet on IRC, and #puppetize and @PuppetLabs on Twitter.
• Stay up-to-date with the Puppet Labs newsletter

Originally published on Eucalyptus.com. Republished with permission.

When I talk to IT organizations about using Eucalyptus to set up a private or hybrid cloud, they often ask “How do I get started?” Recently, we’ve published a series of Reference Architectures which are specific to common use cases deployed in the field. Let’s take a look at how the Dev/Test Reference Architecture might be used as a framework for building a private or hybrid cloud for Dev/Test purposes.

In many IT organizations, a gap exists between developers and the IT operations team. Developers are looking to create new applications and get them out the door quickly. But there are often delays in provisioning the infrastructure and resources that developers need. This gap often leads to time to market delays and lost productivity.

Eucalyptus provides an AWS-compatible private and hybrid cloud solution which provides developers with the same flexibility and agility they’ve come to expect from the public cloud, while still allowing IT operations teams to maintain control and costs. The Eucalyptus cloud platform provides self-service resource provisioning, so developers and QA engineers can streamline workflow processes and get their jobs done quickly and efficiently. Developers don’t need to submit an IT ticket and wait for weeks to get the resources they need. Developers can quickly provision and configure the compute, storage, and networking resources they need on demand.

So where does the Reference Architecture fit in? The Reference Architecture provides a deployment template for IT ops teams to assist them in setting up a Eucalyptus private or hybrid cloud. The Reference Architecture provides the scope of resources they need and the recommended deployment model for their specific use case (in this case, Dev/Test).

This is important for two reasons: First, it prescribes a bill of materials and how to deploy the Eucalyptus cloud platform on it. Second, the Reference Architecture provides a “recipe” for the bill of materials needed to run the cloud.

In the case of a Dev/Test environment, Infrastructure as a Service (IaaS) provides the automation for provisioning and de-provisioning resources as application and workload demand ebbs and flows. The Reference Architecture provides design choices for the IaaS solution, including physical resources required and a deployment topology.

The Reference Architecture also outlines technologies commonly deployed with Eucalyptus to provide a complete IaaS solution – such as configuration management, monitoring, and workflow management.

Dev/Test is a great place to get started with Eucalyptus and the Reference Architecture provides you the tools to get started successfully.

Andy Knosp is the Vice President of Product and is responsible for product strategy, product management, and product marketing at Eucalyptus Systems.

Learn More

• Learn about Puppet Labs and Eucalyptus Clouds
• Install and configure your own Eucalyptus private cloud with the Eucalyptus Puppet Forge module
• Read about more ways to kickstart cloud projects with Puppet Enterprise

Podcast: Puppet Armatures
jQuery(document).ready(function($) { $('#wp_mep_1').mediaelementplayer({ m:1 ,features: ['playpause','current','progress','duration','volume','tracks','fullscreen'] ,audioWidth:400,audioHeight:30 }); });

In this episode of the Puppet Labs Podcast, we talked about the new Puppet Armatures process.

Puppet’s open source community is amazing. People from all over the world have contributed to its ecosystem with everything from modules to manage complex application stacks, to documentation fixes, to new features in the core. The community is big enough now that we wanted to both give and receive feedback earlier in the design process of significant changes, whether they’re developed internally at Puppet Labs or by outside contributors. There are some great examples of this process and we followed the OpenJDK Java Enhancement Proposal (JEP) model closely when thinking about how to do this. Unfortunately, another great community nabbed the most obvious acronym for Puppet Enhancement Proposals and we didn’t want to tread on Python! So some brainstorming ensued. The result is Armatures, or ARMs for short:

armature |ˈärməCHər, -ˌCHo͝or |
2 a metal framework on which a sculpture is molded with clay or similar material.
• a framework or formal structure, esp. of a literary work: Shakespeare’s plots have served as the armature for many novels.
— New Oxford American Dictionary

We’re still just a short time into the process (the process itself is an ARM and subject to community feedback!) but you can view, comment, and contribute to the
Armature repository on GitHub.

In this podcast, Dawn, Kara and I discuss AMRs with Henrik Lindberg (who generated the idea and is the most prolific ARM author to this point).

See the rest of the podcasts.

Learn more:

• We have Puppet Camps coming up in New York, Austin, Raleigh and more. Visit the
  Puppet Camp page to register to attend or
  learn more about submitting talks.
• Attend PuppetConf, our annual operations conference at the Fairmont in San Francisco on August 22 and 23. You can register now or submit a session proposal!
• Find a local Puppet User Group or start one in your city!
• Download the Learning Puppet VM, and get started learning Puppet in a safe environment following our introductory tutorial.
• Already using Puppet, and have questions? Visit our community Ask site, and get answers and advice for using Puppet.
• Join the Puppet community.

Today, we’ve released version 1.3 of PuppetDB.  This release adds some new features around Puppet reports, and we’ve got big plans for the next several months. For more details, read on!

PuppetDB is the next-generation open source storage service for Puppet-produced data. With the initial 1.0 release in September 2012, it provided a high-performance system for capturing all of the catalogs and facts for your Puppet nodes. It could be used as a drop-in replacement for the Puppet inventory service, and for the first time gave users a fast and scalable way to take advantage of Puppet storeconfigs and exported resources.

But you already know all of that, because you’re already using and loving PuppetDB, right? On the off chance that you’re not, take a peek at the “Learn More” section at the end of this document. There you’ll find links to an introductory video, blog post, and other info about getting started with PuppetDB.

What’s New In PuppetDB 1.3 Extended Report/Event Query API

In PuppetDB version 1.1, we added experimental support for storing your Puppet report data.   PuppetDB 1.1 also included a very basic retrieval API for listing your Puppet reports and collecting all of the events that occurred for an individual report.  Now, in version 1.3, we’ve added a much more robust query API for interacting with your report data.

Query Events by Timestamp

Often you’re only going to be interested in events that occurred during a certain time frame.  Now you can query for them by doing something like this:

 [">", "timestamp", "2013-05-02T12:00:00-03:00"]

This query will return all of the events (across all of the Puppet reports stored in your PuppetDB database) with a timestamp later than noon on May 2.  The results might look something like this:

[      {        "certname": "foo.localhost",        "old-value": "absent",        "property": "ensure",        "timestamp": "2013-05-02T19:01:05.000Z",        "resource-type": "File",        "resource-title": "/tmp/reportingfoo",        "new-value": "file",        "message": "defined content as '{md5}49f68a5c8493ec2c0bf489821c21fc3b'",        "report": "38ff2aef3ffb7800fe85b322280ade2b867c8d27",        "status": "success"      },      {        "certname": "bar.localhost",        "old-value": "absent",        "property": "message",        "timestamp": "2013-05-02T19:01:05.000Z",        "resource-type": "Notify",        "resource-title": "notify, yo",        "new-value": "notify, yo",        "message": "defined 'message' as 'notify, yo'",        "report": "87552aef3ffb7800fe85b322280ade2b867c8d27",        "status": "success"      }    ] Compound Queries

The timestamp query shown above is a lot more useful if you can specify both a start and and end time for your query. To accomplish this, we now support the same set of compound / boolean query operations for reports / events that we support for querying catalogs and facts.  For example:

   ["and", ["<", "timestamp", "2013-05-02T12:05:00-03:00"],    [">", "timestamp", "2013-05-02T12:00:00-03:00"]] That would allow you to look at all of the events that occurred within a 5-minute window. Query by Resource Type/Title/Property

What if you’re only interested in events that happened for a specific resource?  You can query for events matching certain resources using something like this:

   ["and", ["=", "resource-type", "Service"],            ["=", "resource-title", "postgresql"],            ["=", "property", "ensure"]]

This will only give you results relating to the Postgresql Service, and specifically only events that relate to the ensure property of that resource.

Query by Event Status

A common scenario that we envision is that you might only be looking for events that had a certain status: e.g., only failed events, or only noop events.  To do that, all you need to do is query against the status field:

["=", "status", "noop"] Regular Expressions

Equality queries are powerful, but sometimes it’s useful to be able to match against a pattern.  This can be really handy when you’re querying against host names, so we now support the same types of regular expressions for reports / events as we do for resources:

["~", "certname", "^www.*"]

That’ll return events for all nodes whose certnames start with www.

Putting it All Together

We can put together some pretty powerful queries by combining all of the features above.  For example, let’s say that you had a known window of time where some bad things happened on your infrastructure.  You’d like to quickly determine all of the services that failed during that time, but you’ve already taken care of the apache service on all of the nodes, and you’ve taken care of all of your nodes whose name begins with database–so you’re not concerned about those.

["and", ["=", "resource-type", "Service"],        ["=", "status", "failed"],        [">", "timestamp", "2013-05-02T12:00:00-03:00"],        ["<", "timestamp", "2013-05-02T12:05:00-03:00"],        ["not", ["=", "resource-title", "apache"]],        ["not", ["~", "certname", "^database.*"]]]

Voila!  For more info, make sure to check out the complete documentation for the report / event query API.

Import and Export Catalogs and Reports

In PuppetDB 1.2, we added two new command line tools: puppetdb-import and puppetdb-export.  These tools provide support for importing and exporting catalog data from PuppetDB.  This feature is mostly intended to help users migrate to a new PuppetDB instance or database.

In PuppetDB 1.3, these tools now support importing and exporting report data in addition to catalogs.

After writing these tools to help with data migrations, we realized that they also provide a unique opportunity for users to play a role in improving our products by sharing their real-world data with us. We’re constantly looking for new ways to optimize PuppetDB and test new features, and doing so with actual user data would provide us with much greater confidence that our new products are going to work well for our users.  If you’d be willing to share some of your data with us, please contact us! (We’ll keep it private. It will only be used for internal testing and performance profiling.) We’ll even give you a free t-shirt!

What’s Next?

We’re working hard to keep cranking out new features and make sure that you consider PuppetDB to be an indispensable part of your Puppet ecosystem. Here are a few things that you can expect to see in the not-too-distant future:

• PuppetDB bundled with Puppet Enterprise: we intend to deliver an absolutely seamless, dead-simple, out-of-the-box experience for leveraging the power of PuppetDB in Puppet Enterprise environments.
• Capture data about Puppet modules: in an upcoming release of Puppet core, we’ll be adding more information to catalogs about what module (if any) each class or resource was defined in. We’ll also capture that data in PuppetDB, and extend the query API to allow you to include module information in your queries.
• Even more report query functionality: we’re looking into options for querying reports based on classes and modules, querying for aggregates (counts, etc.), and several other possibilities.

Tell Us What You Think!

We think that PuppetDB is really cool. But at the end of the day, it’s not what we think that keeps the lights on over here at Puppet Labs — it’s what you think. So, if you have an opinion on any of our current features or suggestions to help us shape and prioritize features for upcoming releases, we really want to hear from you! Here are some great ways to get in touch with us:

• Sign up to be a Puppet Test Pilot! You’ll get free goodies as a reward for your participation if you are able to volunteer a few minutes of your time to tell us what you think about prototypes of upcoming features. We did a lot of user testing around our query API in the past, and we hope to do a lot more in the future.
• Ping us on IRC: we’re usually online in #puppet on freenode. Just mention ‘puppetdb’ and you’re pretty likely to get a quick response.
• Send an e-mail to the puppet-users or puppet-dev mailing lists.

Learn More

Here are some good resources for learning about and getting started with PuppetDB:

• Deepak Giridharagopal’s Intro To PuppetDB talk from PuppetConf 2012
• Nick Lewis’ Introducing Introducing PuppetDB blog post
• PuppetDB official documentation
• Puppet “Module of the Week” blog post on the PuppetDB module, which can help you get PuppetDB up and running in no time

Release management best practices have evolved over time as software tools that manage and automate parts of the process appear. As a result, established structures are ever changing. An example of this is a 2007 piece on Buildmeister about best practices that were inspired by ITIL, the ISO standard IT Infrastructure Library.

The biggest observation I have about the 2007-era best practices suggestions are the barriers erected between the development and the operations team regarding software deployment, configuration, and testing. A modern DevOps culture, where everyone on the team uses a common source code control system, is missing, and it’s replaced by admonitions to keep the teams separate for management and accountability reasons.

What else has changed in the intervening six years? The 2007 view of best practices doesn’t touch on virtual machine configuration management, which is critical to the world of the modern data center.

An admonition to use “a Deployment Management tool … to move the release to different environments or to install onto multiple desktop machines” needs now to accommodate quickly building and spinning up virtual machines for operating and test environments.

Surprisingly enough, many of the rest of the suggestions still make sense. It’s still all too true that “Software Release Management can be a repetitive and error prone manual process, therefore as much as possible should be automated.” Buildmeister has good recommendations about software packaging and the need to put together testable releases. There’s a curious lack of emphasis on an automated test and build environments, perhaps reflecting the relative immaturity of those tools six years ago.

Time and change march on, and if you are using a state of the art systems approach from the previous decade, there may well be ways to improve the software release management process. Look for ways to get from the previous generation of best practices to the current generation.

Learn More

• Can Release Management Be a Competitive Advantage?
• Is Release Management Just Another Form of Change Management?
• A Catalog of Automation Is Needed for Continuous Delivery

Did you miss our April newsletter? Here’s your chance to catch up with the highlights of the past month. If you’d like to skip the bookmark and get May’s newsletter straight to your inbox, subscribe here.

Get Started With Puppet

We’re getting ready for the most exciting IT operations conference of the year: PuppetConf 2013. If you sign up now, you can snag a 25 percent discount. Not convinced? Read our San Francisco holiday guide, then book your trip.

If you’re not even sure what Puppet’s all about, or how it can help you, we have a great podcast from early April where Puppet Enterprise users told us how they got started.

If you’d like to learn more, you can read about the Puppet Labs Classroom, a chance to do some bite-sized learning about all things Puppet.

Puppet Master Power-Ups

If you’re already using Puppet and want to learn how to up your game, we had plenty of posts in April that might keep you busy until the end of May:

• Speed Up Your Web Development with Vagrant and Puppet: Use Puppet and Vagrant to automate away the pain of managing VMs, keeping your web developers up to speed with your production environment.
• Facter 1.7 Introduces External Facts: Make a puppet agent provide custom facts without writing any Ruby. Just drop in a text file and go.
• Bridging the Two Worlds: IT and Networking: Puppet Labs works with Juniper to extend Junos OS and provide a new networking automation solution
• Hiera for Pouncers and Stalkers: We recently gave our Hiera documentation a big shot in the arm. Here’s a tour of the new docs for every type of admin.

Graphic of the Month

The OpenStack conference came to Portland. We had a booth, hosted a party, and talked to the OpenStack community. Listen to how various OpenStack leaders are using Puppet Enterprise to automate public, private and hybrid clouds:

DevOps In Action

We released the results of our 2013 State of DevOps Report in late March, and in April we followed up with additional reporting that’ll help you break out of the silos in your own organization:

• Hiring for the DevOps Toolchain: The Need for Generalists: Hiring for DevOps? Find out why generalists might work for you …
• A Four Quadrant Look at the DevOps Toolchain: DevOps as a service delivery platform, and how to use the toolchain to get the platform to work.

Puppet Community

There was lots going on in the Puppet community, too:

If you’d like to speak at PuppetConf 2013!, we want to hear from you. Accepted speakers get free tickets to PuppetConf, so apply today!

We were also out and about at Puppet Camps. Here are recaps from our London and Amsterdam events:

• Puppet Camp London
• Puppet Camp Amsterdam

Puppet In The News

We’re really happy to welcome our new VP of engineering, Brian Stein. Brian joined us from Red Hat. The Portland Business Journal has a report.

Subscribe Today

If you liked what you read and want more, subscribe here or just browse our archives.

I’ve been working for a nice project the last month. Implementing puppet Enterprise and puppet with the foreman. Why would someone want to use puppet enterprise  and puppet with foreman? Well that’s a good question

Foreman is very versatile with its smart proxy functionality.  This is very important if you have a complex network environment. Like puppet, foreman is written in ruby and can  easily be integrated in the puppet enterprise framework. Foreman also has nice features like locations and organization as well as  provisioning to bare metal,  RHEV, Libvirt, Vmware, Amazon Cloud etc. etc. and you can integrate it into katello and add mcollective, which is very nice too.  This means you could basically have an all in one solution, config and repo management and provisioning. Some of you guys would say oh but puppet Enterprise has a very nice LIve Management Console and can do provisioning to clouds too!  I have to say your right, but not being able to lock the mcollective part down via web interface is a no go in a large environment.

Now where is the benefit of running (Enterprise/Open source) solution in parallel?

• Licenses
• You get the support for the puppet enterprise edition which kind of means also for the open source
• There is no difference between the Puppet Enterprise puppet binary version and the community version. So you can test on the Enterprise binary, and deploy to both environments.
• You can start building your environment with open source components without having to pay a large sum in advance in licenses.
• You can have one interface “Foreman” to manage all of your nodes.
• You get fantastic support on the #puppet or #foreman irc channel

As I could not find any documentation on how to integrate foreman with puppet enterprise and puppet open source in parallel I decided to write 2 documents:

• Set up the Foreman Proxy with passenger on Puppet PE and use foreman as an ENC and reporting engine
• Set up the Foreman Proxy to use mcollective with puppet PE / Puppet