Purge debconf selections

If you’ve been playing with debconf selections, and you need to purge them, you can use “debconf-communitcate” to get that done, e.g.

echo PURGE | debconf-communicate $PACKAGE

See the debconf spec for more info.

Chef 11 - Libraries and LWRP testing in chef-shell

In chef 11 you can no longer load up the chef-shell, load a cookbook and access the libs and LWRPS that it provides.

You need a few extra steps. Credit to this goes to @btmspox who put this in a CHEF ticket which I can’t current locate.

Testing libs in chef shell, for chef 11 and greater

Start chef shell:

$ chef-shell -c /path/to/chef-config.rb -z

All of the following can be run from chef shell, you don’t need to be in recipe mode:

Chef::Config[:cookbook_path] = "/path/to/your/cookbooks"
run_list = "recipe[partial_search]"
cl = Chef::CookbookLoader.new(Chef::Config[:cookbook_path])
cl.load_cookbooks
cookbook_collection = Chef::CookbookCollection.new(cl)
@events = Chef::EventDispatch::Dispatcher.new()
node.run_context = Chef::RunContext.new(node, cookbook_collection, @events)
node.run_list = Chef::RunList.new(run_list)
node.run_context.load(node.expand!)

You can now use partial search from the shell:

partial_search(:node, "chef_environment:production", :keys => { 'fqdn' => ['fqdn'] })

sql postgres

Show postgres CREATE TABLE commands

In MySQL you can see how a table is created using

SHOW CREATE TABLE table_name;

In postgres you don’t have this command, however you can use pg_dump to get the schema definition of a table:

$ pg_dump -U user --schema-only -t table_name database > table_schema.sql

You can also describe the structure of the table with the \d command:

database=> \d+ table_name

apt-get update failing with method died error

If you’ve tried to install or update a package on debian based systems (debian, ubuntu, etc) and you’ve had the following error:

Failed to exec method /usr/lib/apt/methods/
E: Method  has died unexpectedly!
E: Sub-process  returned an error code (100)
E: Method /usr/lib/apt/methods/ did not start correctly

You might find this a pain to debug. There are a few things that seem to cause this problem:

corrupt sources.list entry, you have an entry in your sources.list file (or a file included from sources.list.d) which contains a deb or deb-src line which has a malformed uri method (see sources.list(5) for a list of valid methods) or is just an incorrect entry. Fix the malformed entry, run apt-get update and then all should be well.

corrupt entry in /var/lib/apt/lists - if your sources.list files are ok, you may have a corrupt entry in one of the lists that have been downloaded. To resolve this you can backup the files somewhere, remove the existing files, and then run apt-get update, it’ll download a new set of lists. You then should be able to install/update packages.

Debugging the problem

I debugged this problem with the apt-get manpages, strace to find out what the process was trying to load, and the apt sources to explain what the error message was saying and why it would occur.

Edit

This has been happening a lot on my servers. I’ve further dug into it and found that the eu-west-1.ec2.archive.ubuntu.com_ubuntu_dists_natty_multiverse_i18n_Translation-en and eu-west-1.ec2.archive.ubuntu.com_ubuntu_dists_natty_main_i18n_Translation-en files appear to be the ones causing the problem. After performing an update these files are no longer present. I’ve not had a chance to dig into why this is yet.

Reconnoiter MySQL checks

When using MySQL 5.0 or greater, ensure that you set the “sslmode” parameter in your DSN.

Failure to do this will cause noit to crash as there is a strcmp in the mysql.c module code which fails.

You can set ‘sslmode=”required”’ if you need to use SSL, or to any other string if not (e.g. ‘sslmode=”disabled”’).

reconnoiter graphing sysadmin

Reconnoiter composite graphs

Composite graphing in reconnoiter is pretty handy. To create a composite graph you will use the same RPN functions as you would to manipulate normal graphs.

The datapoints consist of the numeric datapoints that you’ve added to your graph, indexed off of zero.

For example, if you have four data points for system CPU, and you want to make a composite you can put the following in the function field:

[0],[1],[2],[3],+,+,+

Then hide all of the datapoints apart from the composite one.

reconnoiter graphing monitoring sysadmin

Reconnoiter Graph Templates

Reconnoiter graph templates

I’ve spent some time working with reconnoiter and getting the templates setup on the web interface. As there isn’t much information regarding how the templates work I thought I’d share it here to help others.

Firstly there currently isn’t any template editor with reconnoiter, you’ll either have to build one yourself, or you’ll need to perform inserts directly into the database. For all of these examples I’ll be performing the inserts directly into the database.

Where are the templates stored:

In the database the templates are stored in the prism.graph_templates table, when you first look in this table it will be blank. If you describe the table you’ll see that it has three columns:

The templateid should be a UUID value. The title should be a meaningful title you want to give the template, and the json is the actual JSON that you’ll use for the template.

Creating JSON for the templates

Reference: http://lists.omniti.com/pipermail/reconnoiter-users/2009-August/000159.html

The link above gives an example of the JSON that you should use for a template, I found that this example mostly worked, though I had to quote the “sid” values or PHP would throw a fatal error as the JSON was invalid.

Breaking down the JSON:

Using the above JSON, the query you make into postgres then looks like this:

Some example templates

An example of a query that would add a template for SSH duration metrics:

This template could be used for average ping time metrics:

This is more complex, it has three values and a guide for the 95th percentile. In this case I’ve setup a check that pulls down ELB (Elastic Load Balancer) metrics from AWS Cloudwatch. This example is setting up a template for the latency values. Note that the title will be named “ELB Latency (seconds)” followed by the text you enter. See the JSON on the mailing list post linked above for an example of putting other variables in the title:

Using the templates

Once the templates are in place, reload your reconnoiter web interface and browse to the templates section under graph templates. You’ll see a list of the templates you’ve added. When you expand a template you’ll see a list of all of the targets that have metrics that match that template. You can now expand a target, select the metric, give it a name and hit “create graph”.

Now browse to graphs, search for the graph you’ve just added and it should be there.

Things to note

I’ve noticed that if you have multiple metrics with the same name they’ll show up in the templates. For example both the nginx module and the ssh2 module will collectd a metric called “duration”, when you add this as a metric for a template you’ll see both of those listed as metrics you can choose.

Troubleshooting

Your reconnoiter apache error log will log your PHP errors and can be helpful in tracking down problems. The Reconnoiter_GraphTemplate.php, Reconnoiter_DB.php and template_browse_json.php files can be very helpful in working out what’s going on. They are all in the “ui” section of the source.

Ensure your JSON is correct and parseable, you can use http://json.parser.online.fr to verify it.

If you’re using Chrome, the developer tools are excellent for showing the requests that are being made.

iOS emoji