Runa - Cognizant Transmutation

Creating an Amazon EC2 AMI for Opscode Chef 0.8 Client and Server

Robert J Berger — Tue, 12 Jan 2010 09:00:21 +0000

Changes Since Original

1/13/10: Fix various minor inaccuracies and improved description on how to set up the chef-server. Also removed nanite as a requirement (its no longer used)
1/17/10: Add the requirement to build and install mixlib-authentication for the chef-client
1/21/10: Added a mkdir for /var/log/chef
1/22/10: Added step to insure that /tmp permissions are set

Introduction

Here’s my experience setting up an Amazon EC2 AMI and Instance for a Chef Server and Client. It is based mostly on Bryan Mclellan (btm)‘s post of Nov 24, 2009 Installing Chef 0.8 alpha on Ubuntu Karmic and his more up to date GIST: chef 0.8 alpha installation. It has a slightly different focus and is a bit stale if you are building your own 0.8 gems from the source.

Instantiate an Amazon EC2 Instance

We’ll start with the Canonical Ubuntu 9.10 Karmic AMI. I always go to Eric Hammond’s site alestic.com to get the pointers to the right AMIs. In this case we’re using a 32bit image for the US-West Region: ami-7d3c6d38 US-East 32bit: ami-1515f67c. You can use the US-West 64bit image ami-7b3c6d3e, US-East 64bit: ami-ab15f6c2

Start the instance from your local dev machine using the command line ec2-api-tools (available as a package or directly from Amazon) or using something like the Firefox Elasticfox and then ssh into the instance so that you can do the following steps on the instance. For the sake of this example, lets say that the Public DNS name for the instance you started is ec2-204-222-170-10.us-west-1.compute.amazonaws.com and the ssh keypair you associated with this new instance is now on your local dec machine in ~/.ssh/gsg-keypair

Prerequisite preparation

The first set of steps need to be done on the instance you just created so login via ssh:

ssh -i ~/.ssh/gsg-keypair ec2-204-222-170-10.us-west-1.compute.amazonaws.com

If on Amazon us-west

There is a bug in the current us-west Canonical AMI where it does not use the us-west apt server. So you have to correct the apt soruces.list:

sed -i.bak '1,$s/us.ec2.archive.ubuntu.com/us-west-1.ec2.archive.ubuntu.com/' \
/etc/apt/sources.list

For all cases

sudo sed -i.bak2 '1,$s/universe/universe multiverse/' /etc/apt/sources.list
sudo apt-get -y update
sudo apt-get -y upgrade
sudo apt-get -y install emacs23 # Of course this is the first package to install!

# Will need these to manipulate ec2 images
sudo apt-get -y install ec2-api-tools ec2-ami-tools

Set up the ruby environment and install rubygems

Install Ruby and needed packages

sudo apt-get -y install -y ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb \
build-essential wget ssl-cert git-core rake librspec-ruby libxml-ruby \
thin couchdb zlib1g-dev libxml2-dev

Install Rubygems

Rubygems will be installed from source since debian/ubuntu try to control rubygems upgrades. If you don’t care you can install it via apt-get install rubygems

cd /tmp
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar zxf rubygems-1.3.5.tgz
cd rubygems-1.3.5
sudo ruby setup.rb
sudo ln -sfv /usr/bin/gem1.8 /usr/bin/gem
sudo gem sources -a http://gems.opscode.com
sudo gem sources -a http://gemcutter.org

Install Pre-requisit Gems

sudo gem install cucumber merb-core jeweler uuidtools \
json libxml-ruby --no-ri --no-rdoc

Building and Installing Chef Related Gems

Until there are final 0.8.x Chef gems, you will have had to build them on your local machine and upload them to this instance. On your dev machine (this example builds things in ~/src, but it could be anywhere appropriate) follow these instructions to build all the gems and install gems you might need to use your local machine. You will use your local dev machine to develop and manage cookbooks and to manage a remote chef-server:

mkdir ~/src
cd ~/src
git clone git://github.com/opscode/chef.git
git clone git://github.com/opscode/ohai.git
git clone git://github.com/opscode/mixlib-log
git clone git://github.com/opscode/mixlib-authentication.git
# Need to get mixlib-log for client & server and
# mixlib-authentication for the client from git till the 1.1.0 update hits
# See http://tickets.opscode.com/browse/CHEF-823
cd mixlib-log
sudo rake install
cd mixlib-authentication
sudo rake install
cd ../ohai
sudo rake install
cd ../chef
rake gem
# Now cd into ~/src/chef/chef to install the chef client/dev gem on your local machine
cd chef
rake install

Upload the gems needed for the client to your instance. From ~/src on your local dev machine do:

scp -i ~/.ssh/gsg-keypair chef/chef/pkg/chef-0.8.0.gem  ohai/pkg/ohai-0.3.7.gem \
mixlib-authentication/pkg/mixlib-authentication-1.1.0.gem \
mixlib-log/pkg/mixlib-log-1.1.0.gem  ec2-204-222-170-10.us-west-1.compute.amazonaws.com:

Set up the Chef Client on the new Instance

Now back in your home directory on the instance ec2-204-222-170-10.us-west-1.compute.amazonaws.com install the gems you just copied over:

sudo gem install mixlib-log-1.1.0.gem ohai-0.3.7.gem
sudo gem install chef-0.8.0.gem

Create the client config file

mkdir /var/log/chef
mkdir /etc/chef
chown root:root /etc/chef
chmod 755 /etc/chef

Put the following in /etc/chef/client.rb:

# Chef Client Config File

require 'ohai'
require 'json'

o = Ohai::System.new
o.all_plugins
chef_config = JSON.parse(o[:ec2][:userdata])
if chef_config.kind_of?(Array)
  chef_config = chef_config[o[:ec2][:ami_launch_index]]
end

log_level        :info
log_location     "/var/log/chef/client.log"
chef_server_url  chef_config["chef_server"]
registration_url chef_config["chef_server"]
openid_url       chef_config["chef_server"]
template_url     chef_config["chef_server"]
remotefile_url   chef_config["chef_server"]
search_url       chef_config["chef_server"]
role_url         chef_config["chef_server"]
client_url       chef_config["chef_server"]

node_name        o[:ec2][:instance_id]

unless File.exists?("/etc/chef/client.pem")
  File.open("/etc/chef/validation.pem", "w") do |f|
    f.print(chef_config["validation_key"])
  end
end

if chef_config.has_key?("attributes")
  File.open("/etc/chef/client-config.json", "w") do |f|
    f.print(JSON.pretty_generate(chef_config["attributes"]))
  end
  json_attribs "/etc/chef/client-config.json"
end

validation_key "/etc/chef/validation.pem"
validation_client_name chef_config["validation_client_name"]

Mixlib::Log::Formatter.show_time = true

Set up the /etc/init.d/chef-client

Copy the example init.d script (You can also use runit instead, but we’re not going to describe that here)

cp /usr/lib/ruby/gems/1.8/gems/chef-0.8.0/distro/debian/etc/init.d/chef-client /etc/init.d
cd /etc/init.d
update-rc.d chef-client defaults

Create an Init script to set /tmp to proper permmissions

It looks like the Canonical Images will not have /tmp with proper permissions if you exclude /tmp from your bundle process. Eric Hammond recommends doing the following.

Create a file /etc/init.d/ec2-mkdir-tmp with the following contents:

#!/bin/sh
#
# ec2-mkdir-tmp Create /tmp if missing (as it's nice to bundle without it).
#
mkdir -p    /tmp
chmod 01777 /tmp

Then set up the /etc/rc dirs to launch this on boot:

chmod a+x /etc/init.d/ec2-mkdir-tmp
ln -s /etc/init.d/ec2-mkdir-tmp /etc/rcS.d/S36ec2-mkdir-tmp

Build the EC2 Image

The always amazingly helpful Eric Hammond has a post, Creating a New Image for EC2 by Rebundling a Running Instance, that describes the basics of how to do this. The following is pretty much a direct synopsis with minimal explanation. See his blog post for more details.

Clean up potential security holes

Remove stuff you don’t want to freeze into your image.

sudo rm -f /root/.*hist* $HOME/.*hist*
sudo rm -f /var/log/*.gz

Copy AWS Certs to Instance

Back on your local development system, copy your Amazon certificates to the instance.


remotehost=
remoteuser=ubuntu
scp -i  \
  /{cert,pk}-*.pem \
  $remoteuser@$remotehost:/tmp

Create the new Image on the Instance

Back on the ec2 instance, you’ll do the following to create the image.

Define where to store the image on S3

This assumes you have an S3 account setup on AWS. You don’t have to have already created the bucket. Set some bash variables that will be used by the commands that follow. You should set the prefix to something that is meaningful. Below is what I used as an example. You’ll want to make it unique to your environment. The Bucket name must be Globally unique across all of Amazon S3.

bucket=runa-west-amis
prefix=runa-ubuntu-9.10-i386-20100101-base

Define your AWS credentials and target processor

export AWS_USER_ID=
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
else
  arch=i386
fi

Bundle the files
This also runs on the current instance and will bundle the everything on the instance file system except for dirs specified with the -e flag into a copy of the image under /mnt:

sudo -E ec2-bundle-vol           \
  -r $arch                       \
  -d /mnt                        \
  -p $prefix                     \
  -u $AWS_USER_ID                \
  -k /tmp/pk-*.pem               \
  -c /tmp/cert-*.pem             \
  -s 10240                       \
  -e /mnt,/tmp,/root/.ssh,/home/ubuntu/.ssh

If you are deploying to US-West-1 AWS Region

Looks like the Amazon ec2 ami tools are not super aware about us-west yet. So you have to do this extra step right now. You’ll have to change the –kernel and –ramdisk to the ones appropriate for your kernel. You can inspect the values used for the AMI you used to boot the original instance. You can do this with ElasticFox or with the command (specify the AMI and region its in thatyou want to check):

ec2-describe-images ami-7d3c6d38   -C /tmp/cert-*.pem -K /tmp/pk-*.pem --region us-west-1

Then execute the following command and specify the right kernel and ramdisk

sudo -E ec2-migrate-manifest        \
  -c /tmp/cert-*.pem             \
  -k /tmp/pk-*.pem               \
  -m /mnt/$prefix.manifest.xml   \
  --access-key $AWS_ACCESS_KEY_ID  \
  --secret-key $AWS_SECRET_ACCESS_KEY \
  --kernel aki-773c6d32          \
  --ramdisk ari-713c6d34         \
  --region us-west-1

Upload the bundle to a bucket on S3:

sudo -E ec2-upload-bundle        \
    -b $bucket                   \
    -m /mnt/$prefix.manifest.xml \
    -a $AWS_ACCESS_KEY_ID        \
    -s $AWS_SECRET_ACCESS_KEY    \
    --location us-west-1

You may be prompted with something like:

You are bundling in one region, but uploading to another. If the kernel or ramdisk associated with this AMI are not in the target region, AMI registration will fail.
You can use the ec2-migrate-manifest tool to update your manifest file with a kernel and ramdisk that exist in the target region.
Are you sure you want to continue? [y/N]

You should enter y return to accept.

Register the AMI

Back on your local development machine:

ec2-register $bucket/$prefix.manifest.xml --region us-west-1

The output of this will be the ami-id of your new instance. You can use this to instantiate your new ami.

You now have a private ami image you can start just like any other image. If you want to make it public

ec2-modify-image-attribute -l -a all

Using the new AMI Image

You can now use this instance as the basis for chef clients and also the basis to create a Chef Server. Use the Amazon EC2 tool, ElasticFox or whatever you favorite tool for managing EC2 instances to make a new instance first to create a Chef Server. Then after that you can create clients and have them load their roles and recipes from the chef server. Once you have a Chef Server, you can use knife ec2 instance command to create user data that includes a run list, credentials and other json that can be passed to the general ec2 tools to build specific instances.

Creating a Chef Server from your new Image

Using an EC2 tool like ec2-tools or elasticfox, create a new instance based on the AMI created earlier. You should use at least a c1.medium as the m1.small is just too painfully wimpy to use. Assume the new instance has the Public DNS name: ec2-204-203-51-20.us-west-1.compute.amazonaws.com
Copy the chef server gems to the new instance from the ~/src directory in your local dev environment to the new instance:

scp -i ~/.ssh/gsg-keypair chef/*/pkg/*.gem \
ec2-204-203-51-20.us-west-1.compute.amazonaws.com:

ssh to the new instance and do the following:

sudo gem install chef-server-0.8.0.gem chef-server-api-0.8.0.gem \
chef-server-webui-0.8.0.gem chef-solr-0.8.0.gem

Set things up to use bootstrap client using chef-solo

We’ll be using the last part of BTM’s GIST, and danielsdeleo (Dan DeLeo)’s bootstrap cookbook and chef-solo to set up this initial server.

mkdir -p /tmp/chef-solo
cd /tmp/chef-solo
git clone git://github.com/danielsdeleo/cookbooks.git
cd cookbooks
git checkout 08boot

Create ~/chef.json:

{
  "bootstrap": {
    "chef": {
      "url_type": "http",
      "init_style": "runit",
      "path": "/srv/chef",
      "serve_path": "/srv/chef",
      "server_fqdn": "localhost"
    }
  },
  "recipes": "bootstrap::server"
}
# End of file

Create ~/solo.rb with the following content:

file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
# End of ~/solo.rb file

Run chef-solo which will execute the chef bootstrap recipes using the bootstrap params in ~/chef.json to actually setup and configure this chef server

If you had installed rubygems with the ubuntu apt package you may have to specify the path:

/var/lib/gems/1.8/bin/

instead of:

/usr/bin

for the knife and various chef commands in the following code.

/usr/bin/chef-solo -j ~/chef.json -c ~/solo.rb -l debug

You will see a lot of Debug statements go by and it will take several minutes to complete. It should complete with something like:

[Thu, 14 Jan 2010 00:19:38 +0000] INFO: Chef Run complete in 38.59808 seconds
[Thu, 14 Jan 2010 00:19:38 +0000] DEBUG: Exiting

Setup basic cookbooks

The following will install the standard cookbooks on the chef server

cd
git clone git://github.com/opscode/chef-repo.git
cd chef-repo
rm cookbooks/README
git clone git://github.com/opscode/cookbooks.git

Now upload the standard cookbooks using the credentials set up by the bootstrap process (user chef-webui)

knife cookbook upload --all -u chef-webui \
-k /etc/chef/webui.pem -o cookbooks

Startup the Chef Server web ui

Do to a bug (http://tickets.opscode.com/browse/CHEF-839) you have to run this twice, the first time will create the admin user:

sudo /usr/bin/chef-server-webui -p 4002

But the first time will abort with an error message like:

Loading init file from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/init-webui.rb
Loading /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/environments/development.rb
~ Loaded slice 'ChefServerWebui' ...
WARN: HTTP Request Returned 404 Not Found: Cannot load user admin
~ Compiling routes...
~ Could not find resource model Node
~ Could not find resource model Client
~ Could not find resource model Role
~ Could not find resource model Search
~ Could not find resource model Cookbook
~ Could not find resource model Client
~ Could not find resource model Databag
~ Could not find resource model DatabagItem
/usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/init-webui.rb:32: uninitialized constant OpenID (NameError)
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `call'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `each'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:99:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:172:in `bootup'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:42:in `start'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core.rb:173:in `start'
from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/bin/chef-server-webui:76
from /usr/bin/chef-server-webui:19:in `load'
from /usr/bin/chef-server-webui:19

Then again to actually start the WebUI and have it run in the background. You might want to start it in screen for now or possibly redirect its output to a log file The following example shows sending the output of the command to a log file. You’ll want to check that log file after starting to make sure there were no errors.

sudo sh -c '/usr/bin/chef-server-webui -p 4002 > /var/log/chef-server-webui.log' &

If you look at the output of a ps, you’ll see the shell command above, but the real work is being done by a merb instance with the port you specified (4002):

#ps ax | grep webui
5533 pts/0    S      0:00 sh -c /usr/bin/chef-server-webui -p 4002 > /var/log/chef-server-webui.log
#ps ax | grep merb
3694 ?        Sl     0:55 merb : worker (port 4000)
5534 pts/0    Sl     0:07 merb : worker (port 4002)

The first merb worker is the chef-server itself, the second is the WebUI server.

Accessing the Chef Web UI

You can access the Chef Web UI web server using a web browser at the IP address / Public DNS name of this server that was just set up. Assuming the Public DNS is

ec2-204-203-51-20.us-west-1.compute.amazonaws.com

Assuming that you set up this instance to allow you to access port 4002 from the IP adddress of your local dev machine, you should be able to access the Web UI at

http://ec2-204-203-51-20.us-west-1.compute.amazonaws.com:4002

You can allow access to port 4002 from specific ip address ranges by updating your security group. You can do that with ElasticFox (easy) or via the command line tools (a pain for a one off). Eventually you (or hopefully Opscode) will set up an apache or nginx reverse proxy, Passenger or equiv to allow normal port 80 / 443 http/https access.

Conclusion

You should now be able to use knife your local dev environment to develop cookbooks and upload roles and cookbooks to your new Chef Server and spin up new chef cookbook driven instances. You should use the knife documentation from the Opscode main wiki Knife Page NOT the docs in the Alpha Forums / Getting Started With Opscode / Knife – Commandline API as the later is actually more obsolete in terms of the version that you built from the opscode git repository. There is also a man page and knife –help gives you pretty much the same correct info as the wiki.

I hope to have a follow up post on how to do this in more details.

Feel free to leave comments if you find problems or have questions.

The post Creating an Amazon EC2 AMI for Opscode Chef 0.8 Client and Server first appeared on Cognizant Transmutation.

Building Opscode Chef 0.8.x from HEAD of the git repo

Robert J Berger — Wed, 23 Dec 2009 02:55:44 +0000

Update:

I am having problems using the chef dev tools/client from the HEAD of the chef git repository with the Opscode Alpha Server service. I’m not sure if its me or if the latest versions of the chef client from HEAD is compatible with the Alpha Server Service. So the following is still useful for understanding how to build from HEAD, but it will not work with the Opscode Alpha SaaS server. It will work with the server you build from HEAD. See the next article Creating an Amazon EC2 AMI for Opscode Chef 0.8 for info on creating a Chef client and server on EC2.

Introduction

Opscode is introducing a pretty major set of changes in Chef in the 0.8 release. Its a major step forward and has some major changes as to how one interacts with Chef. (as well as some major bug fixes that alone make it worth the move). The Opscode Alpha Program introduces a new service where Opscode runs the actual Chef Server as a service.

This post will describe setting up your User/Dev environment by building your own Chef Client / Dev Gems from the latest HEAD of the Chef repo from Github. It assumes that you did sign up for the Alpha program and have access to the Opscode Alpha Server. Though much of it would be the same if you were running your own chef server also built from the latest source from github. This post does not show how to actually use Chef and the chef-client on a target node. Hope to have a post on that in the next few days.

The documentation on how to move to and use Chef 0.8 is still very sparse, so I figured I would jot down some of the things we are learning as we apply this to our infrastructure at Runa. If any of you OpsChefs out there see something wrong or something I left out, let me know in the comments or via email.

The Opscode Chef Alpha Environment

If you are in the Opscode Alpha program, you would have been given login[s] and some pem keys. I won’t go into the details of this since they do have pretty good docs on setting this up (if you have an alpha login you can get them at http://opscode.zendesk.com/forums/58858/entries/49336). Its probably a good idea to follow these and start with their 0.8.0 gem to make sure you are talking with the Alpha Server before trying to use the Chef Git Repository to build your own gems.

The Alpha instructions use a Chef gem that is frozen at 0.8.0. But the Chef folks have already progressed much further than the Oct 29h release of 0.8.0 in the Chef Git Repository.

The HEAD of the Git Repository has many changes since 0.8.0. Some big ones include:

The Knife sub commands are completely different
There is now a Chef Shell (A REPL like irb but for the chef client)
Lots of Bug Fixes

And if we’re going to be on the bleeding edge, we might as well go all the way! So the rest of this blog will be about using the Chef HEAD branch from the Chef git repository. We’ll still use the Alpha Chef Server at least to start with.

Configuring your Dev Environment

Prerequisites

I’m using Mac OS X 10.6 (snow leopard). Our target environments are Ubuntu Linux on Amazon EC2. But assuming you have *nix, Ruby and Ruby Gems set up on your environment it should generally be the same (don’t know about people stuck in the Legacy Windows environment though).

So you will need to have installed and know how to use:

Ruby
RubyGems
Git

And the following Ruby Gems should be installed (I think this is the minimum you need, these will include their own dependencies:

rake
rspec
cucumber
uuidtools
nanite
gemcutter
jeweler

You will need http://gems.opscode.com as a gem source for the following. You can use the command:

sudo gem sources -a http://gems.opscode.com

mixlib-authentication

Getting and building the code/GEMs for the Dev Environment

The instructions that are in the README.doc of the Chef Git Repository are out of date as of now (Dec 20, 2009). The instructions on the wiki, Installing Chef from HEAD are more accurate. Even though it seems like one can use the mixlib gems as the repository and the gems have the same version number, I found that I needed to install the mixlib libraries from source.

Getting and building Ohai & Mixlib Gems from Github

We won’t be making any changes in these, so we’ll just git clone and build it:

cd to where you want to keep your local repositories
git clone git://github.com/opscode/ohai.git
cd ohai
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-config.git
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-log.git
sudo rake install
cd ..
git clone git://github.com/opscode/mixlib-cli.git
sudo rake install
cd ..

Getting the Chef code from github

You can get the Chef repository from github. The readme there has most of the info you need for

If you plan to submit any patches or other changes back to Opscode, or you would like to have your own repository of this, you can fork the Opscode repository into your own Github account. This is what I did and will demonstrate below. If you don’t want any hardcore forking action, you can just git clone the opscode repository as shown here (assuming your current working directory is where you want the local directory repository placed. It will be named using the default “chef”):

git clone git://github.com/opscode/chef.git

If you have forked into your own github account (mine is rberger), you would git clone using the “Your Clone URL”:

git clone git@github.com:rberger/chef.git rberger-chef

This assumes you want your local directory name for the repository to be rberger-chef, just so you can distinguish it from the official opscode one. (I will refer to the top of the local repository as rberger-chef from now on).

What’s in the Chef Git Repository

Change directory into the local repository and do an ls. You’ll see that there are several components here.


$ cd rberger-chef
$ ls
CHANGELOG         README.rdoc       chef-server       chef-solr         scripts
LICENSE           Rakefile          chef-server-api   cucumber.yml
NOTICE            chef              chef-server-webui features

There are 2 main trees:

chef: chef-client and dev gem
chef-server: Chef Server gem. Used only if you build your own server
- chef-server-api: Implements the REST interface sub-system as part of the full Chef Server
- chef-server-webui: Implements the WebUI as part of the full Chef Server
- chef-solar: Implements the Solar Search sub-system as part of the full Chef Server
- features: Not 100% sure all its used for, definately for the cucumber tests. But is part of the Server as far as I can tell

For now we are only interested in the chef tree. That will be used to set up the local dev environment. We’re not going to follow the outdated instructions that are in the README.doc in the root of the chef repository which assumes you are setting up the whole stack on the Dev machine. We’re going to just install the chef client and tools from the chef sub-tree on the dev machine.

This post will not describe how to build /use the chef-server, though you can pretty much build everything by running

sudo rake install

from the top of the distro. There are more gem dependencies that need to be installed before you can build the chef-server trees.

Building and Installing the Chef Client / Dev tools

Change directory to the chef subdirectory so you should be in rberger-chef/chef (or if you have a direct clone of the opscode chef repository: chef/chef)

cd chef

Some minor tweaks to the Source

(shef is now included in the executables in the latest repository and setting my own sub-version number was lame)

I have done a few mods to the source. Mainly to set the version number to something that will not conflict with the official numbering now or when new releases come out and to have shef be installed by the gem.

Changed line 30 in the Rakefile to s.executables = %w( chef-client chef-solo knife shef ) so the install puts shef in /usr/bin
Changed line 7 in the Rakefile to CHEF_VERSION = "0.8.0.1"
Change line 30 in lib/chef.rb to VERSION = '0.8.0.1'

Build and install

rake install

Its going to eventually ask for your sudo password as it needs to use sudo to do the gem install. The run should look something like:

(in /Users/rberger/work/Chef/rberger-chef/chef)
mkdir -p pkg
WARNING:  no rubyforge_project specified
WARNING:  description and summary are identical
  Successfully built RubyGem
  Name: chef
  Version: 0.8.0.1
  File: chef-0.8.0.1.gem
mv chef-0.8.0.1.gem pkg/chef-0.8.0.1.gem
sudo gem install pkg/chef-0.8.0.1 --no-rdoc --no-ri
Password:
Building native extensions.  This could take a while...
Successfully installed eventmachine-0.12.10
Successfully installed amqp-0.6.5
Successfully installed thor-0.12.0
Successfully installed deep_merge-0.1.0
Successfully installed moneta-0.6.0
Successfully installed chef-0.8.0.1
6 gems installed

Using Chef with the Opscode Alpha SaaS Server

This just touches on some of the things that are described in The Official Guide to Getting Started With Opscode

Setting up your Dev Environment

Its not clear if you really have to do everything as described in the document if you are building the latest release from the chef repository and using the ~/.chef/knife.rb config described below. For instance I didn’t have to set the environment variables for OPSCODE_USER and OPSCODE_KEY since they are now set in the knife.rb nor did I have to create /etc/chef/client.rb. And even without the global Chef config, I was able to use most of the knife commands. But not some like the ec2 instances data seemed to need the organization validation key to be in /etc/chef/validation.pem

Copy your assigned validation key to /etc/chef

When you got your Opscode Alpha welcome stuff, you should have gotten your user keys and a key for your organization. Copy your organization (in our case runa.pem) to /etc/chef/validation.pem. You will probably have to create /etc/chef directory first.

The User Chef/Knife config

You must configure a knife config file in your home directory under ~/.chef/knife.rb and have your key that you got from Opscode somewhere pointed to by a line in ~/.chef/knife.rb. The configuration parameters are described on the Knife Wiki Page. For instance my config file:

log_level        :info
log_location     STDOUT
node_name        'rberger'
client_key       '/Users/rberger/.chef/rberger.pem'
chef_server_url  "https://api.opscode.com/organizations/runa"
cache_type       'BasicFile'
cache_options( :path => '/Users/rberger/.chef/checksums' )

Once you have this set up you can now use knife and the chef rake commands. You can test things out by saying something like:

knife client list

Which should return and empty list assuming you haven’t set up any clients on this server yet.

The first real useful command you want to do is to upload your cookbooks to the Opscode Server:

cd to where your chef cookbook repository is
rake upload_cookbooks

You can also do it with just knife:

knife cookbook upload -a

This may take a while as it will upload all the cookbooks in cookbooks and site-cookbooks in your current repository.

After that you can upload single cookbooks

knife cookbook upload

Just remember the knife documentation on the Alpha site no longer applies to the knife that you get from building from the HEAD of the chef git repository. Strangely enough, the knife documentation on the wiki is accurate.

Conclusion

Once you’ve been thru it, its all quite simple. I hope to post some more on using 0.8.0+ soon. See a more recent blog post for building your own Chef Server Creating an Amazon EC2 AMI for Opscode Chef 0.8

The post Building Opscode Chef 0.8.x from HEAD of the git repo first appeared on Cognizant Transmutation.

Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2

Robert J Berger — Sat, 05 Sep 2009 01:34:41 +0000

NOTE (Sep 7 2009): Updated info on need to use Amazon Private DNS Names and clarified the need for the masters, slaves and regionservers files. Also updated to use HBase 0.20.0 Release Candidate 3

Introduction

As someone who has “skipped” Java and wants to learn as little as possible about it, and as one who has not had much experience with Hadoop so far, HBase deployment has a big learning curve. So some of the things I describe below may be obvious to those who have had experience in those domains.

Where’s the docs for HBase 0.20

If you go to the HBase wiki, you will find that there is not much documentation on the 0.20 version. This puzzled me since all the twittering, blog posting and other buzz was talking about people using 0.20 even though its “pre-release”

One of the great things about going to meetups such as the HBase Meetup is you can talk to the folks who actually wrote the thing and ask them “Where is the documentation for HBase 0.20

Turns out its in the HBase 0.20.0 distribution in the docs directory. The easiest thing is to get the pre-built 0.20.0 release candididate 3. If you download the source from the version control repository you have to build the documentation using Ant. If you are an Java/Ant kind of person it might not be hard. But just to build the docs, you have to meet some dependencies like

What we learnt with 0.19.x

We have been learning a lot about making HBase Cluster work at a basic level. I had a lot of problems getting 0.19.x running beyond a single node in Psuedo Distributed mode. I think a lot of my problems was just not getting how it all fit together with Hadoop and what the different startup/shutdown scripts did.

Then we finally tried the HBase EC2 Scripts even though it uses an AMI based on Fedora 8 and seemed wired to 0.19.0. Its a pretty nice script if you want to have an opionated HBase cluster set up. But it did educate us on how to get a cluster to go. It has a bit of strangeness by having a script in /root/hbase_init that is called at boot time to configure all the hadoop and hbase conf scripts and then call the hadoop and hbase startup scripts. Something like this is kind of needed for Amazon EC2 since you don’t really know what the IP Address/FQDN is until boot time.

The scripts also set up an Amazon Security Group for the cluster master and one for the rest of the cluster. I beleive it then uses this as a way to identify the group as well.

The main thing we did get was by going thru mainly the /root/hbase_init script we were able to figure out what the process was for bringing up Hadoop/HBase as a cluster.

We did build a staging cluster with this script. We were able to pretty easily change the scripts to use 0.19.3 instead of 0.19.0. But its opions were different than ours for many things. Plus after talking to the folks at the HBase Meetup, and having all sort of weird problems with our app on 0.19.3, we were convinced that our future is in HBase 0.20. And 0.20 introduces some new things like using Zookeeper to manage the Master selection so seems like its not worth it for us to continue to use this script. Though it helped in our learning quite a bit!

Building an HBase 0.20.0 Cluster

This post will use the HBase pre-built Release Candidate 3 and the prebuild standard Hadoop 0.20.0.

This post will show how to do all this “by hand”. Hopefully we’ll have an article on how to do all this with Chef sometime soon.

The Hbase folks say that you really should have at least 5 regionservers and one master. The master and several of the regionservers can also run the zookeeper quorum. Of course the master serveris also going to run the Hadoop Nameserver Secondary name server. Then the 5 other nodes are running the Hadoop HDFS Data nodes as well as the HBase region servers. When you build out larger clusters, you will probably want to dedicate machines to Zookeepers and hot-standby Hbase Masters. Name Servers are still the Single Point of Failure (SPOF). Rumour has it that this will be fixed in Hadoop 0.21.

We’re not using Map / Reduce yet so won’t go into that, but its just a mater of different startup scripts to make the same nodes do Map/Reduce as HDFS and HBase.

In this example, we’re installing and running everything as Root. It can also be done as a special user like hadoop as described in the earlier blog post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard

Getting the pre-requisites in order

We started with the vanilla alestic Ubuntu 9.04 Jaunty 64Bit Server AMI: ami-5b46a732 and instantiated 6 High CPU Large Instances. You really want as much memory and cores as you can get. You can do the following by hand or combine it with the shell scripting described below in the section Installing Hadoop and HBase.

apt-get update
apt-get upgrade

Then added via apt-get install:

apt-get install sun-java6-jdk

Downloading Hadoop and HBase

You can use the production Hadoop 0.20.0 release. You can find them at the mirrors at http://www.apache.org/dyn/closer.cgi/hadoop/core/. The examples show from one mirror:

wget http://mirror.cloudera.com/apache/hadoop/core/hadoop-0.20.0/hadoop-0.20.0.tar.gz

You can download the HBase 0.20.0 Release Candidate 3 in a prebuilt form from http://people.apache.org/~stack/hbase-0.20.0-candidate-3/ (You can get the source out of Version Control:http://hadoop.apache.org/hbase/version_control.html but  you'll have to figure out how to build it.)

wget http://people.apache.org/~stack/hbase-0.20.0-candidate-3/hbase-0.20.0.tar.gz

Installing Hadoop and HBase

Assuming that you are running in your home directory on the master server and that the target for the versioned packages is in /mnt/pkgs and that there will be a link in /mnt for the path to the home for hadoop and hbase:

You can do a some simple scripting to do the following on all the nodes at once:

Create a file named servers with the list of the fully qualified domain names of all your servers including “localhost” for the master and call the file “servers”.

Make sure you can ssh to all the servers from the master. Ideally you are using ssh keys. On master:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

On each of your region servers make sure that the id_dsa.pub is also in their authorized_keys (don’t delete any other keys you have in the authorized keys!)

Now with a bit of shell command line scripting you can install on all your servers at once:

for host in `cat servers`
 do
 echo $host
 ssh $host 'apt-get update; apt-get upgrade; apt-get install sun-java6-jdk'
 scp ~/hadoop-0.20.0.tar.gz ~/hbase-0.20.0.tar.gz $host:
 ssh $host 'mkdir -p /mnt/pkgs; cd /mnt/pkgs; tar xzf ~/hadoop-0.20.0.tar.gz; tar xzf ~/hbase-0.20.0.tar.gz; ln -s /mnt/pkgs/hadoop-0.20.0 /mnt/hadoop; ln -s /mnt/pkgs/hbase-0.20.0 /mnt/hbase'
done

Use Amazon Private DNS Names in Config files

So far I have found that its best to use the Amazon Private DNS names in the hadoop and hbase config files. It looks like HBase uses the system hostname to determine various things at runtime. Thie is always the Private DNS name. It also means that its difficult to use the Web GUI interfaces to HBase from outside of the Amazon Cloud. I set up a “desktop” version of Ubuntu that is running in the Amazon Cloud that I VNC (or NX) into and use its browser to view the Web Interface.

In any case, Amazon instances normally have limited TCP/UDP access to the outside world due to the default security group settings. You would have to add the various ports used by HBase and Hadoop to the security group to allow outside access.

If you do use the Amazon Public DNS names in the config files, there will be startup errors like the following for each instance that is assigned to the zookeeper quorum (there may be other errors as well, but these are the most obvious):

ec2-75-101-104-121.compute-1.amazonaws.com: java.io.IOException: Could not find my address: domU-12-31-39-06-9D-51.compute-1.internal in list of ZooKeeper quorum servers
ec2-75-101-104-121.compute-1.amazonaws.com:     at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.writeMyID(HQuorumPeer.java:128)
ec2-75-101-104-121.compute-1.amazonaws.com:     at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.main(HQuorumPeer.java:67)

Configuring Hadoop

Now you have to configure the hadoop on master in /mnt/hadoop/conf:

hadoop-env.sh:

The minimal things to change are:

Set your JAVA_HOME to where the java package is installed. On Ubuntu:

export JAVA_HOME=/usr/lib/jvm/java-6-sun

Add the hbase path to the HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=/mnt/hbase/hbase-0.20.0.jar:/mnt/hbase/hbase-0.20.0-test.jar:/conf

core-site.xml:

Here is what we used. Primarily setting where the hadoop files are and the nameserver path and port:





   
     hadoop.tmp.dir
     /mnt/hadoop
   

   
     fs.default.name
     hdfs://domU-12-31-39-06-9D-51.compute-1.internal:50001
   

   
     tasktracker.http.threads
     80

mapred-site.xml:

Even though we are not currently using Map/Reduce this is a basic config:





   
     mapred.job.tracker
     domU-12-31-39-06-9D-51.compute-1.internal:50002
   

   
     mapred.tasktracker.map.tasks.maximum
     4
   

   
     mapred.tasktracker.reduce.tasks.maximum
     4
   

   
     mapred.output.compress
     true
   

   
     mapred.output.compression.type
     BLOCK

hdfs-site.xml:

The main thing to change based on your config is the dfs.replication. It should be less than the total number of data-nodes / region-servers.





   
     dfs.client.block.write.retries
     3
   

   
     dfs.replication
     3

Put the Fully qualified domain name of your master in the file masters and the names of the data-nodes in the file slaves.

masters:

domU-12-31-39-06-9D-51.compute-1.internal

slaves:

domU-12-31-39-06-9D-C1.compute-1.internal
domU-12-31-39-06-9D-51.compute-1.internal

We did not change any of the other files so far.

Now copy these files to the data-nodes:

for host in `cat slaves`
do
  echo $host
  scp slaves masters hdfs-site.xml hadoop-env.sh core-site.xml ${host}:/mnt/hadoop/conf
done

And also format the hdfs on the master

/mnt/hadoop/bin/hadoop namenode -format

Configuring HBase

hbase-env.sh:

Similar to the hadoop-env.sh, you must set the JAVA_HOME:

export JAVA_HOME=/usr/lib/jvm/java-6-sun

and add the hadoop conf directory to the HBASE_CLASSPATH:

export HBASE_CLASSPATH=/mnt/hadoop/conf

And for the master you will want to say:

export HBASE_MANAGES_ZK=true

hbase-site.xml:

Mainly need to define the hbase master, hbase rootdir and the list of zookeepers. We also had to bump up the hbase.zookeeper.property.maxClientCnxns from the default of 30 to 300.




   
     hbase.master
     domU-12-31-39-06-9D-51.compute-1.internal:60000
   

   
     hbase.rootdir
     hdfs://domU-12-31-39-06-9D-51.compute-1.internal:50001/hbase
   
   
     hbase.zookeeper.quorum
     domU-12-31-39-06-9D-51.compute-1.internal,domU-12-31-39-06-9D-C1.compute-1.internal,domU-12-31-39-06-9D-51.compute-1.internal
   
   
     hbase.cluster.distributed
     true
   
   
     hbase.zookeeper.property.maxClientCnxns
     300

You will also need to have a file called regionservers. Normally it contains the same hostnames as the hadoop slaves:

regionservers:

domU-12-31-39-06-9D-C1.compute-1.internal
domU-12-31-39-06-9D-51.compute-1.internal

Copy the files to the region-servers:

for host in `cat regionservers`
do
  echo $host
  scp hbase-env.sh hbase-site.xml regionservers ${host}:/mnt/hbase/conf
done

Starting Hadoop and HBase

On the master:

(This just starts the Hadoop File System services, not Map/Reduce services)

/mnt/hadoop/bin/start-dfs.sh

Then start hbase:

/mnt/hbase/bin/start-hbase.sh

You can shut things down by doing the reverse:

/mnt/hbase/bin/stop-hbase.sh
/mnt/hadoop/bin/stop-dfs.sh

It is advisable to set up init scripts. This is described in the Ubuntu /etc/init.d style startup scripts section of the earlier blog post:Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard

The post Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2 first appeared on Cognizant Transmutation.

Want to work at a Startup with Cool Tech? (HBase, Clojure, Chef, Swarms, Javascript, Ruby & Rails)

Robert J Berger — Fri, 28 Aug 2009 18:15:01 +0000

Opportunity Knocks

Runa.com, the startup where I am CTO, is looking for great developers to join our small agile team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations). Runa offers a SaaS to on-line merchant that allows them to offer dynamic product and consumer specific promotions embeded in their website. This will be a very large positive disruption to the online retailing world.

Techie keywords: clojure, hadoop, hbase, rabbitmq, erlang, chef, swarm computing, ruby, rails, javascript, amazon EC2, emacs, Macintosh, Linux, selenium, test/behavior driven development, agile, lean, XP, scalability

If you’re interested, email jobs@runa.com

If you want to know more, read on!

What do we do

Runa aims to provide the top of the long tail thru the middle of the top 500 online retailers with tools/services that companies like amazon.com use/provide. These smaller guys can’t afford or don’t have the resources to do anything on that scale, but by using our SaaS services, they can make more money while providing customers with greater value.

The first service we’re building is what we call Dynamic Sale Price.

It’s a simple concept – it allows the online-retailer to offer a sale price for each product on his site, personalized to the individual consumer who is browsing it. By using this service, merchants are able to –

Increase conversion (get them to buy!) and
Offer consumers a special price which maximizes the merchant’s profit

This is different from “dumb-discounting” where something is marked-down, and everyone sees the same price. This service is more like airline or hotel pricing which varies from day to day, but much more dynamic and real-time. Further, it is based on broad statistical factors AND individual consumer behavior. After all, if you lower prices enough, consumers will buy. Instead, we dynamically lower prices to a point where statistically, that consumer is most likely to buy.

How we do it

Runa does this by performing statistical analysis and pattern recognition of what consumers are doing on the merchant sites. This includes browsing products on various pages, adding and removing items from carts, and purchasing or abandoning the carts. We track consumers as they browse, and collect vast quantities of this click-stream data. By mining this data and applying algorithms to determine a price point per consumer based on their behavior, we’re able to maximize both conversion (getting the consumer to buy) AND merchant profit.

We also offer the merchant comprehensive reports based on analysis of the mountains of data we collect. Since the data tracks consumer activity down to the individual product SKU level (for each individual consumer), we can provide very rich analytics. This is a tool that merchants need today, but don’t have the resources to build for themselves.

The business model

For reference, it is useful to understand the affiliate marketing space. Small-to-medium merchants (our target audience) pay affiliates up to 40% of a sale price. Yes, 40%. The average is in the 20% range.

We charge our merchants around 10% of sales the Runa delivers. Our merchants are happy to pay it, because it is a performance-based pay, lower than what they pay affiliates, and there is zero up-front cost to the service. In fact, the above mentioned analytics reports are free.

We’re targeting e-commerce PLATFORMS (as opposed to individual merchants); in this way, we’re able to scale up merchant-acquisition. We have 10 early-customer merchants right now, with about 100 more planned to go live in the next 2-3 months. By the end of next year, we’re targeting about 1,000 merchants and 10,000 merchants the following year. Our channel deployment model makes these goals achievable.

At something like a 5 to 10% service charge, and a typical merchant having between 500K to 1M in sales per year, this is a VERY profitable business model. That is, of course, if we’re successful… but we’re seeing very positive signs so far.

Technology

Most of our front-end stuff (like the merchant-dashboard, reports, campaign management) is built with Ruby on Rails. Our merchant integration requires browser-side Javascript magic. All our analytics (batch-processing) and real-time pricing services are written in Clojure. We use RabbitMQ for all our messaging needs. We store data in HBase. We’re deployed on Amazon’s EC2.

Here are a few blog postings about what we’ve been up to –

Distributed Clojure system in production
Using messaging for scalability
Capjure: a simple HBase persistence layer
Clojure in production
Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2

We’ve also open-sourced a few of our projects –

swarmiji – A distributed computing system to write and run Clojure code in parallel, across CPUs
capjure – Clojure persistence for HBase

Culture at Runa

We’re a small team, very passionate about what we do. We’re focused on delivering a ground-breaking, disruptive service that will allow merchants to really change the way they sell online. We work start-up hours, but we’re flexible and laid-back about it. We know that a healthy personal life is important for a good professional life. We work with each other to support it.

We use an agile process with a lot of influences from the “Lean”:http://en.wikipedia.org/wiki/Lean_software_development and “Kanban”:http://leansoftwareengineering.com/2007/08/29/kanban-systems-for-software-development/ world. We use “Mingle”:http://studios.thoughtworks.com/mingle-agile-project-management to run our development process. Everything, OK mostly everything is covered by automated tests, so we can change things as needed.

We’re all Apple in the office – developers get a MacPro with a nice 30″ screen, and a nice 17″ MacBook Pro. We deploy on Ubuntu servers. Aeron chairs are cliché, yes; but, very comfy.

The environment is chilled out… you can wear shorts and sandals to work… Very flat organization, very non-bureaucratic… nice open spaces (no cubes!). Lunch is brought in on most days! Beer and snacks are always in the fridge.

We’re walking distance to the San Antonio Caltrain station (biking distance from the Mountain View Caltrain/VTA lightrail station).

What’s in it for you

Competitive salaries, and lots of stock-options
Cutting edge technology stack
Fantastic business opportunity, and early-stage (= great time to join!)
Developer #5 – means plenty of influence on foundational architecture and design
Smart, full bandwidth, fun people to work with
Very comfortable, nice office environment
We have a “No Assholes” policy

OK!

So, if you’re interested, email us at jobs@runa.com

No recruiters please!

We would prefer folks who are already in the Bay Area (but if you not local and are really great let’s talk!)

The post Want to work at a Startup with Cool Tech? (HBase, Clojure, Chef, Swarms, Javascript, Ruby & Rails) first appeared on Cognizant Transmutation.

Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard

Robert J Berger — Tue, 06 Jan 2009 02:19:16 +0000

UPDATE: This has been replaced by a newer post Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2 . I found that using the pre-built distributions of Hadoop and HBase much better than trying to build from source. I need more Java/Ant-fu to do the build from scratch. The HBase-0.20.0 Release Candidates are really great and seemingly easier to get the cluster going than previous releases.

Introduction

Hadoop and Map / Reduce are all the rage now days, so we figure we should be using it too.

Hbase is an implementation of Google’s Bigtable. Its built on top of the Hadoop File System (HDFS).

Its trivial to install it as a standalone on top of a filesystem, but I had some difficulty getting it working on top of HDFS in the “Pseudo-Distributed” mode.

Follow the Instructions

I set up Hadoop with no problems following the instructions on the Hadoop sitefor Pseudo-Distributed Operation which runs Hbase on top of HDFS but everything runs on one server (I.E. Its configured pretty much like a cluster but all the pieces are on the same server). Another helpful set of instructions are at Running Hadoop On Ubuntu Linux (Single-Node Cluster).

I followed the HBase installation instructions also for Pseudo-Distributed Operation.

A few things to be aware of:

Make sure that the Hadoop version and the Hbase major version numbers are the same
(I used Hadoop 0.18.2 and Hbase 0.18.1)
Make sure that the Hadoop, Hbase trees as well as the directories and files that hold the hdfs filesystem are owned by hadoop:hadoop (You have to create the user and group)
No need to disable ipv6 as some sites said

You can download the Hadoop tar file from http://www.apache.org/dyn/closer.cgi/hadoop/core/ and the Hbase tar file from http://www.apache.org/dyn/closer.cgi/hadoop/hbase/
They are also available as git repositories via:

git clone git://git.apache.org/hadoop.git
git clone git://git.apache.org/hbase.git

You can track a particular branch with the command (We’re stuck at hadoop 0.19.1 / hbase 0.19.0:

cd hadoop
git branch --track release-0.19.1 origin/tags/release-0.19.1
git checkout release-0.19.1
cd ../hbase
git branch --track 0.19.0 origin/tags/0.19.0
git checkout 0.19.0

Then in each directory build things. As far as I can tell you just need to use the default ant build. But you can build the jar also:

cd ../hadoop
ant
ant jar

cd ../hbase
ant
ant jar

Biggest Problem I Had

The thing that took the longest time to get right was when I wanted to access Hbase from other hosts. You would think you could put the DNS Fully Qualified Domain Name (FQDN) in the config file. Turns out that by default, the Hadoop tools don’t seem to use the host’s DNS resolver and just what is in /etc/hosts (as far as I can tell). So you have to use the IP address in the config file.

I believe there are ways to configure around this but I haven’t found it yet.

Configuration Examples

File System Layout

I untarred the distributions into /usr/local/pkgs and made symbolic links to /usr/local/hadoop and /usr/local/hbase as well as created the directory where Hadoop/HDFS will use for storage.

For Ubuntu:

sudo addgroup hadoop
sudo adduser --ingroup hadoop hadoop

For Mac:

Create a Home Directory

mkdir /Users/_hadoop

Find an unused groupid by seeing what ids are already in use:

sudo dscl . -list /Groups PrimaryGroupID | cut -c 32-34 | sort -rn

Then find an unused userid by seeing what userid’s are in use:

sudo dscl . -list /Users UniqueID | cut -c 20-22 | sort -rn

Pick a number that is in neither list. In our case we will use 402 for both the userid and groupid for _hadoop (Mac OS X has an underscore in front of daemon user/group names. We will also

sudo dscl . -create /Groups/_hadoop PrimaryGroupID 402
sudo dscl . -append /Groups/_hadoop RecordName hadoop

Take the Value of dsAttrTypeStandard:PrimaryGroupID in this case 500, and use it as the groupid in the following command:

sudo dscl . -create /Users/_hadoop UniqueID 402
sudo dscl . -create /Users/_hadoop RealName "Hadoop Service"
sudo dscl . -create /Users/_hadoop PrimaryGroupID 402
sudo dscl . -create /Users/_hadoop NFSHomeDirectory /Users/_hadoop
sudo dscl . -append /Users/_hadoop RecordName hadoop

For both Ubuntu and Mac (Note that the Mac will end up having a user/group id of _hadoop)

cd /usr/local/pkgs
tar xzf hadoop-0.18.2.tar.gz
tar xzf hbase-0.18.1.tar.gz

cd ..
ln -s /usr/local/pkgs/hadoop-0.18.2 hadoop
ln -s /usr/local/pkgs/hbase-0.18.1 hbase
mkdir /var/hadoop_datastore
chown -R hadoop:hadoop hadoop/ hbase/ /var/hadoop_datastore /Users/_hadoop

Hadoop Config files

The following are all in /usr/local/hadoop/conf

hadoop-env.sh

Need to set the JAVA_HOME variable. I installed java 6 via synoptic. You can also install it with:

apt-get install sun-java6-jdk

The Macintosh is a easy if you have a Intel Core 2 Dual (the Intel Core Dual doesn’t count). Apple is only supporting Java 1.6 on their 64 bit processors. If you have a 32 bit processor like the first generation Macbook Pro 17″ or first generation MacMini, or you have a PPC see Tech Tip: How to Set Up JDK 6 and JavaFX on 32-bit Intel Macs

So my config is (only the things I changed, the rest was left as is):

...
# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
 export JAVA_HOME=/usr/lib/jvm/java-6-sun
...

For the Macintosh:

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/Current

hadoop-site.xml






  hadoop.tmp.dir
  /var/hadoop_datastore/hadoop-${user.name}
  A base for other temporary directories.



  fs.default.name
  hdfs://localhost:54310
  The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.



  mapred.job.tracker
  localhost:54311
  The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  



  dfs.replication
  1
  Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  



  dfs.datanode.socket.write.timeout
  0



   dfs.datanode.max.xcievers
   1023

HBase Config Files

The following are all in /usr/local/hbase/conf

hbase-env.sh

Again, just need to set up JAVA_HOME:

...
# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME=/usr/lib/jvm/java-6-sun
...

For the Macintosh:

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/Current

hbase-site.xml

Here is where I wanted to give a FQDN for the host that is the hbase.master, but had to use an IP address instead.




  
    hbase.rootdir
    hdfs://localhost:54310/hbase
    The directory shared by region servers.
    Should be fully-qualified to include the filesystem to use.
    E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR
    
  

  
    hbase.master
    192.168.10.50:60000
    The host and port that the HBase master runs at.

Formatting the Name Node

You must do this as the same user as will be running the daemon (hadoop)

su hadoop -s /bin/sh -c /usr/local/hadoop/bin/hadoop namenode -format

on the Mac:

/usr/bin/su _hadoop /usr/local/hadoop/bin/hadoop namenode -format

Setup passphraseless ssh

Now check that you can ssh to the localhost without a passphrase:

su - hadoop
ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands (as haddop):

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Ubuntu /etc/init.d style startup scripts

I scoured the InterTubes for example hadoop/hbase startup scripts and found absolutely none! I ended up creating a minimal one that is so far only suited for the Pseudo-Distributed Operation mode as it just calls the start-all / stop-all scripts.

/etc/init.d/hadoop

Create the place it will put its startup logs

mkdir /var/log/hadoop

Create /etc/init.d/hadoop with the following:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          hadoop services
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description:       Hadoop services
# Short-Description: Enable Hadoop services including hdfs
### END INIT INFO
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
HADOOP_BIN=/usr/local/hadoop/bin
NAME=hadoop
DESC=hadoop
USER=hadoop
ROTATE_SUFFIX=
test -x $HADOOP_BIN || exit 0
RETVAL=0
set -e
cd /

start_hadoop () {
    set +e
    su $USER -s /bin/sh -c $HADOOP_BIN/start-all.sh > /var/log/hadoop/startup_log
    case "$?" in
      0)
        echo SUCCESS
        RETVAL=0
        ;;
      1)
        echo TIMEOUT - check /var/log/hadoop/startup_log
        RETVAL=1
        ;;
      *)
        echo FAILED - check /var/log/hadoop/startup_log
        RETVAL=1
        ;;
    esac
    set -e
}

stop_hadoop () {
    set +e
    if [ $RETVAL = 0 ] ; then
        su $USER -s /bin/sh -c $HADOOP_BIN/stop-all.sh > /var/log/hadoop/shutdown_log
        RETVAL=$?
        if [ $RETVAL != 0 ] ; then
            echo FAILED - check /var/log/hadoop/shutdown_log
        fi
    else
        echo No nodes running
        RETVAL=0
    fi
    set -e
}

restart_hadoop() {
    stop_hadoop
    start_hadoop
}

case "$1" in
    start)
        echo -n "Starting $DESC: "
        start_hadoop
        echo "$NAME."
        ;;
    stop)
        echo -n "Stopping $DESC: "
        stop_hadoop
        echo "$NAME."
        ;;
    force-reload|restart)
        echo -n "Restarting $DESC: "
        restart_hadoop
        echo "$NAME."
        ;;
    *)
        echo "Usage: $0 {start|stop|restart|force-reload}" >&2
        RETVAL=1
        ;;
esac
exit $RETVAL

/etc/init.d/hbase

Create the place it will put its startup logs

mkdir /var/log/hbase

Create /etc/init.d/hbase with the following:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          hbase services
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Description:       Hbase services
# Short-Description: Enable Hbase services including hdfs
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
HBASE_BIN=/usr/local/hbase/bin
NAME=hbase
DESC=hbase
USER=hadoop
ROTATE_SUFFIX=
test -x $HBASE_BIN || exit 0
RETVAL=0
set -e
cd /

start_hbase () {
    set +e
    su $USER -s /bin/sh -c $HBASE_BIN/start-hbase.sh > /var/log/hbase/startup_log
    case "$?" in
      0)
        echo SUCCESS
        RETVAL=0
        ;;
      1)
        echo TIMEOUT - check /var/log/hbase/startup_log
        RETVAL=1
        ;;
      *)
        echo FAILED - check /var/log/hbase/startup_log
        RETVAL=1
        ;;
    esac
    set -e
}

stop_hbase () {
    set +e
    if [ $RETVAL = 0 ] ; then
        su $USER -s /bin/sh -c $HBASE_BIN/stop-hbase.sh > /var/log/hbase/shutdown_log
        RETVAL=$?
        if [ $RETVAL != 0 ] ; then
            echo FAILED - check /var/log/hbase/shutdown_log
        fi
    else
        echo No nodes running
        RETVAL=0
    fi
    set -e
}

restart_hbase() {
    stop_hbase
    start_hbase
}

case "$1" in
    start)
        echo -n "Starting $DESC: "
        start_hbase
        echo "$NAME."
        ;;
    stop)
        echo -n "Stopping $DESC: "
        stop_hbase
        echo "$NAME."
        ;;
    force-reload|restart)
        echo -n "Restarting $DESC: "
        restart_hbase
        echo "$NAME."
        ;;
    *)
        echo "Usage: $0 {start|stop|restart|force-reload}" >&2
        RETVAL=1
        ;;
esac
exit $RETVAL

Set up the init system

This assumes you put the above init files in /etc/init.d

chmod +x /etc/init.d/{hbase,hadoop}
update-rc.d hadoop defaults
update-rc.d hbase defaults 25

You can now start / stop hadoop by saying:

/etc/init.d/hadoop start

/etc/init.d/hadoop stop

And similarly with hbase

/etc/init.d/hbase start

/etc/init.d/hbase stop

Make sure you start hadoop before hbase and stop hbase before you stop hadoop

Macintosh launchd style startup

Starting proceses on Macintosh Leopard is pretty easy with lauchd/launchctl.

For hadoop, create a file /Library/LaunchAgents/com.yourdomain.hadoop.plist with the following content (replace yourdomain with the domain you want to use for this class of apps):





    GroupName
    _hadoop
    KeepAlive
    
    Label
    com.yourdomain.hadoop
    ProgramArguments
    
        /usr/local/hadoop/bin/start-all.sh
    
    RunAtLoad
    
    ServiceDescription
    Hadoop Process
    UserName
    _hadoop

And for hbase, /Library/LaunchAgents/com.yourdomain.hbase.plist:





	GroupName
	_hadoop
	KeepAlive
	
	Label
	com.ibd.hbase
	ProgramArguments
	
		/usr/local/hbase/bin/start-hbase.sh
	
	RunAtLoad
	
	UserName
	_hadoop

Set the owner to root and the mode to 644:

chown root /Library/LaunchAgents/com.yourdomain.hadoop.plist /Library/LaunchAgents/com.yourdomain.hbase.plist
chmod 644 /Library/LaunchAgents/com.yourdomain.hadoop.plist /Library/LaunchAgents/com.yourdomain.hbase.plist

The next time you restart, it should start hbase and hadoop. You can also start them manually with the commands:

sudo launchctl load /Library/LaunchAgents/com.yourdomain.hadoop.plist
sudo launchctl load /Library/LaunchAgents/com.yourdomain.hbase.plist

Conclusion

You should now be able to see the HBase web interface at http://:60010

If you have problems check /var/log/{hbase,hadoop}/startup_log as well as /usr/local/hadoop/logs/hadoop-hadoop-namenode-yourhostname.log and /usr/local/hbase/logs/hbase-hadoop-master-yourhostname.log

The error messages are pretty poor. (Ie useless as far as I could tell when tracking down the FQDN/IP Address problem). But better than nothing.

I will post an update when I deploy a Full Cluster.

The post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard first appeared on Cognizant Transmutation.

Deploying RabbitMQ and Stomp on Ubuntu

Robert J Berger — Fri, 02 Jan 2009 10:33:31 +0000

Install rabbitmq via synaptic

Make sure that the erlang package is installed

Add a repository from the rabbitmq site

Set up Repository via the Synaptic GUI tool

(http://www.rabbitmq.com/debian/)

Set up Repository via command line

Ubuntu Documentation for Managing Repositories via the Command Line

How to use the RabbitMQ Debian repository and available RabbitMQ Debian packages

The repositories are described in /etc/apt/sources.list

So do the following:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup

Edit /etc/apt/sources.list and add the following line:

deb http://www.rabbitmq.com/debian/ testing main

Then update the apt-get environment:

wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
sudo apt-get update

Install the RabbitMQ Server

Set up Repository via the Synaptic GUI tool

Set up Repository via command line

sudo apt-get install rabbitmq-server

This should have installed the main portion of the code base in /usr/lib/erlang/lib/rabbitmq_server-1.5.1 (The trailing version number may be different than 1.5.1)

After it installed the server, make sure its stopped

# /etc/init.d/rabbitmq-server stop

Install rabbitmq-stomp

I could not find any ubuntu/debian packages so I installed it from the Mercurial repository. If you don’t already have Mercurial (the hg command) then you can install it with the following command:

apt-get install mercurial

Install the rabbitmq-stomp code

This also will go parallel to where the ubuntu package put the stomp server main code and the rabbit-codegen.

cd /usr/lib/erlang/lib/
hg clone http://hg.rabbitmq.com/rabbitmq-stomp/

Compile the stomp code

Build and test run rabbitmq and stomp via make

cd /usr/lib/erlang/lib/rabbitmq-stomp
make RABBIT_SERVER_SOURCE_ROOT=../rabbitmq_server-1.5.1 all

This should produce an output like:

mkdir -p ebin
erlc -I ../rabbitmq_server-1.5.0/include -I include -o ebin -Wall +debug_info  src/rabbit_stomp.erl
erlc -I ../rabbitmq_server-1.5.0/include -I include -o ebin -Wall +debug_info  src/stomp_frame.erl

Add a file /etc/default/rabbitmq and Restart rabbitmq_server

You need to tell the main rabbitmq_server to load and run the rabbitmq-stomp stuff when it starts up. You do that by creating this file with the following content:

SERVER_START_ARGS='
  -pa /usr//lib/erlang/lib/rabbitmq-stomp/ebin
  -rabbit
     stomp_listeners [{"0.0.0.0",61613}]
     extra_startup_steps [{"STOMP-listeners",rabbit_stomp,kickstart,[]}]'

Restart the Rabbitmq_server:

/etc/init.d/rabbitmq_server start

You can do a

ps -ax | grep stomp

and see an erlang process that is running the rabbit-stomp process.

Install ruby stomp client code and test

Install the ruby stomp gems

If you don’t have ruby already installed:

sudo apt-get install ruby
sudo apt-get install rubygems

Then install the ruby stomp gem

sudo gem install stomp

Run the ruby receiver client in one window

ruby /usr/lib/erlang/lib/rabbit-stomp/examples/ruby/cb-receiver.rb

In another window run the ruby sender client

ruby /usr/lib/erlang/lib/rabbit-stomp/examples/ruby/cb-sender.rb

In the receiver window you should see 10,000 test message lines:

...
Test Message number 9998
Test Message number 9999
All Done!

That’s it! Now you can use Stomp

(See later post Updating RabbitMQ and RabbitMQ-Stomp to RabbitMQ 1.5.3)

The post Deploying RabbitMQ and Stomp on Ubuntu first appeared on Cognizant Transmutation.