AWS - Cognizant Transmutation

Accessing AppSync APIs that require Cognito Login outside of Amplify

Robert J Berger — Wed, 01 Sep 2021 00:36:18 +0000

The Need

You have this great Amplify App using AppSync GraphQL. You eventually find that you need to be able to access that data in your AppSync GraphQL database from tools other than your Amplify App. Its easy if you just have your AppSync API protected just by an API Key. But that isn’t great security for your data!

One way to protect your AppSync data is to use Cognito Identity Pools. Amplify makes it pretty transparent if you are using Amplify to build your clients. AppSync lets you do really nice table and record level access control based on logins and roles.

What happens if you want to access that data from something other than an Amplify based client? How do you “login” and get the JWT credentials you need to access your AppSync APIs?

Use AWS CLI

The most general way is to use the AWS CLI to effectively login and retrieve the JWT credentials that can then be passed in the headers of any requests you make to your AppSync APIs.

Unfortunately its not as easy as just having your login and password. It also depends on how you configured your Cognito Identity Pool and its related Client Apps.

Cognito User Pool Client App

You can have multiple Client Apps specified for your Cognito User Pool. I suggest having one dedicated to these external applications. That way you can have custom configuration just for this and not disrupt your main Amplify apps. Also you can easily turn it off if you need too.

In my case I created a new client app shoppabdbe800b-rob-test2 as a way to test a client app with no App Client Secret. This makes it easier to access from the command line as you do not have to generate a Secret Hash (will describe how to deal with that below).

If you want to allow admin level access (ie a user with admin permission) you need to check Enable username password auth for admin APIs for authentication (ALLOW_ADMIN_USER_PASSWORD_AUTH)

If you want to allow regular users to login you must also select Enable username password based authentication (ALLOW_USER_PASSWORD_AUTH)

The defaults for the other fields should be ok. Be sure to save your changes.

Minimal IAM permissions

As far as I can tell, these are the minimal IAM permissions to make the aws cognito-idp command work for admin and regular users of AppSync (replace the Resource arn with the arn of the user pool[s] you want to control):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cognito-idp:AdminInitiateAuth",
                "cognito-idp:AdminGetUser"
            ],
            "Resource": "arn:aws:cognito-idp:us-east-1:XXXXXXXXXXXXX:userpool/us-east-1_XXXXXXXXX"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "cognito-idp:GetUser",
                "cognito-idp:InitiateAuth"
            ],
            "Resource": "*"
        }
    ]
}

Get the Credentials with no App Client Secret

This example is if you did not set the App Client Secret.

You should now be able to get the JWT credentials from the AWS CLI.

This assumes you have set up your ~/.aws/credentials file or whatever is appropriate for your command line environment so that you have the permissions to access this service.

When using the ADMIN_USER_PASSWORD_AUTH

aws cognito-idp admin-initiate-auth --user-pool-id us-east-1_XXXXXXXXXX --auth-flow ADMIN_USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=username1,PASSWORD=XXXXXXXXXXXXX > creds.json

When using the USER_PASSWORD_AUTH

aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=username2,PASSWORD=XXXXXXXXXXXX > creds.json

Of course replace the XXXX‘s with the actual values.

user-pool-id – The pool id found at the top of the User Pool Client Apps page
client-id – The client-id of the app client you are using
USERNAME – The Username normally used to login to your Amplify app
PASSWORD – The Password normally used to login to your Amplify app

The results will be in creds.json. (You could not use the > creds.json if you want to just see the results)

Get the Credentials when there is an App Client Secret

This assumes you have an App Client that has an app secret key set.

The main thing here is you need to generate a secret hash to send along with the command.

You can do that by creating a little python program to generate it for you when you need it:

#!/usr/bin/env python3

import sys
import hmac, hashlib, base64

if (len(sys.argv) == 4):
    username = sys.argv[1]
    app_client_id = sys.argv[2]
    key = sys.argv[3]
    message = bytes(sys.argv[1]+sys.argv[2],'utf-8')
    key = bytes(sys.argv[3],'utf-8')
    secret_hash = base64.b64encode(hmac.new(key, message, digestmod=hashlib.sha256).digest()).decode()

    print("SECRET HASH:",secret_hash)
else:
    (print("len sys.argv: ", len(sys.argv)))
    print("usage: ",  sys.argv[0], "   ")

Save the file someplace that you can execute it from like ~/bin/app-client-secret-hash and make it executable (chmod a+x ~/bin/app-client-secret-hash).

You will need:

app-client-id – The client-id of the app client you are using
app-client-secret – The secret of the app client you are using (its on the App Client page of the User Pool)
USERNAME – The Username normally used to login to your Amplify app

To use:

~/bin/app-client-secret-hash

Where of course you replace the arguments with the actual values.

The result is a secret-hash you will use in the following command to get the actual JWT credentials

aws cognito-idp admin-initiate-auth --user-pool-id us-east-1_XXXXXXXXXX --auth-flow ADMIN_USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=username3,PASSWORD='secret password',SECRET_HASH='secret-hash' > creds.json

You could do the same thing with USER_PASSWORD_AUTH if you nee that instead

aws cognito-idp initiate-auth --auth-flow USER_PASSWORD_AUTH --client-id XXXXXXXXXXXXX --auth-parameters USERNAME=rob+admin,PASSWORD=XXXXXXXXX,SECRET_HASH='secret-hash' > creds.json

Using the Credentials

How you use these credentials depends on what tool or how you are trying to access your AppSync APIs.

From some Javascript

You can just add in the IdToken from the creds.json as an Authorization header when you build the request:

function graphQLFetcher(graphQLParams) {
  const APPSYNC_API_URL = "TYPE_YOUR_APPSYNC_URL";
  const credentialsAppSync = {
    Authorization: "eyJraWQiOiI1dVUwMld...",
  };
  return fetch(APPSYNC_API_URL, {
    method: "post",
    headers: {
      Accept: "application/json",
      "Content-Type": "application/json",
      ...credentialsAppSync,
    },
    body: JSON.stringify(graphQLParams),
    credentials: "omit",
  }).then(function (response) {
    return response.json().catch(function () {
      return response.text();
    });
  });
}

If you are using some GraphQL tool that needs to access your AppSync APIs. The tool should have a way that you can supply the token and it will add it as an Authorization header for its own requests.

Do let me know if you have some examples of tools that would make use of this.

References

Explore AWS AppSync APIs with GraphiQL from your local machine
[How do I troubleshoot “Unable to verify secret hash for client ” errors from my Amazon Cognito user pools API?](https://aws.amazon.com/premiumsupport/knowledge-center/cognito-unable-to-verify-secret-hash/ “How do I troubleshoot “Unable to verify secret hash for client ” errors from my Amazon Cognito user pools API?”)

The post Accessing AppSync APIs that require Cognito Login outside of Amplify first appeared on Cognizant Transmutation.

CLI to Switch Amazon AWS Shell Environment Credentials

Robert J Berger — Mon, 16 Jun 2014 04:54:11 +0000

I work with many different AWS IAM Accounts and need to easily switch between these accounts. The good news is the AWS CLI tools now support a standard config file (~/.aws/config) that allows you to create profiles for multiple accounts in the one config file. You can select them when using the aws-cli with the --profile flag.

But many other tools don’t yet support the new format config file or multi-profiles. But they do support shell environment variables. So I wrote a simple ruby script that

Allows you to specify the profile name as an argument
Reads in the config file ~/.aws/config
Outputs the export statements for publishing the environment variables
- You can eval the output to set the environment of your current shell session

So if you had a config file ~/.aws/config that looked like this:

[default]
aws_access_key_id=AKI***********2A
aws_secret_access_key=jt41************************************p
region=us-east-1

[profile foo]
aws_access_key_id=0K***************K82
aws_secret_access_key=2b+***********************************1g
region=us-east-1

[profile bar]
aws_access_key_id=AKI**************GA
aws_secret_access_key=MG************************************/d
region=us-east-1

If you don’t specify any argument to the command it will output the default profile:
 $ aws_switch
export AWS_ACCESS_KEY_ID=AKI***********2A
export AWS_SECRET_ACCESS_KEY=jt41************************************p
export AMAZON_ACCESS_KEY_ID=AKI***********2A
export AMAZON_SECRET_ACCESS_KEY=jt41************************************p
export AWS_ACCESS_KEY=AKI***********2A
export AWS_SECRET_KEY=jt41************************************p

If you specified a profile (in this case foo):
$ aws_switch foo
export AWS_ACCESS_KEY_ID=0K***************K82
export AWS_SECRET_ACCESS_KEY=2b+***********************************1g
export AMAZON_ACCESS_KEY_ID=0K***************K82
export AMAZON_SECRET_ACCESS_KEY=2b+***********************************1g
export AWS_ACCESS_KEY=0K***************K82
export AWS_SECRET_KEY=2b+***********************************1g

You would actually use it by eval’ing the output of aws_switch so it sets the variables in the environment of yhour current shell:
eval `aws_switch foo`

Here’s the code for aws_switch. Put it in someplace in your $PATH and make sure to chmod 0755 the file so its executable:
#!/usr/bin/env ruby
require 'inifile'

configs = IniFile.load(File.join(File.expand_path('~'), '.aws', 'config'))

profile_name_input = ARGV[0]
case profile_name_input
when 'default'
  profile_name = 'default'
when nil
  profile_name = 'default'
when ""
  profile_name = 'default'
else
  profile_name = "profile #{profile_name_input}"
end

id = configs[profile_name]['aws_access_key_id']
key = configs[profile_name]['aws_secret_access_key']

puts "export AWS_ACCESS_KEY_ID=#{id}"
puts "export AWS_SECRET_ACCESS_KEY=#{key}"
puts "export AMAZON_ACCESS_KEY_ID=#{id}"
puts "export AMAZON_SECRET_ACCESS_KEY=#{key}"
puts "export AWS_ACCESS_KEY=#{id}"
puts "export AWS_SECRET_KEY=#{key}"
The post CLI to Switch Amazon AWS Shell Environment Credentials first appeared on Cognizant Transmutation.

Deploy WordPress to Amazon EC2 Micro Instance with Opscode Chef

Robert J Berger — Mon, 03 Jan 2011 07:08:16 +0000

Updates

September 9, 2011

Included the latest Chef Knife ec2 server create argument that sets the EBS Volume to not be deleted on the termination of the EC2 Instance

Intro

Up until recently a friend lent me a Virtual Machine in he Cloud for my Blog. I didn’t have to do anything to manage it. But his company is no longer supporting those machines so I had to move my blog.
Right around that time Amazon announced their Micro Instances at a very low price. I also wanted to try out the new Opscode Chef knife commands that bootstrap an EC2 instance from scratch as well as their Chef Server SaaS. So this was a good reason to combine all these to create my new Blog Instance. And now Amazon even offers the ability to have a single micro instance free for a year! (You still have to pay for I/O charges but they are really cheap compared to the instance charges, unless you have a blog that is too popular, but then you’ll need a bigger server anyway)
Spoiler Alert: It was way too easy and no problem at all! (Though I did end up having to write a few support cookbooks like vsftpd, but now you don’t have to)

Some Assumptions for this post

You are using a *nix platform for your local development (ie your laptop is a Mac, Linux, *BSD or equivalent) and that your target server you want to deploy to is a relatively recent Ubuntu Linux.
You have or will install git client on your local development box
You followed the directions or have done the equivalent of the instructions in the Opscode How to Get Started pages as noted below

Set up an Account on Amazon Web Services

If you don’t already have an Amazon EC2 Account, go to the Amazon Web Services page and click on the Sign Up Now button. Create all your user info and then Sign Up for Amazon EC2. You’ll need to put in credit card info at this point since you’ll need to pay for the EC2 instance you’ll be using shortly. After you complete your signup, you’ll need to get your credentials at the AWS Security Credentials page. Copy down your Access Key ID and click on Show under the Secret Access Key and get that as well. You will need these values to put into your knife.rb file that you will get to in the following steps.

Get an Opscode Platform Account

Its free and easy. Just go to the Opscode Platform Signup page. Fill in your information and submit. There is no cost for up to 5 client nodes. Once you set up and confirm your account you can go thru the How to Get Started pages which includes how to set up your client development machine (installing Chef Client, Knife and various dependencies) as well as downloading your private key, organization key and your Knife Configuration File. You should go thru all 5 steps of the Getting Started section. And please do follow their examples of using git. The rest of this post assumes you have git installed and will use it for your own repository even if you don’t push it to an upstream git repository.

Once you have completed that you will be ready to use the remaining steps of this blog post. The remaining steps will assume you put your chef-repo in the same location as the Opscode instructions suggested (~/chef-repo). If you put it somewhere else, just adjust your path to your chef-repo as appropriate.

It also assumes you got your private user key (your_user_name.pem) and organization validator key (your_organization-validator.pem) and knife.rb in Section 3 of How to Get Started: Setting Up a Chef Client. In that section you ran the command knife configure client ./client-config inside your ~/chef-repo/ directory. That will have created ~/chef-repo/.chef and put the keys and knife.rb in that directory.

For the use of this blog post, we will use the username: rberger_test and organization name: install_wordpress. So the private user key name for this example will be: rberger_test.pem and the organization validator key will be called install_wordpress-validator.pem. You should copy your keys someplace that you will not loose outside of ~/chef-repo. There are ways to create new ones, but its always easier not to have to. Bottom line, is its expected that your keys and the knife.rb will be in your ~/chef-repo/.chef directory at this point.

Set up your Development Environment

Your development environment is your home or work computer/laptop. Its the machine that is local to you. It is on this machine that you put together your Cookbooks. From here you push your cookbooks to the Opscode Chef Server, issue the commands to configure AWS and launch your AWS instances.

Tweak up your chef-repo

I like to keep the “standard” chef recipes that get downloaded from git or from cookbook.opscode.com in their own directory (called cookbooks) and all the cookbooks I create or highly modify in another directory (site-cookbooks). In Step 2 of the How to Get Started: Setting Up Your User Environment, they had you create a ~/chef-repo directory and populate it from git or from a tar ball. You should add the site-cookbooks directory to your ~/chef-repo. We’re also going to add an empty README.md to the site-cookbooks directory so when we create our own git repository that directory will be there (an empty directory will not be added to a git repository)

cd ~/chef-repo
mkdir site-cookbooks
echo "Directory for customized cookbooks" &gt; site-cookbooks/README.md

You will probably also not want to include your .chef directory with all your keys in what gets uploaded to any outside chef repository. If you are just keeping things local, you can skip this step. Edit ~/chef-repo/.gitignore and add .chef to the file on its own line. You might also want to add client-config to .gitignore as well as any temporary or backup file suffixes you might have. For instance if you use Emacs, you would add ~* (emacs backup files suffix), the .DS_Store which is something left by the Mac filesystem, .rake_test_cache which is left around by Rake and metadata.json which is a file generated by chef. My .gitignore looks like:

.chef
client-config
*~
.DS_Store
.rake_test_cache
metadata.json

If you created the ~chef-repo from the git clone of the Opscode repository, you’ll want to get rid of the git configuration and history from the cloning of the Opscode chef-repo and create your own git repository:

rm -rf .git
git init
git add -A
git commit -a -m "Created my own basic chef-repo"

The above commands will have removed the old git config that came when you did the git clone http://github.com/opscode/chef-repo.git command as part of the Opscode How to Get Started pages. The git init, add and commit will create a new local git repository for your own use not connected to the opscode repository. You can then add a remote repository if you want to be able to push your repository and future changes to another git repository such as github.com.

Updating your knife.rb file with Amazon Credentials

Add the following lines to the end of your ~/chef-repo/.chef/knife.rb file. You should have gotten your AWS Access Key and Secret Access key when you signed up to Amazon Web Services, but you can always go back and get it at AWS Security Credentials page. Your final knife.rb should look something like this, except the various items that are customized to your setup. In the example below rberger_test would be replaced by your Opscode User name and install_wordpress would be replaced by your Opscode Organization name that was used when you went thru the Section 3 of the Opscode How to Get Started: Setting Up a Chef Client.

current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                "rberger_test"
client_key               "#{current_dir}/rberger_test.pem"
validation_client_name   "install_wordpress-validator"
validation_key           "#{current_dir}/install_wordpress-validator.pem"
chef_server_url          "https://api.opscode.com/organizations/install_wordpress"
cache_type               'BasicFile'
cache_options( :path =&gt; "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{current_dir}/../cookbooks",&nbsp;"#{current_dir}/../site-cookbooks"]
knife[:aws_access_key_id]     = "Your Access Key"
knife[:aws_secret_access_key] = "Your Secret Access Key"

You can test that your knife.rb is setup enough to access AWS by issuing the command

knife ec2 server list

And you should see something like this (just the heading and no instances unless you’ve launched some EC2 instances earlier:

Instance ID &nbsp; &nbsp; &nbsp;Public IP &nbsp; &nbsp; &nbsp; &nbsp;Private IP &nbsp; &nbsp; &nbsp; Flavor &nbsp; &nbsp; &nbsp; &nbsp;Image &nbsp; &nbsp; &nbsp; &nbsp;Security Groups &nbsp;State

Get the Appropriate Cookbooks

We’ll get cookbooks using the knife command and the cookbooks.opscode.com web service. We’ll be using the following cookbooks:

chef
apache2
mysql
openssl
php
postfix
sudo
users
vsftpd
wordpress

Use the knife command on your local development machine to pull down the cookbooks you need. The command we’re using (knife cookbook site vendor COOKBOOK) will automatically download the cookbooks and install them in the ~/chef-repo/cookbooks directory. It will also check them into your git repository as a vendor branch (Stay on the master branch at least until you have installed all the cookbooks).

cd ~/chef-repo
knife cookbook site vendor chef -d
knife cookbook site vendor apache2 -d
knife cookbook site vendor mysql -d
knife cookbook site vendor openssl -d
knife cookbook site vendor php -d
knife cookbook site vendor postfix -d
knife cookbook site vendor sudo -d
knife cookbook site vendor users -d
knife cookbook site vendor vsftpd -d
knife cookbook site vendor wordpress -d

Those commands will download all the cookbooks and any other cookbook dependencies they may have into your ~/chef-repo/cookbooks directory and check each one in as a git branch in your repo. If you do an ls on your ~/chef-repo/cookbooks directory you should see:

README.md &nbsp; &nbsp; &nbsp; bluepill &nbsp; &nbsp; &nbsp; &nbsp;couchdb &nbsp; &nbsp; &nbsp; &nbsp; java &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;php &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; runit &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; users &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; xml
apache2 &nbsp; &nbsp; &nbsp; &nbsp; build-essential daemontools &nbsp; &nbsp; mysql &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; postfix &nbsp; &nbsp; &nbsp; &nbsp; sudo &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;vsftpd &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;zlib
apt &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; chef &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;erlang &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;openssl &nbsp; &nbsp; &nbsp; &nbsp; rabbitmq_chef &nbsp; ucspi-tcp &nbsp; &nbsp; &nbsp; wordpress

And if you do a git branch you should see your master branch as the current and a chef-vendor- for each of the cookbooks you installed:

  chef-vendor-apache2
  chef-vendor-apt
  chef-vendor-bluepill
  chef-vendor-build-essential
  chef-vendor-chef
  chef-vendor-couchdb
  chef-vendor-daemontools
  chef-vendor-erlang
  chef-vendor-java
  chef-vendor-mysql
  chef-vendor-openssl
  chef-vendor-php
  chef-vendor-postfix
  chef-vendor-rabbitmq_chef
  chef-vendor-runit
  chef-vendor-sudo
  chef-vendor-ucspi-tcp
  chef-vendor-users
  chef-vendor-vsftpd
  chef-vendor-wordpress
  chef-vendor-xml
  chef-vendor-zlib
* master

If you ever want to update these standard cookbooks, you can just redo the knife cookbook site vendor Cookbook command.

Create site-cookbooks to extend standard cookbooks

It is standard practice to put the official cookbooks in the ~chef-repo/cookbooks directory, as we just did in the previous step. Any cookbook overrides, extensions or custom cookbooks go into the ~chef-repo/site-cookbooks directory. If you create a cookbook directory in ~chef-repo/site-cookbooks with the same name as a cookbook in the ~chef-repo/cookbooks directory, the files, templates and/or recipes in the ~chef-repo/site-cookbook directory will override the matching files, templates and/or recipes in the cookbook of the same name in the ~chef-repo/cookbooks directory. We will now extend two of the cookbooks; users and wordpress.

Extend the Sudo cookbook so its suitable for EC2

The standard sudo cookbook creates a sudoers file that requires passwords to activate sudo. Most EC2 environments do not allow passwords for logins and require that you login only with ssh keys. So we need to modify the Sudo cookbook to create the sudoers file with the NOPASSWORD flag set for all the users we want to have sudo powers. We just need to override the template file used in the standard sudo cookbook.

First have to make a directory for the new template in your site-cookbooks directory:

mkdir -p site-cookbooks/sudo/templates/default

Copy the following into site-cookbooks/sudo/templates/default/sudoers.erb:

#
# /etc/sudoers
#
# Generated by Chef for
#

Defaults !lecture,tty_tickets,!fqdn

# User privilege specification
root  ALL=(ALL) ALL

 ALL=(ALL) NOPASSWD:ALL

# Members of the sysadmin group may gain root privileges
%sysadmin ALL=(ALL) NOPASSWD:ALL

# Members of the group '' may gain root privileges
% ALL=(ALL) NOPASSWD:ALL

Fix a bug in the latest version of the Standard Mysql Cookbook

As I was writing this post, Opscode came out with a new version of the Mysql Cookbook that seems to have a bug with the Chef Client version 0.9.12. It may be fixed by the time you read this. If you are running Chef 0.9.12, check for line 59 of cookbooks/mysql/recipes/client.rb. Change

if platform_version.to_f &gt;= 5.0

to:

if node.platform_version.to_f &gt;= 5.0

Extend the WordPress cookbook to do some custom actions

We need to do a few custom actions after we install wordpress. The main one being to change the onwnership of the wordpress directory and most of the files to the user blog.

We need to add a user named blog that has its home directory the same as the wordpress directory. We will use this blog user to do automatic updates to wordpress. It will use vsftpd for secure ftp and will have only access to the wordpress directory.

We also need to add a swap file to the server. We could create a new cookbook to hold this as its not really wordpress related, but because this is such a simple system, we will just add a new recipe to wordpress to handle these miscellaneous actions.

Create a recipe to add the blog user and change ownership of the wordpress directory

First make the directories in site-cookbooks for extending the wordpress cookbook:

mkdir -p site-cookbooks/wordpress/recipes
mkdir -p site-cookbooks/wordpress/attributes
mkdir -p site-cookbooks/wordpress/templates/default

Create and edit the file site-cookbooks/wordpress/attributes/wordpress.rb and put the following in it (note, this must have a different name than the one used in the standard wordpress cookbook templates directory):

default[:wordpress][:blog_updater][:username] = "blog"

::Chef::Node.send(:include, Opscode::OpenSSL::Password)

default[:wordpress][:blog_updater][:password] = secure_password
# hash set by recipe or manually using makepasswd
default[:wordpress][:blog_updater][:hash] = nil

# For creating the swap partition. Swap_size is in GB
default[:wordpress][:gb_swap_size] = 2
default[:wordpress][:swap_file] = "/swap_file"

This will set the [:wordpress][:blog_updater] to be the name “blog”. This is the default for the username that will have the ability to use vsftpd to update wordpress and its plugins. We actually override this in the wordpress.rb role file. But we put a default here as well for good practice (ie. the cookbook will work even if someone doesn’t override the value in a role).

The ::Chef::Node.send(:include, Opscode::OpenSSL::Password) line is there so we can use the Chef mechanism to create an auto-generated password (secure_password). We then use that mechanism to set the default password for the blog_updater.

Create and edit site-cookbooks/wordpress/recipes/blog_user.rb. put the following as the contents:

# Get the password cryptographic hash for node[:wordpress][:blog_updater][:password
package "makepasswd"
package "libshadow-ruby1.8"
if node[:wordpress][:blog_updater][:hash].nil? || node[:wordpress][:blog_updater][:hash].empty?
  cmd = "echo #{node[:wordpress][:blog_updater][:password]} | /usr/bin/makepasswd --clearfrom=- --crypt-md5 |awk '{ print $2 }'"
  ruby_block "create_blog_updater_pw" do
    block do
      node.set[:wordpress][:blog_updater][:hash] = `#{cmd}`.chomp
    end
    action :create
  end
end

# Create the blog_updater user with their home directory being the wordpress directory and the group as the same group as the Apache runtime group
user "#{node[:wordpress][:blog_updater][:username]}" do
  home "#{node[:wordpress][:dir]}"
  gid "#{node[:apache][:user]}"
  shell "/bin/bash"
  supports :manage_home =&gt; true
  unless node[:wordpress][:blog_updater][:hash].nil? || node[:wordpress][:blog_updater][:hash].empty?
    password "#{node[:wordpress][:blog_updater][:hash]}"
  end
end

# Change the ownership of the wordpress directory so that the blog user can update
execute "chown wordpress home for blog user" do
  cwd "#{node[:wordpress][:dir]}"
  user "root"
  command "chown -R #{node[:wordpress][:blog_updater][:username]}:#{node[:apache][:user]} #{node[:wordpress][:dir]}"
  not_if { node[:wordpress][:dir].nil? || node[:wordpress][:dir].empty? || (not File.exists?(node[:wordpress][:dir])) }
end

The above code will create the blog_user as a Linux user on the target system and set its home directory to be the wordpress directory. This is to make it work with vsftpd.

Create a template to override the default wordpress apache config

The standard WordPress cookbook sets the Apache Server Name the FQDN of the EC2 Public DNS and sets the Server Aliases to the EC2 FQDN Private DNS name. This is pretty useless. We would like to have the cookbook set the Server Alias to the FQDN’s based on our own DNS names. To do this without overriding the whole standard WordPress cookbook, we can override one template and name it: site-cookbooks/wordpress/templates/default/wordpress.conf.erb.


  ServerName
  ServerAlias
  DocumentRoot

  &gt;
    Options FollowSymLinks
    AllowOverride FileInfo
    Order allow,deny
    Allow from all

    Options FollowSymLinks
    AllowOverride None

  LogLevel info
  ErrorLog /-error.log
  CustomLog /-access.log combined

  RewriteEngine On
  RewriteLog /-rewrite.log
  RewriteLogLevel 0

The key changes are the ServerAlias line where we now add the @node[:wordpress][:server_aliases] will add any aliases specified by this attribute which we set in the wordpress.rb role file. We also change the AllowOverride to FileInfo for the docroot

Create a recipe to add a swap file to the server

The t1.micro instance only has 612MB of RAM. You can easily run out of that with a WordPress blog. So we have a recipe to add a swap file system utilizing some space the EBS disk Volume. This recipe creates a 2GB file called /swap_file using dd and then uses the mkswap and swapon commands to make that file into a swap partition. The recipe also updates the /etc/fstab file so that the swap file will be mounted again if the instance reboots.

Create and edit the file site-cookbooks/wordpress/recipes/add_swap.rb with the following content:

mb_block_size = 100
count = (node[:wordpress][:gb_swap_size] * 1024) / mb_block_size
bash "add_swap" do
  user "root"
  code &lt; "#{node[:wordpress][:swap_file]}"
  )
end

Create and edit the file site-cookbooks/wordpress/templates/default/fstab.erb and put the following content:

# /etc/fstab: static file system information.
#
proc                   /proc           proc   nodev,noexec,nosuid     0       0
      none            swap   sw                      0       0
/dev/sda1              /               ext3   defaults                0       0
/dev/sda2              /mnt            auto   defaults,nobootwait,comment=cloudconfig 0       0

Create WordPress Role

This example will use a single role named wordpress. Use your favorite editor to create a file in your repo with the path roles/wordpress.rb with the following contents (Substitute your domain for ibd.com and change the hostnames such as test and wordpress-test to names appropriate for your blog. Replace rberger_test with the userid you want to use to log into your server via ssh):

name "wordpress"
description "Blog using wordpress"
recipes "apt", "build-essential", "chef::client_service", "users::sysadmins",
        "sudo", "postfix", "mysql::server", "wordpress", "wordpress::blog_user",
        "wordpress::add_swap", "vsftpd"

override_attributes(
  "postfix" =&gt; {"myhostname" =&gt; "test.ibd.com", "mydomain" =&gt; "ibd.com"},
  "authorization" =&gt; {
    "sudo" =&gt; {
      "groups" =&gt; [],
      "users" =&gt; ["rberger_test", "ubuntu"]
    }
  },
  "wordpress" =&gt; {
     "server_aliases" =&gt; %w(test.ibd.com wordpress-test.ibd.com),
     "version" =&gt; "3.0.4",
     "checksum" =&gt; "c68588ca831b76ac8342d783b7e3128c9f4f75aad39c43a7f2b33351634b74de",
     "blog_updater" =&gt; {
       "username" =&gt; "blog",
       "password" =&gt; "big-secret"
     }
   },
   "vsftpd" =&gt; {"chroot_users" =&gt; %w(blog)}
)

The recipes line will be used to determine which cookbook/recipes (order is important) should be loaded by Chef when the chef-client is run on your new server.

apt: Configures various APT components on Debian-like systems.
build-essential: Installs C compiler / build tools
chef::client_service: Sets up a Chef client daemon to run periodically
users::sysadmins: Creates users with ssh authorized keys. Requires a databag to be configured with users info
sudo: Installs sudo and configures the /etc/sudoers file
postfix: Installs and configures postfix for outgoing email
mysql::server: Installs & configures packages required for mysql servers
wordpress: Installs and configures WordPress according to the instructions at http://codex.wordpress.org/Installing_WordPress
wordpress::blog_user: Custom add-on recipe to add a user named “blog” to use with vsftpd for automatic wordpress and plugin updates
wordpress::add_swap: Custom add-on recipe to add a swap partition to the instance
vsftpd: Very Basic installation and configuration of vsftpd to support Secure (SSL) SFTP

The override_attributes are used to configure various cookbooks.

postfix – Parameters for the postfix cookbook. Mainly sets the host and domain name to be meaningful
authorization – Configures the sudo cookbook. Tells which users and groups should have sudo capability
wordpress Some of these override values in the base cookbook and others for the site-cookbook version
- server-aliases – Sets aliases for the blog name. Will be used as serveralias names in the apache config.
- version – The version of wordpress to download.
- checksum – The checksum of the tar image of the wordpress download.
- blog_updater– Info needed to create a user that will do auto updates to wordpress via vsftps
  - username – The username of the user
  - password – The password to create for the user
vsftpd – Sets what user should be allowed to access via ftp and have their home directory chroot’d (should be the same as wordpress-blog_updater).

Upload the cookbooks and roles to Opscode Chef Platform

Run the following commands while you are in ~/chef-repo. This will upload the wordpress role and all the cookbooks in your chef-repo to your account on the Opscode Chef Platform:

knife role from file roles/wordpress.rb
knife cookbook upload -a

Create the Users databag

The users cookbook will take info from Opscode Chef Server Data Bag names users. There can be an item for each user that you want to create a login for. The standard Opscode users cookbook expects the users set up in the data bags to be in the group sysadmin and have the ability to sudo and gain root powers.

We’ll need to create an item for each user you would like to have on your system. I suggest you make at least one for yourself. Here is the data bag I used for my setup. I don’t show the ssh key. You’ll have to substitute your own public ssh key for <your public ssh key> that you will use to ssh to the server. Its a requirement that you have an ssh key as described in the next section on the sudo cookbook.

Here is the JSON representation of my user data bag item. Create a directory users in ~/chef-repo/data_bags/users and put the following JSON in the file ~/chef-repo/users/.json (where is the username you want to have on the target system. The id will be the name of the item in the data bag and what will become your username (in this case rberger_test) You will also need to include the public ssh key you want associated with this user. You will need to have created a ssh keypair (private and public) locally using something like ssh-keygen. You don’t really need the openid. You should be able to set that to an empty string (“”):

{
  "id": "rberger_test",
  "comment": "Robert J. Berger",
  "uid": 2001,
  "groups": "sysadmin",
  "shell": "/bin/bash",
  "openid": "rberger_test.myopenid.com",
  "ssh_keys": ""
}

You will need to create the users databag and then upload your version of the user JSON (rberger_test.json in the example) to the Chef server with the following commands:

knife data bag create users
knife data bag from file users data_bags/users/rberger_test.json

With Amazon EC2 instances its best to only allow access without passwords using ssh keys. Since the login is protected by ssh keys, and you don’t have passwords associated with the users, you need to make sure sudo is set up to allow invoking sudo for specific users (sysadmins) without a password. The users cookbook creates such a user based on the users data bag. But the sudo cookbook does not set up sudoers to support not having password. We will modify the sudoers.erb template later. Make sure you don’t deploy without this modification as the default sudo cookbook will make it impossible to sudo on an EC2 instance after its run.

Configure AWS

You can do most of the following by using the a GUI web app such as Amazon’s AWS console, the Firefox plugin ElasticFox other such GUI tools or the command line ec2-api-tools. For now, we’ll show how to do this with the Amazon AWS Console.

Set up Security Group

Add a WordPress group, that enables ssh, http and https. You should open at least http and https to all IP addresses (represented by Source IP: 0.0.0.0/0) You can decide to open up ssh to every IP or just to your own development network or host. In this example we’ll open it up to the world. Note: by default ping (ICMP) is not enabled so you can not ping your instance. You can enable ping by adding a line where it doesn’t matter what is in Connection Method, Protocol is ICMP, From Port and To Port is set to -1 and Source IP is 0.0.0.0/0.

Enter the name and description of the Security Group

Set the Ports that are to be enabled (Select the Connection Method, enter the Source IP, and click Save)

Generate an SSH Key Pair for accessing your instance[s]

You need to use the Amazon Key Pair generator to generate a key that will be used to make initial ssh connections to your new instances after they are created. You can also do this on the AWS Management Console’s EC2 Key Pairs page:

Navigate to the Key Pairs page and click on Create Key Pair

You can name the key pair anything, but you may want to use this key pair to access this and future instances, so you might want to name it something general like aws-east. Here we’re going to name it something more specific: aws-wordpress just for this example.

Enter the name for the key

After the key pair is created, make sure to save the private key that is downloaded automatically

At this point a file named asw-wordpress.pem will have been downloaded by your browser. Make sure not to loose it! Put it into your ~/.ssh directory and chmod it to 0600:

chmod 0600 ~/.ssh/aws-wordpress.pem

The final Key Pairs page on the AWS Management Console should look something like:

Final Key Pair Page

Create the Instance and Bootstrap Chef on the Instance

The Chef Knife command has the ability to launch EC2 (and other cloud) instances. This process automatically installs chef and all its dependencies after the instance is created. If all goes well, it then loads and executes your roles and cookbooks on the instance creating your server.

You can see what options are available to this command:

# knife ec2 server create --help
knife ec2 server create (options)
    -Z, --availability-zone ZONE     The Availability Zone
    -A, --aws-access-key-id KEY      Your AWS Access Key ID
    -K SECRET,                       Your AWS API Secret Access Key
        --aws-secret-access-key
        --user-data USER_DATA_FILE   The EC2 User Data file to provision the instance with
        --bootstrap-version VERSION  The version of Chef to install
    -N, --node-name NAME             The Chef node name for your new node
        --server-url URL             Chef Server URL
    -k, --key KEY                    API Client Key
        --color                      Use colored output
    -c, --config CONFIG              The configuration file to use
        --defaults                   Accept default values for all questions
    -d, --distro DISTRO              Bootstrap a distro using a template
        --ebs-no-delete-on-term      Do not delete EBS volumn on instance termination
        --ebs-size SIZE              The size of the EBS volume in GB, for EBS-backed instances
    -e, --editor EDITOR              Set the editor to use for interactive commands
    -E, --environment ENVIRONMENT    Set the Chef environment
    -f, --flavor FLAVOR              The flavor of server (m1.small, m1.medium, etc)
    -F, --format FORMAT              Which format to use for output
    -i IDENTITY_FILE,                The SSH identity file used for authentication
        --identity-file
    -I, --image IMAGE                The AMI for the server
        --no-color                   Don't use colors in the output
    -n, --no-editor                  Do not open EDITOR, just accept the data as is
        --no-host-key-verify         Disable host key verification
    -u, --user USER                  API Client Username
        --prerelease                 Install the pre-release chef gems
        --print-after                Show the data after a destructive operation
        --region REGION              Your AWS region
    -r, --run-list RUN_LIST          Comma separated list of roles/recipes to apply
    -G, --groups X,Y,Z               The security groups for this server
    -S, --ssh-key KEY                The AWS SSH key id
    -P, --ssh-password PASSWORD      The ssh password
    -x, --ssh-user USERNAME          The ssh username
    -s, --subnet SUBNET-ID           create node in this Virtual Private Cloud Subnet ID (implies VPC mode)
        --template-file TEMPLATE     Full path to location of template to use
    -V, --verbose                    More verbose output. Use twice for max verbosity
    -v, --version                    Show chef version
    -y, --yes                        Say yes to all prompts for confirmation
    -h, --help                       Show this message

The actual command we’ll use is:

knife ec2 server create --run-list 'role[wordpress]' --node-name test-wordpress --flavor t1.micro \
--identity-file ~/.ssh/aws-wordpress.pem --image ami-a2f405cb --groups wordpress \
--ssh-key aws-wordpress --ssh-user ubuntu --ebs-no-delete-on-term

Details of knife command to launch instance

role[wordpress]: The role[s] given to this instance. More than one can be specified by an orderd space separated list of strings: ‘role[role0]’ ‘role[role1]’ …

–node-name test-wordpress: The name of the instance. Used by Chef to name the Node and Client

–flavor t1.micro: The EC2 Instance Type. Here we are using the smallest type. This is the only one that is “free”

–identity-file ~/.ssh/aws-wordpress.pem: The path to the ssh private key that was downloaded earlier from the AWS Management Console. You could potentially not include this if you added the key to your ssh-agent.

–image ami-a2f405cb: The Amazon Machine Image assigned to this instance. It is the image of the root file system for the instance and thus determines what OS and software is booted when the instance is started. In this case it is the Canonical Ubuntu 10.4 32 bit AMI. You can find the latest Ubuntu AMIs for each region at the top of the home page of Eric Hammond’s super helpful site.

–groups wordpress: The Security Group[s] to be assigned to this instance. In this case its “wordpress” Multiple Groups can be assigned as a comma separated list

–ssh-key aws-wordpress: The name of the SSH Key Pair that was downloaded from the AWS Management Console

–ssh-user ubuntu: The user name for ssh access. This AMI uses “ubuntu”. The AMI’s usually are configured to allow only a single user to ssh by default. Different AMI’s use different names such as root or ec2-user.

–ebs-no-delete-on-term: By default, the EBS Volume is deleted when the EC2 instance is terminated. By adding this flag it will instead make it so the EBS volume will continue to exist after the EC2 instance has been terminated. You want this for your final deployed site so that if something goes wrong with the EC2 instance you will still have your EBS volume and can use it to create a new EC2 instance without loosing your data. (That is the topic of another tutorial though!)

Successful launch results

After you fire off the knife ec2 server create command, you’ll see something like:

[WARN] Fog::AWS::EC2#new is deprecated, use Fog::AWS::Compute#new instead (/Library/Ruby/Gems/1.8/gems/chef-0.9.12/lib/chef/knife/ec2_server_create.rb:145:in `run')
Instance ID: i-d10ae5bd
Flavor: t1.micro
Image: ami-a2f405cb
Availability Zone: us-east-1b
Security Groups: wordpress
SSH Key: aws-wordpress

Waiting for server..............
Public DNS Name: ec2-184-73-44-17.compute-1.amazonaws.com
Public IP Address: 184.73.44.17
Private DNS Name: domU-12-31-39-10-60-17.compute-1.internal
Private IP Address: 10.198.99.229

Waiting for sshd...done
INFO: Bootstrapping Chef on&nbsp;ec2-184-73-44-17.compute-1.amazonaws.com

That will be followed by loads of debugging info as the knife command bootstraps chef and its related packages and gems. This can go on for 10 to 20 minutes. Eventually you’ll see something along the lines of:

Instance ID: i-d10ae5bd
Flavor: t1.micro
Image: ami-a2f405cb
Availability Zone: us-east-1b
Security Groups: wordpress
SSH Key: aws-wordpress
Public DNS Name: ec2-184-73-44-17.compute-1.amazonaws.com
Public IP Address: 184.73.44.17
Private DNS Name: domU-12-31-39-10-60-17.compute-1.internal
Private IP Address: 10.198.99.229
Run List: role[wordpress]

You can look just before this block and see if chef finished the running of the wordpress related cookbooks ok. If within a page above the last block you don’t see any errors then all is ok. The last few lines should be something like:

[Mon, 03 Jan 2011 07:23:34 +0000] INFO: Chef Run complete in 10.945359 seconds
[Mon, 03 Jan 2011 07:23:34 +0000] INFO: cleaning the checksum cache
[Mon, 03 Jan 2011 07:23:34 +0000] INFO: Running report handlers
[Mon, 03 Jan 2011 07:23:34 +0000] INFO: Report handlers complete

If there are errors, you’ll have to debug your cookbooks which is beyond the scope of this post.

Now you should be able to log into your instance ether as the ubuntu default user or the user you created in the wordpress role and the Users Databag (rberger_test in this example):

# Using the ubuntu user and an explicit ssh key
ssh -i ~/.ssh/aws-wordpress.pem ubuntu@ec2-184-73-44-17.compute-1.amazonaws.com

# Using the user created by the cookbook and a key that is already on you ssh-agent
ssh rberger_test@ec2-184-73-44-17.compute-1.amazonaws.com

Configure DNS to have preferred FQDNs point to your instance

You can access your site using the Amazon Public DNS name, but that would not be good in general. You probably want to access it via a URL like http://www.myydomain.com. To do this you must configure your DNS to add a CNAME to map your FQDN to the Amazon Public DNS name. How this is done is very specific to your DNS service provider. Bottom line is that you want to do a CNAME not an A record. (I.E. an alias of your FQDN for the Amazon Public DNS name, not an A record that uses the Amazon IP address). There are some issues of using an A record with Amazon. You probably won’t see them for a simple situation such as hosting a single instance. But once you have many instances that need to talk to each other, using the CNAME will make life easier.

Installing your WordPress Blog

At this point you should be able to access your new instance via http. The initial screen will be the WordPress setup dialog. You should be able to access it via http using the Amazon Public DNS name or any CNAME aliases you created and also added in the wordpress.rb role file override attribute for (wordpress =>server_aliases. You should see something like:

WordPress startup installation page

It is possible to move an existing WordPress Blog to this new instance but that is beyond the scope of this post.

Happily Ever After

By default, the chef client runs every 1/2 hour on the instance. If you change any of the cookbooks and push them up to the Opscode Chef Server, those changes will be propogated to the instance the next time the chef-client runs again on the instance.

This is the way to maintain the server. By updating or adding cookbooks, you define the state of the server and the server will converge to that state when the chef-client runs. The inverse is true. If you change something on the server directly and the service you changed is managed by Chef, your direct changes could be reverted by the chef-client the next time it runs.

You shouldn’t need to but you can disable the chef-client from running automatically by running the following command while ssh’d to the instance:

sudo /etc/init.d/chef-client stop

That will be reset (ie automatic chef-client runs will be re-enabled) if you reboot. You can permanently disable the automatic running of chef-client by running the following commands while ssh’d into the instance:

cd /etc/init.d
sudo update-rc.d -f chef-client remove

Using the WordPress Automatic Upgrade Mechanism

At this point you should be able to use your wordpress blog as normal. You should be able to use the automatic update feature of WordPress to update WordPress itself and the plugins. When you are asked to supply the Connection Information, put in:

Hostname: The Public FQDN of the host (ether the EC2 Public DNS Name or one of the DNS CNAMEs you set up)
FTP Username: “blog” (or whatever you set node[:wordpress][:blog_updater][:username] in the wordpress.rb role file)
FTP Password: “big-secret” (or whatever you SHOULD have set node[:wordpress][:blog_updater][:password] in the wordpress.rb role
Connection Type: FTPS (SSL)

For instance for the Plugin Update Page:

Upgrade Plugins Connection Information

That should work and be secure using the vsftpd server that we installed automatically.

Hopefully all will work well for you. I will try to answer questions but can’t guarantee quick response here. A great resource is the Opscode Chef IRC channel irc.freenode.net #chef. And of course the Opscode Chef Wiki and the Opscode Support Site.

Source Code at Github

You can get all the source for this at https://github.com/rberger/ibd-wordpress-repo

The post Deploy WordPress to Amazon EC2 Micro Instance with Opscode Chef first appeared on Cognizant Transmutation.

Modyfying Jets3t S3 GUI tool to work with Walrus (Eucalyptus S3)

Robert J Berger — Fri, 18 Jun 2010 01:37:18 +0000

Jets3t (pronounced “jet-set”) is a free, open-source Java toolkit and application suite for the Amazon Simple Storage Service (Amazon S3) and Amazon CloudFront content delivery network. For some reason almost all the standard tools for accessing S3 will not easily work with the Eucalyptus equivalent to S3 called Walrus. I am use to using the excellent S3Fox add-on for Firefox and wanted some GUI tool that had similar capabilities. I was able to piece together how to get Jets3t to work with Eucalyptus Walrus. This article puts it all together in one place.

The basic build procedure is based on the instructions for downloading and building Jet3t from Source. I got the hints for what to change to make things work with Walrus from the Jets3t Users Forum article eucalyptus walrus. And an almost unrelated article hadoop in eucalyptus private cloud in the Cloudera Support Forum. Search on the page for the section that says my jets3t file has these values.

Prerequisites

These instructions assume you are on Ubuntu (I had 10.4 Lucid) though it should be easily modifiable to work on any platform that supports Java.

You’ll need to install

Mercurial (hg)
Sun Java 6 (I couldn’t get it to work with openjdk-6)
Ant

You can use the command:

sudo apt-get install mercurial sun-java6-jdk ant1.8

Get the Source of Jets3t with Mercurial

cd to where you want to create the directory that will contain the source. Then use the following command to create a local mercurial repository of the source. Then cd into the repository directory jets3t

hg clone http://bitbucket.org/jmurty/jets3t/
cd jets3t

Edit files to make jets3t work with Walrus

Use your favorite editor (emacs of course to make the following edits.

Edit `LoginCredentialsPanel.java` to not check the length of login credentials

Jets3t enforces the length of the Access Key and Access Secret Key. But some newer versions of Walrus do not fit the assumptions. This edit eliminates the checks.

This should be around line 230 of src/org/jets3t/apps/cockpit/gui/LoginCredentialsPanel.java. Comment out the lines that have errors.add. It should look something like the following after you comment out the two lines with errors.add.

    if (getAWSAccessKey().length() == 20) {
        // Correct length for AWS Access Key
    } else if (getAWSAccessKey().length() == 22) {
        // Correct length for Eucalyptus ID
    } else {
        // errors.add("Access Key must have 20 or 22 characters");
    }

    if (getAWSSecretKey().length() == 40) {
        // Correct length for AWS Access Key
    } else if (getAWSSecretKey().length() == 38) {
        // Correct length for Eucalyptus Secret Key
    } else {
        //  errors.add("Secret Key must have 40 or 38 characters");
    }

Edit `jets3t.properties` to use parameters for accessing Walrus instead of AWS S3

You’ll want to set the following for Walrus access in jets3t/configs/jets3t.properties. The following are the values you need to change. s3service.s3-endpoint should be set to the fully qualified domain name of the host that runs Walrus (you could use an ip address I believe). I had to set s3service.https-only to false since I don’t know what it would take to set up SSL/TLS between the Java environment and the Walrus environment. If you do, let me know!


s3service.https-only=false
s3service.s3-endpoint=your_walrus_host_name
s3service.s3-endpoint-http-port=8773
s3service.s3-endpoint-https-port=8443
s3service.disable-dns-buckets=true
s3service.s3-endpoint-virtual-path=/services/Walrus

Optionally edit `build.properties`

If you want to mark the build version in a way that distinguishes from the standard version and or change the debug level.
I changed the version to version=0.7.4-runa.

Build Jets3t with your changes to work with Walrus

The following use the default target of dist which will create a target tree in the top level directory dist.

ant

If that works (it will say BUILD SUCCESSFUL at the end if it was) then there will be a director dist/jets3t-0.7.4-runa (or whatever you set the version value to in build.properties). You should be able to:

cd dist/jets3t-0.7.4-runa/bin
bash cockpit.sh &

This should start up an application window and a window for you to enter your Eucalyptus credentials. Select the Direct Login tab and enter you Eucalyptus Access Key and Access Secret Key.

After you click ok, you should see your Walrus buckets!

Once it all works you can use the Store Credentials option on the login window to store your credentials on Walrus and use a login/password to access Walrus. But that is optional.

The post Modyfying Jets3t S3 GUI tool to work with Walrus (Eucalyptus S3) first appeared on Cognizant Transmutation.

Copy an EBS AMI image to another Amazon EC2 Region

Robert J Berger — Mon, 15 Mar 2010 08:45:24 +0000

Since I’ve already created an image I liked in the us-west-1 region, I would like to reuse it in other regions. Turns out there is no mechanism within Amazon EC2 to do that. (See How do I launch an Amazon EBS volume from a snapshot across Regions?). I did find one post that talked a bit about how it can be done “out of band”. So I figured I would give that a try instead of doing a full recreation in the new region.

Prepare the Source Instance and Volume

Start an instance in the source region

Here I’ll start an instance in us-west-1a where I have the EBS image I want to copy. In this case I’ll use the image I want to copy, but it could be any image as long as its in the same region as the EBS AMI image that is to be copied. Though we are going to use the instance info to figure out some parameters for creating the new AMI. So if you don’t make the source instance the same AMI as the one you are copying you will need to supply some of the parameters yourself.

You can use a tool like ElasticFox to do the following creating of instances. Here we’ll do it with command line tools.

Set some Shell source variables on host machine

Just to make using these instructions as a cookbook, we’ll have some shell variables that you can set once and then all the instructions will use the variables so you can just cut and paste the instructions into your shell.

src_keypair=id_runa-staging-us-west
src_fullpath_keypair=~/.ssh/runa/id_runa-staging-us-west
src_availability_zone=us-west-1a
src_instance_type=m1.large
src_region=us-west-1
src_origin_ami=ami-1f4e1f5a
src_device=/dev/sdh
src_dir=/src
src_user=ubuntu

Start up the source instance and capture the instanceid

src_instanceid=$(ec2-run-instances \
  --key $src_keypair \
  --availability-zone $src_availability_zone \
  --instance-type $src_instance_type \
  $src_origin_ami \
  --region $src_region  | \
  egrep ^INSTANCE | cut -f2)
echo "src_instanceid=$src_instanceid"

# Wait for the instance to move to the “running” state
while src_public_fqdn=$(ec2-describe-instances --region $src_region "$src_instanceid" | \
  egrep ^INSTANCE | cut -f4) && test -z $src_public_fqdn; do echo -n .; sleep 1; done
echo src_public_fqdn=$src_public_fqdn

This should loop till you see something like:

$ echo src_public_fqdn=$src_public_fqdn
src_public_fqdn=ec2-184-72-2-93.us-west-1.compute.amazonaws.com

Create a volume from the EBS AMI snapshot

Normally when starting an EBS AMI instance, it automatically created a volume from the snapshot associated with the AMI. Here we create the volume from the snapshot ourselves

# Get the volume id
ec2-describe-instances --region $src_region "$src_instanceid" > /tmp/src_instance_info
src_volumeid=$(egrep ^BLOCKDEVICE /tmp/src_instance_info | cut -f3); echo $src_volumeid
# Now get the snapshot id from the volume id
ec2-describe-volumes --region $src_region $src_volumeid | egrep ^VOLUME > /tmp/volume_info
src_snapshotid=$(cut /tmp/volume_info | cut -f2)
echo $src_snapshotid
src_size=$(cut /tmp/volume_info | cut -f2)
echo $src_size
# Create a new volume from the snapshot
src_volumeid=$(ec2-create-volume --region $src_region --snapshot $src_snapshotid -z $src_availability_zone | egrep ^VOLUME | cut -f2)
echo $src_volumeid

Mount the EBS Image of the AMI you want to copy

Now we’ll mount the EBS AMI image as a plain mount on the running source instance. In this case we’re going to use the same image as we launched, but it doesn’t have to be the same image or even the same architecture.

ec2-attach-volume --region $src_region $src_volumeid -i $src_instanceid -d $src_device

You should see something like:

ATTACHMENT	vol-6e7fee06	i-fb0804be	/dev/sdh	attaching	2010-03-14T09:02:58+0000

Prepare the Destination Instance and Volume

Set some Shell destination variables on host machine

You’ll want to tune these to your needs. This example makes the destination size the same as the source. You could make the destination an arbitrary size as long as it fits the source data.

dst_keypair=runa-production-us-east
dst_fullpath_keypair=~/.ssh/runa/id_runa-production-us-east
dst_availability_zone=us-east-1b
dst_instance_type=m1.large
dst_region=us-east-1
dst_origin_ami=ami-7d43ae14
dst_size=$src_size
dst_device=/dev/sdh
dst_dir=/dst
dst_user=ubuntu

Start up the destination instance and capture the dst_instanceid

dst_instanceid=$(ec2-run-instances \
  --key $dst_keypair \
  --availability-zone $dst_availability_zone \
  --instance-type $dst_instance_type \
  $dst_origin_ami \
  --region $dst_region  | \
  egrep ^INSTANCE | cut -f2)
echo "dst_instanceid=$dst_instanceid"

# Wait for the instance to move to the “running” state
while dst_public_fqdn=$(ec2-describe-instances --region $dst_region "$dst_instanceid" | \
  egrep ^INSTANCE | cut -f4) && test -z $dst_public_fqdn; do echo -n .; sleep 1; done
echo dst_public_fqdn=$dst_public_fqdn

This should loop till you see something like:

$ echo dst_public_fqdn=$dst_public_fqdn
dst_public_fqdn=ec2-184-73-71-160.compute-1.amazonaws.com

Create an empty destination volume

dst_volumeid=$(ec2-create-volume --region $dst_region --size $dst_size -z $dst_availability_zone | egrep ^VOLUME | cut -f2)
echo $dst_volumeid

Mount the EBS Image of the AMI you want to copy

ec2-attach-volume --region $dst_region $dst_volumeid -i $dst_instanceid -d $dst_device

You should see something like:

ATTACHMENT	vol-450ed02c	i-65be1f0e	/dev/sdh	attaching	2010-03-14T09:39:20+0000

Copy the data from the Source Volume to the Destination Volume

Copy your credentials to the source machine

We’re going to use rsync to copy from the source to the destination tunneled thru ssh. This eliminates any issues with EC2 security groups. But it does mean you have to copy an ssh private key to the source machine that will then be able to access the destination machine via ssh.

scp -i $src_fullpath_keypair $dst_fullpath_keypair ${src_user}@${src_public_fqdn}:.ssh

Mount the source and destination volumes on their instances

ssh -i $src_fullpath_keypair ${src_user}@${src_public_fqdn} sudo mkdir -p $src_dir
ssh -i $src_fullpath_keypair ${src_user}@${src_public_fqdn} sudo mount $src_device $src_dir
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo mkfs.ext3 -F $dst_device
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo mkdir -p $dst_dir
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo mount $dst_device $dst_dir

Get the FQDN of the Amazon internal address of the destination machine

We’re assuming that the dst instance is the us-east equivalent base AMI of the us-west source base AMI so we can use these kernel and ramdisk to build the new AMI later.

ec2-describe-instances --region $dst_region "$dst_instanceid" > /tmp/dst_instance_info
dst_internal_fqdn=$(egrep ^INSTANCE /tmp/dst_instance_info | cut -f5); echo $dst_internal_fqdn
dst_kernel=$(egrep ^INSTANCE /tmp/dst_instance_info | cut -f13); echo $dst_kernel
dst_ramdisk=$(egrep ^INSTANCE /tmp/dst_instance_info | cut -f14) ;echo $dst_ramdisk

Commands to run on the source machine

You could do the rsync by logging into the source machine and do the following. I tried to do this by using ssh commands, but the fact that the first ssh from source to destination has to be authenticated was a blocker for me. You could log into the source machine and then sudo ssh to the destination machine (you have to do sudo ssh since the rsync has to be run with sudo and the keys are stored separately for the sudo user and the regular user).
I’ll show both ways.
Here’s how you can ssh to the source machine:

ssh -i $src_fullpath_keypair ${src_user}@${src_public_fqdn}

Set up some shell variables on the source machine shell environment

# This is the key you just copied over
dst_fullpath_keypair=~/.ssh/id_runa-production-us-east
# You need to use the Public FQDN of the destination since its cross region
dst_keypair=runa-production-us-east
src_public_fqdn=ec2-184-72-2-93.us-west-1.compute.amazonaws.com
dst_public_fqdn=ec2-184-73-71-160.compute-1.amazonaws.com
dst_user=ubuntu
src_user=ubuntu
src_dir=/src
dst_dir=/dst

Do the rsync

We are using the rsync options

P Keep partial transferred files and Show Progress
H Preserve Hard Links
A Preserve ACLs
X Preserve extended attributes
a Archive mode
z Compress files for transfer

rsync -PHAXaz --rsh "ssh -i /home/${src_user}/.ssh/id_${dst_keypair}" --rsync-path "sudo rsync" ${src_dir}/ ${dst_user}@${dst_public_fqdn}:${dst_dir}/

If you want to do the rsync from your local host

I found that I still had to log into the source instance

ssh -i $src_fullpath_keypair ${src_user}@${src_public_fqdn}

and then on the source instance do:

sudo ssh -i /home/${src_user}/.ssh/id_${dst_keypair} ${dst_user}@${dst_public_fqdn}

and accept “The authenticity of host” for the first time so the destination host is in the known keys of the sudo user
Then back on your local host you can issue the remote command that will run on the source instance and rsync to the destination host:

ssh -i $src_fullpath_keypair ${src_user}@${src_public_fqdn} sudo "rsync -PHAXaz --rsh \"ssh -i /home/${src_user}/.ssh/id_${dst_keypair}\" --rsync-path \"sudo rsync\" ${src_dir}/ ${dst_user}@${dst_public_fqdn}:${dst_dir}/"

Complete the new AMI from your Local Host

The remaining steps will be done back on your local host. This assumes that the shell variables we set up earlier are still there.

Some Cleanup for new Region

Ubuntu has their apt sources tied to the region you are in. So we have to update the apt sources for the new region.
We’ll do this by chrooting to the mount /dst directory and running some commands as if they were being run on an ami with the /dst image. We might as well update things at the same time to the latest packages.

# Allow network access from chroot environment
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo cp /etc/resolv.conf $dst_dir/etc/

# Upgrade the system and install packages
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo -E chroot $dst_dir mount -t proc none /proc
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo -E chroot $dst_dir mount -t devpts none /dev/pts

cat < /tmp/policy-rc.d
#!/bin/sh
exit 101
EOF
scp -i $dst_fullpath_keypair /tmp/policy-rc.d ${dst_user}@${dst_public_fqdn}:/tmp
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo mv /tmp/policy-rc.d $dst_dir/usr/sbin/policy-rc.d

ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} chmod 755 $dst_dir/usr/sbin/policy-rc.d

# This has to be done to set up the Locale & apt sources
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} DEBIAN_FRONTEND=noninteractive sudo -E chroot $dst_dir /usr/bin/ec2-set-defaults

# Update the apt sources
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} DEBIAN_FRONTEND=noninteractive sudo -E chroot $dst_dir apt-get update

# Optionally update the packages
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} DEBIAN_FRONTEND=noninteractive sudo -E chroot $dst_dir apt-get dist-upgrade -y

# Optionally update your gems
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo -E chroot $dst_dir gem update --system
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo -E chroot $dst_dir gem update

Clean up from the building of the image

ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo chroot $dst_dir umount /proc
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo -E chroot $dst_dir umount /dev/pts
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo -E rm -f $dst_dir/usr/sbin/policy-rc.d

There are a few more shell variables we’ll need

I got the kernel and ramdisk from the destination instance since it has the alestic.com us-east-1 equivalent base AMI to the us-west-1 one that we are copying from.

# Some info for creating the name and description
codename=karmic
release=9.10
tag=server

# Make sure you set this as appropriate
# 64bit
arch=x86_64

# You will need to set the aki and ari values base on the actual base AMI you used
# It will be different for different regions.  These are set for x86_64 and us-east-1
ebsopts="--kernel=${dst_kernel} --ramdisk=${dst_ramdisk}"
ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"

now=$(date +%Y%m%d-%H%M)
# Make this specific to what you are making
chef_version="0.8.6"
prefix=runa-chef-${chef_version}-ubuntu-${release}-${codename}-${tag}-${arch}-${now}
description="Runa Chef ${chef_version} Ubuntu $release $codename $tag $arch $now"

Snapshot the Destination Volume and register the new AMI in the destination region

# Unmount the destination filesystem
ssh -i $dst_fullpath_keypair ${dst_user}@${dst_public_fqdn} sudo umount $dst_dir

# Detach the Destination Volume (it may speed up the snapshot)
ec2-detach-volume --region $dst_region "$dst_volumeid"

# Make the snapshot
dst_snapshotid=$(ec2-create-snapshot -region $dst_region -d "$description" $dst_volumeid | cut -f2)

# Wait for snapshot to complete. This can take a while
while ec2-describe-snapshots --region $dst_region "$dst_snapshotid" | grep -q pending
  do echo -n .; sleep 1; done

# Register the Destination Snapshot as a new AMI in the Destination Region
new_ami=$(ec2-register \
  --region $dst_region \
  --architecture $arch \
  --name "$prefix" \
  --description "$description" \
  $ebsopts \
  --snapshot "$dst_snapshotid")
echo $new_ami

Conclusion

You should now have a shiny new ami in your destination region. Use the value of $new_ami to start a new instance in your destination region using your favorite tool or technique.

The post Copy an EBS AMI image to another Amazon EC2 Region first appeared on Cognizant Transmutation.

Simple update and clone an Amazon EC2 EBS Boot image

Robert J Berger — Fri, 05 Mar 2010 07:54:41 +0000

Introduction

Well there is already an update to Chef’s Ohai library. At first I thought, “Oh no, I have to generate another EC2 image”. But then I remember reading that you can update and clone a running EBS boot image.

One of the cool features of using an Amazon EC2 instance that boots from an EBS Snapshot is that its easy to create new boot images from an existing running EC2 instance, assuming that you are running an EC2 instance that is itself bootable from an EBS Image.

Prerequisites

The following expects that you have a recent copy of the Amazon ec2-api-tools on the instance and that you have recent version of the ec2-api-tools on your host development system.

Start up an instance, make changes

Start up an instance you can use as a base, for instance the one we created in Using the Official Opscode 0.8.x Gems to build EC2 AMI Chef Client and Server

Get the name of the instance

First you will need the instance name of your instance you want to copy. You can use Elasticfox or other tool. Or run the following command on the instance:

wget -qO- http://instance-data/latest/meta-data/instance-id

On another host

The rest of the instructions will be run on your host development system (not the system you are copying). This makes it so you don’t have to put your Amazon Certs onto the machine you are cloning (you don’t want those keys to end up on the cloned image)

Create some shell defines

To make the instructions easier make some defines we’ll use in commands. Tune them for your environment.

# This will be the instance id of the running instance you want to clone
instanceid=i-07202042

# Some info for creating the name and description
codename=karmic
release=9.10
tag=server
region=us-west-1
availability_zone=us-west-1a

# Make sure you set this as appropriate
# 64bit
arch=x86_64
arch2=amd64
#32bit
arch=i386
arch2=i386
now=$(date +%Y%m%d-%H%M)

# Make this specific to what you are making
prefix=runa-chef-0.8.4-ubuntu-$release-$codename-$tag-$arch-$now
description="Runa Chef 0.8.4 Ubuntu $release $codename $tag $arch $now"

Get the info about your running instance

Use Elasticfox or your favorite tool or the following command to get the volume id of the instance

ec2-describe-instances --region $region "$instanceid" > /tmp/instance_info
volumeid=$(egrep ^BLOCKDEVICE /tmp/instance_info | cut -f3); echo $volumeid
kernel=$(egrep ^INSTANCE /tmp/instance_info | cut -f13); echo $kernel
ramdisk=$(egrep ^INSTANCE /tmp/instance_info | cut -f14) ;echo $ramdisk

Shutdown the instance

Its not clear if you really need to do this. But when I first tried doing it without shuting down the instance, the snapshots took forever.

Create a new snapshot

snapshotid=$(ec2-create-snapshot -region $region -d "$description" $volumeid | cut -f2)

Register the new image

ec2reg --region $region -s $snapshotid -a $arch --kernel $kernel --ramdisk $ramdisk -d "$description" -n "$prefix"

The result of this command will be the ami image name. After this completes, the image and snapshot can be used to create new instances.

The post Simple update and clone an Amazon EC2 EBS Boot image first appeared on Cognizant Transmutation.

Using the Official Opscode 0.8.x Gems to build EC2 AMI Chef Client and Server

Robert J Berger — Wed, 03 Mar 2010 06:50:57 +0000

Updates

Mar 3, 2010 Added call to script ec2-set-defaults that is normally called on ec2 init that sets the locale and apt sources for EC availability Zone

Introduction

Opscode has officially released 0.8.x of Chef. It is now even more fabulous. I’ve been using the pre-release version for the last couple of months and it is rock steady and very powerful. I’ll be having a post soon on how I used it to deploy a pretty complicated cloud stack with multiple Rails/Mysql/Nginx/Unicorn/Postfix apps for front-ends, and a back end made up of a mix of a Clojure/Swarmiji distributed processing swarm, HBase/Hadoop, Redis, RabbitMQ.

But first, I needed to upgrade my Amazon EC2 AMIs for the officially released Chef 0.8.x. I also wanted to try the EBS Boot image as a basis for the AMI.

This is an update to my earlier post, Creating an Amazon EC2 AMI for Opscode Chef 0.8, but now using the official Opscode 0.8.x Gems instead of building your own Gems. A lot of the content is the same, but you can consider this mostly superceding the older one except where mentioned otherwise. This version will use the EBS Boot AMIs as per Eric Hammond’s Tutorial Building EBS Boot AMIs Using Canonical’s Downloadable EC2 Images. Much of this is blog post is taken from Eric’s blog post but in the context of creating a Chef Client base AMI and a Chef Server. Note that Opscode now has their own AMIs, including ones for Chef 0.8.4, but as of this writing, they do not have AMIs for Amazon us-west.

Setup

Prerequisites

On your host development machine (ie your laptop or whatever machine you are developing from) you should have already installed:

ec2-api-tools and ec2-ami-tools (these assume you have a modern Java run time setup)
chef-0.8.4 or later chef client gem (which implies the entire ruby 1.8.x and rubygems toolchain)

Set some Shell variables on host machine

keypair=id_runa-staging-us-west
fullpath_keypair=~/.ssh/runa/id_runa-staging-us-west
availability_zone=us-west-1a
instance_type=m1.large
region=us-west-1

# Pick one of these two AMIs (Note that it will be different for different Amazon Regions)
# 32bit AMI
origin_ami=ami-fd5100b8
#64bit AMI
origin_ami=ami-ff5100ba

Start up an instance and capture the instanceid

instanceid=$(ec2-run-instances \
  --key $keypair \
  --availability-zone $availability_zone \
  --instance-type $instance_type \
  $origin_ami \
  --region $region  |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"

Wait for the instance to move to the “running” state

while host=$(ec2-describe-instances --region $region "$instanceid" |
  egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host

This should loop till you see something like:

$ echo host=$host
host=ec2-184-72-2-93.us-west-1.compute.amazonaws.com

Upload your certs

This assumes that your Amazon certs are in ~/.ec2

rsync                            \
 --rsh="ssh -i $fullpath_keypair" \
 --rsync-path="sudo rsync"      \
 ~/.ec2/{cert,pk}-*.pem         \
 ubuntu@$host:/mnt/

Connect to the instance

ssh -i $fullpath_keypair ubuntu@$host

Update the Amazon ec2 tools on the instance

export DEBIAN_FRONTEND=noninteractive
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
  sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
sudo apt-get update &&
sudo -E apt-get dist-upgrade -y &&
sudo -E apt-get install -y ec2-api-tools

Set some parameters on instance shell environment

Again this makes it easier to cut and paste the instructions.

codename=karmic
release=9.10
tag=server
region=us-west-1
availability_zone=us-west-1a
if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
  arch2=amd64
  # You will need to set the aki and ari values base on the actual base AMI you used
  # It will be different for different regions.  These are set for us-west-1
  ebsopts="--kernel=aki-7f3c6d3a --ramdisk=ari-cf2e7f8a"
  ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
else
  arch=i386
  arch2=i386
  # You will need to set the aki and ari values base on the actual base AMI you used
  # It will be different for different regions. These are set for us-west-1
  ebsopts="--kernel=aki-773c6d32 --ramdisk=ari-c12e7f84"
  ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
fi

Download and unpack the latest released Ubuntu server image file

This contains the output of vmbuilder as run by Canonical.

imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
image=/mnt/$codename-$tag-uec-$arch2.img
imagedir=/mnt/$codename-$tag-uec-$arch2
wget -O- $imagesource |
  sudo tar xzf - -C /mnt
sudo mkdir -p $imagedir
sudo mount -o loop $image $imagedir

Bring the packages on the instance up to date

# Allow network access from chroot environment
sudo cp /etc/resolv.conf $imagedir/etc/

# Fix what I consider to be a bug in vmbuilder
sudo rm -f $imagedir/etc/hostname

# Add multiverse
sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
$imagedir/etc/ec2-init/templates/sources.list.tmpl

# Add Alestic PPA for runurl package (handy in user-data scripts)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
sudo chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571

# Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
sudo chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873

# Upgrade the system and install packages
sudo chroot $imagedir mount -t proc none /proc
sudo chroot $imagedir mount -t devpts none /dev/pts

cat < /tmp/policy-rc.d
#!/bin/sh
exit 101
EOF
sudo mv /tmp/policy-rc.d $imagedir/usr/sbin/policy-rc.d

chmod 755 $imagedir/usr/sbin/policy-rc.d
DEBIAN_FRONTEND=noninteractive

# It seems this has to be done to set up the Locale & apt sources
sudo -E chroot $imagedir /usr/bin/ec2-set-defaults

# Update the apt sources and packages
sudo chroot $imagedir apt-get update &&
sudo -E chroot $imagedir apt-get dist-upgrade -y &&
sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools

Install Chef Client and other customizations

Install Ruby and needed packages

sudo -E chroot $imagedir apt-get -y install ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb \
build-essential wget ssl-cert git-core rake librspec-ruby libxml-ruby \
thin couchdb zlib1g-dev libxml2-dev emacs23-nox

Install Rubygems

Rubygems will be installed from source since debian/ubuntu try to control rubygems upgrades. If you don’t care you can install it via apt-get install rubygems

cd $imagedir/tmp
wget http://rubyforge.org/frs/download.php/69365/rubygems-1.3.6.tgz
tar zxf rubygems-1.3.6.tgz
cd rubygems-1.3.6
sudo -E chroot $imagedir ruby /tmp/rubygems-1.3.6/setup.rb
cd ..
sudo rm -rf rubygems-1.3.6
sudo -E chroot $imagedir ln -sfv /usr/bin/gem1.8 /usr/bin/gem
sudo -E chroot $imagedir gem sources -a http://gems.opscode.com
sudo -E chroot $imagedir gem sources -a http://gemcutter.org
sudo -E chroot $imagedir gem install chef

Use Opscode Chef Solo Bootstrap to configure the Chef Client

The following will set up all the default paths and directories as well as install and configure runit to start and monitor the chef-client. Originally I shied away from runit, but this time I’m going as Opscode Vanilla as possible and they like runit.

Create the solo.rb file

All of the following files should be done in $imagedir as we are going to have to run this as chroot to $imagedir

Create $imagedir/solo.rb with an editor and put in the following:

file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
recipe_url "http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz"

Create the chef.json file

Create $imagedir/chef.json with the following. (set the server_fqdn to the chef server you are using):

{
  "bootstrap": {
    "chef": {
      "url_type": "http",
      "init_style": "runit",
      "path": "/srv/chef",
      "serve_path": "/srv/chef",
      "server_fqdn": "chef-server-staging.runa.com"
    }
  },
  "run_list": [ "recipe[bootstrap::client]" ]
}

Run the chef-solo command

sudo -E chroot $imagedir chef-solo -c solo.rb -j chef.json \
  -r http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz

I had to run it 3 times before it completed with no errors.
After it does work, clean up the chef-solo stuff:

sudo rm $imagedir/{solo.rb,chef.json}

Update the client config file

The Chef Solo Client bootstrap process creates an /etc/chef/client.rb that is not ideal for Amazon EC2. The following will replace that:

mkdir -p /etc/chef
chown root:root /etc/chef
chmod 755 /etc/chef

Put the following in /etc/chef/client.rb:


# Chef Client Config File
# Automatically grabs configuration from ohai ec2 metadata.

require 'ohai'
require 'json'

o = Ohai::System.new
o.all_plugins
chef_config = JSON.parse(o[:ec2][:userdata])
if chef_config.kind_of?(Array)
  chef_config = chef_config[o[:ec2][:ami_launch_index]]
end

log_level        :info
log_location     STDOUT
node_name        o[:ec2][:instance_id]
chef_server_url  chef_config["chef_server"]

unless File.exists?("/etc/chef/client.pem")
  File.open("/etc/chef/validation.pem", "w", 0600) do |f|
    f.print(chef_config["validation_key"])
  end
end

if chef_config.has_key?("attributes")
  File.open("/etc/chef/client-config.json", "w") do |f|
    f.print(JSON.pretty_generate(chef_config["attributes"]))
  end
  json_attribs "/etc/chef/client-config.json"
end

validation_key "/etc/chef/validation.pem"
validation_client_name chef_config["validation_client_name"]

Mixlib::Log::Formatter.show_time = true

Finish creating the new image

Clean up from the building of the image

sudo chroot $imagedir umount /proc
sudo chroot $imagedir umount /dev/pts
sudo rm -f $imagedir/usr/sbin/policy-rc.d

Copy the image files to a new EBS volume, snapshot and register the snapshot

size=15 # root disk in GB
now=$(date +%Y%m%d-%H%M)
prefix=runa-chef-0.8.4-ubuntu-$release-$codename-$tag-$arch-$now
description="Runa Chef 0.8.4 Ubuntu $release $codename $tag $arch $now"
export EC2_CERT=$(echo /mnt/cert-*.pem)
export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)

volumeid=$(ec2-create-volume --region $region --size $size \
  --availability-zone $availability_zone | cut -f2)

instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)

ec2-attach-volume --region $region --device /dev/sdi --instance "$instanceid" "$volumeid"

while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done

sudo mkfs.ext3 -F /dev/sdi
ebsimage=$imagedir-ebs
sudo mkdir $ebsimage
sudo mount /dev/sdi $ebsimage

sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
sudo umount $ebsimage

ec2-detach-volume --region $region "$volumeid"
snapshotid=$(ec2-create-snapshot --region $region "$volumeid" | cut -f2)

ec2-delete-volume --region $region "$volumeid"

# This takes a while
while ec2-describe-snapshots --region $region "$snapshotid" | grep -q pending
  do echo -n .; sleep 1; done

ec2-register \
  --region $region \
  --architecture $arch \
  --name "$prefix" \
  --description "$description" \
  $ebsopts \
  --snapshot "$snapshotid"

Afterward

That will get you an AMI that you can now use as a chef-client. You can use the directions from the section Creating a Chef Server from your new Image in the previous article: Creating an Amazon EC2 AMI for Opscode Chef 0.8.

The post Using the Official Opscode 0.8.x Gems to build EC2 AMI Chef Client and Server first appeared on Cognizant Transmutation.

Creating an Amazon EC2 AMI for Opscode Chef 0.8 Client and Server

Robert J Berger — Tue, 12 Jan 2010 09:00:21 +0000

Changes Since Original

1/13/10: Fix various minor inaccuracies and improved description on how to set up the chef-server. Also removed nanite as a requirement (its no longer used)
1/17/10: Add the requirement to build and install mixlib-authentication for the chef-client
1/21/10: Added a mkdir for /var/log/chef
1/22/10: Added step to insure that /tmp permissions are set

Introduction

Here’s my experience setting up an Amazon EC2 AMI and Instance for a Chef Server and Client. It is based mostly on Bryan Mclellan (btm)‘s post of Nov 24, 2009 Installing Chef 0.8 alpha on Ubuntu Karmic and his more up to date GIST: chef 0.8 alpha installation. It has a slightly different focus and is a bit stale if you are building your own 0.8 gems from the source.

Instantiate an Amazon EC2 Instance

We’ll start with the Canonical Ubuntu 9.10 Karmic AMI. I always go to Eric Hammond’s site alestic.com to get the pointers to the right AMIs. In this case we’re using a 32bit image for the US-West Region: ami-7d3c6d38 US-East 32bit: ami-1515f67c. You can use the US-West 64bit image ami-7b3c6d3e, US-East 64bit: ami-ab15f6c2

Start the instance from your local dev machine using the command line ec2-api-tools (available as a package or directly from Amazon) or using something like the Firefox Elasticfox and then ssh into the instance so that you can do the following steps on the instance. For the sake of this example, lets say that the Public DNS name for the instance you started is ec2-204-222-170-10.us-west-1.compute.amazonaws.com and the ssh keypair you associated with this new instance is now on your local dec machine in ~/.ssh/gsg-keypair

Prerequisite preparation

The first set of steps need to be done on the instance you just created so login via ssh:

ssh -i ~/.ssh/gsg-keypair ec2-204-222-170-10.us-west-1.compute.amazonaws.com

If on Amazon us-west

There is a bug in the current us-west Canonical AMI where it does not use the us-west apt server. So you have to correct the apt soruces.list:

sed -i.bak '1,$s/us.ec2.archive.ubuntu.com/us-west-1.ec2.archive.ubuntu.com/' \
/etc/apt/sources.list

For all cases

sudo sed -i.bak2 '1,$s/universe/universe multiverse/' /etc/apt/sources.list
sudo apt-get -y update
sudo apt-get -y upgrade
sudo apt-get -y install emacs23 # Of course this is the first package to install!

# Will need these to manipulate ec2 images
sudo apt-get -y install ec2-api-tools ec2-ami-tools

Set up the ruby environment and install rubygems

Install Ruby and needed packages

sudo apt-get -y install -y ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb \
build-essential wget ssl-cert git-core rake librspec-ruby libxml-ruby \
thin couchdb zlib1g-dev libxml2-dev

Install Rubygems

Rubygems will be installed from source since debian/ubuntu try to control rubygems upgrades. If you don’t care you can install it via apt-get install rubygems

cd /tmp
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar zxf rubygems-1.3.5.tgz
cd rubygems-1.3.5
sudo ruby setup.rb
sudo ln -sfv /usr/bin/gem1.8 /usr/bin/gem
sudo gem sources -a http://gems.opscode.com
sudo gem sources -a http://gemcutter.org

Install Pre-requisit Gems

sudo gem install cucumber merb-core jeweler uuidtools \
json libxml-ruby --no-ri --no-rdoc

Building and Installing Chef Related Gems

Until there are final 0.8.x Chef gems, you will have had to build them on your local machine and upload them to this instance. On your dev machine (this example builds things in ~/src, but it could be anywhere appropriate) follow these instructions to build all the gems and install gems you might need to use your local machine. You will use your local dev machine to develop and manage cookbooks and to manage a remote chef-server:

mkdir ~/src
cd ~/src
git clone git://github.com/opscode/chef.git
git clone git://github.com/opscode/ohai.git
git clone git://github.com/opscode/mixlib-log
git clone git://github.com/opscode/mixlib-authentication.git
# Need to get mixlib-log for client & server and
# mixlib-authentication for the client from git till the 1.1.0 update hits
# See http://tickets.opscode.com/browse/CHEF-823
cd mixlib-log
sudo rake install
cd mixlib-authentication
sudo rake install
cd ../ohai
sudo rake install
cd ../chef
rake gem
# Now cd into ~/src/chef/chef to install the chef client/dev gem on your local machine
cd chef
rake install

Upload the gems needed for the client to your instance. From ~/src on your local dev machine do:

scp -i ~/.ssh/gsg-keypair chef/chef/pkg/chef-0.8.0.gem  ohai/pkg/ohai-0.3.7.gem \
mixlib-authentication/pkg/mixlib-authentication-1.1.0.gem \
mixlib-log/pkg/mixlib-log-1.1.0.gem  ec2-204-222-170-10.us-west-1.compute.amazonaws.com:

Set up the Chef Client on the new Instance

Now back in your home directory on the instance ec2-204-222-170-10.us-west-1.compute.amazonaws.com install the gems you just copied over:

sudo gem install mixlib-log-1.1.0.gem ohai-0.3.7.gem
sudo gem install chef-0.8.0.gem

Create the client config file

mkdir /var/log/chef
mkdir /etc/chef
chown root:root /etc/chef
chmod 755 /etc/chef

Put the following in /etc/chef/client.rb:

# Chef Client Config File

require 'ohai'
require 'json'

o = Ohai::System.new
o.all_plugins
chef_config = JSON.parse(o[:ec2][:userdata])
if chef_config.kind_of?(Array)
  chef_config = chef_config[o[:ec2][:ami_launch_index]]
end

log_level        :info
log_location     "/var/log/chef/client.log"
chef_server_url  chef_config["chef_server"]
registration_url chef_config["chef_server"]
openid_url       chef_config["chef_server"]
template_url     chef_config["chef_server"]
remotefile_url   chef_config["chef_server"]
search_url       chef_config["chef_server"]
role_url         chef_config["chef_server"]
client_url       chef_config["chef_server"]

node_name        o[:ec2][:instance_id]

unless File.exists?("/etc/chef/client.pem")
  File.open("/etc/chef/validation.pem", "w") do |f|
    f.print(chef_config["validation_key"])
  end
end

if chef_config.has_key?("attributes")
  File.open("/etc/chef/client-config.json", "w") do |f|
    f.print(JSON.pretty_generate(chef_config["attributes"]))
  end
  json_attribs "/etc/chef/client-config.json"
end

validation_key "/etc/chef/validation.pem"
validation_client_name chef_config["validation_client_name"]

Mixlib::Log::Formatter.show_time = true

Set up the /etc/init.d/chef-client

Copy the example init.d script (You can also use runit instead, but we’re not going to describe that here)

cp /usr/lib/ruby/gems/1.8/gems/chef-0.8.0/distro/debian/etc/init.d/chef-client /etc/init.d
cd /etc/init.d
update-rc.d chef-client defaults

Create an Init script to set /tmp to proper permmissions

It looks like the Canonical Images will not have /tmp with proper permissions if you exclude /tmp from your bundle process. Eric Hammond recommends doing the following.

Create a file /etc/init.d/ec2-mkdir-tmp with the following contents:

#!/bin/sh
#
# ec2-mkdir-tmp Create /tmp if missing (as it's nice to bundle without it).
#
mkdir -p    /tmp
chmod 01777 /tmp

Then set up the /etc/rc dirs to launch this on boot:

chmod a+x /etc/init.d/ec2-mkdir-tmp
ln -s /etc/init.d/ec2-mkdir-tmp /etc/rcS.d/S36ec2-mkdir-tmp

Build the EC2 Image

The always amazingly helpful Eric Hammond has a post, Creating a New Image for EC2 by Rebundling a Running Instance, that describes the basics of how to do this. The following is pretty much a direct synopsis with minimal explanation. See his blog post for more details.

Clean up potential security holes

Remove stuff you don’t want to freeze into your image.

sudo rm -f /root/.*hist* $HOME/.*hist*
sudo rm -f /var/log/*.gz

Copy AWS Certs to Instance

Back on your local development system, copy your Amazon certificates to the instance.


remotehost=
remoteuser=ubuntu
scp -i  \
  /{cert,pk}-*.pem \
  $remoteuser@$remotehost:/tmp

Create the new Image on the Instance

Back on the ec2 instance, you’ll do the following to create the image.

Define where to store the image on S3

This assumes you have an S3 account setup on AWS. You don’t have to have already created the bucket. Set some bash variables that will be used by the commands that follow. You should set the prefix to something that is meaningful. Below is what I used as an example. You’ll want to make it unique to your environment. The Bucket name must be Globally unique across all of Amazon S3.

bucket=runa-west-amis
prefix=runa-ubuntu-9.10-i386-20100101-base

Define your AWS credentials and target processor

export AWS_USER_ID=
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
else
  arch=i386
fi

Bundle the files
This also runs on the current instance and will bundle the everything on the instance file system except for dirs specified with the -e flag into a copy of the image under /mnt:

sudo -E ec2-bundle-vol           \
  -r $arch                       \
  -d /mnt                        \
  -p $prefix                     \
  -u $AWS_USER_ID                \
  -k /tmp/pk-*.pem               \
  -c /tmp/cert-*.pem             \
  -s 10240                       \
  -e /mnt,/tmp,/root/.ssh,/home/ubuntu/.ssh

If you are deploying to US-West-1 AWS Region

Looks like the Amazon ec2 ami tools are not super aware about us-west yet. So you have to do this extra step right now. You’ll have to change the –kernel and –ramdisk to the ones appropriate for your kernel. You can inspect the values used for the AMI you used to boot the original instance. You can do this with ElasticFox or with the command (specify the AMI and region its in thatyou want to check):

ec2-describe-images ami-7d3c6d38   -C /tmp/cert-*.pem -K /tmp/pk-*.pem --region us-west-1

Then execute the following command and specify the right kernel and ramdisk

sudo -E ec2-migrate-manifest        \
  -c /tmp/cert-*.pem             \
  -k /tmp/pk-*.pem               \
  -m /mnt/$prefix.manifest.xml   \
  --access-key $AWS_ACCESS_KEY_ID  \
  --secret-key $AWS_SECRET_ACCESS_KEY \
  --kernel aki-773c6d32          \
  --ramdisk ari-713c6d34         \
  --region us-west-1

Upload the bundle to a bucket on S3:

sudo -E ec2-upload-bundle        \
    -b $bucket                   \
    -m /mnt/$prefix.manifest.xml \
    -a $AWS_ACCESS_KEY_ID        \
    -s $AWS_SECRET_ACCESS_KEY    \
    --location us-west-1

You may be prompted with something like:

You are bundling in one region, but uploading to another. If the kernel or ramdisk associated with this AMI are not in the target region, AMI registration will fail.
You can use the ec2-migrate-manifest tool to update your manifest file with a kernel and ramdisk that exist in the target region.
Are you sure you want to continue? [y/N]

You should enter y return to accept.

Register the AMI

Back on your local development machine:

ec2-register $bucket/$prefix.manifest.xml --region us-west-1

The output of this will be the ami-id of your new instance. You can use this to instantiate your new ami.

You now have a private ami image you can start just like any other image. If you want to make it public

ec2-modify-image-attribute -l -a all

Using the new AMI Image

You can now use this instance as the basis for chef clients and also the basis to create a Chef Server. Use the Amazon EC2 tool, ElasticFox or whatever you favorite tool for managing EC2 instances to make a new instance first to create a Chef Server. Then after that you can create clients and have them load their roles and recipes from the chef server. Once you have a Chef Server, you can use knife ec2 instance command to create user data that includes a run list, credentials and other json that can be passed to the general ec2 tools to build specific instances.

Creating a Chef Server from your new Image

Using an EC2 tool like ec2-tools or elasticfox, create a new instance based on the AMI created earlier. You should use at least a c1.medium as the m1.small is just too painfully wimpy to use. Assume the new instance has the Public DNS name: ec2-204-203-51-20.us-west-1.compute.amazonaws.com
Copy the chef server gems to the new instance from the ~/src directory in your local dev environment to the new instance:

scp -i ~/.ssh/gsg-keypair chef/*/pkg/*.gem \
ec2-204-203-51-20.us-west-1.compute.amazonaws.com:

ssh to the new instance and do the following:

sudo gem install chef-server-0.8.0.gem chef-server-api-0.8.0.gem \
chef-server-webui-0.8.0.gem chef-solr-0.8.0.gem

Set things up to use bootstrap client using chef-solo

We’ll be using the last part of BTM’s GIST, and danielsdeleo (Dan DeLeo)’s bootstrap cookbook and chef-solo to set up this initial server.

mkdir -p /tmp/chef-solo
cd /tmp/chef-solo
git clone git://github.com/danielsdeleo/cookbooks.git
cd cookbooks
git checkout 08boot

Create ~/chef.json:

{
  "bootstrap": {
    "chef": {
      "url_type": "http",
      "init_style": "runit",
      "path": "/srv/chef",
      "serve_path": "/srv/chef",
      "server_fqdn": "localhost"
    }
  },
  "recipes": "bootstrap::server"
}
# End of file

Create ~/solo.rb with the following content:

file_cache_path "/tmp/chef-solo"
cookbook_path "/tmp/chef-solo/cookbooks"
# End of ~/solo.rb file

Run chef-solo which will execute the chef bootstrap recipes using the bootstrap params in ~/chef.json to actually setup and configure this chef server

If you had installed rubygems with the ubuntu apt package you may have to specify the path:

/var/lib/gems/1.8/bin/

instead of:

/usr/bin

for the knife and various chef commands in the following code.

/usr/bin/chef-solo -j ~/chef.json -c ~/solo.rb -l debug

You will see a lot of Debug statements go by and it will take several minutes to complete. It should complete with something like:

[Thu, 14 Jan 2010 00:19:38 +0000] INFO: Chef Run complete in 38.59808 seconds
[Thu, 14 Jan 2010 00:19:38 +0000] DEBUG: Exiting

Setup basic cookbooks

The following will install the standard cookbooks on the chef server

cd
git clone git://github.com/opscode/chef-repo.git
cd chef-repo
rm cookbooks/README
git clone git://github.com/opscode/cookbooks.git

Now upload the standard cookbooks using the credentials set up by the bootstrap process (user chef-webui)

knife cookbook upload --all -u chef-webui \
-k /etc/chef/webui.pem -o cookbooks

Startup the Chef Server web ui

Do to a bug (http://tickets.opscode.com/browse/CHEF-839) you have to run this twice, the first time will create the admin user:

sudo /usr/bin/chef-server-webui -p 4002

But the first time will abort with an error message like:

Loading init file from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/init-webui.rb
Loading /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/environments/development.rb
~ Loaded slice 'ChefServerWebui' ...
WARN: HTTP Request Returned 404 Not Found: Cannot load user admin
~ Compiling routes...
~ Could not find resource model Node
~ Could not find resource model Client
~ Could not find resource model Role
~ Could not find resource model Search
~ Could not find resource model Cookbook
~ Could not find resource model Client
~ Could not find resource model Databag
~ Could not find resource model DatabagItem
/usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/config/init-webui.rb:32: uninitialized constant OpenID (NameError)
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `call'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `each'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:1258:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/bootloader.rb:99:in `run'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:172:in `bootup'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/server.rb:42:in `start'
from /usr/lib/ruby/gems/1.8/gems/merb-core-1.0.15/lib/merb-core.rb:173:in `start'
from /usr/lib/ruby/gems/1.8/gems/chef-server-0.8.0/bin/chef-server-webui:76
from /usr/bin/chef-server-webui:19:in `load'
from /usr/bin/chef-server-webui:19

Then again to actually start the WebUI and have it run in the background. You might want to start it in screen for now or possibly redirect its output to a log file The following example shows sending the output of the command to a log file. You’ll want to check that log file after starting to make sure there were no errors.

sudo sh -c '/usr/bin/chef-server-webui -p 4002 > /var/log/chef-server-webui.log' &

If you look at the output of a ps, you’ll see the shell command above, but the real work is being done by a merb instance with the port you specified (4002):

#ps ax | grep webui
5533 pts/0    S      0:00 sh -c /usr/bin/chef-server-webui -p 4002 > /var/log/chef-server-webui.log
#ps ax | grep merb
3694 ?        Sl     0:55 merb : worker (port 4000)
5534 pts/0    Sl     0:07 merb : worker (port 4002)

The first merb worker is the chef-server itself, the second is the WebUI server.

Accessing the Chef Web UI

You can access the Chef Web UI web server using a web browser at the IP address / Public DNS name of this server that was just set up. Assuming the Public DNS is

ec2-204-203-51-20.us-west-1.compute.amazonaws.com

Assuming that you set up this instance to allow you to access port 4002 from the IP adddress of your local dev machine, you should be able to access the Web UI at

http://ec2-204-203-51-20.us-west-1.compute.amazonaws.com:4002

You can allow access to port 4002 from specific ip address ranges by updating your security group. You can do that with ElasticFox (easy) or via the command line tools (a pain for a one off). Eventually you (or hopefully Opscode) will set up an apache or nginx reverse proxy, Passenger or equiv to allow normal port 80 / 443 http/https access.

Conclusion

You should now be able to use knife your local dev environment to develop cookbooks and upload roles and cookbooks to your new Chef Server and spin up new chef cookbook driven instances. You should use the knife documentation from the Opscode main wiki Knife Page NOT the docs in the Alpha Forums / Getting Started With Opscode / Knife – Commandline API as the later is actually more obsolete in terms of the version that you built from the opscode git repository. There is also a man page and knife –help gives you pretty much the same correct info as the wiki.

I hope to have a follow up post on how to do this in more details.

Feel free to leave comments if you find problems or have questions.

The post Creating an Amazon EC2 AMI for Opscode Chef 0.8 Client and Server first appeared on Cognizant Transmutation.

Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2

Robert J Berger — Sat, 05 Sep 2009 01:34:41 +0000

NOTE (Sep 7 2009): Updated info on need to use Amazon Private DNS Names and clarified the need for the masters, slaves and regionservers files. Also updated to use HBase 0.20.0 Release Candidate 3

Introduction

As someone who has “skipped” Java and wants to learn as little as possible about it, and as one who has not had much experience with Hadoop so far, HBase deployment has a big learning curve. So some of the things I describe below may be obvious to those who have had experience in those domains.

Where’s the docs for HBase 0.20

If you go to the HBase wiki, you will find that there is not much documentation on the 0.20 version. This puzzled me since all the twittering, blog posting and other buzz was talking about people using 0.20 even though its “pre-release”

One of the great things about going to meetups such as the HBase Meetup is you can talk to the folks who actually wrote the thing and ask them “Where is the documentation for HBase 0.20

Turns out its in the HBase 0.20.0 distribution in the docs directory. The easiest thing is to get the pre-built 0.20.0 release candididate 3. If you download the source from the version control repository you have to build the documentation using Ant. If you are an Java/Ant kind of person it might not be hard. But just to build the docs, you have to meet some dependencies like

What we learnt with 0.19.x

We have been learning a lot about making HBase Cluster work at a basic level. I had a lot of problems getting 0.19.x running beyond a single node in Psuedo Distributed mode. I think a lot of my problems was just not getting how it all fit together with Hadoop and what the different startup/shutdown scripts did.

Then we finally tried the HBase EC2 Scripts even though it uses an AMI based on Fedora 8 and seemed wired to 0.19.0. Its a pretty nice script if you want to have an opionated HBase cluster set up. But it did educate us on how to get a cluster to go. It has a bit of strangeness by having a script in /root/hbase_init that is called at boot time to configure all the hadoop and hbase conf scripts and then call the hadoop and hbase startup scripts. Something like this is kind of needed for Amazon EC2 since you don’t really know what the IP Address/FQDN is until boot time.

The scripts also set up an Amazon Security Group for the cluster master and one for the rest of the cluster. I beleive it then uses this as a way to identify the group as well.

The main thing we did get was by going thru mainly the /root/hbase_init script we were able to figure out what the process was for bringing up Hadoop/HBase as a cluster.

We did build a staging cluster with this script. We were able to pretty easily change the scripts to use 0.19.3 instead of 0.19.0. But its opions were different than ours for many things. Plus after talking to the folks at the HBase Meetup, and having all sort of weird problems with our app on 0.19.3, we were convinced that our future is in HBase 0.20. And 0.20 introduces some new things like using Zookeeper to manage the Master selection so seems like its not worth it for us to continue to use this script. Though it helped in our learning quite a bit!

Building an HBase 0.20.0 Cluster

This post will use the HBase pre-built Release Candidate 3 and the prebuild standard Hadoop 0.20.0.

This post will show how to do all this “by hand”. Hopefully we’ll have an article on how to do all this with Chef sometime soon.

The Hbase folks say that you really should have at least 5 regionservers and one master. The master and several of the regionservers can also run the zookeeper quorum. Of course the master serveris also going to run the Hadoop Nameserver Secondary name server. Then the 5 other nodes are running the Hadoop HDFS Data nodes as well as the HBase region servers. When you build out larger clusters, you will probably want to dedicate machines to Zookeepers and hot-standby Hbase Masters. Name Servers are still the Single Point of Failure (SPOF). Rumour has it that this will be fixed in Hadoop 0.21.

We’re not using Map / Reduce yet so won’t go into that, but its just a mater of different startup scripts to make the same nodes do Map/Reduce as HDFS and HBase.

In this example, we’re installing and running everything as Root. It can also be done as a special user like hadoop as described in the earlier blog post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard

Getting the pre-requisites in order

We started with the vanilla alestic Ubuntu 9.04 Jaunty 64Bit Server AMI: ami-5b46a732 and instantiated 6 High CPU Large Instances. You really want as much memory and cores as you can get. You can do the following by hand or combine it with the shell scripting described below in the section Installing Hadoop and HBase.

apt-get update
apt-get upgrade

Then added via apt-get install:

apt-get install sun-java6-jdk

Downloading Hadoop and HBase

You can use the production Hadoop 0.20.0 release. You can find them at the mirrors at http://www.apache.org/dyn/closer.cgi/hadoop/core/. The examples show from one mirror:

wget http://mirror.cloudera.com/apache/hadoop/core/hadoop-0.20.0/hadoop-0.20.0.tar.gz

You can download the HBase 0.20.0 Release Candidate 3 in a prebuilt form from http://people.apache.org/~stack/hbase-0.20.0-candidate-3/ (You can get the source out of Version Control:http://hadoop.apache.org/hbase/version_control.html but  you'll have to figure out how to build it.)

wget http://people.apache.org/~stack/hbase-0.20.0-candidate-3/hbase-0.20.0.tar.gz

Installing Hadoop and HBase

Assuming that you are running in your home directory on the master server and that the target for the versioned packages is in /mnt/pkgs and that there will be a link in /mnt for the path to the home for hadoop and hbase:

You can do a some simple scripting to do the following on all the nodes at once:

Create a file named servers with the list of the fully qualified domain names of all your servers including “localhost” for the master and call the file “servers”.

Make sure you can ssh to all the servers from the master. Ideally you are using ssh keys. On master:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

On each of your region servers make sure that the id_dsa.pub is also in their authorized_keys (don’t delete any other keys you have in the authorized keys!)

Now with a bit of shell command line scripting you can install on all your servers at once:

for host in `cat servers`
 do
 echo $host
 ssh $host 'apt-get update; apt-get upgrade; apt-get install sun-java6-jdk'
 scp ~/hadoop-0.20.0.tar.gz ~/hbase-0.20.0.tar.gz $host:
 ssh $host 'mkdir -p /mnt/pkgs; cd /mnt/pkgs; tar xzf ~/hadoop-0.20.0.tar.gz; tar xzf ~/hbase-0.20.0.tar.gz; ln -s /mnt/pkgs/hadoop-0.20.0 /mnt/hadoop; ln -s /mnt/pkgs/hbase-0.20.0 /mnt/hbase'
done

Use Amazon Private DNS Names in Config files

So far I have found that its best to use the Amazon Private DNS names in the hadoop and hbase config files. It looks like HBase uses the system hostname to determine various things at runtime. Thie is always the Private DNS name. It also means that its difficult to use the Web GUI interfaces to HBase from outside of the Amazon Cloud. I set up a “desktop” version of Ubuntu that is running in the Amazon Cloud that I VNC (or NX) into and use its browser to view the Web Interface.

In any case, Amazon instances normally have limited TCP/UDP access to the outside world due to the default security group settings. You would have to add the various ports used by HBase and Hadoop to the security group to allow outside access.

If you do use the Amazon Public DNS names in the config files, there will be startup errors like the following for each instance that is assigned to the zookeeper quorum (there may be other errors as well, but these are the most obvious):

ec2-75-101-104-121.compute-1.amazonaws.com: java.io.IOException: Could not find my address: domU-12-31-39-06-9D-51.compute-1.internal in list of ZooKeeper quorum servers
ec2-75-101-104-121.compute-1.amazonaws.com:     at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.writeMyID(HQuorumPeer.java:128)
ec2-75-101-104-121.compute-1.amazonaws.com:     at org.apache.hadoop.hbase.zookeeper.HQuorumPeer.main(HQuorumPeer.java:67)

Configuring Hadoop

Now you have to configure the hadoop on master in /mnt/hadoop/conf:

hadoop-env.sh:

The minimal things to change are:

Set your JAVA_HOME to where the java package is installed. On Ubuntu:

export JAVA_HOME=/usr/lib/jvm/java-6-sun

Add the hbase path to the HADOOP_CLASSPATH:

export HADOOP_CLASSPATH=/mnt/hbase/hbase-0.20.0.jar:/mnt/hbase/hbase-0.20.0-test.jar:/conf

core-site.xml:

Here is what we used. Primarily setting where the hadoop files are and the nameserver path and port:





   
     hadoop.tmp.dir
     /mnt/hadoop
   

   
     fs.default.name
     hdfs://domU-12-31-39-06-9D-51.compute-1.internal:50001
   

   
     tasktracker.http.threads
     80

mapred-site.xml:

Even though we are not currently using Map/Reduce this is a basic config:





   
     mapred.job.tracker
     domU-12-31-39-06-9D-51.compute-1.internal:50002
   

   
     mapred.tasktracker.map.tasks.maximum
     4
   

   
     mapred.tasktracker.reduce.tasks.maximum
     4
   

   
     mapred.output.compress
     true
   

   
     mapred.output.compression.type
     BLOCK

hdfs-site.xml:

The main thing to change based on your config is the dfs.replication. It should be less than the total number of data-nodes / region-servers.





   
     dfs.client.block.write.retries
     3
   

   
     dfs.replication
     3

Put the Fully qualified domain name of your master in the file masters and the names of the data-nodes in the file slaves.

masters:

domU-12-31-39-06-9D-51.compute-1.internal

slaves:

domU-12-31-39-06-9D-C1.compute-1.internal
domU-12-31-39-06-9D-51.compute-1.internal

We did not change any of the other files so far.

Now copy these files to the data-nodes:

for host in `cat slaves`
do
  echo $host
  scp slaves masters hdfs-site.xml hadoop-env.sh core-site.xml ${host}:/mnt/hadoop/conf
done

And also format the hdfs on the master

/mnt/hadoop/bin/hadoop namenode -format

Configuring HBase

hbase-env.sh:

Similar to the hadoop-env.sh, you must set the JAVA_HOME:

export JAVA_HOME=/usr/lib/jvm/java-6-sun

and add the hadoop conf directory to the HBASE_CLASSPATH:

export HBASE_CLASSPATH=/mnt/hadoop/conf

And for the master you will want to say:

export HBASE_MANAGES_ZK=true

hbase-site.xml:

Mainly need to define the hbase master, hbase rootdir and the list of zookeepers. We also had to bump up the hbase.zookeeper.property.maxClientCnxns from the default of 30 to 300.




   
     hbase.master
     domU-12-31-39-06-9D-51.compute-1.internal:60000
   

   
     hbase.rootdir
     hdfs://domU-12-31-39-06-9D-51.compute-1.internal:50001/hbase
   
   
     hbase.zookeeper.quorum
     domU-12-31-39-06-9D-51.compute-1.internal,domU-12-31-39-06-9D-C1.compute-1.internal,domU-12-31-39-06-9D-51.compute-1.internal
   
   
     hbase.cluster.distributed
     true
   
   
     hbase.zookeeper.property.maxClientCnxns
     300

You will also need to have a file called regionservers. Normally it contains the same hostnames as the hadoop slaves:

regionservers:

domU-12-31-39-06-9D-C1.compute-1.internal
domU-12-31-39-06-9D-51.compute-1.internal

Copy the files to the region-servers:

for host in `cat regionservers`
do
  echo $host
  scp hbase-env.sh hbase-site.xml regionservers ${host}:/mnt/hbase/conf
done

Starting Hadoop and HBase

On the master:

(This just starts the Hadoop File System services, not Map/Reduce services)

/mnt/hadoop/bin/start-dfs.sh

Then start hbase:

/mnt/hbase/bin/start-hbase.sh

You can shut things down by doing the reverse:

/mnt/hbase/bin/stop-hbase.sh
/mnt/hadoop/bin/stop-dfs.sh

It is advisable to set up init scripts. This is described in the Ubuntu /etc/init.d style startup scripts section of the earlier blog post:Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard

The post Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2 first appeared on Cognizant Transmutation.

Want to work at a Startup with Cool Tech? (HBase, Clojure, Chef, Swarms, Javascript, Ruby & Rails)

Robert J Berger — Fri, 28 Aug 2009 18:15:01 +0000

Opportunity Knocks

Runa.com, the startup where I am CTO, is looking for great developers to join our small agile team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations). Runa offers a SaaS to on-line merchant that allows them to offer dynamic product and consumer specific promotions embeded in their website. This will be a very large positive disruption to the online retailing world.

Techie keywords: clojure, hadoop, hbase, rabbitmq, erlang, chef, swarm computing, ruby, rails, javascript, amazon EC2, emacs, Macintosh, Linux, selenium, test/behavior driven development, agile, lean, XP, scalability

If you’re interested, email jobs@runa.com

If you want to know more, read on!

What do we do

Runa aims to provide the top of the long tail thru the middle of the top 500 online retailers with tools/services that companies like amazon.com use/provide. These smaller guys can’t afford or don’t have the resources to do anything on that scale, but by using our SaaS services, they can make more money while providing customers with greater value.

The first service we’re building is what we call Dynamic Sale Price.

It’s a simple concept – it allows the online-retailer to offer a sale price for each product on his site, personalized to the individual consumer who is browsing it. By using this service, merchants are able to –

Increase conversion (get them to buy!) and
Offer consumers a special price which maximizes the merchant’s profit

This is different from “dumb-discounting” where something is marked-down, and everyone sees the same price. This service is more like airline or hotel pricing which varies from day to day, but much more dynamic and real-time. Further, it is based on broad statistical factors AND individual consumer behavior. After all, if you lower prices enough, consumers will buy. Instead, we dynamically lower prices to a point where statistically, that consumer is most likely to buy.

How we do it

Runa does this by performing statistical analysis and pattern recognition of what consumers are doing on the merchant sites. This includes browsing products on various pages, adding and removing items from carts, and purchasing or abandoning the carts. We track consumers as they browse, and collect vast quantities of this click-stream data. By mining this data and applying algorithms to determine a price point per consumer based on their behavior, we’re able to maximize both conversion (getting the consumer to buy) AND merchant profit.

We also offer the merchant comprehensive reports based on analysis of the mountains of data we collect. Since the data tracks consumer activity down to the individual product SKU level (for each individual consumer), we can provide very rich analytics. This is a tool that merchants need today, but don’t have the resources to build for themselves.

The business model

For reference, it is useful to understand the affiliate marketing space. Small-to-medium merchants (our target audience) pay affiliates up to 40% of a sale price. Yes, 40%. The average is in the 20% range.

We charge our merchants around 10% of sales the Runa delivers. Our merchants are happy to pay it, because it is a performance-based pay, lower than what they pay affiliates, and there is zero up-front cost to the service. In fact, the above mentioned analytics reports are free.

We’re targeting e-commerce PLATFORMS (as opposed to individual merchants); in this way, we’re able to scale up merchant-acquisition. We have 10 early-customer merchants right now, with about 100 more planned to go live in the next 2-3 months. By the end of next year, we’re targeting about 1,000 merchants and 10,000 merchants the following year. Our channel deployment model makes these goals achievable.

At something like a 5 to 10% service charge, and a typical merchant having between 500K to 1M in sales per year, this is a VERY profitable business model. That is, of course, if we’re successful… but we’re seeing very positive signs so far.

Technology

Most of our front-end stuff (like the merchant-dashboard, reports, campaign management) is built with Ruby on Rails. Our merchant integration requires browser-side Javascript magic. All our analytics (batch-processing) and real-time pricing services are written in Clojure. We use RabbitMQ for all our messaging needs. We store data in HBase. We’re deployed on Amazon’s EC2.

Here are a few blog postings about what we’ve been up to –

Distributed Clojure system in production
Using messaging for scalability
Capjure: a simple HBase persistence layer
Clojure in production
Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2

We’ve also open-sourced a few of our projects –

swarmiji – A distributed computing system to write and run Clojure code in parallel, across CPUs
capjure – Clojure persistence for HBase

Culture at Runa

We’re a small team, very passionate about what we do. We’re focused on delivering a ground-breaking, disruptive service that will allow merchants to really change the way they sell online. We work start-up hours, but we’re flexible and laid-back about it. We know that a healthy personal life is important for a good professional life. We work with each other to support it.

We use an agile process with a lot of influences from the “Lean”:http://en.wikipedia.org/wiki/Lean_software_development and “Kanban”:http://leansoftwareengineering.com/2007/08/29/kanban-systems-for-software-development/ world. We use “Mingle”:http://studios.thoughtworks.com/mingle-agile-project-management to run our development process. Everything, OK mostly everything is covered by automated tests, so we can change things as needed.

We’re all Apple in the office – developers get a MacPro with a nice 30″ screen, and a nice 17″ MacBook Pro. We deploy on Ubuntu servers. Aeron chairs are cliché, yes; but, very comfy.

The environment is chilled out… you can wear shorts and sandals to work… Very flat organization, very non-bureaucratic… nice open spaces (no cubes!). Lunch is brought in on most days! Beer and snacks are always in the fridge.

We’re walking distance to the San Antonio Caltrain station (biking distance from the Mountain View Caltrain/VTA lightrail station).

What’s in it for you

Competitive salaries, and lots of stock-options
Cutting edge technology stack
Fantastic business opportunity, and early-stage (= great time to join!)
Developer #5 – means plenty of influence on foundational architecture and design
Smart, full bandwidth, fun people to work with
Very comfortable, nice office environment
We have a “No Assholes” policy

OK!

So, if you’re interested, email us at jobs@runa.com

No recruiters please!

We would prefer folks who are already in the Bay Area (but if you not local and are really great let’s talk!)

The post Want to work at a Startup with Cool Tech? (HBase, Clojure, Chef, Swarms, Javascript, Ruby & Rails) first appeared on Cognizant Transmutation.

AWS - Cognizant Transmutation

Accessing AppSync APIs that require Cognito Login outside of Amplify

The Need

Use AWS CLI

Cognito User Pool Client App

Minimal IAM permissions

Get the Credentials with no App Client Secret

Get the Credentials when there is an App Client Secret

Using the Credentials

From some Javascript

References

CLI to Switch Amazon AWS Shell Environment Credentials

Deploy WordPress to Amazon EC2 Micro Instance with Opscode Chef

Updates

September 9, 2011

Intro

Some Assumptions for this post

Set up an Account on Amazon Web Services

Get an Opscode Platform Account

Set up your Development Environment

Tweak up your chef-repo

Updating your knife.rb file with Amazon Credentials

Get the Appropriate Cookbooks

Create site-cookbooks to extend standard cookbooks

Extend the Sudo cookbook so its suitable for EC2

Fix a bug in the latest version of the Standard Mysql Cookbook

Extend the WordPress cookbook to do some custom actions

Create a recipe to add the blog user and change ownership of the wordpress directory

Create a template to override the default wordpress apache config

Create a recipe to add a swap file to the server

Create WordPress Role

Upload the cookbooks and roles to Opscode Chef Platform

Create the Users databag

Configure AWS

Set up Security Group

Generate an SSH Key Pair for accessing your instance[s]

Create the Instance and Bootstrap Chef on the Instance

Details of knife command to launch instance

Successful launch results

Configure DNS to have preferred FQDNs point to your instance

Installing your WordPress Blog

Happily Ever After

Using the WordPress Automatic Upgrade Mechanism

Source Code at Github

Modyfying Jets3t S3 GUI tool to work with Walrus (Eucalyptus S3)

Prerequisites

Get the Source of Jets3t with Mercurial

Edit files to make jets3t work with Walrus

Edit LoginCredentialsPanel.java to not check the length of login credentials

Edit jets3t.properties to use parameters for accessing Walrus instead of AWS S3

Optionally edit build.properties

Build Jets3t with your changes to work with Walrus

Copy an EBS AMI image to another Amazon EC2 Region

Prepare the Source Instance and Volume

Start an instance in the source region

Set some Shell source variables on host machine

Start up the source instance and capture the instanceid

Create a volume from the EBS AMI snapshot

Mount the EBS Image of the AMI you want to copy

Prepare the Destination Instance and Volume

Set some Shell destination variables on host machine

Start up the destination instance and capture the dst_instanceid

Create an empty destination volume

Mount the EBS Image of the AMI you want to copy

Copy the data from the Source Volume to the Destination Volume

Copy your credentials to the source machine

Mount the source and destination volumes on their instances

Get the FQDN of the Amazon internal address of the destination machine

Commands to run on the source machine

Set up some shell variables on the source machine shell environment

Do the rsync

If you want to do the rsync from your local host

Complete the new AMI from your Local Host

Some Cleanup for new Region

Clean up from the building of the image

There are a few more shell variables we’ll need

Snapshot the Destination Volume and register the new AMI in the destination region

Conclusion

Simple update and clone an Amazon EC2 EBS Boot image

Introduction

Edit `LoginCredentialsPanel.java` to not check the length of login credentials

Edit `jets3t.properties` to use parameters for accessing Walrus instead of AWS S3

Optionally edit `build.properties`