The first ones we did had no problem at all. But one of them gave the dreaded multi-lingual kernel panic display on the initial reboot after the user ran the Lion Updater.

At first I suspected the fact that the unit had Software RAID as the initial Googling showed that many folks had similar issues with Software RAID. When I booted from the Lion boot DVD I had created from the Lion installer download and ran the Disk Utilities, it reinforced the assumption that there was some issue with Lion and the RAID as the Disk Utility wouldn’t do anything with the disks (the various First Aid buttons were greyed out). But then I had some similar problems with the Snow Leopard DVD Disk Utility, so I assumed that the disk[s] had gotten corrupt. I spend most of the next day using Disk Warrior to recover the disks, tried various attempts of using Disk Utility and command line diskutil to make the RAID work with Lion.

Of course the itterations were long since it required reboots from DVD, etc. Eventually I got some new drives and did a totally fresh install and would just end up with the Grey Screen with the circle with a line thru it instead of the Apple Symbol on boot.

That was when I realized it probably wasn’t the RAID issue (also found plenty of folks including ourselves where Lion and Software RAID did work). So some different Googling found one article that mentioned resetting the PRAM. And of course, that worked. I had previously done the “hold down the power button till the power led blinks rapidly and hear the reset tone” assuming that reset everything that needed to be reset on modern Macs.

I haven’t had to CMD-OPTION-P-R keyboard chord reset the PRAM in ages. I had thought that it was no longer a thing with Modern Macs and the push and hold the power button had replaced it. But low and behold what was need the was the old fashioned PRAM Reset.

So if you have a problem where your Mac won’t reboot after the initial install of Lion, give the old PRAM reset power chord a try.

The post Your Mac Won’t Reboot when Installing Mac OS X Lion – Reset Your PRAM first appeared on Cognizant Transmutation.

One blog post, HowTo: Make Ubuntu A Perfect Mac File Server And Time Machine Volume did a great job going through all the steps needed to build Netatalk from source and configure it to work very transparently with Ubuntu releases of the past. But with the Ubuntu 10.4 Lucid release, the Netatalk that is in the Ubuntu repository is built and configure to support transparent Apple File Protocol based file sharing.

But there are a few configuration issues, mainly with the Unix implementation of Bonjour resource discovery protocol, that still needs to be done to make it so you can see your Ubuntu Filesystems on your Mac’s Finder like other Macintosh instances. Also we’ll see how to make it so that the Ubuntu instance will show up as an ssh server as well.

Installing Packages

You will need to install the following packages onto your Ubuntu 10.4 instance. This assumes that you already did a clean install of Ubuntu 10.4 and used the update manager to bring it up to date. If you have already installed some of these, it should not be a problem.

Install ssh server

I can’t believe that ubuntu doesn’t install an ssh server by default. But in any case its pretty easy. This is not needed to use netatalk but I wanted to make ssh and netatalk to work and be available via bonjour.

sudo apt-get install openssh-server

Then you’ll need to set up your authorized keys on the ubuntu server. In your home directory do the following:

mkdir -p .ssh
# Copy your public key[s] to .ssh/authorized_keys (not shown here)
# Set the permissions to only allow your user to access the .ssh directory and files in there
chmod -R og-rwx .ssh

Install Netatalk

sudo apt-get install netatalk

Configure Netatalk

You don’t need to change any of the configuration files for netatalk. The defaults will enable the sharing of your home directory. If you want to share any additional filesystems from your Ubuntu instance to your Macs, you can add them to the /etc/netatalk/AppleVolumes.default. That file has explanations of al the options.

You may want to change the default last item in /etc/netatalk/AppleVolumes.default from:

~/			"Home Directory"

to something like:

~/ "$h_$u Home Directory" options:upriv,usedots

This will change the name that shows up in listing to be “hostname_username Home Directory” and will use Unix Privilages. Most importantly the usedots says to not do Hex translation of dot files. If you don’t do this, you’ll see things like
:2e_somefilename instead of .somefilename where filenames start with “dot”.

Configure AVAHI

AVAHI is probably already installed if you did a standard installation.

Copy the avahi ssh service configuration into /etc/avahi/services

sudo cp /usr/share/doc/avahi-daemon/examples/ssh.service /etc/avahi/services/

Create an avahi afpd service configuration by creating a file /etc/avahi/services/afpd.service with the following content:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_afpovertcp._tcp</type>
    <port>548</port>
  </service>
  <service>
    <type>_device-info._tcp</type>
    <port>0</port>
    <txt-record>model=Xserve</txt-record>
  </service>
</service-group>

You should now be able to see the Ubuntu host in your Finder under the SHARED section on the left side of the Finder. You should also see your Ubuntu host in the “New Remote Connection” window of the Mac Terminal app (CMD-SHIFT-K) if you select the “Secure Shell (ssh)” Service.

If you don’t see the Ubuntu hostname in the FInder or in the Terminal New Remote Connection service,  restart the avahi-daemon service:

sudo restart avahi-daemon

TimeMachine Support

The new Ubuntu Netatalk package is supposed to also support TimeMachine storage. You can enable this in /etc/netatalk/AppleVolumes.default and add tm as an option to the filesystems that is published in this file. I have not tried this and many sources consider this a risky way to store Time Machine backups.

Troubleshooting

You should make sure that there is at least one afpd process running on the Ubuntu instance. You can see the log info in /var/log/daemon.log.

That’s it!

The post Bonjour / AVAHI & Netatalk to share files files between Ubuntu 10.4 & Mac OS X first appeared on Cognizant Transmutation.

So I thought it was time to revisit installing Hadoop and HBase on the Mac using the latest versions of everything. This will be deployed as Psuedo-Distributed mode native to Mac OS X. Some folks actually create a set of Linux VMs with a full Hadoop/HBase stack and run that on the Mac, but that is a bit of overkill for now.

These instructions mainly follow the standard instructions for Apache Hadoop and Apache HBase

Prerequisits

Mac OS X Xcode developer tools which includes Java 1.6.x. You can get this for free from the Apple Mac Dev Center. You have to become a member but there is a free membership available.

Download and Unpack Latest Distros

You can get a link to a mirror for Hadoop via the Hadoop Apache Mirror link and for Hbase at the HBase Apache Mirror link. Each of those links will bring you to a suggested link to a mirror for Hadoop or HBase. Once you click on the suggest link, it will bring you to a mirror with the recent releases. You can click on the stable link which will then bring you to a directory that has the latest stable Hadoop (as of this writing: hadoop-0.20.2.tar.gz) or HBase (as of this writing: hbase-0.20.3.tar.gz ). Click on those tar.gz files to download them.

I am going to keep the distros in ~/work/pkgs. I usually create a directory ~/work/pkgs and unpack the tar files there as numbered versions and then create symbolic links to them in ~/work. But you can do this all in any directory that you can control.:

cd ~/work
mkdir -p pkgs
cd pkgs
tar xvzf hadoop-0.20.2.tar.gz
tar xvzf hbase-0.20.3.tar.gz
cd ..
ln -s pkgs/hadoop-0.20.2 hadoop
ln -s pkgs/hbase-020.3 hbase
mkdir -p hadoop/logs
mkdir -p hbase/logs

Now you can have your tools all access ~/work/hadoop or ~/work/hbase and not care what version it is. You can update to later version just by downloading, untarring the distro and then just change the symbolic links.

Configure Hadoop

All the configuration files mentioned here will be in ~/work/hadoop/conf. In this example we are assuming that the Hadoop servers will only be accessed from this localhost. If you need to make it accessable from other hosts or VMs on your lan that support Bonjour, you could use the bonjour name (ie. the name of your mac followed by .local such as mymac.local) instead of localhost in the following Hadoop and HBase configuraitons

hadoop-env.sh

Mainly need to tell Hadoop where your JAVA_HOME is.

Add the following line below the commented out JAVA_HOME line is in hadoop-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>

Make sure you can ssh without a password to the hostname used in the configs

The Hadoop and Hbase start/stop scripts use ssh to access the various servers. In this case of doing a Pseudo-Distributed mode, everything is running on the localhost, but we still need to allow the scripts to ssh to the localhost.

Check that you can ssh to the localhost (or whatever hostname you used in the above configs)

We’re assuming that we’ll be running the Hadoop/HBase servers as the same user as our login. You can set things up to run as the hadoop user, but its kind of complicated on Mac OS X. See the section File System Layout in an earlier post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard. That section and a few other points thru that post describe how to create and use a hadoop user to run the Hadoop and HBase servers.

Back to just doing this as our own user. Test that you can ssh to the localhost without a password:

ssh localhost

If you see something like the following paragraph that ends up with a password prompt, then you need to add a key to your ssh setup that does not need a password (you may need to say yes if you are asked if you want to continue connecting).

The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 3c:5d:6a:39:64:78:02:9d:a3:c9:69:68:50:23:71:eb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password:

To create a passwordless key and add it to your set of authorized keys that can access your host, do the following (as yourself, not as root. The id_dsa file name can be arbitrary):

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa_for_hadoop
cat ~/.ssh/id_dsa_for_hadoop.pub >> ~/.ssh/authorized_keys

If you have strong alternative opinions on how to set up your own keys to accomplish the same thing please do it your own way. This is just the basic way of doing a passwordless ssh. You may want to use a key you already have lying around or some other mechanism.

Start Hadoop

One time format of  Hadoop File System

Only once, before the first time you use Hadoop, you have to create a formated Hadoop File System. Don’t do this again once you have data in your Hadoop file system as it will erase anything you might have saved there. You may have to do this command again if somehow you screw up your file system. But its not something to do lightly the second time.

~/work/hadoop/bin/hadoop namenode -format

If all goes well, you should see something like:

10/05/02 18:45:04 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Psion.local/192.168.50.16
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/05/02 18:45:04 INFO namenode.FSNamesystem: fsOwner=rberger,rberger,admin,com.apple.access_screensharing,_developer,_lpoperator,_lpadmin,_appserveradm,_appserverusr,localaccounts,everyone,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3,dev,com.apple.sharepoint.group.1,workgroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: supergroup=supergroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/05/02 18:45:04 INFO common.Storage: Image file of size 97 saved in 0 seconds.
10/05/02 18:45:04 INFO common.Storage: Storage directory /tmp/hadoop-rberger/dfs/name has been successfully formatted.
10/05/02 18:45:04 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Psion.local/192.168.50.16
************************************************************/

Starting and stopping Hadoop

Now you can start Hadoop. You will use this command to start Hadoop in general:

~/work/hadoop/bin/start-all.sh

You can stop Hadoop with the command

~/work/hadoop/bin/stop-all.sh

But remember if you are running HBase, stop that first, then stop Hadoop.

Making sure Hadoop is working

You can see the Hadoop logs in ~/work/hadoop/logs

You should be able to see the Hadoop Namenode web interface at http://localhost:50070/ and the JobTracker Web Interface at http://localhost:50030/. If not, check that you have 5 java processes running where each of those java processes have one of the following as their last command line (as seen from a ps ax | grep hadoop command) :

org.apache.hadoop.mapred.JobTracker
org.apache.hadoop.hdfs.server.namenode.NameNode
org.apache.hadoop.mapred.TaskTracker
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
org.apache.hadoop.hdfs.server.datanode.DataNode

If you do not see these 5 processes, check the logs in ~work/hadoop/logs/*.{out,log} for messages that might give you a hint as to what went wrong.

Run some example map/reduce jobs

The Hadoop distro comes with some example / test map / reduce jobs. Here we’ll run them and make sure things are working end to end.

cd ~/work/hadoop
# Copy the input files into the distributed filesystem
# (there will be no output visible from the command):
bin/hadoop fs -put conf input
# Run some of the examples provided:
# (there will be a large amount of INFO statements as output)
bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
# Examine the output files:
bin/hadoop fs -cat output/part-00000

The resulting output should be something like:

3	dfs.class
2	dfs.period
1	dfs.file
1	dfs.replication
1	dfs.servers
1	dfsadmin
1	dfsmetrics.log

Configuring HBase

The following config files all reside in ~/work/hbase/conf. As mentioned earlier, use a FQDN or a Bonjour name instead of localhost if you need remote clients to access HBase. But if you don’t use localhost here, make sure you do the same in the Hadoop config.

hbase-env.sh

Add the following line below the commented out JAVA_HOME line is in hbase-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

Add the following line below the commented out HBASE_CLASSPATH= line

export HBASE_CLASSPATH=${HOME}/work/hadoop/conf

hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://localhost:9000/hbase</value>
    <description>The directory shared by region servers.
    </description>
  </property>
</configuration>

Making Sure HBase is Working

If you do a ps ax | grep hbase you should see two java processes. One should end with:
org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
And the other should end with:
org.apache.hadoop.hbase.master.HMaster start
Since we are running in the Pseudo-Distributed mode, there will not be any explicit regionservers running. If you have problems, check the logs in ~/work/hbase/logs/*.{out,log}

Testing HBase using the HBase Shell

From the unix prompt give the following command:

~/work/hbase/bin/hbase shell

Here is some example commands from the Apache HBase Installation Instructions:

base> # Type "help" to see shell help screen
hbase> help
hbase> # To create a table named "mylittletable" with a column family of "mylittlecolumnfamily", type
hbase> create "mylittletable", "mylittlecolumnfamily"
hbase> # To see the schema for you just created "mylittletable" table and its single "mylittlecolumnfamily", type
hbase> describe "mylittletable"
hbase> # To add a row whose id is "myrow", to the column "mylittlecolumnfamily:x" with a value of 'v', do
hbase> put "mylittletable", "myrow", "mylittlecolumnfamily:x", "v"
hbase> # To get the cell just added, do
hbase> get "mylittletable", "myrow"
hbase> # To scan you new table, do
hbase> scan "mylittletable"

You can stop hbase with the command:

~/work/hbase/bin/stop-hbase.sh

Once that has stopped you can stop hadoop:

~/work/hadoop/bin/stop-all.sh

Conclusion

You should now have a fully working Pseudo-Distributed Hadoop / HBase setup on your Mac. This is not suitable for any kind of large data or production project. In fact it will probably fail if you try to do anything with lots of data or high volumes of I/O. HBase seems to not like to work well until you get 4 – 5 regionservers.

But this Pseudo-Distributed version should be fine for doing experiments with tools and small data sets.

Now I can get on with playing with Cascading-Clojure and Cascalog!

The post HBase/Hadoop on Mac OS X (Pseudo-Distributed) first appeared on Cognizant Transmutation.

Gitting the latest source

For instance they show you how to get the latest via SVN or a snapshop via wget. But the wget actually gets it from a git repository, but they don’t tell you how to directly git it! Which is:

git clone git://git.thrift-rpc.org/thrift.git

That will create a a source distribution of thrift in a directory called thrift.

The git repository is where the developers are really working according to the Developers Wiki on the GitRepository. There is also a copy on github.

Requirements

The relevant requirements as stated by the wiki are:

GNU build tools: autoconf 2.59+ (2.60+ recommended), automake 1.9+, libtool 1.5.24+
boost 1.34.0+
g++ 3.3.5+
pkgconfig (Use MacPorts for Mac OS X)
lex and yacc (developed primarily with flex and bison)

Well, for Ubuntu it wasn’t quite clear what was really required. The GettingUbuntuPackages wiki page listed only a few of the required packages. Max Luebbe has a blog page that has a more in depth list:

apt-get install libboost-dev libevent-dev python-dev automake pkg-config libtool flex bison sun-java5-jdk

We already had Sun Java6 installed and that worked fine, so I didn’t include sun-java5-jdk. But we didn’t have g++ installed, so also do:

apt-get install g++

Confusingly, the ./configure did not fail saying there was no g++ but failed by saying there was no boost. It took a while to figure out it was actually not finding boost because it could not compile the little configure test script that was used to detect if boost was installed or not!

So the actual apt-get used on our ubuntu 8.04 server was:

sudo apt-get install g++ libboost-dev libevent-dev python-dev automake pkg-config libtool flex bison

On the Mac you can use the MacPorts to install the required packages.. Max also had a good page on Installing Apache Thrift on Mac OS X 10.5 Leopard that doesn’t require MacPorts.

sudo port selfupdate
sudo port install boost
sudo port install pkgconfig

The pkg.m4 workaround

As noted in the Thrift Wiki FAQ,the ./configure command may generate an error like:

./configure: line 21183: syntax error near unexpected token `MONO,'
./configure: line 21183: `  PKG_CHECK_MODULES(MONO, mono >= 1.2.6, have_mono=yes, have_mono=no)'

This will happen if there is no pkg.m4 file in the aclocal directory of the thrift source tree. For the Macintosh, install pkgconfig via MacPorts and copy /opt/local/share/aclocal to aclocal (assuming you are in the thrift source distro):

cp /opt/local/share/aclocal/pkg.m4 aclocal

This is not necessary in ubuntu if you have installed pkgconfig there.

Actual Build and Installation

In the Thrift directory run:

./bootstrap.sh

on the Mac if boost was installed with MacPorts use the following (If you manually installed boost elsewhere use that path instead):

./configure --with-boost=/opt/local

on Ubunto you can just say”

./configure

On both Mac and Ubuntu:

make
sudo make install

If you want any of the bindings for different languages, cd into lib and there are directories for each language. Its not always clear what to do to build them. For the ruby one what I ended up doing was:

cd lib/rb
sudo ruby setup.rb

Next step

Figure out how to test and use Thrift!

The post Installing Apache Thrift on Ubuntu and Leopard first appeared on Cognizant Transmutation.