brew install wine

Until it failed trying to install libicns:

==> ./configure --prefix=/usr/local/Cellar/libicns/0.8.1
==> make install
_WritePNGImage in icns2png.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [icns2png] Error 1
make: *** [install-recursive] Error 1

READ THIS: https://github.com/mxcl/homebrew/wiki/troubleshooting

These open issues may also help:
https://github.com/mxcl/homebrew/issues/15478

Fortunately the second link showed what needed to be done:

brew rm libpng
brew remove freetype
brew install libpng --universal
brew install freetype --universal
brew install wine

The post Install Wine on Mac Mountain Lion with Homebrew first appeared on Cognizant Transmutation.

So I usually would try to use Homebrew to install it: brew install pgrep

only to get: Error: No available formula for pgrep

After a bit of googling I find again that its: brew install proctools

The post Install Pgrep and PKill for Mac OS X with Brew first appeared on Cognizant Transmutation.

The first ones we did had no problem at all. But one of them gave the dreaded multi-lingual kernel panic display on the initial reboot after the user ran the Lion Updater.

At first I suspected the fact that the unit had Software RAID as the initial Googling showed that many folks had similar issues with Software RAID. When I booted from the Lion boot DVD I had created from the Lion installer download and ran the Disk Utilities, it reinforced the assumption that there was some issue with Lion and the RAID as the Disk Utility wouldn’t do anything with the disks (the various First Aid buttons were greyed out). But then I had some similar problems with the Snow Leopard DVD Disk Utility, so I assumed that the disk[s] had gotten corrupt. I spend most of the next day using Disk Warrior to recover the disks, tried various attempts of using Disk Utility and command line diskutil to make the RAID work with Lion.

Of course the itterations were long since it required reboots from DVD, etc. Eventually I got some new drives and did a totally fresh install and would just end up with the Grey Screen with the circle with a line thru it instead of the Apple Symbol on boot.

That was when I realized it probably wasn’t the RAID issue (also found plenty of folks including ourselves where Lion and Software RAID did work). So some different Googling found one article that mentioned resetting the PRAM. And of course, that worked. I had previously done the “hold down the power button till the power led blinks rapidly and hear the reset tone” assuming that reset everything that needed to be reset on modern Macs.

I haven’t had to CMD-OPTION-P-R keyboard chord reset the PRAM in ages. I had thought that it was no longer a thing with Modern Macs and the push and hold the power button had replaced it. But low and behold what was need the was the old fashioned PRAM Reset.

So if you have a problem where your Mac won’t reboot after the initial install of Lion, give the old PRAM reset power chord a try.

The post Your Mac Won’t Reboot when Installing Mac OS X Lion – Reset Your PRAM first appeared on Cognizant Transmutation.

I really like how Lyx is a true structured document editor. Its What You See Is What You Mean (WYSIWYM) instead of WYSIWYG. You focus on the structure and text of your document and let the tool render the document.

The main users of Lyx seem to be academics. Most of the templates are for Thesis and other academic related documents. But the Report and Article templates are fine for most technical documents and that is what I’m using.

Lyx is not for those who want to tweak the layout via GUIs (you can tweak the layout as much as you want, but you have to understand the underlying LaTeX platform and do most of  your tweaks in code). Its not a slick, ultra-friendly environment. And things get a bit confusing once you get off the beaten track of the pretty good documentation of basic stuff. It doesn’t particularly feel like a Mac App. But its the best I’ve found for doing structured technical docs in a way that is versionable (like with Git) and can support multiple people doing edits and collaboration on a document.

I’m using the 2.0 Beta which is a significant improvement over the 1.6 stable version. I’ve had no problems with the Beta and the functionality is a big improvement. You can download the Beta binary images for the Mac, Linux or if you must, Windows here: ftp://ftp.lyx.org/pub/lyx/devel/lyx-2.0/beta2.

Some things are a bit opaque as how to do. For instance, Lyx has some nice mechanisms to add Notes and Comments to the text. But these Notes and comments don’t show up in the PDF output. This is fine if you are doing all your collaboration with other folks using Lyx. But sometimes you want to share the PDF output and have the notes show up there. I could not find out a way to make Notes or Comments to  show up in the PDF. The “Greyed Out” option will show up in the PDF though.

I did discover a couple of options in the Lyx Wiki. The first one ExportWithCommentsIncluded was a bit confusing as it didn’t seem to apply to the 2.0 Beta. But if I put the code they suggest for version <1.5

\RequirePackage{colortbl, tabularx}
\@ifundefined{comment}{}% do nothing if the comment environment is not defined
  {% redefine the comment environment if it is defined
   \renewenvironment{comment}
    {% replaces \begin{comment}
     \par\medskip\noindent
     \tabularx{\textwidth}{|>{\columncolor[gray]{0.9}}X|}
     \hline
     \emph{\textbf{Comment:}}% You can use any other text instead of "Comment:" or leave it
    }
    {% replaces \end{comment}
     \endtabularx\hrule\par\medskip
    }
  }%

into the Document Preamble, (accessed via Menu Bar Document->Settings…->LaTeX Preamble), then the Comments would show up in the PDF. (Notes still does not).

Another place in the Lyx Wiki, Producing annotations and comments,  mentions you can load a module that directly uses PDF annotations. But it was a bit short on instructions on how exactly to do that. I eventually figured it out. These instructions are for Lyx 2.0 Beta 2 on Mac OS X Snow Leopard.

Save the file from the link uploads:/Modules/pdfcomment.module to ~/Library/Application Support/Lyx-2.0/layouts/pdfcomment.module. Download the file pdfcomment_v2.1.zip, unzip it and copy the file pdfcomment.sty from the unzipped directory to /Application/LyX.app/Contents/Resources/tex/pdfcomment.sty

Then inside of Lyx mouse to Menu Bar Lyx->Reconfigure. Quit Lyx and restart.

Next mouse to Menu Bar Document->Settings…->Modules, select PDF comments in the Available and click Add, then Apply (and maybe Save as Document Defaults so it will apply to all documents from then on)

Now you will have new options in Menu Bar Insert->Custom Insets: PDF-Annotation and PDF-Margin. I couldn’t see a difference between them off hand. They put the PDF annotation Icon at the point of insert and the comments you type for that tag show up in the margin of the PDF document.

You potentially can tweak the look of the PDF annotations but you need to dive into the code.

I’ll try to blog about other experiences and tips I find on Lyx as I go.

The post Getting Comments/Annotations in LyX to show up in PDF first appeared on Cognizant Transmutation.

One blog post, HowTo: Make Ubuntu A Perfect Mac File Server And Time Machine Volume did a great job going through all the steps needed to build Netatalk from source and configure it to work very transparently with Ubuntu releases of the past. But with the Ubuntu 10.4 Lucid release, the Netatalk that is in the Ubuntu repository is built and configure to support transparent Apple File Protocol based file sharing.

But there are a few configuration issues, mainly with the Unix implementation of Bonjour resource discovery protocol, that still needs to be done to make it so you can see your Ubuntu Filesystems on your Mac’s Finder like other Macintosh instances. Also we’ll see how to make it so that the Ubuntu instance will show up as an ssh server as well.

Installing Packages

You will need to install the following packages onto your Ubuntu 10.4 instance. This assumes that you already did a clean install of Ubuntu 10.4 and used the update manager to bring it up to date. If you have already installed some of these, it should not be a problem.

Install ssh server

I can’t believe that ubuntu doesn’t install an ssh server by default. But in any case its pretty easy. This is not needed to use netatalk but I wanted to make ssh and netatalk to work and be available via bonjour.

sudo apt-get install openssh-server

Then you’ll need to set up your authorized keys on the ubuntu server. In your home directory do the following:

mkdir -p .ssh
# Copy your public key[s] to .ssh/authorized_keys (not shown here)
# Set the permissions to only allow your user to access the .ssh directory and files in there
chmod -R og-rwx .ssh

Install Netatalk

sudo apt-get install netatalk

Configure Netatalk

You don’t need to change any of the configuration files for netatalk. The defaults will enable the sharing of your home directory. If you want to share any additional filesystems from your Ubuntu instance to your Macs, you can add them to the /etc/netatalk/AppleVolumes.default. That file has explanations of al the options.

You may want to change the default last item in /etc/netatalk/AppleVolumes.default from:

~/			"Home Directory"

to something like:

~/ "$h_$u Home Directory" options:upriv,usedots

This will change the name that shows up in listing to be “hostname_username Home Directory” and will use Unix Privilages. Most importantly the usedots says to not do Hex translation of dot files. If you don’t do this, you’ll see things like
:2e_somefilename instead of .somefilename where filenames start with “dot”.

Configure AVAHI

AVAHI is probably already installed if you did a standard installation.

Copy the avahi ssh service configuration into /etc/avahi/services

sudo cp /usr/share/doc/avahi-daemon/examples/ssh.service /etc/avahi/services/

Create an avahi afpd service configuration by creating a file /etc/avahi/services/afpd.service with the following content:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_afpovertcp._tcp</type>
    <port>548</port>
  </service>
  <service>
    <type>_device-info._tcp</type>
    <port>0</port>
    <txt-record>model=Xserve</txt-record>
  </service>
</service-group>

You should now be able to see the Ubuntu host in your Finder under the SHARED section on the left side of the Finder. You should also see your Ubuntu host in the “New Remote Connection” window of the Mac Terminal app (CMD-SHIFT-K) if you select the “Secure Shell (ssh)” Service.

If you don’t see the Ubuntu hostname in the FInder or in the Terminal New Remote Connection service,  restart the avahi-daemon service:

sudo restart avahi-daemon

TimeMachine Support

The new Ubuntu Netatalk package is supposed to also support TimeMachine storage. You can enable this in /etc/netatalk/AppleVolumes.default and add tm as an option to the filesystems that is published in this file. I have not tried this and many sources consider this a risky way to store Time Machine backups.

Troubleshooting

You should make sure that there is at least one afpd process running on the Ubuntu instance. You can see the log info in /var/log/daemon.log.

That’s it!

The post Bonjour / AVAHI & Netatalk to share files files between Ubuntu 10.4 & Mac OS X first appeared on Cognizant Transmutation.

So I thought it was time to revisit installing Hadoop and HBase on the Mac using the latest versions of everything. This will be deployed as Psuedo-Distributed mode native to Mac OS X. Some folks actually create a set of Linux VMs with a full Hadoop/HBase stack and run that on the Mac, but that is a bit of overkill for now.

These instructions mainly follow the standard instructions for Apache Hadoop and Apache HBase

Prerequisits

Mac OS X Xcode developer tools which includes Java 1.6.x. You can get this for free from the Apple Mac Dev Center. You have to become a member but there is a free membership available.

Download and Unpack Latest Distros

You can get a link to a mirror for Hadoop via the Hadoop Apache Mirror link and for Hbase at the HBase Apache Mirror link. Each of those links will bring you to a suggested link to a mirror for Hadoop or HBase. Once you click on the suggest link, it will bring you to a mirror with the recent releases. You can click on the stable link which will then bring you to a directory that has the latest stable Hadoop (as of this writing: hadoop-0.20.2.tar.gz) or HBase (as of this writing: hbase-0.20.3.tar.gz ). Click on those tar.gz files to download them.

I am going to keep the distros in ~/work/pkgs. I usually create a directory ~/work/pkgs and unpack the tar files there as numbered versions and then create symbolic links to them in ~/work. But you can do this all in any directory that you can control.:

cd ~/work
mkdir -p pkgs
cd pkgs
tar xvzf hadoop-0.20.2.tar.gz
tar xvzf hbase-0.20.3.tar.gz
cd ..
ln -s pkgs/hadoop-0.20.2 hadoop
ln -s pkgs/hbase-020.3 hbase
mkdir -p hadoop/logs
mkdir -p hbase/logs

Now you can have your tools all access ~/work/hadoop or ~/work/hbase and not care what version it is. You can update to later version just by downloading, untarring the distro and then just change the symbolic links.

Configure Hadoop

All the configuration files mentioned here will be in ~/work/hadoop/conf. In this example we are assuming that the Hadoop servers will only be accessed from this localhost. If you need to make it accessable from other hosts or VMs on your lan that support Bonjour, you could use the bonjour name (ie. the name of your mac followed by .local such as mymac.local) instead of localhost in the following Hadoop and HBase configuraitons

hadoop-env.sh

Mainly need to tell Hadoop where your JAVA_HOME is.

Add the following line below the commented out JAVA_HOME line is in hadoop-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>

Make sure you can ssh without a password to the hostname used in the configs

The Hadoop and Hbase start/stop scripts use ssh to access the various servers. In this case of doing a Pseudo-Distributed mode, everything is running on the localhost, but we still need to allow the scripts to ssh to the localhost.

Check that you can ssh to the localhost (or whatever hostname you used in the above configs)

We’re assuming that we’ll be running the Hadoop/HBase servers as the same user as our login. You can set things up to run as the hadoop user, but its kind of complicated on Mac OS X. See the section File System Layout in an earlier post Hadoop, HDFS and Hbase on Ubuntu & Macintosh Leopard. That section and a few other points thru that post describe how to create and use a hadoop user to run the Hadoop and HBase servers.

Back to just doing this as our own user. Test that you can ssh to the localhost without a password:

ssh localhost

If you see something like the following paragraph that ends up with a password prompt, then you need to add a key to your ssh setup that does not need a password (you may need to say yes if you are asked if you want to continue connecting).

The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 3c:5d:6a:39:64:78:02:9d:a3:c9:69:68:50:23:71:eb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password:

To create a passwordless key and add it to your set of authorized keys that can access your host, do the following (as yourself, not as root. The id_dsa file name can be arbitrary):

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa_for_hadoop
cat ~/.ssh/id_dsa_for_hadoop.pub >> ~/.ssh/authorized_keys

If you have strong alternative opinions on how to set up your own keys to accomplish the same thing please do it your own way. This is just the basic way of doing a passwordless ssh. You may want to use a key you already have lying around or some other mechanism.

Start Hadoop

One time format of  Hadoop File System

Only once, before the first time you use Hadoop, you have to create a formated Hadoop File System. Don’t do this again once you have data in your Hadoop file system as it will erase anything you might have saved there. You may have to do this command again if somehow you screw up your file system. But its not something to do lightly the second time.

~/work/hadoop/bin/hadoop namenode -format

If all goes well, you should see something like:

10/05/02 18:45:04 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Psion.local/192.168.50.16
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/05/02 18:45:04 INFO namenode.FSNamesystem: fsOwner=rberger,rberger,admin,com.apple.access_screensharing,_developer,_lpoperator,_lpadmin,_appserveradm,_appserverusr,localaccounts,everyone,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3,dev,com.apple.sharepoint.group.1,workgroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: supergroup=supergroup
10/05/02 18:45:04 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/05/02 18:45:04 INFO common.Storage: Image file of size 97 saved in 0 seconds.
10/05/02 18:45:04 INFO common.Storage: Storage directory /tmp/hadoop-rberger/dfs/name has been successfully formatted.
10/05/02 18:45:04 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Psion.local/192.168.50.16
************************************************************/

Starting and stopping Hadoop

Now you can start Hadoop. You will use this command to start Hadoop in general:

~/work/hadoop/bin/start-all.sh

You can stop Hadoop with the command

~/work/hadoop/bin/stop-all.sh

But remember if you are running HBase, stop that first, then stop Hadoop.

Making sure Hadoop is working

You can see the Hadoop logs in ~/work/hadoop/logs

You should be able to see the Hadoop Namenode web interface at http://localhost:50070/ and the JobTracker Web Interface at http://localhost:50030/. If not, check that you have 5 java processes running where each of those java processes have one of the following as their last command line (as seen from a ps ax | grep hadoop command) :

org.apache.hadoop.mapred.JobTracker
org.apache.hadoop.hdfs.server.namenode.NameNode
org.apache.hadoop.mapred.TaskTracker
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
org.apache.hadoop.hdfs.server.datanode.DataNode

If you do not see these 5 processes, check the logs in ~work/hadoop/logs/*.{out,log} for messages that might give you a hint as to what went wrong.

Run some example map/reduce jobs

The Hadoop distro comes with some example / test map / reduce jobs. Here we’ll run them and make sure things are working end to end.

cd ~/work/hadoop
# Copy the input files into the distributed filesystem
# (there will be no output visible from the command):
bin/hadoop fs -put conf input
# Run some of the examples provided:
# (there will be a large amount of INFO statements as output)
bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
# Examine the output files:
bin/hadoop fs -cat output/part-00000

The resulting output should be something like:

3	dfs.class
2	dfs.period
1	dfs.file
1	dfs.replication
1	dfs.servers
1	dfsadmin
1	dfsmetrics.log

Configuring HBase

The following config files all reside in ~/work/hbase/conf. As mentioned earlier, use a FQDN or a Bonjour name instead of localhost if you need remote clients to access HBase. But if you don’t use localhost here, make sure you do the same in the Hadoop config.

hbase-env.sh

Add the following line below the commented out JAVA_HOME line is in hbase-env.sh

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

Add the following line below the commented out HBASE_CLASSPATH= line

export HBASE_CLASSPATH=${HOME}/work/hadoop/conf

hbase-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://localhost:9000/hbase</value>
    <description>The directory shared by region servers.
    </description>
  </property>
</configuration>

Making Sure HBase is Working

If you do a ps ax | grep hbase you should see two java processes. One should end with:
org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
And the other should end with:
org.apache.hadoop.hbase.master.HMaster start
Since we are running in the Pseudo-Distributed mode, there will not be any explicit regionservers running. If you have problems, check the logs in ~/work/hbase/logs/*.{out,log}

Testing HBase using the HBase Shell

From the unix prompt give the following command:

~/work/hbase/bin/hbase shell

Here is some example commands from the Apache HBase Installation Instructions:

base> # Type "help" to see shell help screen
hbase> help
hbase> # To create a table named "mylittletable" with a column family of "mylittlecolumnfamily", type
hbase> create "mylittletable", "mylittlecolumnfamily"
hbase> # To see the schema for you just created "mylittletable" table and its single "mylittlecolumnfamily", type
hbase> describe "mylittletable"
hbase> # To add a row whose id is "myrow", to the column "mylittlecolumnfamily:x" with a value of 'v', do
hbase> put "mylittletable", "myrow", "mylittlecolumnfamily:x", "v"
hbase> # To get the cell just added, do
hbase> get "mylittletable", "myrow"
hbase> # To scan you new table, do
hbase> scan "mylittletable"

You can stop hbase with the command:

~/work/hbase/bin/stop-hbase.sh

Once that has stopped you can stop hadoop:

~/work/hadoop/bin/stop-all.sh

Conclusion

You should now have a fully working Pseudo-Distributed Hadoop / HBase setup on your Mac. This is not suitable for any kind of large data or production project. In fact it will probably fail if you try to do anything with lots of data or high volumes of I/O. HBase seems to not like to work well until you get 4 – 5 regionservers.

But this Pseudo-Distributed version should be fine for doing experiments with tools and small data sets.

Now I can get on with playing with Cascading-Clojure and Cascalog!

The post HBase/Hadoop on Mac OS X (Pseudo-Distributed) first appeared on Cognizant Transmutation.

Runa.com, the startup where I am CTO, is looking for great developers to join our small agile team. We’re an early stage, pre-series-A startup (presently funded with strategic investments from two large corporations). Runa offers a SaaS to on-line merchant that allows them to offer dynamic product and consumer specific promotions embeded in their website. This will be a very large positive disruption to the online retailing world.

Techie keywords: clojure, hadoop, hbase, rabbitmq, erlang, chef, swarm computing, ruby, rails, javascript, amazon EC2, emacs, Macintosh, Linux, selenium, test/behavior driven development, agile, lean, XP, scalability

If you’re interested, email  jobs@runa.com

If you want to know more, read on!

What do we do

Runa aims to provide the top of the long tail thru the middle of the top 500 online retailers with tools/services that companies like amazon.com use/provide. These smaller guys can’t afford or don’t have the resources to do anything on that scale, but by using our SaaS services, they can make more money while providing customers with greater value.

The first service we’re building is what we call Dynamic Sale Price.

It’s a simple concept – it allows the online-retailer to offer a sale price for each product on his site, personalized to the individual consumer who is browsing it. By using this service, merchants are able to –

• Increase conversion (get them to buy!) and
• Offer consumers a special price which maximizes the merchant’s profit

This is different from “dumb-discounting” where something is marked-down, and everyone sees the same price. This service is more like airline or hotel pricing which varies from day to day, but much more dynamic and real-time. Further, it is based on broad statistical factors AND individual consumer behavior. After all, if you lower prices enough, consumers will buy. Instead, we dynamically lower prices to a point where statistically, that consumer is most likely to buy.

How we do it

Runa does this by performing statistical analysis and pattern recognition of what consumers are doing on the merchant sites. This includes browsing products on various pages, adding and removing items from carts, and purchasing or abandoning the carts. We track consumers as they browse, and collect vast quantities of this click-stream data. By mining this data and applying algorithms to determine a price point per consumer based on their behavior, we’re able to  maximize both conversion (getting the consumer to buy) AND merchant profit.

We also offer the merchant comprehensive reports based on analysis of the mountains of data we collect. Since the data tracks consumer activity down to the individual product SKU level (for each individual consumer), we can provide very rich analytics.  This is a tool that merchants need today, but don’t have the resources to build for themselves.

The business model

For reference, it is useful to understand the affiliate marketing space. Small-to-medium merchants (our target audience) pay affiliates up to 40% of a sale price. Yes, 40%. The average is in the 20% range.

We charge our merchants around 10% of sales the Runa delivers. Our merchants are happy to pay it, because it is a performance-based pay, lower than what they pay affiliates, and there is zero up-front cost to the service. In fact, the above mentioned analytics reports are free.

We’re targeting e-commerce PLATFORMS (as opposed to individual merchants); in this way, we’re able to scale up merchant-acquisition. We have 10 early-customer merchants right now, with about 100 more planned to go live in the next 2-3 months. By the end of next year, we’re targeting about 1,000 merchants and 10,000 merchants the following year. Our channel deployment model makes these goals achievable.

At something like a 5 to 10% service charge, and a typical merchant having between 500K to 1M in sales per year, this is a VERY profitable business model. That is, of course, if we’re successful… but we’re seeing very positive signs so far.

Technology

Most of our front-end stuff (like the merchant-dashboard, reports, campaign management) is built with Ruby on Rails. Our merchant integration requires browser-side Javascript magic. All our analytics (batch-processing) and real-time pricing services are written in Clojure. We use RabbitMQ for all our messaging needs. We store data in HBase. We’re deployed on Amazon’s EC2.

Here are a few blog postings about what we’ve been up to –

Distributed Clojure system in production
Using messaging for scalability
Capjure: a simple HBase persistence layer
Clojure in production
Experience installing Hbase 0.20.0 Cluster on Ubuntu 9.04 and EC2

We’ve also open-sourced a few of our projects –

swarmiji – A distributed computing system to write and run Clojure code in parallel, across CPUs
capjure – Clojure persistence for HBase

Culture at Runa

We’re a small team, very passionate about what we do. We’re focused on delivering a ground-breaking, disruptive service that will allow merchants to really change the way they sell online. We work start-up hours, but we’re flexible and laid-back about it. We know that a healthy personal life is important for a good professional life. We work with each other to support it.

We use an agile process with a lot of influences from the “Lean”:http://en.wikipedia.org/wiki/Lean_software_development and “Kanban”:http://leansoftwareengineering.com/2007/08/29/kanban-systems-for-software-development/ world. We use “Mingle”:http://studios.thoughtworks.com/mingle-agile-project-management to run our development process. Everything, OK mostly everything is covered by automated tests, so we can change things as needed.

We’re all Apple in the office – developers get a MacPro with a nice 30″ screen, and a nice 17″ MacBook Pro.  We deploy on Ubuntu servers.  Aeron chairs are cliché, yes; but, very comfy.

The environment is chilled out… you can wear shorts and sandals to work…  Very flat organization, very non-bureaucratic… nice open spaces (no cubes!). Lunch is brought in on most days! Beer and snacks are always in the fridge.

We’re walking distance to the San Antonio Caltrain station (biking distance from the Mountain View Caltrain/VTA lightrail station).

What’s in it for you

• Competitive salaries, and lots of stock-options
• Cutting edge technology stack
• Fantastic business opportunity, and early-stage (= great time to join!)
• Developer #5 – means plenty of influence on foundational architecture and design
• Smart, full bandwidth, fun people to work with
• Very comfortable, nice office environment
• We have a “No Assholes” policy

OK!

So, if you’re interested, email us at jobs@runa.com

No recruiters please!

We would prefer folks who are already in the Bay Area (but if you not local and are really great let’s talk!)

The post Want to work at a Startup with Cool Tech? (HBase, Clojure, Chef, Swarms, Javascript, Ruby & Rails) first appeared on Cognizant Transmutation.

The folks who developed iClassify have come out with a total framework that is an alternative to iClassify/Puppet called opscode-chef. I will be looking into that soon and not do anymore work with iClassify (unless Chef turns out to suck or something, but at first glance it looks pretty good!).

[After playing with Chef, my conclusion is forget about iClassify / Puppet and just use Chef unless you are already using Puppet]

iClassify Description

From the creator’s of iClassify HJK Solutions website:

iClassify allows for the easy registration and classification of nodes. Most of the time, a node is a server. With iClassify:

• Nodes register themselves with a central web service, including reporting Facter facts.
• You can then tag those nodes, and add manual attributes.
• You can search the nodes with a full text search engine
• You can write recipies for icagent to auto-classify and auto-attribute your nodes.
• You can tie it in to Puppet as an external node classification tool, enabling you to easily configure hundreds of nodes at a time.
• You can tie it in to Capistrano, and have a dynamic ad-hoc configuration tool.

We are considering using it along with Puppet and Amazon EC2 for deployment of some of our infrastructure.

Install iClassify

Original Instructions are at HJK Solutions. Some of the following quotes liberally from that site but adds the things I learnt along the way as well as how to do it on a Mac.

Prerequisites

Ruby Gems

• Rails 2.0.2
• Rake
• Builder
• UUID Tools
• Mongrel
• Highline
• Net-LDAP

sudo gem rails rake install builder uuidtools mongrel ruby-net-ldap

Non-Gems

• MySQL
• Factor
• Git
• Java
• Runit
• Mongrel Runit

Configure MySQL

The folks at HJK say they use MySQL and it should work with PostgreSQL and sqlite3 as well. We went wtih MySQL.
First create the database iclassify_production

mysqladmin -u root -p  create iclassify_production

Grant it the correct privleges (set yourpass to the password you want to use):

mysql -u root -p iclassify_production
mysql> GRANT ALL ON iclassify_production.* TO 'iclassify'@'localhost' IDENTIFIED BY 'yourpass';

You should have the ruby mysql gem installed

sudo gem install mysql

on the Macintosh OS X Leopard I had to say:

sudo env ARCHFLAGS="-arch i386" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Git

You can install from ubuntu packages

sudo apt-get install git-core

On the Mac Download and install the git OS X package for the Mac from Git OSX Installer on Google Code or

port -uR install git-core

Java

Java is needed for the Solr package that is bundled in the iClassify distro. It can be installed via apt and/or downloaded from the Sun site. Java is already installed on Mac OS X.

Runit

Runit is an alternative / addition to the standard /etc/init.d “systemV” init system. I guess the HJK folks like it and seem to have dependencies on it. They say it should work without it, but I haven’t tried putting this together without the Runit/Mongrel_runit dependencies yet.

Some good info and tips on setting up / running runit on various systems can be found at runit – a UNIX init scheme with service supervision

Runit can be safely installed with apt. It will not replace the standard init system.:

sudo apt-get install runit

For the Mac:

port install runit

On the Mac, you’ll have to start the runit system with the command

sudo launchctl load -w /Library/LaunchDaemons/org.macports.runit.plist

iClassify itself

In a directory you want to keep the source of iclassify:

git clone git://git.hjksolutions.com/iclassify iclassify
cd iclassify

You need to know where you want to install the actual working rails app of iclassify and what user id/group you want to run it under.

The default location and the one we’ll use on Ubuntu is /srv/iclassify and the user id/group is usually the same as the one that runs the apache web services (www-data). Change yourpass to the password used for the iclassify user in MySQL.

sudo rake iclassify:install ICBASE=/srv/iclassify ICUSER=www-data ICGROUP=www-data DBUSER=iclassify DBPASS=yourpass

For the Macintosh:

sudo rake iclassify:install ICBASE=/usr/local/iclassify ICUSER=_www ICGROUP=_www DBUSER=iclassify DBPASS=yourpass

This will create a new iClassify instance in /srv/iclassify, set the right ownership to run iClassify, and set run the migrations to prepare your database instance. I found that I had to run this as root so that it will create the directories properly.

You can test that the iClassify rails app was installed properly by testing it with the built in Rails Server (on the Mac use the /usr/local/iclassify directory and _www user id):

$ cd /srv/iclassify
$ sudo -u www-data env RAILS_ENV=production ./script/server
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails application starting on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with production environment...
** Rails loaded.
** Loading any Rails specific GemPlugins
** Signals ready.  TERM => stop.  USR2 => restart.  INT => stop (no restart).
** Rails signals registered.  HUP => reload (without restart).  It might not work well.
** Mongrel available at 0.0.0.0:3000
** Use CTRL-C to stop.

You can now point your browser to the local instance of iClassify at http://localhost:3000, hit CTRL-C when you are done to terminate the script/server.

Mongrel Runit

You can download the mongrel_runit gem from the Mongrel Runit page at HJK

Then install it with (on the Mac you’ll have to use the /usr/local/iclassify directory instead of /src/iclassify):

For some reason the HJK folks set their runit_service_dir to be /var/service but the runit ubuntu package puts it in /etc/service. So you might want to edit /srv/iclassify/examples/mongrel_runit_iclassify.yml and set runit_service_dir to /etc/service. You can also change the number of mongrels you want to run in that file. Similarly, the DarwinPorts port install of runit expects it to be in /opt/local/var/service. You can change the /srv/iclassify/examples/mongrel_runit_iclassify.yml or you can make a symbolic link from /opt/local/var/service to /var/service

sudo gem install mongrel_runit-0.2.1.gem
sudo mkdir /etc/mongrel_runit
sudo cp /srv/iclassify/examples/mongrel_runit_iclassify.yml /etc/mongrel_runit/iclassify.yml
sudo mongrel_runit -c /etc/mongrel_runit/iclassify.yml create

You should be able to then run the command

mongrel_runit -v status -c /etc/mongrel_runit/iclassify.yml

and see something like (there should be as many lines as you have set for mongrel servers. I changed the iclassify.yml from 5 to 3):

5000: true: run: /etc/sv/mongrel-iclassify-5000: (pid 4403) 119s; run: log: (pid 4402) 119s
5001: true: run: /etc/sv/mongrel-iclassify-5001: (pid 4401) 119s; run: log: (pid 4400) 119s
5002: true: run: /etc/sv/mongrel-iclassify-5002: (pid 4399) 119s; run: log: (pid 4398) 119s

Solr

First create some directories that will be needed for Solr to run its index as the www user (on the Mac replace /srv with /usr/local).

sudo mkdir -p /srv/iclassify/vendor/plugins/acts_as_solr/solr/solr/data/production
sudo chown -R www-data:www-data /srv/iclassify/vendor/plugins/acts_as_solr/solr/solr/data/

You can then test it with the command:

sudo -u www-data env RAILS_ENV=production rake solr:start

It should start and run with no errors.

You can stop it if you want with:

sudo -u www-data env RAILS_ENV=production rake solr:stop

Then set up runit to run it automatically (use /usr/local instead of /srv on the Mac and /var/service or whatevever your system uses for runit service dir if not on ubuntu and using /etc/service):

sudo mkdir -p /etc/sv/iclassify-solr/log/main
sudo cp /srv/iclassify/examples/solr-run /etc/sv/iclassify-solr/run
sudo cp /srv/iclassify/examples/solr-log /etc/sv/iclassify-solr/log/run
sudo chmod a+x /etc/sv/iclassify-solr/run /etc/sv/iclassify-solr/log/run

The following will start the solr process immediately as well as in the future reboots

sudo ln -s /etc/sv/iclassify-solr /etc/service

Apache

iClassify is best configured as a virtual host under Apache, running with SSL and mod_proxy_balancer. Follow the proper steps for configuring your platforms Apache to use mod_ssl, mod_proxy_balance r and mod_rewrite. Create a virtual host config which resembles the following (works with Mac and Ubuntu, On the Mac just change the refs to /srv to /usr/local and the EXAMPLE.com to your domain in any case)

<VirtualHost *:443>
  DocumentRoot /srv/iclassify/public
  LimitRequestBody 8388608
  ServerName iclassify.EXAMPLE.COM
  ServerAlias iclassify
  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>

  <Location /server-status>
    SetHandler server-status
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1 192.168.0.0/255.255.0.0
  </Location>

  <Proxy balancer://iclassify>
    BalancerMember http://localhost:5000
    BalancerMember http://localhost:5001
    BalancerMember http://localhost:5002
    BalancerMember http://localhost:5003
  </Proxy>

  LogLevel info
  ErrorLog /var/log/apache2/iclassify-error.log
  CustomLog /var/log/apache2/iclassify-access.log combined

  RewriteEngine On
  RewriteLog /var/log/apache2/iclassify-rewrite.log
  RewriteLogLevel 0
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]
  RewriteRule ^/server-status$ /server-status$1 [L]

  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RequestHeader set X_FORWARDED_PROTO 'https'
  RewriteRule ^/(.*)$ balancer://iclassify%{REQUEST_URI} [P,QSA,L]
  ProxyPassReverse / balancer://iclassify
  SetEnv proxy-nokeepalive 1

  SSLEngine on
  SSLCertificateFile /etc/apache2/ssl/iclassify.crt
  SSLCertificateKeyFile /etc/apache2/ssl/iclassify.key

  BrowserMatch ".*MSIE.*" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
</VirtualHost>

<VirtualHost *:80>
  DocumentRoot /srv/iclassify/public
  LimitRequestBody 8388608
  ServerName iclassify.EXAMPLE.COM
  ServerAlias iclassify

  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^/(.*) https://%{SERVER_NAME}/ [R,L]
</VirtualHost>

Basic instructions for setting up ssl certificates can be found at Just Samuels blog post HOWTO: Create a self-signed (wildcard) SSL certificate

Take the resulting hosts.cert and copy it to /etc/apache2/ssl/iclassify.crt and hosts.key to /etc/apache2/ssl/iclassify.key (or whereever you put your ssl keys and make sure the SSLCertificateKeyFile and SSLCertificateFile are set the same in your vhosts conf file.

Conclusion

That should get you up and running with iClassify. In a future post I will install Puppet and then figure out how to use these together to deploy to Amazon EC2.

The post HOWTO: Install iClassify on Ubuntu and Mac OS X Leopard first appeared on Cognizant Transmutation.