I especially enjoyed the link to Oracle’s in-database MapReduce example.  Though I find it a really useful tool (using table functions to parallelize workload), it seems a bit odd to consider it a replacement to Hadoop.  I mean, it’s still executing the processes on a completely different architecture (shared storage vs Hadoop’s shared nothing with HDFS replication, fault tolerance, etc).

In any case, check out the article, it’s a fun read.

The first is under the "Analytics" section of the article.  It’s called the LISTAGG function and it’s super-cool.  I can’t think of how many times we’ve wanted to concatenate a list of values into a delimited string – and now it’s really easy to do in 11gR2!  And it’s very functional, allowing the list to be ordered by other columns (whether or not those columns exist in the SELECT clause).  The SQL Language page for LISTAGG has a few simple examples, and Tom’s article has even more, including a comparison with the painful way we had to accomplish the same task in 10g (and 11gR1).

The second new analytic function Tom writes about is also very nice.  NTH_VALUE lets you pick out the Nth entry from a window’s result set.  It’s a nice compliment to the FIRST_VALUE and LAST_VALUE functions (see his example for all three).  Again, the SQL Language page has more details on its usage and a great example.

Finally, one of my favorite new features because it’s one I wish we had in our recent engagements.  At a couple of projects, we were tasked with identifying user sessions across a set of web logs (pretty common and basic stuff – mark a set of records with a common Session ID based on a user cookie’s time on the site, ordering by time, and so on).  Everything in creating sessions is typically centered around a user cookie, so updating the records could benefit from parallelization – we could update a full set of records belonging to a single user cookie as a "chunk", and execute several chunks at the same time.  Now, in our case, the "update" was a bit more complex than a simple DML statement, but if you wanted to chunk data and execute DML per chunk in parallel, DBMS_PARALLEL_EXECUTE may be a huge help.  The package gives us the ability to create chunks (segments of data, based on ROWID or user-defined SQL), and then execute DML (or other) operations on those chunks in parallel, using the Scheduler to perform execution, logging, and error handling.  The Doc page has all the details and examples.  I haven’t tried this myself yet so it may already be possible, but it would be a huge bonus if, instead of basic DML statements, we could execute a Stored Procedure or a dynamic PL/SQL block on each chunk – that would be a huge win in simplifying parallel ETL inside the DB.

There are a bunch of other new features in 11g (R1 and R2) that we haven’t covered yet – several great enhancements that can help in the DW world.  Definitely worth checking out.

The most significant "sell" Vertica (and folks like Exasol) has over other solutions is that it’s product is a column-store (or column-oriented) database, rather than the better known row-store.  Column-store (where "tables" are separated into having one object per column) databases have a significant advantage in the analytics world as a typical BI query tends to filter for only a select set of columns from tables (we typically don’t require all dimension columns from a star query) – that means less data (less I/O) to sift through.  Second, a column-store database will typically get much higher compression since each column object will always be consistent in type and length.  And vendors like Vertica have the ability to perform functions right off the compressed data set (vs. having to decompress first).  According to their benchmarks, the end result is a blazing fast DB for typical analytic queries (note that there is plenty of discussion on when a column-store is not the right choice for an enterprise data warehouse, but that’s for a later blog post).  All exciting stuff …

So that led to checking out what open source alternatives might be out there.  Two of the more talked about column-store databases are Infobright and MonetDB.  Both seem to be pretty active projects, and both seem to be backed by some bigger companies that offer commercial support.  Neither, however, support an MPP architecture at this point.  Infobright has a blog post that seems to indicate they are considering it, but as of now, scalability is upward more than outward.  Not that SMP is necessarily a bad thing; I know it doesn’t have the marketing feel MPP does nowadays, but it really is a more simple approach and we don�t typically see a huge need for it in the data mart world (ODS, maybe, but a client with average volumes won’t see a need for that much processing on summarized data).  Ultimately, it would be nice to have both options and I’d love to see a truly open source MPP database.

That’s where HadoopDB comes in.  Recently released, this project can be the glue that brings it all together.  Here is the description from Daniel Abadi’s release blog post:

"It’s an open source stack that includes PostgreSQL, Hadoop, and Hive, along with some glue between PostgreSQL and Hadoop, a catalog, a data loader, and an interface that accepts queries in MapReduce or SQL and generates query plans that are processed partly in Hadoop and partly in different PostgreSQL instances spread across many nodes in a shared-nothing cluster of machines. In essence it is a hybrid of MapReduce and parallel DBMS technologies. But unlike Aster Data, Greenplum, Pig, and Hive, it is not a hybrid simply at the language/interface level. It is a hybrid at a deeper, systems implementation level. Also unlike Aster Data and Greenplum, it is free and open source."

This is great stuff, especially if "deeper, systems implementation level" means a solution to some of the fault tolerance vs. performance issues Daniel discusses in the article.  Better yet, he claims they can switch out the underlying database (PostgreSQL) to a column-store like MonetDB or Infobright.  Now that would be a huge win!

It feels like we’re going to see a lot of convergence in technologies in the upcoming year, and that’s probably a good thing, at least in the open source world – the more collaboration of projects, the better.  I’d love to see HadoopDB get more coverage in the Hadoop/Apache community – would be great to see this progress along with Hive and Pig.

Overall, it’s exciting to see these (hopefully) lower-TCO solutions maturing – it’s getting harder and harder for some of you to shell out 100k/TB for some of those commercial appliance offerings, so it’s nice to see we have open source some options out there.  They may not be the perfect option just yet, but I see a bright future here, and that’s good for everyone, even for the commercial offerings (competition will drive innovation, and innovation and solid support are what open up wallets).

I haven’t had time yet to do some POCs on the two DB solutions or HadoopDB – I’ll be posting some findings when I do.  But send us your thoughts if you’ve tried these out.  What advantages/disadvantages do you see when comparing to Vertica and other commercial offerings?

Due to a build slippage, binary distributions of MySQL 5.1.12 contained neither NDB Cluster nor Partitioning. We apologize for any inconvenience. Please upgrade to 5.1.14. if you build from source, you can execute configure with the --with-ndbcluster and --with-partition options.

Can some one please explain the phrase “build slippage” to me? I’ll have to ask my QA friends if that’s an industry-specific phrase. And, oh, hey, who missed fixing that in 5.1.13? Or maybe there isn’t a 5.1.13?

It’s ok, MySQL, I’m still a fan and, hey, at least it’s not your GA release. I just want to make sure the main real reason to upgrade to 5.1 (partitioning) is in my binary

Of course, we’re not claiming that’s how it should be done in the long run, just that it may be a way to show the business the true value of BI without emptying their pockets too quickly.

Hope the white paper provides some useful information – it comes our of our real-life experiences, so feel free to write with additional questions/comments (we purposefully did not put too many technical solutions so they can be discussed more on this blog).

Enjoy.

I’m writing this blog post via Deepest Sender, an easy-to-use Firefox extension which logs into our free and very powerful WordPress blog and posts my blogs all from a simple interface. I set up our WP blog and fixed up some of the look and feel in under an hour. Why? Because these powerful tools have been created with simplicity in mind – there is something to be said about the power of an easy-to-use application (keep that in mind, Web 2.0 AJAX folks!).

So thank you, WordPress. Thank you Mozilla and all the extension makers. Thanks Zoho (we love Zoho CRM). Thanks Journyx. Thank you, Trillian creators. And, of course, thank you LAMP folks. Oh boy, so many, many more.

Now, take a moment to pause and thank the hard-working developer next to you. Aren’t they a special breed? I think so.