Clearspace Server Syndication Feed

Four Fundamental Differences Between TwinFin and Exadata

pfrancisco — Mon, 02 Aug 2010 23:56:30 GMT

Today Netezza is launching a new eBook entitled, “Oracle Exadata and Netezza TwinFin™ Compared”. As the name implies, this eBook provides a comparison of the Netezza TwinFin data warehouse appliance and Oracle’s “appliance-like” database machine offering.

Certainly Netezza is not the first company to compare/contrast its flagship system with Oracle’s most recent entry. Richard Burns, a consultant over at Teradata did a laudable job exposing the technical shortcomings of the Exadata v2 machine as they pertain to data warehousing in a May 2010 whitepaper. And there have been several recent pieces written on Oracle’s apparent success although the publicly named customer-list has struck some as a bit underwhelming.

Netezza continues to compete (and win) against Oracle regularly in the marketplace, including in competition with the Exadata v2 product and so, we felt it was high time to put our own comparison story together with today’s eBook and with this little blog posting. Let me know what you think.

So where to begin? Let’s start with the fact that the Netezza TwinFin is built to excel at a specific purpose – as the best price/performance platform for Data Warehousing and Analytics in the market. Conversely, Oracle has tried to “kill two birds with one stone” in the Exadata v2 – aiming it primarily at the On-Line Transaction Processing applications space, but also making bold claims to performance as a Data Warehouse with it’s Sun-based Oracle Database Machine (DBM) and Exadata Storage Server, version 2 (Exadata).

So why does it matter that Oracle is aiming to do both OLTP and DW in the same system – apart, that is, from at least two decades of people trying-and-failing to do exactly that with the likes of Oracle in previous software and hardware instantiations? Let’s start with the workload requirements of the two application areas:

OLTP systems execute many short transactions, typically of extremely small scope (touching only a handful of records) and in extremely predictable, well-understood access and query patterns. They need to excel at handling these small transactions in very high volume, combined with equally small writes to the database in the form of updates, insertions and deletions. This limited scope, high throughput and “regularity” of the access patterns make OLTP systems great candidates for intelligent caching and (multiple) secondary data structures, such as indices to speed their processing.

Conversely, DW systems are typically asked to perform “read-heavy” queries and operations against the current and deep historical data sets. Rather than analyzing just a few records, a DW query might look at millions, even billions, of rows from a single table, combined with join logic with multiple other tables. Data warehouse systems are used by company analysts and managers to find the “needle in the haystack” in guiding enterprise decision-making in a more comprehensive and often ad-hoc manner – frequently mitigating the ability to use “tricks of the trade” such as results caching and/or indices.

So the two applications tend to lead to very different system/platform implications. No special “news” there – as I said earlier, people have been trying-and-failing to use a single system for both applications for years.

Without stealing any more of the thunder of our electronic publication today, let me just lay out what I believe are the fundamental differences between Netezza’s TwinFin and the Oracle Database Machine/Exadata as simply and plainly as I can:

Netezza TwinFin	Oracle Database Machine / Exadata v2
True MPP	Hybrid "SMP-plus" Approach
Data Streaming with a Hardware Assist	CPU-intensive Processing for Basic DB Operations
Deep Analytics Processing	Central Cluster-based Approach
No-Tuning-Required Simplicity	Complex Array of Knobs and Levers

In my view, these are "big deal" differences. They're not the result of a simple feature gap to be closed in an upcoming point-release, but rather go directly to limitations at the heart of the Oracle DBM/Exadata system architecture and/or business culture. To address them would require a major rearchitecting, or at least refactoring, of Oracle's decades-old DBMS code base. They also happen to be highly visible to customers and prospects, which makes for some interesting comparisons in head-to-head on-site Proofs of Concept (POCs).

1) True MPP vs. a Hybrid "SMP-plus" Approach

Netezza’s TwinFin uses a full MPP approach to data warehousing, pushing all of the processing down as close as possible to where the data is stored and maximizing the processing horsepower of MPP for scalability, throughput and performance – for even the most complex workloads. Using the MPP method of dividing the workload and attacking query problems in parallel, Netezza has been able to demonstrate market-leading data warehouse price-performance across four generations of data warehouse appliances.

Oracle’s DBM/Exadata takes a hybrid approach adding Exadata Storage nodes largely to handle data decompression and predicate filtering tasks, but still relying primarily on the SMP cluster of Oracle RAC to handle most of the data warehouse tasks, including complex joins. In addition the SMP cluster also must act as the central distribution point for any data that needs to be redistributed between and across Exadata nodes. To try to minimize this, Oracle and Sun’s solution was to “throw hardware at the problem” (quoting Teradata’s Mr. Burns), over-engineering interconnections, processor rates and other elements required because of all of this data movement, rather than refactoring and solving a fundamental software architecture issue.

The difference between the two is akin to an 8-lane continuous streaming superhighway in the TwinFin instance versus multiple freeways converging on and necking down to a two-lane country road via a “traffic roundabout”. I live in Massachusetts and can attest to the negative impact of taking multiple highways down to a single road – it happens every weekend at the gateway to and from Route 6 on Cape Cod.

2) Data Streaming with a Hardware Assist vs. CPU-intensive Work for Basic DB Operations

In addition to the advantages of the MPP architecture for data warehousing, the TwinFin system makes use of hardware acceleration for increased query and analytics performance. Coming in the form of the "DB Accelerator" that is part of each S-Blade in the TwinFin system architecture, providing four dual-core Field-Programmable Gate Arrays (FPGAs) on each DB Accelerator, this hardware acceleration takes care of fundamental processing steps such as decompression, predicate filtering and ACID-compliant data visibility at the full scan rate of the data from disk. The fact that this device is placed as close as it is to the disks for which it is performing its processing gives the TwinFin system much more performance leverage because data can be filtered, processed and value-added before undergoing any unnecessary CPU processing or having to be transported across an expensive network.

And the fact that it is a field programmable device means that Netezza can use it to introduce additional features and performance through a simple upgrade to our NPS software/firmware – as Netezza has with the introduction of two phases of hybrid column/row-level compression technology (with Release 6.0, scaling as high as 32:1 compression, depending on data patterns) first introduced in 2005, and our high-performance implementation of row-level security. Because it's performed in the FPGA in TwinFin, "Compression = Performance"; so if a customer's data is compressed by a 4:1 factor, the effective data streaming rate for processing queries is increased four-fold.

Conversely, the DBM/Exadata system relies entirely on CPU processing. In fact, the great majority of the functionality provided for by the Exadata nodes in the DBM/Exadata system is to replicate the functionality included in each FPGA core of the TwinFin - data decompression and predicate filtering. Because of the CPU-intensive nature of decompressing data in the DBM/Exadata system, Oracle "strongly suggests" lesser compression when data is required for high-performance data warehousing vs. "cooler" queryable archive purposes. Again, the heavy-lifting for query processing and analytics is left to the central SMP cluster nodes rather than parallel Exadata nodes, forcing Oracle to "throw hardware at the problem".

3) Deep Analytics Processing vs. Central Cluster Analytics

Netezza brings analytics to where the data is stored – as close as possible to where it is stored to do the processing – not just to decompress it and do predicate filtering, but to complete as much of the complex analytics as is possible, in parallel. That’s as true of the “traditional” OLAP analytics of SQL-based data warehousing as it is of the advanced and predictive analytics enabled by the new capabilities of i-Class in the “Second Wave of TwinFin”.

With i-Class, Netezza introduces a comprehensive, scalable and high-performance approach to advanced analytics for both our customers and partners, spanning Linear Algebra/Matrix manipulation, and engines for R and Hadoop along with several programming languages including C, C++, Java, Python and even Fortran. The i-Class functionality also offers plug-ins and packages for the Eclipse IDE and R GUI, and pre-built, analytic functions engineered to deliver performance at scale spanning data preparation, mining, predictive analytics and spatial functions together with access to analytics functions from the GNU Scientific Library and R CRAN repository. Extended by the i-Class embedded analytics capabilities, TwinFin allows our partners and customers to push-down applications, functions and algorithms going well beyond standard set-based SQL, at scale with high performance, freeing them of the latency and sampling requirements demanded by off-board processing platforms for advanced analytics.

The Oracle DBM/Exadata performs the majority of the OLAP analytics in the central cluster (RAC) nodes, after traversing the "traffic roundabout". And apart from basic scoring functionality, virtually ALL of the advanced analytics are performed in the cluster nodes as well. Placing the predominance of processing in the central SMP cluster means that both the functionality and scale of the analytics are limited by the capacity and performance that the SMP cluster can provide - typically limited to the elements included in Oracle's own "Data Mining" package.

The DBM/Exadata’s requirement for shipping the data from the storage arrays to the central cluster for analytics is akin to backhauling full massive truckloads of materials from a mining site to pick out the gold at a central headquarters rather than sifting out the most important nuggets in parallel and sending only those valuable elements back in the case if TwinFin.

4) No-Tuning-Required Simplicity vs. a Complex Array of Knobs and Levers

For a long time, the simplicity of the Netezza data warehouse appliance has shone through most strongly in the extremely limited tuning requirements it imposes on administrators of the system, particularly as compared to Oracle-based systems. Simplifying the system management is core to Netezza’s “appliantization” of the data warehouse and analytics platform. Rather than managing a “coordinated collection” of technology assets, the system and database administrators of TwinFin interact with a single appliance and use the redundant Linux-based SMP host nodes as the interaction point for all activities. Everything from database configuration, data distribution, data mirroring, monitoring, software upgrade and day-to-day management are simplified (in the words of one TwinFin customer, “It’s Netezza-easy – it just works.”).

No indexing is necessary (or even supported) in TwinFin to achieve high performance. Just about the only requisite “tuning” of the system is the definition of the distribution key for spreading data across all the S-Blades – typically the primary keys of the tables. Even in the internal management structure of TwinFin, our system management has been configured to get the maximum performance from the commodity subsystems (blades, chassis, disk arrays and network) by connecting them in novel ways and then managing them at a system level, rather than at the subsystem or rack-level.

While it is true that Oracle has simplified some of the tuning knobs and levers in the DBM/Exadata, prospective customers should ask them if they really have moved into the domain of requiring only a small handful of tuning knobs & settings; or whether they still require, or more colloquially, “strongly suggest” the use of dozens or even hundreds of settings (depending upon the number of objects being maintained and optimized). How many dozens of IP addresses are needed to configure and manage the DBM/Exadata (TwinFin requires only two)? Oracle even have a special service to help DBM/Exadata customers migrate and tune their systems and databases for performance and some of their leading Performance Architects even talk about the requirement of using functions like the Oracle SQL Tuning Advisor as an inevitable fait accompli.

By Oracle’s own admission, the time-savings that customers can expect to achieve in managing and tuning the DBM/Exadata system in Oracle 11g r2 is only 26% less than in Oracle 11g. Contrast that with installation after installation of Netezza appliances where 100s of terabytes of data under management in a data warehouse(s) are being maintained by two or even less then one FTE, rather than a team of Oracle specialists. It all depends on one’s perspective and philosophy in building a real appliance for the data warehouse market. Where others may see the need to tune, partition, index and sub-index data sets for performance purposes as an inevitability, Netezza sees that same need as reason to enhance TwinFin’s capabilities in order to obviate it.

All of this really adds up quickly to a significant price-performance advantage for customers of TwinFin – and with our limited tuning and simplified operations, also translates into much more rapid time-to-value for Netezza’s customers, too. So that’s it – four simple fundamental differences that really set the TwinFin appliance apart from the DBM/Exadata. Agree? Disagree? Let me know what you’re thinking. And now, go over and have a look at today’s eBook release for the rest of the story.

Hadoop & Netezza: Synergy in Data Analytics – PART 2

pfrancisco — Wed, 21 Jul 2010 01:04:26 GMT

I mentioned in my previous post that Netezza is excited about our partnership with Cloudera and Hadoop because we’ve already seen some of our customers benefit from the synergy of Hadoop and Netezza TwinFin™ technologies working together.

As I noted, these types of strategies play to the strengths of both technologies and roughly break down into two categories: 1) the use of a Hadoop Cluster for data ingestion, and 2) using a Hadoop Cluster for long-term data retention, which I’m addressing today.

Netezza TwinFin with a Hadoop Cluster Used for Queryable Archive Analytics

The second pattern we have seen customers deploy is one in which the Hadoop Cluster is used for long-term data retention, or as a “queryable archive”. Here one could think of Hadoop as a complementary analytic extension of the Netezza TwinFin when there is far less premium placed on low-latency or high-performance. In addition to the weblog and unstructured data analysis discussed in Pattern 1, the queryable archive could also retain long-term copies of structured data that had previously been loaded into the high-performance TwinFin appliance.

Hadoop Cluster Used for Queryable Archive

With a mix of structured, semi-structured and unstructured data loaded across the two complementary systems, customers can alter the level of granularity and data retention periods across each and typically use TwinFin for processing “hot” data and the Hadoop Cluster for processing “cool” or “cold” data, perhaps with specialized analytics. A deployment of this pattern could look like the following diagram:

Readers should view this pair of posts as a “point-in-time” look at the market. Our customers continue to innovate and make use of the complementary strengths of TwinFin and Hadoop. And Netezza will continue to innovate both inside the appliance – adding performance, scale, workload management capabilities and especially with the advanced analytics of i-Class, through partnerships like the one announced with Cloudera a week ago, and through expansion of our platform, software and virtualization capabilities beyond the TwinFin and Skimmer™ appliances. Those innovations should help alter and/or enhance some of the deployment directions discussed here.

Now, as I said at the outset of these two posts, I’d like to hear from you on your Netezza & Hadoop co-existence deployment and/or compatibility wish-list ideas. What would you like to see?

Hadoop & Netezza: Synergy in Data Analytics Results in New Customer Deployment Trends – PART 1

pfrancisco — Tue, 20 Jul 2010 02:43:27 GMT

Two things before I begin:

I’ll begin this posting with a call for inputs. Below I will list a few of the most common Hadoop/Netezza co-existence deployment patterns we have seen to date. But I would like to hear from others. As you see the continuing deployment of Hadoop in the enterprise and as the Second Wave of TwinFin™ comes on with the advanced analytics capabilities of i-Class, how do you see the evolving deployment patterns happening in your environment?

A special hat-tip to Krishnan Parasuraman, Netezza’s Chief Architect for our Digital Media group, for his excellent help in aiding and abetting this post! I have used his guidance gratefully and (with his permission) stolen freely from some of his inputs.

You may have noticed a partnership announcement made by Cloudera and Netezza late last week. Together with Cloudera, Netezza will open up data movement and transformation between Cloudera’s Distribution for Hadoop and the Netezza family of appliances applications and data flows for integration of the two systems. We expect that our partnership with Cloudera, together with the Hadoop support in Netezza’s i-Class™ set of advanced analytics capabilities that are included as part of the upcoming release 6.0 software release, will lead to some very innovative and expansive applications for our customers and for both companies.

Even today, Netezza customers are doing some very interesting things with deployment of Hadoop and our TwinFin data warehouse appliance. Far from being the “Hadoop v. SQL” battle that some people might like to make the current market out to be, we have instead noticed a growing number of “co-existence” deployment strategies and design patterns already at work with our customers – particularly among customers in the “Digital Media” vertical market.

These types of strategies can play to the strengths of both technologies and roughly break down into two categories: 1) the use of a Hadoop Cluster for data ingestion, which I’ll write about in further detail today; and 2) using a Hadoop Cluster for long-term data retention, or as a “queryable archive,” for which I’ll go into further detail in a post later this week.

Using a Hadoop Cluster for Raw Data Ingestion

The use of a Hadoop Cluster as the engine for data ingestion is the most common “co-existence” pattern we see in our customers’ mutual deployments of Hadoop and Netezza. The deployment pattern typically arises when the customer has hit specific performance and processing throughput scalability limitations with their existing Data Integration or ETL implementation.

Raw weblog data is the primary data source for most Digital Media analytics and reporting requirements. Weblogs are data rich (e.g., page views, impressions, click-throughs and demographics collected from applications servers). They are typically semi-structured and collected and stored in flat files.

There are some critical facts about weblogs that present real performance challenges in processing them:

sheer volume: millions of rows of weblog data collected throughout the day and loaded daily into the data warehouse;
complex query processing: parsing and decoding encoded character strings requires text processing, pattern matching, tokenizing type capabilities within the ETL process
non-conformed dimensions: collecting page views or impression data defined and represented differently by various systems makes fitting them into conformed dimensions is another very common data ingestion & processing challenge.

There are two common variants of this pattern – dealing with semi-structured (e.g., weblogs) and unstructured (e.g., text) data and often customers will have versions of both variants in operation simultaneously.

Semi-structured data ingest via Hadoop

Semi-structured data is parsed (and possibly aggregated as well) in the Hadoop Cluster and then loaded into a TwinFin where the performance and workload scaling of the appliance is important for deeper analysis, higher throughput and faster reporting.

Unstructured data ingest via Hadoop

Unstructured data in this pattern is contextualized (classified, mined, keyworded and indexed) in Hadoop and then moved into a Netezza TwinFin appliance for the low-latency, high-performance analytics used to drive business decisions.

A Hadoop Cluster provides a scalable ingestion mechanism that is well suited for addressing the challenges described above. The Cluster can be incrementally scaled to handle ingesting the massive volumes of weblog data and it can support text processing and complex data processing through programming languages such as Java or Python. [Note that with the coming i-Class set of analytics functionality, the programmability and some of the complex data processing may also be possible on the TwinFin, depending on a customer’s applications needs or preference.]

Following the data ingest steps, processed weblog information is brought into TwinFin as atomic event information or as summarized tables, depending on the size of the appliance and analytic maturity & scale of the organization where it is deployed. A typical deployment might look like the following diagram:

An alternate, far less common, deployment design of the above co-existence pattern is used by some of our customers. That is the use of an external elastic MapReduce cloud (such as the Amazon Cloud) for the data ingestion purposes.

In cases where the customer may have its application servers in the Amazon’s EC2 cluster, they may also choose to use Amazon’s S3 web services for retaining weblog data. In that case, Amazon would provide the elastic MapReduce infrastructure for the data ingest process into the TwinFin appliance. This alternative deployment scenario would look something like the following:

The bottom line is that the different strengths of TwinFin and Hadoop lend themselves to complementary deployments – and some of our customers have already discovered innovative ways to leverage them together to maximize the value of both their investments.

In my next post, I’ll discuss the second pattern we’re noticing: one in which Netezza customers are using the Hadoop Cluster for long-term data retention.

EMC Swallows a Green Plum?

pfrancisco — Wed, 07 Jul 2010 05:18:39 GMT

News broke on Tuesday that EMC plans to acquire Greenplum to focus on data warehousing and analytics on “big data”. The idea is that by doing so, EMC is officially throwing its hat into the competitive ring for the ‘Data Warehouse Appliance’ (DWA) market – something of a defensive mechanism now that virtually all of the major data warehouse vendors are now selling their own versions of a DWA – and consequently greatly reducing sales pull-through of EMC storage for data warehouse deployments.

Some referred to the merger as “a good fit for a storage vendor with appliance-y ideas” and others hailed it as follows, “the market has shifted as of late moving toward integrated appliances and this move gives EMC a very important arrow in its quiver” and labeled Greenplum as a purveyor of “very high performance database systems”.

One can also reasonably assume that this acquisition not only is intended to shore up a product offering weakness, but that it is also destined for affiliation with EMC’s other major initiative announced earlier this year – the Acadia Virtual Computing Environment (VCE) Joint Venture with Cisco Systems and headed up by Michael Capellas. The Acadia JV includes EMC’s storage and its VMWare virtualization software as well as Cisco Systems’ compute nodes and networking. VCE is built on the concept of modular building blocks, called vblocks that marry computing horsepower to storage capacity. All that’s missing from that story is a data warehouse DBMS to make it a full-on data warehouse appliance, right?

There are two big problems with these assumptions…

Performance: For all the discussion about “scale” and “big data” in the EMC announcement, there is no mention of how either party can address the real issues that mainstream enterprises face every single day with their data warehouse systems – how to get maximum performance out of a complex, highly concurrent operational environment where hundreds if not thousands of users are banging away on the system, night and day.

The fact is that the actual Greenplum target market has clearly NOT been one that focused on high-performance analytics over the past several years. Instead, the few wins publicly announced by the company have been for very high capacity, limited compute platforms – applications more commonly referred to as “queryable archive”.
Curt Monash today again mentioned Greenplum’s lack of support for the “high-concurrency” requirements of a mainstream data warehouse.
This looks much more like adding a very basic set of storage-centric data warehousing capabilities in a move to find a broader channel for EMC’s traditional storage products rather than any strategic move into the world of high performance data analytics. Further to this point, neither company has done much of anything to address a very strong trend in the mainstream data warehouse market – the marriage of advanced, predictive analytics into the busy data warehouse systems.
David Vellante confirmed that to be successful the EMC/Greenplum marriage will need to yield, “optimized sytems[sic]; smokin’ fast performance; reference architectures; scale;” and “federation capabilities; not just big honking systems.” We couldn’t agree more but one can’t help but notice that neither Greenplum nor EMC have brought any of those characteristics to market for data warehousing to date.

Appliances: Since the acquisition is fairly transparent in its defense against moves by the likes of Oracle, Teradata and IBM (as well as Netezza seven years ago) to the appliance model, it’s hard to see how either EMC or Greenplum are effectively equipped now to do battle against those established players.

EMC have never really “sold” data warehousing to anyone previously and Greenplum have nearly prided themselves in going after “Greenfield” high capacity applications rather than head-to-head competition vs. established players. And one need look no further than the limited market penetration of H-P’s NeoView to understand that it takes more than simply deep pockets to succeed in the data warehousing market.
Greenplum is not a purveyor of “integrated appliances” and at best, they can hope to infuse in EMC the ability to make their joint product offering a little more of an “appliance-y idea” (hat tip to Dr. Monash for coining the term) to the market. Instead, Greenplum have fashioned themselves over the past several years as a software only solution.
Assume that the Acadia VCE and “vblock” application is a big piece of this strategy. Neither Cisco nor EMC would claim that their servers, networking or storage arrays offer the lowest price-per-bit or price-per-performance alternative in the market. So one needs to think about what that means in terms of the price-performance competitiveness of this new “appliance-y” joint product.

In short, Greenplum joins the pantheon of “interesting” acquisitions for EMC as it will certainly stir some news cycles and drive some analysts and bloggers to create “fresh, new” content; but it’s not really something that I think will register on the Richter scale of customer market share.

Netezza Migrator™: A Point of Clarification

pfrancisco — Sun, 27 Jun 2010 16:55:22 GMT

It may have been the result of a misunderstanding or a comment heard out of context. But whatever the background for the commentary, let me simply state that Netezza is completely committed to the success of the Netezza Migrator and all the other Netezza products and functionality launched at Enzee Universe 2010 this past week. Migrator eliminates a potential barrier to TwinFin™ adoption (i.e., migration costs) and logically should lead to easier acceptance and broader system sales for Netezza. Furthermore, our partnership with EnterpriseDB at both the corporate and technical levels has been and remains extremely solid and strong.

As I stated in the announcement of the product, “The Netezza Migrator product allows organizations to make data warehouse migration decisions independent of proprietary software lock-in. Organizations using data integration and BI applications with embedded Oracle-proprietary database constructs, interfaces and utilities can now more easily manage their migration from Oracle to a TwinFin appliance. The Netezza Migrator will allow our customers to achieve the performance, scale and cost advantages of their TwinFin systems while maintaining their prior investment in proprietary software.” The Netezza Migrator is specifically designed to reduce the time, complexity and costs required of our customers to move their IT applications to the Netezza TwinFin platform.

With Migrator, Netezza’s customers will be able to extract themselves from the dreaded “Oracle lock-in” of functions and procedures written using Oracle-proprietary techniques and they can decide which of their applications to migrate directly to Netezza and just when, at their own pace. Its capabilities go well beyond the extremely limited capabilities provided by Oracle’s own ‘Database Gateway for ODBC’. Migrator provides an Oracle compatible wrapper around Netezza that is optimized in ways that Oracle could never hope, nor deign, to provide with its "Heterogeneous Services" functionality: including support for Netezza syntax pushdown, high speed API, and Netezza user defined functions.

Migrator makes it even easier for Netezza’s customers to move all of their data warehouse from Oracle to Netezza. In short, this is something we feel is extremely valuable for Netezza and particularly “liberating” for our customers.

A Seriously Happy Customer Exposes a Common Dilemma

pfrancisco — Fri, 21 May 2010 05:55:24 GMT

"I cannot imagine life without Netezza." – from a tweet by "noogle" (Twitter, 11 May 2010)

[Photo credit: 2006 photo of “the scramble” intersection in the Shibuya district of Tokyo, courtesy of "Bantosh" and Wikipedia]

Late in May, members of my team and I were in Tokyo's ultra-bustling Shibuya district for a few days on our "worldwide whirlwind training tour" with the global field sales teams regarding the details of the TwinFin i-Class product offering. The late-night scene, hairstyles and outfits there border on the outrageously-hip. There are high-def billboards, electronic gadgets, and of course the bright lights of retailers, bars, clubs and restaurants all through Shibuya. Advances in high technology are virtually 2nd nature to the people there. So with that as the backdrop, imagine the surprise of hearing over a beer or two (see earlier reference to bars & clubs in Shibuya) that a customer "could not live without Netezza".

We're proud of our highly referenceable customer base at Netezza and our "easy to do business with" relationships with our customers and partners. In my six years with the company I've met a pretty fair number of really enthusiastic customers including people who held "welcome" parties for their Netezza systems, use "Netezza" as a verb ("Did you Netezza that data?") or even an adverb ("It's Netezza easy."). But I can't recall any customer who said that they, "could not imagine living without Netezza".

Simple self-promotion is not the real point of this post though. What is is the thread of a dilemma that noogle presents us with in his 40+ word tweet. It's something that business managers and analysts face on a daily basis: what is more important –

being able look for strategic and/or tactical competitive nuggets by performing SQL OLAP analytics on their full, atomic-level dataset; or
looking for that guidance by using advanced analytical toolsets on subsets or aggregations of their data that are extracted from the data warehouse?

Here's the whole tweet by noogle in it's original form:

データが莫大になると分析が不可能になる。少ないデータを複雑なアルゴリズムで分析するよりも、莫大なデータを単純なアルゴリズムで分析する方が有益。統計学とは逆。アホかという量のデータ分析の手助けするのがNetezza。もうあたしはNetezzaの無い世界では生きていけない。

12:47 AM May 11th via TwitBird iPhone

And here's a translation of it into English [parenthetical comments and emphasis are mine]:

When data is huge, complex analytics are impossible. It’s far more beneficial analyzing massive data with simple [SQL] logic, rather than analyzing small data with complicated analysis. This is opposite of statistics [based on sampling techniques]. Analyzing data which is “crazy massive” is Netezza. I cannot imagine life without Netezza.

It turns out that noogle is a long-time user of advanced analytics and predictive techniques. He knows their value, but his tweet exposes of weakness of today's typical analytical environment. By not being able to perform advanced analysis inside the database, most of that work (if performed at all) is done in external servers based on data sets that are extracted (filtered, sampled and/or aggregated) from the data warehouse.

That adds latency to do the extraction and limits the "currency" of the data. Depending on whom you ask, it also limits the accuracy of the results. For instance, looking at aggregations or samples may give you a sense of the "big picture" but not necessarily uncover the needle in the haystack (e.g., fraud detection) or the impact of a long tail that can be exploited in a particular business.

So noogle's choice is to use the analytic horsepower of TwinFin over the sampling techniques. But if one is limited to the set-based logic of SQL, perhaps aided by user-defined functions, you are again limited in the predictive visibility that those tools can provide. Faced with the dilemma this customer chose being able to analyze all the data over statistically sampling and performing advanced analytics on the sample set. Having an answer to that dilemma is precisely what has driven the advent of the i-Class functionality for TwinFin.

We're excited about TwinFin i-Class, but I'm interested in what others may have to say about this. Does your company employ advanced analytical techniques and how have you reconciled the "sampling" versus "full data set" questions in your business? And what are the prospects and pitfalls of doing "crazy complicated analytics on crazy massive data" all in one simple, high-performance data warehouse appliance from your perspective?

Coming soon to a blog near you: nzLabs

pfrancisco — Mon, 17 May 2010 03:05:59 GMT

"Learning to learn is to know how to navigate in a forest of facts, ideas and theories, a proliferation of constantly changing items of knowledge. Learning to learn is to know what to ignore but at the same time not rejecting innovation and research." – Raymond Queneau (1903-1976), French poet, novelist and co-founder of 'Ouvroir de littérature potentielle'.

A bit of “letting the cat out of the bag” here today.

Coming soon to a blog near you is the public surfacing of nzLabs by Netezza, headed up by our CTO Justin Lindsey. The nzLabs team has been the primary driver for the TwinFin i-Class which Netezza announced back in February. As we head toward our Enzee Universe annual user conference in Boston starting on 21^st June, Justin and his team are launching a website and blog to provide insights, information and opinions about the technology side of industry and the directions being taken.

While the nzLabs organization is the “heart and soul” of the innovation engine behind Netezza's deep-dive into the world of advanced analytics with TwinFin i-Class and a fantastic resource for partners and customers alike, that’s not where they stop. The nzLabs engineers and researchers continue to knock down technology barriers and expand boundaries for Netezza.

So through Justin and the rest of the nzLabs team you can expect to hear more about what they're doing to remove those barriers and increase capabilities in TwinFin i-Class as well as topics like getting maximum performance out of analytic algorithms (by parallelizing and "appliantizing" them), development tools and practices, virtualization and broadening the data analytics capabilities for Netezza’s customers in the near future.

Some of Netezza’s customers and partners who have been active in the Netezza D eveloper Network since Netezza’s first entry into the world of advanced analytics in 2007 will be very familiar with a number of the faces of the labs. Today this is a high-powered team that spans across the United States and Europe.

You should notice a characteristically non-marketing style in the blogs and communications from Justin and the nzLabs team. That’s not an accident. It reflects Justin’s no-nonsense approach to technology, problem solving and his care for the craft of engineering & design. It’s also a continuation of a trend that we set out on at Netezza more than two years ago to be much more open in our communications with partners, customers and analysts about what our capabilities, plans and directions were.

I’ll leave it up Justin to tell you more about nzLabs itself and to introduce you to some of the contributors and what they’re thinking about. But until then, “watch this space” for more news on nzLabs and TwinFin i-Class.

SAS Goes Inside the Netezza Appliance

pfrancisco — Mon, 16 Nov 2009 15:43:14 GMT

“The best vision is insight.” -- Malcolm Forbes (1919-1990), publisher of Forbes magazine, New Jersey state senator and adventure hobbyist.

A couple of big announcements from our friends at SAS today. For the industry at large, SAS’ commitment to in-database analytic processing is a confirmation of trends that we have been discussing for over two years: more and more, the “data warehouse” is becoming the hub of all analytics processing for the enterprise. While that announcement covers multiple database vendors, today’s other announcement from Cary, NC on the availability of the “SAS Scoring Accelerator for Netezza” means that we and SAS are immediately putting this recommitted strategy into action.

Of primary importance to Netezza’s customers is the fact that with SAS’ intensification of In-Database functionality, SAS and Netezza will continue working together to deliver ever more advanced analytic capabilities inside the Netezza appliance. And the first step on that path is an excellent one: the availability SAS Scoring Accelerator for Netezza means that Netezza’s customers are able to execute SAS scoring models directly within the Netezza appliance and in-line with other SQL query processing on their data. The SAS Scoring Accelerator for Netezza will be Generally Available in early 2010, and Netezza and SAS are already working with a small number of early adopter customers such as Catalina Marketing, as they begin to benefit from this powerful functionality.

These scoring models are used in virtually every vertical market in which Netezza sells our products for fraud detection, credit and risk analysis and market segmentation. By embedding them in the Netezza appliance, customers will get the same 10-100X market-leading performance on scoring their data as they do on query processing. By running in-database customers can score all their data and not be reliant on only using samples or aggregates for expediency. And the in-database scoring also means that the inherent delays, or latency, in getting at the data to score it has been eliminated. The best way to deal with the large amounts of data being loaded in today’s data warehouse systems is not move it unless necessary, so Netezza’s AMPP architecture and method of moving the data processing as close as possible to where data is stored delivers huge performance gains for in-database analytics.

The on-going partnering work with SAS, and specifically the Scoring Accelerator, are part of the conversation with customers, partners and the market in general that Netezza began back with our Enzee Universe world tour in September regarding our vision for the industry and for Netezza. It’s known as “Netezza Insight” and CEO Jim Baum used his keynote addresses in seven cities around the world to begin the dialogue of taking Netezza and the concept of data warehousing “deeper”, “higher”, “wider” in a “unified” enterprise-wide platform approach together with other partners in the community. In smaller settings with customers, partners and analysts since then, we’ve continued that dialogue since the Enzee Universe and generated real excitement as they come to understand the full breadth of what Netezza is enabling in the market.

In coming days, we’ll be writing more about Netezza Insight and how it is manifest in product platforms, features and applications. But for today, let’s just say that SAS and Netezza customers are already able to do more, faster, with our combined products than ever before and that this is just a step toward even more powerful capabilities.

As Rick (Humphrey Bogart) said in the closing scenes of Casablanca, “I think this is the beginning of a beautiful friendship.”

[UPDATE: Rather than just reading what I have to say, you can watch SAS Executive Vice President and CTO Keith Collins describe his take on the value of in-database processing and the Scoring Accelerator for Netezza in the following video from the Enzee Universe 2009 show in Boston.]

A "FUD-machine" in overdrive?

pfrancisco — Tue, 04 Aug 2009 18:22:33 GMT

A loyal customer alerted us toan Oracle blog by Jean-Pierre Dijcks earlier today that showed the Oracle FUD machine is fully revved-up and ready to go. I'd like to offer a rebuttal, however in the interest of not intruding on Jean-Pierre's entry with an overly-long comment, I've just put a short response on his blog post with a pointer to this one.

Misconceptions and Misunderstandings, or Errors and Plain-old FUD?

I’m writing to correct *just a few* of the misconceptions about what is really important in high-performance, scalable data warehouse systems, errors, or just plain-old pure “competitive FUD” points from Jean-Pierre's posting earlier today. We certainly have posted some information recently about the TwinFin product and Curt Monash’s postings late Thursday provided more info. If his readers are interested in learning more, or even signing up for a “Test Drive”, they should visit www.netezza.com.

First off, I think this is a “banner day” for Netezza. We believe that TwinFin (and the other products in the new product family) extend both our performance and price-performance advantage over our competitors. We stand by our marketing statements that we regularly demonstrate 10-100X performance advantages over our competitors, particularly competitive offerings of the major incumbent DW system vendors (“Just who are those incumbents?” Jean-Pierre's readers may ask. Well let’s just say that we see Oracle as the incumbent system and/or a challenger system in over 50% of our deal flow.).

Regarding his claims about DBM being “faster than Netezza” (and I can only assume he meant at “real” data warehouse tasks) - we’re ready whenever Oracle feels up to actually taking one of their Database Machines onsite to a customer for a fair, open customer benchmark. So far, Oracle have been, shall we say, “a little reticent” to do on-site benchmark testing against Netezza.

Next, given the large number of incorrect points in the original posting, I think perhaps that just a few of them will be useful enough for readers to get the gist of just how far afield some of the ‘facts’ are:

“It all comes down to data scan rates per rack”: Would that it were true that all of data warehousing boiled down to full-stream data scans (as if the entire world of analytics relied on “select count(*) from lineitem” types of queries), then we could all measure “goodness” on how many GB/sec of data could be burst-scanned in our systems. But that’s not the case. So we build Netezza’s data and analytic appliances to deliver the best possible overall performance at the best price and power requirements. As a consequence, and following from those same numbers as-posted, a single rack of TwinFin can process (not just scan) about 400 million rows of data per second. That’s process, as in: “scan, decompress, project, restrict, AND join, etc.”. Need more processing firepower? Netezza’s system performance scales linearly with the addition of more S-Blades: at the low-end, the TwinFin 3 can deliver as much as 100M rows/second of processing horsepower, while the TwinFin 120 can provide you with 4 billion rows/second. Does a system that still relies on using SMP-based servers running “plain old” Oracle 11g RAC scale similarly for data warehousing?

“Non-open Linux running on FPGAs”: I’m really not sure what (if anything) was meant by this, but saying that Netezza’s FPGAs “are apparently running non-open Linux” is oxymoronic on at least two different levels (FPGAs don’t typically “run” an OS and, “non-open Linux” - really?)

“User data & compresssion”: I also enjoyed the accounting of all that “user data” available to DBM users in the Oracle table and the various comments about compression. When Netezza quotes user data capacities in our systems, the numbers reflect real raw user data space, not space that will be further reduced because of required indexes in an attempt to boost performance. Furthermore, Netezza’s compression & decompression techniques allow us to extract “pure performance” from their use. By not relying on CPU cycles to decompress the data before we can process it any further, the FPGA engines decompress the data, on-the-fly, as fast as it streams off the disk drives. Can Oracle make either of those claims?

“Tolerating node failures without downtime”: In perhaps the most bald-faced inaccuracy, the Oracle blog claimed, that Netezza “continues to lack the ability to tolerate node failures without downtime”. This I can only chock up to pure competitive “FUD-ism” as our capabilities in this area have been quite strong throughout the four generations of Netezza appliances and are further strengthened in TwinFin. Netezza is a fully-redundant system with no single point of failure, even in our smallest systems. Failover in the presence of failures of the disk drives, S-Blades, internal networking or host processors (in short, everything) is automatic and done in-service, with hot-swappable replacement throughout.

“Appliance simplicity”: One thing Jean-Pierre didn’t address that might have been humorous to see his take on is the notion of “appliance simplicity” - basically the ability to build, support and maintain large to very large-sized data warehouses, with heavy workloads, with no or minimal tuning, partitioning, indexing or other “performance duct tape” required. Routinely, this capability in the Netezza systems is what delights our customers most and we have customers managing systems with several hundreds of terabytes of user data (not indexes + data, mind you - real data) with fractions of an FTE (full-time employee) devoted to them.

I hope that clears up some of the misconceptions. If any of Jean-Pierre's readers or Oracle customers would like to see or hear more about TwinFin for themselves, we definitely would invite them to come stop by our booth (#207) at TDWI or come to one or our regional Enzee Universe events coming to a location near you.

Shooting the Curl in San Diego

pfrancisco — Tue, 04 Aug 2009 05:52:26 GMT

"You stay classy, San Diego." -- Ron Burgundy (Will Ferrell) in "Anchorman" (2004)

This morning a few others from the Netezza Marketing and Product Management teams and I are ensconced by the Marina in sunny San Diego, CA for the TDWI World Conference and for an news announcement or two. And who better to bring us "Breaking News!" than the Number 1 newsman in all of San Diego, Ron Burgundy. [For those of you who might have been "hoping for more" from Ron in a quote about San Diego, you can check out the IMDB database for some great ones, including Ron's own historical (and hysterical) etymology for the city's name.]

Though it’s not exactly a state-secret at this point, today we’re launching the 4th generation of Netezza data warehouse and analytic appliances and the first of four initial product lines in it: TwinFin™.

Some of the core characteristics of the TwinFin and the overall platform are:

Resetting Netezza’s price-performance leadership position in the market and extending Netezza’s performance lead;
Disrupting the competitive data warehouse market among the incumbents, just as we did with our initial systems in 2003/’04;
Moving to a commercially-available, blade-based server and storage platform; and
Opening Netezza’s aperture on the broader market with a multi-product platform design to match customers’ data warehouse and analytics needs across their enterprise

After the market disruption Netezza caused with the introduction of the NPS® in 2003 and since, we have seen the entry of dozens of new startups in our wake and virtually every major incumbent data warehouse vendor has retooled its portfolio to include a “response” to the Data Warehouse Appliance (DWA) in a suddenly reenergized market. Several of them, to their credit, have advanced their value propositions and improved their competitive position.

Now it is Netezza’s time once again. With the introduction of TwinFin and the other members of the new family of products, Netezza is once again changing the game; widening the applicability of our systems to more types of customers, applications and partners in the market.

As stated in my response to Curt Monash, my response to Curt Monash last week, we think of this 4^th generation of the Netezza appliance as using “the same architecture with a new physical implementation”. Starting with TwinFin, we moved to a commodity blade-server based system framework, but one that still uses Netezza’s “secret sauce” to deliver as much as a 5X increase in performance over the previous generation of Netezza systems, namely:

· our balanced design and streaming architecture;

· the use of Field Programmable Gate Array (FPGA) technology as a query processing “turbocharger”; and

· our advanced MPP management and optimization software.

And there are more innovations and performance gains on the way! TwinFin, quite simply, will serve as a platform for expanding Netezza’s performance and price-performance advantage in the industry and as the basis for advancing the state-of-the-art for in-database, analytically intensive data processing; all without sacrificing any of the appliance simplicity with which our company is synonymous.

As a couple of us said last week, Netezza has served as “the benchmark” for high-performance DWA pricing in the industry and we are now leading “the market in pivoting to a new competitive price-performance level”. With these new systems, we have embraced a trend that has been happening around the industry – the movement of marginal cost of a bit of disk storage toward $0 – with system-sizing, pricing and even system numbering focused on the performance delivered by a given platform.

We think the net effect of the new, simplified pricing structure for TwinFin and the other members of the Netezza product family will create a major disruption in the market. With starting (US-based) prices that equate to under $20,000 per terabyte, TwinFin’s list price is a fraction of other competitors’ performance-system pricing (after they’re all done playing price-obfuscation games around mirror, swap and index storage).

TwinFin and the other new Netezza data and analytic appliance products give us the opportunity to continue to lead the market and provide our customers with the best value and performance possible for all of their data warehouse and analytic processing needs. Netezza TwinFin - because two fins are faster than one.

Change, but no Change

pfrancisco — Fri, 31 Jul 2009 04:00:20 GMT

Just trying to clarify. Curt Monash's informative blog on the coming Netezza system and family of products includes the following:

<snip>

Beyond the switcheroo in components, Netezza is making substantial changes to its hardware architecture. In current Netezza products, the FPGA plays the role of a disk controller on steroids — it receives data, does some SQL or other analytic operations on it, and then throws it over the wall to the CPU for the rest of the processing. The new Netezza product family, however, adds an actual disk controller. More important, it adds fast interconnects between the FPGAs, the disk controller, and RAM — specifically, as Phil Francisco put it in an email,

using multiple parallel channels of PCIe with much faster interconnection rates and lower contention between the blade server and the “DB accelerator card” with the FPGAs.

DMA (Direct Memory Access) technology also fits into the picture somehow.

<snip>

...which seems to beg further clarification.

While Curt suggests big changes are afoot in Netezza's “architecture” - I think a more appropriate viewpoint would be that it's “the same architecture with a new physical implementation”. That is, the concept of data streaming from disk through the system is just as important now as it ever was.

True, we did move the "disk controller" function to a pair of HBA (Host Bus Adapter) cards that interface with the disk enclosures using multiple, redundant SAS (Serial-Attached SCSI), and providing more than ample bandwidth to stream all the drives per rack continuously to the blades. For those who click-thru on Curt's blog, this function is embedded in the device labeled “SAS Expander Module” (one on both the blade server and the "DB accelerator") in the 3rd chart of the PDF file (and also shown above) and allows data to stream from disk through to memory and then on to the FPGA without delay.

To move data between the blade server and the DB accelerator cards, we use IBM's expansion card (formerly known as "sidecar") technology to provide multiple parallel high-speed PCIe (peripheral component interconnect express) channels delivering the data streams from the disk drives to the memory on each blade server and providing very high-speed interconnect between the FPGA devices and that same memory, using DMA (direct memory access) to effect high-speed memory access without encumbering the CPU to get at it.

With all this high-speed interconnectivity, Netezza has been able to alter the data flow so that data streams to the memory first and then to the various FAST engines (see above diagram and/or refer to Issue 16: The Latest Addition to Netezza's FAST Engines Framework) in the FPGA. Those engines act as a "turbocharger" for query processing, implementing data decompression, restricting, projecting and applying the appropriate visibility rules in a pipelined process; typically filtering out well over 95% of the data scanned. From the FPGA, the resulting reduced data set is passed on to the CPU memory for additional processing to complete the process.

So, the logical streaming model of data from from disk to FPGA to CPU is retained, with significantly higher throughput as a result. But there's an added benefit: the fact that the originally-scanned data can remain in memory, still in compressed & unfiltered form, to be used as a cache avoiding disk scan activity where possible and helping boost system performance even more. In short, "Change, but no Change."

I hope that helps - with Curt's architecture viewpoint as well as with questions about our use of PCIe interconnects to raise performance.

"Catch a wave and you're sittin' on top of the world"

pfrancisco — Thu, 30 Jul 2009 20:54:05 GMT

"Don't be afraid to try the greatest sport around

(catch a wave, catch a wave)
Everybody tries it once
Those who don't just have to put it down
You paddle out turn around and raise
And baby that's all there is to the coastline craze
You gotta catch a wave and you're sittin' on top of the world"
– from "Catch a Wave" by The Beach Boys (1963)

Surf's up! Summer seems to finally have arrived in the Boston area and a number of vendors in the data warehousing and analytics space are hoping to catch a wave riding on a flurry of industry announcements. A few trends continue to build in the news:

Data sizes continue to grow alongside the pressure to increase performance & shrink data latencies;
Workload complexity and user counts continue to grow;
More and more, customers are seeing the value of running advanced analytical processing directly in their primary data repository (see item #1 for reasons why); and
Industry prices for data warehousing and analytics have begun another shift downward.

Today I'd like to address this last point. According to more than one industry analyst, over the last several years, Netezza has served as "the benchmark" for DWA pricing in the industry. Several of our competitors have sought to match and/or undercut Netezza pricing in the market. Some of the incumbent players have tried to, with very limited success, hinge their pricing off Netezza prices, match the performance of the Netezza Performance Server® system, or inoculate their pricey "flagship" products by adding less-expensive, feature-deficient products to their portfolio. But Netezza has continued to succeed in the marketplace, becoming a profitable, publicly-traded company with nearly 300 customers and 400 employees worldwide and one that is listed among the "Leaders" in the Gartner Magic Quadrant.

When we disrupted the data warehousing market with our first generation product in 2003 and 2004, Netezza was one of very few startups in an otherwise moribund industry. Now, with established "street cred" and hundreds of loyal customers, we intend to once again upset our competitors and lead the market in pivoting to a new competitive price-performance level. We're about to launch the fourth generation platform of our data warehouse and analytic appliances, which will advance Netezza's performance leadership and once again establish a new price-performance benchmark.

Admittedly, we won't be the first vendor offering high-performance data warehouse systems to move to a lower pricing plateau. That task is usually done by early-stage start-ups looking to find a way to differentiate themselves. True to form, Dataupia probably can claim establishing a lower price point first and recently another multiyear "start-up" has also started lower. But those are offerings from very modestly-sized startups with no established market "track record". Netezza will be the first company with proven product maturity, customer base and financial viability to do so.

Just how and what are we doing to cause this disruption? Well, let's just say things around the "briefing table" have been quite hectic, and that I and others will have more news about that to follow shortly.

[As you might imagine, it's been getting more and more difficult to keep things under wraps – in recent weeks we've even had to fight people off from getting early "sneak peeks". ]

Until then hey, it's summertime! So here's what I'd recommend –

"So take a lesson from a top-notch surfer boy

(catch a wave, catch a wave)
Get yourself a big board
But don't you treat it like a toy
Just get away from the shady turf
And baby go catch some rays on the sunny surf
And when you catch a wave you'll be sittin' on top of the world

Catch a wave and you'll be sittin' on top of the world"

Twin Fin: A short board (usually 5'8" - 6'8") with a wide tail for maneuverability and a fin near each rail for stability in radical turns.

Purpose: A wider tail area provides more planing area and lift, which creates more speed by efficiently utilizing wave energy. Milking speed and energy from smart surf with extremely sensitive and responsive turning ability are this design's strong points

Setting the Record Straight

pfrancisco — Tue, 16 Dec 2008 22:31:19 GMT

We came across a series of blog posts the other day which seemed to insinuate that Netezza and other competitors might be trying to shape our 10-100X performance message on the backs of comparisons to antiquated, end-of-service life systems and not comparing to current competitors' platforms . When we got to this one - "Database Customer Benchmarketing Reports" - I felt we just had to correct the record, so I wrote a response to Greg Rahn's posting to give Netezza's side of the story, namely that

we are as up-front as possible with prospective customers and use the customer benchmark testing/POC process to prove out Netezza's performance, value and simplicity value propositions;
the results of other products' performance come from our prospects/customers and not the result of Netezza running the tests on those platforms;
not only do we test against the incumbent systems, but there is almost always at least one other current competitive system that is included in the POC process;
the PowerPoint deck on which Greg was doing his analysis contained some rather ancient (in enzee-years, anyway) comparisons with versions of the NPS appliance that we have not sold in as much as 4.5 years & was really not much of a data set on which to base his analysis; and
the "proof of the pudding is in the tasting" - Netezza's success rate of converting prospects to customers through the customer benchmarking process remains very strong.

In short - we make every effort to keep the Netezza website contents both accurate and clear and we definitely feel confident in standing by our 10-100X performance claims. It would be great to have more than the Netezza "product marketing guy" clarify things - while I know some of the excellent results recent customers have seen in POC, no one knows them better than our customers & SI partners themselves.

"Netezza Underground" - NPS at your bookstore

pfrancisco — Wed, 26 Nov 2008 19:15:49 GMT

We had quite a surprise the other day when it came to our attention that Netezza and the NPS data warehouse appliance are now the subjects of a new book: Netezza Underground: The unauthorized tales of derring-do and adventures in resilient data warehousing solutions, by David Birmingham (ISBN: 1-4392-0743-7 and now available in paperback version for $31.54 at Amazon.com).

This is not the first instance of the NPS system being the subject of a book sold by Amazon (e.g., SAS/ACCESS(R) 9.1.3 Supplement for Netezza), but this particular publication certainly brought feelings of both fun and reaching into the mainstream with it, starting right from it's very clever cover art (above) to David's clever turns of phrase and real-life examples.

As the title suggests, it was not written or coordinated with any Netezza authorization. So of course we bought a copy and read/skimmed through it as quickly as we could. I will say this, David's self-publication skills are great - he keeps what could easily have been a boring, heavy technical tome both engaging and fun to read while still imparting lots of great information about the NPS system, its performance and its ease of operation. And the book's publication is incredibly current - with references to Netezza Developer Network and "BI Appliance" announcements made only as recently as the Enzee Universe user conference in September.

While I certainly could quibble with a point made here or there about the system, in general I thought it was an excellent book and even put up the following recommendation for it on the Amazon site:

I commend David Birmingham on a book that is at once as lightly entertaining and interesting to read as it is chock full of details about just the kind of performance and operational simplicity that is possible with the Netezza Performance Server (NPS) system. Straightaway from the opening pages, Birmingham's effusive, engaging style and excitement about Netezza's system is apparent, "It inhales, crunches and publishes Libraries-of-Congress-at-a-time - and fast."

He also captures the essence of the NPS appliance in an ultra-succinct two-sentence paragraph explaining just why his "Administration Stuff" chapter is so short, "It's an appliance. Put it in the corner and let it work." I couldn't have said it better myself!

This book is comprehensive and current - even reflecting some of the more recent announcements from Netezza regarding OnStream programmability, the Netezza Developer Network and analytic appliances.

As the guy who is responsible for projecting the Netezza products and our technology direction forward, I want to recommend David Birmingham's book to current and prospective customers and partners alike, or as David himself says on the book's Dedication page, "to Enzees everywhere".

--Phil Francisco, VP Product Management & Marketing, Netezza Corporation

So "to Enzees everywhere", have a read of David's book and welcome to the "Netezza Underground".

Oracle’s Entry to the DWA Community – just some of the Snarkalicious Commentary

pfrancisco — Fri, 26 Sep 2008 03:32:20 GMT

Okay I'll admit that my first posting about the new Oracle Data Warehouse Appliance (DWA) tonight was a tad on the "snarky" side. But I have to say that I think it was because of all influences in the environment all around me. Straight away since the announcement yesterday afternoon, there's been a healthy degree of skepticism from industry insiders.

Beyond his commentary on Larry Ellison's hairstyle, Gavin Clarke of the UK's Channel Register virtually flogged Larry for flogging the "Oracle server appliance alliance with HP". Some of the best snippets included:

Gavin's subtitle: "(Not) a hardware provider"
"And so to chief executive Larry Ellison, who Wednesday afternoon announced Oracle's third effort in 10 years bundling his company's software with someone else's hardware. This time, it's a high-performance, Oracle data and storage server stack locking arms with old favorite Hewlett-Packard."

And after taking several informative paragraphs to expound on Oracle's two previously-failed attempts at ‘appliantization' - most recently the "Network Computer" initiative circa-2000 - to draw the clear analogy to yesterday's announcement, Clarke closed out his piece with this stinger:

"In a telling sign of how much faith Ellison places in his latest appliance, he did not sit down for his traditional, open-mic smack-down session with OpenWorld attendees to field questions."

Analyst/blogger Curt Monash summarized more than a few skeptical digs in his Oracle Exadata and Oracle data warehouse appliance sound bites posting earlier today. For example, here are a few "bites" from Curt's post:

"Oracle Exadata Release 1 is hardly going to put Teradata, Netezza, or Greenplum out of business."
"After long denying it, Oracle has finally admitted that putting more than 10 TB on Oracle had been an extremely painful thing to do."
"Oracle's Exadata-based appliance doesn't have the out-of-the-box simplicity that other appliances and analytic DBMS do."

VP & Global Marketing CTO Chuck Hollis of EMC weighed in with a couple good shots on his Chuck's Blog post: Oracle does hardware (emphasis mine):

"Of course, there's little in the way of performance comparisons to help us evaluate just how fast this beast might go, except the ‘Up To 10x Faster' which smells a bit optimistic, never mind that it's Oracle comparing with itself, rather than other data warehousing appliances."
"Every year at Oracle Open World, we hear about many "new initiatives" from Oracle. Well, not to be harsh here, but it's my impression that very few of them get talked about at next year's Oracle Open World. I routinely dig up past announcements from previous years, and it's relatively consistent pattern. I think it's fair to ask the question -- just how serious is Oracle about all of this?"

But the lead cynic was none other than Oracle CEO Larry Ellison himself. After years of denying performance issues at scale with various generations of Oracle DBMS software for data warehousing, Larry dropped this 11g-megaton bombshell about Oracle's data warehouse scalability, pre-Exadata - laying out the fundamental reason why Netezza has become the industry leader in Data Warehouse Appliances (source: ZDNet's Larry Dignan):

"Ellison, speaking at Oracle's OpenWorld conference, said large databases are creating a fundamental problem: Disk storage systems can't cope with data that has to be moved off of drives to database servers. He called it a ‘data bandwidth problem.'

"As data gets larger the slowdowns become more unbearable. At one terabyte you will notice data bandwidth slippage. At 10 terabytes, storage systems crawl. ‘At one terabyte the problem rears its ugly head and it gets worse every year,' said Ellison."

And that's not all - the barbs, skepticism and "bites" go on in site-after-site, and commentary-after-commentary. So please forgive my snarky-ness - I blame it on the "nuture" of my environment, not my personal "nature", per se.