Thinking Inside the Box

Thinking about Right-Time Analytics in the Big Data Era at TDWI

razi — Thu, 19 Aug 2010 20:18:31 GMT

Netezza Director of Product Marketing Razi Raziuddin is blogging today.

I’ve been at The 2010 TDWI World Conference in San Diego this week, where the theme is "agile BI that delivers data (I would use the term ‘insights’) at the speed of thought.” Timing is everything when it comes to making decisions – and influencing other to make decisions we’d like to see.

We’ve all experienced Red Car Syndrome at some point or another. You test drive a red car. You like it. Suddenly, you start noticing red cars everywhere – not because the number of red cars has increased, but because the experience of driving a red car is now personalized. Online advertisers use Red Car Syndrome to connect consumers with the products they genuinely want, as I was reminded first-hand recently. While searching for kitchen fixtures online, I noticed that many of the ads featured a pair of pricey fixtures that initially caught our eye, but that we had rejected as exceeding our budget. But the ads seemed to know our tastes better than we did, and ultimately we succumbed and made the purchase.

The experience brought home the power of right-time analytics. Speed is critical in making analytics actionable and delivering real value to the business. The trifecta of huge data volumes, complex analytics and query performance is an increasingly common thread in the BI and data warehousing world. It is true not just for online marketers, but cuts across industry lines. Whether it is an insurance provider trying to prevent fraud, a telco determining the cheapest and best path to route a call or a government agency unearthing criminal activity, time to insight from big data makes the difference in every case.

Doug Henschen recently wrote a good article on this topic for InformationWeek in which he calls out success in the Big Data era as the ability to get faster insights from huge data sets. The article highlights Catalina Marketing’s petascale data warehouse environment and the fast insights they derive from a huge database of 195 million consumers.

Although not every enterprise has a data warehouse environment quite that large, the need to perform complex analytics and derive insight in the shortest time possible is common in every environment, big or small. While scalable MPP architectures address the big data problem quite well, the big math problem associated with complex and advanced analytics is what many customers still wrestle with. There’s general agreement that in-database processing, especially in scalable MPP systems, is the right solution to the big math problem. Doug’s article again highlights Catalina’s use of in-database analytics to radically streamline their analytic modeling environment and gain efficiencies of 10X as a result.

However, not every data warehouse platform is geared up for the challenges of performing in-database analytics at scale. The first and obvious challenge is the additional processing overhead required to run advanced analytic algorithms alongside the traditional data warehouse workload. You need a system architecture that is not overwhelmed by the data volumes typical of data warehouses in the Big Data era. Then there is the question of what analytics you want to perform. The majority of commonly available analytic libraries are written for in-memory processing in SMP systems and need to be parallelized in order to take advantage of MPP architectures. The analytic system should not only offer parallelized versions of the analytics you desire, but also provide primitives to easily parallelize advanced analytic algorithms while hiding the complexity of parallel programming from developers.

Finally, the dearth of universally accepted standards in the advanced analytics world poses yet another challenge. A typical analytic environment may consist of a mish-mash of commercially available tools such as SAS and SPSS, open source ones such as R and Hadoop (which are gaining popularity), and tons of application code written in various languages such as Java and Python. The underlying system must offer tremendous flexibility in integrating with a wide array of analytic tools and support for a variety of frameworks and languages.

In subsequent posts, I’ll talk about Netezza’s advanced analytic capabilities to enable big math on big data. In the meantime, as you plan your analytic infrastructures for the Big Data era, tell us what challenges you are coming up against.

Hadoop & Netezza: Synergy in Data Analytics – PART 2

pfrancisco — Thu, 22 Jul 2010 11:06:27 GMT

I mentioned in my previous post that Netezza is excited about our partnership with Cloudera and Hadoop because we’ve already seen some of our customers benefit from the synergy of Hadoop and Netezza TwinFin™ technologies working together.

As I noted, these types of strategies play to the strengths of both technologies and roughly break down into two categories: 1) the use of a Hadoop Cluster for data ingestion, and 2) using a Hadoop Cluster for long-term data retention, which I’m addressing today.

Netezza TwinFin with a Hadoop Cluster Used for Queryable Archive Analytics

The second pattern we have seen customers deploy is one in which the Hadoop Cluster is used for long-term data retention, or as a “queryable archive”. Here one could think of Hadoop as a complementary analytic extension of the Netezza TwinFin when there is far less premium placed on low-latency or high-performance. In addition to the weblog and unstructured data analysis discussed in Pattern 1, the queryable archive could also retain long-term copies of structured data that had previously been loaded into the high-performance TwinFin appliance.

Hadoop Cluster Used for Queryable Archive

With a mix of structured, semi-structured and unstructured data loaded across the two complementary systems, customers can alter the level of granularity and data retention periods across each and typically use TwinFin for processing “hot” data and the Hadoop Cluster for processing “cool” or “cold” data, perhaps with specialized analytics. A deployment of this pattern could look like the following diagram:

Readers should view this pair of posts as a “point-in-time” look at the market. Our customers continue to innovate and make use of the complementary strengths of TwinFin and Hadoop. And Netezza will continue to innovate both inside the appliance – adding performance, scale, workload management capabilities and especially with the advanced analytics of i-Class, through partnerships like the one announced with Cloudera a week ago, and through expansion of our platform, software and virtualization capabilities beyond the TwinFin and Skimmer™ appliances. Those innovations should help alter and/or enhance some of the deployment directions discussed here.

Now, as I said at the outset of these two posts, I’d like to hear from you on your Netezza & Hadoop co-existence deployment and/or compatibility wish-list ideas. What would you like to see?

SAS Goes Inside the Netezza Appliance

pfrancisco — Mon, 16 Nov 2009 16:07:06 GMT

“The best vision is insight.” -- Malcolm Forbes (1919-1990), publisher of Forbes magazine, New Jersey state senator and adventure hobbyist.

A couple of big announcements from our friends at SAS today. For the industry at large, SAS’ commitment to in-database analytic processing is a confirmation of trends that we have been discussing for over two years: more and more, the “data warehouse” is becoming the hub of all analytics processing for the enterprise. While that announcement covers multiple database vendors, today’s other announcement from Cary, NC on the availability of the “SAS Scoring Accelerator for Netezza” means that we and SAS are immediately putting this recommitted strategy into action.

Of primary importance to Netezza’s customers is the fact that with SAS’ intensification of In-Database functionality, SAS and Netezza will continue working together to deliver ever more advanced analytic capabilities inside the Netezza appliance. And the first step on that path is an excellent one: the availability SAS Scoring Accelerator for Netezza means that Netezza’s customers are able to execute SAS scoring models directly within the Netezza appliance and in-line with other SQL query processing on their data. The SAS Scoring Accelerator for Netezza will be Generally Available in early 2010, and Netezza and SAS are already working with a small number of early adopter customers such as Catalina Marketing, as they begin to benefit from this powerful functionality.

These scoring models are used in virtually every vertical market in which Netezza sells our products for fraud detection, credit and risk analysis and market segmentation. By embedding them in the Netezza appliance, customers will get the same 10-100X market-leading performance on scoring their data as they do on query processing. By running in-database customers can score all their data and not be reliant on only using samples or aggregates for expediency. And the in-database scoring also means that the inherent delays, or latency, in getting at the data to score it has been eliminated. The best way to deal with the large amounts of data being loaded in today’s data warehouse systems is not move it unless necessary, so Netezza’s AMPP architecture and method of moving the data processing as close as possible to where data is stored delivers huge performance gains for in-database analytics.

The on-going partnering work with SAS, and specifically the Scoring Accelerator, are part of the conversation with customers, partners and the market in general that Netezza began back with our Enzee Universe world tour in September regarding our vision for the industry and for Netezza. It’s known as “Netezza Insight” and CEO Jim Baum used his keynote addresses in seven cities around the world to begin the dialogue of taking Netezza and the concept of data warehousing “deeper”, “higher”, “wider” in a “unified” enterprise-wide platform approach together with other partners in the community. In smaller settings with customers, partners and analysts since then, we’ve continued that dialogue since the Enzee Universe and generated real excitement as they come to understand the full breadth of what Netezza is enabling in the market.

In coming days, we’ll be writing more about Netezza Insight and how it is manifest in product platforms, features and applications. But for today, let’s just say that SAS and Netezza customers are already able to do more, faster, with our combined products than ever before and that this is just a step toward even more powerful capabilities.

As Rick (Humphrey Bogart) said in the closing scenes of Casablanca, “I think this is the beginning of a beautiful friendship.”

[UPDATE: Rather than just reading what I have to say, you can watch SAS Executive Vice President and CTO Keith Collins describe his take on the value of in-database processing and the Scoring Accelerator for Netezza in the following video from the Enzee Universe 2009 show in Boston.]

"Catch a wave and you're sittin' on top of the world"

pfrancisco — Fri, 31 Jul 2009 02:00:52 GMT

"Don't be afraid to try the greatest sport around

(catch a wave, catch a wave)
Everybody tries it once
Those who don't just have to put it down
You paddle out turn around and raise
And baby that's all there is to the coastline craze
You gotta catch a wave and you're sittin' on top of the world"
– from "Catch a Wave" by The Beach Boys (1963)

Surf's up! Summer seems to finally have arrived in the Boston area and a number of vendors in the data warehousing and analytics space are hoping to catch a wave riding on a flurry of industry announcements. A few trends continue to build in the news:

Data sizes continue to grow alongside the pressure to increase performance & shrink data latencies;
Workload complexity and user counts continue to grow;
More and more, customers are seeing the value of running advanced analytical processing directly in their primary data repository (see item #1 for reasons why); and
Industry prices for data warehousing and analytics have begun another shift downward.

Today I'd like to address this last point. According to more than one industry analyst, over the last several years, Netezza has served as "the benchmark" for DWA pricing in the industry. Several of our competitors have sought to match and/or undercut Netezza pricing in the market. Some of the incumbent players have tried to, with very limited success, hinge their pricing off Netezza prices, match the performance of the Netezza Performance Server® system, or inoculate their pricey "flagship" products by adding less-expensive, feature-deficient products to their portfolio. But Netezza has continued to succeed in the marketplace, becoming a profitable, publicly-traded company with nearly 300 customers and 400 employees worldwide and one that is listed among the "Leaders" in the Gartner Magic Quadrant.

When we disrupted the data warehousing market with our first generation product in 2003 and 2004, Netezza was one of very few startups in an otherwise moribund industry. Now, with established "street cred" and hundreds of loyal customers, we intend to once again upset our competitors and lead the market in pivoting to a new competitive price-performance level. We're about to launch the fourth generation platform of our data warehouse and analytic appliances, which will advance Netezza's performance leadership and once again establish a new price-performance benchmark.

Admittedly, we won't be the first vendor offering high-performance data warehouse systems to move to a lower pricing plateau. That task is usually done by early-stage start-ups looking to find a way to differentiate themselves. True to form, Dataupia probably can claim establishing a lower price point first and recently another multiyear "start-up" has also started lower. But those are offerings from very modestly-sized startups with no established market "track record". Netezza will be the first company with proven product maturity, customer base and financial viability to do so.

Just how and what are we doing to cause this disruption? Well, let's just say things around the "briefing table" have been quite hectic, and that I and others will have more news about that to follow shortly.

[As you might imagine, it's been getting more and more difficult to keep things under wraps – in recent weeks we've even had to fight people off from getting early "sneak peeks". ]

Until then hey, it's summertime! So here's what I'd recommend –

"So take a lesson from a top-notch surfer boy

(catch a wave, catch a wave)
Get yourself a big board
But don't you treat it like a toy
Just get away from the shady turf
And baby go catch some rays on the sunny surf
And when you catch a wave you'll be sittin' on top of the world

Catch a wave and you'll be sittin' on top of the world"

Twin Fin: A short board (usually 5'8" - 6'8") with a wide tail for maneuverability and a fin near each rail for stability in radical turns.

Purpose: A wider tail area provides more planing area and lift, which creates more speed by efficiently utilizing wave energy. Milking speed and energy from smart surf with extremely sensitive and responsive turning ability are this design's strong points

"Netezza Underground" - NPS at your bookstore

pfrancisco — Wed, 26 Nov 2008 20:37:04 GMT

We had quite a surprise the other day when it came to our attention that Netezza and the NPS data warehouse appliance are now the subjects of a new book: Netezza Underground: The unauthorized tales of derring-do and adventures in resilient data warehousing solutions, by David Birmingham (ISBN: 1-4392-0743-7 and now available in paperback version for $31.54 at Amazon.com).

This is not the first instance of the NPS system being the subject of a book sold by Amazon (e.g., SAS/ACCESS(R) 9.1.3 Supplement for Netezza), but this particular publication certainly brought feelings of both fun and reaching into the mainstream with it, starting right from it's very clever cover art (above) to David's clever turns of phrase and real-life examples.

As the title suggests, it was not written or coordinated with any Netezza authorization. So of course we bought a copy and read/skimmed through it as quickly as we could. I will say this, David's self-publication skills are great - he keeps what could easily have been a boring, heavy technical tome both engaging and fun to read while still imparting lots of great information about the NPS system, its performance and its ease of operation. And the book's publication is incredibly current - with references to Netezza Developer Network and "BI Appliance" announcements made only as recently as the Enzee Universe user conference in September.

While I certainly could quibble with a point made here or there about the system, in general I thought it was an excellent book and even put up the following recommendation for it on the Amazon site:

I commend David Birmingham on a book that is at once as lightly entertaining and interesting to read as it is chock full of details about just the kind of performance and operational simplicity that is possible with the Netezza Performance Server (NPS) system. Straightaway from the opening pages, Birmingham's effusive, engaging style and excitement about Netezza's system is apparent, "It inhales, crunches and publishes Libraries-of-Congress-at-a-time - and fast."

He also captures the essence of the NPS appliance in an ultra-succinct two-sentence paragraph explaining just why his "Administration Stuff" chapter is so short, "It's an appliance. Put it in the corner and let it work." I couldn't have said it better myself!

This book is comprehensive and current - even reflecting some of the more recent announcements from Netezza regarding OnStream programmability, the Netezza Developer Network and analytic appliances.

As the guy who is responsible for projecting the Netezza products and our technology direction forward, I want to recommend David Birmingham's book to current and prospective customers and partners alike, or as David himself says on the book's Dedication page, "to Enzees everywhere".

--Phil Francisco, VP Product Management & Marketing, Netezza Corporation

So "to Enzees everywhere", have a read of David's book and welcome to the "Netezza Underground".