"What's this?" asked the CFO of the Data Warehousing Director, now holding a fresh-off-the-press request for more hardware.

"We've hit capacity," said the Director with a sigh, lowering himself wearily into a padded chair.

"What? That machine is barely a year old! And just last month you claimed it would be two years before we would need more capacity. I didn't allocate any budget for hardware because you said we wouldn't need it!"

"I was wrong. We just implemented an upgrade of the application and it started hammering the machine. We didn't know we had hit capacity. We thought we had enough."

"I've had enough," said the CFO, "I want some answers before I sign off on this."

And so began three weeks of malaise in searching for the right answer. An answer that nobody was particularly trained to find. Throwing in the towel, they sought outside help and eventually discovered some very interesting artifacts.

• Firstly, the solution using the machine had been inefficient from the beginning. The machine's power had masked the inefficiency. The new implementation only created a tipping point.
  
• Secondly, they discovered that of all the SQL statments hitting the machine, over eighty percent of them were singleton operations. Of these, over half of them were singleton updates. Of these, none of them were using a distribution key to support the update. This information was generally available in the query history database, if one only knew where to look. And if one had the time. And the patience.
  
• Thirdly, many other inefficiencies were pervasive, including the use of big-fat transforms (as opposed to more and leaner transforms with intermediate tables) and the inability to see the spikes in application activity.
  
• Lastly, much of this activity went across machines (several machines shared the load of processing) so that finding the nefarious operations was nothing short of a submarine hunt.
  
• Several of the large tables had been distributed on keys that were more suitable for zone maps than distribution, meaning that the vast majority of queries would only leverage one or two CPUs out of hundreds. Sort of like trying to pull liquid cement through a soda straw. This is process skew rather than data skew and is sometimes difficult to diagnose.
  
• Couple the above with the lack of proper controls for update/delete operations, resulting in an enormous count of aborted queries and their attendant instabilties - and the pernicious growth of unrecovered space and we have hidden inefficiencies that require a different level of resolution and investigation.
  
• The cost to remediate these problems was not as substantial as wholesale upgrade, but could have been avoided or at least mitigated had the principals been aware, or had visibility to the lurking dangers and inefficiencies that were literally draining the lifeblood from the systems like parasitic insects on an unsuspecting host.
  

Now the reader may well assert, ahh, we have enterprise tools that allow us to see the machines on the network and in the operations center. Those tools should tell us everything we want to know. Well, sure, about the Linux host of the machine, but not about the health of the various databases, usage stats, trending and most certainly not application-level nuances like misapplied or ignored distribution keys, mangled zone maps and tables that haven't been groomed. What's that, the nzPortal could give us some of this? Sure, on a per-machine basis. But the machine admins want to deal with machine health on one level, while the information architects want a completely different, even oblique level of information that may require deep immersion into the logistics of one or more enteprise applications.

What's the real problem here? Logistical complexity. The simple fact that as we add more technologies, applications and functionality around the machine, it becomes the fulcrum for the enterprise. Without management of the activities both in detail and in context, logistical complexity arises. We want to embrace the simplicity of management and administration, and on a purely administrative level (e.g. what the common DBA role fulfills) it's still a part-time job. But this may hide the fact that the application engineers and implementors have been given, or rather been delegated with, the additional responsibility of logistics. Perhaps they didn't realize that implementation and architectural logistics was now on their plate. After all, they only build applications. Deploying, operating and maintaining them has always been someone else's gig. In any other technology besides Netezza, the DBAs make it their business to know the data and its processing nuances. But in those technologies, there is no power in the machine to take us very high, so the possibility of logistical complexity is held in check by the lack of power in the machine. This constraint released, logistical complexity now becomes a very real threat.

A number of years ago these needs reached critical mass and the ninjas of the Enzee Community that had been addressing these issues as onsite consultants finally congealed these capabilities into a platform that can not only see these trends, nuances and capacity issues, it can do it across multiple machines and applications. In the spirit of Netezza's "driving toward simplicity", this platform plumbs the depths of Netezza's more complex interior and serves up the nuggets in visualized, actionable form. It is essentially the application-administrative level of business intelligence - for the machines.

In the aforementioned "submarine hunt" the principals learned an important lesson of very-powerful machines: they can make even the most horrible implementation look stellar. By correcting the discovered issues, they reduced the machine's load by over eighty percent. Imagine recovering eighty percent of a machine's capacity just by applying some simple fixes. One day the machine is overloaded and we're looking at upgrading for non-trivial costs, and the next day the machine is barely breaching twenty percent capacity and we won't be chatting with the CFO anytime soon. Years even.

Some of you have similar stories (I've heard many of them!) and sincerely want a better way to deal with multiple machines and a wide array of applications and users in a more holistic manner, with an eye on what counts in a Netezza machine, not just what a typical database does. This takes administration to a level that simplifies and clarifies the complex in a form that intersects to the language base and nomenclature of the Enzee Universe.More importantly, the Enzee Universe is an ecosystem where many of us have found, through experience of success or pitfall, what works and what most people are asking for (and wondering why some of the leaders aren't taking the bull by the horns to solve it). Well, many members of the ecosystem have worked around the logistical issues without privy to the expanding capabilities of core observational needs, codified and implemented by people in-the-trench.

The inception of this observation technology was both brilliant and simple. It would faithfully gather stats from each of the machines as a summarized, transmission-safe extract and drop it to disk. Their on-site consultants could install it, kick it off and then go about their other analytic activities. At the end of a day (or two) they could examine the summaries and these would reveal all. As scheduled operations using the machine's power to process the machine's own statistics, it has a very low footprint and can be brought under workload management. As structured extracts. it was only a matter of time before they would put a pretty face on it. Now it's a dashboard to the inner sanctum of the Netezza machines, from the people who have collectively installed and shepherded more Netezza installations into production than any other, with no close second.

The Observation Deck, from Brightlight Consulting.

Next Tuesday, August 30th is a Webinar demonstration of this technology and I would like to cordially invite the Enzee Community (and all other interested parties!) to attend and take a deep look at what it can do, and what it could do for you.No pressure and no sales push. The need for this is obvious and we are simply demonstrating that it can be filled and supported. But the participants will be the judge as to its viability. It is the ecosystem's way after all.

The webinar is on Tuesday, 8/30 at 11 am pacific / 2 pm eastern.  You can get more details and sign up for it at:  http://advancedmonitoring.eventbrite.com/

"You hate me don't you?" she whimpered, teary-eyed.
"Nobody hates you, Spooki," Bill assured, "It's just business."
"Just business?" she spat, the hurt spilling from her mouth, "Who died and made you king?"
"I thought you didn't want to hear about it."
"Never mind. We're already in Season 4, our ratings are tanking and you want to bring in who, exactly?"
"Well, we have some options. You know, other leading characters from bloodsucking tales -"

"What, the Lincoln Lawyer? You'll never get Matt to sign on -"

"No attorneys. People don't like to mix attorneys and bloodsucking, no matter how cute it sounds."

"Worked for Angel."

"Or not. He's in syndication now, right?"
"Then, like Lugosi?"
"Well, Bella anyhow, but that's not important. What's important is that we reel in this storyline so people can actually follow it."
Spooki rubbed her shoulder, pressing the fabric against her skin, "Where did we get this wardrobe? It's itchy."
"Wolfe collection. Has a lot of wool in it, none of us have problems with it."
"Did they think we would boost ratings if they got us all to wear Wolfe?"
"Some of us, anyhow. Were you like, asleep for Season 2?"
"Look, Bill, I've carried this series for three seasons, so - "
"But you're not a name-brand player, and we need some lift. I mean, not just to pull the ratings out, but to scale to more viewers. We're all over the world now. Streaming internet video, graphic novels, action figures -"
"That's just great. Merchandising. It's all about merchandising."
"No, it's all about scalability. The storyline's not scalable unless we bring in some power hitters."
"So the critics snipe at us and all of you go running to momma."
"Speaking of Snipe -"
"Oh no way, you're not bringing in the - the -"
"Daywalker. Yep, he's got more chops that the rest of us combined."
"Karate chops, you mean. He's a vamp-killer, Bill. Bring him on board and the storyline will die because all of you will be dead."
"Hey, the best vamps survived the Mighty Buff, so I think the writers can handle this."
"Speaking of buff -"
"Writers seemed to be cramped on it, I know, but hey, it's about ratings after all."
"So why isn't anyone blaming the writers?" Spooki cried, "I mean, the show is only as good as the writing, right? Make the writing more efficient and we get more lift, right?"
"I know it's easy to blame the soft skills but the problem is the platform itself."
"Me. You mean the problem is me?"
"Look, Spooki, I'm just the messenger so -"
"What's Blade got to offer the franchise? I thought he was off on his website Fantasy Pro Golf Assoociation."
"Yep, well, fantasy golf has taken off, worldwide has more participants that we could home to amass in a hundred seasons. By bringing Blade into the picture, we automatically get the FPGA to join us. It's not about Blade alone, but Blade and the FPGA together."
"I don't get it."
"Following, it's all about following. Blade and the FPGA will eclipse you into the twilight."
"It's so unfair. Let's talk about it over dinner."
"Sure, who are we having?"
"That's not funny."
"Over, I mean, Who are we having over?"
Spooki rolls her eyes, weary of the double-speak.
"We could go into town and have a sit-down, maybe have a nice Merlot -"
"Now why would you say that? Sam's a nice guy even if he is a little shifty."
"Ahh, he's a vendor rep. Some of them are that way. He hangs out with shifty people, you know."
"I'm just trying to get my head around all this."
"Let me put it to you this way. Your character has become too complex. People can't keep up with all the different facets. We certainly can't add any more character features. It's just better if we sunset Spooki and take it into something simpler, easier to manage."

"Are you saying," she swallowed hard, "That I'm a high maintenance character?"
"Well, not to put too fine a point on it -"
"How am I too complex?"
"Rather than staying grounded in the everyday, it's become a fairy tale. Our fans are not just moving away, they're hemorraging. It's a true bleed."
"A fairy tale? Seriously?"
"People want to get back to roots. You know. good and evil. Black hats and white hats. Battle lines obvious. It just simplifies the story for the viever. They need simplification or they will get lost."
"Where's Travolta when you need him?"
"Look, if I get knocked off, I have other options."
"Oh, what that BBC show with the cross-dressers? Get real."
"Hey, Ex-Men is very popular."
"I don't think the viewers are lost. They asked for more meat in the storyline and more depth in the characters."
"Yeah, but like anything else, the extra character features don't lead to scalability of the storyline. We've reduced the entire story to just a few people. Now we're bleeding out because we can't spin off."
"Spin offs? They want spin offs?"
"If you can't spin off, you can't scale."
"Spin off! I'll show you spin off!"
"Now Spooki, don't get carried away. Not all of us have a fairy godmother looking after us."
"Actually, thanks to you, none of us have one of those."
"Let's go into town. They just opened a new Which-Wich and I want to check it out."

"You would.  I just don't see how Blade will change anything."

"He has more bandwidth and better technology. Lookit, how do you think Blade got so successful? He could find vamps practically anywhere, no matter how deep they tried to hide themselves."

"Yeah, how did he do that? And why can't we have writers that good?"

"Feature films, Spooki. Different realm altogether. Just ask Bella."

"Bella has hers, I have mine," she sighed, "So how does Blade do it? Haven't our writers gone to every extent to boost the story?"

"Simple. He knows where the vamps aren't. Not at the supermarket. Not at the cinema. Just by process of elimination, Blade can zone-in on his prey anywhere on the map. He's groomed for it. He doesn't have to go to every extent to find what he wants."

"There's just got to be some way we can reclaim our lead in all this. I can't believe we've been eclipsed."

"Hey, reclaim is a thing of the past. We groom our characters now. It's the only way to fly."

The long-awaited "SQL" to Netezza Underground has hit Amazon.Com.

http://www.amazon.com/Netezza-Transformation-without-alchemy-sorcery/dp/1453874054/ref=sr_1_1?ie=UTF8&s=books&qid=1306624705&sr=1-1

This book tackles some deeper issues around transforming our data warehouse, our approaches and even our thinking to align with the arrival of the TwinFin. No, it's not the TwinFin we'll transform, but that the TwinFin will transform us. Once again, a little tongue-in-cheek irony

The book offers working examples of checkpointing, exception handling (primary/foreign key), windowing (SCDs and deduplication) as well as a cookbook on more details to watch for in a migration project. And of course, is replete with Case Study Short and a whole chapter on Case Studies. There's also an appendix at the end to offer up some simple scripting jump-start routines that can make bash so much easier.

I also threw my hat into the ring on the Exadata question, as well as several other products that need a kick in the pants.

A question I receive often can be characterized as "do I really need to do this for smaller tables? Isn't this only necessary for larger tables?"

And the question could run the gamut from things like when-to-use-bigint to how-to-distribute-tables. The interesting thing - some of it really can be deferred until later, since Netezza has the power to turn-the-ship so to speak. Other things we need to consider for different reasons entirely. For example, will the table always be small? Or can it grow without asking our permission?

Case in point - a discussion arises as to whether we need bigint surrogate key types for small lookup tables that will never exceed hundreds or even thousands. The bigint type seems like overkill. So this makes some sense in that we don't want such overkill mechanics in the data model. Then someone notes, "and we'll save disk space too". Well, not really. On the smaller tables the additional bytes are negligible, add to this the compression ratio and the fact that all tables have a minimum required allocation of space - and we see that the avoid-bigint-to-save-space isn't really valid at all.

The avoid-bigint-to-reduce-domain-size tends to make sense, until we get into the issues of actually filling them with data. If we want to use a hashN() from the toolkit, we'll have to coordinate between the hash8() or the hash4() for example, to make sure we use the right one. If we plan to use sequence generators, we'll also have to coordinate the separate use of those, one for bigint and one for integer etc. In this aspect, the bigint (since it isn't costing anything in disk space) has higher value in its data-type-portability and the portability of the code or SQL leveraging it. We reduce logistical complexity with no penalty. The bigint then acquires a more polymorphic flavor to it, as a pointer to data rather than as a numeric counter or index.

Another aspect is distribution. How do we want to distribute the smaller tables, and what should we choose? For the smaller tables, recall, if they are not distributed on the same key as the larger joining tables  (usually the case) then they will automatically broadcast to the larger table. As this is actually the expected behavior, is there anything that could change this? I have noted on some sites that when queries get complicated, the optimizer does not have enough information to properly make a call on broadcasting, and can actually reverse this outcome - broadcast the larger table toward the smaller. Such things are very difficult to find when the data sizes are marginal and only rear their heads when the data sizes (on the larger table) hit a certain threshold. Meaning: it will work for a while, perhaps even a very long time, and then show signs of inexplicable drag.

Of course. this can also happen if the statistics on the table are stale, so we always want the optimizer to make to right call. We could always keep our stats updated and we could make sure those complex queries never got-that-way. We could-and-should do that over time and maintenance activities, but we should also never get bitten in between.

To avoid pain and injury, the way to make the smaller tables always behave is to distribute them on random. The optmizer then has no basis to broadcast a larger table into the smaller, because no distribution exists. The smaller will always broadcast to the larger (the behavior we expected, right?) and all is well. The beauty on the TwinFin is that this smaller table will be cached in memory at no additional charge, further boosting the query and any subsequent queries using it - a beautiful thing.

At the June conference we covered a lot of practices around harnessing the Netezza machine for fun and profit. But the buzz on ELT was rising then, and when I arrived (and remained after) the November London Roadshow, people were practically tackling me in the hallways to learn more about how ELT works, how well it work, and how practical it is.

Well, the ELT buzz is growing even louder. I've been practically inundated with requests for more information on ELT and how it operates. Of all the opinions on the subject, the conversation usually boils down to the irreducible: How do we get plain vanilla queries to do more than be, well, you know - plain vanilla queries? I mean, the business logic and flow rules are all bound into it! And if the queries can be thrown into the machine with reckless abandon how can we possibly make any sense of them? ELT means forming structured flows of information using intermediate tables as checkpoints. Netezza is one of the few platforms in the world that can do this effortlessly and productively. Anyone now exposed to this (like a common RDBMS developer) would claim that ELT is unworkable on any database. Well, perhaps on your database, but not on Netezza.

Of course, we cannot allow arbitrary SQL to enter the flow can we? But does this mean arbitrarily submitted, or arbitrarily formed? You see, if the SQL has a consistent form, we have all the freedom we need for arbitrary submission. I suppose it's the "degree" of arbitrary we're looking for no?

Case in point, a vendor tells me "Oh, you can send any arbitrary query to the machine and.." Of course he means anything, absolutely anything our developers choose to concoct of their own cleverness or ignorance, or anywhere in between. In fact, cleverness is perhaps the least of our worries since the clever queries often follow some degree of structured discipline. It's the queries produced by the newbies that pose the most danger. It's hard to remediate a lot of work that took place "in the wild", so to speak. Of course, for those who like their wild-game-SQL or pehaps free-range-SQL, I will not disparage their personal tastes. But this is enterprise computing after all, and many of my personal tastes are already checked at the door.

And as I've noted in the past, metadata is where it all happens anyhow so let's take a short dive into the future of ELT and what I suspect is the next wave of stuff we'll see.

1 - insert/select queries, while they are the ultimate bread-and-butter of ELT, their hard-wired form should not rule the roost. One need not fear being enslaved by the queries, but should instead harness them. In fact, the queries should never completely exist in one place, but be manufactured at run time. Now, our BI tools already do this, right? We post a request into our favorite BI tool and it leverages its metadata repository to form-the-perfect-query. That's where we need to be. Provide a template and some steering logic, but the query itself is manufactured on-the-fly.

2 - Automate everything, from DDL to physical model, from intake to reposit, and it should be a fairly simple approach. When we takea look at the patterns of what we're trying to accomplish, we are essentially punting around metadata (tables, columns, data types, rules) and these are the object-oriented assets shaping the solution-at-hand. We should not be in the businss of crafting SQL statements (let a product do that for us) but in the business of shaping logic. The ETL tools purport to do this but cannot scale (for set-based operations) as effortlessly as Netezza can. So in some ways, the ELT aficionado is stuck in a bubble waitng for the technologies to mature.

3 - Think in terms of metadata, factories and steering logic. Okay, what does that mean? Factories mean components that take raw parts and manufacture them into SQL statements and other assets. This allows us to drive practically everything from knowledge of the Netezza catalog. Source table to target table? All we need are the rules in the middle and we're done, right? So if we can make simple lists of rules that are tied to particular table/colums as targets, we are now in the zone to leverage a factory to manufacture the proper insert/select statement to send to the machine. We don't really need to have the predefied insert/select, only a list of target columns tied to rules. The rules then become our source map, and the target columns become our fodder for manufacturing the insert-clause. Bolt a filter / where-clause on the end of it and yes, we've have valid SQL. But wait, what about steering logic? As my factory is manufacturing the query, I can make all kinds of decisions about the table and columns as I encounter them. For example, if I am making an extraction query to face a database, as I encounter each column in the catalog, I can compare this to yet another repository of information that can tell me, does this column use a different name on the extraction source than the target? Perhaps I need to exclude it from the query altogether? Perhaps I need to  alter the value of a given column to a default (data masking) or I may need to bolt-on columns to the end of the structure. In short, the steering logic is a simple set of exports that I can tie by-name to the source and table name so that it is surgically applied as necessary. This means every extraction is now programmable because we've built the programmability into the factory - the switches and signals serve as steering logic as the factory weaves the perfect query.

What is BI anyhow? Some companies who do all-data-warehousing have jumped into BI thinking it's just the icing on the cake, so to speak. Others embrace BI as their primary focus but then drive into a pit of miry clay when they have to access data untouched by the scrubbing bubbles of an enterprise-class data warehouse. So which is it? Data warehousing and BI? Does BI encompass data warehousing? I've had this conversation with so many people who cannot seem to put their finger on the definition. One argued passionately that we should simple consider data warehousing and BI to be one and the same, such that when we talk about BI, we're talking about both. Then not five minutes later he claimed that Microsoft Excel was the most widely used BI tool in the world, to which I asked wryly, "since when did Excel start doing data warehousing and ELT" - he sort of stared at me, not knowing why I had asked the question, forgetting his prior claims as though second thoughts entirely. My simple assertion is that the open marketplace calls them out as different - BI is not data warehousing and likely never will be.

Cached stuff - Like any system, catalog-hits can become cumbersome and the factories leveraging them can begin to take protracted lifecycles for building out the queries, some of which can be very complex. However, if they are based on static metadata (like that in the catalog) it is easier to cache the manufactured assets (less any table/column-centric adjustments) and avoid re-manufacturing these assets each and every time we need them. In one case, manufacturing the assets for each extract/load took about five minutes to complete when this metadata had to be manufactured from scratch each time. After caching, however the entire duration became less than a minute. Many ELT-oriented tools actually do consume the catalog but unfortunately  do not consider this aspect of data processing - that the metadata itself - when manufactured from static assets, becomes a repeatable product. A perfect candidate for a cache.

Transform metadata - talk about "all the rage". Hey, people want ELT, but they don't want to risk losing track of how to trace data lineage through the multiple stages of SQL statements. After all, by the time the data actually arrives in our BI tool, we might see the data undergo hundreds of insert/select queries to arrive at the final data outcome. What if an analyst wants to perform track-back lineage of the data, or perhaps even do an impact analyis of the environment? These are impossible without metadata-driven artifacts, rather than allowing SQL guys to manufacture any arbitrary query on-the-fly.  What does this look like? Well, think in terms of source columns, rules applied to those columns and their outome sent to target columns. Then we need a filter clause (join, where, filter, group) that is easily identified. Of all this, we need to capture rules, participating tables, participating columns (both in the source to target map and in the filter clause). It's not really all that hard to capture it if the developers have followed certain rules.

Rules-based asset manufacturing - (a spinoff of one of the above) is simply providing a structured template where developers can formulate and test a single query for accuracy, or multiple dependent queries. But in either case, the metadata in the catalog can provide us with the the raw structures -  columns, tables, etc and we can then formulate templates from these. We then hand off this template to a developer and they fill in the details. They can't really get themselves into trouble because if the developer tries to do something illegal or ignorant, it's all part of the game to us. Our environment requires the developer to adapt - or the query simply won't run. I was on a site where we had only a handful of basic programming rules. Easy enough - like falling off a log. But rules will not protect us from someone who is willing to break them. Here we have templates that provide the balance between rigidity and infinite flexibility. Make the template work and you are by definition in-compliance.

Without elaborating into infinite detail, the realm of ELT is still maturing. ETL tools jockey for position in push-down approaches, but know that if they are really good at this, they are essentially working themselves out of a job. If for no better reason than they have effectively harnesses a Netezza machine to do all of the heavy-lifting their own hard-won capabilities currently do. And if so, the irony is that any one of their smaller competitors stands to gain much more in this marketing space because they have less to lose. After all, we don't buy a power-hitter ETL tool and relegate it to data-transport/transfer activities (it's new role alongside the Netezza machine) but a smaller player would love to fill this spot, grow it, mature it and deliver it as a packaged method to create fire-from-fingertips.

As some of you know, here in Texas the sport of football is a bit more than a team sport. In some locales, it's practically royalty. Back in high school and college, I was ever-aware of the sports fans that liked to wear the team colors. One in particular invited me to lunch while his parents were in town (I already knew them from back home) but I never knew that his father owned a car that was actually painted in Dallas Cowboys colors. Another colleague of mine was stuck in a parking garage with a dead battery as I was leaving work, and asked if I could help him with a jump. Upon saying yes, he produced a set of maroon-and-white jumper cables, honoring Texas A&M University. Oh yeah, he was a fan.

Then several of my friends kept large jars of team-color makeup and would smear it all over them before a game. No, not to go see the game at the stadium, but in their apartment! Yes, the team makeup was always a big hit, and their parties were solely centered on the tiny color TV in the center of their rather modest apartment. I had once wondered what the people next door thought of their shouts, whoops and antics, until I learned that the folks next door were just as rowdy with their own team. Hey, ya gatta be a fan.

Okay that's not the team makeup I'm talking about here, but those were just such funny memories I thought I'd share. And to make a point of course. The enthusiasm of your developers and architects, and their desire to cheer you on with your goals to achieve, are directly dependent on the team makeup. Of people, that is. So what does it take to make a data warehouse? Or for that matter, what does it take to effectively roll out a new one, or migrate an old one?

What most people fail to estimate in taming such a beast, is the level of testing required to make it a reality. As an example, many of you must produce documents as part of your regular workday. Those documents are often hard to write, but even more work to proofread. In fact, proofreading is the same form of content-and-context testing we would do for a data warehouse. The chief reason is the product - information and knowledge. Business intelligence is the same way - it has a way of taking on a life of its own, but the only way we can reliably roll out a viable business intelligence platform, is to test, test and test some more. Eyeballs-to-page however, may be required for a book or document, but it won't scale for a warehouse.

Many people don't realize that the testing portion of the warehouse can take as much as 80 percent of the project's resources. While we can compress this somewhat with agile methods, we cannot afford to test such quantities of data with simplistic manual approaches. And by that I mean eye-ball examination or screen-shot testing. No, the majority of testing is in the data itself, and on a Netezza machine it's in the billions of records. Eyeballs don't have the bandwidth. We need to use the actual power of the machine to scale this mountain.

So what should the team itself look like? May I suggest that for every Architect you would have several developers, and for every developer you would have two or more testers. Ideally three testers for each developer, regardless of how many developers are actually doing the work. I will also suggest that you keep the total count of developers rather lean. Five is perhaps overkill for the back end. For the front end, three solid developers can be an army, and five is about the upper limit. The reason is simple: logistics. If you have five developers on the back end and five on the front end, and three testers in the wake of each, this is a team of 40 people - which quite frankly is overkill in any sense of the word. Not to say that an overall team might not be comprised of 40 people once we include all of the infrastructure folks, but not for pure develop-and-test. We can and should make it leaner.

As an aside, I had the rather disturbing experience, numerous times, of encountering folks who worked these things out with overblown spreadsheets that they normally used for application development estimations. A data warehouse gig is completely different from an application development gig. But of course, if one of these spreadsheet guys ever showed up and plugged in his metrics, he would spout off that we need a 30-person team to migrate a couple of tables from one machine to the new one. Once this number is in the air, it becomes the de-facto standard by which all discussions are measured, even though it is completely wrong. In another setting, another spreadsheet-guy plugged in his numbers and characterized a project as a $900k gig when our competitors were bidding $300k for the same work. Knock yourself out, dude, because the client ain't a-bitin' three times more expensive projects because they like our faces. True to form, the $300k bid actually won. But the irony was, that the potential client had no desire whatsoever to pay more than about $400k, so the bid fit their budget just fine. The eventual winner of the bid took a bath, however. The truth is always somewhere in the middle.

I still say, watch those spreadsheet-guys. Somethin' up with that.

Perhaps it goes without saying that an architect needs to lay out a framework so that all can work comfortably in the same sandbox. This is a challenge and should not be left to the developers to forge on their own. Harnessing it later will be impossible, because too many opportunities for flow-based consolidation will be lost. Workarounds and repetitive logic will become the rule. Let's not go there.

If we have say, three solid developers in the back and front end each, they can and should cross-test each other's  work. In this case, we have the senior developers and architect working on the core logic, and the junior developers bench-checking their work and zipping it up for a formal testing team. Here we have a synergy, that a senior developer can crank out ten times the work and quality of the juniors (so says Demarco and Lister, your actual mileage may vary)  but nonetheless, we would not want to put a junior developer in the front seat of this chaim because the testers will be waiting on him more often than not. But with the ten-times-more-power driving the front like a locomotive, the junior developers can wrap up the many tactical areas of the warehouse and cross-test each other, but also receive the work products from the senior developer.

Now think about what this kind of model means. The senior developer is literally force-feeding the pipeline with work products and is doing it with the highest quality the team has to offer. The junior developers are learning from the senior without injecting their own inexperience into the mix, which will invariably have to be reviewed by the senior developer anyhow. No, the senior developer is more productive and experienced, so let him/her drive. Seems like every senior developer I talk to, they really, really want to develop and have the testing tedium off their plates. The junior developers really, really want to learn from a senior developer, and of course want to do some development themselves. I'm not saying this is off-limits, but the senior developer can delegate-what-he-knows to the junior developers because they cannot go too far astray without his guidance anyhow, so it's a win-win. And of course, I and every other person who was ever a junior developer had to pay our dues, so not everyone can be the leader. I don't say this dismissively, but we know in a business intelligence project there has to be a driving mind. Too much consensus means too little leadership, and in the famous words of Margaret Thatcher "Consensus is the absence of leadership".

But for the people doing the testing, they need something that will scale. To billions of records. And it had better be solid on the first round or they will be playing catch-up for every round after that. While writing and proofreading a document is an eyeballs-only model, don't you think I could at least do myself a favor and run a spell-check and grammar-check on the contents? Such set-based operations resolve a world of problems and let me focus my eyeballs on the harder stuff. But in a data warehouse, our eyeballs will never have enough bandwidth, and will never scale to the necessary heights. Set-based testing is all we have, but it's also all we need. And with a Netezza machine, we're so in the zone.

Now testing of the report screens can involve eyeball-based activities but doesn't have to be so egregious. Automated testing tools go a long way to mitigate the necessity for eyeballs on these as well (for the subjective parts like positioning, banners or colors especially). However, if the data is wrong, no amount of pretty-pretty will fix it. As Murphy would say "Beauty is only skin deep, but ugly goes to the bone."

Now, no sooner will I write this than I will get feedback from those junior developers who say that they have been relegated, but not to fear. This particular article is in context of a high-productivity bubble of work, normally found with new projects or migrations. The priority is not to make people feel better about their role, but to get past the workload so that everyone can feel better about the work products. I am always looking for opportunities to stretch the developers, both junior and senior. When a junior is ready to sit in the driver's seat of the locomotive, it's because he's passed the Demarco and Lister smoke test. Now what the heck is that, anyhow?

Get a copy of Demarco and Lister's Peopleware, a classic in every decade. Something they have empirically measured, is that a junior developer will start out at one level of productivity, and then in a sudden epiphany will transform into someone who is ten times more productive than before. Something mentally and/or emotionally clicks and they get this whoosh. They claim it is different timing for each developer, but usually takes about two years to make this transition. This is perhaps one reason why so many job-search requirement listings show "X years of experience in Y" and the "X" is never less than two years. Not because the poster has ever read Peopleware, but we who are in the field want folks who are 2 years along because we already know they have (at least) transitioned into a high-productivity asset.

But this is the mechanism driving the team makeup - and the experience of the developers and their known levels of productivity should help us find the right role for them on the team. We don't want a low productivity person in the locomotive chair. But having one in the wake of a strong developer only makes them stronger and exposes them to practices that will accelerate their transition into the higher productivity person we always wanted anyhow. And then, of course, once the person has made the 10x transition and is self-aware of their value, we have another problem: They are self-aware of their salary level too! Making someone stronger makes them more valuable. Be prepared to recognize the value (or rest assured that your competitor will). But all this, is the nature of the beast we purport to tame, no?

Back to set-based testing. This has as much to do with using the right data as it does the right method. The right data means - select a subset of known data that will deliberately exercise all of your business rules and software paths. Nothing is worse than realizing such errors in production. Then, we need set-based testing methods. This means we need three primary assets: (a) source data that we can sluice through our application transforms to get a result (b) a saved baseline result to compare this recent result against and (c) tested components that compare these two results in a reliable manner so that we get a statistical report on what passed and what didn't, and a detailed report on what specific records didn't make the cut. Counts, amounts, checksums and summaries all reveal deviations, especially for regression testing. You might recognize this as an exception report, and this is exactly the spirit of the effort. Our testing has to deal with statistical exceptions, because it is the only practical and scalable way to validate billions of rows.

And also notice that such a practice would be the kiss-of-death in many other "secondhand" data warehouse platforms. Such platforms are in no wise optimized to compare monstrous sets of data to each other column-for-column, row-for-row. Queries like that can dim-the-lights and may not return for hours, if not days. We cannot afford a protracted testing phase, and with Netezza we don't have to. Scan times and comparison times are very objective and knowable. The tests will take the same amount of time each time they run, and we always have the option to optimize them further with the Netezza performance model. Power is in the physics.

And again, why all the focus on testing? I have seen data warehouses blind-side an organization that accounted only for the opposite equation - 80 percent development and 20 percent testing, when more often than not, exactly the opposite is true. This would mean that if a two-month development effort were characterized with one model (the wrong one) it would look like at most a three-month effort. Why then does it metastasize into a ten-month effort? Because 20 percent (2 months) tranlates to 80 percent (8 months) of testing.

That is, if we just embrace the standard model. By embracing the aforementioned model, we get the development out of the way quickly and deliberately, entering the testing phase much sooner, and if more heads are deliberately dedicated to set-based testing we can close this part off even sooner. I have watched very-large-scale projects, with a Netezza team in the middle and strong developers in the locomotive seat, enter their first UAT phase within two months of the project's inception. The funny thing is, the model requires rapid turnaound that only the Netezza workhorse can provide, Try pulling off this team makeup with any other lower-productivity technology, and it won't sing the same key. A high-productivity developer is meaningless on low-productivity technology. And high-productivity testing methods are useless if enslaved to a low-productivity technology.

Start it, shape it, ship it. Netezza is the ticket home.

Whew! The Enzee Universe this year was quite an experience.

I would like to offer my sincerest thanks to all of you who attended the Best Practices session held in marathon-form on Monday before the keynote. Over 300 people signed up, and many of you arrived the evening before, and at the end of the session, you were still there!

Afterwards we took a checkpoint and then added more material to the powerpoint presentations that we used during the sessions, and these will be added to the content of the Enzee Universe downloads for those who attended.

Some of you also asked me about the music selections we played at the intro and during the breaks. These were selected in terms of "Your Theme Songs", because some of them were from superhero movies, and some from action-adventure flicks. Here they are, in no particular order, the tune, origin and reason for selection:

Theme from "The Incredibles" - because we come from a family (The Enzee Universe) of mult-talented superstars

Theme from "Batman/The Dark Knight" - because sometimes we have to work in a thankless role (however personally rewarding, with high-tech toys)

Theme from "Superman" - because to competitors, Netezza is like Kryptonite, and to the rest of us, it solves World Problems and makes us look good without having to wear our underwear on the outside

Theme from "Spirit, Stallion of the Cimarron" - surging music for those who entered the frontier on a Mustang

Theme from "Surf's Up" - probably enough said, here

Theme from "Mission Impossible" - a congenially offered rebuttal to those naysayers who say it can't be done

Theme from "James Bond" - because he, like many of you, is an MTBA - That's Multi-Talented Bad A**

Herbie Hancock - Rockit - because that's what you do

I would also like to thank Netezza for the opportunity to share these ideas, many of which I have gathered over the course of my Netezza derring-do from people just like you, so some of the information is what-is-practiced in the field, and some of it is idea-works that we have re-synthesized into practices that seem work well as a sort of "adaptive composite". The objective of course is to share with you what others are doing, to enrich your base of ideas, but are certainly not hard-and-fast rules. The Netezza appliance is one that unlocks creativity, harnessing it for the Good of All Mankind. So guidelines and practices give us more critical mass to solve problems.

Likewise I would like to thank Netezza for the Enzee Community Voice Award presented to me on Tuesday night at the Gala, in recognition for being such a vocal supporter. But my words then apply as now "The people and the product create a synergy that's like electric current. I love interacting with the Enzee Community, and being a part of it".

I also noted that a larger number of independent consultant/contractors were present on this go-round. In the best practices sessions, there are a wide range of professionals, from those who are Netezza customers, to consultants working for firms, independent consultants, analysts and the like. The Enzee Universe has various video screens constantly running slow-motion surfing videos in keeping with the TwinFin Theme. One day you might be lookin' at that guy on the wave, thinkin' about the Twinfin and wondering if you're on the wave, or just watching it pass.

As with all professions, you might be in the zone where your project is just ending, or is about to, and you're wondering about the next great thing. And as you know, I'm always on the hunt for bright architects and engineers, especially in this economy, so if any of you independent types are looking for an opportunity, give me a shout. I also extend the invitation to anyone else who is reading, with the qualified apology that I am not lurking at the doorways to steal away your company's valuable resources. But I have seen in the past that some bright folks find themselves tapping a pencil on their desk, coming down from the exhilaration of a Netezza 'experience" and wishing for more. I can say - the work is out there. I'm often in contact with people who need someone just like you - hooked on the technology. Hey, who isn't?

Finally I offer a simple salutation to everyone who "gathered round the grill" this past week, sampled the wares, wishes and whatcha-ma-callits of the various vendors, trainers and speakers, and came away enriched and enabled to dream a little stronger, solve a little simpler, and crush those waves with the shredding confidence of parallel power. So I'll either see you in your natural habitat, interact with you here, or catch up with you in person when the Enzee Universe cranks up another adventure.