Feb 23, 2010 4:53 AM
NZSQL Analytic Functions - Partition rowcount limit exceeded error
-
Like (0)
Hi,
We are trying to calculate count for different gouping (8 Levels of grouping) using a single qurey. The query works fine with 2-3 level of grouping but anything above 3 level Netezza throws up the below error.
Error: ERROR: 65534 : Partition rowcount limit exceeded
(State:HY000, Native Code: 2C)
Sample query:
Select
customer_dim.Level1_Name,
customer_dim.Level2_Name,
customer_dim.Level3_Name,
customer_dim.Level4_Name,
customer_dim.Level5_Name,
customer_dim.Level6_Name,
customer_dim.Level7_Name,
customer_dim.Level8_Name,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level1_Name) as cnt_Level1,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level2_Name) as cnt_Level2,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level3_Name) as cnt_Level3,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level4_Name) as cnt_Level4,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level5_Name) as cnt_Level5,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level6_Name) as cnt_Level6,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level7_Name) as cnt_Level7,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level8_Name) as cnt_Level8
From
Customer_fact,customer_dim
Where
Customer_fact.Segment_key = Customer_dim.Segment_key and
month_nm='2009-02'
Please assist...
Does this work for each of the 8 levels when they are ran independantly? Is it just 1 or 2 of the levels that cause the problem or is it just the result of combining all of the levels together in a single query?
hi,
This is a known problem in 4.x series (i havent worked on twinfin).
This problem happens, when one of your partition by clause has more rows to be processed. As you know each partition has to fit in a "single spu", so if data volume is more then system cannot handle that partitions data in another spu ( i wish i could provide exact volume i can give, but i have to dig my old-old emails .. )
eg:
count(distinct cust_key) over (partition by level1) as level1 --
if level1 has only two distinct values and any one distinct value has more records. Then system throws this error.
you can check by running select level1, count(distinct cust_key) from tab group by level1 to see how many partitions level1 creates and count.
output would look something like below
level1 | count
0 | 100000000
1 | 1500000
if you see above, one partition with 0 has more than 100 mil records. So NPS cannot handle this large partition.
Workaround is to create a intermediate table with grouped data.
i'm sure that only one or two levels out of your 8 levels would cause this error. This is system limitation, either remove that one level of analytical calculation or create one temporary table and do process in two steps.
hope this helps.
Thanks bmasna. The problem seems to be related to only one level which is having large number of records. this I suppose is exceeding the SPU memory thus causing the error.
@Shawn - Individual levels are working fine. This error occurs while combining all of the levels together in a single query.

