Skip navigation
17040 Views 3 Replies Latest reply: Feb 28, 2010 11:37 PM by Aravinth RSS
Aravinth New Enzee 2 posts since
Feb 23, 2010
Currently Being Moderated

Feb 23, 2010 4:53 AM

NZSQL Analytic Functions - Partition rowcount limit exceeded error

Hi,

 

We are trying to calculate count for different gouping (8 Levels of grouping) using a single qurey. The query works fine with 2-3 level of grouping but anything above 3 level Netezza throws up the below error.

 

 

Error: ERROR: 65534 : Partition rowcount limit exceeded

(State:HY000, Native Code: 2C)

 

Sample query:

 

Select
customer_dim.Level1_Name,
customer_dim.Level2_Name,
customer_dim.Level3_Name,
customer_dim.Level4_Name,
customer_dim.Level5_Name,
customer_dim.Level6_Name,
customer_dim.Level7_Name,
customer_dim.Level8_Name,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level1_Name) as cnt_Level1,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level2_Name) as cnt_Level2,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level3_Name) as cnt_Level3,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level4_Name) as cnt_Level4,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level5_Name) as cnt_Level5,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level6_Name) as cnt_Level6,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level7_Name) as cnt_Level7,
COUNT(DISTINCT (Customer_fact.Customer_key)) over (partition by customer_dim.Level8_Name) as cnt_Level8
From
Customer_fact,customer_dim
Where
Customer_fact.Segment_key = Customer_dim.Segment_key and
month_nm='2009-02'

 

 

 

Please assist...

  • Shawn Fox Enzee Exraordinaire 1,482 posts since
    Aug 15, 2006

    Does this work for each of the 8 levels when they are ran independantly?  Is it just 1 or 2 of the levels that cause the problem or is it just the result of combining all of the levels together in a single query?

  • bmasna New Enzee 12 posts since
    Mar 6, 2009

    hi,

    This is a known problem in 4.x series (i havent worked on twinfin).

    This problem happens, when one of your partition by clause has more rows to be processed. As you know each partition has to fit in a "single spu", so if data volume is more then system cannot handle that partitions data in another spu ( i wish i could provide exact volume i can give, but i have to dig my old-old emails .. )

    eg:

    count(distinct cust_key) over (partition by level1) as level1 --

    if level1 has only two distinct values and any one distinct value has more records. Then system throws this error.

    you can check by running select level1, count(distinct cust_key) from tab group by level1 to see how many partitions level1 creates and count.

    output would look something like below

     

    level1  | count

    0        | 100000000

    1        | 1500000

    if you see above, one partition with 0 has more than 100 mil records. So NPS cannot handle this large partition.

    Workaround is to create a intermediate table with grouped data.

     

    i'm sure that only one or two levels out of your 8 levels would cause this error. This is system limitation, either remove that one level of analytical calculation or create one temporary table and do process in two steps.

    hope this helps.

More Like This

  • Retrieving data ...

Bookmarked By (0)

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points