[Postgres-xl-general] "ERROR: Failed to get pooled connections" with cross-node join

Tobias Oberstein tobias.oberstein at gmail.com
Thu Oct 1 09:28:50 PDT 2015


> Try increasing max_pool_size on the datanodes too.

Ok, increased to 2048 as well. Rebuilt whole cluster and test data. Same 
issue.

> There are some compile time #defines for worker threads etc, but other
> than that there is no tunable for GTM.

FWIW, when stuff breaks, I see 10k's of messages like the following in 
the GTM log:

1:139736683767552:2015-10-01 17:46:32.257 CEST -LOG:  unexpected EOF on 
client connection
LOCATION:  ReadCommand, main.c:1481

1:139769623611136:2015-10-01 17:46:34.727 CEST -LOG:  Too many threads 
active
LOCATION:  GTM_ThreadAdd, gtm_thread.c:53

==

My unscientific best guess would be: GTM falls apart under cross-node 
joins with 32 nodes. But I have of course no clue what's really going on.

>     I wonder if anyone has run PG-XL with large node counts and driven by
>     more than toy workloads .. anyone?
>
>
> I haven't heard of cases when someone tried or using it with more than
> 16 nodes. But recently we are getting aware of cases where people are

Huh, ok. That is a bummer. 16 nodes is nothing. What we are aiming for 
is scaling to at least 256 nodes (4 boxes a 4 sockets a 16 cores) and 
parallelize to a degree of perhaps 64.

That is: drive 64 cores from a single query. Since there are 256 cores, 
I'd expect to be able to sustain 4 such queries running concurrently 
then to saturate the whole cluster.

> using it in production, but those cases don't get reported here,
> especially if things work out fine.
>
> We (developers) are also limited by the hardware that is accessible to
> us. So doing any tests which involves significant hardware is
> practically impossible for us. Any help in that area is highly
> appreciated. I know Krzysztof is doing many tests with real world use
> cases and reporting issues. Thats really helpful and we are committed to
> keep improving stability of the product, more than adding new features
> at this stage.

I totally understand. My problem is: I am expected to give an opinion on 
feasibility of using PG-XL to drive a finance DWH now. Hell, I'd be 
happy to recommend PG-XL. But. And after reading through the dev list, I 
have more questions ..

>
> Thanks,
> Pavan
>
> --
>   Pavan Deolasee http://www.2ndQuadrant.com/
>   PostgreSQL Development, 24x7 Support, Training & Services





More information about the postgres-xl-general mailing list