[Postgres-xl-general] "ERROR: Failed to get pooled connections" with cross-node join
Tobias Oberstein
tobias.oberstein at gmail.com
Thu Oct 1 09:28:50 PDT 2015
> Try increasing max_pool_size on the datanodes too.
Ok, increased to 2048 as well. Rebuilt whole cluster and test data. Same
issue.
> There are some compile time #defines for worker threads etc, but other
> than that there is no tunable for GTM.
FWIW, when stuff breaks, I see 10k's of messages like the following in
the GTM log:
1:139736683767552:2015-10-01 17:46:32.257 CEST -LOG: unexpected EOF on
client connection
LOCATION: ReadCommand, main.c:1481
1:139769623611136:2015-10-01 17:46:34.727 CEST -LOG: Too many threads
active
LOCATION: GTM_ThreadAdd, gtm_thread.c:53
==
My unscientific best guess would be: GTM falls apart under cross-node
joins with 32 nodes. But I have of course no clue what's really going on.
> I wonder if anyone has run PG-XL with large node counts and driven by
> more than toy workloads .. anyone?
>
>
> I haven't heard of cases when someone tried or using it with more than
> 16 nodes. But recently we are getting aware of cases where people are
Huh, ok. That is a bummer. 16 nodes is nothing. What we are aiming for
is scaling to at least 256 nodes (4 boxes a 4 sockets a 16 cores) and
parallelize to a degree of perhaps 64.
That is: drive 64 cores from a single query. Since there are 256 cores,
I'd expect to be able to sustain 4 such queries running concurrently
then to saturate the whole cluster.
> using it in production, but those cases don't get reported here,
> especially if things work out fine.
>
> We (developers) are also limited by the hardware that is accessible to
> us. So doing any tests which involves significant hardware is
> practically impossible for us. Any help in that area is highly
> appreciated. I know Krzysztof is doing many tests with real world use
> cases and reporting issues. Thats really helpful and we are committed to
> keep improving stability of the product, more than adding new features
> at this stage.
I totally understand. My problem is: I am expected to give an opinion on
feasibility of using PG-XL to drive a finance DWH now. Hell, I'd be
happy to recommend PG-XL. But. And after reading through the dev list, I
have more questions ..
>
> Thanks,
> Pavan
>
> --
> Pavan Deolasee http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
More information about the postgres-xl-general
mailing list