[Postgres-xl-developers] SQueue X, timeout while waiting for Consumers finishing

Pavan Deolasee pavan.deolasee at gmail.com
Wed Aug 16 01:12:44 PDT 2017


On Sat, Aug 12, 2017 at 4:19 AM, Tomas Vondra <tomas.vondra at 2ndquadrant.com>
wrote:

> Hi,
>
> Over the past few days I've been looking into what's causing error
> messages like this
>
>   WARNING: SQueue p_1_2_3, timeout while waiting for Consumers finishing
>
> Typically associated with a 10-second stall due to a timeout. It's quite
> annoying, because it's somewhat unpredictable.
>

I agree.


>
>
> The problem is that "t2" only has a single row, which means it's empty on
> one of the datanodes (say, dn1)  What seems to be happening is dn1 starts
> executing the Hash Join, and the first thing it does is building the hash
> table.
>
> dn1 then realizes it can't possibly produce any rows with an empty hash
> table (for an inner join), and entirely skips the Remote Subquery bit.
> Which means it never even connects to the SQueue on dn2, does not fetch any
> data from it nor does it notify dn2 not to wait for it.
>
> So dn2 ends up with data for dn1 still in the queue, and the way this is
> handled right now is "wait for 10 seconds and then give up". Which is
> exactly what the warning message is about.
>
>
I'd investigated this is the past and concluded that the datanode which
does not join the shared queue actually does send a 'close' message when
the executor is teared down, but if the teardown process tries to destroy
the producer first, it gets into that 10s timed wait. Thus I'd come up with
the attached patch which marks consumers as disconnected and then wakes up
the producer.

The patch seems to work for some of the test cases I'd. For example, the
following test case does not get into the timed wait with the patch applied.

CREATE TABLE t1 (a int, b text, c int);
CREATE TABLE t2 (cc int);

INSERT INTO t1 VALUES (1, 'one', 10);
INSERT INTO t1 VALUES (2, 'two', 20);
INSERT INTO t2 VALUES (10), (20);

SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE t2.cc = t1.c);

But while running regression, I still see at least one timeout, which seems
to be happening during rollback. So the patch needs more work to handle
those cases too.


>
> What I'm pretty sure about is that we should get rid of the timeouts
> altogether, as it's rather unreliable way to deal with this. For example,
> what if building the hash table takes 15 seconds on one of the datanodes,
> for whatever reason (much more data, the node being temporarily overloaded,
> ...)? I'd bet the other datanodes will wait for the 10 seconds, and then
> give up. I don't think think could produce incorrect results, though.
>
> This also means lowering the timeout value is not a viable approach, as it
> makes such failures more likely.
>

Yes, I agree. Do you think the general idea behind the attached patch is
sensible? If so, we can possibly trace out the timeout in failure-handling
and address that. Also, can you please try the patch in the test scenarios
that you may have and see if it helps? The patch is based on the current
XL9_5_STABLE branch.

Thanks,
Pavan

-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.postgres-xl.org/private.cgi/postgres-xl-developers-postgres-xl.org/attachments/20170816/9c094239/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: xl_shared_q_timeout_v2.patch
Type: application/octet-stream
Size: 8457 bytes
Desc: not available
URL: <http://lists.postgres-xl.org/private.cgi/postgres-xl-developers-postgres-xl.org/attachments/20170816/9c094239/attachment.obj>


More information about the Postgres-xl-developers mailing list