[Postgres-xl-bugs] GTM generated global XID not available

Pavan Deolasee pavan.deolasee at gmail.com
Wed Jul 29 22:51:46 PDT 2015


On Wed, Jul 29, 2015 at 2:26 PM, Krzysztof Nienartowicz <
krzysztof.nienartowicz at gmail.com> wrote:

> Hello,
> Usual setup: 4 datanodes collocated with 4 coords, XL patched for to
> enable partitioning triggers from the trunk, gtm collocated with
> coord0/dn0, gtm_proxy on each node.
> After several hours of successful ingestion using copy on partitioned
> table using around 60 workers on a single coord1 we got errors appearing in
> various parts of the system around one hour or so. I will investigate if
> there was a network glitch (it's isolated infiniband between XL nodes), but
> it's unlikely.
>
>
Hi Krysztof,

Thanks for the report. May I ask which sources you are currently using? I
ask because there is a new repository based on 9.5 we recently announced
and I would like to know if you are running from there.

Also, are these intermittent errors which go away after a while or when you
restart the command again?

Another question: is the data is partitioned in a way such that each COPY
command will hit only one datanode?


Thanks,
Pavan


> 1. on the coord1:
>
> 2015-07-29 08:02:49.115 CEST,,,20588,,55b86c84.506c,3,,2015-07-29 08:02:44 CEST,6/2659273,34925596,LOG,00000,"Getting detailed statistics for dpac_or5_s2_epsl_dex.mdb_cu1_completesource_027",,,,,,,,"analyze_rel_coordinator, analyze.c:2865",""
>
> 2015-07-29 08:19:40.223 CEST,,,34820,,55ac294d.8804,897,,2015-07-20 00:48:45 CEST,,0,LOG,00000,"checkpoint starting: time",,,,,,,,"LogCheckpointStart, xlog.c:7885",""
>
> 2015-07-29 08:19:40.246 CEST,,,34820,,55ac294d.8804,898,,2015-07-20 00:48:45 CEST,,0,LOG,00000,"checkpoint complete: wrote 0 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=0.020 s, sync=0.000 s, total=0.022 s; sync files=0, longest=0.000 s, average=0.000 s",,,,,,,,"LogCheckpointEnd, xlog.c:7973",""
>
> 2015-07-29 08:33:14.518 CEST,,,34823,,55ac294d.8807,2,,2015-07-20 00:48:45 CEST,1/985,0,ERROR,58000,"GTM generated global XID not available",,"Check if GTM/GTM-proxy is running @ pigdb01i:20002 and reachable from this host. Your firewall could also block access to a host/port",,,,,,"GetNewTransactionId, varsup.c:210",""
>
> 2015-07-29 08:33:15.533 CEST,,,34823,,55ac294d.8807,3,,2015-07-20 00:48:45 CEST,1/986,34925618,ERROR,XX000,"Cannot set snapshot from global snapshot",,,,,,,,"GetSnapshotFromGlobalSnapshot, procarray.c:2931",""
>
> 2015-07-29 08:33:16.535 CEST,,,34823,,55ac294d.8807,4,,2015-07-20 00:48:45 CEST,1/987,34925618,WARNING,01000,"Unexpected clog condition. curval = 2, status = 1",,,,,,,,"TransactionIdSetStatusBit, clog.c:367",""
>
>
> 2015-07-29 08:49:40.345 CEST,,,34820,,55ac294d.8804,899,,2015-07-20 00:48:45 CEST,,0,LOG,00000,"checkpoint starting: time",,,,,,,,"LogCheckpointStart, xlog.c:7885",""
>
> 2. On GTM:
> 1:140532557694720:2015-07-29 07:15:38.525 CEST -WARNING:
> No transaction handle for gxid: 34919674
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:38.525 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:15:38.525 CEST -ERROR:
> Failed to get a snapshot
> LOCATION:  ProcessGetSnapshotCommandMulti, gtm_snap.c:420
> 1:140532557694720:2015-07-29 07:15:38.544 CEST -WARNING:
> No transaction handle for gxid: 34919674
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:38.544 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:15:39.587 CEST -WARNING:
> No transaction handle for gxid: 34919676
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:39.587 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:15:59.054 CEST -WARNING:
> No transaction handle for gxid: 34919674
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:59.054 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:15:59.055 CEST -WARNING:
> No transaction handle for gxid: 34919902
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:59.055 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:15:59.055 CEST -ERROR:
> Failed to get a snapshot
> LOCATION:  ProcessGetSnapshotCommandMulti, gtm_snap.c:420
> 1:140532557694720:2015-07-29 07:15:59.055 CEST -WARNING:
> No transaction handle for gxid: 34919902
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:59.055 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:15:59.059 CEST -WARNING:
> No transaction handle for gxid: 34919902
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 07:15:59.059 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 07:17:02.563 CEST -LOG:
> Saving transaction info - next_gxid: 34920297
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2822
> 1:140532557694720:2015-07-29 07:17:07.999 CEST -LOG:
> Saving transaction restoration info, backed-up gxid: 34922383
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2839
> 1:140532557694720:2015-07-29 07:19:32.950 CEST -LOG:
> Saving transaction info - next_gxid: 34921298
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2822
> 1:140532557694720:2015-07-29 07:22:13.373 CEST -LOG:
> Saving transaction info - next_gxid: 34922300
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2822
> 1:140532557694720:2015-07-29 07:22:25.412 CEST -LOG:
> Saving transaction restoration info, backed-up gxid: 34924387
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2839
> 1:140532557694720:2015-07-29 07:25:42.497 CEST -LOG:
> Saving transaction info - next_gxid: 34923301
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2822
> 1:140532557694720:2015-07-29 07:28:29.053 CEST -LOG:
> Saving transaction info - next_gxid: 34924302
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2822
> 1:140532557694720:2015-07-29 07:29:17.987 CEST -LOG:
> Saving transaction restoration info, backed-up gxid: 34926391
> LOCATION:  GTM_WriteRestorePointXid, gtm_txn.c:2839
> 1:140532557694720:2015-07-29 07:31:33.749 CEST -LOG:
> Saving transaction info - next_gxid: 34925303
> LOCATION:  GTM_SaveTxnInfo, gtm_txn.c:2822
> 1:140532549302016:2015-07-29 08:15:49.345 CEST -WARNING:
> No transaction handle for gxid: 34851476
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532549302016:2015-07-29 08:15:49.345 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 08:33:16.536 CEST -WARNING:
> No transaction handle for gxid: 34925618
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 08:33:16.536 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 08:33:16.536 CEST -ERROR:
> Failed to get a snapshot
> LOCATION:  ProcessGetSnapshotCommandMulti, gtm_snap.c:420
> 1:140532557694720:2015-07-29 08:33:16.536 CEST -WARNING:
> No transaction handle for gxid: 34925618
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 08:33:16.536 CEST -WARNING:
> Invalid transaction handle: -1
> LOCATION:  GTM_HandleToTransactionInfo, gtm_txn.c:227
> 1:140532557694720:2015-07-29 08:33:16.537 CEST -WARNING:
> No transaction handle for gxid: 34925618
> LOCATION:  GTM_GXIDToHandle, gtm_txn.c:177
> 1:140532557694720:2015-07-29 08:33:16.537 CEST -WARNING:
> Invalid transaction handle: -1
>
>
> 3. on the gtm_proxy:
>
> 1:139673966851840:2015-07-29 08:32:40.811 CEST -LOG:
> cleaning up client disconnection
> LOCATION:  GTMProxy_ProcessPendingCommands, proxy_main.c:2859
> 1:139673966851840:2015-07-29 08:33:16.536 CEST -ERROR:  Wrong result
> LOCATION:  ProcessResponse, proxy_main.c:1863
> 1:139673966851840:2015-07-29 08:33:16.536 CEST -ERROR2:
> Transaction commit failed
> LOCATION:  ProcessResponse, proxy_main.c:1818
> 1:139673966851840:2015-07-29 08:33:16.536 CEST -LOG:
> cleaning up client disconnection
> LOCATION:  GTMProxy_ProcessPendingCommands, proxy_main.c:2859
> 1:139673966851840:2015-07-29 08:33:16.537 CEST -ERROR2:
> Transaction commit failed
> LOCATION:  ProcessResponse, proxy_main.c:1818
> 1:139673966851840:2015-07-29 08:33:16.537 CEST -LOG:
> cleaning up client disconnection
> LOCATION:  GTMProxy_ProcessPendingCommands, proxy_main.c:2859
>
> 4. on datanode1 the only unusual thing was longer begin transaction times
> appearing around same time roughly:
> 2015-07-29 07:25:50.504 CEST,"dpac_or5_s2_epsl_dex","surveys",15943,"
> 10.0.0.171:54107",55b860f9.3e47,16,"COPY",2015-07-29 07:13:29
> CEST,46/128614,34923357,LOG,00000,"duration: 7399.463 ms  statement: COPY
> dpac_or5_s2_epsl_dex.mdb_cu1_completesource_027 FROM STDIN DELIMI
> TER AS ';'",,,,,,,,"exec_simple_query, postgres.c:1354","pgxc"
> 2015-07-29 07:25:50.504 CEST,"dpac_or5_s2_epsl_dex","surveys",18147,"
> 10.0.0.171:55970",55b863da.46e3,2,"BEGIN",2015-07-29 07:25:46
> CEST,6/1184173,0,LOG,00000,"duration: 7679.310 ms  statement:
> BEGIN",,,,,,,,"exec_simple_query, postgres.c:1354","pgxc"
> 2015-07-29 07:25:50.586 CEST,"dpac_or5_s2_epsl_dex","surveys",18146,"
> 10.0.0.171:55965",55b863d7.46e2,2,"COPY",2015-07-29 07:25:43
> CEST,3/53682354,34923381,LOG,00000,"duration: 7812.481 ms  statement: COPY
> dpac_or5_s2_epsl_dex.mdb_cu1_completesource_027 FROM STDIN DELIMI
> TER AS ';'",,,,,,,,"exec_simple_query, postgres.c:1354","pgxc"
> 2015-07-29 07:25:50.723 CEST,"dpac_or5_s2_epsl_dex","surveys",15779,"
> 10.0.0.171:53921",55b860a8.3da3,30,"COPY",2015-07-29 07:12:08
> CEST,38/121950,34923348,LOG,00000,"duration: 7617.987 ms  statement: COPY
> dpac_or5_s2_epsl_dex.mdb_cu1_completesource_027 FROM STDIN DELIMI
> TER AS ';'",,,,,,,,"exec_simple_query, postgres.c:1354","pgxc"
>
> Best regards,
> Krzysztof
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Postgres-xl-bugs mailing list
> Postgres-xl-bugs at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/postgres-xl-bugs
>
>


-- 
 Pavan Deolasee                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.postgres-xl.org/private.cgi/postgres-xl-bugs-postgres-xl.org/attachments/20150730/056e135e/attachment.htm>


More information about the postgres-xl-bugs mailing list