[Postgres-xl-developers] PostgresXL -- Data importing is very slow

Pavan Deolasee pavan.deolasee at gmail.com
Wed May 4 03:41:02 PDT 2016


On Thu, Apr 14, 2016 at 5:41 PM, Sairam V <v.sairam1 at gmail.com> wrote:

> Hi Team,
>
> It is great to work with PostgresXL till now. But I need some support
> here. I have 10 GB size of 8 files. Each file has records/events( 50
> million) that needs to be imported into the Database table.
> My DB Cluster details are as follow :
>
> 1 GTM +1 Coordinator, 1DataNode + 1Coordinator, 1DataNode
> ( 3 instances == > 1 gtm instance, coordinator 01 and datanode 01 in one
> instance, coordinator 02 and datanode 02 in another instance)
>
>
Can you tell us which version of Postgres-XL you're currently using?


> What I observe is that to load a 10GB file without creation of indexes,
> takes nearly 2 plus hours of time when compared to some 45 minutes when the
> same is done in Postgresql.
>

How do you load the data? Using COPY protocol or some other mechanism?


>
> So I have 8 such files and it takes for me 22 hours nearly to just load
> the data in PostgresXL.
>
> My machine details coordinator and datanode instances:
>
> 16 core cpu, 70 GB memroy and 850 GB disk space.
>
> Please help me to optimize things if any and would appreciate any kind of
> support.
>
>
 If you're using a 9.5 version and COPY, I would suggest starting parallel
load using both the coordinators. In my experience, splitting the data into
multiple chunks and starting for than one COPY streams helps XL to use all
its resources more efficiently.

Thanks,
Pavan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.postgres-xl.org/private.cgi/postgres-xl-developers-postgres-xl.org/attachments/20160504/3045a28c/attachment.htm>


More information about the Postgres-xl-developers mailing list