[Postgres-xl-general] How to distribute a table with billions of rows?

Mason Sharp msharp at translattice.com
Sat Dec 6 12:29:02 PST 2014


On Friday, December 5, 2014, Eason Yi <11719668 at qq.com> wrote:

> Thanks, Mason.
>
> I wonder how many partitions will PostgreSQL-XL automatically initially
> build for that big table internally.
>
> If I add one or more data nodes for the PostgreSQL-XL cluster later, then
> what will happen to that big table automatically.
>
>
It will pick a column and hash distribute by it, across the datanodes.

A value is calculated based on the column and it gets mapped to a hash
bucket. These hash buckets are then mapped to datanodes.

It is advisable to choose which column to distribute by and include it in
your Create table statement.


> ------------------
> Regards,
>
> Eason
>
>
>
> ------------------ Original ------------------
> *From: * "Mason Sharp";<msharp at translattice.com
> <javascript:_e(%7B%7D,'cvml','msharp at translattice.com');>>;
> *Date: * Fri, Dec 5, 2014 11:09 PM
> *To: * "Eason Yi"<11719668 at qq.com
> <javascript:_e(%7B%7D,'cvml','11719668 at qq.com');>>;
> *Cc: * "postgres-xl-general"<postgres-xl-general at lists.sourceforge.net
> <javascript:_e(%7B%7D,'cvml','postgres-xl-general at lists.sourceforge.net');>>;
>
> *Subject: * Re: [Postgres-xl-general] How to distribute a table with
> billions of rows?
>
>
>
> On Wed, Dec 3, 2014 at 7:23 AM, Eason Yi <11719668 at qq.com
> <javascript:_e(%7B%7D,'cvml','11719668 at qq.com');>> wrote:
>
>> Thank you very much!
>>
>> If I have that big table already created in the PostgresSQL 9.1 database
>> and about millions of rows had  been  inserted in it. And the table is not
>> partioned, I mean that it's just a simple table and just in one data node.
>>
>> Then how should I migrate that table and its huge data to the
>> PostgreSQL-XL cluster with serveral datanodes?
>>
>
> You should unload it with COPY on the source, and load it into Postgres-XL
> with COPY there.
>
> I suppose you could try piping these together, feeding the COPY TO STDOUT
> on the source into XL and copy FROM STIDN.
>
> Please carefully choose an appropriate DISTRIBUTE BY clause for your
> table, usually a primary key column.
>
> Good luck!
>
> Mason
>
>

-- 
Mason Sharp

TransLattice - http://www.translattice.com
Clustered and Distributed Database Solutions
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.postgres-xl.org/pipermail/postgres-xl-general-postgres-xl.org/attachments/20141206/411d915d/attachment.htm>


More information about the postgres-xl-general mailing list