Geek asks and answers

Smooth a tuple like a bag

My dataset is as follows:

( A, (1,2) )
( B, (2,9) )

I would like to “smooth out” the tuples in Pig, basically repeating each record for each value found in the inner tuple, so the expected result:

( A, 1 )
( A, 2 )
( B, 2 ) 
( B, 9 )

I know this is possible when tuples (1,2) and (2,9) are bags instead.

+5
source share
3 answers

Your understanding is good; it is possible by converting a tuple in a bag. The scheme we want to strive for: {a: chararray, {(chararray)}} for example: (A, {(1), (2)})

Here is the solution to your problem:

A = LOAD 'data.txt' AS (a:chararray,b:(b1:chararray,b2:chararray));
B = FOREACH A GENERATE a, TOBAG(b.b1,b.b2);
C = FOREACH B GENERATE a, FLATTEN($1);

The magic part is the TOBAG operator.

+9
source

I know this is an old thread, but I could not get this method to work. I think I will share my conclusions.

input: (1-2-3, abc)
       (4-5-6, xyz)
desired output:
       (1, abc)
       (2, abc)
       (3, abc)
       (4, xyz)
       (5, xyz)
       (6, xyz)

I originally used STRSPLIT, which generates a tuple leading to a similar input, as mentioned above, but failed.

output = FOREACH input GENERATE FLATTEN(TOBAG(STRSPLIT($0, '-'))), $1

As a result, the result was as follows:

 (1,2,3,abc)
 (4,5,6,xyz)

However, when I used tokenize and replaced functions, I got the desired result.

output = FOREACH input GENERATE FLATTEN(TOKENIZE(REPLACE($0,'-', ' '))), $1;
0
source

More articles:


All Articles