Structure dataset from rows to columns pandas python -


i have dataframe following many feature columns 3 mentioned below:

productid   |feature1   |value1 |feature2    |value2     | feature3    |value3 100001      |weight     | 130g   |                       |price        |$140.50 100002      |weight     | 200g   |pieces     |12 pcs     | dimensions  |150x75cm 100003      |dimensions |70x30cm |price      |$22.90         100004      |price      |$12.90  |manufacturer| abc    |calories    |556kcal 100005      |calories   |1320kcal|dimensions |20x20cm  |manufacturer   | xyz

and want structure in following way using pandas:

productid   weight  dimensions  price   calories    no. of pieces   manufacturer 100001       130g              $140.50           100002       200g    150x75cm                         12 pcs     100003               70x30cm    $22.90           100004                          $12.90   556kcal                          abc 100005               20x20cm            1320kcal                         xyz

i studied various pandas methods reset_index, stack etc didn't convert in required way.

you looking code unpack dataframe. straightforward way (with many features , possibly repeating productids):

import pandas pd import numpy np  def expand(frame):     df = pd.dataframe()     row in frame.iterrows():         data = row[1]         feature_name, feature_value in zip(data[1::2], data[2::2]):             if feature_name:                 df.loc[data.productid, feature_name] = feature_value     return df.replace(np.nan, '')   df = pd.dataframe([("100001", "weight", "130g", none, none, "price", "$140.50"), ("100002", "weight", "200g", "pieces", "12 pcs", "dimensions", "150x75cm"), ("100003", "dimensions", "70x30cm", "price", "$22.90"), ("100004", "price", "$12.90", "manufacturer", "abc", "calories", "556kcal"), ("100005", "calories", "1320kcal", "dimensions", "20x20cm", "manufacturer", "xyz")],                   columns=["productid", "feature1", "value1", "feature2", "value2", "feature3", "value3"])  xdf = expand(df) print(xdf)

output:

       weight    price  pieces dimensions manufacturer  calories 100001   130g  $140.50                                           100002   200g           12 pcs   150x75cm                        100003          $22.90            70x30cm                        100004          $12.90                             abc   556kcal 100005                            20x20cm          xyz  1320kcal

edit1: compressed form: (slow!)

def expand2(frame):     return pd.dataframe.from_dict(         {data.productid: {f: v f, v in zip(data[1::2], data[2::2]) if f} _, data in frame.iterrows()},         orient='index')

edit2: using generator expression:

def expand3(frame):     return pd.dataframe.from_records(         ({f: v f, v in itertools.chain((('productid', data.productid),), zip(data[1::2], data[2::2])) if f}          _, data          in frame.iterrows()), index='productid').replace(np.nan, '')

some testing (decorate function s @timeit):

def timeit(f):     @functools.wraps(f)     def timed(*args, **kwargs):         try:             start_time = time.time()             return f(*args, **kwargs)         finally:             end_time = time.time()             function_invocation = "x"             sys.stdout.flush()             print(f'function {f.__name__}({function_invocation}), took: {end_time - start_time:2.4f} seconds.',                   flush=true, file=sys.stderr)      return timed  def generate_wide_df(n_rows, n_features):     possible_labels = [f'label_{i}' in range(n_features)]     columns = ['productid']     in range(1, n_features):         columns.append(f'feature_{i}')         columns.append(f'value_{i}')      df = pd.dataframe(columns=columns)     row_n in range(n_rows):         df.loc[row_n, 'productid'] = int(1000000 + row_n)         _ in range(n_features):             feature_num = random.randint(1, n_features)             df.loc[row_n, f'feature_{feature_num}'] = random.choice(possible_labels)             df.loc[row_n, f'value_{feature_num}'] = random.randint(1, 10000)     return df.where(df.notnull(), none)   df = generate_wide_df(4000, 30)   expand(df) expand3(df) expand2(df)

and results:

function expand(x), took: 1.1576 seconds. function expand3(x), took: 1.1185 seconds. function expand2(x), took: 16.3055 seconds.

Comments

Post a Comment

Popular posts from this blog

python - Operations inside variables -

this question has answer here: evaluating mathematical expression in string 12 answers i want make python script finds maximum possible output using 2 operations. problem says there syntax error @ 2nd last line, in b in there. how can fix it? x = 0 = int(input("1st number:")) c = int(input("2nd number:" )) e = int(input("3rd number:" )) in range(4): if == 0: b = "+" elif == 1: b = "-" elif == 2: b = "/" else: b = "*" j in range(4): if j == 0: d = "+" elif j == 1: d = "-" elif j == 2: d = "/" else: d = "*" k = b c d e print(k) if you're after variables e on single line, couldn...
Read more

Generic Map Parameter java -

i having lot of methods of type: public static list<eduusers> getdetails(class c, map<string, integer> params) { list<eduusers> details = null; session session = hibernateutil.getsessionfactory().opensession(); transaction tx = null; try { tx = session.begintransaction(); criteria cr = session.createcriteria(c); (map.entry<string, integer> entry : params.entryset()) { cr.add(restrictions.eq(entry.getkey(), entry.getvalue())); } details = cr.list(); tx.commit(); } catch (exception asd) { log.debug(asd.getmessage()); if (tx != null) { tx.commit(); } } { session.close(); } return details; } i trying have generic method them , have written far: public static <t> list<t> getdetails(class<t> c, class<t> b) { l...
Read more

arrays - What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it? -

what arrayindexoutofboundsexception mean , how rid of it? here code sample triggers exception: string[] name = {"tom", "dick", "harry"}; for(int = 0; i<=name.length; i++) { system.out.print(name[i] +'\n'); } your first port of call should documentation explains reasonably clearly: thrown indicate array has been accessed illegal index. index either negative or greater or equal size of array. so example: int[] array = new int[5]; int boom = array[10]; // throws exception as how avoid it... um, don't that. careful array indexes. one problem people run thinking arrays 1-indexed, e.g. int[] array = new int[5]; // ... populate array here ... (int index = 1; index <= array.length; index++) { system.out.println(array[index]); } that miss out first element (index 0) , throw exception when index 5. valid indexes here 0-4 inclusive. correct, idiomatic for statement here be: for (int index = 0; index <...
Read more