python - More efficient way to get various token count stats from array and list -


i'm classifying spam list of email text (stored in csv format), before can this, want simple count stats output. used countvectorizer sklearn first step , implemented following code

import pandas pd import numpy np sklearn.model_selection import train_test_split sklearn.feature_extraction.text import countvectorizer  #import data csv  spam = pd.read_csv('spam.csv') spam['spam'] = np.where(spam['spam']=='spam',1,0)  #split data  x_train, x_test, y_train, y_test = train_test_split(spam_data['text'], spam_data['target'], random_state=0)   #convert 'features' numeric , matrix or list cv = countvectorizer() x_traincv = cv.fit_transform(x_train) = x_traincv.toarray() a_list = cv.inverse_transform(a)

the output stored in matrix (named 'a') or list of arrays (named 'a_list') format looks this

[array(['do', 'i', 'off', 'text', 'where', 'you'],         dtype='<u32'),  array(['ages', 'will', 'did', 'driving', 'have', 'hello', 'hi', 'hol', 'in', 'its', 'just', 'mate', 'message', 'nice', 'off', 'roads', 'say', 'sent', 'so', 'started', 'stay'], dtype='<u32'),              ...  array(['biz', 'for', 'free', 'is', '1991', 'network', 'operator', 'service', 'the', 'visit'], dtype='<u32')]

but found little difficult simple count stats these outputs, such longest/shortest token, average length of tokens, etc. how can these simple count stats matrix or list output generated?

you can load tokens, token counts, , token lengths new pandas dataframe, custom queries.

here simple example toy data set.

import pandas pd import numpy np sklearn.feature_extraction.text import countvectorizer  texts = ["dog cat fish","dog cat cat","fish bird walrus monkey","bird lizard"]  cv = countvectorizer() cv_fit = cv.fit_transform(texts) # https://stackoverflow.com/a/16078639/2491761 tokens_and_counts = zip(cv.get_feature_names(), np.asarray(cv_fit.sum(axis=0)).ravel())  df = pd.dataframe(tokens_and_counts, columns=['token', 'count'])  df['length'] = df.token.str.len() # https://stackoverflow.com/a/29869577/2491761  # tokens length equal min token length: df.loc[df['length'] == df['length'].min(), 'token']  # tokens length equal max token length: df.loc[df['length'] == df['length'].max(), 'token']  # tokens length less mean token length: df.loc[df['length'] < df['length'].mean(), 'token']  # tokens length greater 1 standard deviation mean: df.loc[df['length'] > df['length'].mean() + df['length'].std(), 'token']

can extended if want queries based on counts.


Comments

Post a Comment

Popular posts from this blog

python - Operations inside variables -

this question has answer here: evaluating mathematical expression in string 12 answers i want make python script finds maximum possible output using 2 operations. problem says there syntax error @ 2nd last line, in b in there. how can fix it? x = 0 = int(input("1st number:")) c = int(input("2nd number:" )) e = int(input("3rd number:" )) in range(4): if == 0: b = "+" elif == 1: b = "-" elif == 2: b = "/" else: b = "*" j in range(4): if j == 0: d = "+" elif j == 1: d = "-" elif j == 2: d = "/" else: d = "*" k = b c d e print(k) if you're after variables e on single line, couldn...
Read more

Generic Map Parameter java -

i having lot of methods of type: public static list<eduusers> getdetails(class c, map<string, integer> params) { list<eduusers> details = null; session session = hibernateutil.getsessionfactory().opensession(); transaction tx = null; try { tx = session.begintransaction(); criteria cr = session.createcriteria(c); (map.entry<string, integer> entry : params.entryset()) { cr.add(restrictions.eq(entry.getkey(), entry.getvalue())); } details = cr.list(); tx.commit(); } catch (exception asd) { log.debug(asd.getmessage()); if (tx != null) { tx.commit(); } } { session.close(); } return details; } i trying have generic method them , have written far: public static <t> list<t> getdetails(class<t> c, class<t> b) { l...
Read more

arrays - What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it? -

what arrayindexoutofboundsexception mean , how rid of it? here code sample triggers exception: string[] name = {"tom", "dick", "harry"}; for(int = 0; i<=name.length; i++) { system.out.print(name[i] +'\n'); } your first port of call should documentation explains reasonably clearly: thrown indicate array has been accessed illegal index. index either negative or greater or equal size of array. so example: int[] array = new int[5]; int boom = array[10]; // throws exception as how avoid it... um, don't that. careful array indexes. one problem people run thinking arrays 1-indexed, e.g. int[] array = new int[5]; // ... populate array here ... (int index = 1; index <= array.length; index++) { system.out.println(array[index]); } that miss out first element (index 0) , throw exception when index 5. valid indexes here 0-4 inclusive. correct, idiomatic for statement here be: for (int index = 0; index <...
Read more