In PHP(Codeigniter) After inserting few data in Mongodb insertion is getting slow -


in mongodb after inserting few data insertion getting slow, using batch_insert insertion. while inserting need check conditions also.

to insert 20k data takes more 1 hour.

in temporary_table having 1l data

$batchsize = 20; $documents = array(); $count = count($pending_contacts_data); $count =1; $temporary_data = array();  for($i=0;$i<$count;$i++){    $multiple_temporary_data = $this->mongo_db->select('*')->where(array('contact_id'=>(int)4,'status'=>1))->limit(10000)->get('temporary_table');   $temporary_data = array_merge($temporary_data,$multiple_temporary_data); }  $count1 = count($temporary_data); $documents = array(); ($i=0; $i<$count1; $i++) {    $obj_id[] = $temporary_data[$i]->_id;   $test_email = $this->mongo_db->select('*')->where(array('encrypted_email'=>$temporary_data[$i]->encrypted_email))->get('email_table');    if(empty($test_email)){     $document = array('email_id'=>$temporary_data[$i]->email,                       'encrypted_email'=>$temporary_data[$i]->encrypted_email,                       'encrypted_key'=>$temporary_data[$i]->encrypted_key,                       'encrypted_iv'=>$temporary_data[$i]->encrypted_iv,                       'status'=>(int)1,                       'opend_supression_status'=>''                      );       array_push($documents, $document);      if ((($i % $batchsize) === 0)) {            $insert = $this->mongo_db->batch_insert('opend_contacts_email_new1',$documents);           $update_temporary  =$this->mongo_db->where_in('_id',$obj_id)->set(array('status'=>13))->update_all('temporary_data');           $documents = array();     }   } }

what went wrong code after 500th element each new item acted single batch insert. , $documents array of 1 element rest of loop. u should have used modulus operator %

$count = 20000; $batchsize = 50;  ($i=1; $i<=$count; ++$i){    $test_email = $this->mongo_db->select('*')->where(array('email_id'=>$temporary_data[$i]->email_id))->get('email_table');    if(empty($test_email)){     $document = array('email_id'=>$temporary_data[$i]->email,                           'encrypted_email'=>$temporary_data[$i]->encrypted_email,                           'encrypted_key'=>$temporary_data[$i]->encrypted_key,                           'encrypted_iv'=>$temporary_data[$i]->encrypted_iv,                           'status'=>(int)1,                           'suppression_status'=>''     );     array_push($documents, $document);      if (($i % $batchsize) === 0) {                    $insert = $this->mongo_db->batch_insert('email_table',$documents);           $documents = array();     }   } }

Comments

Post a Comment

Popular posts from this blog

python - Operations inside variables -

this question has answer here: evaluating mathematical expression in string 12 answers i want make python script finds maximum possible output using 2 operations. problem says there syntax error @ 2nd last line, in b in there. how can fix it? x = 0 = int(input("1st number:")) c = int(input("2nd number:" )) e = int(input("3rd number:" )) in range(4): if == 0: b = "+" elif == 1: b = "-" elif == 2: b = "/" else: b = "*" j in range(4): if j == 0: d = "+" elif j == 1: d = "-" elif j == 2: d = "/" else: d = "*" k = b c d e print(k) if you're after variables e on single line, couldn...
Read more

Generic Map Parameter java -

i having lot of methods of type: public static list<eduusers> getdetails(class c, map<string, integer> params) { list<eduusers> details = null; session session = hibernateutil.getsessionfactory().opensession(); transaction tx = null; try { tx = session.begintransaction(); criteria cr = session.createcriteria(c); (map.entry<string, integer> entry : params.entryset()) { cr.add(restrictions.eq(entry.getkey(), entry.getvalue())); } details = cr.list(); tx.commit(); } catch (exception asd) { log.debug(asd.getmessage()); if (tx != null) { tx.commit(); } } { session.close(); } return details; } i trying have generic method them , have written far: public static <t> list<t> getdetails(class<t> c, class<t> b) { l...
Read more

arrays - What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it? -

what arrayindexoutofboundsexception mean , how rid of it? here code sample triggers exception: string[] name = {"tom", "dick", "harry"}; for(int = 0; i<=name.length; i++) { system.out.print(name[i] +'\n'); } your first port of call should documentation explains reasonably clearly: thrown indicate array has been accessed illegal index. index either negative or greater or equal size of array. so example: int[] array = new int[5]; int boom = array[10]; // throws exception as how avoid it... um, don't that. careful array indexes. one problem people run thinking arrays 1-indexed, e.g. int[] array = new int[5]; // ... populate array here ... (int index = 1; index <= array.length; index++) { system.out.println(array[index]); } that miss out first element (index 0) , throw exception when index 5. valid indexes here 0-4 inclusive. correct, idiomatic for statement here be: for (int index = 0; index <...
Read more