python - Is there a faster way of reading in GML files than this R code? -


i've got 11,000 gml files total 0.5tb when uncompressed. i've written code in r job files take 10 minutes each. code reads each file, takes out building polygons , saves rds file, rds file smaller , contain code need - make quicker read in on future uses.

any suggestions different ways of writing code appreciated, or there different language better @ stuff - python example? current approach run same code on multiple r sessions.

here's code:

library(rgdal)  #get files in folders os_in_dir <- "d:/rhome/os_topo" os_out_dir <- "d:/rhome/os_topo_out" folders <- list.files(path = os_in_dir) # files <- list.files(path = paste0(os_in_dir,"/",folders[1]), pattern = "*.gz$") # o_files <- sub(".gz",".rds",files)  (i in 1:length(folders)) {     files <- list.files(path = paste0(os_in_dir,"/",folders[i]), pattern = "*.gz$")     o_files <- sub(".gz",".rds",files)     logevent(paste("processing folder:",folders[i],"-",length(files),"files process."))     (j in ifelse(i==23,95,1):length(files)) {         logevent(paste("processing file:",files[j]))         flag <- true         ta <- na         trycatch({             ta <<- readogr(paste0(os_in_dir,"/",folders[i],"/",files[j]), layer = 'topographicarea') #read in building polygon data         },         error = function(e) {             logevent(paste("error on file:",files[j]))             flag <<- false         }         )         if (!flag) next #if errored skip on next file          o_rc <- nrow(ta)         ta <- ta[ta@data$theme1 == "buildings",] #get buildings data         b_rc <- nrow(ta)         pct_b <- round((b_rc/o_rc)*100)         logevent(paste("done file:",files[j],"-",b_rc,"out of",o_rc,"were buildings (",pct_b,"% )"))         if (b_rc > 0) {             logevent(paste("saving file",files[j],"as",o_files[j]))             saverds(ta, paste0(os_out_dir,"/",o_files[j])) #save file rds file         }         gc()     }     gc() }


Comments

Post a Comment

Popular posts from this blog

python - Operations inside variables -

this question has answer here: evaluating mathematical expression in string 12 answers i want make python script finds maximum possible output using 2 operations. problem says there syntax error @ 2nd last line, in b in there. how can fix it? x = 0 = int(input("1st number:")) c = int(input("2nd number:" )) e = int(input("3rd number:" )) in range(4): if == 0: b = "+" elif == 1: b = "-" elif == 2: b = "/" else: b = "*" j in range(4): if j == 0: d = "+" elif j == 1: d = "-" elif j == 2: d = "/" else: d = "*" k = b c d e print(k) if you're after variables e on single line, couldn...
Read more

Generic Map Parameter java -

i having lot of methods of type: public static list<eduusers> getdetails(class c, map<string, integer> params) { list<eduusers> details = null; session session = hibernateutil.getsessionfactory().opensession(); transaction tx = null; try { tx = session.begintransaction(); criteria cr = session.createcriteria(c); (map.entry<string, integer> entry : params.entryset()) { cr.add(restrictions.eq(entry.getkey(), entry.getvalue())); } details = cr.list(); tx.commit(); } catch (exception asd) { log.debug(asd.getmessage()); if (tx != null) { tx.commit(); } } { session.close(); } return details; } i trying have generic method them , have written far: public static <t> list<t> getdetails(class<t> c, class<t> b) { l...
Read more

arrays - What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it? -

what arrayindexoutofboundsexception mean , how rid of it? here code sample triggers exception: string[] name = {"tom", "dick", "harry"}; for(int = 0; i<=name.length; i++) { system.out.print(name[i] +'\n'); } your first port of call should documentation explains reasonably clearly: thrown indicate array has been accessed illegal index. index either negative or greater or equal size of array. so example: int[] array = new int[5]; int boom = array[10]; // throws exception as how avoid it... um, don't that. careful array indexes. one problem people run thinking arrays 1-indexed, e.g. int[] array = new int[5]; // ... populate array here ... (int index = 1; index <= array.length; index++) { system.out.println(array[index]); } that miss out first element (index 0) , throw exception when index 5. valid indexes here 0-4 inclusive. correct, idiomatic for statement here be: for (int index = 0; index <...
Read more