Spark application fails due to "FileSystem closed" exception -


my spark application failed due "filesystem closed" exception. typical stacktrace attached @ end. did research , indicates executor got shutdown (see this post).

this happen due large shuffle or out-of-memory exceptions, couldn't find of in yarn log.

my question is:

  1. if exception happened, find them if not in yarn logs?
  2. if exceptions never happen, should other investigating yarn log?

thanks!

here snippet of yarn log:

2017-08-21 01:55:10,668 error  org.apache.spark.scheduler.livelistenerbus: listener eventlogginglistener threw exception java.io.ioexception: filesystem closed         @ org.apache.hadoop.hdfs.dfsclient.checkopen(dfsclient.java:837)         @ org.apache.hadoop.hdfs.dfsoutputstream.flushorsync(dfsoutputstream.java:2170)         @ org.apache.hadoop.hdfs.dfsoutputstream.hflush(dfsoutputstream.java:2116)         @ org.apache.hadoop.fs.fsdataoutputstream.hflush(fsdataoutputstream.java:130)         @ sun.reflect.generatedmethodaccessor11.invoke(unknown source)         @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)         @ java.lang.reflect.method.invoke(method.java:498)         @ com.pepperdata.common.reflect.b.b(sourcefile:149)         @ com.pepperdata.common.reflect.b.c(sourcefile:205)         @ com.pepperdata.supervisor.agent.resource.t.b(sourcefile:102)         @ com.pepperdata.supervisor.agent.resource.i.hflush(sourcefile:61)         @ org.apache.spark.scheduler.eventlogginglistener$$anonfun$logevent$3.apply(eventlogginglistener.scala:140)         @ org.apache.spark.scheduler.eventlogginglistener$$anonfun$logevent$3.apply(eventlogginglistener.scala:140)         @ scala.option.foreach(option.scala:236)         @ org.apache.spark.scheduler.eventlogginglistener.logevent(eventlogginglistener.scala:140)         @ org.apache.spark.scheduler.eventlogginglistener.onjobstart(eventlogginglistener.scala:163)         @ org.apache.spark.scheduler.sparklistenerbus$class.dopostevent(sparklistenerbus.scala:37)         @ org.apache.spark.scheduler.livelistenerbus.dopostevent(livelistenerbus.scala:36)         @ org.apache.spark.scheduler.livelistenerbus.dopostevent(livelistenerbus.scala:36)         @ org.apache.spark.util.listenerbus$class.posttoall(listenerbus.scala:63)         @ org.apache.spark.scheduler.livelistenerbus.posttoall(livelistenerbus.scala:36)         @ org.apache.spark.scheduler.livelistenerbus$$anon$1$$anonfun$run$1$$anonfun$apply$mcv$sp$1.apply$mcv$sp(livelistenerbus.scala:94)         @ org.apache.spark.scheduler.livelistenerbus$$anon$1$$anonfun$run$1$$anonfun$apply$mcv$sp$1.apply(livelistenerbus.scala:79)         @ org.apache.spark.scheduler.livelistenerbus$$anon$1$$anonfun$run$1$$anonfun$apply$mcv$sp$1.apply(livelistenerbus.scala:79)         @ scala.util.dynamicvariable.withvalue(dynamicvariable.scala:57)         @ org.apache.spark.scheduler.livelistenerbus$$anon$1$$anonfun$run$1.apply$mcv$sp(livelistenerbus.scala:78)         @ org.apache.spark.util.utils$.tryorstopsparkcontext(utils.scala:1245)         @ org.apache.spark.scheduler.livelistenerbus$$anon$1.run(livelistenerbus.scala:77)

well found out cause of problem same filesystem accidentally gets closed() multiple times. not calling close() makes exception disappear.


Comments

Post a Comment

Popular posts from this blog

python - Operations inside variables -

this question has answer here: evaluating mathematical expression in string 12 answers i want make python script finds maximum possible output using 2 operations. problem says there syntax error @ 2nd last line, in b in there. how can fix it? x = 0 = int(input("1st number:")) c = int(input("2nd number:" )) e = int(input("3rd number:" )) in range(4): if == 0: b = "+" elif == 1: b = "-" elif == 2: b = "/" else: b = "*" j in range(4): if j == 0: d = "+" elif j == 1: d = "-" elif j == 2: d = "/" else: d = "*" k = b c d e print(k) if you're after variables e on single line, couldn...
Read more

Generic Map Parameter java -

i having lot of methods of type: public static list<eduusers> getdetails(class c, map<string, integer> params) { list<eduusers> details = null; session session = hibernateutil.getsessionfactory().opensession(); transaction tx = null; try { tx = session.begintransaction(); criteria cr = session.createcriteria(c); (map.entry<string, integer> entry : params.entryset()) { cr.add(restrictions.eq(entry.getkey(), entry.getvalue())); } details = cr.list(); tx.commit(); } catch (exception asd) { log.debug(asd.getmessage()); if (tx != null) { tx.commit(); } } { session.close(); } return details; } i trying have generic method them , have written far: public static <t> list<t> getdetails(class<t> c, class<t> b) { l...
Read more

arrays - What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it? -

what arrayindexoutofboundsexception mean , how rid of it? here code sample triggers exception: string[] name = {"tom", "dick", "harry"}; for(int = 0; i<=name.length; i++) { system.out.print(name[i] +'\n'); } your first port of call should documentation explains reasonably clearly: thrown indicate array has been accessed illegal index. index either negative or greater or equal size of array. so example: int[] array = new int[5]; int boom = array[10]; // throws exception as how avoid it... um, don't that. careful array indexes. one problem people run thinking arrays 1-indexed, e.g. int[] array = new int[5]; // ... populate array here ... (int index = 1; index <= array.length; index++) { system.out.println(array[index]); } that miss out first element (index 0) , throw exception when index 5. valid indexes here 0-4 inclusive. correct, idiomatic for statement here be: for (int index = 0; index <...
Read more