hdfs - Spark Streaming program to process a file with more than 10 lines -
i need solve problem program should process files have lines greater 10.
below code:
val sparkconf = new sparkconf().setappname("spooldirspark") val ssc = new streamingcontext(sparkconf, seconds(2)) val lines = ssc .filestream[longwritable, text, textinputformat]("hdfs:///loclahost:8020/avin") .map{ case (x, y) => (x.tostring, y.tostring) } val data = lines.count() ssc.start() ssc.awaittermination()
Comments
Post a Comment