Reading binaryFile with Spark Streaming -


does 1 know how setup `

streamingcontext.filestream [keyclass, valueclass, inputformatclass] (datadirectory)

to consume binary files.

  • where can find inputformatclass ? documentation give no links that. imagine valueclass related inputformatclass somehow.

  • in non-streaming version using method binaryfiles, can bytearrays each files. there way can same sparkstreaming ? if not can find details. meaning inputformat supportted , value class produces. can 1 pick keyclass, aren't element connected ?

if clarify use of method.

edit1

i have tried following:

val bfiles = ssc.filestreambyteswritable, byteswritable, sequencefileasbinaryinputformat

however compiler complain such: 

[error] /xxxxxxxxx/src/main/scala/estimatorstreamingapp.scala:14: type arguments [org.apache.hadoop.io.byteswritable,org.apache.hadoop.io.byteswritable,org.apache.hadoop.mapred.sequencefileasbinaryinputformat] conform bounds of none of overloaded alternatives of [error]  value filestream: [k, v, f <: org.apache.hadoop.mapreduce.inputformat[k,v]](directory: string, filter: org.apache.hadoop.fs.path => boolean, newfilesonly: boolean, conf: org.apache.hadoop.conf.configuration)(implicit evidence$10: scala.reflect.classtag[k], implicit evidence$11: scala.reflect.classtag[v], implicit evidence$12: scala.reflect.classtag[f])org.apache.spark.streaming.dstream.inputdstream[(k, v)] <and> [k, v, f <: org.apache.hadoop.mapreduce.inputformat[k,v]](directory: string, filter: org.apache.hadoop.fs.path => boolean, newfilesonly: boolean)(implicit evidence$7: scala.reflect.classtag[k], implicit evidence$8: scala.reflect.classtag[v], implicit evidence$9: scala.reflect.classtag[f])org.apache.spark.streaming.dstream.inputdstream[(k, v)] <and> [k, v, f <: org.apache.hadoop.mapreduce.inputformat[k,v]](directory: string)(implicit evidence$4: scala.reflect.classtag[k], implicit evidence$5: scala.reflect.classtag[v], implicit evidence$6: scala.reflect.classtag[f])org.apache.spark.streaming.dstream.inputdstream[(k, v)] [error]   val bfiles = ssc.filestream[byteswritable, byteswritable, sequencefileasbinaryinputformat]("/xxxxxxxxx/casalini_streamed")

what doing wrong ?

follow link read about hadoop input formats

i found here documented answer sequence file format.

you facing compilation issue because of import missmatch. hadoop mapred vs mapreduce

e.g.

java

javapairinputdstream<text,byteswritable> dstream=         sc.filestream("/somepath",org.apache.hadoop.io.text.class,         org.apache.hadoop.io.byteswritable.class,     org.apache.hadoop.mapreduce.lib.input.sequencefileasbinaryinputformat.class);

i didn't try in scala should similar;

val dstream = sc.filestream("/somepath",          classof[org.apache.hadoop.io.text], classof[org.apache.hadoop.io.byteswritable],         classof[org.apache.hadoop.mapreduce.lib.input.sequencefileasbinaryinputformat] ) ;

Comments

Post a Comment

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

i running os ububtu 16.04. ultimate goal able run php script cron job every minute. here php script located @ /var/www/html/tests/test/index.php : <?php $directorypath = $_server['document_root']."/tests/test/data/"; echo "directorypath: $directorypath<br><br>";//check $imagefilepathsarray = glob($directorypath . "*.{png,gif}", glob_brace); echo "<br><br>imagefilepathsarray: "; print_r($imagefilepathsarray);//check ?> when open php script in browser visiting url my-ip-address/tests/test/index.php , get: directorypath: /var/www/html/tests/test/data/ imagefilepathsarray:array( [0] => /var/www/html/tests/test/data/pic.png [1] => /var/www/html/tests/test/data/pic.gif ) before writing cron job in crontab file, need check if can run anywhere using terminal . opened ubuntu terminal @ /home/ location, , executed in terminal: sudo php /var/www/html/tests/test/index.php i got following out...
Read more

php - How can i create a user dashboard -

today have questions you, want create user dashboard site, include (obviously), users data, example -email -username -and others stuff how can information, , print them in html page? if can, how can these information via php sessions? shuld use session? example, if user login, , start new session ( session_start() ), how can data print in dashboard? should use php "project"? in advice, have day , thank time!
Read more

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -

i have jquery mobile page fixed full-screen toolbar, data-tap-toggle enabled. straight under toolbar have positioned banner should slide upwards when toolbar hiding , slide downward when toolbar showing. jquery mobile toggles toolbar applying , removing ui-fixed-hidden class - sadly, can't find in documentation of toolbar widget toggle , hide or show event that. how can detect when toolbar toggled, reposition banner? .banner { position: fixed; background-color: darkseagreen; top: 46px; min-height: 48px; width: 100%; text-align: center; line-height: 48px; } <!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"> <script src="https://code.jquery.com/jquery-1....
Read more