Is it good to create Spark batch job for every new Use cases -


i run 100sof computer in network , 100sof user access machines. every day, thousands or more syslogsare generated machines. syslog log including system failures, network, firewall, application errors etc.

sample log below

may 11 11:32:40 scrooge sg_child[1829]: [id 748625 user.info] m:wr-sg-block-111- 00 c:y th:block , no allow rule matched request entryurl:http:url on  mapping:bali [ rid:t6zcuh8aaaeaagxyaqyaaaaq sid:a6bbd3447766384f3bccc3ca31dbd50n ip:192.24.61.1]

from logs, extract fields timestamp, loghost, msg, process, facility etc , store them in hdfs. logsare stored in json format. now, want build system can type query in web application , analysis on logs. able queries like

  • get logs message contains "firewall blocked" keywords.
  • get logs generated user jason
  • get logs containing "access denied" msg.
  • get log count grouped user, process, loghost etc. there thousands of different types of analytics want do. add more, want combined results of historical data , real time data i.e. combining batch , realtime results.

now questions is

  • to batch result, need run batch spark jobs. should creating batch jobs every unique query user makes. if so, end creating 1000s of batch jobs. if not, kind of batch jobs should run can results type of analytics.
  • am thinking right way. if approach wrong, share should correct procedure.

while it's possible (via thrift server example), apache spark main objective not query engine building data pipelines stream , batch data sources.

if transformation projecting fields , want enable ad-hoc queries, sounds need data store - such elasticsearch example. additional benefit comes kibana enable analytics extent.

another option use sql engine such apache drill.


Comments

Post a Comment

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

i running os ububtu 16.04. ultimate goal able run php script cron job every minute. here php script located @ /var/www/html/tests/test/index.php : <?php $directorypath = $_server['document_root']."/tests/test/data/"; echo "directorypath: $directorypath<br><br>";//check $imagefilepathsarray = glob($directorypath . "*.{png,gif}", glob_brace); echo "<br><br>imagefilepathsarray: "; print_r($imagefilepathsarray);//check ?> when open php script in browser visiting url my-ip-address/tests/test/index.php , get: directorypath: /var/www/html/tests/test/data/ imagefilepathsarray:array( [0] => /var/www/html/tests/test/data/pic.png [1] => /var/www/html/tests/test/data/pic.gif ) before writing cron job in crontab file, need check if can run anywhere using terminal . opened ubuntu terminal @ /home/ location, , executed in terminal: sudo php /var/www/html/tests/test/index.php i got following out...
Read more

php - How can i create a user dashboard -

today have questions you, want create user dashboard site, include (obviously), users data, example -email -username -and others stuff how can information, , print them in html page? if can, how can these information via php sessions? shuld use session? example, if user login, , start new session ( session_start() ), how can data print in dashboard? should use php "project"? in advice, have day , thank time!
Read more

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -

i have jquery mobile page fixed full-screen toolbar, data-tap-toggle enabled. straight under toolbar have positioned banner should slide upwards when toolbar hiding , slide downward when toolbar showing. jquery mobile toggles toolbar applying , removing ui-fixed-hidden class - sadly, can't find in documentation of toolbar widget toggle , hide or show event that. how can detect when toolbar toggled, reposition banner? .banner { position: fixed; background-color: darkseagreen; top: 46px; min-height: 48px; width: 100%; text-align: center; line-height: 48px; } <!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"> <script src="https://code.jquery.com/jquery-1....
Read more