Kafka rack-id and min in-sync replicas -


kafka has introduced rack-id provide redundancy capabilities if whole rack fails. there min in-sync replica setting specify minimum number of replicas need in-sync before producer receives ack (-1 / config). there unclean leader election setting specify whether leader can elected when not in-sync.

so, given following scenario:

  • two racks. rack 1, 2.
  • replication count 4.
  • min in-sync replicas = 2
  • producer ack=-1 (all).
  • unclean leader election = false

aiming have @ least once message delivery, redundancy of nodes , tolerant rack failure.

is possible there moment 2 in-sync replicas both come rack 1, producer receives ack , @ point rack 1 crashes (before replicas rack 2 in-sync)? means rack 2 contain unclean replicas , no producers able add messages partition grinding halt. replicas unclean no new leader elected in case.

is analysis correct, or there under hood ensure replicas forming min in-sync replicas have different racks?
since replicas on same rack have lower latency seems above scenario reasonably likely.

the scenario shown in image below:

enter image description here

yes, think possible. because kafka can maintain isr according runtime's fact, not spirit.

words https://engineering.linkedin.com/kafka/intra-cluster-replication-apache-kafka

for each partition of topic, maintain in-sync replica set (isr). set of replicas alive , have caught leader (note leader in isr). when partition created initially, every replica in isr. when new message published, leader waits until reaches replicas in isr before committing message. if follower replica fails, dropped out of isr , leader continues commit new messages fewer replicas in isr. notice now, system running in under replicated mode.

words https://cwiki.apache.org/confluence/display/kafka/kafka+replication

after configured timeout period, leader drop failed follower isr , writes continue on remaining replicas in isr. if failed follower comes back, first truncates log last checkpointed hw. starts catch messages after hw leader. when follower catches up, leader add current isr.

the min in-sync replicas mentioned limit number, isr size not depend on it. settings means if producer's ack "all" , isr size less min, kafka refuse write message.

so in first time, isr {1,2,3,4}, , if broker 3 or 4 fall down, kicked out isr. , case mentioned happen. when rack 1's broker failed, unclean leader election.


Comments

Post a Comment

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

i running os ububtu 16.04. ultimate goal able run php script cron job every minute. here php script located @ /var/www/html/tests/test/index.php : <?php $directorypath = $_server['document_root']."/tests/test/data/"; echo "directorypath: $directorypath<br><br>";//check $imagefilepathsarray = glob($directorypath . "*.{png,gif}", glob_brace); echo "<br><br>imagefilepathsarray: "; print_r($imagefilepathsarray);//check ?> when open php script in browser visiting url my-ip-address/tests/test/index.php , get: directorypath: /var/www/html/tests/test/data/ imagefilepathsarray:array( [0] => /var/www/html/tests/test/data/pic.png [1] => /var/www/html/tests/test/data/pic.gif ) before writing cron job in crontab file, need check if can run anywhere using terminal . opened ubuntu terminal @ /home/ location, , executed in terminal: sudo php /var/www/html/tests/test/index.php i got following out...
Read more

php - How can i create a user dashboard -

today have questions you, want create user dashboard site, include (obviously), users data, example -email -username -and others stuff how can information, , print them in html page? if can, how can these information via php sessions? shuld use session? example, if user login, , start new session ( session_start() ), how can data print in dashboard? should use php "project"? in advice, have day , thank time!
Read more

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -

i have jquery mobile page fixed full-screen toolbar, data-tap-toggle enabled. straight under toolbar have positioned banner should slide upwards when toolbar hiding , slide downward when toolbar showing. jquery mobile toggles toolbar applying , removing ui-fixed-hidden class - sadly, can't find in documentation of toolbar widget toggle , hide or show event that. how can detect when toolbar toggled, reposition banner? .banner { position: fixed; background-color: darkseagreen; top: 46px; min-height: 48px; width: 100%; text-align: center; line-height: 48px; } <!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"> <script src="https://code.jquery.com/jquery-1....
Read more