Raft协议概览Raft是什么？Raft是一种共识协议。与Raft完成相同任务的系统有Chubby和Zookeeper，以及一些系统内置的完成类似功能的组件，例如Elasticsearch里的Zen-Discovery。简而言之，共识协议的目的是让一组节点在响应外界输入时能表现的像一个节点一样。共识系统往往应用于有主备结构的存储系统中，如果缺少这种协议，就无法避免数据错误。
Raft协议实现学习之—初始化和Leader Election过程
实验代码和输出实验代码是基于作者原来的代码稍微修改而来，main函数如下所示:12345678910111213141516171819var ( nodes = make(map[int]*node))func main() { // start a small cluster nodes[1] = newNode(1, []raft.Peer{{ID: 1}, {ID: 2}, {ID: 3}}) go nodes[1].run() nodes[2] = newNode(2, []raft.Peer{{ID: 1}, {ID: 2}, {ID: 3}}) go nodes[2].run() nodes[3] = newNode(3, []raft.Peer{{ID: 1}, {ID: 2}, {ID: 3}}) go nodes[3].run() // Wait for leader election for { time.Sleep(100 * time.Millisecond) }}
Reading "State Management in Apache Flink"
Updated on 2018-09-09
I’m playing with Flink (1.6) and Structured Streaming (2.3.1) recently. I’m no expert for either framework, so my opinion is based on my very short experience with each of them.
Things about replication in Elasticsearch
Updated on 2018-04-18
Elasticsearch is evolving fast in the past few years. There have been quite some discussions on data loss during node crashes, which can be found here and here. Most of the issues have been fixed as described here. However, since Elasticsearch carried out a major upgrade to version 5+, some serious issues still remain for low versions, e.g., the stale replica problem described here.
Notes on HBase
So far as I know, HBase is the first open source “table” style storage in the big data scope. It is an implementation of the BigTable paper presented by Google. If you read the paper or the reference guide, HBase does not look like a table. The paper tells you that it is a
sparse, distributed, persistent, multidimensional sorted map.
投资的心法
为什么要投资？自从货币与黄金脱钩以后，整体的发展趋势总是超发的。政治家还经常会以各种借口来增发货币，比如“量化宽松”这个听起来不知所云的名词，本质就是货币超发。你辛苦工作后换来的货币随着时间流逝，其内在价值在逐渐变少。怎么能尽量减少由于一些不确定性造成的经济损失呢？一般人除了投资似乎没有别的办法。
Eventual Consistency vs. Strong Consistency
Here is a very good explanation about eventual consistency and strong consistency. I’d like to borrow the two figures on that page below:
Fig. 1 figure for eventual consistency
Notes on MR memory issues
Updated on 2018-02-05
I recently encountered several OOMs from mapper tasks reading parquet files. The yarn container is killed due to running out of physical memory. Since I already set the JVM memory to 0.8 of the container size, I’m pretty sure that this is due to off-heap memory allocation issues. I found the two jira issues here and here, pointing me to the snappy codec used by parquet for decompression. There aren’t so much I can do except allocating more memory beside the JVM.
Understanding the SSD
Reading the chapter 13.5 “Arranging data on disk” in the book “DATABASE SYSTEM: IMPLEMENTATION” makes me think of a question: How data should be arranged on a SSD (Solid-State Drive)? This is indeed an old question, so after doing some research with Google, I find some very good explanations.
An Overview of Pages, Blocks and FTLs in a Solid-State Drive (SSD)
How Do SSDs Work?