2019/03/28

Redis Cluster 3.0.2 on CentOS 7

這篇其實寫完很久，但一直忘記按發佈XD

Redis Cluster是Redis用於解決分散式的方案，又稱為Redis叢集
是一個讓數據再多個Node之間相互傳輸的服務，Redis Cluster的優勢主要有以下兩個點

  1. 自動分割數據到不同Node
  2. 部分Node當機或不可用情況能夠繼續維持服務


不過Redis Cluster並不支持處理多個keys的命令，在不同Node移動數據並不像Redis那樣高性能，高負載時有可能會遇到不可預期錯誤
Redis Cluster並非採用Consistent Hashing而是用Hash Slots，它其實就是代表一個Keys的集合
Redis Cluster共有16384 Hash Slots，將Key做CRC16校驗接著Mod 16384決定Key會放置於哪個Slot，Redis Cluster每個Node負責一部分的Hash Slots。例如，目前Cluster中有三個Master Node，則：

  • Node A包含0到3000 Hash Slots
  • Node B包含3001到9000 Hash Slots
  • Node C包含9001到16383 Hash Slots

如果新增了Node D，僅需將原本Node上的Hash Slots添加至Node D上；相對的要刪除一個Node A，將Node A上的Hash Slots遷移至Node B / C / D上，再將沒有任何Hash Slots的Node A移除即可；且新增、刪除或搬遷Hash Slots時無須停止任何服務。

Redis Cluster為了使部分Node失敗或大部分的Node無法通訊時仍可以使用，Redis Cluster使用Master-Slave複製模型，則每個Master Node則最少有會1個Slave Node；以上述的例子而言，如果沒有使用該模型的情況下，Node A失效則會導致Cluster因0到3000的Hash Slots不可用而失效。
最後Redis Cluster並不保證數據的一致性，這也意味著Redis Cluster在特定條件下有可能會丟失數據。


讓我們開始來建置Redis Cluster吧

接著我們開始來建立Redis Cluster吧，根據官方文獻建置一個Redis Cluster最少需要三個Master，也意味著還需要三個Slave來確保每個Master失效時才能將角色轉移至Slave上。
所以需先準備6台Host或container，系統為CentOS 7 minimal


Name
IP
Port
Cluster BUS Port

Master 1
192.168.126.135
6379
16379

Master 2
192.168.126.136
6380
16380

Master 3
192.168.126.141
6381
16381

Slave 1
192.168.126.142
6382
6382

Slave 2
192.168.126.143
6383
6383

Slave 3
192.168.126.144
6384
6384



安裝教學請參考「Install Redis Server on CentOS 7」


Master與Slave配置如下
# Master 1配置
bind 192.168.126.135 127.0.0.1
port 6379
protected-mode yes
daemonize yes
supervised systemd
pidfile /var/run/redis_6379.pid
masterauth password
requirepass password
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file nodes-6379.conf
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage no

# Master 2配置
bind 192.168.126.136 127.0.0.1
port 6380
protected-mode yes
daemonize yes
supervised systemd
pidfile /var/run/redis_6380.pid
masterauth password
requirepass password
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file nodes-6380.conf
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage no

# Master 3配置
bind 192.168.126.141 127.0.0.1
port 6381
protected-mode yes
daemonize yes
supervised systemd
pidfile /var/run/redis_6381.pid
masterauth password
requirepass password
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file nodes-6381.conf
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage no

# Slave 1配置
bind 192.168.126.142 127.0.0.1
port 6382
protected-mode yes
daemonize yes
supervised systemd
pidfile /var/run/redis_6382.pid
masterauth password
requirepass password
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file nodes-6382.conf
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage no

# Slave 2配置
bind 192.168.126.143 127.0.0.1
port 6383
protected-mode yes
daemonize yes
supervised systemd
pidfile /var/run/redis_6383.pid
masterauth password
requirepass password
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file nodes-6383.conf
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage no

# Slave 3配置
bind 192.168.126.144 127.0.0.1
port 6384
protected-mode yes
daemonize yes
supervised systemd
pidfile /var/run/redis_6384.pid
masterauth password
requirepass password
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file nodes-6384.conf
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage no

防火牆規則：
firewall-cmd --add-port=6379/tcp --permanent
firewall-cmd --add-port=6380/tcp --permanent
firewall-cmd --add-port=6381/tcp --permanent
firewall-cmd --add-port=6382/tcp --permanent
firewall-cmd --add-port=6383/tcp --permanent
firewall-cmd --add-port=6384/tcp --permanent
firewall-cmd --add-port=16379/tcp --permanent
firewall-cmd --add-port=16380/tcp --permanent
firewall-cmd --add-port=16381/tcp --permanent
firewall-cmd --add-port=16382/tcp --permanent
firewall-cmd --add-port=16383/tcp --permanent
firewall-cmd --add-port=16384/tcp --permanent
firewall-cmd --reload


透過redis-rb或redis-cli讓Node互相Handshake
不過依稀記得redis-rb輸入完就設定完成了
redis-cli則是需一個命令一個命令接著下

redis-rb：
redis-trib create --replicas 1 192.168.126.135:6379 192.168.126.142:6382 192.168.126.136:6380 192.168.126.133:6383 192.168.126.141:6381 192.168.126.144:6384

redis-cli：
redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER meet 192.168.126.136 6380 
redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER meet 192.168.126.141 6381 
redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER meet 192.168.126.142 6382 
redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER meet 192.168.126.143 6383 
redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER meet 192.168.126.144 6384


透過下方指令來查看目前狀態
備註：
Handshake失敗原因大部分都是Redis Port / Cluster BUS port其中一個沒有連接導致Handshake失敗
Cluster BUS port = Redis Port + 10000
redis-cli –h 192.168.126.135 -p 6379 –a password –c cluster nodes



分配Hash Slots
redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER ADDSLOTS {0..5461}
redis-cli -h 192.168.126.136 -p 6380 -a password -c CLUSTER ADDSLOTS {5462..10922}
redis-cli -h 192.168.126.141 -p 6381 -a password -c CLUSTER ADDSLOTS {10923..16383}


沒有分配完會如下圖

分配完成如下圖


配置都沒問題，就輸入下方指令保存至每個Node config

redis-cli -h 192.168.126.135 -p 6379 -a password -c CLUSTER SAVECONFIG


要測試角色切換最簡單的方式就是停止服務
systemctl stop redis

我將135停止服務

142就會變成Master，並將該訊息通知給其他Node
等135重啟後變成Slave角色


Java程式碼：
package cy.com;

import com.sun.istack.internal.NotNull;
import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.JedisPoolConfig;

import java.io.IOException;
import java.io.InputStream;
import java.util.*;


class RedisCluster {
    /**Map*/
    private static Map<String, Object> map;

    /**
     * @see JedisCluster
     */
    private JedisCluster jedisCluster;

    /**
     * @see JedisPoolConfig
     */
    private static JedisPoolConfig getJedisPoolConfig() {
        JedisPoolConfig config = new JedisPoolConfig();
        config.setMaxTotal(Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_POOL_MAXACTIVE).toString()));
        config.setMinIdle(Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_POOL_MIN_IDLE).toString()));
        config.setMaxIdle(Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_POOL_MAX_IDLE).toString()));
        config.setMaxWaitMillis(Long.parseLong(RedisCluster.map.get(RedisInfo.REDIS_POOL_MAX_WAIT_MILLIS).toString()));
        config.setNumTestsPerEvictionRun(Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_POOL_NUM_TESTS_PER_EVICTION_RUN).toString()));
        config.setTimeBetweenEvictionRunsMillis(Long.parseLong(RedisCluster.map.get(RedisInfo.REDIS_POOL_TIME_BETWEEN_EVICTION_RUNS_MILLIS).toString()));
        config.setMinEvictableIdleTimeMillis(Long.parseLong(RedisCluster.map.get(RedisInfo.REDIS_POOL_MIN_EVICTABLE_IDLE_TIME_MILLIS).toString()));
        config.setSoftMinEvictableIdleTimeMillis(Long.parseLong(RedisCluster.map.get(RedisInfo.REDIS_POOL_SOFT_MIN_EVICTABLE_IDLE_TIME_MILLIS).toString()));
        config.setTestOnBorrow(Boolean.parseBoolean(RedisCluster.map.get(RedisInfo.REDIS_POOL_TEST_ON_BORROW).toString()));
        config.setTestWhileIdle(Boolean.parseBoolean(RedisCluster.map.get(RedisInfo.REDIS_POOL_TEST_WHILE_IDLE).toString()));
        config.setTestOnReturn(Boolean.parseBoolean(RedisCluster.map.get(RedisInfo.REDIS_POOL_TEST_ON_RETURN).toString()));
        config.setBlockWhenExhausted(Boolean.parseBoolean(RedisCluster.map.get(RedisInfo.REDIS_POOL_BLOCK_WHEN_EXHAUSTED).toString()));
        return config;
    }

    /**
     * get Cluster host and port
     *
     * @return HostAndPort
     */
    private static Set<HostAndPort> getHostAndPortSet() {
        Set<HostAndPort> set = new HashSet<HostAndPort>();
        String[] array = RedisCluster.map.get(RedisInfo.REDIS_CLUSTERS).toString().split(";");
        for (String c : array) {
            String[] hostAndPort = c.split(":");
            set.add(new HostAndPort(hostAndPort[0], Integer.parseInt(hostAndPort[1])));
        }

        return set;
    }

    static {
        if (RedisCluster.map == null) {
            final InputStream inputStream = RedisInfo.class.getClassLoader()
                    .getResourceAsStream("Redis.properties");
            final Properties properties = new Properties();
            try {
                properties.load(inputStream);
                inputStream.close();
                RedisCluster.map = new HashMap<String, Object>();
                //Master Name
                RedisCluster.map.put(RedisInfo.REDIS_MASTER_NAME, properties.getProperty(RedisInfo.REDIS_MASTER_NAME));
                //Host
                RedisCluster.map.put(RedisInfo.REDIS_HOST, properties.getProperty(RedisInfo.REDIS_HOST));
                //Port
                RedisCluster.map.put(RedisInfo.REDIS_PORT, properties.getProperty(RedisInfo.REDIS_PORT));
                //Sentinels
                RedisCluster.map.put(RedisInfo.REDIS_SENTINELS, properties.getProperty(RedisInfo.REDIS_SENTINELS));
                //Clusters
                RedisCluster.map.put(RedisInfo.REDIS_CLUSTERS, properties.getProperty(RedisInfo.REDIS_CLUSTERS));
                //Timeout
                RedisCluster.map.put(RedisInfo.REDIS_TIMEOUT, properties.getProperty(RedisInfo.REDIS_TIMEOUT));
                //So Timeout
                RedisCluster.map.put(RedisInfo.REDIS_SO_TIMEOUT, properties.getProperty(RedisInfo.REDIS_SO_TIMEOUT));
                //Password
                RedisCluster.map.put(RedisInfo.REDIS_PASSWORD, properties.getProperty(RedisInfo.REDIS_PASSWORD));
                //Max attempts
                RedisCluster.map.put(RedisInfo.REDIS_MAX_ATTEMPTS, properties.getProperty(RedisInfo.REDIS_MAX_ATTEMPTS));
                //Max active
                RedisCluster.map.put(RedisInfo.REDIS_POOL_MAXACTIVE, properties.getProperty(RedisInfo.REDIS_POOL_MAXACTIVE));
                //Max idle
                RedisCluster.map.put(RedisInfo.REDIS_POOL_MAX_IDLE, properties.getProperty(RedisInfo.REDIS_POOL_MAX_IDLE));
                //Min idle
                RedisCluster.map.put(RedisInfo.REDIS_POOL_MIN_IDLE, properties.getProperty(RedisInfo.REDIS_POOL_MIN_IDLE));
                //Max wait millis
                RedisCluster.map.put(RedisInfo.REDIS_POOL_MAX_WAIT_MILLIS, properties.getProperty(RedisInfo.REDIS_POOL_MAX_WAIT_MILLIS));
                //Num tests per eviction run
                RedisCluster.map.put(RedisInfo.REDIS_POOL_NUM_TESTS_PER_EVICTION_RUN, properties.getProperty(RedisInfo.REDIS_POOL_NUM_TESTS_PER_EVICTION_RUN));
                //Time between eviction runs millis
                RedisCluster.map.put(RedisInfo.REDIS_POOL_TIME_BETWEEN_EVICTION_RUNS_MILLIS, properties.getProperty(RedisInfo.REDIS_POOL_TIME_BETWEEN_EVICTION_RUNS_MILLIS));
                //Min evictable idle time millis
                RedisCluster.map.put(RedisInfo.REDIS_POOL_MIN_EVICTABLE_IDLE_TIME_MILLIS, properties.getProperty(RedisInfo.REDIS_POOL_MIN_EVICTABLE_IDLE_TIME_MILLIS));
                //Soft min evictable idle time millis
                RedisCluster.map.put(RedisInfo.REDIS_POOL_SOFT_MIN_EVICTABLE_IDLE_TIME_MILLIS, properties.getProperty(RedisInfo.REDIS_POOL_SOFT_MIN_EVICTABLE_IDLE_TIME_MILLIS));
                //Test on borrow
                RedisCluster.map.put(RedisInfo.REDIS_POOL_TEST_ON_BORROW, properties.getProperty(RedisInfo.REDIS_POOL_TEST_ON_BORROW));
                //Test while idle
                RedisCluster.map.put(RedisInfo.REDIS_POOL_TEST_WHILE_IDLE, properties.getProperty(RedisInfo.REDIS_POOL_TEST_WHILE_IDLE));
                //Test on return
                RedisCluster.map.put(RedisInfo.REDIS_POOL_TEST_ON_RETURN, properties.getProperty(RedisInfo.REDIS_POOL_TEST_ON_RETURN));
                //Block when exhausted
                RedisCluster.map.put(RedisInfo.REDIS_POOL_BLOCK_WHEN_EXHAUSTED, properties.getProperty(RedisInfo.REDIS_POOL_BLOCK_WHEN_EXHAUSTED));
            } catch (IOException e) {
            }
        }
    }

    /**
     * Redis Cluster Constructor
     */
    public RedisCluster() {
        //Timeout
        final int TIMEOUT = Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_TIMEOUT).toString());
        //So Timeout
        final int SO_TIMEOUT = Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_SO_TIMEOUT).toString());
        //Password
        final String PASSWORD = RedisCluster.map.get(RedisInfo.REDIS_PASSWORD).toString();
        //Max Attempts
        final int MAX_ATTEMPTS = Integer.parseInt(RedisCluster.map.get(RedisInfo.REDIS_MAX_ATTEMPTS).toString());
        //Nodes
        final Set<HostAndPort> NODES = getHostAndPortSet();
        //Config
        final JedisPoolConfig CONFIG = getJedisPoolConfig();
        //Setting Redis Cluster
        this.jedisCluster = new JedisCluster(NODES, TIMEOUT, SO_TIMEOUT, MAX_ATTEMPTS, PASSWORD, CONFIG);
    }

    /**
     * @param key   Key
     * @param value Value
     */
    public String set(final @NotNull String key, final @NotNull String value) {
        return this.jedisCluster.set(key, value);
    }

    /**
     * @param key Key
     * @see <a href="https://redis.io/commands/get">GET</a>
     */
    public String get(@NotNull final String key) {
        return this.jedisCluster.get(key);
    }

    /**
     * Close connection
     */
    public void close() throws IOException {

        this.jedisCluster.close();
        this.jedisCluster = null;
    }

}

Redis.properties：
redis.masterName=mymaster
redis.host=192.168.126.135
redis.port=6379
redis.sentinels=192.168.126.135:26379;192.168.126.136:26379
redis.clusters=192.168.126.135:6379;192.168.126.136:6380;192.168.126.141:6381;192.168.126.142:6382;192.168.126.143:6383;192.168.126.144:6384
redis.timeout=10000
redis.sotimeout=10000
redis.password=password
redis.maxAttempts=5
redis.pool.maxActive=128
redis.pool.maxIdle=10
redis.pool.minIdle=1
redis.pool.maxWaitMillis=3000
redis.pool.numTestsPerEvictionRun=50
redis.pool.timeBetweenEvictionRunsMillis=3000
redis.pool.minEvictableIdleTimeMillis=1800000
redis.pool.softMinEvictableIdleTimeMillis=10000
redis.pool.testOnBorrow=true
redis.pool.testWhileIdle=true
redis.pool.testOnReturn=true
redis.pool.blockWhenExhausted=true
By at
標籤: , , ,