Naming module Raft protocol migration (#3989)

* fix-#3595, delete the unnecessary code (#3596) * [ISSUE #3566] move the permission code of nacos-core module to nacos-auth module (#3593) * move the permission code of nacos-core module to nacos-auth module. * Fix some code style issues * address server module auth package name change. * test change * Incorrect package name correction * [ISSUE #3592] Fix incorrect prompt when accessing the restricted namespace (#3603) * Fix incorrect prompt when accessing the restricted namespace * Modify variable name * [ISSUE #3600] Replace the deprecated api of jwt (#3616) * replace the deprecated api of jwt * transfer secretKey to byte array just using String encode with utf-8 * [ISSUE #3613] Fix `unit test method not be static` & update publish config listener in `ConfigTest.java` (#3614) * fix `unit test method not be static` & update publish config listener in `ConfigTest.java` * fix `unit test method not be static` & update publish config listener in `ConfigTest.java` * move jwt dependency from console,core to auth. (#3624) * refactor: unified implementation of http client api adjustment. (#3639) * [ISSUE #3628] set naming client updateTask interval more flexible (#3637) * 1.use server cacheMillis event service deleted 2.naming client UpdateTask's interval will inc by failCount that connect with server * 1.move failCount to updateTask 2.redefine the updateService method name. updateServiceNow -> updateService, wrap updateService in updateServiceNow when first getServiceInfo * 1.create push client even service is not exist 2.serviceInfo's hosts is empty or can't connect to server both add the updateTsk interval * format the indent * fix: create kvstorage * refactor: create kv storage] * refactor: refactor rocksdb storage code * Fix issue 3661 (#3662) * improvement: merge upstream/develop * feat: merge upstream develop * fix: fix issue #3661 * [ISSUE#3658] refactor TaskManager and move to nacos-common module (#3663) * Add TaskManagerTest * Move Abstract Task and Task processor to nacos-common * Add Nacos execute engine interface and nacos task interface * Refactor Task Manager to NacosDelayTaskExecuteEngine * for code style * [ISSUE #3671] move some tools class into common package (#3672) * for #3621 (#3668) * [ISSUE#3315]Nacos client support https (#3654) * [ISSUE #3315] nacos client support https * common module add tls related classes * JdkHttpClientRequest support https * unified IpUtils * [ISSUE #3315] nacos client support https * common module add tls related classes * JdkHttpClientRequest support https * unified IpUtils * [ISSUE #3315] nacos client support https * format code * fix typo and doc format of README file (#3688) 1. It’s a little weird to use Chinese comma in English 2. Fix typo: change 'reposity' to 'repository', 'instance' to 'instances' 3. Format other details of the doc * Try to fix nacos server CLOSE_WAIT (#3703) * Fix revert chunk isn't work in Content Comparison page (#3686) * fix: fixed cluster node version issue * Fix Logging in with the wrong username or password will cause jackson serialization results to fail (#3695) (#3721) * [ISSUE#3712] add apache http client factory (#3716) * refactor: Add apache http client Factory. * refactor: Add apache http client Factory. * add license * refactor: class name change * [Issue#3692] Use new distro task engine replace old task dispatcher. (#3715) * Add ServiceManagerTest * Add DistroConsistencyServiceImplTest * Add new distro entities * Add new distro sync data change * add new VerifyTask to do checksum. * Replace old sync task and checksum task * Add retry sync change task. * Fix high cpu load * For checkstyle * Combine naming sync task to reduce http cost * Move some classes * Refactor keys combined logic * enhance package construct * Fix unit test * For pmd * [Issue #3692] Use new distro implmentation to handle sync datum and checksum request (#3734) * Add receive data and verify data distro protocol interface * Use new distro implmentation to handle sync datum and checksum request * Use new distro implmentation to handle get datum request. * refactor: adjust the data loading logic when a new protocol is started * [ISSUE#3692] Use new distro implmentation to handle init all datum request. (#3744) * Add load data task in DistroProtocol * Use new distro implmentation to handle init all datum request. * Fix unit test * fix: #3617 (#3678) * fix #3617 * 调整代码格式 * 修改抛出的Exception类型 * 调整逻辑 * 移除没用到的方法 * 修改注释 * [ISSUE#3658] Move distro sync code to nacos-core module (#3750) * Move distro sync code to nacos-core module * Update unit test * feat: none * fix: fixing serialization problems * replace the deprecated api com.fasterxml.jackson.databind.node.ObjectNode#put(java.lang.String, com.fasterxml.jackson.databind.JsonNode) * Revert "[#3368]Cancel empty Long polling thread to improve performance. (#ISSUE3432)" (#3778) This reverts commit 95c8bf2. * [ISSUE #3658] Some enhance refactor for naming distro (#3765) * Some enhance refactor for naming distro * Remove null code * [ISSUE #3687] check serviceName's format(groupName@@serviceName) in server and client (#3767) * 1.in server, check serviceName's format 'groupName@@serviceName', groupName and serviceName can't be blank 2.in client, check 'groupName@@serviceName', groupName and serviceName can't be blank * ignore the check to groupName * check split's length instead of exception to check argument * 1.add some notes 2.remove unnecessary code * modify the notes * [ISSUE#3790] Supplement http response Content-Encoding processing. (#3791) * bug: fix issue #3790, Supplement http response Content-Encoding processing * bug: fix issue #3790, Supplement http response Content-Encoding processing. * bug: fix issue #3790, Supplement http response Content-Encoding processing. * Update ConvertUtils.java (#3789) * fix ConvertUtils can not handle FormatException. * fix: fixed some logic errors * [ISSUE#3192] naming module replace http client (#3763) * naming module replace http client * refactor: naming module replace http client. * refactor: naming module replace http client. * refactor: Add apache http client Factory. * refactor: naming module replace http client. * fix code style * refactor: Add http client config * refactor: naming module HttpClientManager change * refactor: naming module HttpClientManager change * refactor: naming module replace http client. * fix code style * refactor: fix JDK http client Use error problem. * refactor: Query And Header entity init Add non-empty judgment * Enhance the asynchronous http delete request method to support body passing parameters. * refactor: apache http client set MaxConnTotal and maxConnPerRoute. * Fix NullPointerException when no subscriber for slow event (#3835) * Set mediaType charset as utf8 (#3837) * Fix code style problem in DiskUtils (#3842) * Fix http client contentType charset problem (#3848) * feature issue #3804 (#3805) * improvement: merge upstream/develop * feat: merge upstream develop * feat: manage the loading of configuration files uniformly * fix: fix copyright style * style: fix code style * fix: fix code style * bug: fix the problem of incorrect judgment of http response code in SubscribeManager#getSubscribers() method (#3879) * [ISSUE #3867] replace the way which get version (#3872) * replace the way which get version * remove version sign application.properties in nacos-api * [ISSUE #3871] fix description don't match the error (#3886) * Fix jraft problem * Fix chinese string are truncated in ConcunrrentDiskUtil (#3883) * Temp fix raft server can't refresh raft configuration problem * update spring boot dependencies version (#3900) Co-authored-by: yanlinly <[email protected]> Co-authored-by: 杨翊 SionYang <[email protected]> * [ISSUE #3781]Fix service list intermittently lost service (#3891) * update service init * commit futureMap.remove() * update serviceManager Co-authored-by: yanlinly <[email protected]> Co-authored-by: 杨翊 SionYang <[email protected]> * Fix Listener do not listen new consistency problem * Fix Performance logger thread call old raft error * Use datum in new raft processor to compatible old data * Use multiple kv storage in new raft processor to compatible old data * Revert PR#2849. * Move datum key check to KeyBuilder * Fix 1.3.2 upgrade 1.4.0 can't notify service change problem * [ISSUE #3850] ignore socket exception when client destroy already (#3906) * if client destroy already, ignore socket exception. * remove the exception's judgement * remove unuseful import * fix: fix merge conflict * isAddressServerHealth set as true when request success (#3952) * [ISSUE#3533] change cache dir with namespace -- part 1: unify the cache dir (#3859) * [ISSUE#3533] change cache dir with namespace -- part 2: unify the log dir (#3882) * fix typo error (#3954) * [ISSUE #3909] add domain's judgement (#3913) * add domain's judgement * modify domain's judgement, can resolve = true * remove judgement in 'if' code block * replace Ip to IP in InetUtils * log warn info when domain can not be resolved * fix vaiable name * 1.fix unit test can't pass (#3956) 2.update the unit test * Move remove listener logic to ServiceManager * Move remove listener logic to ServiceManager * Fix#3973 (#3974) * fix #3973 * 重复代码抽取到一个方法 * 删除私有方法的注释 * 处理namespace参数的方法提出到一个工具类中 * 修改注释 * 添加licences * 增加 TenantUtil 的测试 * TenantUtil 改名为 NamespaceUtil * For #3384, Fix member extend info do not update error. (#3982) * refactor: code refactor Co-authored-by: 赵延 <[email protected]> Co-authored-by: mai.jh <[email protected]> Co-authored-by: ljhrot <[email protected]> Co-authored-by: Xarrow <[email protected]> Co-authored-by: 杨翊 SionYang <[email protected]> Co-authored-by: Gagharv <[email protected]> Co-authored-by: syapollo <[email protected]> Co-authored-by: S2W <[email protected]> Co-authored-by: Tboy <[email protected]> Co-authored-by: 邪影oO <[email protected]> Co-authored-by: yanlinly <[email protected]> Co-authored-by: Mark4z <[email protected]> Co-authored-by: Marcus <[email protected]> Co-authored-by: shizhengxing <[email protected]> Co-authored-by: ljhrot <[email protected]> Co-authored-by: sanxun0325 <[email protected]> Co-authored-by: JackSun-Developer <[email protected]>
alibaba · Oct 14, 2020 · 8a54b1f · 8a54b1f
1 parent fdee6b8
commit 8a54b1f
Show file tree

Hide file tree

Showing 195 changed files with 3,265 additions and 2,035 deletions.
diff --git a/api/src/main/java/com/alibaba/nacos/api/naming/utils/NamingUtils.java b/api/src/main/java/com/alibaba/nacos/api/naming/utils/NamingUtils.java
@@ -28,6 +28,9 @@
 public class NamingUtils {
 
     public static String getGroupedName(final String serviceName, final String groupName) {
+        if (StringUtils.isBlank(serviceName)) {
+            throw new IllegalArgumentException("Param 'serviceName' is illegal, serviceName is blank");
+        }
         final String resultGroupedName = groupName + Constants.SERVICE_INFO_SPLITER + serviceName;
         return resultGroupedName.intern();
     }

diff --git a/auth/pom.xml b/auth/pom.xml
@@ -40,6 +40,10 @@
             <groupId>${project.groupId}</groupId>
             <artifactId>nacos-common</artifactId>
         </dependency>
+        <dependency>
+            <groupId>${project.groupId}</groupId>
+            <artifactId>nacos-sys</artifactId>
+        </dependency>
 
         <dependency>
             <groupId>org.springframework.boot</groupId>

diff --git a/auth/src/main/java/com/alibaba/nacos/auth/common/AuthConfigs.java b/auth/src/main/java/com/alibaba/nacos/auth/common/AuthConfigs.java
@@ -16,12 +16,11 @@
 
 package com.alibaba.nacos.auth.common;
 
-import com.alibaba.nacos.auth.common.env.ReloadableConfigs;
 import com.alibaba.nacos.common.JustForTest;
+import com.alibaba.nacos.sys.utils.ApplicationUtils;
 import io.jsonwebtoken.io.Decoders;
 import org.apache.commons.lang3.BooleanUtils;
 import org.apache.commons.lang3.StringUtils;
-import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.beans.factory.annotation.Value;
 import org.springframework.context.annotation.Configuration;
 
@@ -40,9 +39,6 @@ public class AuthConfigs {
     @JustForTest
     private static Boolean cachingEnabled = null;
 
-    @Autowired
-    private ReloadableConfigs reloadableConfigs;
-
     /**
      * secret key.
      */
@@ -93,7 +89,7 @@ public boolean isAuthEnabled() {
             return BooleanUtils.toBoolean(enabled);
         }
         return BooleanUtils
-                .toBoolean(reloadableConfigs.getProperties().getProperty("nacos.core.auth.enabled", "false"));
+                .toBoolean(ApplicationUtils.getProperty("nacos.core.auth.enabled", "false"));
     }
 
     /**
@@ -106,7 +102,7 @@ public boolean isCachingEnabled() {
             return cachingEnabled;
         }
         return BooleanUtils
-                .toBoolean(reloadableConfigs.getProperties().getProperty("nacos.core.auth.caching.enabled", "true"));
+                .toBoolean(ApplicationUtils.getProperty("nacos.core.auth.caching.enabled", "true"));
     }
 
     @JustForTest

diff --git a/auth/src/main/java/com/alibaba/nacos/auth/common/env/ReloadableConfigs.java b/auth/src/main/java/com/alibaba/nacos/auth/common/env/ReloadableConfigs.java
diff --git a/client/src/main/java/com/alibaba/nacos/client/config/impl/ClientWorker.java b/client/src/main/java/com/alibaba/nacos/client/config/impl/ClientWorker.java
@@ -32,7 +32,6 @@
 import com.alibaba.nacos.client.utils.TenantUtil;
 import com.alibaba.nacos.common.http.HttpRestResult;
 import com.alibaba.nacos.common.lifecycle.Closeable;
-import com.alibaba.nacos.common.utils.ConcurrentHashSet;
 import com.alibaba.nacos.common.utils.ConvertUtils;
 import com.alibaba.nacos.common.utils.MD5Utils;
 import com.alibaba.nacos.common.utils.StringUtils;
@@ -166,7 +165,6 @@ private void removeCache(String dataId, String group) {
             copy.remove(groupKey);
             cacheMap.set(copy);
         }
-        reMakeCacheDataTaskId();
         LOGGER.info("[{}] [unsubscribe] {}", this.agent.getName(), groupKey);
 
         MetricsMonitor.getListenConfigCountMonitor().set(cacheMap.get().size());
@@ -179,38 +177,11 @@ void removeCache(String dataId, String group, String tenant) {
             copy.remove(groupKey);
             cacheMap.set(copy);
         }
-        reMakeCacheDataTaskId();
         LOGGER.info("[{}] [unsubscribe] {}", agent.getName(), groupKey);
 
         MetricsMonitor.getListenConfigCountMonitor().set(cacheMap.get().size());
     }
 
-    /**
-     * Remake cacheData taskId.
-     */
-    private void reMakeCacheDataTaskId() {
-        int listenerSize = cacheMap.get().size();
-        int remakeTaskId = (int) Math.ceil(listenerSize / ParamUtil.getPerTaskConfigSize());
-        if (remakeTaskId < (int) currentLongingTaskCount) {
-            for (int i = 0; i < remakeTaskId; i++) {
-                int count = 0;
-                for (String key : cacheMap.get().keySet()) {
-                    if (count == ParamUtil.getPerTaskConfigSize()) {
-                        break;
-                    }
-                    CacheData cacheData = cacheMap.get().get(key);
-                    cacheData.setTaskId(i);
-                    synchronized (cacheMap) {
-                        Map<String, CacheData> copy = new HashMap<String, CacheData>(this.cacheMap.get());
-                        copy.put(key, cacheData);
-                        cacheMap.set(copy);
-                    }
-                    count++;
-                }
-            }
-        }
-    }
-
     /**
      * Add cache data if absent.
      *
@@ -277,8 +248,6 @@ public CacheData addCacheDataIfAbsent(String dataId, String group, String tenant
                 cache.setInitializing(true);
             } else {
                 cache = new CacheData(configFilterChainManager, agent.getName(), dataId, group, tenant);
-                int taskId = cacheMap.get().size() / (int) ParamUtil.getPerTaskConfigSize();
-                cache.setTaskId(taskId);
                 // fix issue # 1317
                 if (enableRemoteSyncConfig) {
                     String[] ct = getServerConfig(dataId, group, tenant, 3000L);
@@ -425,16 +394,11 @@ public void checkConfigInfo() {
         int longingTaskCount = (int) Math.ceil(listenerSize / ParamUtil.getPerTaskConfigSize());
         if (longingTaskCount > currentLongingTaskCount) {
             for (int i = (int) currentLongingTaskCount; i < longingTaskCount; i++) {
-                taskIdSet.add(i);
                 // The task list is no order.So it maybe has issues when changing.
                 executorService.execute(new LongPollingRunnable(i));
             }
-        } else if (longingTaskCount < currentLongingTaskCount) {
-            for (int i = longingTaskCount; i < (int) currentLongingTaskCount; i++) {
-                taskIdSet.remove(i);
-            }
+            currentLongingTaskCount = longingTaskCount;
         }
-        currentLongingTaskCount = longingTaskCount;
     }
 
     /**
@@ -692,9 +656,7 @@ public void run() {
                 }
                 inInitializingCacheList.clear();
 
-                if (taskIdSet.contains(taskId)) {
-                    executorService.execute(this);
-                }
+                executorService.execute(this);
 
             } catch (Throwable e) {
 
@@ -723,11 +685,6 @@ private void setHealthServer(boolean isHealthServer) {
     private final AtomicReference<Map<String, CacheData>> cacheMap = new AtomicReference<Map<String, CacheData>>(
             new HashMap<String, CacheData>());
 
-    /**
-     * Store the running taskId.
-     */
-    private final ConcurrentHashSet<Integer> taskIdSet = new ConcurrentHashSet<Integer>();
-
     private final HttpAgent agent;
 
     private final ConfigFilterChainManager configFilterChainManager;

diff --git a/client/src/main/java/com/alibaba/nacos/client/logging/AbstractNacosLogging.java b/client/src/main/java/com/alibaba/nacos/client/logging/AbstractNacosLogging.java
@@ -19,6 +19,8 @@
 import com.alibaba.nacos.common.utils.ConvertUtils;
 import com.alibaba.nacos.common.utils.StringUtils;
 
+import java.io.File;
+
 /**
  * Abstract nacos logging.
  *
@@ -31,13 +33,13 @@ public abstract class AbstractNacosLogging {
 
     private static final String NACOS_LOGGING_DEFAULT_CONFIG_ENABLED_PROPERTY = "nacos.logging.default.config.enabled";
 
-    private static final String NACOS_LOGGING_PATH_PROPERTY = "nacos.logging.path";
+    private static final String NACOS_LOGGING_PATH_PROPERTY = "JM.LOG.PATH";
 
     static {
         String loggingPath = System.getProperty(NACOS_LOGGING_PATH_PROPERTY);
         if (StringUtils.isBlank(loggingPath)) {
             String userHome = System.getProperty("user.home");
-            System.setProperty(NACOS_LOGGING_PATH_PROPERTY, userHome + "/logs/nacos");
+            System.setProperty(NACOS_LOGGING_PATH_PROPERTY, userHome + File.separator + "logs");
         }
     }
 

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/NacosNamingService.java b/client/src/main/java/com/alibaba/nacos/client/naming/NacosNamingService.java
@@ -39,6 +39,7 @@
 import com.alibaba.nacos.common.utils.ConvertUtils;
 import com.alibaba.nacos.common.utils.StringUtils;
 
+import java.io.File;
 import java.util.ArrayList;
 import java.util.Iterator;
 import java.util.List;
@@ -151,9 +152,13 @@ private void initLogName(Properties properties) {
     }
 
     private void initCacheDir() {
-        cacheDir = System.getProperty("com.alibaba.nacos.naming.cache.dir");
-        if (StringUtils.isEmpty(cacheDir)) {
-            cacheDir = System.getProperty("user.home") + "/nacos/naming/" + namespace;
+        String jmSnapshotPath = System.getProperty("JM.SNAPSHOT.PATH");
+        if (!StringUtils.isBlank(jmSnapshotPath)) {
+            cacheDir = jmSnapshotPath + File.separator + "nacos" + File.separator + "naming"
+                    + File.separator + namespace;
+        } else {
+            cacheDir = System.getProperty("user.home") + File.separator + "nacos" + File.separator + "naming"
+                    + File.separator + namespace;
         }
     }
 

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/cache/ConcurrentDiskUtil.java b/client/src/main/java/com/alibaba/nacos/client/naming/cache/ConcurrentDiskUtil.java
@@ -143,11 +143,12 @@ public static Boolean writeFileContent(File file, String content, String charset
                 }
             } while (null == lock);
 
-            ByteBuffer sendBuffer = ByteBuffer.wrap(content.getBytes(charsetName));
+            byte[] contentBytes = content.getBytes(charsetName);
+            ByteBuffer sendBuffer = ByteBuffer.wrap(contentBytes);
             while (sendBuffer.hasRemaining()) {
                 channel.write(sendBuffer);
             }
-            channel.truncate(content.length());
+            channel.truncate(contentBytes.length);
         } catch (FileNotFoundException e) {
             throw new IOException("file not exist");
         } finally {

diff --git a/client/src/main/java/com/alibaba/nacos/client/naming/core/PushReceiver.java b/client/src/main/java/com/alibaba/nacos/client/naming/core/PushReceiver.java
@@ -107,6 +107,9 @@ public void run() {
                 udpSocket.send(new DatagramPacket(ack.getBytes(UTF_8), ack.getBytes(UTF_8).length,
                         packet.getSocketAddress()));
             } catch (Exception e) {
+                if (closed) {
+                    return;
+                }
                 NAMING_LOGGER.error("[NA] error while receiving push data", e);
             }
         }

diff --git a/client/src/main/java/com/alibaba/nacos/client/utils/ParamUtil.java b/client/src/main/java/com/alibaba/nacos/client/utils/ParamUtil.java
@@ -20,9 +20,9 @@
 import com.alibaba.nacos.api.SystemPropertyKeyConst;
 import com.alibaba.nacos.api.common.Constants;
 import com.alibaba.nacos.common.utils.StringUtils;
+import com.alibaba.nacos.common.utils.VersionUtils;
 import org.slf4j.Logger;
 
-import java.io.InputStream;
 import java.util.Properties;
 import java.util.concurrent.Callable;
 import java.util.regex.Pattern;
@@ -80,19 +80,7 @@ public class ParamUtil {
         }
         LOGGER.info("[settings] [http-client] connect timeout:{}", connectTimeout);
 
-        try {
-            InputStream in = ValidatorUtils.class.getClassLoader().getResourceAsStream("application.properties");
-            Properties props = new Properties();
-            props.load(in);
-            String val = null;
-            val = props.getProperty("version");
-            if (val != null) {
-                clientVersion = val;
-            }
-            LOGGER.info("NACOS_CLIENT_VERSION: {}", clientVersion);
-        } catch (Exception e) {
-            LOGGER.error("[500] read application.properties", e);
-        }
+        clientVersion = VersionUtils.version;
 
         try {
             perTaskConfigSize = Double.valueOf(System.getProperty("PER_TASK_CONFIG_SIZE", "3000"));

diff --git a/client/src/main/resources/application.properties b/client/src/main/resources/application.properties
diff --git a/client/src/main/resources/nacos-log4j2.xml b/client/src/main/resources/nacos-log4j2.xml
@@ -17,8 +17,8 @@
 
 <Configuration status="WARN">
     <Appenders>
-        <RollingFile name="CONFIG_LOG_FILE" fileName="${sys:nacos.logging.path}/config.log"
-            filePattern="${sys:nacos.logging.path}/config.log.%d{yyyy-MM-dd}.%i">
+        <RollingFile name="CONFIG_LOG_FILE" fileName="${sys:JM.LOG.PATH}/nacos/config.log"
+            filePattern="${sys:JM.LOG.PATH}/nacos/config.log.%d{yyyy-MM-dd}.%i">
             <PatternLayout>
                 <Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p [%-5t:%c{2}] %m%n</Pattern>
             </PatternLayout>
@@ -31,8 +31,8 @@
             <DefaultRolloverStrategy max="${sys:JM.LOG.RETAIN.COUNT:-7}"/>
         </RollingFile>
 
-        <RollingFile name="NAMING_LOG_FILE" fileName="${sys:nacos.logging.path}/naming.log"
-            filePattern="${sys:nacos.logging.path}/naming.log.%d{yyyy-MM-dd}.%i">
+        <RollingFile name="NAMING_LOG_FILE" fileName="${sys:JM.LOG.PATH}/nacos/naming.log"
+            filePattern="${sys:JM.LOG.PATH}/nacos/naming.log.%d{yyyy-MM-dd}.%i">
             <PatternLayout>
                 <Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %p [%-5t:%c{2}] %m%n</Pattern>
             </PatternLayout>