[Feature-6586][Server]add some ds process definition demo when init (a…

…pache#11759) * [Feature-6586][Server]add some ds process definition demo when init 1.add some ds process definition demo when init, to display what task type can run and make user easy to use ds. 2.need configure the JVM parameters (-Ddemo=true) to turn on the StandaloneServer service 3.need modify the tenant information in it
hstdream · Nov 2, 2022 · 72c985c · 72c985c
1 parent b37bf64
commit 72c985c
Show file tree

Hide file tree

Showing 16 changed files with 1,271 additions and 7 deletions.
diff --git a/docs/configs/docsdev.js b/docs/configs/docsdev.js
@@ -393,6 +393,10 @@ export default {
                         title: 'Expansion and Reduction',
                         link: '/en-us/docs/dev/user_doc/guide/expansion-reduction.html',
                     },
+                    {
+                        title: 'Demo',
+                        link: '/en-us/docs/dev/user_doc/guide/demo.html',
+                    },
                 ],
             },
             {
@@ -1013,6 +1017,10 @@ export default {
                         title: '扩/缩容',
                         link: '/zh-cn/docs/dev/user_doc/guide/expansion-reduction.html',
                     },
+                    {
+                        title: 'Demo',
+                        link: '/zh-cn/docs/dev/user_doc/guide/demo.html',
+                    },
                 ],
             },
             {

diff --git a/docs/docs/en/guide/demo.md b/docs/docs/en/guide/demo.md
@@ -0,0 +1,39 @@
+# DolphinScheduler Initialize The Workflow Demo
+
+## Prepare
+
+### Backup Previous Version's Files and Database
+
+To prevent data loss by some miss-operation, it is recommended to back up data before initializing the workflow demo. The backup way according to your environment.
+
+### Download the Latest Version Installation Package
+
+Download the latest binary distribute package from [download](/en-us/download/download.html) and then put it in the different
+directory where current service running. And all below command is running in this directory.
+
+## Start
+
+### Start Services of DolphinScheduler
+
+Start all services of dolphinscheduler according to your deployment method. If you deploy your dolphinscheduler according to [cluster deployment](installation/cluster.md), you can start all services by command `sh ./script/start-all.sh`.
+
+### Database Configuration
+
+Initializing the workflow demo needs to store metabase in other database like MySQL or PostgreSQL, they have to change some configuration. Follow the instructions in [datasource-setting](howto/datasource-setting.md) `Standalone Switching Metadata Database Configuration` section to create and initialize database.
+
+### Tenant Configuration
+
+#### Change `dolphinscheduler-tools/resources/application.yaml` Placement Details
+
+```
+demo:
+  tenant-code: default
+  domain-name: localhost
+  api-server-port: 5173
+```
+
+Mentioned above, tenant-code is the default tenant, users can modify the user name according to their operating system, this replaces the manual tenant creation action, api-server-port is the port number of the service.
+
+Then execute the startup script that initializes the workflow demo service: `sh ./tools/bin/create-demo-processes.sh` to start the service.
+
+To create a demo, you can refer to [Quick Start](start/quick-start.md)
diff --git a/docs/docs/zh/guide/demo.md b/docs/docs/zh/guide/demo.md
@@ -0,0 +1,40 @@
+# DolphinScheduler 初始化工作流 demo
+
+## 准备工作
+
+### 备份上一版本文件和数据库
+
+为了防止操作错误导致数据丢失，建议初始化工作流 demo 服务之前备份数据，备份方法请结合你数据库的情况来定
+
+### 下载新版本的安装包
+
+在[下载](/zh-cn/download/download.html)页面下载最新版本的二进制安装包，并将二进制包放到与当前 dolphinscheduler 服务不一样的路径中，以下服务启动操作都需要在新版本的目录进行。
+
+## 服务启动步骤
+
+### 开启 dolphinscheduler 服务
+
+根据你部署方式开启 dolphinscheduler 的所有服务，如果你是通过 [集群部署](installation/cluster.md) 来部署你的 dolphinscheduler 的话，可以通过 `sh ./script/start-all.sh` 开启全部服务。
+
+### 数据库配置
+
+初始化工作流 demo 服务需要使用 MySQL 或 PostgreSQL 等其他数据库作为其元数据存储数据，因此必须更改一些配置。
+请参考[数据源配置](howto/datasource-setting.md) `Standalone 切换元数据库`创建并初始化数据库 ，然后运行 demo 服务启动脚本。
+
+### 租户配置
+
+#### 修改 `dolphinscheduler-tools/resources/application.yaml` 配置内容
+
+```
+demo:
+  tenant-code: default
+  domain-name: localhost
+  api-server-port: 5173
+```
+
+其中 tenant-code 是默认租户 default ，用户可以根据自己操作系统用户名修改，从而代替手动创建租户操作。api-server-port 是 dolphinscheduler 服务的端口号
+
+然后执行初始化工作流 demo 服务的启动脚本：`sh ./tools/bin/create-demo-processes.sh` 来启动服务。
+
+创建 demo 可以参考[快速上手](start/quick-start.md)
+
diff --git a/dolphinscheduler-common/pom.xml b/dolphinscheduler-common/pom.xml
@@ -107,6 +107,11 @@
             <artifactId>netty-all</artifactId>
         </dependency>
 
+        <dependency>
+            <groupId>com.squareup.okhttp3</groupId>
+            <artifactId>okhttp</artifactId>
+        </dependency>
+
         <dependency>
             <groupId>javax.servlet</groupId>
             <artifactId>servlet-api</artifactId>

diff --git a/...er/plugin/task/api/utils/OkHttpUtils.java → ...inscheduler/common/utils/OkHttpUtils.java b/...er/plugin/task/api/utils/OkHttpUtils.java → ...inscheduler/common/utils/OkHttpUtils.java
@@ -15,14 +15,13 @@
  * limitations under the License.
  */
 
-package org.apache.dolphinscheduler.plugin.task.api.utils;
-
-import org.apache.dolphinscheduler.common.utils.JSONUtils;
+package org.apache.dolphinscheduler.common.utils;
 
 import org.apache.http.HttpStatus;
 
 import java.io.IOException;
 import java.util.Map;
+import java.util.concurrent.TimeUnit;
 
 import javax.annotation.Nullable;
 
@@ -36,7 +35,11 @@
 
 public class OkHttpUtils {
 
-    private static final OkHttpClient CLIENT = new OkHttpClient();
+    private static final OkHttpClient CLIENT = new OkHttpClient.Builder()
+            .connectTimeout(5, TimeUnit.MINUTES) // connect timeout
+            .writeTimeout(5, TimeUnit.MINUTES) // write timeout
+            .readTimeout(5, TimeUnit.MINUTES) // read timeout
+            .build();
 
     public static @NonNull String get(@NonNull String url,
                                       @Nullable Map<String, String> httpHeaders,
@@ -65,6 +68,32 @@ public class OkHttpUtils {
         }
     }
 
+    public static @NonNull String demoPost(@NonNull String url,
+                                           @Nullable String token,
+                                           @Nullable Map<String, Object> requestBodyMap) throws IOException {
+
+        StringBuffer stringBuffer = new StringBuffer();
+        if (requestBodyMap != null) {
+            for (String key : requestBodyMap.keySet()) {
+                stringBuffer.append(key + "=" + requestBodyMap.get(key) + "&");
+            }
+        }
+
+        RequestBody body =
+                RequestBody.create(MediaType.parse("application/x-www-form-urlencoded"), stringBuffer.toString());
+
+        Request request = new Request.Builder()
+                .url(url)
+                .header("token", token)
+                .addHeader("accpect", "application/json")
+                .post(body)
+                .build();
+
+        try (Response response = CLIENT.newCall(request).execute()) {
+            return response.body().string();
+        }
+
+    }
     private static String addUrlParams(@Nullable Map<String, Object> requestParams, @NonNull String url) {
         if (requestParams == null) {
             return url;

diff --git a/...ler/plugin/task/api/loop/template/http/method/HttpLoopTaskCancelTaskMethodDefinition.java b/...ler/plugin/task/api/loop/template/http/method/HttpLoopTaskCancelTaskMethodDefinition.java
@@ -17,10 +17,10 @@
 
 package org.apache.dolphinscheduler.plugin.task.api.loop.template.http.method;
 
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskCancelMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskMethodDefinition;
-import org.apache.dolphinscheduler.plugin.task.api.utils.OkHttpUtils;
 
 import org.apache.commons.lang3.StringUtils;
 

diff --git a/...er/plugin/task/api/loop/template/http/method/HttpLoopTaskQueryStatusMethodDefinition.java b/...er/plugin/task/api/loop/template/http/method/HttpLoopTaskQueryStatusMethodDefinition.java
@@ -17,13 +17,13 @@
 
 package org.apache.dolphinscheduler.plugin.task.api.loop.template.http.method;
 
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceStatus;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskQueryStatusMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskInstanceStatus;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.utils.JsonPathUtils;
-import org.apache.dolphinscheduler.plugin.task.api.utils.OkHttpUtils;
 
 import org.apache.commons.lang3.StringUtils;
 

diff --git a/...ler/plugin/task/api/loop/template/http/method/HttpLoopTaskSubmitTaskMethodDefinition.java b/...ler/plugin/task/api/loop/template/http/method/HttpLoopTaskSubmitTaskMethodDefinition.java
@@ -17,12 +17,12 @@
 
 package org.apache.dolphinscheduler.plugin.task.api.loop.template.http.method;
 
+import org.apache.dolphinscheduler.common.utils.OkHttpUtils;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.LoopTaskSubmitTaskMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskInstanceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.loop.template.http.HttpLoopTaskMethodDefinition;
 import org.apache.dolphinscheduler.plugin.task.api.utils.JsonPathUtils;
-import org.apache.dolphinscheduler.plugin.task.api.utils.OkHttpUtils;
 
 import org.apache.commons.lang3.StringUtils;
 

diff --git a/dolphinscheduler-tools/src/main/bin/create-demo-processes.sh b/dolphinscheduler-tools/src/main/bin/create-demo-processes.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BIN_DIR=$(dirname $0)
+DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/../..; pwd)}
+
+if [ "$DOCKER" != "true" ]; then
+  source "$DOLPHINSCHEDULER_HOME/bin/env/dolphinscheduler_env.sh"
+fi
+
+JAVA_OPTS=${JAVA_OPTS:-"-server -Duser.timezone=${SPRING_JACKSON_TIME_ZONE} -Xms1g -Xmx1g -Xmn512m -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof"}
+
+$JAVA_HOME/bin/java $JAVA_OPTS \
+  -cp "$DOLPHINSCHEDULER_HOME/tools/conf":"$DOLPHINSCHEDULER_HOME/tools/libs/*":"$DOLPHINSCHEDULER_HOME/tools/sql" \
+  -Dspring.profiles.active=demo,${DATABASE} \
+  org.apache.dolphinscheduler.tools.demo.CreateProcessDemo
diff --git a/...cheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/CreateDemoTenant.java b/...cheduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/CreateDemoTenant.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+import org.apache.dolphinscheduler.dao.entity.Tenant;
+import org.apache.dolphinscheduler.dao.mapper.TenantMapper;
+
+import java.util.Date;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+
+public class CreateDemoTenant {
+
+    private static final Logger logger = LoggerFactory.getLogger(CreateDemoTenant.class);
+    @Autowired
+    private TenantMapper tenantMapper;
+
+    public void createTenantCode(String tenantCode) {
+        Date now = new Date();
+
+        if (!tenantCode.equals("default")) {
+            Boolean existTenant = tenantMapper.existTenant(tenantCode);
+            if (!Boolean.TRUE.equals(existTenant)) {
+                Tenant tenant = new Tenant();
+                tenant.setTenantCode(tenantCode);
+                tenant.setQueueId(1);
+                tenant.setDescription("");
+                tenant.setCreateTime(now);
+                tenant.setUpdateTime(now);
+                // save
+                tenantMapper.insert(tenant);
+                logger.info("create tenant success");
+            } else {
+                logger.warn("os tenant code already exists");
+            }
+        }
+    }
+}
diff --git a/...heduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/CreateProcessDemo.java b/...heduler-tools/src/main/java/org/apache/dolphinscheduler/tools/demo/CreateProcessDemo.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.tools.demo;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.context.annotation.ComponentScan;
+import org.springframework.context.annotation.FilterType;
+import org.springframework.context.annotation.Profile;
+import org.springframework.stereotype.Component;
+
+@SpringBootApplication
+@ComponentScan(value = "org.apache.dolphinscheduler", excludeFilters = {
+        @ComponentScan.Filter(type = FilterType.REGEX, pattern = {
+                "org.apache.dolphinscheduler.tools.datasource.*",
+        })
+})
+public class CreateProcessDemo {
+
+    public static void main(String[] args) {
+        SpringApplication.run(CreateProcessDemo.class, args);
+    }
+
+    @Component
+    @Profile("demo")
+    static class DemoRunner implements CommandLineRunner {
+
+        private static final Logger logger = LoggerFactory.getLogger(DemoRunner.class);
+
+        private final ProcessDefinitionDemo processDefinitionDemo;
+
+        DemoRunner(ProcessDefinitionDemo processDefinitionDemo) {
+            this.processDefinitionDemo = processDefinitionDemo;
+        }
+
+        @Override
+        public void run(String... args) throws Exception {
+            processDefinitionDemo.createProcessDefinitionDemo();
+            logger.info("create process definition demo success");
+        }
+    }
+}