Skip to content

Commit 97b9e59

Browse files
Davidhua1996Davidhua1996
authored
Merge pull request #38 from WeDataSphere/dev-0.2.0-docs-appconn
Update English StreamisAppConnDesignDocument
2 parents ffb3ffe + 50beefd commit 97b9e59

File tree

9 files changed

+66
-107
lines changed

9 files changed

+66
-107
lines changed

‎README-ZH.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com
5555
| 依赖的应用工具 | 描述 | Streamis 兼容版本 |
5656
|--------------|---------------------------------------------------------------|--------------|
5757
| [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | 数据应用开发管理集成框架。以工作流式的图形化拖拽开发体验，将满足从数据交换、脱敏清洗、分析挖掘、质量检测、可视化展现、定时调度到数据输出应用等，数据应用开发全流程场景需求。 | >= DSS1.1.0（已发布）|
58-
| [Linkis](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis，通过提供 REST/WebSocket/JDBC/SDK 等标准接口，上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎。 | >= Linkis1.1.1（已发布） |
58+
| [Linkis](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis，通过提供 REST/WebSocket/JDBC/SDK 等标准接口，上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎。 | >= Linkis1.1.1（已发布），部分功能需要Linkis 1.1.2支持 |
5959

6060
----
6161

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,16 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/
3535

3636
       Supports multi-version management, full life cycle management, monitoring alarm, checkpoint and savepoint management capabilities of streaming jobs.
3737

38-
![prod center](docs/images/homePage_en.png)
38+
![prod center](docs/images/stream_product_center.png)
3939

4040
       Running information page:
4141

42-
![Running information](docs/images/statusDetail_en.png)
42+
![Running information](docs/images/stream_job_detail.png)
4343

4444
       Configurations page：
4545

46-
![Configurations](docs/images/config_en.png)
46+
![Configurations](docs/images/stream_job_config_1.png)
47+
![Configurations](docs/images/stream_job_config_2.png)
4748

4849
       For more features, please refer to: [User Manual](docs/en_US/userManual/StreamisUserManual.md).
4950

@@ -54,7 +55,7 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/
5455
| Depended Component | Description | Streamis compatibility |
5556
| -------------- | -------------------------------------------------------------- | --------------|
5657
| [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | Data application development management framework. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. | >= DSS1.1.0 (Released) |
57-
| [Linkis](https://github.com/apache/incubator-linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | &gt= Linkis1.1.1 (Released) |
58+
| [Linkis](https://github.com/apache/incubator-linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | >= Linkis1.1.1 (Released),some functions need to be supported by linkis 1.1.2 |
5859

5960
## Demo Trial environment
6061

‎docs/en_US/0.2.0/FlinkEnginePluginCompileDocument.md‎

Lines changed: 0 additions & 54 deletions
This file was deleted.

‎docs/en_US/0.2.0/StreamisDeployment.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,10 @@ After the compilation is successful, the installation package will be generated
4040
- JDK (above 1.8.0_141), [How to install JDK](https://www.runoob.com/java/java-environment-setup.html)
4141

4242
### 3.2 Linkis and DSS environments
43-
- The execution of Streamis depends on Linkis, and it needs to be version 1.1.1 and above, so you need to install Linkis above 1.1.1 and ensure that the Flink engine can be used normally. Specifically, you can create a new and edit a flinksql script on Scriptis
43+
- The execution of Streamis depends on Linkis, and it needs to be version 1.1.1 and above, so you need to install Linkis above 1.1.1 and ensure that the Flink engine can be used normally.Some functions need to be supported by linkis-1.1.2.
4444
- Datasphere studio (> =1.1.0), the development and debugging of streaming jobs depend on DSS scriptis, and the streaming production center needs to be embedded in the DSS engineering framework system, so it depends on * * dss-1.1.0 * * and above.
4545

46-
Before the formal installation of streamis, please install linkis1.1.1 and dss1.1.0 or above, and ensure that the linkis Flink engine and DSS can be used normally. For the installation of DSS and linkis, please refer to the [dss & linkis one click installation and deployment document](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/DSS%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md).
46+
Before the formal installation of streamis, please install linkis-1.1.1 and dss-1.1.0 or above, and ensure that the linkis Flink engine and DSS can be used normally. For the installation of DSS and linkis, please refer to the [dss & linkis one click installation and deployment document](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/DSS%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md).
4747

4848
How to verify that DSS and linkis are basically available? You can create a flinksql script on DSS scriptis and execute it. If flinksql can execute correctly and return the result set, it means that the DSS and linkis environments are available.
4949

‎docs/en_US/0.2.0/StreamisAppConnDesignDocument.md‎ renamed to ‎docs/en_US/0.2.0/architecture/StreamisAppConnDesignDocument.md‎

Lines changed: 14 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,19 @@
11
# Streamis access AppConn
22

33
## Overall flow chart
4-
![Streamis access DSS](../../images/zh_CN/streamis_appconn.png)
4+
![Streamis access DSS](../../../images/streamis_appconn_en.png)
55

66
## DSS project APPCONN plug-in streamis-appconn
77

88
### The configuration table
9-
Configure the following three tables:dss_appconn、dss_workspace_menu_appconn、dss_appconn_instance，appconn_name for realTimeJobCenter is appconn accessed by the graphical interface，appconn_name for streamis is appconn accessed by the API，The StreamisAppConn object is instantiated based on the configuration information in the table when DSS is started。
9+
Configure the following four tables:dss_workspace_dictionary、dss_appconn、dss_workspace_menu_appconn、dss_appconn_instance，appconn_name for realTimeJobCenter is appconn accessed by the graphical interface，appconn_name for streamis is appconn accessed by the API，The StreamisAppConn object is instantiated based on the configuration information in the table when DSS is started.Appconn in the following SQL_ INSTALL_ IP and appconn_ INSTALL_ When executing DSS installation script for automatic installation, port will enter through interactive commands.
1010
```roomsql
11+
delete from `dss_workspace_dictionary` WHERE `appconn_name` = 'streamis';
12+
13+
INSERT INTO `dss_workspace_dictionary` ( `workspace_id`, `parent_key`, `dic_name`, `dic_name_en`, `dic_key`, `dic_value`, `dic_value_en`, `title`, `title_en`, `url`, `url_type`,`icon`, `order_num`, `remark`, `create_user`, `create_time`, `update_user`, `update_time`, appconn_name)
14+
VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center','pdp_streamis_product_center','streamis_prod',NULL,NULL,NULL,
15+
'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/#/realtimeJobCenter?projectName=${projectName}&workspaceName=${workspaceName}','0','kaifa-icon','1','工程开发流程-流式生产中心','SYSTEM','2020-12-28 17:32:35',NULL,'2022-06-30 17:49:02','streamis');
16+
1117
select @old_dss_appconn_id:=id from `dss_appconn` where `appconn_name` = 'streamis';
1218
1319
delete from `dss_workspace_menu_appconn` WHERE `appconn_id` = @old_dss_appconn_id;
@@ -37,11 +43,11 @@ VALUES(@jobcenter_dss_appconn_id, 1, 'StreamSQL development', 'StreamSQL开发',
3743
3844
INSERT INTO dss_appconn_instance
3945
(appconn_id, label, url, enhance_json, homepage_uri)
40-
VALUES(@dss_appconn_id, 'DEV', 'http://Streamis_INSTALL_IP:Streamis_INSTALL_PORT/', '', 'http://Streamis_INSTALL_IP:Streamis_INSTALL_PORT/#/realTimeJobCenter');
46+
VALUES(@dss_appconn_id, 'DEV', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/', '', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/#/realTimeJobCenter');
4147
4248
INSERT INTO dss_appconn_instance
4349
(appconn_id, label, url, enhance_json, homepage_uri)
44-
VALUES(@jobcenter_dss_appconn_id, 'DEV', 'http://Streamis_INSTALL_IP:Streamis_INSTALL_PORT/#/realTimeJobCenter', NULL, NULL);
50+
VALUES(@jobcenter_dss_appconn_id, 'DEV', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/#/realTimeJobCenter', NULL, NULL);
4551
```
4652

4753
### Concrete implementation description
@@ -55,7 +61,7 @@ To create a StreamisProjectService by rewriting the methods in StreamisStructure
5561

5662
## API
5763
1 API name: query project
58-
- API path：GET/streamis/project/searchProject
64+
- API path：GET /streamis/project/searchProject
5965
- Request parameters
6066

6167
|Parameter name |Whether it is necessary |Example |remarks |
@@ -73,7 +79,7 @@ To create a StreamisProjectService by rewriting the methods in StreamisStructure
7379
|- projectId |number |yes |
7480

7581
2 API name：create project
76-
- API path：GET/streamis/project/createProject
82+
- API path：POST /streamis/project/createProject
7783
- Request parameters
7884

7985
|Parameter name |Whether it is necessary |Example |remarks |
@@ -96,7 +102,7 @@ To create a StreamisProjectService by rewriting the methods in StreamisStructure
96102
|- projectName |string |no |
97103

98104
3 API name：update project
99-
- API path：GET/streamis/project/updateProject
105+
- API path：PUT /streamis/project/updateProject
100106
- Request parameters
101107

102108
|Parameter name |Whether it is necessary |Example |remarks |
@@ -118,7 +124,7 @@ To create a StreamisProjectService by rewriting the methods in StreamisStructure
118124
|data |object |no |
119125

120126
4 API name：delete project
121-
- API path：GET/streamis/project/deleteProject
127+
- API path：DELETE /streamis/project/deleteProject
122128
- Request parameters
123129

124130
|Parameter name |Whether it is necessary |Example |remarks |
@@ -170,40 +176,3 @@ ALTER TABLE `linkis_stream_project` ADD is_deleted tinyint unsigned DEFAULT 0;
170176
- The creation operation will insert the project information (projectName、workspaceId) in the request parameters into the project table linkis_stream_project and auto increment the generated ID, associate the user in the permission information data (releaseUsers、editUsers、accessUsers) in the request parameters with the generated ID of the project table, and insert the table linkis_stream_project_privilege, the status value returned from the successful creation operation is 0, and the project table generation ID will be returned as the value of projectId;
171177
- The modification operation will update the request information data to the table linkis_stream_project and linkis_stream_project_privilege, the status value returned successfully is 0;
172178
- In the delete operation, the is_deleted field of the table linkis_stream_project will be marked as 1 according to the projectId. The relevant data in the table linkis_stream_project_privilege will be deleted, the status value returned successfully is 0.
173-
174-
# Authentication
175-
176-
## Authentication flow chart
177-
![Streamis project authentication operation](../../images/zh_CN/streamis_project_privilege.png)
178-
179-
### edit privilege API：
180-
181-
|RequestMethod |API path |name |
182-
|------|----------------------------------------------------------|-----------------|
183-
|POST |/streamis/streamProjectManager/project/files/upload |Project resource file - Import |
184-
|GET |/streamis/streamProjectManager/project/files/delete |Delete all versions of the file under the project |
185-
|GET |/streamis/streamProjectManager/project/files/version/delete |Delete version file |
186-
|GET |/streamis/streamProjectManager/project/files/download |Task details - Download |
187-
|POST |/streamis/streamJobManager/job/upload |Upload file |
188-
|POST |/streamis/streamJobManager/job/execute |start-up |
189-
|GET |/streamis/streamJobManager/job/stop |stop |
190-
|PUT |/streamis/streamJobManager/job//snapshot/{jobId:\w+} |Snapshot generation |
191-
|GET |/streamis/streamJobManager/config/json/{jobId:\w+} |Configuration - save |
192-
|POST |/streamis/streamJobManager/job/bulk/execution |Batch start |
193-
|POST |/streamis/streamJobManager/job/bulk/pause |Batch stop |
194-
195-
196-
### access privilege API：
197-
198-
|RequestMethod |API path |name |
199-
|------|----------------------------------------------------------|-------------|
200-
|GET |streamis/streamJobManager/job/list |Query the jobs that the current user can view |
201-
|GET |/streamis/streamProjectManager/project/files/list |prokect resource document |
202-
|GET |/streamis/streamProjectManager/project/files/version/list |Obtain all versions of the file under the project |
203-
|GET |/streamis/streamJobManager/job/version |Query job version |
204-
|GET |/streamis/streamJobManager/job/execute/history |Job execution history |
205-
|GET |/streamis/streamJobManager/job/progress |Get the latest task status of the current version of the job |
206-
|GET |/streamis/streamJobManager/job/jobContent |Task details |
207-
|GET |/streamis/streamJobManager/job/logs |Get log |
208-
|POST |/streamis/streamJobManager/config/json/{jobId:\w+} |Get task configuration |
209-
|GET |/streamis/streamJobManager/config/view |Query the current job configuration information |

‎docs/en_US/0.2.0/architecture/StreamisAuthenticationDesignDocument.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Authentication
2+
3+
## Authentication flow chart
4+
In Streamis, the module that needs authentication does not rely on the Streamis project server module. The rest interface is called to handle authentication.
5+
6+
![Streamis project authentication operation](../../../images/streamis_project_privilege_en.png)
7+
8+
## Specific implementation instructions
9+
Get the set of all permissions according to the current user name and item id/ name. If the permission set contains RELEASE permission, you have the permission to publish / edit / view; if the permission set contains EDIT permission, you have the permission to edit / view; if the permission set contains ACCESS permission, you have the permission to view;
10+
Permission inclusion relationship: RELEASE permission includes EDIT permission and ACCESS permission; edit permission includes ACCESS permission.
11+
12+
### edit privilege API：
13+
14+
|RequestMethod |API path |name |
15+
|------|----------------------------------------------------------|-----------------|
16+
|POST |/streamis/streamProjectManager/project/files/upload |Project resource file - Import |
17+
|GET |/streamis/streamProjectManager/project/files/delete |Delete all versions of the file under the project |
18+
|GET |/streamis/streamProjectManager/project/files/version/delete |Delete version file |
19+
|GET |/streamis/streamProjectManager/project/files/download |Task details - Download |
20+
|POST |streamis/streamJobManager/job/createOrUpdate |create or Update streamis-job|
21+
|POST |/streamis/streamJobManager/job/upload |Upload file |
22+
|POST |/streamis/streamJobManager/job/execute |start-up |
23+
|GET |/streamis/streamJobManager/job/stop |stop |
24+
|PUT |/streamis/streamJobManager/job//snapshot/{jobId:\w+} |Snapshot generation |
25+
|GET |/streamis/streamJobManager/config/json/{jobId:\w+} |Configuration - save |
26+
|POST |/streamis/streamJobManager/job/bulk/execution |Batch start |
27+
|POST |/streamis/streamJobManager/job/bulk/pause |Batch stop |
28+
29+
30+
### access privilege API：
31+
32+
|RequestMethod |API path |name |
33+
|------|----------------------------------------------------------|-------------|
34+
|GET |streamis/streamJobManager/job/list |Query the jobs that the current user can view |
35+
|GET |/streamis/streamProjectManager/project/files/list |prokect resource document |
36+
|GET |/streamis/streamProjectManager/project/files/version/list |Obtain all versions of the file under the project |
37+
|GET |/streamis/streamJobManager/job/version |Query job version |
38+
|GET |/streamis/streamJobManager/job/execute/history |Job execution history |
39+
|GET |/streamis/streamJobManager/job/progress |Get the latest task status of the current version of the job |
40+
|GET |/streamis/streamJobManager/job/jobContent |Task details |
41+
|GET |/streamis/streamJobManager/job/logs |Get log |
42+
|POST |/streamis/streamJobManager/config/json/{jobId:\w+} |Get task configuration |
43+
|GET |/streamis/streamJobManager/config/view |Query the current job configuration information |

‎docs/images/streamis_appconn_en.png‎

75.8 KB
Loading

‎docs/images/streamis_project_privilege_en.png‎

106 KB
Loading

0 commit comments

Comments
 (0)