Skip to content

Commit 04650fc

Browse files
JinXingYoungJinXingYoung
committed
Proposal: Adapt Multiple Type of Database for Harbor
Update support-multiDB.md Basic structure add png feat: db proposal initial verion (#2) Update the figure Update adapt_multiple_type_of_database.md (#3) * Update adapt_multiple_type_of_database.md Some update * Update adapt_multiple_type_of_database.md Update authors Update adapt_multiple_type_of_database.md Signed-off-by: JinXingYoung <[email protected]>
1 parent 29b42cd commit 04650fc

File tree

6 files changed

+400
-0
lines changed

6 files changed

+400
-0
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Workgroup is a virtual team of aggregating the efforts of all the interested par
1919
|[Multi Architecture](https://github.com/goharbor/community/tree/master/workgroups/wg-multiarch)|The purpose of this working group is to enable Harbor to support multiple architectures, such as x86 arm.|ZhiPeng Yu@Alauda ([yuzp1996](https://github.com/yuzp1996))| Steven Zou@VMware ([steven-zou](https://github.com/steven-zou)) |[8 Members](https://github.com/goharbor/community/tree/master/workgroups/wg-multiarch#members)|
2020
|[Performance](https://github.com/goharbor/community/tree/master/workgroups/wg-performance)|The purpose of this work group is to improve the performance and scalability of harbor in the use case of large data volumes.|ChenYu Zhang@Alauda ([chlins](https://github.com/chlins))| Steven Zou@VMware ([steven-zou](https://github.com/steven-zou))/ Daniel Morinigo@Alauda ([danielfbm](https://github.com/danielfbm)) |[11 Members](https://github.com/goharbor/community/tree/master/workgroups/wg-performance#members)|
2121
|[Image Acceleration](https://github.com/goharbor/community/tree/master/workgroups/wg-image-accel)|The objective of this working group is to aggregate efforts to discuss, design and finally integrate accelerated container image format support in Harbor.|Bo Liu@AlibabaCloud ([liubogithub](https://github.com/liubogithub))|Steven Zou@VMware ([steven-zou](https://github.com/steven-zou))/Wang Yan@VMWare ([wy65701436](https://github.com/wy65701436))|[7 members](https://github.com/goharbor/community/tree/master/workgroups/wg-image-accel#members)|
22+
|[Databases](https://github.com/goharbor/community/tree/master/workgroups/wg-databases)|The objective of this working group is to aggregate efforts to support different databases in Harbor in addition to PostgreSQL.|Yvonne@Tencent[@JinXingYoung](https://github.com/JinXingYoung)|Yiyang Huang@ByteDance[@hyy0322](https://github.com/hyy0322)|[3 members](https://github.com/goharbor/community/tree/master/workgroups/wg-databases#members)|
2223

2324
## Structure
2425

‎proposals/adapt_multiple_type_of_database.md

Lines changed: 383 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,383 @@
1+
# Proposal: `Adapt Multiple Type of Database for Harbor`
2+
3+
Author:
4+
5+
- Yvonne [@JinXingYoung](https://github.com/JinXingYoung)
6+
- Yiyang Huang [@hyy0322](https://github.com/hyy0322)
7+
- De Chen [@cd1989](https://github.com/cd1989)
8+
- Minglu [@ConnieHan2019]([email protected])
9+
10+
Links:
11+
12+
- Previous Discussion: [goharbor/harbor#6534](https://github.com/goharbor/harbor/issues/6534)
13+
- Related PR from Others Before: [goharbor/harbor#14265](https://github.com/goharbor/harbor/pull/14265)
14+
15+
## Abstract
16+
17+
Propose to support other databases in Harbor, the first step is to support MySQL/MariaDB. This proposal introduces an abstract DAO layer in Harbor, different database can have their drivers to implement the interface. So that Harbor can adapt to other databases as long as a well tested database driver provided.
18+
19+
20+
## Background
21+
22+
As previous discussion([goharbor/harbor#6534](https://github.com/goharbor/harbor/issues/6534)) shown, there are certain amount of users(especially in China) lack the experiences of maintaining PostgreSQL for HA, disaster recovery, etc. And meanwhile they are more familiar with other databases such as MySQL/MariaDB. They prefer to use MariaDB/MySQL instead of PostgreSQL to keep their production environments stable.
23+
24+
As we all know that Harbor used MySQL before. But scanner clair use PostgreSQL as database. In order to keep consistency with clair and reduce maintenance difficulties. Harbor unified database using PostgreSQL.
25+
26+
Since Harbor v2.0 use trivy as default scanner instead of clair，there's no strongly requirement to use PostgreSQL anymore. Therefore, it is possible to adapt different kind of database now.
27+
28+
## Proposal
29+
30+
Support other databases in Harbor other than PostgreSQL.
31+
32+
### Goals
33+
34+
- Keep using PostgreSQL as default database. The Implementation will be compatible with current version of Harbor.
35+
- Abstract DAO layer for different type of databases.
36+
- Support MariaDB(10.5.9), MySQL(8.0) by implementing corresponding drivers and resolving sql compatibility.
37+
- Provide migration tool or guide for users to migrate data from PostgreSQL to MariaDB/MySQL.
38+
39+
### Non-Goals
40+
41+
- Support other type of database this time, such as MongoDB, Oracle, CockroachDB. (This is restricted to this effort, we welcome other data implementations in our framework and welcome to join us to support more databases)
42+
- Implement Mariadb/MySQL operator for internal database case.
43+
44+
## Implementation
45+
46+
### Overview
47+
48+
![overview.png](images/multidb/overview.png)
49+
50+
![initdb.png](images/multidb/initdb.png)
51+
52+
![dbdriver.png](images/multidb/dbdriver.png)
53+
54+
### Component Detail
55+
56+
**Common**
57+
58+
Add mysql config settings: src/lib/config/metadata/metadatalist.go
59+
```go
60+
var (
61+
// ConfigList - All configure items used in harbor
62+
// Steps to onboard a new setting
63+
// 1. Add configure item in metadatalist.go
64+
// 2. Get/Set config settings by CfgManager
65+
// 3. CfgManager.Load()/CfgManager.Save() to load/save from configure storage.
66+
ConfigList = []Item{
67+
...
68+
{Name: common.MySQLDatabase, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_DATABASE", DefaultValue: "registry", ItemType: &StringType{}, Editable: false},
69+
{Name: common.MySQLHOST, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_HOST", DefaultValue: "mysql", ItemType: &StringType{}, Editable: false},
70+
{Name: common.MySQLPassword, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_PASSWORD", DefaultValue: "root123", ItemType: &PasswordType{}, Editable: false},
71+
{Name: common.MySQLPort, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_PORT", DefaultValue: "3306", ItemType: &PortType{}, Editable: false},
72+
{Name: common.MySQLUsername, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_USERNAME", DefaultValue: "root", ItemType: &StringType{}, Editable: false},
73+
{Name: common.MySQLMaxIdleConns, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_MAX_IDLE_CONNS", DefaultValue: "2", ItemType: &IntType{}, Editable: false},
74+
{Name: common.MySQLMaxOpenConns, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_MAX_OPEN_CONNS", DefaultValue: "0", ItemType: &IntType{}, Editable: false},
75+
...
76+
}
77+
```
78+
79+
Add DB type in configs: make/photon/prepare/utils/configs.py
80+
```
81+
if external_db_configs:
82+
config_dict['external_database'] = True
83+
# harbor db
84+
config_dict['harbor_db_type'] = external_db_configs['harbor']['type']
85+
config_dict['harbor_db_host'] = external_db_configs['harbor']['host']
86+
config_dict['harbor_db_port'] = external_db_configs['harbor']['port']
87+
config_dict['harbor_db_name'] = external_db_configs['harbor']['db_name']
88+
config_dict['harbor_db_username'] = external_db_configs['harbor']['username']
89+
...
90+
```
91+
92+
MySQL struct: src/common/models/database.go
93+
```go
94+
type MySQL struct {
95+
Host string `json:"host"`
96+
Port int `json:"port"`
97+
Username string `json:"username"`
98+
Password string `json:"password,omitempty"`
99+
Database string `json:"database"`
100+
MaxIdleConns int `json:"max_idle_conns"`
101+
MaxOpenConns int `json:"max_open_conns"`
102+
}
103+
```
104+
105+
Get database infos for different db types: src/common/dao/base.go
106+
```go
107+
func getDatabase(database *models.Database) (db Database, err error) {
108+
switch database.Type {
109+
case "", "postgresql":
110+
db = NewPGSQL(
111+
database.PostGreSQL.Host,
112+
strconv.Itoa(database.PostGreSQL.Port),
113+
database.PostGreSQL.Username,
114+
database.PostGreSQL.Password,
115+
database.PostGreSQL.Database,
116+
database.PostGreSQL.SSLMode,
117+
database.PostGreSQL.MaxIdleConns,
118+
database.PostGreSQL.MaxOpenConns,
119+
)
120+
case "mariadb", "mysql":
121+
db = NewMySQL(
122+
database.MySQL.Host,
123+
strconv.Itoa(database.MySQL.Port),
124+
database.MySQL.Username,
125+
database.MySQL.Password,
126+
database.MySQL.Database,
127+
database.MySQL.MaxIdleConns,
128+
database.MySQL.MaxOpenConns,
129+
)
130+
default:
131+
err = fmt.Errorf("invalid database: %s", database.Type)
132+
}
133+
return
134+
}
135+
```
136+
137+
MigrateDB with certain sql file: src/migration/migration.go
138+
```go
139+
// MigrateDB upgrades DB schema and do necessary transformation of the data in DB
140+
func MigrateDB(database *models.Database) error {
141+
var migrator *migrate.Migrate
142+
var err error
143+
144+
// check the database schema version
145+
switch database.Type {
146+
case "", "postgresql":
147+
migrator, err = dao.NewMigrator(database.PostGreSQL) // migrate db with postgres sql file
148+
case "mariadb", "mysql":
149+
migrator, err = dao.NewMysqlMigrator(database.MySQL) // migrate db with mysql sql file
150+
}
151+
152+
...
153+
}
154+
```
155+
156+
Migration sql file architecture
157+
```
158+
harbor/
159+
├── make
160+
│ ├── migrations
161+
│ │ ├── mysql
162+
│ │ │ ├── 0001_initial_schema.up.sql // File content may different with postgresql
163+
│ │ │ ├── ...
164+
│ │ ├── postgresql
165+
│ │ │ ├── 0001_initial_schema.up.sql
166+
│ │ │ ├── ...
167+
```
168+
169+
DAO layer make different databases compatible. We can extend database support by implementing mysql implementation.
170+
171+
DAO interface in src/pkg/repository/dao/dao.go
172+
```go
173+
// DAO is the data access object interface for repository
174+
type DAO interface {
175+
// Count returns the total count of repositories according to the query
176+
Count(ctx context.Context, query *q.Query) (count int64, err error)
177+
// List repositories according to the query
178+
List(ctx context.Context, query *q.Query) (repositories []*model.RepoRecord, err error)
179+
// Get the repository specified by ID
180+
Get(ctx context.Context, id int64) (repository *model.RepoRecord, err error)
181+
// Create the repository
182+
Create(ctx context.Context, repository *model.RepoRecord) (id int64, err error)
183+
// Delete the repository specified by ID
184+
Delete(ctx context.Context, id int64) (err error)
185+
// Update updates the repository. Only the properties specified by "props" will be updated if it is set
186+
Update(ctx context.Context, repository *model.RepoRecord, props ...string) (err error)
187+
// AddPullCount increase pull count for the specified repository
188+
AddPullCount(ctx context.Context, id int64, count uint64) error
189+
// NonEmptyRepos returns the repositories without any artifact or all the artifacts are untagged.
190+
NonEmptyRepos(ctx context.Context) ([]*model.RepoRecord, error)
191+
}
192+
```
193+
194+
**Core**
195+
196+
make/photon/prepare/templates/core/env.jinja:
197+
```
198+
DATABASE_TYPE={{harbor_db_type}}
199+
{% if ( harbor_db_type == "mysql" or harbor_db_type == "mariadb" ) %}
200+
MYSQL_HOST={{harbor_db_host}}
201+
MYSQL_PORT={{harbor_db_port}}
202+
MYSQL_USERNAME={{harbor_db_username}}
203+
MYSQL_PASSWORD={{harbor_db_password}}
204+
MYSQL_DATABASE={{harbor_db_name}}
205+
MYSQL_MAX_IDLE_CONNS={{harbor_db_max_idle_conns}}
206+
MYSQL_MAX_OPEN_CONNS={{harbor_db_max_open_conns}}
207+
{% else %}
208+
POSTGRESQL_HOST={{harbor_db_host}}
209+
POSTGRESQL_PORT={{harbor_db_port}}
210+
POSTGRESQL_USERNAME={{harbor_db_username}}
211+
POSTGRESQL_PASSWORD={{harbor_db_password}}
212+
POSTGRESQL_DATABASE={{harbor_db_name}}
213+
POSTGRESQL_SSLMODE={{harbor_db_sslmode}}
214+
POSTGRESQL_MAX_IDLE_CONNS={{harbor_db_max_idle_conns}}
215+
POSTGRESQL_MAX_OPEN_CONNS={{harbor_db_max_open_conns}}
216+
{% endif %}
217+
```
218+
219+
**Exporter**
220+
221+
make/photon/prepare/templates/exporter/env.jinja
222+
```
223+
HARBOR_DATABASE_TYPE={{harbor_db_type}}
224+
HARBOR_DATABASE_HOST={{harbor_db_host}}
225+
HARBOR_DATABASE_PORT={{harbor_db_port}}
226+
HARBOR_DATABASE_USERNAME={{harbor_db_username}}
227+
HARBOR_DATABASE_PASSWORD={{harbor_db_password}}
228+
HARBOR_DATABASE_DBNAME={{harbor_db_name}}
229+
HARBOR_DATABASE_SSLMODE={{harbor_db_sslmode}}
230+
```
231+
232+
**JobService**
233+
234+
JobService get configs from Core, then initDatabase with db config obtained.
235+
src/pkg/config/manager.go
236+
```go
237+
// GetDatabaseCfg - Get database configurations
238+
func (c *CfgManager) GetDatabaseCfg() *models.Database {
239+
ctx := context.Background()
240+
database := &models.Database{}
241+
database.Type = c.Get(ctx, common.DatabaseType).GetString()
242+
243+
switch database.Type {
244+
case "", "postgresql":
245+
postgresql := &models.PostGreSQL{
246+
Host: c.Get(ctx, common.PostGreSQLHOST).GetString(),
247+
Port: c.Get(ctx, common.PostGreSQLPort).GetInt(),
248+
Username: c.Get(ctx, common.PostGreSQLUsername).GetString(),
249+
Password: c.Get(ctx, common.PostGreSQLPassword).GetString(),
250+
Database: c.Get(ctx, common.PostGreSQLDatabase).GetString(),
251+
SSLMode: c.Get(ctx, common.PostGreSQLSSLMode).GetString(),
252+
MaxIdleConns: c.Get(ctx, common.PostGreSQLMaxIdleConns).GetInt(),
253+
MaxOpenConns: c.Get(ctx, common.PostGreSQLMaxOpenConns).GetInt(),
254+
}
255+
database.PostGreSQL = postgresql
256+
case "mariadb", "mysql":
257+
mysql := &models.MySQL{
258+
Host: c.Get(ctx, common.MySQLHOST).GetString(),
259+
Port: c.Get(ctx, common.MySQLPort).GetInt(),
260+
Username: c.Get(ctx, common.MySQLUsername).GetString(),
261+
Password: c.Get(ctx, common.MySQLPassword).GetString(),
262+
Database: c.Get(ctx, common.MySQLDatabase).GetString(),
263+
MaxIdleConns: c.Get(ctx, common.MySQLMaxIdleConns).GetInt(),
264+
MaxOpenConns: c.Get(ctx, common.MySQLMaxOpenConns).GetInt(),
265+
}
266+
database.MySQL = mysql
267+
}
268+
269+
return database
270+
}
271+
```
272+
273+
### Migration
274+
275+
#### Write a document Using official migration tool
276+
277+
MySQL community provide a [tool](https://dev.mysql.com/doc/workbench/en/wb-migration-database-postgresql.html) to migrate data.(Need further test)
278+
279+
#### Write a script to migrate data.
280+
281+
1. Define data models for postgreSQL and MariaDB/MySQL.
282+
2. Read data from postgreSQL and map to data model.
283+
3. Transfer postgreSQL data model to MariaDB/MySQL data model.
284+
4. Write data to MariaDB/MySQL database
285+
286+
### Database Compatibility Testing
287+
288+
**MySQL 8.0**
289+
290+
We have done some test for SQL compatibility. Here we list some SQL Incompatible points.
291+
292+
- TRIGGER is not needed in MySQL. Just use default CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP.
293+
- SERIAL type in MySQL is bigint unsigned. So a column reference other column type SERIAL must define as bigint unsigned.
294+
- Alter column type in MySQL must use MODIFY.
295+
- Loop keyword is different between MySQL and PostgreSQL.
296+
- Create index not support IF NOT EXIST in MySQL.
297+
298+
sql file | Compatibility test | comment
299+
------------|------------|------------
300+
| 0001_initial_schema.up.sql | Pass | xxx
301+
| 0002_1.7.0_schema.up.sql | Pass | xxx
302+
| 0003_add_replication_op_uuid.up.sql | Pass | xxx
303+
| 0004_1.8.0_schema.up.sql | Pass | xxx
304+
| 0005_1.8.2_schema.up.sql | Pass | xxx
305+
| 0010_1.9.0_schema.up.sql | Pass | xxx
306+
| 0011_1.9.1_schema.up.sql | Pass | xxx
307+
| 0012_1.9.4_schema.up.sql | Pass | xxx
308+
| 0015_1.10.0_schema.up.sql | Pass | xxx
309+
| 0030_2.0.0_schema.up.sql | Pass | xxx
310+
| 0031_2.0.3_schema.up.sql | Pass | xxx
311+
| 0040_2.1.0_schema.up.sql | Pass | xxx
312+
| 0041_2.1.4_schema.up.sql | Pass | xxx
313+
| 0050_2.2.0_schema.up.sql | Pass | xxx
314+
| 0051_2.2.1_schema.up.sql | Pass | xxx
315+
| 0052_2.2.2_schema.up.sql | Pass | xxx
316+
| 0053_2.2.3_schema.up.sql | Pass | xxx
317+
| 0060_2.3.0_schema.up.sql | Pass | xxx
318+
| 0061_2.3.4_schema.up.sql | Pass | xxx
319+
| 0070_2.4.0_schema.up.sql | Pass | xxx
320+
| 0071_2.4.2_schema.up.sql | Pass | xxx
321+
| 0080_2.5.0_schema.up.sql | Pass | xxx
322+
323+
**MariaDB 10.5.9**
324+
325+
sql file | Compatibility test | comment
326+
------------|------------|------------
327+
| 0001_initial_schema.up.sql | Pass | xxx
328+
| 0002_1.7.0_schema.up.sql | Pass | xxx
329+
| 0003_add_replication_op_uuid.up.sql | Pass | xxx
330+
| 0004_1.8.0_schema.up.sql | Pass | xxx
331+
| 0005_1.8.2_schema.up.sql | Pass | xxx
332+
| 0010_1.9.0_schema.up.sql | Pass | xxx
333+
| 0011_1.9.1_schema.up.sql | Pass | xxx
334+
| 0012_1.9.4_schema.up.sql | Pass | xxx
335+
| 0015_1.10.0_schema.up.sql | Pass | xxx
336+
| 0030_2.0.0_schema.up.sql | Pass | xxx
337+
| 0031_2.0.3_schema.up.sql | Pass | xxx
338+
| 0040_2.1.0_schema.up.sql | | xxx
339+
| 0041_2.1.4_schema.up.sql | | xxx
340+
| 0050_2.2.0_schema.up.sql | | xxx
341+
| 0051_2.2.1_schema.up.sql | | xxx
342+
| 0052_2.2.2_schema.up.sql | | xxx
343+
| 0053_2.2.3_schema.up.sql | | xxx
344+
| 0060_2.3.0_schema.up.sql | | xxx
345+
| 0061_2.3.4_schema.up.sql | | xxx
346+
| 0070_2.4.0_schema.up.sql | | xxx
347+
| 0071_2.4.2_schema.up.sql | | xxx
348+
| 0080_2.5.0_schema.up.sql | | xxx
349+
350+
### How To Use
351+
352+
Users can configure database type to use MariaDB/MySQL in external_database mode. PostgreSQL will be used by default.
353+
354+
Set harbor_db_type configuration under external_database and db type under notary db configuration:
355+
```
356+
external_database:
357+
harbor:
358+
# database type, default is postgresql, options include postgresql, mariadb and mysql
359+
type: mysql
360+
host: mysql.test.cn
361+
port: 3306
362+
db_name: harbor
363+
username: root
364+
password: root
365+
ssl_mode: disable
366+
max_idle_conns: 2
367+
max_open_conns: 0
368+
notary_signer:
369+
host: mysql.test.cn
370+
port: 3306
371+
db_name: notary_signer
372+
username: root
373+
password: root
374+
ssl_mode: disable
375+
notary_server:
376+
host: mysql.test.cn
377+
port: 3306
378+
db_name: notary_server
379+
username: root
380+
password: root
381+
ssl_mode: disable
382+
```
383+

‎proposals/images/multidb/dbdriver.png

57 KB
Loading

‎proposals/images/multidb/initdb.png

94.5 KB
Loading

‎proposals/images/multidb/overview.png

71.2 KB
Loading

0 commit comments

Comments
 (0)