Skip to content

Commit 436a181

Browse files
mariscalromeroalejandrograduta
mariscalromeroalejandro
and
graduta
authored
[OGUI-1620] Create Sequelize seeders for mocking data (#3111)
* introduces seeders that will be use to mock layout data for test and dev environments. * updates the configuration of the database to execute the migrations and the seeders in a different way depending on the environment in which they are executed. --------- Co-authored-by: George Raduta <[email protected]>
1 parent d53f173 commit 436a181

16 files changed

+632
-34
lines changed

‎QualityControl/config-default.js‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@
1313
*/
1414

1515
export const config = {
16-
demoData: false,
1716

1817
http: {
1918
port: 8080,
@@ -49,9 +48,11 @@ export const config = {
4948
timezone: '+00:00',
5049
logging: false,
5150
retryThrottle: 5000,
51+
forceSeed: false,
52+
drop: false,
5253
},
5354
bookkeeping: {
54-
url: 'http://localhost:4000', // local insance
55+
url: 'http://localhost:4000', // local instance
5556
token: '<paste a token from bookkeeping here>',
5657
runTypesRefreshInterval: 15000,
5758
runStatusRefreshInterval: 30000,

‎QualityControl/docker-compose.yml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
version: '3.8'
2-
31
services:
42
database:
53
# Pinned to 11.8 LTS version, community support till 04-07-2028, enterprise: 15-10-2033.

‎QualityControl/docs/Configuration.md‎

Lines changed: 19 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Example:
1414
```javascript
1515
http: {
1616
hostname: 'localhost',
17-
port: 3000
17+
port: 8080
1818
}
1919
```
2020

@@ -83,19 +83,23 @@ qc: {
8383
}
8484
```
8585

86-
### Database Configuration
86+
### Database Object Configuration
8787

8888
The application requires the following database configuration parameters:
89-
| **Field** | **Description** |
90-
|-----------------|---------------------------------------------------------------------------------------------|
91-
| `host` | Hostname or IP address of the database server. |
92-
| `port` | Port number used to connect to the database server. |
93-
| `username` | Username for authenticating with the database. |
94-
| `password` | Password for the specified database user. |
95-
| `database` | Name of the database to connect to. |
96-
| `charset` | Character encoding used for the connection. |
97-
| `collate` | Collation setting used for string comparison and sorting. |
98-
| `timezone` | Time zone used for all date/time values in the database connection. |
99-
| `logging` | Enables or disables SQL query logging (useful for debugging). |
100-
| `retryThrottle` | Time in milliseconds to wait before retrying a failed database connection. |
101-
| `migrationSeed` | *(Optional)* Set to `true` to execute seeders that populate the database with mock data. |
89+
90+
| **Field** | **Type** | **Description** | **Default Value** |
91+
|-----------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
92+
| `host` | `string` | Hostname or IP address of the database server. | `'database'` |
93+
| `port` | `number` | Port number used to connect to the database server. | `3306` |
94+
| `username` | `string` | Username for authenticating with the database. | `'cern'` |
95+
| `password` | `string` | Password for the specified database user. | `'cern'` |
96+
| `database` | `string` | Name of the database to connect to. | `'qcg'` |
97+
| `charset` | `string` | Character encoding used for the connection. | `'utf8mb4'` |
98+
| `collate` | `string` | Collation setting used for string comparison and sorting. | `'utf8mb4_unicode_ci'` |
99+
| `timezone` | `string` | Time zone used for all date/time values in the database connection. | `'+00:00'` |
100+
| `logging` | `boolean` | Enables or disables SQL query logging (useful for debugging). | `false` |
101+
| `retryThrottle` | `number` | Time in milliseconds to wait before retrying a failed database connection. | `5000` |
102+
| `forceSeed` | `boolean` | (used by dev mode only) Force seeding the database with mock data. **Warning:** Not recommended for production use. | `false` |
103+
| `drop` | `boolean` | (used by dev mode only) Force deleting the data from the database when server starts. **Warning:** This will erase all data—never use in production. | `false` |
104+
105+
To know more about the database configuration, please go to: [Database Setup](./Database.md)

‎QualityControl/docs/Database.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
- [Database Setup and Configuration](#database-setup-and-configuration)
2+
- [Development Database Setup](#development-database-setup)
3+
- [Option 1: Docker (Recommended)](#option-1-docker-recommended)
4+
- [Option 2: Local MariaDB Installation](#option-2-local-mariadb-installation)
5+
- [Testing Database Setup](#testing-database-setup)
6+
- [Database Seeding and Configuration](#database-seeding-and-configuration)
7+
- [Development Seeding](#development-seeding)
8+
- [Migrations](#migrations)
9+
- [Current Status](#current-status)
10+
- [Database Management Commands](#database-management-commands)
11+
12+
# Database Setup and Configuration
13+
14+
This document explains how to set up and configure the database for development and testing purposes.
15+
16+
## Development Database Setup
17+
18+
There are two ways to set up a database for development:
19+
20+
### Option 1: Docker (Recommended)
21+
22+
The easiest way to get started is using Docker with the default configuration:
23+
24+
```bash
25+
npm run docker-dev
26+
```
27+
28+
This command will:
29+
- Start the database container using Docker Compose
30+
- Use the default configuration settings
31+
- Set up the database ready for development
32+
33+
### Option 2: Local MariaDB Installation
34+
35+
For a local MariaDB setup, you'll need to install MariaDB on your system first.
36+
37+
**Installation Guide:** [MariaDB Getting Started](https://mariadb.com/get-started-with-mariadb/)
38+
39+
After installing MariaDB locally, configure your application to connect to your local database instance by updating the appropriate configuration files.
40+
41+
## Testing Database Setup
42+
43+
For running tests, Docker must be installed and running on your system.
44+
45+
To set up the test database:
46+
47+
```bash
48+
npm run docker-test
49+
```
50+
51+
This command will:
52+
- Start a test database container
53+
- Automatically seed the database with mock data prepared specifically for testing
54+
- Ensure a clean testing environment
55+
56+
## Database Seeding and Configuration
57+
58+
### Development Seeding
59+
60+
For development purposes, you can seed the database with mock data by setting `forceSeed` to `true` in your configuration.
61+
62+
**Important Notes:**
63+
- When `forceSeed` is enabled, seeding will execute every time the server reloads
64+
- The `drop` property will clean all data from the database if set to `true` every time the server reloads
65+
- Use these options carefully to avoid unintended data loss
66+
67+
### Migrations
68+
69+
Database migrations will be executed automatically if they haven't been run before.
70+
71+
**Clean Setup Recommendation:**
72+
For a completely clean database setup, it's recommended to run:
73+
74+
```bash
75+
docker compose down database -v
76+
```
77+
78+
And delete all existing data.
79+
80+
**WARNING:** Only perform this clean setup the first time or when you specifically need to reset everything, as it will permanently delete all database data.
81+
82+
## Current Status
83+
84+
**Important:** The database is currently not being used by the application or tests, as the services and controllers are not yet configured to utilize the database layer. This functionality is planned for future implementation.
85+
86+
## Database Management Commands
87+
88+
- `npm run docker-dev` - Start development database
89+
- `npm run docker-test` - Start test database with mock data
90+
- `docker compose down database -v` - Clean database setup (removes all data)

‎QualityControl/jsconfig.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,7 @@
99
"public/**/*.js",
1010
"lib/**/*.js",
1111
"test/**/*.js",
12-
"lib/database/migrations/*.mjs" ]
12+
"lib/database/migrations/*.mjs",
13+
"lib/database/seeders/*.mjs"
14+
]
1315
}

‎QualityControl/lib/QCModel.js‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,18 @@ export const setupQcModel = async (eventEmitter) => {
6464
const packageJSON = JSON.parse(readFileSync(`${__dirname}/../package.json`));
6565

6666
const jsonFileService = new JsonFileService(config.dbFile || `${__dirname}/../db.json`);
67-
if (config.database) {
68-
initDatabase(new SequelizeDatabase(config?.database || {}));
67+
68+
const databaseConfig = config.database || {};
69+
if (Object.keys(databaseConfig).length > 0) {
70+
try {
71+
const sequelizeDatabase = new SequelizeDatabase(databaseConfig);
72+
await initDatabase(sequelizeDatabase, { forceSeed: config?.database?.forceSeed, drop: config?.database?.drop });
73+
logger.infoMessage('Database initialized successfully');
74+
} catch (error) {
75+
logger.errorMessage(`Database initialization failed: ${error.message}`);
76+
}
77+
} else {
78+
logger.warnMessage('No database configuration found, skipping database initialization');
6979
}
7080

7181
if (config?.kafka?.enabled) {

‎QualityControl/lib/config/database.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,5 +29,7 @@ export function getDbConfig(config) {
2929
timezone: config.timezone ?? '+00:00',
3030
logging: config.logging ?? false,
3131
retryThrottle: config.retryThrottle ?? 5000,
32+
forceSeed: config.forceSeed ?? false,
33+
drop: config.drop ?? false,
3234
};
3335
};

‎QualityControl/lib/database/SequelizeDatabase.js‎

Lines changed: 44 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ import { fileURLToPath } from 'url';
1919
import { createUmzug } from './umzug.js';
2020
import { getDbConfig } from '../config/database.js';
2121
import models from './models/index.js';
22-
import { SequelizeStorage } from 'umzug';
22+
import { memoryStorage, SequelizeStorage } from 'umzug';
2323

2424
const LOG_FACILITY = `${process.env.npm_config_log_label ?? 'qcg'}/database`;
2525

@@ -29,10 +29,9 @@ const LOG_FACILITY = `${process.env.npm_config_log_label ?? 'qcg'}/database`;
2929
export class SequelizeDatabase {
3030
constructor(config) {
3131
this._logger = LogManager.getLogger(LOG_FACILITY);
32+
const filename = fileURLToPath(import.meta.url);
33+
this._dirname = dirname(filename);
3234

33-
if (!config) {
34-
this._logger.warnMessage('No configuration provided for SequelizeDatabase. Using default configuration.');
35-
}
3635
this._dbConfig = getDbConfig(config);
3736
const {
3837
database,
@@ -99,23 +98,60 @@ export class SequelizeDatabase {
9998
async migrate() {
10099
this._logger.debugMessage('Executing pending migrations...');
101100
try {
102-
const __filename = fileURLToPath(import.meta.url);
103-
const __dirname = dirname(__filename);
104101
const umzug = createUmzug(
105102
this.sequelize,
106-
join(__dirname, 'migrations'),
103+
join(this._dirname, 'migrations'),
107104
new SequelizeStorage({
108105
sequelize: this.sequelize,
109106
}),
110107
);
108+
const pendingMigrations = await umzug.pending();
111109
await umzug.up();
112-
this._logger.infoMessage('Migrations completed successfully.');
110+
this._logger.infoMessage(pendingMigrations.length > 0
111+
? `Executed ${pendingMigrations.length} pending migrations`
112+
: 'No pending migrations to execute');
113113
} catch (error) {
114114
this._logger.errorMessage(`Error executing migrations: ${error}`);
115115
throw error;
116116
}
117117
}
118118

119+
/**
120+
* Executes seed files to populate the database with initial data.
121+
* @returns {Promise<void>}
122+
*/
123+
async seed() {
124+
try {
125+
const umzug = createUmzug(
126+
this.sequelize,
127+
join(this._dirname, 'seeders'),
128+
memoryStorage(),
129+
);
130+
await umzug.up();
131+
this._logger.infoMessage('Seeders executed successfully');
132+
} catch (error) {
133+
this._logger.errorMessage(`Error while executing seeders: ${error}`);
134+
return Promise.reject(error);
135+
}
136+
}
137+
138+
/**
139+
* Drops all tables in the database.
140+
* @returns {Promise<void>}
141+
*/
142+
async dropAllTables() {
143+
this._logger.warnMessage('Dropping all tables!');
144+
145+
try {
146+
await this.sequelize.getQueryInterface().dropAllTables();
147+
} catch (error) {
148+
this._logger.errorMessage(`Error while dropping all tables: ${error}`);
149+
return Promise.reject(error);
150+
}
151+
152+
this._logger.infoMessage('Dropped all tables!');
153+
}
154+
119155
/**
120156
* Gets the models.
121157
* @returns {object} The models.

‎QualityControl/lib/database/index.js‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,19 +13,36 @@
1313
*/
1414

1515
import { LogManager } from '@aliceo2/web-ui';
16+
import { isRunningInDevelopment, isRunningInProduction, isRunningInTest } from '../utils/environment.js';
1617

1718
const LOG_FACILITY = `${process.env.npm_config_log_label ?? 'qcg'}/database`;
1819

1920
/**
2021
* Initializes the database connection and runs migrations.
2122
* @param {object} sequelizeDatabase - The Sequelize database instance.
23+
* @param {object} options - Options for database initialization.
24+
* @param {boolean} [options.forceSeed=false] - Whether to force seeding the database.
25+
* @param {boolean} [options.drop=false] - Whether to drop existing tables before migration (development only).
2226
* @returns {Promise<void>} A promise that resolves when the database is initialized.
2327
*/
24-
export const initDatabase = async (sequelizeDatabase) => {
28+
export const initDatabase = async (sequelizeDatabase, { forceSeed = false, drop = false }) => {
2529
const _logger = LogManager.getLogger(LOG_FACILITY);
2630
try {
27-
await sequelizeDatabase.connect();
28-
await sequelizeDatabase.migrate();
31+
if (isRunningInTest) {
32+
await sequelizeDatabase.dropAllTables();
33+
await sequelizeDatabase.migrate();
34+
await sequelizeDatabase.seed();
35+
} else if (isRunningInDevelopment) {
36+
if (drop) {
37+
await sequelizeDatabase.dropAllTables();
38+
}
39+
await sequelizeDatabase.migrate();
40+
if (forceSeed) {
41+
await sequelizeDatabase.seed();
42+
}
43+
} else if (isRunningInProduction) {
44+
await sequelizeDatabase.migrate();
45+
}
2946
} catch (error) {
3047
_logger.errorMessage(`Failed to initialize database: ${error.message}`);
3148
}

‎QualityControl/lib/database/migrations/20250424083717-create-tables.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ export const up = async (queryInterface, Sequelize) => {
150150
},
151151
object_name: {
152152
type: Sequelize.STRING(255),
153-
allowNull: true,
153+
allowNull: false,
154154
},
155155
ignore_defaults: {
156156
type: Sequelize.BOOLEAN,

0 commit comments

Comments
 (0)