Start Locally from Source Code
Tutorial Video
Important
If you encounter environment configuration or task execution exceptions during deployment, we recommend watching the official qData Data Platform troubleshooting video collection.
This video series explains common troubleshooting methods and covers typical scenarios such as task failure, unable to add or execute tasks, and no task instances, helping you quickly locate and resolve issues.
This document explains how to start qData locally using the source code method. Follow the steps below to prepare the environment, configure services, and start the application.
1. Prerequisites
1.1. Install Docker and Docker Compose
Before installing qData, make sure your device meets the following minimum requirements. We recommend installing Docker-related components on a Linux server or virtual machine:
- CPU: >= 4 cores
- Memory (RAM): >= 8 GiB
⚠️ Note: Make sure Docker and Docker Compose are correctly installed and can be executed normally in the terminal.
2. Version Requirements
| Component | Version Requirement |
|---|---|
| JDK | 1.8 |
| Maven | 3.6+ |
| yarn | v1.22.22+ |
| Node.js | 18.0+ |
| Python | 3.x |
| Docker | 19.03 or later |
| Docker Compose | 1.28 or later |
3. Clone the qData Repository
git clone https://gitee.com/qiantongtech/qData.git4. Prepare DataX
4.1. Install Python 3
DataX is launched with Python 3. Install Python 3 and verify that the command for your operating system runs successfully:
python3 --versionpython --versionOn Windows, also change python-command to python in application-dev.yml.
:::
4.2. Prepare the DataX Directory
🔗 Baidu Netdisk: Download DataX source code
🔑 Extraction code: Join the QQ discussion group and find the code in the group files. Join the QQ discussion group.
Build notes
The download contains DataX source code, which must be built before deployment. The directory tree below shows the deployment structure after building.
For details, see the “打包发布” (Packaging and Release) section in dataxPluginDev.md in the DataX source root.
After building, copy the complete <DataX_source_root>/target/datax/datax/ directory into the qData project root, keeping the directory name datax.
Deployment directory structure:
qData/
├── datax/
│ ├── bin/
│ │ └── datax.py
│ └── lib/
├── qdata-server/
└── ...⚠️ Note: Keep the directory name as
dataxand verify thatdatax/bin/datax.pyanddatax/libexist.
4.3. Configure DataX
Check the DataX settings in qdata-server/src/main/resources/application-dev.yml:
datax:
home: ${user.dir}/datax/
python-command: python3
datax-py-path: ${user.dir}/datax/bin/datax.py
lib-dir: ${user.dir}/datax/lib
job-dir: ${user.dir}/upload/datax/jobOn Windows, set python-command to python. ${user.dir} is the qData working directory. When starting qData in IDEA, set the working directory to the qData project root.
5. Start Middleware with Docker Compose
The qData backend depends on the following middleware:
- Databases: DM8 / MySQL, PostgreSQL
- Cache and message queue: Redis, RabbitMQ
- Files and storage: MongoDB, Hadoop, kkFileView
- Scheduling: DolphinScheduler
Start middleware through Docker Compose:
# Local source-code mode in the Docker Compose deployment document
docker-compose --profile local up -d👉 See Docker Compose deployment documentation
6. Start the Backend
6.1. Environment Preparation
- JDK = 1.8
- Maven >= 3.6
6.2. Start the Project in IDEA
Open the project with IntelliJ IDEA.
6.3. Build the Project
mvn clean -DskipTests=true install6.4. Required Configuration
All of the following configuration items must be modified in the application-dev.yml file.
6.4.1. qdata-server Configuration
- Modify the qData service Redis connection information by editing
qdata-server/src/main/resources/application-dev.yml.
spring:
redis:
host: IP
port: PORT
database: 0
password: PASSWORD- Modify RabbitMQ connection information by editing
qdata-server/src/main/resources/application-dev.yml.
spring:
rabbitmq:
host: IP
port: PORT
username: ACCOUNT
password: PASSWORD- Modify database connection information by editing
qdata-server/src/main/resources/application-dev.yml.
# Primary data source selection
datasource:
type: mysql # mysql or dm8. Fill in the selected type and modify the corresponding settings below.
# MySQL configuration
mysql:
# JDBC driver class name
driver-class-name: com.mysql.cj.jdbc.Driver
# Primary database JDBC connection URL
url: jdbc:mysql://IP:PORT/qdata?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8
# Primary database username
username: ACCOUNT
# Primary database password
password: PASSWORD
# DM configuration
dm8:
# JDBC driver class name
driver-class-name: dm.jdbc.driver.DmDriver
# Primary database JDBC connection URL
url: jdbc:dm://IP:PORT/QDATA?STU&zeroDateTimeBehavior=convertToNull&useUnicode=true&characterEncoding=utf-8&schema=QDATA&serverTimezone=Asia/Shanghai
# Primary database username
username: ACCOUNT
# Primary database password
password: PASSWORD- Modify scheduler-related configuration by editing
qdata-server/src/main/resources/application-dev.yml.
# Scheduler-related configuration
ds:
# Scheduler API prefix
base_url: http://IP:12345/dolphinscheduler
# Data quality service component HTTP callback address
quality_url: http://127.0.0.1:8083/quality/qualityTaskExecutor/runExecuteTask
# DolphinScheduler project code for data quality tasks
http_quality_projectCode: 134799536571008
# Scheduler token, which can be changed as needed
token: b07f0e57c4818043a57ba05a28da291a
# Spark-related configuration
spark:
# Spark master address
master_url: spark://spark:7077
# JAR path in DolphinScheduler
main_jar: file:/dolphinscheduler/default/resources/spark-jar/qdata-etl-3.8.8.jar
# JAR main entry class
main_class: tech.qiantong.qdata.spark.etl.EtlApplication
# HDFS-related configuration
hdfs:
# NameNode RPC address, the entry address for client connections to HDFS
url: hdfs://namenode:8020
# Scheduler Redis configuration, used to obtain the latest data source information and ensure task execution
redis:
host: redis
port: 6379
database: 0
password: PASSWORD- Modify data quality service configuration by editing
qdata-server/src/main/resources/application-dev.yml.
path:
# Data quality service prefix
quality_url: http://127.0.0.1:8083/quality/qualityTaskExecutor6.4.2. qdata-quality Configuration
- Modify RabbitMQ connection information by editing
qdata-quality/src/main/resources/application-dev.yml.
spring:
rabbitmq:
host: IP
port: PORT
username: ACCOUNT
password: PASSWORD- Modify MongoDB connection information by editing
qdata-quality/src/main/resources/application-dev.yml.
spring:
data:
mongodb:
# Connection address
uri: mongodb://ACCOUNT:PASSWORD@IP:PORT/data?authSource=admin
field-naming-strategy: org.springframework.data.mapping.model.SnakeCaseFieldNamingStrategy
print: true
slowQuery: true
slowTime: 1000- Modify database connection information by editing
qdata-quality/src/main/resources/application-dev.yml.
# Primary data source selection
datasource:
type: mysql # mysql or dm8. Fill in the selected type and modify the corresponding settings below.
# MySQL configuration
mysql:
# JDBC driver class name
driver-class-name: com.mysql.cj.jdbc.Driver
# Primary database JDBC connection URL
url: jdbc:mysql://IP:PORT/qdata?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8
# Primary database username
username: ACCOUNT
# Primary database password
password: PASSWORD
# DM configuration
dm8:
# JDBC driver class name
driver-class-name: dm.jdbc.driver.DmDriver
# Primary database JDBC connection URL
url: jdbc:dm://IP:PORT/QDATA?STU&zeroDateTimeBehavior=convertToNull&useUnicode=true&characterEncoding=utf-8&schema=QDATA&serverTimezone=Asia/Shanghai
# Primary database username
username: ACCOUNT
# Primary database password
password: PASSWORD- Modify RabbitMQ connection information by editing
qdata-quality/src/main/resources/application-dev.yml.
spring:
rabbitmq:
host: IP
port: PORT
username: ACCOUNT
password: PASSWORD6.4.3. Start the qdata-server Service
Run qdata-server/src/main/java/tech/qiantong/qdata/server/QDataApplication.java
6.4.4. Start the qdata-quality Service
Run qdata-quality/src/main/java/tech/qiantong/qdata/quality/QualityApplication.java
7. Start the Frontend
7.1. Environment Preparation
- Node.js >= 18.0
- yarn >= v1.22.22
7.2. Start the Frontend Service
# Enter the frontend directory
cd qdata-ui
# Install dependencies
yarn install
# Start
yarn run dev7.3. Reset the admin Password
# Execute the following SQL in the database
UPDATE SYSTEM_USER
SET PASSWORD='$2a$10$3dzj0nGSmlNjPw4iuZNHse5.lxrNzumTCQ8aEaV7nNEtn2pXmmimW'
WHERE USER_ID=1;7.4. Access qData
Open http://localhost:81 in your browser.
Default account/password: admin/qData123
If the login page displays correctly, login succeeds, and menus and pages display normally, the environment has been set up successfully.
