MySQL服务搭建(通用linux系统安装)

环境: Centos7.2 一：获取mysq包下载mysql https://dev.mysql.com/downloads/mysql/

二：Install MySQL Server

1 解压xxx.tar.gz。

2 进入解压后的目录

3 添加组和用户 #groupadd mysql #useradd -r -g mysql mysql -r禁止用户mysql进行登录操作。

4 改变目录拥有者 chown -R mysql:mysql ./

5 新建数据目录 mkdir data

6 /etc/profile添加PATH路径

7 修改配置文件

cp $MYSQL_HOME/support-files/mysql.server /etc/init.d/mysql 修改/etc/init.d/mysql中的datadir和basedir

vim /etc/my.cnf

[mysql]
default-character-set=utf8

[mysqld]
port=8282
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

server-id=5

log-bin=mysql-bin
character_set_server=utf8


binlog-ignore-db=information_schema
binlog-ignore-db=mysql

log-slave-updates
slave-skip-errors=all
slave-net-timeout=60

symbolic-links=0
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

8 初始化mysqld #mysqld –initialize –user=mysql –datadir=xx –basedir=xx 这一步会创建用户root，并且生成登录密码，需要记住密码。

可能会出现的问题：

1 mysqld: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

A yum install libaio

2 mysqld: error while loading shared libraries: libnuma.so.1: cannot open shared object file: No such file or directory

A yum install numactl.x86_64

3 Can’t open the mysql.plugin table. Please run mysql_upgrade to create it.

A mysql_install_db –user=mysql –basedir=/soft/mysql-5.7.21-linux-glibc2.12-x86_64 –datadir=/soft/mysql-5.7.21-linux-glibc2.12-x86_64/data

9 启动mysql

mysqld_safe –user=mysql

10 登录mysql

mysql -u root -p 这一步输入前面步骤中生成的密码。

11 进入mysql shell修改密码 mysql»set password=password(‘1234’);

12 加入开机启动

chkconfig --add mysql

chkconfig mysql on

MySQL服务搭建(RPM包安装)

这个主要是针对hive的安装过程。

一：需要的rpm包:

mysql-community-common-5.7.21-1.el7.x86_64

mysql-community-libs-5.7.21-1.el7.x86_64

mysql-community-client-5.7.21-1.el7.x86_64

mysql-community-server-5.7.21-1.el7.x86_64

二：安装

将上面的包上传到服务器，注意上传过程中请勿使用ASCLL上传，否则安装报错。

依次安装上面4个rpm包

rpm -ivh xxxx

安装过程可能会出现mariadb-lib冲突（会产生冲突），因为集群对这个包有依赖，所以这里不要卸载。

直接使用

rpm -ivh mysql-community-client-5.7.21-1.el7.x86_64 --nodeps --force
rpm -ivh  mysql-community-server-5.7.21-1.el7.x86_64  --nodeps  --force

然后在 /etc/my.cnf中配置用户不授权（否则Hive的Metastore启动会报错）。

在[mysqld]下面添加

skip-grant-tables

mysql问题排查

1 mysql重启失败

A1: 检查SELINUX是否关闭，setenforce 0 (立即生效)，修改/etc/selinux/config。

Mysql主从配置

实例环境 Centos 7.2

Mysql 5.7.21

主机A：192.168.24.172（Mysql-Master）

主机B：10.251.44.84（Mysql-Slave）

复制现有数据

同步之前需要拷贝现有数据，为了保证数据的完整性和正确性，拷贝的同时要禁止数据的写入,这里使用 mysqldump 复制数据.

在Master Mysql中

mysqldump -uroot -p1234 xxx(database)>xxx.sql

将sql文件复制到Slave，root登陆Slave Mysql

先建立相关库，年后运行sql文件

mysql>create database xxx;
mysql>source xxx.sql

配置Master

1 在 /etc/my.cnf中添加：

log-bin=mysql-bin
id一定要唯一
server-id=2
binlog-ignore-db=information_schema
binlog-ignore-db=cluster
binlog-ignore-db=mysql
#如果不配置则同步会针对所有库
binlog-do-db=ufind_db

2 重启Master Mysql

service mysqld restart

3 用root登陆Master Mysql，授予同步用户的 file 和 replication slave 权限。

mysql>set global validate_password_policy=0;
mysql>GRANT FILE ON *.* TO 'root'@'%' IDENTIFIED BY 'wss_*0817';
mysql>GRANT REPLICATION SLAVE ON *.* TO 'root'@'%' IDENTIFIED BY 'wss_*0817';
mysql>FLUSH PRIVILEGES;

4 检查Master status

mysql>mysql> show master status;
mysql> show master status;
+------------------+----------+--------------+----------------------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB                 | Executed_Gtid_Set |
+------------------+----------+--------------+----------------------------------+-------------------+
| mysql-bin.000004 |    28125 | ufind_db     | information_schema,cluster,mysql |                   |
+------------------+----------+--------------+----------------------------------+-------------------+
1 row in set (0.00 sec)

mysql>

如果结果是 empty set则需要检查/etc/my.cnf。

配置Slave

1 vim /etc/my.cnf

log-bin=mysql-bin
server-id=3
log-slave-updates
slave-skip-errors=all
slave-net-timeout=60

binlog-ignore-db=information_schema
binlog-ignore-db=cluster
binlog-ignore-db=mysql

2 root登陆Slave控制台，连接Master

mysql>stop slave;
mysql>change master to master_host='192.168.24.172',master_user='wss',master_password='wss',master_port=3307,master_log_file='mysql-bin.000004',master_log_pos=36399068;
mysql>start slave;
mysql>show slave status \G;
mysql>省略结果

以上查看Slave status结果中主要查看

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

2项是否为yes，如果是则同步正常。

ntpd服务搭建

没有外网环境，可以使用本地centos的ios yum源。

yum install ntp

修改ntp的配置文件（红色表示需要修改的地方） vim /etc/ntp.conf

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).

#不使用外网服务
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

#开启本地时间服务
server 127.127.1.0
fudge  127.127.1.0 stratum 10

#以下ip换成自己时间服务器的ip
restrict 192.168.72.22 mask 255.255.255.0 nomodify notrap

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor

配置文件修改好之后，启动ntp服务 service ntp start chkconfig ntp on

ntpd服务测试

在另外的机器上（一般是集群中另外的机器）测试是否可以与ntp服务端的机器同步

ntpdate $hostname

测试成功后需要在/etc/crontab中添加自动时间同步

echo “ */10 * * * * root ntpdate $hostname”»/etc/crontab

Azkaban服务搭建

一：预安装

1）：前提。安装JDK，安装Hadoop，Hive客户端环境

2）：前提。安装 git 的命令：apt-get install git

3）：把github的azkaban源码下载

git clone https://github.com/azkaban/azkaban.git

4）进入azkaban，然后命令

vim azkaban-common/src/main/java/azkaban/jobExecutor/ProcessJob.java 将final boolean isExecuteAsUser = this.sysProps.getBoolean(EXECUTE_AS_USER,true);true修改false（修改它之后，预防后面安装出错）

进入azkaban路径，编译源码

./gradlew

编译完成会生成5个xxx.tar.gz包。

二：服务搭建和配置

1 Web服务配置

进入目录

cd  /soft/azkaban/azkaban-tar/azkaban-web-server-3.41.0-9-geae1f9d/conf

修改文件azkaban.properties（以下是一份配置文件的样例）

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/soft/azkaban/azkaban-tar/azkaban-web-server-3.41.0-9-geae1f9d/web/
default.timezone.id=Asia/Shanghai

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/soft/azkaban/azkaban-tar/azkaban-web-server-3.41.0-9-geae1f9d/conf/azkaban-users.xml

# Loader for projects
#executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=192.168.25.97
mysql.database=azkaban
mysql.user=wss
mysql.password=wss
mysql.numconnections=100

# Velocity dev mode
velocity.dev.mode=false

# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12322
executor.flow.threads=30

# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
azkaban.native.lib=true
#Multiple Executor Mode
#open it if you use the multiple executor mode
azkaban.use.multiple.executors=true
#azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=2
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

2 Executor服务配置进入目录

cd  /soft/azkaban/azkaban-tar/azkaban-exec-server-3.41.0-9-geae1f9d/conf

修改配置文件 vim azkaban.properties（以下是一份样例配置文件）

# Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai

# Loader for projects
#executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=192.168.25.97
mysql.database=azkaban
mysql.user=wss
mysql.password=wss
mysql.numconnections=100


# Azkaban Executor settings
executor.maxThreads=50
executor.port=12322
executor.flow.threads=30

# JMX stats
jetty.connector.stats=true
executor.connector.stats=true

# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/soft/azkaban/azkaban-tar/azkaban-exec-server-3.41.0-9-geae1f9d/plugins/jobtypes

azkaban.native.lib=false
execute.as.user=false

导入Azkaban表结构到mysql

Mysql>
source  /soft/azkaban/azkaban-tar/azkaban-db-3.41.0-9-geae1f9d/xxx-all-xxx.sql

导入所有表结构之后需要进入mysql将表executor中的activate的值全部修改为1

3 启动executor：

一定要先启动启动executor：,然后再启动web

cd  /soft/azkaban/azkaban-tar/azkaban-exec-server-3.41.0-9-geae1f9d/bin
nohup azkaban-executor-start.sh

4 启动web：

cd  /soft/azkaban/azkaban-tar/azkaban-web-server-3.41.0-9-geae1f9d/bin
nohup azkaban-web-start.sh &

MongoDB 搭建

1 利用yum安装，repo配置如下：

[mongodb-org-3.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/
gpgcheck=0
enabled=1
#gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc

Haproxy配置端口映射

一般在需要配置端口映射场景下，除了nginx的各大模块，利用Haproxy也是一个不错的选择。

安装haproxy

yum install haproxy

使用配置

以下配置是监听本机的21端口，然后将请求交给9019端口的服务处理，这里处理的是TCP协议（实际上这里就是转发FTP服务的功能）。

vim /etc/haproxy/haproxy.cfg

listen ftp21_9019
bind 0.0.0.0:21
mode tcp
balance roundrobin
server ftp9019 192.168.73.101:9019

启动

haproxy -f /etc/haproxy/haproxy.cfg  -D

hdfs-ftp服务搭建

hdfs-ftp 一个将FTP功能使用在hdfs上的工具，工具包下载密码：s5np

1 解压工具包后直接配置使用即可。

2 配置

  cd   ~/hdfs-ftp/conf/

  #hdfs/Kerberos配置
  将hdfs的 hdfs-site.xml  core-site.xml   yarn-site.xml配置文件拷贝过来。如果hdfs没有启用Kerberos，则将  platform-site.xml 删除，如果启用了 kerberos，在  platform-site.xml 中添加对应的kerberos配置。

   #用户配置
   在文件 users.properties 中添加对应的用户和密码即可，密码使用的是6位的AMD5加密。修改配置文件 hdfs-over-ftp.properties，修改所需端口和数据端口。

   #应用端口
   port = 9019
   #数据端口范围（每次会在此范围选择一个端口进行数据的）
   data-ports = 8600-9599

   #uncomment this to run ssl ftp server（这个需要注释掉）
   #ssl-port = 2226
   #ssl-data-ports = 2227-2229

下面是一份可用的配置样例

   #uncomment this to run ftp server
   port = 9019
   data-ports = 8600-9599

   #uncomment this to run ssl ftp server
   #ssl-port = 2226
   #ssl-data-ports = 2227-2229

   # not hava to be set
   # The maximum number of simultaneous users
   maxLogins = 1000

   # not hava to be set
   # Are anonymous logins enabled?
   anonymousLoginEnabled = true

   # not hava to be set
   # this allows you to set The maximum number of simultaneous anonymous users
   maxAnonymousLogins = 10

   # not hava to be set
   # this allows you to set The number of failed login attempts before the connection is closed
   maxLoginFailures = 3

   # not hava to be set
   # this allows you to set The number of milliseconds that the connection is delayed after a failed login attempt.
   # Used to limit to possibility of brute force guessing passwords.
   loginFailureDelay = 100

   # not hava to be set
   # this allows you to set The maximum number of threads used
   # in the thread pool for handling client connections
   maxThreads = 1000

   # hdfs uri
   hdfs-uri = hdfs://aep/

   superuser=aep_bill

需要将 hdfs-over-ftp.properties users.properties 同样拷贝到 ~/hdfs-ftp/conf/conf下面才能生效。

3 启动hdfs-ftp

bin目录下面后台启动

cd  ~/hdfs-ftp/bin
nohup ./dfs.sh hdfs-ftp&

Linux服务搭建