K8s部署mysql案例，学习资源类型（nfs、pv、pvc、configmap、secret）

一、部署背景

使用k8s-1.20版本，通过部署mysql应用，对nfs存储和pv、pvc进行配置，通过网络存储达到数据持久化存储的目的，配置configmap资源用来挂载mysql的配置文件，灵活管理配置，使用secret资源管理mysqlroot用户密码，防止在资源配置文件中泄露关键密钥等信息，在此过程中对以上k8s的资源类型进行初步的配置和了解，掌握基本的使用方法。

1.1 相关概念

NFS(Network File System)即网络文件系统

NFS允许一个系统在网络上与它人共享目录和文件。通过使用NFS，用户和程序可以像访问本地文件一样访问远端系统上的文件。

nfs可以挂载已经存在的共享的Pod中，和emptyDir不同的是，当Pod被删除的时候emptyDir会被删除，但是nfs不会被删除，仅仅是解除挂载状态而已，这就意味着NFS能够允许提前对数据进行处理，而且这些数据可以在Pod之间相互传递，并且，nfs可以同时被多个pod挂载并进行读写。

Secret

Secret解决了密码、token、密钥等敏感数据的配置问题，而不需要把这些敏感数据暴露到镜像或者Pod Spec中。Secret可以以Volume或者环境变量的方式使用。

Secret有三种类型：

Service Account：用来访问Kubernetes API，由Kubernetes自动创建，并且会自动挂载到Pod的/run/secrets/kubernetes.io/serviceaccount目录中；
Opaque：base64编码格式的Secret，用来存储密码、密钥等；
kubernetes.io/dockerconfigjson：用来存储私有docker registry的认证信息。
ConfigMap

ConfigMap用于保存配置数据的键值对，可以用来保存单个属性，也可以用来保存配置文件。ConfigMap跟secret很类似，但它可以更方便地处理不包含敏感信息的字符串。

PersistentVolume（PV）

PersistentVolume（PV）是集群中已由管理员配置的一段网络存储。集群中的资源就像一个节点是一个集群资源。 PV是诸如卷之类的卷插件，但是具有独立于使用PV的任何单个pod的生命周期。该API对象捕获存储的实现细节，即NFS，iSCSI或云提供商特定的存储系统。

更多介绍可查看官方文档持久卷|Kubernetes

PersistentVolumeClaim（PVC）

PersistentVolumeClaim（PVC）是用户存储的请求。它类似于pod。Pod消耗节点资源，PVC消耗存储资源。 pod可以请求特定级别的资源（CPU和内存）。权限要求可以请求特定的大小和访问模式。

二、部署应用

2.1 搭建nfs服务

安装相关组件

yum -y install rpcbind nfs-utils

编辑/etc/exports，并启动nfs

[root@ecs-1213 mysql]# cat /etc/exports
/data/service_data      192.168.0.0/16(rw,sync,no_root_squash,no_all_squash) 10.0.0.0/8(rw,sync,no_root_squash,no_all_squash)

第一列为共享的目录，第二列是可访问的ip地址段和相关权限配置

rw：read-write，可读写的权限

sync：数据同步写入到内存与硬盘当中；

no_root_squash：具有 root 的权限

no_all_squash: 不做来宾账户映射

systemctl start rpcbind
systemctl enable rpcbind
systemctl start nfs
systemctl enable nfs

[root@ecs-1213 mysql]# exportfs -arv         #不用重启nfs服务，配置文件就会生效
exporting 192.168.0.0/16:/data/service_data
exporting 10.0.0.0/8:/data/service_data

2.2 编辑pv资源文件

[root@ecs-1213 mysql]# cat pv.yaml
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-data-pv
  labels:
    name: mysql-data
spec:
  accessModes:
    - ReadWriteOnce                          #单node的读写
  capacity:
    storage: 5Gi                             #分配的空间大小
  persistentVolumeReclaimPolicy: Retain      #回收策略是手工回收
  storageClassName: nfs                      #存储类型是nfs
  nfs:
    path: /data/service_data/mysql2/data     #共享的路径
    server: 192.168.0.191                    #nfs服务器的地址

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-log-pv
  labels:
    name: mysql-log
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /data/service_data/mysql2/log
    server: 192.168.0.191

2.2 编辑pvc资源文件

[root@ecs-1213 mysql]# cat pvc.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-data-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: nfs
  selector:
    matchLabels:
      name: mysql-data
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-log-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
  selector:
    matchLabels:
      name: mysql-log

按照以上配置，已经分别创建了mysql-data-pv大小为5G，匹配mysql-data-pvc，mysql-log-pv大小为1G，匹配mysql-log-pvc,通过标签精确绑定

分别用来持久化存储mysql的数据目录和binlog目录，下面对mysql的相关配置资源进行编辑。

2.3 编辑secret资源存储mysql密码

[root@ecs-1213 mysql]# cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
  namespace: default
  labels:
    app: mysql
type: Opaque
data:
  password: MTIzNDU2 # echo -n '123456' |base64

type类型为Opaque 是base64编码格式，可以通过注释的命令获取到编码后的密码，打上labels标签会和应用进行匹配

2.4 编辑configmap资源挂载mysql配置

[root@ecs-1213 mysql]# cat config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
  namespace: default
data:
  mysqld.cnf: |-
    [mysqld_safe]
    socket          = /var/run/mysqld/mysqld.sock
    nice            = 0
    [mysqld]
    user            = mysql
    pid-file        = /var/run/mysqld/mysqld.pid
    socket          = /var/run/mysqld/mysqld.sock
    port            = 3306
    basedir         = /usr
    datadir         = /var/lib/mysql
    tmpdir          = /tmp
    lc-messages-dir = /usr/share/mysql
    skip-external-locking
    key_buffer_size         = 16M
    max_allowed_packet      = 16M
    thread_stack            = 192K
    thread_cache_size       = 8
    myisam-recover-options  = BACKUP
    query_cache_limit       = 1M
    query_cache_size        = 16M
    server-id               = 191
    log_bin                 = /var/log/mysql/master-bin.log
    relay-log               = slave-relay-bin
    expire_logs_days        = 30
    replicate_ignore_db     = information_schema,performation_schema,sys,mysql,metabase
    character_set_server=utf8mb4
    collation-server=utf8mb4_unicode_ci
    sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
    [mysql]
    default-character-set=utf8mb4

注意写法，data下是配置文件名称，然后加 |- 表示保留块中已有的回车换行，并对特殊字符不进行转义

2.5 编辑mysql应用和服务资源文件

[root@ecs-1213 mysql]# cat mysql-deploy.yaml
---
kind: Service
apiVersion: v1
metadata:
  name: mysql
  namespace: default
spec:
  type: NodePort
  selector:
    app: mysql
  ports:
  - name: http2
    port: 3306
    nodePort: 31000
    targetPort: 3306

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.7.32
        imagePullPolicy: Never
        env:
        - name: "MYSQL_ROOT_PASSWORD"
          valueFrom:                                     #引用secret的key,就是mysql的root密码
            secretKeyRef:
              name: mysql-secret
              key: password
        - name: "TZ"
          value: "Asiz/Shanghai"
        ports:
        - containerPort: 3306
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mysql-data
        - mountPath: /etc/mysql/mysql.conf.d/
          name: mysql-conf
        - mountPath: /var/log/mysql
          name: mysql-binlog
      volumes:
      - name: mysql-data
        persistentVolumeClaim:                   #挂载对应的pvc名称
          claimName: mysql-data-pvc
      - name: mysql-conf
        configMap:                               #挂载对应的configmap配置名称
          name: mysql-config
      - name: mysql-binlog
        persistentVolumeClaim:                   #挂载对应的pvc名称
          claimName: mysql-log-pvc

三、部署测试

以上资源配置文件准备完毕后就可以应用配置进行测试了

3.1 部署secret

[root@ecs-1213 mysql]# kubectl apply -f secret.yaml
secret/mysql-secret created
[root@ecs-1213 mysql]# kubectl get secret
NAME                                     TYPE                                  DATA   AGE
default-token-gqq4x                      kubernetes.io/service-account-token   3      116d
mysql-secret                             Opaque                                1      12s
sh.helm.release.v1.redis-1649381748.v1   helm.sh/release.v1                    1      5h2m

3.2 部署configmap

[root@ecs-1213 mysql]# kubectl apply -f config.yaml
configmap/mysql-config created
[root@ecs-1213 mysql]# kubectl get configmap
NAME                             DATA   AGE
kube-root-ca.crt                 1      116d
mysql-config                     1      11s
[root@ecs-1213 mysql]# kubectl describe configmap mysql-config
Name:         mysql-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
mysqld.cnf:
----
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0
[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
key_buffer_size         = 16M
max_allowed_packet      = 16M
thread_stack            = 192K
thread_cache_size       = 8
myisam-recover-options  = BACKUP
query_cache_limit       = 1M
query_cache_size        = 16M
server-id               = 191
log_bin                 = /var/log/mysql/master-bin.log
relay-log               = slave-relay-bin
expire_logs_days        = 30
replicate_ignore_db     = information_schema,performation_schema,sys,mysql,metabase
character_set_server=utf8mb4
collation-server=utf8mb4_unicode_ci
sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
[mysql]
default-character-set=utf8mb4
Events:  <none>

3.3 部署pv、pvc

[root@ecs-1213 mysql]# kubectl apply -f pv.yaml
persistentvolume/mysql-data-pv created
persistentvolume/mysql-log-pv created
[root@ecs-1213 mysql]# kubectl apply -f pvc.yaml
persistentvolumeclaim/mysql-data-pvc created
persistentvolumeclaim/mysql-log-pvc created
[root@ecs-1213 mysql]# kubectl get pv
NAME            CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
mysql-data-pv   5Gi        RWO            Retain           Bound    default/mysql-data-pvc   nfs                     8s
mysql-log-pv    1Gi        RWO            Retain           Bound    default/mysql-log-pvc    nfs                     8s

3.4 部署mysql应用和服务

[root@ecs-1213 mysql]# kubectl apply -f mysql-deploy.yaml
service/mysql created
deployment.apps/mysql created
[root@ecs-1213 mysql]# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
dns-example              1/1     Running   4          44d
mysql-575ffc99b7-qqk25   1/1     Running   0          5s

3.5 进入mysql命令行，测试密码和配置是否生效

[root@ecs-1213 mysql]# kubectl exec -it mysql-575ffc99b7-qqk25 -- bash
root@mysql-575ffc99b7-qqk25:/# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.32-0ubuntu0.18.04.1-log (Ubuntu)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin       | ON    |
+---------------+-------+
1 row in set (0.01 sec)

mysql> show variables like 'expire_logs_days';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| expire_logs_days | 30    |
+------------------+-------+
1 row in set (0.00 sec)

mysql>

可以发现能够用secret存储的密码进入mysql，并且configmap中配置了mysql的binlog和expire_logs_days 参数都已经生效。

3.6 查看存储目录是否存在mysql相关数据

[root@ecs-1213 mysql]# ll /data/service_data/mysql2/data/
total 188476
-rw-r----- 1 polkitd input       56 Apr  8 09:17 auto.cnf
-rw------- 1 polkitd input     1676 Apr  8 09:17 ca-key.pem
-rw-r--r-- 1 polkitd input     1112 Apr  8 09:17 ca.pem
-rw-r--r-- 1 polkitd input     1112 Apr  8 09:17 client-cert.pem
-rw------- 1 polkitd input     1680 Apr  8 09:17 client-key.pem
-rw-r----- 1 polkitd input      460 Apr  8 15:27 ib_buffer_pool
-rw-r----- 1 polkitd input 79691776 Apr  8 15:27 ibdata1
-rw-r----- 1 polkitd input 50331648 Apr  8 15:27 ib_logfile0
-rw-r----- 1 polkitd input 50331648 Apr  8 09:17 ib_logfile1
-rw-r----- 1 polkitd input 12582912 Apr  8 15:27 ibtmp1
drwxr-x--- 2 polkitd input     4096 Apr  8 09:17 mysql
drwxr-x--- 2 polkitd input     4096 Apr  8 09:17 performance_schema
-rw------- 1 polkitd input     1680 Apr  8 09:17 private_key.pem
-rw-r--r-- 1 polkitd input      452 Apr  8 09:17 public_key.pem
-rw-r--r-- 1 polkitd input     1112 Apr  8 09:17 server-cert.pem
-rw------- 1 polkitd input     1680 Apr  8 09:17 server-key.pem
drwxr-x--- 2 polkitd input    12288 Apr  8 09:17 sys
[root@ecs-1213 mysql]# ll /data/service_data/mysql2/log/
total 3012
-rw-r----- 1 polkitd input     177 Apr  8 09:17 master-bin.000001
-rw-r----- 1 polkitd input 3063627 Apr  8 09:17 master-bin.000002
-rw-r----- 1 polkitd input     177 Apr  8 14:36 master-bin.000003
-rw-r----- 1 polkitd input     177 Apr  8 15:27 master-bin.000004
-rw-r----- 1 polkitd input     154 Apr  8 15:27 master-bin.000005
-rw-r----- 1 polkitd input     165 Apr  8 15:27 master-bin.index
[root@ecs-1213 mysql]#

数据通过nfs服务已实现持久化存储。