通过 Helm 搭建 Docker 镜像仓库 Harbor
参数地址:
系统环境:
- kubernetes 版本:1.20.1
- Traefik Ingress 版本:2.4.3
- Harbor Chart 版本:1.5.2
- Harbor 版本:2.1.3
- Helm 版本:3.2.1
- 持久化存储驱动:NFS
一、Harbor 简介
1、简介
Harbor 是一个开放源代码容器镜像注册表,可通过基于角色权限的访问控制来管理镜像,还能扫描镜像中的漏洞并将映像签名为受信任。Harbor 是 CNCF 孵化项目,可提供合规性,性能和互操作性,以帮助跨 Kubernetes 和 Docker 等云原生计算平台持续,安全地管理镜像。
2、特性
- 管理:多租户、可扩展
- 安全:安全和漏洞分析、内容签名与验证
二、准备环境
1、安装 Helm
关于如何安装 Helm 3,请查看之前的博文 安装 Helm3 管理 Kubernetes 应用 进行安装。
2、创建 Namespace
由于 Harbor 组件较多,一般我们会采取新建一个 Namespace 专用于部署 Harbor 相关组件,输入下面命令创建名为 mydlq-hub 的命名空间。
$ kubectl create namespace mydlq-hub3、挂载 NFS 与创建目录
这里使用的是 NFS 存储驱动,如果使用其他存储驱动,请自行配置。
#挂载 NFS$ mount -o vers=4.1 192.168.2.11:/nfs/ /nfs
#创建文件夹mkdir -p /nfs/harbor/registrymkdir -p /nfs/harbor/chartmuseummkdir -p /nfs/harbor/jobservicemkdir -p /nfs/harbor/databasemkdir -p /nfs/harbor/redismkdir -p /nfs/harbor/trivy4、创建 PV 与 PVC
(1)、创建 PV 部署文件 harbor-pv.yaml
harbor-pv.yaml
#registry-PVapiVersion: v1kind: PersistentVolumemetadata: name: harbor-registry labels: app: harbor-registryspec: capacity: storage: 100Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: "hub" mountOptions: - hard - nfsvers=4.1 nfs: path: /nfs/harbor/registry server: 192.168.2.11---#harbor-chartmuseum-pvapiVersion: v1kind: PersistentVolumemetadata: name: harbor-chartmuseum labels: app: harbor-chartmuseumspec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: "hub" mountOptions: - hard - nfsvers=4.1 nfs: path: /nfs/harbor/chartmuseum server: 192.168.2.11---#harbor-jobservice-pvapiVersion: v1kind: PersistentVolumemetadata: name: harbor-jobservice labels: app: harbor-jobservicespec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: "hub" mountOptions: - hard - nfsvers=4.1 nfs: path: /nfs/harbor/jobservice server: 192.168.2.11---#harbor-database-pvapiVersion: v1kind: PersistentVolumemetadata: name: harbor-database labels: app: harbor-databasespec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: "hub" mountOptions: - hard - nfsvers=4.1 nfs: path: /nfs/harbor/database server: 192.168.2.11---#harbor-redis-pvapiVersion: v1kind: PersistentVolumemetadata: name: harbor-redis labels: app: harbor-redisspec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: "hub" mountOptions: - hard - nfsvers=4.1 nfs: path: /nfs/harbor/redis server: 192.168.2.11---#harbor-trivy-pvapiVersion: v1kind: PersistentVolumemetadata: name: harbor-trivy labels: app: harbor-trivyspec: capacity: storage: 5Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: "hub" mountOptions: - hard - nfsvers=4.1 nfs: path: /nfs/harbor/trivy server: 192.168.2.11执行 Kuberctl 命令创建 PV 资源:
- -f:指定资源配置文件
- -n:指定创建资源的命名空间
$ kubectl apply -f harbor-pv.yaml(2)、创建 PVC 部署文件 harbor-pvc.yaml
harbor-pvc.yaml
#harbor-registry-pvckind: PersistentVolumeClaimapiVersion: v1metadata: name: harbor-registryspec: accessModes: - ReadWriteOnce storageClassName: "hub" resources: requests: storage: 100Gi selector: matchLabels: app: harbor-registry---#harbor-chartmuseum-pvckind: PersistentVolumeClaimapiVersion: v1metadata: name: harbor-chartmuseumspec: accessModes: - ReadWriteOnce storageClassName: "hub" resources: requests: storage: 5Gi selector: matchLabels: app: harbor-chartmuseum---#harbor-jobservice-pvckind: PersistentVolumeClaimapiVersion: v1metadata: name: harbor-jobservicespec: accessModes: - ReadWriteOnce storageClassName: "hub" resources: requests: storage: 5Gi selector: matchLabels: app: harbor-jobservice---#harbor-database-pvckind: PersistentVolumeClaimapiVersion: v1metadata: name: harbor-databasespec: accessModes: - ReadWriteOnce storageClassName: "hub" resources: requests: storage: 5Gi selector: matchLabels: app: harbor-database---#harbor-redis-pvckind: PersistentVolumeClaimapiVersion: v1metadata: name: harbor-redisspec: accessModes: - ReadWriteOnce storageClassName: "hub" resources: requests: storage: 5Gi selector: matchLabels: app: harbor-redis---#harbor-trivy-pvckind: PersistentVolumeClaimapiVersion: v1metadata: name: harbor-trivyspec: accessModes: - ReadWriteOnce storageClassName: "hub" resources: requests: storage: 5Gi selector: matchLabels: app: harbor-trivy执行 Kuberctl 命令创建 PVC 资源:
- -f:指定要创建资源的文件
- -n:指定应用创建的命名空间
$ kubectl apply -f harbor-pvc.yaml -n mydlq-hub三、创建自定义证书
安装 Harbor 我们会默认使用 HTTPS 协议,需要 TLS 证书,如果我们没用自己设定自定义证书文件,那么 Harbor 将自动创建证书文件,不过这个有效期只有一年时间,所以这里我们生成自签名证书,为了避免频繁修改证书,将证书有效期为 10 年,操作如下:
1、生成证书文件:
下面执行步骤时,需要输入一些证书信息,其中 Common Name 必须要设置为和你要给 Harbor 的域名保持一致,如 Common Name (eg, your name or your server's hostname) []:hub.mydlq.club。
## 获得证书$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 3650 -out ca.crt
## 生成证书签名请求$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout tls.key -out tls.csr
## 生成证书$ openssl x509 -req -days 3650 -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt2、生成 Secret 资源
创建 Kubernetes 的 Secret 资源,且将证书文件导入:
-
- n:指定创建资源的 Namespace
- —from-file:指定要导入的文件地址
$ kubectl create secret generic hub-mydlq-tls --from-file=tls.crt --from-file=tls.key --from-file=ca.crt -n mydlq-hub查看是否创建成功:
$ kubectl get secret hub-mydlq-tls -n mydlq-hub可以观察到:
NAME TYPE DATA AGEhub-mydlq-tls Opaque 3 52m四、设置 Harbor 配置清单
由于我们需要通过 Helm 安装 Harbor 仓库,需要提前创建 Harbor Chart 的配置清单文件,里面是对要创建的应用 Harbor 进行一系列参数配置,由于参数过多,关于都有 Harbor Chart 都能够配置哪些参数这里就不一一罗列,可以通过访问 Harbor-helm 的 Github 地址 进行了解。
下面描述下,需要的一些配置参数:
values.yaml
#Ingress 网关入口配置expose: type: ingress tls: ### 是否启用 https 协议,如果不想启用 HTTPS,则可以设置为 false enabled: true ### 指定使用 sectet 挂载证书模式,且使用上面创建的 secret 资源 certSource: secret secret: secretName: "hub-mydlq-tls" notarySecretName: "hub-mydlq-tls" ingress: hosts: ### 配置 Harbor 的访问域名,需要注意的是配置 notary 域名要和 core 处第一个单词外,其余保持一致 core: hub.mydlq.club notary: notary.mydlq.club controller: default annotations: ingress.kubernetes.io/ssl-redirect: "true" ingress.kubernetes.io/proxy-body-size: "0" #### 如果是 traefik ingress,则按下面配置: kubernetes.io/ingress.class: "traefik" traefik.ingress.kubernetes.io/router.tls: 'true' traefik.ingress.kubernetes.io/router.entrypoints: websecure #### 如果是 nginx ingress,则按下面配置: #nginx.ingress.kubernetes.io/ssl-redirect: "true" #nginx.ingress.kubernetes.io/proxy-body-size: "0" ## 如果不想使用 Ingress 方式,则可以配置下面参数,配置为 NodePort #clusterIP: # name: harbor # ports: # httpPort: 80 # httpsPort: 443 # notaryPort: 4443 #nodePort: # name: harbor # ports: # http: # port: 80 # nodePort: 30011 # https: # port: 443 # nodePort: 30012 # notary: # port: 4443 # nodePort: 30013
## 如果Harbor部署在代理后,将其设置为代理的URL,这个值一般要和上面的 Ingress 配置的地址保存一致externalURL: https://hub.mydlq.club
### Harbor 各个组件的持久化配置,并设置各个组件 existingClaim 参数为上面创建的对应 PVC 名称persistence: enabled: true ### 存储保留策略,当PVC、PV删除后,是否保留存储数据 resourcePolicy: "keep" persistentVolumeClaim: registry: existingClaim: "harbor-registry" size: 100Gi chartmuseum: existingClaim: "harbor-chartmuseum" size: 5Gi jobservice: existingClaim: "harbor-jobservice" size: 5Gi database: existingClaim: "harbor-database" size: 5Gi redis: existingClaim: "harbor-redis" size: 5Gi trivy: existingClaim: "harbor-trivy" size: 5Gi
### 默认用户名 admin 的密码配置,注意:密码中一定要包含大小写字母与数字harborAdminPassword: "Mydlq123456"
### 设置日志级别logLevel: info
#各个组件 CPU & Memory 资源相关配置nginx: resources: requests: memory: 256Mi cpu: 500mportal: resources: requests: memory: 256Mi cpu: 500mcore: resources: requests: memory: 256Mi cpu: 1000mjobservice: resources: requests: memory: 256Mi cpu: 500mregistry: registry: resources: requests: memory: 256Mi cpu: 500m controller: resources: requests: memory: 256Mi cpu: 500mclair: clair: resources: requests: memory: 256Mi cpu: 500m adapter: resources: requests: memory: 256Mi cpu: 500mnotary: server: resources: requests: memory: 256Mi cpu: 500m signer: resources: requests: memory: 256Mi cpu: 500mdatabase: internal: resources: requests: memory: 256Mi cpu: 500mredis: internal: resources: requests: memory: 256Mi cpu: 500mtrivy: enabled: true resources: requests: cpu: 200m memory: 512Mi limits: cpu: 1000m memory: 1024Mi
#开启 chartmuseum,使 Harbor 能够存储 Helm 的 chartchartmuseum: enabled: true resources: requests: memory: 256Mi cpu: 500m五、安装 Harbor
1、添加 Helm 仓库
$ helm repo add harbor https://helm.goharbor.io2、部署 Harbor
$ helm install harbor harbor/harbor --version 1.5.2 -f values.yaml -n mydlq-hub3、查看应用是否部署完成
$ kubectl get deployment -n mydlq-hub
NAME READY UP-TO-DATE AVAILABLE AGEharbor-harbor-chartmuseum 1/1 1 1 5mharbor-harbor-clair 1/1 1 1 5mharbor-harbor-core 1/1 1 1 5mharbor-harbor-jobservice 1/1 1 1 5mharbor-harbor-notary-server 1/1 1 1 5mharbor-harbor-notary-signer 1/1 1 1 5mharbor-harbor-portal 1/1 1 1 5mharbor-harbor-registry 1/1 1 1 5m4、Host 配置域名
接下来配置 Hosts,客户端想通过域名访问服务,必须要进行 DNS 解析,由于这里没有 DNS 服务器进行域名解析,所以修改 hosts 文件将 Harbor 指定节点的 IP 和自定义 host 绑定。打开电脑的 Hosts 配置文件,往其加入下面配置:
192.168.2.11 hub.mydlq.club5、访问 Harbor
输入地址 https://hub.mydlq.club 访问 Harbor 仓库。
- 用户:admin
- 密码:Mydlq123456 (在安装配置中自定义的密码)

进入后可以看到 Harbor 的管理后台:

六、服务器配置镜像仓库
1、查看 Harbor 证书
再上面部署 harbor 过程中创建了 https 证书 ca.crt。这里我们查看证书,内容如下:
-----BEGIN CERTIFICATE-----MIIC9TCCAd2gAwIBAgIRALztT/b8wlhjw50UECEOTR8wDQYJKoZIhvcNAQELBQAwFDESMBAGA1UEAxMJaGFyYm9yLWNhMB4XDTIwMDIxOTA3NTgwMFoXDTIxMDIxODA3NTgwMFowFDESMBAGA1UEAxMJaGFyYm9yLWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArYbsxYmNksU5eQhVIM3OKac4l6MV/5u5belAlWSdpbbQCwMFG/gAliTSQMgqcmhQ3odYTKImvx+5zrhP5b1CWXCQCVOlOFSLrs3ZLv68ZpKoDLkg6XhoQFVPLM0v5V+YzWCGAson81LfX3tDhltnOItSpe2KESABVH+5L/2vo25P7Mvw4bWEWMyY4AS/3toiDZjhwNMrMb2lpICrlH9Sc3dAOzUteyVznA5/WF8IyPI64aKntl0gxLOZgUBTkBoxVhPj7dNNZu8lMnqAYXmhWt+oRr7t1HHp2lOtk2u/ndyV0kKLxufx5FYVJQel2yRBGc/C1QLN18nC1y6u5pITaQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACFT92PWBFeCT7By8y8+EkB2TD1QVMZmNDpBS75q5s2yIumFwJrbY6YsHtRkN1Zx9jc4LiJFHC6r0ES3tbCDapsxocvzn7dWXLNTtnSx0zPxNXZzgmTsamfunBd4gszdXMshJ+bKsEoTXhJEXVjZq/k0EZS8L4MpNZ7ciPqwAI1Tg+mFGp5UOvzxYLyW8nCLPykC73y3ob1tiO6xdyD/orTAbA6pIMc97ajTfwYj4Q6JPY/QAmu0S+4hJHs724IrC6hiXUlQNVVRW/d3k+nXbYttnnmPnQXCRyK2ru7R8H43Zlwj26kQJo6naQoQ0+Xcjcyk5llPqJxCrk3uoHF0r4U=-----END CERTIFICATE-----2、服务器 Docker 中配置 Harbor 证书
然后进入服务器,在服务器上 /etc/docker 目录下创建 certs.d 文件夹,然后在 certs.d 文件夹下创建 Harobr 域名文件夹,可以输入下面命令创建对应文件夹:
$ mkdir -p /etc/docker/certs.d/hub.mydlq.club然后再 /etc/docker/certs.d/hub.mydlq.club 目录下创建上面的 ca 证书文件:
$ cat > /etc/docker/certs.d/hub.mydlq.club/ca.crt << EOF-----BEGIN CERTIFICATE-----MIIC9TCCAd2gAwIBAgIRALztT/b8wlhjw50UECEOTR8wDQYJKoZIhvcNAQELBQAwFDESMBAGA1UEAxMJaGFyYm9yLWNhMB4XDTIwMDIxOTA3NTgwMFoXDTIxMDIxODA3NTgwMFowFDESMBAGA1UEAxMJaGFyYm9yLWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArYbsxYmNksU5eQhVIM3OKac4l6MV/5u5belAlWSdpbbQCwMFG/gAliTSQMgqcmhQ3odYTKImvx+5zrhP5b1CWXCQCVOlOFSLrs3ZLv68ZpKoDLkg6XhoQFVPLM0v5V+YzWCGAson81LfX3tDhltnOItSpe2KESABVH+5L/2vo25P7Mvw4bWEWMyY4AS/3toiDZjhwNMrMb2lpICrlH9Sc3dAOzUteyVznA5/WF8IyPI64aKntl0gxLOZgUBTkBoxVhPj7dNNZu8lMnqAYXmhWt+oRr7t1HHp2lOtk2u/ndyV0kKLxufx5FYVJQel2yRBGc/C1QLN18nC1y6u5pITaQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACFT92PWBFeCT7By8y8+EkB2TD1QVMZmNDpBS75q5s2yIumFwJrbY6YsHtRkN1Zx9jc4LiJFHC6r0ES3tbCDapsxocvzn7dWXLNTtnSx0zPxNXZzgmTsamfunBd4gszdXMshJ+bKsEoTXhJEXVjZq/k0EZS8L4MpNZ7ciPqwAI1Tg+mFGp5UOvzxYLyW8nCLPykC73y3ob1tiO6xdyD/orTAbA6pIMc97ajTfwYj4Q6JPY/QAmu0S+4hJHs724IrC6hiXUlQNVVRW/d3k+nXbYttnnmPnQXCRyK2ru7R8H43Zlwj26kQJo6naQoQ0+Xcjcyk5llPqJxCrk3uoHF0r4U=-----END CERTIFICATE-----EOF3、登录 Harbor 仓库
只有登录成功后才能将镜像推送到镜像仓库,所以配置完证书后尝试登录,测试是否能够登录成功:
如果提示 ca 证书错误,则重建检测证书配置是否有误。
$ docker login -u admin -p Mydlq123456 hub.mydlq.club七、服务器配置 Helm Chart 仓库
1、配置 Helm 证书
跟配置 Docker 仓库一样,配置 Helm 仓库也得提前配置证书,首先进入 ca 签名目录
如果下面执行的目录不存在,请用 yum 安装 ca-certificates 包。
$ cat > /etc/pki/ca-trust/source/anchors/ca.crt << EOF-----BEGIN CERTIFICATE-----MIIC9TCCAd2gAwIBAgIRALztT/b8wlhjw50UECEOTR8wDQYJKoZIhvcNAQELBQAwFDESMBAGA1UEAxMJaGFyYm9yLWNhMB4XDTIwMDIxOTA3NTgwMFoXDTIxMDIxODA3NTgwMFowFDESMBAGA1UEAxMJaGFyYm9yLWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArYbsxYmNksU5eQhVIM3OKac4l6MV/5u5belAlWSdpbbQCwMFG/gAliTSQMgqcmhQ3odYTKImvx+5zrhP5b1CWXCQCVOlOFSLrs3ZLv68ZpKoDLkg6XhoQFVPLM0v5V+YzWCGAson81LfX3tDhltnOItSpe2KESABVH+5L/2vo25P7Mvw4bWEWMyY4AS/3toiDZjhwNMrMb2lpICrlH9Sc3dAOzUteyVznA5/WF8IyPI64aKntl0gxLOZgUBTkBoxVhPj7dNNZu8lMnqAYXmhWt+oRr7t1HHp2lOtk2u/ndyV0kKLxufx5FYVJQel2yRBGc/C1QLN18nC1y6u5pITaQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAqQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACFT92PWBFeCT7By8y8+EkB2TD1QVMZmNDpBS75q5s2yIumFwJrbY6YsHtRkN1Zx9jc4LiJFHC6r0ES3tbCDapsxocvzn7dWXLNTtnSx0zPxNXZzgmTsamfunBd4gszdXMshJ+bKsEoTXhJEXVjZq/k0EZS8L4MpNZ7ciPqwAI1Tg+mFGp5UOvzxYLyW8nCLPykC73y3ob1tiO6xdyD/orTAbA6pIMc97ajTfwYj4Q6JPY/QAmu0S+4hJHs724IrC6hiXUlQNVVRW/d3k+nXbYttnnmPnQXCRyK2ru7R8H43Zlwj26kQJo6naQoQ0+Xcjcyk5llPqJxCrk3uoHF0r4U=-----END CERTIFICATE-----EOF执行更新命令,使证书生效:
$ update-ca-trust extract2、添加 Helm 仓库
添加 Helm 仓库:
$ helm repo add myrepo --username=admin --password=Mydlq123456 https://hub.mydlq.club/chartrepo/library- —username:harbor仓库用户名
- —password:harbor仓库密码
- —ca-file:指向ca.crt证书地址
- chartrepo:如果是chart仓库地址,中间必须加chartrepo
- library:仓库的项目名称
查看仓库列表:
$ helm repo list
NAME URLstable https://kubernetes-charts.storage.googleapis.comharbor https://helm.goharbor.iomyrepo https://hub.mydlq.club/chartrepo/library八、测试功能
1、推送与拉取 Docker 镜像
这里为了测试推送镜像,先下载一个用于测试的 helloworld 小镜像,然后推送到 hub.mydlq.club 仓库:
### 拉取 Helloworld 镜像$ docker pull hello-world:latest
### 将下载的镜像使用 tag 命令改变镜像名$ docker tag hello-world:latest hub.mydlq.club/library/hello-world:latest
### 推送镜像到镜像仓库$ docker push hub.mydlq.club/library/hello-world:latest将之前的下载的镜像删除,然后测试从 hub.mydlq.club 下载镜像进行测试:
### 删除之前镜像$ docker rmi hello-world:latest$ docker rmi hello-world:latest hub.mydlq.club/library/hello-world:latest
### 测试从 `hub.mydlq.club` 下载新镜像$ docker pull hub.mydlq.club/library/hello-world:latest2、推送与拉取 Chart
Helm 要想推送 Chart 到 Helm 仓库,需要提前安装上传插件:
$ helm plugin install https://github.com/chartmuseum/helm-push然后创建一个测试的 Chart 进行推送测试:
### 创建一个测 试chart$ helm create hello
### 打包chart,将chart打包成tgz格式$ helm package hello
### 推送 chart 进行测试$ helm push hello-0.1.0.tgz myrepo
Pushing hello-0.1.0.tgz to myrepo...Done.---END---
