Jenkins 持续集成实战

如何定制Jenkins jnlp docker镜像

为什么需要定制 jnlp 镜像?

你是否有如下需求:

  • 要在容器内使用命令行工具 wget, curl, telnet, tcpdump
  • 构建制品的过程中要使用 nodejs, npm, maven, jdk
  • 要在 K8S 容器中构建 docker 镜像就要使用到 kaniko 工具
  • 需要 python3 来执行一些脚本
  • 要使用 kubectl, helm 命令行工具发布 K8S 应用

如果你有如上需求,那么需要为你的业务环境量身定做一个 jnlp docker 镜像了。

当然也可以在流水线阶段中指定 agent 来完成不同类型的任务。

分析 jenkins/inbound-agent Dockerfile

Github 地址为:Docker image for inbound Jenkins agents 从中选用 jdk11 + debian 版本的 Dockerfile,其对应的 DockerHub 地址为:Docker image for inbound Jenkins agents

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# The MIT License
#
# Copyright (c) 2015-2017, CloudBees, Inc.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

ARG version=3107.v665000b_51092-10
FROM jenkins/agent:${version}-jdk11

ARG version=3107.v665000b_51092-10
LABEL Description="This is a base image, which allows connecting Jenkins agents via JNLP protocols" Vendor="Jenkins project" Version="$version"

ARG user=jenkins

USER root
COPY ../../jenkins-agent /usr/local/bin/jenkins-agent
RUN chmod +x /usr/local/bin/jenkins-agent &&\
ln -s /usr/local/bin/jenkins-agent /usr/local/bin/jenkins-slave
USER ${user}

ENTRYPOINT ["/usr/local/bin/jenkins-agent"]

我们来分析这个 Dockerfile 是如何制作一个 jenkins agent 的:

  1. FROM jenkins/agent:3107.v665000b_51092-10-jdk11 首先使用的是这个上层镜像作为基础镜像,jdk 版本为 11
  2. USER root 其次将身份切换为 root,再把项目根路径下的 jenkins-agent 启动脚本放进容器并授权
  3. USER jenkins 最后切换身份为 jenkins,以 ENTRYPOINT 模式启动脚本

在这个 Dockerfile 中并没有体现软件包的安装,我们可以先把这个上层镜像 docker pull 下来看下 debian 的版本号:

1
2
3
4
5
6
7
8
9
10
11
$ docker run -it --rm -u root --entrypoint bash uhub.service.ucloud.cn/gao7public/agent:3107.v665000b_51092-10-jdk11
root@d25e8d950855:/home/jenkins# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

我这里使用了优刻得镜像加速器

继续翻看这个上层镜像的 Dockerfile 去分析它是如何构建的。

继续分析 jenkins/agent Dockerfile

Github: Jenkins Agent Docker image
Dockerhub: Jenkins Agent Docker image

可以去 Dockerhub 直接搜索镜像名称 jenkins/agent,找到 Dockerhub 地址 Jenkins Agent Docker image,这正是 jnlp 更名前使用的上层镜像。再找到它对应的 Github 仓库地址:Jenkins Agent Docker image

同样的,找到仓库 jdk11 + debian 版本对应的 Dockerfile,这正是刚才 jenkins/inbound-agent 使用的上层基础镜像:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
# The MIT License
#
# Copyright (c) 2015-2023, CloudBees, Inc. and other Jenkins contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

ARG JAVA_VERSION=17.0.7_7
FROM eclipse-temurin:"${JAVA_VERSION}"-jdk-focal AS jre-build

# This Build ARG is populated by Docker
# Ref. https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope
ARG TARGETPLATFORM

SHELL ["/bin/bash","-e", "-u", "-o", "pipefail", "-c"]

# Generate smaller java runtime without unneeded files
# for now we include the full module path to maintain compatibility
# while still saving space (approx 200mb from the full distribution)
RUN if test "${TARGETPLATFORM}" != 'linux/arm/v7'; then \
case "$(jlink --version 2>&1)" in \
# jlink version 11 has less features than jdk11+
"11."*) strip_java_debug_flags=("--strip-debug") ;; \
*) strip_java_debug_flags=("--strip-java-debug-attributes") ;; \
esac; \
jlink \
--add-modules ALL-MODULE-PATH \
"${strip_java_debug_flags[@]}" \
--no-man-pages \
--no-header-files \
--compress=2 \
--output /javaruntime; \
# It is acceptable to have a larger image in arm/v7 (arm 32 bits) environment.
# Because jlink fails with the error "jmods: Value too large for defined data type" error.
else cp -r /opt/java/openjdk /javaruntime; \
fi

FROM debian:bullseye-20230502 AS build

ARG user=jenkins
ARG group=jenkins
ARG uid=1000
ARG gid=1000

RUN groupadd -g "${gid}" "${group}" \
&& useradd -l -c "Jenkins user" -d /home/"${user}" -u "${uid}" -g "${gid}" -m "${user}"

ARG AGENT_WORKDIR=/home/"${user}"/agent
ENV TZ=Etc/UTC

## Always use the latest Debian packages: no need for versions
# hadolint ignore=DL3008
RUN apt-get update \
&& apt-get --yes --no-install-recommends install \
ca-certificates \
curl \
fontconfig \
git \
git-lfs \
less \
netbase \
openssh-client \
patch \
tzdata \
&& apt-get clean \
&& rm -rf /tmp/* /var/cache/* /usr/share/doc/* /usr/share/man/* /var/lib/apt/lists/*

ARG VERSION=3107.v665000b_51092
ADD --chown="${user}":"${group}" "https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/${VERSION}/remoting-${VERSION}.jar" /usr/share/jenkins/agent.jar
RUN chmod 0644 /usr/share/jenkins/agent.jar \
&& ln -sf /usr/share/jenkins/agent.jar /usr/share/jenkins/slave.jar

ENV LANG C.UTF-8

ENV JAVA_HOME=/opt/java/openjdk
COPY --from=jre-build /javaruntime "$JAVA_HOME"
ENV PATH="${JAVA_HOME}/bin:${PATH}"

USER "${user}"
ENV AGENT_WORKDIR=${AGENT_WORKDIR}
RUN mkdir /home/${user}/.jenkins && mkdir -p "${AGENT_WORKDIR}"

VOLUME /home/"${user}"/.jenkins
VOLUME "${AGENT_WORKDIR}"
WORKDIR /home/"${user}"
ENV user=${user}
LABEL \
org.opencontainers.image.vendor="Jenkins project" \
org.opencontainers.image.title="Official Jenkins Agent Base Docker image" \
org.opencontainers.image.description="This is a base image, which provides the Jenkins agent executable (agent.jar)" \
org.opencontainers.image.version="${VERSION}" \
org.opencontainers.image.url="https://www.jenkins.io/" \
org.opencontainers.image.source="https://github.com/jenkinsci/docker-agent" \
org.opencontainers.image.licenses="MIT"

看到 apt-get install 就知道找到我们需要的东西了,只要模仿其结构添加我们需要额外安装的软件包就可以了。

在开始构建之前,我们仍然需要知道这个镜像所使用的上层基础镜像有:

  • FROM eclipse-temurin:17.0.7_7-jdk-focal
  • FROM debian:bullseye-20230502

我们仅做了解,正式构建我们仍然使用 jenkins/inbound-agent Dockerfile

开始构建自定义 jnlp docker image

获取构建所需二进制

1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建构建目录
mkdir -p /data/inbound-agent && cd /data/inbound-agent

# 获取agent启动脚本:jenkins-agent
git clone https://github.com/jenkinsci/docker-inbound-agent.git
cp docker-inbound-agent/jenkins-agent ./
rm -rf docker-inbound-agent/

# 获取 helm 二进制命令:
wget https://get.helm.sh/helm-v3.8.1-linux-amd64.tar.gz
tar xvf helm-v3.8.1-linux-amd64.tar.gz
cp linux-amd64/helm ./
rm -rf linux-amd64/ helm-v3.8.1-linux-amd64.tar.gz

查看:

1
2
3
4
5
$ ls -l /data/inbound-agent/
total 147068
-rw-r--r-- 1 root root 1152 May 15 14:52 Dockerfile
-rwxr-xr-x 1 root root 45072384 May 15 10:58 helm
-rwxr-xr-x 1 root root 5052 May 15 14:34 jenkins-agent

编排 Dockerfile

  1. 二进制本地文件:
    • jenkins-agent entrypoint 启动文件,来自本地脚本文件
    • helm v3.8.1 来自本地自二进制文件
  2. docker hub 二进制容器镜像:
    • kubectl v1.20.6 来自二进制容器镜像 kubectl
    • kaniko v1.9.0 来自二进制容器镜像,并创建工作目录 /workspace
  3. apt install:
    • nodejs 12.x.x, npm 7.x.x
    • curl, wget, rsync
  4. 容器运行身份切换为 root

整理如上业务环境构建需求后,以多阶段的方式来构建 Dockerfile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# https://hub.docker.com/r/bitnami/kubectl
FROM uhub.service.ucloud.cn/gao7public/kubectl:1.20.6 as kubectl
# https://hub.docker.com/r/jenkins/agent
FROM uhub.service.ucloud.cn/gao7public/agent:3107.v665000b_51092-10-jdk11

ENV TZ=Asia/Shanghai

USER root
RUN apt-get update \
&& apt-get --yes --no-install-recommends install \
vim \
curl \
wget \
rsync \
iputils-ping \
telnet \
python2 \
python3 \
nodejs \
npm \
&& apt-get clean \
&& rm -rf /tmp/* /var/cache/* /usr/share/doc/* /usr/share/man/* /var/lib/apt/lists/*

COPY --from=kubectl /opt/bitnami/kubectl/bin/kubectl /usr/bin/kubectl
COPY helm /usr/local/bin/helm
COPY jenkins-agent /usr/local/bin/jenkins-agent
RUN chmod +x /usr/local/bin/jenkins-agent &&\
ln -s /usr/local/bin/jenkins-agent /usr/local/bin/jenkins-slave &&\
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone &&\
chmod +x /usr/local/bin/helm

ENTRYPOINT ["/usr/local/bin/jenkins-agent"]

构建镜像:

1
2
3
4
5
6
# 构建
cd /data/inbound-agent
docker image build -t uhub.service.ucloud.cn/gao7public/inbound-agent:0.0.8 .

# 上传
docker push uhub.service.ucloud.cn/gao7public/inbound-agent:0.0.8

创建一个容器,对各种构建工具查看版本号:

1
2
3
4
5
6
7
8
$ docker run --rm -it --entrypoint bash uhub.service.ucloud.cn/gao7public/inbound-agent:0.0.8
rsync --version
node -v
npm -v
python3 -V
/kaniko/executor version
kubectl version
helm version

Jenkins 固定 agent 测试 jnlp 镜像

jnlp 的方式创建一个 Jenkins 从节点:

获取这个从节点的密钥:

Docker image for inbound Jenkins agents 得到 docker run 所需的命令行参数:

1
docker run --name inbound-agent --privileged=true -d --init uhub.service.ucloud.cn/gao7public/inbound-agent:0.0.8 -url http://192.168.50.57:8080/ 2d2f77b41ba3d84670e5285e47b326907dd3035398fb02504150590931a0e3b7 inbound-agent

查看 Jenkins agent 状态:

创建一个流水线进行测试Pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
pipeline {
agent { label "test" }
stages {
stage("build") {
steps {
script {
sh """
pwd
whoami
rsync --version
python3 -V
node -v
npm -v
/kaniko/executor version
helm version
kubectl version
sleep 300
"""
}
}
}
}
}

查看日志,各种构建工具能够正确输出版本号:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
Started by user admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on inbound-agent in /home/jenkins/agent/workspace/workspace/test1
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] script
[Pipeline] {
[Pipeline] sh
+ pwd
/home/jenkins/agent/workspace/workspace/test1
+ whoami
root
+ rsync --version
rsync version 3.2.3 protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
symtimes, prealloc, stop-at, no crtimes
Optimizations:
SIMD, asm, openssl-crypto
Checksum list:
xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
+ python3 -V
Python 3.9.2
+ node -v
v12.22.12
+ npm -v
7.5.2
+ /kaniko/executor version
Kaniko version : v1.9.0
+ helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}
+ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE