It has been a long time already since Kubernetes is considered the most popular platform for application deployment and management. However, when you’re a little (just a little) stuck with Java of version 8, which is more than 9 years old now, and deploy your applications to the WebLogic server, it can be quite a long way to make it work in Kubernetes. Especially when you provide your applications to 4 countries, and each of them has at least 4 environments. So in this article, I will describe how we organized our deployment process, what tools we used for it, and how we had to modify our applications for it.
Migration to Java 17
The first step to prepare our applications was to migrate them to Java 17. As already noted above, back in time when it all started, Java 8 was already too old, while Java 17 was already released and was offering a wide range of improvements, which we were eager to use. Plus it was clear, that frameworks and libraries, that we use, will sooner or later stop their support for the 8th version (for example, Spring). Moreover, there were known problems with the 8 version inside the containers, when the JVM couldn’t figure out the memory and CPU allocated to the container. Even though it was backported to Java 8u191, it was obvious that future similar fixes won’t be backported.
We decided to immediately jump to the latest LTS version of Java. It would bring us longer support as well as many language advantages such as pattern matching, sealed classes, and improved switch expressions.
GitOps
GitOps is a part of DevOps that is based on using Git code repositories to manage infrastructure and application code deployments. Its key components are very well described here:
- The Git repository is the single source of truth for infrastructure definitions.
- The CD pipeline is used for building, testing, and deploying the application.
- The deployment tool is used to manage the application resources in the target environment.
- The monitoring system tracks the application performance and provides feedback to the development team.
According to these principles, we made every application (or say, microservice) on each environment have its own repository. Each repository has its own app-specific env-specific kubernetes yaml files. Take a look at the screenshot of such a repository.
It shows the repository of the OFS application deployed to the in00a1 environment, which is the first testing environment for India. And here you can see that we have all yaml files to configure the required Kubernetes objects.
Argo CD
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It is responsible to deliver all the yaml files from the described above repository to Kubernetes. Every 3m (by default) Argo CD checks for changes in these repositories and syncs them. Or you can trigger the sync yourself using the argocd client.
Argo CD has lots of useful features. One of them, which we use, is “Sync Phases and Waves“, which allows you to define the order of synchronization. As you can see in the screenshot above, there is a db-update-job.yaml, which is responsible for creating a kubernetes job. This job applies database migrations and we want it to run before the deployment.yaml is synced and a new pod is started. So all we need to do is to declare an annotation with a pre-sync hook:
|
1 2 3 |
metadata: annotations: argocd.argoproj.io/hook: PreSync |
Or you can define an order in which it will be synced using argo-waves:
|
1 2 3 |
metadata: annotations: argocd.argoproj.io/sync-wave: "5" |
Helm & GitLab CI/CD
Since every environment has its own repository and we have many environments, there must be an automated process of how yaml files end up there with env-specific values. Helm is an ideal tool for it. But we don’t use it in a common way via helm install, because we don’t want helm to deploy our application itself. Our intention is just to let it generate yaml files for us, which we will commit later to the environment repository. That’s what helm template is ideal for:
|
1 2 3 4 |
helm template "${CI_PROJECT_DIR}/deployment/helm/${APP_NAME}" -f "${CI_PROJECT_DIR}/deployment/helm/${APP_NAME}/conf/${COUNTRY}/${ENVIRONMENT}.yaml" --set version="${VERSION}" --output-dir ${ENVIRONMENT}/${APP_NAME} |
where we
- say to helm template, that our app’s helm config is in "${CI_PROJECT_DIR}/deployment/helm/${APP_NAME}"
- pass environment-specific properties as -f "${CI_PROJECT_DIR}/deployment/helm/${APP_NAME}/conf/${COUNTRY}/${ENVIRONMENT}.yaml"
- say where to output the resulting yaml files: --output-dir ${ENVIRONMENT}/${APP_NAME}, which is a git repository we mentioned above.
As you can see, the helm template command above has variables from the GitLab CI. This way we can have one common gitlabci project with all the deployment steps, which is independent of any application. Applications must only follow the same structure of directories for helm, e.g. they all must have a ${CI_PROJECT_DIR}/deployment/helm directory with helm configuration inside. All that a specific project needs to do is just to reference this common project:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
include: - project: 'CoolBankProjects/common-gitlabci' file: 'common-spring-gitlabci.yaml' variables: - APP_NAME: ofs stages: - build - deploy deploy_in00a1: stage: deploy variables: - COUNTRY: IN - ENVIRONMENT: in00a1 |
SealedSecret
After all yaml files are generated, we don’t want sensitive data to be visible to everybody in the environment repositories. Kubernetes secrets have data in a base64-encoded format, which can be easily decoded. The solution is to use SealedSecret. The SealedSecret can be decrypted only by the controller in the target cluster and nobody else is able to obtain the original Secret from the SealedSecret. All that is needed for encryption is a key certificate (public key portion). We store it together with cluster configuration, so during the gitlabci build we just need to fetch it from the cluster repository. To do so, firstly we clone the cluster repo:
|
1 2 3 |
git clone --branch master --single-branch "https://${TECH_USER_NAME}:${TECH_USER_TOKEN}@git.homecredit.net/deploy/clusters/${CLUSTER_GROUP}/${CLUSTER}.git" "/tmp/${ENVIRONMENT}" |
And then use the kubeseal command to seal the secret.yaml:
|
1 2 3 4 |
cat "${INPUT_SECRET}" | kubeseal \ --namespace "${NAMESPACE}" \ --cert "/tmp/${ENVIRONMENT}/sealed-secrets-cert.pem" --format yaml > "sealedsecret-ofs.yaml" |
Git-secret
As stated above, we need a kubernetes secret in order to get a SealedSecret. And as well, we want to make it universal for all environments. That’s why let’s define a secret like this:
|
1 2 3 4 5 6 |
apiVersion: v1 kind: Secret metadata: name: secrets-{{ .Chart.Name }} data: {{ .Files.Get (printf "%s-passwords.yaml" .Values.configuration.environment) | indent 2 }} |
"%s-passwords.yaml" (e.g. in00a1-passwords.yaml) file contains all sensitive data base64 encoded. But we can’t have the "%s-passwords.yaml" file committed to the application repository as it is. We need to hide sensitive data in the application repository too. Git-secret is a suitable solution in this case. It uses gpg to encrypt files with the public keys of the users that you trust. Then these users can decrypt these files using their personal secret keys. So all we need to do is to add public keys of all our application developers to git-secret. Also don’t forget about the technical user ${TECH_USER_NAME}, so that we can further work with it in gitlabci.
Now, let’s imagine we have the following property file in00a1_credentials in our application repository:
|
1 2 |
spring.datasource.username: dmitrii.bocharov spring.datasource.password: password123 |
Before committing any changes to this file, we just need to run git secret hide -d and git-secret will create a secret file in00a1_credentials.secret file out of it and delete the original file in00a1_credentials. Now it’s safe to commit. Similar steps must be added to the gitlabci.
Firstly, we need to import gpg private key from cicd-pipeline variables:
|
1 2 3 4 |
echo Importing gpg private key echo "$GPG_PRIVATE_KEY" | base64 -d > ./private_key.gpg gpg --batch --yes --pinentry-mode loopback --import private_key.gpg rm private_key.gpg |
After that, we can reveal the secret file using “$GPG_PASSPHRASE” which is also stored in the cicd-pipeline variables:
|
1 |
git secret reveal -p "$GPG_PASSPHRASE" |
Now we have a in00a1_credentials file. The final step to prepare everything for sealing a secret is to encode the value of all the properties into base64:
|
1 2 3 4 5 6 |
while IFS=': ' read -r KEY VALUE || [ -n "${VALUE}" ]; do if [ -n "${KEY}" ]; then VALUE_BASE64=$(echo -n "${VALUE}" | base64 -w 0) echo "${KEY}: ${VALUE_BASE64}" >> "in00a1-passwords.yaml" fi done < "in00a1_credentials" |
This bash snippet will create a in00a1-passwords.yaml file with base64 encoded data, which will be used by helm when creating a secret.
Conclusion
We’ve shown some technical details of how we do the deployment to kubernetes. We hope you find something useful and interesting in this article. If you have any questions or offers for improvement, you’re very welcome to the discussion in the comments below.