python-gitlab CLI Usage Notes

It took us about two weeks to migrate our GitHub code to Gitlab, and the biggest impact was on our product release pipeline, which needed to be adapted to Gitlab and some services on our intranet environment. Basically, we had to rewrite the entire product release pipeline, and it was exhausting 🥺. At the time, I thought migrating the code to Gitlab was purely backwards 😅, but after all the adaptations were done, I suddenly realized that Gitlab was really good!

Ultimately, the Gitlab network on the intranet is ten times better than GitHub. As we all know, GitHub’s direct connection speed and stability is very poor in learning wall countries. That’s why the GitHub pipeline was often disturbed by network jitter, sometimes fetching a repo for 10 or 20 minutes! After migrating to Gitlab, it’s been a blast! It used to take at least 10 minutes to complete the pipeline, but now it takes less than 5 minutes to complete 😂.

So today I’m going to write a blog about what I’ve learned while tossing around Gitlab 👓.

Gitlab

Documentation and tools for Gitlab used in the tossing process.

Gitlab workflow: learn about Gitlab’s workflow, which is different from GitHub’s PR, in that Gitlab uses the MR approach.
Gitlab API: the official documentation for the Gitlab API, which you can use with the following tools.
python-gitlab API client: A Python implementation of the Gitlab API client, which you can use for specific needs tools, such as getting a file or directory in a repo based on a tag or branch.
python-gitlab CLI: based on the python-gitlab API client wrapped into a command-line tool that can be easily integrated into some pipeline scripts because it is a CLI tool.
go-gitlab API client: A Gitlab API client implemented in Golang. Since one of the phases of the release pipeline is to collect specific files and directories from other repo’s based on a list, the tools used are written in golang. The tools are written in golang, and go-gitlab is used instead of python-gitlab to reduce the amount of code changes.

Gitlab workflow

PR

On GitHub, we generally use PR to merge code, and the process is as follows.

clone the Fork repository to the local repository
Create a new branch locally, make changes and commits based on the new branch, and push the new branch to the Fork repository
Launch a Pull Request to the target branch of the original repository based on the new branch in the Fork repository
@ Reviewer in the comments of the PR, requesting review changes
The reviewer receives the request email, reviews the code, and comments directly at the suggestion
The committer continues to commit changes based on the suggestions and responds to the comments
After the reviewer has no objections, @ admin in the comments of the PR, request to merge the code, the admin accepts the PR, and the code is merged into the master branch

MR

But when we got to Gitlab, we used MR to do the code merge, and the process was as follows.

member Clone the original repository locally, create a new branch based on the branch you want to modify
local changes and commits, pushing the new branch to the original repository
Merge Request is launched in the original repository to the target protected branch based on the new branch.
Reviewers review code, administrators merge code

In comparison, the MR and approach are more suitable for collaborative development within a team, and the PR approach is suitable for collaborative development of open source projects.

Gitlab API

The main GitLab API is a REST API. Because of this, the documentation in this section assumes that you’re familiar with REST concepts.

Referring to the official document API resources, we can see that there are three API groups, Projects, Groups, and Standalone.

Projects: These are repo-related APIs, such as tag, commit, branch, MR, and Issue.
Groups: This is similar to Organizations on GitHub. Generally speaking, repo’s in a company are organized by team, and repo’s in the same team are placed under the same Groups in gitlab, rather than being stored as individuals.
Standalone: API resources other than Projects and Groups, such as user

Most of the time, we use the Projects API to add, delete, and check repo’s. After a brief introduction to the Gitlab API types, this article will introduce a few tools for using the Gitlab API. If you encounter any errors while using these tools, you can use the Status codes API to return status codes to troubleshoot the problem.

python-gitlab CLI

This is a gitlab command-line tool wrapped with the python-gitlab API client, and you can use it to do most of the operations supported by the Gitlab API. Because many of the operations in the previous pipeline were accessed from GitHub, such as committing PRs, getting repo tags, viewing PR labels, etc., they were written in Jenkinsfile and called with various scripts and tools. If you switch to Gitlab, you’ll need a tool to do the same thing. The python-gitlab CLI is a great tool for this, and it’s even more convenient than the previous tool. I haven’t seen anything like python-gitlab on GitHub so far. Anyway, for anyone who uses Gitlab and wants to automate some of the work you do with your repo, this CLI tool is highly recommended, and it can be easily integrated into your pipeline.

安装

The python-gitlab CLI relies on Python 2.7 or 3.4+, so don’t use Python 2.7 in 2021 😊. After installing python3 and pip3 locally, use the following command to install it.

1
2

# 在这里使用清华的 pypi 源来加速安装，毕竟是墙🧱国
$ sudo pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple requests PyYAML python-gitlab

Since most of the scenarios using this tool are executed in the slave pod created by Jenkins, it is also possible to build a docker image with the following Dockerfile

FROM debian:buster
RUN apt update \
    && apt install -y --no-install-recommends \
        git \
        python3 \
        python3-pip \
        jq \
    && rm -rf /var/lib/apt/lists/* \
    && pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple requests PyYAML python-gitlab

COPY python-gitlab.cfg /etc/python-gitlab.cfg

Configuration

The Gitlab CLI tool requires a python-gitlab.cfg configuration file to connect to the Gitlab server and to complete some authentication, in the form of an ini file as follows.

[global]
default = gitlab.com
ssl_verify = true
timeout = 5
per_page = 100
api_version = 4

[gitlab.com]
url = https://gitlab.com
private_token = xxxxxxxxx

Global connection parameters

Option	Possible values	Description
ssl_verify	True or False	Whether to enable SSL encryption authentication
timeout	Certificates	Connection timeout time
api_version	4	API version, the default is 4, refer to API V3 to API V4
per_page	1 ～ 100	The number of elements returned at a time is limited to 100 by Gitlab, and you can get all of them with the –all parameter

Customizing GitLab server parameters

Option	Description
url	GitLab server URL
private_token	Generate tokens by accessing -/profile/personal_access_tokens on the gitlab server
oauth_token
job_token
api_version	API version, the default is 4, or you can leave it undefined and use the global parameters
http_username	Gitlab username, which is not recommended for connecting to the Gitlab server
http_password	Gitlab password, which is not recommended for connecting to the Gitlab server

Save the file in ~/.python-gitlab.cfg or /etc/python-gitlab.cfg, or you can use the environment variable PYTHON_GITLAB_CFG or -config-file to execute the path to the configuration file, but to save yourself the trouble put it in ~/. python-gitlab.cfg.

Once the configuration is done, you can use the -gitlab current-user get command to test if the connection is working, and if it returns the correct username, the configuration is successful.

1
2

$ gitlab current-user get
username: muzi502

Basic Use

The gitlab command line tool is used to add, delete, and update various objects on the Gitlab server, such as user, project, file, repo, mr, tag, commit, and so on (get, list, create, delete, and update). The command line format used is as follows.

`1`	`$ gitlab <option> [object] [action] <option>`

In general, only 4 parameters are needed.

The first argument is the one immediately following the gitlab command. It is the output and configuration argument for the gitlab command line, such as the -o argument specifies the format of the output; the -f argument stores the output to a file; and the -g argument enforces which Gitlab server to connect to.
The second parameter specifies the object you want to operate on, such as project-merge-request, project-tag, etc. There are a number of supported objects, which basically cover all of the objects supported by the Gitlab API, as follows.

1
2
3

$ gitlab -h
usage: gitlab [-h] [--version] [-v] [-d] [-c CONFIG_FILE] [-g GITLAB] [-o {json,legacy,yaml}] [-f FIELDS]
{application,application-appearance,application-settings,audit-event,broadcast-message,current-user,current-user-email,current-user-gp-gkey,current-user-key,current-user-status,deploy-key,deploy-token,dockerfile,event,feature,geo-node,gitignore,gitlabciyml,group,group-access-request,group-badge,group-board,group-board-list,group-cluster,group-custom-attribute,group-deploy-token,group-epic,group-epic-issue,group-epic-resource-label-event,group-export,group-import,group-issue,group-label,group-member,group-merge-request,group-milestone,group-notification-settings,group-package,group-project,group-runner,group-subgroup,group-variable,hook,issue,l-da-pgroup,license,merge-request,namespace,notification-settings,pages-domain,project,project-access-request,project-additional-statistics,project-approval,project-approval-rule,project-badge,project-board,project-board-list,project-branch,project-cluster,project-commit,project-commit-comment,project-commit-discussion,project-commit-discussion-note,project-commit-status,project-custom-attribute,project-deploy-token,project-deployment,project-environment,project-event,project-export,project-file,project-fork,project-hook,project-import,project-issue,project-issue-award-emoji,project-issue-discussion,project-issue-discussion-note,project-issue-link,project-issue-note,project-issue-note-award-emoji,project-issue-resource-label-event,project-issue-resource-milestone-event,project-issues-statistics,project-job,project-key,project-label,project-member,project-merge-request,project-merge-request-approval,project-merge-request-approval-rule,project-merge-request-award-emoji,project-merge-request-diff,project-merge-request-discussion,project-merge-request-discussion-note,project-merge-request-note,project-merge-request-note-award-emoji,project-merge-request-resource-label-event,project-merge-request-resource-milestone-event,project-milestone,project-note,project-notification-settings,project-package,project-pages-domain,project-pipeline,project-pipeline-bridge,project-pipeline-job,project-pipeline-schedule,project-pipeline-schedule-variable,project-pipeline-variable,project-protected-branch,project-protected-tag,project-push-rules,project-registry-repository,project-registry-tag,project-release,project-remote-mirror,project-runner,project-service,project-snippet,project-snippet-award-emoji,project-snippet-discussion,project-snippet-discussion-note,project-snippet-note,project-snippet-note-award-emoji,project-tag,project-trigger,project-user,project-variable,project-wiki,runner,runner-job,snippet,todo,user,user-activities,user-custom-attribute,user-email,user-event,user-gp-gkey,user-impersonation-token,user-key,user-membership,user-project,user-status,variable}

The third argument is the action argument, which specifies what kind of operation is to be performed on the object being operated on, and generally supports add, delete, change and check operations (get, list, create, update, delete) $ gitlab project-tag usage: gitlab project-tag [-h] {list,get,create,delete,set-release-description} …

1
2

$ gitlab project-tag
usage: gitlab project-tag [-h] {list,get,create,delete,set-release-description} ...

The fourth parameter is the parameter on which the object + action depends, for example, specifying the project id

1
2
3

$ gitlab project-tag list
usage: gitlab project-tag list --project-id PROJECT_ID [--page PAGE] [--per-page PER_PAGE] [--all]
gitlab project-tag list: error: the following arguments are required: --project-id

Project-ID: is the unique ID of the repo on gitlab, there are two types, one is in the form of group/project, where / should be translated to %2F like: muzi502%2Fkubespray; the other is in the form of numbers, which can be seen on the web page of the repo, the second one is recommended.

# 也可以使用gitlab 命令获取 repo 的 id
$ gitlab project get --id muzi502%2Fkubespray
id: 25099880
path: kubespray

In the pipeline you can get the user’s username and email based on the token, which can be used to configure the repo git information in the pipeline to avoid failing due to CLA.

1
2
3

gitlab -o json current-user get | jq '.id'
git config --global user.email $(gitlab -o json current-user get | jq -r '.email')
git config --global user.name $(gitlab -o json current-user get | jq -r '.username')

project

Get repo ssh url address

Since the repo url for clone in the Jenkins pipeline uses token+ https, if you want to push code to the repo in the pipeline, you need to change it to ssh, and you can use the following method to get the ssh url of the repo based on the project id.

1
2
3

$ gitlab -o json  project get --id ${PROJECT_ID} | jq -r '.ssh_url_to_repo'
$ git remote remove origin
$ git remote add origin $(gitlab -o json  project get --id ${PROJECT_ID} | jq -r '.ssh_url_to_repo')

file

For manipulation of files in the repo use project-file

1
2

$ gitlab project-file
usage: gitlab project-file [-h] {get,create,update,delete,raw,blame} ...

Get the file, via the get operation of the project-file object

$ gitlab -o json project-file get --project-id ${PROJECT_ID} --file-path .gitignore --ref master  | jq '.'
{
  "file_name": ".gitignore",
  "file_path": ".gitignore",
  "size": 1208,
  "encoding": "base64",
  "content_sha256": "91f1d50ba3a4f79f96d9371afc70b389f75dfb2ac5190b8fb01051aa8679fd04",
  "ref": "master",
  "blob_id": "b09ca9d3b101034c7e34430177c1d64738df5fbb",
  "commit_id": "a9c97e5253c455546c2c7fdd794147eeb9b8ab7a",
  "last_commit_id": "4ffc106c58fc5865b6d72a52365e25b8c268d4d8",
  "content": "LnZhZ3JhbnQKKi5yZXRyeQoqKi92YWdyYW50X2Fuc2libGVfaW52ZW50b3J5CiouaW1sCnRlbXAKLmlkZWEKLnRveAouY2FjaGUKKi5iYWsKKi50ZnN0YXRlCioudGZzdGF0ZS5iYWNrdXAKLnRlcnJhZm9ybS8KY29udHJpYi90ZXJyYWZvcm0vYXdzL2NyZWRlbnRpYWxzLnRmdmFycwovc3NoLWJhc3Rpb24uY29uZgoqKi8qLnN3W3Bvbl0KKn4KdmFncmFudC8KcGx1Z2lucy9taXRvZ2VuCgojIEFuc2libGUgaW52ZW50b3J5CmludmVudG9yeS8qCiFpbnZlbnRvcnkvbG9jYWwKIWludmVudG9yeS9zYW1wbGUKaW52ZW50b3J5LyovYXJ0aWZhY3RzLwoKIyBCeXRlLWNvbXBpbGVkIC8gb3B0aW1pemVkIC8gRExMIGZpbGVzCl9fcHljYWNoZV9fLwoqLnB5W2NvZF0KKiRweS5jbGFzcwoKIyBEaXN0cmlidXRpb24gLyBwYWNrYWdpbmcKLlB5dGhvbgplbnYvCmJ1aWxkLwpjcmVkZW50aWFscy8KZGV2ZWxvcC1lZ2dzLwpkaXN0Lwpkb3dubG9hZHMvCmVnZ3MvCi5lZ2dzLwpwYXJ0cy8Kc2Rpc3QvCnZhci8KKi5lZ2ctaW5mby8KLmluc3RhbGxlZC5jZmcKKi5lZ2cKCiMgUHlJbnN0YWxsZXIKIyAgVXN1YWxseSB0aGVzZSBmaWxlcyBhcmUgd3JpdHRlbiBieSBhIHB5dGhvbiBzY3JpcHQgZnJvbSBhIHRlbXBsYXRlCiMgIGJlZm9yZSBQeUluc3RhbGxlciBidWlsZHMgdGhlIGV4ZSwgc28gYXMgdG8gaW5qZWN0IGRhdGUvb3RoZXIgaW5mb3MgaW50byBpdC4KKi5tYW5pZmVzdAoqLnNwZWMKCiMgSW5zdGFsbGVyIGxvZ3MKcGlwLWxvZy50eHQKcGlwLWRlbGV0ZS10aGlzLWRpcmVjdG9yeS50eHQKCiMgVW5pdCB0ZXN0IC8gY292ZXJhZ2UgcmVwb3J0cwpodG1sY292LwoudG94LwouY292ZXJhZ2UKLmNvdmVyYWdlLioKLmNhY2hlCm5vc2V0ZXN0cy54bWwKY292ZXJhZ2UueG1sCiosY292ZXIKLmh5cG90aGVzaXMvCgojIFRyYW5zbGF0aW9ucwoqLm1vCioucG90CgojIERqYW5nbyBzdHVmZjoKKi5sb2cKbG9jYWxfc2V0dGluZ3MucHkKCiMgRmxhc2sgc3R1ZmY6Cmluc3RhbmNlLwoud2ViYXNzZXRzLWNhY2hlCgojIFNjcmFweSBzdHVmZjoKLnNjcmFweQoKIyBTcGhpbnggZG9jdW1lbnRhdGlvbgpkb2NzL19idWlsZC8KCiMgUHlCdWlsZGVyCnRhcmdldC8KCiMgSVB5dGhvbiBOb3RlYm9vawouaXB5bmJfY2hlY2twb2ludHMKCiMgcHllbnYKLnB5dGhvbi12ZXJzaW9uCgojIGRvdGVudgouZW52CgojIHZpcnR1YWxlbnYKdmVudi8KRU5WLwo="
}

# The content of the file is base64 encoded and needs to be decoded using base64 to get the original content.
$ gitlab -g gitlab -o json project-file get --project-id 25099880 --file-path .gitignore --ref master  | jq -r '.content' | base64 -d
.vagrant
*.retry
**/vagrant_ansible_inventory
*.iml
temp
.idea
.tox
…………

The raw method of project-file can get the raw content of the file without base64 decoding

`1`	`$ gitlab project-file get --project-id ${PROJECT_ID} --file-path .gitignore --ref master`

To create a file, add, delete, or change it, you need to commit it, so you need to specify the branch you want to operate on and the commit-message. In addition, if the file to be operated is a master branch to get other protected branches, make sure the current user has write permission, otherwise the following error will be prompted.

1
2

gitlab project-file update
Impossible to update object (400: You are not allowed to push into this branch)

There was a set of CIs on GitHub that didn’t work for Gitlab, so you need to create CI flowlines for all branches to check if the code is in spec, and you can create files to all branches in bulk by doing the following.

ID=123456
BRANCHS=$(gitlab -o json project-branch list --project-id ${ID} --all | jq -r ".[].name")
for branch in ${BRANCHS}; do
	gitlab project-file create --project-id ${ID} --file-path .gitlab-ci.yml --branch ${branch} --content @.gitlab-ci.yml --commit-message "feat(gitlab-ci): add gitlab-ci.yml for ci"
done

Update files

For example, bulk update github.com to gitlab.com in the Makefile of all branches

ID=123456
BRANCHS=$(gitlab -o json project-branch list --project-id ${ID} --all | jq -r ".[].name")
for branch in ${BRANCHS}; do
    rm -f Makefile Makefile-
    gitlab project-file raw --project-id ${ID} --file-path Makefile --ref ${branch} > Makefile
    sed -i- "s|github.com/muzi502|gitlab.com/muzi502" Makefile
    gitlab project-file update --project-id ${ID} --file-path Makefile --branch ${branch} --content @Makefile \
    --commit-message "chore(Makefile): update repo url in Makefile for migrate gitlab"
done

Delete files

1
2

$ gitlab project-file delete --project-id ${PROJECT_ID} --file-path .gitignore --ref master \
--commit-message "test delete file"

MR

Create MR, specifying the source branch and target branch and the title of the mr. It is better to add -o json parameter in front to get the iid of the mr, which can be used to add, delete and check the mr.

`1`	`$ gitlab -o json project-merge-request create --project-id ${PROJECT_ID} --source-branch --target-branch ${BASE_BRANCH} --title "${MR_TITLE}"`

The -o json parameter returns information about this mr, where iid is the unique identifier of this mr in this repo

$ gitlab -g gitlab -o json project-merge-request create --project-id 25099880 --source-branch release-2.14 --target-branch  master --title "mr create test"
{
  "id": 92872102,
  "iid": 1,
  "project_id": 25099880,
  "title": "mr create test",
  "description": null,
  "state": "opened",
  "created_at": "2021-03-21T12:42:52.893Z",
  "updated_at": "2021-03-21T12:42:52.893Z",
  "merged_by": null,
  "merged_at": null,
  "closed_by": null,
  "closed_at": null,
  "target_branch": "master",
  "source_branch": "release-2.14",
  "user_notes_count": 0,
  "upvotes": 0,
  "downvotes": 0,
  "author": {
    "id": 5599205,
    "name": "muzi502",
    "username": "muzi502",
    "state": "active",
    "avatar_url": "https://secure.gravatar.com/avatar/f91578ffea9a538eedd8fbaf3007289b?s=80&d=identicon",
    "web_url": "https://gitlab.com/muzi502"
  }

Merge MR

`1`	`$ gitlab project-merge-request merge --project-id ${PROJECT_ID} --iid @mr_iid`

View MR Status

`1`	`$ gitlab -o json project-merge-request get --project-id ${PROJECT_ID} --iid @mr_iid \| jq -r ".state"`

Integrating in Jenkinsfile to create MRs, merge MRs, and check MRs

When you call it, you only need to pass the parameters SOURCE_BRANCH, TARGET_BRANCH, MR_TITLE.

def makeMR(SOURCE_BRANCH, TARGET_BRANCH, MR_TITLE) {
    container("debian") {
        sh """
        gitlab -o json project-merge-request create --project-id ${PROJECT_ID}  --title \"${MR_TITLE}\" \
        --source-branch ${SOURCE_BRANCH} --target-branch ${TARGET_BRANCH} > mr_info.json

        jq -r '.iid' mr_info.json > mr_iid
        jq -r '.web_url' mr_info.json > mr_url
        """
    }
}

def checkMR() {
    container("debian") {
        retry(120) {
        sh """
        if [ ! -s mr_iid ]; then exit 0; else sleep 60s; fi
        gitlab -o json project-merge-request get --project-id ${PROJECT_ID} --iid @mr_iid | jq -r ".labels[]" | grep 'approve'
        """
        }
    }
}

def mergeMR(){
    container("debian") {
        retry(10){
        sh """
        if [ ! -s mr_iid ]; then exit 0; else sleep 60s; fi
        if gitlab project-merge-request merge --project-id ${PROJECT_ID} --iid @mr_iid; then sleep 10s; fi
        gitlab -o json project-merge-request get --project-id ${PROJECT_ID} --iid @mr_iid | jq -r ".state" | grep 'merged'
        """
        }
    }
}

Tag

List repo tags

We recommend using the git tag method to get the repo tag, because the Gitlab API is limited to 100 values per request, so you can add the -all argument to return all the values.

1
2

$ gitlab -o json project-tag list --project-id ${ID} | jq -r '.[].name'
$ gitlab -o json project-tag list --project-id ${ID} --all | jq -r '.[].name'

Create tag

`1`	`$ gitlab project-tag create --project-id ${ID} --tag-name v1.0.0-rc.2 --ref master`

Delete tag

The upstream repo tag can only be deleted from Gitlab, but not from the local repo, so you can delete the Gitlab repo tag with the following command.

`1`	`$ gitlab project-tag delete --project-id ${ID} --tag-name v1.0.0-rc.2`

Create a protected repo tag

Since our pipeline tasks rely on repo tags to do versioning, we need to protect each repo tag, but there are special cases where we need to overwrite repo tags, so there is no suitable method for protected repo tags yet, so we have to create them manually first and then delete them when we need to. You can use the following command to create protected repo tags in bulk.

`1`	`$ git tag \| xargs -L1 -I {} gitlab project-protected-tag create --project-id ${ID} --name {}`

Lint flow line

After migrating to Gitlab, I couldn’t use my old pipeline on Gitlab’s intranet, so I used Gitlab’s own CI to reduce maintenance costs. The repo I maintain only uses Gitlab CI to do a few lint checks, so the CI configuration is very simple. For example, the CI configuration for kubespray is

.gitlab-ci.yml

---
lint:
  image: 'quay.io/kubespray/kubespray:v2.15.0'
  script:
    - chmod -R o-w .
    - make lint
  tags:
    - shared

Add a .gitlab-ci.yml file to all branches using the gitlab CLI tool

ID=123456
BRANCHS=$(gitlab -o json project-branch list --project-id ${ID} --all | jq -r ".[].name")
for branch in ${BRANCHS}; do
	gitlab project-file create --project-id ${ID} --file-path .gitlab-ci.yml --branch ${branch} --content @.gitlab-ci.yml --commit-message "feat(gitlab-ci): add gitlab-ci.yml for ci"
done

Other

repo migration

Since Gitlab doesn’t support importing git URLs internally, you have to manually clone your GitHub repo locally and then push it to Gitlab. If you use git clone, you will only have one master branch locally, so you have to track all the branches of the repo on GitHub and then push it to Gitlab.

# 使用 git clone 下来的 repo 默认为 master 分支
$ git clone git@gitlab.com/muzi502/kubespray.git
# track 出 origin 上的所有分支
$ git branch -r | grep -v '\->' | while read remote; do git branch --track "${remote#origin/}" "$remote"; done
$ git fetch --all
$ git pull --all
$ git remote remove origin
$ git remote add origin git@gitlab/gitlab502/kubespray.git
$ git push origin --all

Table of Contents