Commit Graph

80 Commits

Author SHA1 Message Date
Niels de Vos
5ea99fdd5b build: add yamlgen to build deployment files
This initial version of yamlgen generates deploy/scc.yaml based on the
deployment artifact that is provided by the new api/deploy/ocp package.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-10-05 11:26:50 +00:00
Niels de Vos
36e099d939 deploy: add API for getting OpenShift SecurityContextConstraints
This new Go module allows other projects (like Rook) to consume
deployment details (the SCC to get started) directly from Ceph-CSI.

Based-on: https://github.com/rook/rook/blob/3c93eb3/cluster/examples/kubernetes/ceph/operator-openshift.yaml#L47-L90
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-10-05 11:26:50 +00:00
Yati Padia
e077c1fdf5 cleanup: run codespell on containerized testing
This commit adds a new target codespell to the
make containerized-test.

Fixes: #2229

Signed-off-by: Yati Padia <ypadia@redhat.com>
2021-08-12 09:42:54 +05:30
Niels de Vos
538e36f7a7 ci: pass GITHUB_BASE_REF when running commitlint
GitHub Actions include a merge commit for the PR, which will defeat the
commitlint checking of all the commits inside the PR (only the merge
commit is checked).

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-06-07 18:19:23 +05:30
Niels de Vos
2f6fca0862 build: regenerate scripts/golangci.yml if build.env was updated
In case build.env was updated, scripts/golangci.yml needs to be
regenerated. It contains a reference to the build-tag that is used to
identify the Ceph version to link against. Failing to update the
scripts/golangci.yml configuration, may cause running tests fail.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-17 13:38:46 +00:00
Niels de Vos
92190dee5d ci: use "devel" branch instead of "master"
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2021-03-01 10:51:30 +05:30
Niels de Vos
61a16012b7 build: only use --cpuset options when the cgroup controller is available
Fixes: #1670
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-12-23 02:53:51 +00:00
Madhu Rajanna
cd96b3359b build: use CSI_IMAGE_VERSION in makefile
updated makefile to use CSI_IMAGE_VERSION
from build.env
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-12-09 17:14:19 +00:00
Madhu Rajanna
39b1f2b4d3 cleanup: fix mispell words
fixed mispell words in the repo.
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-11-29 12:47:46 +05:30
Niels de Vos
cba466c163 ci: do not run "make mod-check" for containerized-tests
An updated CI job will run "make mod-check" in parallel with the full
containerized-test and containerized-build targets. This will hopefully
reduced the time that is needed for the whole
ci/centos/containerized-tests job.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:36:07 +00:00
Niels de Vos
afd6994e19 ci: allow passing USE_PULLED_IMAGE=yes to use pre-pulled images
When passing USE_PULLED_IMAGE=yes to the containerized-test or
containerized-build make targets, it is now possible to use pre-pulled
container images. This saves time as the container images will not get
created from scratch.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-11-04 05:36:07 +00:00
Prasanna Kumar Kalever
e9d5e731dc build: change default to use podman
With '# make image-cephcsi GOARCH=amd64'
On Fedora observed errors like:

 ---> Running in 3e2dbf48ebc6
OCI runtime create failed: this version of runc doesn't work on cgroups
v2: unknown
make: *** [Makefile:175: image-cephcsi] Error 1

hence it is recommended to switch to podman if available.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>
2020-08-19 11:52:10 +00:00
Niels de Vos
6d5456cc77 ci: use FETCH_HEAD like "commitlint --from" does
After fetching the target branch for the PR, the GIT_SINCE ref in the
git repository may not be set (in CI environments). This causes 'git
rebase' to fail. Use FETCH_HEAD instead, so that the checked out PR gets
rebased correctly.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 14:34:08 +00:00
Niels de Vos
104203ed03 ci: pass REBASE=1 to have commitlint rebase the branch
When Mergify adds a merge commit to the branch that is being tested with
commitlint, the tool tries to detect the most recent changes based on
the newly merged commit. This is for most PRs the master branch, and
that contains incorrect commit messages in the history. Because of this,
commitlint will fail.

By adding an option (REBASE=1) to the commitlint make target, CI jobs
can request a rebase so that the history of the PR becomes linear again
and commitlint should be able to detect only the new commits.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 14:34:08 +00:00
Niels de Vos
84b7e7b11c build: store detected container-cmd to prevent rebuilds
The 'need-container-cmd' make target is marked as .PHONY which will
cause it to be run every time. That triggers a rebuild of the container
images, even when that is not required.

By removing the 'need-container-cmd' target from .PHONY, and storing the
contents of CONTAINER_CMD in .container-cmd, unneeded rebuilds are
prevented.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-18 14:34:08 +00:00
Niels de Vos
9bd91bc9d3 ci: use last "git fetch" output to test commitlint
When running in the CI the git repository is not completely cloned. This
causes the 'commitlint' job to be unable to resolve the history of the
commits.

By using FETCH_HEAD, the last 'git fetch' output will be used.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-13 12:12:58 +05:30
Niels de Vos
35c5afcd3e ci: always fetch GIT_SINCE for commitlint
When running the commitlint CI job, the branch that the PR is based on
may not be available. That makes it impossible for commitlint to detect
the changes between the HEAD of the branch, and the commits in the PR.

By fetching GIT_SINCE unconditionally, commitlint should be able to
detect the changes and only run the tests against the commits that were
added through the PR.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-08-11 08:11:37 +00:00
Niels de Vos
74ba85f87b ci: add "make run-e2e"
The keeps the standard arguments for e2e testing in a single location
instead of spread over multiple files and CI jobs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-31 16:03:54 +00:00
Madhu Rajanna
b949d5f9c7 build: Build and push multi architecture images to dockerhub
As quay.io doesnot support the multi architecture
images, We need to switch to dockerhub as it supports
multi-architecture images.

closes: #1003

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-07-16 08:57:38 +00:00
Niels de Vos
969b032f1c build: error out when Podman or Docker is not available
In case Podman or Docker is not installed, running containerized tests
or builds will fail with unclear errors.

Introduce a 'need-container-cmd' make target that checks if the
CONTAINER_CMD is available (and non-empty). When running without Podman
and Docker, the following error is displayed:

    $ make containerized-build
    cephcsi image settings: quay.io/cephcsi/cephcsi version canary
    Missing container support, install Podman or Docker
    make: *** [Makefile:135: need-container-cmd] Error 1

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-07 09:49:45 +00:00
Niels de Vos
f9f9fba26c ci: generate golangci.yml with correct CEPH_VERSION
When building against go-ceph, the most recent version of Ceph is
assumed to be available (currently Octopus). In case an older version of
the development packages is installed, building go-ceph will fail.

Golangci-lint does not accept the `-tags nautilus` parameter like other
Golang tools. Instead, the build-constraints need to be configured in a
confguration file.

This change takes care of the following:
- move the current scripts/golangci.yml to a template
- add the @@CEPH_VERSION@@ substitute
- generate the configuration file when needed

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-07-02 14:24:02 +00:00
Niels de Vos
c02413a663 build: Add -tags=nautilus to go test and go build
Build `GO_TAGS` based on the `CEPH_VERSION` from build.env. In case the
version is non-empty, pass `-tags=${CEPH_VERSION}` to any of the go
commandline and script that call go programs.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-30 07:55:37 +00:00
Yug Gupta
306d5b1da0 ci: add pylint as part of CI
add pylint to catch static issues of python
files in the repo.

User can now run make lint-py for pylint
check on python files.

Signed-off-by: Yug Gupta <ygupta@redhat.com>
2020-06-30 04:43:45 +00:00
Niels de Vos
44bf398c79 build: move GOARCH environment variable to targets that use it
This prevents some clutter while running 'make' commands on an
environment where 'go' is not installed. There is no requirement on 'go'
in the case containers are used for building and testing.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
3cc791c7c4 ci: document TEST_COVERAGE and GO_COVER_DIR in buid.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
4fd973b924 build: use BASE_IMAGE from build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
bc2ac1ccac build: move GOLANGCI_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
f83a065c8a build: move GOLANG_VERSION to build.env
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-28 17:46:37 +00:00
Niels de Vos
c0b53d3ad4 build: add option to compile e2e.test in a container
Run `make containerized-build TARGET=e2e.test` to build the e2e.test
executable in a container. This makes it possible for environments to
only have the dependencies for the runtime installed (most notably Ceph
shared libraries) and, run the e2e tests directly.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-06-02 08:28:54 +00:00
Niels de Vos
ed30fa19d4 build: use CURDIR instead of PWD
`make -C ${GOPATH/src/github.com/ceph/ceph-csi container...` commands
fail as the PWD variable gets substituted with the path where `make` is
called, not with the path of the Makefile that is passed with the -C
option.

The CURDIR variable is expended to the correct path, so `make -C ...`
works for all make targets now.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-31 15:03:58 +00:00
Niels de Vos
686db53fde build: prevent error message when 'go' is unavailable
In case the `go` executable is not available, any `make` command shows
the error `go: command not found` even when running the actual task in a
container (that will have `go` available).

So, redirect the error message to `/dev/null`. In case `go` is really
not available in the build environment, the `check-env` make target will
report it.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-19 07:31:58 +00:00
Niels de Vos
eff4f4ba50 ci: add commitlint target to the Makefile
When passing `GIT_SINCE` as an argument, the commits from then on
(defaults to origin/master) are used. A CI job can pass the base branch
or ID so a range of patches gets checked.

To test the last four commits, one can run:

    $ make containerized-test TARGET=commitlint GIT_SINCE=HEAD~4

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-15 18:20:29 +00:00
Madhu Rajanna
3482cb7091 ci: updated shell script to run individual test
Earlier we were running all the linter for non-go
files in one short, this wont be helpful for the
users who want to run particular tests.

now the Makefile as different target to
run separate lint test for different type
of non-go files.

Fixes: #979

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-05-14 16:54:19 +00:00
Niels de Vos
b89f7fa1d6 build: add check for functional environment
Updates: #1000
Reported-by: Madhu Rajanna <madhupr007@gmail.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-05-12 10:52:59 +00:00
Yaniv Kaul
1f7d9760d5 Makefile: allow building with Podman
If Docker does not exist, use Podman.
Use all CPUs for the build, which hopefully will make it faster.

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
Fixes: #987
2020-04-27 05:21:26 +00:00
Niels de Vos
d1e12cb0e2 add optional TARGET=.. parameter for 'make containerized-test'
Running the containerized-tests is relatively time consuming as it runs
all the tests. By adding a TARGET=.. option to the Makefile, it is now
possible to run a selected test, for example:

    make containerized-test TARGET=static-check

This reduces the time the tests take, so it is more friendly to the
developers while addressing problems.

When no TARGET=.. parameter is passed, the default tests are run.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-24 15:41:46 +00:00
Niels de Vos
32839948ef cleanup: move pkg/ to internal/
The internal/ directory in Go has a special meaning, and indicates that
those packages are not meant for external consumption. Ceph-CSI does
provide public APIs for other projects to consume. There is no plan to
keep the API of the internally used packages stable.

Closes: #903
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 11:00:59 +00:00
Niels de Vos
2c6c0decbe containerized-build: set WORKDIR and use in make target
By setting the WORKDIR in the container image, there is no need to pass
it on the commandline in the Makefile. This makes the line for the make
target a little cleaner.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-23 03:41:06 +00:00
Madhu Rajanna
bfa6064b4d Read baseimage from the dockerfile
Updated deploy.sh and build scripts
to read base image from the dockerfile

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-22 15:41:40 +00:00
Madhu Rajanna
4b540ffecc Update makefile and deploy.sh to pass build arguments
updated makefile to pass the required build
arguments for multistage docker build

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-22 15:41:40 +00:00
Madhu Rajanna
32e2a713e6 Fix multi architecture dockerfile
Add support for multi architecture build
for cephcsi. with multistage docker build
we we build cephcsi binary for both arm64
and amd64 architecture.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-22 15:41:40 +00:00
Niels de Vos
22c0baa4b0 tests: fail when "make mod-check" detects a modified go.mod file
`go mod verify` does not fail when all moduled under vendor/ are
updated. However it does update `go.mod` in case it does not reflect the
downloaded versions.

By adding this additional verification in `make mod-check`, future
updates to vendor/ will require a correct go.mod as well.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-22 06:26:12 +00:00
Niels de Vos
4d60e76241 travis: split static checks and continue with tests in case one fails
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 07:01:10 +00:00
Niels de Vos
3c6054ddf9 add make target for containerized tests
`make containerized-test` has been added as a make target. This runs the
'make test' target in a container. All test dependencies are installed
in the container once, and the container is kept around for running
`make containerized-test` subsequently.

The test container is based on Fedora:latest so that all test tools get
easily installed and are available in a recent version. The production
container is based on the Ceph container which has CentOS as Operating
System and therefor a more stable (too old) toolset.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-04-21 05:26:32 +00:00
Madhu Rajanna
d09ffbd6de helm: add helm charts E2E
This PR adds the support for helm
installation, and cephcsi helm charts
deployment and teardown and also runs E2E
on for helm charts.

Add socat to provide port forwadring access for helm

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-04-06 11:01:25 +00:00
Niels de Vos
7381253ee0 build: add an option to compile in a container
This makes it possible to build on any platform that supports Linux
containers. The container image used for building is created once, or on
updating the `scripts/Dockerfile.build` and is cached afterwards.

To build the executable in a container, use `make containerized-build`
and everything will be done automatically. The executable will also be
available on the usual location.

Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-26 08:45:53 +00:00
Madhu Rajanna
d5a0606c33 Migrate from dep to go module
Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>
2020-03-17 10:44:07 +00:00
Niels de Vos
3226b17d08 enable building with cgo for go-ceph usage
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2020-03-11 16:09:10 +00:00
Yibo Cai
4b8b52e0de Support Arm64 image
Update CI merge job to build and push Arm64 image to
quay.io/cephcsi/cephcsi:version-arm64.

Add CI PR job running on Travis Arm64 nodes to make sure cephcsi
compiles successfully on Arm64.

No CI test job is availabe for Arm64 now due to below issues
- k8s-csi sidecar images for Arm64 are not available
- Travis Arm64 CI job runs inside unprivileged LXD which blocks
  launching minikube test environment

Signed-off-by: Yibo Cai <yibo.cai@arm.com>
2020-01-09 09:53:50 +00:00
Niels de Vos
8f133e03b8 Add 'gosec' to the static-checks
Run static security scanning tool 'gosec' while testing.

URL: https://github.com/securego/gosec
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2019-09-04 11:48:37 +00:00