build: move e2e dependencies into e2e/go.mod

Several packages are only used while running the e2e suite. These
packages are less important to update, as the they can not influence the
final executable that is part of the Ceph-CSI container-image.

By moving these dependencies out of the main Ceph-CSI go.mod, it is
easier to identify if a reported CVE affects Ceph-CSI, or only the
testing (like most of the Kubernetes CVEs).

Signed-off-by: Niels de Vos <ndevos@ibm.com>
This commit is contained in:
Niels de Vos
2025-03-04 08:57:28 +01:00
committed by mergify[bot]
parent 15da101b1b
commit bec6090996
8047 changed files with 1407827 additions and 3453 deletions

View File

@ -0,0 +1,56 @@
## pwalkdir: parallel implementation of filepath.WalkDir
This is a wrapper for [filepath.WalkDir](https://pkg.go.dev/path/filepath#WalkDir)
which may speed it up by calling multiple callback functions (WalkDirFunc)
in parallel, utilizing goroutines.
By default, it utilizes 2\*runtime.NumCPU() goroutines for callbacks.
This can be changed by using WalkN function which has the additional
parameter, specifying the number of goroutines (concurrency).
### pwalk vs pwalkdir
This package is very similar to
[pwalk](https://pkg.go.dev/github.com/opencontainers/selinux/pkg/pwalkdir),
but utilizes `filepath.WalkDir` (added to Go 1.16), which does not call stat(2)
on every entry and is therefore faster (up to 3x, depending on usage scenario).
Users who are OK with requiring Go 1.16+ should switch to this
implementation.
### Caveats
Please note the following limitations of this code:
* Unlike filepath.WalkDir, the order of calls is non-deterministic;
* Only primitive error handling is supported:
* fs.SkipDir is not supported;
* ErrNotExist errors from filepath.WalkDir are silently ignored for any path
except the top directory (WalkDir argument); any other error is returned to
the caller of WalkDir;
* once any error is returned from any walkDirFunc instance, no more calls
to WalkDirFunc are made, and the error is returned to the caller of WalkDir;
* if more than one WalkDirFunc instance will return an error, only one
of such errors will be propagated to and returned by WalkDir, others
will be silently discarded.
### Documentation
For the official documentation, see
https://pkg.go.dev/github.com/opencontainers/selinux/pkg/pwalkdir
### Benchmarks
For a WalkDirFunc that consists solely of the return statement, this
implementation is about 15% slower than the standard library's
filepath.WalkDir.
Otherwise (if a WalkDirFunc is actually doing something) this is usually
faster, except when the WalkDirN(..., 1) is used. Run `go test -bench .`
to see how different operations can benefit from it, as well as how the
level of parallelism affects the speed.

View File

@ -0,0 +1,123 @@
//go:build go1.16
// +build go1.16
package pwalkdir
import (
"errors"
"fmt"
"io/fs"
"path/filepath"
"runtime"
"sync"
)
// Walk is a wrapper for filepath.WalkDir which can call multiple walkFn
// in parallel, allowing to handle each item concurrently. A maximum of
// twice the runtime.NumCPU() walkFn will be called at any one time.
// If you want to change the maximum, use WalkN instead.
//
// The order of calls is non-deterministic.
//
// Note that this implementation only supports primitive error handling:
//
// - no errors are ever passed to walkFn;
//
// - once a walkFn returns any error, all further processing stops
// and the error is returned to the caller of Walk;
//
// - filepath.SkipDir is not supported;
//
// - if more than one walkFn instance will return an error, only one
// of such errors will be propagated and returned by Walk, others
// will be silently discarded.
func Walk(root string, walkFn fs.WalkDirFunc) error {
return WalkN(root, walkFn, runtime.NumCPU()*2)
}
// WalkN is a wrapper for filepath.WalkDir which can call multiple walkFn
// in parallel, allowing to handle each item concurrently. A maximum of
// num walkFn will be called at any one time.
//
// Please see Walk documentation for caveats of using this function.
func WalkN(root string, walkFn fs.WalkDirFunc, num int) error {
// make sure limit is sensible
if num < 1 {
return fmt.Errorf("walk(%q): num must be > 0", root)
}
files := make(chan *walkArgs, 2*num)
errCh := make(chan error, 1) // Get the first error, ignore others.
// Start walking a tree asap.
var (
err error
wg sync.WaitGroup
rootLen = len(root)
rootEntry *walkArgs
)
wg.Add(1)
go func() {
err = filepath.WalkDir(root, func(p string, entry fs.DirEntry, err error) error {
if err != nil {
// Walking a file tree can race with removal,
// so ignore ENOENT, except for root.
// https://github.com/opencontainers/selinux/issues/199.
if errors.Is(err, fs.ErrNotExist) && len(p) != rootLen {
return nil
}
close(files)
return err
}
if len(p) == rootLen {
// Root entry is processed separately below.
rootEntry = &walkArgs{path: p, entry: entry}
return nil
}
// Add a file to the queue unless a callback sent an error.
select {
case e := <-errCh:
close(files)
return e
default:
files <- &walkArgs{path: p, entry: entry}
return nil
}
})
if err == nil {
close(files)
}
wg.Done()
}()
wg.Add(num)
for i := 0; i < num; i++ {
go func() {
for file := range files {
if e := walkFn(file.path, file.entry, nil); e != nil {
select {
case errCh <- e: // sent ok
default: // buffer full
}
}
}
wg.Done()
}()
}
wg.Wait()
if err == nil {
err = walkFn(rootEntry.path, rootEntry.entry, nil)
}
return err
}
// walkArgs holds the arguments that were passed to the Walk or WalkN
// functions.
type walkArgs struct {
entry fs.DirEntry
path string
}