mirrors/woodpecker
1
0
Fork 0
mirror of https://github.com/woodpecker-ci/woodpecker.git synced 2024-05-20 09:18:15 +00:00
Code Issues Projects Releases Wiki Activity

Add support for pipeline root.when conditions (#770)

Browse source 
Co-authored-by: Zav Shotan <zshotan@bloomberg.net>
Co-authored-by: Anbraten <anton@ju60.de>
Co-authored-by: 6543 <6543@obermui.de>
This commit is contained in:
Zav Shotan 2022-09-26 03:27:20 -04:00 committed by GitHub
parent 62d82765fd
commit ec9b0a62a7
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
12 changed files with 440 additions and 110 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

240
docs/docs/20-usage/20-pipeline-syntax.md
View file

@ -21,88 +21,6 @@ pipeline:
In the above example we define two pipeline steps, `frontend` and `backend`. The names of these steps are completely arbitrary.
## Global Pipeline Conditionals
Woodpecker gives the ability to skip whole pipelines (not just steps) based on certain conditions.
### `branches`
Woodpecker can skip commits based on the target branch. If the branch matches the `branches:` block the pipeline is executed, otherwise it is skipped.
Example skipping a commit when the target branch is not master:
```diff
pipeline:
build:
image: golang
commands:
- go build
- go test
+branches: master
```
Example matching multiple target branches:
```diff
pipeline:
build:
image: golang
commands:
- go build
- go test
+branches: [ master, develop ]
```
Example uses glob matching:
```diff
pipeline:
build:
image: golang
commands:
- go build
- go test
+branches: [ master, feature/* ]
```
Example includes branches:
```diff
pipeline:
build:
image: golang
commands:
- go build
- go test
+branches:
+ include: [ master, feature/* ]
```
Example excludes branches:
```diff
pipeline:
build:
image: golang
commands:
- go build
- go test
+branches:
+ exclude: [ develop, feature/* ]
```
The branch matching is done using [doublestar](https://github.com/bmatcuk/doublestar/#usage), note that a pattern starting with `*` should be put between quotes and a literal `/` needs to be escaped. A few examples:
- `*\\/*` to match patterns with exactly 1 `/`
- `*\\/**` to match patters with at least 1 `/`
- `*` to match patterns without `/`
- `**` to match everything
### Skip Commits
Woodpecker gives the ability to skip individual commits by adding `[CI SKIP]` to the commit message. Note this is case-insensitive.
@ -334,6 +252,13 @@ when:
- branch: prefix/*
```
The branch matching is done using [doublestar](https://github.com/bmatcuk/doublestar/#usage), note that a pattern starting with `*` should be put between quotes and a literal `/` needs to be escaped. A few examples:
- `*\\/*` to match patterns with exactly 1 `/`
- `*\\/**` to match patters with at least 1 `/`
- `*` to match patterns without `/`
- `**` to match everything
Execute a step using custom include and exclude logic:
```yaml
@ -469,7 +394,7 @@ when:
:::info
Path conditions are applied only to **push** and **pull_request** events.
It is currently **only available** for GitHub, GitLab.
Gitea only support **push** at the moment ([go-gitea/gitea#18228](https://github.com/go-gitea/gitea/pull/18228)).
Gitea only supports **push** at the moment ([go-gitea/gitea#18228](https://github.com/go-gitea/gitea/pull/18228)).
:::
Execute a step only on a pipeline with certain files being changed:
@ -731,6 +656,155 @@ pipeline:
...
```
## `when` - Global pipeline conditions
Woodpecker gives the ability to skip whole pipelines (not just steps #when---conditional-execution-1) based on certain conditions by a `when` block. If all conditions in the `when` block evaluate to true the pipeline is executed, otherwise it is skipped, but treated as successful and other pipelines depending on it will still continue.
### `repo`
Example conditional execution by repository:
```diff
pipeline:
slack:
image: plugins/slack
settings:
channel: dev
+ when:
+ repo: test/test
```
### `branch`
:::note
Branch conditions are not applied to tags.
:::
Example conditional execution by branch:
```diff
pipeline:
slack:
image: plugins/slack
settings:
channel: dev
+ when:
+ branch: master
```
> The step now triggers on master, but also if the target branch of a pull request is `master`. Add an event condition to limit it further to pushes on master only.
Execute a step if the branch is `master` or `develop`:
```diff
when:
branch: [master, develop]
```
Execute a step if the branch starts with `prefix/*`:
```diff
when:
branch: prefix/*
```
Execute a step using custom include and exclude logic:
```diff
when:
branch:
include: [ master, release/* ]
exclude: [ release/1.0.0, release/1.1.* ]
```
### `event`
Execute a step if the build event is a `tag`:
```diff
when:
event: tag
```
Execute a step if the pipeline event is a `push` to a specified branch:
```diff
when:
event: push
+ branch: main
```
Execute a step for all non-pull request events:
```diff
when:
event: [push, tag, deployment]
```
Execute a step for all build events:
```diff
when:
event: [push, pull_request, tag, deployment]
```
### `tag`
This filter only applies to tag events.
Use glob expression to execute a step if the tag name starts with `v`:
```diff
when:
event: tag
tag: v*
```
### `environment`
Execute a step for deployment events matching the target deployment environment:
```diff
when:
environment: production
event: deployment
```
### `instance`
Execute a step only on a certain Woodpecker instance matching the specified hostname:
```diff
when:
instance: stage.woodpecker.company.com
```
### `path`
:::info
Path conditions are applied only to **push** and **pull_request** events.
It is currently **only available** for GitHub, GitLab.
Gitea only supports **push** at the moment ([go-gitea/gitea#18228](https://github.com/go-gitea/gitea/pull/18228)).
:::
Execute a step only on a pipeline with certain files being changed:
```diff
when:
path: "src/*"
```
You can use [glob patterns](https://github.com/bmatcuk/doublestar#patterns) to match the changed files and specify if the step should run if a file matching that pattern has been changed `include` or if some files have **not** been changed `exclude`.
```diff
when:
path:
include: [ '.woodpecker/*.yml', '*.ini' ]
exclude: [ '*.md', 'docs/**' ]
ignore_message: "[ALL]"
```
**Hint:** Passing a defined ignore-message like `[ALL]` inside the commit message will ignore all path conditions.
## `depends_on`
Woodpecker supports to define multiple pipelines for a repository. Those pipelines will run independent from each other. To depend them on each other you can use the [`depends_on`](https://woodpecker-ci.org/docs/usage/multi-pipeline#flow-control) keyword.

12
pipeline/frontend/yaml/compiler/compiler.go
View file

@ -85,6 +85,12 @@ func New(opts ...Option) *Compiler {
func (c *Compiler) Compile(conf *yaml.Config) *backend.Config {
config := new(backend.Config)
if !conf.When.Match(c.metadata, true) {
// This pipeline does not match the configured filter so return an empty config and stop further compilation.
// An empty pipeline will just be skipped completely.
return config
}
// create a default volume
config.Volumes = append(config.Volumes, &backend.Volume{
Name: fmt.Sprintf("%s_default", c.prefix),
@ -149,7 +155,7 @@ func (c *Compiler) Compile(conf *yaml.Config) *backend.Config {
config.Stages = append(config.Stages, stage)
} else if !c.local && !conf.SkipClone {
for i, container := range conf.Clone.Containers {
if !container.When.Match(c.metadata) {
if !container.When.Match(c.metadata, false) {
continue
}
stage := new(backend.Stage)
@ -176,7 +182,7 @@ func (c *Compiler) Compile(conf *yaml.Config) *backend.Config {
stage.Alias = nameServices
for i, container := range conf.Services.Containers {
if !container.When.Match(c.metadata) {
if !container.When.Match(c.metadata, false) {
continue
}
@ -196,7 +202,7 @@ func (c *Compiler) Compile(conf *yaml.Config) *backend.Config {
continue
}
if !container.When.Match(c.metadata) {
if !container.When.Match(c.metadata, false) {
continue
}

4
pipeline/frontend/yaml/config.go
View file

@ -10,9 +10,9 @@ import (
type (
// Config defines a pipeline configuration.
Config struct {
When constraint.When `yaml:"when,omitempty"`
Cache types.Stringorslice
Platform string
Branches constraint.List
Workspace Workspace
Clone Containers
Pipeline Containers
@ -23,6 +23,8 @@ type (
DependsOn []string `yaml:"depends_on,omitempty"`
RunsOn []string `yaml:"runs_on,omitempty"`
SkipClone bool `yaml:"skip_clone"`
// Deprecated use When.Branch
Branches constraint.List
}
// Workspace defines a pipeline workspace.

51
pipeline/frontend/yaml/config_test.go
View file

@ -5,6 +5,7 @@ import (
"github.com/franela/goblin"
"github.com/woodpecker-ci/woodpecker/pipeline/frontend"
"github.com/woodpecker-ci/woodpecker/pipeline/frontend/yaml/types"
)
@ -19,6 +20,8 @@ func TestParse(t *testing.T) {
g.Fail(err)
}
g.Assert(out.When.Constraints[0].Event.Match("tester")).Equal(true)
g.Assert(out.Workspace.Base).Equal("/go")
g.Assert(out.Workspace.Path).Equal("src/github.com/octocat/hello-world")
g.Assert(out.Volumes.Volumes[0].Name).Equal("custom")
@ -61,17 +64,65 @@ func TestParse(t *testing.T) {
}
g.Assert(out.Pipeline.Containers[0].Name).Equal("notify_fail")
g.Assert(out.Pipeline.Containers[0].Image).Equal("plugins/slack")
g.Assert(out.Pipeline.Containers[1].Name).Equal("notify_success")
g.Assert(out.Pipeline.Containers[1].Image).Equal("plugins/slack")
g.Assert(len(out.Pipeline.Containers[0].When.Constraints)).Equal(0)
g.Assert(out.Pipeline.Containers[1].Name).Equal("notify_success")
g.Assert(out.Pipeline.Containers[1].Image).Equal("plugins/slack")
g.Assert(out.Pipeline.Containers[1].When.Constraints[0].Event.Include).Equal([]string{"success"})
})
matchConfig, err := ParseString(sampleYaml)
if err != nil {
g.Fail(err)
}
g.It("Should match event tester", func() {
g.Assert(matchConfig.When.Match(frontend.Metadata{
Curr: frontend.Build{
Event: "tester",
},
}, false)).Equal(true)
})
g.It("Should match event tester2", func() {
g.Assert(matchConfig.When.Match(frontend.Metadata{
Curr: frontend.Build{
Event: "tester2",
},
}, false)).Equal(true)
})
g.It("Should match branch tester", func() {
g.Assert(matchConfig.When.Match(frontend.Metadata{
Curr: frontend.Build{
Commit: frontend.Commit{
Branch: "tester",
},
},
}, true)).Equal(true)
})
g.It("Should not match event push", func() {
g.Assert(matchConfig.When.Match(frontend.Metadata{
Curr: frontend.Build{
Event: "push",
},
}, false)).Equal(false)
})
})
})
}
var sampleYaml = `
image: hello-world
when:
- event:
- tester
- tester2
- branch:
- tester
build:
context: .
dockerfile: Dockerfile

39
pipeline/frontend/yaml/constraint/constraint.go
View file

@ -58,9 +58,9 @@ func (when *When) IsEmpty() bool {
}
// Returns true if at least one of the internal constraints is true.
func (when *When) Match(metadata frontend.Metadata) bool {
func (when *When) Match(metadata frontend.Metadata, global bool) bool {
for _, c := range when.Constraints {
if c.Match(metadata) {
if c.Match(metadata, global) {
return true
}
}
@ -68,7 +68,7 @@ func (when *When) Match(metadata frontend.Metadata) bool {
if when.IsEmpty() {
// test against default Constraints
empty := &Constraint{}
return empty.Match(metadata)
return empty.Match(metadata, global)
}
return false
}
@ -126,24 +126,21 @@ func (when *When) UnmarshalYAML(value *yaml.Node) error {
// Match returns true if all constraints match the given input. If a single
// constraint fails a false value is returned.
func (c *Constraint) Match(metadata frontend.Metadata) bool {
// if event filter is not set, set default
if c.Event.IsEmpty() {
c.Event.Include = []string{
frontend.EventPush,
frontend.EventPull,
frontend.EventTag,
frontend.EventDeploy,
}
func (c *Constraint) Match(metadata frontend.Metadata, global bool) bool {
match := true
if !global {
c.SetDefaultEventFilter()
// apply step only filters
match = c.Matrix.Match(metadata.Job.Matrix)
}
match := c.Platform.Match(metadata.Sys.Platform) &&
match = match && c.Platform.Match(metadata.Sys.Platform) &&
c.Environment.Match(metadata.Curr.Target) &&
c.Event.Match(metadata.Curr.Event) &&
c.Repo.Match(metadata.Repo.Name) &&
c.Ref.Match(metadata.Curr.Commit.Ref) &&
c.Instance.Match(metadata.Sys.Host) &&
c.Matrix.Match(metadata.Job.Matrix)
c.Instance.Match(metadata.Sys.Host)
// changed files filter apply only for pull-request and push events
if metadata.Curr.Event == frontend.EventPull || metadata.Curr.Event == frontend.EventPush {
@ -161,6 +158,18 @@ func (c *Constraint) Match(metadata frontend.Metadata) bool {
return match
}
// SetDefaultEventFilter set default e event filter if not event filter is already set
func (c *Constraint) SetDefaultEventFilter() {
if c.Event.IsEmpty() {
c.Event.Include = []string{
frontend.EventPush,
frontend.EventPull,
frontend.EventTag,
frontend.EventDeploy,
}
}
}
// IsEmpty return true if a constraint has no conditions
func (c List) IsEmpty() bool {
return len(c.Include) == 0 && len(c.Exclude) == 0

2
pipeline/frontend/yaml/constraint/constraint_test.go
View file

@ -489,7 +489,7 @@ func TestConstraints(t *testing.T) {
for _, test := range testdata {
t.Run(test.desc, func(t *testing.T) {
c := parseConstraints(t, test.conf)
got, want := c.Match(test.with), test.want
got, want := c.Match(test.with, false), test.want
if got != want {
t.Errorf("Expect %+v matches %q is %v", test.with, test.conf, want)
}

18
pipeline/schema/.woodpecker/test-pipeline-when.yml Normal file
View file

@ -0,0 +1,18 @@
when:
- branch: [master, deploy]
event: push
path:
- "folder/**"
- "**/*.c"
- tag: "v**"
event: tag
- event: cron
cron:
include:
- hello
pipeline:
echo:
image: alpine
commands:
- echo "test"

96
pipeline/schema/schema.json
View file

@ -98,6 +98,100 @@
{ "type": "array", "items": { "$ref": "#/definitions/step" }, "minLength": 1 }
]
},
"pipeline_when": {
"description": "Whole pipelines can be skipped based on conditions. Read more: TODO",
"oneOf": [
{
"type": "array",
"minLength": 1,
"items": { "$ref": "#/definitions/pipeline_when_condition" }
},
{
"$ref": "#/definitions/pipeline_when_condition"
}
]
},
"pipeline_when_condition": {
"type": "object",
"additionalProperties": false,
"properties": {
"repo": {
"description": "Execute a step only on a specific repository. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#repo",
"$ref": "#/definitions/constraint_list"
},
"branch": {
"description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#branch",
"$ref": "#/definitions/constraint_list"
},
"event": {
"description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#event",
"default": [],
"oneOf": [
{
"type": "array",
"minLength": 1,
"items": { "$ref": "#/definitions/event_enum" }
},
{
"$ref": "#/definitions/event_enum"
}
]
},
"tag": {
"description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#tag",
"type": "string"
},
"cron": {
"description": "filter cron by title. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#cron",
"$ref": "#/definitions/constraint_list"
},
"platform": {
"description": "Execute a step only on a specific platform. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#platform",
"$ref": "#/definitions/constraint_list"
},
"environment": {
"description": "Execute a step only for a specific environment. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#environment",
"$ref": "#/definitions/constraint_list"
},
"instance": {
"description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#instance",
"$ref": "#/definitions/constraint_list"
},
"path": {
"description": "Execute a step only on commit with certain files added/removed/modified. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#path",
"oneOf": [
{ "type": "string" },
{
"type": "array",
"items": {
"type": "string"
}
},
{
"type": "object",
"properties": {
"include": {
"type": "array",
"items": {
"type": "string"
}
},
"exclude": {
"type": "array",
"items": {
"type": "string"
}
},
"ignore_message": {
"type": "string"
}
},
"additionalProperties": false
}
]
}
}
},
"step": {
"description": "Every step of your pipeline executes arbitrary commands inside a specified docker container. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#steps",
"type": "object",
@ -165,6 +259,7 @@
},
"event": {
"description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#event",
"default": ["push", "pull_request", "tag", "deployment"],
"oneOf": [
{
"type": "array",
@ -253,7 +348,6 @@
}
},
"event_enum": {
"default": ["push", "pull_request", "tag", "deployment"],
"enum": ["push", "pull_request", "tag", "deployment", "cron"]
},
"constraint_list": {

2
server/pipeline/create.go
View file

@ -63,7 +63,7 @@ func Create(ctx context.Context, _store store.Store, repo *model.Repo, build *mo
configFetcher := shared.NewConfigFetcher(server.Config.Services.Remote, server.Config.Services.ConfigService, repoUser, repo, build)
remoteYamlConfigs, configFetchErr = configFetcher.Fetch(ctx)
if configFetchErr == nil {
filtered, parseErr = branchFiltered(build, remoteYamlConfigs)
filtered, parseErr = checkIfFiltered(build, remoteYamlConfigs)
if parseErr == nil {
if filtered {
err := ErrFiltered{Msg: "branch does not match restrictions defined in yaml"}

27
server/pipeline/filter.go
View file

@ -19,6 +19,7 @@ package pipeline
import (
"github.com/rs/zerolog/log"
"github.com/woodpecker-ci/woodpecker/pipeline/frontend"
"github.com/woodpecker-ci/woodpecker/pipeline/frontend/yaml"
"github.com/woodpecker-ci/woodpecker/server/model"
"github.com/woodpecker-ci/woodpecker/server/remote"
@ -49,11 +50,17 @@ func zeroSteps(build *model.Build, remoteYamlConfigs []*remote.FileMeta) bool {
}
// TODO: parse yaml once and not for each filter function
func branchFiltered(build *model.Build, remoteYamlConfigs []*remote.FileMeta) (bool, error) {
// Check if at least one pipeline step will be execute otherwise we will just ignore this webhook
func checkIfFiltered(build *model.Build, remoteYamlConfigs []*remote.FileMeta) (bool, error) {
log.Trace().Msgf("hook.branchFiltered(): build branch: '%s' build event: '%s' config count: %d", build.Branch, build.Event, len(remoteYamlConfigs))
if build.Event == model.EventTag || build.Event == model.EventDeploy {
return false, nil
matchMetadata := frontend.Metadata{
Curr: frontend.Build{
Event: string(build.Event),
Commit: frontend.Commit{
Branch: build.Branch,
},
},
}
for _, remoteYamlConfig := range remoteYamlConfigs {
@ -64,10 +71,20 @@ func branchFiltered(build *model.Build, remoteYamlConfigs []*remote.FileMeta) (b
}
log.Trace().Msgf("config '%s': %#v", remoteYamlConfig.Name, parsedPipelineConfig)
if parsedPipelineConfig.Branches.Match(build.Branch) {
return false, nil
// ignore if the pipeline was filtered by matched constraints
if !parsedPipelineConfig.When.Match(matchMetadata, true) {
continue
}
// ignore if the pipeline was filtered by the branch (legacy)
if !parsedPipelineConfig.Branches.Match(build.Branch) {
continue
}
// at least one config yielded in a valid run.
return false, nil
}
// no configs yielded a valid run.
return true, nil
}

13
server/shared/procBuilder.go
View file

@ -23,6 +23,7 @@ import (
"strings"
"github.com/drone/envsubst"
"github.com/rs/zerolog/log"
backend "github.com/woodpecker-ci/woodpecker/pipeline/backend/types"
"github.com/woodpecker-ci/woodpecker/pipeline/frontend"
@ -117,7 +118,19 @@ func (b *ProcBuilder) Build() ([]*BuildItem, error) {
return nil, &yaml.PipelineParseError{Err: err}
}
// checking if filtered.
if !parsed.When.Match(metadata, true) {
log.Debug().Str("pipeline", proc.Name).Msg(
"Marked as skipped, dose not match metadata",
)
proc.State = model.StatusSkipped
}
// TODO: deprecated branches filter => remove after some time
if !parsed.Branches.Match(b.Curr.Branch) && (b.Curr.Event != model.EventDeploy && b.Curr.Event != model.EventTag) {
log.Debug().Str("pipeline", proc.Name).Msg(
"Marked as skipped, dose not match branch",
)
proc.State = model.StatusSkipped
}

46
server/shared/procBuilder_test.go
View file

@ -326,6 +326,52 @@ pipeline:
}
}
func TestRootWhenFilter(t *testing.T) {
t.Parallel()
b := ProcBuilder{
Repo: &model.Repo{},
Curr: &model.Build{Event: "tester"},
Last: &model.Build{},
Netrc: &model.Netrc{},
Secs: []*model.Secret{},
Regs: []*model.Registry{},
Link: "",
Yamls: []*remote.FileMeta{
{Data: []byte(`
when:
event:
- tester
pipeline:
xxx:
image: scratch
`)},
{Data: []byte(`
when:
event:
- push
pipeline:
xxx:
image: scratch
`)},
{Data: []byte(`
pipeline:
build:
image: scratch
`)},
},
}
buildItems, err := b.Build()
if err != nil {
t.Fatal(err)
}
if len(buildItems) != 2 {
t.Fatal("Should have generated 2 buildItems")
}
}
func TestZeroSteps(t *testing.T) {
t.Parallel()