Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Summary

Unify the charts and operator repositories by moving the contents of charts into the operator. Also, release the charts whenever the operator is released.

Motivation

Changes to the operator frequently require changes to be made to the charts. Some features require simultaneous changes in the operator, charts and mirrord (3 separate PRs) as well as docs and config docs (2 more separate PRs). This:

  1. Makes local development and testing more difficult
  2. Increases admin time overhead for PR authors and reviewers
  3. Can result in the repositories being out of sync with eachother
  4. Is annoying

COR-1109 reduces the number of PRs to 4, and by merging the charts and the operator, it can be reduced to 3.

Problem 1: Hidden dependencies

For example, take COR-1149, which required 5 separate PRs. In this case, the feature (which was a customer request) was announced as released before it was present in the charts and before user documentation was available. This was due to the charts being released without an operator release, which resulted in a misleading changelog entry in the charts. The hidden dependency here was that the changes in the charts release required the operator to be released.

Problem 2: Misleading changelogs/ when is the feature available to use?

The operator changelog is updated upon operator release automatically, but until the charts are released the latest operator features are not available to users. Since we publish the operator changelog in our docs, users may see features that are not yet available fo use. By forcing an operator release to release the charts, we can ensure released operator features are available immediately.

Guide-level explanation

The contents of metalbear-co/charts is a copy of metalbear-co/operator/charts. Changes are made to metalbear-co/operator/charts and then copied to metalbear-co/charts when the operator is released.

  • For developers writing code changes this means that changes to the charts occur in the same commit as the related changes to the operator, in metalbear-co/operator.
  • For developers performing a release this means that you must only make one release commit, in metalbear-co/operator.
  • For non-developers at MetalBear this means it is easy to tell when operator features are released AND available to customers, which is useful if a feature has been requested.
  • For users and customers this means that features in the operator changelog will always be available in the latest chart release.

Reference-level explanation

Assumptions:

  • Releasing the operator should always trigger a charts release
    • Impact: developers will always have to update all changelogs and operator dependencies even when charts do not change otherwise
  • Code will not be committed to the charts repository UNLESS it is pushed by the operator release, ever. Otherwise, the repositories would have to sync both ways
    • Impact: it must be very clear or enforced that code cannot be committed directly to the charts repo

Repository structure:

metalbear-co/operator

  • visibility: private
  • allows PRs?: yes
  • main HEAD points to: most recent release commit or unreleased changes
  • code changes come from: developer PRs

metalbear-co/charts

  • visibility: public
  • allows PRs?: no
  • main HEAD points to: most recent (charts) release commit
  • code changes come from: pushed automatically from metalbear-co/operator/charts

New release process:

  • To release the operator and charts:
    • Create a release PR on the operator repository as before, bumping the versions in Cargo.toml and each chart.yaml, all to the same new version
      • Update the changelog as before (the operator and charts now all share a changelog)
    • When merged, create a GitHub release tagged with the new version, containing the changelog with the Internal section omitted
    • Check that the operator is released and the charts are released with up to date code

Release flow:

  • Release PR opened, CI runs with extra checks for release
  • Release commit merges, GitHub release is created with a tag
  • Tag push triggers a workflow that:
    • Triggers the operator release workflow
    • Pushes charts dir contents to charts repo
    • Triggers a charts release in the charts repo

Implementation order

To minimise impact if something goes wrong:

    1. Copy the charts source code to the operator repo, within its own directory
    • Trigger charts CI from the operator CI when files in charts dir change
      • Move charts CI to the operator repository and run using the current (unreleased) operator image
    • Adjust files changed checks to account for charts/ non charts files
    • Unify chart version for operator and license-server charts (allows for single charts changelog)
    1. In the operator repo, add a new job that will:
    • Push current state of charts source code to the charts repository main
    • Trigger a release there
    1. Move any open PRs/issues of ours from the charts to the operator repo, leave any PRs/issues opened by the users
    • Stop new PRs from being opened
    1. Adjust operator release job to also trigger job from 2.
    • Operator release must finish before charts release triggered, so the charts are able to fetch operator (and if operator release fails, charts do not get released)
    • The trigger job from 2. should run when a semver version tag is created (same as current operator release)
    1. Add an option to manually trigger job from 2., without operator release
    • Add check to require up-to-date changelog in charts (if changelogs are not merged)
    1. Update the operator release.sh script and CONTRIBUTING.md to include all the manual charts release steps
    • Update release.sh script and CONTRIBUTING.md to reflect new changelog layout

Drawbacks

  • This change has the potential to break operator and/or charts releases if there are bugs
    • Customers will be directly affected if something is released incorrectly (e.g. incorrect dependencies) or if releases cannot be performed at all
  • Operator CI/ actions may be made more complex
    • For example, separate changelog checks will be necessary for the operator and charts
    • Changed files checks for the operator may need to be altered to exclude the charts directory

Unresolved questions

Resolve before implementation

  • Will there any negative changes to development experience for the charts or operator?
  • Will the 3 changelogs be kept separate? Can we combine them into 1?
    • How will this work during charts only releases?
    • Either: releases are always operator and charts AND changelogs can be unified
    • or releases can be charts only or both AND changelogs must stay separate
  • Would there ever be a reason for code to be pushed to the charts repository when a release is not happening?
    • For example, if there are changes to the (public) charts README.md
    • Can we account for/ allow/ disallow this?
  • How do we ensure that updates to the charts code that merge after a release PR dont get included in code pushed to the charts repository when performing a charts-only release?
    • ie. we only push code up to and including release commit (may require specific commit name)
  • How do we avoid charts CI failures on release PRs for the operator and charts?
    • The charts CI will attempt to install the chart being released, which depends on the operator version being released, which doesn’t exist yet
    • It is safe to disable the charts CI on release PRs (there are no extra checks in the charts CI for release PRs)?

Future possibilities

  • Fully automate operator releases
    • Ideally, developers wouldn’t have to create a release commit, wait for it to merge and create a GitHub release
    • If we could automate creation of the release commit and GitHub release with tag, we could have a manual dispatch only job to perform release
      • Would we need a way to point to the specific commit to be released?
  • Move to a monorepo: eventually, we could move all of our code into one repo
    • When considering this in the past, issues raised included reluctance to use git submodules and concerns about code that generates or manages licenses being public
  • Release a nightly version of the operator and charts every day, with a version tag like X.Y.Z-nightly.yyyy.mm.dd