Release process: changelog generation and docs (#3446)

* Remove extra pyprojects and update changelog docs

* Add script for rendering the changelog

* Add docs for how to release the spec

* Move legacy changelogs out of the way
pull/977/head
Travis Ralston 3 years ago committed by GitHub
parent 61ac438871
commit e8674688e6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

1
.gitignore vendored

@ -14,3 +14,4 @@ _rendered.rst
/.vscode/ /.vscode/
/.idea/ /.idea/
/spec/ /spec/
changelogs/rendered.*

@ -1,60 +1,15 @@
<!-- Note: This is a markdown file so the build script's RST processing doesn't grab it -->
# Changelogs # Changelogs
[Towncrier](https://github.com/hawkowl/towncrier) is used to manage the changelog and [Towncrier](https://github.com/hawkowl/towncrier) is used to manage the changelog and
keep it up to date. Because of this, updating a changelog is really easy. keep it up to date. Because of this, updating a changelog is really easy.
## How to update a changelog when releasing an API ## Generating the changelog
1. Ensure you're in your Python 3 virtual environment
2. `cd` your way to the API you're releasing (eg: `cd changelogs/client_server`)
3. Run `towncrier --version "r0.4.0" --name "client-server" --yes` substituting the
variables as approprite. Note that `--name` is required although the value is ignored.
4. Commit the changes and finish the release process.
## How to prepare a changelog for a new API
For this example, we're going to pretend that the `server_server` API doesn't exist.
1. Create the file `changelogs/server_server.rst`
2. Create the folder `changelogs/server_server`
3. In the new folder, create a `pyproject.toml` file with these contents:
```toml
[tool.towncrier]
filename = "../server_server.rst"
directory = "newsfragments"
issue_format = "`#{issue} <https://github.com/matrix-org/matrix-doc/issues/{issue}>`_"
title_format = "{version}"
[[tool.towncrier.type]]
directory = "breaking"
name = "Breaking Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "Deprecations"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "New Endpoints"
showcontent = true
[[tool.towncrier.type]] Please see the [release docs](../meta/releasing.md) for more information.
directory = "removal"
name = "Removed Endpoints"
showcontent = true
[[tool.towncrier.type]] ## Creating a new changelog
directory = "feature"
name = "Backwards Compatible Changes"
showcontent = true
[[tool.towncrier.type]] There are a few places you'll have to update:
directory = "clarification" * `/layouts/shortcodes/changelog/changelog-changes.html` to account for the new changelog.
name = "Spec Clarifications" * `/scripts/generate-changelog.sh` to render the changelog for releases.
showcontent = true * Supporting documentation such as the contributing guidelines.
```
4. Create a `.gitignore` in `changelogs/server_server/newsfragments` with the contents `!.gitignore`

@ -1,35 +0,0 @@
[tool.towncrier]
filename = "../application_service.rst"
directory = "newsfragments"
issue_format = "`#{issue} <https://github.com/matrix-org/matrix-doc/issues/{issue}>`_"
title_format = "{version}"
[[tool.towncrier.type]]
directory = "breaking"
name = "Breaking Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "Deprecations"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "New Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "removal"
name = "Removed Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "feature"
name = "Backwards Compatible Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "clarification"
name = "Spec Clarifications"
showcontent = true

@ -1,35 +0,0 @@
[tool.towncrier]
filename = "../client_server.rst"
directory = "newsfragments"
issue_format = "`#{issue} <https://github.com/matrix-org/matrix-doc/issues/{issue}>`_"
title_format = "{version}"
[[tool.towncrier.type]]
directory = "breaking"
name = "Breaking Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "Deprecations"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "New Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "removal"
name = "Removed Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "feature"
name = "Backwards Compatible Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "clarification"
name = "Spec Clarifications"
showcontent = true

@ -0,0 +1,16 @@
<!--
This is a header file for the generated changelog.
Variables:
VERSION = Replaced by the version number (eg: v1.2)
DATE = Replaced by the date (eg: April 01, 2021)
-->
## VERSION
<table class="release-info">
<tr><th>Git commit</th><td><a href="https://github.com/matrix-org/matrix-doc/tree/VERSION">https://github.com/matrix-org/matrix-doc/tree/VERSION</a></td>
<tr><th>Release date</th><td>DATE</td>
</table>
<!-- Intentionally blank line to ensure headers work in the concatenated changelog -->

@ -1,35 +0,0 @@
[tool.towncrier]
filename = "../identity_service.rst"
directory = "newsfragments"
issue_format = "`#{issue} <https://github.com/matrix-org/matrix-doc/issues/{issue}>`_"
title_format = "{version}"
[[tool.towncrier.type]]
directory = "breaking"
name = "Breaking Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "Deprecations"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "New Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "removal"
name = "Removed Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "feature"
name = "Backwards Compatible Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "clarification"
name = "Spec Clarifications"
showcontent = true

@ -1,35 +0,0 @@
[tool.towncrier]
filename = "../push_gateway.rst"
directory = "newsfragments"
issue_format = "`#{issue} <https://github.com/matrix-org/matrix-doc/issues/{issue}>`_"
title_format = "{version}"
[[tool.towncrier.type]]
directory = "breaking"
name = "Breaking Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "Deprecations"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "New Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "removal"
name = "Removed Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "feature"
name = "Backwards Compatible Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "clarification"
name = "Spec Clarifications"
showcontent = true

@ -0,0 +1,39 @@
[tool.towncrier]
version = "unused"
filename = "../rendered.md"
issue_format = "[#{issue}](https://github.com/matrix-org/matrix-doc/issues/{issue})"
title_format = "### {name}" # Matches rendered spec, even if awkward
underlines = " " # 3 spaces intentionally to hide RST headings
# Note: The names below have the <strong> tag built-in so the rendered spec *and* the generated
# changelog can benefit from sane headings.
[[tool.towncrier.type]]
directory = "breaking"
name = "<strong>Breaking Changes</strong>"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "<strong>Deprecations</strong>"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "<strong>New Endpoints</strong>"
showcontent = true
[[tool.towncrier.type]]
directory = "removal"
name = "<strong>Removed Endpoints</strong>"
showcontent = true
[[tool.towncrier.type]]
directory = "feature"
name = "<strong>Backwards Compatible Changes</strong>"
showcontent = true
[[tool.towncrier.type]]
directory = "clarification"
name = "<strong>Spec Clarifications</strong>"
showcontent = true

@ -1,35 +0,0 @@
[tool.towncrier]
filename = "../server_server.rst"
directory = "newsfragments"
issue_format = "`#{issue} <https://github.com/matrix-org/matrix-doc/issues/{issue}>`_"
title_format = "{version}"
[[tool.towncrier.type]]
directory = "breaking"
name = "Breaking Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "deprecation"
name = "Deprecations"
showcontent = true
[[tool.towncrier.type]]
directory = "new"
name = "New Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "removal"
name = "Removed Endpoints"
showcontent = true
[[tool.towncrier.type]]
directory = "feature"
name = "Backwards Compatible Changes"
showcontent = true
[[tool.towncrier.type]]
directory = "clarification"
name = "Spec Clarifications"
showcontent = true

@ -8,6 +8,8 @@ weight: 1000
{{% changelog/changelog-changes %}} {{% changelog/changelog-changes %}}
<!-- DO NOT REMOVE OR CHANGE - Release script puts next release here -->
<h2 id="historical-versions" class="no-numbers">Historical versions</h2> <h2 id="historical-versions" class="no-numbers">Historical versions</h2>
Before version 1.1, versioning was applied at the level of individual API specifications. This section includes links to these versions of the APIs. Before version 1.1, versioning was applied at the level of individual API specifications. This section includes links to these versions of the APIs.

@ -40,7 +40,7 @@
{{ define "partials/render-api-changes" }} {{ define "partials/render-api-changes" }}
<h3 id="{{.id}}">{{ .title }}</h3> <h3 id="{{.id}}">{{ .title }}</h3>
{{ $api_path := .path }} {{ $api_path := .path }}
{{ $config_file := path.Join $api_path "pyproject.toml" }} {{ $config_file := path.Join $api_path ".." "pyproject.toml" }}
{{ $config := readFile $config_file | transform.Unmarshal }} {{ $config := readFile $config_file | transform.Unmarshal }}
{{ $news_path := path.Join $api_path "newsfragments" }} {{ $news_path := path.Join $api_path "newsfragments" }}
{{ partial "render-newsfragments" (dict "config" $config "news_path" $news_path )}} {{ partial "render-newsfragments" (dict "config" $config "news_path" $news_path )}}
@ -74,7 +74,7 @@
{{ range $config.tool.towncrier.type }} {{ range $config.tool.towncrier.type }}
{{ $changes_of_type := (index $types .directory) }} {{ $changes_of_type := (index $types .directory) }}
{{ if $changes_of_type }} {{ if $changes_of_type }}
<li><strong>{{.name}}</strong> <li>{{.name | safeHTML}}
<p><ul> <p><ul>
{{ range $changes_of_type }} {{ range $changes_of_type }}
<li><a href="https://github.com/matrix-org/matrix-doc/issues/{{.ticket}}"><strong>{{ .ticket }}: </strong></a>{{ .description | markdownify }}</li> <li><a href="https://github.com/matrix-org/matrix-doc/issues/{{.ticket}}"><strong>{{ .ticket }}: </strong></a>{{ .description | markdownify }}</li>

@ -0,0 +1,2 @@
{{ $partial := .Params.p }}
{{ partial $partial . }}

@ -0,0 +1,67 @@
# How to release the specification
The whole specification is now released as a single unit/artifact. This document is
the process for releasing the specification and a description of how the (public)
machinery works.
## Prerequisites / preparation
First, can we even release the spec? This stage is mostly preperation work needed
to ensure a consistent and reliable specification.
1. Ensure `main` is committed with all the spec changes you expect to be there.
2. Review the changelog to look for typos, wording inconsistencies, or lines which
can be merged. For example, "Fix typos" and "Fix spelling" can be condensed to
"Fix various typos throughout the specification".
3. Do a quick skim to ensure changelogs reference the MSCs which brought the changes
in. They should be linked to the GitHub MSC PR (not the markdown document).
## The release
Assuming the preparation work is complete, all that remains is the actual specification
release. This is done directly on `main`, though local branching for safety is also
welcome.
1. Update the `params.version` section of `config.toml` to use the following template:
```toml
[params.version]
status = "stable"
current_version_url = "https://spec.matrix.org/latest"
# This will be the spec version you're releasing. If that's v1.2, then `major = "1"`
# and `minor = "2"`
major = "1"
minor = "2"
# Today's date. Please use the format implied here for consistency.
release_date = "October 01, 2021"
```
2. Commit the changes.
3. Tag `main` with the spec release with a format of `v1.2` (if releasing Matrix 1.2).
4. Push `main` and the tag.
5. GitHub Actions will run its build steps. Wait until these are successful. If fixes
need to be made to repair the pipeline or spec build, delete and re-tag the release.
6. Generate the changelog. This is done *after* the tagging to ensure the rendered
changelog makes sense.
1. Activate your python virtual environment.
2. Run `./scripts/generate-changelog.sh v1.2 "October 01, 2021"` (using the correct
version number and same `release_date` format from the hugo config).
3. Commit the result.
7. Create a new release on GitHub from the newly created tag.
* The title should be just "v1.2" (for example).
* The description should be a copy/paste of the changelog. The generated changelog
will be at `content/partials/changelogs/v1.2.md` - copy/paste verbatim.
* Upload the artifacts of the GitHub Actions build for the release to the GitHub
release as artifacts themselves. This should be the tarball that got deployed
to spec.matrix.org.
8. Commit a reversion to `params.version` of `config.toml` on `main`:
```toml
[params.version]
status = "unstable"
current_version_url = "https://spec.matrix.org/latest"
# major = "1"
# minor = "2"
# release_date = "October 01, 2021"
```
9. Push pending commits and ensure the unstable spec updates accordingly from the
GitHub Actions pipeline.

@ -1,52 +0,0 @@
# How to release a specification
There are several specifications that belong to matrix, such as the client-server
specification, server-server specification, and identity service specification. Each
of these gets released independently of each other with their own version numbers.
Once a specification is ready for release, a branch should be created to track the
changes in and to hold potential future hotfixes. This should be the name of the
specification (as it appears in the directory structure of this project) followed
by "release-" and the release version. For example, if the Client-Server Specification
was getting an r0.4.0 release, the branch name would be `client_server/release-r0.4.0`.
*Note*: Historical releases prior to this process may or may not have an appropriate
release branch. Releases after this document came into place will have an appropriate
branch.
The remainder of the process is as follows:
1. Activate your Python 3 virtual environment.
1. Having checked out the new release branch, navigate your way over to `./changelogs`.
1. Follow the release instructions provided in the README.md located there.
1. Update any version/link references across all specifications.
1. From translate.riot.im, push repository changes and merge the subsequent PR.
1. Run `./scripts/i18n.py` to ensure all translatable files are up to date.
1. Generate the specification using `./scripts/gendoc.py`, specifying all the
API versions at the time of generation. For example: `./scripts/gendoc.py -c r0.4.0 -s r0.1.0 -i r0.1.0 #etc`
1. PR the changes to the matrix-org/matrix.org repository (for historic tracking).
* This is done by making a PR to the `unstyled_docs/spec` folder for the version and
specification you're releasing.
* Don't forget to symlink the new release as `latest`.
* For the client-server API, don't forget to generate the swagger JSON by using
`./scripts/dump-swagger.py -c r0.4.0`. This will also need symlinking to `latest`.
1. Commit the changes and PR them to master. **Wait for review from the spec core team.**
* Link to your matrix-org/matrix.org so both can be reviewed at the same time.
1. Tag the release with the format `client_server/r0.4.0`.
1. Perform a release on GitHub to tag the release.
1. Yell from the mountaintop to the world about the new release.
### Creating a release for a brand-new specification
Some specifications may not have ever had a release, and therefore need a bit more work
to become ready.
1. Activate your Python 3 virtual environment.
1. Having checked out the new release branch, navigate your way over to `./changelogs`.
1. Follow the "new changelog" instructions provided in the README.md located there.
1. Open the specification RST file and make some changes:
* Using a released specification as a template, update the changelog section.
* Use the appropriate changelog variable in the RST.
1. Create/define the appropriate variables in `gendoc.py`.
1. Update `targets.yml`.
1. Update any version/link references across all specifications.
1. Follow the regular release process.

@ -0,0 +1,35 @@
# /bin/bash
# Usage: ./generate.sh v1.2 "April 01, 2021"
set -e
MAGIC_STRING="<!-- DO NOT REMOVE OR CHANGE - Release script puts next release here -->"
cd changelogs
# Pre-cleanup just in case it wasn't done on the last run
rm -f rendered.*
# Reversed order so the push gateway ends up on the bottom
towncrier --name "Push Gateway API" --dir "./push_gateway" --config "./pyproject.toml" --yes
towncrier --name "Identity Service API" --dir "./identity_service" --config "./pyproject.toml" --yes
towncrier --name "Application Service API" --dir "./application_service" --config "./pyproject.toml" --yes
towncrier --name "Server-Server API" --dir "./server_server" --config "./pyproject.toml" --yes
towncrier --name "Client-Server API" --dir "./client_server" --config "./pyproject.toml" --yes
# Prepare the header
cp header.md rendered.header.md
sed -i "s/VERSION/$1/g" rendered.header.md
sed -i "s/DATE/$2/g" rendered.header.md
cat rendered.header.md rendered.md > rendered.final.md
# Remove trailing whitespace (such as our intentionally blank RST headings)
sed -i "s/[ ]*$//" rendered.final.md
# Put the changelog in place
mv rendered.final.md ../layouts/partials/changelogs/$1.md
sed -i "s/$MAGIC_STRING/$MAGIC_STRING\n{{% changelog\\/changelog-rendered p=\"changelogs\\/$1.md\" %}}/" ../content/changelog.md
# Cleanup
rm -v rendered.*

@ -7,4 +7,4 @@ jsonschema >= 2.6.0, < 3.0.0
PyYAML >= 3.12 PyYAML >= 3.12
requests >= 2.18.4 requests >= 2.18.4
towncrier == 18.6.0 towncrier == 21.9.0rc1

Loading…
Cancel
Save