From 170260b7f5953731ff13eb89880dc99b864720b6 Mon Sep 17 00:00:00 2001 From: Rafa de la Torre Date: Fri, 12 Aug 2016 18:17:13 +0200 Subject: [PATCH 1/3] Revamp the dev process --- README.md | 77 ++++++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index ee72c18..683fb38 100644 --- a/README.md +++ b/README.md @@ -1,70 +1,65 @@ -# crankshaft [![Build Status](https://travis-ci.org/CartoDB/crankshaft.svg?branch=develop)](https://travis-ci.org/CartoDB/crankshaft) +# Crankshaft [![Build Status](https://travis-ci.org/CartoDB/crankshaft.svg?branch=develop)](https://travis-ci.org/CartoDB/crankshaft) CartoDB Spatial Analysis extension for PostgreSQL. ## Code organization -* *doc* documentation -* *src* source code -* - *src/pg* contains the PostgreSQL extension source code -* - *src/py* Python module source code -* *release* reseleased versions +* `doc/` documentation +* `src/` source code + - `pg/` contains the PostgreSQL extension source code + - `py/` Python module source code +* `release` reseleased versions ## Requirements -* pip, PostgreSQL -* python-scipy system package (see [src/py/README.md](https://github.com/CartoDB/crankshaft/blob/master/src/py/README.md)) +* PostgreSQL +* plpythonu and postgis extensions +* python-scipy system package (see [src/py/README.md](https://github.com/CartoDB/crankshaft/blob/develop/src/py/README.md)) -# Working Process -- Quickstart Guide +# Development Process -We distinguish two roles regarding the development cycle of crankshaft: +We distinguish two roles: * *developers* will implement new functionality and bugfixes into - the codebase and will request for new releases of the extension. -* A *release manager* will attend these requests and will handle - the release process. The release process is sequential: - no concurrent releases will ever be in the works. + the codebase. +* A *release manager* will handle the release process. -We use the default `develop` branch as the basis for development. -The `master` branch is used to merge and tag releases to be -deployed in production. +We use the branch `develop` as the main integration branch for development. The `master` is reserved to handle releases. -Developers shall create a new topic branch from `develop` for any new feature -or bugfix and commit their changes to it and eventually merge back into -the `develop` branch. When a new release is required a Pull Request -will be open against the `develop` branch. +The process is as follows: + +1. Create a new **topic branch** from `develop` for any new feature +or bugfix and commit their changes to it: +```shell +git fetch && git checkout -b my-cool-feature origin/develop +``` +1. Code, commit, push, repeat. +1. Write some **tests** for your feature or bugfix. +1. Create a pull request and mention relevant people for a **peer review**. +1. Address the comments and improvements you get from the peer review. +1. Mention `@CartoDB/dataservices` in the PR to get it merged into `develop`. + +In order for a pull request to be accepted, the following criteria should be met: +* The peer review should pass and no major issue should be left unaddressed. +* CI tests must pass (travis will take care of that). -The `develop` pull requests will be handled by the release manage, -who will merge into master where new releases are prepared and tagged. -The `master` branch is the sole responsibility of the release masters -and developers must not commit or merge into it. ## Development Guidelines For a detailed description of the development process please see -the [CONTRIBUTING.md](https://github.com/CartoDB/crankshaft/blob/master/CONTRIBUTING.md) guide. +the [CONTRIBUTING.md](https://github.com/CartoDB/crankshaft/blob/develop/CONTRIBUTING.md) guide. -Any modification to the source code (`src/pg/sql` for the SQL extension, -`src/py/crankshaft` for the Python package) shall always be done -in a topic branch created from the `develop` branch. -Tests, documentation and peer code reviewing are required for all -modifications. +## Testing -The tests (both for SQL and Python) are executed by running, -from the top directory: +The tests (both for SQL and Python) are executed by running, from the top directory: -``` +```shell sudo make install make test ``` -To request a new release, which will be handled by them -release manager, a Pull Request must be created in the `develop` -branch. - ## Release -The release and deployment process is described in the -[RELEASE.md](https://github.com/CartoDB/crankshaft/blob/master/RELEASE.md) guide and it is the responsibility of the designated -release manager. +The release process is described in the +[RELEASE.md](https://github.com/CartoDB/crankshaft/blob/develop/RELEASE.md) guide and is the responsibility of the designated *release manager*. From 065dc476b4021d19602621eaf0b95acc8f84ee80 Mon Sep 17 00:00:00 2001 From: Rafa de la Torre Date: Fri, 12 Aug 2016 18:34:13 +0200 Subject: [PATCH 2/3] Revamp the dev process --- CONTRIBUTING.md | 86 ++++++++++++++----------------------------------- README.md | 1 + 2 files changed, 26 insertions(+), 61 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 42385dc..ed694bd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,10 +1,7 @@ # Development process -Please read the Working Process/Quickstart Guide in [README.md](https://github.com/CartoDB/crankshaft/blob/master/README.md) first. - For any modification of crankshaft, such as adding new features, -refactoring or bug-fixing, topic branch must be created out of the `develop` -branch and be used for the development process. +refactoring or bugfixing, a topic branch must be created out of the `develop`. Modifications are done inside `src/pg/sql` and `src/py/crankshaft`. @@ -14,80 +11,47 @@ Take into account: (inside `src/pg/test`, `src/py/crankshaft/test`) as well as to detect any bugs that are being fixed. * Add or modify the corresponding documentation files in the `doc` folder. - Since we expect to have highly technical functions here, an extense - background explanation would be of great help to users of this extension. -* Convention: snake case(i.e. `snake_case` and not `CamelCase`) - shall be used for all function names. - Prefix function names intended for public use with `cdb_` - and private functions (to be used only internally inside - the extension) with `_cdb_`. +* Naming conventions for function names: + - use `CamelCase` + - prefix "public" functions with `CDB_`. E.g: `CDB_SpatialMarkovTrend` + - prefix "private" functions with an underscore. E.g: `_CDB_MyObscureInternalImplementationDetail` Once the code is ready to be tested, update the local development installation with `sudo make install`. This will update the 'dev' version of the extension in `src/pg/` and make it available to PostgreSQL. -It will also install the python package (crankshaft) in a virtual -environment `env/dev`. - -The version number of the Python package, defined in -`src/pg/crankshaft/setup.py` will be overridden when -the package is released and always match the extension version number, -but for development it shall be kept as '0.0.0'. Run the tests with `make test`. -To use the python extension for custom tests, activate the virtual -environment with: - -``` -source envs/dev/bin/activate -``` - Update extension in a working database with: -* `ALTER EXTENSION crankshaft UPDATE TO 'current';` - `ALTER EXTENSION crankshaft UPDATE TO 'dev';` - -Note: we keep the current development version install as 'dev' always; -we update through the 'current' alias to allow changing the extension -contents but not the version identifier. This will fail if the -changes involve incompatible function changes such as a different -return type; in that case the offending function (or the whole extension) -should be dropped manually before the update. +```sql +ALTER EXTENSION crankshaft UPDATE TO 'current'; +ALTER EXTENSION crankshaft UPDATE TO 'dev'; +``` If the extension has not previously been installed in a database, it can be installed directly with: - -* `CREATE EXTENSION IF NOT EXISTS plpythonu;` - `CREATE EXTENSION IF NOT EXISTS postgis;` - `CREATE EXTENSION crankshaft WITH VERSION 'dev';` - -Note: the development extension uses the development python virtual -environment automatically. - -Before proceeding to the release process peer code reviewing of the code is -a must. +```sql +CREATE EXTENSION IF NOT EXISTS plpythonu; +CREATE EXTENSION IF NOT EXISTS postgis; +CREATE EXTENSION crankshaft WITH VERSION 'dev'; +``` Once the feature or bugfix is completed and all the tests are passing -a Pull-Request shall be created on the topic branch, reviewed by a peer -and then merged back into the `develop` branch when all CI tests pass. +a pull request shall be created, reviewed by a peer +and then merged back into the `develop` branch once all the CI tests pass. -When the changes in the `develop` branch are to be released in a new -version of the extension, a PR must be created on the `develop` branch. -The release manage will take hold of the PR at this moment to proceed -to the release process for a new revision of the extension. +## Relevant development targets in the Makefile -## Relevant development tasks available in the Makefile +```shell +# Show a short description of the available targets +make help -``` -* `make help` show a short description of the available targets - -* `sudo make install` will generate the extension scripts for the development - version ('dev'/'current') and install the python package into the - development virtual environment `envs/dev`. - Intended for use by developers. - -* `make test` will run the tests for the installed development extension. - Intended for use by developers. +# Generate the extension scripts and install the python package. +sudo make install + +# Run the tests against the installed extension. +make test ``` diff --git a/README.md b/README.md index 683fb38..9dad032 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,7 @@ git fetch && git checkout -b my-cool-feature origin/develop ``` 1. Code, commit, push, repeat. 1. Write some **tests** for your feature or bugfix. +1. Update the [NEWS.md](https://github.com/CartoDB/crankshaft/blob/develop/NEWS.md) doc. 1. Create a pull request and mention relevant people for a **peer review**. 1. Address the comments and improvements you get from the peer review. 1. Mention `@CartoDB/dataservices` in the PR to get it merged into `develop`. From 8953bf92ee85067a96aaa547a986fd7da145dc8d Mon Sep 17 00:00:00 2001 From: Rafa de la Torre Date: Fri, 12 Aug 2016 18:56:03 +0200 Subject: [PATCH 3/3] Update release process --- RELEASE.md | 107 +++++++++++++++-------------------------------------- 1 file changed, 29 insertions(+), 78 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 0db48a2..005557a 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,93 +1,44 @@ # Release & Deployment Process -Please read the Working Process/Quickstart Guide in README.md -and the Development guidelines in CONTRIBUTING.md. - The release process of a new version of the extension shall be performed by the designated *Release Manager*. -Note that we expect to gradually automate more of this process. - -Having checked PR to be released it shall be -merged back into the `master` branch to prepare the new release. - -The version number in `pg/cranckshaft.control` must first be updated. -To do so [Semantic Versioning 2.0](http://semver.org/) is in order. - -Thew `NEWS.md` will be updated. - -We now will explain the process for the case of backwards-compatible -releases (updating the minor or patch version numbers). - -TODO: document the complex case of major releases. - -The next command must be executed to produce the main installation -script for the new release, `release/cranckshaft--X.Y.Z.sql` and -also to copy the python package to `release/python/X.Y.Z/crankshaft`. - -``` +## Release steps +1. Make sure `develop` branch passes all the tests. +1. Merge `develop` into `master` +1. Update the version number in `src/pg/crankshaft.control`. +1. Generate the next release files with this command: +```shell make release ``` +1. Generate an upgrade path from the previous to the next release by copying the generated release file. E.g: +```shell +cp release/cranckshaft--X.Y.Z.sql release/cranckshaft--A.B.C--X.Y.Z.sql +``` +NOTE: you can rely on this thanks to the compatibility checks. TODO: automate this step [#94](https://github.com/CartoDB/crankshaft/issues/94) +1. Commit and push the generated files. +1. Tag the release: +``` +git tag -a X.Y.Z -m "Release X.Y.Z" +git push origin X.Y.Z +``` +1. Deploy and test in staging -Then, the release manager shall produce upgrade and downgrade scripts -to migrate to/from the previous release. In the case of minor/patch -releases this simply consist in extracting the functions that have changed -and placing them in the proper `release/cranckshaft--X.Y.Z--A.B.C.sql` -file. + +## Some remarks +* Version numbers shall follow [Semantic Versioning 2.0](http://semver.org/). +* CI tests will take care of **forward compatibility** of the extension at postgres level. +* **Major version changes** (breaking forward compatibility) are a major event and are out of the scope of this doc. They **shall be avoided as much as we can**. +* We will go forward, never backwards. **Generating upgrade paths automatically is easy** and we'll rely on the CI checks for that. + +## Deploy commands The new release can be deployed for staging/smoke tests with this command: - -``` +```shell sudo make deploy ``` -This will copy the current 'X.Y.Z' released version of the extension to -PostgreSQL. The corresponding Python extension will be installed in a -virtual environment in `envs/X.Y.Z`. - -It can be activated with: - -``` -source envs/X.Y.Z/bin/activate -``` - -But note that this is needed only for using the package directly; -the 'X.Y.Z' version of the extension will automatically use the -python package from this virtual environment. - -The `sudo make deploy` operation can be also used for installing -the new version after it has been released. - -To install a specific version 'X.Y.Z' different from the current one -(which must be present in `releases/`) you can: - -``` +To install a specific version 'X.Y.Z' different from the default one: +```shell sudo make deploy RELEASE_VERSION=X.Y.Z ``` - -TODO: testing procedure for the new release. - -TODO: procedure for staging deployment. - -TODO: procedure for merging to master, tagging and deploying -in production. - -## Relevant release & deployment tasks available in the Makefile - -``` -* `make help` show a short description of the available targets - -* `make release` will generate a new release (version number defined in - `src/pg/crankshaft.control`) into `release/`. - Intended for use by the release manager. - -* `sudo make deploy` will install the current release X.Y.Z from the - `release/` files into PostgreSQL and a Python virtual environment - `envs/X.Y.Z`. - Intended for use by the release manager and deployment jobs. - -* `sudo make deploy RELEASE_VERSION=X.Y.Z` will install specified version - previously generated in `release/` - into PostgreSQL and a Python virtual environment `envs/X.Y.Z`. - Intended for use by the release manager and deployment jobs. -```