From 0206cc6c4457e6ca91f7a2cee1dee89e9df8b40f Mon Sep 17 00:00:00 2001 From: Javier Goizueta Date: Thu, 10 Mar 2016 19:13:46 +0100 Subject: [PATCH] Update documentation --- CONTRIBUTING.md | 84 ---------------------------------- README.md | 77 ++++++++++++++++++------------- src/pg/Makefile | 2 - src/py/README.md | 117 ++++++++++++++++++++++++++++++----------------- 4 files changed, 118 insertions(+), 162 deletions(-) delete mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 63670ca..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,84 +0,0 @@ -# Contributing guide - -## How to add new functions - -Try to put as little logic in the SQL extension as possible and -just use it as a wrapper to the Python module functionality. - -Once a function is defined it should never change its signature in subsequent -versions. To change a function's signature a new function with a different -name must be created. - -### Version numbers - -The version of both the SQL extension and the Python package shall -follow the [Semantic Versioning 2.0](http://semver.org/) guidelines: - -* When backwards incompatibility is introduced the major number is incremented -* When functionally is added (in a backwards-compatible manner) the minor number - is incremented -* When only fixes are introduced (backwards-compatible) the patch number is - incremented - -### Python Package - -... - -### SQL Extension - -* Generate a **new subfolder version** for `sql` and `test` folders to define - the new functions and tests - - Use symlinks to avoid file duplication between versions that don't update them - - Add new files or modify copies of the old files to add new functions or - modify existing functions (remember to rename a function if the signature - changes) - - Add or modify the corresponding documentation files in the `doc` folder. - Since we expect to have highly technical functions here, an extense - background explanation would be of great help to users of this extension. - - Create tests for the new functions/behaviour - -* Generate the **upgrade and downgrade files** for the extension - -* Update the control file and the Makefile to generate the complete SQL - file for the new created version. After running `make` a new - file `crankshaft--X.Y.Z.sql` will be created for the current version. - Additional files for migrating to/from the previous version A.B.Z should be - created: - - `crankshaft--X.Y.Z--A.B.C.sql` - - `crankshaft--A.B.C--X.Y.Z.sql` - All these new files must be added to git and pushed. - -* Update the public docs! ;-) - -## Conventions - -# SQL - -Use snake case (i.e. `snake_case` and not `CamelCase`) for all -functions. Prefix functions intended for public use with `cdb_` -and private functions (to be used only internally inside -the extension) with `_cdb_`. - -# Python - -... - -## Testing - -Running just the Python tests: - -``` -(cd python && make test) -``` - -Installing the Extension and running just the PostgreSQL tests: - -``` -(cd pg && sudo make install && PGUSER=postgres make installcheck) -``` - -Installing and testing everything: - -``` -sudo make install && PGUSER=postgres make testinstalled -``` diff --git a/README.md b/README.md index c8869e7..c9dd529 100644 --- a/README.md +++ b/README.md @@ -8,19 +8,32 @@ CartoDB Spatial Analysis extension for PostgreSQL. * *src* source code * - *src/pg* contains the PostgreSQL extension source code * - *src/py* Python module source code -* *release* reselesed versions +* *release* reseleased versions ## Requirements * pip, virtualenv, PostgreSQL -* python-scipy system package +* python-scipy system package (see src/py/README.md) # Working Process ## Development Work in `src/pg/sql`, `src/py/crankshaft`; -use topic branch. +use a topic branch. See src/py/README.md +for the procedure to work with the Python local environment. + +Take into account: + +* Always remember to add tests for any new functionality + documentation. +* Add or modify the corresponding documentation files in the `doc` folder. + Since we expect to have highly technical functions here, an extense + background explanation would be of great help to users of this extension. +* Convention: Use snake case (i.e. `snake_case` and not `CamelCase`) for all + functions. Prefix functions intended for public use with `cdb_` + and private functions (to be used only internally inside + the extension) with `_cdb_`. Update local installation with `sudo make install` (this will update the 'dev' version of the extension in 'src/pg/') @@ -42,50 +55,48 @@ should be dropped manually before the update. If the extension has not previously been installed in a database we can: -Add tests... - * `CREATE EXTENSION crankshaft WITH VERSION 'dev';` -Test +Once the tests are succeeding a new Pull-Request can be created. +CI-tests must be checked to be successfull. + +Before merging a topic branch peer code reviewing of the code is a must. -Commit, push, create PR, wait for CI tests, CR, ... ## Release -To release current development version -(working directory should be clean in dev branch) +The release process of a new version of the extension +shall by performed by the designated *Release Manager*. -(process to be gradually automated) +Note that we expect to gradually automate this process. -For backwards compatible changes (no return value, num of arguments, etc. changes...) -new version number increasing either patch level (no new functionality) -or minor level (new functionality) => 'X.Y.Z'. -Update version in src/pg/crankshaft.control -Copy release/crankshaft--current.sql to release/crankshaft--X.Y.Z.sql -Prepare incremental downgrade, upgrade scripts.... +Having checkout the topic branch of the PR to be released: -Python: ... +The version number in `pg/cranckshaft.control` must first be updated. +To do so [Semantic Versioning 2.0](http://semver.org/) is in order. -Install the new release +We now will explain the process for the case of backwards-compatible +releases (updating the minor or patch version numbers). -`make install-release` +TODO: document the complex case of major releases. -Test the new release +The next command must be executed to produce the main installation +script for the new release, `release/cranckshaft--X.Y.Z.sql`. -`make test-release` +``` +make release +``` -Push the release +Then, the release manager shall produce upgrade and downgrade scripts +to migrate to/from the previous release. In the case of minor/patch +releases this simply consist in extracting the functions that have changed +and placing them in the proper `release/cranckshaft--X.Y.Z--A.B.C.sql` +file. -Wait for CI tests +TODO: configure the local enviroment to be used by the release; +currently should be directory `src/py/X.Y.Z`, but this must be fixed; +a possibility to explore is to use the `cdb_conf` table. -Merge into master +TODO: testing procedure for the new release -Deploy: install extension and python to production hosts, -update extension in databases (limited to team users, data observatory, ...) - -Release manager role: ... - -.sql release scripts -commit -tests: staging.... -merge, tag, deploy... +TODO: push, merge, tag, deploy procedures. diff --git a/src/pg/Makefile b/src/pg/Makefile index ded7121..ed65eba 100644 --- a/src/pg/Makefile +++ b/src/pg/Makefile @@ -46,5 +46,3 @@ EXTVERSION = $(shell grep default_version $(EXTENSION).control | sed -e "s/def release: ../release/$(EXTENSION).control cp $(EXTENSION)--dev.sql $(EXTENSION)--$(EXTVERSION).sql - # pending: create upgrade/downgrade scripts, - # commit, push, tag.... diff --git a/src/py/README.md b/src/py/README.md index 03926c3..d55b7d7 100644 --- a/src/py/README.md +++ b/src/py/README.md @@ -20,80 +20,111 @@ nosetests test/ --- -### Sample session with virtualenv +We have two possible approaches being considered as to how manage +the Python virtual environment: using a pure virtual enviroment +or combine it with some system packages that include depencencies +for the *hard-to-compile* packages (and pin them in somewhat old versions). + +### Alternative A: pure virtual environment + +In this case we will install all the packages needed in the +virtual environment. +This will involve, specially for the numerical packages compiling +and linking code that uses a number of third party libraries, +and requires having theses depencencies solved for the production +environments. + #### Create and use a virtual env +We'll use a virtual enviroment directory `dev` +under the `src/pg` directory. + # Create the virtual environment for python - $ virtualenv myenv + $ virtualenv dev # Activate the virtualenv - $ source myenv/bin/activate + $ source dev/bin/activate # Install all the requirements # expect this to take a while, as it will trigger a few compilations - (myenv) $ pip install -r requirements.txt + (dev) $ pip install -r requirements.txt # Add a new pip to the party - (myenv) $ pip install pandas + (dev) $ pip install pandas #### Test the libraries with that virtual env + ##### Test numpy library dependency: import numpy numpy.test('full') -output: -``` -====================================================================== -ERROR: test_multiarray.TestNewBufferProtocol.test_relaxed_strides ----------------------------------------------------------------------- -Traceback (most recent call last): - File "/home/ubuntu/www/crankshaft/src/py/dev2/lib/python2.7/site-packages/nose/case.py", line 197, in runTest - self.test(*self.arg) - File "/home/ubuntu/www/crankshaft/src/py/dev2/lib/python2.7/site-packages/numpy/core/tests/test_multiarray.py", line 5366, in test_relaxed_strides - fd.write(c.data) -TypeError: 'buffer' does not have the buffer interface - ----------------------------------------------------------------------- -Ran 6153 tests in 84.561s - -FAILED (KNOWNFAIL=3, SKIP=5, errors=1) -Out[2]: -``` - -NOTE: this is expected to fail with Python 2.7.3, which is the version embedded in our postgresql installation - - ##### Run scipy tests import scipy scipy.test('full') -Output: -``` -Ran 21562 tests in 321.610s - -OK (KNOWNFAIL=130, SKIP=1840) -Out[2]: -``` -Ok, this looks good... - ##### Testing pysal + See [http://pysal.readthedocs.org/en/latest/developers/testing.html] +This will require putting this into `dev/lib/python2.7/site-packages/setup.cfg`: + +``` +[nosetests] +ignore-files=collection +exclude-dir=pysal/contrib + +[wheel] +universal=1 +``` + +And copying some files before executing the tests: +(we'll use a temporary directory from where the tests will be executed because +some tests expect some files in the current directory). Next must be executed +from + +``` +cp dev/lib/python2.7/site-packages/pysal/examples/geodanet/* dev/local/lib/python2.7/site-packages/pysal/examples +mkdir -p test_tmp && cd test_tmp && cp ../dev/lib/python2.7/site-packages/pysal/examples/geodanet/* ./ +``` + +Then, execute the tests with: + import pysal import nose nose.runmodule('pysal') -``` -Ran 537 tests in 42.182s -FAILED (errors=48, failures=17) -An exception has occurred, use %tb to see the full traceback. +### Alternative B: using some packaged modules + +This option avoids troublesome compilations/linkings, at the cost +of freezing some module versions as available in system packages, +namely numpy 1.6.1 and scipy 0.9.0. (in turn, this implies +the most recent version of PySAL we can use is 1.9.1) + + +TODO: to use this alternative the python-scipy package must be +installed (this will have to be included in server provisioning) + +``` +apt-get install -y python-scipy ``` -This doesn't look good... Taking a deeper look at the failures, many have the `IOError: [Errno 2] No such file or directory: 'streets.shp'` +#### Create and use a virtual env -In the source code, there's the following [config](https://github.com/pysal/pysal/blob/master/setup.cfg) that seems to be missing in the pip package. By copying it to `lib/python2.7/site-packages` within the environment, it goes down to 17 failures. +We'll use a `dev` enviroment as before, but will configure it to +use also system modules. -The remaining failures don't look good. I see two types: precision calculation errors and arrays/matrices missing 1 element when comparing... TODO: FIX this + + # Create the virtual environment for python + $ virtualenv --system-site-packages dev + + # Activate the virtualenv + $ source dev/bin/activate + + # Install all the requirements + # expect this to take a while, as it will trigger a few compilations + (dev) $ pip install -I ./crankshaft + +Then we can proceed to testing as in Alternative A.