cartodb4/crankshaft
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update documentation

Browse Source
This commit is contained in:
Javier Goizueta
2016-03-10 19:13:46 +01:00
parent b754ffe42a
commit 0206cc6c44
4 changed files with 118 additions and 162 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
CONTRIBUTING.md
View File

@@ -1,84 +0,0 @@
# Contributing guide
## How to add new functions
Try to put as little logic in the SQL extension as possible and
just use it as a wrapper to the Python module functionality.
Once a function is defined it should never change its signature in subsequent
versions. To change a function's signature a new function with a different
name must be created.
### Version numbers
The version of both the SQL extension and the Python package shall
follow the [Semantic Versioning 2.0](http://semver.org/) guidelines:
* When backwards incompatibility is introduced the major number is incremented
* When functionally is added (in a backwards-compatible manner) the minor number
is incremented
* When only fixes are introduced (backwards-compatible) the patch number is
incremented
### Python Package
...
### SQL Extension
* Generate a **new subfolder version** for `sql` and `test` folders to define
the new functions and tests
- Use symlinks to avoid file duplication between versions that don't update them
- Add new files or modify copies of the old files to add new functions or
modify existing functions (remember to rename a function if the signature
changes)
- Add or modify the corresponding documentation files in the `doc` folder.
Since we expect to have highly technical functions here, an extense
background explanation would be of great help to users of this extension.
- Create tests for the new functions/behaviour
* Generate the **upgrade and downgrade files** for the extension
* Update the control file and the Makefile to generate the complete SQL
file for the new created version. After running `make` a new
file `crankshaft--X.Y.Z.sql` will be created for the current version.
Additional files for migrating to/from the previous version A.B.Z should be
created:
- `crankshaft--X.Y.Z--A.B.C.sql`
- `crankshaft--A.B.C--X.Y.Z.sql`
All these new files must be added to git and pushed.
* Update the public docs! ;-)
## Conventions
# SQL
Use snake case (i.e. `snake_case` and not `CamelCase`) for all
functions. Prefix functions intended for public use with `cdb_`
and private functions (to be used only internally inside
the extension) with `_cdb_`.
# Python
...
## Testing
Running just the Python tests:
```
(cd python && make test)
```
Installing the Extension and running just the PostgreSQL tests:
```
(cd pg && sudo make install && PGUSER=postgres make installcheck)
```
Installing and testing everything:
```
sudo make install && PGUSER=postgres make testinstalled
```

77
README.md
View File

@@ -8,19 +8,32 @@ CartoDB Spatial Analysis extension for PostgreSQL.
* *src* source code
* - *src/pg* contains the PostgreSQL extension source code
* - *src/py* Python module source code
* *release* reselesed versions
* *release* reseleased versions
## Requirements
* pip, virtualenv, PostgreSQL
* python-scipy system package
* python-scipy system package (see src/py/README.md)
# Working Process
## Development
Work in `src/pg/sql`, `src/py/crankshaft`;
use topic branch.
use a topic branch. See src/py/README.md
for the procedure to work with the Python local environment.
Take into account:
* Always remember to add tests for any new functionality
documentation.
* Add or modify the corresponding documentation files in the `doc` folder.
Since we expect to have highly technical functions here, an extense
background explanation would be of great help to users of this extension.
* Convention: Use snake case (i.e. `snake_case` and not `CamelCase`) for all
functions. Prefix functions intended for public use with `cdb_`
and private functions (to be used only internally inside
the extension) with `_cdb_`.
Update local installation with `sudo make install`
(this will update the 'dev' version of the extension in 'src/pg/')
@@ -42,50 +55,48 @@ should be dropped manually before the update.
If the extension has not previously been installed in a database
we can:
Add tests...
* `CREATE EXTENSION crankshaft WITH VERSION 'dev';`
Test
Once the tests are succeeding a new Pull-Request can be created.
CI-tests must be checked to be successfull.
Before merging a topic branch peer code reviewing of the code is a must.
Commit, push, create PR, wait for CI tests, CR, ...
## Release
To release current development version
(working directory should be clean in dev branch)
The release process of a new version of the extension
shall by performed by the designated *Release Manager*.
(process to be gradually automated)
Note that we expect to gradually automate this process.
For backwards compatible changes (no return value, num of arguments, etc. changes...)
new version number increasing either patch level (no new functionality)
or minor level (new functionality) => 'X.Y.Z'.
Update version in src/pg/crankshaft.control
Copy release/crankshaft--current.sql to release/crankshaft--X.Y.Z.sql
Prepare incremental downgrade, upgrade scripts....
Having checkout the topic branch of the PR to be released:
Python: ...
The version number in `pg/cranckshaft.control` must first be updated.
To do so [Semantic Versioning 2.0](http://semver.org/) is in order.
Install the new release
We now will explain the process for the case of backwards-compatible
releases (updating the minor or patch version numbers).
`make install-release`
TODO: document the complex case of major releases.
Test the new release
The next command must be executed to produce the main installation
script for the new release, `release/cranckshaft--X.Y.Z.sql`.
`make test-release`
```
make release
```
Push the release
Then, the release manager shall produce upgrade and downgrade scripts
to migrate to/from the previous release. In the case of minor/patch
releases this simply consist in extracting the functions that have changed
and placing them in the proper `release/cranckshaft--X.Y.Z--A.B.C.sql`
file.
Wait for CI tests
TODO: configure the local enviroment to be used by the release;
currently should be directory `src/py/X.Y.Z`, but this must be fixed;
a possibility to explore is to use the `cdb_conf` table.
Merge into master
TODO: testing procedure for the new release
Deploy: install extension and python to production hosts,
update extension in databases (limited to team users, data observatory, ...)
Release manager role: ...
.sql release scripts
commit
tests: staging....
merge, tag, deploy...
TODO: push, merge, tag, deploy procedures.

2
src/pg/Makefile
View File

@@ -46,5 +46,3 @@ EXTVERSION = $(shell grep default_version $(EXTENSION).control | sed -e "s/def
release: ../release/$(EXTENSION).control
cp $(EXTENSION)--dev.sql $(EXTENSION)--$(EXTVERSION).sql
# pending: create upgrade/downgrade scripts,
# commit, push, tag....

117
src/py/README.md
View File

@@ -20,80 +20,111 @@ nosetests test/
---
### Sample session with virtualenv
We have two possible approaches being considered as to how manage
the Python virtual environment: using a pure virtual enviroment
or combine it with some system packages that include depencencies
for the *hard-to-compile* packages (and pin them in somewhat old versions).
### Alternative A: pure virtual environment
In this case we will install all the packages needed in the
virtual environment.
This will involve, specially for the numerical packages compiling
and linking code that uses a number of third party libraries,
and requires having theses depencencies solved for the production
environments.
#### Create and use a virtual env
We'll use a virtual enviroment directory `dev`
under the `src/pg` directory.
# Create the virtual environment for python
$ virtualenv myenv
$ virtualenv dev
# Activate the virtualenv
$ source myenv/bin/activate
$ source dev/bin/activate
# Install all the requirements
# expect this to take a while, as it will trigger a few compilations
(myenv) $ pip install -r requirements.txt
(dev) $ pip install -r requirements.txt
# Add a new pip to the party
(myenv) $ pip install pandas
(dev) $ pip install pandas
#### Test the libraries with that virtual env
##### Test numpy library dependency:
import numpy
numpy.test('full')
output:
```
======================================================================
ERROR: test_multiarray.TestNewBufferProtocol.test_relaxed_strides
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/www/crankshaft/src/py/dev2/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/ubuntu/www/crankshaft/src/py/dev2/lib/python2.7/site-packages/numpy/core/tests/test_multiarray.py", line 5366, in test_relaxed_strides
fd.write(c.data)
TypeError: 'buffer' does not have the buffer interface
----------------------------------------------------------------------
Ran 6153 tests in 84.561s
FAILED (KNOWNFAIL=3, SKIP=5, errors=1)
Out[2]: <nose.result.TextTestResult run=6153 errors=1 failures=0>
```
NOTE: this is expected to fail with Python 2.7.3, which is the version embedded in our postgresql installation
##### Run scipy tests
import scipy
scipy.test('full')
Output:
```
Ran 21562 tests in 321.610s
OK (KNOWNFAIL=130, SKIP=1840)
Out[2]: <nose.result.TextTestResult run=21562 errors=0 failures=0>
```
Ok, this looks good...
##### Testing pysal
See [http://pysal.readthedocs.org/en/latest/developers/testing.html]
This will require putting this into `dev/lib/python2.7/site-packages/setup.cfg`:
```
[nosetests]
ignore-files=collection
exclude-dir=pysal/contrib
[wheel]
universal=1
```
And copying some files before executing the tests:
(we'll use a temporary directory from where the tests will be executed because
some tests expect some files in the current directory). Next must be executed
from
```
cp dev/lib/python2.7/site-packages/pysal/examples/geodanet/* dev/local/lib/python2.7/site-packages/pysal/examples
mkdir -p test_tmp && cd test_tmp && cp ../dev/lib/python2.7/site-packages/pysal/examples/geodanet/* ./
```
Then, execute the tests with:
import pysal
import nose
nose.runmodule('pysal')
```
Ran 537 tests in 42.182s
FAILED (errors=48, failures=17)
An exception has occurred, use %tb to see the full traceback.
### Alternative B: using some packaged modules
This option avoids troublesome compilations/linkings, at the cost
of freezing some module versions as available in system packages,
namely numpy 1.6.1 and scipy 0.9.0. (in turn, this implies
the most recent version of PySAL we can use is 1.9.1)
TODO: to use this alternative the python-scipy package must be
installed (this will have to be included in server provisioning)
```
apt-get install -y python-scipy
```
This doesn't look good... Taking a deeper look at the failures, many have the `IOError: [Errno 2] No such file or directory: 'streets.shp'`
#### Create and use a virtual env
In the source code, there's the following [config](https://github.com/pysal/pysal/blob/master/setup.cfg) that seems to be missing in the pip package. By copying it to `lib/python2.7/site-packages` within the environment, it goes down to 17 failures.
We'll use a `dev` enviroment as before, but will configure it to
use also system modules.
The remaining failures don't look good. I see two types: precision calculation errors and arrays/matrices missing 1 element when comparing... TODO: FIX this
# Create the virtual environment for python
$ virtualenv --system-site-packages dev
# Activate the virtualenv
$ source dev/bin/activate
# Install all the requirements
# expect this to take a while, as it will trigger a few compilations
(dev) $ pip install -I ./crankshaft
Then we can proceed to testing as in Alternative A.