Update documentation

Also, version the virtualenv

CONTRIBUTING.md

@@ -1,84 +0,0 @@
-# Contributing guide
-
-## How to add new functions
-
-Try to put as little logic in the SQL extension as possible and
-just use it as a wrapper to the Python module functionality.
-
-Once a function is defined it should never change its signature in subsequent
-versions. To change a function's signature a new function with a different
-name must be created.
-
-### Version numbers
-
-The version of both the SQL extension and the Python package shall
-follow the [Semantic Versioning 2.0](http://semver.org/) guidelines:
-
-* When backwards incompatibility is introduced the major number is incremented
-* When functionally is added (in a backwards-compatible manner) the minor number
-  is incremented
-* When only fixes are introduced (backwards-compatible) the patch number is
-  incremented
-
-### Python Package
-
-...
-
-### SQL Extension
-
-* Generate a **new subfolder version** for `sql` and `test` folders to define
-  the new functions and tests
-  - Use symlinks to avoid file duplication between versions that don't update them
-  - Add new files or modify copies of the old files to add new functions or
-    modify existing functions (remember to rename a function if the signature
-    changes)
-  - Add or modify the corresponding documentation files in the `doc` folder.
-    Since we expect to have highly technical functions here, an extense
-    background explanation would be of great help to users of this extension.
-  - Create tests for the new functions/behaviour
-
-* Generate the **upgrade and downgrade files** for the extension
-
-* Update the control file and the Makefile to generate the complete SQL
-  file for the new created version. After running `make` a new
-  file `crankshaft--X.Y.Z.sql` will be created for the current version.
-  Additional files for migrating to/from the previous version A.B.Z should be
-  created:
-  - `crankshaft--X.Y.Z--A.B.C.sql`
-  - `crankshaft--A.B.C--X.Y.Z.sql`
-  All these new files must be added to git and pushed.
-
-* Update the public docs! ;-)
-
-## Conventions
-
-# SQL
-
-Use snake case (i.e. `snake_case` and not `CamelCase`) for all
-functions. Prefix functions intended for public use with `cdb_`
-and private functions (to be used only internally inside
-the extension)  with `_cdb_`.
-
-# Python
-
-...
-
-## Testing
-
-Running just the Python tests:
-
-```
-(cd python && make test)
-```
-
-Installing the Extension and running just the PostgreSQL tests:
-
-```
-(cd pg && sudo make install && PGUSER=postgres make installcheck)
-```
-
-Installing and testing everything:
-
-```
-sudo make install && PGUSER=postgres make testinstalled
-```

README.md

@@ -8,18 +8,32 @@ CartoDB Spatial Analysis extension for PostgreSQL.
 * *src* source code
 * - *src/pg* contains the PostgreSQL extension source code
 * - *src/py* Python module source code
-* *release* reselesed versions
+* *release* reseleased versions
 
 ## Requirements
 
 * pip, virtualenv, PostgreSQL
+* python-scipy system package (see src/py/README.md)
 
 # Working Process
 
 ## Development
 
 Work in `src/pg/sql`, `src/py/crankshaft`;
-use topic branch.
+use a topic branch. See src/py/README.md
+for the procedure to work with the Python local environment.
+
+Take into account:
+
+*  Always remember to add tests for any new functionality
+   documentation.
+*  Add or modify the corresponding documentation files in the `doc` folder.
+   Since we expect to have highly technical functions here, an extense
+   background explanation would be of great help to users of this extension.
+*  Convention: Use snake case (i.e. `snake_case` and not `CamelCase`) for all
+   functions. Prefix functions intended for public use with `cdb_`
+   and private functions (to be used only internally inside
+   the extension)  with `_cdb_`.
 
 Update local installation with `sudo make install`
 (this will update the 'dev' version of the extension in 'src/pg/')
@@ -41,50 +55,48 @@ should be dropped manually before the update.
 If the extension has not previously been installed in a database
 we can:
 
-Add tests...
-
 * `CREATE EXTENSION crankshaft WITH VERSION 'dev';`
 
-Test
+Once the tests are succeeding a new Pull-Request can be created.
+CI-tests must be checked to be successfull.
+
+Before merging a topic branch peer code reviewing of the code is a must.
 
-Commit, push, create PR, wait for CI tests, CR, ...
 
 ## Release
 
-To release current development version
+The release process of a new version of the extension
-(working directory should be clean in dev branch)
+shall by performed by the designated *Release Manager*.
 
-(process to be gradually automated)
+Note that we expect to gradually automate this process.
 
-For backwards compatible changes (no return value, num of arguments, etc. changes...)
+Having checkout the topic branch of the PR to be released:
-new version number increasing either patch level (no new functionality)
-or minor level (new functionality) => 'X.Y.Z'.
-Update version in src/pg/crankshaft.control
-Copy release/crankshaft--current.sql to release/crankshaft--X.Y.Z.sql
-Prepare incremental downgrade, upgrade scripts....
 
-Python: ...
+The version number in `pg/cranckshaft.control` must first be updated.
+To do so [Semantic Versioning 2.0](http://semver.org/) is in order.
 
-Install the new release
+We now will explain the process for the case of backwards-compatible
+releases (updating the minor or patch version numbers).
 
-`make install-release`
+TODO: document the complex case of major releases.
 
-Test the new release
+The next command must be executed to produce the main installation
+script for the new release, `release/cranckshaft--X.Y.Z.sql`.
 
-`make test-release`
+```
+make release
+```
 
-Push the release
+Then, the release manager shall produce upgrade and downgrade scripts
+to migrate to/from the previous release. In the case of minor/patch
+releases this simply consist in extracting the functions that have changed
+and placing them in the proper `release/cranckshaft--X.Y.Z--A.B.C.sql`
+file.
 
-Wait for CI tests
+TODO: configure the local enviroment to be used by the release;
+currently should be directory `src/py/X.Y.Z`, but this must be fixed;
+a possibility to explore is to use the `cdb_conf` table.
 
-Merge into master
+TODO: testing procedure for the new release
 
-Deploy: install extension and python to production hosts,
+TODO: push, merge, tag, deploy procedures.
-update extension in databases (limited to team users, data observatory, ...)
-
-Release manager role: ...
-
-.sql release scripts
-commit
-tests: staging....
-merge, tag, deploy...

src/pg/Makefile

@@ -5,6 +5,8 @@
 #                      and make it available to PostgreSQL
 # PGUSER=postgres make installcheck -- test the 'dev' extension
 
+SED = sed
+
 EXTENSION    = crankshaft
 
 DATA         = $(EXTENSION)--dev.sql \
@@ -14,8 +16,14 @@ DATA         = $(EXTENSION)--dev.sql \
 SOURCES_DATA_DIR = sql
 SOURCES_DATA = $(wildcard $(SOURCES_DATA_DIR)/*.sql)
 
+VIRTUALENV_PATH = $(realpath ../py/)
+ESC_VIRVIRTUALENV_PATH = $(subst /,\/,$(VIRTUALENV_PATH))
+
+REPLACEMENTS = -e 's/@@VERSION@@/$(EXTVERSION)/g' \
+               -e 's/@@VIRTUALENV_PATH@@/$(ESC_VIRVIRTUALENV_PATH)/g'
+
 $(DATA): $(SOURCES_DATA)
-	cat $(SOURCES_DATA_DIR)/*.sql > $@
+	$(SED) $(REPLACEMENTS) $(SOURCES_DATA_DIR)/*.sql > $@
 
 TEST_DIR = test
 REGRESS = $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql)))
@@ -28,6 +36,7 @@ include $(PGXS)
 # This seems to be needed at least for PG 9.3.11
 all: $(DATA)
 
+
 # WIP: goals for releasing the extension...
 
 EXTVERSION   = $(shell grep default_version $(EXTENSION).control | sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")
@@ -37,5 +46,3 @@ EXTVERSION   = $(shell grep default_version $(EXTENSION).control | sed -e "s/def
 
 release: ../release/$(EXTENSION).control
 	cp $(EXTENSION)--dev.sql $(EXTENSION)--$(EXTVERSION).sql
-	# pending: create upgrade/downgrade scripts,
-	#          commit, push, tag....

src/pg/sql/01_py.sql

@@ -1,18 +0,0 @@
--- Use the crankshaft python module
-CREATE OR REPLACE FUNCTION _cdb_crankshaft_activate_py()
-RETURNS VOID
-AS $$
-    # activate virtualenv
-    # TODO: parameterize with environment variables or something
-    venv_path = '/home/ubuntu/crankshaft/src/py/dev'
-    activate_path = venv_path + '/bin/activate_this.py'
-    exec(open(activate_path).read(),
-         dict(__file__=activate_path))
-
-    # import something from virtualenv
-    # from crankshaft import random_seeds
-
-    # do some stuff
-    # random_seeds.set_random_seeds(123)
-    # plpy.notice('here we are')
-$$ LANGUAGE plpythonu;

src/pg/sql/01_version.sql

@@ -0,0 +1,12 @@
+-- Version number of the extension release
+CREATE OR REPLACE FUNCTION cdb_crankshaft_version()
+RETURNS text AS $$
+  SELECT '@@VERSION@@'::text;
+$$ language 'sql' IMMUTABLE STRICT;
+
+-- Internal identifier of the installed extension instence
+-- e.g. 'dev' for current development version
+CREATE OR REPLACE FUNCTION cdb_crankshaft_internal_version()
+RETURNS text AS $$
+  SELECT installed_version FROM pg_available_extensions where name='crankshaft' and pg_available_extensions IS NOT NULL;
+$$ language 'sql' IMMUTABLE STRICT;

src/pg/sql/02_py.sql

@@ -0,0 +1,23 @@
+CREATE OR REPLACE FUNCTION _cdb_crankshaft_virtualenvs_path()
+RETURNS text
+AS $$
+  BEGIN
+    -- RETURN '/opt/virtualenvs/crankshaft';
+    RETURN '@@VIRTUALENV_PATH@@';
+  END;
+$$ language plpgsql IMMUTABLE STRICT;
+
+-- Use the crankshaft python module
+CREATE OR REPLACE FUNCTION _cdb_crankshaft_activate_py()
+RETURNS VOID
+AS $$
+    import os
+    # plpy.notice('%',str(os.environ))
+    # activate virtualenv
+    crankshaft_version = plpy.execute('SELECT cdb_crankshaft.cdb_crankshaft_internal_version()')[0]['cdb_crankshaft_internal_version']
+    base_path = plpy.execute('SELECT cdb_crankshaft._cdb_crankshaft_virtualenvs_path()')[0]['_cdb_crankshaft_virtualenvs_path']
+    default_venv_path = os.path.join(base_path, crankshaft_version)
+    venv_path =  os.environ.get('CRANKSHAFT_VENV', default_venv_path)
+    activate_path = venv_path + '/bin/activate_this.py'
+    exec(open(activate_path).read(), dict(__file__=activate_path))
+$$ LANGUAGE plpythonu;

src/py/Makefile

@@ -1,8 +1,8 @@
 # Install the package locally for development
 install:
-	virtualenv dev
+	virtualenv --system-site-packages dev
-	./dev/bin/pip install ./crankshaft --upgrade
+	./dev/bin/pip install -I ./crankshaft
-	./dev/bin/pip install nose
+	./dev/bin/pip install -I nose
 
 # Test develpment install
 testinstalled:

src/py/README.md

@@ -7,3 +7,124 @@
 cd crankshaft
 nosetests test/
 ```
+
+## Notes about python dependencies
+* This extension is targeted at production databases. Therefore certain restrictions must be assumed about the production environment vs other experimental environments.
+* We're using `pip` and `virtualenv` to generate a suitable isolated environment for python code that has  all the dependencies
+* Every dependency should be:
+  - Added to the `setup.py` file
+  - Installed through it
+  - Tested, when they have a test suite.
+  - Fixed in the `requirements.txt`
+* At present we use Python version 2.7.3
+
+---
+
+We have two possible approaches being considered as to how manage
+the Python virtual environment: using a pure virtual enviroment
+or combine it with some system packages that include depencencies
+for the *hard-to-compile* packages (and pin them in somewhat old versions).
+
+### Alternative A: pure virtual environment
+
+In this case we will install all the packages needed in the
+virtual environment.
+This will involve, specially for the numerical packages compiling
+and linking code that uses a number of third party libraries,
+and requires having theses depencencies solved for the production
+environments.
+
+#### Create and use a virtual env
+
+We'll use a virtual enviroment directory `dev`
+under the `src/pg` directory.
+
+    # Create the virtual environment for python
+    $ virtualenv dev
+
+    # Activate the virtualenv
+    $ source dev/bin/activate
+
+    # Install all the requirements
+    # expect this to take a while, as it will trigger a few compilations
+    (dev) $ pip install -r requirements.txt
+
+    # Add a new pip to the party
+    (dev) $ pip install pandas
+
+#### Test the libraries with that virtual env
+
+##### Test numpy library dependency:
+
+    import numpy
+    numpy.test('full')
+
+##### Run scipy tests
+
+    import scipy
+    scipy.test('full')
+
+##### Testing pysal
+
+See [http://pysal.readthedocs.org/en/latest/developers/testing.html]
+
+This will require putting this into `dev/lib/python2.7/site-packages/setup.cfg`:
+
+```
+[nosetests]
+ignore-files=collection
+exclude-dir=pysal/contrib
+
+[wheel]
+universal=1
+```
+
+And copying some files before executing the tests:
+(we'll use a temporary directory from where the tests will be executed because
+some tests expect some files in the current directory). Next must be executed
+from
+
+```
+cp dev/lib/python2.7/site-packages/pysal/examples/geodanet/* dev/local/lib/python2.7/site-packages/pysal/examples
+mkdir -p test_tmp && cd test_tmp && cp ../dev/lib/python2.7/site-packages/pysal/examples/geodanet/* ./
+```
+
+Then, execute the tests with:
+
+    import pysal
+    import nose
+    nose.runmodule('pysal')
+
+
+### Alternative B: using some packaged modules
+
+This option avoids troublesome compilations/linkings, at the cost
+of freezing some module versions as available in system packages,
+namely numpy 1.6.1 and scipy 0.9.0. (in turn, this implies
+the most recent version of PySAL we can use is 1.9.1)
+
+
+TODO: to use this alternative the python-scipy package must be
+installed (this will have to be included in server provisioning)
+
+```
+apt-get install -y python-scipy
+```
+
+#### Create and use a virtual env
+
+We'll use a `dev` enviroment as before, but will configure it to
+use also system modules.
+
+
+    # Create the virtual environment for python
+    $ virtualenv --system-site-packages dev
+
+    # Activate the virtualenv
+    $ source dev/bin/activate
+
+    # Install all the requirements
+    # expect this to take a while, as it will trigger a few compilations
+    (dev) $ pip install -I ./crankshaft
+
+Then we can proceed to testing as in Alternative A.

src/py/crankshaft/setup.py

@@ -40,9 +40,9 @@ setup(
 
     # The choice of component versions is dictated by what's
     # provisioned in the production servers.
-    install_requires=['pysal==1.11.0','numpy==1.6.1','scipy==0.17.0'],
+    install_requires=['pysal==1.9.1'],
 
-    requires=['pysal', 'numpy'],
+    requires=['pysal', 'numpy' ],
 
     test_suite='test'
 )