Implement release with variant python package names

Instead of having to use different virtual environments for development
and for each version to be used in development here we're using
differnt names for each release's python package.
There a pending problem here: pip replaces underscores in the package name
by dashes when installing it.

CONTRIBUTING.md

@@ -1,84 +0,0 @@
-# Contributing guide
-
-## How to add new functions
-
-Try to put as little logic in the SQL extension as possible and
-just use it as a wrapper to the Python module functionality.
-
-Once a function is defined it should never change its signature in subsequent
-versions. To change a function's signature a new function with a different
-name must be created.
-
-### Version numbers
-
-The version of both the SQL extension and the Python package shall
-follow the [Semantic Versioning 2.0](http://semver.org/) guidelines:
-
-* When backwards incompatibility is introduced the major number is incremented
-* When functionally is added (in a backwards-compatible manner) the minor number
-  is incremented
-* When only fixes are introduced (backwards-compatible) the patch number is
-  incremented
-
-### Python Package
-
-...
-
-### SQL Extension
-
-* Generate a **new subfolder version** for `sql` and `test` folders to define
-  the new functions and tests
-  - Use symlinks to avoid file duplication between versions that don't update them
-  - Add new files or modify copies of the old files to add new functions or
-    modify existing functions (remember to rename a function if the signature
-    changes)
-  - Add or modify the corresponding documentation files in the `doc` folder.
-    Since we expect to have highly technical functions here, an extense
-    background explanation would be of great help to users of this extension.
-  - Create tests for the new functions/behaviour
-
-* Generate the **upgrade and downgrade files** for the extension
-
-* Update the control file and the Makefile to generate the complete SQL
-  file for the new created version. After running `make` a new
-  file `crankshaft--X.Y.Z.sql` will be created for the current version.
-  Additional files for migrating to/from the previous version A.B.Z should be
-  created:
-  - `crankshaft--X.Y.Z--A.B.C.sql`
-  - `crankshaft--A.B.C--X.Y.Z.sql`
-  All these new files must be added to git and pushed.
-
-* Update the public docs! ;-)
-
-## Conventions
-
-# SQL
-
-Use snake case (i.e. `snake_case` and not `CamelCase`) for all
-functions. Prefix functions intended for public use with `cdb_`
-and private functions (to be used only internally inside
-the extension)  with `_cdb_`.
-
-# Python
-
-...
-
-## Testing
-
-Running just the Python tests:
-
-```
-(cd python && make test)
-```
-
-Installing the Extension and running just the PostgreSQL tests:
-
-```
-(cd pg && sudo make install && PGUSER=postgres make installcheck)
-```
-
-Installing and testing everything:
-
-```
-sudo make install && PGUSER=postgres make testinstalled
-```

Makefile

@@ -8,6 +8,12 @@ install:
 	$(MAKE) -C $(PYP_DIR) install
 	$(MAKE) -C $(EXT_DIR) install
 
-testinstalled:
-	$(MAKE) -C $(PYP_DIR) testinstalled
-	$(MAKE) -C $(EXT_DIR) installcheck
+test:
+	$(MAKE) -C $(PYP_DIR) test
+	$(MAKE) -C $(EXT_DIR) test
+
+release: install
+	$(MAKE) -C $(EXT_DIR) release
+
+deploy:
+	 echo 'not yet implemented'

README.md

@@ -8,25 +8,46 @@ CartoDB Spatial Analysis extension for PostgreSQL.
 * *src* source code
 * - *src/pg* contains the PostgreSQL extension source code
 * - *src/py* Python module source code
-* *release* reselesed versions
+* *release* reseleased versions
 
 ## Requirements
 
 * pip, virtualenv, PostgreSQL
+* python-scipy system package (see src/py/README.md)
 
 # Working Process
 
 ## Development
 
 Work in `src/pg/sql`, `src/py/crankshaft`;
-use topic branch.
+use a topic branch. See src/py/README.md
+for the procedure to work with the Python local environment.
+
+Take into account:
+
+*  Always remember to add tests for any new functionality
+   documentation.
+*  Add or modify the corresponding documentation files in the `doc` folder.
+   Since we expect to have highly technical functions here, an extense
+   background explanation would be of great help to users of this extension.
+*  Convention: Use snake case (i.e. `snake_case` and not `CamelCase`) for all
+   functions. Prefix functions intended for public use with `cdb_`
+   and private functions (to be used only internally inside
+   the extension)  with `_cdb_`.
 
 Update local installation with `sudo make install`
 (this will update the 'dev' version of the extension in 'src/pg/')
 
 Run the tests with `PGUSER=postgres make test`
 
-Update extension in working database with
+The commands mentioned, executed from the top directory,
+install and test both the Pyhton package and the PostgreSQL extension.
+
+When developing the Pyhon package, it can be
+installed and tested in its own from the `src/pg` directory with the same commands.
+
+When a new development extension has beeen installed it is available
+in PostgreSQL; update extension in any specific database with
 
 * `ALTER EXTENSION crankshaft VERSION TO 'current';`
   `ALTER EXTENSION crankshaft VERSION TO 'dev';`
@@ -41,50 +62,48 @@ should be dropped manually before the update.
 If the extension has not previously been installed in a database
 we can:
 
-Add tests...
-
 * `CREATE EXTENSION crankshaft WITH VERSION 'dev';`
 
-Test
+Once the tests are succeeding a new Pull-Request can be created.
+CI-tests must be checked to be successfull.
+
+Before merging a topic branch peer code reviewing of the code is a must.
 
-Commit, push, create PR, wait for CI tests, CR, ...
 
 ## Release
 
-To release current development version
-(working directory should be clean in dev branch)
+The release process of a new version of the extension
+shall by performed by the designated *Release Manager*.
 
-(process to be gradually automated)
+Note that we expect to gradually automate this process.
 
-For backwards compatible changes (no return value, num of arguments, etc. changes...)
-new version number increasing either patch level (no new functionality)
-or minor level (new functionality) => 'X.Y.Z'.
-Update version in src/pg/crankshaft.control
-Copy release/crankshaft--current.sql to release/crankshaft--X.Y.Z.sql
-Prepare incremental downgrade, upgrade scripts....
+Having checkout the topic branch of the PR to be released:
 
-Python: ...
+The version number in `pg/cranckshaft.control` must first be updated.
+To do so [Semantic Versioning 2.0](http://semver.org/) is in order.
 
-Install the new release
+We now will explain the process for the case of backwards-compatible
+releases (updating the minor or patch version numbers).
 
-`make install-release`
+TODO: document the complex case of major releases.
 
-Test the new release
+The next command must be executed to produce the main installation
+script for the new release, `release/cranckshaft--X.Y.Z.sql`.
 
-`make test-release`
+```
+make release
+```
 
-Push the release
+Then, the release manager shall produce upgrade and downgrade scripts
+to migrate to/from the previous release. In the case of minor/patch
+releases this simply consist in extracting the functions that have changed
+and placing them in the proper `release/cranckshaft--X.Y.Z--A.B.C.sql`
+file.
 
-Wait for CI tests
+TODO: configure the local enviroment to be used by the release;
+currently should be directory `src/py/X.Y.Z`, but this must be fixed;
+a possibility to explore is to use the `cdb_conf` table.
 
-Merge into master
+TODO: testing procedure for the new release
 
-Deploy: install extension and python to production hosts,
-update extension in databases (limited to team users, data observatory, ...)
-
-Release manager role: ...
-
-.sql release scripts
-commit
-tests: staging....
-merge, tag, deploy...
+TODO: push, merge, tag, deploy procedures.

src/pg/Makefile

@@ -5,6 +5,8 @@
 #                      and make it available to PostgreSQL
 # PGUSER=postgres make installcheck -- test the 'dev' extension
 
+SED = sed
+
 EXTENSION    = crankshaft
 
 DATA         = $(EXTENSION)--dev.sql \
@@ -14,8 +16,14 @@ DATA         = $(EXTENSION)--dev.sql \
 SOURCES_DATA_DIR = sql
 SOURCES_DATA = $(wildcard $(SOURCES_DATA_DIR)/*.sql)
 
+VIRTUALENV_PATH = $(realpath ../py/environment)
+ESC_VIRVIRTUALENV_PATH = $(subst /,\/,$(VIRTUALENV_PATH))
+
+REPLACEMENTS = -e 's/@@VERSION@@/$(EXTVERSION)/g' \
+               -e 's/@@VIRTUALENV_PATH@@/$(ESC_VIRVIRTUALENV_PATH)/g'
+
 $(DATA): $(SOURCES_DATA)
-	cat $(SOURCES_DATA_DIR)/*.sql > $@
+	$(SED) $(REPLACEMENTS) $(SOURCES_DATA_DIR)/*.sql > $@
 
 TEST_DIR = test
 REGRESS = $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql)))
@@ -28,14 +36,31 @@ include $(PGXS)
 # This seems to be needed at least for PG 9.3.11
 all: $(DATA)
 
+
 # WIP: goals for releasing the extension...
 
 EXTVERSION   = $(shell grep default_version $(EXTENSION).control | sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")
 
-../release/$(EXTENSION).control: $(EXTENSION).control
+RELEASED_PACKAGE = crankshaft_$(subst .,_,$(EXTVERSION))
+
+RELEASE_REPLACEMENTS = -e 's/import\scrankshaft/import $(RELEASED_PACKAGE)/g' \
+                       -e 's/from\scrankshaft/from $(RELEASED_PACKAGE)/g'
+
+../../release/$(EXTENSION).control: $(EXTENSION).control
 	cp $< $@
 
-release: ../release/$(EXTENSION).control
-	cp $(EXTENSION)--dev.sql $(EXTENSION)--$(EXTVERSION).sql
-	# pending: create upgrade/downgrade scripts,
-	#          commit, push, tag....
+release: ../../release/$(EXTENSION).control
+	# ALTERNATIVE: use release virtualenv; copy here files from src replacing VIRTUALENV_PATH
+	$(SED) $(RELEASE_REPLACEMENTS) $(EXTENSION)--dev.sql > ../../release/$(EXTENSION)--$(EXTVERSION).sql
+	cp -r ../py/crankshaft ../../release/$(RELEASED_PACKAGE)
+	$(SED) -i -e 's/name='"'"'crankshaft'"'"'/name='"'"'$(RELEASED_PACKAGE)'"'"'/g'  ../../release/$(RELEASED_PACKAGE)/setup.py
+
+install_release:
+	$(INSTALL_DATA) ../../release/$(EXTENSION).control '$(DESTDIR)$(datadir)/extension/'
+	$(INSTALL_DATA) ../../release/*.sql '$(DESTDIR)$(datadir)/extension/'
+	# virtualenv --system-site-packages ../py/dev
+	../py/environment/bin/pip install -I -U ../../release/$(RELEASED_PACKAGE)
+	../py/environment/bin/pip install -I nose
+
+test: export PGUSER=postgres
+test: installcheck

src/pg/sql/01_py.sql

@@ -1,18 +0,0 @@
--- Use the crankshaft python module
-CREATE OR REPLACE FUNCTION _cdb_crankshaft_activate_py()
-RETURNS VOID
-AS $$
-    # activate virtualenv
-    # TODO: parameterize with environment variables or something
-    venv_path = '/home/ubuntu/crankshaft/src/py/dev'
-    activate_path = venv_path + '/bin/activate_this.py'
-    exec(open(activate_path).read(),
-         dict(__file__=activate_path))
-
-    # import something from virtualenv
-    # from crankshaft import random_seeds
-
-    # do some stuff
-    # random_seeds.set_random_seeds(123)
-    # plpy.notice('here we are')
-$$ LANGUAGE plpythonu;

src/pg/sql/01_version.sql

@@ -0,0 +1,12 @@
+-- Version number of the extension release
+CREATE OR REPLACE FUNCTION cdb_crankshaft_version()
+RETURNS text AS $$
+  SELECT '@@VERSION@@'::text;
+$$ language 'sql' IMMUTABLE STRICT;
+
+-- Internal identifier of the installed extension instence
+-- e.g. 'dev' for current development version
+CREATE OR REPLACE FUNCTION cdb_crankshaft_internal_version()
+RETURNS text AS $$
+  SELECT installed_version FROM pg_available_extensions where name='crankshaft' and pg_available_extensions IS NOT NULL;
+$$ language 'sql' IMMUTABLE STRICT;

src/pg/sql/02_py.sql

@@ -0,0 +1,21 @@
+CREATE OR REPLACE FUNCTION _cdb_crankshaft_virtualenv_path()
+RETURNS text
+AS $$
+  BEGIN
+    -- RETURN '/opt/virtualenvs/crankshaft';
+    RETURN '@@VIRTUALENV_PATH@@';
+  END;
+$$ language plpgsql IMMUTABLE STRICT;
+
+-- Use the crankshaft python module
+CREATE OR REPLACE FUNCTION _cdb_crankshaft_activate_py()
+RETURNS VOID
+AS $$
+    import os
+    # plpy.notice('%',str(os.environ))
+    # activate virtualenv
+    default_venv_path = plpy.execute('SELECT cdb_crankshaft._cdb_crankshaft_virtualenv_path()')[0]['_cdb_crankshaft_virtualenv_path']
+    venv_path =  os.environ.get('CRANKSHAFT_VENV', default_venv_path)
+    activate_path = venv_path + '/bin/activate_this.py'
+    exec(open(activate_path).read(), dict(__file__=activate_path))
+$$ LANGUAGE plpythonu;

src/py/.gitignore

@@ -1,2 +1,2 @@
 *.pyc
-dev/
+environment/

src/py/Makefile

@@ -1,9 +1,9 @@
 # Install the package locally for development
 install:
-	virtualenv dev
-	./dev/bin/pip install ./crankshaft --upgrade
-	./dev/bin/pip install nose
+	virtualenv --system-site-packages environment
+	./environment/bin/pip install -I -U ./crankshaft
+	./environment/bin/pip install -I nose
 
 # Test develpment install
-testinstalled:
-	./dev/bin/nosetests crankshaft/test/
+test:
+	./environment/bin/nosetests crankshaft/test/

src/py/README.md

@@ -7,3 +7,124 @@
 cd crankshaft
 nosetests test/
 ```
+
+## Notes about python dependencies
+* This extension is targeted at production databases. Therefore certain restrictions must be assumed about the production environment vs other experimental environments.
+* We're using `pip` and `virtualenv` to generate a suitable isolated environment for python code that has  all the dependencies
+* Every dependency should be:
+  - Added to the `setup.py` file
+  - Installed through it
+  - Tested, when they have a test suite.
+  - Fixed in the `requirements.txt`
+* At present we use Python version 2.7.3
+
+---
+
+We have two possible approaches being considered as to how manage
+the Python virtual environment: using a pure virtual enviroment
+or combine it with some system packages that include depencencies
+for the *hard-to-compile* packages (and pin them in somewhat old versions).
+
+### Alternative A: pure virtual environment
+
+In this case we will install all the packages needed in the
+virtual environment.
+This will involve, specially for the numerical packages compiling
+and linking code that uses a number of third party libraries,
+and requires having theses depencencies solved for the production
+environments.
+
+#### Create and use a virtual env
+
+We'll use a virtual enviroment directory `dev`
+under the `src/pg` directory.
+
+    # Create the virtual environment for python
+    $ virtualenv dev
+
+    # Activate the virtualenv
+    $ source dev/bin/activate
+
+    # Install all the requirements
+    # expect this to take a while, as it will trigger a few compilations
+    (dev) $ pip install -r requirements.txt
+
+    # Add a new pip to the party
+    (dev) $ pip install pandas
+
+#### Test the libraries with that virtual env
+
+##### Test numpy library dependency:
+
+    import numpy
+    numpy.test('full')
+
+##### Run scipy tests
+
+    import scipy
+    scipy.test('full')
+
+##### Testing pysal
+
+See [http://pysal.readthedocs.org/en/latest/developers/testing.html]
+
+This will require putting this into `dev/lib/python2.7/site-packages/setup.cfg`:
+
+```
+[nosetests]
+ignore-files=collection
+exclude-dir=pysal/contrib
+
+[wheel]
+universal=1
+```
+
+And copying some files before executing the tests:
+(we'll use a temporary directory from where the tests will be executed because
+some tests expect some files in the current directory). Next must be executed
+from
+
+```
+cp dev/lib/python2.7/site-packages/pysal/examples/geodanet/* dev/local/lib/python2.7/site-packages/pysal/examples
+mkdir -p test_tmp && cd test_tmp && cp ../dev/lib/python2.7/site-packages/pysal/examples/geodanet/* ./
+```
+
+Then, execute the tests with:
+
+    import pysal
+    import nose
+    nose.runmodule('pysal')
+
+
+### Alternative B: using some packaged modules
+
+This option avoids troublesome compilations/linkings, at the cost
+of freezing some module versions as available in system packages,
+namely numpy 1.6.1 and scipy 0.9.0. (in turn, this implies
+the most recent version of PySAL we can use is 1.9.1)
+
+
+TODO: to use this alternative the python-scipy package must be
+installed (this will have to be included in server provisioning)
+
+```
+apt-get install -y python-scipy
+```
+
+#### Create and use a virtual env
+
+We'll use a `dev` enviroment as before, but will configure it to
+use also system modules.
+
+
+    # Create the virtual environment for python
+    $ virtualenv --system-site-packages dev
+
+    # Activate the virtualenv
+    $ source dev/bin/activate
+
+    # Install all the requirements
+    # expect this to take a while, as it will trigger a few compilations
+    (dev) $ pip install -I ./crankshaft
+
+Then we can proceed to testing as in Alternative A.

src/py/crankshaft/setup.py

@@ -40,9 +40,9 @@ setup(
 
     # The choice of component versions is dictated by what's
     # provisioned in the production servers.
-    install_requires=['pysal==1.11.0','numpy==1.6.1','scipy==0.17.0'],
+    install_requires=['pysal==1.9.1'],
 
-    requires=['pysal', 'numpy'],
+    requires=['pysal', 'numpy' ],
 
     test_suite='test'
 )