Merge pull request #313 from CartoDB/development

Release 0.20.0 and python 0.12.0

.gitignore

@@ -1,5 +1,6 @@
 .DS_Store
 *.pyc
+.coverage
 cartodb_services.egg-info/
 build/
 dist/

NEWS.md

@@ -1,3 +1,39 @@
+November 7st, 2016
+==================
+* Version 0.17.0 of the server and version 0.10.0 of the python package
+    * Added metrics context manager to gather data from different parts of the server functions
+    * Support multiple response data for one server function call: For example in the one_to_many matrix client
+    * Metrics files configuration is not mandatory
+    * All the services covered and gathering metrics
+
+October 27st, 2016
+==================
+* Version 0.9.4 of the python package
+    * Added timeouts to all the third-party connections using requests because requests by default doesn't add timeouts.
+
+October 26st, 2016
+==================
+* Version 0.9.3 of the python package
+    * Fixes https://github.com/CartoDB/dataservices-api/issues/293
+    * Mitigate problem with 504 errors coming from Mapzen
+
+October 21st, 2016
+==================
+* Version 0.9.2 of the python package
+    * mapzen routing quota now is configurable per user
+
+September 28, 2016
+==========
+* Released version 0.8.1 of Python package cartodb\_services
+  * Improvements in QPS retry decorator for requests to external services
+
+https://github.com/CartoDB/dataservices-api/releases/tag/python-0.8.1
+
+September 8, 2016
+===========
+* Released version 0.11.1 of the client
+  * Minor change in the name of the function parameter sent to server and Observatory backend for compatibility with the last observatory-extension framework updates
+
 September 1, 2016
 ===========
 * Released version 0.11.0 of the client
@@ -22,7 +58,7 @@ August 11, 2016
 ===========
 * Released server version 0.14.2
 * Released client version 0.10.2
-* Always default arguments for DO functions to NULL, which prevents duplication & overwrite 
+* Always default arguments for DO functions to NULL, which prevents duplication & overwrite
 
 August 5, 2016
 ===========
@@ -78,7 +114,7 @@ July 15, 2016:
   * __OBS_ConnectUserTable
 
   This change does not require any client change.
- 
+
  https://github.com/CartoDB/dataservices-api/releases/tag/0.13.1-server
 
 Jul 12, 2016:
@@ -88,14 +124,14 @@ Jul 12, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.13.0-server
 
-Jul 12, 2016: 
+Jul 12, 2016:
 ===========
 * Release client 0.10.0
 * [Client] Add beta augment functions, isoline fixes, observatory dump version
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.10.0-client
 
-Jul 7, 2016: 
+Jul 7, 2016:
 ===========
 * Release client 0.9.0
 * This release adds two new functions in the Data Services extension client:
@@ -105,7 +141,7 @@ Jul 7, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.9.0-client
 
-Jul 7, 2016: 
+Jul 7, 2016:
 ===========
 * Release server 0.12.0
 * This release adds four new functions in the Data Services extension server:
@@ -117,7 +153,7 @@ Jul 7, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.12.0-server
 
-Jul 5, 2016: 
+Jul 5, 2016:
 ===========
 * Release server 0.11.0
 * Added three new public functions for each geocoding provider:
@@ -130,40 +166,40 @@ Jul 5, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.11.0-server
 
-Jul 5, 2016: 
+Jul 5, 2016:
 ===========
 * Release client 0.8.0
 * Expose providers in high-precision geocoder functions
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.8.0-client
 
-Jun 15, 2016: 
+Jun 15, 2016:
 ===========
 * [server python] Write quota info from services with zero padding. Closes issue #204.
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.10.0-server3
 
-Jun 13, 2016: 
+Jun 13, 2016:
 ===========
 * [server python] Read quota info from services with and without zero padding. Closes issue #201.
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.10.0-server2
 
-May 31, 2016: 
+May 31, 2016:
 ===========
 * Release client 0.7.0
 * Adds new function OBS_GetMeasureById
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.7.0-client
 
-May 31, 2016: 
+May 31, 2016:
 ===========
 * Release server 0.10.0
 * Adds new function OBS_GetMeasureById
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.10.0-server
 
-May 25, 2016: 
+May 25, 2016:
 ===========
 * Release server 0.9.0
 * Added a new routing function which allows to generate routes from an origin to a destination, which passes through a set of defined locations:
@@ -180,7 +216,7 @@ May 25, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.9.0-server
 
-May 25, 2016: 
+May 25, 2016:
 ===========
 * Release client 0.6.0
 * Includes new client function to obtain a route with waypoints:
@@ -188,7 +224,7 @@ May 25, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.6.0-client
 
-May 18, 2016: 
+May 18, 2016:
 ===========
 * Release client 0.5.0
 * Added new functions for the data observatory:
@@ -211,7 +247,7 @@ May 18, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.5.0-client
 
-May 18, 2016: 
+May 18, 2016:
 ===========
 * Release server 0.8.0: Data Observatory release
 * Added new functions for the data observatory:
@@ -235,28 +271,28 @@ May 18, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.8.0-server
 
-May 10, 2016: 
+May 10, 2016:
 ===========
 * Release server 0.7.4
 * In case we receive a 4xx error from one of the services: isolines, here geocoder, etc we have to return an empty value an increment the empty counter. We have to raise exception in 5xx or unhandled exceptions
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.7.4-server
 
-May 10, 2016: 
+May 10, 2016:
 ===========
 * Release server 0.7.3
 * Change how the blue/green system is working in the server side. Now the loopback is only in the observatory extension functions call instead in all the dataservices-api function for observatory
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.7.3-server
 
-May 4, 2016: 
+May 4, 2016:
 ===========
 * Release server 0.7.2
 * Added Blue/Green capability to the data observatory functions in order to be able to use staging or production databases
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.7.2-server
 
-Apr 25, 2016: 
+Apr 25, 2016:
 ===========
 * Release server 0.7.1
 * Use redis based config if exists, if not use the db config value
@@ -264,7 +300,7 @@ Apr 25, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.7.1-server
 
-Apr 21, 2016: 
+Apr 21, 2016:
 ===========
 * Release client 0.4.0
 * Remove old versioning system for client side
@@ -274,7 +310,7 @@ Apr 21, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.4.0-client
 
-Apr 21, 2016: 
+Apr 21, 2016:
 ===========
 * Release server 0.7.0
 * Added obs_get_demography_snapshot function
@@ -283,14 +319,14 @@ Apr 21, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.7.0-server
 
-Apr 19, 2016: 
+Apr 19, 2016:
 ===========
 * Release server 0.6.2
 * Add Mapzen routing and geocoder quota check
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.6.2-server
 
-Apr 14, 2016: 
+Apr 14, 2016:
 ===========
 * Release server 0.6.1
 * Now the implementation knows how to get the iso3 for the passed country in order to pass it to Mapzen
@@ -311,8 +347,8 @@ Mar 28, 2016:
 * Integrated Mapzen geocoder for street level geocoding function
 
  https://github.com/CartoDB/dataservices-api/releases/tag/0.6.0-server
- 
-Mar 23, 2016: 
+
+Mar 23, 2016:
 ===========
 * Release server 0.5.2
 * Deleted old versioning system
@@ -321,7 +357,7 @@ Mar 23, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.5.2-server
 
-Mar 17, 2016: 
+Mar 17, 2016:
 ===========
 * Release server 0.5.1
 * Renamed the python library metrics functions
@@ -331,7 +367,7 @@ Mar 17, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.5.1-server
 
-Mar 14, 2016: 
+Mar 14, 2016:
 ===========
 * Release server 0.5.0
 * Mapzen routing functions to calculate a route point to point
@@ -339,28 +375,28 @@ Mar 14, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.5.0-server
 
-Mar 14, 2016: 
+Mar 14, 2016:
 ===========
 * Release client 0.3.0
 * Added cdb_routing_point_to_point function using Mapzen as provider
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.3.0-client
 
-Feb 26, 2016: 
+Feb 26, 2016:
 ===========
 * Release client 0.2.0
 * Added routing isolines capabilities to the client and public API
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.2.0-client
 
-Feb 26, 2016: 
+Feb 26, 2016:
 ===========
 * Release server 0.4.0
 * Added routing isolines capabilities
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.4.0-server
 
-Feb 11, 2016: 
+Feb 11, 2016:
 ===========
 * Release server 0.3.0
 * Extension refactor, now is called cdb_dataservices_[client|server] in order to include more services aside the geocoder.
@@ -369,7 +405,7 @@ Feb 11, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.3.0-server
 
-Feb 4, 2016: 
+Feb 4, 2016:
 ===========
 * Release server 0.2.0
 * Logic for the google geocoder so the users with this geocoder setted up can use street level geocoding too
@@ -377,7 +413,7 @@ Feb 4, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.2.0-server
 
-Jan 25, 2016: 
+Jan 25, 2016:
 ===========
 * Release Geocoder API 0.1.0
 * Street geocoding available through the cdb_geocoder_street_point_v2 function (only working Heremaps geocoder)
@@ -386,27 +422,27 @@ Jan 25, 2016:
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.1.0
 
-Jan 25, 2016: 
+Jan 25, 2016:
 ===========
 * Release Geocoder API 0.0.1 production ready
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.0.1
 
-Dec 23, 2015: 
+Dec 23, 2015:
 ===========
 * Release Geocoder API 0.0.2 beta
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.0.2
 
-Dec 3, 2015: 
+Dec 3, 2015:
 ===========
 * Release Geocoder API 0.0.1 Beta2
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.0.1-beta2
 
-Nov 27, 2015: 
+Nov 27, 2015:
 ===========
-* Release Geocoder API BETA 1 
+* Release Geocoder API BETA 1
   * Added the organization public user to the api key check
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.0.1-beta

README.md

@@ -5,7 +5,7 @@ The CARTO Data Services SQL API
 Steps to deploy a new Data Services API version :
 
 - Deploy new version of dataservices API to all servers
-- Update the server user using: ALTER EXTENSION cdb_dataservices_server UPDATE TO '<CURRENT_VERSION>';
+- Update the server user using: ALTER EXTENSION cdb_dataservices_server UPDATE TO '\<CURRENT_VERSION\>';
 - Update the python dependencies if needed: **cartodb_geocoder** and **heremaps**
 - Add the needed config in the `cdb_conf` table:
   - `redis_metadata_config` and `redis_metrics_conf`

client/Makefile

@@ -4,29 +4,31 @@ EXTENSION = cdb_dataservices_client
 EXTVERSION = $(shell grep default_version $(EXTENSION).control | sed -e "s/default_version[[:space:]]*=[[:space:]]*'\([^']*\)'/\1/")
 
 # The new version to be generated from templates
+SED = sed
+ERB = erb
+REPLACEMENTS = -i 's/$(EXTVERSION)/$(NEW_VERSION)/g'
 NEW_EXTENSION_ARTIFACT = $(EXTENSION)--$(EXTVERSION).sql
+
+# OLD_VERSIONS = $(wildcard old_versions/*.sql)
+# DATA =  $(NEW_EXTENSION_ARTIFACT) \
+#   $(OLD_VERSIONS) \
+#   cdb_dataservices_client--0.11.0--0.11.1.sql \
+#   cdb_dataservices_client--0.11.1--0.11.0.sql
 OLD_VERSIONS = $(wildcard old_versions/*.sql)
+DATA = $(NEW_EXTENSION_ARTIFACT) \
+		 $(EXTENSION)--*--*.sql \
+		 $(OLD_VERSIONS)
+SOURCES_DATA_DIR = sql/
+
+
+REGRESS = $(notdir $(basename $(wildcard test/sql/*test.sql)))
+TEST_DIR = test/
+REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)' --user='postgres'
 
 # DATA is a special variable used by postgres build infrastructure
 # These are the files to be installed in the server shared dir,
 # for installation from scratch, upgrades and downgrades.
 # @see http://www.postgresql.org/docs/current/static/extend-pgxs.html
-DATA =  $(NEW_EXTENSION_ARTIFACT) \
-  $(OLD_VERSIONS) \
-  cdb_dataservices_client--0.10.2--0.11.0.sql \
-  cdb_dataservices_client--0.11.0--0.10.2.sql
-
-
-REGRESS = $(notdir $(basename $(wildcard test/sql/*test.sql)))
-TEST_DIR = test/
-REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)'
-
-# postgres build stuff
-PG_CONFIG = pg_config
-PGXS := $(shell $(PG_CONFIG) --pgxs)
-include $(PGXS)
-
-SOURCES_DATA_DIR = sql/
 
 # The interface definition is used along with some templates to automatically generate code
 RENDERER = renderer/sql-template-renderer
@@ -35,6 +37,11 @@ TEMPLATE_DIR = renderer/templates
 TEMPLATE_FILES = $(wildcard $(TEMPLATE_DIR)/*.erb)
 GENERATED_SQL_FILES = $(patsubst $(TEMPLATE_DIR)/%.erb, $(SOURCES_DATA_DIR)/%.sql, $(TEMPLATE_FILES))
 
+# postgres build stuff
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+
 $(GENERATED_SQL_FILES): $(SOURCES_DATA_DIR)/%.sql: $(TEMPLATE_DIR)/%.erb $(INTERFACE_FILE) $(RENDERER)
 	$(RENDERER) $(INTERFACE_FILE) $< > $@
 
@@ -44,9 +51,20 @@ $(NEW_EXTENSION_ARTIFACT): $(SOURCES_DATA)
 	rm -f $@
 	cat $(SOURCES_DATA_DIR)/*.sql >> $@
 
+.PHONY: all
 all: $(DATA)
 
+.PHONY: release
+release: $(EXTENSION).control $(SOURCES_DATA)
+	test -n "$(NEW_VERSION)"  # $$NEW_VERSION VARIABLE MISSING. Eg. make release NEW_VERSION=0.x.0
+	mv *.sql old_versions
+	$(SED) $(REPLACEMENTS) $(EXTENSION).control
+	cat $(SOURCES_DATA_DIR)/*.sql > $(EXTENSION)--$(NEW_VERSION).sql
+	$(ERB) version=$(NEW_VERSION) upgrade_downgrade_template.erb > $(EXTENSION)--$(EXTVERSION)--$(NEW_VERSION).sql
+	$(ERB) version=$(EXTVERSION) upgrade_downgrade_template.erb > $(EXTENSION)--$(NEW_VERSION)--$(EXTVERSION).sql
+
 # Only meant for development time, do not use once a version is released
+.PHONY: devclean
 devclean:
 	rm -f $(NEW_EXTENSION_ARTIFACT)
 	rm -f $(GENERATED_SQL_FILES)

client/cdb_dataservices_client--0.12.1--0.13.0.sql

@@ -0,0 +1,115 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.13.0'" to load this file. \quit
+
+-- Added for consistency (SELECT func instead of SELECT * FROM)
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_dumpversion ()
+RETURNS text AS $$
+DECLARE
+  ret text;
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    SELECT cdb_dataservices_client._obs_dumpversion(username, orgname) INTO ret;
+    RETURN ret;
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_dumpversion (username text, organization_name text)
+RETURNS text AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT cdb_dataservices_server.obs_dumpversion (username, organization_name);
+  
+$$ LANGUAGE plproxy;
+
+
+-- For quotas and services configuration
+CREATE TYPE cdb_dataservices_client.service_type AS ENUM (
+    'isolines',
+    'hires_geocoder',
+    'routing',
+    'observatory'
+);
+
+CREATE TYPE cdb_dataservices_client.service_quota_info AS (
+    service cdb_dataservices_client.service_type,
+    monthly_quota NUMERIC,
+    used_quota NUMERIC,
+    soft_limit BOOLEAN,
+    provider TEXT
+);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_service_quota_info ()
+RETURNS SETOF service_quota_info AS $$
+DECLARE
+  
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    RETURN QUERY
+    SELECT * FROM cdb_dataservices_client._cdb_service_quota_info(username, orgname);
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_enough_quota (service TEXT ,input_size NUMERIC)
+RETURNS BOOLEAN AS $$
+DECLARE
+  ret BOOLEAN;
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    SELECT cdb_dataservices_client._cdb_enough_quota(username, orgname, service, input_size) INTO ret;
+    RETURN ret;
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_quota_info (username text, organization_name text)
+RETURNS SETOF service_quota_info AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT * FROM cdb_dataservices_server.cdb_service_quota_info (username, organization_name);
+  
+$$ LANGUAGE plproxy;
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_enough_quota (username text, organization_name text, service TEXT, input_size NUMERIC)
+RETURNS BOOLEAN AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT cdb_dataservices_server.cdb_enough_quota (username, organization_name, service, input_size);
+  
+$$ LANGUAGE plproxy;
+
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_service_quota_info() TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_enough_quota(service TEXT, input_size NUMERIC) TO publicuser;

client/cdb_dataservices_client--0.13.0--0.12.1.sql

@@ -0,0 +1,44 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.12.1'" to load this file. \quit
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_dumpversion ()
+RETURNS text AS $$
+DECLARE
+  ret text;
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+    SELECT * FROM cdb_dataservices_client._obs_dumpversion(username, orgname) INTO ret;
+    RETURN ret;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_dumpversion (username text, organization_name text)
+RETURNS text AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT * FROM cdb_dataservices_server.obs_dumpversion (username, organization_name);
+$$ LANGUAGE plproxy;
+
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_enough_quota (username text, organization_name text, service TEXT, input_size NUMERIC);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_quota_info (username text, organization_name text);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_enough_quota (service TEXT ,input_size NUMERIC)
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_service_quota_info ();
+
+DROP TYPE IF EXISTS cdb_dataservices_client.service_quota_info;
+
+DROP TYPE IF EXISTS cdb_dataservices_client.service_type;

client/cdb_dataservices_client.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB dataservices client API extension'
-default_version = '0.11.0'
+default_version = '0.13.0'
 requires = 'plproxy, cartodb'
 superuser = true
 schema = cdb_dataservices_client

client/old_versions/cdb_dataservices_client--0.11.0--0.11.1.sql

@@ -0,0 +1,140 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.11.1'" to load this file. \quit
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PrepareTableOBS_GetMeasure(
+    username text,
+    orgname text,
+    user_db_role text,
+    user_schema text,
+    output_table_name text,
+    params json
+) RETURNS boolean AS $$
+    function_name = 'OBS_GetMeasure'
+    # Obtain return types for augmentation procedure
+    ds_return_metadata = plpy.execute("SELECT colnames, coltypes "
+        "FROM cdb_dataservices_client._DST_GetReturnMetadata({username}::text, {orgname}::text, {function_name}::text, {params}::json);"
+        .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            function_name=plpy.quote_literal(function_name),
+            params=plpy.quote_literal(params)
+            )
+        )
+    if ds_return_metadata[0]["colnames"]:
+        colnames_arr = ds_return_metadata[0]["colnames"]
+        coltypes_arr = ds_return_metadata[0]["coltypes"]
+    else:
+        raise Exception('Error retrieving OBS_GetMeasure metadata')
+
+
+    # Prepare column and type strings required in the SQL queries
+    columns_with_types_arr = [colnames_arr[i] + ' ' + coltypes_arr[i] for i in range(0,len(colnames_arr))]
+    columns_with_types = ','.join(columns_with_types_arr)
+
+    # Create a new table with the required columns
+    plpy.execute('CREATE TABLE "{schema}".{table_name} ( '
+        'cartodb_id int, the_geom geometry, {columns_with_types} '
+        ');'
+        .format(schema=user_schema, table_name=output_table_name, columns_with_types=columns_with_types)
+        )
+
+    plpy.execute('ALTER TABLE "{schema}".{table_name} OWNER TO "{user}";'
+        .format(schema=user_schema, table_name=output_table_name, user=user_db_role)
+        )
+
+    return True
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PopulateTableOBS_GetMeasure(
+    username text,
+    orgname text,
+    user_db_role text,
+    user_schema text,
+    dbname text,
+    table_name text,
+    output_table_name text,
+    params json
+) RETURNS boolean AS $$
+    function_name = 'OBS_GetMeasure'
+    # Obtain return types for augmentation procedure
+    ds_return_metadata = plpy.execute(
+        "SELECT colnames, coltypes "
+        "FROM cdb_dataservices_client._DST_GetReturnMetadata({username}::text, {orgname}::text, {function_name}::text, {params}::json);" .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            function_name=plpy.quote_literal(function_name),
+            params=plpy.quote_literal(params)))
+
+    if ds_return_metadata[0]["colnames"]:
+        colnames_arr = ds_return_metadata[0]["colnames"]
+        coltypes_arr = ds_return_metadata[0]["coltypes"]
+    else:
+        raise Exception('Error retrieving OBS_GetMeasure metadata')
+
+    # Prepare column and type strings required in the SQL queries
+    columns_with_types_arr = [
+        colnames_arr[i] +
+        ' ' +
+        coltypes_arr[i] for i in range(
+            0,
+            len(colnames_arr))]
+    columns_with_types = ','.join(columns_with_types_arr)
+    aliased_colname_list = ','.join(
+        ['result.' + name for name in colnames_arr])
+
+    # Instruct the OBS server side to establish a FDW
+    # The metadata is obtained as well in order to:
+    #   - (a) be able to write the query to grab the actual data to be executed in the remote server via pl/proxy,
+    #   - (b) be able to tell OBS to free resources when done.
+    ds_fdw_metadata = plpy.execute(
+        "SELECT schemaname, tabname, servername "
+        "FROM cdb_dataservices_client._DST_ConnectUserTable({username}::text, {orgname}::text, {user_db_role}::text, "
+        "{schema}::text, {dbname}::text, {table_name}::text);" .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            user_db_role=plpy.quote_literal(user_db_role),
+            schema=plpy.quote_literal(user_schema),
+            dbname=plpy.quote_literal(dbname),
+            table_name=plpy.quote_literal(table_name)))
+
+    if ds_fdw_metadata[0]["schemaname"]:
+        server_schema = ds_fdw_metadata[0]["schemaname"]
+        server_table_name = ds_fdw_metadata[0]["tabname"]
+        server_name = ds_fdw_metadata[0]["servername"]
+    else:
+        raise Exception('Error connecting dataset via FDW')
+
+    # Create a new table with the required columns
+    plpy.execute(
+        'INSERT INTO "{schema}".{analysis_table_name} '
+        'SELECT ut.cartodb_id, ut.the_geom, {colname_list} '
+        'FROM "{schema}".{table_name} ut '
+        'LEFT JOIN _DST_FetchJoinFdwTableData({username}::text, {orgname}::text, {server_schema}::text, {server_table_name}::text, '
+        '{function_name}::text, {params}::json) '
+        'AS result ({columns_with_types}, cartodb_id int)  '
+        'ON result.cartodb_id = ut.cartodb_id;' .format(
+            schema=user_schema,
+            analysis_table_name=output_table_name,
+            colname_list=aliased_colname_list,
+            table_name=table_name,
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            server_schema=plpy.quote_literal(server_schema),
+            server_table_name=plpy.quote_literal(server_table_name),
+            function_name=plpy.quote_literal(function_name),
+            params=plpy.quote_literal(params),
+            columns_with_types=columns_with_types))
+
+    # Wipe user FDW data from the server
+    wiped = plpy.execute(
+        "SELECT cdb_dataservices_client._DST_DisconnectUserTable({username}::text, {orgname}::text, {server_schema}::text, "
+        "{server_table_name}::text, {fdw_server}::text)" .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            server_schema=plpy.quote_literal(server_schema),
+            server_table_name=plpy.quote_literal(server_table_name),
+            fdw_server=plpy.quote_literal(server_name)))
+
+    return True
+$$ LANGUAGE plpythonu;

client/old_versions/cdb_dataservices_client--0.11.1--0.11.0.sql

@@ -0,0 +1,140 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.11.0'" to load this file. \quit
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PrepareTableOBS_GetMeasure(
+    username text,
+    orgname text,
+    user_db_role text,
+    user_schema text,
+    output_table_name text,
+    params json
+) RETURNS boolean AS $$
+    function_name = 'GetMeasure'
+    # Obtain return types for augmentation procedure
+    ds_return_metadata = plpy.execute("SELECT colnames, coltypes "
+        "FROM cdb_dataservices_client._DST_GetReturnMetadata({username}::text, {orgname}::text, {function_name}::text, {params}::json);"
+        .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            function_name=plpy.quote_literal(function_name),
+            params=plpy.quote_literal(params)
+            )
+        )
+    if ds_return_metadata[0]["colnames"]:
+        colnames_arr = ds_return_metadata[0]["colnames"]
+        coltypes_arr = ds_return_metadata[0]["coltypes"]
+    else:
+        raise Exception('Error retrieving OBS_GetMeasure metadata')
+
+
+    # Prepare column and type strings required in the SQL queries
+    columns_with_types_arr = [colnames_arr[i] + ' ' + coltypes_arr[i] for i in range(0,len(colnames_arr))]
+    columns_with_types = ','.join(columns_with_types_arr)
+
+    # Create a new table with the required columns
+    plpy.execute('CREATE TABLE "{schema}".{table_name} ( '
+        'cartodb_id int, the_geom geometry, {columns_with_types} '
+        ');'
+        .format(schema=user_schema, table_name=output_table_name, columns_with_types=columns_with_types)
+        )
+
+    plpy.execute('ALTER TABLE "{schema}".{table_name} OWNER TO "{user}";'
+        .format(schema=user_schema, table_name=output_table_name, user=user_db_role)
+        )
+
+    return True
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PopulateTableOBS_GetMeasure(
+    username text,
+    orgname text,
+    user_db_role text,
+    user_schema text,
+    dbname text,
+    table_name text,
+    output_table_name text,
+    params json
+) RETURNS boolean AS $$
+    function_name = 'GetMeasure'
+    # Obtain return types for augmentation procedure
+    ds_return_metadata = plpy.execute(
+        "SELECT colnames, coltypes "
+        "FROM cdb_dataservices_client._DST_GetReturnMetadata({username}::text, {orgname}::text, {function_name}::text, {params}::json);" .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            function_name=plpy.quote_literal(function_name),
+            params=plpy.quote_literal(params)))
+
+    if ds_return_metadata[0]["colnames"]:
+        colnames_arr = ds_return_metadata[0]["colnames"]
+        coltypes_arr = ds_return_metadata[0]["coltypes"]
+    else:
+        raise Exception('Error retrieving OBS_GetMeasure metadata')
+
+    # Prepare column and type strings required in the SQL queries
+    columns_with_types_arr = [
+        colnames_arr[i] +
+        ' ' +
+        coltypes_arr[i] for i in range(
+            0,
+            len(colnames_arr))]
+    columns_with_types = ','.join(columns_with_types_arr)
+    aliased_colname_list = ','.join(
+        ['result.' + name for name in colnames_arr])
+
+    # Instruct the OBS server side to establish a FDW
+    # The metadata is obtained as well in order to:
+    #   - (a) be able to write the query to grab the actual data to be executed in the remote server via pl/proxy,
+    #   - (b) be able to tell OBS to free resources when done.
+    ds_fdw_metadata = plpy.execute(
+        "SELECT schemaname, tabname, servername "
+        "FROM cdb_dataservices_client._DST_ConnectUserTable({username}::text, {orgname}::text, {user_db_role}::text, "
+        "{schema}::text, {dbname}::text, {table_name}::text);" .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            user_db_role=plpy.quote_literal(user_db_role),
+            schema=plpy.quote_literal(user_schema),
+            dbname=plpy.quote_literal(dbname),
+            table_name=plpy.quote_literal(table_name)))
+
+    if ds_fdw_metadata[0]["schemaname"]:
+        server_schema = ds_fdw_metadata[0]["schemaname"]
+        server_table_name = ds_fdw_metadata[0]["tabname"]
+        server_name = ds_fdw_metadata[0]["servername"]
+    else:
+        raise Exception('Error connecting dataset via FDW')
+
+    # Create a new table with the required columns
+    plpy.execute(
+        'INSERT INTO "{schema}".{analysis_table_name} '
+        'SELECT ut.cartodb_id, ut.the_geom, {colname_list} '
+        'FROM "{schema}".{table_name} ut '
+        'LEFT JOIN _DST_FetchJoinFdwTableData({username}::text, {orgname}::text, {server_schema}::text, {server_table_name}::text, '
+        '{function_name}::text, {params}::json) '
+        'AS result ({columns_with_types}, cartodb_id int)  '
+        'ON result.cartodb_id = ut.cartodb_id;' .format(
+            schema=user_schema,
+            analysis_table_name=output_table_name,
+            colname_list=aliased_colname_list,
+            table_name=table_name,
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            server_schema=plpy.quote_literal(server_schema),
+            server_table_name=plpy.quote_literal(server_table_name),
+            function_name=plpy.quote_literal(function_name),
+            params=plpy.quote_literal(params),
+            columns_with_types=columns_with_types))
+
+    # Wipe user FDW data from the server
+    wiped = plpy.execute(
+        "SELECT cdb_dataservices_client._DST_DisconnectUserTable({username}::text, {orgname}::text, {server_schema}::text, "
+        "{server_table_name}::text, {fdw_server}::text)" .format(
+            username=plpy.quote_nullable(username),
+            orgname=plpy.quote_nullable(orgname),
+            server_schema=plpy.quote_literal(server_schema),
+            server_table_name=plpy.quote_literal(server_table_name),
+            fdw_server=plpy.quote_literal(server_name)))
+
+    return True
+$$ LANGUAGE plpythonu;

client/old_versions/cdb_dataservices_client--0.11.1--0.12.0.sql

@@ -0,0 +1,172 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.12.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+CREATE TYPE cdb_dataservices_client.obs_meta_numerator AS (numer_id text, numer_name text, numer_description text, numer_weight text, numer_license text, numer_source text, numer_type text, numer_aggregate text, numer_extra jsonb, numer_tags jsonb, valid_denom boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE TYPE cdb_dataservices_client.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE TYPE cdb_dataservices_client.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean);
+
+CREATE TYPE cdb_dataservices_client.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailablenumerators (bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, denom_id text DEFAULT NULL, geom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_numerator AS $$
+DECLARE
+  
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    RETURN QUERY
+    SELECT * FROM cdb_dataservices_client._obs_getavailablenumerators(username, orgname, bounds, filter_tags, denom_id, geom_id, timespan);
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailabledenominators (bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, geom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_denominator AS $$
+DECLARE
+  
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    RETURN QUERY
+    SELECT * FROM cdb_dataservices_client._obs_getavailabledenominators(username, orgname, bounds, filter_tags, numer_id, geom_id, timespan);
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailablegeometries (bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
+DECLARE
+  
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    RETURN QUERY
+    SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailabletimespans (bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, geom_id text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_timespan AS $$
+DECLARE
+  
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    RETURN QUERY
+    SELECT * FROM cdb_dataservices_client._obs_getavailabletimespans(username, orgname, bounds, filter_tags, numer_id, denom_id, geom_id);
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_legacybuildermetadata (aggregate_type text DEFAULT NULL)
+RETURNS TABLE(name text, subsection json) AS $$
+DECLARE
+  
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  
+    RETURN QUERY
+    SELECT * FROM cdb_dataservices_client._obs_legacybuildermetadata(username, orgname, aggregate_type);
+  
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablenumerators (username text, organization_name text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, denom_id text DEFAULT NULL, geom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_numerator AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT * FROM cdb_dataservices_server.obs_getavailablenumerators (username, organization_name, bounds, filter_tags, denom_id, geom_id, timespan);
+  
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailabledenominators (username text, organization_name text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, geom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+
+RETURNS SETOF cdb_dataservices_client.obs_meta_denominator AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT * FROM cdb_dataservices_server.obs_getavailabledenominators (username, organization_name, bounds, filter_tags, numer_id, geom_id, timespan);
+  
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries (username text, organization_name text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+
+RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT * FROM cdb_dataservices_server.obs_getavailablegeometries (username, organization_name, bounds, filter_tags, numer_id, denom_id, timespan);
+  
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailabletimespans (username text, organization_name text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, geom_id text DEFAULT NULL)
+
+RETURNS SETOF cdb_dataservices_client.obs_meta_timespan AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT * FROM cdb_dataservices_server.obs_getavailabletimespans (username, organization_name, bounds, filter_tags, numer_id, denom_id, geom_id);
+  
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_legacybuildermetadata (username text, organization_name text, aggregate_type text DEFAULT NULL)
+
+RETURNS TABLE(name text, subsection json) AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  
+  SELECT * FROM cdb_dataservices_server.obs_legacybuildermetadata (username, organization_name, aggregate_type);
+  
+$$ LANGUAGE plproxy;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_getavailablenumerators(bounds geometry(Geometry, 4326), filter_tags text[], denom_id text, geom_id text, timespan text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_getavailabledenominators(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, geom_id text, timespan text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_getavailablegeometries(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, timespan text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_getavailabletimespans(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, geom_id text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_legacybuildermetadata(aggregate_type text) TO publicuser;

client/old_versions/cdb_dataservices_client--0.12.0--0.11.1.sql

@@ -0,0 +1,21 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.11.1'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+DROP FUNCTION IF EXISTS cdb_dataservices_client.obs_getavailablenumerators (geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.obs_getavailabledenominators (geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.obs_getavailablegeometries (geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.obs_getavailabletimespans (geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.obs_legacybuildermetadata(text);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_getavailablenumerators (text, text, geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_getavailabledenominators (text, text, geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_getavailablegeometries (text, text, geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_getavailabletimespans (text, text, geometry(Geometry, 4326), text[], text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_legacybuildermetadata(text, text, text);
+
+DROP TYPE IF EXISTS cdb_dataservices_client.obs_meta_numerator;
+DROP TYPE IF EXISTS cdb_dataservices_client.obs_meta_denominator;
+DROP TYPE IF EXISTS cdb_dataservices_client.obs_meta_geometry;
+DROP TYPE IF EXISTS cdb_dataservices_client.obs_meta_timespan;

client/old_versions/cdb_dataservices_client--0.12.0--0.12.1.sql

@@ -0,0 +1,12 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.12.1'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE score numeric;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE numtiles bigint;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE notnull_percent numeric;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE numgeoms numeric;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE percentfill numeric;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE estnumgeoms numeric;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE meanmediansize numeric;

client/old_versions/cdb_dataservices_client--0.12.1--0.12.0.sql

@@ -0,0 +1,12 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.12.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE score;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE numtiles;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE notnull_percent;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE numgeoms;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE percentfill;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE estnumgeoms;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE meanmediansize;

client/renderer/interface.yaml

@@ -323,6 +323,69 @@
 
 - name: obs_dumpversion
   return_type: text
-  no_params: true
   params:
     - {}
+
+- name: obs_getavailablenumerators
+  return_type: SETOF cdb_dataservices_client.obs_meta_numerator
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: bounds, type: "geometry(Geometry, 4326)", default: 'NULL' }
+    - { name: filter_tags, type: "text[]", default: 'NULL' }
+    - { name: denom_id, type: text, default: 'NULL' }
+    - { name: geom_id, type: text, default: 'NULL' }
+    - { name: timespan, type: text, default: 'NULL'}
+
+- name: obs_getavailabledenominators
+  return_type: SETOF cdb_dataservices_client.obs_meta_denominator
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: bounds, type: "geometry(Geometry, 4326)", default: 'NULL' }
+    - { name: filter_tags, type: "text[]", default: 'NULL' }
+    - { name: numer_id, type: text, default: 'NULL' }
+    - { name: geom_id, type: text, default: 'NULL' }
+    - { name: timespan, type: text, default: 'NULL'}
+
+- name: obs_getavailablegeometries
+  return_type: SETOF cdb_dataservices_client.obs_meta_geometry
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: bounds, type: "geometry(Geometry, 4326)", default: 'NULL' }
+    - { name: filter_tags, type: "text[]", default: 'NULL' }
+    - { name: numer_id, type: text, default: 'NULL' }
+    - { name: denom_id, type: text, default: 'NULL' }
+    - { name: timespan, type: text, default: 'NULL'}
+
+- name: obs_getavailabletimespans
+  return_type: SETOF cdb_dataservices_client.obs_meta_timespan
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: bounds, type: "geometry(Geometry, 4326)", default: 'NULL' }
+    - { name: filter_tags, type: "text[]", default: 'NULL' }
+    - { name: numer_id, type: text, default: 'NULL' }
+    - { name: denom_id, type: text, default: 'NULL' }
+    - { name: geom_id, type: text, default: 'NULL'}
+
+- name: obs_legacybuildermetadata
+  return_type: TABLE(name text, subsection json)
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: aggregate_type, type: text, default: 'NULL' }
+
+- name: cdb_service_quota_info
+  return_type: SETOF service_quota_info
+  multi_row: true
+  multi_field: true
+  params:
+    - {}
+
+- name: cdb_enough_quota
+  return_type: BOOLEAN
+  params:
+    - { name: service, type: TEXT }
+    - { name: input_size, type: NUMERIC }

client/renderer/sql-template-renderer

@@ -36,10 +36,6 @@ class SqlTemplateRenderer
     @function_signature['multi_row']
   end
 
-  def no_params
-    @function_signature['no_params']
-  end
-
   def user_config_key
     @function_signature['user_config_key']
   end
@@ -49,11 +45,11 @@ class SqlTemplateRenderer
   end
 
   def params
-    @function_signature['params'].reject(&:empty?).map { |p| "#{p['name']}"}.join(', ')
+    @function_signature['params'].reject(&:empty?).map { |p| "#{p['name']}"}
   end
 
   def params_with_type
-    @function_signature['params'].reject(&:empty?).map { |p| "#{p['name']} #{p['type']}" }.join(', ')
+    @function_signature['params'].reject(&:empty?).map { |p| "#{p['name']} #{p['type']}" }
   end
 
   def params_with_type_and_default
@@ -64,7 +60,7 @@ class SqlTemplateRenderer
         "#{p['name']} #{p['type']}"
       end
     end
-    return parameters.join(', ')
+    return parameters
   end
 end
 

client/renderer/templates/20_public_functions.erb

@@ -4,7 +4,7 @@
 -- These are the only ones with permissions to publicuser role
 -- and should also be the only ones with SECURITY DEFINER
 
-CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>.<%= name %> (<%= params_with_type_and_default %>)
+CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>.<%= name %> (<%= params_with_type_and_default.join(' ,') %>)
 RETURNS <%= return_type %> AS $$
 DECLARE
   <% if not multi_row %>ret <%= return_type %>;<% end %>
@@ -21,15 +21,12 @@ BEGIN
   END IF;
   <% if multi_row %>
     RETURN QUERY
-    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(username, orgname, <%= params %>);
+    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>);
   <% elsif multi_field %>
-    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(username, orgname, <%= params %>) INTO ret;
-    RETURN ret;
-  <% elsif no_params %>
-    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(username, orgname) INTO ret;
+    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>) INTO ret;
     RETURN ret;
   <% else %>
-    SELECT <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(username, orgname, <%= params %>) INTO ret;
+    SELECT <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>) INTO ret;
     RETURN ret;
   <% end %>
 END;

client/renderer/templates/30_plproxy_functions.erb

@@ -1,15 +1,9 @@
-<% if no_params %>
-CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (username text, organization_name text)
-<% else %>
-CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (username text, organization_name text, <%= params_with_type_and_default %>)
-<% end %>
+CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (<%= ['username text', 'organization_name text'].concat(params_with_type_and_default).join(', ') %>)
 RETURNS <%= return_type %> AS $$
   CONNECT <%= DATASERVICES_CLIENT_SCHEMA %>._server_conn_str();
   <% if multi_field %>
-  SELECT * FROM <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (username, organization_name, <%= params %>);
-  <% elsif no_params %>
-  SELECT * FROM <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (username, organization_name);
+  SELECT * FROM <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= ['username', 'organization_name'].concat(params).join(', ') %>);
   <% else %>
-  SELECT <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (username, organization_name, <%= params %>);
+  SELECT <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= ['username', 'organization_name'].concat(params).join(', ') %>);
   <% end %>
 $$ LANGUAGE plproxy;

client/renderer/templates/90_grant_execute.erb

@@ -1 +1 @@
-GRANT EXECUTE ON FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>.<%= name %>(<%= params_with_type %>) TO publicuser;
+GRANT EXECUTE ON FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>.<%= name %>(<%= params_with_type.join(', ') %>) TO publicuser;

client/sql/16_custom_types.sql

@@ -8,4 +8,30 @@ CREATE TYPE cdb_dataservices_client.simple_route AS (
     shape geometry(LineString,4326),
     length real,
     duration integer
-);
+);
+
+-- For the OBS_Meta functions
+CREATE TYPE cdb_dataservices_client.obs_meta_numerator AS (numer_id text, numer_name text, numer_description text, numer_weight text, numer_license text, numer_source text, numer_type text, numer_aggregate text, numer_extra jsonb, numer_tags jsonb, valid_denom boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE TYPE cdb_dataservices_client.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE TYPE cdb_dataservices_client.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric);
+
+CREATE TYPE cdb_dataservices_client.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean);
+
+
+-- For quotas and services configuration
+CREATE TYPE cdb_dataservices_client.service_type AS ENUM (
+    'isolines',
+    'hires_geocoder',
+    'routing',
+    'observatory'
+);
+
+CREATE TYPE cdb_dataservices_client.service_quota_info AS (
+    service cdb_dataservices_client.service_type,
+    monthly_quota NUMERIC,
+    used_quota NUMERIC,
+    soft_limit BOOLEAN,
+    provider TEXT
+);

client/sql/20_table_augmentation.sql

@@ -100,7 +100,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PrepareTableOBS_GetMeas
     output_table_name text,
     params json
 ) RETURNS boolean AS $$
-    function_name = 'GetMeasure'
+    function_name = 'OBS_GetMeasure'
     # Obtain return types for augmentation procedure
     ds_return_metadata = plpy.execute("SELECT colnames, coltypes "
         "FROM cdb_dataservices_client._DST_GetReturnMetadata({username}::text, {orgname}::text, {function_name}::text, {params}::json);"
@@ -146,7 +146,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PopulateTableOBS_GetMea
     output_table_name text,
     params json
 ) RETURNS boolean AS $$
-    function_name = 'GetMeasure'
+    function_name = 'OBS_GetMeasure'
     # Obtain return types for augmentation procedure
     ds_return_metadata = plpy.execute(
         "SELECT colnames, coltypes "

client/upgrade_downgrade_template.erb

@@ -0,0 +1,5 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '<%= version %>'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade

doc/demographic_functions.md

@@ -1,6 +1,6 @@
 # Demographic Functions
 
-The Demographic Snapshot enables you to collect demographic reports around a point location. For example, you can take the coordinates of a coffee shop and find the average population characteristics, such as total population, educational attainment, housing and income information around that location. You can use raw street addresses by combining the Demographic Snapshot with CARTO's geocoding features. If you need help creating coordinates from addresses, see the [Geocoding Functions](/carto-engine/dataservices-api/geocoding-functions/) documentation.
+The Demographic Snapshot enables you to collect demographic reports around a point location. For example, you can take the coordinates of a coffee shop and find the average population characteristics, such as total population, educational attainment, housing and income information around that location. You can use raw street addresses by combining the Demographic Snapshot with CARTO's geocoding features. If you need help creating coordinates from addresses, see the [Geocoding Functions](https://carto.com/docs/carto-engine/dataservices-api/geocoding-functions/) documentation.
 
 _**Note:** The Demographic Snapshot functions are only available for the United States._
 

doc/geocoding_functions.md

@@ -215,7 +215,7 @@ INSERT INTO {tablename} (the_geom) SELECT cdb_geocode_namedplace_point('New York
 
 ## Postal Code Geocoder
 
-This function geocodes your data into point, or polygon, geometries for postal codes. The postal code polygon geocoder covers the United States, France, Australia and Canada; a request for a different country will return an empty response.
+These functions geocode your data into point, or polygon, geometries for postal codes. The postal code geocoder covers the United States, France, Australia and Canada; a request for a different country will return an empty response.
 
 **Note:** For the USA, US Census [Zip Code Tabulation Areas](https://www.census.gov/geo/reference/zctas.html) (ZCTA) are used to reference geocodes for USPS postal codes service areas. See the [FAQs](https://carto.com/docs/faqs/datasets-and-data/#why-does-carto-use-census-bureau-zctas-and-not-usps-zip-codes-for-postal-codes) about datasets and data for details.
 
@@ -290,8 +290,7 @@ Geocodes a postal code from a specified country into an IP address, displayed as
 
 Name | Type | Description
 --- | --- | ---
-`ip_address` | `text` | Postal code
-`country_name` | `text` | IPv4 or IPv6 address
+`ip_address` | `text` | IPv4 or IPv6 address
 
 #### Returns
 

doc/overview.md

@@ -10,7 +10,7 @@ _The Data Services API is collaborating with [Mapzen](https://mapzen.com/), and
 
 By using the SQL API to query the Data Services API functions, you can manage specific operations and the corresponding geometries (a `polygon` or a `point`), according to the input information.
 
-The Data Services API decouples the geocoding and isoline services from the CARTO Editor. The API allows you to geocode data (from single rows, complete datasets, or simple inputs) and to perform trade areas analysis (computing isodistances or isochrones) programatically, through authenticated requests.
+The Data Services API also exposes its services directly through CARTO Builder. For example, you can geocode data (from single rows, complete datasets, or simple inputs) and perform trade areas analysis (computing isodistances or isochrones) programatically, through authenticated SQL requests, or by using the ANALYSIS options. 
 
 The geometries provided by this API are projected in the projection [WGS 84 SRID 4326](http://spatialreference.org/ref/epsg/wgs-84/).
 

doc/routing_functions.md

@@ -12,7 +12,7 @@ Name | Type | Description | Accepted values
 --- | --- | --- | ---
 `origin` | `geometry(Point)` | Origin point, in 4326 projection, which defines the start location. |
 `destination` | `geometry(Point)` | Destination point, in 4326 projection, which defines the end location. |
-`mode` | `text` | Type of transport used to calculate the isolines. | `car`, `walk`, `bicycle` or `public_transport`
+`mode` | `text` | Type of transport used to calculate the routes. | `car`, `walk`, `bicycle` or `public_transport`
 `options` | `text[]` | (Optional) Multiple options to add more capabilities to the analysis. See [Optional routing parameters](#optional-routing-parameters) for details.
 `units` | `text` | Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`
 
@@ -47,7 +47,7 @@ Returns a route that goes from origin to destination and whose path travels thro
 Name | Type | Description | Accepted values
 --- | --- | --- | ---
 `waypoints` | `geometry(Point)[]` | Array of ordered points, in 4326 projection, which defines the origin point, one or more locations for the route path to travel through, and the destination. The first element of the array defines the origin and the last element the destination of the route. |
-`mode` | `text` | Type of transport used to calculate the isolines. | `car`, `walk`, `bicycle` or `public_transport`
+`mode` | `text` | Type of transport used to calculate the routes. | `car`, `walk`, `bicycle` or `public_transport`
 `options` | `text[]` | (Optional) Multiple options to add more capabilities to the analysis. See [Optional routing parameters](#optional-routing-parameters) for details.
 `units` | `text` | Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`
 

server/extension/cdb_dataservices_server--0.19.0--0.20.0.sql

@@ -0,0 +1,187 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.20.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapzen_isolines(TEXT, TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]);
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isodistance(
+   username TEXT,
+   orgname TEXT,
+   source geometry(Geometry, 4326),
+   mode TEXT,
+   data_range integer[],
+   options text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  import json
+  from cartodb_services.mapzen import MatrixClient, MapzenIsolines
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  quota_service = QuotaService(user_isolines_routing_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  try:
+    client = MatrixClient(user_isolines_routing_config.mapzen_matrix_api_key, logger)
+    mapzen_isolines = MapzenIsolines(client, logger)
+
+    if source:
+      lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
+      lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
+      origin = {'lat': lat, 'lon': lon}
+    else:
+      raise Exception('source is NULL')
+
+    # -- TODO Support options properly
+    isolines = {}
+    for r in data_range:
+        isoline = mapzen_isolines.calculate_isodistance(origin, mode, r)
+        isolines[r] = isoline
+
+    result = []
+    for r in data_range:
+
+      if len(isolines[r]) >= 3:
+        # -- TODO encapsulate this block into a func/method
+        locations = isolines[r] + [ isolines[r][0] ] # close the polygon repeating the first point
+        wkt_coordinates = ','.join(["%f %f" % (l['lon'], l['lat']) for l in locations])
+        sql = "SELECT ST_MPolyFromText('MULTIPOLYGON((({0})))', 4326) as geom".format(wkt_coordinates)
+        multipolygon = plpy.execute(sql, 1)[0]['geom']
+      else:
+        multipolygon = None
+
+      result.append([source, r, multipolygon])
+
+    quota_service.increment_success_service_use()
+    quota_service.increment_isolines_service_use(len(isolines))
+    return result
+  except BaseException as e:
+    import sys
+    quota_service.increment_failed_service_use()
+    logger.error('Error trying to get mapzen isolines', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to get mapzen isolines')
+  finally:
+    quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isochrones(
+   username TEXT,
+   orgname TEXT,
+   source geometry(Geometry, 4326),
+   mode TEXT,
+   data_range integer[],
+   options text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  import json
+  from cartodb_services.mapzen import MatrixClient, MapzenIsochrones
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+  from cartodb_services.mapzen.types import coordinates_to_polygon
+
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  # -- Check the quota
+  quota_service = QuotaService(user_isolines_routing_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  try:
+    mapzen_isochrones = MapzenIsochrones(user_isolines_routing_config.mapzen_matrix_api_key,
+                                         logger)
+
+    if source:
+      lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
+      lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
+      origin = {'lat': lat, 'lon': lon}
+    else:
+      raise Exception('source is NULL')
+
+    resp = mapzen_isochrones.isochrone(origin, mode, data_range)
+
+    if resp:
+      result = []
+      for isochrone in resp:
+        result_polygon = coordinates_to_polygon(isochrone.coordinates)
+        if result_polygon:
+          quota_service.increment_success_service_use()
+          result.append([source, isochrone.duration, result_polygon])
+        else:
+          quota_service.increment_empty_service_use()
+          result.append([source, isochrone.duration, None])
+      quota_service.increment_success_service_use()
+      quota_service.increment_isolines_service_use(len(result))
+      return result
+    else:
+      quota_service.increment_empty_service_use()
+      return []
+  except BaseException as e:
+    import sys
+    quota_service.increment_failed_service_use()
+    logger.error('Error trying to get mapzen isochrones', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to get mapzen isochrones')
+  finally:
+    quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+  result = plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
+
+  return result
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isochrones($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+  result = plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
+  return result
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+
+  if user_isolines_config.google_services_user:
+    raise Exception('This service is not available for google service users.')
+
+  with metrics('cb_isodistance', user_isolines_config, logger):
+    if user_isolines_config.heremaps_provider:
+      here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+      return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
+    elif user_isolines_config.mapzen_provider:
+      mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+      return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
+    else:
+      raise Exception('Requested isolines provider is not available')
+$$ LANGUAGE plpythonu;

server/extension/cdb_dataservices_server--0.20.0--0.19.0.sql

@@ -0,0 +1,121 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.19.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapzen_isodistance(TEXT, TEXT,  geometry(Geometry, 4326), TEXT, integer[], text[]);
+DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapzen_isochrones(TEXT, TEXT,  geometry(Geometry, 4326), TEXT, integer[], text[]);
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isolines(
+   username TEXT,
+   orgname TEXT,
+   isotype TEXT,
+   source geometry(Geometry, 4326),
+   mode TEXT,
+   data_range integer[],
+   options text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  import json
+  from cartodb_services.mapzen import MatrixClient, MapzenIsolines
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  quota_service = QuotaService(user_isolines_routing_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  try:
+    client = MatrixClient(user_isolines_routing_config.mapzen_matrix_api_key, logger)
+    mapzen_isolines = MapzenIsolines(client, logger)
+
+    if source:
+      lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
+      lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
+      origin = {'lat': lat, 'lon': lon}
+    else:
+      raise Exception('source is NULL')
+
+    # -- TODO Support options properly
+    isolines = {}
+    for r in data_range:
+        isoline = mapzen_isolines.calculate_isodistance(origin, mode, r)
+        isolines[r] = isoline
+
+    result = []
+    for r in data_range:
+
+      if len(isolines[r]) >= 3:
+        # -- TODO encapsulate this block into a func/method
+        locations = isolines[r] + [ isolines[r][0] ] # close the polygon repeating the first point
+        wkt_coordinates = ','.join(["%f %f" % (l['lon'], l['lat']) for l in locations])
+        sql = "SELECT ST_MPolyFromText('MULTIPOLYGON((({0})))', 4326) as geom".format(wkt_coordinates)
+        multipolygon = plpy.execute(sql, 1)[0]['geom']
+      else:
+        multipolygon = None
+
+      result.append([source, r, multipolygon])
+
+    quota_service.increment_success_service_use()
+    quota_service.increment_isolines_service_use(len(isolines))
+    return result
+  except BaseException as e:
+    import sys
+    quota_service.increment_failed_service_use()
+    logger.error('Error trying to get mapzen isolines', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to get mapzen isolines')
+  finally:
+    quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  type = 'isochrone'
+
+  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+  result = plpy.execute(mapzen_plan, [username, orgname, type, source, mode, range, options])
+  return result
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  type = 'isodistance'
+
+  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+  result = plpy.execute(mapzen_plan, [username, orgname, type, source, mode, range, options])
+
+  return result
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  if user_isolines_config.google_services_user:
+    raise Exception('This service is not available for google service users.')
+
+  if user_isolines_config.heremaps_provider:
+    here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+    return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
+    elif user_isolines_config.mapzen_provider:
+    mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+    return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
+  else:
+    raise Exception('Requested isolines provider is not available')
+$$ LANGUAGE plpythonu;

server/extension/cdb_dataservices_server.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB dataservices server extension'
-default_version = '0.15.1'
+default_version = '0.20.0'
 requires = 'plpythonu, plproxy, postgis, cdb_geocoder'
 superuser = true
 schema = cdb_dataservices_server

server/extension/old_versions/cdb_dataservices_server--0.15.1--0.16.0.sql

@@ -0,0 +1,69 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.16.0'" to load this file. \quit
+
+-- Here goes your code to upgrade/downgrade
+
+-- This is done in order to avoid an undesired depedency on cartodb extension
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_conf_getconf(input_key text)
+RETURNS JSON AS $$
+    SELECT VALUE FROM cartodb.cdb_conf WHERE key = input_key;
+$$ LANGUAGE SQL STABLE SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
+RETURNS Geometry AS $$
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+  from cartodb_services.mapzen import MapzenGeocoder
+  from cartodb_services.mapzen.types import country_to_iso3
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger
+  from cartodb_services.refactor.tools.logger import LoggerConfigBuilder
+  from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
+  from cartodb_services.refactor.core.environment import ServerEnvironmentBuilder
+  from cartodb_services.refactor.backend.server_config import ServerConfigBackendFactory
+  from cartodb_services.refactor.backend.user_config import UserConfigBackendFactory
+  from cartodb_services.refactor.backend.org_config import OrgConfigBackendFactory
+  from cartodb_services.refactor.backend.redis_metrics_connection import RedisMetricsConnectionFactory
+
+  server_config_backend = ServerConfigBackendFactory().get()
+  environment = ServerEnvironmentBuilder(server_config_backend).get()
+  user_config_backend = UserConfigBackendFactory(username, environment, server_config_backend).get()
+  org_config_backend = OrgConfigBackendFactory(orgname, environment, server_config_backend).get()
+
+  logger_config = LoggerConfigBuilder(environment, server_config_backend).get()
+  logger = Logger(logger_config)
+
+  mapzen_geocoder_config = MapzenGeocoderConfigBuilder(server_config_backend, user_config_backend, org_config_backend, username, orgname).get()
+
+  redis_metrics_connection = RedisMetricsConnectionFactory(environment, server_config_backend).get()
+
+  quota_service = QuotaService(mapzen_geocoder_config, redis_metrics_connection)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  try:
+    geocoder = MapzenGeocoder(mapzen_geocoder_config.mapzen_api_key, logger)
+    country_iso3 = None
+    if country:
+      country_iso3 = country_to_iso3(country)
+    coordinates = geocoder.geocode(searchtext=searchtext, city=city,
+                                   state_province=state_province,
+                                   country=country_iso3, search_type='address')
+    if coordinates:
+      quota_service.increment_success_service_use()
+      plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
+      point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
+      return point['st_setsrid']
+    else:
+      quota_service.increment_empty_service_use()
+      return None
+  except BaseException as e:
+    import sys
+    quota_service.increment_failed_service_use()
+    logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to geocode street point using mapzen')
+  finally:
+    quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.16.0--0.15.1.sql

@@ -0,0 +1,54 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.15.1'" to load this file. \quit
+
+-- Here goes your code to upgrade/downgrade
+
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_conf_getconf(text);
+
+-- Geocodes a street address given a searchtext and a state and/or country
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
+RETURNS Geometry AS $$
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  from cartodb_services.config.user import User
+  from cartodb_services.config.configs import ConfigsFactory
+  from cartodb_services.config.hires_geocoder_config import HiResGeocoderConfigFactory
+  from cartodb_services.request.request import RequestFactory
+
+  user = User(username, orgname)
+  configs = ConfigsFactory.get(user)
+  request = RequestFactory().create(user, configs, 'cdb_geocode_street_point')
+
+  # TODO change to hires_geocoder_config = HiResGeocoderConfigFactory.get(request)
+  hires_geocoder_config = HiResGeocoderConfigFactory(configs).get(user)
+
+  if hires_geocoder_config.provider == 'here':
+    here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+    return plpy.execute(here_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+  elif hires_geocoder_config.provider == 'google':
+    google_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_google_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+    return plpy.execute(google_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+  elif hires_geocoder_config.provider == 'mapzen':
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+  else:
+    raise Exception('Requested geocoder is not available')
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
+RETURNS Geometry AS $$
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
+
+  if user_geocoder_config.heremaps_geocoder:
+    here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+    return plpy.execute(here_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+  else:
+    raise Exception('Here geocoder is not available for your account.')
+
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.17.0--0.18.0.sql

@@ -0,0 +1,73 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.18.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+CREATE TYPE cdb_dataservices_server.obs_meta_numerator AS (numer_id text, numer_name text, numer_description text, numer_weight text, numer_license text, numer_source text, numer_type text, numer_aggregate text, numer_extra jsonb, numer_tags jsonb, valid_denom boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableNumerators(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan);
+$$ LANGUAGE plproxy;
+
+CREATE TYPE cdb_dataservices_server.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableDenominators(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_denominator AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan);
+$$ LANGUAGE plproxy;
+
+CREATE TYPE cdb_dataservices_server.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_geometry AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan);
+$$ LANGUAGE plproxy;
+
+CREATE TYPE cdb_dataservices_server.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableTimespans(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_timespan AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id);
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_LegacyBuilderMetadata(
+  username TEXT,
+  orgname TEXT,
+  aggregate_type TEXT DEFAULT NULL)
+RETURNS TABLE(name TEXT, subsection JSON) AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_LegacyBuilderMetadata(aggregate_type);
+$$ LANGUAGE plproxy;

server/extension/old_versions/cdb_dataservices_server--0.18.0--0.17.0.sql

@@ -0,0 +1,15 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.17.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+DROP FUNCTION IF EXISTS cdb_dataservices_server.obs_getavailablenumerators(TEXT, TEXT, geometry(Geometry, 4326), TEXT[], TEXT, TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.obs_getavailabledenominators(TEXT, TEXT, geometry(Geometry, 4326), TEXT[], TEXT, TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.obs_getavailablegeometries(TEXT, TEXT, geometry(Geometry, 4326), TEXT[], TEXT, TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.obs_getavailabletimespans(TEXT, TEXT, geometry(Geometry, 4326), TEXT[], TEXT, TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.obs_legacybuildermetadata(TEXT, TEXT, TEXT);
+
+DROP TYPE IF EXISTS cdb_dataservices_server.obs_meta_numerator;
+DROP TYPE IF EXISTS cdb_dataservices_server.obs_meta_denominator;
+DROP TYPE IF EXISTS cdb_dataservices_server.obs_meta_geometry;
+DROP TYPE IF EXISTS cdb_dataservices_server.obs_meta_timespan;

server/extension/old_versions/cdb_dataservices_server--0.18.0--0.18.1.sql

@@ -0,0 +1,12 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.18.1'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE score numeric;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE numtiles bigint;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE notnull_percent numeric;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE numgeoms numeric;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE percentfill numeric;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE estnumgeoms numeric;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE meanmediansize numeric;

server/extension/old_versions/cdb_dataservices_server--0.18.1--0.18.0.sql

@@ -0,0 +1,12 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.18.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE score;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE numtiles;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE notnull_percent;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE numgeoms;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE percentfill;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE estnumgeoms;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE meanmediansize;

server/extension/old_versions/cdb_dataservices_server--0.18.1--0.19.0.sql

@@ -0,0 +1,109 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.19.0'" to load this file. \quit
+
+DO $$
+BEGIN
+  IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'service_type') THEN
+    CREATE TYPE cdb_dataservices_server.service_type AS ENUM (
+      'isolines',
+      'hires_geocoder',
+      'routing',
+      'observatory'
+    );
+  END IF;
+END $$;
+
+DO $$
+BEGIN
+  IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'service_quota_info') THEN
+    CREATE TYPE cdb_dataservices_server.service_quota_info AS (
+      service cdb_dataservices_server.service_type,
+      monthly_quota NUMERIC,
+      used_quota NUMERIC,
+      soft_limit BOOLEAN,
+      provider TEXT
+    );
+  END IF;
+END $$;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_quota_info(
+  username TEXT,
+  orgname TEXT)
+RETURNS SETOF cdb_dataservices_server.service_quota_info AS $$
+  from cartodb_services.metrics.user import UserMetricsService
+  from datetime import date
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+
+  today = date.today()
+  ret = []
+
+  #-- Isolines
+  service = 'isolines'
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  user_service = UserMetricsService(user_isolines_config, redis_conn)
+
+  monthly_quota = user_isolines_config.isolines_quota
+  used_quota = user_service.used_quota(user_isolines_config.service_type, today)
+  soft_limit = user_isolines_config.soft_isolines_limit
+  provider = user_isolines_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  #-- Hires Geocoder
+  service = 'hires_geocoder'
+  plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
+  user_service = UserMetricsService(user_geocoder_config, redis_conn)
+
+  monthly_quota = user_geocoder_config.geocoding_quota
+  used_quota = user_service.used_quota(user_geocoder_config.service_type, today)
+  soft_limit = user_geocoder_config.soft_geocoding_limit
+  provider = user_geocoder_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  #-- Routing
+  service = 'routing'
+  plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_routing_config = GD["user_routing_config_{0}".format(username)]
+  user_service = UserMetricsService(user_routing_config, redis_conn)
+
+  monthly_quota = user_routing_config.monthly_quota
+  used_quota = user_service.used_quota(user_routing_config.service_type, today)
+  soft_limit = user_routing_config.soft_limit
+  provider = user_routing_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  #-- Observatory
+  service = 'observatory'
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+  user_service = UserMetricsService(user_obs_config, redis_conn)
+
+  monthly_quota = user_obs_config.monthly_quota
+  used_quota = user_service.used_quota(user_obs_config.service_type, today)
+  soft_limit = user_obs_config.soft_limit
+  provider = user_obs_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  return ret
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_enough_quota(
+  username TEXT,
+  orgname TEXT,
+  service_ TEXT,
+  input_size NUMERIC)
+returns BOOLEAN AS $$
+  DECLARE
+    params cdb_dataservices_server.service_quota_info;
+  BEGIN
+    SELECT * INTO params
+      FROM cdb_dataservices_server.cdb_service_quota_info(username, orgname) AS p
+      WHERE p.service = service_::cdb_dataservices_server.service_type;
+    RETURN params.soft_limit OR ((params.used_quota + input_size) <= params.monthly_quota);
+  END
+$$ LANGUAGE plpgsql;

server/extension/old_versions/cdb_dataservices_server--0.19.0--0.18.1.sql

@@ -0,0 +1,17 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.18.1'" to load this file. \quit
+
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_enough_quota(
+  username TEXT,
+  orgname TEXT,
+  service_ TEXT,
+  input_size NUMERIC);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_service_quota_info(
+  username TEXT,
+  orgname TEXT);
+
+DROP TYPE IF EXISTS cdb_dataservices_server.service_quota_info;
+
+DROP TYPE IF EXISTS cdb_dataservices_server.service_type;

server/extension/sql/105_route_between_points.sql

@@ -7,15 +7,21 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_point_to_point(
   options text[] DEFAULT ARRAY[]::text[],
   units text DEFAULT 'kilometers')
 RETURNS cdb_dataservices_server.simple_route AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_routing_config = GD["user_routing_config_{0}".format(username)]
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
 
-  waypoints = [origin, destination]
-  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
-  result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
-  return [result[0]['shape'],result[0]['length'], result[0]['duration']]
+  with metrics('cdb_route_with_point', user_routing_config, logger):
+    waypoints = [origin, destination]
+    mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
+    result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
+    return [result[0]['shape'],result[0]['length'], result[0]['duration']]
 $$ LANGUAGE plpythonu;
 
 
@@ -27,12 +33,18 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_with_waypoints(
   options text[] DEFAULT ARRAY[]::text[],
   units text DEFAULT 'kilometers')
 RETURNS cdb_dataservices_server.simple_route AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_routing_config = GD["user_routing_config_{0}".format(username)]
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
 
-  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
-  result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
-  return [result[0]['shape'],result[0]['length'], result[0]['duration']]
+  with metrics('cdb_route_with_waypoints', user_routing_config, logger):
+    mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
+    result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
+    return [result[0]['shape'],result[0]['length'], result[0]['duration']]
 $$ LANGUAGE plpythonu;

server/extension/sql/110_data_observatory_augmentation.sql

@@ -11,9 +11,9 @@ RETURNS text AS $$
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
-  user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
+  user_obs_config = GD["user_obs_snapshot_config_{0}".format(username)]
 
-  return user_obs_snapshot_config.connection_str
+  return user_obs_config.connection_str
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetDemographicSnapshotJSON(
@@ -34,6 +34,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_demographic_snapshot(
   time_span TEXT DEFAULT NULL,
   geometry_level TEXT DEFAULT NULL)
 RETURNS json AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
   import json
@@ -41,31 +42,32 @@ RETURNS json AS $$
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
-  user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
+  user_obs_config = GD["user_obs_snapshot_config_{0}".format(username)]
 
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
-  quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
+  quota_service = QuotaService(user_obs_config, redis_conn)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetDemographicSnapshotJSON($1, $2, $3, $4, $5) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, time_span, geometry_level])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['snapshot']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to obs_get_demographic_snapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to obs_get_demographic_snapshot')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getdemographicsnapshot', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetDemographicSnapshotJSON($1, $2, $3, $4, $5) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, time_span, geometry_level])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['snapshot']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to obs_get_demographic_snapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to obs_get_demographic_snapshot')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetDemographicSnapshot(
@@ -86,41 +88,43 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetDemographicSnapshot(
   time_span TEXT DEFAULT NULL,
   geometry_level TEXT DEFAULT NULL)
 RETURNS SETOF JSON AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
-  user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
+  user_obs_config = GD["user_obs_snapshot_config_{0}".format(username)]
 
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
-  quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
+  quota_service = QuotaService(user_obs_config, redis_conn)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetDemographicSnapshot($1, $2, $3, $4, $5) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, time_span, geometry_level])
-      if result:
-        resp = []
-        for element in result:
-          value = element['snapshot']
-          resp.append(value)
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return []
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to obs_get_demographic_snapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to obs_get_demographic_snapshot')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getdemographicsnapshot', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetDemographicSnapshot($1, $2, $3, $4, $5) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, time_span, geometry_level])
+        if result:
+          resp = []
+          for element in result:
+            value = element['snapshot']
+            resp.append(value)
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to obs_get_demographic_snapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to obs_get_demographic_snapshot')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetSegmentSnapshotJSON(
@@ -139,6 +143,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_segment_snapshot(
   geom geometry(Geometry, 4326),
   geometry_level TEXT DEFAULT NULL)
 RETURNS json AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
   import json
@@ -146,31 +151,32 @@ RETURNS json AS $$
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
-  user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
+  user_obs_config = GD["user_obs_snapshot_config_{0}".format(username)]
 
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
-  quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
+  quota_service = QuotaService(user_obs_config, redis_conn)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetSegmentSnapshotJSON($1, $2, $3, $4) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, geometry_level])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['snapshot']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to obs_get_segment_snapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to obs_get_segment_snapshot')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getsegmentsnapshot', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetSegmentSnapshotJSON($1, $2, $3, $4) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, geometry_level])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['snapshot']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to obs_get_segment_snapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to obs_get_segment_snapshot')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetSegmentSnapshot(
@@ -189,41 +195,43 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetSegmentSnapshot(
   geom geometry(Geometry, 4326),
   geometry_level TEXT DEFAULT NULL)
 RETURNS SETOF JSON AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
-  user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
+  user_obs_config = GD["user_obs_snapshot_config_{0}".format(username)]
 
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
-  quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
+  quota_service = QuotaService(user_obs_config, redis_conn)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetSegmentSnapshot($1, $2, $3, $4) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, geometry_level])
-      if result:
-        resp = []
-        for element in result:
-          value = element['snapshot']
-          resp.append(value)
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return []
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetSegmentSnapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetSegmentSnapshot')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getsegmentsnapshot', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetSegmentSnapshot($1, $2, $3, $4) as snapshot;", ["text", "text", "geometry(Geometry, 4326)", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, geometry_level])
+        if result:
+          resp = []
+          for element in result:
+            value = element['snapshot']
+            resp.append(value)
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetSegmentSnapshot', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetSegmentSnapshot')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeasure(
@@ -248,6 +256,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeasure(
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL)
 RETURNS NUMERIC AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -263,22 +272,23 @@ RETURNS NUMERIC AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeasure($1, $2, $3, $4, $5, $6, $7) as measure;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, measure_id, normalize, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['measure']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetMeasure', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetMeasure')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getmeasure', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeasure($1, $2, $3, $4, $5, $6, $7) as measure;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, measure_id, normalize, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['measure']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetMeasure', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetMeasure')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetCategory(
@@ -301,6 +311,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetCategory(
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL)
 RETURNS TEXT AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -316,22 +327,23 @@ RETURNS TEXT AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetCategory($1, $2, $3, $4, $5, $6) as category;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, category_id, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['category']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetCategory', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetCategory')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getcategory', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetCategory($1, $2, $3, $4, $5, $6) as category;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, category_id, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['category']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetCategory', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetCategory')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetUSCensusMeasure(
@@ -356,6 +368,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetUSCensusMeasure(
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL)
 RETURNS NUMERIC AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -371,22 +384,23 @@ RETURNS NUMERIC AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetUSCensusMeasure($1, $2, $3, $4, $5, $6, $7) as census_measure;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, name, normalize, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['census_measure']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetUSCensusMeasure', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetUSCensusMeasure')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getuscensusmeasure', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetUSCensusMeasure($1, $2, $3, $4, $5, $6, $7) as census_measure;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, name, normalize, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['census_measure']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetUSCensusMeasure', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetUSCensusMeasure')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetUSCensusCategory(
@@ -409,6 +423,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetUSCensusCategory(
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL)
 RETURNS TEXT AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -424,22 +439,23 @@ RETURNS TEXT AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetUSCensusCategory($1, $2, $3, $4, $5, $6) as census_category;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, name, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['census_category']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetUSCensusCategory', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetUSCensusCategory')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getuscensuscategory', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetUSCensusCategory($1, $2, $3, $4, $5, $6) as census_category;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, name, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['census_category']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetUSCensusCategory', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetUSCensusCategory')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPopulation(
@@ -462,6 +478,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetPopulation(
   boundary_id TEXT DEFAULT NULL,
   time_span TEXT DEFAULT NULL)
 RETURNS NUMERIC AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -477,22 +494,23 @@ RETURNS NUMERIC AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetPopulation($1, $2, $3, $4, $5, $6) as population;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, normalize, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['population']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetPopulation', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetPopulation')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getpopulation', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetPopulation($1, $2, $3, $4, $5, $6) as population;", ["text", "text", "geometry(Geometry, 4326)", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, normalize, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['population']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetPopulation', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetPopulation')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeasureById(
@@ -515,6 +533,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeasureById(
   boundary_id TEXT,
   time_span TEXT DEFAULT NULL)
 RETURNS NUMERIC AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -530,20 +549,21 @@ RETURNS NUMERIC AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeasureById($1, $2, $3, $4, $5, $6) as measure;", ["text", "text", "text", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom_ref, measure_id, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['measure']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetMeasureById', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetMeasureById')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getmeasurebyid', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeasureById($1, $2, $3, $4, $5, $6) as measure;", ["text", "text", "text", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom_ref, measure_id, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['measure']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetMeasureById', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetMeasureById')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;

server/extension/sql/115_data_observatory_exploration.sql

@@ -14,6 +14,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_Search(
   search_term TEXT,
   relevant_boundary TEXT DEFAULT NULL)
 RETURNS TABLE(id text, description text, name text, aggregate text, source text) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -29,30 +30,31 @@ RETURNS TABLE(id text, description text, name text, aggregate text, source text)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_Search($1, $2, $3, $4);", ["text", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, search_term, relevant_boundary])
-      if result:
-        resp = []
-        for element in result:
-          id = element['id']
-          description = element['description']
-          name = element['name']
-          aggregate = element['aggregate']
-          source = element['source']
-          resp.append([id, description, name, aggregate, source])
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return [None, None, None, None, None]
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_Search', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_Search')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_search', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_Search($1, $2, $3, $4);", ["text", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, search_term, relevant_boundary])
+        if result:
+          resp = []
+          for element in result:
+            id = element['id']
+            description = element['description']
+            name = element['name']
+            aggregate = element['aggregate']
+            source = element['source']
+            resp.append([id, description, name, aggregate, source])
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return [None, None, None, None, None]
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_Search', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_Search')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetAvailableBoundaries(
@@ -71,6 +73,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableBoundaries(
   geom geometry(Geometry, 4326),
   time_span TEXT DEFAULT NULL)
 RETURNS TABLE(boundary_id text, description text, time_span text, tablename text) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -86,27 +89,28 @@ RETURNS TABLE(boundary_id text, description text, time_span text, tablename text
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetAvailableBoundaries($1, $2, $3, $4) as available_boundaries;", ["text", "text", "geometry(Geometry, 4326)", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, time_span])
-      if result:
-        resp = []
-        for element in result:
-          id = element['boundary_id']
-          description = element['description']
-          tspan = element['time_span']
-          tablename = element['tablename']
-          resp.append([id, description, tspan, tablename])
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return []
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetMeasureById', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetMeasureById')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getavailableboundaries', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetAvailableBoundaries($1, $2, $3, $4) as available_boundaries;", ["text", "text", "geometry(Geometry, 4326)", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, time_span])
+        if result:
+          resp = []
+          for element in result:
+            id = element['boundary_id']
+            description = element['description']
+            tspan = element['time_span']
+            tablename = element['tablename']
+            resp.append([id, description, tspan, tablename])
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetMeasureById', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetMeasureById')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;

server/extension/sql/120_data_observatory_geometries.sql

@@ -16,6 +16,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundary(
   boundary_id TEXT,
   time_span TEXT DEFAULT NULL)
 RETURNS geometry(Geometry, 4326) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -31,22 +32,23 @@ RETURNS geometry(Geometry, 4326) AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetBoundary($1, $2, $3, $4) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['boundary']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetBoundary', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetBoundary')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getboundary', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetBoundary($1, $2, $3, $4) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['boundary']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetBoundary', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetBoundary')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundaryId(
@@ -67,6 +69,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundaryId(
   boundary_id TEXT,
   time_span TEXT DEFAULT NULL)
 RETURNS TEXT AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -82,22 +85,23 @@ RETURNS TEXT AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetBoundaryId($1, $2, $3, $4, $5) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['boundary']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetBoundaryId', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetBoundaryId')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getboundaryid', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetBoundaryId($1, $2, $3, $4, $5) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['boundary']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetBoundaryId', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetBoundaryId')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundaryById(
@@ -118,6 +122,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundaryById(
   boundary_id TEXT,
   time_span TEXT DEFAULT NULL)
 RETURNS geometry(Geometry, 4326) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -133,22 +138,23 @@ RETURNS geometry(Geometry, 4326) AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetBoundaryById($1, $2, $3, $4, $5) as boundary;", ["text", "text", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geometry_id, boundary_id, time_span])
-      if result:
-        quota_service.increment_success_service_use()
-        return result[0]['boundary']
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetBoundaryById', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetBoundaryById')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getboundarybyid', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetBoundaryById($1, $2, $3, $4, $5) as boundary;", ["text", "text", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geometry_id, boundary_id, time_span])
+        if result:
+          quota_service.increment_success_service_use()
+          return result[0]['boundary']
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetBoundaryById', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetBoundaryById')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundariesByGeometry(
@@ -171,6 +177,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundariesByGeometry(
   time_span TEXT DEFAULT NULL,
   overlap_type TEXT DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -186,27 +193,28 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetBoundariesByGeometry($1, $2, $3, $4, $5, $6) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span, overlap_type])
-      if result:
-        resp = []
-        for element in result:
-          the_geom = element['the_geom']
-          geom_refs = element['geom_refs']
-          resp.append([the_geom, geom_refs])
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return []
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetBoundariesByGeometry', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetBoundariesByGeometry')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getboundariesbygeometry', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetBoundariesByGeometry($1, $2, $3, $4, $5, $6) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span, overlap_type])
+        if result:
+          resp = []
+          for element in result:
+            the_geom = element['the_geom']
+            geom_refs = element['geom_refs']
+            resp.append([the_geom, geom_refs])
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetBoundariesByGeometry', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetBoundariesByGeometry')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundariesByPointAndRadius(
@@ -231,6 +239,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundariesByPointAndRa
   time_span TEXT DEFAULT NULL,
   overlap_type TEXT DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -246,27 +255,28 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetBoundariesByPointAndRadius($1, $2, $3, $4, $5, $6, $7) as boundary;", ["text", "text", "geometry(Point, 4326)", "numeric", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, radius, boundary_id, time_span, overlap_type])
-      if result:
-        resp = []
-        for element in result:
-          the_geom = element['the_geom']
-          geom_refs = element['geom_refs']
-          resp.append([the_geom, geom_refs])
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return []
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetBoundariesByPointAndRadius', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetBoundariesByPointAndRadius')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getboundariesbypointandradius', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetBoundariesByPointAndRadius($1, $2, $3, $4, $5, $6, $7) as boundary;", ["text", "text", "geometry(Point, 4326)", "numeric", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, radius, boundary_id, time_span, overlap_type])
+        if result:
+          resp = []
+          for element in result:
+            the_geom = element['the_geom']
+            geom_refs = element['geom_refs']
+            resp.append([the_geom, geom_refs])
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetBoundariesByPointAndRadius', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetBoundariesByPointAndRadius')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPointsByGeometry(
@@ -289,6 +299,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetPointsByGeometry(
   time_span TEXT DEFAULT NULL,
   overlap_type TEXT DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -304,27 +315,28 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetPointsByGeometry($1, $2, $3, $4, $5, $6) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span, overlap_type])
-      if result:
-        resp = []
-        for element in result:
-          the_geom = element['the_geom']
-          geom_refs = element['geom_refs']
-          resp.append([the_geom, geom_refs])
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return []
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetPointsByGeometry', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetPointsByGeometry')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getpointsbygeometry', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetPointsByGeometry($1, $2, $3, $4, $5, $6) as boundary;", ["text", "text", "geometry(Point, 4326)", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, boundary_id, time_span, overlap_type])
+        if result:
+          resp = []
+          for element in result:
+            the_geom = element['the_geom']
+            geom_refs = element['geom_refs']
+            resp.append([the_geom, geom_refs])
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetPointsByGeometry', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetPointsByGeometry')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPointsByPointAndRadius(
@@ -349,6 +361,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetPointsByPointAndRadius
   time_span TEXT DEFAULT NULL,
   overlap_type TEXT DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.tools import Logger,LoggerConfig
 
@@ -364,25 +377,26 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-      obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetPointsByPointAndRadius($1, $2, $3, $4, $5, $6, $7) as boundary;", ["text", "text", "geometry(Point, 4326)", "numeric", "text", "text", "text"])
-      result = plpy.execute(obs_plan, [username, orgname, geom, radius, boundary_id, time_span, overlap_type])
-      if result:
-        resp = []
-        for element in result:
-          the_geom = element['the_geom']
-          geom_refs = element['geom_refs']
-          resp.append([the_geom, geom_refs])
-        quota_service.increment_success_service_use()
-        return resp
-      else:
-        quota_service.increment_empty_service_use()
-        return None
-  except BaseException as e:
-      import sys
-      quota_service.increment_failed_service_use()
-      logger.error('Error trying to OBS_GetPointsByPointAndRadius', sys.exc_info(), data={"username": username, "orgname": orgname})
-      raise Exception('Error trying to OBS_GetPointsByPointAndRadius')
-  finally:
-      quota_service.increment_total_service_use()
+  with metrics('obs_getpointsbypointandradius', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetPointsByPointAndRadius($1, $2, $3, $4, $5, $6, $7) as boundary;", ["text", "text", "geometry(Point, 4326)", "numeric", "text", "text", "text"])
+        result = plpy.execute(obs_plan, [username, orgname, geom, radius, boundary_id, time_span, overlap_type])
+        if result:
+          resp = []
+          for element in result:
+            the_geom = element['the_geom']
+            geom_refs = element['geom_refs']
+            resp.append([the_geom, geom_refs])
+          quota_service.increment_success_service_use()
+          return resp
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to OBS_GetPointsByPointAndRadius', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetPointsByPointAndRadius')
+    finally:
+        quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;

server/extension/sql/130_data_observatory_helper.sql

@@ -3,3 +3,73 @@ RETURNS text AS $$
   CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
   SELECT cdb_observatory.obs_dumpversion();
 $$ LANGUAGE plproxy;
+
+-- We could create a super type for the common data like id, name and so on but we need to parse inside the functions because the -- the return data tha comes from OBS is a TABLE() with them
+CREATE TYPE cdb_dataservices_server.obs_meta_numerator AS (numer_id text, numer_name text, numer_description text, numer_weight text, numer_license text, numer_source text, numer_type text, numer_aggregate text, numer_extra jsonb, numer_tags jsonb, valid_denom boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableNumerators(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan);
+$$ LANGUAGE plproxy;
+
+CREATE TYPE cdb_dataservices_server.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableDenominators(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_denominator AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan);
+$$ LANGUAGE plproxy;
+
+CREATE TYPE cdb_dataservices_server.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_geometry AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan);
+$$ LANGUAGE plproxy;
+
+CREATE TYPE cdb_dataservices_server.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean);
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableTimespans(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_server.obs_meta_timespan AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id);
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_LegacyBuilderMetadata(
+  username TEXT,
+  orgname TEXT,
+  aggregate_type TEXT DEFAULT NULL)
+RETURNS TABLE(name TEXT, subsection JSON) AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_LegacyBuilderMetadata(aggregate_type);
+$$ LANGUAGE plproxy;

server/extension/sql/15_config_helper.sql

@@ -10,6 +10,12 @@ RETURNS boolean AS $$
     return True
 $$ LANGUAGE plpythonu SECURITY DEFINER;
 
+-- This is done in order to avoid an undesired depedency on cartodb extension
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_conf_getconf(input_key text)
+RETURNS JSON AS $$
+    SELECT VALUE FROM cartodb.cdb_conf WHERE key = input_key;
+$$ LANGUAGE SQL STABLE SECURITY DEFINER;
+
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_geocoder_config(username text, orgname text, provider text DEFAULT NULL)
 RETURNS boolean AS $$
   cache_key = "user_geocoder_config_{0}".format(username)

server/extension/sql/200_quotas.sql

@@ -0,0 +1,105 @@
+DO $$
+BEGIN
+  IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'service_type') THEN
+    CREATE TYPE cdb_dataservices_server.service_type AS ENUM (
+      'isolines',
+      'hires_geocoder',
+      'routing',
+      'observatory'
+    );
+  END IF;
+END $$;
+
+DO $$
+BEGIN
+  IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'service_quota_info') THEN
+    CREATE TYPE cdb_dataservices_server.service_quota_info AS (
+      service cdb_dataservices_server.service_type,
+      monthly_quota NUMERIC,
+      used_quota NUMERIC,
+      soft_limit BOOLEAN,
+      provider TEXT
+    );
+  END IF;
+END $$;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_quota_info(
+  username TEXT,
+  orgname TEXT)
+RETURNS SETOF cdb_dataservices_server.service_quota_info AS $$
+  from cartodb_services.metrics.user import UserMetricsService
+  from datetime import date
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+
+  today = date.today()
+  ret = []
+
+  #-- Isolines
+  service = 'isolines'
+  plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  user_service = UserMetricsService(user_isolines_config, redis_conn)
+
+  monthly_quota = user_isolines_config.isolines_quota
+  used_quota = user_service.used_quota(user_isolines_config.service_type, today)
+  soft_limit = user_isolines_config.soft_isolines_limit
+  provider = user_isolines_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  #-- Hires Geocoder
+  service = 'hires_geocoder'
+  plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
+  user_service = UserMetricsService(user_geocoder_config, redis_conn)
+
+  monthly_quota = user_geocoder_config.geocoding_quota
+  used_quota = user_service.used_quota(user_geocoder_config.service_type, today)
+  soft_limit = user_geocoder_config.soft_geocoding_limit
+  provider = user_geocoder_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  #-- Routing
+  service = 'routing'
+  plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_routing_config = GD["user_routing_config_{0}".format(username)]
+  user_service = UserMetricsService(user_routing_config, redis_conn)
+
+  monthly_quota = user_routing_config.monthly_quota
+  used_quota = user_service.used_quota(user_routing_config.service_type, today)
+  soft_limit = user_routing_config.soft_limit
+  provider = user_routing_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  #-- Observatory
+  service = 'observatory'
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+  user_service = UserMetricsService(user_obs_config, redis_conn)
+
+  monthly_quota = user_obs_config.monthly_quota
+  used_quota = user_service.used_quota(user_obs_config.service_type, today)
+  soft_limit = user_obs_config.soft_limit
+  provider = user_obs_config.provider
+  ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
+
+  return ret
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_enough_quota(
+  username TEXT,
+  orgname TEXT,
+  service_ TEXT,
+  input_size NUMERIC)
+returns BOOLEAN AS $$
+  DECLARE
+    params cdb_dataservices_server.service_quota_info;
+  BEGIN
+    SELECT * INTO params
+      FROM cdb_dataservices_server.cdb_service_quota_info(username, orgname) AS p
+      WHERE p.service = service_::cdb_dataservices_server.service_type;
+    RETURN params.soft_limit OR ((params.used_quota + input_size) <= params.monthly_quota);
+  END
+$$ LANGUAGE plpgsql;

server/extension/sql/20_geocode_street.sql

@@ -1,22 +1,28 @@
 -- Geocodes a street address given a searchtext and a state and/or country
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger,LoggerConfig
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
 
-  if user_geocoder_config.heremaps_geocoder:
-    here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
-    return plpy.execute(here_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
-  elif user_geocoder_config.google_geocoder:
-    google_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_google_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
-    return plpy.execute(google_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
-  elif user_geocoder_config.mapzen_geocoder:
-    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
-    return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
-  else:
-    raise Exception('Requested geocoder is not available')
+  with metrics('cdb_geocode_street_point', user_geocoder_config, logger):
+    if user_geocoder_config.heremaps_geocoder:
+      here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+      return plpy.execute(here_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+    elif user_geocoder_config.google_geocoder:
+      google_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_google_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+      return plpy.execute(google_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+    elif user_geocoder_config.mapzen_geocoder:
+      mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
+      return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
+    else:
+      raise Exception('Requested geocoder is not available')
 
 $$ LANGUAGE plpythonu;
 
@@ -137,23 +143,38 @@ $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
 RETURNS Geometry AS $$
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
   from cartodb_services.mapzen import MapzenGeocoder
   from cartodb_services.mapzen.types import country_to_iso3
   from cartodb_services.metrics import QuotaService
-  from cartodb_services.tools import Logger,LoggerConfig
+  from cartodb_services.tools import Logger
+  from cartodb_services.refactor.tools.logger import LoggerConfigBuilder
+  from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
+  from cartodb_services.refactor.core.environment import ServerEnvironmentBuilder
+  from cartodb_services.refactor.backend.server_config import ServerConfigBackendFactory
+  from cartodb_services.refactor.backend.user_config import UserConfigBackendFactory
+  from cartodb_services.refactor.backend.org_config import OrgConfigBackendFactory
+  from cartodb_services.refactor.backend.redis_metrics_connection import RedisMetricsConnectionFactory
 
-  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
-  user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
+  server_config_backend = ServerConfigBackendFactory().get()
+  environment = ServerEnvironmentBuilder(server_config_backend).get()
+  user_config_backend = UserConfigBackendFactory(username, environment, server_config_backend).get()
+  org_config_backend = OrgConfigBackendFactory(orgname, environment, server_config_backend).get()
 
-  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
-  logger_config = GD["logger_config"]
+  logger_config = LoggerConfigBuilder(environment, server_config_backend).get()
   logger = Logger(logger_config)
-  quota_service = QuotaService(user_geocoder_config, redis_conn)
+
+  mapzen_geocoder_config = MapzenGeocoderConfigBuilder(server_config_backend, user_config_backend, org_config_backend, username, orgname).get()
+
+  redis_metrics_connection = RedisMetricsConnectionFactory(environment, server_config_backend).get()
+
+  quota_service = QuotaService(mapzen_geocoder_config, redis_metrics_connection)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
   try:
-    geocoder = MapzenGeocoder(user_geocoder_config.mapzen_api_key, logger)
+    geocoder = MapzenGeocoder(mapzen_geocoder_config.mapzen_api_key, logger)
     country_iso3 = None
     if country:
       country_iso3 = country_to_iso3(country)

server/extension/sql/30_admin0.sql

@@ -2,6 +2,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_admin0_polygon(us
 RETURNS Geometry AS $$
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
+  from cartodb_services.metrics import metrics
   from cartodb_services.tools import Logger,LoggerConfig
 
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
@@ -13,23 +14,24 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_admin0_polygon(trim($1)) AS mypolygon", ["text"])
-    rv = plpy.execute(plan, [country_name], 1)
-    result = rv[0]["mypolygon"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode admin0 polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode admin0 polygon')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_admin0_polygon', user_geocoder_config, logger):
+    try:
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_admin0_polygon(trim($1)) AS mypolygon", ["text"])
+      rv = plpy.execute(plan, [country_name], 1)
+      result = rv[0]["mypolygon"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode admin0 polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode admin0 polygon')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 

server/extension/sql/40_admin1.sql

@@ -1,6 +1,7 @@
 ---- cdb_geocode_admin1_polygon(admin1_name text)
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_admin1_polygon(username text, orgname text, admin1_name text)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
   from cartodb_services.tools import Logger,LoggerConfig
@@ -14,44 +15,11 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_admin1_polygon(trim($1)) AS mypolygon", ["text"])
-    rv = plpy.execute(plan, [admin1_name], 1)
-    result = rv[0]["mypolygon"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode admin1 polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode admin1 polygon')
-  finally:
-    quota_service.increment_total_service_use()
-$$ LANGUAGE plpythonu;
 
----- cdb_geocode_admin1_polygon(admin1_name text, country_name text)
-CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_admin1_polygon(username text, orgname text, admin1_name text, country_name text)
-RETURNS Geometry AS $$
-    from cartodb_services.metrics import QuotaService
-    from cartodb_services.metrics import InternalGeocoderConfig
-    from cartodb_services.tools import Logger,LoggerConfig
-
-    plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
-    redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
-    plpy.execute("SELECT cdb_dataservices_server._get_internal_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
-    user_geocoder_config = GD["user_internal_geocoder_config_{0}".format(username)]
-
-    plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
-    logger_config = GD["logger_config"]
-    logger = Logger(logger_config)
-    quota_service = QuotaService(user_geocoder_config, redis_conn)
+  with metrics('cdb_geocode_admin1_polygon', user_geocoder_config, logger):
     try:
-      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_admin1_polygon(trim($1), trim($2)) AS mypolygon", ["text", "text"])
-      rv = plpy.execute(plan, [admin1_name, country_name], 1)
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_admin1_polygon(trim($1)) AS mypolygon", ["text"])
+      rv = plpy.execute(plan, [admin1_name], 1)
       result = rv[0]["mypolygon"]
       if result:
         quota_service.increment_success_service_use()
@@ -68,6 +36,44 @@ RETURNS Geometry AS $$
       quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
+---- cdb_geocode_admin1_polygon(admin1_name text, country_name text)
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_admin1_polygon(username text, orgname text, admin1_name text, country_name text)
+RETURNS Geometry AS $$
+    from cartodb_services.metrics import metrics
+    from cartodb_services.metrics import QuotaService
+    from cartodb_services.metrics import InternalGeocoderConfig
+    from cartodb_services.tools import Logger,LoggerConfig
+
+    plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+    redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+    plpy.execute("SELECT cdb_dataservices_server._get_internal_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+    user_geocoder_config = GD["user_internal_geocoder_config_{0}".format(username)]
+
+    plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+    logger_config = GD["logger_config"]
+    logger = Logger(logger_config)
+    quota_service = QuotaService(user_geocoder_config, redis_conn)
+
+    with metrics('cdb_geocode_admin1_polygon', user_geocoder_config, logger):
+      try:
+        plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_admin1_polygon(trim($1), trim($2)) AS mypolygon", ["text", "text"])
+        rv = plpy.execute(plan, [admin1_name, country_name], 1)
+        result = rv[0]["mypolygon"]
+        if result:
+          quota_service.increment_success_service_use()
+          return result
+        else:
+          quota_service.increment_empty_service_use()
+          return None
+      except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use()
+        logger.error('Error trying to geocode admin1 polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to geocode admin1 polygon')
+      finally:
+        quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu;
+
 --------------------------------------------------------------------------------
 
 -- Implementation of the server extension

server/extension/sql/50_namedplaces.sql

@@ -35,7 +35,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_namedplac
 RETURNS Geometry AS $$
   from cartodb_services.mapzen import MapzenGeocoder
   from cartodb_services.mapzen.types import country_to_iso3
-  from cartodb_services.metrics import QuotaService
+  from cartodb_services.metrics import QuotaService, metrics
   from cartodb_services.tools import Logger,LoggerConfig
 
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
@@ -50,35 +50,36 @@ RETURNS Geometry AS $$
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
 
-  try:
-    geocoder = MapzenGeocoder(user_geocoder_config.mapzen_api_key, logger)
-    country_iso3 = None
-    if country_name:
-      country_iso3 = country_to_iso3(country_name)
-    coordinates = geocoder.geocode(searchtext=city_name, city=None,
-                                   state_province=admin1_name,
-                                   country=country_iso3, search_type='locality')
-    if coordinates:
-      quota_service.increment_success_service_use()
-      plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
-      point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
-      return point['st_setsrid']
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode city point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode city point using mapzen')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_namedplace_point', user_geocoder_config, logger):
+    try:
+      geocoder = MapzenGeocoder(user_geocoder_config.mapzen_api_key, logger)
+      country_iso3 = None
+      if country_name:
+        country_iso3 = country_to_iso3(country_name)
+      coordinates = geocoder.geocode(searchtext=city_name, city=None,
+                                    state_province=admin1_name,
+                                    country=country_iso3, search_type='locality')
+      if coordinates:
+        quota_service.increment_success_service_use()
+        plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
+        point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
+        return point['st_setsrid']
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode city point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode city point using mapzen')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_internal_geocode_namedplace(username text, orgname text, city_name text, admin1_name text DEFAULT NULL, country_name text DEFAULT NULL)
 RETURNS Geometry AS $$
   from cartodb_services.metrics import QuotaService
-  from cartodb_services.metrics import InternalGeocoderConfig
+  from cartodb_services.metrics import InternalGeocoderConfig, metrics
   from cartodb_services.tools import Logger,LoggerConfig
 
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
@@ -90,30 +91,32 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    if admin1_name and country_name:
-      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_namedplace_point(trim($1), trim($2), trim($3)) AS mypoint", ["text", "text", "text"])
-      rv = plpy.execute(plan, [city_name, admin1_name, country_name], 1)
-    elif country_name:
-      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_namedplace_point(trim($1), trim($2)) AS mypoint", ["text", "text"])
-      rv = plpy.execute(plan, [city_name, country_name], 1)
-    else:
-      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_namedplace_point(trim($1)) AS mypoint", ["text"])
-      rv = plpy.execute(plan, [city_name], 1)
-    result = rv[0]["mypoint"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode namedplace point', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode namedplace point')
-  finally:
-    quota_service.increment_total_service_use()
+
+  with metrics('cdb_geocode_namedplace_point', user_geocoder_config, logger):
+    try:
+      if admin1_name and country_name:
+        plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_namedplace_point(trim($1), trim($2), trim($3)) AS mypoint", ["text", "text", "text"])
+        rv = plpy.execute(plan, [city_name, admin1_name, country_name], 1)
+      elif country_name:
+        plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_namedplace_point(trim($1), trim($2)) AS mypoint", ["text", "text"])
+        rv = plpy.execute(plan, [city_name, country_name], 1)
+      else:
+        plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_namedplace_point(trim($1)) AS mypoint", ["text"])
+        rv = plpy.execute(plan, [city_name], 1)
+      result = rv[0]["mypoint"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode namedplace point', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode namedplace point')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 --------------------------------------------------------------------------------

server/extension/sql/60_postalcodes.sql

@@ -1,5 +1,6 @@
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code text)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
   from cartodb_services.tools import Logger,LoggerConfig
@@ -13,27 +14,29 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_point(trim($1)) AS mypoint", ["text"])
-    rv = plpy.execute(plan, [code], 1)
-    result = rv[0]["mypoint"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode postal code point', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode postal code point')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_postalcode_point', user_geocoder_config, logger):
+    try:
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_point(trim($1)) AS mypoint", ["text"])
+      rv = plpy.execute(plan, [code], 1)
+      result = rv[0]["mypoint"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode postal code point', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode postal code point')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code text, country text)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
   from cartodb_services.tools import Logger,LoggerConfig
@@ -47,27 +50,29 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_point(trim($1), trim($2)) AS mypoint", ["TEXT", "TEXT"])
-    rv = plpy.execute(plan, [code, country], 1)
-    result = rv[0]["mypoint"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode postal code point', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode postal code point')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_postalcode_point', user_geocoder_config, logger):
+    try:
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_point(trim($1), trim($2)) AS mypoint", ["TEXT", "TEXT"])
+      rv = plpy.execute(plan, [code, country], 1)
+      result = rv[0]["mypoint"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode postal code point', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode postal code point')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code text)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
   from cartodb_services.tools import Logger,LoggerConfig
@@ -81,27 +86,29 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_polygon(trim($1)) AS mypolygon", ["text"])
-    rv = plpy.execute(plan, [code], 1)
-    result = rv[0]["mypolygon"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode postal code polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode postal code polygon')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_postalcode_point', user_geocoder_config, logger):
+    try:
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_polygon(trim($1)) AS mypolygon", ["text"])
+      rv = plpy.execute(plan, [code], 1)
+      result = rv[0]["mypolygon"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode postal code polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode postal code polygon')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code text, country text)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
   from cartodb_services.tools import Logger,LoggerConfig
@@ -115,23 +122,24 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_polygon(trim($1), trim($2)) AS mypolygon", ["TEXT", "TEXT"])
-    rv = plpy.execute(plan, [code, country], 1)
-    result = rv[0]["mypolygon"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode postal code polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode postal code polygon')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_postalcode_point', user_geocoder_config, logger):
+    try:
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_postalcode_polygon(trim($1), trim($2)) AS mypolygon", ["TEXT", "TEXT"])
+      rv = plpy.execute(plan, [code, country], 1)
+      result = rv[0]["mypolygon"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode postal code polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode postal code polygon')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 --------------------------------------------------------------------------------

server/extension/sql/70_ips.sql

@@ -1,5 +1,6 @@
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_ipaddress_point(username text, orgname text, ip text)
 RETURNS Geometry AS $$
+  from cartodb_services.metrics import metrics
   from cartodb_services.metrics import QuotaService
   from cartodb_services.metrics import InternalGeocoderConfig
   from cartodb_services.tools import Logger,LoggerConfig
@@ -13,23 +14,24 @@ RETURNS Geometry AS $$
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
   quota_service = QuotaService(user_geocoder_config, redis_conn)
-  try:
-    plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_ipaddress_point(trim($1)) AS mypoint", ["TEXT"])
-    rv = plpy.execute(plan, [ip], 1)
-    result = rv[0]["mypoint"]
-    if result:
-      quota_service.increment_success_service_use()
-      return result
-    else:
-      quota_service.increment_empty_service_use()
-      return None
-  except BaseException as e:
-    import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode postal code polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
-    raise Exception('Error trying to geocode postal code polygon')
-  finally:
-    quota_service.increment_total_service_use()
+  with metrics('cdb_geocode_ipaddress_point', user_geocoder_config, logger):
+    try:
+      plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_geocode_ipaddress_point(trim($1)) AS mypoint", ["TEXT"])
+      rv = plpy.execute(plan, [ip], 1)
+      result = rv[0]["mypoint"]
+      if result:
+        quota_service.increment_success_service_use()
+        return result
+      else:
+        quota_service.increment_empty_service_use()
+        return None
+    except BaseException as e:
+      import sys
+      quota_service.increment_failed_service_use()
+      logger.error('Error trying to geocode postal code polygon', sys.exc_info(), data={"username": username, "orgname": orgname})
+      raise Exception('Error trying to geocode postal code polygon')
+    finally:
+      quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 --------------------------------------------------------------------------------

server/extension/sql/80_isolines_helper.sql

@@ -57,11 +57,9 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
     quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu SECURITY DEFINER;
 
-
-CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isolines(
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isodistance(
    username TEXT,
    orgname TEXT,
-   isotype TEXT,
    source geometry(Geometry, 4326),
    mode TEXT,
    data_range integer[],
@@ -78,7 +76,6 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
   logger_config = GD["logger_config"]
   logger = Logger(logger_config)
-  # -- Check the quota
   quota_service = QuotaService(user_isolines_routing_config, redis_conn)
   if not quota_service.check_user_quota():
     raise Exception('You have reached the limit of your quota')
@@ -96,14 +93,9 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
 
     # -- TODO Support options properly
     isolines = {}
-    if isotype == 'isodistance':
-      for r in data_range:
-          isoline = mapzen_isolines.calculate_isodistance(origin, mode, r)
-          isolines[r] = isoline
-    elif isotype == 'isochrone':
-      for r in data_range:
-          isoline = mapzen_isolines.calculate_isochrone(origin, mode, r)
-          isolines[r] = isoline
+    for r in data_range:
+        isoline = mapzen_isolines.calculate_isodistance(origin, mode, r)
+        isolines[r] = isoline
 
     result = []
     for r in data_range:
@@ -130,3 +122,67 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
   finally:
     quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isochrones(
+   username TEXT,
+   orgname TEXT,
+   source geometry(Geometry, 4326),
+   mode TEXT,
+   data_range integer[],
+   options text[])
+RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  import json
+  from cartodb_services.mapzen import MatrixClient, MapzenIsochrones
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+  from cartodb_services.mapzen.types import coordinates_to_polygon
+
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  # -- Check the quota
+  quota_service = QuotaService(user_isolines_routing_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  try:
+    mapzen_isochrones = MapzenIsochrones(user_isolines_routing_config.mapzen_matrix_api_key,
+                                         logger)
+
+    if source:
+      lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
+      lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
+      origin = {'lat': lat, 'lon': lon}
+    else:
+      raise Exception('source is NULL')
+
+    resp = mapzen_isochrones.isochrone(origin, mode, data_range)
+
+    if resp:
+      result = []
+      for isochrone in resp:
+        result_polygon = coordinates_to_polygon(isochrone.coordinates)
+        if result_polygon:
+          quota_service.increment_success_service_use()
+          result.append([source, isochrone.duration, result_polygon])
+        else:
+          quota_service.increment_empty_service_use()
+          result.append([source, isochrone.duration, None])
+      quota_service.increment_success_service_use()
+      quota_service.increment_isolines_service_use(len(result))
+      return result
+    else:
+      quota_service.increment_empty_service_use()
+      return []
+  except BaseException as e:
+    import sys
+    quota_service.increment_failed_service_use()
+    logger.error('Error trying to get mapzen isochrones', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to get mapzen isochrones')
+  finally:
+    quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu SECURITY DEFINER;

server/extension/sql/85_isodistance.sql

@@ -1,21 +1,28 @@
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
 RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger
+
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
 
   if user_isolines_config.google_services_user:
     raise Exception('This service is not available for google service users.')
 
-  if user_isolines_config.heremaps_provider:
-    here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
-    return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
-  elif user_isolines_config.mapzen_provider:
-    mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
-    return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
-  else:
-    raise Exception('Requested isolines provider is not available')
+  with metrics('cb_isodistance', user_isolines_config, logger):
+    if user_isolines_config.heremaps_provider:
+      here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+      return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
+    elif user_isolines_config.mapzen_provider:
+      mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+      return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
+    else:
+      raise Exception('Requested isolines provider is not available')
 $$ LANGUAGE plpythonu;
 
 -- heremaps isodistance
@@ -40,10 +47,9 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
-  type = 'isodistance'
 
-  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
-  result = plpy.execute(mapzen_plan, [username, orgname, type, source, mode, range, options])
+  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+  result = plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
 
   return result
 $$ LANGUAGE plpythonu;

server/extension/sql/90_isochrone.sql

@@ -1,21 +1,28 @@
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
 RETURNS SETOF cdb_dataservices_server.isoline AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger
+
   plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
 
   if user_isolines_config.google_services_user:
     raise Exception('This service is not available for google service users.')
 
-  if user_isolines_config.heremaps_provider:
-    here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
-    return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
-  elif user_isolines_config.mapzen_provider:
-    mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
-    return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
-  else:
-    raise Exception('Requested isolines provider is not available')
+  with metrics('cb_isochrone', user_isolines_config, logger):
+    if user_isolines_config.heremaps_provider:
+      here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+      return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
+    elif user_isolines_config.mapzen_provider:
+      mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+      return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
+    else:
+      raise Exception('Requested isolines provider is not available')
 $$ LANGUAGE plpythonu;
 
 -- heremaps isochrone
@@ -40,9 +47,8 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
   redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
   plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
   user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
-  type = 'isochrone'
 
-  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
-  result = plpy.execute(mapzen_plan, [username, orgname, type, source, mode, range, options])
+  mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isochrones($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
+  result = plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
   return result
 $$ LANGUAGE plpythonu;

server/extension/test/expected/100_data_observatory_test.out

@@ -185,3 +185,58 @@ SELECT exists(SELECT *
  t
 (1 row)
 
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailablenumerators'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailabledenominators'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailablegeometries'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailabletimespans'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_legacybuildermetadata'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text');
+ exists 
+--------
+ t
+(1 row)
+

server/extension/test/sql/100_data_observatory_test.sql

@@ -115,4 +115,39 @@ SELECT exists(SELECT *
               INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
               WHERE ns.nspname = 'cdb_dataservices_server'
               AND proname = 'obs_getpointsbypointandradius'
-              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, numeric, text, text, text');
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, numeric, text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailablenumerators'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailabledenominators'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailablegeometries'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_getavailabletimespans'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text[], text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_legacybuildermetadata'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text');

server/lib/python/cartodb_services/README.md

@@ -1,4 +1,4 @@
-# CartoDB dataservices API python module
+# CARTO dataservices API python module
 
 This directory contains the python library used by the server side of CARTO LDS (Location Data Services).
 
@@ -29,12 +29,12 @@ NOTE: a system installation is required at present because the library is meant
 
 
 ## Running the unit tests
-Just run `nosetests`
+Just run `nosetests test/`
 ```shell
-$ nosetests
-.................................................
+$ nosetests test/
+......................................................................................................
 ----------------------------------------------------------------------
-Ran 49 tests in 0.131s
+Ran 102 tests in 0.122s
 
 OK
 ```

server/lib/python/cartodb_services/cartodb_services/__init__.py

@@ -0,0 +1,35 @@
+# NOTE: This init function must be called from plpythonu entry points to
+# initialize cartodb_services module properly. E.g:
+#
+# CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(...)
+# RETURNS SETOF cdb_dataservices_server.isoline AS $$
+#
+#     import cartodb_services
+#     cartodb_services.init(plpy, GD)
+#
+#     # rest of the code here
+#     cartodb_services.GD[key] = val
+#     cartodb_services.plpy.execute('SELECT * FROM ...')
+#
+# $$ LANGUAGE plpythonu;
+
+plpy = None
+GD = None
+
+def init(_plpy, _GD):
+    global plpy
+    global GD
+
+    if plpy is None:
+        plpy = _plpy
+
+    if GD is None:
+        GD = _GD
+
+def _reset():
+    # NOTE: just for testing
+    global plpy
+    global GD
+
+    plpy = None
+    GD = None

server/lib/python/cartodb_services/cartodb_services/here/geocoder.py

@@ -5,15 +5,18 @@ import json
 import requests
 
 from exceptions import *
+from cartodb_services.metrics import Traceable
 
 
-class HereMapsGeocoder:
+class HereMapsGeocoder(Traceable):
     'A Here Maps Geocoder wrapper for python'
 
     PRODUCTION_GEOCODE_JSON_URL = 'https://geocoder.api.here.com/6.2/geocode.json'
     STAGING_GEOCODE_JSON_URL = 'https://geocoder.cit.api.here.com/6.2/geocode.json'
     DEFAULT_MAXRESULTS = 1
     DEFAULT_GEN = 9
+    READ_TIMEOUT = 60
+    CONNECT_TIMEOUT = 10
 
     ADDRESS_PARAMS = [
         'city',
@@ -85,7 +88,9 @@ class HereMapsGeocoder:
             'gen': self.gen
         }
         request_params.update(params)
-        response = requests.get(self.host, params=request_params)
+        response = requests.get(self.host, params=request_params,
+                                timeout=(self.CONNECT_TIMEOUT, self.READ_TIMEOUT))
+        self.add_response_data(response, self._logger)
         if response.status_code == requests.codes.ok:
             return json.loads(response.text)
         elif response.status_code == requests.codes.bad_request:

server/lib/python/cartodb_services/cartodb_services/here/routing.py

@@ -2,14 +2,17 @@ import requests
 import json
 
 from exceptions import WrongParams
+from cartodb_services.metrics import Traceable
 
 
-class HereMapsRoutingIsoline:
+class HereMapsRoutingIsoline(Traceable):
     'A Here Maps Routing wrapper for python'
 
     PRODUCTION_ROUTING_BASE_URL = 'https://isoline.route.api.here.com'
     STAGING_ROUTING_BASE_URL = 'https://isoline.route.cit.api.here.com'
     ISOLINE_PATH = '/routing/7.2/calculateisoline.json'
+    READ_TIMEOUT = 60
+    CONNECT_TIMEOUT = 10
 
     ACCEPTED_MODES = {
         "walk": "pedestrian",
@@ -50,7 +53,9 @@ class HereMapsRoutingIsoline:
                                                          data_range,
                                                          range_type,
                                                          parsed_options)
-        response = requests.get(self._url, params=request_params)
+        response = requests.get(self._url, params=request_params,
+                                timeout=(self.CONNECT_TIMEOUT, self.READ_TIMEOUT))
+        self.add_response_data(response, self._logger)
         if response.status_code == requests.codes.ok:
             return self.__parse_isolines_response(response.text)
         elif response.status_code == requests.codes.bad_request:

server/lib/python/cartodb_services/cartodb_services/mapzen/__init__.py

@@ -2,3 +2,4 @@ from routing import MapzenRouting, MapzenRoutingResponse
 from isolines import MapzenIsolines
 from geocoder import MapzenGeocoder
 from matrix_client import MatrixClient
+from isochrones import MapzenIsochrones

server/lib/python/cartodb_services/cartodb_services/mapzen/exceptions.py

@@ -19,3 +19,15 @@ class MalformedResult(Exception):
 class TimeoutException(Exception):
     def __str__(self):
             return repr('Timeout requesting to mapzen server')
+
+
+class ServiceException(Exception):
+    def __init__(self, message, response):
+        self.message = message
+        self.response = response
+
+    def response(self):
+        return self.response
+
+    def __str__(self):
+        return self.message

server/lib/python/cartodb_services/cartodb_services/mapzen/geocoder.py

@@ -2,28 +2,34 @@ import requests
 import json
 import re
 
-from exceptions import WrongParams, MalformedResult
+from exceptions import WrongParams, MalformedResult, ServiceException
 from qps import qps_retry
 from cartodb_services.tools import Coordinate, PolyLine
+from cartodb_services.metrics import Traceable
 
 
-class MapzenGeocoder:
+class MapzenGeocoder(Traceable):
     'A Mapzen Geocoder wrapper for python'
 
     BASE_URL = 'https://search.mapzen.com/v1/search'
+    READ_TIMEOUT = 60
+    CONNECT_TIMEOUT = 10
 
     def __init__(self, app_key, logger, base_url=BASE_URL):
         self._app_key = app_key
         self._url = base_url
         self._logger = logger
 
-    @qps_retry
-    def geocode(self, searchtext, city=None, state_province=None, country=None, search_type=None):
+    @qps_retry(qps=20)
+    def geocode(self, searchtext, city=None, state_province=None,
+                country=None, search_type=None):
         request_params = self._build_requests_parameters(searchtext, city,
                                                          state_province,
                                                          country, search_type)
         try:
-            response = requests.get(self._url, params=request_params)
+            response = requests.get(self._url, params=request_params,
+                                    timeout=(self.CONNECT_TIMEOUT, self.READ_TIMEOUT))
+            self.add_response_data(response, self._logger)
             if response.status_code == requests.codes.ok:
                 return self.__parse_response(response.text)
             elif response.status_code == requests.codes.bad_request:
@@ -31,21 +37,27 @@ class MapzenGeocoder:
             else:
                 self._logger.error('Error trying to geocode using mapzen',
                                 data={"response_status": response.status_code,
-                                        "response_reason": response.reason,
-                                        "response_content": response.text,
-                                        "reponse_url": response.url,
-                                        "response_headers": response.headers,
-                                        "searchtext": searchtext,
-                                        "city": city, "country": country,
-                                        "state_province": state_province })
-                raise Exception('Error trying to geocode {0} using mapzen'.format(searchtext))
+                                      "response_reason": response.reason,
+                                      "response_content": response.text,
+                                      "reponse_url": response.url,
+                                      "response_headers": response.headers,
+                                      "searchtext": searchtext,
+                                      "city": city, "country": country,
+                                      "state_province": state_province})
+                raise ServiceException('Error trying to geocode {0} using mapzen'.format(searchtext),
+                                       response)
+        except requests.Timeout as te:
+            # In case of timeout we want to stop the job because the server
+            # could be down
+            self._logger.error('Timeout connecting to Mapzen geocoding server', te)
+            raise ServiceException('Error trying to geocode {0} using mapzen'.format(searchtext),
+                                    None)
         except requests.ConnectionError as e:
             # Don't raise the exception to continue with the geocoding job
             self._logger.error('Error connecting to Mapzen geocoding server',
                                exception=e)
             return []
 
-
     def _build_requests_parameters(self, searchtext, city=None,
                                    state_province=None, country=None,
                                    search_type=None):

server/lib/python/cartodb_services/cartodb_services/mapzen/isochrones.py

@@ -0,0 +1,130 @@
+import requests
+import json
+import re
+
+from exceptions import WrongParams, MalformedResult, ServiceException
+from qps import qps_retry
+
+
+class MapzenIsochrones:
+    'A Mapzen Isochrones wrapper for python'
+
+    BASE_URL = 'https://matrix.mapzen.com/isochrone'
+    READ_TIMEOUT = 60
+    CONNECT_TIMEOUT = 10
+
+    ACCEPTED_MODES = {
+        "walk": "pedestrian",
+        "car": "auto"
+    }
+
+    def __init__(self, app_key, logger, base_url=BASE_URL):
+        self._app_key = app_key
+        self._url = base_url
+        self._logger = logger
+
+    @qps_retry
+    def isochrone(self, locations, costing, ranges):
+        request_params = self._parse_request_params(locations, costing,
+                                                    ranges)
+        try:
+            response = requests.get(self._url, params=request_params,
+                                    timeout=(self.CONNECT_TIMEOUT,
+                                            self.READ_TIMEOUT))
+
+            if response.status_code is requests.codes.ok:
+                return self._parse_response(response)
+            elif response.status_code == requests.codes.bad_request:
+                return []
+            else:
+                self._logger.error('Error trying to get isochrones from mapzen',
+                                data={"response_status": response.status_code,
+                                      "response_reason": response.reason,
+                                      "response_content": response.text,
+                                      "reponse_url": response.url,
+                                      "response_headers": response.headers,
+                                      "locations": locations,
+                                      "costing": costing})
+                raise ServiceException('Error trying to get isochrones from mapzen',
+                                       response)
+        except requests.Timeout as te:
+            # In case of timeout we want to stop the job because the server
+            # could be down
+            self._logger.error('Timeout connecting to Mapzen isochrones server', exception=te)
+            raise ServiceException('Error trying to calculate isochrones using mapzen',
+                                    None)
+        except requests.ConnectionError as e:
+            # Don't raise the exception to continue with the geocoding job
+            self._logger.error('Error connecting to Mapzen isochrones server',
+                               exception=e)
+            return []
+
+    def _parse_request_params(self, locations, costing, ranges):
+        if costing in self.ACCEPTED_MODES:
+            mode_source = self.ACCEPTED_MODES[costing]
+        else:
+            raise WrongParams("{0} is not an accepted mode".format(costing))
+
+        contours = []
+        for r in ranges:
+            # range is in seconds but mapzen uses minutes
+            range_minutes = r / 60
+            contours.append({"time": range_minutes, "color": 'tbd'})
+        request_params = {
+            'json': json.dumps({'locations': [locations],
+                                'costing': mode_source,
+                                'contours': contours,
+                                'generalize': 50,
+                                'denoise': .3}),
+            'api_key': self._app_key
+        }
+
+        return request_params
+
+    def _parse_response(self, response):
+        try:
+            json_response = response.json()
+            isochrones = []
+            for feature in json_response['features']:
+                # Coordinates could have more than one isochrone. For the
+                # moment we're getting the first polygon only
+                coordinates = feature['geometry']['coordinates']
+                duration = feature['properties']['contour']
+                mapzen_response = MapzenIsochronesResponse(coordinates,
+                                                           duration)
+                isochrones.append(mapzen_response)
+            return isochrones
+        except IndexError:
+            return []
+        except KeyError:
+            self._logger.error('Non existing key for mapzen isochrones response',
+                               data={"response_status": response.status_code,
+                                     "response_reason": response.reason,
+                                     "response_content": response.text,
+                                     "reponse_url": response.url,
+                                     "response_headers": response.headers})
+            raise MalformedResult()
+        except ValueError:
+            # JSON decode error
+            self._logger.error('JSON decode error for Mapzen isochrones',
+                               data={"response_status": response.status_code,
+                                     "response_reason": response.reason,
+                                     "response_content": response.text,
+                                     "reponse_url": response.url,
+                                     "response_headers": response.headers})
+            return []
+
+
+class MapzenIsochronesResponse:
+
+    def __init__(self, coordinates, duration):
+        self._coordinates = coordinates
+        self._duration = duration
+
+    @property
+    def coordinates(self):
+        return self._coordinates
+
+    @property
+    def duration(self):
+        return self._duration

server/lib/python/cartodb_services/cartodb_services/mapzen/isolines.py

@@ -86,7 +86,7 @@ class MapzenIsolines:
     def calculate_isoline(self, origin, costing_model, isorange, upper_rmax, cost_variable, unit_factor=1.0):
 
         # NOTE: not for production
-        self._logger.debug('Calculate isoline', data={"origin": origin, "costing_model": costing_model, "isorange": isorange})
+        # self._logger.debug('Calculate isoline', data={"origin": origin, "costing_model": costing_model, "isorange": isorange})
 
         # Formally, a solution is an array of {angle, radius, lat, lon, cost} with cardinality NUMBER_OF_ANGLES
         # we're looking for a solution in which abs(cost - isorange) / isorange <= TOLERANCE
@@ -105,14 +105,16 @@ class MapzenIsolines:
 
             response = self._matrix_client.one_to_many([origin] + location_estimates,  costing_model)
             costs = [None] * self.NUMBER_OF_ANGLES
+            if not response:
+                # In case the matrix client doesn't return any data
+                break
+
             for idx, c in enumerate(response['one_to_many'][0][1:]):
                 if c[cost_variable]:
                     costs[idx] = c[cost_variable]*unit_factor
                 else:
                     costs[idx] = isorange
 
-            # self._logger.debug('i = %d, costs = %s' % (i, costs))
-
             errors = [(cost - isorange) / float(isorange) for cost in costs]
             max_abs_error = max([abs(e) for e in errors])
             if max_abs_error <= self.TOLERANCE:

server/lib/python/cartodb_services/cartodb_services/mapzen/matrix_client.py

@@ -1,9 +1,11 @@
 import requests
 import json
 from qps import qps_retry
+from exceptions import ServiceException
+from cartodb_services.metrics import Traceable
 
 
-class MatrixClient:
+class MatrixClient(Traceable):
 
     """
     A minimal client for Mapzen Time-Distance Matrix Service
@@ -18,6 +20,8 @@ class MatrixClient:
     """
 
     ONE_TO_MANY_URL = 'https://matrix.mapzen.com/one_to_many'
+    READ_TIMEOUT = 60
+    CONNECT_TIMEOUT = 10
 
     def __init__(self, matrix_key, logger):
         self._matrix_key = matrix_key
@@ -40,9 +44,11 @@ class MatrixClient:
             'costing': costing,
             'api_key': self._matrix_key
         }
-        response = requests.get(self.ONE_TO_MANY_URL, params=request_params)
+        response = requests.get(self.ONE_TO_MANY_URL, params=request_params,
+                                timeout=(self.CONNECT_TIMEOUT, self.READ_TIMEOUT))
+        self.add_response_data(response, self._logger)
 
-        if not requests.codes.ok:
+        if response.status_code != requests.codes.ok:
             self._logger.error('Error trying to get matrix distance from mapzen',
                                data={"response_status": response.status_code,
                                      "response_reason": response.reason,
@@ -51,6 +57,22 @@ class MatrixClient:
                                      "response_headers": response.headers,
                                      "locations": locations,
                                      "costing": costing})
-            raise Exception('Error trying to get matrix distance from mapzen')
+            # In case 4xx error we return empty because the error comes from
+            # the provided info by the user and we don't want to top the
+            # isolines generation
+            if response.status_code == requests.codes.bad_request:
+                return {}
+            elif response.status_code == 504:
+                # Due to some unsolved problems in the Mapzen Matrix API we're
+                # getting randomly 504, probably timeouts. To avoid raise an
+                # exception in all the jobs, for now we're going to return
+                # empty in that case
+                return {}
+            else:
+                raise ServiceException("Error trying to get matrix distance from mapzen", response)
 
-        return response.json()
+        # response could return with empty json
+        try:
+            return response.json()
+        except:
+            return {}

server/lib/python/cartodb_services/cartodb_services/mapzen/qps.py

@@ -4,18 +4,38 @@ from datetime import datetime
 from exceptions import TimeoutException
 
 DEFAULT_RETRY_TIMEOUT = 60
+DEFAULT_QUERIES_PER_SECOND = 10
 
-
-def qps_retry(f):
-    def wrapped_f(*args, **kw):
-        return QPSService().call(f, *args, **kw)
-    return wrapped_f
+def qps_retry(original_function=None,**options):
+    """ Query Per Second retry decorator
+        The intention of this decorator is to retry requests against third
+        party services that has QPS restriction.
+        Parameters:
+            - timeout: Maximum number of seconds to retry
+            - qps: Allowed queries per second. This parameter is used to
+                   calculate the next time to retry the request
+    """
+    if original_function is not None:
+        def wrapped_function(*args, **kwargs):
+            if 'timeout' in options:
+                timeout = options['timeout']
+            else:
+                timeout = DEFAULT_RETRY_TIMEOUT
+            if 'qps' in options:
+                qps = options['qps']
+            else:
+                qps = DEFAULT_QUERIES_PER_SECOND
+            return QPSService(retry_timeout=timeout, queries_per_second=qps).call(original_function, *args, **kwargs)
+        return wrapped_function
+    else:
+        def partial_wrapper(func):
+            return qps_retry(func, **options)
+        return partial_wrapper
 
 
 class QPSService:
 
-    def __init__(self, queries_per_second=10,
-                 retry_timeout=DEFAULT_RETRY_TIMEOUT):
+    def __init__(self, queries_per_second, retry_timeout):
         self._queries_per_second = queries_per_second
         self._retry_timeout = retry_timeout
 
@@ -27,7 +47,7 @@ class QPSService:
                 return fn(*args, **kwargs)
             except Exception as e:
                 response = getattr(e, 'response', None)
-                if response and (response.status_code == 429):
+                if response is not None and (response.status_code == 429):
                     self.retry(start_time, attempt_number)
                 else:
                     raise e
@@ -35,7 +55,7 @@ class QPSService:
 
     def retry(self, first_request_time, retry_count):
         elapsed = datetime.now() - first_request_time
-        if elapsed.seconds > self._retry_timeout:
+        if elapsed.microseconds  > (self._retry_timeout * 1000.0):
             raise TimeoutException()
 
         # inverse qps * (1.5 ^ i) is an increased sleep time of 1.5x per

server/lib/python/cartodb_services/cartodb_services/mapzen/routing.py

@@ -2,15 +2,18 @@ import requests
 import json
 import re
 
-from exceptions import WrongParams, MalformedResult
+from exceptions import WrongParams, MalformedResult, ServiceException
 from qps import qps_retry
 from cartodb_services.tools import Coordinate, PolyLine
+from cartodb_services.metrics import MetricsDataGatherer, Traceable
 
 
-class MapzenRouting:
+class MapzenRouting(Traceable):
     'A Mapzen Routing wrapper for python'
 
     PRODUCTION_ROUTING_BASE_URL = 'https://valhalla.mapzen.com/route'
+    READ_TIMEOUT = 60
+    CONNECT_TIMEOUT = 10
 
     ACCEPTED_MODES = {
         "walk": "pedestrian",
@@ -43,7 +46,9 @@ class MapzenRouting:
                                                            mode_param,
                                                            units)
         request_params = self.__parse_request_parameters(json_request_params)
-        response = requests.get(self._url, params=request_params)
+        response = requests.get(self._url, params=request_params,
+                                timeout=(self.CONNECT_TIMEOUT, self.READ_TIMEOUT))
+        self.add_response_data(response, self._logger)
         if response.status_code == requests.codes.ok:
             return self.__parse_routing_response(response.text)
         elif response.status_code == requests.codes.bad_request:
@@ -57,7 +62,7 @@ class MapzenRouting:
                                      "response_headers": response.headers,
                                      "waypoints": waypoints, "mode": mode,
                                      "options": options})
-            raise Exception('Error trying to calculate route using Mapzen')
+            raise ServiceException('Error trying to calculate route using Mapzen', response)
 
     def __parse_options(self, options):
         return dict(option.split('=') for option in options)

server/lib/python/cartodb_services/cartodb_services/mapzen/types.py

@@ -19,6 +19,24 @@ def polyline_to_linestring(polyline):
 
     return geometry
 
+
+def coordinates_to_polygon(coordinates):
+    """Convert a Mapzen coordinates to a PostGIS polygon"""
+    result_coordinates = []
+    for coordinate in coordinates:
+        result_coordinates.append("%s %s" % (coordinate[0], coordinate[1]))
+    wkt_coordinates = ','.join(result_coordinates)
+
+    try:
+        sql = "SELECT ST_MakePolygon(ST_GeomFromText('LINESTRING({0})', 4326)) as geom".format(wkt_coordinates)
+        geometry = plpy.execute(sql, 1)[0]['geom']
+    except BaseException as e:
+        plpy.warning("Can't generate POLYGON from coordinates: {0}".format(e))
+        geometry = None
+
+    return geometry
+
+
 def country_to_iso3(country):
     """ Convert country to its iso3 code """
     try:

server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py

@@ -1,3 +1,4 @@
 from config import GeocoderConfig, IsolinesRoutingConfig, InternalGeocoderConfig, RoutingConfig, ConfigException, ObservatorySnapshotConfig, ObservatoryConfig
 from quota import QuotaService
 from user import UserMetricsService
+from log import metrics, MetricsDataGatherer, Traceable

server/lib/python/cartodb_services/cartodb_services/metrics/config.py

@@ -15,6 +15,7 @@ class ServiceConfig(object):
         self._username = username
         self._orgname = orgname
         self._db_config = ServicesDBConfig(db_conn, username, orgname)
+        self._metrics_log_path = self.__get_metrics_log_path()
         self._environment = self._db_config._server_environment
         if redis_connection:
             self._redis_config = ServicesRedisConfig(redis_connection).build(
@@ -38,9 +39,20 @@ class ServiceConfig(object):
     def environment(self):
         return self._environment
 
+    @property
+    def metrics_log_path(self):
+        return self._metrics_log_path
+
+    def __get_metrics_log_path(self):
+        if self.METRICS_LOG_KEY:
+            return self._db_config.logger_config.get(self.METRICS_LOG_KEY, None)
+        else:
+            return None
 
 class DataObservatoryConfig(ServiceConfig):
 
+    METRICS_LOG_KEY = 'do_log_path'
+
     def __init__(self, redis_connection, db_conn, username, orgname=None):
         super(DataObservatoryConfig, self).__init__(redis_connection, db_conn,
                                             username, orgname)
@@ -61,6 +73,10 @@ class DataObservatoryConfig(ServiceConfig):
     def connection_str(self):
         return self._connection_str
 
+    @property
+    def provider(self):
+        return 'data observatory'
+
 
 class ObservatorySnapshotConfig(DataObservatoryConfig):
 
@@ -78,7 +94,7 @@ class ObservatorySnapshotConfig(DataObservatoryConfig):
             self._soft_limit = False
         self._monthly_quota = 0
         if self.QUOTA_KEY in self._redis_config:
-            self._monthly_quota = float(self._redis_config[self.QUOTA_KEY])
+            self._monthly_quota = int(self._redis_config[self.QUOTA_KEY])
         self._connection_str = self._db_config.data_observatory_connection_str
 
     @property
@@ -102,7 +118,7 @@ class ObservatoryConfig(DataObservatoryConfig):
             self._soft_limit = False
         self._monthly_quota = 0
         if self.QUOTA_KEY in self._redis_config:
-            self._monthly_quota = float(self._redis_config[self.QUOTA_KEY])
+            self._monthly_quota = int(self._redis_config[self.QUOTA_KEY])
         self._connection_str = self._db_config.data_observatory_connection_str
 
     @property
@@ -116,6 +132,9 @@ class RoutingConfig(ServiceConfig):
     ROUTING_PROVIDER_KEY = 'routing_provider'
     MAPZEN_PROVIDER = 'mapzen'
     DEFAULT_PROVIDER = 'mapzen'
+    QUOTA_KEY = 'mapzen_routing_quota'
+    SOFT_LIMIT_KEY = 'soft_mapzen_routing_limit'
+    METRICS_LOG_KEY = 'routing_log_path'
 
     def __init__(self, redis_connection, db_conn, username, orgname=None):
         super(RoutingConfig, self).__init__(redis_connection, db_conn,
@@ -124,7 +143,8 @@ class RoutingConfig(ServiceConfig):
         if not self._routing_provider:
             self._routing_provider = self.DEFAULT_PROVIDER
         self._mapzen_api_key = self._db_config.mapzen_routing_api_key
-        self._monthly_quota = self._db_config.mapzen_routing_monthly_quota
+        self._set_monthly_quota()
+        self._set_soft_limit()
         self._period_end_date = date_parse(self._redis_config[self.PERIOD_END_DATE])
 
     @property
@@ -132,6 +152,10 @@ class RoutingConfig(ServiceConfig):
         if self._routing_provider == self.MAPZEN_PROVIDER:
             return 'routing_mapzen'
 
+    @property
+    def provider(self):
+        return self._routing_provider
+
     @property
     def mapzen_api_key(self):
         return self._mapzen_api_key
@@ -144,6 +168,26 @@ class RoutingConfig(ServiceConfig):
     def period_end_date(self):
         return self._period_end_date
 
+    @property
+    def soft_limit(self):
+        return self._soft_limit
+
+    def _set_monthly_quota(self):
+        self._monthly_quota = self._get_effective_monthly_quota()
+
+    def _get_effective_monthly_quota(self):
+        quota_from_redis = self._redis_config.get(self.QUOTA_KEY)
+        if quota_from_redis and quota_from_redis <> '':
+            return int(quota_from_redis)
+        else:
+            return self._db_config.mapzen_routing_monthly_quota
+
+    def _set_soft_limit(self):
+        if self.SOFT_LIMIT_KEY in self._redis_config and self._redis_config[self.SOFT_LIMIT_KEY].lower() == 'true':
+            self._soft_limit = True
+        else:
+            self._soft_limit = False
+
 
 class IsolinesRoutingConfig(ServiceConfig):
 
@@ -159,6 +203,7 @@ class IsolinesRoutingConfig(ServiceConfig):
     MAPZEN_PROVIDER = 'mapzen'
     HEREMAPS_PROVIDER = 'heremaps'
     DEFAULT_PROVIDER = 'heremaps'
+    METRICS_LOG_KEY = 'isolines_log_path'
 
     def __init__(self, redis_connection, db_conn, username, orgname=None):
         super(IsolinesRoutingConfig, self).__init__(redis_connection, db_conn,
@@ -173,7 +218,7 @@ class IsolinesRoutingConfig(ServiceConfig):
         self._geocoder_provider = filtered_config[self.GEOCODER_PROVIDER_KEY].lower()
         self._period_end_date = date_parse(filtered_config[self.PERIOD_END_DATE])
         if self._isolines_provider == self.HEREMAPS_PROVIDER:
-            self._isolines_quota = float(filtered_config[self.QUOTA_KEY])
+            self._isolines_quota = int(filtered_config[self.QUOTA_KEY])
             self._heremaps_app_id = db_config.heremaps_isolines_app_id
             self._heremaps_app_code = db_config.heremaps_isolines_app_code
             if filtered_config[self.SOFT_LIMIT_KEY].lower() == 'true':
@@ -235,11 +280,12 @@ class IsolinesRoutingConfig(ServiceConfig):
 
 class InternalGeocoderConfig(ServiceConfig):
 
+    METRICS_LOG_KEY = 'geocoder_log_path'
+
     def __init__(self, redis_connection, db_conn, username, orgname=None):
         # For now, internal geocoder doesn't use the redis config
         super(InternalGeocoderConfig, self).__init__(None, db_conn,
                                                      username, orgname)
-        self._log_path = self._db_config.geocoder_log_path
 
     @property
     def service_type(self):
@@ -258,8 +304,8 @@ class InternalGeocoderConfig(ServiceConfig):
         return None
 
     @property
-    def log_path(self):
-        return self._log_path
+    def provider(self):
+        return 'internal'
 
 
 class GeocoderConfig(ServiceConfig):
@@ -285,6 +331,7 @@ class GeocoderConfig(ServiceConfig):
     ORGNAME_KEY = 'orgname'
     PERIOD_END_DATE = 'period_end_date'
     DEFAULT_PROVIDER = 'mapzen'
+    METRICS_LOG_KEY = 'geocoder_log_path'
 
     def __init__(self, redis_connection, db_conn, username, orgname=None, forced_provider=None):
         super(GeocoderConfig, self).__init__(redis_connection, db_conn,
@@ -314,9 +361,8 @@ class GeocoderConfig(ServiceConfig):
             self._geocoder_provider = filtered_config[self.GEOCODER_PROVIDER].lower()
         else:
             self._geocoder_provider = self.DEFAULT_PROVIDER
-        self._geocoding_quota = float(filtered_config[self.QUOTA_KEY])
+        self._geocoding_quota = int(filtered_config[self.QUOTA_KEY])
         self._period_end_date = date_parse(filtered_config[self.PERIOD_END_DATE])
-        self._log_path = db_config.geocoder_log_path
         if filtered_config[self.SOFT_LIMIT_KEY].lower() == 'true':
             self._soft_geocoding_limit = True
         else:
@@ -399,8 +445,8 @@ class GeocoderConfig(ServiceConfig):
         return self._cost_per_hit
 
     @property
-    def log_path(self):
-        return self._log_path
+    def provider(self):
+        return self._geocoder_provider
 
 
 class ServicesDBConfig:
@@ -415,7 +461,6 @@ class ServicesDBConfig:
         self._get_server_config()
         self._get_here_config()
         self._get_mapzen_config()
-        self._get_logger_config()
         self._get_data_observatory_config()
 
     def _get_server_config(self):
@@ -468,13 +513,6 @@ class ServicesDBConfig:
             else:
                 self._data_observatory_connection_str = do_conf['connection']['production']
 
-    def _get_logger_config(self):
-        logger_conf_json = self._get_conf('logger_conf')
-        if not logger_conf_json:
-            raise ConfigException('Logger configuration missing')
-        else:
-            logger_conf = json.loads(logger_conf_json)
-            self._geocoder_log_path = logger_conf['geocoder_log_path']
 
     def _get_conf(self, key):
         try:
@@ -482,7 +520,7 @@ class ServicesDBConfig:
             conf = self._db_conn.execute(sql, 1)
             return conf[0]['conf']
         except Exception as e:
-            raise ConfigException("Malformed config for {0}: {1}".format(key, e))
+            raise ConfigException("Error trying to get config for {0}: {1}".format(key, e))
 
     @property
     def server_environment(self):
@@ -532,14 +570,18 @@ class ServicesDBConfig:
     def mapzen_geocoder_monthly_quota(self):
         return self._mapzen_geocoder_quota
 
-    @property
-    def geocoder_log_path(self):
-        return self._geocoder_log_path
-
     @property
     def data_observatory_connection_str(self):
         return self._data_observatory_connection_str
 
+    @property
+    def logger_config(self):
+        logger_conf_json = self._get_conf('logger_conf')
+        if not logger_conf_json:
+            raise ConfigException('Logger configuration missing')
+        else:
+            return json.loads(logger_conf_json)
+
 
 class ServicesRedisConfig:
 
@@ -547,6 +589,7 @@ class ServicesRedisConfig:
     GOOGLE_GEOCODER_CLIENT_ID = 'google_maps_client_id'
     QUOTA_KEY = 'geocoding_quota'
     ISOLINES_QUOTA_KEY = 'here_isolines_quota'
+    ROUTING_QUOTA_KEY = 'mapzen_routing_quota'
     OBS_SNAPSHOT_QUOTA_KEY = 'obs_snapshot_quota'
     OBS_GENERAL_QUOTA_KEY = 'obs_general_quota'
     PERIOD_END_DATE = 'period_end_date'
@@ -585,8 +628,12 @@ class ServicesRedisConfig:
         if not org_config:
             raise ConfigException("""There is no organization config available. Please check your configuration.'""")
         else:
-            user_config[self.QUOTA_KEY] = org_config[self.QUOTA_KEY]
-            user_config[self.ISOLINES_QUOTA_KEY] = org_config[self.ISOLINES_QUOTA_KEY]
+            if self.QUOTA_KEY in org_config:
+                user_config[self.QUOTA_KEY] = org_config[self.QUOTA_KEY]
+            if self.ISOLINES_QUOTA_KEY in org_config:
+                user_config[self.ISOLINES_QUOTA_KEY] = org_config[self.ISOLINES_QUOTA_KEY]
+            if self.ROUTING_QUOTA_KEY in org_config:
+                user_config[self.ROUTING_QUOTA_KEY] = org_config[self.ROUTING_QUOTA_KEY]
             if self.OBS_SNAPSHOT_QUOTA_KEY in org_config:
                 user_config[self.OBS_SNAPSHOT_QUOTA_KEY] = org_config[self.OBS_SNAPSHOT_QUOTA_KEY]
             if self.OBS_GENERAL_QUOTA_KEY in org_config:

server/lib/python/cartodb_services/cartodb_services/metrics/log.py

@@ -1,15 +1,130 @@
-from datetime import datetime
 import abc
 import json
 import re
+import time
+import uuid
+import plpy
+from datetime import datetime
+from contextlib import contextmanager
+from urlparse import urlparse
 
 
-class MetricsLoggerFactory:
+@contextmanager
+def metrics(function, service_config, logger=None):
+    try:
+        start_time = time.time()
+        yield
+    finally:
+        end_time = time.time()
+        MetricsDataGatherer.add('uuid', str(uuid.uuid1()))
+        MetricsDataGatherer.add('function_name', function)
+        MetricsDataGatherer.add('function_execution_time', (end_time - start_time))
+        metrics_logger = MetricsServiceLoggerFactory.build(service_config,
+                                                           logger)
+        if metrics_logger:
+            data = MetricsDataGatherer.get()
+            metrics_logger.log(data)
+        MetricsDataGatherer.clean()
+
+
+class Traceable:
+    """
+    Module to add metrics traceability, for example to get response object
+    in order to add to the metrics dump
+    """
+    def add_response_data(self, response, logger=None):
+        try:
+            response_data = {}
+            response_data['type'] = "request"
+            response_data['date'] = datetime.now().isoformat()
+            response_data['elapsed_time'] = response.elapsed.total_seconds()
+            response_data['code'] = response.status_code
+            response_data['message'] = response.reason
+            response_data['url'] = self._parse_response_url(response.url)
+            stored_data = MetricsDataGatherer.get_element('response')
+            if stored_data:
+                stored_data.append(response_data)
+            else:
+                MetricsDataGatherer.add('response', [response_data])
+        except BaseException as e:
+            # We don't want to stop the job for some error processing response
+            if logger:
+                logger.error("Error trying to process response metricd data",
+                             exception=e)
+
+    def _parse_response_url(self, url):
+        u = urlparse(url)
+        return "{0}://{1}{2}".format(u.scheme, u.netloc, u.path)
+
+
+class MetricsDataGatherer:
+    """
+    Metrics gatherer used as a singleton. The intend is to use it as a global
+    storage for the metrics along the function request.
+    """
+
+    class __MetricsDataGatherer:
+        def __init__(self):
+            self.data = {}
+
+        def add(self, key, value):
+            self.data[key] = value
+
+        def get(self):
+            return self.data
+
+        def get_element(self, key):
+            return self.data.get(key, None)
+
+        def clean(self):
+            self.data = {}
+
+
+    #  We use pgbouncer so we need to have multiples instances per request id
+    __instance = {}
 
     @classmethod
-    def build(self, service_config):
-        if re.match('geocoder_*', service_config.service_type):
-            return MetricsGeocoderLogger(service_config)
+    def add(self, key, value):
+        MetricsDataGatherer.instance().add(key, value)
+
+    @classmethod
+    def get(self):
+        return MetricsDataGatherer.instance().get()
+
+    @classmethod
+    def get_element(self, key):
+        return MetricsDataGatherer.instance().get_element(key)
+
+    @classmethod
+    def clean(self):
+        MetricsDataGatherer.instance().clean()
+
+    @classmethod
+    def instance(self):
+        txid = MetricsDataGatherer._get_txid()
+        if txid not in MetricsDataGatherer.__instance:
+            MetricsDataGatherer.__instance[txid] = MetricsDataGatherer.__MetricsDataGatherer()
+
+        return MetricsDataGatherer.__instance[txid]
+
+    @classmethod
+    def _get_txid(self):
+        result = plpy.execute('select txid_current() as txid')
+        return result[0]['txid']
+
+
+class MetricsServiceLoggerFactory:
+
+    @classmethod
+    def build(self, service_config, logger=None):
+        if re.search('^geocoder_*', service_config.service_type):
+            return MetricsGeocoderLogger(service_config, logger)
+        elif re.search('^routing_*', service_config.service_type):
+            return MetricsGenericLogger(service_config, logger)
+        elif re.search('_isolines$', service_config.service_type):
+            return MetricsIsolinesLogger(service_config, logger)
+        elif re.search('^obs_*', service_config.service_type):
+            return MetricsGenericLogger(service_config, logger)
         else:
             return None
 
@@ -17,58 +132,119 @@ class MetricsLoggerFactory:
 class MetricsLogger(object):
     __metaclass__ = abc.ABCMeta
 
-    def __init__(self, file_path):
-        self._file_path = file_path
+    def __init__(self, service_config, logger):
+        self._service_config = service_config
+        self._logger = logger
 
     def dump_to_file(self, data):
-        with open(self._file_path, 'a') as logfile:
-            json.dump(data, logfile)
-            logfile.write('\n')
+        try:
+            log_path = self.service_config.metrics_log_path
+            response_data = data.pop('response', [])
+            uuid = data.get('uuid')
+            if log_path:
+                with open(log_path, 'a') as logfile:
+                    self._dump_response_to_file(uuid, response_data, logfile)
+                    json.dump(data, logfile)
+                    logfile.write('\n')
+        except BaseException as e:
+            self._logger("Error dumping metrics to file {0}".format(log_path),
+                         exception=e)
+
+    def collect_data(self, data):
+        return {
+            "uuid": data.get('uuid', uuid.uuid1()),
+            "type": 'function',
+            "function_name": data.get('function_name', None),
+            "function_execution_time": data.get('function_execution_time',
+                                                None),
+            "service": self._service_config.service_type,
+            "processable_rows": 1,
+            "success": data.get('success', False),
+            "successful_rows": data.get('successful_rows', 0),
+            "failed_rows": data.get('failed_rows', 0),
+            "empty_rows": data.get('empty_rows', 0),
+            "created_at": datetime.now().isoformat(),
+            "provider": self._service_config.provider,
+            "username": self._service_config.username,
+            "organization": self._service_config.organization,
+            "response": data.get('response', [])
+        }
+
+    def _dump_response_to_file(self, uuid, response_data, log_file):
+        for r in response_data:
+            r['uuid'] = uuid
+            json.dump(r, log_file)
+            log_file.write('\n')
+
+    @property
+    def service_config(self):
+        return self._service_config
 
     @abc.abstractproperty
-    def log(self, **data):
+    def log(self, data):
         raise NotImplementedError('log method must be defined')
 
 
 class MetricsGeocoderLogger(MetricsLogger):
 
-    def __init__(self, service_config):
-        super(MetricsGeocoderLogger, self).__init__(service_config.log_path)
-        self._service_config = service_config
+    def __init__(self, service_config, logger):
+        super(MetricsGeocoderLogger, self).__init__(service_config, logger)
 
-    def log(self, **data):
-        dump_data = self._dump_data(**data)
+    def log(self, data):
+        dump_data = self.collect_data(data)
         self.dump_to_file(dump_data)
 
-    def _dump_data(self, **data):
-        if data['success']:
-            cost = self._service_config.cost_per_hit
-            failed_rows = 0
-            successful_rows = 1
+    def collect_data(self, data):
+        dump_data = super(MetricsGeocoderLogger, self).collect_data(data)
+        if data.get('success', False):
+            cost = self.service_config.cost_per_hit
         else:
             cost = 0
-            failed_rows = 1
-            successful_rows = 0
 
-        if self._service_config.is_high_resolution:
+        if self.service_config.is_high_resolution:
             kind = 'high-resolution'
         else:
             kind = 'internal'
 
-        return {
+        dump_data.update({
             "batched": False,
             "cache_hits": 0,  # Always 0 because no cache involved
             # https://github.com/CartoDB/cartodb/blob/master/app/models/geocoding.rb#L208-L211
             "cost": cost,
-            "created_at": datetime.now().isoformat(),
-            "failed_rows": failed_rows,
-            "geocoder_type": self._service_config.service_type,
+            "geocoder_type": self.service_config.service_type,
             "kind": kind,
-            "processable_rows": 1,
-            "processed_rows": successful_rows,
-            "real_rows": successful_rows,
-            "success": data['success'],
-            "successful_rows": successful_rows,
-            "username": self._service_config.username,
-            "organization": self._service_config.organization
-        }
+            "processed_rows": data.get('successful_rows', 0),
+            "real_rows": data.get('successful_rows', 0),
+        })
+
+        return dump_data
+
+
+class MetricsGenericLogger(MetricsLogger):
+
+    def __init__(self, service_config, logger):
+        super(MetricsGenericLogger, self).__init__(service_config, logger)
+
+    def log(self, data):
+        dump_data = self.collect_data(data)
+        self.dump_to_file(dump_data)
+
+    def collect_data(self, data):
+        return super(MetricsGenericLogger, self).collect_data(data)
+
+
+class MetricsIsolinesLogger(MetricsLogger):
+
+    def __init__(self, service_config, logger):
+        super(MetricsIsolinesLogger, self).__init__(service_config, logger)
+
+    def log(self, data):
+        dump_data = self.collect_data(data)
+        self.dump_to_file(dump_data)
+
+    def collect_data(self, data):
+        dump_data = super(MetricsIsolinesLogger, self).collect_data(data)
+        dump_data.update({
+            "isolines_generated": data.get('isolines_generated', 0)
+        })
+        return dump_data

server/lib/python/cartodb_services/cartodb_services/metrics/quota.py

@@ -1,5 +1,5 @@
 from user import UserMetricsService
-from log import MetricsLoggerFactory
+from log import MetricsDataGatherer
 from datetime import date
 import re
 
@@ -14,7 +14,6 @@ class QuotaService:
                                            redis_connection)
         self._user_service = UserMetricsService(self._user_service_config,
                                                 redis_connection)
-        self._metrics_logger = MetricsLoggerFactory.build(user_service_config)
 
     def check_user_quota(self):
         return self._quota_checker.check()
@@ -46,13 +45,19 @@ class QuotaService:
         self._user_service.increment_service_use(
             self._user_service_config.service_type, "isolines_generated",
             amount=amount)
+        MetricsDataGatherer.add('isolines_generated', amount)
 
     def _log_service_process(self, event):
-        if self._metrics_logger:
-            if event is 'success' or event is 'empty':
-                self._metrics_logger.log(success=True)
-            elif event is 'empty':
-                self._metrics_logger.log(success=False)
+        if event is 'success':
+            MetricsDataGatherer.add('success', True)
+            MetricsDataGatherer.add('successful_rows', 1)
+        elif event is 'empty':
+            MetricsDataGatherer.add('success', True)
+            MetricsDataGatherer.add('successful_rows', 1)
+            MetricsDataGatherer.add('empty_rows', 1)
+        elif event is 'fail':
+            MetricsDataGatherer.add('success', False)
+            MetricsDataGatherer.add('failed_rows', 1)
 
 
 class QuotaChecker:
@@ -93,7 +98,7 @@ class QuotaChecker:
         current_used = self._user_service.used_quota(service_type, today)
         soft_geocoding_limit = self._user_service_config.soft_geocoding_limit
 
-        if soft_geocoding_limit or (user_quota > 0 and current_used <= user_quota):
+        if soft_geocoding_limit or (user_quota > 0 and current_used < user_quota):
             return True
         else:
             return False
@@ -105,7 +110,7 @@ class QuotaChecker:
         current_used = self._user_service.used_quota(service_type, today)
         soft_isolines_limit = self._user_service_config.soft_isolines_limit
 
-        if soft_isolines_limit or (user_quota > 0 and current_used <= user_quota):
+        if soft_isolines_limit or (user_quota > 0 and current_used < user_quota):
             return True
         else:
             return False
@@ -115,8 +120,9 @@ class QuotaChecker:
         today = date.today()
         service_type = self._user_service_config.service_type
         current_used = self._user_service.used_quota(service_type, today)
+        soft_limit = self._user_service_config.soft_limit
 
-        if (user_quota > 0 and current_used <= user_quota):
+        if soft_limit or (user_quota > 0 and current_used < user_quota):
             return True
         else:
             return False
@@ -128,7 +134,7 @@ class QuotaChecker:
         service_type = self._user_service_config.service_type
         current_used = self._user_service.used_quota(service_type, today)
 
-        if soft_limit or (user_quota > 0 and current_used <= user_quota):
+        if soft_limit or (user_quota > 0 and current_used < user_quota):
             return True
         else:
             return False

server/lib/python/cartodb_services/cartodb_services/metrics/user.py

@@ -8,6 +8,8 @@ class UserMetricsService:
     SERVICE_GEOCODER_NOKIA = 'geocoder_here'
     SERVICE_GEOCODER_CACHE = 'geocoder_cache'
     SERVICE_HERE_ISOLINES = 'here_isolines'
+    SERVICE_MAPZEN_ROUTING = 'routing_mapzen'
+    SERVICE_OBSERVATORY = 'obs_general'
     DAY_OF_MONTH_ZERO_PADDED = '%d'
 
     def __init__(self, user_geocoder_config, redis_connection):
@@ -19,6 +21,10 @@ class UserMetricsService:
     def used_quota(self, service_type, date):
         if service_type == self.SERVICE_HERE_ISOLINES:
             return self.__used_isolines_quota(service_type, date)
+        elif service_type == self.SERVICE_MAPZEN_ROUTING:
+            return self.__used_routing_quota(service_type, date)
+        elif service_type == self.SERVICE_OBSERVATORY:
+            return self.__used_observatory_quota(service_type, date)
         else:
             return self.__used_geocoding_quota(service_type, date)
 
@@ -69,6 +75,19 @@ class UserMetricsService:
 
         return current_use
 
+    def __used_observatory_quota(self, service_type, date):
+        date_from, date_to = self.__current_billing_cycle()
+        current_use = 0
+        success_responses = self.get_metrics(service_type,
+                                             'success_responses', date_from,
+                                             date_to)
+        empty_responses = self.get_metrics(service_type,
+                                           'empty_responses', date_from,
+                                           date_to)
+        current_use += (success_responses + empty_responses)
+
+        return current_use
+
     def increment_service_use(self, service_type, metric, date=date.today(),
                               amount=1):
         """ Increment the services uses in monthly and daily basis"""
@@ -85,11 +104,11 @@ class UserMetricsService:
             redis_prefix = self.__parse_redis_prefix(key_prefix, entity_name,
                                                      service, metric, date)
             score = self._redis_connection.zscore(redis_prefix, date.day)
-            aggregated_metric += score if score else 0
+            aggregated_metric += int(score) if score else 0
             zero_padded_day = date.strftime(self.DAY_OF_MONTH_ZERO_PADDED)
             if str(date.day) != zero_padded_day:
                 score = self._redis_connection.zscore(redis_prefix, zero_padded_day)
-                aggregated_metric += score if score else 0
+                aggregated_metric += int(score) if score else 0
 
         return aggregated_metric
 

server/lib/python/cartodb_services/cartodb_services/refactor/backend/org_config.py

@@ -0,0 +1,24 @@
+from cartodb_services.refactor.storage.redis_connection_config import RedisMetadataConnectionConfigBuilder
+from cartodb_services.refactor.storage.redis_connection import RedisConnectionBuilder
+from cartodb_services.refactor.storage.redis_config import RedisOrgConfigStorageBuilder
+
+class OrgConfigBackendFactory(object):
+    """
+    This class abstracts the creation of an org configuration backend. It will return
+    an implementation of the ConfigBackendInterface appropriate to the org, depending
+    on the environment.
+    """
+
+    def __init__(self, orgname, environment, server_config_backend):
+        self._orgname = orgname
+        self._environment = environment
+        self._server_config_backend = server_config_backend
+
+    def get(self):
+        if self._environment.is_onpremise:
+            org_config_backend = self._server_config_backend
+        else:
+            redis_metadata_connection_config = RedisMetadataConnectionConfigBuilder(self._server_config_backend).get()
+            redis_metadata_connection = RedisConnectionBuilder(redis_metadata_connection_config).get()
+            org_config_backend = RedisOrgConfigStorageBuilder(redis_metadata_connection, self._orgname).get()
+        return org_config_backend

server/lib/python/cartodb_services/cartodb_services/refactor/backend/redis_metrics_connection.py

@@ -0,0 +1,17 @@
+from cartodb_services.refactor.tools.redis_mock import RedisConnectionMock
+from cartodb_services.refactor.storage.redis_connection_config import RedisMetricsConnectionConfigBuilder
+from cartodb_services.refactor.storage.redis_connection import RedisConnectionBuilder
+
+class RedisMetricsConnectionFactory(object):
+    def __init__(self, environment, server_config_storage):
+        self._environment = environment
+        self._server_config_storage = server_config_storage
+
+    def get(self):
+        if self._environment.is_onpremise:
+           redis_metrics_connection = RedisConnectionMock()
+        else:
+          redis_metrics_connection_config = RedisMetricsConnectionConfigBuilder(self._server_config_storage).get()
+          redis_metrics_connection = RedisConnectionBuilder(redis_metrics_connection_config).get()
+        return redis_metrics_connection
+

server/lib/python/cartodb_services/cartodb_services/refactor/backend/server_config.py

@@ -0,0 +1,13 @@
+from cartodb_services.refactor.storage.server_config import InDbServerConfigStorage
+
+
+class ServerConfigBackendFactory(object):
+    """
+    This class creates a backend to retrieve server configurations (implementing the ConfigBackendInterface).
+
+    At this moment it will always return an InDbServerConfigStorage, but nothing prevents from changing the
+    implementation. To something that reads from a file, memory or whatever. It is mostly there to keep
+    the layers separated.
+    """
+    def get(self):
+        return InDbServerConfigStorage()

server/lib/python/cartodb_services/cartodb_services/refactor/backend/user_config.py

@@ -0,0 +1,24 @@
+from cartodb_services.refactor.storage.redis_connection_config import RedisMetadataConnectionConfigBuilder
+from cartodb_services.refactor.storage.redis_connection import RedisConnectionBuilder
+from cartodb_services.refactor.storage.redis_config import RedisUserConfigStorageBuilder
+
+class UserConfigBackendFactory(object):
+    """
+    This class abstracts the creation of a user configuration backend. It will return
+    an implementation of the ConfigBackendInterface appropriate to the user, depending
+    on the environment.
+    """
+
+    def __init__(self, username, environment, server_config_backend):
+        self._username = username
+        self._environment = environment
+        self._server_config_backend = server_config_backend
+
+    def get(self):
+        if self._environment.is_onpremise:
+            user_config_backend = self._server_config_backend
+        else:
+            redis_metadata_connection_config = RedisMetadataConnectionConfigBuilder(self._server_config_backend).get()
+            redis_metadata_connection = RedisConnectionBuilder(redis_metadata_connection_config).get()
+            user_config_backend = RedisUserConfigStorageBuilder(redis_metadata_connection, self._username).get()
+        return user_config_backend

server/lib/python/cartodb_services/cartodb_services/refactor/config/exceptions.py

@@ -0,0 +1,2 @@
+class ConfigException(Exception):
+    pass

server/lib/python/cartodb_services/cartodb_services/refactor/core/environment.py

@@ -0,0 +1,57 @@
+class ServerEnvironment(object):
+
+    DEVELOPMENT = 'development'
+    STAGING = 'staging'
+    PRODUCTION = 'production'
+    ONPREMISE = 'onpremise'
+
+    VALID_ENVIRONMENTS = [
+        DEVELOPMENT,
+        STAGING,
+        PRODUCTION,
+        ONPREMISE
+    ]
+
+    def __init__(self, environment_str):
+        assert environment_str in self.VALID_ENVIRONMENTS
+        self._environment_str = environment_str
+
+    def __str__(self):
+        return self._environment_str
+
+    @property
+    def is_development(self):
+        return self._environment_str == self.DEVELOPMENT
+
+    @property
+    def is_staging(self):
+        return self._environment_str == self.STAGING
+
+    @property
+    def is_production(self):
+        return self._environment_str == self.PRODUCTION
+
+    @property
+    def is_onpremise(self):
+        return self._environment_str == self.ONPREMISE
+
+    def __eq__(self, other):
+        return self._environment_str == other._environment_str
+
+
+class ServerEnvironmentBuilder(object):
+
+    DEFAULT_ENVIRONMENT = ServerEnvironment.DEVELOPMENT
+
+    def __init__(self, server_config_storage):
+        self._server_config_storage = server_config_storage
+
+    def get(self):
+        server_config = self._server_config_storage.get('server_conf')
+
+        if not server_config or 'environment' not in server_config:
+            environment_str = self.DEFAULT_ENVIRONMENT
+        else:
+            environment_str = server_config['environment']
+
+        return ServerEnvironment(environment_str)