Version 0.15.5 of the python library

GoogleMapsClientFactory to persist clients #401

NEWS.md

@@ -1,3 +1,81 @@
+October 6th, 2017
+=================
+* Version `0.15.5` of the python library
+    * googlemaps dependency updated to v2.5.1
+    * Google geocoder performance boost: client connections are now reused between queries. See #401
+    * Fixed issue with Google keys validity check. See #382
+    * Fixed inconsistency in service usage failed requests tracking. See f0a3249
+* Client extension tests are now compatible with PostgreSQL 9.5 and 9.6
+
+August 30th, 2017
+=============
+* Version `0.15.4` of the python library
+    * Fixed invalid geometries for isochrones due to `generalize` option. See #397
+
+August 24th, 2017
+=============
+* Improved the documentation
+* Version `0.15.3` of the python library
+    * Disabled DO quota check for users that have it configured . See #395
+
+August 23th, 2017
+=============
+* Version `0.27.0` of the server
+    * New public DO function to perform metadata validation, `obs_metadatavalidation`. See #392
+* Version `0.20.0` of the client
+    * New private function to precheck the DO requests, `_obs_precheck`. See #392
+    * New public DO function to perform metadata validation, `obs_metadatavalidation`. See #392
+
+July 20th, 2017
+=============
+* Version `0.26.0` of the server
+    * New private function to be used by the UI, `_obs_getnumerators`. See #386
+* Version `0.19.0` of the client
+    * New private function to be used by the UI, `_obs_getnumerators`. See #386
+
+July 13th, 2017
+=============
+* Version `0.25.0` of the server
+    * Add postalcode numeric type function. See #387
+
+* Version `0.15.2` of the python library
+    * Fixed some test. See #387
+
+* Version `0.18.0` of the client and version `0.25.0` of the server
+    * Add postalcode numeric type function. See #387
+
+June 28th, 2017
+=============
+* Not version specific, content change only
+    * Removed link to /docs/faqs and enhanced ZCTA note directly in the Postal Coder docs.
+
+May 26th, 2017
+=============
+* Version `0.24.2` of the server
+    * Fixed fallback logic for namedplaces geocoding functions
+
+* Version `0.15.1` of the python library
+    * Fixed some typos and improve exception messages
+    * Added a check for the google client credentials in order to improve the error message
+
+May 16th, 2017
+=============
+* Version `0.24.1` of the server
+    * Fixed an interface mismatch between DS API and OBS backend, when returning no data. See #366
+
+May 9th, 2017
+=============
+* Version `0.17.0` of the client and version `0.24.0` of the server
+    * Fixed some missing return values documented but not present.
+        *  `OBS_GetAvailableGeometries` now returns `geom_type`, `geom_extra` and `geom_tags` in addition to existing values.
+        *  `OBS_GetAvailableTimespans` now returns `timespan_type`, `timespan_extra`, `timespan_tags` in addition to existing values.
+
+March 28th, 2017
+================
+
+* Version 0.16.0 of the client, version 0.23.0 of the server and version 0.15.0 of the python library
+  * Added support for rate-limiting the geocoding services. See #365
+
 March 13th, 2017
 ================
 * Version `0.14.1` of the python library:
@@ -141,7 +219,7 @@ July 25, 2016
 ===========
 * Release server 0.13.3
 * Add provider per service
-* Default provider in case the provider is not setted
+* Default provider in case the provider is not set
 * Refactor and improvements in the multiprovider services functions
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.13.3-server
@@ -457,7 +535,7 @@ https://github.com/CartoDB/dataservices-api/releases/tag/0.3.0-server
 Feb 4, 2016:
 ===========
 * Release server 0.2.0
-* Logic for the google geocoder so the users with this geocoder setted up can use street level geocoding too
+* Logic for the google geocoder so the users with this geocoder set up can use street level geocoding too
 * Refactor of the python library in order to reflect the change to a services extension more than only geocoder
 
 https://github.com/CartoDB/dataservices-api/releases/tag/0.2.0-server

README.md

@@ -23,47 +23,81 @@ Steps to deploy a new Data Services API version :
 
 ### Local install instructions
 
-- install data services geocoder extension
-
-    ```
-    git clone https://github.com/CartoDB/data-services.git
-    cd data-services/geocoder/extension
-    sudo make install
-    ```
-
-- install observatory extension
-
-    ```
-    git clone https://github.com/CartoDB/observatory-extension.git
-    cd observatory
-    sudo make install
-    ```
-
 - install server and client extensions
 
     ```
-    # in data-services repo root path:
+    # in your workspace root path
+    git clone https://github.com/CartoDB/dataservices-api.git
+    cd dataservices-api
     cd client && sudo make install
+    cd -
     cd server/extension && sudo make install
     ```
 
 - install python library
 
     ```
-    # in data-services repo root path:
-    cd server/lib/python/cartodb_services && sudo pip install . --upgrade
+    # in dataservices-api repo root path:
+    cd server/lib/python/cartodb_services && pip install -r requirements.txt && sudo pip install . --upgrade
     ```
 
-- install extensions in user database
+- Create a database to hold all the server part and a user for it
+
+  ```sql
+  CREATE DATABASE dataservices_db ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
+  CREATE USER dataservices_user;
+  ```
+
+- Install needed extensions in `dataservices_db` database
 
     ```
-    create extension cdb_geocoder;
-    create extension plproxy;
-    create extension observatory;
-    create extension cdb_dataservices_server;
-    create extension cdb_dataservices_client;
+    psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS plproxy; COMMIT" -e
+    psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS cdb_dataservices_server; COMMIT" -e
     ```
 
+- [optional] install internal geocoder
+   - Make the extension available in postgres
+     ```
+     git clone https://github.com/CartoDB/data-services.git
+     cd data-services/geocoder/extension
+     sudo make install
+     ```
+
+   - Make sure you have `wget` installed because is needed for the next step.
+
+   - Go to `geocoder` folder and execute the `geocoder_dowload_dumps` script to download the internal geocoder data.
+
+   - Once the data is downloaded, execute this command:
+     ```bash
+     geocoder_restore_dump postgres dataservices_db {DOWNLOADED_DUMPS_FOLDER}/*.sql
+     ```
+
+   - Now we have to make available the extension to be installed by postgres. Follow [this](https://github.com/CartoDB/data-services/tree/master/geocoder/extension) instructions.
+
+   - Now install the extension with:
+     ```
+     psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS cdb_geocoder; COMMIT" -e
+     ```
+
+- [optional] install data observatory extension
+   - Make the extension available in postgresql to be installed
+     ```
+     git clone https://github.com/CartoDB/observatory-extension.git
+     cd observatory
+     sudo make install
+     ```
+   - This extension needs data, dumps are not available so we're going to use the test fixtures to make it work. Execute:
+     ```
+     psql -U postgres -d dataservices_db -f src/pg/test/fixtures/load_fixtures.sql
+     ```
+   - Give permission to execute and select to the `dataservices_user` user:
+     ```
+     psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS observatory VERSION 'dev'; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO dataservices_user; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA observatory TO dataservices_user; COMMIT" -e
+     ```
 
 ### Server configuration
 
@@ -220,14 +254,16 @@ jsonb_set(
 ```
 ### User database configuration
 
+#### Option 1 (manually)
+
 User (client) databases need also some configuration so that the client extension can access the server:
-#### Users/Organizations
+##### Users/Organizations
 
 ```sql
 SELECT CDB_Conf_SetConf('user_config', '{"is_organization": false, "entity_name": "<YOUR_USERNAME>"}');
 ```
 
-#### Dataservices server
+##### Dataservices server
 
 The `geocoder_server_config` (the name is not accurate for historical reasons) entry points
 to the dataservices server DB (you can use a specific database for the server or your same user's):
@@ -238,10 +274,14 @@ SELECT CDB_Conf_SetConf(
     '{ "connection_str": "host=localhost port=5432 dbname=<SERVER_DB_NAME> user=postgres"}'
 );
 ```
-#### Search path
+##### Search path
 
 The search path must be configured in order to be able to execute the functions without using the schema:
 
 ```sql
 ALTER ROLE "<USER_ROLE>" SET search_path="$user", public, cartodb, cdb_dataservices_client;
 ```
+
+#### Option 2 (from builder)
+
+See [the **Configuring Dataservices** documentation](http://cartodb.readthedocs.io/en/latest/operations/configure_data_services.html)

client/Makefile

@@ -21,7 +21,7 @@ DATA = $(NEW_EXTENSION_ARTIFACT) \
 SOURCES_DATA_DIR = sql/
 
 
-REGRESS = $(notdir $(basename $(wildcard test/sql/*test.sql)))
+REGRESS = $(notdir $(basename $(sort $(wildcard test/sql/*test.sql))))
 TEST_DIR = test/
 REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)' --user='postgres'
 

client/cdb_dataservices_client--0.19.0--0.20.0.sql

@@ -0,0 +1,122 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.20.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+-- HERE goes your code to upgrade/downgrade
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._OBS_PreCheck(
+    source_query text,
+    parameters json
+) RETURNS boolean AS $$
+DECLARE
+    errors text[];
+    validator_errors text[];
+    validator_error text;
+    valid boolean;
+    geoms record;
+BEGIN
+    errors := (ARRAY[])::TEXT[];
+    FOR geoms IN EXECUTE format('SELECT ST_GeometryType(the_geom) as geom_type,
+                                 bool_and(st_isvalid(the_geom)) as valid,
+                                 avg(st_npoints(the_geom)) as avg_vertex,
+                                 ST_SetSRID(ST_Extent(the_geom), 4326) as extent,
+                                 count(*)::INT as numgeoms
+                                 FROM (%s) as _source
+                                 GROUP BY ST_GeometryType(the_geom)', source_query)
+    LOOP
+        IF geoms.geom_type NOT IN ('ST_Polygon', 'ST_MultiPolygon', 'ST_Point') THEN
+            errors := array_append(errors, format($data$'Geometry type %s not supported'$data$, geoms.geom_type));
+        END IF;
+
+        IF geoms.valid IS FALSE THEN
+            errors := array_append(errors, 'There are invalid geometries in the input data, please review them');
+        END IF;
+
+        -- 1000 vertex for a geometry is a limit we have in the obs_getdata function. You can check here
+        -- https://github.com/CartoDB/observatory-extension/blob/1.6.0/src/pg/sql/41_observatory_augmentation.sql#L813
+        IF geoms.avg_vertex > 1000 THEN
+            errors := array_append(errors, 'The average number of geometries vertex is greater than 1000, please try to simplify them');
+        END IF;
+
+        -- OBS specific part
+        EXECUTE 'SELECT valid, errors
+        FROM cdb_dataservices_client.OBS_MetadataValidation($1, $2, $3, $4)'
+        INTO valid, validator_errors
+        USING geoms.extent, geoms.geom_type, parameters, geoms.numgeoms;
+        IF valid is FALSE THEN
+            FOR validator_error IN EXECUTE 'SELECT unnest($1)' USING validator_errors
+            LOOP
+                errors := array_append(errors, validator_error);
+            END LOOP;
+        END IF;
+    END LOOP;
+
+    IF CARDINALITY(errors) > 0 THEN
+        RAISE EXCEPTION '%', errors;
+    END IF;
+
+    RETURN TRUE;
+END;
+$$ LANGUAGE 'plpgsql';
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_metadatavalidation (username text, orgname text, geom_extent Geometry(Geometry, 4326), geom_type text, params json, target_geoms integer DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+  SELECT * FROM cdb_dataservices_server.obs_metadatavalidation (username, orgname, geom_extent, geom_type, params, target_geoms);
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_metadatavalidation (geom_extent Geometry(Geometry, 4326) ,geom_type text ,params json ,target_geoms integer DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+DECLARE
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_metadatavalidation(username, orgname, geom_extent, geom_type, params, target_geoms);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_metadatavalidation_exception_safe (geom_extent Geometry(Geometry, 4326) ,geom_type text ,params json ,target_geoms integer DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+DECLARE
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+  BEGIN
+    RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_metadatavalidation(username, orgname, geom_extent, geom_type, params, target_geoms);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._OBS_PreCheck(source_query text, params JSON) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_metadatavalidation(geom_extent Geometry(Geometry, 4326), geom_type text, params json, target_geoms integer) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._obs_metadatavalidation_exception_safe(geom_extent Geometry(Geometry, 4326), geom_type text, params json, target_geoms integer )  TO publicuser;
+

client/cdb_dataservices_client--0.20.0--0.19.0.sql

@@ -0,0 +1,12 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.19.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+-- HERE goes your code to upgrade/downgrade
+DROP FUNCTION IF EXISTS _OBS_PreCheck(text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.obs_metadatavalidation (Geometry(Geometry, 4326), text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_metadatavalidation_exception_safe (Geometry(Geometry, 4326), text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._obs_metadatavalidation (text, text, Geometry(Geometry, 4326), text, json);

client/cdb_dataservices_client.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB dataservices client API extension'
-default_version = '0.15.0'
+default_version = '0.20.0'
 requires = 'plproxy, cartodb'
 superuser = true
 schema = cdb_dataservices_client

client/old_versions/cdb_dataservices_client--0.15.0--0.16.0.sql

@@ -0,0 +1,277 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.16.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_service_get_rate_limit (service text)
+RETURNS json AS $$
+DECLARE
+  ret json;
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  SELECT cdb_dataservices_client._cdb_service_get_rate_limit(username, orgname, service) INTO ret; RETURN ret;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Public dataservices API function
+--
+-- These are the only ones with permissions to publicuser role
+-- and should also be the only ones with SECURITY DEFINER
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_service_set_user_rate_limit (username text ,orgname text ,service text ,rate_limit json)
+RETURNS void AS $$
+DECLARE
+
+
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  PERFORM cdb_dataservices_client._cdb_service_set_user_rate_limit(username, orgname, service, rate_limit);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Public dataservices API function
+--
+-- These are the only ones with permissions to publicuser role
+-- and should also be the only ones with SECURITY DEFINER
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_service_set_org_rate_limit (username text ,orgname text ,service text ,rate_limit json)
+RETURNS void AS $$
+DECLARE
+
+
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  PERFORM cdb_dataservices_client._cdb_service_set_org_rate_limit(username, orgname, service, rate_limit);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Public dataservices API function
+--
+-- These are the only ones with permissions to publicuser role
+-- and should also be the only ones with SECURITY DEFINER
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_service_set_server_rate_limit (username text ,orgname text ,service text ,rate_limit json)
+RETURNS void AS $$
+DECLARE
+
+
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  PERFORM cdb_dataservices_client._cdb_service_set_server_rate_limit(username, orgname, service, rate_limit);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Exception-safe private DataServices API function
+--
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_get_rate_limit_exception_safe (service text)
+RETURNS json AS $$
+DECLARE
+  ret json;
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    SELECT cdb_dataservices_client._cdb_service_get_rate_limit(username, orgname, service) INTO ret; RETURN ret;
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+         RETURN ret;
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Exception-safe private DataServices API function
+--
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_set_user_rate_limit_exception_safe (username text ,orgname text ,service text ,rate_limit json)
+RETURNS void AS $$
+DECLARE
+
+
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    PERFORM cdb_dataservices_client._cdb_service_set_user_rate_limit(username, orgname, service, rate_limit);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Exception-safe private DataServices API function
+--
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_set_org_rate_limit_exception_safe (username text ,orgname text ,service text ,rate_limit json)
+RETURNS void AS $$
+DECLARE
+
+
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    PERFORM cdb_dataservices_client._cdb_service_set_org_rate_limit(username, orgname, service, rate_limit);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+--
+-- Exception-safe private DataServices API function
+--
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_set_server_rate_limit_exception_safe (username text ,orgname text ,service text ,rate_limit json)
+RETURNS void AS $$
+DECLARE
+
+
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+    -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    PERFORM cdb_dataservices_client._cdb_service_set_server_rate_limit(username, orgname, service, rate_limit);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_get_rate_limit (username text, orgname text, service text)
+RETURNS json AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT cdb_dataservices_server.cdb_service_get_rate_limit (username, orgname, service);
+
+$$ LANGUAGE plproxy;
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_set_user_rate_limit (username text, orgname text, service text, rate_limit json)
+RETURNS void AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT cdb_dataservices_server.cdb_service_set_user_rate_limit (username, orgname, service, rate_limit);
+
+$$ LANGUAGE plproxy;
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_set_org_rate_limit (username text, orgname text, service text, rate_limit json)
+RETURNS void AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT cdb_dataservices_server.cdb_service_set_org_rate_limit (username, orgname, service, rate_limit);
+
+$$ LANGUAGE plproxy;
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_service_set_server_rate_limit (username text, orgname text, service text, rate_limit json)
+RETURNS void AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT cdb_dataservices_server.cdb_service_set_server_rate_limit (username, orgname, service, rate_limit);
+
+$$ LANGUAGE plproxy;
+
+
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client.cdb_service_set_user_rate_limit (username text ,orgname text ,service text ,rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client.cdb_service_set_org_rate_limit (username text ,orgname text ,service text ,rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client.cdb_service_set_server_rate_limit (username text ,orgname text ,service text ,rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_set_user_rate_limit_exception_safe (username text ,orgname text ,service text ,rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_set_org_rate_limit_exception_safe (username text ,orgname text ,service text ,rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_set_server_rate_limit_exception_safe (username text ,orgname text ,service text ,rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_get_rate_limit (username text, orgname text, service text) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_set_user_rate_limit (username text, orgname text, service text, rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_set_org_rate_limit (username text, orgname text, service text, rate_limit json) FROM PUBLIC, publicuser;
+REVOKE EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_set_server_rate_limit (username text, orgname text, service text, rate_limit json) FROM PUBLIC, publicuser;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_service_get_rate_limit(service text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._cdb_service_get_rate_limit_exception_safe(service text )  TO publicuser;

client/old_versions/cdb_dataservices_client--0.16.0--0.15.0.sql

@@ -0,0 +1,21 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.15.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_service_get_rate_limit (text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_service_set_user_rate_limit (text, text, text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_service_set_org_rate_limit (text, text, text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_service_set_server_rate_limit (text, text, text, json);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_get_rate_limit_exception_safe (text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_set_user_rate_limit_exception_safe (text, text, text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_set_org_rate_limit_exception_safe (text, text, text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_set_server_rate_limit_exception_safe (text, text, text, json);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_get_rate_limit (text, text, text);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_set_user_rate_limit (text, text, text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_set_org_rate_limit (text, text, text, json);
+DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_service_set_server_rate_limit (text, text, text, json);

client/old_versions/cdb_dataservices_client--0.16.0--0.17.0.sql

@@ -0,0 +1,14 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.17.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE geom_type text;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE geom_extra jsonb;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry ADD ATTRIBUTE geom_tags jsonb;
+
+ALTER TYPE cdb_dataservices_client.obs_meta_timespan ADD ATTRIBUTE timespan_type text;
+ALTER TYPE cdb_dataservices_client.obs_meta_timespan ADD ATTRIBUTE timespan_extra jsonb;
+ALTER TYPE cdb_dataservices_client.obs_meta_timespan ADD ATTRIBUTE timespan_tags jsonb;

client/old_versions/cdb_dataservices_client--0.17.0--0.16.0.sql

@@ -0,0 +1,14 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.16.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE geom_type;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE geom_extra;
+ALTER TYPE cdb_dataservices_client.obs_meta_geometry DROP ATTRIBUTE geom_tags;
+
+ALTER TYPE cdb_dataservices_client.obs_meta_timespan DROP ATTRIBUTE timespan_type;
+ALTER TYPE cdb_dataservices_client.obs_meta_timespan DROP ATTRIBUTE timespan_extra;
+ALTER TYPE cdb_dataservices_client.obs_meta_timespan DROP ATTRIBUTE timespan_tags;

client/old_versions/cdb_dataservices_client--0.17.0--0.18.0.sql

@@ -0,0 +1,325 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.18.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+
+-- Due to an inconsistency between the v0.17.0 script on one side, and the source sql files
+-- (namely 16_custon_types) and incremental migration scripts on the other (from/to 0.16.0),
+-- the order of the fields of the obs_meta_geometry and obs_meta_timespan types
+-- is different between extensions that have been up upgraded to 0.17.0 from prior versions
+-- and those that have been installed directly.
+-- For version 0.18.0 we will keep the "correct" ordering which is the one in the source sql
+-- file and in the migration scripts (new fields added in 0.17.0 at the end).
+-- When upgrading from 0.17.0 we must take into account that these types may or may not have
+-- been defined with the correct ordering, so we need to redefine them.
+-- ALTERing a type does not allow to reorder fields, so we'll need to DROP the types
+-- (and all functions depending on them) and recreate the types and functions affected.
+--          (≧︿≦)                          (╯°□°)╯ ︵ ┻━┻
+
+DROP FUNCTION cdb_dataservices_client.obs_getavailablegeometries(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, timespan text);
+DROP FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, timespan text);
+DROP FUNCTION cdb_dataservices_client._obs_getavailablegeometries(username text, orgname text, bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, timespan text);
+
+DROP FUNCTION cdb_dataservices_client.obs_getavailabletimespans(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, geom_id text);
+DROP FUNCTION cdb_dataservices_client._obs_getavailabletimespans_exception_safe(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, geom_id text);
+DROP FUNCTION cdb_dataservices_client._obs_getavailabletimespans(username text, orgname text, bounds geometry(Geometry, 4326) , filter_tags text[] , numer_id text , denom_id text , geom_id text);
+
+DROP TYPE IF EXISTS cdb_dataservices_client.obs_meta_geometry;
+DROP TYPE IF EXISTS cdb_dataservices_client.obs_meta_timespan;
+
+CREATE TYPE cdb_dataservices_client.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric, geom_type text, geom_extra jsonb, geom_tags jsonb);
+CREATE TYPE cdb_dataservices_client.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean, timespan_type text, timespan_extra jsonb, timespan_tags jsonb);
+
+
+-- TYPE: cdb_dataservices_client.obs_meta_geometry
+
+--
+-- Public dataservices API function
+--
+-- These are the only ones with permissions to publicuser role
+-- and should also be the only ones with SECURITY DEFINER
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailablegeometries (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
+DECLARE
+
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+--
+-- Exception-safe private DataServices API function
+--
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
+DECLARE
+
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries (username text, orgname text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, timespan text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT * FROM cdb_dataservices_server.obs_getavailablegeometries (username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
+
+$$ LANGUAGE plproxy;
+
+-- TYPE: cdb_dataservices_client.obs_meta_timespan
+
+--
+-- Public dataservices API function
+--
+-- These are the only ones with permissions to publicuser role
+-- and should also be the only ones with SECURITY DEFINER
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailabletimespans (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,geom_id text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_timespan AS $$
+DECLARE
+
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailabletimespans(username, orgname, bounds, filter_tags, numer_id, denom_id, geom_id);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+--
+-- Exception-safe private DataServices API function
+--
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailabletimespans_exception_safe (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,geom_id text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_timespan AS $$
+DECLARE
+
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailabletimespans(username, orgname, bounds, filter_tags, numer_id, denom_id, geom_id);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailabletimespans (username text, orgname text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, geom_id text DEFAULT NULL)
+RETURNS SETOF cdb_dataservices_client.obs_meta_timespan AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT * FROM cdb_dataservices_server.obs_getavailabletimespans (username, orgname, bounds, filter_tags, numer_id, denom_id, geom_id);
+
+$$ LANGUAGE plproxy;
+
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_getavailablegeometries(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, timespan text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, timespan text )  TO publicuser;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.obs_getavailabletimespans(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, geom_id text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._obs_getavailabletimespans_exception_safe(bounds geometry(Geometry, 4326), filter_tags text[], numer_id text, denom_id text, geom_id text )  TO publicuser;
+
+
+-- Real migration changes for path 0.17.0 to 0.18.0
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_geocode_postalcode_polygon (postal_code double precision ,country_name text)
+RETURNS Geometry AS $$
+DECLARE
+  ret Geometry;
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  SELECT cdb_dataservices_client._cdb_geocode_postalcode_polygon(username, orgname, postal_code, country_name) INTO ret; RETURN ret;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_geocode_postalcode_point (postal_code double precision ,country_name text)
+RETURNS Geometry AS $$
+DECLARE
+  ret Geometry;
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  SELECT cdb_dataservices_client._cdb_geocode_postalcode_point(username, orgname, postal_code, country_name) INTO ret; RETURN ret;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_polygon_exception_safe (postal_code double precision ,country_name text)
+RETURNS Geometry AS $$
+DECLARE
+  ret Geometry;
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    SELECT cdb_dataservices_client._cdb_geocode_postalcode_polygon(username, orgname, postal_code, country_name) INTO ret; RETURN ret;
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+         RETURN ret;
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_point_exception_safe (postal_code double precision ,country_name text)
+RETURNS Geometry AS $$
+DECLARE
+  ret Geometry;
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    SELECT cdb_dataservices_client._cdb_geocode_postalcode_point(username, orgname, postal_code, country_name) INTO ret; RETURN ret;
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+         RETURN ret;
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_polygon (username text, orgname text, postal_code double precision, country_name text)
+RETURNS Geometry AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon (username, orgname, postal_code, country_name);
+
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_point (username text, orgname text, postal_code double precision, country_name text)
+RETURNS Geometry AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_point (username, orgname, postal_code, country_name);
+
+$$ LANGUAGE plproxy;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_geocode_postalcode_polygon(postal_code double precision, country_name text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_polygon_exception_safe(postal_code double precision, country_name text )  TO publicuser;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_geocode_postalcode_point(postal_code double precision, country_name text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_point_exception_safe(postal_code double precision, country_name text )  TO publicuser;

client/old_versions/cdb_dataservices_client--0.18.0--0.17.0.sql

@@ -0,0 +1,13 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.17.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+DROP FUNCTION cdb_dataservices_client.cdb_geocode_postalcode_polygon (postal_code double precision ,country_name text);
+DROP FUNCTION cdb_dataservices_client.cdb_geocode_postalcode_point (postal_code double precision ,country_name text);
+DROP FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_polygon_exception_safe (postal_code double precision ,country_name text);
+DROP FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_point_exception_safe (postal_code double precision ,country_name text);
+DROP FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_polygon (username text, orgname text, postal_code double precision, country_name text);
+DROP FUNCTION cdb_dataservices_client._cdb_geocode_postalcode_point (username text, orgname text, postal_code double precision, country_name text);

client/old_versions/cdb_dataservices_client--0.18.0--0.19.0.sql

@@ -0,0 +1,71 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.19.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+-- HERE goes your code to upgrade/downgrade
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getnumerators (bounds geometry(Geometry, 4326) DEFAULT NULL ,section_tags text[] DEFAULT ARRAY[]::TEXT[] ,subsection_tags text[] DEFAULT ARRAY[]::TEXT[] ,other_tags text[] DEFAULT ARRAY[]::TEXT[] ,ids text[] DEFAULT ARRAY[]::TEXT[] ,name text DEFAULT NULL ,denom_id text DEFAULT '' ,geom_id text DEFAULT '' ,timespan text DEFAULT '')
+RETURNS SETOF cdb_dataservices_client.obs_meta_numerator AS $$
+DECLARE
+
+  username text;
+  orgname text;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+  RETURN QUERY SELECT * FROM cdb_dataservices_client.__obs_getnumerators(username, orgname, bounds, section_tags, subsection_tags, other_tags, ids, name, denom_id, geom_id, timespan);
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.__obs_getnumerators_exception_safe (bounds geometry(Geometry, 4326) DEFAULT NULL ,section_tags text[] DEFAULT ARRAY[]::TEXT[] ,subsection_tags text[] DEFAULT ARRAY[]::TEXT[] ,other_tags text[] DEFAULT ARRAY[]::TEXT[] ,ids text[] DEFAULT ARRAY[]::TEXT[] ,name text DEFAULT NULL ,denom_id text DEFAULT '' ,geom_id text DEFAULT '' ,timespan text DEFAULT '')
+RETURNS SETOF cdb_dataservices_client.obs_meta_numerator AS $$
+DECLARE
+
+  username text;
+  orgname text;
+  _returned_sqlstate TEXT;
+  _message_text TEXT;
+  _pg_exception_context TEXT;
+BEGIN
+  IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
+    RAISE EXCEPTION 'The api_key must be provided';
+  END IF;
+  SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
+  -- JSON value stored "" is taken as literal
+  IF username IS NULL OR username = '' OR username = '""' THEN
+    RAISE EXCEPTION 'Username is a mandatory argument, check it out';
+  END IF;
+
+
+  BEGIN
+    RETURN QUERY SELECT * FROM cdb_dataservices_client.__obs_getnumerators(username, orgname, bounds, section_tags, subsection_tags, other_tags, ids, name, denom_id, geom_id, timespan);
+  EXCEPTION
+    WHEN OTHERS THEN
+        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
+                                _message_text = MESSAGE_TEXT,
+                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
+        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
+
+  END;
+END;
+$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_client.__obs_getnumerators (username text, orgname text, bounds geometry(Geometry, 4326) DEFAULT NULL, section_tags text[] DEFAULT ARRAY[]::TEXT[], subsection_tags text[] DEFAULT ARRAY[]::TEXT[], other_tags text[] DEFAULT ARRAY[]::TEXT[], ids text[] DEFAULT ARRAY[]::TEXT[], name text DEFAULT NULL, denom_id text DEFAULT '', geom_id text DEFAULT '', timespan text DEFAULT '')
+RETURNS SETOF cdb_dataservices_client.obs_meta_numerator AS $$
+  CONNECT cdb_dataservices_client._server_conn_str();
+
+  SELECT * FROM cdb_dataservices_server._obs_getnumerators (username, orgname, bounds, section_tags, subsection_tags, other_tags, ids, name, denom_id, geom_id, timespan);
+
+$$ LANGUAGE plproxy;
+
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._obs_getnumerators(bounds geometry(Geometry, 4326), section_tags text[], subsection_tags text[], other_tags text[], ids text[], name text, denom_id text, geom_id text, timespan text) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client.__obs_getnumerators_exception_safe(bounds geometry(Geometry, 4326), section_tags text[], subsection_tags text[], other_tags text[], ids text[], name text, denom_id text, geom_id text, timespan text )  TO publicuser;

client/old_versions/cdb_dataservices_client--0.19.0--0.18.0.sql

@@ -0,0 +1,11 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.18.0'" to load this file. \quit
+
+-- Make sure we have a sane search path to create/update the extension
+SET search_path = "$user",cartodb,public,cdb_dataservices_client;
+
+-- HERE goes your code to upgrade/downgrade
+DROP IF EXISTS FUNCTION cdb_dataservices_client._obs_getnumerators (geometry(Geometry, 4326), text[], text[] , text[], text[] , text, text, text,text);
+DROP IF EXISTS FUNCTION cdb_dataservices_client.__obs_getnumerators_exception_safe (geometry(Geometry, 4326), text[], text[] , text[], text[] , text, text, text,text);
+DROP IF EXISTS FUNCTION cdb_dataservices_client.__obs_getnumerators (text, text, geometry(Geometry, 4326), text[], text[] , text[], text[] , text, text, text,text);

client/renderer/interface.yaml

@@ -39,12 +39,24 @@
     - { name: postal_code,  type: text}
     - { name: country_name, type: text}
 
+- name: cdb_geocode_postalcode_polygon
+  return_type: Geometry
+  params:
+    - { name: postal_code,  type: double precision}
+    - { name: country_name, type: text}
+
 - name: cdb_geocode_postalcode_point
   return_type: Geometry
   params:
     - { name: postal_code,  type: text}
     - { name: country_name, type: text}
 
+- name: cdb_geocode_postalcode_point
+  return_type: Geometry
+  params:
+    - { name: postal_code,  type: double precision}
+    - { name: country_name, type: text}
+
 - name: cdb_geocode_ipaddress_point
   return_type: Geometry
   params:
@@ -293,6 +305,16 @@
     - { name: max_score_rank, type: integer, default: 'NULL' }
     - { name: target_geoms, type: integer, default: 'NULL' }
 
+- name: obs_metadatavalidation
+  return_type: TABLE(valid boolean, errors text[])
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: geom_extent,  type: "Geometry(Geometry, 4326)" }
+    - { name: geom_type,  type: text }
+    - { name: params, type: json }
+    - { name: target_geoms, type: integer, default: 'NULL' }
+
 - name: obs_getcategory
   return_type: text
   params:
@@ -369,6 +391,21 @@
     - { name: geom_id, type: text, default: 'NULL' }
     - { name: timespan, type: text, default: 'NULL'}
 
+- name: _obs_getnumerators
+  return_type: SETOF cdb_dataservices_client.obs_meta_numerator
+  multi_row: true
+  multi_field: true
+  params:
+    - { name: bounds, type: "geometry(Geometry, 4326)", default: 'NULL' }
+    - { name: section_tags, type: "text[]", default: 'ARRAY[]::TEXT[]' }
+    - { name: subsection_tags, type: "text[]", default: 'ARRAY[]::TEXT[]' }
+    - { name: other_tags, type: "text[]", default: 'ARRAY[]::TEXT[]' }
+    - { name: ids, type: "text[]", default: 'ARRAY[]::TEXT[]' }
+    - { name: name, type: text, default: 'NULL' }
+    - { name: denom_id, type: text, default: "''" }
+    - { name: geom_id, type: text, default: "''" }
+    - { name: timespan, type: text, default: "''"}
+
 - name: obs_getavailabledenominators
   return_type: SETOF cdb_dataservices_client.obs_meta_denominator
   multi_row: true
@@ -421,3 +458,29 @@
   params:
     - { name: service, type: TEXT }
     - { name: input_size, type: NUMERIC }
+
+- name: cdb_service_get_rate_limit
+  return_type: json
+  params:
+    - { name: service,  type: "text" }
+
+- name: cdb_service_set_user_rate_limit
+  superuser: true
+  return_type: void
+  params:
+    - { name: service,  type: "text" }
+    - { name: rate_limit, type: json }
+
+- name: cdb_service_set_org_rate_limit
+  superuser: true
+  return_type: void
+  params:
+    - { name: service,  type: "text" }
+    - { name: rate_limit, type: json }
+
+- name: cdb_service_set_server_rate_limit
+  superuser: true
+  return_type: void
+  params:
+    - { name: service,  type: "text" }
+    - { name: rate_limit, type: json }

client/renderer/sql-template-renderer

@@ -17,7 +17,7 @@ class SqlTemplateRenderer
   end
 
   def render
-    ERB.new(@template).result(binding)
+    ERB.new(@template, _save_level=nil, _trim_mode='-').result(binding)
   end
 
   def name
@@ -44,16 +44,29 @@ class SqlTemplateRenderer
     @function_signature['geocoder_config_key']
   end
 
-  def params
-    @function_signature['params'].reject(&:empty?).map { |p| "#{p['name']}"}
+  def parameters_info(with_user_org)
+    parameters = []
+    if with_user_org
+      parameters << { 'name' => 'username', 'type' => 'text' }
+      parameters << { 'name' => 'orgname',  'type' => 'text' }
+    end
+    parameters + @function_signature['params'].reject(&:empty?)
   end
 
-  def params_with_type
-    @function_signature['params'].reject(&:empty?).map { |p| "#{p['name']} #{p['type']}" }
+  def user_org_declaration()
+    "username text;\n  orgname text;" unless superuser_function?
   end
 
-  def params_with_type_and_default
-    parameters = @function_signature['params'].reject(&:empty?).map do |p|
+  def params(with_user_org = superuser_function?)
+    parameters_info(with_user_org).map { |p| p['name'].to_s }
+  end
+
+  def params_with_type(with_user_org = superuser_function?)
+    parameters_info(with_user_org).map { |p| "#{p['name']} #{p['type']}" }
+  end
+
+  def params_with_type_and_default(with_user_org = superuser_function?)
+    parameters = parameters_info(with_user_org).map do |p|
       if not p['default'].nil?
         "#{p['name']} #{p['type']} DEFAULT #{p['default']}"
       else
@@ -62,6 +75,49 @@ class SqlTemplateRenderer
     end
     return parameters
   end
+
+  def superuser_function?
+    !!@function_signature['superuser']
+  end
+
+  def void_return_type?
+    return_type.downcase == 'void'
+  end
+
+  def return_declaration
+     "ret #{return_type};" unless void_return_type? || multi_row
+  end
+
+  def return_statement(&block)
+    if block
+      erb_out = block.binding.eval('_erbout')
+
+      if multi_row
+        erb_out << 'RETURN QUERY SELECT * FROM '
+      elsif multi_field
+        erb_out << 'SELECT * FROM '
+      elsif void_return_type?
+        erb_out << 'PERFORM '
+      else
+        erb_out << 'SELECT '
+      end
+      yield
+      if multi_row || void_return_type?
+        erb_out << ';'
+      else
+        erb_out << ' INTO ret;'
+      end
+      if !multi_row && !void_return_type?
+        erb_out << ' RETURN ret;'
+      end
+    else
+      if !multi_row && !void_return_type?
+        ' RETURN ret;'
+      end
+    end
+  end
+
+
 end
 
 

client/renderer/templates/20_public_functions.erb

@@ -7,27 +7,18 @@
 CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>.<%= name %> (<%= params_with_type_and_default.join(' ,') %>)
 RETURNS <%= return_type %> AS $$
 DECLARE
-  <% if not multi_row %>ret <%= return_type %>;<% end %>
-  username text;
-  orgname text;
+  <%= return_declaration if not multi_row %>
+  <%= user_org_declaration %>
 BEGIN
   IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
     RAISE EXCEPTION 'The api_key must be provided';
   END IF;
-  SELECT u, o INTO username, orgname FROM <%= DATASERVICES_CLIENT_SCHEMA %>._cdb_entity_config() AS (u text, o text);
+  <% unless superuser_function? -%>SELECT u, o INTO username, orgname FROM <%= DATASERVICES_CLIENT_SCHEMA %>._cdb_entity_config() AS (u text, o text);<% end %>
   -- JSON value stored "" is taken as literal
   IF username IS NULL OR username = '' OR username = '""' THEN
     RAISE EXCEPTION 'Username is a mandatory argument, check it out';
   END IF;
-  <% if multi_row %>
-    RETURN QUERY
-    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>);
-  <% elsif multi_field %>
-    SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>) INTO ret;
-    RETURN ret;
-  <% else %>
-    SELECT <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>) INTO ret;
-    RETURN ret;
-  <% end %>
+
+  <% return_statement do %><%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= params(_with_user_org=true).join(', ') %>)<% end %>
 END;
 $$ LANGUAGE 'plpgsql' SECURITY DEFINER;

client/renderer/templates/25_exception_safe_private_functions.erb

@@ -5,9 +5,8 @@
 CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>_exception_safe (<%= params_with_type_and_default.join(' ,') %>)
 RETURNS <%= return_type %> AS $$
 DECLARE
-  <% if not multi_row %>ret <%= return_type %>;<% end %>
-  username text;
-  orgname text;
+  <%= return_declaration %>
+  <%= user_org_declaration %>
   _returned_sqlstate TEXT;
   _message_text TEXT;
   _pg_exception_context TEXT;
@@ -15,47 +14,22 @@ BEGIN
   IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
     RAISE EXCEPTION 'The api_key must be provided';
   END IF;
-  SELECT u, o INTO username, orgname FROM <%= DATASERVICES_CLIENT_SCHEMA %>._cdb_entity_config() AS (u text, o text);
+  <% unless superuser_function? -%>SELECT u, o INTO username, orgname FROM <%= DATASERVICES_CLIENT_SCHEMA %>._cdb_entity_config() AS (u text, o text);<% end %>
   -- JSON value stored "" is taken as literal
   IF username IS NULL OR username = '' OR username = '""' THEN
     RAISE EXCEPTION 'Username is a mandatory argument, check it out';
   END IF;
 
-  <% if multi_row %>
-    BEGIN
-      RETURN QUERY
-      SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>);
-    EXCEPTION
-      WHEN OTHERS THEN
+
+  BEGIN
+    <% return_statement do %><%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= params(_with_user_org=true).join(', ') %>)<% end %>
+  EXCEPTION
+    WHEN OTHERS THEN
         GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
                                 _message_text = MESSAGE_TEXT,
                                 _pg_exception_context = PG_EXCEPTION_CONTEXT;
         RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
-    END;
-  <% elsif multi_field %>
-    BEGIN
-      SELECT * FROM <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>) INTO ret;
-      RETURN ret;
-    EXCEPTION
-      WHEN OTHERS THEN
-        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
-                                _message_text = MESSAGE_TEXT,
-                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
-        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
-        RETURN ret;
-    END;
-  <% else %>
-    BEGIN
-      SELECT <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= ['username', 'orgname'].concat(params).join(', ') %>) INTO ret;
-      RETURN ret;
-    EXCEPTION
-      WHEN OTHERS THEN
-        GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
-                                _message_text = MESSAGE_TEXT,
-                                _pg_exception_context = PG_EXCEPTION_CONTEXT;
-        RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
-        RETURN ret;
-    END;
-  <% end %>
+        <%= return_statement %>
+  END;
 END;
 $$ LANGUAGE 'plpgsql' SECURITY DEFINER;

client/renderer/templates/30_plproxy_functions.erb

@@ -1,9 +1,9 @@
-CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (<%= ['username text', 'organization_name text'].concat(params_with_type_and_default).join(', ') %>)
+CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (<%= params_with_type_and_default(_with_user_org=true).join(', ') %>)
 RETURNS <%= return_type %> AS $$
   CONNECT <%= DATASERVICES_CLIENT_SCHEMA %>._server_conn_str();
   <% if multi_field %>
-  SELECT * FROM <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= ['username', 'organization_name'].concat(params).join(', ') %>);
+  SELECT * FROM <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= params(_with_user_org=true).join(', ') %>);
   <% else %>
-  SELECT <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= ['username', 'organization_name'].concat(params).join(', ') %>);
+  SELECT <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= params(_with_user_org=true).join(', ') %>);
   <% end %>
 $$ LANGUAGE plproxy;

client/renderer/templates/90_grant_execute.erb

@@ -1,2 +1,4 @@
+<% unless superuser_function? %>
 GRANT EXECUTE ON FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>.<%= name %>(<%= params_with_type.join(', ') %>) TO publicuser;
-GRANT EXECUTE ON FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>_exception_safe(<%= params_with_type.join(', ') %>) TO publicuser;
+GRANT EXECUTE ON FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>_exception_safe(<%= params_with_type.join(', ') %> )  TO publicuser;
+<% end %>

client/sql/16_custom_types.sql

@@ -15,9 +15,9 @@ CREATE TYPE cdb_dataservices_client.obs_meta_numerator AS (numer_id text, numer_
 
 CREATE TYPE cdb_dataservices_client.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
 
-CREATE TYPE cdb_dataservices_client.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric);
+CREATE TYPE cdb_dataservices_client.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric, geom_type text, geom_extra jsonb, geom_tags jsonb);
 
-CREATE TYPE cdb_dataservices_client.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean);
+CREATE TYPE cdb_dataservices_client.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean, timespan_type text, timespan_extra jsonb, timespan_tags jsonb);
 
 
 -- For quotas and services configuration

client/sql/35_precheck_functions.sql

@@ -0,0 +1,54 @@
+CREATE OR REPLACE FUNCTION cdb_dataservices_client._OBS_PreCheck(
+    source_query text,
+    parameters json
+) RETURNS boolean AS $$
+DECLARE
+    errors text[];
+    validator_errors text[];
+    validator_error text;
+    valid boolean;
+    geoms record;
+BEGIN
+    errors := (ARRAY[])::TEXT[];
+    FOR geoms IN EXECUTE format('SELECT ST_GeometryType(the_geom) as geom_type,
+                                 bool_and(st_isvalid(the_geom)) as valid,
+                                 avg(st_npoints(the_geom)) as avg_vertex,
+                                 ST_SetSRID(ST_Extent(the_geom), 4326) as extent,
+                                 count(*)::INT as numgeoms
+                                 FROM (%s) as _source
+                                 GROUP BY ST_GeometryType(the_geom)', source_query)
+    LOOP
+        IF geoms.geom_type NOT IN ('ST_Polygon', 'ST_MultiPolygon', 'ST_Point') THEN
+            errors := array_append(errors, format($data$'Geometry type %s not supported'$data$, geoms.geom_type));
+        END IF;
+
+        IF geoms.valid IS FALSE THEN
+            errors := array_append(errors, 'There are invalid geometries in the input data, please try to fix them');
+        END IF;
+
+        -- 1000 vertex for a geometry is a limit we have in the obs_getdata function. You can check here
+        -- https://github.com/CartoDB/observatory-extension/blob/1.6.0/src/pg/sql/41_observatory_augmentation.sql#L813
+        IF geoms.avg_vertex > 1000 THEN
+            errors := array_append(errors, 'The average number of vertices per geometry is greater than 1000, please try to simplify them');
+        END IF;
+
+        -- OBS specific part
+        EXECUTE 'SELECT valid, errors
+        FROM cdb_dataservices_client.OBS_MetadataValidation($1, $2, $3, $4)'
+        INTO valid, validator_errors
+        USING geoms.extent, geoms.geom_type, parameters, geoms.numgeoms;
+        IF valid is FALSE THEN
+            FOR validator_error IN EXECUTE 'SELECT unnest($1)' USING validator_errors
+            LOOP
+                errors := array_append(errors, validator_error);
+            END LOOP;
+        END IF;
+    END LOOP;
+
+    IF CARDINALITY(errors) > 0 THEN
+        RAISE EXCEPTION '%', errors;
+    END IF;
+
+    RETURN TRUE;
+END;
+$$ LANGUAGE 'plpgsql';

client/sql/95_grant_execute_manual.sql

@@ -1,2 +1,3 @@
 GRANT EXECUTE ON FUNCTION cdb_dataservices_client._DST_PrepareTableOBS_GetMeasure(output_table_name text, params json) TO publicuser;
 GRANT EXECUTE ON FUNCTION cdb_dataservices_client._DST_PopulateTableOBS_GetMeasure(table_name text, output_table_name text, params json) TO publicuser;
+GRANT EXECUTE ON FUNCTION cdb_dataservices_client._OBS_PreCheck(source_query text, params JSON) TO publicuser;

client/test/expected/00_installation_test.out

@@ -1,6 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION schema_triggers;
 CREATE EXTENSION plpythonu;
 CREATE EXTENSION cartodb;
 CREATE EXTENSION plproxy;

client/test/expected/10_admin0_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server function
@@ -11,8 +12,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied function
 SELECT cdb_geocode_admin0_polygon('Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_admin0_polygon(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_admin0_polygon invoked with params (test_user, <NULL>, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_admin0_polygon(username, orgname, country_name)"
-PL/pgSQL function cdb_geocode_admin0_polygon(text) line 16 at SQL statement
  cdb_geocode_admin0_polygon 
 ----------------------------
  

client/test/expected/20_admin1_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions
@@ -18,8 +19,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied function
 SELECT cdb_geocode_admin1_polygon('California');
 NOTICE:  cdb_dataservices_client._cdb_geocode_admin1_polygon(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_admin1_polygon invoked with params (test_user, <NULL>, California)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_admin1_polygon(username, orgname, admin1_name)"
-PL/pgSQL function cdb_geocode_admin1_polygon(text) line 16 at SQL statement
  cdb_geocode_admin1_polygon 
 ----------------------------
  
@@ -27,8 +26,6 @@ PL/pgSQL function cdb_geocode_admin1_polygon(text) line 16 at SQL statement
 
 SELECT cdb_geocode_admin1_polygon('California', 'United States');
 NOTICE:  cdb_dataservices_client._cdb_geocode_admin1_polygon(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_admin1_polygon invoked with params (test_user, <NULL>, California, United States)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_admin1_polygon(username, orgname, admin1_name, country_name)"
-PL/pgSQL function cdb_geocode_admin1_polygon(text,text) line 16 at SQL statement
  cdb_geocode_admin1_polygon 
 ----------------------------
  

client/test/expected/25_exception_safe_private_functions_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 SET client_min_messages TO warning;
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions to raise exceptions
@@ -31,8 +32,6 @@ SET ROLE test_regular_user;
 -- Exercise the exception safe and the proxied functions
 SELECT _cdb_geocode_street_point_exception_safe('One street, 1');
 WARNING:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE ERROR: Not enough quota or any other exception whatsoever.
-DETAIL:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function _cdb_geocode_street_point_exception_safe(text,text,text,text) line 21 at SQL statement
  _cdb_geocode_street_point_exception_safe 
 ------------------------------------------
  
@@ -40,15 +39,12 @@ PL/pgSQL function _cdb_geocode_street_point_exception_safe(text,text,text,text)
 
 SELECT * FROM _cdb_isodistance_exception_safe('POINT(-3.70568 40.42028)'::geometry, 'walk', ARRAY[300]::integer[]);
 WARNING:  cdb_dataservices_client._cdb_isodistance(6): [contrib_regression] REMOTE ERROR: Not enough quota or any other exception whatsoever.
-DETAIL:  PL/pgSQL function _cdb_isodistance_exception_safe(geometry,text,integer[],text[]) line 21 at RETURN QUERY
  center | data_range | the_geom 
 --------+------------+----------
 (0 rows)
 
 SELECT * FROM _cdb_route_point_to_point_exception_safe('POINT(-3.70237112 40.41706163)'::geometry,'POINT(-3.69909883 40.41236875)'::geometry, 'car', ARRAY['mode_type=shortest']::text[]);
 WARNING:  cdb_dataservices_client._cdb_route_point_to_point(7): [contrib_regression] REMOTE ERROR: Not enough quota or any other exception whatsoever.
-DETAIL:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_point_to_point(username, orgname, origin, destination, mode, options, units)"
-PL/pgSQL function _cdb_route_point_to_point_exception_safe(geometry,geometry,text,text[],text) line 21 at SQL statement
  shape | length | duration 
 -------+--------+----------
        |        |         

client/test/expected/30_namedplaces_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions
@@ -25,8 +26,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied function
 SELECT cdb_geocode_namedplace_point('Elx');
 NOTICE:  cdb_dataservices_client._cdb_geocode_namedplace_point(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_namedplace_point invoked with params (test_user, <NULL>, Elx)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_namedplace_point(username, orgname, city_name)"
-PL/pgSQL function cdb_geocode_namedplace_point(text) line 16 at SQL statement
  cdb_geocode_namedplace_point 
 ------------------------------
  
@@ -34,8 +33,6 @@ PL/pgSQL function cdb_geocode_namedplace_point(text) line 16 at SQL statement
 
 SELECT cdb_geocode_namedplace_point('Elx', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_namedplace_point(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_namedplace_point invoked with params (test_user, <NULL>, Elx, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_namedplace_point(username, orgname, city_name, country_name)"
-PL/pgSQL function cdb_geocode_namedplace_point(text,text) line 16 at SQL statement
  cdb_geocode_namedplace_point 
 ------------------------------
  
@@ -43,8 +40,6 @@ PL/pgSQL function cdb_geocode_namedplace_point(text,text) line 16 at SQL stateme
 
 SELECT cdb_geocode_namedplace_point('Elx', 'Valencia', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_namedplace_point(5): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_namedplace_point invoked with params (test_user, <NULL>, Elx, Valencia, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_namedplace_point(username, orgname, city_name, admin1_name, country_name)"
-PL/pgSQL function cdb_geocode_namedplace_point(text,text,text) line 16 at SQL statement
  cdb_geocode_namedplace_point 
 ------------------------------
  

client/test/expected/40_postalcodes_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions
@@ -18,8 +19,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied function
 SELECT cdb_geocode_postalcode_polygon('03204', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_postalcode_polygon(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_postalcode_polygon invoked with params (test_user, <NULL>, 03204, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_postalcode_polygon(username, orgname, postal_code, country_name)"
-PL/pgSQL function cdb_geocode_postalcode_polygon(text,text) line 16 at SQL statement
  cdb_geocode_postalcode_polygon 
 --------------------------------
  
@@ -27,8 +26,6 @@ PL/pgSQL function cdb_geocode_postalcode_polygon(text,text) line 16 at SQL state
 
 SELECT cdb_geocode_postalcode_point('03204', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_postalcode_point(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_postalcode_point invoked with params (test_user, <NULL>, 03204, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_postalcode_point(username, orgname, postal_code, country_name)"
-PL/pgSQL function cdb_geocode_postalcode_point(text,text) line 16 at SQL statement
  cdb_geocode_postalcode_point 
 ------------------------------
  

client/test/expected/50_ipaddresses_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions
@@ -11,8 +12,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied function
 SELECT cdb_geocode_ipaddress_point('8.8.8.8');
 NOTICE:  cdb_dataservices_client._cdb_geocode_ipaddress_point(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_ipaddress_point invoked with params (test_user, <NULL>, 8.8.8.8)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_ipaddress_point(username, orgname, ip_address)"
-PL/pgSQL function cdb_geocode_ipaddress_point(text) line 16 at SQL statement
  cdb_geocode_ipaddress_point 
 -----------------------------
  

client/test/expected/60_street_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions
@@ -11,8 +12,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied function
 SELECT cdb_geocode_street_point('One street, 1');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, 1, <NULL>, <NULL>, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -20,8 +19,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, <NULL>, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -29,8 +26,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city', 'state');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, state, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -38,8 +33,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city', 'state', 'country');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, state, country)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -47,8 +40,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city', NULL, 'country');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, <NULL>, country)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -56,8 +47,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street, 1');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, 1, <NULL>, <NULL>, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -65,8 +54,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, <NULL>, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -74,8 +61,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city', 'state');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, state, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -83,8 +68,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city', 'state', 'country');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, state, country)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -92,8 +75,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_geocode_street_point('One street', 'city', NULL, 'country');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, One street, city, <NULL>, country)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  

client/test/expected/80_route_point_to_point_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 -- Mock the server functions
@@ -24,8 +25,6 @@ $$ LANGUAGE 'plpgsql';
 -- Exercise the public and the proxied functions
 SELECT cdb_route_point_to_point('POINT(-87.81406 41.89308)'::geometry,'POINT(-87.79209 41.86138)'::geometry, 'car');
 NOTICE:  cdb_dataservices_client._cdb_route_point_to_point(7): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_point_to_point invoked with params (test_user, <NULL>, 0101000000D53E1D8F19F455C0185B087250F24440, 0101000000465F419AB1F255C0D8B628B341EE4440, car, {}, kilometers)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_point_to_point(username, orgname, origin, destination, mode, options, units)"
-PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) line 16 at SQL statement
  cdb_route_point_to_point 
 --------------------------
  (,5.33,100)
@@ -33,8 +32,6 @@ PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) l
 
 SELECT cdb_route_point_to_point('POINT(-87.81406 41.89308)'::geometry,'POINT(-87.79209 41.86138)'::geometry, 'car', ARRAY['mode_type=shortest']::text[]);
 NOTICE:  cdb_dataservices_client._cdb_route_point_to_point(7): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_point_to_point invoked with params (test_user, <NULL>, 0101000000D53E1D8F19F455C0185B087250F24440, 0101000000465F419AB1F255C0D8B628B341EE4440, car, {mode_type=shortest}, kilometers)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_point_to_point(username, orgname, origin, destination, mode, options, units)"
-PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) line 16 at SQL statement
  cdb_route_point_to_point 
 --------------------------
  (,5.33,100)
@@ -42,8 +39,6 @@ PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) l
 
 SELECT cdb_route_point_to_point('POINT(-87.81406 41.89308)'::geometry,'POINT(-87.79209 41.86138)'::geometry, 'car', ARRAY[]::text[], 'miles');
 NOTICE:  cdb_dataservices_client._cdb_route_point_to_point(7): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_point_to_point invoked with params (test_user, <NULL>, 0101000000D53E1D8F19F455C0185B087250F24440, 0101000000465F419AB1F255C0D8B628B341EE4440, car, {}, miles)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_point_to_point(username, orgname, origin, destination, mode, options, units)"
-PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) line 16 at SQL statement
  cdb_route_point_to_point 
 --------------------------
  (,5.33,100)
@@ -51,8 +46,6 @@ PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) l
 
 SELECT cdb_route_with_waypoints(Array['POINT(-87.81406 41.89308)'::geometry,'POINT(-87.80406 41.87308)'::geometry,'POINT(-87.79209 41.86138)'::geometry], 'car');
 NOTICE:  cdb_dataservices_client._cdb_route_with_waypoints(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_with_waypoints invoked with params (test_user, <NULL>, {0101000000D53E1D8F19F455C0185B087250F24440:0101000000650113B875F355C05665DF15C1EF4440:0101000000465F419AB1F255C0D8B628B341EE4440}, car, {}, kilometers)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_with_waypoints(username, orgname, waypoints, mode, options, units)"
-PL/pgSQL function cdb_route_with_waypoints(geometry[],text,text[],text) line 16 at SQL statement
  cdb_route_with_waypoints 
 --------------------------
  (,2.22,500)
@@ -60,8 +53,6 @@ PL/pgSQL function cdb_route_with_waypoints(geometry[],text,text[],text) line 16
 
 SELECT cdb_route_with_waypoints(Array['POINT(-87.81406 41.89308)'::geometry,'POINT(-87.80406 41.87308)'::geometry,'POINT(-87.79209 41.86138)'::geometry], 'car', ARRAY['mode_type=shortest']::text[]);
 NOTICE:  cdb_dataservices_client._cdb_route_with_waypoints(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_with_waypoints invoked with params (test_user, <NULL>, {0101000000D53E1D8F19F455C0185B087250F24440:0101000000650113B875F355C05665DF15C1EF4440:0101000000465F419AB1F255C0D8B628B341EE4440}, car, {mode_type=shortest}, kilometers)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_with_waypoints(username, orgname, waypoints, mode, options, units)"
-PL/pgSQL function cdb_route_with_waypoints(geometry[],text,text[],text) line 16 at SQL statement
  cdb_route_with_waypoints 
 --------------------------
  (,2.22,500)
@@ -69,8 +60,6 @@ PL/pgSQL function cdb_route_with_waypoints(geometry[],text,text[],text) line 16
 
 SELECT cdb_route_with_waypoints(Array['POINT(-87.81406 41.89308)'::geometry,'POINT(-87.80406 41.87308)'::geometry,'POINT(-87.79209 41.86138)'::geometry], 'car', ARRAY[]::text[], 'miles');
 NOTICE:  cdb_dataservices_client._cdb_route_with_waypoints(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_with_waypoints invoked with params (test_user, <NULL>, {0101000000D53E1D8F19F455C0185B087250F24440:0101000000650113B875F355C05665DF15C1EF4440:0101000000465F419AB1F255C0D8B628B341EE4440}, car, {}, miles)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_with_waypoints(username, orgname, waypoints, mode, options, units)"
-PL/pgSQL function cdb_route_with_waypoints(geometry[],text,text[],text) line 16 at SQL statement
  cdb_route_with_waypoints 
 --------------------------
  (,2.22,500)

client/test/expected/999_permissions_test.out

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Use regular user role
 SET ROLE test_regular_user;
 -- Add to the search path the schema
@@ -6,8 +7,6 @@ SET search_path TO public,cartodb,cdb_dataservices_client;
 -- it is public, it shall work
 SELECT cdb_geocode_admin0_polygon('Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_admin0_polygon(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_admin0_polygon invoked with params (test_user, <NULL>, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_admin0_polygon(username, orgname, country_name)"
-PL/pgSQL function cdb_geocode_admin0_polygon(text) line 16 at SQL statement
  cdb_geocode_admin0_polygon 
 ----------------------------
  
@@ -15,8 +14,6 @@ PL/pgSQL function cdb_geocode_admin0_polygon(text) line 16 at SQL statement
 
 SELECT cdb_geocode_admin1_polygon('California');
 NOTICE:  cdb_dataservices_client._cdb_geocode_admin1_polygon(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_admin1_polygon invoked with params (test_user, <NULL>, California)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_admin1_polygon(username, orgname, admin1_name)"
-PL/pgSQL function cdb_geocode_admin1_polygon(text) line 16 at SQL statement
  cdb_geocode_admin1_polygon 
 ----------------------------
  
@@ -24,8 +21,6 @@ PL/pgSQL function cdb_geocode_admin1_polygon(text) line 16 at SQL statement
 
 SELECT cdb_geocode_admin1_polygon('California', 'United States');
 NOTICE:  cdb_dataservices_client._cdb_geocode_admin1_polygon(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_admin1_polygon invoked with params (test_user, <NULL>, California, United States)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_admin1_polygon(username, orgname, admin1_name, country_name)"
-PL/pgSQL function cdb_geocode_admin1_polygon(text,text) line 16 at SQL statement
  cdb_geocode_admin1_polygon 
 ----------------------------
  
@@ -33,8 +28,6 @@ PL/pgSQL function cdb_geocode_admin1_polygon(text,text) line 16 at SQL statement
 
 SELECT cdb_geocode_namedplace_point('Elx');
 NOTICE:  cdb_dataservices_client._cdb_geocode_namedplace_point(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_namedplace_point invoked with params (test_user, <NULL>, Elx)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_namedplace_point(username, orgname, city_name)"
-PL/pgSQL function cdb_geocode_namedplace_point(text) line 16 at SQL statement
  cdb_geocode_namedplace_point 
 ------------------------------
  
@@ -42,8 +35,6 @@ PL/pgSQL function cdb_geocode_namedplace_point(text) line 16 at SQL statement
 
 SELECT cdb_geocode_namedplace_point('Elx', 'Valencia');
 NOTICE:  cdb_dataservices_client._cdb_geocode_namedplace_point(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_namedplace_point invoked with params (test_user, <NULL>, Elx, Valencia)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_namedplace_point(username, orgname, city_name, country_name)"
-PL/pgSQL function cdb_geocode_namedplace_point(text,text) line 16 at SQL statement
  cdb_geocode_namedplace_point 
 ------------------------------
  
@@ -51,8 +42,6 @@ PL/pgSQL function cdb_geocode_namedplace_point(text,text) line 16 at SQL stateme
 
 SELECT cdb_geocode_namedplace_point('Elx', 'Valencia', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_namedplace_point(5): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_namedplace_point invoked with params (test_user, <NULL>, Elx, Valencia, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_namedplace_point(username, orgname, city_name, admin1_name, country_name)"
-PL/pgSQL function cdb_geocode_namedplace_point(text,text,text) line 16 at SQL statement
  cdb_geocode_namedplace_point 
 ------------------------------
  
@@ -60,8 +49,6 @@ PL/pgSQL function cdb_geocode_namedplace_point(text,text,text) line 16 at SQL st
 
 SELECT cdb_geocode_postalcode_polygon('03204', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_postalcode_polygon(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_postalcode_polygon invoked with params (test_user, <NULL>, 03204, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_postalcode_polygon(username, orgname, postal_code, country_name)"
-PL/pgSQL function cdb_geocode_postalcode_polygon(text,text) line 16 at SQL statement
  cdb_geocode_postalcode_polygon 
 --------------------------------
  
@@ -69,8 +56,6 @@ PL/pgSQL function cdb_geocode_postalcode_polygon(text,text) line 16 at SQL state
 
 SELECT cdb_geocode_postalcode_point('03204', 'Spain');
 NOTICE:  cdb_dataservices_client._cdb_geocode_postalcode_point(4): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_postalcode_point invoked with params (test_user, <NULL>, 03204, Spain)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_postalcode_point(username, orgname, postal_code, country_name)"
-PL/pgSQL function cdb_geocode_postalcode_point(text,text) line 16 at SQL statement
  cdb_geocode_postalcode_point 
 ------------------------------
  
@@ -78,8 +63,6 @@ PL/pgSQL function cdb_geocode_postalcode_point(text,text) line 16 at SQL stateme
 
 SELECT cdb_geocode_ipaddress_point('8.8.8.8');
 NOTICE:  cdb_dataservices_client._cdb_geocode_ipaddress_point(3): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_ipaddress_point invoked with params (test_user, <NULL>, 8.8.8.8)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_ipaddress_point(username, orgname, ip_address)"
-PL/pgSQL function cdb_geocode_ipaddress_point(text) line 16 at SQL statement
  cdb_geocode_ipaddress_point 
 -----------------------------
  
@@ -87,8 +70,6 @@ PL/pgSQL function cdb_geocode_ipaddress_point(text) line 16 at SQL statement
 
 SELECT cdb_geocode_street_point('one street, 1');
 NOTICE:  cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_geocode_geocoder_street_point invoked with params (test_user, <NULL>, one street, 1, <NULL>, <NULL>, <NULL>)
-CONTEXT:  SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
-PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL statement
  cdb_geocode_street_point 
 --------------------------
  
@@ -96,8 +77,6 @@ PL/pgSQL function cdb_geocode_street_point(text,text,text,text) line 16 at SQL s
 
 SELECT cdb_route_point_to_point('POINT(-87.81406 41.89308)'::geometry,'POINT(-87.79209 41.86138)'::geometry, 'car');
 NOTICE:  cdb_dataservices_client._cdb_route_point_to_point(7): [contrib_regression] REMOTE NOTICE: cdb_dataservices_server.cdb_route_point_to_point invoked with params (test_user, <NULL>, 0101000000D53E1D8F19F455C0185B087250F24440, 0101000000465F419AB1F255C0D8B628B341EE4440, car, {}, kilometers)
-CONTEXT:  SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_point_to_point(username, orgname, origin, destination, mode, options, units)"
-PL/pgSQL function cdb_route_point_to_point(geometry,geometry,text,text[],text) line 16 at SQL statement
  cdb_route_point_to_point 
 --------------------------
  (,5.33,100)

client/test/sql/00_installation_test.sql

@@ -1,6 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION schema_triggers;
 CREATE EXTENSION plpythonu;
 CREATE EXTENSION cartodb;
 CREATE EXTENSION plproxy;

client/test/sql/10_admin0_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/20_admin1_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/25_exception_safe_private_functions_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 SET client_min_messages TO warning;
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/30_namedplaces_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/40_postalcodes_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/50_ipaddresses_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/60_street_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/80_route_point_to_point_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 

client/test/sql/90_data_observatory_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Add to the search path the schema
 SET search_path TO public,cartodb,cdb_dataservices_client;
 
@@ -218,6 +219,14 @@ BEGIN
 END;
 $$ LANGUAGE 'plpgsql';
 
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_metadatavalidation(username text, orgname text, geom_extent geometry(Geometry, 4326), geom_type text, params JSON, target_geoms INTEGER DEFAULT NULL)
+RETURNS TABLE (valid boolean, errors text[]) AS $$
+BEGIN
+  RAISE NOTICE 'cdb_dataservices_server.obs_metadatavalidation invoked with params (%, %, %, %, %, %)', username, orgname, geom_extent, geom_type, params, target_geoms;
+  RETURN QUERY SELECT true AS valid, ARRAY[]::TEXT[] AS errors;
+END;
+$$ LANGUAGE 'plpgsql';
+
 -- Exercise the public and the proxied function
 SELECT obs_get_demographic_snapshot(ST_SetSRID(ST_Point(-73.936669 , 40.704512), 4326), '2009 - 2013'::text, '"us.census.tiger".block_group'::text);
 SELECT obs_get_segment_snapshot(ST_SetSRID(ST_Point(-73.936669 , 40.704512), 4326), '"us.census.tiger".block_group'::text);
@@ -241,4 +250,5 @@ SELECT obs_getavailableboundaries(ST_SetSRID(ST_Point(-73.936669 , 40.704512), 4
 SELECT obs_getmeta(ST_SetSRID(ST_Point(-73.9, 40.7), 4326), '[{"numer_id": "us.census.acs.B01003001"}]', 1, 1, 1000);
 SELECT obs_getdata(ARRAY['36047'], obs_getmeta(st_setsrid(st_point(-73.9, 40.7), 4326), '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]', 1, 1, 1000));
 SELECT obs_getdata(ARRAY[(ST_SetSRID(ST_Point(-73.9, 40.7), 4326), 1)::geomval], obs_getmeta(st_setsrid(st_point(-73.9, 40.7), 4326), '[{"numer_id": "us.census.acs.B01003001"}]'));
+SELECT obs_metadatavalidation(ST_SetSRID(ST_Point(-73.9, 40.7), 4326), 'ST_Polygon', '[{"numer_id": "us.census.acs.B01003001"}]', 1000);
 

client/test/sql/999_permissions_test.sql

@@ -1,3 +1,4 @@
+\set VERBOSITY terse
 -- Use regular user role
 SET ROLE test_regular_user;
 

doc/geocoding_functions.md

@@ -217,7 +217,7 @@ INSERT INTO {tablename} (the_geom) SELECT cdb_geocode_namedplace_point('New York
 
 These functions geocode your data into point, or polygon, geometries for postal codes. The postal code geocoder covers the United States, France, Australia and Canada; a request for a different country will return an empty response.
 
-**Note:** For the USA, US Census [Zip Code Tabulation Areas](https://www.census.gov/geo/reference/zctas.html) (ZCTA) are used to reference geocodes for USPS postal codes service areas. See the [FAQs](https://carto.com/docs/faqs/datasets-and-data/#why-does-carto-use-census-bureau-zctas-and-not-usps-zip-codes-for-postal-codes) about datasets and data for details.
+**Note:** For the USA, US Census Zip Code Tabulation Areas (ZCTA) are used to reference geocodes for USPS postal codes service areas. This is not a CARTO restriction, this is a US Government licensing protection of their zip code data source; which is not publicly available. Additionally, zip codes are considered service areas and are not actually geometric areas. As a solution, the US Census provides ZCTA data, which tabulates GIS postal codes for USPS locations by aggregating census blocks. For details about how ZCTAs are created, see [ZIP Code™ Tabulation Areas (ZCTAs™)](https://www.census.gov/geo/reference/zctas.html). If you are geocoding data and your zip codes fail, ensure you are using ZCTAs for the postal code.
 
 ### cdb_geocode_postalcode_polygon(_postal_code text, country_name text_)
 

doc/internal/exception_safe.md

@@ -18,7 +18,7 @@ These functions are useful in cases when it is undesirable to rollback a transac
 Fo example if a table is geocoded with:
 
 ```sql
-UPDATE table SET the_geom=cdb_geocode_street_point(user,NULL,address,city,NULL,country);
+UPDATE table SET the_geom=cdb_geocode_street_point(address,city,NULL,country);
 ```
 
 In case of the user geocoding quota being exhausted mid-process, the user could
@@ -28,7 +28,7 @@ transaction rollback.
 We can avoid the problem using the corresponding exception-safe function:
 
 ```sql
-UPDATE table SET the_geom=_cdb_geocode_street_point_exception_safe(user,NULL,address,city,NULL,country);
+UPDATE table SET the_geom=_cdb_geocode_street_point_exception_safe(address,city,NULL,country);
 ```
 
 # Addition Information

doc/rate_limits.md

@@ -0,0 +1,197 @@
+# Rate limits
+
+Services can be rate-limited. (currently only gecoding is limited)
+
+The rate limits configuration can be established at server, organization or user levels, the latter having precedence over the earlier.
+
+The default configuration (a null or empty configuration) doesn't impose any limits.
+
+The configuration consist of a JSON object with two attributes:
+
+* `period`: the rate-limiting period, in seconds.
+* `limit`: the maximum number of request in the established period.
+
+If a service request exceeds the configured rate limits
+(i.e. if more than `limit` calls are performe in a fixed interval of
+duration `period` seconds) the call will fail with an "Rate limit exceeded" error.
+
+## Server-side interface
+
+There's a server-side SQL interface to query or change the configuration.
+
+### cdb_dataservices_server.cdb_service_get_rate_limit(username, orgname, service)
+
+This function returns the rate limit configuration for a given user and service.
+
+#### Returns
+
+The result is a JSON object with the configuration (`period` and `limit` attributes as explained above).
+
+### cdb_dataservices_server.cdb_service_set_user_rate_limit(username, orgname, service, rate_limit)
+
+This function sets the rate limit configuration for the user. This overrides any other configuration.
+
+The configuration is provided as a JSON literal. To remove the user-level configuration `NULL` should be passed as the `rate_limit`.
+
+#### Returns
+
+This functions doesn't return any value.
+
+### cdb_dataservices_server.cdb_service_set_org_rate_limit(username, orgname, service, rate_limit)
+
+This function sets the rate limit configuration for the organization.
+This overrides server level configuration and is overriden by user configuration if present.
+
+The configuration is provided as a JSON literal. To remove the organization-level configuration `NULL` should be passed as the `rate_limit`.
+
+#### Returns
+
+This functions doesn't return any value.
+
+### cdb_dataservices_server.cdb_service_set_server_rate_limit(username, orgname, service, rate_limit)
+
+This function sets the default rate limit configuration for all users accesing the dataservices server. This is overriden by organization of user configuration.
+
+The configuration is provided as a JSON literal. To remove the organization-level configuration `NULL` should be passed as the `rate_limit`.
+
+#### Returns
+
+This functions doesn't return any value.
+
+## Client-side interface
+
+For convenience there's also a client-side interface (in the client dataservices-api extension), consisting
+of public functions to get the current configuration and privileged functions to change it.
+
+### Public functions
+
+These functions are accesible to non-privileged roles, and should only be executed
+using the role corresponding to a CARTO user, since that will determine the
+user and organization to which the rate limits configuration applies.
+
+### cdb_dataservices_client.cdb_service_get_rate_limit(service)
+
+This function returns the rate limit configuration in effect for the specified service
+and the user corresponding to the role which makes the calls. The effective configuration
+may come from any of the configuration levels (server/organization/user); only the
+existing configuration with most precedence is returned.
+
+#### Returns
+
+The result is a JSON object with the configuration (`period` and `limit` attributes as explained above).
+
+#### Example:
+
+```
+SELECT cdb_dataservices_client.cdb_service_get_rate_limit('geocoder');
+
+   cdb_service_get_rate_limit
+---------------------------------
+ {"limit": 1000, "period": 86400}
+(1 row)
+```
+
+
+### Privileged (superuser) functions
+
+Thes functions are not accessible by regular user roles, and the user and organization names must be provided as parameters.
+
+### cdb_dataservices_client.cdb_service_set_user_rate_limit(username, orgname, service, rate_limit)
+
+This function sets the rate limit configuration for the user. This overrides any other configuration.
+
+The configuration is provided as a JSON literal. To remove the user-level configuration `NULL` should be passed as the `rate_limit`.
+
+#### Returns
+
+This functions doesn't return any value.
+
+#### Example
+
+This will configure the geocoder service rate limit for  user  `myusername`, a non-organization user.
+The limit will be set at 1000 requests per day. Since the user doesn't belong to any organization,
+`NULL` will be passed to the organization argument; otherwise the name of the user's organization should
+be provided.
+
+Note that the name of the geocoding services is `geocoder` and not `geocoding`.
+
+```
+SELECT cdb_dataservices_client.cdb_service_set_user_rate_limit(
+    'myusername',
+    NULL,
+    'geocoder',
+    '{"limit":1000,"period":86400}'
+);
+
+ cdb_service_set_user_rate_limit
+---------------------------------
+
+(1 row)
+```
+
+### cdb_dataservices_client.cdb_service_set_org_rate_limit(username, orgname, service, rate_limit)
+
+This function sets the rate limit configuration for the organization.
+This overrides server level configuration and is overriden by user configuration if present.
+
+The configuration is provided as a JSON literal. To remove the organization-level configuration `NULL` should be passed as the `rate_limit`.
+
+#### Returns
+
+This functions doesn't return any value.
+
+#### Example
+
+This will configure the geocoder service rate limit for the `myorg` organization.
+The limit will be set at 100 requests per hour.
+Note that even we're setting the default configuration for the whole organization,
+the name of a user of the organization must be provided for technical reasons.
+
+```
+SELECT cdb_dataservices_client.cdb_service_set_org_rate_limit(
+    'myorgadmin',
+    'myorg',
+    'geocoder',
+    '{"limit":100,"period":3600}'
+);
+
+
+ cdb_service_set_org_rate_limit
+---------------------------------
+
+(1 row)
+```
+
+### cdb_dataservices_client.cdb_service_set_server_rate_limit(username, orgname, service, rate_limit)
+
+This function sets the default rate limit configuration for all users accesing the dataservices server. This is overriden by organization of user configuration.
+
+The configuration is provided as a JSON literal. To remove the organization-level configuration `NULL` should be passed as the `rate_limit`.
+
+#### Returns
+
+This functions doesn't return any value.
+
+#### Example
+
+This will configure the default geocoder service rate limit for all users
+accesing the data-services server.
+The limit will be set at 10000 requests per month.
+Note that even we're setting the default configuration for the server,
+the name of a user and the name of the corresponding organization (or NULL)
+must be provided for technical reasons.
+
+```
+SELECT cdb_dataservices_client.cdb_service_set_server_rate_limit(
+    'myorgadmin',
+    'myorg',
+    'geocoder',
+    '{"limit":10000,"period":108000}'
+);
+
+
+ cdb_service_set_server_rate_limit
+---------------------------------
+
+(1 row)
+```

server/extension/cdb_dataservices_server--0.26.0--0.27.0.sql

@@ -0,0 +1,50 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.27.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_MetadataValidation(
+  username TEXT,
+  orgname TEXT,
+  geometry_extent Geometry(Geometry, 4326),
+  geometry_type text,
+  params JSON,
+  target_geoms INTEGER DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_MetadataValidation(geometry_extent, geometry_type, params, target_geoms);
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_MetadataValidation(
+  username TEXT,
+  orgname TEXT,
+  geometry_extent Geometry(Geometry, 4326),
+  geometry_type text,
+  params JSON,
+  target_geoms INTEGER DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+
+  with metrics('obs_metadatavalidation', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_MetadataValidation($1, $2, $3, $4, $5, $6);", ["text", "text", "Geometry (Geometry, 4326)",  "text", "json", "integer"])
+        result = plpy.execute(obs_plan, [username, orgname, geometry_extent, geometry_type, params, target_geoms])
+        if result:
+          return result
+        else:
+          return []
+    except BaseException as e:
+        import sys
+        logger.error('Error trying to OBS_MetadataValidation', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_MetadataValidation')
+$$ LANGUAGE plpythonu;

server/extension/cdb_dataservices_server--0.27.0--0.26.0.sql

@@ -0,0 +1,7 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.26.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+DROP FUNCTION IF EXISTS cdb_dataservices_server.OBS_MetadataValidation(TEXT, TEXT, Geometry(Geometry, 4326), text, JSON, INTEGER);
+DROP FUNCTION IF EXISTS cdb_dataservices_server._OBS_MetadataValidation(Geometry(Geometry, 4326), text, JSON, INTEGER);

server/extension/cdb_dataservices_server.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB dataservices server extension'
-default_version = '0.22.0'
+default_version = '0.27.0'
 requires = 'plpythonu, plproxy, postgis, cdb_geocoder'
 superuser = true
 schema = cdb_dataservices_server

server/extension/old_versions/cdb_dataservices_server--0.22.0--0.23.0.sql

@@ -0,0 +1,207 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.23.0'" to load this file. \quit
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_get_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT)
+RETURNS JSON AS $$
+  import json
+  from cartodb_services.config import ServiceConfiguration, RateLimitsConfigBuilder
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  service_config = ServiceConfiguration(service, username, orgname)
+  rate_limit_config = RateLimitsConfigBuilder(service_config.server, service_config.user, service_config.org, service=service, username=username, orgname=orgname).get()
+  if rate_limit_config.is_limited():
+      return json.dumps({'limit': rate_limit_config.limit, 'period': rate_limit_config.period})
+  else:
+      return None
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_user_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT,
+  rate_limit_json JSON)
+RETURNS VOID AS $$
+  import json
+  from cartodb_services.config import RateLimitsConfig, RateLimitsConfigSetter
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  config_setter = RateLimitsConfigSetter(service=service, username=username, orgname=orgname)
+  if rate_limit_json:
+      rate_limit = json.loads(rate_limit_json)
+      limit = rate_limit.get('limit', None)
+      period = rate_limit.get('period', None)
+  else:
+      limit = None
+      period = None
+  config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
+  config_setter.set_user_rate_limits(config)
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_org_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT,
+  rate_limit_json JSON)
+RETURNS VOID AS $$
+  import json
+  from cartodb_services.config import RateLimitsConfig, RateLimitsConfigSetter
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  config_setter = RateLimitsConfigSetter(service=service, username=username, orgname=orgname)
+  if rate_limit_json:
+      rate_limit = json.loads(rate_limit_json)
+      limit = rate_limit.get('limit', None)
+      period = rate_limit.get('period', None)
+  else:
+      limit = None
+      period = None
+  config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
+  config_setter.set_org_rate_limits(config)
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_server_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT,
+  rate_limit_json JSON)
+RETURNS VOID AS $$
+  import json
+  from cartodb_services.config import RateLimitsConfig, RateLimitsConfigSetter
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  config_setter = RateLimitsConfigSetter(service=service, username=username, orgname=orgname)
+  if rate_limit_json:
+      rate_limit = json.loads(rate_limit_json)
+      limit = rate_limit.get('limit', None)
+      period = rate_limit.get('period', None)
+  else:
+      limit = None
+      period = None
+  config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
+  config_setter.set_server_rate_limits(config)
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE
+FUNCTION cdb_dataservices_server.CDB_Conf_SetConf(key text, value JSON)
+    RETURNS void AS $$
+BEGIN
+    PERFORM cdb_dataservices_server.CDB_Conf_RemoveConf(key);
+    EXECUTE 'INSERT INTO cartodb.CDB_CONF (KEY, VALUE) VALUES ($1, $2);' USING key, value;
+END
+$$ LANGUAGE PLPGSQL VOLATILE SECURITY DEFINER;
+
+CREATE OR REPLACE
+FUNCTION cdb_dataservices_server.CDB_Conf_RemoveConf(key text)
+    RETURNS void AS $$
+BEGIN
+    EXECUTE 'DELETE FROM cartodb.CDB_CONF WHERE KEY = $1;' USING key;
+END
+$$ LANGUAGE PLPGSQL VOLATILE SECURITY DEFINER;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
+RETURNS Geometry AS $$
+  from cartodb_services.tools import LegacyServiceManager
+  from cartodb_services.here import HereMapsGeocoder
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
+  service_manager.assert_within_limits()
+
+  try:
+    geocoder = HereMapsGeocoder(service_manager.config.heremaps_app_id, service_manager.config.heremaps_app_code, service_manager.logger, service_manager.config.heremaps_service_params)
+    coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
+    if coordinates:
+      service_manager.quota_service.increment_success_service_use()
+      plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
+      point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
+      return point['st_setsrid']
+    else:
+      service_manager.quota_service.increment_empty_service_use()
+      return None
+  except BaseException as e:
+    import sys
+    service_manager.quota_service.increment_failed_service_use()
+    service_manager.logger.error('Error trying to geocode street point using here maps', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to geocode street point using here maps')
+  finally:
+    service_manager.quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_google_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
+RETURNS Geometry AS $$
+  from cartodb_services.tools import LegacyServiceManager
+  from cartodb_services.google import GoogleMapsGeocoder
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
+  service_manager.assert_within_limits(quota=False)
+
+  try:
+    geocoder = GoogleMapsGeocoder(service_manager.config.google_client_id, service_manager.config.google_api_key, service_manager.logger)
+    coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
+    if coordinates:
+      service_manager.quota_service.increment_success_service_use()
+      plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
+      point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
+      return point['st_setsrid']
+    else:
+      service_manager.quota_service.increment_empty_service_use()
+      return None
+  except BaseException as e:
+    import sys
+    service_manager.quota_service.increment_failed_service_use()
+    service_manager.logger.error('Error trying to geocode street point using google maps', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to geocode street point using google maps')
+  finally:
+    service_manager.quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
+RETURNS Geometry AS $$
+  from cartodb_services.tools import ServiceManager
+  from cartodb_services.mapzen import MapzenGeocoder
+  from cartodb_services.mapzen.types import country_to_iso3
+  from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
+  service_manager.assert_within_limits()
+
+  try:
+    geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
+    country_iso3 = None
+    if country:
+      country_iso3 = country_to_iso3(country)
+    coordinates = geocoder.geocode(searchtext=searchtext, city=city,
+                                   state_province=state_province,
+                                   country=country_iso3, search_type='address')
+    if coordinates:
+      service_manager.quota_service.increment_success_service_use()
+      plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
+      point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
+      return point['st_setsrid']
+    else:
+      service_manager.quota_service.increment_empty_service_use()
+      return None
+  except BaseException as e:
+    import sys
+    service_manager.quota_service.increment_failed_service_use()
+    service_manager.logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
+    raise Exception('Error trying to geocode street point using mapzen')
+  finally:
+    service_manager.quota_service.increment_total_service_use()
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.23.0--0.22.0.sql

@@ -0,0 +1,15 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.22.0'" to load this file. \quit
+
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_service_get_rate_limit(TEXT, TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_service_set_user_rate_limit(TEXT, TEXT, TEXT, JSON);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_service_set_org_rate_limit(TEXT, TEXT, TEXT, JSON);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_service_set_server_rate_limit(TEXT, TEXT, TEXT, JSON);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_server.CDB_Conf_SetConf(text, JSON);
+DROP FUNCTION IF EXISTS cdb_dataservices_server.CDB_Conf_RemoveConf(text);
+
+DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_here_geocode_street_point(TEXT, TEXT, TEXT, TEXT,  TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_google_geocode_street_point(TEXT, TEXT, TEXT, TEXT,  TEXT, TEXT);
+DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapzen_geocode_street_point(TEXT, TEXT, TEXT, TEXT,  TEXT, TEXT);

server/extension/old_versions/cdb_dataservices_server--0.23.0--0.24.0.sql

@@ -0,0 +1,11 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.0'" to load this file. \quit
+
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE geom_type text;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE geom_extra jsonb;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry ADD ATTRIBUTE geom_tags jsonb;
+
+ALTER TYPE cdb_dataservices_server.obs_meta_timespan ADD ATTRIBUTE timespan_type text;
+ALTER TYPE cdb_dataservices_server.obs_meta_timespan ADD ATTRIBUTE timespan_extra jsonb;
+ALTER TYPE cdb_dataservices_server.obs_meta_timespan ADD ATTRIBUTE timespan_tags jsonb;

server/extension/old_versions/cdb_dataservices_server--0.24.0--0.23.0.sql

@@ -0,0 +1,12 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.23.0'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE geom_type;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE geom_extra;
+ALTER TYPE cdb_dataservices_server.obs_meta_geometry DROP ATTRIBUTE geom_tags;
+
+ALTER TYPE cdb_dataservices_server.obs_meta_timespan DROP ATTRIBUTE timespan_type;
+ALTER TYPE cdb_dataservices_server.obs_meta_timespan DROP ATTRIBUTE timespan_extra;
+ALTER TYPE cdb_dataservices_server.obs_meta_timespan DROP ATTRIBUTE timespan_tags;

server/extension/old_versions/cdb_dataservices_server--0.24.0--0.24.1.sql

@@ -0,0 +1,100 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.1'" to load this file. \quit
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
+  username TEXT,
+  orgname TEXT,
+  geomvals geomval[],
+  params JSON,
+  merge BOOLEAN DEFAULT True)
+RETURNS TABLE (
+  id INT,
+  data JSON
+) AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  quota_service = QuotaService(user_obs_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  with metrics('obs_getdata', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4, $5);", ["text", "text", "geomval[]", "json", "boolean"])
+        result = plpy.execute(obs_plan, [username, orgname, geomvals, params, merge])
+        empty_results = len(geomvals) - len(result)
+        if empty_results > 0:
+          quota_service.increment_empty_service_use(empty_results)
+        if result:
+          quota_service.increment_success_service_use(len(result))
+          return result
+        else:
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use(len(geomvals))
+        logger.error('Error trying to OBS_GetData', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetData')
+    finally:
+        quota_service.increment_total_service_use(len(geomvals))
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
+  username TEXT,
+  orgname TEXT,
+  geomrefs TEXT[],
+  params JSON)
+RETURNS TABLE (
+  id TEXT,
+  data JSON
+) AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  quota_service = QuotaService(user_obs_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  with metrics('obs_getdata', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4);", ["text", "text", "text[]", "json"])
+        result = plpy.execute(obs_plan, [username, orgname, geomrefs, params])
+        empty_results = len(geomrefs) - len(result)
+        if empty_results > 0:
+          quota_service.increment_empty_service_use(empty_results)
+        if result:
+          quota_service.increment_success_service_use(len(result))
+          return result
+        else:
+          return []
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use(len(geomrefs))
+        exc_info = sys.exc_info()
+        logger.error('%s, %s, %s' % (exc_info[0], exc_info[1], exc_info[2]))
+        logger.error('Error trying to OBS_GetData', exc_info, data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetData')
+    finally:
+        quota_service.increment_total_service_use(len(geomrefs))
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.24.1--0.24.0.sql

@@ -0,0 +1,100 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.0'" to load this file. \quit
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
+  username TEXT,
+  orgname TEXT,
+  geomvals geomval[],
+  params JSON,
+  merge BOOLEAN DEFAULT True)
+RETURNS TABLE (
+  id INT,
+  data JSON
+) AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  quota_service = QuotaService(user_obs_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  with metrics('obs_getdata', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4, $5);", ["text", "text", "geomval[]", "json", "boolean"])
+        result = plpy.execute(obs_plan, [username, orgname, geomvals, params, merge])
+        empty_results = len(geomvals) - len(result)
+        if empty_results > 0:
+          quota_service.increment_empty_service_use(empty_results)
+        if result:
+          quota_service.increment_success_service_use(len(result))
+          return result
+        else:
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use(len(geomvals))
+        logger.error('Error trying to OBS_GetData', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetData')
+    finally:
+        quota_service.increment_total_service_use(len(geomvals))
+$$ LANGUAGE plpythonu;
+
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
+  username TEXT,
+  orgname TEXT,
+  geomrefs TEXT[],
+  params JSON)
+RETURNS TABLE (
+  id TEXT,
+  data JSON
+) AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.metrics import QuotaService
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+  quota_service = QuotaService(user_obs_config, redis_conn)
+  if not quota_service.check_user_quota():
+    raise Exception('You have reached the limit of your quota')
+
+  with metrics('obs_getdata', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4);", ["text", "text", "text[]", "json"])
+        result = plpy.execute(obs_plan, [username, orgname, geomrefs, params])
+        empty_results = len(geomrefs) - len(result)
+        if empty_results > 0:
+          quota_service.increment_empty_service_use(empty_results)
+        if result:
+          quota_service.increment_success_service_use(len(result))
+          return result
+        else:
+          return None
+    except BaseException as e:
+        import sys
+        quota_service.increment_failed_service_use(len(geomrefs))
+        exc_info = sys.exc_info()
+        logger.error('%s, %s, %s' % (exc_info[0], exc_info[1], exc_info[2]))
+        logger.error('Error trying to OBS_GetData', exc_info, data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_GetData')
+    finally:
+        quota_service.increment_total_service_use(len(geomrefs))
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.24.1--0.24.2.sql

@@ -0,0 +1,40 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.2'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+---- cdb_geocode_namedplace_point(city_name text)
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text)
+RETURNS Geometry AS $$
+  import spiexceptions
+  try:
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, city_name])[0]['point']
+  except spiexceptions.ExternalRoutineException as e:
+    internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
+    return plpy.execute(internal_plan, [username, orgname, city_name])[0]['point']
+$$ LANGUAGE plpythonu;
+
+---- cdb_geocode_namedplace_point(city_name text, country_name text)
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, country_name text)
+RETURNS Geometry AS $$
+  import spiexceptions
+  try:
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, city_name, country_name])[0]['point']
+  except spiexceptions.ExternalRoutineException as e:
+    internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
+    return plpy.execute(internal_plan, [username, orgname, city_name, country_name])[0]['point']
+$$ LANGUAGE plpythonu;
+
+---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, admin1_name text, country_name text)
+RETURNS Geometry AS $$
+  import spiexceptions
+  try:
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
+  except spiexceptions.ExternalRoutineException as e:
+    internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
+    return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.24.2--0.24.1.sql

@@ -0,0 +1,36 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.1'" to load this file. \quit
+
+-- HERE goes your code to upgrade/downgrade
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text)
+RETURNS Geometry AS $$
+  try:
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, city_name])[0]['point']
+  except BaseException as e:
+    internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
+    return plpy.execute(internal_plan, [username, orgname, city_name])[0]['point']
+$$ LANGUAGE plpythonu;
+
+---- cdb_geocode_namedplace_point(city_name text, country_name text)
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, country_name text)
+RETURNS Geometry AS $$
+  try:
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, city_name, country_name])[0]['point']
+  except BaseException as e:
+    internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
+    return plpy.execute(internal_plan, [username, orgname, city_name, country_name])[0]['point']
+$$ LANGUAGE plpythonu;
+
+---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, admin1_name text, country_name text)
+RETURNS Geometry AS $$
+  try:
+    mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
+    return plpy.execute(mapzen_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
+  except BaseException as e:
+    internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
+    return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
+$$ LANGUAGE plpythonu;

server/extension/old_versions/cdb_dataservices_server--0.24.2--0.25.0.sql

@@ -0,0 +1,24 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.25.0'" to load this file. \quit
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_point(username, orgname, code::text);
+$$ LANGUAGE SQL;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision, country text)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_point(username, orgname, code::text, country);
+$$ LANGUAGE SQL;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon(username, orgname, code::text)
+$$ LANGUAGE SQL;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision, country text)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon(username, orgname, code::text, country);
+$$ LANGUAGE SQL;
+

server/extension/old_versions/cdb_dataservices_server--0.25.0--0.24.2.sql

@@ -0,0 +1,8 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.2'" to load this file. \quit
+
+DROP FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision);
+DROP FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision, country text);
+DROP FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision);
+DROP FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision, country text);

server/extension/old_versions/cdb_dataservices_server--0.25.0--0.26.0.sql

@@ -0,0 +1,20 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.25.0'" to load this file. \quit
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetNumerators(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  section_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  subsection_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  other_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  ids TEXT[] DEFAULT ARRAY[]::TEXT[],
+  name TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT '',
+  geom_id TEXT DEFAULT '',
+  timespan TEXT DEFAULT '')
+RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory._OBS_GetNumerators(bounds, section_tags, subsection_tags, other_tags, ids, name, denom_id, geom_id, timespan);
+$$ LANGUAGE plproxy;

server/extension/old_versions/cdb_dataservices_server--0.26.0--0.25.0.sql

@@ -0,0 +1,5 @@
+--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
+-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.24.2'" to load this file. \quit
+
+DROP IF EXISTS FUNCTION cdb_dataservices_server._OBS_GetNumerators (text, text, geometry(Geometry, 4326), text[], text[] , text[], text[] , text, text, text,text);

server/extension/sql/110_data_observatory_augmentation.sql

@@ -619,7 +619,7 @@ RETURNS TABLE (
           quota_service.increment_success_service_use(len(result))
           return result
         else:
-          return None
+          return []
     except BaseException as e:
         import sys
         quota_service.increment_failed_service_use(len(geomvals))
@@ -678,7 +678,7 @@ RETURNS TABLE (
           quota_service.increment_success_service_use(len(result))
           return result
         else:
-          return None
+          return []
     except BaseException as e:
         import sys
         quota_service.increment_failed_service_use(len(geomrefs))
@@ -737,3 +737,49 @@ RETURNS JSON AS $$
         logger.error('Error trying to OBS_GetMeta', sys.exc_info(), data={"username": username, "orgname": orgname})
         raise Exception('Error trying to OBS_GetMeta')
 $$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_MetadataValidation(
+  username TEXT,
+  orgname TEXT,
+  geometry_extent Geometry(Geometry, 4326),
+  geometry_type text,
+  params JSON,
+  target_geoms INTEGER DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory.OBS_MetadataValidation(geometry_extent, geometry_type, params, target_geoms);
+$$ LANGUAGE plproxy;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_MetadataValidation(
+  username TEXT,
+  orgname TEXT,
+  geometry_extent Geometry(Geometry, 4326),
+  geometry_type text,
+  params JSON,
+  target_geoms INTEGER DEFAULT NULL)
+RETURNS TABLE(valid boolean, errors text[]) AS $$
+  from cartodb_services.metrics import metrics
+  from cartodb_services.tools import Logger,LoggerConfig
+
+  plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
+  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
+  plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
+  user_obs_config = GD["user_obs_config_{0}".format(username)]
+
+  plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
+  logger_config = GD["logger_config"]
+  logger = Logger(logger_config)
+
+  with metrics('obs_metadatavalidation', user_obs_config, logger):
+    try:
+        obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_MetadataValidation($1, $2, $3, $4, $5, $6);", ["text", "text", "Geometry (Geometry, 4326)",  "text", "json", "integer"])
+        result = plpy.execute(obs_plan, [username, orgname, geometry_extent, geometry_type, params, target_geoms])
+        if result:
+          return result
+        else:
+          return []
+    except BaseException as e:
+        import sys
+        logger.error('Error trying to OBS_MetadataValidation', sys.exc_info(), data={"username": username, "orgname": orgname})
+        raise Exception('Error trying to OBS_MetadataValidation')
+$$ LANGUAGE plpythonu;

server/extension/sql/130_data_observatory_helper.sql

@@ -20,6 +20,23 @@ RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
   SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan);
 $$ LANGUAGE plproxy;
 
+CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetNumerators(
+  username TEXT,
+  orgname TEXT,
+  bounds geometry(Geometry, 4326) DEFAULT NULL,
+  section_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  subsection_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  other_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  ids TEXT[] DEFAULT ARRAY[]::TEXT[],
+  name TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT '',
+  geom_id TEXT DEFAULT '',
+  timespan TEXT DEFAULT '')
+RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
+  CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
+  SELECT * FROM cdb_observatory._OBS_GetNumerators(bounds, section_tags, subsection_tags, other_tags, ids, name, denom_id, geom_id, timespan);
+$$ LANGUAGE plproxy;
+
 CREATE TYPE cdb_dataservices_server.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableDenominators(
@@ -35,7 +52,7 @@ RETURNS SETOF cdb_dataservices_server.obs_meta_denominator AS $$
   SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan);
 $$ LANGUAGE plproxy;
 
-CREATE TYPE cdb_dataservices_server.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric);
+CREATE TYPE cdb_dataservices_server.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric, geom_type text, geom_extra jsonb, geom_tags jsonb);
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(
   username TEXT,
@@ -50,7 +67,7 @@ RETURNS SETOF cdb_dataservices_server.obs_meta_geometry AS $$
   SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan);
 $$ LANGUAGE plproxy;
 
-CREATE TYPE cdb_dataservices_server.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean);
+CREATE TYPE cdb_dataservices_server.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean, timespan_type text, timespan_extra jsonb, timespan_tags jsonb);
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableTimespans(
   username TEXT,

server/extension/sql/15_config_helper.sql

@@ -16,6 +16,24 @@ RETURNS JSON AS $$
     SELECT VALUE FROM cartodb.cdb_conf WHERE key = input_key;
 $$ LANGUAGE SQL STABLE SECURITY DEFINER;
 
+CREATE OR REPLACE
+FUNCTION cdb_dataservices_server.CDB_Conf_SetConf(key text, value JSON)
+    RETURNS void AS $$
+BEGIN
+    PERFORM cdb_dataservices_server.CDB_Conf_RemoveConf(key);
+    EXECUTE 'INSERT INTO cartodb.CDB_CONF (KEY, VALUE) VALUES ($1, $2);' USING key, value;
+END
+$$ LANGUAGE PLPGSQL VOLATILE SECURITY DEFINER;
+
+CREATE OR REPLACE
+FUNCTION cdb_dataservices_server.CDB_Conf_RemoveConf(key text)
+    RETURNS void AS $$
+BEGIN
+    EXECUTE 'DELETE FROM cartodb.CDB_CONF WHERE KEY = $1;' USING key;
+END
+$$ LANGUAGE PLPGSQL VOLATILE SECURITY DEFINER;
+
+
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_geocoder_config(username text, orgname text, provider text DEFAULT NULL)
 RETURNS boolean AS $$
   cache_key = "user_geocoder_config_{0}".format(username)

server/extension/sql/20_geocode_street.sql

@@ -72,109 +72,77 @@ $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
 RETURNS Geometry AS $$
+  from cartodb_services.tools import LegacyServiceManager
   from cartodb_services.here import HereMapsGeocoder
-  from cartodb_services.metrics import QuotaService
-  from cartodb_services.tools import Logger,LoggerConfig
-
-  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
-  user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
 
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
-  logger_config = GD["logger_config"]
-  logger = Logger(logger_config)
-  # -- Check the quota
-  quota_service = QuotaService(user_geocoder_config, redis_conn)
-  if not quota_service.check_user_quota():
-    raise Exception('You have reached the limit of your quota')
+  service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
+  service_manager.assert_within_limits()
 
   try:
-    geocoder = HereMapsGeocoder(user_geocoder_config.heremaps_app_id, user_geocoder_config.heremaps_app_code, logger, user_geocoder_config.heremaps_service_params)
+    geocoder = HereMapsGeocoder(service_manager.config.heremaps_app_id, service_manager.config.heremaps_app_code, service_manager.logger, service_manager.config.heremaps_service_params)
     coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
     if coordinates:
-      quota_service.increment_success_service_use()
+      service_manager.quota_service.increment_success_service_use()
       plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
       point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
       return point['st_setsrid']
     else:
-      quota_service.increment_empty_service_use()
+      service_manager.quota_service.increment_empty_service_use()
       return None
   except BaseException as e:
     import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode street point using here maps', sys.exc_info(), data={"username": username, "orgname": orgname})
+    service_manager.quota_service.increment_failed_service_use()
+    service_manager.logger.error('Error trying to geocode street point using here maps', sys.exc_info(), data={"username": username, "orgname": orgname})
     raise Exception('Error trying to geocode street point using here maps')
   finally:
-    quota_service.increment_total_service_use()
+    service_manager.quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_google_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
 RETURNS Geometry AS $$
+  from cartodb_services.tools import LegacyServiceManager
   from cartodb_services.google import GoogleMapsGeocoder
-  from cartodb_services.metrics import QuotaService
-  from cartodb_services.tools import Logger,LoggerConfig
-
-  redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
-  user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
 
   plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
-  logger_config = GD["logger_config"]
-  logger = Logger(logger_config)
-  quota_service = QuotaService(user_geocoder_config, redis_conn)
+  service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
+  service_manager.assert_within_limits(quota=False)
 
   try:
-    geocoder = GoogleMapsGeocoder(user_geocoder_config.google_client_id, user_geocoder_config.google_api_key, logger)
+    geocoder = GoogleMapsGeocoder(service_manager.config.google_client_id, service_manager.config.google_api_key, service_manager.logger)
     coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
     if coordinates:
-      quota_service.increment_success_service_use()
+      service_manager.quota_service.increment_success_service_use()
       plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
       point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
       return point['st_setsrid']
     else:
-      quota_service.increment_empty_service_use()
+      service_manager.quota_service.increment_empty_service_use()
       return None
   except BaseException as e:
     import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode street point using google maps', sys.exc_info(), data={"username": username, "orgname": orgname})
+    service_manager.quota_service.increment_failed_service_use()
+    service_manager.logger.error('Error trying to geocode street point using google maps', sys.exc_info(), data={"username": username, "orgname": orgname})
     raise Exception('Error trying to geocode street point using google maps')
   finally:
-    quota_service.increment_total_service_use()
+    service_manager.quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
 CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
 RETURNS Geometry AS $$
-  import cartodb_services
-  cartodb_services.init(plpy, GD)
+  from cartodb_services.tools import ServiceManager
   from cartodb_services.mapzen import MapzenGeocoder
   from cartodb_services.mapzen.types import country_to_iso3
-  from cartodb_services.metrics import QuotaService
-  from cartodb_services.tools import Logger
-  from cartodb_services.refactor.tools.logger import LoggerConfigBuilder
   from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
-  from cartodb_services.refactor.core.environment import ServerEnvironmentBuilder
-  from cartodb_services.refactor.backend.server_config import ServerConfigBackendFactory
-  from cartodb_services.refactor.backend.user_config import UserConfigBackendFactory
-  from cartodb_services.refactor.backend.org_config import OrgConfigBackendFactory
-  from cartodb_services.refactor.backend.redis_metrics_connection import RedisMetricsConnectionFactory
 
-  server_config_backend = ServerConfigBackendFactory().get()
-  environment = ServerEnvironmentBuilder(server_config_backend).get()
-  user_config_backend = UserConfigBackendFactory(username, environment, server_config_backend).get()
-  org_config_backend = OrgConfigBackendFactory(orgname, environment, server_config_backend).get()
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
 
-  logger_config = LoggerConfigBuilder(environment, server_config_backend).get()
-  logger = Logger(logger_config)
-
-  mapzen_geocoder_config = MapzenGeocoderConfigBuilder(server_config_backend, user_config_backend, org_config_backend, username, orgname).get()
-
-  redis_metrics_connection = RedisMetricsConnectionFactory(environment, server_config_backend).get()
-
-  quota_service = QuotaService(mapzen_geocoder_config, redis_metrics_connection)
-  if not quota_service.check_user_quota():
-    raise Exception('You have reached the limit of your quota')
+  service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
+  service_manager.assert_within_limits()
 
   try:
-    geocoder = MapzenGeocoder(mapzen_geocoder_config.mapzen_api_key, logger, mapzen_geocoder_config.service_params)
+    geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
     country_iso3 = None
     if country:
       country_iso3 = country_to_iso3(country)
@@ -182,18 +150,18 @@ RETURNS Geometry AS $$
                                    state_province=state_province,
                                    country=country_iso3, search_type='address')
     if coordinates:
-      quota_service.increment_success_service_use()
+      service_manager.quota_service.increment_success_service_use()
       plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
       point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
       return point['st_setsrid']
     else:
-      quota_service.increment_empty_service_use()
+      service_manager.quota_service.increment_empty_service_use()
       return None
   except BaseException as e:
     import sys
-    quota_service.increment_failed_service_use()
-    logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
+    service_manager.quota_service.increment_failed_service_use()
+    service_manager.logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
     raise Exception('Error trying to geocode street point using mapzen')
   finally:
-    quota_service.increment_total_service_use()
+    service_manager.quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;

server/extension/sql/210_rates.sql

@@ -0,0 +1,91 @@
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_get_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT)
+RETURNS JSON AS $$
+  import json
+  from cartodb_services.config import ServiceConfiguration, RateLimitsConfigBuilder
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  service_config = ServiceConfiguration(service, username, orgname)
+  rate_limit_config = RateLimitsConfigBuilder(service_config.server, service_config.user, service_config.org, service=service, username=username, orgname=orgname).get()
+  if rate_limit_config.is_limited():
+      return json.dumps({'limit': rate_limit_config.limit, 'period': rate_limit_config.period})
+  else:
+      return None
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_user_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT,
+  rate_limit_json JSON)
+RETURNS VOID AS $$
+  import json
+  from cartodb_services.config import RateLimitsConfig, RateLimitsConfigSetter
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  config_setter = RateLimitsConfigSetter(service=service, username=username, orgname=orgname)
+  if rate_limit_json:
+      rate_limit = json.loads(rate_limit_json)
+      limit = rate_limit.get('limit', None)
+      period = rate_limit.get('period', None)
+  else:
+      limit = None
+      period = None
+  config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
+  config_setter.set_user_rate_limits(config)
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_org_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT,
+  rate_limit_json JSON)
+RETURNS VOID AS $$
+  import json
+  from cartodb_services.config import RateLimitsConfig, RateLimitsConfigSetter
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  config_setter = RateLimitsConfigSetter(service=service, username=username, orgname=orgname)
+  if rate_limit_json:
+      rate_limit = json.loads(rate_limit_json)
+      limit = rate_limit.get('limit', None)
+      period = rate_limit.get('period', None)
+  else:
+      limit = None
+      period = None
+  config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
+  config_setter.set_org_rate_limits(config)
+$$ LANGUAGE plpythonu;
+
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_server_rate_limit(
+  username TEXT,
+  orgname TEXT,
+  service TEXT,
+  rate_limit_json JSON)
+RETURNS VOID AS $$
+  import json
+  from cartodb_services.config import RateLimitsConfig, RateLimitsConfigSetter
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  config_setter = RateLimitsConfigSetter(service=service, username=username, orgname=orgname)
+  if rate_limit_json:
+      rate_limit = json.loads(rate_limit_json)
+      limit = rate_limit.get('limit', None)
+      period = rate_limit.get('period', None)
+  else:
+      limit = None
+      period = None
+  config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
+  config_setter.set_server_rate_limits(config)
+$$ LANGUAGE plpythonu;

server/extension/sql/50_namedplaces.sql

@@ -1,10 +1,11 @@
 ---- cdb_geocode_namedplace_point(city_name text)
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text)
 RETURNS Geometry AS $$
+  import spiexceptions
   try:
     mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
     return plpy.execute(mapzen_plan, [username, orgname, city_name])[0]['point']
-  except BaseException as e:
+  except spiexceptions.ExternalRoutineException as e:
     internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
     return plpy.execute(internal_plan, [username, orgname, city_name])[0]['point']
 $$ LANGUAGE plpythonu;
@@ -12,10 +13,11 @@ $$ LANGUAGE plpythonu;
 ---- cdb_geocode_namedplace_point(city_name text, country_name text)
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, country_name text)
 RETURNS Geometry AS $$
+  import spiexceptions
   try:
     mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
     return plpy.execute(mapzen_plan, [username, orgname, city_name, country_name])[0]['point']
-  except BaseException as e:
+  except spiexceptions.ExternalRoutineException as e:
     internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
     return plpy.execute(internal_plan, [username, orgname, city_name, country_name])[0]['point']
 $$ LANGUAGE plpythonu;
@@ -23,10 +25,11 @@ $$ LANGUAGE plpythonu;
 ---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, admin1_name text, country_name text)
 RETURNS Geometry AS $$
+  import spiexceptions
   try:
     mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
     return plpy.execute(mapzen_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
-  except BaseException as e:
+  except spiexceptions.ExternalRoutineException as e:
     internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
     return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
 $$ LANGUAGE plpythonu;

server/extension/sql/60_postalcodes.sql

@@ -34,6 +34,12 @@ RETURNS Geometry AS $$
       quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_point(username, orgname, code::text);
+$$ LANGUAGE SQL;
+
+
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code text, country text)
 RETURNS Geometry AS $$
   from cartodb_services.metrics import metrics
@@ -70,6 +76,12 @@ RETURNS Geometry AS $$
       quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision, country text)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_point(username, orgname, code::text, country);
+$$ LANGUAGE SQL;
+
+
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code text)
 RETURNS Geometry AS $$
   from cartodb_services.metrics import metrics
@@ -106,6 +118,13 @@ RETURNS Geometry AS $$
       quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon(username, orgname, code::text)
+$$ LANGUAGE SQL;
+
+
+
 CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code text, country text)
 RETURNS Geometry AS $$
   from cartodb_services.metrics import metrics
@@ -142,6 +161,11 @@ RETURNS Geometry AS $$
       quota_service.increment_total_service_use()
 $$ LANGUAGE plpythonu;
 
+CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision, country text)
+RETURNS Geometry AS $$
+  SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon(username, orgname, code::text, country);
+$$ LANGUAGE SQL;
+
 --------------------------------------------------------------------------------
 
 -- Implementation of the server extension

server/extension/test/expected/00_install_test.out

@@ -37,7 +37,7 @@ SELECT cartodb.cdb_conf_setconf('logger_conf', '{"geocoder_log_path": "/dev/null
  
 (1 row)
 
-SELECT cartodb.cdb_conf_setconf('data_observatory_conf', '{"connection": {"whitelist": ["ethervoid"], "production": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api", "staging": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api"}, "monthly_quota": 100000}');
+SELECT cartodb.cdb_conf_setconf('data_observatory_conf', '{"connection": {"whitelist": ["ethervoid"], "production": "host=localhost port=5432 dbname=contrib_regression user=geocoder_api", "staging": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api"}, "monthly_quota": 100000}');
  cdb_conf_setconf 
 ------------------
  

server/extension/test/expected/100_data_observatory_test.out

@@ -273,3 +273,14 @@ SELECT exists(SELECT *
  t
 (1 row)
 
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_metadatavalidation'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text, json, integer');
+ exists 
+--------
+ t
+(1 row)
+

server/extension/test/expected/210_rates_test.out

@@ -0,0 +1,44 @@
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_get_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_set_user_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text, json');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_set_org_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text, json');
+ exists 
+--------
+ t
+(1 row)
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_set_server_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text, json');
+ exists 
+--------
+ t
+(1 row)
+

server/extension/test/expected/366_empty_table_test.out

@@ -0,0 +1,9 @@
+\set ECHO none
+ id | data 
+----+------
+(0 rows)
+
+ id | data 
+----+------
+(0 rows)
+

server/extension/test/sql/00_install_test.sql

@@ -14,7 +14,7 @@ SELECT cartodb.cdb_conf_setconf('redis_metadata_config', '{"redis_host": "localh
 SELECT cartodb.cdb_conf_setconf('heremaps_conf', '{"geocoder": {"app_id": "dummy_id", "app_code": "dummy_code", "geocoder_cost_per_hit": 1}, "isolines": {"app_id": "dummy_id", "app_code": "dummy_code"}}');
 SELECT cartodb.cdb_conf_setconf('mapzen_conf', '{"routing": {"api_key": "routing_dummy_api_key", "monthly_quota": 1500000}, "geocoder": {"api_key": "geocoder_dummy_api_key", "monthly_quota": 1500000}, "matrix": {"api_key": "matrix_dummy_api_key", "monthly_quota": 1500000}}');
 SELECT cartodb.cdb_conf_setconf('logger_conf', '{"geocoder_log_path": "/dev/null"}');
-SELECT cartodb.cdb_conf_setconf('data_observatory_conf', '{"connection": {"whitelist": ["ethervoid"], "production": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api", "staging": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api"}, "monthly_quota": 100000}');
+SELECT cartodb.cdb_conf_setconf('data_observatory_conf', '{"connection": {"whitelist": ["ethervoid"], "production": "host=localhost port=5432 dbname=contrib_regression user=geocoder_api", "staging": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api"}, "monthly_quota": 100000}');
 
 -- Mock the varnish invalidation function
 -- (used by cdb_geocoder tests)

server/extension/test/sql/100_data_observatory_test.sql

@@ -172,3 +172,10 @@ SELECT exists(SELECT *
               WHERE ns.nspname = 'cdb_dataservices_server'
               AND proname = 'obs_legacybuildermetadata'
               AND oidvectortypes(p.proargtypes)  = 'text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'obs_metadatavalidation'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, geometry, text, json, integer');

server/extension/test/sql/210_rates_test.sql

@@ -0,0 +1,27 @@
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_get_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_set_user_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text, json');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_set_org_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text, json');
+
+SELECT exists(SELECT *
+              FROM pg_proc p
+              INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
+              WHERE ns.nspname = 'cdb_dataservices_server'
+              AND proname = 'cdb_service_set_server_rate_limit'
+              AND oidvectortypes(p.proargtypes)  = 'text, text, text, json');

server/extension/test/sql/366_empty_table_test.sql

@@ -0,0 +1,65 @@
+\set ECHO none
+\set VERBOSITY verbose
+SET client_min_messages TO error;
+
+-- Set configuration for a user 'foo'
+DO $$
+  import json
+  from cartodb_services.config import ServiceConfiguration
+
+  import cartodb_services
+  cartodb_services.init(plpy, GD)
+
+  service_config = ServiceConfiguration('observatory', 'foo', None)
+  service_config.user.set('soft_obs_general_limit', True)
+  service_config.user.set('period_end_date', '20170516')
+
+$$ LANGUAGE plpythonu;
+
+
+-- Mock Observatory backend function
+CREATE SCHEMA cdb_observatory;
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetData(geomvals geomval[], params JSON, merge BOOLEAN DEFAULT TRUE)
+RETURNS TABLE (
+  id INT,
+  data JSON
+) AS $$
+BEGIN
+  -- this will return an empty set
+  RAISE NOTICE 'Mocked OBS_GetData()';
+END;
+$$ LANGUAGE plpgsql;
+GRANT USAGE ON SCHEMA cdb_observatory TO geocoder_api;
+GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO geocoder_api;
+
+
+-- Test it
+SELECT * FROM cdb_dataservices_server.OBS_GetData(
+    'foo',
+    NULL,
+    '{"(0103000020E61000000100000005000000010000E0F67F52C096D88AE6B25F4440010000E0238052C0BF6D8A1A8D5D4440010000D0DA7E52C05F03F3CC265D444001000020F47E52C0F2DD78AB5D5F4440010000E0F67F52C096D88AE6B25F4440,1)"}'::public._geomval,
+    '[{"id": 1, "score": 52.7515548093083898758340051256007949661290516400338, "geom_id": "us.census.tiger.census_tract", "denom_id": "us.census.acs.B01003001", "numer_id": "us.census.acs.B03002003", "geom_name": "US Census Tracts", "geom_type": "Geometry", "num_geoms": 2.86483076549783307739486952736, "denom_name": "Total Population", "denom_type": "Numeric", "numer_name": "White Population", "numer_type": "Numeric", "score_rank": 1, "target_area": 0.000307374806576033, "geom_colname": "the_geom", "score_rownum": 1, "target_geoms": null, "denom_colname": "total_pop", "denom_reltype": '
+         '"denominator", "geom_timespan": "2015", "normalization": "prenormalized", "numer_colname": "white_pop", "timespan_rank": 1, "geom_tablename": "obs_87a814e485deabe3b12545a537f693d16ca702c2", "max_score_rank": null, "numer_timespan": "2010 - 2014", "suggested_name": "white_pop_2010_2014", "denom_aggregate": "sum", "denom_tablename": "obs_b393b5b88c6adda634b2071a8005b03c551b609a", "numer_aggregate": "sum", "numer_tablename": "obs_b393b5b88c6adda634b2071a8005b03c551b609a", "timespan_rownum": 1, "geom_description": "Census tracts are small, relatively permanent statistical subdivisions of a county or equivalent entity that are updated by local participants prior to each decennial census as part of the Census Bureau’s Participant Statistical Areas Program. The Census Bureau delineates census tracts in situations where no local participant existed or where state, local, or tribal governments'
+         'declined to participate. The primary purpose of census tracts is to provide a stable set of geographic units for the presentation of statistical data.\r\n\r\nCensus tracts generally have a population size between 1,200 and 8,000 people, with an optimum size of 4,000 people. A census tract usually covers a contiguous area; however, the spatial size of census tracts varies widely depending on the density of settlement. Census tract boundaries are delineated with the intention of being maintained over a long time so that statistical comparisons can be made from census to census. Census tracts occasionally are split due to population growth or merged as a result of substantial population decline.\r\n\r\nCensus tract boundaries generally follow visible and identifiable features. They may follow nonvisible legal boundaries, such as minor civil division (MCD) or incorporated place boundaries'
+         'in some states and situations, to allow for census-tract-to-governmental-unit relationships where the governmental boundaries tend to remain unchanged between censuses. State and county boundaries always are census tract boundaries in the standard census geographic hierarchy. Tribal census tracts are a unique geographic entity defined within federally recognized American Indian reservations and off-reservation trust lands and can cross state and county boundaries. Tribal census tracts may be completely different from the census tracts and block groups defined by state and county (see “Tribal Census Tract”).", "denom_description": "The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.", "max_timespan_rank": null, "numer_description": "The number of people identifying as white, non-Hispanic in each'
+         'geography.", "geom_t_description": null, "denom_t_description": null, "numer_t_description": null, "geom_geomref_colname": "geoid", "denom_geomref_colname": "geoid", "numer_geomref_colname": "geoid"}]'::json,
+     true);
+
+
+
+-- Mock another observatory backend function (overloaded, different params)
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetData(geomrefs TEXT[], params JSON)
+RETURNS TABLE (
+  id INT,
+  data JSON
+) AS $$
+BEGIN
+  -- this will return an empty set
+  RAISE NOTICE 'Mocked OBS_GetData()';
+END;
+$$ LANGUAGE plpgsql;
+GRANT USAGE ON SCHEMA cdb_observatory TO geocoder_api;
+GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO geocoder_api;
+
+-- Test it
+SELECT * FROM cdb_dataservices_server.OBS_GetData('foo', NULL, '{bar, baz}'::TEXT[], '[]'::JSON);

server/lib/python/cartodb_services/README.md

@@ -7,12 +7,7 @@ It is used from pl/python functions contained in the `cdb_dataservices_server` e
 On the other hand, it is pretty independent from the client, as long as the signatures of the public pl/python functions match.
 
 ## Dependencies
-See the [[`requirements.txt`]] or better the Basically:
-- pip
-- redis and hiredis
-- dateutil
-- googlemaps
-- request
+Take a look at [`requirements.txt`](requirements.txt) for details about the required dependencies.
 
 ## Installation
 Install the requirements:
@@ -40,7 +35,7 @@ OK
 ```
 
 ## Running the integration tests
-See the [[../../../../test/README.md]]. Basically, move to the `/test` directory at the top level of this repo and execute the `run_tests.py` script:
+See this [`README`](../../../../test/README.md) in the `/test` directory for details. Basically, you have to move to the `/test` directory at the top level of this repo and execute the `run_tests.py` script:
 ```sh
 cd $(git rev-parse --show-toplevel)/test
 python run_tests.py --host=$YOUR_HOST $YOUR_USERNAME $YOUR_API_KEY

server/lib/python/cartodb_services/cartodb_services/config/__init__.py

@@ -0,0 +1,3 @@
+from service_configuration import ServiceConfiguration
+from rate_limits import RateLimitsConfig, RateLimitsConfigBuilder, RateLimitsConfigSetter
+from legacy_rate_limits import RateLimitsConfigLegacyBuilder

server/lib/python/cartodb_services/cartodb_services/config/legacy_rate_limits.py

@@ -0,0 +1,46 @@
+import json
+from rate_limits import RateLimitsConfig
+
+class RateLimitsConfigLegacyBuilder(object):
+    """
+    Build a RateLimitsConfig object using the *legacy* configuration classes
+    """
+
+    def __init__(self, redis_connection, db_conn, service, username, orgname):
+        self._service = service
+        self._username = username
+        self._orgname = orgname
+        self._redis_connection = redis_connection
+        self._db_conn = db_conn
+
+    def get(self):
+        rate_limit = self.__get_rate_limit()
+        return RateLimitsConfig(self._service,
+                                self._username,
+                                rate_limit.get('limit', None),
+                                rate_limit.get('period', None))
+
+    def __get_rate_limit(self):
+        rate_limit = {}
+        rate_limit_key = "{0}_rate_limit".format(self._service)
+        user_key = "rails:users:{0}".format(self._username)
+        rate_limit_json = self.__get_redis_config(user_key, rate_limit_key)
+        if not rate_limit_json and self._orgname:
+            org_key = "rails:orgs:{0}".format(self._orgname)
+            rate_limit_json = self.__get_redis_config(org_key, rate_limit_key)
+        if rate_limit_json:
+            rate_limit = json.loads(rate_limit_json)
+        else:
+            conf_key = 'rate_limits'
+            sql = "SELECT cdb_dataservices_server.CDB_Conf_GetConf('{0}') as conf".format(conf_key)
+            try:
+                conf = self._db_conn.execute(sql, 1)[0]['conf']
+            except Exception:
+                conf = None
+            if conf:
+                rate_limit = json.loads(conf).get(self._service)
+        return rate_limit or {}
+
+    def __get_redis_config(self, basekey, param):
+        config = self._redis_connection.hgetall(basekey)
+        return config and config.get(param)

server/lib/python/cartodb_services/cartodb_services/config/rate_limits.py

@@ -0,0 +1,113 @@
+import json
+
+from service_configuration import ServiceConfiguration
+
+class RateLimitsConfig(object):
+    """
+    Value object that represents the configuration needed to rate-limit services
+    """
+
+    def __init__(self,
+                 service,
+                 username,
+                 limit,
+                 period):
+        self._service = service
+        self._username = username
+        self._limit = limit and int(limit)
+        self._period = period and int(period)
+
+    def __eq__(self, other):
+        return self.__dict__ == other.__dict__
+
+    # service this limit applies to
+    @property
+    def service(self):
+        return self._service
+
+    # user this limit applies to
+    @property
+    def username(self):
+        return self._username
+
+    # rate period in seconds
+    @property
+    def period(self):
+        return self._period
+
+    # rate limit in seconds
+    @property
+    def limit(self):
+        return self._limit
+
+    def is_limited(self):
+        return self._limit and self._limit > 0 and self._period and self._period > 0
+
+class RateLimitsConfigBuilder(object):
+    """
+    Build a rate limits configuration obtaining the parameters
+    from the user/org/server configuration.
+    """
+
+    def __init__(self, server_conf, user_conf, org_conf, service, username, orgname):
+        self._server_conf = server_conf
+        self._user_conf = user_conf
+        self._org_conf = org_conf
+        self._service = service
+        self._username = username
+        self._orgname = orgname
+
+    def get(self):
+        # Order of precedence is user_conf, org_conf, server_conf
+
+        rate_limit_key = "{0}_rate_limit".format(self._service)
+
+        rate_limit_json = self._user_conf.get(rate_limit_key, None) or self._org_conf.get(rate_limit_key, None)
+        if (rate_limit_json):
+            rate_limit = rate_limit_json and json.loads(rate_limit_json)
+        else:
+            rate_limit = self._server_conf.get('rate_limits', {}).get(self._service, {})
+
+        return RateLimitsConfig(self._service,
+                                self._username,
+                                rate_limit.get('limit', None),
+                                rate_limit.get('period', None))
+
+
+class RateLimitsConfigSetter(object):
+
+    def __init__(self, service, username, orgname=None):
+        self._service = service
+        self._service_config = ServiceConfiguration(service, username, orgname)
+
+    def set_user_rate_limits(self, rate_limits_config):
+        # Note we allow copying a config from another user/service, so we
+        # ignore rate_limits:config.service and rate_limits:config.username
+        rate_limit_key = "{0}_rate_limit".format(self._service)
+        if rate_limits_config.is_limited():
+            rate_limit = {'limit': rate_limits_config.limit, 'period': rate_limits_config.period}
+            rate_limit_json = json.dumps(rate_limit)
+            self._service_config.user.set(rate_limit_key, rate_limit_json)
+        else:
+            self._service_config.user.remove(rate_limit_key)
+
+    def set_org_rate_limits(self, rate_limits_config):
+        rate_limit_key = "{0}_rate_limit".format(self._service)
+        if rate_limits_config.is_limited():
+            rate_limit = {'limit': rate_limits_config.limit, 'period': rate_limits_config.period}
+            rate_limit_json = json.dumps(rate_limit)
+            self._service_config.org.set(rate_limit_key, rate_limit_json)
+        else:
+            self._service_config.org.remove(rate_limit_key)
+
+    def set_server_rate_limits(self, rate_limits_config):
+        rate_limits = self._service_config.server.get('rate_limits', {})
+        if rate_limits_config.is_limited():
+            rate_limits[self._service] = {'limit': rate_limits_config.limit, 'period': rate_limits_config.period}
+        else:
+            rate_limits.pop(self._service, None)
+        if rate_limits:
+            self._service_config.server.set('rate_limits', rate_limits)
+        else:
+            self._service_config.server.remove('rate_limits')
+