diff --git a/README.md b/README.md index 0159356..809cd71 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ -# geocoder-api -The CartoDB Geocoder SQL API (server and client FTM) +# Data Services API +The CartoDB Data Services SQL API (server and client FTM) ### Deploy instructions -Steps to deploy a new Geocoder API version : +Steps to deploy a new Data Services API version : - Deploy new version of dataservices API to all servers - Update the server user using: ALTER EXTENSION cdb_dataservices_server UPDATE TO ''; @@ -20,3 +20,58 @@ Steps to deploy a new Geocoder API version : - `RAILS_ENV=production bundle exec rake cartodb:db:configure_geocoder_extension_for_organizations['', true]` - `RAILS_ENV=production bundle exec rake cartodb:db:configure_geocoder_extension_for_non_org_users['', true]` - Freeze the generated SQL file for the version. Eg. cdb_dataservices_server--0.0.1.sql + +### Local install instructions + +- install data services extension + + ``` + git clone git@github.com:CartoDB/data-services.git + data-services/geocoder/extension + sudo make install + ``` + +- install server and client extensions + + ``` + cd client && sudo make install + cd server && sudo make install + ``` + +- install python library + + ``` + cd server/lib/python/cartodb_services && python setup.py install + ``` + +- install extensions in user database + + ``` + create extension cdb_geocoder; + create extension plproxy; + create extension cdb_dataservices_server; + create extension cdb_dataservices_client; + ``` + +- add configuration for different services in user database + + + ``` + # select CDB_Conf_SetConf('redis_metadata_config', '{"sentinel_host": "localhost", "sentinel_port": 26379, "sentinel_master_id": "mymaster", "timeout": 0.1, "redis_db": 5}'); + # select CDB_Conf_SetConf('redis_metrics_config', '{"sentinel_host": "localhost", "sentinel_port": 26379, "sentinel_master_id": "mymaster", "timeout": 0.1, "redis_db": 5}'); + + # select CDB_Conf_SetConf('heremaps_conf', '{"app_id": "APP_ID", "app_code": "APP_CODE"}'); + # select CDB_Conf_SetConf('user_config', '{"is_organization": false, "entity_name": ""}') + ``` + +- configure plproxy to point to the same user database (you could do in a different one) + + ``` + select CDB_Conf_SetConf('geocoder_server_config', '{ "connection_str": "host=localhost port=5432 dbname=cartodb_dev_user_accf0647-d942-4e37-b129-8287c117e687_db user=postgres"}'); + ``` + +- configure the search path in order to be able to execute the functions without use the schema: + + ``` + alter role "" set search_path='"$user", public, cartodb, cdb_dataservices_client'; + ``` diff --git a/doc/internal/internal_doc.md b/doc/internal/internal_doc.md new file mode 100644 index 0000000..aaef516 --- /dev/null +++ b/doc/internal/internal_doc.md @@ -0,0 +1,69 @@ +# Data Services API Internal documentation + +* [Existent services](#existent-services) +* [How to add a new service](#how-to-add-a-new-service) + +## Existent services + +Available at [cartodb_services](https://github.com/CartoDB/geocoder-api/tree/master/server/lib/python/cartodb_services/cartodb_services). + +* Google + * [Geocoding](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/google/geocoder.py) + +* Here + * [Geocoding](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/here/geocoder.py) + * [Routing](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/here/routing.py) + +## How to add a new service + +These are the steps that need to be followed when creating a new service in the API or updating an existent one. + +### Creating a new service function or editing an existent one +In this scenario, both client and server sides require to be edited/created. + +* **Update the interface file** with the function addition or update + * Interfaces are stored in `client/renderer/interfaces` + * Interface YAML filenames follow the client versioning schema with the format `interface-x.y.z.yaml`. + +* Update the renderer templates or script, *if applicable* + * Renderer templates are stored in `client/renderer/templates` + * The Renderer script (`client/renderer/sql-template-renderer`) generates SQL from the defined interfaces + +* Generate a **new subfolder version** for `sql` and `test` folders to define the new functions and tests + * TODO: Use symlinks to avoid file duplication between versions that don't update them + * The `client/sql` folder contents are generated from the interfaces in the first step + * Add or upgrade your SQL server functions + * Create tests for the client and server functions -- at least, to check that those are created + +* Generate the **upgrade and downgrade files** for the extension for both client and server + +* Update the control files and the Makefiles to generate the complete SQL file for the new created version + * These new version files (`cdb_dataservices_client--X.Y.Z.sql and cdb_dataservices_server--X.Y.X.sql`) must be pushed and frozen. You can add these to the `.gitignore` file. + +* Update the public docs! ;-) + +### Updating an existing server side function + +With no changes in client side. + +#### Extension + +* Generate a **new subfolder version** for `sql` and `test` folders to define the new functions and tests + * **TODO:** Use symlinks to avoid file duplication between versions that don't update them + * **Add or upgrade your SQL server functions** + * For example, if a new street geocoder service is implemented, it will require a change in the main function (`cdb_dataservices_server.cdb_geocode_street_point`) and generate a new `cdb_dataservices_server._cdb_newservice_geocode_street_point` + +* Generate the upgrade and downgrade files for the extension for the client + +#### Python + +* Add, if needed, [new configuration elements](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/config.py#L100) + +* Add the new **functionality** into the provider folder. If the provider is new, create a new folder(`server/lib/python/cartodb_services/cartodb_services/{provider_name}` and add the service (`geocoder.py`) + +* Check the `__init__.py` files to follow the existent [import conventions](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py) + +* Add a **new metric**, if needed, at the [corresponding service](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py#L37-L60) + +* **Update the package version** in [setup.py](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/setup.py) + diff --git a/server/extension/cdb_dataservices_server--0.3.0--0.4.0.sql b/server/extension/cdb_dataservices_server--0.3.0--0.4.0.sql index 7af507e..221898d 100644 --- a/server/extension/cdb_dataservices_server--0.3.0--0.4.0.sql +++ b/server/extension/cdb_dataservices_server--0.3.0--0.4.0.sql @@ -1,12 +1,12 @@ -- Get the Redis configuration from the _conf table -- -CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text) +CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text) RETURNS boolean AS $$ - cache_key = "user_routing_config_{0}".format(username) + cache_key = "user_isolines_routing_config_{0}".format(username) if cache_key in GD: return False else: import json - from cartodb_services.metrics import RoutingConfig + from cartodb_services.metrics import IsolinesRoutingConfig plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection'] heremaps_conf_json = plpy.execute("SELECT cartodb.CDB_Conf_GetConf('heremaps_conf') as heremaps_conf", 1)[0]['heremaps_conf'] @@ -17,7 +17,7 @@ RETURNS boolean AS $$ heremaps_conf = json.loads(heremaps_conf_json) heremaps_app_id = heremaps_conf['app_id'] heremaps_app_code = heremaps_conf['app_code'] - routing_config = RoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code) + routing_config = IsolinesRoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code) # --Think about the security concerns with this kind of global cache, it should be only available # --for this user session but... GD[cache_key] = routing_config @@ -34,12 +34,15 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$ from cartodb_services.here.types import geo_polyline_to_multipolygon redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - user_routing_config = GD["user_routing_config_{0}".format(username)] + user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)] - quota_service = QuotaService(user_routing_config, redis_conn) + # -- Check the quota + quota_service = QuotaService(user_isolines_routing_config, redis_conn) + if not quota_service.check_user_quota(): + plpy.error('You have reach the limit of your quota') try: - client = HereMapsRoutingIsoline(user_routing_config.heremaps_app_id, user_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL) + client = HereMapsRoutingIsoline(user_isolines_routing_config.heremaps_app_id, user_isolines_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL) if source: lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat'] @@ -80,8 +83,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT RETURNS SETOF cdb_dataservices_server.isoline AS $$ plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) - user_isolines_config = GD["user_routing_config_{0}".format(username)] + plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) + user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)] type = 'isodistance' here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"]) @@ -99,8 +102,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT, RETURNS SETOF cdb_dataservices_server.isoline AS $$ plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) - user_isolines_config = GD["user_routing_config_{0}".format(username)] + plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) + user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)] type = 'isochrone' here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"]) diff --git a/server/extension/cdb_dataservices_server--0.4.0--0.3.0.sql b/server/extension/cdb_dataservices_server--0.4.0--0.3.0.sql index bb764a1..a67074f 100644 --- a/server/extension/cdb_dataservices_server--0.4.0--0.3.0.sql +++ b/server/extension/cdb_dataservices_server--0.4.0--0.3.0.sql @@ -1,5 +1,5 @@ DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_isochrone(TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]); DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_isodistance(TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]); DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_here_routing_isolines(TEXT, TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]); -DROP FUNCTION IF EXISTS cdb_dataservices_server._get_routing_config(text, text); +DROP FUNCTION IF EXISTS cdb_dataservices_server._get_isolines_routing_config(text, text); DROP TYPE IF EXISTS cdb_dataservices_server.isoline; \ No newline at end of file diff --git a/server/extension/cdb_dataservices_server--0.4.0.sql b/server/extension/cdb_dataservices_server--0.4.0.sql index df953c3..85cffae 100644 --- a/server/extension/cdb_dataservices_server--0.4.0.sql +++ b/server/extension/cdb_dataservices_server--0.4.0.sql @@ -85,14 +85,14 @@ RETURNS boolean AS $$ $$ LANGUAGE plpythonu SECURITY DEFINER; -- Get the Redis configuration from the _conf table -- -CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text) +CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text) RETURNS boolean AS $$ - cache_key = "user_routing_config_{0}".format(username) + cache_key = "user_isolines_routing_config_{0}".format(username) if cache_key in GD: return False else: import json - from cartodb_services.metrics import RoutingConfig + from cartodb_services.metrics import IsolinesRoutingConfig plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection'] heremaps_conf_json = plpy.execute("SELECT cartodb.CDB_Conf_GetConf('heremaps_conf') as heremaps_conf", 1)[0]['heremaps_conf'] @@ -103,10 +103,10 @@ RETURNS boolean AS $$ heremaps_conf = json.loads(heremaps_conf_json) heremaps_app_id = heremaps_conf['app_id'] heremaps_app_code = heremaps_conf['app_code'] - routing_config = RoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code) + isolines_routing_config = IsolinesRoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code) # --Think about the security concerns with this kind of global cache, it should be only available # --for this user session but... - GD[cache_key] = routing_config + GD[cache_key] = isolines_routing_config return True $$ LANGUAGE plpythonu SECURITY DEFINER; -- Geocodes a street address given a searchtext and a state and/or country @@ -814,12 +814,15 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$ from cartodb_services.here.types import geo_polyline_to_multipolygon redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - user_routing_config = GD["user_routing_config_{0}".format(username)] + user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)] - quota_service = QuotaService(user_routing_config, redis_conn) + # -- Check the quota + quota_service = QuotaService(user_isolines_routing_config, redis_conn) + if not quota_service.check_user_quota(): + plpy.error('You have reach the limit of your quota') try: - client = HereMapsRoutingIsoline(user_routing_config.heremaps_app_id, user_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.STAGING_ROUTING_BASE_URL ) + client = HereMapsRoutingIsoline(user_isolines_routing_config.heremaps_app_id, user_isolines_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL) if source: lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat'] @@ -859,8 +862,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT RETURNS SETOF cdb_dataservices_server.isoline AS $$ plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) - user_isolines_config = GD["user_routing_config_{0}".format(username)] + plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) + user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)] type = 'isodistance' here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"]) @@ -877,8 +880,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT, RETURNS SETOF cdb_dataservices_server.isoline AS $$ plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) - user_isolines_config = GD["user_routing_config_{0}".format(username)] + plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) + user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)] type = 'isochrone' here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"]) diff --git a/server/extension/sql/0.4.0/15_config_helper.sql b/server/extension/sql/0.4.0/15_config_helper.sql index 47aad42..c49ef50 100644 --- a/server/extension/sql/0.4.0/15_config_helper.sql +++ b/server/extension/sql/0.4.0/15_config_helper.sql @@ -25,14 +25,14 @@ RETURNS boolean AS $$ $$ LANGUAGE plpythonu SECURITY DEFINER; -- Get the Redis configuration from the _conf table -- -CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text) +CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text) RETURNS boolean AS $$ - cache_key = "user_routing_config_{0}".format(username) + cache_key = "user_isolines_routing_config_{0}".format(username) if cache_key in GD: return False else: import json - from cartodb_services.metrics import RoutingConfig + from cartodb_services.metrics import IsolinesRoutingConfig plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection'] heremaps_conf_json = plpy.execute("SELECT cartodb.CDB_Conf_GetConf('heremaps_conf') as heremaps_conf", 1)[0]['heremaps_conf'] @@ -43,9 +43,9 @@ RETURNS boolean AS $$ heremaps_conf = json.loads(heremaps_conf_json) heremaps_app_id = heremaps_conf['app_id'] heremaps_app_code = heremaps_conf['app_code'] - routing_config = RoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code) + isolines_routing_config = IsolinesRoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code) # --Think about the security concerns with this kind of global cache, it should be only available # --for this user session but... - GD[cache_key] = routing_config + GD[cache_key] = isolines_routing_config return True $$ LANGUAGE plpythonu SECURITY DEFINER; diff --git a/server/extension/sql/0.4.0/80_routing_helper.sql b/server/extension/sql/0.4.0/80_isolines_helper.sql similarity index 80% rename from server/extension/sql/0.4.0/80_routing_helper.sql rename to server/extension/sql/0.4.0/80_isolines_helper.sql index 634c2d7..eb15963 100644 --- a/server/extension/sql/0.4.0/80_routing_helper.sql +++ b/server/extension/sql/0.4.0/80_isolines_helper.sql @@ -8,12 +8,15 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$ from cartodb_services.here.types import geo_polyline_to_multipolygon redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - user_routing_config = GD["user_routing_config_{0}".format(username)] + user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)] - quota_service = QuotaService(user_routing_config, redis_conn) + # -- Check the quota + quota_service = QuotaService(user_isolines_routing_config, redis_conn) + if not quota_service.check_user_quota(): + plpy.error('You have reach the limit of your quota') try: - client = HereMapsRoutingIsoline(user_routing_config.heremaps_app_id, user_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL) + client = HereMapsRoutingIsoline(user_isolines_routing_config.heremaps_app_id, user_isolines_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL) if source: lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat'] diff --git a/server/extension/sql/0.4.0/85_isodistance.sql b/server/extension/sql/0.4.0/85_isodistance.sql index b5b553c..c6d99ae 100644 --- a/server/extension/sql/0.4.0/85_isodistance.sql +++ b/server/extension/sql/0.4.0/85_isodistance.sql @@ -2,8 +2,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT RETURNS SETOF cdb_dataservices_server.isoline AS $$ plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) - user_isolines_config = GD["user_routing_config_{0}".format(username)] + plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) + user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)] type = 'isodistance' here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"]) diff --git a/server/extension/sql/0.4.0/90_isochrone.sql b/server/extension/sql/0.4.0/90_isochrone.sql index 31f9d3a..5ca7272 100644 --- a/server/extension/sql/0.4.0/90_isochrone.sql +++ b/server/extension/sql/0.4.0/90_isochrone.sql @@ -2,8 +2,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT, RETURNS SETOF cdb_dataservices_server.isoline AS $$ plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username)) redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection'] - plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) - user_isolines_config = GD["user_routing_config_{0}".format(username)] + plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname))) + user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)] type = 'isochrone' here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"]) diff --git a/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py b/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py index 51147f8..208b2be 100644 --- a/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py +++ b/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py @@ -1,3 +1,3 @@ -from config import GeocoderConfig, RoutingConfig, InternalGeocoderConfig, ConfigException +from config import GeocoderConfig, IsolinesRoutingConfig, InternalGeocoderConfig, ConfigException from quota import QuotaService from user import UserMetricsService diff --git a/server/lib/python/cartodb_services/cartodb_services/metrics/config.py b/server/lib/python/cartodb_services/cartodb_services/metrics/config.py index 2b8c51b..484cbb7 100644 --- a/server/lib/python/cartodb_services/cartodb_services/metrics/config.py +++ b/server/lib/python/cartodb_services/cartodb_services/metrics/config.py @@ -28,18 +28,76 @@ class ServiceConfig(object): return self._orgname -class RoutingConfig(ServiceConfig): +class IsolinesRoutingConfig(ServiceConfig): + + ROUTING_CONFIG_KEYS = ['here_isolines_quota', 'soft_here_isolines_limit', + 'period_end_date', 'username', 'orgname', + 'heremaps_app_id', 'heremaps_app_code'] + NOKIA_APP_ID_KEY = 'heremaps_app_id' + NOKIA_APP_CODE_KEY = 'heremaps_app_code' + QUOTA_KEY = 'here_isolines_quota' + SOFT_LIMIT_KEY = 'soft_here_isolines_limit' + USERNAME_KEY = 'username' + ORGNAME_KEY = 'orgname' + PERIOD_END_DATE = 'period_end_date' def __init__(self, redis_connection, username, orgname=None, heremaps_app_id=None, heremaps_app_code=None): - super(RoutingConfig, self).__init__(redis_connection, - username, orgname) - self._heremaps_app_id = heremaps_app_id - self._heremaps_app_code = heremaps_app_code + super(IsolinesRoutingConfig, self).__init__(redis_connection, username, + orgname) + config = self.__get_user_config(username, orgname, heremaps_app_id, + heremaps_app_code) + filtered_config = {key: config[key] for key in self.ROUTING_CONFIG_KEYS if key in config.keys()} + self.__parse_config(filtered_config) + + def __get_user_config(self, username, orgname=None, heremaps_app_id=None, + heremaps_app_code=None): + user_config = self._redis_connection.hgetall( + "rails:users:{0}".format(username)) + if not user_config: + raise ConfigException("""There is no user config available. Please check your configuration.'""") + else: + user_config[self.NOKIA_APP_ID_KEY] = heremaps_app_id + user_config[self.NOKIA_APP_CODE_KEY] = heremaps_app_code + if orgname: + self.__get_organization_config(orgname, user_config) + + return user_config + + def __get_organization_config(self, orgname, user_config): + org_config = self._redis_connection.hgetall( + "rails:orgs:{0}".format(orgname)) + if not org_config: + raise ConfigException("""There is no organization config available. Please check your configuration.'""") + else: + user_config[self.QUOTA_KEY] = org_config[self.QUOTA_KEY] + user_config[self.PERIOD_END_DATE] = org_config[self.PERIOD_END_DATE] + + def __parse_config(self, filtered_config): + self._isolines_quota = float(filtered_config[self.QUOTA_KEY]) + self._period_end_date = date_parse(filtered_config[self.PERIOD_END_DATE]) + if filtered_config[self.SOFT_LIMIT_KEY].lower() == 'true': + self._soft_isolines_limit = True + else: + self._soft_isolines_limit = False + self._heremaps_app_id = filtered_config[self.NOKIA_APP_ID_KEY] + self._heremaps_app_code = filtered_config[self.NOKIA_APP_CODE_KEY] @property def service_type(self): - return 'routing_here' + return 'here_isolines' + + @property + def isolines_quota(self): + return self._isolines_quota + + @property + def soft_isolines_limit(self): + return self._soft_isolines_limit + + @property + def period_end_date(self): + return self._period_end_date @property def heremaps_app_id(self): diff --git a/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py b/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py index 340642c..aba6135 100644 --- a/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py +++ b/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py @@ -1,32 +1,22 @@ from user import UserMetricsService from datetime import date +import re class QuotaService: """ Class to manage all the quota operation for the Geocoder SQL API Extension """ - def __init__(self, user_geocoder_config, redis_connection): - self._user_geocoder_config = user_geocoder_config + def __init__(self, user_service_config, redis_connection): + self._user_service_config = user_service_config + # TODO First step to extract to a factory if needed in the future + self._quota_checker = QuotaChecker(user_service_config, + redis_connection) self._user_service = UserMetricsService( - self._user_geocoder_config, redis_connection) + self._user_service_config, redis_connection) def check_user_quota(self): - """ Check if the current user quota surpasses the current quota """ - # We don't have quota check for google geocoder - if self._user_geocoder_config.google_geocoder: - return True - - user_quota = self._user_geocoder_config.geocoding_quota - today = date.today() - service_type = self._user_geocoder_config.service_type - current_used = self._user_service.used_quota(service_type, today) - soft_geocoding_limit = self._user_geocoder_config.soft_geocoding_limit - - if soft_geocoding_limit or current_used <= user_quota: - return True - else: - return False + return self._quota_checker.check() # TODO # We are going to change this class to be the generic one and @@ -36,25 +26,72 @@ class QuotaService: def increment_success_geocoder_use(self, amount=1): self._user_service.increment_service_use( - self._user_geocoder_config.service_type, "success_responses", + self._user_service_config.service_type, "success_responses", amount=amount) def increment_empty_geocoder_use(self, amount=1): self._user_service.increment_service_use( - self._user_geocoder_config.service_type, "empty_responses", + self._user_service_config.service_type, "empty_responses", amount=amount) def increment_failed_geocoder_use(self, amount=1): self._user_service.increment_service_use( - self._user_geocoder_config.service_type, "fail_responses", + self._user_service_config.service_type, "fail_responses", amount=amount) def increment_total_geocoder_use(self, amount=1): self._user_service.increment_service_use( - self._user_geocoder_config.service_type, "total_requests", + self._user_service_config.service_type, "total_requests", amount=amount) def increment_isolines_service_use(self, amount=1): self._user_service.increment_service_use( - self._user_geocoder_config.service_type, "isolines_generated", + self._user_service_config.service_type, "isolines_generated", amount=amount) + + +class QuotaChecker: + + def __init__(self, user_service_config, redis_connection): + self._user_service_config = user_service_config + self._user_service = UserMetricsService( + self._user_service_config, redis_connection) + + def check(self): + """ Check if the current user quota surpasses the current quota """ + if re.match('geocoder_*', + self._user_service_config.service_type) is not None: + return self.__check_geocoder_quota() + elif re.match('here_isolines', + self._user_service_config.service_type) is not None: + return self.__check_isolines_quota() + else: + return False + + def __check_geocoder_quota(self): + # We don't have quota check for google geocoder + if self._user_service_config.google_geocoder: + return True + + user_quota = self._user_service_config.geocoding_quota + today = date.today() + service_type = self._user_service_config.service_type + current_used = self._user_service.used_quota(service_type, today) + soft_geocoding_limit = self._user_service_config.soft_geocoding_limit + + if soft_geocoding_limit or (user_quota > 0 and current_used <= user_quota): + return True + else: + return False + + def __check_isolines_quota(self): + user_quota = self._user_service_config.isolines_quota + today = date.today() + service_type = self._user_service_config.service_type + current_used = self._user_service.used_quota(service_type, today) + soft_isolines_limit = self._user_service_config.soft_isolines_limit + + if soft_isolines_limit or (user_quota > 0 and current_used <= user_quota): + return True + else: + return False diff --git a/server/lib/python/cartodb_services/cartodb_services/metrics/user.py b/server/lib/python/cartodb_services/cartodb_services/metrics/user.py index 947f782..782746d 100644 --- a/server/lib/python/cartodb_services/cartodb_services/metrics/user.py +++ b/server/lib/python/cartodb_services/cartodb_services/metrics/user.py @@ -6,16 +6,8 @@ class UserMetricsService: """ Class to manage all the user info """ SERVICE_GEOCODER_NOKIA = 'geocoder_here' - SERVICE_GEOCODER_GOOGLE = 'geocoder_google' SERVICE_GEOCODER_CACHE = 'geocoder_cache' - - GEOCODING_QUOTA_KEY = "geocoding_quota" - GEOCODING_SOFT_LIMIT_KEY = "soft_geocoder_limit" - - REDIS_CONNECTION_KEY = "redis_connection" - REDIS_CONNECTION_HOST = "redis_host" - REDIS_CONNECTION_PORT = "redis_port" - REDIS_CONNECTION_DB = "redis_db" + SERVICE_HERE_ISOLINES = 'here_isolines' def __init__(self, user_geocoder_config, redis_connection): self._user_geocoder_config = user_geocoder_config @@ -24,6 +16,12 @@ class UserMetricsService: self._orgname = user_geocoder_config.organization def used_quota(self, service_type, date): + if service_type == self.SERVICE_HERE_ISOLINES: + return self.__used_isolines_quota(service_type, date) + else: + return self.__used_geocoding_quota(service_type, date) + + def __used_geocoding_quota(self, service_type, date): """ Recover the used quota for the user in the current month """ date_from, date_to = self.__current_billing_cycle() current_use = 0 @@ -42,6 +40,20 @@ class UserMetricsService: return current_use + def __used_isolines_quota(self, service_type, date): + date_from, date_to = self.__current_billing_cycle() + current_use = 0 + isolines_generated = self.get_metrics(service_type, + 'isolines_generated', date_from, + date_to) + empty_responses = self.get_metrics(service_type, + 'empty_responses', date_from, + date_to) + current_use += (isolines_generated + empty_responses) + + return current_use + + def increment_service_use(self, service_type, metric, date=date.today(), amount=1): """ Increment the services uses in monthly and daily basis""" self.__increment_user_uses(service_type, metric, date, amount) diff --git a/test/helpers/integration_test_helper.py b/test/helpers/integration_test_helper.py index 799bf30..537fd31 100644 --- a/test/helpers/integration_test_helper.py +++ b/test/helpers/integration_test_helper.py @@ -29,4 +29,4 @@ class IntegrationTestHelper: raise Exception(json.loads(query_response.text)['error']) query_response_data = json.loads(query_response.text) - return query_response_data['rows'][0]['geometry'] + return query_response_data['rows'][0] diff --git a/test/integration/test_admin0_functions.py b/test/integration/test_admin0_functions.py index ff8912a..8958502 100644 --- a/test/integration/test_admin0_functions.py +++ b/test/integration/test_admin0_functions.py @@ -16,16 +16,15 @@ class TestAdmin0Functions(TestCase): def test_if_select_with_admin0_is_ok(self): query = "SELECT cdb_geocode_admin0_polygon(country) as geometry " \ - "FROM {0} LIMIT 1&api_key={1}".format( - self.env_variables['table_name'], - self.env_variables['api_key']) + "FROM {0} LIMIT 1&api_key={1}".format( + self.env_variables['table_name'], + self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_admin0_without_api_key_raise_error(self): query = "SELECT cdb_geocode_admin0_polygon(country) as geometry " \ - "FROM {0} LIMIT 1".format( - self.env_variables['table_name']) + "FROM {0} LIMIT 1".format(self.env_variables['table_name']) try: IntegrationTestHelper.execute_query(self.sql_api_url, query) except Exception as e: diff --git a/test/integration/test_admin1_functions.py b/test/integration/test_admin1_functions.py index 082cc43..8e09d70 100644 --- a/test/integration/test_admin1_functions.py +++ b/test/integration/test_admin1_functions.py @@ -20,7 +20,7 @@ class TestAdmin1Functions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_admin1_with_country_is_ok(self): query = "SELECT cdb_geocode_admin1_polygon(province,country)" \ @@ -28,7 +28,7 @@ class TestAdmin1Functions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_admin1_without_api_key_raise_error(self): query = "SELECT cdb_geocode_admin1_polygon(province) as geometry " \ diff --git a/test/integration/test_ipaddress_functions.py b/test/integration/test_ipaddress_functions.py index 42c30a0..69056b6 100644 --- a/test/integration/test_ipaddress_functions.py +++ b/test/integration/test_ipaddress_functions.py @@ -20,7 +20,7 @@ class TestPostalcodeFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_ipaddress_without_api_key_raise_error(self): query = "SELECT cdb_geocode_ipaddress_point(ip) " \ diff --git a/test/integration/test_isolines_functions.py b/test/integration/test_isolines_functions.py new file mode 100644 index 0000000..936699c --- /dev/null +++ b/test/integration/test_isolines_functions.py @@ -0,0 +1,45 @@ +from unittest import TestCase +from nose.tools import assert_raises +from nose.tools import assert_not_equal, assert_equal +from ..helpers.integration_test_helper import IntegrationTestHelper + + +class TestIsolinesFunctions(TestCase): + + def setUp(self): + self.env_variables = IntegrationTestHelper.get_environment_variables() + self.sql_api_url = "https://{0}.{1}/api/v2/sql".format( + self.env_variables['username'], + self.env_variables['host'], + self.env_variables['api_key'] + ) + + def test_if_select_with_isochrones_is_ok(self): + query = "SELECT * FROM cdb_isochrone('POINT(-3.70568 40.42028)'::geometry, " \ + "'car', ARRAY[300]::integer[]);&api_key={0}".format( + self.env_variables['api_key']) + isolines = IntegrationTestHelper.execute_query(self.sql_api_url, query) + assert_not_equal(isolines['the_geom'], None) + + def test_if_select_with_isochrones_without_api_key_raise_error(self): + query = "SELECT * FROM cdb_isochrone('POINT(-3.70568 40.42028)'::geometry, " \ + "'car', ARRAY[300]::integer[]);" + try: + IntegrationTestHelper.execute_query(self.sql_api_url, query) + except Exception as e: + assert_equal(e.message[0], "The api_key must be provided") + + def test_if_select_with_isodistance_is_ok(self): + query = "SELECT * FROM cdb_isodistance('POINT(-3.70568 40.42028)'::geometry, " \ + "'car', ARRAY[300]::integer[]);&api_key={0}".format( + self.env_variables['api_key']) + isolines = IntegrationTestHelper.execute_query(self.sql_api_url, query) + assert_not_equal(isolines['the_geom'], None) + + def test_if_select_with_isodistance_without_api_key_raise_error(self): + query = "SELECT * FROM cdb_isodistance('POINT(-3.70568 40.42028)'::geometry, " \ + "'car', ARRAY[300]::integer[]);" + try: + IntegrationTestHelper.execute_query(self.sql_api_url, query) + except Exception as e: + assert_equal(e.message[0], "The api_key must be provided") diff --git a/test/integration/test_namedplace_functions.py b/test/integration/test_namedplace_functions.py index e3c7c07..6b7abf5 100644 --- a/test/integration/test_namedplace_functions.py +++ b/test/integration/test_namedplace_functions.py @@ -20,7 +20,7 @@ class TestNameplaceFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_namedplace_city_country_is_ok(self): query = "SELECT cdb_geocode_namedplace_point(city,country) " \ @@ -28,7 +28,7 @@ class TestNameplaceFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_namedplace_city_province_country_is_ok(self): query = "SELECT cdb_geocode_namedplace_point(city,province,country) " \ @@ -36,7 +36,7 @@ class TestNameplaceFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_namedplace_without_api_key_raise_error(self): query = "SELECT cdb_geocode_namedplace_point(city) as geometry " \ diff --git a/test/integration/test_postalcode_functions.py b/test/integration/test_postalcode_functions.py index 7845a32..07055f9 100644 --- a/test/integration/test_postalcode_functions.py +++ b/test/integration/test_postalcode_functions.py @@ -20,7 +20,7 @@ class TestPostalcodeFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_postalcode_point_is_ok(self): query = "SELECT cdb_geocode_postalcode_point(postalcode, country) " \ @@ -28,7 +28,7 @@ class TestPostalcodeFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_postalcode_without_api_key_raise_error(self): query = "SELECT cdb_geocode_postalcode_polygon(postalcode, country) " \ diff --git a/test/integration/test_street_functions.py b/test/integration/test_street_functions.py index e4b1adf..8b82f68 100644 --- a/test/integration/test_street_functions.py +++ b/test/integration/test_street_functions.py @@ -20,7 +20,7 @@ class TestStreetFunctions(TestCase): self.env_variables['table_name'], self.env_variables['api_key']) geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query) - assert_not_equal(geometry, None) + assert_not_equal(geometry['geometry'], None) def test_if_select_with_street_without_api_key_raise_error(self): query = "SELECT cdb_geocode_street_point(street) " \