cartodb4/dataservices-api
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' into new_dataservices_api

Browse Source
This commit is contained in:
Mario de Frutos
2016-02-25 20:19:19 +01:00
parent 2090ed56bc 926655fdf9
commit f42d007b77
21 changed files with 378 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
README.md
View File

@@ -1,8 +1,8 @@
# geocoder-api
The CartoDB Geocoder SQL API (server and client FTM)
# Data Services API
The CartoDB Data Services SQL API (server and client FTM)
### Deploy instructions
Steps to deploy a new Geocoder API version :
Steps to deploy a new Data Services API version :
- Deploy new version of dataservices API to all servers
- Update the server user using: ALTER EXTENSION cdb_dataservices_server UPDATE TO '<CURRENT_VERSION>';
@@ -20,3 +20,58 @@ Steps to deploy a new Geocoder API version :
- `RAILS_ENV=production bundle exec rake cartodb:db:configure_geocoder_extension_for_organizations['', true]`
- `RAILS_ENV=production bundle exec rake cartodb:db:configure_geocoder_extension_for_non_org_users['', true]`
- Freeze the generated SQL file for the version. Eg. cdb_dataservices_server--0.0.1.sql
### Local install instructions
- install data services extension
```
git clone git@github.com:CartoDB/data-services.git
data-services/geocoder/extension
sudo make install
```
- install server and client extensions
```
cd client && sudo make install
cd server && sudo make install
```
- install python library
```
cd server/lib/python/cartodb_services && python setup.py install
```
- install extensions in user database
```
create extension cdb_geocoder;
create extension plproxy;
create extension cdb_dataservices_server;
create extension cdb_dataservices_client;
```
- add configuration for different services in user database
```
# select CDB_Conf_SetConf('redis_metadata_config', '{"sentinel_host": "localhost", "sentinel_port": 26379, "sentinel_master_id": "mymaster", "timeout": 0.1, "redis_db": 5}');
# select CDB_Conf_SetConf('redis_metrics_config', '{"sentinel_host": "localhost", "sentinel_port": 26379, "sentinel_master_id": "mymaster", "timeout": 0.1, "redis_db": 5}');
# select CDB_Conf_SetConf('heremaps_conf', '{"app_id": "APP_ID", "app_code": "APP_CODE"}');
# select CDB_Conf_SetConf('user_config', '{"is_organization": false, "entity_name": "<YOUR_USERNAME>"}')
```
- configure plproxy to point to the same user database (you could do in a different one)
```
select CDB_Conf_SetConf('geocoder_server_config', '{ "connection_str": "host=localhost port=5432 dbname=cartodb_dev_user_accf0647-d942-4e37-b129-8287c117e687_db user=postgres"}');
```
- configure the search path in order to be able to execute the functions without use the schema:
```
alter role "<USER_ROLE>" set search_path='"$user", public, cartodb, cdb_dataservices_client';
```

69
doc/internal/internal_doc.md Normal file
View File

@@ -0,0 +1,69 @@
# Data Services API Internal documentation
* [Existent services](#existent-services)
* [How to add a new service](#how-to-add-a-new-service)
## Existent services
Available at [cartodb_services](https://github.com/CartoDB/geocoder-api/tree/master/server/lib/python/cartodb_services/cartodb_services).
* Google
* [Geocoding](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/google/geocoder.py)
* Here
* [Geocoding](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/here/geocoder.py)
* [Routing](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/here/routing.py)
## How to add a new service
These are the steps that need to be followed when creating a new service in the API or updating an existent one.
### Creating a new service function or editing an existent one
In this scenario, both client and server sides require to be edited/created.
* **Update the interface file** with the function addition or update
* Interfaces are stored in `client/renderer/interfaces`
* Interface YAML filenames follow the client versioning schema with the format `interface-x.y.z.yaml`.
* Update the renderer templates or script, *if applicable*
* Renderer templates are stored in `client/renderer/templates`
* The Renderer script (`client/renderer/sql-template-renderer`) generates SQL from the defined interfaces
* Generate a **new subfolder version** for `sql` and `test` folders to define the new functions and tests
* TODO: Use symlinks to avoid file duplication between versions that don't update them
* The `client/sql` folder contents are generated from the interfaces in the first step
* Add or upgrade your SQL server functions
* Create tests for the client and server functions -- at least, to check that those are created
* Generate the **upgrade and downgrade files** for the extension for both client and server
* Update the control files and the Makefiles to generate the complete SQL file for the new created version
* These new version files (`cdb_dataservices_client--X.Y.Z.sql and cdb_dataservices_server--X.Y.X.sql`) must be pushed and frozen. You can add these to the `.gitignore` file.
* Update the public docs! ;-)
### Updating an existing server side function
With no changes in client side.
#### Extension
* Generate a **new subfolder version** for `sql` and `test` folders to define the new functions and tests
* **TODO:** Use symlinks to avoid file duplication between versions that don't update them
* **Add or upgrade your SQL server functions**
* For example, if a new street geocoder service is implemented, it will require a change in the main function (`cdb_dataservices_server.cdb_geocode_street_point`) and generate a new `cdb_dataservices_server._cdb_newservice_geocode_street_point`
* Generate the upgrade and downgrade files for the extension for the client
#### Python
* Add, if needed, [new configuration elements](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/config.py#L100)
* Add the new **functionality** into the provider folder. If the provider is new, create a new folder(`server/lib/python/cartodb_services/cartodb_services/{provider_name}` and add the service (`geocoder.py`)
* Check the `__init__.py` files to follow the existent [import conventions](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py)
* Add a **new metric**, if needed, at the [corresponding service](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/cartodb_services/metrics/quota.py#L37-L60)
* **Update the package version** in [setup.py](https://github.com/CartoDB/geocoder-api/blob/983440086d3fabf03aedc66f53dcf4c4a8cb2323/server/lib/python/cartodb_services/setup.py)

25
server/extension/cdb_dataservices_server--0.3.0--0.4.0.sql
View File

@@ -1,12 +1,12 @@
-- Get the Redis configuration from the _conf table --
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text)
RETURNS boolean AS $$
cache_key = "user_routing_config_{0}".format(username)
cache_key = "user_isolines_routing_config_{0}".format(username)
if cache_key in GD:
return False
else:
import json
from cartodb_services.metrics import RoutingConfig
from cartodb_services.metrics import IsolinesRoutingConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection']
heremaps_conf_json = plpy.execute("SELECT cartodb.CDB_Conf_GetConf('heremaps_conf') as heremaps_conf", 1)[0]['heremaps_conf']
@@ -17,7 +17,7 @@ RETURNS boolean AS $$
heremaps_conf = json.loads(heremaps_conf_json)
heremaps_app_id = heremaps_conf['app_id']
heremaps_app_code = heremaps_conf['app_code']
routing_config = RoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code)
routing_config = IsolinesRoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code)
# --Think about the security concerns with this kind of global cache, it should be only available
# --for this user session but...
GD[cache_key] = routing_config
@@ -34,12 +34,15 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.here.types import geo_polyline_to_multipolygon
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
user_routing_config = GD["user_routing_config_{0}".format(username)]
user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
quota_service = QuotaService(user_routing_config, redis_conn)
# -- Check the quota
quota_service = QuotaService(user_isolines_routing_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reach the limit of your quota')
try:
client = HereMapsRoutingIsoline(user_routing_config.heremaps_app_id, user_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL)
client = HereMapsRoutingIsoline(user_isolines_routing_config.heremaps_app_id, user_isolines_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
@@ -80,8 +83,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
type = 'isodistance'
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"])
@@ -99,8 +102,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT,
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
type = 'isochrone'
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"])

2
server/extension/cdb_dataservices_server--0.4.0--0.3.0.sql
View File

@@ -1,5 +1,5 @@
DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_isochrone(TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_isodistance(TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_here_routing_isolines(TEXT, TEXT, TEXT, geometry(Geometry, 4326), TEXT, integer[], text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_server._get_routing_config(text, text);
DROP FUNCTION IF EXISTS cdb_dataservices_server._get_isolines_routing_config(text, text);
DROP TYPE IF EXISTS cdb_dataservices_server.isoline;

27
server/extension/cdb_dataservices_server--0.4.0.sql
View File

@@ -85,14 +85,14 @@ RETURNS boolean AS $$
$$ LANGUAGE plpythonu SECURITY DEFINER;
-- Get the Redis configuration from the _conf table --
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text)
RETURNS boolean AS $$
cache_key = "user_routing_config_{0}".format(username)
cache_key = "user_isolines_routing_config_{0}".format(username)
if cache_key in GD:
return False
else:
import json
from cartodb_services.metrics import RoutingConfig
from cartodb_services.metrics import IsolinesRoutingConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection']
heremaps_conf_json = plpy.execute("SELECT cartodb.CDB_Conf_GetConf('heremaps_conf') as heremaps_conf", 1)[0]['heremaps_conf']
@@ -103,10 +103,10 @@ RETURNS boolean AS $$
heremaps_conf = json.loads(heremaps_conf_json)
heremaps_app_id = heremaps_conf['app_id']
heremaps_app_code = heremaps_conf['app_code']
routing_config = RoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code)
isolines_routing_config = IsolinesRoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code)
# --Think about the security concerns with this kind of global cache, it should be only available
# --for this user session but...
GD[cache_key] = routing_config
GD[cache_key] = isolines_routing_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
-- Geocodes a street address given a searchtext and a state and/or country
@@ -814,12 +814,15 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.here.types import geo_polyline_to_multipolygon
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
user_routing_config = GD["user_routing_config_{0}".format(username)]
user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
quota_service = QuotaService(user_routing_config, redis_conn)
# -- Check the quota
quota_service = QuotaService(user_isolines_routing_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reach the limit of your quota')
try:
client = HereMapsRoutingIsoline(user_routing_config.heremaps_app_id, user_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.STAGING_ROUTING_BASE_URL )
client = HereMapsRoutingIsoline(user_isolines_routing_config.heremaps_app_id, user_isolines_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
@@ -859,8 +862,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
type = 'isodistance'
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"])
@@ -877,8 +880,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT,
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
type = 'isochrone'
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"])

10
server/extension/sql/0.4.0/15_config_helper.sql
View File

@@ -25,14 +25,14 @@ RETURNS boolean AS $$
$$ LANGUAGE plpythonu SECURITY DEFINER;
-- Get the Redis configuration from the _conf table --
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text)
RETURNS boolean AS $$
cache_key = "user_routing_config_{0}".format(username)
cache_key = "user_isolines_routing_config_{0}".format(username)
if cache_key in GD:
return False
else:
import json
from cartodb_services.metrics import RoutingConfig
from cartodb_services.metrics import IsolinesRoutingConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection']
heremaps_conf_json = plpy.execute("SELECT cartodb.CDB_Conf_GetConf('heremaps_conf') as heremaps_conf", 1)[0]['heremaps_conf']
@@ -43,9 +43,9 @@ RETURNS boolean AS $$
heremaps_conf = json.loads(heremaps_conf_json)
heremaps_app_id = heremaps_conf['app_id']
heremaps_app_code = heremaps_conf['app_code']
routing_config = RoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code)
isolines_routing_config = IsolinesRoutingConfig(redis_conn, username, orgname, heremaps_app_id, heremaps_app_code)
# --Think about the security concerns with this kind of global cache, it should be only available
# --for this user session but...
GD[cache_key] = routing_config
GD[cache_key] = isolines_routing_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;

9
server/extension/sql/0.4.0/80_routing_helper.sql → server/extension/sql/0.4.0/80_isolines_helper.sql
View File

@@ -8,12 +8,15 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.here.types import geo_polyline_to_multipolygon
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
user_routing_config = GD["user_routing_config_{0}".format(username)]
user_isolines_routing_config = GD["user_isolines_routing_config_{0}".format(username)]
quota_service = QuotaService(user_routing_config, redis_conn)
# -- Check the quota
quota_service = QuotaService(user_isolines_routing_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reach the limit of your quota')
try:
client = HereMapsRoutingIsoline(user_routing_config.heremaps_app_id, user_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL)
client = HereMapsRoutingIsoline(user_isolines_routing_config.heremaps_app_id, user_isolines_routing_config.heremaps_app_code, base_url = HereMapsRoutingIsoline.PRODUCTION_ROUTING_BASE_URL)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']

4
server/extension/sql/0.4.0/85_isodistance.sql
View File

@@ -2,8 +2,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
type = 'isodistance'
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"])

4
server/extension/sql/0.4.0/90_isochrone.sql
View File

@@ -2,8 +2,8 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT,
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
type = 'isochrone'
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_routing_isolines($1, $2, $3, $4, $5, $6, $7) as isoline; ", ["text", "text", "text", "geometry(Geometry, 4326)", "text", "integer[]", "text[]"])

2
server/lib/python/cartodb_services/cartodb_services/metrics/__init__.py
View File

@@ -1,3 +1,3 @@
from config import GeocoderConfig, RoutingConfig, InternalGeocoderConfig, ConfigException
from config import GeocoderConfig, IsolinesRoutingConfig, InternalGeocoderConfig, ConfigException
from quota import QuotaService
from user import UserMetricsService

70
server/lib/python/cartodb_services/cartodb_services/metrics/config.py
View File

@@ -28,18 +28,76 @@ class ServiceConfig(object):
return self._orgname
class RoutingConfig(ServiceConfig):
class IsolinesRoutingConfig(ServiceConfig):
ROUTING_CONFIG_KEYS = ['here_isolines_quota', 'soft_here_isolines_limit',
'period_end_date', 'username', 'orgname',
'heremaps_app_id', 'heremaps_app_code']
NOKIA_APP_ID_KEY = 'heremaps_app_id'
NOKIA_APP_CODE_KEY = 'heremaps_app_code'
QUOTA_KEY = 'here_isolines_quota'
SOFT_LIMIT_KEY = 'soft_here_isolines_limit'
USERNAME_KEY = 'username'
ORGNAME_KEY = 'orgname'
PERIOD_END_DATE = 'period_end_date'
def __init__(self, redis_connection, username, orgname=None,
heremaps_app_id=None, heremaps_app_code=None):
super(RoutingConfig, self).__init__(redis_connection,
username, orgname)
self._heremaps_app_id = heremaps_app_id
self._heremaps_app_code = heremaps_app_code
super(IsolinesRoutingConfig, self).__init__(redis_connection, username,
orgname)
config = self.__get_user_config(username, orgname, heremaps_app_id,
heremaps_app_code)
filtered_config = {key: config[key] for key in self.ROUTING_CONFIG_KEYS if key in config.keys()}
self.__parse_config(filtered_config)
def __get_user_config(self, username, orgname=None, heremaps_app_id=None,
heremaps_app_code=None):
user_config = self._redis_connection.hgetall(
"rails:users:{0}".format(username))
if not user_config:
raise ConfigException("""There is no user config available. Please check your configuration.'""")
else:
user_config[self.NOKIA_APP_ID_KEY] = heremaps_app_id
user_config[self.NOKIA_APP_CODE_KEY] = heremaps_app_code
if orgname:
self.__get_organization_config(orgname, user_config)
return user_config
def __get_organization_config(self, orgname, user_config):
org_config = self._redis_connection.hgetall(
"rails:orgs:{0}".format(orgname))
if not org_config:
raise ConfigException("""There is no organization config available. Please check your configuration.'""")
else:
user_config[self.QUOTA_KEY] = org_config[self.QUOTA_KEY]
user_config[self.PERIOD_END_DATE] = org_config[self.PERIOD_END_DATE]
def __parse_config(self, filtered_config):
self._isolines_quota = float(filtered_config[self.QUOTA_KEY])
self._period_end_date = date_parse(filtered_config[self.PERIOD_END_DATE])
if filtered_config[self.SOFT_LIMIT_KEY].lower() == 'true':
self._soft_isolines_limit = True
else:
self._soft_isolines_limit = False
self._heremaps_app_id = filtered_config[self.NOKIA_APP_ID_KEY]
self._heremaps_app_code = filtered_config[self.NOKIA_APP_CODE_KEY]
@property
def service_type(self):
return 'routing_here'
return 'here_isolines'
@property
def isolines_quota(self):
return self._isolines_quota
@property
def soft_isolines_limit(self):
return self._soft_isolines_limit
@property
def period_end_date(self):
return self._period_end_date
@property
def heremaps_app_id(self):

83
server/lib/python/cartodb_services/cartodb_services/metrics/quota.py
View File

@@ -1,32 +1,22 @@
from user import UserMetricsService
from datetime import date
import re
class QuotaService:
""" Class to manage all the quota operation for
the Geocoder SQL API Extension """
def __init__(self, user_geocoder_config, redis_connection):
self._user_geocoder_config = user_geocoder_config
def __init__(self, user_service_config, redis_connection):
self._user_service_config = user_service_config
# TODO First step to extract to a factory if needed in the future
self._quota_checker = QuotaChecker(user_service_config,
redis_connection)
self._user_service = UserMetricsService(
self._user_geocoder_config, redis_connection)
self._user_service_config, redis_connection)
def check_user_quota(self):
""" Check if the current user quota surpasses the current quota """
# We don't have quota check for google geocoder
if self._user_geocoder_config.google_geocoder:
return True
user_quota = self._user_geocoder_config.geocoding_quota
today = date.today()
service_type = self._user_geocoder_config.service_type
current_used = self._user_service.used_quota(service_type, today)
soft_geocoding_limit = self._user_geocoder_config.soft_geocoding_limit
if soft_geocoding_limit or current_used <= user_quota:
return True
else:
return False
return self._quota_checker.check()
# TODO
# We are going to change this class to be the generic one and
@@ -36,25 +26,72 @@ class QuotaService:
def increment_success_geocoder_use(self, amount=1):
self._user_service.increment_service_use(
self._user_geocoder_config.service_type, "success_responses",
self._user_service_config.service_type, "success_responses",
amount=amount)
def increment_empty_geocoder_use(self, amount=1):
self._user_service.increment_service_use(
self._user_geocoder_config.service_type, "empty_responses",
self._user_service_config.service_type, "empty_responses",
amount=amount)
def increment_failed_geocoder_use(self, amount=1):
self._user_service.increment_service_use(
self._user_geocoder_config.service_type, "fail_responses",
self._user_service_config.service_type, "fail_responses",
amount=amount)
def increment_total_geocoder_use(self, amount=1):
self._user_service.increment_service_use(
self._user_geocoder_config.service_type, "total_requests",
self._user_service_config.service_type, "total_requests",
amount=amount)
def increment_isolines_service_use(self, amount=1):
self._user_service.increment_service_use(
self._user_geocoder_config.service_type, "isolines_generated",
self._user_service_config.service_type, "isolines_generated",
amount=amount)
class QuotaChecker:
def __init__(self, user_service_config, redis_connection):
self._user_service_config = user_service_config
self._user_service = UserMetricsService(
self._user_service_config, redis_connection)
def check(self):
""" Check if the current user quota surpasses the current quota """
if re.match('geocoder_*',
self._user_service_config.service_type) is not None:
return self.__check_geocoder_quota()
elif re.match('here_isolines',
self._user_service_config.service_type) is not None:
return self.__check_isolines_quota()
else:
return False
def __check_geocoder_quota(self):
# We don't have quota check for google geocoder
if self._user_service_config.google_geocoder:
return True
user_quota = self._user_service_config.geocoding_quota
today = date.today()
service_type = self._user_service_config.service_type
current_used = self._user_service.used_quota(service_type, today)
soft_geocoding_limit = self._user_service_config.soft_geocoding_limit
if soft_geocoding_limit or (user_quota > 0 and current_used <= user_quota):
return True
else:
return False
def __check_isolines_quota(self):
user_quota = self._user_service_config.isolines_quota
today = date.today()
service_type = self._user_service_config.service_type
current_used = self._user_service.used_quota(service_type, today)
soft_isolines_limit = self._user_service_config.soft_isolines_limit
if soft_isolines_limit or (user_quota > 0 and current_used <= user_quota):
return True
else:
return False

30
server/lib/python/cartodb_services/cartodb_services/metrics/user.py
View File

@@ -6,16 +6,8 @@ class UserMetricsService:
""" Class to manage all the user info """
SERVICE_GEOCODER_NOKIA = 'geocoder_here'
SERVICE_GEOCODER_GOOGLE = 'geocoder_google'
SERVICE_GEOCODER_CACHE = 'geocoder_cache'
GEOCODING_QUOTA_KEY = "geocoding_quota"
GEOCODING_SOFT_LIMIT_KEY = "soft_geocoder_limit"
REDIS_CONNECTION_KEY = "redis_connection"
REDIS_CONNECTION_HOST = "redis_host"
REDIS_CONNECTION_PORT = "redis_port"
REDIS_CONNECTION_DB = "redis_db"
SERVICE_HERE_ISOLINES = 'here_isolines'
def __init__(self, user_geocoder_config, redis_connection):
self._user_geocoder_config = user_geocoder_config
@@ -24,6 +16,12 @@ class UserMetricsService:
self._orgname = user_geocoder_config.organization
def used_quota(self, service_type, date):
if service_type == self.SERVICE_HERE_ISOLINES:
return self.__used_isolines_quota(service_type, date)
else:
return self.__used_geocoding_quota(service_type, date)
def __used_geocoding_quota(self, service_type, date):
""" Recover the used quota for the user in the current month """
date_from, date_to = self.__current_billing_cycle()
current_use = 0
@@ -42,6 +40,20 @@ class UserMetricsService:
return current_use
def __used_isolines_quota(self, service_type, date):
date_from, date_to = self.__current_billing_cycle()
current_use = 0
isolines_generated = self.get_metrics(service_type,
'isolines_generated', date_from,
date_to)
empty_responses = self.get_metrics(service_type,
'empty_responses', date_from,
date_to)
current_use += (isolines_generated + empty_responses)
return current_use
def increment_service_use(self, service_type, metric, date=date.today(), amount=1):
""" Increment the services uses in monthly and daily basis"""
self.__increment_user_uses(service_type, metric, date, amount)

2
test/helpers/integration_test_helper.py
View File

@@ -29,4 +29,4 @@ class IntegrationTestHelper:
raise Exception(json.loads(query_response.text)['error'])
query_response_data = json.loads(query_response.text)
return query_response_data['rows'][0]['geometry']
return query_response_data['rows'][0]

11
test/integration/test_admin0_functions.py
View File

@@ -16,16 +16,15 @@ class TestAdmin0Functions(TestCase):
def test_if_select_with_admin0_is_ok(self):
query = "SELECT cdb_geocode_admin0_polygon(country) as geometry " \
"FROM {0} LIMIT 1&api_key={1}".format(
self.env_variables['table_name'],
self.env_variables['api_key'])
"FROM {0} LIMIT 1&api_key={1}".format(
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_admin0_without_api_key_raise_error(self):
query = "SELECT cdb_geocode_admin0_polygon(country) as geometry " \
"FROM {0} LIMIT 1".format(
self.env_variables['table_name'])
"FROM {0} LIMIT 1".format(self.env_variables['table_name'])
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e:

4
test/integration/test_admin1_functions.py
View File

@@ -20,7 +20,7 @@ class TestAdmin1Functions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_admin1_with_country_is_ok(self):
query = "SELECT cdb_geocode_admin1_polygon(province,country)" \
@@ -28,7 +28,7 @@ class TestAdmin1Functions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_admin1_without_api_key_raise_error(self):
query = "SELECT cdb_geocode_admin1_polygon(province) as geometry " \

2
test/integration/test_ipaddress_functions.py
View File

@@ -20,7 +20,7 @@ class TestPostalcodeFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_ipaddress_without_api_key_raise_error(self):
query = "SELECT cdb_geocode_ipaddress_point(ip) " \

45
test/integration/test_isolines_functions.py Normal file
View File

@@ -0,0 +1,45 @@
from unittest import TestCase
from nose.tools import assert_raises
from nose.tools import assert_not_equal, assert_equal
from ..helpers.integration_test_helper import IntegrationTestHelper
class TestIsolinesFunctions(TestCase):
def setUp(self):
self.env_variables = IntegrationTestHelper.get_environment_variables()
self.sql_api_url = "https://{0}.{1}/api/v2/sql".format(
self.env_variables['username'],
self.env_variables['host'],
self.env_variables['api_key']
)
def test_if_select_with_isochrones_is_ok(self):
query = "SELECT * FROM cdb_isochrone('POINT(-3.70568 40.42028)'::geometry, " \
"'car', ARRAY[300]::integer[]);&api_key={0}".format(
self.env_variables['api_key'])
isolines = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(isolines['the_geom'], None)
def test_if_select_with_isochrones_without_api_key_raise_error(self):
query = "SELECT * FROM cdb_isochrone('POINT(-3.70568 40.42028)'::geometry, " \
"'car', ARRAY[300]::integer[]);"
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e:
assert_equal(e.message[0], "The api_key must be provided")
def test_if_select_with_isodistance_is_ok(self):
query = "SELECT * FROM cdb_isodistance('POINT(-3.70568 40.42028)'::geometry, " \
"'car', ARRAY[300]::integer[]);&api_key={0}".format(
self.env_variables['api_key'])
isolines = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(isolines['the_geom'], None)
def test_if_select_with_isodistance_without_api_key_raise_error(self):
query = "SELECT * FROM cdb_isodistance('POINT(-3.70568 40.42028)'::geometry, " \
"'car', ARRAY[300]::integer[]);"
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e:
assert_equal(e.message[0], "The api_key must be provided")

6
test/integration/test_namedplace_functions.py
View File

@@ -20,7 +20,7 @@ class TestNameplaceFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_namedplace_city_country_is_ok(self):
query = "SELECT cdb_geocode_namedplace_point(city,country) " \
@@ -28,7 +28,7 @@ class TestNameplaceFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_namedplace_city_province_country_is_ok(self):
query = "SELECT cdb_geocode_namedplace_point(city,province,country) " \
@@ -36,7 +36,7 @@ class TestNameplaceFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_namedplace_without_api_key_raise_error(self):
query = "SELECT cdb_geocode_namedplace_point(city) as geometry " \

4
test/integration/test_postalcode_functions.py
View File

@@ -20,7 +20,7 @@ class TestPostalcodeFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_postalcode_point_is_ok(self):
query = "SELECT cdb_geocode_postalcode_point(postalcode, country) " \
@@ -28,7 +28,7 @@ class TestPostalcodeFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_postalcode_without_api_key_raise_error(self):
query = "SELECT cdb_geocode_postalcode_polygon(postalcode, country) " \

2
test/integration/test_street_functions.py
View File

@@ -20,7 +20,7 @@ class TestStreetFunctions(TestCase):
self.env_variables['table_name'],
self.env_variables['api_key'])
geometry = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(geometry, None)
assert_not_equal(geometry['geometry'], None)
def test_if_select_with_street_without_api_key_raise_error(self):
query = "SELECT cdb_geocode_street_point(street) " \