cartodb4/dataservices-api
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #182 from CartoDB/development

Browse Source 
Quota management docs and integrations tests fixes
This commit is contained in:
Mario de Frutos
2016-05-18 15:21:36 +02:00
parent 77e31db729 9a49a20358
commit 2a98a673d1
2 changed files with 76 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
doc/internal/quota_management.md Normal file
View File

@@ -0,0 +1,66 @@
All the services from Dataservices API are subject to quota management: check, limit, etc.
There are three main fields in the quota management:
- **Quota**: Number of requests of this kind the user could make, eg. Number of street geocoding requests
- **Soft limit**: This flag enables the user to surpass their assigned quota. When this flag is activated, there is no quota check so the user could make all the requests that they want. Consequently, the `hard_limit` flag, when enabled, indicates that the user cannot surpass the assigned quota.
- **Block price**: Price for every 1000 requests
All the user/organization quota information is stored in the user metadata in Redis but managed by the CartoDB Rails app through the User/Organization models. Nevertheless you could change/read the Redis information through the following keys:
- `hgetall rails:users:username`
- `hgetall rails:orgs:orgname`
This whole information is managed by the CartoDB Rails App too so we could make a numer of useful operations in order to know how much quota do you have, how much quota have you spent this month and so on.
###How can I know the current quota, number of uses, etc for a user?###
You could use the following endpoint to know it: https://<username>.cartodb.com/api/v1/users/<user_id>?api_key=<api_key>
In the result of this endpoint you can see blocks with all the information. Eg:
```
"geocoding": {
"quota": 1000,
"block_price": 1500,
"monthly_use": 743,
"hard_limit": true
}
```
### How can I set a new quota for a user###
This operation could be done through the rails console:
- First you have to connect to the rails console: `bundle exec rails c`
- Once in the console you have to get the target user/organization:
- `u = User.find(username: '<username>')`
- `o = Organization.find(name: '<orgname>')`
- After we have the user/organization, we could change the quota or the hard limit flag for the desired service. I'm going to use geocoding as an example but it could be done with all the services:
```
u.geocoding_quota = 2000
[u.soft_geocoding_limit = true|false]
u.save
```
```
o.geocoding_quota = 2000
o.save
```
- This way the user now has 2000 requests as their current quota
- We can only change the hard limit flag for users, not for organizations
### What services we could change?###
The following list numbers all the current services but this is a living list so it could keep growing in the future:
- Geocoding (street level): `geocoding_quota`, `soft_geocoding_limit`
- Isolines: `here_isolines_quota`, `soft_here_isolines_limit`
- Data observatory snapshot: `obs_snapshot_quota`, `soft_obs_snapshot_limit`
- Data observatory general: `obs_general_quota`, `soft_obs_general_limit`
### How is the quota spent?###
Almost in all the services: geocoding, data observatory snapshot and general the number of spent credits is calculated per request made (either successful or empty request).
In the case of the isolines service, the number of credits is calculated based on the number of isolines generated by the request. Ie. If your query generates 3 isolines for the request, you've spent 3 isolines credits.

20
test/integration/test_data_observatory_functions.py
View File

@@ -39,32 +39,32 @@ class TestDataObservatoryFunctions(TestCase):
assert_equal(e.message[0], "The api_key must be provided")
def test_if_get_measure_with_point_is_ok(self):
query = "SELECT OBS_GetMeasure(CDB_LatLng(40.704512, -73.936669), 'us.census.acs.B01001001') as measure;&api_key={0}".format(self.env_variables['api_key'])
query = "SELECT OBS_GetMeasure(CDB_LatLng(40.704512, -73.936669), 'us.census.acs.B01003001') as measure;&api_key={0}".format(self.env_variables['api_key'])
result = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(result['measure'], None)
assert_not_equal(result['measure'], 10923.097048937793740)
assert_equal(result['measure'], 10923.097048937794)
def test_if_get_measure_with_area_is_ok(self):
query = "SELECT OBS_GetMeasure('0103000020E61000000100000021000000BB366F6D917B52C0E7BB6EA82B5A444067224C85937B52C0810205E70E5A4440596D6342997B52C09BED2952F35944400A96386CA27B52C07B6D31F9D95944402B81A0A8AE7B52C0D16B73D5C3594440C34C397FBD7B52C0D8B5B7C0B1594440CFD90A5ECE7B52C0BE8DD86CA45944405C7E229FE07B52C0A904EE5C9C59444017C1F28EF37B52C05E2745E099594440AE8A3873067C52C0FD62540F9D5944402F272292187C52C05502CBCAA5594440C3F17139297C52C0B3FBC4BCB3594440DDAE56C5377C52C00C46175CC6594440E96AB6A6437C52C0355C94F1DC5944402313AE684C7C52C05340159FF6594440B3B90FB5517C52C031F30168125A44407F43B357537C52C0DF95053B2F5A444059C17840517C52C0D9E08EFC4B5A44406580E8834B7C52C046B5B591675A444035736A5A427C52C0AF9A1AEB805A44402E721C1E367C52C0D226550F975A4440506D5C47277C52C0A2968A24A95A4440507C2868167C52C0D51FCE78B65A4440C4438226047C52C02369F888BE5A4440C9F10C36F17B52C027B1B205C15A4440AABE2451DE7B52C0F5E383D6BD5A44402A18B431CC7B52C01785C11AB55A444018320D8ABB7B52C070265B28A75A4440E28A0EFEAC7B52C09E518C88945A44401D08D61CA17B52C09E8195F27D5A4440D7C3405B987B52C0643CB044645A444047B16D0F937B52C04EC9837B485A4440BB366F6D917B52C0E7BB6EA82B5A4440'::geometry, 'us.census.acs.B01001001') as measure;&api_key={0}".format(self.env_variables['api_key'])
query = "SELECT OBS_GetMeasure('0103000020E61000000100000021000000BB366F6D917B52C0E7BB6EA82B5A444067224C85937B52C0810205E70E5A4440596D6342997B52C09BED2952F35944400A96386CA27B52C07B6D31F9D95944402B81A0A8AE7B52C0D16B73D5C3594440C34C397FBD7B52C0D8B5B7C0B1594440CFD90A5ECE7B52C0BE8DD86CA45944405C7E229FE07B52C0A904EE5C9C59444017C1F28EF37B52C05E2745E099594440AE8A3873067C52C0FD62540F9D5944402F272292187C52C05502CBCAA5594440C3F17139297C52C0B3FBC4BCB3594440DDAE56C5377C52C00C46175CC6594440E96AB6A6437C52C0355C94F1DC5944402313AE684C7C52C05340159FF6594440B3B90FB5517C52C031F30168125A44407F43B357537C52C0DF95053B2F5A444059C17840517C52C0D9E08EFC4B5A44406580E8834B7C52C046B5B591675A444035736A5A427C52C0AF9A1AEB805A44402E721C1E367C52C0D226550F975A4440506D5C47277C52C0A2968A24A95A4440507C2868167C52C0D51FCE78B65A4440C4438226047C52C02369F888BE5A4440C9F10C36F17B52C027B1B205C15A4440AABE2451DE7B52C0F5E383D6BD5A44402A18B431CC7B52C01785C11AB55A444018320D8ABB7B52C070265B28A75A4440E28A0EFEAC7B52C09E518C88945A44401D08D61CA17B52C09E8195F27D5A4440D7C3405B987B52C0643CB044645A444047B16D0F937B52C04EC9837B485A4440BB366F6D917B52C0E7BB6EA82B5A4440'::geometry, 'us.census.acs.B01003001') as measure;&api_key={0}".format(self.env_variables['api_key'])
result = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(result['measure'], None)
assert_not_equal(result['measure'], 12327.3133495107)
assert_equal(result['measure'], 12327.3133495107)
def test_if_get_measure_without_api_key_raise_error(self):
query = "SELECT OBS_GetMeasure(CDB_LatLng(40.704512, -73.936669), 'us.census.acs.B01001001');"
query = "SELECT OBS_GetMeasure(CDB_LatLng(40.704512, -73.936669), 'us.census.acs.B01003001');"
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e:
assert_equal(e.message[0], "The api_key must be provided")
def test_if_get_category_is_ok(self):
query = "SELECT OBS_GetCategory(CDB_LatLng(40.704512, -73.936669), 'us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2009 - 2013') as category;&api_key={0}".format(self.env_variables['api_key'])
query = "SELECT OBS_GetCategory(CDB_LatLng(40.704512, -73.936669), 'us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014') as category;&api_key={0}".format(self.env_variables['api_key'])
result = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(result['category'], None)
assert_equal(result['category'], 'Wealthy, urban without Kids')
def test_if_get_category_without_api_key_raise_error(self):
query = "SELECT OBS_GetCategory(CDB_LatLng(40.704512, -73.936669), 'us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2009 - 2013');"
query = "SELECT OBS_GetCategory(CDB_LatLng(40.704512, -73.936669), 'us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014');"
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e:
@@ -106,7 +106,7 @@ class TestDataObservatoryFunctions(TestCase):
query = "SELECT OBS_GetPopulation(CDB_LatLng(40.704512, -73.936669)) as population;&api_key={0}".format(self.env_variables['api_key'])
result = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(result['population'], None)
assert_equal(result['population'], 9514.5924294695)
assert_equal(result['population'], 10923.097048937794)
def test_if_get_population_without_api_key_raise_error(self):
query = "SELECT OBS_GetPopulation(CDB_LatLng(40.704512, -73.936669));"
@@ -119,7 +119,7 @@ class TestDataObservatoryFunctions(TestCase):
query = "SELECT id FROM OBS_Search('total_pop') LIMIT 1;&api_key={0}".format(self.env_variables['api_key'])
result = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(result['id'], None)
assert_equal(result['id'], 'es.ine.total_pop')
assert_equal(result['id'], 'us.census.acs.B01003001_quantile')
def test_if_obs_search_without_api_key_raise_error(self):
query = "SELECT id FROM OBS_Search('total_pop') LIMIT 1;"
@@ -132,7 +132,7 @@ class TestDataObservatoryFunctions(TestCase):
query = "SELECT boundary_id FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.704512, -73.936669)) LIMIT 1;&api_key={0}".format(self.env_variables['api_key'])
result = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(result['boundary_id'], None)
assert_equal(result['boundary_id'], 'us.census.tiger.place')
assert_equal(result['boundary_id'], 'whosonfirst.wof_continent_geom')
def test_if_obs_get_available_boundaries_without_api_key_raise_error(self):
query = "SELECT boundary_id FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.704512, -73.936669)) LIMIT 1;"