Compare commits

...

43 Commits

Author SHA1 Message Date
Mario de Frutos
ee13920a8e Server SQL file for version 0.7.1 2016-04-25 11:12:46 +02:00
Mario de Frutos
5efb81d809 Merge pull request #162 from CartoDB/data_observatory_quota_override
Use redis or db data observatory quota value
2016-04-25 11:12:01 +02:00
Mario de Frutos
4407ff630e Change data observatory config file to segragate more in order to prepare for future observatory elements 2016-04-25 10:03:39 +02:00
Mario de Frutos
a9c420ba7b Use redis or db data observatory quota value 2016-04-25 09:33:18 +02:00
csobier
976697c81e Merge pull request #127 from CartoDB/mapzen_geocoder_doc
[do NOT merge] Add mapzen geocoder as provider in the geocoding docs
2016-04-21 09:58:43 -04:00
csobier
3de15de1c9 Merge pull request #102 from CartoDB/routing_function_docs
[do NOT merge] Routing function docs
2016-04-21 09:58:21 -04:00
Mario de Frutos
303c7c81fb Merge pull request #160 from CartoDB/development
Check for snapshot instead of the_geom in the data observatory integr…
2016-04-21 12:32:23 +02:00
Mario de Frutos
b9757d2026 Check for snapshot instead of the_geom in the data observatory integration tests 2016-04-21 12:31:39 +02:00
Mario de Frutos
8aaaba3737 Merge pull request #159 from CartoDB/development
Add api_key parameter to data observatory integration tests
2016-04-21 12:15:36 +02:00
Mario de Frutos
e73d6b84ff Add api_key parameter to data observatory integration tests 2016-04-21 12:14:48 +02:00
Mario de Frutos
178651150c Merge pull request #158 from CartoDB/development
More fixes to data observatory integration tests
2016-04-21 12:11:30 +02:00
Mario de Frutos
b0d614d032 More fixes to data observatory integration tests 2016-04-21 12:10:35 +02:00
Mario de Frutos
b3e6d6731a Merge pull request #157 from CartoDB/development
Rename data observatory funtion call in the integration tests
2016-04-21 12:07:01 +02:00
Mario de Frutos
403f0d2164 Rename data observatory funtion call in the integration tests 2016-04-21 12:05:45 +02:00
csobier
4c5c31cfea removed null select example, woo 2016-04-18 09:19:05 -04:00
csobier
5ac9bd884f what about the SELECT examples for street? 2016-04-18 08:46:05 -04:00
csobier
6b6c30ff17 merged master, ready to roll 2016-04-15 13:50:37 -04:00
csobier
3ed89c09d5 clarified actual behavior of geocoding descriptions 2016-04-15 13:41:24 -04:00
Mario de Frutos
2ffee03e7b Merge branch 'master' into mapzen_geocoder_doc 2016-04-15 16:49:08 +02:00
csobier
9fce6b558b applied update for DataServices API issue#145 2016-04-15 07:29:53 -04:00
csobier
6eb8df2955 removed typo from copypaste review, and added more descriptions for geocoding functions 2016-04-13 13:28:42 -04:00
csobier
1fbab0617f change point to geometries 2016-04-13 12:32:00 -04:00
csobier
d18eb95575 clarified equalness in isolines 2016-04-13 12:21:38 -04:00
Sergio Álvarez Leiva
3c1250ee72 Update overview.md 2016-04-13 13:16:24 +02:00
Sergio Álvarez Leiva
7e8d666c8c Update geocoding_functions.md 2016-04-13 13:14:25 +02:00
csobier
f223ad9ca0 fixing indentation 2016-04-12 19:54:13 -04:00
csobier
2c8cfa3032 hyperlink fix 2016-04-12 19:50:30 -04:00
csobier
0b53ede06c applied some formatting changes 2016-04-12 19:41:34 -04:00
csobier
6bd854a395 added more geocoding funtion descriptions 2016-04-12 19:15:51 -04:00
csobier
30cb5869f1 reorg of docs format and editing, removed mentions of providers 2016-04-12 17:19:51 -04:00
csobier
187fe8f849 updated routing functions description 2016-04-12 14:35:05 -04:00
csobier
ee9e5aba38 updated renamed link 2016-04-11 10:21:45 -04:00
csobier
6105f677af added mapzen basemap details 2016-04-11 10:20:43 -04:00
csobier
9e90ee3e1f clarified how to access Mapzen features 2016-04-11 09:01:49 -04:00
csobier
ea24aa938f added link in overview to quickstart rregarding mapzen setup 2016-04-11 08:21:26 -04:00
csobier
75d58dc836 fix url 2016-04-06 07:57:51 -04:00
csobier
652bd6abd6 fixed url 2016-04-06 07:56:15 -04:00
csobier
dab46269d4 url fixes 2016-04-06 07:42:38 -04:00
csobier
8eeef94eab added quick start info about mapzen 2016-04-05 16:21:34 -04:00
csobier
14d02fd63f added link to mapzen routing 2016-04-05 15:48:57 -04:00
csobier
28b4b12ce4 minor edits to mapzen geocoder provider inserts 2016-03-28 11:00:21 -04:00
Mario de Frutos
dd2af74b03 Add mapzen geocoder as provider in the geocoding docs 2016-03-28 16:47:22 +02:00
Mario de Frutos
30001b68a7 Routing functions docs 2016-03-17 20:05:49 +01:00
27 changed files with 1600 additions and 207 deletions

View File

@@ -2,14 +2,10 @@
The CartoDB Data Services API offers a set of location based services that can be used programatically to empower your geospatial applications.
The geocoder functions allow you to match your data with geometries on your map. This geocoding service can be used programatically to geocode datasets via the CartoDB SQL API. It is fed from _Open Data_ and it serves geometries for countries, provinces, states, cities, postal codes, IP addresses and street addresses.
The isoline functions provide a way to generate isolines in terms of distance and time, by means of the available isodistance and isochrone functions useful for Trade Areas Analysis.
## Documentation
* [Quickstart](quickstart.md)
* [General Concepts](general_concepts.md)
* [Overview](overview.md)
* [Geocoding Functions](geocoding_functions.md)
* [Isoline Functions](isoline_functions.md)
* [Routing Functions](routing_functions.md)
* [Quota Information](quota_information.md)

View File

@@ -1,33 +0,0 @@
# General Concepts
The Data Services API offers geocoding and isoline services on top of the CartoDB SQL API by means of a set of functions. Each one of these functions is oriented to one kind of operation and returns the corresponding geometry (a `polygon` or a `point`), according to the input information.
The Data Services API decouples the geocoding and isoline services from the CartoDB Editor. The API allows you to geocode data (from single rows, complete datasets, or simple inputs) and to perform trade areas analysis (computing isodistances or isochrones) programatically through authenticated requests.
The geometries provided by this API are projected in the projection [WGS 84 SRID 4326](http://spatialreference.org/ref/epsg/wgs-84/).
The Geocoder functions can return different types of geometries (points or polygons) as result of different geocoding processes. The CartoDB platform does not support multigeometry layers or datasets, therefore the final users of this Data Services API must check that they are using consistent geometry types inside a table to avoid further conflicts in the map visualization.
## Authentication
All requests performed to the CartoDB Data Services API must be authenticated with the user API Key. For more information about where to find your API Key, and how to authenticate your SQL API requests, view the [SQL API authentication](/cartodb-platform/sql-api/authentication/) guide.
## Errors
Errors are described in the response of the request. An example is as follows:
```json
{
error: [
"The api_key must be provided"
]
}
```
Since the Data Services API is used on top of the CartoDB SQL API, you can refer to the [Making calls to the SQL API](/cartodb-platform/sql-api/making-calls/) documentation for help debugging your SQL errors.
If the requested information is not in the CartoDB geocoding database, or if CartoDB is unable to recognize your input and match it with a result, the geocoding function returns `null` as a result.
## Limits
Usage of the Data Services API is subject to the CartoDB SQL API limits, stated in our [Terms of Service](https://cartodb.com/terms/#excessive).

View File

@@ -1,14 +1,31 @@
# Geocoding Functions
The [geocoder](https://cartodb.com/data/geocoder-api/) functions allow you to match your data with geometries on your map. This geocoding service can be used programatically to geocode datasets via the CartoDB SQL API. It is fed from _Open Data_ and it serves geometries for countries, provinces, states, cities, postal codes, IP addresses and street addresses. CartoDB provides functions for several different categories of geocoding through the Data Services API.
_**This service is subject to quota limitations, and extra fees may apply**. View the [Quota Information](http://docs.cartodb.com/cartodb-platform/dataservices-api/quota-information/) section for details, and recommendations, about to quota consumption._
Here is an example of how to geocode a single country:
```bash
https://{username}.cartodb.com/api/v2/sql?q=SELECT cdb_geocode_admin0_polygon('USA')&api_key={api_key}
```
In order to geocode an existent CartoDB dataset, an SQL UPDATE statement must be used to populate the geometry column in the dataset with the results of the Data Services API. For example, if the column where you are storing the country names for each one of our rows is called `country_column`, run the following statement in order to geocode the dataset:
```bash
https://{username}.cartodb.com/api/v2/sql?q=UPDATE {tablename} SET the_geom = cdb_geocode_admin0_
```
The following geocoding functions are available, grouped by categories.
## Country Geocoder
## Country geocoder
This function provides a country geocoding service. It recognizes the names of the different countries from different synonyms, such as their English name, their endonym, or their ISO2 or ISO3 codes.
This function geocodes your data into country border geometries. It recognizes the names of the different countries either by different synonyms (such as their English name or their endonym), or by ISO (ISO2 or ISO3) codes.
### cdb_geocode_admin0_polygon(_country_name text_)
Geocodes the text name of a country into a country_name geometry, displayed as polygon data.
#### Arguments
Name | Type | Description
@@ -34,12 +51,14 @@ UPDATE {tablename} SET the_geom = cdb_geocode_admin0_polygon({country_column})
```
## Level-1 Administrative regions geocoder
## Level-1 Administrative Regions Geocoder
The following functions provide a geocoding service for administrative regions of level 1 (or NUTS-1) such as states for the United States, *départements* in France or autonomous communities in Spain.
This function geocodes your data into polygon geometries for [Level 1](https://en.wikipedia.org/wiki/Table_of_administrative_divisions_by_country), or [NUTS-1](https://en.wikipedia.org/wiki/NUTS_1_statistical_regions_of_England), administrative divisions (or units) of countries. For example, a "state" in the United States, "départements" in France, or an autonomous community in Spain.
### cdb_geocode_admin1_polygon(_admin1_name text_)
Geocodes the name of the province/state into a Level-1 administrative region, displayed as a polygon geometry.
#### Arguments
Name | Type | Description
@@ -66,6 +85,8 @@ UPDATE {tablename} SET the_geom = cdb_geocode_admin1_polygon({province_column})
### cdb_geocode_admin1_polygon(_admin1_name text, country_name text_)
Geocodes the name of the province/state for a specified country into a Level-1 administrative region, displayed as a polygon geometry.
#### Arguments
Name | Type | Description
@@ -92,12 +113,14 @@ UPDATE {tablename} SET the_geom = cdb_geocode_admin1_polygon({province_column},
```
## City geocoder
## City Geocoder
The following functions provide a city geocoder service. It is recommended to use the more specific geocoding function -- the one that requires more parameters — in order for the result to be as accurate as possible when several cities share their name. If there are duplicate results for a city name, the city name with the highest population will be returned.
This function geocodes your data into point geometries for names of cities. It is recommended to use geocoding functions that require more defined parameters — this returns more accurate results when several cities have the same name. _If there are duplicate results for a city name, the city name with the highest population will be returned._
### cdb_geocode_namedplace_point(_city_name text_)
Geocodes the text name of a city into a named place geometry, displayed as point data.
#### Arguments
Name | Type | Description
@@ -124,6 +147,8 @@ UPDATE {tablename} SET the_geom = cdb_geocode_namedplace_point({city_column})
### cdb_geocode_namedplace_point(_city_name text, country_name text_)
Geocodes the text name of a city for a specified country into a named place point geometry.
#### Arguments
Name | Type | Description
@@ -151,6 +176,7 @@ UPDATE {tablename} SET the_geom = cdb_geocode_namedplace_point({city_column}, 'S
### cdb_geocode_namedplace_point(_city_name text, admin1_name text, country_name text_)
Geocodes your data into a named place point geometry, containing the text name of a city, for a specified province/state and country. This is recommended for the most accurate geocoding of city data.
#### Arguments
Name | Type | Description
@@ -177,14 +203,16 @@ SELECT cdb_geocode_namedplace_point('New York', 'New York', 'USA')
UPDATE {tablename} SET the_geom = cdb_geocode_namedplace_point({city_column}, {province_column}, 'USA')
```
## Postal codes geocoder
## Postal Code Geocoder
The following functions provide a postal code geocoding service that can be used to obtain points or polygon results. The postal code polygon geocoder covers the United States, France, Australia and Canada; a request for a different country will return an empty response.
This function geocodes your data into point, or polygon, geometries for postal codes. The postal code polygon geocoder covers the United States, France, Australia and Canada; a request for a different country will return an empty response.
**Note:** For the USA, US Census [Zip Code Tabulation Areas](https://www.census.gov/geo/reference/zctas.html) (ZCTA) are used to reference geocodes for USPS postal codes service areas. See the [FAQs](http://docs.cartodb.com/faqs/datasets-and-data/#why-does-cartodb-use-census-bureau-zctas-and-not-usps-zip-codes-for-postal-codes) about datasets and data for details.
### cdb_geocode_postalcode_polygon(_postal_code text, country_name text_)
Goecodes the postal code for a specified country into a **polygon** geometry.
#### Arguments
Name | Type | Description
@@ -212,6 +240,8 @@ UPDATE {tablename} SET the_geom = cdb_geocode_postalcode_polygon({postal_code_co
### cdb_geocode_postalcode_point(_code text, country_name text_)
Goecodes the postal code for a specified country into a **point** geometry.
#### Arguments
Name | Type | Description
@@ -237,12 +267,14 @@ SELECT cdb_geocode_postalcode_point('11211', 'USA')
UPDATE {tablename} SET the_geom = cdb_geocode_postalcode_point({postal_code_column}, 'USA')
```
## IP addresses geocoder
## IP Addresses Geocoder
This function provides an IP address geocoding service, for both IPv4 and IPv6 addresses.
This function geocodes your data into point geometries for IP addresses. This is useful if you are analyzing location based data, based on a set of user's IP addresses.
### cdb_geocode_ipaddress_point(_ip_address text_)
Geocodes a postal code from a specified country into an IP address, displayed as a point geometry.
#### Arguments
Name | Type | Description
@@ -267,22 +299,25 @@ SELECT cdb_geocode_ipaddress_point('102.23.34.1')
```bash
UPDATE {tablename} SET the_geom = cdb_geocode_ipaddress_point('102.23.34.1')
```
## Street-level geocoder
This function provides a street-level geocoding service. This service uses the street level geocoder defined for the user.
## Street-Level Geocoder
**This service is subject to quota limitations, and extra fees may apply**. Please view our [terms and conditions](https://cartodb.com/terms/) and check out the [Quota information section](http://docs.cartodb.com/cartodb-platform/dataservices-api/quota-information/) for details and recommendations related with quota usage.
This function geocodes your data into a point geometry for a street address. CartoDB uses several different service providers for street-level geocoding, depending on your platform. If you access CartoDB on a Google Cloud Platform, [Google Maps geocoding](https://developers.google.com/maps/documentation/geocoding/intro) is applied. All other platform users are provided with [HERE geocoding services](https://developer.here.com/rest-apis/documentation/geocoder/topics/quick-start.html). Additional service providers will be implemented in the future.
**This service is subject to quota limitations, and extra fees may apply**. View the [Quota information](http://docs.cartodb.com/cartodb-platform/dataservices-api/quota-information/) for details and recommendations about quota consumption.
### cdb_geocode_street_point(_search_text text, [city text], [state text], [country text]_)
Geocodes a complete address into a single street geometry, displayed as point data.
#### Arguments
Name | Type | Description
--- | --- | ---
`searchtext` | `text` | searchtext contains free-form text containing address elements. You can specify the searchtext parameter by itself, or you can specify it with other parameters to narrow your search. For example, you can specify the state or country parameters, along with a free-form address in the searchtext field.
`city` | `text` | (Optional) Name of the city
`state` | `text` | (Optional) Name of the state
`country` | `text` | (Optional) Name of the country
--- | --- | --- | ---
`searchtext` | `text` | searchtext contains free-form text containing address elements. You can specify the searchtext parameter by itself, or with other parameters, to narrow your search. For example, you can specify the state or country parameters, along with a free-form address in the searchtext field.
`city` | `text` | (Optional) Name of the city.
`state` | `text` | (Optional) Name of the state.
`country` | `text` | (Optional) Name of the country.
#### Returns
@@ -292,6 +327,8 @@ Geometry (point, EPSG 4326) or null
##### Select
Using SELECT for geocoding functions
```bash
SELECT cdb_geocode_street_point('651 Lombard Street, San Francisco, California, United States')
SELECT cdb_geocode_street_point('651 Lombard Street', 'San Francisco')

View File

@@ -1,10 +1,27 @@
# Isoline Functions
The following functions provide an isolines generator service based on time or distance. This service uses the isolines service defined for the user (currently, only the [HERE](https://developer.here.com/coverage-info) isolines service is available).
[Isolines](https://cartodb.com/data/isolines/) are contoured lines that display equally calculated levels over a given surface area. This enables you to view polygon dimensions by forward or reverse measurements. Isoline functions are calculated as the intersection of areas from the origin point, measured by distance (isodistance) or time (isochrone). For example, the distance of a road from a sidewalk. Isoline services through CartoDB are available by requesting a single function in the Data Services API.
**This service is subject to quota limitations, and extra fees may apply**. Please view our [terms and conditions](https://cartodb.com/terms/) and check out the [Quota information section](http://docs.cartodb.com/cartodb-platform/dataservices-api/quota-information/) for details and recommendations related with quota usage.
_**This service is subject to quota limitations, and extra fees may apply**. View the [Quota Information](http://docs.cartodb.com/cartodb-platform/dataservices-api/quota-information/) section for details, and recommendations, about to quota consumption._
### cdb_isodistance(_source geometry, mode text, range integer[], [options text[]]_)
You can use the isoline functions to retrieve, for example, isochrone lines from a certain location, specifying the mode and the ranges that will define each of the isolines. The following query calculates isolines for areas that are 5, 10 and 15 minutes (300, 600 and 900 seconds, respectively) away from the location by following a path defined by car routing.
polygon({country_column})&api_key={api_key}
```bash
https://{username}.cartodb.com/api/v2/sql?q=SELECT cdb_isochrone('POINT(-3.70568 40.42028)'::geometry, 'car', ARRAY[300,600,900]::integer[])&api_key={api_key}
```
Notice that you can make use of Postgres or PostGIS functions in your Data Services API requests, as the result is a geometry that can be handled by the system. For example, suppose you need to retrieve the centroid of a specific country, you can wrap the resulting geometry from the geocoder functions inside the PostGIS `ST_Centroid` function:
```bash
https://{username}.cartodb.com/api/v2/sql?q=SELECT ST_Centroid(cdb_geocode_admin0_polygon('USA'))&api_key={api_key}
```
The following functions provide an isoline generator service, based on time or distance. This service uses the isolines service defined for your account. The default service limits the usage of displayed polygons represented on top of [HERE](https://developer.here.com/coverage-info) maps.
## cdb_isodistance(_source geometry, mode text, range integer[], [options text[]]_)
Displays a contoured line on a map, connecting geometries to a defined area, measured by an equal range of distance (in meters).
#### Arguments
@@ -48,7 +65,9 @@ SELECT the_geom FROM cdb_isodistance('POINT(-3.70568 40.42028)'::geometry, 'walk
INSERT INTO {table} (the_geom) SELECT (cdb_isodistance(the_geom, 'walk', string_to_array(distance, ',')::integer[])).the_geom FROM {points_table}
```
### cdb_isochrone(_source geometry, mode text, range integer[], [options text[]]_)
## cdb_isochrone(_source geometry, mode text, range integer[], [options text[]]_)
Displays a contoured line on a map, connecting geometries to a defined area, measured by an equal range of time (in seconds).
#### Arguments

41
doc/overview.md Normal file
View File

@@ -0,0 +1,41 @@
# Overview
By using CartoDB libraries and the SQL API, you can apply location data services to your maps with select functions. These functions are integrated with a number of internal and external services, enabling you to programatically customize subsets of data for your visualizations. For example, you can transform an address to a geometry with [geocoding functions](http://docs.cartodb.com/cartodb-platform/dataservices-api/geocoding-functions/#geocoding-functions). You can also use [isoline functions](http://docs.cartodb.com/cartodb-platform/dataservices-api/isoline-functions/#isoline-functions) to calculate the travel distance for a defined area. These features are useful for geospatial analysis and the results can be saved, and stored, for additional location data service operations.
**Note:** Based on your account plan, some of these data services are subject to different [quota limitations](http://docs.cartodb.com/cartodb-platform/dataservices-api/quota-information/#quota-information).
_The Data Services API is collaborating with [Mapzen](https://mapzen.com/), and several other geospatial service providers, in order to supply the best location data services from within our CartoDB Platform._
## Data Services Integration
By using the SQL API to query the Data Services API functions, you can manage specific operations and the corresponding geometries (a `polygon` or a `point`), according to the input information.
The Data Services API decouples the geocoding and isoline services from the CartoDB Editor. The API allows you to geocode data (from single rows, complete datasets, or simple inputs) and to perform trade areas analysis (computing isodistances or isochrones) programatically, through authenticated requests.
The geometries provided by this API are projected in the projection [WGS 84 SRID 4326](http://spatialreference.org/ref/epsg/wgs-84/).
**Note:** The Data Services API [geocoding functions](http://docs.cartodb.com/cartodb-platform/dataservices-api/geocoding-functions/#geocoding-functions) return different types of geometries (points or polygons) as result of different geocoding processes. The CartoDB Platform does not support multi-geometry layers or datasets, therefore you must confirm that you are using consistent geometry types inside a table, to avoid future conflicts in your map visualization.
## Authentication
All requests performed to the CartoDB Data Services API must be authenticated with the user API Key. For more information about where to find your API Key, and how to authenticate your SQL API requests, view the [SQL API authentication](/cartodb-platform/sql-api/authentication/) documentation.
## Errors
Errors are described in the response of the request. An example is as follows:
```json
{
error: [
"The api_key must be provided"
]
}
```
Since the Data Services API is used on top of the CartoDB SQL API, you can refer to the [Making calls to the SQL API](/cartodb-platform/sql-api/making-calls/) documentation for help debugging your SQL errors.
If the requested information is not in the CartoDB geocoding database, or if CartoDB is unable to recognize your input and match it with a result, the geocoding function returns `null` as a result.
## Limits
Usage of the Data Services API is subject to the CartoDB SQL API limits, stated in our [Terms of Service](https://cartodb.com/terms/#excessive).

View File

@@ -1,26 +0,0 @@
# Quickstart
If you are using the set of APIs and libraries that CartoDB offers, and you are handling your data with the SQL API, you can make your data visible in your maps by geocoding the dataset programatically. The set of isoline functions allow you to calculate the area which can be reached by travelling a given distance or time, useful for geospatial analysis, such as Trade Area Analysis.
Here is an example of how to geocode a single country:
```bash
https://{username}.cartodb.com/api/v2/sql?q=SELECT cdb_geocode_admin0_polygon('USA')&api_key={Your API key}
```
In order to geocode an existent CartoDB dataset, an SQL UPDATE statement must be used to populate the geometry column in the dataset with the results of the Geocoding API. For example, if the column where you are storing the country names for each one of our rows is called `country_column`, run the following statement in order to geocode the dataset:
```bash
https://{username}.cartodb.com/api/v2/sql?q=UPDATE {tablename} SET the_geom = cdb_geocode_admin0_polygon({country_column})&api_key={Your API key}
```
You can use the isoline functions to retrieve, for example, isochrone lines from a certain location, specifying the mode and the ranges that will define each of the isolines. The following query calculates isolines for areas that are 5, 10 and 15 minutes (300, 600 and 900 seconds, respectively) away from the location by following a path defined by car routing.
```bash
https://{username}.cartodb.com/api/v2/sql?q=SELECT cdb_isochrone('POINT(-3.70568 40.42028)'::geometry, 'car', ARRAY[300,600,900]::integer[])&api_key={Your API key}
```
Notice that you can make use of Postgres or PostGIS functions in your Data Services API requests, as the result is a geometry that can be handled by the system. For example, suppose you need to retrieve the centroid of a specific country, you can wrap the resulting geometry from the geocoder functions inside the PostGIS `ST_Centroid` function:
```bash
https://{username}.cartodb.com/api/v2/sql?q=SELECT ST_Centroid(cdb_geocode_admin0_polygon('USA'))&api_key={Your API key}
```

View File

@@ -1,17 +1,11 @@
# Quota Information
**This Data Services API provides functions which are subject to quota limitations, and extra fees may apply**. Please check our [terms and conditions](https://cartodb.com/terms/).
**Based on your account plan, some of the Data Services API functions are subject to quota limitations and extra fees may apply.** View our [terms and conditions](https://cartodb.com/terms/), or [contact us](mailto:sales@cartodb.com) for details about which functions require service credits to your account.
The functions that require service credits to be used are as follows:
## Quota Consumption
* cdb_geocode_street_point(_search_text text, [city text], [state text], [country text]_); from [Geocoding functions](http://docs.cartodb.com/cartodb-platform/dataservices-api/geocoding-functions/)
* cdb_isodistance(_source geometry, mode text, range integer[], [options text[]]_); from [Isoline functions](http://docs.cartodb.com/cartodb-platform/dataservices-api/isoline-functions/)
* cdb_isochrone(_source geometry, mode text, range integer[], [options text[]]_); from [Isoline functions](http://docs.cartodb.com/cartodb-platform/dataservices-api/isoline-functions/)
Quota consumption is calculated based on the number of request made for each function. Be mindful of the following usage recommendations when using the Data Services API functions:
## Quota consumption information and usage recommendations
Be mindful of the following when using the abovementioned functions:
* One credit per function call will be consumed, and the results are not cached. If the query is applied to a _N_ rows dataset, then _N_ credits will be used.
* You are discouraged from using dynamic queries to these functions in your maps. This can result in credits consumption per map view. **Note:** Queries to the Data Services API and any of its functions in your maps may be forbidden in the future.
* You are advised to store results of these queries into your datasets and refresh them as needed, so that you can have finer control on your credits' usage.
* One credit per function call will be consumed. The results are not cached. If the query is applied to a _N_ rows dataset, then _N_ credits are consumed
* Avoid running dynamic queries to these functions in your maps. This can result in credit consumption per map view. **Note:** Queries to the Data Services API, and any of its functions in your maps, may be forbidden in the future
* It is advised to store results of these queries into your datasets, and refresh them as needed. This ensure more control of quota credits for your account

45
doc/routing_functions.md Normal file
View File

@@ -0,0 +1,45 @@
# Routing Functions
Routing is the navigation from a defined start location to a defined end location. The calculated results are displayed as turn-by-turn directions on your map, based on the transportation mode that you specified. Routing services through CartoDB are available by requesting a single function in the Data Services API.
### cdb_route_point_to_point(_origin geometry(Point), destination geometry(Point), mode text, [options text[], units text]_)
#### Arguments
Name | Type | Description | Accepted values
--- | --- | --- | ---
`origin` | `geometry(Point)` | Origin point, in 4326 projection, which defines the start location. |
`destination` | `geometry(Point)` | Destination point, in 4326 projection, which defines the end location. |
`mode` | `text` | Type of transport used to calculate the isolines. | `car`, `walk`, `bicycle` or `public_transport`
`options` | `text[]` | (Optional) Multiple options to add more capabilities to the analysis. See [Optional routing parameters](#optional-routing-parameters) for details.
`units` | `text` | Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`
#### Returns
Name | Type | Description
--- | --- | ---
`duration` | `integer` | Duration in seconds of the calculated route.
`length` | `real` | Length in the defined unit in the `units` field. `kilometers` by default .
`the_geom` | `geometry(LineString)` | LineString geometry of the calculated route in the 4326 projection.
#### Examples
##### Insert the values from the calculated route in your table
```bash
INSERT INTO <TABLE> (duration, length, the_geom) SELECT duration, length, shape FROM cdb_route_point_to_point('POINT(-3.70237112 40.41706163)'::geometry,'POINT(-3.69909883 40.41236875)'::geometry, 'car')
```
##### Update the geometry field with the calculated route shape
```bash
UPDATE <TABLE> SET the_geom = (SELECT shape FROM cdb_route_point_to_point('POINT(-3.70237112 40.41706163)'::geometry,'POINT(-3.69909883 40.41236875)'::geometry, 'car', ARRAY['mode_type=shortest']::text[]))
```
### Optional routing parameters
The optional value parameters must be passed using the format: `option=value`.
Name | Type | Description | Accepted values
--- | --- | --- | ---
`mode_type` | `text` | Type of route calculation | `shortest` (this option only applies to the car transport mode)

View File

@@ -13,8 +13,8 @@ OLD_VERSIONS = $(wildcard old_versions/*.sql)
# @see http://www.postgresql.org/docs/current/static/extend-pgxs.html
DATA = $(NEW_EXTENSION_ARTIFACT) \
$(OLD_VERSIONS) \
cdb_dataservices_server--0.6.2--0.7.0.sql \
cdb_dataservices_server--0.7.0--0.6.2.sql
cdb_dataservices_server--0.7.0--0.7.1.sql \
cdb_dataservices_server--0.7.1--0.7.0.sql
REGRESS = $(notdir $(basename $(wildcard test/sql/*test.sql)))
TEST_DIR = test/

View File

@@ -0,0 +1,94 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.7.1'" to load this file. \quit
DROP FUNCTION IF EXISTS cdb_dataservices_server._get_data_observatory_config(text, text);
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_obs_snapshot_config(username text, orgname text)
RETURNS boolean AS $$
cache_key = "user_obs_snapshot_config_{0}".format(username)
if cache_key in GD:
return False
else:
from cartodb_services.metrics import ObservatorySnapshotConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection']
obs_snapshot_config = ObservatorySnapshotConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = obs_snapshot_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_demographic_snapshot(
username TEXT,
orgname TEXT,
geom geometry(Geometry, 4326),
time_span TEXT DEFAULT '2009 - 2013',
geometry_level TEXT DEFAULT '"us.census.tiger".block_group')
RETURNS json AS $$
from cartodb_services.metrics import QuotaService
import json
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reached the limit of your quota')
try:
obs_plan = plpy.prepare("SELECT cdb_observatory.OBS_GetDemographicSnapshot($1, $2, $3) as snapshot;", ["geometry(Geometry, 4326)", "text", "text"])
result = plpy.execute(obs_plan, [geom, time_span, geometry_level])
if result:
quota_service.increment_success_service_use()
return result[0]['snapshot']
else:
quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys, traceback
type_, value_, traceback_ = sys.exc_info()
quota_service.increment_failed_service_use()
error_msg = 'There was an error trying to use get_geographic_snapshot: {0}'.format(e)
plpy.notice(traceback.format_tb(traceback_))
plpy.error(error_msg)
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_segment_snapshot(
username TEXT,
orgname TEXT,
geom geometry(Geometry, 4326),
geometry_level TEXT DEFAULT '"us.census.tiger".block_group')
RETURNS json AS $$
from cartodb_services.metrics import QuotaService
import json
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reached the limit of your quota')
try:
obs_plan = plpy.prepare("SELECT cdb_observatory.OBS_GetSegmentSnapshot($1, $2) as snapshot;", ["geometry(Geometry, 4326)", "text"])
result = plpy.execute(obs_plan, [geom, geometry_level])
if result:
quota_service.increment_success_service_use()
return result[0]['snapshot']
else:
quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys, traceback
type_, value_, traceback_ = sys.exc_info()
quota_service.increment_failed_service_use()
error_msg = 'There was an error trying to use get_segment_snapshot: {0}'.format(e)
plpy.notice(traceback.format_tb(traceback_))
plpy.error(error_msg)
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;

View File

@@ -0,0 +1,94 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.7.0'" to load this file. \quit
DROP FUNCTION IF EXISTS cdb_dataservices_server._get_obs_snapshot_config(text, text);
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_data_observatory_config(username text, orgname text)
RETURNS boolean AS $$
cache_key = "user_data_observatory_config_{0}".format(username)
if cache_key in GD:
return False
else:
from cartodb_services.metrics import DataObservatoryConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection']
data_observatory_config = DataObservatoryConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = data_observatory_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_demographic_snapshot(
username TEXT,
orgname TEXT,
geom geometry(Geometry, 4326),
time_span TEXT DEFAULT '2009 - 2013',
geometry_level TEXT DEFAULT '"us.census.tiger".block_group')
RETURNS json AS $$
from cartodb_services.metrics import QuotaService
import json
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_data_observatory_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_data_observatory_config = GD["user_data_observatory_config_{0}".format(username)]
quota_service = QuotaService(user_data_observatory_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reached the limit of your quota')
try:
obs_plan = plpy.prepare("SELECT cdb_observatory.OBS_GetDemographicSnapshot($1, $2, $3) as snapshot;", ["geometry(Geometry, 4326)", "text", "text"])
result = plpy.execute(obs_plan, [geom, time_span, geometry_level])
if result:
quota_service.increment_success_service_use()
return result[0]['snapshot']
else:
quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys, traceback
type_, value_, traceback_ = sys.exc_info()
quota_service.increment_failed_service_use()
error_msg = 'There was an error trying to use get_geographic_snapshot: {0}'.format(e)
plpy.notice(traceback.format_exception(type_, value_, traceback_))
plpy.error(error_msg)
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_segment_snapshot(
username TEXT,
orgname TEXT,
geom geometry(Geometry, 4326),
geometry_level TEXT DEFAULT '"us.census.tiger".block_group')
RETURNS json AS $$
from cartodb_services.metrics import QuotaService
import json
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_data_observatory_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_data_observatory_config = GD["user_data_observatory_config_{0}".format(username)]
quota_service = QuotaService(user_data_observatory_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reached the limit of your quota')
try:
obs_plan = plpy.prepare("SELECT cdb_observatory.OBS_GetSegmentSnapshot($1, $2) as snapshot;", ["geometry(Geometry, 4326)", "text"])
result = plpy.execute(obs_plan, [geom, geometry_level])
if result:
quota_service.increment_success_service_use()
return result[0]['snapshot']
else:
quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys, traceback
type_, value_, traceback_ = sys.exc_info()
quota_service.increment_failed_service_use()
error_msg = 'There was an error trying to use get_segment_snapshot: {0}'.format(e)
plpy.notice(traceback.format_tb(traceback_))
plpy.error(error_msg)
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,5 @@
comment = 'CartoDB dataservices server extension'
default_version = '0.7.0'
default_version = '0.7.1'
requires = 'plpythonu, postgis, cdb_geocoder'
superuser = true
schema = cdb_dataservices_server

View File

@@ -10,10 +10,10 @@ RETURNS json AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_data_observatory_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_data_observatory_config = GD["user_data_observatory_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
quota_service = QuotaService(user_data_observatory_config, redis_conn)
quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reached the limit of your quota')
@@ -48,10 +48,10 @@ RETURNS json AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_data_observatory_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_data_observatory_config = GD["user_data_observatory_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_obs_snapshot_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_snapshot_config = GD["user_obs_snapshot_config_{0}".format(username)]
quota_service = QuotaService(user_data_observatory_config, redis_conn)
quota_service = QuotaService(user_obs_snapshot_config, redis_conn)
if not quota_service.check_user_quota():
plpy.error('You have reached the limit of your quota')

View File

@@ -54,16 +54,16 @@ RETURNS boolean AS $$
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_data_observatory_config(username text, orgname text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_obs_snapshot_config(username text, orgname text)
RETURNS boolean AS $$
cache_key = "user_data_observatory_config_{0}".format(username)
cache_key = "user_obs_snapshot_config_{0}".format(username)
if cache_key in GD:
return False
else:
from cartodb_services.metrics import DataObservatoryConfig
from cartodb_services.metrics import ObservatorySnapshotConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metadata_connection']
data_observatory_config = DataObservatoryConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = data_observatory_config
obs_snapshot_config = ObservatorySnapshotConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = obs_snapshot_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;

View File

@@ -1,3 +1,3 @@
from config import GeocoderConfig, IsolinesRoutingConfig, InternalGeocoderConfig, RoutingConfig, ConfigException, DataObservatoryConfig
from config import GeocoderConfig, IsolinesRoutingConfig, InternalGeocoderConfig, RoutingConfig, ConfigException, ObservatorySnapshotConfig
from quota import QuotaService
from user import UserMetricsService

View File

@@ -32,19 +32,31 @@ class ServiceConfig(object):
def organization(self):
return self._orgname
class DataObservatoryConfig(ServiceConfig):
class ObservatorySnapshotConfig(ServiceConfig):
SOFT_LIMIT_KEY = 'soft_obs_snapshot_limit'
QUOTA_KEY = 'obs_snapshot_quota'
PERIOD_END_DATE = 'period_end_date'
def __init__(self, redis_connection, db_conn, username, orgname=None):
super(DataObservatoryConfig, self).__init__(redis_connection, db_conn,
super(ObservatorySnapshotConfig, self).__init__(redis_connection, db_conn,
username, orgname)
self._monthly_quota = self._db_config.data_observatory_monthly_quota
self._period_end_date = date_parse(self._redis_config[self.PERIOD_END_DATE])
if self.SOFT_LIMIT_KEY in self._redis_config and self._redis_config[self.SOFT_LIMIT_KEY].lower() == 'true':
self._soft_limit = True
else:
self._soft_limit = False
# Mixed config between db and redis. If we don't update all the users
# in redis, we could use the db value as default
if self.QUOTA_KEY in self._redis_config:
self._monthly_quota = float(self._redis_config[self.QUOTA_KEY])
else:
self._monthly_quota = float(self._db_config.data_observatory_monthly_quota)
@property
def service_type(self):
return 'data_observatory'
return 'obs_snapshot'
@property
def monthly_quota(self):
@@ -54,6 +66,10 @@ class DataObservatoryConfig(ServiceConfig):
def period_end_date(self):
return self._period_end_date
@property
def soft_limit(self):
return self._soft_limit
class RoutingConfig(ServiceConfig):
@@ -200,28 +216,6 @@ class GeocoderConfig(ServiceConfig):
self.__parse_config(filtered_config, self._db_config)
self.__check_config(filtered_config)
def __get_user_config(self, username, orgname):
user_config = self._redis_connection.hgetall(
"rails:users:{0}".format(username))
if not user_config:
raise ConfigException("""There is no user config available. Please check your configuration.'""")
else:
if orgname:
self.__get_organization_config(orgname, user_config)
return user_config
def __get_organization_config(self, orgname, user_config):
org_config = self._redis_connection.hgetall(
"rails:orgs:{0}".format(orgname))
if not org_config:
raise ConfigException("""There is no organization config available. Please check your configuration.'""")
else:
user_config[self.QUOTA_KEY] = org_config[self.QUOTA_KEY]
user_config[self.PERIOD_END_DATE] = org_config[self.PERIOD_END_DATE]
user_config[self.GOOGLE_GEOCODER_CLIENT_ID] = org_config[self.GOOGLE_GEOCODER_CLIENT_ID]
user_config[self.GOOGLE_GEOCODER_API_KEY] = org_config[self.GOOGLE_GEOCODER_API_KEY]
def __check_config(self, filtered_config):
if filtered_config[self.GEOCODER_TYPE].lower() == self.NOKIA_GEOCODER:
if not set(self.NOKIA_GEOCODER_REDIS_MANDATORY_KEYS).issubset(set(filtered_config.keys())) or \
@@ -438,24 +432,24 @@ class ServicesRedisConfig:
GOOGLE_GEOCODER_CLIENT_ID = 'google_maps_client_id'
QUOTA_KEY = 'geocoding_quota'
ISOLINES_QUOTA_KEY = 'here_isolines_quota'
OBS_SNAPSHOT_QUOTA_KEY = 'obs_snapshot_quota'
PERIOD_END_DATE = 'period_end_date'
def __init__(self, redis_conn):
self._redis_connection = redis_conn
def build(self, username, password):
return self.__get_user_config(username, password)
def build(self, username, orgname):
return self.__get_user_config(username, orgname)
def __get_user_config(self, username, orgname):
user_config = self._redis_connection.hgetall(
"rails:users:{0}".format(username))
if not user_config:
raise ConfigException("""There is no user config available. Please check your configuration.'""")
else:
if orgname:
self.__get_organization_config(orgname, user_config)
elif orgname:
self.__get_organization_config(orgname, user_config)
return user_config
return user_config
def __get_organization_config(self, orgname, user_config):
org_config = self._redis_connection.hgetall(
@@ -465,6 +459,8 @@ class ServicesRedisConfig:
else:
user_config[self.QUOTA_KEY] = org_config[self.QUOTA_KEY]
user_config[self.ISOLINES_QUOTA_KEY] = org_config[self.ISOLINES_QUOTA_KEY]
if self.OBS_SNAPSHOT_QUOTA_KEY in org_config:
user_config[self.OBS_SNAPSHOT_QUOTA_KEY] = org_config[self.OBS_SNAPSHOT_QUOTA_KEY]
user_config[self.PERIOD_END_DATE] = org_config[self.PERIOD_END_DATE]
user_config[self.GOOGLE_GEOCODER_CLIENT_ID] = org_config[self.GOOGLE_GEOCODER_CLIENT_ID]
user_config[self.GOOGLE_GEOCODER_API_KEY] = org_config[self.GOOGLE_GEOCODER_API_KEY]

View File

@@ -73,9 +73,9 @@ class QuotaChecker:
elif re.match('routing_mapzen',
self._user_service_config.service_type) is not None:
return self.__check_routing_quota()
elif re.match('data_observatory',
elif re.match('obs_snapshot',
self._user_service_config.service_type) is not None:
return self.__check_data_observatory_quota()
return self.__check_obs_snapshot_quota()
else:
return False
@@ -118,7 +118,7 @@ class QuotaChecker:
else:
return False
def __check_data_observatory_quota(self):
def __check_obs_snapshot_quota(self):
user_quota = self._user_service_config.monthly_quota
today = date.today()
service_type = self._user_service_config.service_type

View File

@@ -10,7 +10,7 @@ from setuptools import setup, find_packages
setup(
name='cartodb_services',
version='0.5.0',
version='0.5.1',
description='CartoDB Services API Python Library',

View File

@@ -1,5 +1,5 @@
import test_helper
from cartodb_services.metrics import GeocoderConfig, ConfigException
from cartodb_services.metrics import GeocoderConfig, ObservatorySnapshotConfig, ConfigException
from unittest import TestCase
from nose.tools import assert_raises
from mockredis import MockRedis
@@ -32,10 +32,29 @@ class TestConfig(TestCase):
assert geocoder_config.soft_geocoding_limit is False
assert geocoder_config.period_end_date.date() == yesterday.date()
def test_should_return_config_for_obs_snapshot(self):
yesterday = datetime.today() - timedelta(days=1)
test_helper.build_redis_user_config(self.redis_conn, 'test_user',
do_quota=100, soft_do_limit=True,
end_date=yesterday)
do_config = ObservatorySnapshotConfig(self.redis_conn, self.plpy_mock,
'test_user')
assert do_config.monthly_quota == 100
assert do_config.soft_limit is True
assert do_config.period_end_date.date() == yesterday.date()
def test_should_return_db_quota_if_not_redis_quota_config_obs_snapshot(self):
yesterday = datetime.today() - timedelta(days=1)
test_helper.build_redis_user_config(self.redis_conn, 'test_user',
end_date=yesterday)
do_config = ObservatorySnapshotConfig(self.redis_conn, self.plpy_mock,
'test_user')
assert do_config.monthly_quota == 100000
assert do_config.soft_limit is False
assert do_config.period_end_date.date() == yesterday.date()
def test_should_raise_exception_when_missing_parameters(self):
plpy_mock = test_helper.build_plpy_mock(empty=True)
test_helper.build_redis_user_config(self.redis_conn, 'test_user')
assert_raises(ConfigException,
GeocoderConfig,
self.redis_conn, plpy_mock, 'test_user',
None)
assert_raises(ConfigException, GeocoderConfig, self.redis_conn,
plpy_mock, 'test_user', None)

View File

@@ -4,6 +4,7 @@ from mock import Mock
def build_redis_user_config(redis_conn, username, quota=100, soft_limit=False,
service="heremaps", isolines_quota=0,
do_quota=None, soft_do_limit=None,
end_date=datetime.today()):
user_redis_name = "rails:users:{0}".format(username)
redis_conn.hset(user_redis_name, 'soft_geocoding_limit', soft_limit)
@@ -11,23 +12,31 @@ def build_redis_user_config(redis_conn, username, quota=100, soft_limit=False,
redis_conn.hset(user_redis_name, 'here_isolines_quota', isolines_quota)
redis_conn.hset(user_redis_name, 'geocoder_type', service)
redis_conn.hset(user_redis_name, 'period_end_date', end_date)
if do_quota:
redis_conn.hset(user_redis_name, 'obs_snapshot_quota', do_quota)
if soft_do_limit:
redis_conn.hset(user_redis_name, 'soft_obs_snapshot_limit',
soft_do_limit)
redis_conn.hset(user_redis_name, 'google_maps_client_id', '')
redis_conn.hset(user_redis_name, 'google_maps_api_key', '')
def build_redis_org_config(redis_conn, orgname, quota=100, isolines_quota=0,
def build_redis_org_config(redis_conn, orgname, quota=100, service="heremaps",
isolines_quota=0, do_quota=None,
end_date=datetime.today()):
org_redis_name = "rails:orgs:{0}".format(orgname)
redis_conn.hset(org_redis_name, 'geocoding_quota', quota)
redis_conn.hset(org_redis_name, 'here_isolines_quota', isolines_quota)
if do_quota:
redis_conn.hset(org_redis_name, 'obs_snapshot_quota', do_quota)
redis_conn.hset(org_redis_name, 'period_end_date', end_date)
redis_conn.hset(org_redis_name, 'google_maps_client_id', '')
redis_conn.hset(org_redis_name, 'google_maps_api_key', '')
def increment_geocoder_uses(redis_conn, username, orgname=None,
date=date.today(), service='geocoder_here',
metric='success_responses', amount=20):
def increment_service_uses(redis_conn, username, orgname=None,
date=date.today(), service='geocoder_here',
metric='success_responses', amount=20):
prefix = 'org' if orgname else 'user'
entity_name = orgname if orgname else username
yearmonth = date.strftime('%Y%m')

View File

@@ -1,7 +1,7 @@
import test_helper
from mockredis import MockRedis
from cartodb_services.metrics import QuotaService
from cartodb_services.metrics import GeocoderConfig, RoutingConfig, DataObservatoryConfig
from cartodb_services.metrics import GeocoderConfig, RoutingConfig, ObservatorySnapshotConfig
from unittest import TestCase
from nose.tools import assert_raises
from datetime import datetime, date
@@ -28,40 +28,40 @@ class TestQuotaService(TestCase):
def test_should_return_true_if_user_quota_is_not_completely_used(self):
qs = self.__build_geocoder_quota_service('test_user')
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user')
test_helper.increment_service_uses(self.redis_conn, 'test_user')
assert qs.check_user_quota() is True
def test_should_return_true_if_org_quota_is_not_completely_used(self):
qs = self.__build_geocoder_quota_service('test_user',
orgname='test_org')
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
orgname='test_org')
test_helper.increment_service_uses(self.redis_conn, 'test_user',
orgname='test_org')
assert qs.check_user_quota() is True
def test_should_return_false_if_user_quota_is_surpassed(self):
qs = self.__build_geocoder_quota_service('test_user')
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
amount=300)
test_helper.increment_service_uses(self.redis_conn, 'test_user',
amount=300)
assert qs.check_user_quota() is False
def test_should_return_false_if_org_quota_is_surpassed(self):
qs = self.__build_geocoder_quota_service('test_user',
orgname='test_org')
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
orgname='test_org', amount=400)
test_helper.increment_service_uses(self.redis_conn, 'test_user',
orgname='test_org', amount=400)
assert qs.check_user_quota() is False
def test_should_return_true_if_user_quota_is_surpassed_but_soft_limit_is_enabled(self):
qs = self.__build_geocoder_quota_service('test_user', soft_limit=True)
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
amount=300)
test_helper.increment_service_uses(self.redis_conn, 'test_user',
amount=300)
assert qs.check_user_quota() is True
def test_should_return_true_if_org_quota_is_surpassed_but_soft_limit_is_enabled(self):
qs = self.__build_geocoder_quota_service('test_user',
orgname='test_org',
soft_limit=True)
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
test_helper.increment_service_uses(self.redis_conn, 'test_user',
orgname='test_org', amount=400)
assert qs.check_user_quota() is True
@@ -109,29 +109,34 @@ class TestQuotaService(TestCase):
qs.increment_success_service_use(amount=1500000)
assert qs.check_user_quota() is False
def test_should_check_user_data_observatory_quota_correctly(self):
qs = self.__build_data_observatory_quota_service('test_user')
def test_should_check_user_obs_snapshot_quota_correctly(self):
qs = self.__build_obs_snapshot_quota_service('test_user')
qs.increment_success_service_use()
assert qs.check_user_quota() is True
qs.increment_success_service_use(amount=100000)
assert qs.check_user_quota() is False
def test_should_check_org_data_observatory_quota_correctly(self):
qs = self.__build_data_observatory_quota_service('test_user', orgname='testorg')
def test_should_check_org_obs_snapshot_quota_correctly(self):
qs = self.__build_obs_snapshot_quota_service('test_user',
orgname='testorg')
qs.increment_success_service_use()
assert qs.check_user_quota() is True
qs.increment_success_service_use(amount=100000)
assert qs.check_user_quota() is False
def __prepare_quota_service(self, username, quota, service, orgname,
soft_limit, end_date):
soft_limit, do_quota, soft_do_limit, end_date):
test_helper.build_redis_user_config(self.redis_conn, username,
quota=quota, service=service,
soft_limit=soft_limit,
soft_do_limit=soft_do_limit,
do_quota=do_quota,
end_date=end_date)
if orgname:
test_helper.build_redis_org_config(self.redis_conn, orgname,
quota=quota, end_date=end_date)
quota=quota, service=service,
do_quota=do_quota,
end_date=end_date)
self._plpy_mock = test_helper.build_plpy_mock()
def __build_geocoder_quota_service(self, username, quota=100,
@@ -139,27 +144,27 @@ class TestQuotaService(TestCase):
soft_limit=False,
end_date=datetime.today()):
self.__prepare_quota_service(username, quota, service, orgname,
soft_limit, end_date)
soft_limit, 0, False, end_date)
geocoder_config = GeocoderConfig(self.redis_conn, self._plpy_mock,
username, orgname)
return QuotaService(geocoder_config, redis_connection=self.redis_conn)
def __build_routing_quota_service(self, username, quota=100,
service='routing_mapzen', orgname=None,
soft_limit=False,
end_date=datetime.today()):
def __build_routing_quota_service(self, username, service='routing_mapzen',
orgname=None, soft_limit=False,
quota=100, end_date=datetime.today()):
self.__prepare_quota_service(username, quota, service, orgname,
soft_limit, end_date)
soft_limit, 0, False, end_date)
routing_config = RoutingConfig(self.redis_conn, self._plpy_mock,
username, orgname)
return QuotaService(routing_config, redis_connection=self.redis_conn)
def __build_data_observatory_quota_service(self, username, quota=100,
service='data_observatory', orgname=None,
def __build_obs_snapshot_quota_service(self, username, quota=100,
service='obs_snapshot',
orgname=None,
soft_limit=False,
end_date=datetime.today()):
self.__prepare_quota_service(username, quota, service, orgname,
soft_limit, end_date)
do_config = DataObservatoryConfig(self.redis_conn, self._plpy_mock,
self.__prepare_quota_service(username, 0, service, orgname, False,
quota, soft_limit, end_date)
do_config = ObservatorySnapshotConfig(self.redis_conn, self._plpy_mock,
username, orgname)
return QuotaService(do_config, redis_connection=self.redis_conn)

View File

@@ -18,15 +18,15 @@ class TestUserService(TestCase):
def test_user_used_quota_for_a_day(self):
us = self.__build_user_service('test_user')
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
amount=400)
test_helper.increment_service_uses(self.redis_conn, 'test_user',
amount=400)
assert us.used_quota(self.NOKIA_GEOCODER, date.today()) == 400
def test_org_used_quota_for_a_day(self):
us = self.__build_user_service('test_user', orgname='test_org')
test_helper.increment_geocoder_uses(self.redis_conn, 'test_user',
orgname='test_org',
amount=400)
test_helper.increment_service_uses(self.redis_conn, 'test_user',
orgname='test_org',
amount=400)
assert us.used_quota(self.NOKIA_GEOCODER, date.today()) == 400
def test_user_not_amount_in_used_quota_for_month_should_be_0(self):

View File

@@ -15,30 +15,24 @@ class TestDataObservatoryFunctions(TestCase):
)
def test_if_get_demographic_snapshot_is_ok(self):
query = "SELECT duration, length, shape as the_geom " \
"FROM cdb_do_get_demographic_snapshot(CDB_LatLng(40.704512, -73.936669))".format(
self.env_variables['api_key'])
routing = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(routing['the_geom'], None)
query = "SELECT obs_get_demographic_snapshot(CDB_LatLng(40.704512, -73.936669)) as snapshot;&api_key={0}".format(self.env_variables['api_key'])
snapshot = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(snapshot['snapshot'], None)
def test_if_get_demographic_snapshot_without_api_key_raise_error(self):
query = "SELECT duration, length, shape as the_geom " \
"FROM cdb_do_get_demographic_snapshot(CDB_LatLng(40.704512, -73.936669))"
query = "SELECT obs_get_demographic_snapshot(CDB_LatLng(40.704512, -73.936669));"
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e:
assert_equal(e.message[0], "The api_key must be provided")
def test_if_get_segment_snapshot_is_ok(self):
query = "SELECT duration, length, shape as the_geom " \
"FROM cdb_do_get_segment_snapshot(CDB_LatLng(40.704512, -73.936669))".format(
self.env_variables['api_key'])
routing = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(routing['the_geom'], None)
query = "SELECT obs_get_segment_snapshot(CDB_LatLng(40.704512, -73.936669)) as snapshot;&api_key={0}".format(self.env_variables['api_key'])
snapshot = IntegrationTestHelper.execute_query(self.sql_api_url, query)
assert_not_equal(snapshot['snapshot'], None)
def test_if_get_segment_snapshot_without_api_key_raise_error(self):
query = "SELECT duration, length, shape as the_geom " \
"FROM cdb_do_get_segment_snapshot(CDB_LatLng(40.704512, -73.936669))"
query = "SELECT obs_get_segment_snapshot(CDB_LatLng(40.704512, -73.936669));"
try:
IntegrationTestHelper.execute_query(self.sql_api_url, query)
except Exception as e: