Merge pull request #464 from CartoDB/development

Release 0.16.7

NEWS.md

@@ -1,3 +1,22 @@
+
+February 13th, 2018
+==================
+* Version `0.16.7` of the python library
+    * Pick the first result when Mapbox geocoder returns multiple results #462
+    * Normalize input for Mapbox geocoder #462
+
+February 12th, 2018
+==================
+* Version `0.16.6` of the python library
+    * Using Mapbox permanent geocoder #455
+
+February 5th, 2018
+==================
+* Version `0.16.5` of the python library
+    * Fix displaced routing shape #443
+    * Check for empty coordinates object before converting it to polygon
+    * 422 errors that come from Mapbox now returns an empty result because is a bad input from the user data
+
 February 2th, 2018
 ==================
 * Version `0.16.4` of the python library

README.md

@@ -45,7 +45,7 @@ Steps to deploy a new Data Services API version :
 
   ```sql
   CREATE DATABASE dataservices_db ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
-  CREATE USER dataservices_user;
+  CREATE USER geocoder_api;
   ```
 
 - Install needed extensions in `dataservices_db` database
@@ -90,19 +90,23 @@ Steps to deploy a new Data Services API version :
      ```
      psql -U postgres -d dataservices_db -f src/pg/test/fixtures/load_fixtures.sql
      ```
-   - Give permission to execute and select to the `dataservices_user` user:
+   - Give permission to execute and select to the `geocoder_api` user:
      ```
      psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS observatory VERSION 'dev'; COMMIT" -e
-     psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
-     psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
-     psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO dataservices_user; COMMIT" -e
-     psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA observatory TO dataservices_user; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO geocoder_api; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO geocoder_api; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT USAGE ON SCHEMA cdb_observatory TO geocoder_api; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO geocoder_api; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA observatory TO geocoder_api; COMMIT" -e
+     psql -U postgres -d dataservices_db -c "BEGIN;GRANT USAGE ON SCHEMA observatory TO geocoder_api; COMMIT" -e
      ```
 
 ### Server configuration
 
 Configuration for the different services must be stored in the server database using `CDB_Conf_SetConf()`.
 
+**All the configuration inside brackets [] is optional**
+
 #### Redis configuration
 
 If sentinel is used:

doc/geocoding_functions.md

@@ -4,7 +4,7 @@ The [geocoder](https://carto.com/data/geocoder-api/) functions allow you to matc
 
 _**This service is subject to quota limitations and extra fees may apply**. View the [Quota Information](https://carto.com/docs/carto-engine/dataservices-api/quota-information/) section for details and recommendations about to quota consumption._
 
-Here is an example of how to geocode a single country:
+The following example displays how to geocode a single country:
 
 ```bash
 https://{username}.carto.com/api/v2/sql?q=SELECT cdb_geocode_admin0_polygon('USA')&api_key={api_key}
@@ -312,7 +312,7 @@ INSERT INTO {tablename} (the_geom) SELECT cdb_geocode_ipaddress_point('102.23.34
 
 ## Street-Level Geocoder
 
-This function geocodes your data into a point geometry for a street address. CARTO uses several different service providers for street-level geocoding, depending on your platform. If you access CARTO on a Google Cloud Platform, [Google Maps geocoding](https://developers.google.com/maps/documentation/geocoding/intro) is applied. All other platform users are provided with [HERE geocoding services](https://developer.here.com/rest-apis/documentation/geocoder/topics/quick-start.html). Additional service providers will be implemented in the future.
+This function geocodes your data into a point geometry for a street address. CARTO uses several different service providers for street-level geocoding, depending on your platform. If you access CARTO on a Google Cloud Platform, [Google Maps geocoding](https://developers.google.com/maps/documentation/geocoding/intro) is applied. All other platform users are provided with [Mapbox geocoding services](https://www.mapbox.com/). [Contact us](mailto:sales@carto.com) if you have any specific questions or requirements about the location data service provider being used with your account._.
 
 **This service is subject to quota limitations, and extra fees may apply**. View the [Quota information](https://carto.com/docs/carto-engine/dataservices-api/quota-information/) for details and recommendations about quota consumption.
 

doc/isoline_functions.md

@@ -10,7 +10,7 @@ You can use the isoline functions to retrieve, for example, isochrone lines from
 https://{username}.carto.com/api/v2/sql?q=INSERT INTO {table} (the_geom) SELECT  the_geom FROM cdb_isodistance('POINT(-3.70568 40.42028)'::geometry, 'car', ARRAY[300, 600, 900]::integer[])&api_key={api_key}
 ```
 
-The following functions provide an isoline generator service, based on time or distance. This service uses the isolines service defined for your account. The default service limits the usage of displayed polygons represented on top of [HERE](https://developer.here.com/coverage-info) maps.
+The following functions provide an isoline generator service, based on time or distance. This service uses the isolines service defined for your account. The default service limits the usage of displayed polygons represented on top of [Mapbox](https://www.mapbox.com/) maps.
 
 ## cdb_isodistance(_source geometry, mode text, range integer[], [options text[]]_)
 

doc/overview.md

@@ -4,7 +4,7 @@ By using CARTO libraries and the SQL API, you can apply location data services t
 
 **Note:** Based on your account plan, some of these data services are subject to different [quota limitations](https://carto.com/docs/carto-engine/dataservices-api/quota-information/#quota-information).
 
-_The Data Services API is collaborating with [Mapzen](https://mapzen.com/), and several other geospatial service providers, in order to supply the best location data services from within our CARTO Engine._
+_In order to supply the best location data services from within our CARTO Engine, the Data Services API collaborates with [Mapbox](https://www.mapbox.com/) and several other geospatial service providers. [Contact us](mailto:sales@carto.com) if you have any specific questions or requirements about the location data service provider being used with your account._
 
 ## Data Services Integration
 

doc/quota_information.md

@@ -59,9 +59,9 @@ Result:
 ```sql
     service     | monthly_quota | used_quota | soft_limit |     provider
 ----------------+---------------+------------+------------+------------------
- isolines       |           100 |          0 | f          | mapzen
- hires_geocoder |           100 |          0 | f          | mapzen
- routing        |            50 |          0 | f          | mapzen
+ isolines       |           100 |          0 | f          | mapbox
+ hires_geocoder |           100 |          0 | f          | mapbox
+ routing        |            50 |          0 | f          | mapbox
  observatory    |             0 |          0 | f          | data observatory
 (4 rows)
 
@@ -100,7 +100,7 @@ Suppose you want to geocode a whole table. In order to check that you have enoug
 SELECT COUNT(*) FROM {tablename} WHERE {street_name_column} IS NOT NULL;
 ```
 
-Result: here's a sample result of 10000 records:
+Result: A sample result of 10000 records:
 
 ```sql
  count

server/lib/python/cartodb_services/cartodb_services/mapbox/geocoder.py

@@ -8,11 +8,12 @@ from mapbox import Geocoder
 from cartodb_services.metrics import Traceable
 from cartodb_services.tools.exceptions import ServiceException
 from cartodb_services.tools.qps import qps_retry
+from cartodb_services.tools.normalize import normalize
 
 GEOCODER_NAME = 'geocoder_name'
 EPHEMERAL_GEOCODER = 'mapbox.places'
 PERMANENT_GEOCODER = 'mapbox.places-permanent'
-DEFAULT_GEOCODER = EPHEMERAL_GEOCODER
+DEFAULT_GEOCODER = PERMANENT_GEOCODER
 
 ENTRY_FEATURES = 'features'
 ENTRY_CENTER = 'center'
@@ -32,17 +33,23 @@ class MapboxGeocoder(Traceable):
         self._token = token
         self._logger = logger
         self._geocoder_name = service_params.get(GEOCODER_NAME,
-                                                 EPHEMERAL_GEOCODER)
+                                                 DEFAULT_GEOCODER)
         self._geocoder = Geocoder(access_token=self._token,
                                   name=self._geocoder_name)
 
     def _parse_geocoder_response(self, response):
         json_response = json.loads(response)
 
-        if json_response and json_response[ENTRY_FEATURES]:
-            feature = json_response[ENTRY_FEATURES][0]
+        # If Mapbox returns more that one result, take the first one
+        if json_response:
+            if type(json_response) == list:
+                json_response = json_response[0]
 
-            return self._extract_lng_lat_from_feature(feature)
+            if json_response[ENTRY_FEATURES]:
+                feature = json_response[ENTRY_FEATURES][0]
+                return self._extract_lng_lat_from_feature(feature)
+            else:
+                return []
         else:
             return []
 
@@ -61,11 +68,11 @@ class MapboxGeocoder(Traceable):
     def geocode(self, searchtext, city=None, state_province=None,
                 country=None):
         if searchtext and searchtext.strip():
-            address = [searchtext]
+            address = [normalize(searchtext)]
             if city:
-                address.append(city)
+                address.append(normalize(city))
             if state_province:
-                address.append(state_province)
+                address.append(normalize(state_province))
         else:
             return []
 
@@ -80,6 +87,8 @@ class MapboxGeocoder(Traceable):
                 return self._parse_geocoder_response(response.text)
             elif response.status_code == requests.codes.bad_request:
                 return []
+            elif response.status_code == requests.codes.unprocessable_entity:
+                return []
             else:
                 raise ServiceException(response.status_code, response)
         except requests.Timeout as te:

server/lib/python/cartodb_services/cartodb_services/mapbox/isolines.py

@@ -49,6 +49,8 @@ class MapboxIsolines():
         response = self._matrix_client.matrix([origin] + targets,
                                               profile)
         json_response = json.loads(response)
+        if not json_response:
+            return []
 
         costs = [None] * number_of_angles
 
@@ -125,6 +127,9 @@ class MapboxIsolines():
                                 unit_factor=unit_factor,
                                 number_of_angles=number_of_angles)
 
+            if not costs:
+                continue
+
             errors = [(cost - isorange) / float(isorange) for cost in costs]
             max_abs_error = max([abs(e) for e in errors])
             if max_abs_error <= tolerance:

server/lib/python/cartodb_services/cartodb_services/mapbox/matrix_client.py

@@ -72,6 +72,8 @@ class MapboxMatrixClient(Traceable):
                 return response.text
             elif response.status_code == requests.codes.bad_request:
                 return '{}'
+            elif response.status_code == requests.codes.unprocessable_entity:
+                return '{}'
             else:
                 raise ServiceException(response.status_code, response)
         except requests.Timeout as te:

server/lib/python/cartodb_services/cartodb_services/mapbox/routing.py

@@ -91,6 +91,8 @@ class MapboxRouting(Traceable):
                 return self._parse_routing_response(response.text)
             elif response.status_code == requests.codes.bad_request:
                 return MapboxRoutingResponse(None, None, None)
+            elif response.status_code == requests.codes.unprocessable_entity:
+                return MapboxRoutingResponse(None, None, None)
             else:
                 raise ServiceException(response.status_code, response)
         except requests.Timeout as te:

server/lib/python/cartodb_services/cartodb_services/tools/coordinates.py

@@ -45,6 +45,8 @@ def marshall_coordinates(coordinates):
 
 def coordinates_to_polygon(coordinates):
     """Convert a Coordinate array coordinates to a PostGIS polygon"""
+    if not coordinates:
+        return None
     coordinates.append(coordinates[0])  # Close the ring
     result_coordinates = []
     for coordinate in coordinates:

server/lib/python/cartodb_services/cartodb_services/tools/normalize.py

@@ -0,0 +1,3 @@
+def normalize(str_input):
+    return str_input.replace('"', '"') \
+                    .replace(';', ',')

server/lib/python/cartodb_services/cartodb_services/tools/polyline.py

@@ -51,9 +51,7 @@ def polyline_to_linestring(polyline):
     """Convert a Mapzen polyline shape to a PostGIS linestring"""
     coordinates = []
     for point in polyline:
-        # Divide by 10 because mapzen uses one more decimal than the
-        # google standard (https://mapzen.com/documentation/turn-by-turn/decoding/)
-        coordinates.append("%s %s" % (point[1]/10, point[0]/10))
+        coordinates.append("%s %s" % (point[1], point[0]))
     wkt_coordinates = ','.join(coordinates)
 
     try:

server/lib/python/cartodb_services/setup.py

@@ -10,7 +10,7 @@ from setuptools import setup, find_packages
 setup(
     name='cartodb_services',
 
-    version='0.16.4',
+    version='0.16.7',
 
     description='CartoDB Services API Python Library',
 

server/lib/python/cartodb_services/test/test_mapboxgeocoder.py

@@ -35,6 +35,11 @@ class MapboxGeocoderTestCase(unittest.TestCase):
 
         assert place
 
+    def test_odd_characters(self):
+        place = self.geocoder.geocode(searchtext='Barcelona; "Spain"')
+
+        assert place
+
     def test_empty_request(self):
         place = self.geocoder.geocode(searchtext='', country=None, city=None, state_province=None)