Compare commits

...

196 Commits

Author SHA1 Message Date
Mario de Frutos
462773a138 Merge pull request #453 from CartoDB/development
Release python library version 0.16.5
2018-02-05 09:39:47 +01:00
Mario de Frutos
fad2f25183 Updated NEWS.md 2018-02-05 09:35:40 +01:00
Mario de Frutos
2f54ef7e4e Merge pull request #452 from CartoDB/fix_misplaced_coordinate_polyline
Multiple fixes
2018-02-05 09:30:24 +01:00
Mario de Frutos
ab4d77edf2 Merge pull request #377 from CartoDB/logger_conf_wrong_square_brackets
Removed wrong square brackets
2018-02-02 18:16:36 +01:00
Mario de Frutos
0b12f26f47 Add info about square brackets in server configuration of the README 2018-02-02 18:15:40 +01:00
Mario de Frutos
a116aba660 Merge pull request #379 from CartoDB/missing_grant_usage
Missing GRANT USAGE
2018-02-02 18:11:47 +01:00
Mario de Frutos
29d5a332b6 Change to reflect the current used user for the DS
Check this issue https://github.com/CartoDB/dataservices-api/issues/380
which talks about the renaming for a more proper name
2018-02-02 18:10:02 +01:00
Juan Ignacio Sánchez Lara
b2825f46a4 Missing GRANT USAGE 2018-02-02 18:08:01 +01:00
Mario de Frutos
a791d02dcc 422 errors from mapbox return empty response 2018-02-02 14:11:22 +01:00
Mario de Frutos
e2612645c3 Bump to version 0.16.5 2018-02-02 13:18:47 +01:00
Mario de Frutos
c5fed2cc80 Check for empty coordinates when we transform to polygon 2018-02-02 13:18:32 +01:00
Mario de Frutos
a1f339376e By default the polyline comes with 6 decimals so we avoid to use 5 by default 2018-02-02 13:17:56 +01:00
Mario de Frutos
019df2b4e0 Merge pull request #451 from CartoDB/development
Hotfix circular dependencies exceptions
2018-02-02 12:38:23 +01:00
Mario de Frutos
9e67beab3d Merge pull request #450 from CartoDB/move_exceptions_to_avoid_circular_dependencies
Move exceptions to a proper place
2018-02-02 12:37:14 +01:00
Mario de Frutos
5c7d1e9ac1 Solve some dependencies bugs 2018-02-02 12:29:48 +01:00
Mario de Frutos
6e54507b5a Move exceptions to a proper place 2018-02-02 12:06:43 +01:00
Mario de Frutos
14925ea448 Add missing imports for google gecode street 2018-02-02 11:18:56 +01:00
Mario de Frutos
26e0a5e00f Merge pull request #448 from CartoDB/development
Release server 0.30.2 and python library 0.16.4
2018-02-02 10:37:16 +01:00
Mario de Frutos
a99e0cb513 Updated NEWS.md 2018-02-02 10:32:24 +01:00
Mario de Frutos
1e4fef5868 Merge pull request #447 from CartoDB/dont_send_quota_exceptions_rollbar
Avoid treating quota exception as bug exception
2018-02-02 10:29:04 +01:00
Mario de Frutos
3973483af2 Move limits check inside the try 2018-02-01 19:03:32 +01:00
Mario de Frutos
88edad9414 Release artifacts for version 0.30.2 2018-02-01 18:46:24 +01:00
Mario de Frutos
88ad551e4a Bump python version to 0.16.4 2018-02-01 18:40:21 +01:00
Mario de Frutos
94eaf84f91 Treat quota exception as a failure 2018-02-01 18:27:32 +01:00
Mario de Frutos
0df0b0d49c Avoid treating quota exception as bug exception 2018-02-01 18:21:15 +01:00
Mario de Frutos
b260c29b8c Merge pull request #445 from CartoDB/development
Updated routing documentation
2018-02-01 12:10:20 +01:00
Mario de Frutos
da8aa5a8c0 Merge pull request #444 from CartoDB/fix_routing_documentation
Change routing docs with the Mapbox addition as provider
2018-02-01 12:05:27 +01:00
Mario de Frutos
423534068a Change routing docs with the Mapbox addition as provider 2018-02-01 11:55:22 +01:00
Mario de Frutos
6f508b550d Merge pull request #441 from CartoDB/development
Release python library 0.16.3
2018-01-31 17:06:00 +01:00
Mario de Frutos
1462f87d97 Merge pull request #440 from CartoDB/439_filter_empty_requests_mapbox_geocoder
Improve empty requests treatment in geocoders
2018-01-31 16:31:11 +01:00
Mario de Frutos
6ae8aa45fd CR changes 2018-01-31 16:13:20 +01:00
Mario de Frutos
02cb4862f9 Update NEWS.md 2018-01-31 13:18:09 +01:00
Mario de Frutos
07f00cc0ae Added tests for the client/channel part of google 2018-01-31 13:17:36 +01:00
Mario de Frutos
544e0fa763 Going green with non empty request strings 2018-01-31 12:25:35 +01:00
Mario de Frutos
2bbd6bac91 Going red with non empty request string 2018-01-31 12:25:18 +01:00
Mario de Frutos
2f8dbbb5dc Dont raise exception when empty params passed to HERE geocoder 2018-01-31 12:16:13 +01:00
Mario de Frutos
3484cce88b Update NEWS.md 2018-01-31 12:13:02 +01:00
Mario de Frutos
56c90cc541 Bump version to 0.16.3 2018-01-31 12:12:52 +01:00
Mario de Frutos
3583cc6f47 Going green with empty requests that leads to 404 responses 2018-01-31 12:11:25 +01:00
Mario de Frutos
850dc09a4f Going green with empty responses 2018-01-31 12:11:25 +01:00
Mario de Frutos
876bae6b56 Going red with bad request parameters 2018-01-31 12:11:25 +01:00
Mario de Frutos
8f30359cc7 Added rate limits topic to the README 2018-01-30 15:02:40 +01:00
Mario de Frutos
7be27969fa Merge pull request #438 from CartoDB/development
Release python version 0.16.2
2018-01-30 09:58:49 +01:00
Mario de Frutos
502039796f Merge pull request #437 from CartoDB/436-Fixing_valid_types
Fixed valid Mapbox types
2018-01-30 09:42:50 +01:00
Antonio
e0b1632fa8 Fixed tests 2018-01-30 09:13:43 +01:00
Antonio
51bf6c2a43 Fixed valid Mapbox types 2018-01-29 18:39:11 +01:00
Mario de Frutos
5f9185ee6b Merge pull request #435 from CartoDB/development
Release 0.30.1 server side
2018-01-29 17:43:51 +01:00
Mario de Frutos
08e21e3999 Updated NEWS.md 2018-01-29 17:42:30 +01:00
Mario de Frutos
212cbda9a0 Merge pull request #434 from CartoDB/433-Fixing_encoding_issues
Fixing encoding issues
2018-01-29 17:40:12 +01:00
Antonio
05718ce58c Fixed KeyError Exception 2018-01-29 17:32:44 +01:00
Antonio
fed444ff6c Version bumped 2018-01-29 17:18:18 +01:00
Antonio
d1c8f8ced0 Fixing encoding issues 2018-01-29 17:09:17 +01:00
Mario de Frutos
96d2bf6218 Update NEWS.md 2018-01-18 17:27:34 +01:00
Mario de Frutos
b4dc616590 Merge pull request #432 from CartoDB/development
Release python library 0.16.1
2018-01-18 17:24:23 +01:00
Mario de Frutos
4f907053f5 Merge pull request #431 from CartoDB/430-Fixed_encoding_problems_with_mapbox_geocoding
Fixed encoding problem with Mapbox geocoding
2018-01-18 17:21:20 +01:00
Antonio
1a2785dff4 Version bumped 2018-01-18 15:21:06 +01:00
Antonio
d442fab9da Fixed encoding problem with Mapbox geocoding 2018-01-18 11:55:25 +01:00
Antonio
236837ebb6 Release January 17th 2018 2018-01-17 16:19:57 +01:00
Antonio Carlón
6edad7b810 Merge pull request #429 from CartoDB/development
Release server 0.30.0 , client 0.23.0 and python library 0.16.0
2018-01-17 16:05:00 +01:00
Mario de Frutos
7d8cc4ae87 Fix bug that checks only for name of the custom type and not uses the
namespace
2018-01-16 18:28:54 +01:00
Mario de Frutos
41916eed18 Merge pull request #428 from CartoDB/427-Fixed_country_codes_for_Mapbox
Added iso3166 import
2018-01-16 17:31:20 +01:00
Antonio
9921f0f394 Added iso3166 import 2018-01-16 17:25:12 +01:00
Mario de Frutos
72b99a1729 Merge pull request #426 from CartoDB/425-Fixed_country_code_problem
Fixed country code problem
2018-01-16 16:44:12 +01:00
Antonio
6c53846e33 Fixed country code problem. Added test 2018-01-16 16:36:24 +01:00
Mario de Frutos
5781f78c7f Revert "Removed suspect import"
This reverts commit 9c1ec1ddde.
2018-01-16 11:04:29 +01:00
Mario de Frutos
8127314965 Merge pull request #424 from CartoDB/423-Fix_staging_error
Removed suspect import
2018-01-16 10:18:13 +01:00
Antonio
9c1ec1ddde Removed suspect import 2018-01-16 10:02:41 +01:00
Antonio
d1d8713f05 Generated sql files 2018-01-15 11:22:34 +01:00
Antonio
0ed477d264 Generated sql files 2018-01-15 11:18:04 +01:00
Antonio Carlón
42abfd962c Merge pull request #420 from CartoDB/Clients_for_Mapbox_services
Python implementation of clients for Mapbox services
2018-01-15 11:08:57 +01:00
Antonio
61e8273a47 Removed unneccessary drop functions 2018-01-15 11:07:50 +01:00
Antonio
6339623ad9 Added ServiceManager for Mapbox services 2018-01-12 12:24:17 +01:00
elenatorro
ae32f91c7c Fix Logger config 2018-01-12 09:53:58 +01:00
Antonio
a02a634e76 Added metrics for DO (related to #419 and removed for easy merging) 2018-01-11 11:13:37 +01:00
Antonio
7dc87ad38e Rotating API keys for Mapbox 2018-01-11 10:18:01 +01:00
Mario de Frutos
af0125197d Merge pull request #422 from CartoDB/development
Release python 0.15.7
2018-01-10 16:36:47 +01:00
Antonio Carlón
b10f48166d Merge branch 'development' into Clients_for_Mapbox_services 2018-01-10 12:50:20 +01:00
Mario de Frutos
051453898e Merge pull request #419 from CartoDB/260-Tracking_OBS_usage
Logging GetMeta and GetData parameters
2018-01-10 12:49:13 +01:00
Mario de Frutos
6047300259 Move the new params parameter to the end to avoid breaking backward compatibility 2018-01-10 12:41:08 +01:00
Antonio
b282c7db5b Fixed tests 2018-01-10 12:33:10 +01:00
Mario de Frutos
84749a2631 Revert SQL changes just leave the python ones 2018-01-10 12:31:49 +01:00
Antonio
c0cc5fe6a7 Logging GetMeta and GetData parameters 2018-01-10 12:31:49 +01:00
Antonio
7d99ede28d Error fixed 2018-01-10 11:55:02 +01:00
Antonio
cd5dba920e Using user's configured geocoder for namedplaces via street point 2018-01-10 10:34:18 +01:00
Antonio
6d978deb16 Using user's configured geocoder for namedplaces via street point 2018-01-10 10:16:59 +01:00
Antonio
d3e01b4d53 Reverted default providers to mapzen to allow dark deployment 2018-01-09 17:15:03 +01:00
Antonio
dd6ad0119c Added CR suggestions 2018-01-09 16:21:55 +01:00
Antonio
3bbb3c6bcc Autogenerated sql file 2018-01-04 18:02:49 +01:00
Antonio
fa3f2fbc3a Modified SQL files 2018-01-04 17:58:46 +01:00
Antonio
dc5dbe8c0a Fixing tests 2018-01-04 16:01:32 +01:00
Antonio
e7c0628751 Solving errors and refactor 2018-01-04 15:00:52 +01:00
Antonio
51d97228dc Added CR suggestions 2018-01-04 12:30:58 +01:00
Antonio
cc34a8b19b Added Mapbox configuration to README.md 2018-01-03 14:19:49 +01:00
Antonio
801c5a3cee Solved data conversion issues 2018-01-03 14:14:25 +01:00
Antonio
941af522ae Bumped Python library version 2018-01-03 09:47:55 +01:00
Antonio
bb86f080fc Added limits/quotas and tests for Mapbox services 2018-01-02 16:21:57 +01:00
Antonio
15340fedad Added cdb_dataservices_client functions 2017-12-29 15:28:29 +01:00
Antonio
c46174210d Added cdb_dataservices_server functions 2017-12-29 14:07:51 +01:00
Antonio
af07def7fc Added parameters for PLPython functions 2017-12-28 11:36:02 +01:00
Antonio
bc9dbdbb72 Removed routing client from isolines 2017-12-27 16:31:29 +01:00
Antonio
dabd596271 Added QPS decorator 2017-12-27 14:55:50 +01:00
Antonio
8c0af7d51d Pure Python implementation of clients for Mapbox services 2017-12-27 13:43:14 +01:00
Mario de Frutos
b16f7e6ed5 Merge pull request #416 from CartoDB/Added_sudo_to_pip_install
Added sudo to pip install
2017-12-12 12:46:00 +01:00
Rafa de la Torre
1ea5400e40 Merge pull request #418 from CartoDB/417-fix-client-upgrade
Fix for client upgrade #417
2017-12-01 17:15:15 +01:00
Rafa de la Torre
c7bae90e93 Update NEWS.md 2017-12-01 17:03:05 +01:00
Rafa de la Torre
604e802807 Add DROP FUNCTION's to the upgrade script #417 2017-12-01 17:00:49 +01:00
Rafa de la Torre
27c17ddc8b Re-generated extension file #417 2017-12-01 16:56:45 +01:00
Rafa de la Torre
feff20ac5a Use params_with_type to avoid DEFAULT #417 2017-12-01 16:40:39 +01:00
Rafa de la Torre
4c996733ce Fix for client upgrade #417 2017-12-01 16:31:57 +01:00
Rafa de la Torre
046b8f71f9 Revert "Fix bad renaming of function parameter"
This reverts commit 63786caaa6.
2017-12-01 13:35:45 +01:00
Antonio
9b8045f2ed Added sudo to pip install requirements.txt 2017-12-01 13:27:11 +01:00
Rafa de la Torre
63786caaa6 Fix bad renaming of function parameter
This fixes the following error when trying to upgrade:

```
cannot change name of input parameter "organization_name"
```

The problem was introduced ages ago. I fixed it by running a script:

```bash
BAD_REGEXP='CREATE OR REPLACE FUNCTION cdb_dataservices_client\._cdb_geocode_admin0_polygon (username text, orgname text, country_name text)'
REPLACEMENT='CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_geocode_admin0_polygon (username text, organization_name text, country_name text)'
git grep -l "$BAD_REGEXP" | xargs sed -i "s/$BAD_REGEXP/$REPLACEMENT/g"
```
2017-12-01 12:52:17 +01:00
Rafa de la Torre
9d9262c0e7 Update NEWS.md 2017-12-01 11:18:22 +01:00
Rafa de la Torre
580f6d9867 Make the install target depend on deploy
The deploy target strips the PARALLEL tag when needed. This is done for
versions not supporting it.
2017-12-01 10:47:30 +01:00
Rafa de la Torre
48942cbb03 Fix typo in PARALLEL stripping 2017-12-01 10:25:10 +01:00
Rafa de la Torre
e4a2e13f69 Merge pull request #413 from CartoDB/server-0.29.0
Server 0.29.0
2017-11-30 13:11:43 +01:00
Rafa de la Torre
048aebedd9 Remove permissions stuff from downgrade script 2017-11-30 13:08:54 +01:00
Rafa de la Torre
1bcfa30938 Remove type creation from downgrade script 2017-11-30 13:08:26 +01:00
Rafa de la Torre
24098a6d2c Remove duplicated header 2017-11-30 13:07:31 +01:00
Rafa de la Torre
645818838b Append old script to downgrade stub 2017-11-30 13:06:57 +01:00
Rafa de la Torre
a2929e88cc Remove permissions stuff from upgrade script 2017-11-30 13:04:34 +01:00
Rafa de la Torre
7e7432b4ec Remove type creation from upgrade script 2017-11-30 13:04:09 +01:00
Rafa de la Torre
82db791706 Delete duplicate header from upgrade script 2017-11-30 13:02:28 +01:00
Rafa de la Torre
d21aa07673 Apend contents of extension to upgrade stub 2017-11-30 13:01:32 +01:00
Rafa de la Torre
dec7842240 Stub new server v0.29.0 2017-11-30 13:00:23 +01:00
Rafa de la Torre
0cb359a9bc Add some instructions to make release target 2017-11-30 12:59:11 +01:00
Rafa de la Torre
5bdf10936e Merge pull request #412 from CartoDB/client-0.22.0
Client 0.22.0
2017-11-30 12:49:57 +01:00
Rafa de la Torre
bf424a8f1e Remove permissions stuff from downgrade script 2017-11-30 12:43:28 +01:00
Rafa de la Torre
ccf27eaa5c Remove type creation from downgrade script 2017-11-30 12:42:51 +01:00
Rafa de la Torre
d2980aa30c Remove duplicate header from downgrade script 2017-11-30 12:41:49 +01:00
Rafa de la Torre
bffd6e8200 Append contents of old version to downgrade script
I just did this:

```
cat old_versions/cdb_dataservices_client--0.21.0.sql >> cdb_dataservices_client--0.22.0--0.21.0.sql
```
2017-11-30 12:40:57 +01:00
Rafa de la Torre
89792b2d0c Remove permissions stuff from client upgrade 2017-11-30 12:38:18 +01:00
Rafa de la Torre
8446c9c388 Remove type creation from client upgrade 2017-11-30 12:37:33 +01:00
Rafa de la Torre
29693ddc81 Remove duplicated header from upgrade 2017-11-30 12:36:44 +01:00
Rafa de la Torre
2213900831 Copy content of new version into upgrade script
I just ran the following:

```
cat cdb_dataservices_client--0.22.0.sql >> cdb_dataservices_client--0.21.0--0.22.0.sql
```
2017-11-30 12:35:25 +01:00
Rafa de la Torre
f645766235 Stub new client version 0.22.0 2017-11-30 12:31:36 +01:00
Rafa de la Torre
657e4064ef Add some instructions to the make release target 2017-11-30 12:27:07 +01:00
Rafa de la Torre
b5c0e135f9 Revert "PG release: Include all files in git upon release"
This reverts commit b44778c586.
2017-11-30 12:19:11 +01:00
Raúl Marín
f4d57007c2 Merge pull request #411 from Algunenano/parallel_deploy
PG release: Strip PARALLEL on deploy instead of release
2017-11-28 17:59:31 +01:00
Raul Marin
b44778c586 PG release: Include all files in git upon release 2017-11-28 16:27:45 +01:00
Raul Marin
723a3d02ff PG release: Strip PARALLEL on deploy instead of release 2017-11-28 15:44:30 +01:00
Raúl Marín
6180b00525 Merge pull request #408 from Algunenano/parallel
Client: Add PARALLEL and VOLATILE categories
2017-11-27 13:56:01 +01:00
Raul Marin
5086ee908d Update CONTRIBUTING with information about SQL function labels 2017-11-24 16:44:47 +01:00
Mario de Frutos
f751530dd2 Merge pull request #410 from CartoDB/development
Release 0.15.6 of the python library
2017-11-17 11:24:49 +01:00
Mario de Frutos
d42fb3cf8e Updated NEWS.md 2017-11-17 11:20:06 +01:00
Mario de Frutos
4f5117611f Merge pull request #409 from CartoDB/s1060-google_geocoding_returns_null
Support for multiple parameters at client (such as channel)
2017-11-17 11:17:45 +01:00
Raul Marin
ff71a406ae Server: Categorize functions
Redis/Service functions declared as STATIC to avoid unneded calls
but it's really questionable if they trully are
2017-11-14 16:48:48 +01:00
Raul Marin
98206a3296 Server: Mark all plproxy functions as VOLATILE PARALLEL UNSAFE 2017-11-14 16:48:48 +01:00
Raul Marin
a69da0084b Server: Add PARALLEL support 2017-11-14 16:48:48 +01:00
Raul Marin
80fc32e1ea Categorize renderer template functions 2017-11-14 16:45:36 +01:00
Raul Marin
b8527514b7 Client: Add PARALLEL and VOLATILE categories
I'm not 100% sure what's appropiate for plproxy functions so, for now,
I've set them up as VOLATILE and PARALLEL UNSAFE
2017-11-14 16:45:36 +01:00
Juan Ignacio Sánchez Lara
da8e5f6e36 Extract parsing and return fix 2017-11-14 13:15:11 +01:00
Juan Ignacio Sánchez Lara
f540a3d6b3 Bump version 2017-11-13 19:09:52 +01:00
Juan Ignacio Sánchez Lara
e26bf2a803 Proper parsing of client and channel parameters 2017-11-13 19:09:17 +01:00
Juan Ignacio Sánchez Lara
43dd9f6ada Better regexp and wrong input handling 2017-11-08 17:33:00 +01:00
Juan Ignacio Sánchez Lara
2cf4072b21 Support for multiple parameters at client (such as channel) 2017-11-08 11:07:35 +01:00
Mario de Frutos
d8a20a61b2 Merge pull request #404 from CartoDB/development
Release client 0.21.0 and server 0.28.0
2017-10-18 10:28:18 +02:00
Mario de Frutos
423431460d Merge pull request #403 from CartoDB/numgeoms_getavailablegeometries
Number of geometries for OBS_GetAvailableGeometries
2017-10-18 10:26:48 +02:00
Mario de Frutos
90e3229e00 Updated NEWS.md 2017-10-18 10:25:14 +02:00
Mario de Frutos
68e87ca477 Release 0.28.0 server extension artifact 2017-10-18 10:25:04 +02:00
Mario de Frutos
b56a6bfa45 Release 0.21.0 client extension artifact 2017-10-18 10:24:50 +02:00
Mario de Frutos
4c7a3a25ec Remove duplicated functions after the new ones are added 2017-10-10 16:23:15 +02:00
Mario de Frutos
e68cc3e820 OBS_GetAvailableGeometries now has number of geometries parameter
We've add a new parameter, number of geometries, that would be used
by DO to compute the geometry that better fit with the input provided
2017-10-09 17:20:57 +02:00
Mario de Frutos
baf9a48dad Stub version 0.28.0 for server 2017-10-09 17:05:07 +02:00
Mario de Frutos
8ecd69d0bf Stub version 0.21.0 for client 2017-10-09 17:03:40 +02:00
Rafa de la Torre
1936374124 Version 0.15.5 of the python library 2017-10-06 16:03:03 +02:00
Rafa de la Torre
988a0c31dc Merge pull request #402 from CartoDB/401-persist-gloogle-client-connections
GoogleMapsClientFactory to persist clients #401
2017-10-06 15:46:54 +02:00
Rafa de la Torre
0cd01ee9c9 Merge pull request #400 from CartoDB/383-upgrade-googlemaps-lib-v2.5.1
Upgrade googlemaps python library to v2.5.1
2017-10-06 15:45:05 +02:00
Rafa de la Torre
3ed41b065a Marginally improve tests for #382 2017-10-06 15:27:51 +02:00
Rafa de la Torre
6c3260b6ee Move credentials tests to GoogleMapsClientFactoryTestCase 2017-10-06 15:23:44 +02:00
Rafa de la Torre
5005572f89 Add tests for GoogleMapsClientFactory 2017-10-06 15:18:51 +02:00
Rafa de la Torre
27fde5a910 Use "{id}:{secret}" as caching key 2017-10-06 15:12:05 +02:00
Rafa de la Torre
b8e1120169 Revert "Use a different client_id for the test"
This reverts commit 854fbb803c.
2017-10-06 15:03:08 +02:00
Rafa de la Torre
8e33cf9723 Fix google services key validity check #382 2017-10-06 14:40:53 +02:00
Rafa de la Torre
7b988e3f78 Add tests for #382
The examples are not arbitrary: base64 by default ignores characters
that are not in its alphabet. When decoding, the dashes and underscores
are not b64-valid and therefore ignored, resulting in a sequence with
the wrong padding.
2017-10-06 14:37:26 +02:00
Rafa de la Torre
f79ac9297d Add again the check for valid credentials
As our check is more strict that the one provided in googlemaps library.
2017-10-06 13:44:08 +02:00
Rafa de la Torre
854fbb803c Use a different client_id for the test 2017-10-06 13:41:50 +02:00
Rafa de la Torre
d029ad7041 GoogleMapsClientFactory to persist clients #401
Add a GoogleMapsClientFactory and remove the check for valid key,
which is no longer needed, as it is done in the google library.
2017-10-06 12:05:34 +02:00
Rafa de la Torre
24c29c0847 Make test order execution deterministic 2017-10-04 17:26:30 +02:00
Rafa de la Torre
31e79cb626 Remove installation of schema_triggers
Schema triggers dependency was deprecated time ago. See
5d43faecaf/NEWS.md (0120-2016-01-27)
2017-10-04 17:26:30 +02:00
Rafa de la Torre
98eef7ae02 Improve robustness of client tests
All tests that were failing in CI with this diffs showing traces with
context, which correspond to a default `VERBOSITY verbose` in the server.

Since we migrated the CI to postgres 9.6 it makes sense to see such
differences in server configuration.

Enforcing the verbosity level makes the tests more robust indeed.
2017-10-04 17:26:13 +02:00
Rafa de la Torre
445d4cf97d Upgrade googlemaps python library to v2.5.1
This closes #383
2017-10-04 16:22:33 +02:00
Rafa de la Torre
3a86ffba71 Merge pull request #399 from CartoDB/keep-metrics-keys-consistent
Keep metrics keys consistent
2017-09-07 11:54:55 +02:00
Rafa de la Torre
f0a3249b4e Keep metrics keys consistent
See cartodb repo and our internal documentation:
https://github.com/CartoDB/cartodb/search?utf8=%E2%9C%93&q=failed_responses&type=
2017-09-07 11:24:38 +02:00
Mario de Frutos
ef7c5d9218 Merge pull request #398 from CartoDB/development
Release python library version 0.15.4
2017-08-30 17:46:25 +02:00
Mario de Frutos
bd63346c50 Updated NEWS.md 2017-08-30 17:44:33 +02:00
Mario de Frutos
c878384955 Merge pull request #397 from CartoDB/fix_invalid_mapzen_isochrones
Fix invalid geometries due to generalize option
2017-08-30 17:42:50 +02:00
Mario de Frutos
475df918c7 Fix invalid geometries due to generalize option
They add a note saying that use their simplification option could
lead to a self-intersection (which is the problem we have) and it's
creating invalid geometries

See it here https://mapzen.com/documentation/mobility/isochrone/api-reference/#other-request-parameters
2017-08-30 17:35:08 +02:00
Mario de Frutos
937440c79a Update NEWS.md 2017-08-24 16:09:10 +02:00
Mario de Frutos
cb39640179 Merge pull request #396 from CartoDB/development
Release 0.15.3 for the python library
2017-08-24 16:06:34 +02:00
Mario de Frutos
727e5c09a9 Merge pull request #395 from CartoDB/remove_quota_do
Remove quota DO
2017-08-24 16:04:40 +02:00
Mario de Frutos
c13dba71a4 Deactivated DO credits usage for users that have quota
See https://github.com/CartoDB/bigmetadata/issues/215
2017-08-24 14:35:20 +02:00
Javier Torres
55cd62f6dc Merge pull request #394 from CartoDB/UpdatedUserDatabaseConfiguration
Updated configuration
2017-08-24 11:55:33 +02:00
Antonio
00424a0314 Added suggested changes 2017-08-24 10:05:23 +02:00
Antonio
e30b081cbc Added suggestions from CR 2017-08-16 11:11:33 +02:00
Antonio
797a2dbd24 Updated the User database configuration section 2017-08-11 10:21:10 +02:00
144 changed files with 48770 additions and 379 deletions

1
.gitignore vendored
View File

@@ -4,3 +4,4 @@
cartodb_services.egg-info/
build/
dist/
.vscode/

View File

@@ -5,6 +5,23 @@ The issue tracker is at [github.com/CartoDB/dataservices-api](https://github.com
We love pull requests from everyone, see [Contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/#contributing).
## PostgreSQL
When adding or modifying PostgreSQL functions make sure that the
[VOLATILITY](https://www.postgresql.org/docs/current/static/xfunc-volatility.html) and [PARALLEL](https://www.postgresql.org/docs/9.6/static/parallel-safety.html) categories are updated accordingly.
Please mark public [client functions](client/sql) or [templates](client/renderer/templates)
as STABLE even if the internals are VOLATILE to allow the planner to cache
results inside a query. For example, in this query we need `cdb_geocode_admin1_polygon`
to be STABLE so it gets called only once (instead of once per row):
```sql
SELECT * japank WHERE NOT (the_geom && cdb_geocode_admin1_polygon('Madrid', 'Spain'));
```
As PARALLEL labels need to be stripped for incompatible PostgreSQL versions,
please use _PARALLEL SAFE/RESTRICTED/UNSAFE_ in uppercase so it's handled
automatically.
## Submitting Contributions

86
NEWS.md
View File

@@ -1,3 +1,89 @@
February 5th, 2018
==================
* Version `0.16.5` of the python library
* Fix displaced routing shape #443
* Check for empty coordinates object before converting it to polygon
* 422 errors that come from Mapbox now returns an empty result because is a bad input from the user data
February 2th, 2018
==================
* Version `0.16.4` of the python library
* Create a QuotaExceededException instead of using a generic one
* Version `0.30.2` of server side
* Return empty value when the quota is exceeded and don't send the exception to external loggers
to avoid noise
January 31th, 2018
==================
* Version `0.16.3` of the python library
* Fix for Mapbox geocoder to handle empty requests and empty responses
* Remove raising an exception when non parameters are passed to the HERE geocoder
* Fix for HERE geocoder with non empty requests
* Added more coverage to the google geocoder credentials parse logic
January 29th, 2018
==================
* Version `0.30.1` of server side
* Fix for Mapbox geocoding function due to the iso3166 library doesn't support UTF-8 names for the countries
* Version `0.16.2` of the python library
* Fix for Mapbox cycling profile
January 18th, 2018
==================
* Version `0.16.1` of the python library
* Fixed encoding problem with Mapbox geocoding (using the Mapbox Python library)
January 17th, 2018
==================
* Version `0.16.0` of the python library
* Added Mapbox provider for routing, geocoding and isolines
* Version `0.30.0` of the server extension
* Added Mapbox provider for routing, geocoding and isolines
* Fixed a bug that makes impossible to install server side if the client is already installed due a namespace problem
* Version `0.23.0` of the client
* Added Mapbox provider for routing, geocoding and isolines
December 1st, 2017
==================
* Version `0.29.0` of the server extension
* Functions qualified with PARALLEL and volatility
* Version `0.22.0` of the client
* Functions qualified with PARALLEL and volatility
* Fixed an issue with upgrades from older versions. See #417
November 17th, 2017
=================
* Version `0.15.6` of the python library
* Added support for channels for google geocoder #409
* Improved the way we manage the client_id #409
October 18th, 2017
=================
* Version `0.21.0` of the client extension
* Added new parameter `number_geometries` to the `obs_getavailablegeometries` to improve the score calculation
* Version `0.28.0` of the server extension
* Added new parameter `number_geometries` to the `obs_getavailablegeometries` to improve the score calculation
October 6th, 2017
=================
* Version `0.15.5` of the python library
* googlemaps dependency updated to v2.5.1
* Google geocoder performance boost: client connections are now reused between queries. See #401
* Fixed issue with Google keys validity check. See #382
* Fixed inconsistency in service usage failed requests tracking. See f0a3249
* Client extension tests are now compatible with PostgreSQL 9.5 and 9.6
August 30th, 2017
=============
* Version `0.15.4` of the python library
* Fixed invalid geometries for isochrones due to `generalize` option. See #397
August 24th, 2017
=============
* Improved the documentation
* Version `0.15.3` of the python library
* Disabled DO quota check for users that have it configured . See #395
August 23th, 2017
=============
* Version `0.27.0` of the server

View File

@@ -38,14 +38,14 @@ Steps to deploy a new Data Services API version :
```
# in dataservices-api repo root path:
cd server/lib/python/cartodb_services && pip install -r requirements.txt && sudo pip install . --upgrade
cd server/lib/python/cartodb_services && sudo pip install -r requirements.txt && sudo pip install . --upgrade
```
- Create a database to hold all the server part and a user for it
```sql
CREATE DATABASE dataservices_db ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
CREATE USER dataservices_user;
CREATE USER geocoder_api;
```
- Install needed extensions in `dataservices_db` database
@@ -90,19 +90,23 @@ Steps to deploy a new Data Services API version :
```
psql -U postgres -d dataservices_db -f src/pg/test/fixtures/load_fixtures.sql
```
- Give permission to execute and select to the `dataservices_user` user:
- Give permission to execute and select to the `geocoder_api` user:
```
psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS observatory; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS observatory VERSION 'dev'; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO geocoder_api; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO geocoder_api; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT USAGE ON SCHEMA cdb_observatory TO geocoder_api; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO geocoder_api; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA observatory TO geocoder_api; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT USAGE ON SCHEMA observatory TO geocoder_api; COMMIT" -e
```
### Server configuration
Configuration for the different services must be stored in the server database using `CDB_Conf_SetConf()`.
**All the configuration inside brackets [] is optional**
#### Redis configuration
If sentinel is used:
@@ -158,6 +162,15 @@ SELECT CDB_Conf_SetConf(
);
```
#### Mapbox configuration
```sql
SELECT CDB_Conf_SetConf(
'mapbox_conf',
'{"routing": {"api_keys": ["your_api_key"], "monthly_quota": 999999}, "geocoder": {"api_keys": ["your_api_key"], "monthly_quota": 999999}, "matrix": {"api_keys": ["your_api_key"], "monthly_quota": 1500000}}'
);
```
#### Data Observatory
```sql
@@ -172,10 +185,12 @@ SELECT CDB_Conf_SetConf(
```sql
SELECT CDB_Conf_SetConf(
'logger_conf',
'{"geocoder_log_path": "/tmp/geocodings.log", [ "min_log_level": "[debug|info|warning|error]", "rollbar_api_key": "SERVER_SIDE_API_KEY", "log_file_path": "LOG_FILE_PATH"]}'
'{"geocoder_log_path": "/tmp/geocodings.log", "min_log_level": "LOG_LEVEL", "rollbar_api_key": "SERVER_SIDE_API_KEY", "log_file_path": "LOG_FILE_PATH"}'
);
```
Where `LOG_LEVEL` can be `debug`, `info`, `warning` or `error`.
#### Environment
The execution environment (development/staging/production) affects rollbar messages and other details.
@@ -254,14 +269,16 @@ jsonb_set(
```
### User database configuration
#### Option 1 (manually)
User (client) databases need also some configuration so that the client extension can access the server:
#### Users/Organizations
##### Users/Organizations
```sql
SELECT CDB_Conf_SetConf('user_config', '{"is_organization": false, "entity_name": "<YOUR_USERNAME>"}');
```
#### Dataservices server
##### Dataservices server
The `geocoder_server_config` (the name is not accurate for historical reasons) entry points
to the dataservices server DB (you can use a specific database for the server or your same user's):
@@ -272,10 +289,17 @@ SELECT CDB_Conf_SetConf(
'{ "connection_str": "host=localhost port=5432 dbname=<SERVER_DB_NAME> user=postgres"}'
);
```
#### Search path
##### Search path
The search path must be configured in order to be able to execute the functions without using the schema:
```sql
ALTER ROLE "<USER_ROLE>" SET search_path="$user", public, cartodb, cdb_dataservices_client;
```
#### Option 2 (from builder)
See [the **Configuring Dataservices** documentation](http://cartodb.readthedocs.io/en/latest/operations/configure_data_services.html)
### Rate limits
See [docs](https://github.com/CartoDB/dataservices-api/blob/master/doc/rate_limits.md)

View File

@@ -8,6 +8,11 @@ SED = sed
ERB = erb
REPLACEMENTS = -i 's/$(EXTVERSION)/$(NEW_VERSION)/g'
NEW_EXTENSION_ARTIFACT = $(EXTENSION)--$(EXTVERSION).sql
AWK = awk
# Parallel support macros
PG_CONFIG = pg_config
PG_PARALLEL := $(shell $(PG_CONFIG) --version | ($(AWK) '{$$2*=1000; if ($$2 >= 9600) print 1; else print 0;}' 2> /dev/null || echo 0))
# OLD_VERSIONS = $(wildcard old_versions/*.sql)
# DATA = $(NEW_EXTENSION_ARTIFACT) \
@@ -21,7 +26,7 @@ DATA = $(NEW_EXTENSION_ARTIFACT) \
SOURCES_DATA_DIR = sql/
REGRESS = $(notdir $(basename $(wildcard test/sql/*test.sql)))
REGRESS = $(notdir $(basename $(sort $(wildcard test/sql/*test.sql))))
TEST_DIR = test/
REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)' --user='postgres'
@@ -50,6 +55,13 @@ SOURCES_DATA = $(wildcard $(SOURCES_DATA_DIR)/*.sql) $(GENERATED_SQL_FILES)
$(NEW_EXTENSION_ARTIFACT): $(SOURCES_DATA)
rm -f $@
cat $(SOURCES_DATA_DIR)/*.sql >> $@
ifeq ($(PG_PARALLEL), 0)
# Remove PARALLEL in aggregates and functions
$(eval TMPFILE := $(shell mktemp /tmp/$(basename $0).XXXXXXXX))
$(SED) -e 's/PARALLEL \= [A-Z]*,/''/g' \
-e 's/PARALLEL [A-Z]*/''/g' $@ > $(TMPFILE)
mv $(TMPFILE) $@
endif
.PHONY: all
all: $(DATA)
@@ -64,9 +76,30 @@ release: $(EXTENSION).control $(SOURCES_DATA)
git add $(EXTENSION)--$(NEW_VERSION).sql
$(ERB) version=$(NEW_VERSION) upgrade_downgrade_template.erb > $(EXTENSION)--$(EXTVERSION)--$(NEW_VERSION).sql
$(ERB) version=$(EXTVERSION) upgrade_downgrade_template.erb > $(EXTENSION)--$(NEW_VERSION)--$(EXTVERSION).sql
@echo
@echo "Please review the file $(EXTENSION)--$(EXTVERSION)--$(NEW_VERSION).sql and add any code needed to upgrade $(EXTVERSION) to $(NEW_VERSION)"
@echo "Please review the file $(EXTENSION)--$(NEW_VERSION)--$(EXTVERSION).sql and add any code needed to downgrade $(NEW_VERSION) to $(EXTVERSION)"
@echo
# Only meant for development time, do not use once a version is released
.PHONY: devclean
devclean:
rm -f $(NEW_EXTENSION_ARTIFACT)
rm -f $(GENERATED_SQL_FILES)
# If needed remove PARALLEL tags from the release files
release_remove_parallel_deploy:
ifeq ($(PG_PARALLEL), 0)
for n in $(wildcard old_versions/*.sql *.sql); do \
$(eval TMPFILE := $(shell mktemp /tmp/XXXXXXXXXX)) \
$(SED) -e 's/PARALLEL \= [A-Z]*,/''/g' -e 's/PARALLEL [A-Z]*/''/g' $$n > $(TMPFILE); \
mv $(TMPFILE) $$n; \
done
endif
# Install the current release into the PostgreSQL extensions directory
deploy: release_remove_parallel_deploy
$(INSTALL_DATA) $(EXTENSION).control '$(DESTDIR)$(datadir)/extension/'
$(INSTALL_DATA) old_versions/*.sql *.sql '$(DESTDIR)$(datadir)/extension/'
install: deploy

View File

@@ -0,0 +1,200 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.23.0'" to load this file. \quit
-- Make sure we have a sane search path to create/update the extension
SET search_path = "$user",cartodb,public,cdb_dataservices_client;
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_mapbox_geocode_street_point (searchtext text, city text DEFAULT NULL, state_province text DEFAULT NULL, country text DEFAULT NULL)
RETURNS Geometry AS $$
DECLARE
ret Geometry;
username text;
orgname text;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
SELECT cdb_dataservices_client._cdb_mapbox_geocode_street_point(username, orgname, searchtext, city, state_province, country) INTO ret; RETURN ret;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_mapbox_isochrone (source geometry(Geometry, 4326), mode text, range integer[], options text[] DEFAULT ARRAY[]::text[])
RETURNS SETOF cdb_dataservices_client.isoline AS $$
DECLARE
username text;
orgname text;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
RETURN QUERY SELECT * FROM cdb_dataservices_client._cdb_mapbox_isochrone(username, orgname, source, mode, range, options);
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client.cdb_mapbox_isodistance (source geometry(Geometry, 4326), mode text, range integer[], options text[] DEFAULT ARRAY[]::text[])
RETURNS SETOF cdb_dataservices_client.isoline AS $$
DECLARE
username text;
orgname text;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
RETURN QUERY SELECT * FROM cdb_dataservices_client._cdb_mapbox_isodistance(username, orgname, source, mode, range, options);
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_mapbox_geocode_street_point_exception_safe (searchtext text, city text DEFAULT NULL, state_province text DEFAULT NULL, country text DEFAULT NULL)
RETURNS Geometry AS $$
DECLARE
ret Geometry;
username text;
orgname text;
_returned_sqlstate TEXT;
_message_text TEXT;
_pg_exception_context TEXT;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
BEGIN
SELECT cdb_dataservices_client._cdb_mapbox_geocode_street_point(username, orgname, searchtext, city, state_province, country) INTO ret; RETURN ret;
EXCEPTION
WHEN OTHERS THEN
GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
_message_text = MESSAGE_TEXT,
_pg_exception_context = PG_EXCEPTION_CONTEXT;
RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
RETURN ret;
END;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_mapbox_isochrone_exception_safe(source geometry(Geometry, 4326), mode text, range integer[], options text[] DEFAULT ARRAY[]::text[])
RETURNS SETOF cdb_dataservices_client.isoline AS $$
DECLARE
username text;
orgname text;
_returned_sqlstate TEXT;
_message_text TEXT;
_pg_exception_context TEXT;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
BEGIN
RETURN QUERY SELECT * FROM cdb_dataservices_client._cdb_mapbox_isochrone(username, orgname, source, mode, range, options);
EXCEPTION
WHEN OTHERS THEN
GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
_message_text = MESSAGE_TEXT,
_pg_exception_context = PG_EXCEPTION_CONTEXT;
RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
END;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_mapbox_isodistance_exception_safe(source geometry(Geometry, 4326), mode text, range integer[], options text[] DEFAULT ARRAY[]::text[])
RETURNS SETOF cdb_dataservices_client.isoline AS $$
DECLARE
username text;
orgname text;
_returned_sqlstate TEXT;
_message_text TEXT;
_pg_exception_context TEXT;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
BEGIN
RETURN QUERY SELECT * FROM cdb_dataservices_client._cdb_mapbox_isodistance(username, orgname, source, mode, range, options);
EXCEPTION
WHEN OTHERS THEN
GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
_message_text = MESSAGE_TEXT,
_pg_exception_context = PG_EXCEPTION_CONTEXT;
RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
END;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_mapbox_geocode_street_point (username text, orgname text, searchtext text, city text DEFAULT NULL, state_province text DEFAULT NULL, country text DEFAULT NULL)
RETURNS Geometry AS $$
CONNECT cdb_dataservices_client._server_conn_str();
SELECT cdb_dataservices_server.cdb_mapbox_geocode_street_point (username, orgname, searchtext, city, state_province, country);
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_mapbox_isochrone (username text, orgname text, source geometry(Geometry, 4326), mode text, range integer[], options text[] DEFAULT ARRAY[]::text[])
RETURNS SETOF cdb_dataservices_client.isoline AS $$
CONNECT cdb_dataservices_client._server_conn_str();
SELECT * FROM cdb_dataservices_server.cdb_mapbox_isochrone (username, orgname, source, mode, range, options);
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._cdb_mapbox_isodistance (username text, orgname text, source geometry(Geometry, 4326), mode text, range integer[], options text[] DEFAULT ARRAY[]::text[])
RETURNS SETOF cdb_dataservices_client.isoline AS $$
CONNECT cdb_dataservices_client._server_conn_str();
SELECT * FROM cdb_dataservices_server.cdb_mapbox_isodistance (username, orgname, source, mode, range, options);
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_mapbox_geocode_street_point(searchtext text, city text, state_province text, country text) TO publicuser;
GRANT EXECUTE ON FUNCTION cdb_dataservices_client._cdb_mapbox_geocode_street_point_exception_safe(searchtext text, city text, state_province text, country text ) TO publicuser;
GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_mapbox_isochrone(source geometry(Geometry, 4326), mode text, range integer[], options text[]) TO publicuser;
GRANT EXECUTE ON FUNCTION cdb_dataservices_client._cdb_mapbox_isochrone_exception_safe(source geometry(Geometry, 4326), mode text, range integer[], options text[] ) TO publicuser;
GRANT EXECUTE ON FUNCTION cdb_dataservices_client.cdb_mapbox_isodistance(source geometry(Geometry, 4326), mode text, range integer[], options text[]) TO publicuser;
GRANT EXECUTE ON FUNCTION cdb_dataservices_client._cdb_mapbox_isodistance_exception_safe(source geometry(Geometry, 4326), mode text, range integer[], options text[] ) TO publicuser;

View File

@@ -0,0 +1,18 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.22.0'" to load this file. \quit
-- Make sure we have a sane search path to create/update the extension
SET search_path = "$user",cartodb,public,cdb_dataservices_client;
-- HERE goes your code to upgrade/downgrade
DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_mapbox_geocode_street_point (searchtext text, city text, state_province text, country text);
DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_mapbox_isochrone (source geometry(Geometry, 4326), mode text, range integer[], options text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_client.cdb_mapbox_isodistance (source geometry(Geometry, 4326), mode text, range integer[], options text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_mapbox_geocode_street_point_exception_safe (searchtext text, city text, state_province text, country text);
DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_mapbox_isochrone_exception_safe(source geometry(Geometry, 4326), mode text, range integer[], options text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_mapbox_isodistance_exception_safe(source geometry(Geometry, 4326), mode text, range integer[], options text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_mapbox_geocode_street_point (username text, orgname text, searchtext text, city text, state_province text, country text);
DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_mapbox_isochrone (username text, orgname text, source geometry(Geometry, 4326), mode text, range integer[], options text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_client._cdb_mapbox_isodistance (username text, orgname text, source geometry(Geometry, 4326), mode text, range integer[], options text[]);

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,5 @@
comment = 'CartoDB dataservices client API extension'
default_version = '0.20.0'
default_version = '0.23.0'
requires = 'plproxy, cartodb'
superuser = true
schema = cdb_dataservices_client

View File

@@ -0,0 +1,72 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.21.0'" to load this file. \quit
-- Make sure we have a sane search path to create/update the extension
SET search_path = "$user",cartodb,public,cdb_dataservices_client;
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries (username text, orgname text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, timespan text DEFAULT NULL, number_geometries integer DEFAULT NULL)
RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
CONNECT cdb_dataservices_client._server_conn_str();
SELECT * FROM cdb_dataservices_server.obs_getavailablegeometries (username, orgname, bounds, filter_tags, numer_id, denom_id, timespan, number_geometries);
$$ LANGUAGE plproxy;
CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailablegeometries (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,timespan text DEFAULT NULL ,number_geometries integer DEFAULT NULL)
RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
DECLARE
username text;
orgname text;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan, number_geometries);
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,timespan text DEFAULT NULL ,number_geometries integer DEFAULT NULL)
RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
DECLARE
username text;
orgname text;
_returned_sqlstate TEXT;
_message_text TEXT;
_pg_exception_context TEXT;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
BEGIN
RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan, number_geometries);
EXCEPTION
WHEN OTHERS THEN
GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
_message_text = MESSAGE_TEXT,
_pg_exception_context = PG_EXCEPTION_CONTEXT;
RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
END;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
DROP FUNCTION cdb_dataservices_client.obs_getavailablegeometries (geometry(Geometry, 4326), text[], text, text, text);
DROP FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe (geometry(Geometry, 4326), text[], text, text, text);
DROP FUNCTION cdb_dataservices_client._obs_getavailablegeometries (text, text, geometry(Geometry, 4326), text[], text, text, text);

View File

@@ -0,0 +1,65 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_client UPDATE TO '0.20.0'" to load this file. \quit
-- Make sure we have a sane search path to create/update the extension
SET search_path = "$user",cartodb,public,cdb_dataservices_client;
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries (username text, orgname text, bounds geometry(Geometry, 4326) DEFAULT NULL, filter_tags text[] DEFAULT NULL, numer_id text DEFAULT NULL, denom_id text DEFAULT NULL, timespan text DEFAULT NULL)
RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
CONNECT cdb_dataservices_client._server_conn_str();
SELECT * FROM cdb_dataservices_server.obs_getavailablegeometries (username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
$$ LANGUAGE plproxy;
CREATE OR REPLACE FUNCTION cdb_dataservices_client.obs_getavailablegeometries (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,timespan text DEFAULT NULL)
RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
DECLARE
username text;
orgname text;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe (bounds geometry(Geometry, 4326) DEFAULT NULL ,filter_tags text[] DEFAULT NULL ,numer_id text DEFAULT NULL ,denom_id text DEFAULT NULL ,timespan text DEFAULT NULL)
RETURNS SETOF cdb_dataservices_client.obs_meta_geometry AS $$
DECLARE
username text;
orgname text;
_returned_sqlstate TEXT;
_message_text TEXT;
_pg_exception_context TEXT;
BEGIN
IF session_user = 'publicuser' OR session_user ~ 'cartodb_publicuser_*' THEN
RAISE EXCEPTION 'The api_key must be provided';
END IF;
SELECT u, o INTO username, orgname FROM cdb_dataservices_client._cdb_entity_config() AS (u text, o text);
-- JSON value stored "" is taken as literal
IF username IS NULL OR username = '' OR username = '""' THEN
RAISE EXCEPTION 'Username is a mandatory argument, check it out';
END IF;
BEGIN
RETURN QUERY SELECT * FROM cdb_dataservices_client._obs_getavailablegeometries(username, orgname, bounds, filter_tags, numer_id, denom_id, timespan);
EXCEPTION
WHEN OTHERS THEN
GET STACKED DIAGNOSTICS _returned_sqlstate = RETURNED_SQLSTATE,
_message_text = MESSAGE_TEXT,
_pg_exception_context = PG_EXCEPTION_CONTEXT;
RAISE WARNING USING ERRCODE = _returned_sqlstate, MESSAGE = _message_text, DETAIL = _pg_exception_context;
END;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
DROP FUNCTION cdb_dataservices_client.obs_getavailablegeometries (geometry(Geometry, 4326), text[], text, text, text, integer);
DROP FUNCTION cdb_dataservices_client._obs_getavailablegeometries_exception_safe (geometry(Geometry, 4326), text[], text, text, text, integer);
DROP FUNCTION cdb_dataservices_client._obs_getavailablegeometries (text, text, geometry(Geometry, 4326), text[], text, text, text, integer);

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -86,6 +86,14 @@
- { name: state_province, type: text, default: 'NULL'}
- { name: country, type: text, default: 'NULL'}
- name: cdb_mapbox_geocode_street_point
return_type: Geometry
params:
- { name: searchtext, type: text}
- { name: city, type: text, default: 'NULL'}
- { name: state_province, type: text, default: 'NULL'}
- { name: country, type: text, default: 'NULL'}
- name: cdb_mapzen_geocode_street_point
return_type: Geometry
params:
@@ -114,6 +122,16 @@
- { name: range, type: "integer[]" }
- { name: options, type: "text[]", default: 'ARRAY[]::text[]' }
- name: cdb_mapbox_isochrone
return_type: SETOF cdb_dataservices_client.isoline
multi_row: true
multi_field: true
params:
- { name: source, type: "geometry(Geometry, 4326)" }
- { name: mode, type: text }
- { name: range, type: "integer[]" }
- { name: options, type: "text[]", default: 'ARRAY[]::text[]' }
- name: cdb_mapzen_isochrone
return_type: SETOF cdb_dataservices_client.isoline
multi_row: true
@@ -124,6 +142,16 @@
- { name: range, type: "integer[]" }
- { name: options, type: "text[]", default: 'ARRAY[]::text[]' }
- name: cdb_mapbox_isodistance
return_type: SETOF cdb_dataservices_client.isoline
multi_row: true
multi_field: true
params:
- { name: source, type: "geometry(Geometry, 4326)" }
- { name: mode, type: text }
- { name: range, type: "integer[]" }
- { name: options, type: "text[]", default: 'ARRAY[]::text[]' }
- name: cdb_mapzen_isodistance
return_type: SETOF cdb_dataservices_client.isoline
multi_row: true
@@ -427,6 +455,7 @@
- { name: numer_id, type: text, default: 'NULL' }
- { name: denom_id, type: text, default: 'NULL' }
- { name: timespan, type: text, default: 'NULL'}
- { name: number_geometries, type: integer, default: 'NULL'}
- name: obs_getavailabletimespans
return_type: SETOF cdb_dataservices_client.obs_meta_timespan

View File

@@ -21,4 +21,4 @@ BEGIN
<% return_statement do %><%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %>(<%= params(_with_user_org=true).join(', ') %>)<% end %>
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;

View File

@@ -32,4 +32,4 @@ BEGIN
<%= return_statement %>
END;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL UNSAFE;

View File

@@ -1,3 +1,4 @@
DROP FUNCTION IF EXISTS <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (<%= params_with_type(true).join(', ') %>);
CREATE OR REPLACE FUNCTION <%= DATASERVICES_CLIENT_SCHEMA %>._<%= name %> (<%= params_with_type_and_default(_with_user_org=true).join(', ') %>)
RETURNS <%= return_type %> AS $$
CONNECT <%= DATASERVICES_CLIENT_SCHEMA %>._server_conn_str();
@@ -6,4 +7,4 @@ RETURNS <%= return_type %> AS $$
<% else %>
SELECT <%= DATASERVICES_SERVER_SCHEMA %>.<%= name %> (<%= params(_with_user_org=true).join(', ') %>);
<% end %>
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;

View File

@@ -13,4 +13,4 @@ BEGIN
SELECT trim(both '"' FROM db_connection_str) INTO db_connection_str;
RETURN db_connection_str;
END;
$$ LANGUAGE 'plpgsql';
$$ LANGUAGE 'plpgsql' STABLE PARALLEL SAFE;

View File

@@ -34,4 +34,4 @@ BEGIN
result.organization_name = organization_name;
RETURN result;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER STABLE PARALLEL SAFE;

View File

@@ -41,7 +41,7 @@ BEGIN
RETURN result;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_PopulateTableOBS_GetMeasure(
table_name text,
@@ -89,7 +89,7 @@ BEGIN
RETURN result;
END;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER;
$$ LANGUAGE 'plpgsql' SECURITY DEFINER VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PrepareTableOBS_GetMeasure(
@@ -134,7 +134,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PrepareTableOBS_GetMeas
)
return True
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PopulateTableOBS_GetMeasure(
username text,
@@ -227,7 +227,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client.__DST_PopulateTableOBS_GetMea
fdw_server=plpy.quote_literal(server_name)))
return True
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_ConnectUserTable(
username text,
@@ -239,7 +239,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_ConnectUserTable(
)RETURNS cdb_dataservices_client.ds_fdw_metadata AS $$
CONNECT cdb_dataservices_client._server_conn_str();
TARGET cdb_dataservices_server._DST_ConnectUserTable;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_GetReturnMetadata(
username text,
@@ -249,7 +249,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_GetReturnMetadata(
) RETURNS cdb_dataservices_client.ds_return_metadata AS $$
CONNECT cdb_dataservices_client._server_conn_str();
TARGET cdb_dataservices_server._DST_GetReturnMetadata;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_FetchJoinFdwTableData(
username text,
@@ -261,7 +261,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_FetchJoinFdwTableData(
) RETURNS SETOF record AS $$
CONNECT cdb_dataservices_client._server_conn_str();
TARGET cdb_dataservices_server._DST_FetchJoinFdwTableData;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_DisconnectUserTable(
username text,
@@ -272,4 +272,4 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_client._DST_DisconnectUserTable(
) RETURNS boolean AS $$
CONNECT cdb_dataservices_client._server_conn_str();
TARGET cdb_dataservices_server._DST_DisconnectUserTable;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;

View File

@@ -51,4 +51,4 @@ BEGIN
RETURN TRUE;
END;
$$ LANGUAGE 'plpgsql';
$$ LANGUAGE 'plpgsql' VOLATILE PARALLEL UNSAFE;

View File

@@ -1,6 +1,5 @@
-- Install dependencies
CREATE EXTENSION postgis;
CREATE EXTENSION schema_triggers;
CREATE EXTENSION plpythonu;
CREATE EXTENSION cartodb;
CREATE EXTENSION plproxy;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server function

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
SET client_min_messages TO warning;
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions to raise exceptions
@@ -31,8 +32,6 @@ SET ROLE test_regular_user;
-- Exercise the exception safe and the proxied functions
SELECT _cdb_geocode_street_point_exception_safe('One street, 1');
WARNING: cdb_dataservices_client._cdb_geocode_street_point(6): [contrib_regression] REMOTE ERROR: Not enough quota or any other exception whatsoever.
DETAIL: SQL statement "SELECT cdb_dataservices_client._cdb_geocode_street_point(username, orgname, searchtext, city, state_province, country)"
PL/pgSQL function _cdb_geocode_street_point_exception_safe(text,text,text,text) line 21 at SQL statement
_cdb_geocode_street_point_exception_safe
------------------------------------------
@@ -40,15 +39,12 @@ PL/pgSQL function _cdb_geocode_street_point_exception_safe(text,text,text,text)
SELECT * FROM _cdb_isodistance_exception_safe('POINT(-3.70568 40.42028)'::geometry, 'walk', ARRAY[300]::integer[]);
WARNING: cdb_dataservices_client._cdb_isodistance(6): [contrib_regression] REMOTE ERROR: Not enough quota or any other exception whatsoever.
DETAIL: PL/pgSQL function _cdb_isodistance_exception_safe(geometry,text,integer[],text[]) line 21 at RETURN QUERY
center | data_range | the_geom
--------+------------+----------
(0 rows)
SELECT * FROM _cdb_route_point_to_point_exception_safe('POINT(-3.70237112 40.41706163)'::geometry,'POINT(-3.69909883 40.41236875)'::geometry, 'car', ARRAY['mode_type=shortest']::text[]);
WARNING: cdb_dataservices_client._cdb_route_point_to_point(7): [contrib_regression] REMOTE ERROR: Not enough quota or any other exception whatsoever.
DETAIL: SQL statement "SELECT * FROM cdb_dataservices_client._cdb_route_point_to_point(username, orgname, origin, destination, mode, options, units)"
PL/pgSQL function _cdb_route_point_to_point_exception_safe(geometry,geometry,text,text[],text) line 21 at SQL statement
shape | length | duration
-------+--------+----------
| |

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;
-- Mock the server functions

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Use regular user role
SET ROLE test_regular_user;
-- Add to the search path the schema

View File

@@ -1,6 +1,5 @@
-- Install dependencies
CREATE EXTENSION postgis;
CREATE EXTENSION schema_triggers;
CREATE EXTENSION plpythonu;
CREATE EXTENSION cartodb;
CREATE EXTENSION plproxy;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
SET client_min_messages TO warning;
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Add to the search path the schema
SET search_path TO public,cartodb,cdb_dataservices_client;

View File

@@ -1,3 +1,4 @@
\set VERBOSITY terse
-- Use regular user role
SET ROLE test_regular_user;

View File

@@ -14,7 +14,7 @@ Name | Type | Description | Accepted values
`destination` | `geometry(Point)` | Destination point, in 4326 projection, which defines the end location. |
`mode` | `text` | Type of transport used to calculate the routes. | `car`, `walk`, `bicycle` or `public_transport`
`options` | `text[]` | (Optional) Multiple options to add more capabilities to the analysis. See [Optional routing parameters](#optional-routing-parameters) for details.
`units` | `text` | Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`
`units` | `text` | (Optional) Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`. This option is not supported by Mapbox provider
#### Returns
@@ -22,7 +22,7 @@ Name | Type | Description | Accepted values
Name | Type | Description
--- | --- | ---
`duration` | `integer` | Duration in seconds of the calculated route.
`length` | `real` | Length in the defined unit in the `units` field. `kilometers` by default .
`length` | `real` | Length in the defined unit in the `units` field. `meters` by default .
`the_geom` | `geometry(LineString)` | LineString geometry of the calculated route in the 4326 projection.
#### Examples
@@ -49,7 +49,7 @@ Name | Type | Description | Accepted values
`waypoints` | `geometry(Point)[]` | Array of ordered points, in 4326 projection, which defines the origin point, one or more locations for the route path to travel through, and the destination. The first element of the array defines the origin and the last element the destination of the route. |
`mode` | `text` | Type of transport used to calculate the routes. | `car`, `walk`, `bicycle` or `public_transport`
`options` | `text[]` | (Optional) Multiple options to add more capabilities to the analysis. See [Optional routing parameters](#optional-routing-parameters) for details.
`units` | `text` | Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`
`units` | `text` | (Optional) Unit used to represent the length of the route. | `kilometers`, `miles`. By default is `kilometers`. This option is not supported by Mapbox provider
#### Returns
@@ -57,7 +57,7 @@ Name | Type | Description | Accepted values
Name | Type | Description
--- | --- | ---
`duration` | `integer` | Duration in seconds of the calculated route.
`length` | `real` | Length in the defined unit in the `units` field. `kilometers` by default .
`length` | `real` | Length in the defined unit in the `units` field. `meters` by default .
`the_geom` | `geometry(LineString)` | LineString geometry of the calculated route in the 4326 projection.
*Note*: A request to the function _cdb\_route\_with\_waypoints(waypoints geometry(Point)[], mode text, [options text[], units text])_ with only two points in the geometry array are automatically defined as origin and destination. It is equivalent to performing the following request with these two locations as parameters: _cdb\_route\_point\_to\_point(origin geometry(Point), destination geometry(Point), mode text, [options text[], units text])_.
@@ -75,10 +75,9 @@ INSERT INTO <TABLE> (duration, length, the_geom) SELECT duration, length, shape
UPDATE <TABLE> SET the_geom = (SELECT shape FROM cdb_route_with_waypoints(Array['POINT(-3.7109 40.4234)'::GEOMETRY, 'POINT(-3.7059 40.4203)'::geometry, 'POINT(-3.7046 40.4180)'::geometry]::geometry[], 'car', ARRAY['mode_type=shortest']::text[]))
```
### Optional routing parameters
The optional value parameters must be passed using the format: `option=value`.
The optional value parameters must be passed using the format: `option=value`. Not all are available for all the routing providers
Name | Type | Description | Accepted values
--- | --- | --- | ---

View File

@@ -7,6 +7,11 @@ SED = sed
ERB = erb
REPLACEMENTS = -i 's/$(EXTVERSION)/$(NEW_VERSION)/g'
NEW_EXTENSION_ARTIFACT = $(EXTENSION)--$(EXTVERSION).sql
AWK = awk
# Parallel support macros
PG_CONFIG = pg_config
PG_PARALLEL := $(shell $(PG_CONFIG) --version | ($(AWK) '{$$2*=1000; if ($$2 >= 9600) print 1; else print 0;}' 2> /dev/null || echo 0))
REGRESS = $(notdir $(basename $(sort $(wildcard test/sql/*test.sql))))
TEST_DIR = test/
@@ -31,6 +36,13 @@ include $(PGXS)
$(NEW_EXTENSION_ARTIFACT): $(SOURCES_DATA)
rm -f $@
cat $(SOURCES_DATA_DIR)/*.sql >> $@
ifeq ($(PG_PARALLEL), 0)
# Remove PARALLEL in aggregates and functions
$(eval TMPFILE := $(shell mktemp /tmp/$(basename $0).XXXXXXXX))
$(SED) -e 's/PARALLEL \= [A-Z]*,/''/g' \
-e 's/PARALLEL [A-Z]*/''/g' $@ > $(TMPFILE)
mv $(TMPFILE) $@
endif
.PHONY: all
all: $(DATA)
@@ -40,11 +52,34 @@ release: $(EXTENSION).control $(SOURCES_DATA)
test -n "$(NEW_VERSION)" # $$NEW_VERSION VARIABLE MISSING. Eg. make release NEW_VERSION=0.x.0
git mv *.sql old_versions
$(SED) $(REPLACEMENTS) $(EXTENSION).control
git add $(EXTENSION).control
cat $(SOURCES_DATA_DIR)/*.sql > $(EXTENSION)--$(NEW_VERSION).sql
$(ERB) version=$(NEW_VERSION) upgrade_downgrade_template.erb > $(EXTENSION)--$(EXTVERSION)--$(NEW_VERSION).sql
$(ERB) version=$(EXTVERSION) upgrade_downgrade_template.erb > $(EXTENSION)--$(NEW_VERSION)--$(EXTVERSION).sql
git add $(EXTENSION)--$(NEW_VERSION).sql
@echo
@echo "Please review the file $(EXTENSION)--$(EXTVERSION)--$(NEW_VERSION).sql and add any code needed to upgrade $(EXTVERSION) to $(NEW_VERSION)"
@echo "Please review the file $(EXTENSION)--$(NEW_VERSION)--$(EXTVERSION).sql and add any code needed to downgrade $(NEW_VERSION) to $(EXTVERSION)"
@echo
# Only meant for development time, do not use once a version is released
.PHONY: devclean
devclean:
rm -f $(NEW_EXTENSION_ARTIFACT)
# If needed remove PARALLEL tags from the release files
release_remove_parallel_deploy:
ifeq ($(PG_PARALLEL), 0)
for n in $(wildcard old_versions/*.sql *.sql); do \
$(eval TMPFILE := $(shell mktemp /tmp/XXXXXXXXXX)) \
$(SED) -e 's/PARALLEL \= [A-Z]*,/''/g' -e 's/PARALLEL [A-Z]*/''/g' $$n > $(TMPFILE); \
mv $(TMPFILE) $$n; \
done
endif
# Install the current release into the PostgreSQL extensions directory
deploy: release_remove_parallel_deploy
$(INSTALL_DATA) $(EXTENSION).control '$(DESTDIR)$(datadir)/extension/'
$(INSTALL_DATA) old_versions/*.sql *.sql '$(DESTDIR)$(datadir)/extension/'
install: deploy

View File

@@ -0,0 +1,156 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.30.2'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import LegacyServiceManager
from cartodb_services.tools import QuotaExceededException
from cartodb_services.here import HereMapsGeocoder
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
try:
service_manager.assert_within_limits()
geocoder = HereMapsGeocoder(service_manager.config.heremaps_app_id, service_manager.config.heremaps_app_code, service_manager.logger, service_manager.config.heremaps_service_params)
coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using here maps', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using here maps')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_google_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import LegacyServiceManager, QuotaExceededException
from cartodb_services.google import GoogleMapsGeocoder
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
try:
service_manager.assert_within_limits(quota=False)
geocoder = GoogleMapsGeocoder(service_manager.config.google_client_id, service_manager.config.google_api_key, service_manager.logger)
coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using google maps', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using google maps')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import ServiceManager, QuotaExceededException
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
try:
service_manager.assert_within_limits()
geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3 = None
if country:
country_iso3 = country_to_iso3(country)
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3, search_type='address')
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapzen')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from iso3166 import countries
from cartodb_services.tools import ServiceManager, QuotaExceededException
from cartodb_services.mapbox import MapboxGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapbox_geocoder_config import MapboxGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapboxGeocoderConfigBuilder, username, orgname, GD)
try:
service_manager.assert_within_limits()
geocoder = MapboxGeocoder(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3166 = None
if country:
country_iso3 = country_to_iso3(country)
if country_iso3:
country_iso3166 = countries.get(country_iso3).alpha2.lower()
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3166)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapbox', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapbox')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -0,0 +1,114 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.30.1'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import LegacyServiceManager
from cartodb_services.here import HereMapsGeocoder
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
service_manager.assert_within_limits()
try:
geocoder = HereMapsGeocoder(service_manager.config.heremaps_app_id, service_manager.config.heremaps_app_code, service_manager.logger, service_manager.config.heremaps_service_params)
coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using here maps', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using here maps')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
service_manager.assert_within_limits()
try:
geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3 = None
if country:
country_iso3 = country_to_iso3(country)
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3, search_type='address')
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapzen')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from iso3166 import countries
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapbox_geocoder_config import MapboxGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapboxGeocoderConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
geocoder = MapboxGeocoder(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3166 = None
if country:
country_iso3 = country_to_iso3(country)
if country_iso3:
country_iso3166 = countries.get(country_iso3).alpha2.lower()
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3166)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapbox', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapbox')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,5 @@
comment = 'CartoDB dataservices server extension'
default_version = '0.27.0'
default_version = '0.30.2'
requires = 'plpythonu, plproxy, postgis, cdb_geocoder'
superuser = true
schema = cdb_dataservices_server

View File

@@ -0,0 +1,19 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.28.0'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(
username TEXT,
orgname TEXT,
bounds geometry(Geometry, 4326) DEFAULT NULL,
filter_tags TEXT[] DEFAULT NULL,
numer_id TEXT DEFAULT NULL,
denom_id TEXT DEFAULT NULL,
timespan TEXT DEFAULT NULL,
number_geometries INTEGER DEFAULT NULL)
RETURNS SETOF cdb_dataservices_server.obs_meta_geometry AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan, number_geometries);
$$ LANGUAGE plproxy;
DROP FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(TEXT, TEXT, geometry(Geometry, 4326), TEXT[], TEXT, TEXT, TEXT);

View File

@@ -0,0 +1,18 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.27.0'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(
username TEXT,
orgname TEXT,
bounds geometry(Geometry, 4326) DEFAULT NULL,
filter_tags TEXT[] DEFAULT NULL,
numer_id TEXT DEFAULT NULL,
denom_id TEXT DEFAULT NULL,
timespan TEXT DEFAULT NULL)
RETURNS SETOF cdb_dataservices_server.obs_meta_geometry AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan);
$$ LANGUAGE plproxy;
DROP FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(TEXT, TEXT, geometry(Geometry, 4326), TEXT[], TEXT, TEXT, TEXT, INTEGER);

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,656 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.30.0'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_route_with_waypoints(
username TEXT,
orgname TEXT,
waypoints geometry(Point, 4326)[],
mode TEXT)
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxRouting
from cartodb_services.mapbox.types import TRANSPORT_MODE_TO_MAPBOX
from cartodb_services.tools import Coordinate
from cartodb_services.tools.polyline import polyline_to_linestring
from cartodb_services.refactor.service.mapbox_routing_config import MapboxRoutingConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('routing', MapboxRoutingConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
client = MapboxRouting(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
if not waypoints or len(waypoints) < 2:
service_manager.logger.info("Empty origin or destination")
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
if len(waypoints) > 25:
service_manager.logger.info("Too many waypoints (max 25)")
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
waypoint_coords = []
for waypoint in waypoints:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % waypoint)[0]['lat']
lon = plpy.execute("SELECT ST_X('%s') AS lon" % waypoint)[0]['lon']
waypoint_coords.append(Coordinate(lon,lat))
profile = TRANSPORT_MODE_TO_MAPBOX.get(mode)
resp = client.directions(waypoint_coords, profile)
if resp and resp.shape:
shape_linestring = polyline_to_linestring(resp.shape)
if shape_linestring:
service_manager.quota_service.increment_success_service_use()
return [shape_linestring, resp.length, int(round(resp.duration))]
else:
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
else:
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to calculate Mapbox routing', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to calculate Mapbox routing')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_point_to_point(
username TEXT,
orgname TEXT,
origin geometry(Point, 4326),
destination geometry(Point, 4326),
mode TEXT,
options text[] DEFAULT ARRAY[]::text[],
units text DEFAULT 'kilometers')
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_routing_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('cdb_route_with_point', user_routing_config, logger):
waypoints = [origin, destination]
if user_routing_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
elif user_routing_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapbox_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
else:
raise Exception('Requested routing method is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_with_waypoints(
username TEXT,
orgname TEXT,
waypoints geometry(Point, 4326)[],
mode TEXT,
options text[] DEFAULT ARRAY[]::text[],
units text DEFAULT 'kilometers')
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_routing_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('cdb_route_with_waypoints', user_routing_config, logger):
if user_routing_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
elif user_routing_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapbox_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
else:
raise Exception('Requested routing method is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('cdb_geocode_street_point', user_geocoder_config, logger):
if user_geocoder_config.heremaps_geocoder:
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(here_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
elif user_geocoder_config.google_geocoder:
google_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_google_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(google_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
elif user_geocoder_config.mapzen_geocoder:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
elif user_geocoder_config.mapbox_geocoder:
mapbox_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapbox_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapbox_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
else:
raise Exception('Requested geocoder is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
# The configuration is retrieved but no checks are performed on it
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapbox_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
service_manager.assert_within_limits()
try:
geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3 = None
if country:
country_iso3 = country_to_iso3(country)
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3, search_type='address')
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapzen')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from iso3166 import countries
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxGeocoder
from cartodb_services.refactor.service.mapbox_geocoder_config import MapboxGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapboxGeocoderConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
geocoder = MapboxGeocoder(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3166 = None
if country:
country_iso3166 = countries.get(country).alpha2.lower()
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3166)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapbox', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapbox')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text)
RETURNS Geometry AS $$
import spiexceptions
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
try:
street_point = plpy.prepare("SELECT cdb_dataservices_server.cdb_geocode_street_point($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(street_point, [username, orgname, city_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
import sys
logger.error('Error geocoding namedplace using geocode street point, falling back to internal geocoder', sys.exc_info(), data={"username": username, "orgname": orgname})
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, country_name text)
RETURNS Geometry AS $$
import spiexceptions
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
try:
street_point = plpy.prepare("SELECT cdb_dataservices_server.cdb_geocode_street_point($1, $2, $3, NULL, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(street_point, [username, orgname, city_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
import sys
logger.error('Error geocoding namedplace using geocode street point, falling back to internal geocoder', sys.exc_info(), data={"username": username, "orgname": orgname})
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, country_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, admin1_name text, country_name text)
RETURNS Geometry AS $$
import spiexceptions
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
try:
street_point = plpy.prepare("SELECT cdb_dataservices_server.cdb_geocode_street_point($1, $2, $3, NULL, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(street_point, [username, orgname, city_name, admin1_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
import sys
logger.error('Error geocoding namedplace using geocode street point, falling back to internal geocoder', sys.exc_info(), data={"username": username, "orgname": orgname})
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_isodistance(
username TEXT,
orgname TEXT,
source geometry(Geometry, 4326),
mode TEXT,
data_range integer[],
options text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxMatrixClient, MapboxIsolines
from cartodb_services.mapbox.types import TRANSPORT_MODE_TO_MAPBOX
from cartodb_services.tools import Coordinate
from cartodb_services.refactor.service.mapbox_isolines_config import MapboxIsolinesConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('isolines', MapboxIsolinesConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
client = MapboxMatrixClient(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
mapbox_isolines = MapboxIsolines(client, service_manager.logger)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
origin = Coordinate(lon,lat)
else:
raise Exception('source is NULL')
profile = TRANSPORT_MODE_TO_MAPBOX.get(mode)
# -- TODO Support options properly
isolines = {}
for r in data_range:
isoline = mapbox_isolines.calculate_isodistance(origin, r, profile)
isolines[r] = isoline
result = []
for r in data_range:
if len(isolines[r]) >= 3:
# -- TODO encapsulate this block into a func/method
locations = isolines[r] + [ isolines[r][0] ] # close the polygon repeating the first point
wkt_coordinates = ','.join(["%f %f" % (l.longitude, l.latitude) for l in locations])
sql = "SELECT ST_MPolyFromText('MULTIPOLYGON((({0})))', 4326) as geom".format(wkt_coordinates)
multipolygon = plpy.execute(sql, 1)[0]['geom']
else:
multipolygon = None
result.append([source, r, multipolygon])
service_manager.quota_service.increment_success_service_use()
service_manager.quota_service.increment_isolines_service_use(len(isolines))
return result
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to get Mapbox isolines', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to get Mapbox isolines')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_isochrones(
username TEXT,
orgname TEXT,
source geometry(Geometry, 4326),
mode TEXT,
data_range integer[],
options text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxMatrixClient, MapboxIsolines
from cartodb_services.mapbox.types import TRANSPORT_MODE_TO_MAPBOX
from cartodb_services.tools import Coordinate
from cartodb_services.tools.coordinates import coordinates_to_polygon
from cartodb_services.refactor.service.mapbox_isolines_config import MapboxIsolinesConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('isolines', MapboxIsolinesConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
client = MapboxMatrixClient(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
mapbox_isolines = MapboxIsolines(client, service_manager.logger)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
origin = Coordinate(lon,lat)
else:
raise Exception('source is NULL')
profile = TRANSPORT_MODE_TO_MAPBOX.get(mode)
resp = mapbox_isolines.calculate_isochrone(origin, data_range, profile)
if resp:
result = []
for isochrone in resp:
result_polygon = coordinates_to_polygon(isochrone.coordinates)
if result_polygon:
service_manager.quota_service.increment_success_service_use()
result.append([source, isochrone.duration, result_polygon])
else:
service_manager.quota_service.increment_empty_service_use()
result.append([source, isochrone.duration, None])
service_manager.quota_service.increment_success_service_use()
service_manager.quota_service.increment_isolines_service_use(len(result))
return result
else:
service_manager.quota_service.increment_empty_service_use()
return []
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to get Mapbox isochrones', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to get Mapbox isochrones')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
if user_isolines_config.google_services_user:
raise Exception('This service is not available for google service users.')
with metrics('cb_isodistance', user_isolines_config, logger):
if user_isolines_config.heremaps_provider:
here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapbox_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
else:
raise Exception('Requested isolines provider is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- mapbox isodistance
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapbox_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
result = plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
return result
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
if user_isolines_config.google_services_user:
raise Exception('This service is not available for google service users.')
with metrics('cb_isochrone', user_isolines_config, logger):
if user_isolines_config.heremaps_provider:
here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapbox_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
else:
raise Exception('Requested isolines provider is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- mapbox isochrone
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapbox_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_isochrones($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
result = plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
return result
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
username TEXT,
orgname TEXT,
geomvals geomval[],
params JSON,
merge BOOLEAN DEFAULT True)
RETURNS TABLE (
id INT,
data JSON
) AS $$
from cartodb_services.metrics import metrics
from cartodb_services.metrics import QuotaService
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_config = GD["user_obs_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
quota_service = QuotaService(user_obs_config, redis_conn)
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('obs_getdata', user_obs_config, logger, params):
try:
obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4, $5);", ["text", "text", "geomval[]", "json", "boolean"])
result = plpy.execute(obs_plan, [username, orgname, geomvals, params, merge])
empty_results = len(geomvals) - len(result)
if empty_results > 0:
quota_service.increment_empty_service_use(empty_results)
if result:
quota_service.increment_success_service_use(len(result))
return result
else:
return []
except BaseException as e:
import sys
quota_service.increment_failed_service_use(len(geomvals))
logger.error('Error trying to OBS_GetData', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetData')
finally:
quota_service.increment_total_service_use(len(geomvals))
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
username TEXT,
orgname TEXT,
geomrefs TEXT[],
params JSON)
RETURNS TABLE (
id TEXT,
data JSON
) AS $$
from cartodb_services.metrics import metrics
from cartodb_services.metrics import QuotaService
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_config = GD["user_obs_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
quota_service = QuotaService(user_obs_config, redis_conn)
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('obs_getdata', user_obs_config, logger, params):
try:
obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4);", ["text", "text", "text[]", "json"])
result = plpy.execute(obs_plan, [username, orgname, geomrefs, params])
empty_results = len(geomrefs) - len(result)
if empty_results > 0:
quota_service.increment_empty_service_use(empty_results)
if result:
quota_service.increment_success_service_use(len(result))
return result
else:
return []
except BaseException as e:
import sys
quota_service.increment_failed_service_use(len(geomrefs))
exc_info = sys.exc_info()
logger.error('%s, %s, %s' % (exc_info[0], exc_info[1], exc_info[2]))
logger.error('Error trying to OBS_GetData', exc_info, data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetData')
finally:
quota_service.increment_total_service_use(len(geomrefs))
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeta(
username TEXT,
orgname TEXT,
geom Geometry(Geometry, 4326),
params JSON,
max_timespan_rank INTEGER DEFAULT NULL,
max_score_rank INTEGER DEFAULT NULL,
target_geoms INTEGER DEFAULT NULL)
RETURNS JSON AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_config = GD["user_obs_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('obs_getmeta', user_obs_config, logger, params):
try:
obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeta($1, $2, $3, $4, $5, $6, $7) as meta;", ["text", "text", "Geometry (Geometry, 4326)", "json", "integer", "integer", "integer"])
result = plpy.execute(obs_plan, [username, orgname, geom, params, max_timespan_rank, max_score_rank, target_geoms])
if result:
return result[0]['meta']
else:
return None
except BaseException as e:
import sys
logger.error('Error trying to OBS_GetMeta', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetMeta')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,400 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.29.0'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapbox_route_with_waypoints(username TEXT, orgname TEXT, waypoints geometry(Point, 4326)[], mode TEXT);
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_point_to_point(
username TEXT,
orgname TEXT,
origin geometry(Point, 4326),
destination geometry(Point, 4326),
mode TEXT,
options text[] DEFAULT ARRAY[]::text[],
units text DEFAULT 'kilometers')
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_routing_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('cdb_route_with_point', user_routing_config, logger):
waypoints = [origin, destination]
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_with_waypoints(
username TEXT,
orgname TEXT,
waypoints geometry(Point, 4326)[],
mode TEXT,
options text[] DEFAULT ARRAY[]::text[],
units text DEFAULT 'kilometers')
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_routing_config = GD["user_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('cdb_route_with_waypoints', user_routing_config, logger):
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('cdb_geocode_street_point', user_geocoder_config, logger):
if user_geocoder_config.heremaps_geocoder:
here_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_here_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(here_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
elif user_geocoder_config.google_geocoder:
google_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_google_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(google_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
elif user_geocoder_config.mapzen_geocoder:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
else:
raise Exception('Requested geocoder is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT, state_province TEXT, country TEXT);
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.mapzen.types import country_to_iso3
from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
service_manager.assert_within_limits()
try:
geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3 = None
if country:
country_iso3 = country_to_iso3(country)
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3, search_type='address')
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapzen')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT, state_province TEXT, country TEXT);
---- cdb_geocode_namedplace_point(city_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text)
RETURNS Geometry AS $$
import spiexceptions
try:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, city_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, country_name text)
RETURNS Geometry AS $$
import spiexceptions
try:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, city_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, country_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, admin1_name text, country_name text)
RETURNS Geometry AS $$
import spiexceptions
try:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapbox_geocode_namedplace(username text, orgname text, city_name text, admin1_name text, country_name text);
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_namedplace(username text, orgname text, city_name text, admin1_name text DEFAULT NULL, country_name text DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.mapzen.types import country_to_iso3
from cartodb_services.metrics import QuotaService, metrics
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1}, {2})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname), plpy.quote_nullable('mapzen')))
user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
quota_service = QuotaService(user_geocoder_config, redis_conn)
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('cdb_geocode_namedplace_point', user_geocoder_config, logger):
try:
geocoder = MapzenGeocoder(user_geocoder_config.mapzen_api_key, logger)
country_iso3 = None
if country_name:
country_iso3 = country_to_iso3(country_name)
coordinates = geocoder.geocode(searchtext=city_name, city=None,
state_province=admin1_name,
country=country_iso3, search_type='locality')
if coordinates:
quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
quota_service.increment_failed_service_use()
logger.error('Error trying to geocode city point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode city point using mapzen')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapbox_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, data_range integer[], options text[]);
DROP FUNCTION IF EXISTS cdb_dataservices_server._cdb_mapbox_isochrones(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, data_range integer[], options text[]);
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
if user_isolines_config.google_services_user:
raise Exception('This service is not available for google service users.')
with metrics('cb_isodistance', user_isolines_config, logger):
if user_isolines_config.heremaps_provider:
here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
else:
raise Exception('Requested isolines provider is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_mapbox_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[]);
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
if user_isolines_config.google_services_user:
raise Exception('This service is not available for google service users.')
with metrics('cb_isochrone', user_isolines_config, logger):
if user_isolines_config.heremaps_provider:
here_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_here_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(here_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
else:
raise Exception('Requested isolines provider is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
DROP FUNCTION IF EXISTS cdb_dataservices_server.cdb_mapbox_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[]);
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
username TEXT,
orgname TEXT,
geomvals geomval[],
params JSON,
merge BOOLEAN DEFAULT True)
RETURNS TABLE (
id INT,
data JSON
) AS $$
from cartodb_services.metrics import metrics
from cartodb_services.metrics import QuotaService
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_config = GD["user_obs_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
quota_service = QuotaService(user_obs_config, redis_conn)
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('obs_getdata', user_obs_config, logger):
try:
obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4, $5);", ["text", "text", "geomval[]", "json", "boolean"])
result = plpy.execute(obs_plan, [username, orgname, geomvals, params, merge])
empty_results = len(geomvals) - len(result)
if empty_results > 0:
quota_service.increment_empty_service_use(empty_results)
if result:
quota_service.increment_success_service_use(len(result))
return result
else:
return []
except BaseException as e:
import sys
quota_service.increment_failed_service_use(len(geomvals))
logger.error('Error trying to OBS_GetData', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetData')
finally:
quota_service.increment_total_service_use(len(geomvals))
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
username TEXT,
orgname TEXT,
geomrefs TEXT[],
params JSON)
RETURNS TABLE (
id TEXT,
data JSON
) AS $$
from cartodb_services.metrics import metrics
from cartodb_services.metrics import QuotaService
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_config = GD["user_obs_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
quota_service = QuotaService(user_obs_config, redis_conn)
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('obs_getdata', user_obs_config, logger):
try:
obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4);", ["text", "text", "text[]", "json"])
result = plpy.execute(obs_plan, [username, orgname, geomrefs, params])
empty_results = len(geomrefs) - len(result)
if empty_results > 0:
quota_service.increment_empty_service_use(empty_results)
if result:
quota_service.increment_success_service_use(len(result))
return result
else:
return []
except BaseException as e:
import sys
quota_service.increment_failed_service_use(len(geomrefs))
exc_info = sys.exc_info()
logger.error('%s, %s, %s' % (exc_info[0], exc_info[1], exc_info[2]))
logger.error('Error trying to OBS_GetData', exc_info, data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetData')
finally:
quota_service.increment_total_service_use(len(geomrefs))
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeta(
username TEXT,
orgname TEXT,
geom Geometry(Geometry, 4326),
params JSON,
max_timespan_rank INTEGER DEFAULT NULL,
max_score_rank INTEGER DEFAULT NULL,
target_geoms INTEGER DEFAULT NULL)
RETURNS JSON AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_obs_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_obs_config = GD["user_obs_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('obs_getmeta', user_obs_config, logger):
try:
obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeta($1, $2, $3, $4, $5, $6, $7) as meta;", ["text", "text", "Geometry (Geometry, 4326)", "json", "integer", "integer", "integer"])
result = plpy.execute(obs_plan, [username, orgname, geom, params, max_timespan_rank, max_score_rank, target_geoms])
if result:
return result[0]['meta']
else:
return None
except BaseException as e:
import sys
logger.error('Error trying to OBS_GetMeta', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetMeta')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -0,0 +1,47 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.30.1'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from iso3166 import countries
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapbox_geocoder_config import MapboxGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapboxGeocoderConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
geocoder = MapboxGeocoder(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3166 = None
if country:
country_iso3 = country_to_iso3(country)
if country_iso3:
country_iso3166 = countries.get(country_iso3).alpha2.lower()
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3166)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapbox', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapbox')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,44 @@
--DO NOT MODIFY THIS FILE, IT IS GENERATED AUTOMATICALLY FROM SOURCES
-- Complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION cdb_dataservices_server UPDATE TO '0.30.0'" to load this file. \quit
-- HERE goes your code to upgrade/downgrade
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from iso3166 import countries
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxGeocoder
from cartodb_services.refactor.service.mapbox_geocoder_config import MapboxGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapboxGeocoderConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
geocoder = MapboxGeocoder(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3166 = None
if country:
country_iso3166 = countries.get(country).alpha2.lower()
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3166)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapbox', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapbox')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

File diff suppressed because it is too large Load Diff

View File

@@ -4,6 +4,67 @@ CREATE TYPE cdb_dataservices_server.simple_route AS (
duration integer
);
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_route_with_waypoints(
username TEXT,
orgname TEXT,
waypoints geometry(Point, 4326)[],
mode TEXT)
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxRouting
from cartodb_services.mapbox.types import TRANSPORT_MODE_TO_MAPBOX
from cartodb_services.tools import Coordinate
from cartodb_services.tools.polyline import polyline_to_linestring
from cartodb_services.refactor.service.mapbox_routing_config import MapboxRoutingConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('routing', MapboxRoutingConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
client = MapboxRouting(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
if not waypoints or len(waypoints) < 2:
service_manager.logger.info("Empty origin or destination")
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
if len(waypoints) > 25:
service_manager.logger.info("Too many waypoints (max 25)")
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
waypoint_coords = []
for waypoint in waypoints:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % waypoint)[0]['lat']
lon = plpy.execute("SELECT ST_X('%s') AS lon" % waypoint)[0]['lon']
waypoint_coords.append(Coordinate(lon,lat))
profile = TRANSPORT_MODE_TO_MAPBOX.get(mode)
resp = client.directions(waypoint_coords, profile)
if resp and resp.shape:
shape_linestring = polyline_to_linestring(resp.shape)
if shape_linestring:
service_manager.quota_service.increment_success_service_use()
return [shape_linestring, resp.length, int(round(resp.duration))]
else:
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
else:
service_manager.quota_service.increment_empty_service_use()
return [None, None, None]
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to calculate Mapbox routing', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to calculate Mapbox routing')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_route_with_waypoints(
username TEXT,
orgname TEXT,
@@ -63,4 +124,4 @@ RETURNS cdb_dataservices_server.simple_route AS $$
raise Exception('Error trying to calculate mapzen routing')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;

View File

@@ -9,6 +9,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_point_to_point(
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
@@ -19,10 +20,18 @@ RETURNS cdb_dataservices_server.simple_route AS $$
with metrics('cdb_route_with_point', user_routing_config, logger):
waypoints = [origin, destination]
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
$$ LANGUAGE plpythonu;
if user_routing_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
elif user_routing_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapbox_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
else:
raise Exception('Requested routing method is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_with_waypoints(
@@ -35,6 +44,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_route_with_waypoints(
RETURNS cdb_dataservices_server.simple_route AS $$
from cartodb_services.metrics import metrics
from cartodb_services.tools import Logger
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
@@ -44,7 +54,14 @@ RETURNS cdb_dataservices_server.simple_route AS $$
logger = Logger(logger_config)
with metrics('cdb_route_with_waypoints', user_routing_config, logger):
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4, $5, $6) as route;", ["text", "text", "geometry(Point, 4326)[]", "text", "text[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode, options, units])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
$$ LANGUAGE plpythonu;
if user_routing_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapzen_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
elif user_routing_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_route_with_waypoints($1, $2, $3, $4) as route;", ["text", "text", "geometry(Point, 4326)[]", "text"])
result = plpy.execute(mapbox_plan, [username, orgname, waypoints, mode])
return [result[0]['shape'],result[0]['length'], result[0]['duration']]
else:
raise Exception('Requested routing method is not available')
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -15,4 +15,4 @@ RETURNS boolean AS $$
'redis_metrics_connection': redis_metrics_connection,
}
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;

View File

@@ -14,7 +14,7 @@ RETURNS text AS $$
user_obs_config = GD["user_obs_snapshot_config_{0}".format(username)]
return user_obs_config.connection_str
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetDemographicSnapshotJSON(
username TEXT,
@@ -25,7 +25,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetDemographicSnapshotJS
RETURNS json AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetDemographicSnapshot(geom, time_span, geometry_level);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_demographic_snapshot(
username TEXT,
@@ -68,7 +68,7 @@ RETURNS json AS $$
raise Exception('Error trying to obs_get_demographic_snapshot')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetDemographicSnapshot(
username TEXT,
@@ -79,7 +79,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetDemographicSnapshot(
RETURNS SETOF json AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetDemographicSnapshot(geom, time_span, geometry_level);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetDemographicSnapshot(
username TEXT,
@@ -125,7 +125,7 @@ RETURNS SETOF JSON AS $$
raise Exception('Error trying to obs_get_demographic_snapshot')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetSegmentSnapshotJSON(
username TEXT,
@@ -135,7 +135,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetSegmentSnapshotJSON(
RETURNS json AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetSegmentSnapshot(geom, geometry_level);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_get_segment_snapshot(
username TEXT,
@@ -177,7 +177,7 @@ RETURNS json AS $$
raise Exception('Error trying to obs_get_segment_snapshot')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetSegmentSnapshot(
username TEXT,
@@ -187,7 +187,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetSegmentSnapshot(
RETURNS SETOF json AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetSegmentSnapshot(geom, geometry_level);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetSegmentSnapshot(
username TEXT,
@@ -232,7 +232,7 @@ RETURNS SETOF JSON AS $$
raise Exception('Error trying to OBS_GetSegmentSnapshot')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeasure(
username TEXT,
@@ -245,7 +245,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeasure(
RETURNS NUMERIC AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetMeasure(geom, measure_id, normalize, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeasure(
username TEXT,
@@ -289,7 +289,7 @@ RETURNS NUMERIC AS $$
raise Exception('Error trying to OBS_GetMeasure')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetCategory(
username TEXT,
@@ -301,7 +301,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetCategory(
RETURNS TEXT AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetCategory(geom, category_id, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetCategory(
username TEXT,
@@ -344,7 +344,7 @@ RETURNS TEXT AS $$
raise Exception('Error trying to OBS_GetCategory')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetUSCensusMeasure(
username TEXT,
@@ -357,7 +357,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetUSCensusMeasure(
RETURNS NUMERIC AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetUSCensusMeasure(geom, name, normalize, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetUSCensusMeasure(
username TEXT,
@@ -401,7 +401,7 @@ RETURNS NUMERIC AS $$
raise Exception('Error trying to OBS_GetUSCensusMeasure')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetUSCensusCategory(
username TEXT,
@@ -413,7 +413,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetUSCensusCategory(
RETURNS TEXT AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetUSCensusCategory(geom, name, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetUSCensusCategory(
username TEXT,
@@ -456,7 +456,7 @@ RETURNS TEXT AS $$
raise Exception('Error trying to OBS_GetUSCensusCategory')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPopulation(
username TEXT,
@@ -468,7 +468,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPopulation(
RETURNS NUMERIC AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetPopulation(geom, normalize, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetPopulation(
username TEXT,
@@ -511,7 +511,7 @@ RETURNS NUMERIC AS $$
raise Exception('Error trying to OBS_GetPopulation')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeasureById(
username TEXT,
@@ -523,7 +523,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeasureById(
RETURNS NUMERIC AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetMeasureById(geom_ref, measure_id, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeasureById(
username TEXT,
@@ -566,7 +566,7 @@ RETURNS NUMERIC AS $$
raise Exception('Error trying to OBS_GetMeasureById')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetData(
username TEXT,
@@ -580,7 +580,7 @@ RETURNS TABLE (
) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetData(geomvals, params, merge);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
username TEXT,
@@ -608,7 +608,7 @@ RETURNS TABLE (
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('obs_getdata', user_obs_config, logger):
with metrics('obs_getdata', user_obs_config, logger, params):
try:
obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4, $5);", ["text", "text", "geomval[]", "json", "boolean"])
result = plpy.execute(obs_plan, [username, orgname, geomvals, params, merge])
@@ -627,7 +627,7 @@ RETURNS TABLE (
raise Exception('Error trying to OBS_GetData')
finally:
quota_service.increment_total_service_use(len(geomvals))
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetData(
username TEXT,
@@ -640,7 +640,7 @@ RETURNS TABLE (
) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetData(geomrefs, params);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetData(
username TEXT,
@@ -667,7 +667,7 @@ RETURNS TABLE (
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('obs_getdata', user_obs_config, logger):
with metrics('obs_getdata', user_obs_config, logger, params):
try:
obs_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._OBS_GetData($1, $2, $3, $4);", ["text", "text", "text[]", "json"])
result = plpy.execute(obs_plan, [username, orgname, geomrefs, params])
@@ -688,7 +688,7 @@ RETURNS TABLE (
raise Exception('Error trying to OBS_GetData')
finally:
quota_service.increment_total_service_use(len(geomrefs))
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeta(
username TEXT,
@@ -701,7 +701,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetMeta(
RETURNS JSON AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetMeta(geom, params, max_timespan_rank, max_score_rank, target_geoms);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetMeta(
username TEXT,
@@ -724,7 +724,7 @@ RETURNS JSON AS $$
logger_config = GD["logger_config"]
logger = Logger(logger_config)
with metrics('obs_getmeta', user_obs_config, logger):
with metrics('obs_getmeta', user_obs_config, logger, params):
try:
obs_plan = plpy.prepare("SELECT cdb_dataservices_server._OBS_GetMeta($1, $2, $3, $4, $5, $6, $7) as meta;", ["text", "text", "Geometry (Geometry, 4326)", "json", "integer", "integer", "integer"])
result = plpy.execute(obs_plan, [username, orgname, geom, params, max_timespan_rank, max_score_rank, target_geoms])
@@ -736,7 +736,7 @@ RETURNS JSON AS $$
import sys
logger.error('Error trying to OBS_GetMeta', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_GetMeta')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_MetadataValidation(
username TEXT,
@@ -748,7 +748,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_MetadataValidation(
RETURNS TABLE(valid boolean, errors text[]) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_MetadataValidation(geometry_extent, geometry_type, params, target_geoms);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_MetadataValidation(
username TEXT,
@@ -782,4 +782,4 @@ RETURNS TABLE(valid boolean, errors text[]) AS $$
import sys
logger.error('Error trying to OBS_MetadataValidation', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to OBS_MetadataValidation')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -6,7 +6,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_Search(
RETURNS TABLE(id text, description text, name text, aggregate text, source text) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_Search(search_term, relevant_boundary);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_Search(
username TEXT,
@@ -55,7 +55,7 @@ RETURNS TABLE(id text, description text, name text, aggregate text, source text)
raise Exception('Error trying to OBS_Search')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetAvailableBoundaries(
username TEXT,
@@ -65,7 +65,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetAvailableBoundaries(
RETURNS TABLE(boundary_id text, description text, time_span text, tablename text) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableBoundaries(geom, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableBoundaries(
username TEXT,
@@ -113,4 +113,4 @@ RETURNS TABLE(boundary_id text, description text, time_span text, tablename text
raise Exception('Error trying to OBS_GetMeasureById')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -7,7 +7,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundary(
RETURNS geometry(Geometry, 4326) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetBoundary(geom, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundary(
username TEXT,
@@ -49,7 +49,7 @@ RETURNS geometry(Geometry, 4326) AS $$
raise Exception('Error trying to OBS_GetBoundary')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundaryId(
username TEXT,
@@ -60,7 +60,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundaryId(
RETURNS TEXT AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetBoundaryId(geom, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundaryId(
username TEXT,
@@ -102,7 +102,7 @@ RETURNS TEXT AS $$
raise Exception('Error trying to OBS_GetBoundaryId')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundaryById(
username TEXT,
@@ -113,7 +113,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundaryById(
RETURNS geometry(Geometry, 4326) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.OBS_GetBoundaryById(geometry_id, boundary_id, time_span);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundaryById(
username TEXT,
@@ -155,7 +155,7 @@ RETURNS geometry(Geometry, 4326) AS $$
raise Exception('Error trying to OBS_GetBoundaryById')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundariesByGeometry(
username TEXT,
@@ -167,7 +167,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundariesByGeometry(
RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetBoundariesByGeometry(geom, boundary_id, time_span, overlap_type);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundariesByGeometry(
username TEXT,
@@ -215,7 +215,7 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
raise Exception('Error trying to OBS_GetBoundariesByGeometry')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundariesByPointAndRadius(
username TEXT,
@@ -228,7 +228,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetBoundariesByPointAndR
RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetBoundariesByPointAndRadius(geom, radius, boundary_id, time_span, overlap_type);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetBoundariesByPointAndRadius(
username TEXT,
@@ -277,7 +277,7 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
raise Exception('Error trying to OBS_GetBoundariesByPointAndRadius')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPointsByGeometry(
username TEXT,
@@ -289,7 +289,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPointsByGeometry(
RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetPointsByGeometry(geom, boundary_id, time_span, overlap_type);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetPointsByGeometry(
username TEXT,
@@ -337,7 +337,7 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
raise Exception('Error trying to OBS_GetPointsByGeometry')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPointsByPointAndRadius(
username TEXT,
@@ -350,7 +350,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetPointsByPointAndRadiu
RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetPointsByPointAndRadius(geom, radius, boundary_id, time_span, overlap_type);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetPointsByPointAndRadius(
username TEXT,
@@ -399,4 +399,4 @@ RETURNS TABLE(the_geom geometry, geom_refs text) AS $$
raise Exception('Error trying to OBS_GetPointsByPointAndRadius')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -8,29 +8,29 @@ RETURNS cdb_dataservices_server.ds_fdw_metadata AS $$
return plpy.execute("SELECT * FROM cdb_dataservices_server.__DST_ConnectUserTable({username}::text, {orgname}::text, {user_db_role}::text, {schema}::text, {dbname}::text, {host_addr}::text, {table_name}::text)"
.format(username=plpy.quote_nullable(username), orgname=plpy.quote_nullable(orgname), user_db_role=plpy.quote_literal(user_db_role), schema=plpy.quote_literal(input_schema), dbname=plpy.quote_literal(dbname), table_name=plpy.quote_literal(table_name), host_addr=plpy.quote_literal(host_addr))
)[0]
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.__DST_ConnectUserTable(username text, orgname text, user_db_role text, input_schema text, dbname text, host_addr text, table_name text)
RETURNS cdb_dataservices_server.ds_fdw_metadata AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
TARGET cdb_observatory._OBS_ConnectUserTable;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._DST_GetReturnMetadata(username text, orgname text, function_name text, params json)
RETURNS cdb_dataservices_server.ds_return_metadata AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
TARGET cdb_observatory._OBS_GetReturnMetadata;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._DST_FetchJoinFdwTableData(username text, orgname text, table_schema text, table_name text, function_name text, params json)
RETURNS SETOF record AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
TARGET cdb_observatory._OBS_FetchJoinFdwTableData;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._DST_DisconnectUserTable(username text, orgname text, table_schema text, table_name text, servername text)
RETURNS boolean AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
TARGET cdb_observatory._OBS_DisconnectUserTable;
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;

View File

@@ -2,7 +2,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.obs_dumpversion(username text
RETURNS text AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT cdb_observatory.obs_dumpversion();
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
-- We could create a super type for the common data like id, name and so on but we need to parse inside the functions because the -- the return data tha comes from OBS is a TABLE() with them
CREATE TYPE cdb_dataservices_server.obs_meta_numerator AS (numer_id text, numer_name text, numer_description text, numer_weight text, numer_license text, numer_source text, numer_type text, numer_aggregate text, numer_extra jsonb, numer_tags jsonb, valid_denom boolean, valid_geom boolean, valid_timespan boolean);
@@ -18,7 +18,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableNumerators(
RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetNumerators(
username TEXT,
@@ -35,7 +35,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server._OBS_GetNumerators(
RETURNS SETOF cdb_dataservices_server.obs_meta_numerator AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory._OBS_GetNumerators(bounds, section_tags, subsection_tags, other_tags, ids, name, denom_id, geom_id, timespan);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE TYPE cdb_dataservices_server.obs_meta_denominator AS (denom_id text, denom_name text, denom_description text, denom_weight text, denom_license text, denom_source text, denom_type text, denom_aggregate text, denom_extra jsonb, denom_tags jsonb, valid_numer boolean, valid_geom boolean, valid_timespan boolean);
@@ -50,7 +50,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableDenominators(
RETURNS SETOF cdb_dataservices_server.obs_meta_denominator AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE TYPE cdb_dataservices_server.obs_meta_geometry AS (geom_id text, geom_name text, geom_description text, geom_weight text, geom_aggregate text, geom_license text, geom_source text, valid_numer boolean, valid_denom boolean, valid_timespan boolean, score numeric, numtiles bigint, notnull_percent numeric, numgeoms numeric, percentfill numeric, estnumgeoms numeric, meanmediansize numeric, geom_type text, geom_extra jsonb, geom_tags jsonb);
@@ -61,11 +61,12 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableGeometries(
filter_tags TEXT[] DEFAULT NULL,
numer_id TEXT DEFAULT NULL,
denom_id TEXT DEFAULT NULL,
timespan TEXT DEFAULT NULL)
timespan TEXT DEFAULT NULL,
number_geometries INTEGER DEFAULT NULL)
RETURNS SETOF cdb_dataservices_server.obs_meta_geometry AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan);
$$ LANGUAGE plproxy;
SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan, number_geometries);
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE TYPE cdb_dataservices_server.obs_meta_timespan AS (timespan_id text, timespan_name text, timespan_description text, timespan_weight text, timespan_aggregate text, timespan_license text, timespan_source text, valid_numer boolean, valid_denom boolean, valid_geom boolean, timespan_type text, timespan_extra jsonb, timespan_tags jsonb);
@@ -80,7 +81,7 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_GetAvailableTimespans(
RETURNS SETOF cdb_dataservices_server.obs_meta_timespan AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_LegacyBuilderMetadata(
username TEXT,
@@ -89,4 +90,4 @@ CREATE OR REPLACE FUNCTION cdb_dataservices_server.OBS_LegacyBuilderMetadata(
RETURNS TABLE(name TEXT, subsection JSON) AS $$
CONNECT cdb_dataservices_server._obs_server_conn_str(username, orgname);
SELECT * FROM cdb_observatory.OBS_LegacyBuilderMetadata(aggregate_type);
$$ LANGUAGE plproxy;
$$ LANGUAGE plproxy VOLATILE PARALLEL UNSAFE;

View File

@@ -8,13 +8,13 @@ RETURNS boolean AS $$
logger_config = LoggerConfig(plpy)
GD[cache_key] = logger_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
-- This is done in order to avoid an undesired depedency on cartodb extension
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_conf_getconf(input_key text)
RETURNS JSON AS $$
SELECT VALUE FROM cartodb.cdb_conf WHERE key = input_key;
$$ LANGUAGE SQL STABLE SECURITY DEFINER;
$$ LANGUAGE SQL SECURITY DEFINER STABLE PARALLEL SAFE;
CREATE OR REPLACE
FUNCTION cdb_dataservices_server.CDB_Conf_SetConf(key text, value JSON)
@@ -23,7 +23,7 @@ BEGIN
PERFORM cdb_dataservices_server.CDB_Conf_RemoveConf(key);
EXECUTE 'INSERT INTO cartodb.CDB_CONF (KEY, VALUE) VALUES ($1, $2);' USING key, value;
END
$$ LANGUAGE PLPGSQL VOLATILE SECURITY DEFINER;
$$ LANGUAGE PLPGSQL SECURITY DEFINER VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE
FUNCTION cdb_dataservices_server.CDB_Conf_RemoveConf(key text)
@@ -31,7 +31,7 @@ FUNCTION cdb_dataservices_server.CDB_Conf_RemoveConf(key text)
BEGIN
EXECUTE 'DELETE FROM cartodb.CDB_CONF WHERE KEY = $1;' USING key;
END
$$ LANGUAGE PLPGSQL VOLATILE SECURITY DEFINER;
$$ LANGUAGE PLPGSQL SECURITY DEFINER VOLATILE PARALLEL UNSAFE ;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_geocoder_config(username text, orgname text, provider text DEFAULT NULL)
@@ -46,7 +46,7 @@ RETURNS boolean AS $$
geocoder_config = GeocoderConfig(redis_conn, plpy, username, orgname, provider)
GD[cache_key] = geocoder_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_internal_geocoder_config(username text, orgname text)
RETURNS boolean AS $$
@@ -60,7 +60,7 @@ RETURNS boolean AS $$
geocoder_config = InternalGeocoderConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = geocoder_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_isolines_routing_config(username text, orgname text)
RETURNS boolean AS $$
@@ -74,7 +74,7 @@ RETURNS boolean AS $$
isolines_routing_config = IsolinesRoutingConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = isolines_routing_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_routing_config(username text, orgname text)
RETURNS boolean AS $$
@@ -102,7 +102,7 @@ RETURNS boolean AS $$
obs_snapshot_config = ObservatorySnapshotConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = obs_snapshot_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._get_obs_config(username text, orgname text)
RETURNS boolean AS $$
@@ -116,4 +116,4 @@ RETURNS boolean AS $$
obs_config = ObservatoryConfig(redis_conn, plpy, username, orgname)
GD[cache_key] = obs_config
return True
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;

View File

@@ -1,6 +1,8 @@
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'service_type') THEN
IF NOT EXISTS (SELECT 1 FROM pg_type inner join pg_namespace ON (pg_type.typnamespace = pg_namespace.oid)
WHERE pg_type.typname = 'service_type'
AND pg_namespace.nspname = 'cdb_dataservices_server') THEN
CREATE TYPE cdb_dataservices_server.service_type AS ENUM (
'isolines',
'hires_geocoder',
@@ -12,7 +14,9 @@ END $$;
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'service_quota_info') THEN
IF NOT EXISTS (SELECT 1 FROM pg_type inner join pg_namespace ON (pg_type.typnamespace = pg_namespace.oid)
WHERE pg_type.typname = 'service_quota_info'
AND pg_namespace.nspname = 'cdb_dataservices_server') THEN
CREATE TYPE cdb_dataservices_server.service_quota_info AS (
service cdb_dataservices_server.service_type,
monthly_quota NUMERIC,
@@ -85,7 +89,7 @@ RETURNS SETOF cdb_dataservices_server.service_quota_info AS $$
ret += [[service, monthly_quota, used_quota, soft_limit, provider]]
return ret
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_enough_quota(
@@ -102,4 +106,4 @@ returns BOOLEAN AS $$
WHERE p.service = service_::cdb_dataservices_server.service_type;
RETURN params.soft_limit OR ((params.used_quota + input_size) <= params.monthly_quota);
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL RESTRICTED;

View File

@@ -1,4 +1,5 @@
-- Geocodes a street address given a searchtext and a state and/or country
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.metrics import metrics
@@ -21,10 +22,13 @@ RETURNS Geometry AS $$
elif user_geocoder_config.mapzen_geocoder:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
elif user_geocoder_config.mapbox_geocoder:
mapbox_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapbox_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapbox_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
else:
raise Exception('Requested geocoder is not available')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
@@ -40,7 +44,7 @@ RETURNS Geometry AS $$
else:
raise Exception('Here geocoder is not available for your account.')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_google_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
@@ -55,7 +59,7 @@ RETURNS Geometry AS $$
else:
raise Exception('Google geocoder is not available for your account.')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
@@ -68,18 +72,32 @@ RETURNS Geometry AS $$
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
# The configuration is retrieved but no checks are performed on it
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapbox_geocode_street_point($1, $2, $3, $4, $5, $6) as point; ", ["text", "text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, searchtext, city, state_province, country], 1)[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_here_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import LegacyServiceManager
from cartodb_services.tools import QuotaExceededException
from cartodb_services.here import HereMapsGeocoder
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
service_manager.assert_within_limits()
try:
service_manager.assert_within_limits()
geocoder = HereMapsGeocoder(service_manager.config.heremaps_app_id, service_manager.config.heremaps_app_code, service_manager.logger, service_manager.config.heremaps_service_params)
coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
if coordinates:
@@ -90,6 +108,9 @@ RETURNS Geometry AS $$
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
@@ -97,18 +118,18 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode street point using here maps')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_google_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import LegacyServiceManager
from cartodb_services.tools import LegacyServiceManager, QuotaExceededException
from cartodb_services.google import GoogleMapsGeocoder
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
service_manager = LegacyServiceManager('geocoder', username, orgname, GD)
service_manager.assert_within_limits(quota=False)
try:
service_manager.assert_within_limits(quota=False)
geocoder = GoogleMapsGeocoder(service_manager.config.google_client_id, service_manager.config.google_api_key, service_manager.logger)
coordinates = geocoder.geocode(searchtext=searchtext, city=city, state=state_province, country=country)
if coordinates:
@@ -119,6 +140,9 @@ RETURNS Geometry AS $$
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
@@ -126,22 +150,22 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode street point using google maps')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.tools import ServiceManager, QuotaExceededException
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.mapzen.types import country_to_iso3
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapzen_geocoder_config import MapzenGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapzenGeocoderConfigBuilder, username, orgname)
service_manager.assert_within_limits()
try:
service_manager.assert_within_limits()
geocoder = MapzenGeocoder(service_manager.config.mapzen_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3 = None
if country:
@@ -157,6 +181,9 @@ RETURNS Geometry AS $$
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
@@ -164,4 +191,50 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode street point using mapzen')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_geocode_street_point(username TEXT, orgname TEXT, searchtext TEXT, city TEXT DEFAULT NULL, state_province TEXT DEFAULT NULL, country TEXT DEFAULT NULL)
RETURNS Geometry AS $$
from iso3166 import countries
from cartodb_services.tools import ServiceManager, QuotaExceededException
from cartodb_services.mapbox import MapboxGeocoder
from cartodb_services.tools.country import country_to_iso3
from cartodb_services.refactor.service.mapbox_geocoder_config import MapboxGeocoderConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('geocoder', MapboxGeocoderConfigBuilder, username, orgname, GD)
try:
service_manager.assert_within_limits()
geocoder = MapboxGeocoder(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
country_iso3166 = None
if country:
country_iso3 = country_to_iso3(country)
if country_iso3:
country_iso3166 = countries.get(country_iso3).alpha2.lower()
coordinates = geocoder.geocode(searchtext=searchtext, city=city,
state_province=state_province,
country=country_iso3166)
if coordinates:
service_manager.quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
service_manager.quota_service.increment_empty_service_use()
return None
except QuotaExceededException as qe:
service_manager.quota_service.increment_failed_service_use()
return None
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to geocode street point using mapbox', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode street point using mapbox')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -16,7 +16,7 @@ RETURNS JSON AS $$
return json.dumps({'limit': rate_limit_config.limit, 'period': rate_limit_config.period})
else:
return None
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_user_rate_limit(
username TEXT,
@@ -40,7 +40,7 @@ RETURNS VOID AS $$
period = None
config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
config_setter.set_user_rate_limits(config)
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu VOLATILE PARALLEL UNSAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_org_rate_limit(
username TEXT,
@@ -64,7 +64,7 @@ RETURNS VOID AS $$
period = None
config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
config_setter.set_org_rate_limits(config)
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_service_set_server_rate_limit(
username TEXT,
@@ -88,4 +88,4 @@ RETURNS VOID AS $$
period = None
config = RateLimitsConfig(service=service, username=username, limit=limit, period=period)
config_setter.set_server_rate_limits(config)
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu VOLATILE PARALLEL UNSAFE;

View File

@@ -32,7 +32,7 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode admin0 polygon')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
--------------------------------------------------------------------------------
@@ -52,4 +52,4 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;

View File

@@ -34,7 +34,7 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode admin1 polygon')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_admin1_polygon(admin1_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_admin1_polygon(username text, orgname text, admin1_name text, country_name text)
@@ -72,7 +72,7 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode admin1 polygon')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
--------------------------------------------------------------------------------
@@ -101,7 +101,7 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;
---- cdb_geocode_admin1_polygon(admin1_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_geocode_admin1_polygon(admin1_name text, country_name text)
@@ -125,5 +125,5 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;

View File

@@ -2,82 +2,61 @@
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text)
RETURNS Geometry AS $$
import spiexceptions
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
try:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, city_name])[0]['point']
street_point = plpy.prepare("SELECT cdb_dataservices_server.cdb_geocode_street_point($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(street_point, [username, orgname, city_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
import sys
logger.error('Error geocoding namedplace using geocode street point, falling back to internal geocoder', sys.exc_info(), data={"username": username, "orgname": orgname})
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3) as point;", ["text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name])[0]['point']
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, country_name text)
RETURNS Geometry AS $$
import spiexceptions
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
try:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, city_name, country_name])[0]['point']
street_point = plpy.prepare("SELECT cdb_dataservices_server.cdb_geocode_street_point($1, $2, $3, NULL, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(street_point, [username, orgname, city_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
import sys
logger.error('Error geocoding namedplace using geocode street point, falling back to internal geocoder', sys.exc_info(), data={"username": username, "orgname": orgname})
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, NULL, $4) as point;", ["text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, country_name])[0]['point']
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_namedplace_point(username text, orgname text, city_name text, admin1_name text, country_name text)
RETURNS Geometry AS $$
import spiexceptions
try:
mapzen_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_mapzen_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(mapzen_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
$$ LANGUAGE plpythonu;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_geocode_namedplace(username text, orgname text, city_name text, admin1_name text DEFAULT NULL, country_name text DEFAULT NULL)
RETURNS Geometry AS $$
from cartodb_services.mapzen import MapzenGeocoder
from cartodb_services.mapzen.types import country_to_iso3
from cartodb_services.metrics import QuotaService, metrics
from cartodb_services.tools import Logger,LoggerConfig
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_geocoder_config({0}, {1}, {2})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname), plpy.quote_nullable('mapzen')))
user_geocoder_config = GD["user_geocoder_config_{0}".format(username)]
plpy.execute("SELECT cdb_dataservices_server._get_logger_config()")
logger_config = GD["logger_config"]
logger = Logger(logger_config)
quota_service = QuotaService(user_geocoder_config, redis_conn)
if not quota_service.check_user_quota():
raise Exception('You have reached the limit of your quota')
with metrics('cdb_geocode_namedplace_point', user_geocoder_config, logger):
try:
geocoder = MapzenGeocoder(user_geocoder_config.mapzen_api_key, logger)
country_iso3 = None
if country_name:
country_iso3 = country_to_iso3(country_name)
coordinates = geocoder.geocode(searchtext=city_name, city=None,
state_province=admin1_name,
country=country_iso3, search_type='locality')
if coordinates:
quota_service.increment_success_service_use()
plan = plpy.prepare("SELECT ST_SetSRID(ST_MakePoint($1, $2), 4326); ", ["double precision", "double precision"])
point = plpy.execute(plan, [coordinates[0], coordinates[1]], 1)[0]
return point['st_setsrid']
else:
quota_service.increment_empty_service_use()
return None
except BaseException as e:
import sys
quota_service.increment_failed_service_use()
logger.error('Error trying to geocode city point using mapzen', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to geocode city point using mapzen')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
try:
street_point = plpy.prepare("SELECT cdb_dataservices_server.cdb_geocode_street_point($1, $2, $3, NULL, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(street_point, [username, orgname, city_name, admin1_name, country_name])[0]['point']
except spiexceptions.ExternalRoutineException as e:
import sys
logger.error('Error geocoding namedplace using geocode street point, falling back to internal geocoder', sys.exc_info(), data={"username": username, "orgname": orgname})
internal_plan = plpy.prepare("SELECT cdb_dataservices_server._cdb_internal_geocode_namedplace($1, $2, $3, $4, $5) as point;", ["text", "text", "text", "text", "text"])
return plpy.execute(internal_plan, [username, orgname, city_name, admin1_name, country_name])[0]['point']
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_internal_geocode_namedplace(username text, orgname text, city_name text, admin1_name text DEFAULT NULL, country_name text DEFAULT NULL)
RETURNS Geometry AS $$
@@ -120,7 +99,7 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode namedplace point')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
--------------------------------------------------------------------------------
@@ -144,7 +123,7 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;
---- cdb_geocode_namedplace_point(city_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_geocode_namedplace_point(city_name text, country_name text)
@@ -164,7 +143,7 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;
---- cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_geocode_namedplace_point(city_name text, admin1_name text, country_name text)
@@ -188,5 +167,5 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;

View File

@@ -32,12 +32,12 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode postal code point')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision)
RETURNS Geometry AS $$
SELECT cdb_dataservices_server.cdb_geocode_postalcode_point(username, orgname, code::text);
$$ LANGUAGE SQL;
$$ LANGUAGE SQL STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code text, country text)
@@ -74,12 +74,12 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode postal code point')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_point(username text, orgname text, code double precision, country text)
RETURNS Geometry AS $$
SELECT cdb_dataservices_server.cdb_geocode_postalcode_point(username, orgname, code::text, country);
$$ LANGUAGE SQL;
$$ LANGUAGE SQL STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code text)
@@ -116,12 +116,12 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode postal code polygon')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision)
RETURNS Geometry AS $$
SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon(username, orgname, code::text)
$$ LANGUAGE SQL;
$$ LANGUAGE SQL STABLE PARALLEL RESTRICTED;
@@ -159,12 +159,12 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode postal code polygon')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_geocode_postalcode_polygon(username text, orgname text, code double precision, country text)
RETURNS Geometry AS $$
SELECT cdb_dataservices_server.cdb_geocode_postalcode_polygon(username, orgname, code::text, country);
$$ LANGUAGE SQL;
$$ LANGUAGE SQL STABLE PARALLEL RESTRICTED;
--------------------------------------------------------------------------------
@@ -189,7 +189,7 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_geocode_postalcode_point(code text, country text)
RETURNS Geometry AS $$
@@ -214,7 +214,7 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_geocode_postalcode_polygon(code text)
RETURNS Geometry AS $$
@@ -235,7 +235,7 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_geocode_postalcode_polygon(code text, country text)
RETURNS Geometry AS $$
@@ -260,4 +260,4 @@ RETURNS Geometry AS $$
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;

View File

@@ -32,7 +32,7 @@ RETURNS Geometry AS $$
raise Exception('Error trying to geocode postal code polygon')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
--------------------------------------------------------------------------------
@@ -63,4 +63,4 @@ RETURNS Geometry AS $$
FROM matches;
RETURN ret;
END
$$ LANGUAGE plpgsql;
$$ LANGUAGE plpgsql STABLE PARALLEL SAFE;

View File

@@ -55,7 +55,7 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
raise Exception('Error trying to get mapzen isolines')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isodistance(
username TEXT,
@@ -121,8 +121,72 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
raise Exception('Error trying to get mapzen isolines')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_isodistance(
username TEXT,
orgname TEXT,
source geometry(Geometry, 4326),
mode TEXT,
data_range integer[],
options text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxMatrixClient, MapboxIsolines
from cartodb_services.mapbox.types import TRANSPORT_MODE_TO_MAPBOX
from cartodb_services.tools import Coordinate
from cartodb_services.refactor.service.mapbox_isolines_config import MapboxIsolinesConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('isolines', MapboxIsolinesConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
client = MapboxMatrixClient(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
mapbox_isolines = MapboxIsolines(client, service_manager.logger)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
origin = Coordinate(lon,lat)
else:
raise Exception('source is NULL')
profile = TRANSPORT_MODE_TO_MAPBOX.get(mode)
# -- TODO Support options properly
isolines = {}
for r in data_range:
isoline = mapbox_isolines.calculate_isodistance(origin, r, profile)
isolines[r] = isoline
result = []
for r in data_range:
if len(isolines[r]) >= 3:
# -- TODO encapsulate this block into a func/method
locations = isolines[r] + [ isolines[r][0] ] # close the polygon repeating the first point
wkt_coordinates = ','.join(["%f %f" % (l.longitude, l.latitude) for l in locations])
sql = "SELECT ST_MPolyFromText('MULTIPOLYGON((({0})))', 4326) as geom".format(wkt_coordinates)
multipolygon = plpy.execute(sql, 1)[0]['geom']
else:
multipolygon = None
result.append([source, r, multipolygon])
service_manager.quota_service.increment_success_service_use()
service_manager.quota_service.increment_isolines_service_use(len(isolines))
return result
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to get Mapbox isolines', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to get Mapbox isolines')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapzen_isochrones(
username TEXT,
@@ -185,4 +249,65 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
raise Exception('Error trying to get mapzen isochrones')
finally:
quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER;
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;
CREATE OR REPLACE FUNCTION cdb_dataservices_server._cdb_mapbox_isochrones(
username TEXT,
orgname TEXT,
source geometry(Geometry, 4326),
mode TEXT,
data_range integer[],
options text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
from cartodb_services.tools import ServiceManager
from cartodb_services.mapbox import MapboxMatrixClient, MapboxIsolines
from cartodb_services.mapbox.types import TRANSPORT_MODE_TO_MAPBOX
from cartodb_services.tools import Coordinate
from cartodb_services.tools.coordinates import coordinates_to_polygon
from cartodb_services.refactor.service.mapbox_isolines_config import MapboxIsolinesConfigBuilder
import cartodb_services
cartodb_services.init(plpy, GD)
service_manager = ServiceManager('isolines', MapboxIsolinesConfigBuilder, username, orgname, GD)
service_manager.assert_within_limits()
try:
client = MapboxMatrixClient(service_manager.config.mapbox_api_key, service_manager.logger, service_manager.config.service_params)
mapbox_isolines = MapboxIsolines(client, service_manager.logger)
if source:
lat = plpy.execute("SELECT ST_Y('%s') AS lat" % source)[0]['lat']
lon = plpy.execute("SELECT ST_X('%s') AS lon" % source)[0]['lon']
origin = Coordinate(lon,lat)
else:
raise Exception('source is NULL')
profile = TRANSPORT_MODE_TO_MAPBOX.get(mode)
resp = mapbox_isolines.calculate_isochrone(origin, data_range, profile)
if resp:
result = []
for isochrone in resp:
result_polygon = coordinates_to_polygon(isochrone.coordinates)
if result_polygon:
service_manager.quota_service.increment_success_service_use()
result.append([source, isochrone.duration, result_polygon])
else:
service_manager.quota_service.increment_empty_service_use()
result.append([source, isochrone.duration, None])
service_manager.quota_service.increment_success_service_use()
service_manager.quota_service.increment_isolines_service_use(len(result))
return result
else:
service_manager.quota_service.increment_empty_service_use()
return []
except BaseException as e:
import sys
service_manager.quota_service.increment_failed_service_use()
service_manager.logger.error('Error trying to get Mapbox isochrones', sys.exc_info(), data={"username": username, "orgname": orgname})
raise Exception('Error trying to get Mapbox isochrones')
finally:
service_manager.quota_service.increment_total_service_use()
$$ LANGUAGE plpythonu SECURITY DEFINER STABLE PARALLEL RESTRICTED;

View File

@@ -21,9 +21,12 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
elif user_isolines_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapbox_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
else:
raise Exception('Requested isolines provider is not available')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- heremaps isodistance
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_here_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
@@ -38,7 +41,7 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
result = plpy.execute(here_plan, [username, orgname, type, source, mode, range, options])
return result
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- mapzen isodistance
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
@@ -52,4 +55,18 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
result = plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
return result
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- mapbox isodistance
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapbox_isodistance(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_isodistance($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
result = plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
return result
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -21,9 +21,12 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
elif user_isolines_config.mapzen_provider:
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapzen_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
elif user_isolines_config.mapbox_provider:
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server.cdb_mapbox_isochrone($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
return plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
else:
raise Exception('Requested isolines provider is not available')
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- heremaps isochrone
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_here_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
@@ -38,7 +41,7 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
result = plpy.execute(here_plan, [username, orgname, type, source, mode, range, options])
return result
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- mapzen isochrone
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapzen_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
@@ -51,4 +54,17 @@ RETURNS SETOF cdb_dataservices_server.isoline AS $$
mapzen_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapzen_isochrones($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
result = plpy.execute(mapzen_plan, [username, orgname, source, mode, range, options])
return result
$$ LANGUAGE plpythonu;
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;
-- mapbox isochrone
CREATE OR REPLACE FUNCTION cdb_dataservices_server.cdb_mapbox_isochrone(username TEXT, orgname TEXT, source geometry(Geometry, 4326), mode TEXT, range integer[], options text[] DEFAULT array[]::text[])
RETURNS SETOF cdb_dataservices_server.isoline AS $$
plpy.execute("SELECT cdb_dataservices_server._connect_to_redis('{0}')".format(username))
redis_conn = GD["redis_connection_{0}".format(username)]['redis_metrics_connection']
plpy.execute("SELECT cdb_dataservices_server._get_isolines_routing_config({0}, {1})".format(plpy.quote_nullable(username), plpy.quote_nullable(orgname)))
user_isolines_config = GD["user_isolines_routing_config_{0}".format(username)]
mapbox_plan = plpy.prepare("SELECT * FROM cdb_dataservices_server._cdb_mapbox_isochrones($1, $2, $3, $4, $5, $6) as isoline; ", ["text", "text", "geometry(geometry, 4326)", "text", "integer[]", "text[]"])
result = plpy.execute(mapbox_plan, [username, orgname, source, mode, range, options])
return result
$$ LANGUAGE plpythonu STABLE PARALLEL RESTRICTED;

View File

@@ -31,6 +31,12 @@ SELECT cartodb.cdb_conf_setconf('mapzen_conf', '{"routing": {"api_key": "routing
(1 row)
SELECT cartodb.cdb_conf_setconf('mapbox_conf', '{"routing": {"api_keys": ["routing_dummy_api_key"], "monthly_quota": 1500000}, "geocoder": {"api_keys": ["geocoder_dummy_api_key"], "monthly_quota": 1500000}, "matrix": {"api_keys": ["matrix_dummy_api_key"], "monthly_quota": 1500000}}');
cdb_conf_setconf
------------------
(1 row)
SELECT cartodb.cdb_conf_setconf('logger_conf', '{"geocoder_log_path": "/dev/null"}');
cdb_conf_setconf
------------------

View File

@@ -245,7 +245,7 @@ SELECT exists(SELECT *
INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
WHERE ns.nspname = 'cdb_dataservices_server'
AND proname = 'obs_getavailablegeometries'
AND oidvectortypes(p.proargtypes) = 'text, text, geometry, text[], text, text, text');
AND oidvectortypes(p.proargtypes) = 'text, text, geometry, text[], text, text, text, integer');
exists
--------
t

View File

@@ -1,18 +1,21 @@
-- Check that the public function is callable, even with no data
-- It should return NULL
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elx');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
------------------------------
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elx', 'Spain');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
------------------------------
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elx', 'Valencia', 'Spain');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
------------------------------
@@ -32,36 +35,42 @@ INSERT INTO country_decoder (synonyms, iso2) VALUES (Array['spain', 'Spain'], 'E
INSERT INTO admin1_decoder (admin1, synonyms, iso2) VALUES ('Valencia', Array['valencia', 'Valencia'], 'ES');
-- This should return the point inserted above
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elx');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
----------------------------------------------------
0101000020E6100000637FD93D7958E63F2ECA6C9049A24340
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elche');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
----------------------------------------------------
0101000020E6100000637FD93D7958E63F2ECA6C9049A24340
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elx', 'Spain');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
----------------------------------------------------
0101000020E6100000637FD93D7958E63F2ECA6C9049A24340
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elche', 'Spain');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
----------------------------------------------------
0101000020E6100000637FD93D7958E63F2ECA6C9049A24340
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elx', 'Valencia', 'Spain');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
----------------------------------------------------
0101000020E6100000637FD93D7958E63F2ECA6C9049A24340
(1 row)
SELECT cdb_dataservices_server.cdb_geocode_namedplace_point('test_user', 'test_orgname', 'Elche', 'valencia', 'Spain');
WARNING: Error geocoding namedplace using geocode street point, falling back to internal geocoder
cdb_geocode_namedplace_point
----------------------------------------------------
0101000020E6100000637FD93D7958E63F2ECA6C9049A24340

View File

@@ -13,6 +13,7 @@ SELECT cartodb.cdb_conf_setconf('redis_metrics_config', '{"redis_host": "localho
SELECT cartodb.cdb_conf_setconf('redis_metadata_config', '{"redis_host": "localhost", "redis_port": 6379, "timeout": 0.1, "redis_db": 5}');
SELECT cartodb.cdb_conf_setconf('heremaps_conf', '{"geocoder": {"app_id": "dummy_id", "app_code": "dummy_code", "geocoder_cost_per_hit": 1}, "isolines": {"app_id": "dummy_id", "app_code": "dummy_code"}}');
SELECT cartodb.cdb_conf_setconf('mapzen_conf', '{"routing": {"api_key": "routing_dummy_api_key", "monthly_quota": 1500000}, "geocoder": {"api_key": "geocoder_dummy_api_key", "monthly_quota": 1500000}, "matrix": {"api_key": "matrix_dummy_api_key", "monthly_quota": 1500000}}');
SELECT cartodb.cdb_conf_setconf('mapbox_conf', '{"routing": {"api_keys": ["routing_dummy_api_key"], "monthly_quota": 1500000}, "geocoder": {"api_keys": ["geocoder_dummy_api_key"], "monthly_quota": 1500000}, "matrix": {"api_keys": ["matrix_dummy_api_key"], "monthly_quota": 1500000}}');
SELECT cartodb.cdb_conf_setconf('logger_conf', '{"geocoder_log_path": "/dev/null"}');
SELECT cartodb.cdb_conf_setconf('data_observatory_conf', '{"connection": {"whitelist": ["ethervoid"], "production": "host=localhost port=5432 dbname=contrib_regression user=geocoder_api", "staging": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api"}, "monthly_quota": 100000}');

View File

@@ -157,7 +157,7 @@ SELECT exists(SELECT *
INNER JOIN pg_namespace ns ON (p.pronamespace = ns.oid)
WHERE ns.nspname = 'cdb_dataservices_server'
AND proname = 'obs_getavailablegeometries'
AND oidvectortypes(p.proargtypes) = 'text, text, geometry, text[], text, text, text');
AND oidvectortypes(p.proargtypes) = 'text, text, geometry, text[], text, text, text, integer');
SELECT exists(SELECT *
FROM pg_proc p

View File

@@ -0,0 +1,39 @@
#!/usr/local/bin/python
# -*- coding: utf-8 -*-
import googlemaps
import base64
from exceptions import InvalidGoogleCredentials
class GoogleMapsClientFactory():
clients = {}
@classmethod
def get(cls, client_id, client_secret, channel=None):
cache_key = "{}:{}:{}".format(client_id, client_secret, channel)
client = cls.clients.get(cache_key)
if not client:
cls.assert_valid_crendentials(client_secret)
client = googlemaps.Client(
client_id=client_id,
client_secret=client_secret,
channel=channel)
cls.clients[cache_key] = client
return client
@classmethod
def assert_valid_crendentials(cls, client_secret):
if not cls.valid_credentials(client_secret):
raise InvalidGoogleCredentials
@staticmethod
def valid_credentials(client_secret):
try:
# Only fails if the string dont have a correct padding for b64
# but this way we could provide a more clear error than
# TypeError: Incorrect padding
b64_secret = client_secret.replace('-', '+').replace('_', '/')
base64.b64decode(b64_secret)
return True
except TypeError:
return False

View File

@@ -1,22 +1,23 @@
#!/usr/local/bin/python
# -*- coding: utf-8 -*-
import base64
import googlemaps
from urlparse import parse_qs
from exceptions import MalformedResult, InvalidGoogleCredentials
from exceptions import MalformedResult
from cartodb_services.google.exceptions import InvalidGoogleCredentials
from client_factory import GoogleMapsClientFactory
class GoogleMapsGeocoder:
"""A Google Maps Geocoder wrapper for python"""
def __init__(self, client_id, client_secret, logger):
if not self._valid_credentials(client_secret):
raise InvalidGoogleCredentials('Invalid google secret key')
self.client_id = self._clean_client_id(client_id)
if client_id is None:
raise InvalidGoogleCredentials
self.client_id, self.channel = self.parse_client_id(client_id)
self.client_secret = client_secret
self.geocoder = googlemaps.Client(
client_id=self.client_id, client_secret=self.client_secret)
self.geocoder = GoogleMapsClientFactory.get(self.client_id, self.client_secret, self.channel)
self._logger = logger
def geocode(self, searchtext, city=None, state=None,
@@ -49,16 +50,8 @@ class GoogleMapsGeocoder:
optional_params['country'] = country
return optional_params
def _clean_client_id(self, client_id):
# Consistency with how the client_id is saved in metadata
return client_id.replace('client=', '')
def _valid_credentials(self, private_key):
try:
# Only fails if the string dont have a correct padding for b64
# but this way we could provide a more clear error than
# TypeError: Incorrect padding
base64.b64decode(private_key)
return True
except TypeError:
return False
def parse_client_id(self, client_id):
arguments = parse_qs(client_id)
client = arguments['client'][0] if arguments.has_key('client') else client_id
channel = arguments['channel'][0] if arguments.has_key('channel') else None
return client, channel

View File

@@ -67,10 +67,10 @@ class HereMapsGeocoder(Traceable):
def geocode(self, **kwargs):
params = {}
for key, value in kwargs.iteritems():
if value:
if value and value.strip():
params[key] = value
if not params:
raise NoGeocodingParams()
return []
return self._execute_geocode(params)
def _execute_geocode(self, params):

View File

@@ -0,0 +1,4 @@
from routing import MapboxRouting, MapboxRoutingResponse
from geocoder import MapboxGeocoder
from isolines import MapboxIsolines, MapboxIsochronesResponse
from matrix_client import MapboxMatrixClient

View File

@@ -0,0 +1,98 @@
'''
Python client for the Mapbox Geocoder service.
'''
import json
import requests
from mapbox import Geocoder
from cartodb_services.metrics import Traceable
from cartodb_services.tools.exceptions import ServiceException
from cartodb_services.tools.qps import qps_retry
GEOCODER_NAME = 'geocoder_name'
EPHEMERAL_GEOCODER = 'mapbox.places'
PERMANENT_GEOCODER = 'mapbox.places-permanent'
DEFAULT_GEOCODER = EPHEMERAL_GEOCODER
ENTRY_FEATURES = 'features'
ENTRY_CENTER = 'center'
ENTRY_GEOMETRY = 'geometry'
ENTRY_COORDINATES = 'coordinates'
ENTRY_TYPE = 'type'
TYPE_POINT = 'Point'
class MapboxGeocoder(Traceable):
'''
Python wrapper for the Mapbox Geocoder service.
'''
def __init__(self, token, logger, service_params=None):
service_params = service_params or {}
self._token = token
self._logger = logger
self._geocoder_name = service_params.get(GEOCODER_NAME,
EPHEMERAL_GEOCODER)
self._geocoder = Geocoder(access_token=self._token,
name=self._geocoder_name)
def _parse_geocoder_response(self, response):
json_response = json.loads(response)
if json_response and json_response[ENTRY_FEATURES]:
feature = json_response[ENTRY_FEATURES][0]
return self._extract_lng_lat_from_feature(feature)
else:
return []
def _extract_lng_lat_from_feature(self, feature):
geometry = feature[ENTRY_GEOMETRY]
if geometry[ENTRY_TYPE] == TYPE_POINT:
location = geometry[ENTRY_COORDINATES]
else:
location = feature[ENTRY_CENTER]
longitude = location[0]
latitude = location[1]
return [longitude, latitude]
@qps_retry(qps=10)
def geocode(self, searchtext, city=None, state_province=None,
country=None):
if searchtext and searchtext.strip():
address = [searchtext]
if city:
address.append(city)
if state_province:
address.append(state_province)
else:
return []
country = [country] if country else None
try:
response = self._geocoder.forward(address=', '.join(address).decode('utf-8'),
country=country,
limit=1)
if response.status_code == requests.codes.ok:
return self._parse_geocoder_response(response.text)
elif response.status_code == requests.codes.bad_request:
return []
elif response.status_code == requests.codes.unprocessable_entity:
return []
else:
raise ServiceException(response.status_code, response)
except requests.Timeout as te:
# In case of timeout we want to stop the job because the server
# could be down
self._logger.error('Timeout connecting to Mapbox geocoding server',
te)
raise ServiceException('Error geocoding {0} using Mapbox'.format(
searchtext), None)
except requests.ConnectionError as ce:
# Don't raise the exception to continue with the geocoding job
self._logger.error('Error connecting to Mapbox geocoding server',
exception=ce)
return []

View File

@@ -0,0 +1,173 @@
'''
Python implementation for Mapbox services based isolines.
Uses the Mapbox Time Matrix service.
'''
import json
from cartodb_services.tools.spherical import (get_angles,
calculate_dest_location)
from cartodb_services.mapbox.matrix_client import (validate_profile,
DEFAULT_PROFILE,
PROFILE_WALKING,
PROFILE_DRIVING,
PROFILE_CYCLING,
ENTRY_DURATIONS)
MAX_SPEEDS = {
PROFILE_WALKING: 3.3333333, # In m/s, assuming 12km/h walking speed
PROFILE_CYCLING: 16.67, # In m/s, assuming 60km/h max speed
PROFILE_DRIVING: 41.67 # In m/s, assuming 140km/h max speed
}
DEFAULT_NUM_ANGLES = 24
DEFAULT_MAX_ITERS = 5
DEFAULT_TOLERANCE = 0.1
MATRIX_NUM_ANGLES = DEFAULT_NUM_ANGLES
MATRIX_MAX_ITERS = DEFAULT_MAX_ITERS
MATRIX_TOLERANCE = DEFAULT_TOLERANCE
UNIT_FACTOR_ISOCHRONE = 1.0
UNIT_FACTOR_ISODISTANCE = 1000.0
DEFAULT_UNIT_FACTOR = UNIT_FACTOR_ISOCHRONE
class MapboxIsolines():
'''
Python wrapper for Mapbox services based isolines.
'''
def __init__(self, matrix_client, logger, service_params=None):
service_params = service_params or {}
self._matrix_client = matrix_client
self._logger = logger
def _calculate_matrix_cost(self, origin, targets, isorange,
profile=DEFAULT_PROFILE,
unit_factor=UNIT_FACTOR_ISOCHRONE,
number_of_angles=MATRIX_NUM_ANGLES):
response = self._matrix_client.matrix([origin] + targets,
profile)
json_response = json.loads(response)
if not json_response:
return []
costs = [None] * number_of_angles
for idx, cost in enumerate(json_response[ENTRY_DURATIONS][0][1:]):
if cost:
costs[idx] = cost * unit_factor
else:
costs[idx] = isorange
return costs
def calculate_isochrone(self, origin, time_ranges,
profile=DEFAULT_PROFILE):
validate_profile(profile)
max_speed = MAX_SPEEDS[profile]
isochrones = []
for time_range in time_ranges:
upper_rmax = max_speed * time_range # an upper bound for the radius
coordinates = self.calculate_isoline(origin=origin,
isorange=time_range,
upper_rmax=upper_rmax,
cost_method=self._calculate_matrix_cost,
profile=profile,
unit_factor=UNIT_FACTOR_ISOCHRONE,
number_of_angles=MATRIX_NUM_ANGLES,
max_iterations=MATRIX_MAX_ITERS,
tolerance=MATRIX_TOLERANCE)
isochrones.append(MapboxIsochronesResponse(coordinates,
time_range))
return isochrones
def calculate_isodistance(self, origin, distance_range,
profile=DEFAULT_PROFILE):
validate_profile(profile)
max_speed = MAX_SPEEDS[profile]
time_range = distance_range / max_speed
return self.calculate_isochrone(origin=origin,
time_ranges=[time_range],
profile=profile)[0].coordinates
def calculate_isoline(self, origin, isorange, upper_rmax,
cost_method=_calculate_matrix_cost,
profile=DEFAULT_PROFILE,
unit_factor=DEFAULT_UNIT_FACTOR,
number_of_angles=DEFAULT_NUM_ANGLES,
max_iterations=DEFAULT_MAX_ITERS,
tolerance=DEFAULT_TOLERANCE):
# Formally, a solution is an array of {angle, radius, lat, lon, cost}
# with cardinality number_of_angles
# we're looking for a solution in which
# abs(cost - isorange) / isorange <= TOLERANCE
# Initial setup
angles = get_angles(number_of_angles)
rmax = [upper_rmax] * number_of_angles
rmin = [0.0] * number_of_angles
location_estimates = [calculate_dest_location(origin, a,
upper_rmax / 2.0)
for a in angles]
# Iterate to refine the first solution
for i in xrange(0, max_iterations):
# Calculate the "actual" cost for each location estimate.
# NOTE: sometimes it cannot calculate the cost and returns None.
# Just assume isorange and stop the calculations there
costs = cost_method(origin=origin, targets=location_estimates,
isorange=isorange, profile=profile,
unit_factor=unit_factor,
number_of_angles=number_of_angles)
if not costs:
continue
errors = [(cost - isorange) / float(isorange) for cost in costs]
max_abs_error = max([abs(e) for e in errors])
if max_abs_error <= tolerance:
# good enough, stop there
break
# let's refine the solution, binary search
for j in xrange(0, number_of_angles):
if abs(errors[j]) > tolerance:
if errors[j] > 0:
rmax[j] = (rmax[j] + rmin[j]) / 2.0
else:
rmin[j] = (rmax[j] + rmin[j]) / 2.0
location_estimates[j] = calculate_dest_location(origin,
angles[j],
(rmax[j] + rmin[j]) / 2.0)
# delete points that got None
location_estimates_filtered = []
for i, c in enumerate(costs):
if c != isorange:
location_estimates_filtered.append(location_estimates[i])
return location_estimates_filtered
class MapboxIsochronesResponse:
def __init__(self, coordinates, duration):
self._coordinates = coordinates
self._duration = duration
@property
def coordinates(self):
return self._coordinates
@property
def duration(self):
return self._duration

Some files were not shown because too many files have changed in this diff Show More