Compare commits

...

540 Commits

Author SHA1 Message Date
Sandro Santilli
1bf7bf66b3 Release 1.10.1 2014-03-21 15:16:19 +01:00
Sandro Santilli
9e495b42ee Do not cache non-success jsonp responses
Closes #186
Includes testcase
2014-03-21 13:58:20 +01:00
Sandro Santilli
898f717254 Prepare for 1.10.1 2014-03-20 17:10:39 +01:00
Sandro Santilli
800ef32959 Release 1.10.0 2014-03-20 17:08:35 +01:00
Sandro Santilli
609d69c4c9 Upgrade of windshaft fixed connection details to client
Closes #183.
2014-03-20 10:21:28 +01:00
Sandro Santilli
9e1be39774 Switch to 3-clause BSD license
Closes #184
2014-03-20 10:20:51 +01:00
Sandro Santilli
87ac44a1f1 Upgrade windshaft to 0.20.0
Reduces noise on the "error" channel (now optionally writing to rollbar)
and avoids caching bogus mapnik renderers.
Details: http://github.com/CartoDB/Windshaft/blob/0.20.0/NEWS
2014-03-20 10:18:33 +01:00
Sandro Santilli
9c4feac19b Ensure make check fails if database preparation fails 2014-03-19 17:04:06 +01:00
Sandro Santilli
471edabe4d Reword uncaught exception error, and log full stack 2014-03-13 11:58:29 +01:00
Sandro Santilli
86841f80ca Use version of node-mapnik with temptative fix for glibc detected corruptions 2014-03-13 10:26:11 +01:00
Sandro Santilli
79348178a7 Upgrade node-varnish to 0.3.0 2014-03-12 18:11:19 +01:00
Sandro Santilli
60b552027b Add optional support for rollbar
Re-targets to 1.10.0
Also installs an uncaught exception handler

Closes #150
2014-03-12 17:21:35 +01:00
Sandro Santilli
62cbb15089 Include tiler version in startup log 2014-03-11 12:21:00 +01:00
Sandro Santilli
667b911023 Prepare for 1.9.1 2014-03-10 17:41:44 +01:00
Sandro Santilli
071e86799b Release 1.9.0 2014-03-10 17:40:55 +01:00
Sandro Santilli
4164cf7adb Set release date for 1.8.5 2014-03-10 17:37:19 +01:00
Sandro Santilli
b61aee36e7 More format changes 2014-03-06 16:29:26 +01:00
Sandro Santilli
7b16676f63 Retarget to 1.9.0 2014-03-06 16:28:13 +01:00
javi
ff4f46abcc Merge branch 'server_metadata' 2014-03-06 16:27:01 +01:00
javi santana
09c1bd96df fix formating 2014-03-06 16:22:25 +01:00
javi santana
40a190c29c added cdn_url option 2014-03-06 16:22:04 +01:00
javi
5bfc360856 added serverMetadata option for layer group, close #182 CDB-1940 2014-03-06 15:19:12 +01:00
Sandro Santilli
7eb26a7326 Upgrade windshaft to 0.19.3, fixing crash on dns error
Closes #180
2014-03-05 18:16:42 +01:00
Sandro Santilli
0afc9c154b Cleanly catch exceptions from sendResponse
Closes #178
2014-03-04 18:04:58 +01:00
Sandro Santilli
97e00fb47d Do not send duplicated stats on template instanciation
Closes #179
2014-03-04 17:51:50 +01:00
Sandro Santilli
dbae0eeb31 It is "cacheDns", not "dnsCache"
See https://github.com/sivy/node-statsd/issues/38
2014-03-04 17:37:19 +01:00
Sandro Santilli
bd9a21b805 Add "dnsCache" statsd setting in the example configs 2014-03-04 16:52:16 +01:00
Sandro Santilli
033f8df500 Include API docs, moved from wiki
Closes #164
2014-03-04 15:39:21 +01:00
Sandro Santilli
ffda103d61 Do not UNWATCH on every redis client release
Closes #161
2014-03-04 15:36:08 +01:00
Sandro Santilli
ecc9ea1226 Use 403 for forbidden, not 401
Includes upgrade of windshaft to 0.19.3
Includes upgrade of redis-mpool to 0.0.4
2014-03-04 15:32:31 +01:00
Sandro Santilli
93345a19b2 Do not log an error on GET /
Closes #177
2014-03-04 14:26:41 +01:00
Sandro Santilli
1741a20575 Do not cache map creation responses
Closes #176
CDB-1908 #resolve
CDB-1901 #resolve

Includes testcase
2014-03-04 10:46:15 +01:00
Sandro Santilli
30eb939dc7 Fix error message on missing requested signature
We don't really distinguish between missing or non-authorizing
signature. And that's fine. See #170
2014-03-03 18:14:17 +01:00
Sandro Santilli
40a254922a Raise 403 forbidden on missing requested signature
Closes #170
Includes testcase
2014-03-03 18:06:39 +01:00
Sandro Santilli
7bc5bab432 Properly prefix statsd labels for all endpoints
CDB-1861 #resolve
Will be 100% complete with update of Windshaft to 0.19.3+
2014-03-03 16:24:20 +01:00
Sandro Santilli
6034f49f40 Prepare for 1.8.5 2014-03-03 11:45:23 +01:00
Sandro Santilli
087eff4734 Release 1.8.4 2014-03-03 11:26:16 +01:00
Sandro Santilli
ed5b045a15 Allow using NODE_ENV env variable to determine app configuration
Default to "development" environment.
Forward NODE_ENV variable to childrens (for example, to hush
millstone).
2014-02-28 16:22:24 +01:00
Sandro Santilli
c1a3cbc28c Hush millstone during testsuite 2014-02-28 16:14:44 +01:00
Sandro Santilli
bddc65a504 Forbid instanciating templates of foreign users
Closes #173
Includes testcase
2014-02-28 16:05:46 +01:00
Sandro Santilli
ddd2628c19 Fix database connection settings on template instanciation
Closes #174
Enhances testsuite to ensure test.js settings are read
2014-02-28 15:56:31 +01:00
Sandro Santilli
cf0c33a85d Oops, previous commit closed #172, not #173
Closes #172
Reopens #173
2014-02-28 13:25:28 +01:00
Sandro Santilli
f46dc90035 Forbid using map signatures of foreign users
Closes #173
Includes testcase
2014-02-28 13:24:38 +01:00
Sandro Santilli
73276b1003 Upgrade windshaft to 0.19.2
Fixes obscure "ECONNREFUSED" error message (closes #171)
Change some http status responses to be more appropriate to the case
2014-02-28 10:54:18 +01:00
Sandro Santilli
16e67387c9 Tell npm to use known registrars
See http://blog.npmjs.org/post/78085451721/npms-self-signed-certificate-is-no-more
2014-02-28 10:50:45 +01:00
Sandro Santilli
ca1b31bd9c Add example MapConfig using a torque layer 2014-02-27 17:20:23 +01:00
Sandro Santilli
55f333c0b7 Call userByReq() only once in req2params 2014-02-27 16:40:59 +01:00
Sandro Santilli
f24e4f8a0a Really skip CDB_TableMetadata lookup for sql affected by no tables
Closes #169
2014-02-27 15:34:09 +01:00
Sandro Santilli
eec9933fb8 Accept a slightly different error message on timeout
Node 0.10 uses ESOCKETTIMEDOUT while 0.8 uses ETIMEDOUT
See http://travis-ci.org/CartoDB/Windshaft-cartodb/builds/19722727
2014-02-27 13:37:44 +01:00
Sandro Santilli
238e8f39f2 Fix ticket referenc ein NEWS entry of 1.8.3 2014-02-27 12:46:56 +01:00
Sandro Santilli
919bcb6888 Prepare for 1.8.4 2014-02-27 12:46:43 +01:00
Sandro Santilli
50ebb25205 Release 1.8.3 2014-02-27 12:45:02 +01:00
Sandro Santilli
625642ca33 Oops, previous commit closed #168, not #16
Closes #168
2014-02-27 12:43:15 +01:00
Sandro Santilli
36632c762e Do not query CDB_TableMetadata for queries affected by no tables
Closes #16
2014-02-27 12:32:34 +01:00
Sandro Santilli
f284362988 Reduce sql-api communication timeout, and allow overriding it
Introduces new sqlapi.timeout directive, defaults to 100 ms
Includes testcase.
Closes #167
2014-02-27 10:33:32 +01:00
Sandro Santilli
cf01f01bc9 Upgrades windshaft to 0.19.1 with many performance improvements
Among others:

- Improve speed of instanciating a map
- Give meaningful error on attempts to use map tokens with
  attribute service

Closes #156 -- CDB-1796 #resolve
Closes #147
Closes #159
Closes #165
2014-02-26 17:26:17 +01:00
Sandro Santilli
5d0c71d292 Prepare for 1.8.3 2014-02-25 11:10:56 +01:00
Sandro Santilli
b3d3269d3d Release 1.8.2 2014-02-25 10:52:55 +01:00
Sandro Santilli
a13c1f61af Do not log an error for a legit request requiring no X-Cache-Channel 2014-02-24 17:34:00 +01:00
Sandro Santilli
4064b8f254 Add test for lack of X-Cache-Channel in response to root request 2014-02-24 16:24:01 +01:00
Sandro Santilli
5c466c51a8 Revert order of hostname components for statsd.prefix 2014-02-21 17:25:10 +01:00
Sandro Santilli
36628ce78e Also enable the profiler in the example test config
This is again for #157 without closing it
2014-02-21 17:06:29 +01:00
Sandro Santilli
d2d7bba357 Add statsd prefix in test example config
Still doesn't add automated testing (#157) but makes manual
testing easier.
2014-02-21 16:57:02 +01:00
Sandro Santilli
8e68716d16 Give more info on failure 2014-02-21 16:56:50 +01:00
Sandro Santilli
6824c09916 Change example test user and database names
This is to avoid a clash with cartodb test databases
2014-02-20 18:03:43 +01:00
Sandro Santilli
09ea924eb2 Allow using GET with sql-api for queries shorter than configured len
Introduces new sqlapi.max_get_sql_length directive, defaults to 2048.
Closes #155
Includes testcases.
2014-02-20 10:17:48 +01:00
Sandro Santilli
c8a042abdd Expand "addCacheChannel" stats 2014-02-19 18:10:33 +01:00
Sandro Santilli
019540e622 Set example statsd prefix with :host placeholder 2014-02-19 16:16:39 +01:00
Sandro Santilli
9a5243ade3 Fix munin plugin after log format changes
Closes #154
2014-02-19 15:38:14 +01:00
Sandro Santilli
b4fc8ec4a5 Allow using ":host" as part of statsd.prefix
It'll be replaced with hostname.
Closes #153
2014-02-19 15:31:12 +01:00
Sandro Santilli
30a2d85e92 Prepare for 1.8.2 2014-02-19 15:26:43 +01:00
Sandro Santilli
98603594b1 Release 1.8.1 2014-02-19 12:24:43 +01:00
Sandro Santilli
7410d98d56 Require windsahft 0.19.0 final 2014-02-19 11:25:42 +01:00
Sandro Santilli
1f552a9e24 Do not duplicate date in logs (already injected by logger) 2014-02-19 11:16:37 +01:00
Sandro Santilli
6c6f3d02f6 Always generate X-Cache-Channel for token-based tile responses
Closes #152
2014-02-19 10:09:54 +01:00
Sandro Santilli
36a135f02b Refactor addCacheChannel using Step 2014-02-19 07:19:41 +01:00
Sandro Santilli
1c3734fde7 Make server_option a callable function, to reduce globals
Updates acceptance test for #152 to not mess wit internals
2014-02-19 06:45:29 +01:00
Sandro Santilli
3c09be64ce Add pending test for X-Cache-Channel on tiler restart (#152) 2014-02-18 18:33:00 +01:00
Sandro Santilli
719346a472 Use log4js logger
Closes #138.
The logger will be automatically used by Windshaft on upgrade
to 0.18.1, see https://github.com/CartoDB/Windshaft/issues/140
2014-02-18 15:12:08 +01:00
Sandro Santilli
69693acea0 Add statsd prefix in example configs 2014-02-18 10:38:15 +01:00
Sandro Santilli
3873fdf5db Prepare for 1.8.1 2014-02-18 10:38:01 +01:00
Sandro Santilli
c3a05e5041 Set 1.8.0 release date 2014-02-18 10:09:17 +01:00
Sandro Santilli
4c0ab92771 Default to installed mapnik_version during testing
See https://github.com/CartoDB/Windshaft/issues/117
2014-02-18 09:55:00 +01:00
Sandro Santilli
c14378ca5d Avoid checking for table privacy when not using table maps
See #147
2014-02-17 18:20:18 +01:00
Sandro Santilli
26b9c8123d Set maxSocket to allow more than 5 concurrent connections to sql-api 2014-02-17 18:03:11 +01:00
Sandro Santilli
5a504ac1dc Require Windshaft-0.18.2 for statsd-profiler integration 2014-02-17 16:56:44 +01:00
Sandro Santilli
8401dcf6d7 Use /api/v1/map route for unified map api in the examples
Closes #146
2014-02-17 16:29:23 +01:00
Sandro Santilli
1f2e4edd35 Comments cleanup 2014-02-17 11:10:08 +01:00
Sandro Santilli
212eec2ca6 Pass profiler back to windshaft on createLayergroup 2014-02-17 08:50:12 +01:00
Sandro Santilli
935826ed1a Integrate statsd in template instanciation endpoint
NOTE: stats are only enabled using windshaft 0.18.2+
2014-02-15 08:23:43 +01:00
Sandro Santilli
8f3c6c3c87 Add profiler calls in template instanciation endpoint 2014-02-15 08:06:57 +01:00
Sandro Santilli
cd3f8dcf89 Only print layergroupid on template instanciation 2014-02-14 17:33:20 +01:00
Sandro Santilli
9ff192366a Add example mapconfig 2014-02-14 17:24:18 +01:00
Sandro Santilli
63401ca3df Use a single redis client in SignedMap.isAuthorized 2014-02-14 17:07:52 +01:00
Sandro Santilli
8e323a6c07 Remove more commas (see previous commit) 2014-02-14 16:08:32 +01:00
xavijam
d50c6c6dc3 close #144 - removed unnecessary comma 2014-02-14 13:12:52 +01:00
Sandro Santilli
def474c611 Skip getting geometry type if request has no table 2014-02-14 12:26:34 +01:00
Sandro Santilli
c1b2d16119 rename tablePrivacy_getUserDBName profile label 2014-02-14 11:47:43 +01:00
Sandro Santilli
678d653ee9 Allow configuring TTL of mapConfigs via "mapConfigTTL" 2014-02-13 15:44:54 +01:00
Sandro Santilli
4a6af108b4 Fix use of maxUserTemplate configuration variable 2014-02-13 15:01:58 +01:00
Sandro Santilli
e4cd37647e Allow limiting number of allowed user templates
Adds maxUserTemplates directive.
Closes #136
2014-02-13 14:55:31 +01:00
Sandro Santilli
4254f56093 Fix output from list_template to be more readable 2014-02-13 12:56:44 +01:00
Sandro Santilli
f7cef9dcd8 Fix bogus reference in SignedMaps when globals.environment is not set 2014-02-13 10:57:41 +01:00
Sandro Santilli
1c69eb1ae4 Add example template to use with the commandline tools 2014-02-13 10:31:28 +01:00
Sandro Santilli
b673cb2a1f Add more detailed profile info about the "authorize" step
Closes #142
2014-02-13 10:25:28 +01:00
Sandro Santilli
e88e49001a Do not retrive user's api key if no api key was provided
Reduces redis interaction, see #142
2014-02-13 10:16:11 +01:00
Sandro Santilli
6a599ccb5d Add script to list templates 2014-02-13 09:19:19 +01:00
Sandro Santilli
a90bf2e87b Require windshaft 0.18.1 for improve garbage collection 2014-02-13 08:54:47 +01:00
Sandro Santilli
115b1a5267 Add reference to node-0.10 ticket 2014-02-13 08:46:16 +01:00
Sandro Santilli
84e346057e Have travis also build with node-0.10
Closes #141
2014-02-13 08:15:43 +01:00
Sandro Santilli
333de67ed5 Require Windshaft 0.18.0 2014-02-12 22:51:52 +01:00
Sandro Santilli
e4dd215808 Tested with node-0.10.25, works fine 2014-02-12 22:45:00 +01:00
javi
c214e269e9 added statsd to npm-shrinkwrap.json 2014-02-12 16:48:48 +01:00
Sandro Santilli
466eac18a7 Recommend setting "from" for other packages too 2014-02-12 16:23:52 +01:00
Sandro Santilli
6e5d8b2d30 Loosen node-varnish dep 2014-02-12 16:23:12 +01:00
Sandro Santilli
bf45bbea56 Do not send multiple equal commands to Varnish on connect
Closes #135
Also accept varnish "secret" in config
2014-02-12 16:14:27 +01:00
Sandro Santilli
cdbcc7dc18 Put statsd config in all example configs 2014-02-12 16:01:50 +01:00
Sandro Santilli
66e57606d2 Retarget to 1.8.0 for the statsd addition 2014-02-12 16:00:15 +01:00
Sandro Santilli
c7f3bb5722 Target 1.7.2 2014-02-12 15:54:21 +01:00
Sandro Santilli
0e2f921b7e Add flush_cache script. Closes #140 2014-02-12 15:54:21 +01:00
javi
c421ea6bfc added basic statsd cnfiguration options in sample file #139 2014-02-12 15:33:41 +01:00
javi
01feeae6f4 include state configuration for windshaft fixes #139 2014-02-12 15:27:42 +01:00
Sandro Santilli
1ff52fcd00 Add windshaft.from tweak for npm-shrinkwrap.json 2014-02-12 10:51:47 +01:00
Sandro Santilli
6db25c3b6a Release 1.7.1 2014-02-12 10:42:51 +01:00
Sandro Santilli
88deded0fe Workaround npm registry bug by downloading windshaft from github
See the problem here:
https://travis-ci.org/CartoDB/Windshaft-cartodb/builds/18663972
2014-02-11 19:33:05 +01:00
Sandro Santilli
fc0f2b5952 Require windshaft 0.17.2 for further reducing log noise
Closes #137
2014-02-11 17:31:21 +01:00
Sandro Santilli
e211e944e5 Set target version to 1.7.1 2014-02-11 16:37:22 +01:00
Sandro Santilli
a948038ff4 Disable debug logging unless "debug" config param evaluates to true
Closes #137
2014-02-11 16:34:43 +01:00
Sandro Santilli
c70d192987 Release 1.7.0 2014-02-11 15:19:36 +01:00
Sandro Santilli
3fc8630634 Require newer windshaft, regenerated shrinkwrap 2014-02-11 15:19:13 +01:00
Sandro Santilli
8c013ed2d1 Rename Step function in setDBConn 2014-02-11 13:42:44 +01:00
Sandro Santilli
7a749631e8 Fix profiler labels 2014-02-11 13:40:17 +01:00
Sandro Santilli
e3a5f398e4 Add test for instace token changing on template change 2014-02-10 15:48:35 +01:00
Sandro Santilli
747f4803ba Include hash of template in the maptoken returned from instanciation
Doing so basically removes the need to include the template identifier
in the surrogate keys of the responses for resources fetched via
the instance whenever template is updated. See #105
2014-02-10 15:30:35 +01:00
Sandro Santilli
24709e8341 Add acceptance test for use of attributes service from template
Closes #120
2014-02-10 12:31:36 +01:00
Sandro Santilli
53861ad327 Populate test private table 2014-02-10 12:31:00 +01:00
Sandro Santilli
399bed34ad Do not try to replace template variables in undefined elements
See #133
2014-02-10 11:26:21 +01:00
Sandro Santilli
6b41fef96c Fix sendError calls to receive the full Error instance 2014-02-10 11:11:35 +01:00
Sandro Santilli
031e2a2e0c Add test for missing cartocss from mapnik layer on layergroup post
See #133
2014-02-10 11:05:02 +01:00
Sandro Santilli
9b4787c4b7 Reword in bug fixes NEWS entries 2014-02-07 18:10:45 +01:00
Sandro Santilli
fe6e915c0d Always set database access parameters from req2params
Fixes privileged database access from unauthorized users while
fetching torque tiles or feature attributes (unreleased feature).
Closes #132.

Includes testcase, which closes #119
2014-02-07 18:08:41 +01:00
javi
b5d67ec6c0 updated news for #130 2014-02-06 17:46:39 +01:00
javi
f5e0d06e2f fixed when default value in a template attribute is a number and type = number checking fails fixed #130 2014-02-06 17:45:48 +01:00
javi
78f69d5236 template variables with spaces are not replaced correctly fixed #129 2014-02-06 17:33:26 +01:00
Sandro Santilli
ab7d603171 Drop unified.js acceptance test.
Does not really make sense now that endpoints are configurable.
See #126 and #127
2014-02-06 12:58:10 +01:00
Sandro Santilli
b4936ffafa Do not allow creating template with auth='token' and no valid tokens
Closes #128
Includes acceptance test for both creation and update
2014-02-06 12:24:14 +01:00
Sandro Santilli
752e9ec655 Add checkInvalidCertificate method for SignedMap class
Includes unit test
2014-02-06 12:05:01 +01:00
Sandro Santilli
9018e39762 Make endpoints configurable
Closes #127
Uses /api/v1/maps* in the production and staging example configs,
keeps /maps* for development and test (they are examples...)
2014-02-05 15:14:47 +01:00
Sandro Santilli
a964ed5fe6 Implement Unified Map API
Closes #126
2014-02-04 19:04:59 +01:00
Sandro Santilli
b862904506 Be explicit about the map output srid configuration 2014-02-04 16:26:26 +01:00
javi
7197cc2d62 added stack to response in development mode 2014-02-04 14:58:21 +01:00
Sandro Santilli
b01570924d Add support for torque tiles and attributes fetching
Retargets self to 1.7.0
Upgrades Windshaft to 0.16.0

Closes #118 -- CDB-1525 #resolve
Closes #112 -- CDB-1329 #resolve
2014-02-04 13:30:59 +01:00
Sandro Santilli
db478579c5 Fix example development configuration to avoid use of empty sqlapi.domain
This is because as of CartoDB-SQL-API-1.8.2 the "user_from_host"
default configuration for "development" environment is:

  '^(.*)\\.localhost'

Which would not match a domain-less hostname

Closes #117 for real now.
2014-01-30 16:56:23 +01:00
Sandro Santilli
978ea9cd04 Fix sqlapi request header to be "Host", not "Hostname"
Closes (better) #117 -- automated test included
2014-01-30 16:46:26 +01:00
Sandro Santilli
ca4f3d2025 Re-introduce sqlapi.host directive, allowing DNS lookups drop
For backward compatibility, sqlapi.host is only used if domain
is also defined and has a different value (empty string allowed).

Closes #117
2014-01-30 16:12:37 +01:00
Sandro Santilli
c0020fd75a Release 1.6.3 2014-01-30 12:44:25 +01:00
Sandro Santilli
add4255bdc Update windshaft to 0.15.1, fixing maxzoom in layergroup
Regenerates shrinkwrap, which includes other minor updates
in dependency modules.
2014-01-30 12:42:11 +01:00
Sandro Santilli
1f0faba71c Stop processing XML on renderer creation
Not needed anymore since 1.6.1 introduced on-demand XML generation.
2014-01-30 11:14:52 +01:00
Sandro Santilli
e3f2658d53 Port show_style to node (really needed now) 2014-01-29 16:01:27 +01:00
Sandro Santilli
f7cdb5f0b7 Typo 2014-01-29 15:14:47 +01:00
Sandro Santilli
d32278b227 Rename template instanciation function 2014-01-29 14:30:27 +01:00
Sandro Santilli
76acc5af99 Indent and other minor tweaks 2014-01-29 13:34:22 +01:00
javi
5755e382fb Merge branch 'master' of github.com:Vizzuality/Windshaft-cartodb 2014-01-29 13:12:40 +01:00
javi
95c450fe99 update NEWS for #116 2014-01-29 13:12:19 +01:00
javi
ad0b2ffc8e added support for template instanciation with jsonp closes #116 2014-01-29 13:11:37 +01:00
Sandro Santilli
1b1b6b975e Add test for malformed CartoCSS error (#115)
The test is disabled for it's failing, it isn't yet decided if
the regression has to be fixed or not.
2014-01-29 10:40:35 +01:00
Sandro Santilli
67e4e7e99b Set api_key to signer's when instanciating a template map
Closes #114
2014-01-28 12:37:41 +01:00
javi
ac31c69c80 added spec to test instanciation of open templated maps without api_key 2014-01-28 12:12:33 +01:00
javi
92ca447c06 fixed #91 2014-01-28 12:05:01 +01:00
javi
bdea9f10fc fixed sqlemu to return forbidden when table name contains "private" in its name 2014-01-28 12:04:10 +01:00
Sandro Santilli
dc3d36e0a5 Prepare for 1.6.3 2014-01-23 12:27:39 +01:00
Sandro Santilli
99ef396aeb Release 1.6.2 2014-01-23 12:25:34 +01:00
javi
69d7fb0344 fixed news #113 2014-01-22 19:12:17 +01:00
javi
e4e08db0b4 Merge branch 'master' of github.com:Vizzuality/Windshaft-cartodb 2014-01-22 19:10:37 +01:00
javi
164d952e56 support CORS in template instanciation endpoint, fixes #113 2014-01-22 19:10:09 +01:00
Sandro Santilli
c711dc328e Fix XML print from in show_style for token styles (#110) 2014-01-17 17:47:37 +01:00
Sandro Santilli
8b80ad8ba1 Restore XML print from the show_style tool
Closes #110
2014-01-16 18:51:02 +01:00
Sandro Santilli
5772c81590 Fix support for long (>64k chars) queries in layergroup creation
Closes #111. Includes testcase.
2014-01-16 17:20:30 +01:00
Sandro Santilli
09d4467e22 Prepare for 1.6.2 2014-01-16 17:19:55 +01:00
Sandro Santilli
d22f399f18 Release 1.6.1 2014-01-15 19:23:20 +01:00
Sandro Santilli
f89fd98ed7 Expect malformed response objects (#109)
Include test for sql errors on layergroup creation
Closes #109
2014-01-15 11:53:19 +01:00
Sandro Santilli
b01ce9d4cc Regenerate shrinkwrap for 1.6.1 2014-01-14 18:09:36 +01:00
Sandro Santilli
18ccd3cbaf Localize external CartoCSS resources at renderer creation time
Closes #108. JIRA CDB-1422 #resolve
2014-01-14 16:20:06 +01:00
Sandro Santilli
d6fe5339cf Do not choke on headers cleanup when response headers are not set
Raise a WARNING instead.
See #107 (github) and CDB-1438 (JIRA)
2014-01-13 18:56:09 +01:00
Sandro Santilli
2690ef3f05 Drop cache headers from error responses.
Closes #107 (github), #resolve CDB-1423 (JIRA)
2014-01-13 11:20:02 +01:00
Sandro Santilli
ae82d0ab47 Expect overrides of mapnik_version to be honoured
Reported on http://gis.stackexchange.com/questions/81450/cartodb-windshaft-error
2014-01-10 13:20:26 +01:00
Sandro Santilli
90e0a5dc30 Prepare for 1.6.1 2014-01-10 11:32:03 +01:00
Sandro Santilli
c1b6b865a7 Release 1.6.0 2014-01-10 11:30:10 +01:00
Sandro Santilli
d849ae216d Keep build status line within 80 cols (and use http) 2014-01-09 18:01:22 +01:00
Sandro Santilli
4ee4492490 Yet another username extraction fix. Thanks again @demimismo.
Closes #100 (yet again)
2014-01-09 16:46:47 +01:00
Sandro Santilli
fcd17692ee Fix username extraction in another two places. Thanks @demimismo.
Closes #100 (again)
2014-01-09 15:36:16 +01:00
Sandro Santilli
36159a7697 Change stresstester to always create a different template 2013-12-20 13:47:28 +01:00
Sandro Santilli
7886189bce Add script to stress-test templates API 2013-12-20 13:14:52 +01:00
Sandro Santilli
3a681b6670 Exit with error if template creation response text contains a space
Should really check for response code, but dunno how to do that
right away
2013-12-20 13:11:32 +01:00
Sandro Santilli
3e4c141913 Add command-line script to delete a template 2013-12-20 12:55:01 +01:00
Sandro Santilli
ef3733aebe Improve error on attempt to delete missing template 2013-12-20 12:54:38 +01:00
Sandro Santilli
b5f54ff534 Rename script to create multilayer 2013-12-20 10:44:11 +01:00
Sandro Santilli
ba494374d0 Add command-line script to instanciate a template 2013-12-20 10:43:48 +01:00
Sandro Santilli
c7465479a2 Improve error on a signature certificate with no or broken auth 2013-12-20 10:41:27 +01:00
Sandro Santilli
b14830e4e3 Add command-line script to update a template 2013-12-20 10:17:31 +01:00
Sandro Santilli
288f23eea2 Add script for command-line template creation 2013-12-20 10:15:18 +01:00
Sandro Santilli
50a902a90b Fix english of error message for sql-api connection problems 2013-12-18 12:59:26 +01:00
Sandro Santilli
277c00c7f8 Advertise `infowindow and map_metadata` as deprecated APIs 2013-12-18 12:54:40 +01:00
Sandro Santilli
4a09ac5b8f Add reference to template maps API extention 2013-12-18 12:54:09 +01:00
Sandro Santilli
d5e9e0559b Add info about Imagemagick requirement for running tests 2013-12-18 12:53:22 +01:00
Sandro Santilli
0dffb0fe85 We don't use Windshaft directly, no need to require it 2013-12-17 17:48:36 +01:00
Sandro Santilli
0f90d687c7 Implement signed teplate maps
Closes #98

Raises minimum required redis version to 2.4.0+ (Debian stable has 2.4.14)
2013-12-17 17:39:21 +01:00
Sandro Santilli
84b7d78ea4 Add an utility authorizedByAPIKey method for reuse 2013-12-17 17:17:17 +01:00
Sandro Santilli
241480bb23 cartodb-redis is upgraded to 0.3.0 2013-12-17 17:17:17 +01:00
Sandro Santilli
73a065c1cc Make sure user from domain is always computed locally
Involved upgrade of cartodb-redis to 0.3.0
Really closes #100
2013-12-17 17:17:17 +01:00
Sandro Santilli
1f693c6c78 Add 'user_from_host' directive to generalize username extraction
Closes #100
Default extractor is backward compatible
2013-12-17 17:17:17 +01:00
Sandro Santilli
e9db535dd8 Drop the idea that we can distinguish a "dbowner" from the domain
We only recognize "users"
2013-12-17 17:17:17 +01:00
Sandro Santilli
7b7408dab7 Revert "Drop /map_metadata API entry point"
This reverts commit b37b07a06a1dd3cf05d60f4aa613ab5c48b90700.

This was too light of a decision...
2013-12-17 17:17:17 +01:00
Sandro Santilli
9c897a91a9 Drop /map_metadata API entry point
Closes #101
2013-12-17 17:17:17 +01:00
Sandro Santilli
4189f8187f Simplify redis test setup using HMSET
See http://redis.io/commands/hmset
2013-12-17 17:17:16 +01:00
Sandro Santilli
98565b0c6b Shrinkwrap cartodb-redis dependency to "~0.2.0"
npm-shrinkwrap takes precedence over package.json...
See https://travis-ci.org/CartoDB/Windshaft-cartodb/builds/15036101
2013-12-17 17:17:16 +01:00
Sandro Santilli
38342a7f5f Refactor req2params to make setting db credential easier 2013-12-17 17:17:16 +01:00
Sandro Santilli
6f689745c0 Fix lzma testcase 2013-12-17 17:17:16 +01:00
Sandro Santilli
63fd660eb1 Fix error handling in testcase 2013-12-17 17:17:16 +01:00
Sandro Santilli
fa14b6045d Retarget to 1.6.0 2013-12-17 17:17:16 +01:00
Sandro Santilli
f2528fb462 Release 1.5.2 2013-12-17 17:17:16 +01:00
Sandro Santilli
0db0809146 Fix use of old layergroups on mapnik upgrade (#97) 2013-12-17 17:17:16 +01:00
Sandro Santilli
276422f4be Set grainstore's GC run probability, for documentation purpose
It sets it to the current grainstore default, so nothing changes.
2013-12-17 17:17:16 +01:00
Sandro Santilli
e6b55ac034 Allow requesting run_test.sh to prepare redis but not postgresql
Adds --nocreate-pg, --nocreate-redis, --nodrop-pg, --nodrop-redis
NOTE that dropping pg is still unimplemented
2013-12-17 17:17:16 +01:00
Sandro Santilli
58af35fdea Add backward-compatibility fix item in NEWS (#96) 2013-12-17 17:17:16 +01:00
Sandro Santilli
763989bc87 Prepare for 1.5.2 2013-12-17 17:17:16 +01:00
Sandro Santilli
385022de80 Revert "fixed #91" -- the fix was for an unconfirmed bug
This reverts commit 9155724082.
See #38 for further action
2013-12-17 17:17:16 +01:00
Sandro Santilli
6c104e2aca Enable test for fetcing tiles of private tables using api_key
See #39 and #91
2013-12-17 17:17:16 +01:00
Sandro Santilli
363c0d28f4 Add test for fetching tile of private table showing api_key
See #38 and #91
2013-12-17 17:17:16 +01:00
javi
a378fc4e68 fixed #91 2013-12-17 17:17:16 +01:00
javi
01de288c35 fixed #96 2013-12-17 17:17:15 +01:00
Sandro Santilli
f1a68e4451 Release 1.5.1 2013-12-17 17:17:15 +01:00
Sandro Santilli
f429b86f48 Accept unused CartoCSS directives
Closes #93

An example unused CartoCSS directive is
"point-transform" without "point-file"
or "point-url". Unused means it has no effect.

It used to be accepted but regressed in release 1.5.0
2013-12-17 17:17:15 +01:00
Sandro Santilli
ccfdacff5b Fix test for invalid font usage after Windshaft update (#90)
NOTE: the error is less friendly now, see
      http://github.com/mapbox/carto/issues/242
2013-12-17 17:17:15 +01:00
Sandro Santilli
a9d9b765e8 Survive presence of malformed CartoCSS in redis
Closes #94, enable relative testcase
2013-12-17 17:17:15 +01:00
Sandro Santilli
5298f4b517 Add package keywords 2013-12-17 17:17:15 +01:00
Sandro Santilli
53d03e82ab Set test redis port to 6335 2013-12-17 17:17:15 +01:00
Sandro Santilli
2fa288fc4d Add (pending) test for getting unrenderable stored styles (#94)
Required upgrading mocha tester to ~0.14.0
2013-12-17 17:17:15 +01:00
Sandro Santilli
73819579f3 Notify travis builds on #cartodb @ freenode.irc 2013-12-17 17:17:15 +01:00
Sandro Santilli
271ff4faeb Use a variable to hold the name of test database 2013-12-17 17:17:15 +01:00
Sandro Santilli
c04ac4fc7e Reduce ppa and explicit package usage
Should fix travis builds despite package compatibilit bugs
(https://travis-ci.org/CartoDB/Windshaft-cartodb/builds/14314805)
2013-12-17 17:17:15 +01:00
Sandro Santilli
5a87a16311 Add note about new directives in the 1.5.0 section 2013-12-17 17:17:15 +01:00
Sandro Santilli
dd48aa73e2 Improve documentation for postgres_auth_* configuration directives 2013-12-17 17:17:15 +01:00
Sandro Santilli
6dd046a1a4 Prepare for 1.5.1 2013-12-17 17:17:15 +01:00
Sandro Santilli
baaacbed31 Release 1.5.0 2013-12-17 17:17:15 +01:00
Sandro Santilli
0b3fdb07f6 Drop unneeded include from outermost app 2013-12-17 17:17:15 +01:00
Sandro Santilli
cc09a8b66f Update to cartodb-redis 0.1.0 2013-12-17 17:17:15 +01:00
Sandro Santilli
a60a3adc12 CartoDB redis interaction delegated to "cartodb-redis" module 2013-12-17 17:17:14 +01:00
Sandro Santilli
e412a0f4b6 Require windshaft-0.14.3 to get 3 new bugfixes:
- Return CORS headers when creating layergroups via GET
 - Fix http status on database authentication error
 - Ensure bogus text-face-name error raises at layergroup creation
2013-12-17 17:17:14 +01:00
Sandro Santilli
ed23d10364 Remember per-environment ./configure parameters
This is to avoid breaking test.js configuration while switching
between branches.
2013-12-17 17:17:14 +01:00
Sandro Santilli
4c95af2c69 Fix ticket reference 2013-12-17 17:17:14 +01:00
Sandro Santilli
baa95a62d1 Add support for reading user-specific database_password from redis
This commits adds support for CartoDB-2.5.0 model.
Closes #89.
Change is backward compatible.
2013-12-17 17:17:14 +01:00
Sandro Santilli
c7494c3c73 Avoid caches during test for user-specific database_host 2013-12-17 17:17:14 +01:00
Sandro Santilli
12f0826d32 Do not force ending dot in SQL-API hostname, for easier testing 2013-12-17 17:17:14 +01:00
Sandro Santilli
428e8631e2 Improve tests robustness on failure 2013-12-17 17:17:14 +01:00
Sandro Santilli
d3e3cfa385 Add NEWS item about CartoDB-2.5.0+ user-specific database_host (#88) 2013-12-17 17:17:14 +01:00
Sandro Santilli
3120d56e80 Add test for redis-specifid database_host. Closes #88 2013-12-17 17:17:14 +01:00
Sandro Santilli
07cb36ebc7 Read user's database_host from redis, when available (#88)
Still lacks a testcase
2013-12-17 17:17:14 +01:00
Sandro Santilli
d7c82e7a51 Indent fixes 2013-12-17 17:17:14 +01:00
Sandro Santilli
bf340e684a Tweak error messages on missing redis variables, update tests 2013-12-17 17:17:14 +01:00
Luis Bosque
8d1b394df1 Add function to read database host from redis 2013-12-17 17:17:14 +01:00
Sandro Santilli
d305dbd468 Style only change 2013-12-17 17:17:13 +01:00
Sandro Santilli
eb51d18012 Add support for specifying database connection passwords 2013-12-17 17:17:13 +01:00
Sandro Santilli
4f3f87fc13 Release 1.14.1 2013-12-17 17:17:12 +01:00
Sandro Santilli
3e6070bd9b Fix support for exponential notation in CartoCSS filter values
Closes #87.
Includes testcase
2013-12-17 17:17:12 +01:00
Sandro Santilli
0daba348fe Prepare for 1.4.1 2013-12-17 17:17:12 +01:00
Sandro Santilli
f874e8844c Add Support for Mapnik-2.2.0. Closes #78. 2013-12-17 17:17:12 +01:00
Sandro Santilli
a8fef04455 Prepare for mapnik-2.2.0 support (#78)
- Tolerate change in CartoCSS error message between 0.9.3 and 0.9.5
- Expect default style to be different for mapnik-2.2.0+ target
2013-12-17 17:17:12 +01:00
Sandro Santilli
2f74a080ee Prepare for 1.3.7 2013-12-17 17:17:12 +01:00
Sandro Santilli
198748feea Release 1.3.6, fixing support for node-0.8.9 2013-12-17 17:17:12 +01:00
Sandro Santilli
9f73be0d5c Prepare for 1.3.6 2013-12-17 17:17:12 +01:00
Sandro Santilli
8aea5041c7 Release 1.3.5 2013-12-17 17:17:12 +01:00
Sandro Santilli
1856b824cb Fix support for apostrophes in CartoCSS
Requires windshaft 0.13.7
Jira ref CDB-414
2013-12-17 17:17:12 +01:00
Sandro Santilli
a27cf1b41c Do not let anonymous requests use authorized renderer caches
Puts dbuser in params, for correct use by Windshaft renderer cache.
Before this fix, and after commit 1c9f63c9, the renderer cache key
did not contain the db user.
2013-12-17 17:17:12 +01:00
Sandro Santilli
b610b9aca2 tweak test description 2013-12-17 17:17:12 +01:00
Sandro Santilli
f5c24cf252 Add more profile slots 2013-12-17 17:17:11 +01:00
Sandro Santilli
8303068310 Remove spaces from configuration input, to make editing easier :) 2013-12-17 17:17:11 +01:00
Sandro Santilli
c17fd3b254 Make testsuite accept an installed mapnik version 2.1.0
See https://travis-ci.org/CartoDB/Windshaft-cartodb/builds/11286823
2013-12-17 17:17:11 +01:00
Sandro Santilli
d82838a137 Add travis widget, fix documented node dependency 2013-12-17 17:17:11 +01:00
Sandro Santilli
4506a9e905 Add travis configuration 2013-12-17 17:17:11 +01:00
Sandro Santilli
b4580943e8 Read test redis port configuration from test.js env 2013-12-17 17:17:11 +01:00
Sandro Santilli
730f9534dc Clean handling of redis connection failures in testcase 2013-12-17 17:17:11 +01:00
Sandro Santilli
a7cc7ceeb8 Fix error for invalid text-name in CartoCSS. Closes #81. 2013-12-17 17:17:11 +01:00
Sandro Santilli
7861852078 Add backward compatibility sqlapi configuration item in NEWS 2013-12-17 17:17:11 +01:00
Sandro Santilli
dbf6bb5fca Only use sqlapi configuration "host" if "domain" is undefined
We'll consider an empty string domain as valid (it's actually used
for testsuite).
2013-12-17 17:17:11 +01:00
Javier Arce
d4d5272bf2 Sets the sqlapi domain. Fixes #82 2013-12-17 17:17:11 +01:00
Sandro Santilli
0c4bcca7c9 Read redis port from test.js environment when running tests 2013-12-17 17:17:11 +01:00
Sandro Santilli
0414307679 Fix use of blank-prefixed "zoom" variable in CartoCSS 2013-12-17 17:17:11 +01:00
Luis Bosque
0f3a5501d4 Target v1.3.5 2013-12-17 17:17:11 +01:00
Luis Bosque
9d4ce3f070 Merge branch 'release/staging' 2013-09-06 13:54:04 +02:00
Sandro Santilli
17fc934aa3 Fix check-submodules rule not to fail if a submodule is missing 2013-09-04 18:19:09 +02:00
Sandro Santilli
27eaad932a Fix race condition in localization of external resources 2013-09-04 17:56:17 +02:00
Sandro Santilli
602b255ce5 Upgrade to latest windshaft for more stable test results 2013-09-03 18:15:37 +02:00
Sandro Santilli
efc6d5c857 Fix config/environments/test.js Makefile rule 2013-08-21 10:50:49 +02:00
Sandro Santilli
633e8d164b Rename sqlapi.host configuration to sqlapi.domain. Closes #79.
Support for "host" is retained for backward compatibility.
2013-08-21 10:11:30 +02:00
Sandro Santilli
db951234aa Add note about cache dir need to be writable by server user
As per
https://groups.google.com/d/msg/cartodb/z06r9SwaoOM/b34In4TTdd0J
2013-08-21 10:04:43 +02:00
Sandro Santilli
60617e7641 Upgrade windshaft to improve CSS error messages 2013-08-13 13:03:24 +02:00
Luis Bosque
7b43a0f0bd Merge branch 'release/staging' 2013-07-24 13:33:22 +02:00
Luis Bosque
06e68c1518 Target v1.3.4 2013-07-23 19:43:32 +02:00
Sandro Santilli
b4045353ea Fix contributors, use short form for author 2013-07-19 12:34:20 +02:00
Sandro Santilli
22130f37df Fix URLS and repository, add contributors, change author
Author became Vizzuality and contributors are the actual committers
2013-07-19 12:28:03 +02:00
Sandro Santilli
bce024961f Add note about layergroup logging at creation
The change is due to Windshaft upgrade.
Closes #76
2013-07-18 11:51:00 +02:00
Sandro Santilli
3819d0d47b Upgrade windshaft to improve profiling 2013-07-18 11:13:41 +02:00
Sandro Santilli
7cb69d1db9 Add example of including profile in response log line 2013-07-18 09:53:59 +02:00
Sandro Santilli
ec97381820 Add more timing in the profile, add useProfiler config variable
Default to useProfiler:true in staging and development
2013-07-16 16:33:03 +02:00
Sandro Santilli
25c46962cb Add --environment switch to restrict output to given env 2013-07-15 13:59:55 +02:00
Sandro Santilli
23da8538a6 Add --with-sqlapi-port switch 2013-07-15 13:55:13 +02:00
Sandro Santilli
381b9a9edf Take cache_buster value, if present, as a Last-Modified timestamp
This makes the Last-Modified header consistent across requests
using the same cache_buster (embedded in the token for multilayer
API).
2013-07-15 13:48:06 +02:00
Sandro Santilli
76c056c7a1 Revert "Use a constant Last-Modified time with cache_policy=persist"
This reverts commit 4b5899ff1a.

The reason is that setting Last-Modified to a remote date in the past
triggers early expiration of cache (as max-age will be reached sooner)
2013-07-15 13:14:06 +02:00
Sandro Santilli
4b5899ff1a Use a constant Last-Modified time with cache_policy=persist
After all if the client is asking for persistance it doesn't make
sense to set a different Last-Modified for different incoming
requests (even if we don't expect any) ....
2013-07-15 12:09:13 +02:00
Sandro Santilli
afd4c3b460 Set Last-Modified header to allow for 304 responses 2013-07-15 12:02:54 +02:00
Luis Bosque
8b7cc64567 Merge branch 'release/staging' 2013-07-09 16:08:07 +02:00
Luis Bosque
308246f324 Target v1.3.3 2013-07-09 10:26:43 +02:00
Luis Bosque
b0b40933d8 Merge branch 'release/staging' 2013-07-09 10:25:50 +02:00
Sandro Santilli
0e13228f5c Update NEWS file 2013-07-08 12:15:01 +02:00
Sandro Santilli
65c7c5fc9c Always serve multilayer tiles and grids with persisting cache request 2013-07-08 12:13:45 +02:00
Sandro Santilli
60242c80f4 Set default layergroup time to live in redis to 2 hours 2013-07-08 11:50:19 +02:00
Luis Bosque
5213216fc4 Target v1.3.2 2013-07-05 15:32:47 +02:00
Sandro Santilli
1c65cec6ed Do not consider broken layergroup configs as good on second look
The fix is really in Windshaft, this commit simply requires a
newer version of it.
2013-07-04 17:05:59 +02:00
Sandro Santilli
316c9c209d Add support for specifying tiler url 2013-07-04 12:50:57 +02:00
Sandro Santilli
563764dc4f Only make curl verbose when -v is passed to commandline 2013-07-04 11:15:21 +02:00
Sandro Santilli
8c51c97102 Be more verbose about curl errors 2013-07-04 11:04:12 +02:00
Sandro Santilli
6416af3f16 Add support for listing all map styles that belong to a user
Also report user database name, as a facility
2013-07-03 09:59:42 +02:00
Sandro Santilli
00c18ab8dd Raise windshaft dependency importing following fixes:
- support for CartoCSS attachments (#layer0::label)
 - only check layergroup validity once
 - use higher zoom level for checking layergroup validity
2013-06-28 19:05:39 +02:00
Sandro Santilli
632d75a7c8 specify units for rendererConfig.cache_ttl 2013-06-28 17:58:11 +02:00
Sandro Santilli
d7b1ff9a80 Set default layergroup ttl locally 2013-06-26 16:26:02 +02:00
Sandro Santilli
10a66fbd66 Fix SQL error reporting to NOT split on newline 2013-06-26 13:26:53 +02:00
Sandro Santilli
4b1d6cd729 Add tile and grid fetching checks at layergroup creation time
Basically requires windshaft 0.12.6, which implements this
2013-06-21 12:46:13 +02:00
Sandro Santilli
e984811eae Fix SQL bug in testcase 2013-06-19 17:24:01 +02:00
Sandro Santilli
eb83851bb7 Fix database authentication with multi-table layergroups 2013-06-17 17:24:09 +02:00
Sandro Santilli
850d4cd6ba Drop unused cluster support 2013-06-17 11:57:35 +02:00
Sandro Santilli
6cb8c85da0 Only return token by default and full response on request (-v) 2013-06-13 10:43:09 +02:00
Sandro Santilli
3d4af14315 Fix deadlock on layergroup post
Required upgrading windshaft to 0.12.5.
Took the chance to also upgrade redis dependency to latest stable.
2013-06-12 18:42:10 +02:00
Sandro Santilli
c07616f889 Run silently, prepare for using against windshaft direct 2013-06-12 18:30:32 +02:00
Sandro Santilli
d53327bc33 Utility to fetch a token from a layergroup config
It's in its infancy, still need to edit the script to use against
remote service or non-standard local deploys
2013-06-11 17:15:02 +02:00
David Arango
f20c98e49c Merge branch 'release/staging' 2013-06-11 14:41:12 +02:00
Luis Bosque
9807a5e12b Target v1.3.1 2013-06-11 10:53:07 +02:00
Sandro Santilli
70f535d13a Properly report error from unsuccessful source table fetching
Report terse error to user, verbose to log
2013-06-11 10:28:05 +02:00
Sandro Santilli
8d326dba2f Add test for CartoCSS error reporting on POST to layergroup 2013-06-10 11:53:28 +02:00
Sandro Santilli
8d0da4d517 Require latest grainstore and windshaft, for multilayer css fixes 2013-06-06 15:45:33 +02:00
Sandro Santilli
63296a87cb Do not increment undefined mapview stat tags 2013-06-06 13:26:59 +02:00
Sandro Santilli
b67a487c48 Fix path to grainstore module 2013-06-06 11:59:11 +02:00
Sandro Santilli
d977f83bd1 Change stats format for multilayer map token request
See https://github.com/Vizzuality/Windshaft-cartodb/wiki/Redis-stats-format

Target 1.3.0
2013-06-04 13:30:28 +02:00
Sandro Santilli
5b6919e0c6 Fix unit of measure for lastUpdated info extraction 2013-05-30 16:48:40 +02:00
Luis Bosque
92a3d52885 Merge branch 'release/staging' into develop 2013-05-30 11:18:57 +02:00
Luis Bosque
627b3f084d Merge branch 'release/staging' 2013-05-30 11:18:36 +02:00
Sandro Santilli
3763232c14 Require latest windshaft to fix confusion between colors and layer names 2013-05-29 19:14:45 +02:00
Sandro Santilli
3e4f98cb51 Require higher Windshaft to fix handling of CartoCSS layer name 2013-05-29 11:43:27 +02:00
Luis Bosque
977d051518 Target v1.2.2 2013-05-28 12:40:20 +02:00
Sandro Santilli
d7ae28095f Require Windshaft-0.12.1 to fix multilayer post from firefox 2013-05-21 14:43:00 +02:00
Luis Bosque
b98a32c296 Merge branch 'release/staging' 2013-05-20 14:41:41 +02:00
Sandro Santilli
5ab4caea7d Make test for LZMA more robust in case of failure 2013-05-16 09:41:38 +02:00
Luis Bosque
9ce88bcc98 Target v1.2.1 2013-04-29 15:38:36 +02:00
Luis Bosque
94ddbf69d8 Target v1.2.0 in package.json 2013-04-29 15:36:54 +02:00
javi
c9e3cc00be merged with develop 2013-04-24 15:16:20 +02:00
javi
efa79b243c fixed lzma decoding to fix browser requirements 2013-04-24 15:10:58 +02:00
Sandro Santilli
06f781334d Drop unused variable 2013-04-24 12:01:32 +02:00
Sandro Santilli
f0fc44aac9 Fix fetching of affected tables when mapnik tokens are used
We'll replace !bbox! with an empty box and !pixel_width! and
!pixel_height! with 1 before passing the query to CDB_QueryTable
2013-04-23 17:29:49 +02:00
Fabio Rueda
b678f82be8 Merge branch 'release/staging' 2013-04-17 16:17:02 +02:00
Sandro Santilli
44a8db03e2 Add node-varnish to the list of submodules tested by check-full 2013-04-15 09:33:41 +02:00
Sandro Santilli
4a12ed1f19 Oops, forgot to drop --nodrop switch after unifying local tests 2013-04-15 09:32:32 +02:00
Sandro Santilli
de24e7fc34 Update to new LZMA to fix global variable leakage
Stop ignoring leaks during testing
2013-04-15 09:22:50 +02:00
Sandro Santilli
1b6a662a72 Tweak PATH during submodule check, to find mocha from subdirs 2013-04-15 09:20:51 +02:00
Sandro Santilli
6f8fd69101 Add note about the stats addition 2013-04-12 17:57:57 +02:00
Sandro Santilli
78a6f4de1b Keep a counter of layergroup created per user.
The counter is in redis db 5, in a field "mapviews" of an hash
"tiler:users:USERNAME". It's incremented whenever the layergroup
token for a configuration is requested.
2013-04-12 17:28:34 +02:00
Sandro Santilli
afb875c32a Add missing fixtures (thanks Fabio) 2013-04-12 14:34:45 +02:00
Sandro Santilli
369b0f6110 Ignore variable leaks exposed by server.js test (due to LZMA mod)
This will have to be fixed once LZMA module is updated.
See https://github.com/nmrugg/LZMA-JS/issues/8

With this change "make check-full" completes successfully for me.
Looking forward to Jenkins experience.
2013-04-12 11:52:09 +02:00
Sandro Santilli
1c910ec513 Insert test data in a single statement 2013-04-11 18:33:51 +02:00
Sandro Santilli
08d1d73b4f Fix "make check" to exit with FAILURE on failure
Also upgrade Windshaft for the same purpose (for make check-full)
2013-04-11 18:31:01 +02:00
Sandro Santilli
83e6e0d457 More verbose logging for SQL api connection errors 2013-04-09 18:07:53 +02:00
Sandro Santilli
e5af3b90f4 Revert "Require interactivity param in single-layer grid fetching request"
This reverts commit 3383c44eb7.

Fixes regression with default interactivity parameter.
Closes #74. See #69.
2013-04-05 18:11:36 +02:00
Luis Bosque
7c82498f8f Merge branch 'release/staging' into develop 2013-04-04 14:23:39 +02:00
Luis Bosque
9162d2cd43 Merge branch 'release/staging' 2013-04-04 14:22:47 +02:00
Sandro Santilli
a0b6f467b1 HOTFIX: require windshaft-0.11.1 to drop tilelive internal cache 2013-04-03 11:10:09 +02:00
Luis Bosque
782edd9506 Target v1.1.10 2013-04-02 14:02:00 +02:00
Sandro Santilli
7ed4320493 Merge branch 'develop-multilayer2' into develop
Target 1.1.9
2013-04-02 13:49:22 +02:00
Sandro Santilli
113b70cf98 Add support for creating layergroups via GET 2013-04-02 13:30:49 +02:00
Sandro Santilli
106e95940d Revert backward incompatible changes in multilayer handling 2013-04-02 12:24:03 +02:00
Luis Bosque
4329ffd61a Target v1.2.1 2013-04-02 11:42:21 +02:00
Sandro Santilli
3383c44eb7 Require interactivity param in single-layer grid fetching request
Closes #69
2013-03-29 18:25:28 +01:00
Sandro Santilli
aea107f1af Upgrade Windshaft to 0.10.0, changing multilayer interface
WARNING: starting from this commit the grid fetching route changed
         to NOT include layer name nor interactivity (which is now
         specified solely as part of layergroup configuration)

Target 1.2.0 release
2013-03-29 16:37:37 +01:00
Sandro Santilli
2b2c22cdd5 Add test showing use of mapnik subtitution tokens in multilayer config 2013-03-28 12:48:00 +01:00
Sandro Santilli
001bf97d69 Add support for LZMA compressed GET parameters
You can now replace the whole query string with a single `lzma`
parameter having as value an hex encoded LZMA compressed version
of the whole query string as a JSON object.
2013-03-22 18:55:59 +01:00
Sandro Santilli
2da8c44914 Fix support for ampersend characters in CartoCSS
Required upgrading grainstore dependency to 0.12.2
2013-03-22 12:42:38 +01:00
javi
e53122de7e fixed last_update in laytergroup response 2013-03-21 11:39:55 +01:00
Sandro Santilli
4e7f92c60d Write a deprecation warning when handling SIGUSR1 (#71) 2013-03-18 18:14:18 +01:00
Sandro Santilli
d95ac85b17 Deprecate USR1 signal handling, add USR2 with same semantic
Closes #71
2013-03-18 16:38:50 +01:00
Sandro Santilli
3ff3dc2c97 Cleanup, handle error in req2param on flushCache 2013-03-15 19:25:13 +01:00
Sandro Santilli
4605bd1e1d Add last_modified field to POST layergroup response (#72)
Includes testcases
2013-03-13 18:41:37 +01:00
Sandro Santilli
dfc4a02398 Fix X-Cache-Channel for multilayer (by token) responses
Required upgrading Windshaft to 0.9.2
Includes testcases
2013-03-13 16:45:15 +01:00
Sandro Santilli
899d8a3c64 Run multilayer test as part of "make check" 2013-03-13 16:40:54 +01:00
Sandro Santilli
402fc90e63 Absence of X-Cache-Channel will be enough for Varnish to skip caching
Do not override Cache-Control in this case, which means let the
clients or geographical proxies cache the response with usual TTL.
2013-03-13 12:01:35 +01:00
Sandro Santilli
fcd6d55ba4 Draft test for X-Cache-Channel in multilayer response
The test is disabled because it fails
2013-03-13 11:55:57 +01:00
Sandro Santilli
4622dac81b Update NEWS 2013-03-13 10:40:06 +01:00
Sandro Santilli
e8cbc666e2 Handle SQL API errors by logging them and requesting NO cache
SQL api is used to determine the list of source tables affected
by a query. Before this commit, the X-Cache-Channel header set
on sql api error was an arbitrary 'table' string, now the header
is omitted, the error logged and Cache-Control and Pragma headers
are sent as an attempt to request no caching.

The code includes test for this mechanism.
2013-03-13 10:39:00 +01:00
Sandro Santilli
49aad435b9 More ignores 2013-03-13 08:48:30 +01:00
Luis Bosque
23f6c82f0e Merge branch 'release/staging' into develop
Conflicts:
	package.json
2013-03-05 14:15:53 +01:00
Luis Bosque
4995011f1e Merge branch 'release/staging'
Conflicts:
	NEWS.md
2013-03-05 14:15:07 +01:00
Luis Bosque
e038aa7522 V1.1.8 in package.json 2013-03-04 11:38:43 +01:00
Luis Bosque
c77dce105c Target v1.1.9 2013-03-04 11:37:10 +01:00
Sandro Santilli
2fa09aee88 Require Windshaft-0.9.1, to reduce harmfulness of cache_buster param 2013-03-01 13:11:46 +01:00
Luis Bosque
a7afdac67e Target v1.1.8 2013-02-28 15:09:54 +01:00
Sandro Santilli
f6d50fafb1 Expose renderer settings in the environment config files
These are: metatile, bufferSize and cache_ttl
2013-02-25 17:05:59 +01:00
Sandro Santilli
f076b0c4d1 Require windshaft-0.9 for multilayer support 2013-02-25 15:08:28 +01:00
Sandro Santilli
9b04abb36c Add UTF grid checker function 2013-02-22 11:34:03 +01:00
Sandro Santilli
14e3ead06e Add redis.max configuration setting and document it. 2013-02-21 12:56:04 +01:00
Sandro Santilli
b98a0f9104 Do not let /etc/services confuse FD checker (munin plugin) 2013-02-20 12:14:16 +01:00
David Arango
dc188817a8 Bump to 1.1.7 2013-02-19 13:57:14 +01:00
Sandro Santilli
b81c83aace Add maxConnection and cache dump notice in NEWS for 1.1.6 2013-02-19 13:37:11 +01:00
Sandro Santilli
9dcf6a1acf Set 'base_url_notable' config for Windshaft-0.9 (multilayer) 2013-02-12 18:53:41 +01:00
Sandro Santilli
c3f53a7987 Rename munin plugin, put all under munin dir 2013-02-11 18:24:18 +01:00
Sandro Santilli
3101dbd433 Add cache count to the stats reported by checkfds.sh (munin plugin) 2013-02-11 17:35:17 +01:00
Sandro Santilli
3c5e358c32 Regenerated npm-shrinkwrap.json with npm-1.1.x
See https://github.com/isaacs/npm/issues/3145 for the rationale
2013-02-11 17:05:44 +01:00
Sandro Santilli
219caf7eab Require windshaft 0.8.5 with some stability issue fixed
Additionally we can now control renderer cache time to live and
metatiling
2013-02-11 16:39:21 +01:00
Sandro Santilli
fb33583df9 Update news file 2013-02-11 15:10:41 +01:00
Sandro Santilli
a79b999e7a Do not try to send commands to an unoconnected redis client
This changes "Cannot read property 'HGET' of null" messages into
"Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED".
2013-02-11 15:05:23 +01:00
Sandro Santilli
031c6ee0cd Allow "staging" as env name from app.js 2013-02-11 13:41:51 +01:00
Sandro Santilli
b3f8515fd5 Fix regexp to find incoming http connections (munin fds plugin) 2013-02-08 18:17:15 +01:00
Sandro Santilli
6e6c1101fe avoid scaling counters (munin) 2013-02-08 17:48:54 +01:00
Sandro Santilli
39c0d3a9a4 tweak graph labels of munin plugin 2013-02-08 17:05:43 +01:00
Sandro Santilli
6476b975d4 Ignore generated munin plugin config file 2013-02-08 16:48:42 +01:00
Sandro Santilli
60b578ae56 Report environment name in munin graph 2013-02-08 16:47:59 +01:00
Sandro Santilli
7dd7d09cf5 Add Makefile rule to install munin plugin 2013-02-08 16:44:34 +01:00
Sandro Santilli
c314640c3b Remove debugging output (munin wouldn't like it) 2013-02-08 16:10:56 +01:00
Sandro Santilli
d1d2074146 Further improvements: support specifying a TILER_ENVIRONMENT env variable
.. or the same thing (a filename) as a command line argument
2013-02-08 16:08:24 +01:00
Sandro Santilli
56319a5ac0 Improvements to the fd checker
1. Finds listening processes automatically
2. Reports highest value among processes for each connection type
3. Reduces lsof calls
4. Uses munin-like output
2013-02-08 15:24:25 +01:00
Sandro Santilli
6b71cde56e Do not throw an Error embedding another Error
Should fix #68, but doesn't come with an automated test
2013-02-08 12:27:49 +01:00
Sandro Santilli
7d34cbbd63 Add item about async throws 2013-02-08 12:23:25 +01:00
Sandro Santilli
cb57dfb27d Fix async throws in getGeometryType, getInfoWindow and getMapMetadata 2013-02-08 12:14:53 +01:00
Sandro Santilli
0b56dd09d9 checkfds: use lsof only once, then grep in the report 2013-02-07 13:26:11 +01:00
Sandro Santilli
2d35a2c269 Add checkfds utility script 2013-02-07 12:32:06 +01:00
Sandro Santilli
6c40d936ef Add a log on cluster startup, including date and pid 2013-02-01 11:28:08 +01:00
Sandro Santilli
61f4212ba2 Remove use of ANSI colors from all example log formats 2013-01-30 12:59:05 +01:00
Sandro Santilli
e408d04fc5 Require Windshaft 0.8.4 to add dump of cache stats on SIGUSR1 2013-01-30 11:29:44 +01:00
Sandro Santilli
2510a47262 Add maxConnection environment configuration, have it default to 128
The number gives the maximum number of contemporary connections and
is dimensioned on the limit of open file descriptors found to be
needed on a per-connection basis.
2013-01-29 17:36:50 +01:00
Sandro Santilli
8d5baebf1d Require latest grainstore and windshaft stable releases
Fixes some http response status to be 400 rather than 500
2013-01-29 13:13:14 +01:00
Sandro Santilli
42b3d0ab9c Add redis timeout and reap interval options in production.js.example
Use default values, for documentation purposes
2013-01-29 13:01:17 +01:00
Sandro Santilli
8d4f033a56 Revert "getDatabase: properly handle redis connection failures"
This reverts commit dd19d74149.

The code was already correct
2013-01-28 17:39:50 +01:00
Sandro Santilli
dd19d74149 getDatabase: properly handle redis connection failures 2013-01-28 17:30:58 +01:00
Sandro Santilli
ac49abe750 Do not leak redis client connections on redis command error 2013-01-28 17:13:49 +01:00
Sandro Santilli
b130b67f24 Check redis connection at pool creation time 2013-01-28 17:12:21 +01:00
Sandro Santilli
093d3de66e We don't run tests with expresso anymore... 2012-12-21 17:32:00 +01:00
Luis Bosque
fea30dcea4 Merge branch 'release/staging'
Conflicts:
	NEWS.md
2012-12-21 14:10:15 +01:00
Sandro Santilli
6e8bfffff1 Merge branch 'master' into develop
Conflicts:
	Makefile
	NEWS.md
	package.json
2012-12-21 13:30:14 +01:00
Sandro Santilli
0ff403fa83 Require grainstore 0.10.9, fixing an issue with multi-geom markers 2012-12-21 13:24:38 +01:00
Sandro Santilli
ab1a0e0339 Add check-full to run testsuite of submodules
This model should be improved to avoid double-checking when the same
submodule could be installed or referenced by multiple levels...
2012-12-20 13:15:47 +01:00
Luis Bosque
290e005e14 Target v1.1.5 2012-12-20 12:28:43 +01:00
Sandro Santilli
d1979a5265 Fix bogus cached return of utf grid for fully contained tiles (#67) 2012-12-20 11:30:29 +01:00
Sandro Santilli
9db49a25e4 Require windshaft-0.8.1 for cache debugging facilities
Also upgrades node-mapnik to 0.7.18
2012-12-19 18:31:58 +01:00
Sandro Santilli
a69afe4cd7 Enhance run_tests.sh to allow running single tests and skipping preparation 2012-12-19 09:04:53 +01:00
David Arango
556aaf3330 Bump to 1.1.5 2012-12-13 11:45:14 +01:00
David Arango
7b1184db8f Bump to 1.1.5 2012-12-13 11:43:11 +01:00
David Arango
e1070d3998 Merge branch 'release/staging' 2012-12-10 17:26:48 +01:00
Sandro Santilli
cc91e0dcff Add a row limitof 65535 (256x256) in the example configs.
Closes #64
2012-12-07 20:34:24 +01:00
Sandro Santilli
081cc41a34 Drop extended styles. Closes #58. Add some documentation in header. 2012-12-07 20:21:13 +01:00
Sandro Santilli
9d52b8b11f Update NEWS 2012-12-07 20:06:26 +01:00
Sandro Santilli
a8a3e739ad Change interface of reset_style to read full config from env files
Closes #62
It is highly recommended to invoke reset_styles after upgrade
2012-12-07 20:04:31 +01:00
Sandro Santilli
b884fe00ea The parameter to simplify geometries is "simplify_geometries"
Closes #63
2012-12-07 19:50:57 +01:00
Sandro Santilli
4f8b855f1f Add reference to the list of accepted parameters for the postgres section 2012-12-07 18:58:31 +01:00
Sandro Santilli
6f5e3837e3 Use an env config parameter for socket timeout 2012-12-05 13:59:20 +01:00
Luis Bosque
de1df4f33c Merge branch 'release/v2.0' into develop
Conflicts:
	NEWS.md
2012-11-30 16:53:55 +01:00
Luis Bosque
7231864e1c Merge branch 'release/v2.0' 2012-11-30 16:52:55 +01:00
Luis Bosque
e813209768 Updated NEWS vor 1.1.3 2012-11-30 16:51:31 +01:00
Sandro Santilli
69508f05d7 Reduce default extent to allow for consistent proj4 round-tripping
See https://github.com/Vizzuality/grainstore/issues/42
2012-11-29 10:20:37 +01:00
Sandro Santilli
2f70640340 Require grainstore-0.10.8 to workaround bubble map bug
See https://github.com/Vizzuality/grainstore/issues/44
2012-11-28 20:17:02 +01:00
Luis Bosque
02d4473766 Target version 1.1.4 2012-11-28 16:43:19 +01:00
David Arango
72cf824235 Fix on convert_database_styles script 2012-11-28 15:34:05 +01:00
Sandro Santilli
a3d09339de Require grainstore-0.10.7 to enhance marker type transform (2.0 -> 2.1)
See https://github.com/Vizzuality/grainstore/blob/0.10.7/NEWS.md
2012-11-28 12:36:47 +01:00
Sandro Santilli
a0cd4354a7 Enlarge default map extent
See https://github.com/Vizzuality/grainstore/issues/42
2012-11-28 11:24:04 +01:00
Sandro Santilli
820c836050 Require grainstore-0.10.6 to enhance marker type transform (2.0 -> 2.1)
See https://github.com/Vizzuality/grainstore/issues/39
2012-11-28 10:06:43 +01:00
Sandro Santilli
bbdc29faae Set max_size=500 in the example configurations
TODO: add a ./configure switch to set it
2012-11-27 18:30:08 +01:00
Sandro Santilli
c976506c67 Require grainstore-0.10.5 to resize arrow markers ( 2.0 -> 2.1 ) 2012-11-27 17:31:53 +01:00
Sandro Santilli
11025eb7c4 Update with the change in reset_styles 2012-11-26 13:20:39 +01:00
Sandro Santilli
08c75843de Really skip extended keys 2012-11-26 12:51:55 +01:00
Sandro Santilli
c3169745ee Require grainstore ~0.10.4 for mapnik-2.1.1 support 2012-11-23 12:39:34 +01:00
Sandro Santilli
32007403c0 Fix testcase 2012-11-22 19:12:21 +01:00
Sandro Santilli
59a1911950 Add test for #57 (succeeds)
The ticket is actually invalid, there's no such bug here...
2012-11-22 16:39:00 +01:00
Sandro Santilli
2be0ebb808 Add --with-mapnik-version configure switch 2012-11-21 13:23:04 +01:00
Sandro Santilli
6ccb7f6f15 Shrinkwrap latest grainstore, to not be fooled by CartoCSS comments 2012-11-20 19:07:47 +01:00
David Arango
bb59524914 Adds script to convert single tables 2012-11-19 12:54:27 +01:00
Sandro Santilli
9645d0fb58 Require grainstore 0.10.1 to handle conditional markers in style transform 2012-11-15 16:12:55 +01:00
Sandro Santilli
635b5db3e6 Nicer indent in CartoCSS (less likely to be converted) 2012-11-15 16:01:34 +01:00
Sandro Santilli
19436a8b14 Let "style_convert" pass by, add tests for GET and POST with it 2012-11-14 15:28:58 +01:00
Sandro Santilli
1ebe862594 Add style_convert and style_version parameters in the docs 2012-11-14 13:45:11 +01:00
Sandro Santilli
2f0ef03cd3 Accept style_convert parameter to GET /styl request
Require grainstore-0.10 and windshaft-0.8 to allow for it
2012-11-14 13:41:46 +01:00
Luis Bosque
597008d9d4 Merge branch 'release/staging' 2012-11-14 12:48:07 +01:00
Sandro Santilli
f9b78e2cb2 Use grainstore 0.9.7 for mapnik version dependent default styles 2012-11-14 10:42:20 +01:00
Luis Bosque
33e68d9069 target v1.1.3 2012-11-12 12:28:32 +01:00
Sandro Santilli
b5658a9f43 Fix point markers shift due to polygon clipping (2.0 -> 2.1)
Done by requiring grainstore 0.9.6
2012-11-09 17:28:22 +01:00
Sandro Santilli
4b8c165261 Fix usage string for "show_style" tool 2012-11-08 18:58:09 +01:00
Sandro Santilli
cd6002879e Add tool to show formatted style stored in redis from user/table 2012-11-07 10:59:39 +01:00
Sandro Santilli
43bb96cc60 Require grainstore 0.9.5, fixing one type of style transform 2012-11-06 18:02:46 +01:00
Sandro Santilli
52303e7821 Fix use of "style_version" with GET (inline styles)
It took a lot of time to produce a testcase for this as the test
config was setting srid to 4326 but not changing geom column name
thus all tiles fetched by tests returned blank (ouch!)
2012-11-06 12:45:04 +01:00
Luis Bosque
37cba2836c timeout is in milseconds, not in seconds 2012-11-02 23:48:08 +01:00
Luis Bosque
8ef952f138 set 600 seconds timeout in cluster.js 2012-11-02 23:24:45 +01:00
Sandro Santilli
b5db3a8138 Require grainstore 0.9.4, for better 2.0 -> 2.1 css transforms 2012-11-02 17:53:25 +01:00
Luis Bosque
d1d321194d target 1.1.2 version 2012-11-02 15:47:54 +01:00
Luis Bosque
8120e6579a updated NEWS 2012-11-02 15:46:54 +01:00
Luis Bosque
f9e0b8708c target version 1.1.1 2012-10-30 18:53:49 +01:00
Luis Bosque
6401278317 Merge branch 'release/staging' into develop
Conflicts:
	NEWS.md
	npm-shrinkwrap.json
	package.json
	test/acceptance/server.js
2012-10-30 18:49:42 +01:00
Luis Bosque
1873322a90 Merge branch 'release/staging'
Conflicts:
	package.json
2012-10-30 18:45:47 +01:00
Luis Bosque
2d2a14e9a6 updated NEW for v1.1.0 2012-10-30 18:43:31 +01:00
Sandro Santilli
1da7484c2a Shrinkwrap newest windshaft and grainstore 2012-10-30 18:00:16 +01:00
Sandro Santilli
2bc09a61cf Add support for cache_policy=persistent
When cache_policy=persistent is given the response will contain
a Cache-Control header requesting for 1 year lifetime caching
2012-10-24 09:40:05 +02:00
Sandro Santilli
d9e6aeb254 Fix crash on unknown user. Closes #55. 2012-10-22 15:30:16 +02:00
Sandro Santilli
1a60c55fea Rename carto version parameters and add its support to GET tile
Also upgrades "carto" to 0.9.3 and "millstone" to 0.5.11
2012-10-19 12:54:11 +02:00
Sandro Santilli
ab8cb5bbb3 Add Windshaft-cartodb version to the /version route 2012-10-15 17:03:57 +02:00
Sandro Santilli
4c6d74b69e Use windshaft-0.6.2 sendError function to send non-200 responses
Ensures all errors are logged
2012-10-11 16:58:11 +02:00
Sandro Santilli
20dca2e8f8 Use windshaft-0.6.2 sendError function to send non-200 responses
Ensures all errors are logged
2012-10-11 16:48:41 +02:00
Sandro Santilli
90d726c0cb Fix test expectance after windshaft/grainstore upgrade
Now GET /style response includes CartoCSS version...
2012-10-11 11:23:35 +02:00
Sandro Santilli
00dccd4c27 Use 'host' configuration for HTTP listening (both app and cluster) 2012-10-11 11:23:33 +02:00
Sandro Santilli
d691f94978 Require grainstore 0.9.1 for automatic styles reset on mapnik upgrade 2012-10-11 11:23:32 +02:00
Sandro Santilli
11910bb218 Use "undefined" mapnik_version in the example configs
Using "undefined" for mapnik_version triggers autodetection,
which is more appropriate.
2012-10-11 11:23:31 +02:00
Sandro Santilli
f0c655294f Upgrade windshaft/grainstore to fix /version route 2012-10-11 11:23:30 +02:00
Sandro Santilli
8e3c900580 Print a warning when configured mapnik version doesn't match installed 2012-10-11 11:23:28 +02:00
Sandro Santilli
65e4cf1510 Batch-convert and mapnik version detection support in reset_styles 2012-10-11 11:23:26 +02:00
Sandro Santilli
41d7000daf Update windshaft to 0.6 exposing CartoCSS versioning support 2012-10-11 11:23:25 +02:00
Sandro Santilli
0e37b32e52 Updated 2012-10-11 11:23:24 +02:00
Sandro Santilli
9ad574efdc Autodetect target mapnik version and let config override it
Closes #40
2012-10-11 11:23:23 +02:00
Sandro Santilli
8d5c52ce1b Make test tolerant to additional fields in responses to POST style 2012-10-11 11:23:22 +02:00
Sandro Santilli
926b47b009 Fix test expectance after windshaft/grainstore upgrade
Now GET /style response includes CartoCSS version...
2012-10-09 18:35:27 +02:00
Sandro Santilli
bb00bf1c05 Use 'host' configuration for HTTP listening (both app and cluster) 2012-10-09 18:31:06 +02:00
Sandro Santilli
bc51b14485 Require grainstore 0.9.1 for automatic styles reset on mapnik upgrade 2012-10-09 14:39:38 +02:00
Sandro Santilli
9cdd2800fa Use "undefined" mapnik_version in the example configs
Using "undefined" for mapnik_version triggers autodetection,
which is more appropriate.
2012-10-09 12:12:33 +02:00
Sandro Santilli
0697873a64 Upgrade windshaft/grainstore to fix /version route 2012-10-09 11:53:23 +02:00
Sandro Santilli
6a1933bed9 Print a warning when configured mapnik version doesn't match installed 2012-10-09 11:45:57 +02:00
Sandro Santilli
703c63d5d6 Batch-convert and mapnik version detection support in reset_styles 2012-10-08 18:06:12 +02:00
Sandro Santilli
89ca4e1a5a Update windshaft to 0.6 exposing CartoCSS versioning support 2012-10-08 18:02:28 +02:00
Sandro Santilli
f8e88153f4 Updated 2012-10-08 17:45:52 +02:00
Sandro Santilli
961269fa1f Autodetect target mapnik version and let config override it
Closes #40
2012-10-08 17:45:03 +02:00
Sandro Santilli
90f6e464d2 Make test tolerant to additional fields in responses to POST style 2012-10-08 17:45:03 +02:00
Luis Bosque
a42f03c224 target 1.1.0 version 2012-10-08 12:50:50 +02:00
Luis Bosque
34df118160 fixed problem in cluster2 with pidfile name 2012-10-04 11:02:31 +02:00
Luis Bosque
b6f28d7dd2 version 1.0.0 in package.json 2012-10-03 16:56:50 +02:00
Luis Bosque
43298f58fb Merge branch 'develop0.8' 2012-10-03 16:42:03 +02:00
Luis Bosque
5d62c6b588 Merge branch 'release/staging' 2012-09-25 17:31:58 +02:00
Sandro Santilli
4216d43db2 Fix grainstore include after direct dependency drop 2008-06-17 05:25:38 +02:00
Sandro Santilli
f85ca16c62 Change LZMA expected encoding from HEX to base64, reducing its size 2013-04-19 16:16:20 +02:00
Sandro Santilli
14953e992f Multilayer API changes, target 1.2.0
- Layers passed by index in grid fetching url
 - Interactivity only specified in layergroup config
 - Encode cache_buster as part of the token
2013-04-15 18:51:28 +02:00
Fabio Rueda
0122c6a386 targeted 1.1.11 2013-04-15 13:05:16 +02:00
69 changed files with 10339 additions and 1143 deletions

6
.gitignore vendored
View File

@@ -1,3 +1,7 @@
node_modules
node_modules*
config.status*
config/environments/*.js
.idea
tools/munin/windshaft.conf
logs/
pids/

23
.travis.yml Normal file
View File

@@ -0,0 +1,23 @@
before_install:
- sudo apt-add-repository --yes ppa:mapnik/v2.1.0
- sudo apt-get update -q
- sudo apt-get install -q libmapnik-dev
- createdb template_postgis
- psql -c "CREATE EXTENSION postgis" template_postgis
# Tell npm to use known registrars:
# see http://blog.npmjs.org/post/78085451721/npms-self-signed-certificate-is-no-more
- npm config set ca ""
env:
- NPROCS=1 JOBS=1
language: node_js
node_js:
- "0.8"
- "0.10"
notifications:
irc:
channels:
- "irc.freenode.org#cartodb"
use_notice: true

13
HOWTO_RELEASE Normal file
View File

@@ -0,0 +1,13 @@
1. Ensure proper version in package.json
2. Ensure NEWS section exists for the new version, review it, add release date
3. Drop npm-shrinkwrap.json
4. Run npm install
5. Test (make check or npm test), fix if broken before proceeding
6. Run npm shrinkwrap
7. Set "from" in npm-shrinkwrap.json for known packages
(windshaft, node-varnish, grainstore...)
8. Commit package.json, npm-shrinwrap.json, NEWS
9. git tag -a Major.Minor.Patch # use NEWS section as content
10. Announce
11. Stub NEWS/package for next version

27
LICENCE
View File

@@ -1,27 +0,0 @@
Copyright (c) 2011, Vizzuality
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software
must display the following acknowledgement:
This product includes software developed by Vizzuality.
4. Neither the name of Vizzuality nor the
names of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

27
LICENSE Normal file
View File

@@ -0,0 +1,27 @@
Copyright (c) 2014, Vizzuality
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@@ -1,11 +1,35 @@
srcdir=$(shell pwd)
all:
npm install
clean:
rm -rf node_modules/*
config/environments/test.js: config/environments/test.js.example
./configure
distclean: clean
rm config.status*
config.status--test:
./configure --environment=test
config/environments/test.js: config.status--test
./config.status--test
check-local: config/environments/test.js
./run_tests.sh ${RUNTESTFLAGS} \
test/unit/cartodb/*.js \
test/acceptance/*.js
check-submodules:
PATH="$$PATH:$(srcdir)/node_modules/.bin/"; \
for sub in windshaft grainstore node-varnish mapnik; do \
if test -e node_modules/$${sub}; then \
echo "Testing submodule $${sub}"; \
make -C node_modules/$${sub} check || exit 1; \
fi; \
done
check-full: check-local check-submodules
check: check-local
check: config/environments/test.js
./run_tests.sh

412
NEWS.md
View File

@@ -1,11 +1,421 @@
1.1.0 (DD/MM/YY)
1.10.1 -- 2014-03-21
-------------------
Bug fixes:
- Do not cache non-success jsonp responses (#186)
1.10.0 -- 2014-03-20
-------------------
New features:
- Add optional support for rollbar (#150)
Enhancements:
- Do not send connection details to client (#183)
- Upgrade node-varnish to 0.3.0
- Upgrade Windshaft to 0.20.0, see
http://github.com/CartoDB/Windshaft/blob/0.20.0/NEWS
- Include tiler version in startup log
- Install an uncaught exception handler
- Require own fork of node-mapnik, with temptative fix
for libxml usage (glibc detected corruptions)
Other changes:
- Switch to 3-clause BSD license (#184)
1.9.0 -- 2014-03-10
-------------------
New features:
- Allow to set server related configuration in serverMetadata (#182)
1.8.5 -- 2014-03-10
-------------------
Enhancements:
- Set statsd prefix for all endpoints
- Respond with a permission denied on attempt to access map tiles waiving
signature of someone who had not left any (#170)
- Do not log an error on GET / (#177)
- Do not UNWATCH on every redis client release (#161)
- Include API docs (#164)
- Add "cacheDns" statsd setting in the example configs
- Do not send duplicated stats on template instanciation
- Do not die on dns resolution errors (#178, #180)
Bug fixes:
- Do not cache map creation responses (#176)
1.8.4 -- 2014-03-03
-------------------
Enhancements:
- Really skip CDB_TableMetadata lookup for sql affected by no tables (#169)
- Upgrade windshaft to 0.19.2, see node_modules/windshaft/NEWS
- Clarify obscure "ECONNREFUSED" error message (#171)
- Change some http status responses to be more appropriate to the case
- Forbid using map signatures of foreign users (#172)
- Forbid instanciating templates of foreign users (#173)
- Allow passing environment configuration name via NODE_ENV to app.js
- Print environment configuration name on app start
Bug fixes:
- Fix database connection settings on template instanciation (#174)
1.8.3 -- 2014-02-27
-------------------
Enhancements:
- Upgrades windshaft to 0.19.1 with many performance improvements,
See node_modules/windshaft/NEWS
- Improve speed of instanciating a map (#147, #159, #165)
- Give meaningful error on attempts to use map tokens
with attribute service (#156)
- Reduce sql-api communication timeout, and allow overriding (#167)
[ new sqlapi.timeout directive, defaults to 100 ms ]
- Do not query CDB_TableMetadata for queries affected by no tables (#168)
1.8.2 -- 2014-02-25
-------------------
Enhancements:
* Allow using ":host" as part of statsd.prefix (#153)
* Expand "addCacheChannel" stats
* Allow using GET with sql-api for queries shorter than configured len (#155)
[ new sqlapi.max_get_sql_length directive, defaults to 2048 ]
* Do not log an error for a legit request requiring no X-Cache-Channel
Bug fixes:
* Fix munin plugin after log format changes (#154)
1.8.1 -- 2014-02-19
-------------------
Enhancements:
* Use log4js logger (#138)
Bug fixes:
* Always generate X-Cache-Channel for token-based tile responses (#152)
1.8.0 -- 2014-02-18
-------------------
Enhancements:
* Add script to flush caches (#140)
* Add script to list templates
* Add statsd support (#139)
* Add support for specifying a varnish password
* Avoid sending multiple varnish invalidation at once (#135)
* Tested with node-0.10 (#141)
* Use single redis pooler for torque and grainstore
* Reduce cost of garbage collection for localized resources
* Allow limiting number of templates for each user (#136)
* Allow configuring TTL of mapConfigs via "mapConfigTTL"
1.7.1 -- 2014-02-11
-------------------
Enhancements:
* Disable debug logging unless "debug" config param evaluates to true (#137)
* Require windshaft 0.17.2 for further reducing log noise (#137)
1.7.0 -- 2014-02-11
-------------------
New features:
* Add support for torque tiles (#112)
* Add attributes service (#118)
* Implement Unified Map API (#126)
* Make endpoints configurable (#127)
Enhancements:
* Allow specifying fixed sqlapi host address (#117)
* Include template hash in template instance response, to keep caches
of different instances separated (#105)
Bug fixes:
* Allow space padding in template variables usage (#129)
* Allow passing numbers as values for numeric template variables (#130)
1.6.3 -- 2014-01-30
-------------------
Bug fixes:
* layergroup accept both map_key and api_key (#91)
* Fix public instanciation of signed template accessing private data (#114)
* Fix show_style in presence of complex styles
* Fix use of maxzoom in layergroup config (via windshaft-0.15.1)
Enhancements:
* Add support for instanciating a template map with JSONP (#116)
* Stop processing XML on renderer creation, not needed anymore since 1.6.1
introduced on-demand XML generation.
1.6.2 -- 2014-01-23
-------------------
Bug fixes:
* Fix support for long (>64k chars) queries in layergroup creation (#111)
Enhancements:
* Enhance tools/show_style to accept an environment parameter and
print XML style now it is not in redis anymore (#110)
* Support CORS in template instanciation endpoint (#113)
1.6.1 -- 2014-01-15
-------------------
Bug fixes:
* Drop cache headers from error responses (#107)
* Localize external CartoCSS resources at renderer creation time (#108)
1.6.0 -- 2014-01-10
-------------------
New features:
* Add 'user_from_host' directive to generalize username extraction (#100)
* Implement signed template maps (#98)
Other changes:
* Update cartodb-redis dependency to "~0.3.0"
* Update redis-server dependency to "2.4.0+"
1.5.2 -- 2013-12-05
-------------------
Bug fixes:
* Fix configuration-level compatibility with versions prior to 1.5 (#96)
* Fix use of old layergroups on mapnik upgrade (#97)
1.5.1 -- 2013-11-28
-------------------
Bug fixes:
* Survive presence of malformed CartoCSS in redis (#94)
* Accept useless point-transform:scale directives (#93)
1.5.0 -- 2013-11-19
-------------------
NOTE: new configuration directives `postgres_auth_pass` and
`postgres.password` added; see config/environments/*.example
for documentation.
Improvements:
* Add support for configuring database connection passwords
* Optionally read user-specific database_host and database_password
from redis as per CartoDB-2.5.0 model (#88, #89)
* Do not force ending dot in SQL-API hostname, for easier testing
Bug fixes:
* Return CORS headers when creating layergroups via GET (windshaft/#92)
* Fix http status on database authentication error (windshaft/#94)
* Fix text-face-name error at layergroup creation (windshaft/#93)
Other changes:
* CartoDB redis interaction delegated to "cartodb-redis" module
1.4.1 -- 2013-11-08
-------------------
* Fix support for exponential notation in CartoCSS filter values (#87)
1.4.0 -- 2013-10-31
-------------------
* Add Support for Mapnik-2.2.0 (#78)
1.3.6 -- 2013-10-11
-------------------
* Restore support for node-0.8.9 accidentally dropped by 1.3.5
NOTE: needs removing node_modules/windshaft and re-running npm install
1.3.5 -- 2013-10-03
-------------------
* Fixing apostrophes in CartoCSS
* Fix "sql/table must contain zoom variable" error when using
"[ zoom > 3]" CartoCSS snippets (note the space)
* Fix backward compatibility handling of sqlapi.host configuration (#82)
* Fix error for invalid text-name in CartoCSS (#81)
* Do not let anonymous requests use authorized renderer caches
1.3.4
------
NOTE: configuration sqlapi.host renamed to sqlapi.domain
(support for "sqlapi.host" is retained for backward compatibility)
* Improve empty CartoCSS error message
* Improve invalid mapnik-geometry-type CSS error message
* Fix race condition in localization of network resources
1.3.3
------
* Set Last-Modified header to allow for 304 responses
* Add profiling support (needs useProfiler in env config file)
* Fix double-checking for layergroups with no interactivity
* Log full layergroup config at creation time (#76)
1.3.2
------
* Set default layergroup TTL to 2 hours
* Serve multilayer tiles and grid with persistent cache control
1.3.1
------
* Fix deadlock on new style creation
* Fix database authentication with multi-table layergroups
* Add tile and grid fetching checks at layergroup creation time
* Fix SQL error reporting to NOT split on newline
* Fix support for CartoCSS attachments
1.3.0
------
* Change stats format for multilayer map token request, see
http://github.com/Vizzuality/Windshaft-cartodb/wiki/Redis-stats-format
1.2.1
------
* Fix multilayer post from firefox
* Fix multilayer cartocss layer name handling
1.2.0
------
* Multilayer API changes
* Layers passed by index in grid fetching url
* Interactivity only specified in layergroup config
* Embed cache_buster within token
* Use ISO format for last_modified timestamp
* Expected LZMA encoding changed to base64
1.1.10
------
* Fix regression with default interactivity parameter (#74)
* More verbose logging for SQL api connection errors
* Write stats for multilayer map token request
1.1.9
-----
* Handle SQL API errors by requesting no Varnish cache
* Fix X-Cache-Channel for multilayer (by token) responses
* Add last_modified field to layergroup creation response (#72)
* Deprecate signal handler for USR1, add handler for USR2 (#71)
* Fix support for ampersend characters in CartoCSS
* Add support for LZMA compressed GET parameters
* Add support for creating layergroups via GET
1.1.8
-----
* Require Windshaft-0.9.1, to reduce harmfulness of cache_buster param
1.1.7 (DD//MM//YY)
-----
* Do not let /etc/services confuse FD checker (munin plugin)
* Multilayer support (#72)
* Expose renderer settings in the environment config files
1.1.6 (19//02//13)
-----
* Require windshaft 0.8.5, fixing some stability issues
and providing cache info on request
* Require grainstore 0.10.9, fixing an issue with multi-geom markers
* Enhance run_tests.sh to allow running single tests and skipping preparation
* Fix async throws in getGeometryType, getInfoWindow and getMapMetadata
* Survive connection refusals from redis
* Add maxConnection environment configuration, default to 128
1.1.5 (DD//MM//YY)
-----
* Fix bogus cached return of utf grid for fully contained tiles (#67)
1.1.4 (DD//MM//YY)
-----
* Reduce default extent to allow for consistent proj4 round-tripping
* Enhance reset_styles script to use full configuration (#62)
* Have reset_styles script also drop extended keys (#58)
* Fix example postgis parameter for simplifying input geoms (#63)
* Add row_limit to example config (#64)
1.1.3 (30//11//12)
-----
* Fix reset_styles script to really skip extended keys
* CartoCSS versioning
* Mapnik-version dependent default styles
* Enhance 2.0 -> 2.1 transforms:
* styles with conditional markers
* scale arrow markers by 50%
1.1.2 (DD//MM//YY)
-----
* CartoCSS versioning
* Fix use of "style_version" with GET (inline styles)
* Enhance 2.0 -> 2.1 transforms:
* styles with no semicolon
* markers shift due to geometry clipping
1.1.1 (DD//MM//YY)
-----
* Add support for persistent client cache headers
* Fix crash on unknown user (#55)
* Add /version entry point
* CartoCSS versioning
* Include style_version in GET /style response
* Support style_version and style_convert parameters in POST /style request
* Support style_version in GET /:z/:x/:y request
1.1.0 (30/10/12)
=======
* Add /version entry point
* CartoCSS versioning
* Include version in GET /style response
* Support version and convert parameters in POST /style request
* Autodetect target mapnik version and let config override it
* Add tools/reset_styles script to batch-reset (and optionally convert) styles
* Configurable logging format (#4)
* Detailed error on missing user metadata
* Properly handle unauthenticated requests for metadata
* Accept "api_key" in addition to "map_key",
both in query_string and POST body (#38)
* Add ./configure script
* Allow listening on host IP
* Replaced environment configs by .example ones
* Fixed some issues with cluster2
1.0.0 (03/10/12)
-----

View File

@@ -1,7 +1,8 @@
Windshaft-CartoDB
==================
NOTE: requires node-0.8.x
[![Build Status](http://travis-ci.org/CartoDB/Windshaft-cartodb.png)]
(http://travis-ci.org/CartoDB/Windshaft-cartodb)
This is the CartoDB map tiler. It extends Windshaft with some extra
functionality and custom filters for authentication
@@ -11,23 +12,28 @@ functionality and custom filters for authentication
* gets the default geometry type from the cartodb redis store
* allows tiles to be styled individually
* provides a link to varnish high speed cache
* provides a infowindow endpoint for windshaft
* provides a ``map_metadata`` endpoint for windshaft
* provides a ``infowindow`` endpoint for windshaft (DEPRECATED)
* provides a ``map_metadata`` endpoint for windshaft (DEPRECATED)
* provides signed template maps API
(http://github.com/CartoDB/Windshaft-cartodb/wiki/Template-maps)
Requirements
------------
[core]
- node-0.6.x+
- node-0.8.x+
- PostgreSQL-8.3+
- PostGIS-1.5.0+
- Redis 2.2.0+ (http://www.redis.io)
- Redis 2.4.0+ (http://www.redis.io)
- Mapnik 2.0 or 2.1
[for cache control]
- CartoDB-SQL-API 1.0.0+
- CartoDB 0.9.5+ (for ``CDB_QueryTables``)
- Varnish (https://www.varnish-cache.org)
- Varnish (http://www.varnish-cache.org)
[for running the testsuite]
- Imagemagick (http://www.imagemagick.org)
Configure
---------
@@ -80,8 +86,16 @@ Args:
* sql - plain SQL arguments
* interactivity - specify the column to use in UTFGrid
* cache_buster - if needed you can add a cachebuster to make sure you're
rendering new
* cache_buster - Specify an identifier for the internal tile cache.
Requesting tiles with the same cache_buster value may
result in being served a cached version of the tile
(even when requesting a tile for the first time, as tiles
can be prepared in advance)
* cache_policy - Set to "persist" to have the server send an Cache-Control
header requesting caching devices to keep the response
cached as much as possible. This is best used with a
timestamp value in cache_buster for manual control of
updates.
* geom_type - override the cartodb default
* style - override the default map style with Carto
@@ -93,6 +107,8 @@ Args:
Args:
* style - the style in CartoCSS you want to set
* style_version - the version of the style for POST
* style_convert - request conversion to target version (both POST and GET)
**INFOWINDOW**

72
app.js
View File

@@ -8,30 +8,78 @@
*/
if ( process.argv[2] ) ENV = process.argv[2];
else if ( process.env['NODE_ENV'] ) ENV = process.env['NODE_ENV'];
else ENV = 'development';
process.env['NODE_ENV'] = ENV;
// sanity check
var ENV = process.argv[2]
if (ENV != 'development' && ENV != 'production'){
if (ENV != 'development' && ENV != 'production' && ENV != 'staging' ){
console.error("\nnode app.js [environment]");
console.error("environments: [development, production]\n");
console.error("environments: development, production, staging\n");
process.exit(1);
}
var _ = require('underscore')
, Step = require('step')
, CartodbWindshaft = require('./lib/cartodb/cartodb_windshaft');
;
// set environment specific variables
global.settings = require(__dirname + '/config/settings');
global.environment = require(__dirname + '/config/environments/' + ENV);
_.extend(global.settings, global.environment);
// Include cart_data.js only _after_ the "global" variable is set
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/28
var cartoData = require('./lib/cartodb/carto_data');
global.log4js = require('log4js')
log4js_config = {
appenders: [
{ type: "console", layout: { type:'basic' } }
],
replaceConsole:true
};
var Windshaft = require('windshaft');
var serverOptions = require('./lib/cartodb/server_options');
if ( global.environment.rollbar ) {
log4js_config.appenders.push({
type: __dirname + "/lib/cartodb/log4js_rollbar.js",
options: global.environment.rollbar
});
}
log4js.configure(log4js_config);
global.logger = log4js.getLogger();
// Include cartodb_windshaft only _after_ the "global" variable is set
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/28
var CartodbWindshaft = require('./lib/cartodb/cartodb_windshaft');
var serverOptions = require('./lib/cartodb/server_options')();
ws = CartodbWindshaft(serverOptions);
ws.listen(global.environment.port);
console.log("Windshaft tileserver started on port " + global.environment.port);
// Maximum number of connections for one process
// 128 is a good number if you have up to 1024 filedescriptors
// 4 is good if you have max 32 filedescriptors
// 1 is good if you have max 16 filedescriptors
ws.maxConnections = global.environment.maxConnections || 128;
ws.listen(global.environment.port, global.environment.host);
var version = require("./package").version;
ws.on('listening', function() {
console.log("Windshaft tileserver " + version + " started on "
+ global.environment.host + ':' + global.environment.port
+ " (" + ENV + ")");
});
// DEPRECATED, use SIGUSR2
process.on('SIGUSR1', function() {
console.log('WARNING: handling of SIGUSR1 by Windshaft-CartoDB is deprecated, please send SIGUSR2 instead');
ws.dumpCacheStats();
});
process.on('SIGUSR2', function() {
ws.dumpCacheStats();
});
process.on('uncaughtException', function(err) {
logger.error('Uncaught exception: ' + err.stack);
});

View File

@@ -1,52 +0,0 @@
/*
* Windshaft-CartoDB
* ===============
*
* ./app.js [environment]
*
* environments: [development, production]
*/
var Cluster = require('cluster2');
// sanity check
var ENV = process.argv[2]
if (ENV != 'development' && ENV != 'production' && ENV != 'staging'){
console.error("\nnode app.js [environment]");
console.error("environments: [development, production, staging]\n");
process.exit(1);
}
var _ = require('underscore')
, Step = require('step')
, CartodbWindshaft = require('./lib/cartodb/cartodb_windshaft');
// set environment specific variables
global.settings = require(__dirname + '/config/settings');
global.environment = require(__dirname + '/config/environments/' + ENV);
_.extend(global.settings, global.environment);
// Include cart_data.js only _after_ the "global" variable is set
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/28
var cartoData = require('./lib/cartodb/carto_data');
var Windshaft = require('windshaft');
var serverOptions = require('./lib/cartodb/server_options');
var ws = CartodbWindshaft(serverOptions);
//.use(cluster.logger('logs'))
//.use(cluster.stats())
//.use(cluster.pidfiles('pids'))
var cluster = new Cluster({
port: global.environment.port,
monPort: global.environment.port+1,
noWorkers: 1 // .set('workers', 1)
});
cluster.listen(function(cb) {
cb(ws);
});
console.log("Windshaft tileserver started on port " + global.environment.port);

View File

@@ -2,43 +2,126 @@ var config = {
environment: 'development'
,port: 8181
,host: '127.0.0.1'
// Regular expression pattern to extract username
// from hostname. Must have a single grabbing block.
,user_from_host: '^(.*)\\.localhost'
// Base URLs for the APIs
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "maps" is the the new API,
// "tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/tiles/layergroup)'
// Base url for the Inline Maps and Table Maps API
,base_url_legacy: '/tiles/:table'
// Maximum number of connections for one process
// 128 is a good value with a limit of 1024 open file descriptors
,maxConnections:128
// Maximum number of templates per user. Unlimited by default.
,maxUserTemplates:1024
// Seconds since "last creation" before a detached
// or template instance map expires. Or: how long do you want
// to be able to navigate the map without a reload ?
// Defaults to 7200 (2 hours)
,mapConfigTTL: 7200
// idle socket timeout, in miliseconds
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: false
,log_format: '[:date] :req[X-Real-IP] \033[90m:method\033[0m \033[36m:req[Host]:url\033[0m \033[90m:status :response-time ms -> :res[Content-Type]\033[0m'
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler])'
// Templated database username for authorized user
// Supported labels: 'user_id' (read from redis)
,postgres_auth_user: 'development_cartodb_user_<%= user_id %>'
// Templated database password for authorized user
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: '<%= user_password %>'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
type: "postgis",
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
extent: "-20005048.4188,-20005048.4188,20005048.4188,20005048.4188",
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
/* experimental
geometry_field: "the_geom",
extent: "-180,-90,180,90",
srid: 4326,
*/
simplify: true
row_limit: 65535,
simplify_geometries: true,
max_size: 500
}
,mapnik_version: undefined
,statsd: {
host: 'localhost',
port: 8125,
prefix: 'dev.',
cacheDns: true
// support all allowed node-statsd options
}
,renderer: {
// Milliseconds since last access before renderer cache item expires
cache_ttl: 60000,
metatile: 4,
bufferSize: 64
}
,millstone: {
// Needs to be writable by server user
cache_basedir: '/tmp/cdb-tiler-dev/millstone-dev'
}
,redis: {
host: '127.0.0.1',
port: 6379,
idleTimeoutMillis: 1,
reapIntervalMillis: 1
// Max number of connections in each pool.
// Users will be put on a queue when the limit is hit.
// Set to maxConnection to have no possible queues.
// There are currently 3 pools involved in serving
// windshaft-cartodb requests so multiply this number
// by 3 to know how many possible connections will be
// kept open by the server. The default is 50.
max: 50,
idleTimeoutMillis: 1, // idle time before dropping connection
reapIntervalMillis: 1 // time between cleanups
}
,sqlapi: {
protocol: 'http',
host: 'localhost.lan',
// If "host" is given, it will be used
// to connect to the SQL-API without a
// DNS lookup
host: '127.0.0.1',
port: 8080,
version: 'v1'
// The "domain" part will be appended to
// the cartodb username and passed to
// SQL-API requests in the Host HTTP header
domain: 'localhost.lan',
version: 'v1',
// Maximum lenght of SQL query for GET
// requests. Longer queries will be sent
// using POST. Defaults to 2048
max_get_sql_length: 2048,
// Maximum time to wait for a response,
// in milliseconds. Defaults to 100.
timeout: 100
}
,varnish: {
host: 'localhost',
port: 6082,
secret: 'xxx',
ttl: 86400
}
// If useProfiler is true every response will be served with an
// X-Tiler-Profile header containing elapsed timing for various
// steps taken for producing the response.
,useProfiler:true
};
module.exports = config;

View File

@@ -2,35 +2,135 @@ var config = {
environment: 'production'
,port: 8181
,host: '127.0.0.1'
// Regular expression pattern to extract username
// from hostname. Must have a single grabbing block.
,user_from_host: '^(.*)\\.cartodb\\.com$'
// Base URLs for the APIs
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "maps" is the the new API,
// "tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/tiles/layergroup)'
// Base url for the Inline Maps and Table Maps API
,base_url_legacy: '/tiles/:table'
// Maximum number of connections for one process
// 128 is a good value with a limit of 1024 open file descriptors
,maxConnections:128
// Maximum number of templates per user. Unlimited by default.
,maxUserTemplates:1024
// Seconds since "last creation" before a detached
// or template instance map expires. Or: how long do you want
// to be able to navigate the map without a reload ?
// Defaults to 7200 (2 hours)
,mapConfigTTL: 7200
// idle socket timeout, in miliseconds
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: true
,log_format: '[:date] :req[X-Real-IP] \033[90m:method\033[0m \033[36m:req[Host]:url\033[0m \033[90m:status :response-time ms -> :res[Content-Type]\033[0m'
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler])'
// Templated database username for authorized user
// Supported labels: 'user_id' (read from redis)
,postgres_auth_user: 'cartodb_user_<%= user_id %>'
// Templated database password for authorized user
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: '<%= user_password %>'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 6432,
extent: "-20005048.4188,-20005048.4188,20005048.4188,20005048.4188",
simplify: true
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
row_limit: 65535,
simplify_geometries: true,
max_size: 500
}
,mapnik_version: undefined
,statsd: {
host: 'localhost',
port: 8125,
prefix: ':host.', // could be hostname, better not containing dots
cacheDns: true
// support all allowed node-statsd options
}
,renderer: {
// Milliseconds since last access before renderer cache item expires
cache_ttl: 60000,
metatile: 4,
bufferSize: 64
}
,millstone: {
// Needs to be writable by server user
cache_basedir: '/home/ubuntu/tile_assets/'
}
,redis: {
host: '127.0.0.1',
port: 6379
port: 6379,
// Max number of connections in each pool.
// Users will be put on a queue when the limit is hit.
// Set to maxConnection to have no possible queues.
// There are currently 3 pools involved in serving
// windshaft-cartodb requests so multiply this number
// by 3 to know how many possible connections will be
// kept open by the server. The default is 50.
max: 50,
idleTimeoutMillis: 30000, // idle time before dropping connection
reapIntervalMillis: 1000 // time between cleanups
}
,sqlapi: {
protocol: 'https',
host: 'cartodb.com',
// If "host" is given, it will be used
// to connect to the SQL-API without a
// DNS lookup
//host: '127.0.0.1',
port: 8080,
version: 'v2'
// The "domain" part will be appended to
// the cartodb username and passed to
// SQL-API requests in the Host HTTP header
domain: 'cartodb.com',
version: 'v2',
// Maximum lenght of SQL query for GET
// requests. Longer queries will be sent
// using POST. Defaults to 2048
max_get_sql_length: 2048,
// Maximum time to wait for a response,
// in milliseconds. Defaults to 100.
timeout: 100
}
,varnish: {
host: 'localhost',
port: 6082,
secret: 'xxx',
ttl: 86400
}
// If useProfiler is true every response will be served with an
// X-Tiler-Profile header containing elapsed timing for various
// steps taken for producing the response.
,useProfiler:false
,serverMetadata: {
cdn_url: {
http: 'api.cartocdn.com',
https: 'cartocdn.global.ssl.fastly.net'
}
}
// Optional rollbar support
,rollbar: {
token: 'secret',
// See http://github.com/rollbar/node_rollbar#configuration-reference
options: {
endpoint: 'https://api.rollbar.com/api/1/',
handler: 'inline'
}
}
};
module.exports = config;

View File

@@ -2,35 +2,135 @@ var config = {
environment: 'production'
,port: 8181
,host: '127.0.0.1'
// Regular expression pattern to extract username
// from hostname. Must have a single grabbing block.
,user_from_host: '^(.*)\\.cartodb\\.com$'
// Base URLs for the APIs
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/maps/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/maps/named|/tiles/template)'
// Base url for the Detached Maps API
// "/api/v1/maps" is the the new API,
// "/tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/maps|/tiles/layergroup)'
// Base url for the Inline Maps and Table Maps API
,base_url_legacy: '/tiles/:table'
// Maximum number of connections for one process
// 128 is a good value with a limit of 1024 open file descriptors
,maxConnections:128
// Maximum number of templates per user. Unlimited by default.
,maxUserTemplates:1024
// Seconds since "last creation" before a detached
// or template instance map expires. Or: how long do you want
// to be able to navigate the map without a reload ?
// Defaults to 7200 (2 hours)
,mapConfigTTL: 7200
// idle socket timeout, in miliseconds
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: true
,log_format: '[:date] :req[X-Real-IP] \033[90m:method\033[0m \033[36m:req[Host]:url\033[0m \033[90m:status :response-time ms -> :res[Content-Type]\033[0m'
,log_format: ':req[X-Real-IP] :method :req[Host]:url :status :response-time ms (:res[X-Tiler-Profiler]) -> :res[Content-Type]'
// Templated database username for authorized user
// Supported labels: 'user_id' (read from redis)
,postgres_auth_user: 'cartodb_staging_user_<%= user_id %>'
// Templated database password for authorized user
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: '<%= user_password %>'
,postgres: {
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "publicuser",
password: "public",
host: '127.0.0.1',
port: 6432,
extent: "-20005048.4188,-20005048.4188,20005048.4188,20005048.4188",
simplify: true
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
row_limit: 65535,
simplify_geometries: true,
max_size: 500
}
,mapnik_version: undefined
,statsd: {
host: 'localhost',
port: 8125,
prefix: 'stage.:host.',
cacheDns: true
// support all allowed node-statsd options
}
,renderer: {
// Milliseconds since last access before renderer cache item expires
cache_ttl: 60000,
metatile: 4,
bufferSize: 64
}
,millstone: {
// Needs to be writable by server user
cache_basedir: '/home/ubuntu/tile_assets/'
}
,redis: {
host: '127.0.0.1',
port: 6379
port: 6379,
// Max number of connections in each pool.
// Users will be put on a queue when the limit is hit.
// Set to maxConnection to have no possible queues.
// There are currently 3 pools involved in serving
// windshaft-cartodb requests so multiply this number
// by 3 to know how many possible connections will be
// kept open by the server. The default is 50.
max: 50,
idleTimeoutMillis: 30000, // idle time before dropping connection
reapIntervalMillis: 1000 // time between cleanups
}
,sqlapi: {
protocol: 'https',
host: 'cartodb.com',
// If "host" is given, it will be used
// to connect to the SQL-API without a
// DNS lookup
//host: '127.0.0.1',
port: 8080,
version: 'v2'
// The "domain" part will be appended to
// the cartodb username and passed to
// SQL-API requests in the Host HTTP header
domain: 'cartodb.com',
version: 'v2',
// Maximum lenght of SQL query for GET
// requests. Longer queries will be sent
// using POST. Defaults to 2048
max_get_sql_length: 2048,
// Maximum time to wait for a response,
// in milliseconds. Defaults to 100.
timeout: 100
}
,varnish: {
host: 'localhost',
port: 6082,
secret: 'xxx',
ttl: 86400
}
// If useProfiler is true every response will be served with an
// X-Tiler-Profile header containing elapsed timing for various
// steps taken for producing the response.
,useProfiler:true
,serverMetadata: {
cdn_url: {
http: 'api.cartocdn.com',
https: 'cartocdn.global.ssl.fastly.net'
}
}
// Optional rollbar support
,rollbar: {
token: 'secret',
// See http://github.com/rollbar/node_rollbar#configuration-reference
options: {
endpoint: 'https://api.rollbar.com/api/1/',
handler: 'inline'
}
}
};
module.exports = config;

View File

@@ -2,38 +2,122 @@ var config = {
environment: 'test'
,port: 8888
,host: '127.0.0.1'
// Regular expression pattern to extract username
// from hostname. Must have a single grabbing block.
,user_from_host: '(.*)'
// Base URLs for the APIs
//
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Unified-Map-API
//
// Base url for the Templated Maps API
// "/api/v1/map/named" is the new API,
// "/tiles/template" is for compatibility with versions up to 1.6.x
,base_url_templated: '(?:/api/v1/map/named|/tiles/template)'
// Base url for the Detached Maps API
// "maps" is the the new API,
// "tiles/layergroup" is for compatibility with versions up to 1.6.x
,base_url_detached: '(?:/api/v1/map|/tiles/layergroup)'
// Base url for the Inline Maps and Table Maps API
,base_url_legacy: '/tiles/:table'
// Maximum number of connections for one process
// 128 is a good value with a limit of 1024 open file descriptors
,maxConnections:128
// Maximum number of templates per user. Unlimited by default.
,maxUserTemplates:1024
// Seconds since "last creation" before a detached
// or template instance map expires. Or: how long do you want
// to be able to navigate the map without a reload ?
// Defaults to 7200 (2 hours)
,mapConfigTTL: 7200
// idle socket timeout, in miliseconds
,socket_timeout: 600000
,enable_cors: true
,cache_enabled: false
,log_format: '[:date] :req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type]'
,postgres_auth_user: 'test_cartodb_user_<%= user_id %>'
,log_format: '[:date] :req[X-Real-IP] :method :req[Host]:url :status :response-time ms -> :res[Content-Type] (:res[X-Tiler-Profiler])'
// Templated database username for authorized user
// Supported labels: 'user_id' (read from redis)
,postgres_auth_user: 'test_windshaft_cartodb_user_<%= user_id %>'
// Templated database password for authorized user
// Supported labels: 'user_id', 'user_password' (both read from redis)
,postgres_auth_pass: 'test_windshaft_cartodb_user_<%= user_id %>_pass'
,postgres: {
user: "publicuser",
// Parameters to pass to datasource plugin of mapnik
// See http://github.com/mapnik/mapnik/wiki/PostGIS
user: "testpublicuser",
password: "public",
host: '127.0.0.1',
port: 5432,
srid: 4326,
extent: "-20005048.4188,-20005048.4188,20005048.4188,20005048.4188",
simplify: true
extent: "-20037508.3,-20037508.3,20037508.3,20037508.3",
row_limit: 65535,
simplify_geometries: true,
max_size: 500
}
,mapnik_version: ''
,statsd: {
host: 'localhost',
port: 8125,
prefix: 'test.:host.',
cacheDns: true
// support all allowed node-statsd options
}
,renderer: {
// Milliseconds since last access before renderer cache item expires
cache_ttl: 60000,
metatile: 4,
bufferSize: 64
}
,millstone: {
// Needs to be writable by server user
cache_basedir: '/tmp/cdb-tiler-test/millstone'
}
,redis: {
host: '127.0.0.1',
port: 6333,
idleTimeoutMillis: 1,
reapIntervalMillis: 1
port: 6335,
// Max number of connections in each pool.
// Users will be put on a queue when the limit is hit.
// Set to maxConnection to have no possible queues.
// There are currently 3 pools involved in serving
// windshaft-cartodb requests so multiply this number
// by 3 to know how many possible connections will be
// kept open by the server. The default is 50.
max: 50,
idleTimeoutMillis: 1, // idle time before dropping connection
reapIntervalMillis: 1 // time between cleanups
}
,sqlapi: {
protocol: 'http',
host: 'localhost.lan',
port: 8080,
version: 'v1'
// If "host" is given, it will be used
// to connect to the SQL-API without a
// DNS lookup
host: '127.0.0.1',
port: 1080,
// The "domain" part will be appended to
// the cartodb username and passed to
// SQL-API requests in the Host HTTP header
domain: 'donot_look_this_up',
// This port will be used by "make check" for testing purposes
// It must be available
version: 'v1',
// Maximum lenght of SQL query for GET
// requests. Longer queries will be sent
// using POST. Defaults to 2048
max_get_sql_length: 2048,
// Maximum time to wait for a response,
// in milliseconds. Defaults to 100.
timeout: 100
}
,varnish: {
host: '',
port: null,
secret: 'xxx',
ttl: 86400
}
// If useProfiler is true every response will be served with an
// X-Tiler-Profile header containing elapsed timing for various
// steps taken for producing the response.
,useProfiler:true
};
module.exports = config;

61
configure vendored
View File

@@ -17,16 +17,26 @@
# --strk(2012-07-23)
#
ENVDIR=config/environments
PGPORT=
SQLAPI_PORT=
MAPNIK_VERSION=
ENVIRONMENT=development
STATUS="$0 $*"
usage() {
echo "Usage: $0 [OPTION]"
echo
echo "Configuration:"
echo " --help display this help and exit"
echo " --with-pgport=NUM access PostgreSQL server on TCP port NUM"
echo " --help display this help and exit"
echo " --with-pgport=NUM access PostgreSQL server on TCP port NUM [$PGPORT]"
echo " --with-sqlapi-port=NUM access SQL-API server on TCP port NUM [$SQLAPI_PORT]"
echo " --with-mapnik-version=STRING set mapnik version string [$MAPNIK_VERSION]"
echo " --environment=STRING set output environment name [$ENVIRONMENT]"
}
PGPORT=5432
while test -n "$1"; do
case "$1" in
--help|-h)
@@ -36,6 +46,15 @@ while test -n "$1"; do
--with-pgport=*)
PGPORT=`echo "$1" | cut -d= -f2`
;;
--with-sqlapi-port=*)
SQLAPI_PORT=`echo "$1" | cut -d= -f2`
;;
--with-mapnik-version=*)
MAPNIK_VERSION=`echo "$1" | cut -d= -f2`
;;
--environment=*)
ENVIRONMENT=`echo "$1" | cut -d= -f2`
;;
*)
echo "Unknown option '$1'" >&2
usage >&2
@@ -44,12 +63,30 @@ while test -n "$1"; do
shift
done
echo "PGPORT: $PGPORT"
ENVEX=./${ENVDIR}/${ENVIRONMENT}.js.example
if [ -z "$PGPORT" ]; then
PGPORT=`node -e "console.log(require('${ENVEX}').postgres.port)"`
fi
if [ -z "$SQLAPI_PORT" ]; then
SQLAPI_PORT=`node -e "console.log(require('${ENVEX}').sqlapi.port)"`
fi
echo "PGPORT: $PGPORT"
echo "SQLAPI_PORT: $SQLAPI_PORT"
echo "MAPNIK_VERSION: $MAPNIK_VERSION"
echo "ENVIRONMENT: $ENVIRONMENT"
o=`dirname "${ENVEX}"`/`basename "${ENVEX}" .example`
echo "Writing $o"
# See http://austinmatzko.com/2008/04/26/sed-multi-line-search-and-replace/
sed -n "1h;1!H;\${;g;s/\(,postgres: {[^}]*port: *'\?\)[^',]*\('\?,\)/\1$PGPORT\2/;p;}" < "${ENVEX}" \
| sed "s/mapnik_version:.*/mapnik_version: '$MAPNIK_VERSION'/" \
| sed -n "1h;1!H;\${;g;s/\(,sqlapi: {[^}]*port: *'\?\)[^',]*\('\?,\)/\1$SQLAPI_PORT\2/;p;}" \
> "$o"
STATUSFILE=config.status--${ENVIRONMENT}
echo "Writing ${STATUSFILE}"
echo ${STATUS} > ${STATUSFILE} && chmod +x ${STATUSFILE}
# TODO: allow specifying configuration settings !
for f in config/environments/*.example; do
o=`dirname "$f"`/`basename "$f" .example`
echo "Writing $o"
# See http://austinmatzko.com/2008/04/26/sed-multi-line-search-and-replace/
sed -n "1h;1!H;\${;g;s/\(,postgres: {[^}]*port: *'\?\)[^',]*\('\?,\)/\1$PGPORT\2/;p;}" < "$f" > "$o"
done

111
docs/Map-API.md Normal file
View File

@@ -0,0 +1,111 @@
# Kind of maps
Windshaft-CartoDB supports these kind of maps:
- [Temporary maps](#temporary-maps) (created by anyone)
- [Detached maps](#detached-maps)
- [Inline maps](#inline-maps) (legacy)
- [Persistent maps](#peristent-maps) (created by CartDB user)
- [Template maps](#template-maps)
- [Table maps](#table-maps) (legacy, deprecated)
## Temporary maps
Temporary maps have no owners and are anonymous in nature.
There are two kind of temporary maps:
- Detached maps (aka MultiLayer-API)
- Inline maps
### Detached maps
Detached maps are maps which are configured with a request
obtaining a temporary token and then used by referencing
the obtained token. The token expires automatically when unused.
Anyone can create detached maps, but users will need read access
to the data source of the map layers.
The configuration format is a [MapConfig]
(http://github.com/CartoDB/Windshaft/wiki/MapConfig-specification) document.
The HTTP endpoints for creating the map and using it are described [here]
(http://github.com/CartoDB/Windshaft-cartodb/wiki/MultiLayer-API)
*TODO* cleanup the referenced document
### Inline maps
Inline maps are maps that only exist for a single request,
being the request for a specific map resource (tile).
Inline maps are always bound to a table, and can only be
obtained by those having read access to the that table.
Additionally, users need to have access to any datasource
specified as part of the configuration.
Inline maps only support PNG and UTF8GRID tiles.
The configuration consist in a set of parameters, to be
specified in the query string of the tile request:
* sql - the query to run as datasource, can be an array
* style - the CartoCSS style for the datasource, can be an array
* style_version - version of the CartoCSS style, can be an array
* interactivity - only for fetching UTF8GRID,
If the style is not provided, style of the associated table is
used; if the sql is not provided, all records of the associated
table are used as the datasource; the two possibilities result
in a mix between _inline_ maps and [Table maps][].
*TODO* specify (or link) api endpoints
## Persistent maps
Persistent maps can only be created by a CartoDB user who has full
responsibility over editing and deleting them. There are two
kind of persistent maps:
- Template maps
- Table maps (legacy, deprecated)
### Templated maps
Templated maps are templated [MapConfig]
(http://github.com/CartoDB/Windshaft/wiki/MapConfig-specification) documents
associated with an authorization certificate.
The authorization certificate determines who can instanciate the
template and use the resulting map. Authorized users of the instanciated
maps will have the same database access privilege of the template owner.
The HTTP endpoints for creating and using templated maps are described [here]
(http://github.com/CartoDB/Windshaft-cartodb/wiki/Template-maps).
*TODO* cleanup the referenced document
### Table maps
Table maps are maps associated with a table.
Configuration of such maps is limited to the CartoCSS style.
* style - the CartoCSS style for the datasource, can be an array
* style_version - version of the CartoCSS style, can be an array
You can only fetch PNG or UTF8GRID tiles from these maps.
Access method is the same as the one for [Inline maps](#inline-maps)
# Endpoints description
- **/api/maps/** (same interface than https://github.com/CartoDB/Windshaft/wiki/Multilayer-API)
- **/api/maps/named** (same interface than https://github.com/CartoDB/Windshaft-cartodb/wiki/Template-maps)
NOTE: in case Multilayer-API does not contain this info yet, the
endpoint for fetching attributes is this:
- **/api/maps/:map_id/:layer_index/attributes/:feature_id**
- would return { c: 1, d: 2 }

28
docs/MultiLayer-API.md Normal file
View File

@@ -0,0 +1,28 @@
The Windshaft-CartoDB MultiLayer API extends the [Windshaft MultiLayer API](https://github.com/Vizzuality/Windshaft/wiki/Multilayer-API) in a few ways.
## Last modification timestamp embedded in the token
It encodes a timestamp of 'last modification time' into the map token (token:EPOCH) returned to the client.
It accepts tokens with encoded timestamp from the client considering the token suffix as a cache_buster value.
Clients don't need to be aware of the extension but rather use the API as they would use the base one.
The only difference will be that the _same_ layergroup configuration may result in different tokens if source data was modified between the mapview requests.
## Additional attributes in the response object
Windshaft-CartoDB adds the following attributes in the response object
- ``last_update`` field with ISO format (2013-11-30T12:23:10).
- ``cdn_url`` object containing CDN url client should use (not mandatory) to access the tiles. It's in the form:
```json
{
http: 'http://cdn_url.com/'
https: 'https://secure.cdn_url.com/'
}
```
## Stats tag
Windshaft-CartoDB adds support for a ``stat_tag`` element in the multilayer configuration to help [stats](Redis-stats-format) gathering.

292
docs/Template-maps.md Normal file
View File

@@ -0,0 +1,292 @@
Template maps are layergroup configurations that rather than being
fully defined contain variables that can be set to produce a different
layergroup configurations (instantiation).
Template maps are persistent, can only be created and deleted by the
CartoDB user showing a valid API_KEY.
Instantiating a signed template map would result in a [signed
map](https://github.com/CartoDB/Windshaft-cartodb/wiki/Signed-maps)
instance that would be signed with the same signature as the template.
Deleting a signed template results in deletion of all signatures created
as a result of instantiation.
# Template format
A templated layergroup would allow using placeholders
in the "cartocss" and "sql" elements in the "option"
field of any "layer" of a layergroup configuration
(see https://github.com/CartoDB/Windshaft/wiki/MapConfig-specification).
Valid placeholder names start with a letter and can only
contain letters, numbers or underscores. They have to be
written between ``<%= `` and `` %>`` strings in order to be
replaced. Example: ``<%= my_color %>``.
The set of supported placeholders for a template will need to be
explicitly defined specifying type and default value for each.
**placeholder types**
Placeholder type will determine the kind of escaping for the
associated value. Supported types are:
* sql_literal (internal single-quotes will be sql-escaped)
* sql_ident (internal double-quotes will be sql-escaped)
* number (can only contain numerical representation)
* css_color (can only contain color names or hex-values)
* ... (add more as need arises)
Placeholder default value will be used when not provided at
instantiation time and could be used to test validity of the
template by creating a default instance.
Additionally you'll be able to embed an authorization
certificate that would be used to sign any instance of the template.
```js
// template.json
{
version: '0.0.1',
// there can be at most 1 template with the same name for any user
// valid names start with a letter and only contains letter, numbers
// or underscores
name: 'template_name',
// embedded authorization certificate
auth: {
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Signed-maps
method: 'token', // or "open" (the default if no "method" is given)
valid_tokens: ['auth_token1','auth_token2'] // only (required and non empty) for 'token' method
},
// Variables not listed here are not substituted
// Variable not provided at instantiation time trigger an error
// A default is required for optional variables
// Type specification is used for quoting, to avoid injections
placeholders: {
color: {
type:'css_color',
default:'red'
},
cartodb_id: {
type:'number',
default: 1
}
},
layergroup: {
// see https://github.com/CartoDB/Windshaft/wiki/MapConfig-specification
"version": "1.0.1",
"layers": [{
"type": "cartodb",
"options": {
"cartocss_version": "2.1.1",
"cartocss": "#layer { polygon-fill: <%= color %>; }",
"sql": "select * from european_countries_e WHERE cartodb_id = <%= cartodb_id %>"
}
}]
}
}
```
# Creating a templated map
You can create a signed template map with a single call (for simplicity).
You'd use a POST sending JSON data:
```sh
curl -X POST \
-H 'Content-Type: application/json' \
-d @template.json \
'https://docs.cartodb.com/tiles/template?api_key=APIKEY'
```
The response would be like this:
```js
{
"template_id":"@template_name"
}
```
If a template with the same name exists in the user storage,
a 400 response is generated.
Errors are in this form:
```js
{
"error":"Some error string here"
}
```
# Updating an existing template
Update of a template map implies removal all signatures from previous
map instances.
You can update a signed template map with a PUT:
```sh
curl -X PUT \
-H 'Content-Type: application/json' \
-d @template.json \
'https://docs.cartodb.com/tiles/template/:template_name?api_key=APIKEY'
```
A template with the same name will be updated, if any.
The response would be like this:
```js
{
"template_id":"@template_name"
}
```
If a template with the same name does NOT exist,
a 400 HTTP response is generated with an error in this format:
```js
{
"error":"Some error string here"
}
```
# Listing available templates
You can get a list of available templates with a GET to ``/template``.
A valid api_key is required.
```sh
curl -X GET 'https://docs.cartodb.com/tiles/template?api_key=APIKEY'
```
The response would be like this:
```js
{
"template_ids": ["@template_name1","@template_name2"]
}
```
Or, on error:
```js
{
"error":"Some error string here"
}
```
# Getting a specific template
You can get the definition of a template with a
GET to ``/template/:template_name``.
A valid api_key is required.
Example:
```sh
curl -X GET 'https://docs.cartodb.com/tiles/template/@template_name?auth_token=AUTH_TOKEN'
```
The response would be like this:
```js
{
"template": {...} // see template.json above
}
```
Or, on error:
```js
{
"error":"Some error string here"
}
```
# Instantiating a template map
You can instantiate a template map passing all required parameters with
a POST to ``/template/:template_name``.
Valid credentials will be needed, if required by the template.
```js
// params.js
{
color: '#ff0000',
cartodb_id: 3
}
```
```sh
curl -X POST \
-H 'Content-Type: application/json' \
-d @params.js \
'https://docs.cartodb.com/tiles/template/@template_name?auth_token=AUTH_TOKEN'
```
The response would be like this:
```js
{
"layergroupid":"docs@fd2861af@c01a54877c62831bb51720263f91fb33:123456788",
"last_updated":"2013-11-14T11:20:15.000Z"
}
```
or, on error:
```js
{
"error":"Some error string here"
}
```
You can then use the ``layergroupid`` for fetching tiles and grids as you do
normally ( see https://github.com/CartoDB/Windshaft/wiki/Multilayer-API).
But you'll still have to show the ``auth_token``, if required by the template
(see https://github.com/CartoDB/Windshaft-cartodb/wiki/Signed-maps)
Instances of a signed template map will be signed with the same signature
certificate associated with the template. Such certificate would contain
a reference to the template identifier, so that it can be revoked every
time the template is updated or deleted.
### using JSONP
There is also a special endpoint to be able to instanciate using JSONP (for old browsers)
```
curl 'https://docs.cartodb.com/tiles/template/@template_name/jsonp?auth_token=AUTH_TOKEN&callback=function_name&config=template_params_json'
```
it takes the ``callback`` function (required), ``auth_token`` in case the template needs auth and ``config`` which is the variabñes for the template (in case it has variables). For example config may be created (using javascript)
```
url += "config=" + encodeURIComponent(
JSON.stringify({ color: 'red' });
```
the response it's in this format:
```
jQuery17205720721024554223_1390996319118(
{
layergroupid: "dev@744bd0ed9b047f953fae673d56a47b4d:1390844463021.1401",
last_updated: "2014-01-27T17:41:03.021Z"
}
)
```
# Deleting a template map
Deletion of a template map will imply removal all instance signatures
You can delete a templated map with a DELETE to ``/template/:template_name``:
```sh
curl -X DELETE 'https://docs.cartodb.com/tiles/template/@template_name?auth_token=AUTH_TOKEN'
```
On success, a 204 (No Content) response would be issued.
Otherwise a 4xx response with this format:
```js
{
"error":"Some error string here"
}
```

View File

@@ -1,89 +1,26 @@
var _ = require('underscore'),
Varnish = require('node-varnish'),
request = require('request'),
crypto = require('crypto'),
channelCache = {},
varnish_queue = null;
function init(host, port) {
varnish_queue = new Varnish.VarnishQueue(host, port);
function init(host, port, secret) {
varnish_queue = new Varnish.VarnishQueue(host, port, secret);
varnish_queue.on('error', function(e) {
console.log("[CACHE VALIDATOR ERROR] " + e);
});
}
function invalidate_db(dbname, table) {
var cmd = 'purge obj.http.X-Cache-Channel ~ "^' + dbname +
':(.*'+ table +'.*)|(table)$"';
try{
varnish_queue.run_cmd('purge obj.http.X-Cache-Channel ~ "^' + dbname + ':(.*'+ table +'.*)|(table)$"');
console.log('[SUCCESS FLUSHING CACHE]');
varnish_queue.run_cmd(cmd, false);
} catch (e) {
console.log("[ERROR FLUSHING CACHE] Is enable_cache set to true? Failed for: " + 'purge obj.http.X-Cache-Channel ~ "^' + dbname + ':(.*'+ table +'.*)|(table)$"');
console.log("[CACHE VALIDATOR ERROR] could not queue command " +
cmd + " -- " + e);
}
}
function generateCacheChannel(req, callback){
var cacheChannel = "";
// use key to call sql api with sql request if present, else just return dbname and table name
// base key
var tableNames = req.params.table;
var dbName = req.params.dbname;
var username = req.headers.host.split('.')[0];
// replace tableNames with the results of the explain if present
if (_.isString(req.params.sql) && req.params.sql != ''){
// initialise MD5 key of sql for cache lookups
var sql_md5 = generateMD5(req.params.sql);
var api = global.environment.sqlapi;
var qs = {};
// use cache if present
if (!_.isNull(channelCache[sql_md5]) && !_.isUndefined(channelCache[sql_md5])) {
callback(channelCache[sql_md5]);
} else{
// strip out windshaft/mapnik inserted sql if present
var sql = req.params.sql.match(/^\((.*)\)\sas\scdbq$/);
sql = (sql != null) ? sql[1] : req.params.sql;
// build up api string
var sqlapi = api.protocol + '://' + username + '.' + api.host + ':' + api.port + '/api/' + api.version + '/sql'
// add query to querystring
qs.q = 'SELECT CDB_QueryTables($windshaft$' + sql + '$windshaft$)';
// add api_key if present in tile request (means table is private)
if (_.isString(req.params.map_key) && req.params.map_key != ''){
qs.api_key = req.params.map_key;
}
// call sql api
request.get({url:sqlapi, qs:qs, json:true}, function(err, response, body){
if (!err && response.statusCode == 200) {
tableNames = body.rows[0].cdb_querytables.split(/^\{(.*)\}$/)[1];
} else {
//oops, no SQL API. Just cache using fallback 'table' key
tableNames = 'table';
}
cacheChannel = buildCacheChannel(dbName,tableNames);
channelCache[sql_md5] = cacheChannel; // store for caching
callback(cacheChannel);
});
}
} else {
cacheChannel = buildCacheChannel(dbName,tableNames);
callback(cacheChannel);
}
}
function buildCacheChannel(dbName, tableNames){
return dbName + ':' + tableNames;
}
function generateMD5(data){
var hash = crypto.createHash('md5');
hash.update(data);
return hash.digest('hex');
}
module.exports = {
init: init,
invalidate_db: invalidate_db,
generateCacheChannel: generateCacheChannel
invalidate_db: invalidate_db
}

View File

@@ -1,243 +0,0 @@
/**
* User: simon
* Date: 30/08/2011
* Time: 21:10
* Desc: CartoDB helper.
* Retrieves dbname (based on subdomain/username)
* and geometry type from the redis stores of cartodb
*/
var RedisPool = require("./redis_pool")
, _ = require('underscore')
, Step = require('step');
module.exports = function() {
var redis_pool = new RedisPool(global.environment.redis);
var me = {
user_metadata_db: 5,
table_metadata_db: 0,
user_key: "rails:users:<%= username %>",
table_key: "rails:<%= database_name %>:<%= table_name %>"
};
/**
* Get the database name for this particular subdomain/username
*
* @param req - standard express req object. importantly contains host information
* @param callback - gets called with args(err, dbname)
*/
me.getDatabase = function(req, callback) {
// strip subdomain from header host
var username = req.headers.host.split('.')[0]
var redisKey = _.template(this.user_key, {username: username});
this.retrieve(this.user_metadata_db, redisKey, 'database_name', function(err, dbname) {
if ( err ) callback(err, null);
else if ( dbname === null ) {
callback(new Error("missing " + username + "'s dbname in redis (try CARTODB/script/restore_redis)"), null);
}
else callback(err, dbname);
});
};
/**
* Get the user id for this particular subdomain/username
*
* @param req - standard express req object. importantly contains host information
* @param callback
*/
me.getId= function(req, callback) {
// strip subdomain from header host
var username = req.headers.host.split('.')[0];
var redisKey = _.template(this.user_key, {username: username});
this.retrieve(this.user_metadata_db, redisKey, 'id', function(err, dbname) {
if ( err ) callback(err, null);
else if ( dbname === null ) {
callback(new Error("missing " + username + "'s dbuser in redis (try CARTODB/script/restore_redis)"), null);
}
else callback(err, dbname);
});
};
/**
* Check the user map key for this particular subdomain/username
*
* @param req - standard express req object. importantly contains host information
* @param callback
*/
me.checkMapKey = function(req, callback) {
// strip subdomain from header host
var username = req.headers.host.split('.')[0];
var redisKey = "rails:users:" + username;
this.retrieve(this.user_metadata_db, redisKey, "map_key", function(err, val) {
var valid = 0;
if ( val ) {
if ( val == req.query.map_key ) valid = 1;
else if ( val == req.query.api_key ) valid = 1;
// check also in request body
else if ( req.body && req.body.map_key && val == req.body.map_key ) valid = 1;
else if ( req.body && req.body.api_key && val == req.body.api_key ) valid = 1;
}
callback(err, valid);
});
};
/**
* Get privacy for cartodb table
*
* @param req - standard req object. Importantly contains table and host information
* @param callback - is the table private or not?
*/
me.authorize= function(req, callback) {
var that = this;
Step(
function(){
that.checkMapKey(req, this);
},
function checkIfInternal(err, check_result){
if (err) throw err;
if (check_result === 1) {
// authorized by key, login as db owner
that.getId(req, function(err, user_id) {
if (err) throw new Error(err);
var dbuser = _.template(global.settings.postgres_auth_user, {user_id: user_id});
_.extend(req, {dbuser:dbuser});
callback(err, true);
});
} else {
return true; // continue to check if the table is public/private
}
}
,function (err, data){
if (err) throw err;
that.getDatabase(req, this);
},
function(err, data){
if (err) throw err;
var redisKey = _.template(that.table_key, {database_name: data, table_name: req.params.table});
that.retrieve(that.table_metadata_db, redisKey, 'privacy', this);
},
function(err, data){
if (err) throw err;
callback(err, data);
}
);
};
/**
* Get the geometry type for this particular table;
* @param req - standard req object. Importantly contains table and host information
* @param callback
*/
me.getGeometryType = function(req, callback){
var that = this;
Step(
function(){
that.getDatabase(req, this)
},
function(err, data){
if (err) throw err;
var redisKey = _.template(that.table_key, {database_name: data, table_name: req.params.table});
that.retrieve(that.table_metadata_db, redisKey, 'the_geom_type', this);
},
function(err, data){
if (err) throw err;
callback(err, data);
}
);
};
me.getInfowindow = function(req, callback){
var that = this;
Step(
function(){
that.getDatabase(req, this);
},
function(err, data) {
if (err) throw err;
var redisKey = _.template(that.table_key, {database_name: data, table_name: req.params.table});
that.retrieve(that.table_metadata_db, redisKey, 'infowindow', this);
},
function(err, data){
if (err) throw err;
callback(err, data);
}
);
};
me.getMapMetadata = function(req, callback){
var that = this;
Step(
function(){
that.getDatabase(req, this);
},
function(err, data) {
if (err) throw err;
var redisKey = _.template(that.table_key, {database_name: data, table_name: req.params.table});
that.retrieve(that.table_metadata_db, redisKey, 'map_metadata', this);
},
function(err, data){
if (err) throw err;
callback(err, data);
}
);
};
// Redis Hash lookup
// @param callback will be invoked with args (err, reply)
// note that reply is null when the key is missing
me.retrieve = function(db, redisKey, hashKey, callback) {
this.redisCmd(db,'HGET',[redisKey, hashKey], callback);
};
// Redis Set member check
me.inSet = function(db, setKey, member, callback) {
this.redisCmd(db,'SISMEMBER',[setKey, member], callback);
};
/**
* Use Redis
*
* @param db - redis database number
* @param redisFunc - the redis function to execute
* @param redisArgs - the arguments for the redis function in an array
* @param callback - function to pass results too.
*/
me.redisCmd = function(db, redisFunc, redisArgs, callback) {
var redisClient;
Step(
function getRedisClient() {
redis_pool.acquire(db, this);
},
function executeQuery(err, data) {
redisClient = data;
redisArgs.push(this);
redisClient[redisFunc.toUpperCase()].apply(redisClient, redisArgs);
},
function releaseRedisClient(err, data) {
if (err) throw err;
redis_pool.release(db, redisClient);
callback(err, data);
}
);
};
return me;
}();

View File

@@ -2,13 +2,30 @@
var _ = require('underscore')
, Step = require('step')
, Windshaft = require('windshaft')
, Cache = require('./cache_validator');
, redisPool = new require('redis-mpool')(global.environment.redis)
// TODO: instanciate cartoData with redisPool
, cartoData = require('cartodb-redis')(global.environment.redis)
, SignedMaps = require('./signed_maps.js')
, TemplateMaps = require('./template_maps.js')
, Cache = require('./cache_validator')
, os = require('os')
;
var CartodbWindshaft = function(serverOptions) {
var debug = global.environment.debug;
// Perform keyword substitution in statsd
// See https://github.com/CartoDB/Windshaft-cartodb/issues/153
if ( global.environment.statsd ) {
if ( global.environment.statsd.prefix ) {
var host_token = os.hostname().split('.').reverse().join('.');
global.environment.statsd.prefix = global.environment.statsd.prefix.replace(/:host/, host_token);
}
}
if(serverOptions.cache_enabled) {
console.log("cache invalidation enabled, varnish on ", serverOptions.varnish_host, ' ', serverOptions.varnish_port);
Cache.init(serverOptions.varnish_host, serverOptions.varnish_port);
Cache.init(serverOptions.varnish_host, serverOptions.varnish_port, serverOptions.varnish_secret);
serverOptions.afterStateChange = function(req, data, callback) {
Cache.invalidate_db(req.params.dbname, req.params.table);
callback(null, data);
@@ -17,19 +34,109 @@ var CartodbWindshaft = function(serverOptions) {
serverOptions.beforeStateChange = function(req, callback) {
var err = null;
if ( ! req.hasOwnProperty('dbuser') ) {
if ( ! req.params.hasOwnProperty('_authorizedByApiKey') ) {
err = new Error("map state cannot be changed by unauthenticated request!");
}
callback(err, req);
}
// This is for Templated maps
//
// "named" is the official, "template" is for backward compatibility up to 1.6.x
//
var template_baseurl = global.environment.base_url_templated || '(?:/maps/named|/tiles/template)';
serverOptions.signedMaps = new SignedMaps(redisPool);
var templateMapsOpts = {
max_user_templates: global.environment.maxUserTemplates
}
var templateMaps = new TemplateMaps(redisPool, serverOptions.signedMaps, templateMapsOpts);
// boot
var ws = new Windshaft.Server(serverOptions);
// Override getVersion to include cartodb-specific versions
var wsversion = ws.getVersion;
ws.getVersion = function() {
var version = wsversion();
version.windshaft_cartodb = require('../../package.json').version;
return version;
}
var ws_sendResponse = ws.sendResponse;
// GET routes for which we don't want to request any caching.
// POST/PUT/DELETE requests are never cached anyway.
var noCacheGETRoutes = [
'/',
// See https://github.com/CartoDB/Windshaft-cartodb/issues/176
serverOptions.base_url_mapconfig,
template_baseurl + '/:template_id/jsonp'
];
ws.sendResponse = function(res, args) {
var that = this;
var thatArgs = arguments;
var statusCode;
if ( res._windshaftStatusCode ) {
// Added by our override of sendError
statusCode = res._windshaftStatusCode;
} else {
if ( args.length > 2 ) statusCode = args[2];
else {
statusCode = args[1] || 200;
}
}
var req = res.req;
Step (
function addCacheChannel() {
if ( ! req ) {
// having no associated request can happen when
// using fake response objects for testing layergroup
// creation
return false;
}
if ( ! req.params ) {
// service requests (/version, /)
// have no need for an X-Cache-Channel
return false;
}
if ( statusCode != 200 ) {
// We do not want to cache
// unsuccessful responses
return false;
}
if ( _.contains(noCacheGETRoutes, req.route.path) ) {
//console.log("Skipping cache channel in route:\n" + req.route.path);
return false;
}
//console.log("Adding cache channel to route\n" + req.route.path + " not matching any in:\n" + mapCreateRoutes.join("\n"));
serverOptions.addCacheChannel(that, req, this);
},
function sendResponse(err, added) {
if ( err ) console.log(err + err.stack);
ws_sendResponse.apply(that, thatArgs);
return null;
},
function finish(err) {
if ( err ) console.log(err + err.stack);
}
);
};
var ws_sendError = ws.sendError;
ws.sendError = function() {
var res = arguments[0];
var statusCode = arguments[2];
res._windshaftStatusCode = statusCode;
ws_sendError.apply(this, arguments);
};
/**
* Helper to allow access to the layer to be used in the maps infowindow popup.
*/
ws.get(serverOptions.base_url + '/infowindow', function(req, res){
if ( req.profiler && req.profiler.statsd_client ) {
req.profiler.start('windshaft-cartodb.get_infowindow');
}
ws.doCORS(res);
Step(
function(){
@@ -37,9 +144,10 @@ var CartodbWindshaft = function(serverOptions) {
},
function(err, data){
if (err){
res.send({error: err.message}, 500);
ws.sendError(res, {error: err.message}, 500, 'GET INFOWINDOW', err);
//ws.sendResponse(res, [{error: err.message}, 500]);
} else {
res.send({infowindow: data}, 200);
ws.sendResponse(res, [{infowindow: data}, 200]);
}
}
);
@@ -50,6 +158,9 @@ var CartodbWindshaft = function(serverOptions) {
* Helper to allow access to metadata to be used in embedded maps.
*/
ws.get(serverOptions.base_url + '/map_metadata', function(req, res){
if ( req.profiler && req.profiler.statsd_client ) {
req.profiler.start('windshaft-cartodb.get_map_metadata');
}
ws.doCORS(res);
Step(
function(){
@@ -57,9 +168,10 @@ var CartodbWindshaft = function(serverOptions) {
},
function(err, data){
if (err){
res.send(err.message, 500);
ws.sendError(res, {error: err.message}, 500, 'GET MAP_METADATA', err);
//ws.sendResponse(res, [err.message, 500]);
} else {
res.send({map_metadata: data}, 200);
ws.sendResponse(res, [{map_metadata: data}, 200]);
}
}
);
@@ -70,20 +182,493 @@ var CartodbWindshaft = function(serverOptions) {
* TODO: Move?
*/
ws.del(serverOptions.base_url + '/flush_cache', function(req, res){
if ( req.profiler && req.profiler.statsd_client ) {
req.profiler.start('windshaft-cartodb.flush_cache');
}
ws.doCORS(res);
Step(
function(){
function flushCache(){
serverOptions.flushCache(req, serverOptions.cache_enabled ? Cache : null, this);
},
function(err, data){
function sendResponse(err, data){
if (err){
res.send(500);
ws.sendError(res, {error: err.message}, 500, 'DELETE CACHE', err);
//ws.sendResponse(res, [500]);
} else {
res.send({status: 'ok'}, 200);
ws.sendResponse(res, [{status: 'ok'}, 200]);
}
}
);
});
// ---- Template maps interface starts @{
ws.userByReq = function(req) {
return serverOptions.userByReq(req);
}
// Add a template
ws.post(template_baseurl, function(req, res) {
ws.doCORS(res);
var that = this;
var response = {};
var cdbuser = ws.userByReq(req);
Step(
function checkPerms(){
serverOptions.authorizedByAPIKey(req, this);
},
function addTemplate(err, authenticated) {
if ( err ) throw err;
if (authenticated !== 1) {
err = new Error("Only authenticated user can create templated maps");
err.http_status = 403;
throw err;
}
var next = this;
if ( ! req.headers['content-type'] || req.headers['content-type'].split(';')[0] != 'application/json' )
throw new Error('template POST data must be of type application/json');
var cfg = req.body;
templateMaps.addTemplate(cdbuser, cfg, this);
},
function prepareResponse(err, tpl_id){
if ( err ) throw err;
// NOTE: might omit "cdbuser" if == dbowner ...
return { template_id: cdbuser + '@' + tpl_id };
},
function finish(err, response){
if ( req.profiler ) {
var report = req.profiler.toString();
res.header('X-Tiler-Profiler', report);
}
if (err){
response = { error: ''+err };
var statusCode = 400;
if ( ! _.isUndefined(err.http_status) ) {
statusCode = err.http_status;
}
ws.sendError(res, response, statusCode, 'POST TEMPLATE', err);
} else {
ws.sendResponse(res, [response, 200]);
}
}
);
});
// Update a template
ws.put(template_baseurl + '/:template_id', function(req, res) {
ws.doCORS(res);
var that = this;
var response = {};
var cdbuser = ws.userByReq(req);
var template;
var tpl_id;
Step(
function checkPerms(){
serverOptions.authorizedByAPIKey(req, this);
},
function updateTemplate(err, authenticated) {
if ( err ) throw err;
if (authenticated !== 1) {
err = new Error("Only authenticated user can list templated maps");
err.http_status = 403;
throw err;
}
if ( ! req.headers['content-type'] || req.headers['content-type'].split(';')[0] != 'application/json' )
throw new Error('template PUT data must be of type application/json');
template = req.body;
tpl_id = req.params.template_id.split('@');
if ( tpl_id.length > 1 ) {
if ( tpl_id[0] != cdbuser ) {
err = new Error("Invalid template id '"
+ req.params.template_id + "' for user '" + cdbuser + "'");
err.http_status = 404;
throw err;
}
tpl_id = tpl_id[1];
}
templateMaps.updTemplate(cdbuser, tpl_id, template, this);
},
function prepareResponse(err){
if ( err ) throw err;
return { template_id: cdbuser + '@' + tpl_id };
},
function finish(err, response){
if ( req.profiler ) {
var report = req.profiler.toString();
res.header('X-Tiler-Profiler', report);
}
if (err){
var statusCode = 400;
response = { error: ''+err };
if ( ! _.isUndefined(err.http_status) ) {
statusCode = err.http_status;
}
ws.sendError(res, response, statusCode, 'PUT TEMPLATE', err);
} else {
ws.sendResponse(res, [response, 200]);
}
}
);
});
// Get a specific template
ws.get(template_baseurl + '/:template_id', function(req, res) {
if ( req.profiler && req.profiler.statsd_client ) {
req.profiler.start('windshaft-cartodb.get_template');
}
ws.doCORS(res);
var that = this;
var response = {};
var cdbuser = ws.userByReq(req);
var template;
var tpl_id;
Step(
function checkPerms(){
serverOptions.authorizedByAPIKey(req, this);
},
function updateTemplate(err, authenticated) {
if ( err ) throw err;
if (authenticated !== 1) {
err = new Error("Only authenticated users can get template maps");
err.http_status = 403;
throw err;
}
tpl_id = req.params.template_id.split('@');
if ( tpl_id.length > 1 ) {
if ( tpl_id[0] != cdbuser ) {
var err = new Error("Cannot get template id '"
+ req.params.template_id + "' for user '" + cdbuser + "'");
err.http_status = 404;
throw err;
}
tpl_id = tpl_id[1];
}
templateMaps.getTemplate(cdbuser, tpl_id, this);
},
function prepareResponse(err, tpl_val){
if ( err ) throw err;
if ( ! tpl_val ) {
err = new Error("Cannot find template '" + tpl_id + "' of user '" + cdbuser + "'");
err.http_status = 404;
throw err;
}
// auth_id was added by ourselves,
// so we remove it before returning to the user
delete tpl_val.auth_id;
return { template: tpl_val };
},
function finish(err, response){
if (err){
var statusCode = 400;
response = { error: ''+err };
if ( ! _.isUndefined(err.http_status) ) {
statusCode = err.http_status;
}
ws.sendError(res, response, statusCode, 'GET TEMPLATE', err);
} else {
ws.sendResponse(res, [response, 200]);
}
}
);
});
// Delete a specific template
ws.del(template_baseurl + '/:template_id', function(req, res) {
if ( req.profiler && req.profiler.statsd_client ) {
req.profiler.start('windshaft-cartodb.delete_template');
}
ws.doCORS(res);
var that = this;
var response = {};
var cdbuser = ws.userByReq(req);
var template;
var tpl_id;
Step(
function checkPerms(){
serverOptions.authorizedByAPIKey(req, this);
},
function updateTemplate(err, authenticated) {
if ( err ) throw err;
if (authenticated !== 1) {
err = new Error("Only authenticated users can delete template maps");
err.http_status = 403;
throw err;
}
tpl_id = req.params.template_id.split('@');
if ( tpl_id.length > 1 ) {
if ( tpl_id[0] != cdbuser ) {
var err = new Error("Cannot find template id '"
+ req.params.template_id + "' for user '" + cdbuser + "'");
err.http_status = 404;
throw err;
}
tpl_id = tpl_id[1];
}
templateMaps.delTemplate(cdbuser, tpl_id, this);
},
function prepareResponse(err, tpl_val){
if ( err ) throw err;
return { status: 'ok' };
},
function finish(err, response){
if (err){
var statusCode = 400;
response = { error: ''+err };
if ( ! _.isUndefined(err.http_status) ) {
statusCode = err.http_status;
}
ws.sendError(res, response, statusCode, 'DELETE TEMPLATE', err);
} else {
ws.sendResponse(res, ['', 204]);
}
}
);
});
// Get a list of owned templates
ws.get(template_baseurl, function(req, res) {
if ( req.profiler && req.profiler.statsd_client ) {
req.profiler.start('windshaft-cartodb.get_template_list');
}
ws.doCORS(res);
var that = this;
var response = {};
var cdbuser = ws.userByReq(req);
Step(
function checkPerms(){
serverOptions.authorizedByAPIKey(req, this);
},
function listTemplates(err, authenticated) {
if ( err ) throw err;
if (authenticated !== 1) {
err = new Error("Only authenticated user can list templated maps");
err.http_status = 403;
throw err;
}
templateMaps.listTemplates(cdbuser, this);
},
function prepareResponse(err, tpl_ids){
if ( err ) throw err;
// NOTE: might omit "cbduser" if == dbowner ...
var ids = _.map(tpl_ids, function(id) { return cdbuser + '@' + id; })
return { template_ids: ids };
},
function finish(err, response){
var statusCode = 200;
if (err){
response = { error: ''+err };
if ( ! _.isUndefined(err.http_status) ) {
statusCode = err.http_status;
}
ws.sendError(res, response, statusCode, 'GET TEMPLATE LIST', err);
} else {
ws.sendResponse(res, [response, statusCode]);
}
}
);
});
ws.setDBParams = function(cdbuser, params, callback) {
Step(
function setAuth() {
serverOptions.setDBAuth(cdbuser, params, this);
},
function setConn(err) {
if ( err ) throw err;
serverOptions.setDBConn(cdbuser, params, this);
},
function finish(err) {
callback(err);
}
);
};
ws.options(template_baseurl + '/:template_id', function(req, res) {
ws.doCORS(res, "Content-Type");
return next();
});
// Instantiate a template
function instanciateTemplate(req, res, template_params, callback) {
ws.doCORS(res);
if ( req.profiler ) req.profiler.done('cors');
var that = this;
var response = {};
var template;
var signedMaps = serverOptions.signedMaps;
var layergroup;
var layergroupid;
var fakereq; // used for call to createLayergroup
var cdbuser = ws.userByReq(req);
// Format of template_id: [<template_owner>]@<template_id>
var tpl_id = req.params.template_id.split('@');
if ( tpl_id.length > 1 ) {
if ( tpl_id[0] && tpl_id[0] != cdbuser ) {
var err = new Error('Cannot instanciate map of user "'
+ tpl_id[0] + '" on database of user "'
+ cdbuser + '"')
err.http_status = 403;
callback(err);
return;
}
tpl_id = tpl_id[1];
}
var auth_token = req.query.auth_token;
Step(
function getTemplate(){
templateMaps.getTemplate(cdbuser, tpl_id, this);
},
function checkAuthorized(err, data) {
if ( req.profiler ) req.profiler.done('getTemplate');
if ( err ) throw err;
if ( ! data ) {
err = new Error("Template '" + tpl_id + "' of user '" + cdbuser + "' not found");
err.http_status = 404;
throw err;
}
template = data;
var cert = templateMaps.getTemplateCertificate(template);
var authorized = false;
try {
// authorizedByCert will throw if unauthorized
authorized = signedMaps.authorizedByCert(cert, auth_token);
} catch (err) {
// we catch to add http_status
err.http_status = 403;
throw err;
}
if ( ! authorized ) {
err = new Error('Unauthorized template instanciation');
err.http_status = 403;
throw err;
}
/*if ( (! req.headers['content-type'] || req.headers['content-type'].split(';')[0] != 'application/json') && req.query.callback === undefined) {
throw new Error('template POST data must be of type application/json, it is instead ');
}*/
//var template_params = req.body;
if ( req.profiler ) req.profiler.done('authorizedByCert');
return templateMaps.instance(template, template_params);
},
function prepareParams(err, instance){
if ( req.profiler ) req.profiler.done('TemplateMaps_instance');
if ( err ) throw err;
layergroup = instance;
fakereq = { query: {}, params: {}, headers: _.clone(req.headers),
profiler: req.profiler
};
ws.setDBParams(cdbuser, fakereq.params, this);
},
function setApiKey(err){
if ( req.profiler ) req.profiler.done('setDBParams');
if ( err ) throw err;
cartoData.getUserMapKey(cdbuser, this);
},
function createLayergroup(err, val) {
if ( req.profiler ) req.profiler.done('getUserMapKey');
if ( err ) throw err;
fakereq.params.api_key = val;
ws.createLayergroup(layergroup, fakereq, this);
},
function signLayergroup(err, resp) {
// NOTE: createLayergroup uses profiler.start()/end() internally
//if ( req.profiler ) req.profiler.done('createLayergroup');
if ( err ) throw err;
response = resp;
var signer = cdbuser;
var map_id = response.layergroupid.split(':')[0]; // dropping last_updated
var crt_id = template.auth_id; // check ?
if ( ! crt_id ) {
var errmsg = "Template '" + tpl_id + "' of user '" + cdbuser + "' has no signature";
// Is this really illegal ?
// Maybe we could just return an unsigned layergroupid
// in this case...
err = new Error(errmsg);
err.http_status = 403; // Forbidden, we refuse to respond to this
throw err;
}
signedMaps.signMap(signer, map_id, crt_id, this);
},
function prepareResponse(err) {
if ( req.profiler ) req.profiler.done('signMap');
if ( err ) throw err;
//console.log("Response from createLayergroup: "); console.dir(response);
// Add the signature part to the token!
var tplhash = templateMaps.fingerPrint(template).substring(0,8);
if ( req.profiler ) req.profiler.done('fingerPrint');
response.layergroupid = cdbuser + '@' + tplhash + '@' + response.layergroupid;
return response;
},
callback
);
}
function finish_instanciation(err, response, res, req) {
if ( req.profiler ) {
var report = req.profiler.toString();
res.header('X-Tiler-Profiler', report);
}
if (err) {
var statusCode = 400;
response = { error: ''+err };
if ( ! _.isUndefined(err.http_status) ) {
statusCode = err.http_status;
}
if(debug) {
response.stack = err.stack;
}
ws.sendError(res, response, statusCode, 'POST INSTANCE TEMPLATE', err);
} else {
ws.sendResponse(res, [response, 200]);
}
}
ws.post(template_baseurl + '/:template_id', function(req, res) {
if ( req.profiler && req.profiler.statsd_client) {
req.profiler.start('windshaft-cartodb.instance_template_post');
}
Step(
function() {
if ( ! req.headers['content-type'] || req.headers['content-type'].split(';')[0] != 'application/json') {
throw new Error('template POST data must be of type application/json, it is instead ');
}
instanciateTemplate(req, res, req.body, this);
}, function(err, response) {
finish_instanciation(err, response, res, req);
}
);
});
/**
* jsonp endpoint, allows to instanciate a template with a json call.
* callback query argument is mandartoy
*/
ws.get(template_baseurl + '/:template_id/jsonp', function(req, res) {
if ( req.profiler && req.profiler.statsd_client) {
req.profiler.start('windshaft-cartodb.instance_template_get');
}
Step(
function() {
if ( req.query.callback === undefined || req.query.callback.length === 0) {
throw new Error('callback parameter should be present and be a function name');
}
var config = {};
if(req.query.config) {
try {
config = JSON.parse(req.query.config);
} catch(e) {
throw new Error('badformed config parameter, should be a valid JSON');
}
}
instanciateTemplate(req, res, config, this);
}, function(err, response) {
finish_instanciation(err, response, res, req);
}
);
});
// ---- Template maps interface ends @}
return ws;
}

View File

@@ -0,0 +1,49 @@
var rollbar = require("rollbar");
/**
* Rollbar Appender. Sends logging events to Rollbar using node-rollbar
*
* @param config object with rollbar configuration data
* {
* token: 'your-secret-token',
* options: node-rollbar options
* }
*/
function rollbarAppender(config) {
var opt = config.options;
rollbar.init(opt.token, opt.options);
return function(loggingEvent) {
/*
For logger.trace('one','two','three'):
{ startTime: Wed Mar 12 2014 16:27:40 GMT+0100 (CET),
categoryName: '[default]',
data: [ 'one', 'two', 'three' ],
level: { level: 5000, levelStr: 'TRACE' },
logger: { category: '[default]', _events: { log: [Object] } } }
*/
// Levels:
// TRACE 5000
// DEBUG 10000
// INFO 20000
// WARN 30000
// ERROR 40000
// FATAL 50000
//
// We only log error and higher errors
//
if ( loggingEvent.level.level < 40000 ) return;
rollbar.reportMessage(loggingEvent.data);
};
}
function configure(config) {
return rollbarAppender(config);
}
exports.name = "rollbar";
exports.appender = rollbarAppender;
exports.configure = configure;

View File

@@ -1,77 +0,0 @@
/**
* RedisPool. A database specific redis pooling lib
*
*/
var redis = require('redis')
, _ = require('underscore')
, Pool = require('generic-pool').Pool;
// constructor.
//
// - `opts` {Object} optional config for redis and pooling
var RedisPool = function(opts){
var opts = opts || {};
var defaults = {
host: '127.0.0.1',
port: '6379',
max: 50,
idleTimeoutMillis: 10000,
reapIntervalMillis: 1000,
log: false
};
var options = _.defaults(opts, defaults)
var me = {
pools: {} // cached pools by DB name
};
// Acquire resource.
//
// - `database` {String} redis database name
// - `callback` {Function} callback to call once acquired. Takes the form
// `callback(err, resource)`
me.acquire = function(database, callback) {
if (!this.pools[database]) {
this.pools[database] = this.makePool(database);
}
this.pools[database].acquire(function(err,resource) {
callback(err, resource);
});
};
// Release resource.
//
// - `database` {String} redis database name
// - `resource` {Object} resource object to release
me.release = function(database, resource) {
this.pools[database] && this.pools[database].release(resource);
};
// Factory for pool objects.
me.makePool = function(database) {
return Pool({
name: database,
create: function(callback){
var client = redis.createClient(options.port, options.host);
client.on('connect', function () {
client.send_anyway = true;
client.select(database);
client.send_anyway = false;
});
return callback(null, client);
},
destroy: function(client) {
return client.quit();
},
max: options.max,
idleTimeoutMillis: options.idleTimeoutMillis,
reapIntervalMillis: options.reapIntervalMillis,
log: options.log
});
};
return me;
};
module.exports = RedisPool;

View File

@@ -1,21 +1,312 @@
var _ = require('underscore')
, Step = require('step')
, cartoData = require('./carto_data')
, Cache = require('./cache_validator');
, cartoData = require('cartodb-redis')(global.environment.redis)
, Cache = require('./cache_validator')
, mapnik = require('mapnik')
, crypto = require('crypto')
, request = require('request')
, LZMA = require('lzma/lzma_worker.js').LZMA
;
// This is for backward compatibility with 1.3.3
if ( _.isUndefined(global.environment.sqlapi.domain) ) {
// Only use "host" as "domain" if it contains alphanumeric characters
var host = global.environment.sqlapi.host;
if ( host && host.match(/[a-zA-Z]/) ) {
global.environment.sqlapi.domain = host;
}
}
module.exports = function(){
var rendererConfig = _.defaults(global.environment.renderer || {}, {
cache_ttl: 60000, // milliseconds
metatile: 4,
bufferSize: 64
});
var me = {
base_url: '/tiles/:table',
// This is for inline maps and table maps
base_url: global.environment.base_url_legacy || '/tiles/:table',
/// @deprecated with Windshaft-0.17.0
///base_url_notable: '/tiles',
// This is for Detached maps
//
// "maps" is the official, while
// "tiles/layergroup" is for backward compatibility up to 1.6.x
//
base_url_mapconfig: global.environment.base_url_detached || '(?:/maps|/tiles/layergroup)',
grainstore: {
map: {
// TODO: allow to specify in configuration
srid: 3857
},
datasource: global.environment.postgres,
cachedir: global.environment.millstone.cache_basedir
cachedir: global.environment.millstone.cache_basedir,
mapnik_version: global.environment.mapnik_version || mapnik.versions.mapnik,
default_layergroup_ttl: global.environment.mapConfigTTL || 7200,
gc_prob: 0.01 // @deprecated since Windshaft-1.8.0
},
mapnik: {
metatile: rendererConfig.metatile,
bufferSize: rendererConfig.bufferSize
},
statsd: global.environment.statsd,
renderCache: {
ttl: rendererConfig.cache_ttl
},
redis: global.environment.redis,
enable_cors: global.environment.enable_cors,
varnish_host: global.environment.varnish.host,
varnish_port: global.environment.varnish.port,
varnish_secret: global.environment.varnish.secret,
cache_enabled: global.environment.cache_enabled,
log_format: global.environment.log_format
log_format: global.environment.log_format,
useProfiler: global.environment.useProfiler
};
// Do not send unwatch on release
// See http://github.com/CartoDB/Windshaft-cartodb/issues/161
me.redis.unwatchOnRelease = false;
// Be nice and warn if configured mapnik version
// is != instaled mapnik version
if ( mapnik.versions.mapnik != me.grainstore.mapnik_version ) {
console.warn("WARNING: detected mapnik version ("
+ mapnik.versions.mapnik + ") != configured mapnik version ("
+ me.grainstore.mapnik_version + ")");
}
/* This whole block is about generating X-Cache-Channel { */
// TODO: review lifetime of elements of this cache
// NOTE: by-token indices should only be dropped when
// the corresponding layegroup is dropped, because
// we have no SQL after layer creation.
me.channelCache = {};
// Run a query through the SQL api
me.sqlQuery = function (username, api_key, sql, callback) {
var api = global.environment.sqlapi;
// build up api string
var sqlapihostname = username;
if ( api.domain ) sqlapihostname += '.' + api.domain;
var sqlapi = api.protocol + '://';
if ( api.host && api.host != api.domain ) sqlapi += api.host;
else sqlapi += sqlapihostname;
sqlapi += ':' + api.port + '/api/' + api.version + '/sql';
var qs = { q: sql }
// add api_key if given
if (_.isString(api_key) && api_key != '') { qs.api_key = api_key; }
// call sql api
//
// NOTE: using POST to avoid size limits:
// See http://github.com/CartoDB/Windshaft-cartodb/issues/111
//
// NOTE: uses "host" header to allow IP based specification
// of sqlapi address (and avoid a DNS lookup)
//
// NOTE: allows for keeping up to "maxConnections" concurrent
// sockets opened per SQL-API host.
// See http://nodejs.org/api/http.html#http_agent_maxsockets
//
var maxSockets = global.environment.maxConnections || 128;
var maxGetLen = api.max_get_sql_length || 2048;
var maxSQLTime = api.timeout || 100; // 1/10 of a second by default
var reqSpec = {
url:sqlapi,
json:true,
headers:{host: sqlapihostname}
// http://nodejs.org/api/http.html#http_agent_maxsockets
,pool:{maxSockets:maxSockets}
// timeout in milliseconds
,timeout:maxSQLTime
}
if ( sql.length > maxGetLen ) {
reqSpec.method = 'POST';
reqSpec.body = qs;
} else {
reqSpec.method = 'GET';
reqSpec.qs = qs;
}
request(reqSpec, function(err, res, body) {
if (err){
console.log('ERROR connecting to SQL API on ' + sqlapi + ': ' + err);
callback(err);
return;
}
if (res.statusCode != 200) {
var msg = res.body.error ? res.body.error : res.body;
callback(new Error(msg));
console.log('unexpected response status (' + res.statusCode + ') for sql query: ' + sql + ': ' + msg);
return;
}
callback(null, body.rows);
});
};
//
// Invoke callback with number of milliseconds since
// last update in any of the given tables
//
me.findLastUpdated = function (username, api_key, tableNames, callback) {
var sql = 'SELECT EXTRACT(EPOCH FROM max(updated_at)) as max FROM CDB_TableMetadata m WHERE m.tabname::name = any (\'{'
+ tableNames.join(',') + '}\')';
// call sql api
me.sqlQuery(username, api_key, sql, function(err, rows){
if (err){
var msg = err.message ? err.message : err;
callback(new Error('could not find last updated timestamp: ' + msg));
return;
}
// when the table has not updated_at means it hasn't been changed so a default last_updated is set
var last_updated = 0;
if(rows.length !== 0) {
last_updated = rows[0].max || 0;
}
callback(null, last_updated*1000);
});
};
me.affectedTables = function (username, api_key, sql, callback) {
// Replace mapnik tokens
sql = sql.replace(RegExp('!bbox!', 'g'), 'ST_MakeEnvelope(0,0,0,0)')
.replace(RegExp('!pixel_width!', 'g'), '1')
.replace(RegExp('!pixel_height!', 'g'), '1')
;
// Pass to CDB_QueryTables
sql = 'SELECT CDB_QueryTables($windshaft$' + sql + '$windshaft$)';
// call sql api
me.sqlQuery(username, api_key, sql, function(err, rows){
if (err){
var msg = err.message ? err.message : err;
callback(new Error('could not fetch source tables: ' + msg));
return;
}
var qtables = rows[0].cdb_querytables;
var tableNames = qtables.split(/^\{(.*)\}$/)[1];
tableNames = tableNames ? tableNames.split(',') : [];
callback(null, tableNames);
});
};
me.buildCacheChannel = function (dbName, tableNames){
return dbName + ':' + tableNames.join(',');
};
me.generateMD5 = function(data){
var hash = crypto.createHash('md5');
hash.update(data);
return hash.digest('hex');
}
me.generateCacheChannel = function(app, req, callback){
// Build channelCache key
var dbName = req.params.dbname;
var cacheKey = [ dbName ];
if ( req.params.token ) cacheKey.push(req.params.token);
else if ( req.params.sql ) cacheKey.push( me.generateMD5(req.params.sql) );
cacheKey = cacheKey.join(':');
var that = this;
Step (
function checkCached() {
if ( me.channelCache.hasOwnProperty(cacheKey) ) {
callback(null, me.channelCache[cacheKey]);
return;
}
return null;
},
function extractSQL(err) {
if ( err ) throw err;
if ( req.params.token ) {
// TODO: cached cache channel for token-based access should
// be constructed at renderer cache creation time
// See http://github.com/CartoDB/Windshaft-cartodb/issues/152
if ( ! app.mapStore ) {
throw new Error('missing channel cache for token ' + req.params.token);
return;
}
var next = this;
var mapStore = app.mapStore;
Step(
function loadFromStore() {
mapStore.load(req.params.token, this);
},
function getSQL(err, mapConfig) {
if (req.profiler) req.profiler.done('mapStore_load');
if ( err ) throw err;
var sql = [];
_.each(mapConfig.obj().layers, function(lyr) {
sql.push(lyr.options.sql);
});
sql = sql.join(';');
return sql;
},
function finish(err, sql) {
next(err, sql);
}
);
return;
}
if ( ! req.params.sql ) {
return null; // no sql
}
// We have sql, and no token...
// strip out windshaft/mapnik inserted sql if present
var sql = req.params.sql.match(/^\((.*)\)\sas\scdbq$/);
sql = (sql != null) ? sql[1] : req.params.sql;
return sql;
},
function findAffectedTables(err, sql) {
if ( err ) throw err;
if ( ! sql ) {
if ( ! req.params.table ) {
throw new Error("this request doesn't need an X-Cache-Channel generated");
}
return [req.params.table];
}
var username = that.userByReq(req);
me.affectedTables(username, req.params.map_key, sql, this);
},
function buildCacheChannel(err, tableNames) {
if ( err ) throw err;
if (req.profiler && ! req.params.table ) {
req.profiler.done('affectedTables');
}
var dbName = req.params.dbname;
var cacheChannel = me.buildCacheChannel(dbName,tableNames);
// store for caching from me.generateCacheChannel
// (not worth when table was specified in params)
if ( ! req.params.table ) {
me.channelCache[cacheKey] = cacheChannel;
}
return cacheChannel;
},
function finish(err, cacheChannel) {
callback(err, cacheChannel);
}
);
};
// Set the cache chanel info to invalidate the cache on the frontend server
@@ -27,19 +318,381 @@ module.exports = function(){
// @param cb function(err, channel) will be called when ready.
// the channel parameter will be null if nothing was added
//
me.addCacheChannel = function(req, cb) {
me.addCacheChannel = function(app, req, cb) {
// skip non-GET requests, or requests for which there's no response
if ( req.method != 'GET' || ! req.res ) { cb(null, null); return; }
if (req.profiler) req.profiler.start('addCacheChannel');
var res = req.res;
var ttl = global.environment.varnish.ttl || 86400;
Cache.generateCacheChannel(req, function(channel){
res.header('X-Cache-Channel', channel);
res.header('Last-Modified', new Date().toUTCString());
res.header('Cache-Control', 'no-cache,max-age='+ttl+',must-revalidate, public');
cb(null, channel); // add last-modified too ?
var cache_policy = req.query.cache_policy;
if ( req.params.token ) cache_policy = 'persist';
if ( cache_policy == 'persist' ) {
res.header('Cache-Control', 'public,max-age=31536000'); // 1 year
} else {
var ttl = global.environment.varnish.ttl || 86400;
res.header('Cache-Control', 'no-cache,max-age='+ttl+',must-revalidate, public');
}
// Set Last-Modified header
var lastUpdated;
if ( req.params.cache_buster ) {
// Assuming cache_buster is a timestamp
// FIXME: store lastModified in the cache channel instead
lastUpdated = new Date(parseInt(req.params.cache_buster));
} else {
lastUpdated = new Date();
}
res.header('Last-Modified', lastUpdated.toUTCString());
me.generateCacheChannel(app, req, function(err, channel){
if (req.profiler) req.profiler.done('generateCacheChannel');
if (req.profiler) req.profiler.end();
if ( ! err ) {
res.header('X-Cache-Channel', channel);
cb(null, channel);
} else {
console.log('ERROR generating cache channel: ' + ( err.message ? err.message : err ));
// TODO: evaluate if we should bubble up the error instead
cb(null, 'ERROR');
}
});
};
me.afterLayergroupCreate = function(req, mapconfig, response, callback) {
var token = response.layergroupid;
var username = this.userByReq(req);
var tasksleft = 2; // redis key and affectedTables
var errors = [];
var done = function(err) {
if ( err ) {
errors.push('' + err);
}
if ( ! --tasksleft ) {
err = errors.length ? new Error(errors.join('\n')) : null;
callback(err);
}
}
// include in layergroup response the variables in serverMedata
// those variables are useful to send to the client information
// about how to reach this server or information about it
var serverMetadata = global.environment.serverMetadata;
if (serverMetadata) {
_.extend(response, serverMetadata);
}
// Don't wait for the mapview count increment to
// take place before proceeding. Error will be logged
// asyncronously
cartoData.incMapviewCount(username, mapconfig.stat_tag, function(err) {
if (req.profiler) req.profiler.done('incMapviewCount');
if ( err ) console.log("ERROR: failed to increment mapview count for user '" + username + "': " + err);
done();
});
var sql = [];
_.each(mapconfig.layers, function(lyr) {
sql.push(lyr.options.sql);
});
sql = sql.join(';');
var dbName = req.params.dbname;
var usr = this.userByReq(req);
var key = req.params.map_key || req.params.api_key;
var cacheKey = dbName + ':' + token;
var tabNames;
Step(
function getTables() {
me.affectedTables(usr, key, sql, this);
},
function getLastupdated(err, tableNames) {
if (req.profiler) req.profiler.done('affectedTables');
if ( err ) throw err;
var cacheChannel = me.buildCacheChannel(dbName,tableNames);
// store for caching from me.afterLayergroupCreate
me.channelCache[cacheKey] = cacheChannel;
// find last updated
if ( ! tableNames.length ) return 0; // skip for no affected tables
tabNames = tableNames;
me.findLastUpdated(usr, key, tableNames, this);
},
function(err, lastUpdated) {
if ( err ) throw err;
if (req.profiler && tabNames) req.profiler.done('findLastUpdated');
response.layergroupid = response.layergroupid + ':' + lastUpdated; // use epoch
response.last_updated = new Date(lastUpdated).toISOString();
return null;
},
function finish(err) {
done(err);
}
);
};
/* X-Cache-Channel generation } */
me.re_userFromHost = new RegExp(
global.environment.user_from_host ||
'^([^\\.]+)\\.' // would extract "strk" from "strk.cartodb.com"
);
me.userByReq = function(req) {
var host = req.headers.host;
var mat = host.match(this.re_userFromHost);
if ( ! mat ) {
console.error("ERROR: user pattern '" + this.re_userFromHost
+ "' does not match hostname '" + host + "'");
return;
}
// console.log("Matches: "); console.dir(mat);
if ( ! mat.length === 2 ) {
console.error("ERROR: pattern '" + this.re_userFromHost
+ "' gave unexpected matches against '" + host + "': " + mat);
return;
}
return mat[1];
}
// Set db authentication parameters to those of the given username
//
// @param username the cartodb username, mapped to a database username
// via CartodbRedis metadata records
//
// @param params the parameters to set auth options into
// added params are: "dbuser" and "dbpassword"
//
// @param callback function(err)
//
me.setDBAuth = function(username, params, callback) {
var user_params = {};
var auth_user = global.environment.postgres_auth_user;
var auth_pass = global.environment.postgres_auth_pass;
Step(
function getId() {
cartoData.getUserId(username, this);
},
function(err, user_id) {
if (err) throw err;
user_params['user_id'] = user_id;
var dbuser = _.template(auth_user, user_params);
_.extend(params, {dbuser:dbuser});
// skip looking up user_password if postgres_auth_pass
// doesn't contain the "user_password" label
if (!auth_pass || ! auth_pass.match(/\buser_password\b/) ) return null;
cartoData.getUserDBPass(username, this);
},
function(err, user_password) {
if (err) throw err;
user_params['user_password'] = user_password;
if ( auth_pass ) {
var dbpass = _.template(auth_pass, user_params);
_.extend(params, {dbpassword:dbpass});
}
return true;
},
function finish(err) {
callback(err);
}
);
};
// Set db connection parameters to those for the given username
//
// @param dbowner cartodb username of database owner,
// mapped to a database username
// via CartodbRedis metadata records
//
// @param params the parameters to set connection options into
// added params are: "dbname", "dbhost"
//
// @param callback function(err)
//
me.setDBConn = function(dbowner, params, callback) {
// Add default database connection parameters
// if none given
_.defaults(params, {
dbuser: global.environment.postgres.user,
dbpassword: global.environment.postgres.password,
dbhost: global.environment.postgres.host,
dbport: global.environment.postgres.port
});
Step(
function getDatabaseHost(){
cartoData.getUserDBHost(dbowner, this);
},
function getDatabase(err, data){
if(err) throw err;
if ( data ) _.extend(params, {dbhost:data});
cartoData.getUserDBName(dbowner, this);
},
function extendParams(err, data){
if (err) throw err;
if ( data ) _.extend(params, {dbname:data});
return null;
},
function finish(err) {
callback(err);
}
);
};
// Check if a request is authorized by a signer
//
// Any existing signature for the given request will verified
// for authorization to this specific request (may require auth_token)
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Signed-maps
//
// @param req express request object
// @param callback function(err, signed_by) signed_by will be
// null if the request is not signed by anyone
// or will be a string cartodb username otherwise.
//
me.authorizedBySigner = function(req, callback)
{
if ( ! req.params.token || ! req.params.signer ) {
//console.log("No signature provided"); // debugging
callback(null, null); // no signer requested
return;
}
var signer = req.params.signer;
var layergroup_id = req.params.token;
var auth_token = req.params.auth_token;
//console.log("Checking authorization from signer " + signer + " for resource " + layergroup_id + " with auth_token " + auth_token);
me.signedMaps.isAuthorized(signer, layergroup_id, auth_token,
function(err, authorized) {
callback(err, authorized ? signer : null);
});
};
// Check if a request is authorized by api_key
//
// @param req express request object
// @param callback function(err, authorized)
// NOTE: authorized is expected to be 0 or 1 (integer)
//
me.authorizedByAPIKey = function(req, callback)
{
var givenKey = req.query.api_key || req.query.map_key;
if ( ! givenKey && req.body ) {
// check also in request body
givenKey = req.body.api_key || req.body.map_key;
}
if ( ! givenKey ) {
callback(null, 0); // no api key, no authorization...
return;
}
//console.log("given ApiKey: " + givenKey);
var user = me.userByReq(req);
Step(
function (){
cartoData.getUserMapKey(user, this);
},
function checkApiKey(err, val){
if (err) throw err;
return ( val && givenKey == val ) ? 1 : 0;
},
function finish(err, authorized) {
callback(err, authorized);
}
);
};
/**
* Check access authorization
*
* @param req - standard req object. Importantly contains table and host information
* @param callback function(err, allowed) is access allowed not?
*/
me.authorize = function(req, callback) {
var that = this;
var user = me.userByReq(req);
Step(
function (){
that.authorizedByAPIKey(req, this);
},
function checkApiKey(err, authorized){
if (req.profiler) req.profiler.done('authorizedByAPIKey');
if (err) throw err;
// if not authorized by api_key, continue
if (authorized !== 1) {
// not authorized by api_key,
// check if authorized by signer
that.authorizedBySigner(req, this);
return;
}
_.extend(req.params, { _authorizedByApiKey: true });
// authorized by api key, login as the given username and stop
that.setDBAuth(user, req.params, function(err) {
callback(err, true); // authorized (or error)
});
},
function checkSignAuthorized(err, signed_by){
if (err) throw err;
if (req.profiler) {
if ( req.params._authorizedByApiKey ) {
req.profiler.done('setDBAuth');
} else {
req.profiler.done('authorizedBySigner');
}
}
if ( ! signed_by ) {
// request not authorized by signer.
// if table was given, continue to check table privacy
if ( req.params.table ) return null;
// if no signer name was given, let dbparams and
// PostgreSQL do the rest.
//
if ( ! req.params.signer ) {
callback(null, true); // authorized so far
return;
}
// if signer name was given, return no authorization
callback(null, false);
return;
}
// Authorized by "signed_by" !
that.setDBAuth(signed_by, req.params, function(err) {
if (req.profiler) req.profiler.done('setDBAuth');
callback(err, true); // authorized (or error)
});
},
function getDatabase(err){
if (err) throw err;
// NOTE: only used to get to table privacy
cartoData.getUserDBName(user, this);
},
function getPrivacy(err, dbname){
if (err) throw err;
if (req.profiler) req.profiler.done('tablePrivacy_getUserDBName');
cartoData.getTablePrivacy(dbname, req.params.table, this);
},
function(err, privacy){
if (req.profiler) req.profiler.done('getTablePrivacy');
callback(err, privacy !== "0");
}
);
};
/**
* Whitelist input and get database name & default geometry type from
* subdomain/user metadata held in CartoDB Redis
@@ -48,57 +701,117 @@ module.exports = function(){
*/
me.req2params = function(req, callback){
if ( req.query.lzma ) {
// TODO: check ?
//console.log("type of req.query.lzma is " + typeof(req.query.lzma));
// Decode (from base64)
var lzma = (new Buffer(req.query.lzma, 'base64').toString('binary')).split('').map(function(c) { return c.charCodeAt(0) - 128 })
// Decompress
LZMA.decompress(
lzma,
function(result) {
if (req.profiler) req.profiler.done('LZMA decompress');
try {
delete req.query.lzma
_.extend(req.query, JSON.parse(result))
me.req2params(req, callback);
} catch (err) {
callback(new Error('Error parsing lzma as JSON: ' + err));
}
},
function(percent) { // progress
//console.log("LZMA decompression " + percent + "%");
}
);
return;
}
// Whitelist query parameters and attach format
var good_query = ['sql', 'geom_type', 'cache_buster','callback', 'interactivity', 'map_key', 'api_key', 'style'];
var good_query = ['sql', 'geom_type', 'cache_buster', 'cache_policy', 'callback', 'interactivity', 'map_key', 'api_key', 'auth_token', 'style', 'style_version', 'style_convert', 'config' ];
var bad_query = _.difference(_.keys(req.query), good_query);
_.each(bad_query, function(key){ delete req.query[key]; });
req.params = _.extend({}, req.params); // shuffle things as request is a strange array/object
var user = me.userByReq(req);
if ( req.params.token ) {
//console.log("Request parameters include token " + req.params.token);
var tksplit = req.params.token.split(':');
req.params.token = tksplit[0];
if ( tksplit.length > 1 ) req.params.cache_buster= tksplit[1];
tksplit = req.params.token.split('@');
if ( tksplit.length > 1 ) {
req.params.signer = tksplit.shift();
if ( ! req.params.signer ) req.params.signer = user;
else if ( req.params.signer != user ) {
var err = new Error('Cannot use map signature of user "' + req.params.signer + '" on database of user "' + user + '"')
err.http_status = 403;
callback(err);
return;
}
if ( tksplit.length > 1 ) {
var template_hash = tksplit.shift(); // unused
}
req.params.token = tksplit.shift();
//console.log("Request for token " + req.params.token + " with signature from " + req.params.signer);
}
}
// bring all query values onto req.params object
_.extend(req.params, req.query);
// for cartodb, ensure interactivity is cartodb_id or user specified
req.params.interactivity = req.params.interactivity || 'cartodb_id';
req.params.processXML = function(req, xml, callback) {
var dbuser = req.dbuser ? req.dbuser : global.settings.postgres.user;
if ( ! me.rx_dbuser ) me.rx_dbuser = /(<Parameter name="user"><!\[CDATA\[)[^\]]*(]]><\/Parameter>)/;
xml = xml.replace(me.rx_dbuser, "$1" + dbuser + "$2");
callback(null, xml);
}
var that = this;
if (req.profiler) req.profiler.done('req2params.setup');
Step(
function getPrivacy(){
cartoData.authorize(req, this);
me.authorize(req, this);
},
function gatekeep(err, data){
function gatekeep(err, authorized){
if (req.profiler) req.profiler.done('authorize');
if(err) throw err;
if(data === "0") throw new Error("Sorry, you are unauthorized (permission denied)");
return data;
if(!authorized) {
err = new Error("Sorry, you are unauthorized (permission denied)");
err.http_status = 403;
throw err;
}
return null;
},
function getDatabase(err, data){
function getDatabase(err){
if(err) throw err;
cartoData.getDatabase(req, this);
that.setDBConn(user, req.params, this);
},
function getGeometryType(err, data){
function getGeometryType(err){
if (req.profiler) req.profiler.done('setDBConn');
if (err) throw err;
_.extend(req.params, {dbname:data});
cartoData.getGeometryType(req, this);
if ( ! req.params.table ) return null;
cartoData.getTableGeometryType(req.params.dbname, req.params.table, this);
},
function finishSetup(err, data){
if (req.profiler) req.profiler.done('cartoData.getTableGeometryType');
if ( err ) { callback(err, req); return; }
if (!_.isNull(data))
_.extend(req.params, {geom_type: data});
that.addCacheChannel(req, function(err, chan) {
callback(err, req);
// Add default database connection parameters
// if none given
_.defaults(req.params, {
dbuser: global.environment.postgres.user,
dbpassword: global.environment.postgres.password,
dbhost: global.environment.postgres.host,
dbport: global.environment.postgres.port
});
callback(null, req);
}
);
};
@@ -110,14 +823,23 @@ module.exports = function(){
*/
me.getInfowindow = function(req, callback){
var that = this;
var user = me.userByReq(req);
Step(
function(){
// TODO: if this step really needed ?
that.req2params(req, this);
},
function getDatabase(err){
if (err) throw err;
cartoData.getUserDBName(user, this);
},
function getInfowindow(err, dbname){
if (err) throw err;
cartoData.getTableInfowindow(dbname, req.params.table, this);
},
function(err, data){
if (err) callback(err, null);
else cartoData.getInfowindow(data, callback);
callback(err, data);
}
);
};
@@ -129,14 +851,23 @@ module.exports = function(){
*/
me.getMapMetadata = function(req, callback){
var that = this;
var user = me.userByReq(req);
Step(
function(){
// TODO: if this step really needed ?
that.req2params(req, this);
},
function getDatabase(err){
if (err) throw err;
cartoData.getUserDBName(user, this);
},
function getMapMetadata(err, dbname){
if (err) throw err;
cartoData.getTableMapMetadata(dbname, req.params.table, this);
},
function(err, data){
if (err) callback(err, null);
else cartoData.getMapMetadata(data, callback);
callback(err, data);
}
);
};
@@ -150,11 +881,17 @@ module.exports = function(){
var that = this;
Step(
function(){
function getParams(){
// this is mostly to compute req.params.dbname
that.req2params(req, this);
},
function(err, data){
if (err) throw err;
function flushInternalCache(err){
// TODO: implement this, see
// http://github.com/Vizzuality/Windshaft-cartodb/issues/73
return true;
},
function flushVarnishCache(err){
if (err) { callback(err); return; }
if(Cache) {
Cache.invalidate_db(req.params.dbname, req.params.table);
}
@@ -164,4 +901,4 @@ module.exports = function(){
};
return me;
}();
};

401
lib/cartodb/signed_maps.js Normal file
View File

@@ -0,0 +1,401 @@
var crypto = require('crypto');
var Step = require('step');
var _ = require('underscore');
var debug = global.environment ? global.environment.debug : undefined;
// Class handling map signatures and user certificates
//
// See https://github.com/CartoDB/Windshaft-cartodb/wiki/Signed-maps
//
// @param redis_pool an instance of a "redis-mpool"
// See https://github.com/CartoDB/node-redis-mpool
// Needs version 0.x.x of the API.
//
function SignedMaps(redis_pool) {
this.redis_pool = redis_pool;
// Database containing signatures
// TODO: allow configuring ?
// NOTE: currently it is the same as
// the one containing layergroups
this.db_signatures = 0;
//
// Map signatures in redis are reference to signature certificates
// We have the following datastores:
//
// 1. User certificates: set of per-user authorization certificates
// 2. Map signatures: set of per-map certificate references
// 3. Certificate applications: set of per-certificate signed maps
// User certificates (HASH:crt_id->crt_val)
this.key_map_crt = "map_crt|<%= signer %>";
// Map signatures (SET:crt_id)
this.key_map_sig = "map_sig|<%= signer %>|<%= map_id %>";
// Certificates applications (SET:map_id)
//
// Everytime a map is signed, the map identifier (layergroup_id)
// is added to this set. The purpose of this set is to drop
// all map signatures when a certificate is removed
//
this.key_crt_sig = "crt_sig|<%= signer %>|<%= crt_id %>";
};
var o = SignedMaps.prototype;
//--------------- PRIVATE METHODS --------------------------------
o._acquireRedis = function(callback) {
this.redis_pool.acquire(this.db_signatures, callback);
};
o._releaseRedis = function(client) {
this.redis_pool.release(this.db_signatures, client);
};
/**
* Internal function to communicate with redis
*
* @param redisFunc - the redis function to execute
* @param redisArgs - the arguments for the redis function in an array
* @param callback - function to pass results too.
*/
o._redisCmd = function(redisFunc, redisArgs, callback) {
var redisClient;
var that = this;
var db = that.db_signatures;
Step(
function getRedisClient() {
that.redis_pool.acquire(db, this);
},
function executeQuery(err, data) {
if ( err ) throw err;
redisClient = data;
redisArgs.push(this);
redisClient[redisFunc.toUpperCase()].apply(redisClient, redisArgs);
},
function releaseRedisClient(err, data) {
if ( ! _.isUndefined(redisClient) ) that.redis_pool.release(db, redisClient);
callback(err, data);
}
);
};
o._getAuthMethod = function(auth) {
return auth.method || 'open';
};
//--------------- PUBLIC API -------------------------------------
/// Check formal validity of a certificate
//
/// Return an Error instance if invalid, null otherwise
///
o.checkInvalidCertificate = function(cert) {
//console.log("Checking cert: "); console.dir(cert);
if ( cert.version !== "0.0.1" ) {
return new Error("Unsupported certificate version " + cert.version);
}
if ( ! cert.auth ) {
console.log("Cert is : "); console.dir(cert);
return new Error("No certificate authorization");
}
var method = this._getAuthMethod(cert.auth);
switch ( method ) {
case 'open':
break;
case 'token':
if ( ! _.isArray(cert.auth.valid_tokens) )
return new Error("Invalid 'token' authentication: missing valid_tokens");
if ( ! cert.auth.valid_tokens.length )
return new Error("Invalid 'token' authentication: no valid_tokens");
break;
default:
return new Error("Unsupported authentication method: " + cert.auth.method);
break;
}
return null; // all valid
}
// Check if the given certificate authorizes waiver of "auth"
o.authorizedByCert = function(cert, auth) {
var err = this.checkInvalidCertificate(cert);
if ( err ) throw err;
var method = this._getAuthMethod(cert.auth);
// Open authentication certificates are always authorized
if ( method === 'open' ) return true;
// Token based authentication requires valid token
if ( method === 'token' ) {
var found = cert.auth.valid_tokens.indexOf(auth);
//if ( found !== -1 ) {
//console.log("Token " + auth + " is found at position " + found + " in valid tokens " + cert.auth.valid_tokens);
// return true;
//} else return false;
return cert.auth.valid_tokens.indexOf(auth) !== -1;
}
throw new Error("Unsupported authentication method: " + cert.auth.method);
};
// Check if shown credential are authorized to access a map
// by the given signer.
//
// @param signer a signer name (cartodb username)
// @param map_id a layergroup_id
// @param auth an authentication token, or undefined if none
// (can still be authorized by signature)
//
// @param callback function(Error, Boolean)
//
o.isAuthorized = function(signer, map_id, auth, callback) {
var that = this;
var redisClient;
var db = that.db_signatures;
var authorized = false;
var certificate_id_list;
var missing_certificates = [];
if ( debug ) {
console.log("Check auth from signer '" + signer + "' on map '" + map_id + "' with auth '" + auth + "'");
}
Step(
function getRedisClient() {
that.redis_pool.acquire(db, this);
},
function getMapSignatures(err, client) {
if ( err ) throw err;
redisClient = client;
var map_sig_key = _.template(that.key_map_sig, {signer:signer, map_id:map_id});
redisClient.SMEMBERS(map_sig_key, this);
//that._redisCmd('SMEMBERS', [ map_sig_key ], this);
},
function getCertificates(err, crt_lst) {
if ( err ) throw err;
if ( debug ) {
console.log("Map '" + map_id + "' is signed by " + crt_lst.length + " certificates of user '" + signer);
}
certificate_id_list = crt_lst;
if ( ! crt_lst.length ) {
// No certs, avoid calling redis with short args list.
// Next step expects a list of certificate values so
// we directly send the empty list.
return crt_lst;
}
var map_crt_key = _.template(that.key_map_crt, {signer:signer});
//that._redisCmd('HMGET', [ map_crt_key ].concat(crt_lst), this);
redisClient.HMGET(map_crt_key, crt_lst, this);
},
function checkCertificates(err, certs) {
if ( err ) throw err;
for (var i=0; i<certs.length; ++i) {
var crt_id = certificate_id_list[i];
if ( _.isNull(certs[i]) ) {
missing_certificates.push(crt_id);
continue;
}
var cert;
try {
//console.log("cert " + crt_id + ": " + certs[i]);
cert = JSON.parse(certs[i]);
authorized = that.authorizedByCert(cert, auth);
} catch (err) {
console.log("Certificate " + certificate_id_list[i] + " by user '" + signer + "' is malformed: " + err);
continue;
}
if ( authorized ) {
if ( debug ) {
console.log("Access to map '" + map_id + "' authorized by cert '"
+ certificate_id_list[i] + "' of user '" + signer + "'");
}
//console.dir(cert);
break; // no need to further check certs
}
}
return null;
},
function finish(err) {
if ( missing_certificates.length ) {
console.log("WARNING: map '" + map_id + "' is signed by '" + signer
+ "' with " + missing_certificates.length
+ " missing certificates: "
+ missing_certificates + " (TODO: give cleanup instructions)");
}
if ( redisClient ) that.redis_pool.release(db, redisClient);
callback(err, authorized);
}
);
};
// Add an authorization certificate from a user.
//
// @param signer a signer name (cartodb username)
// @param cert certificate object, see
// http://github.com/CartoDB/Windshaft-cartodb/wiki/Signed-maps
//
// @param callback function(err, crt_id) return certificate id
//
// TODO: allow for requesting error when certificate already exists ?
//
o.addCertificate = function(signer, cert, callback) {
var crt_val = JSON.stringify(cert);
var crt_id = crypto.createHash('md5').update(crt_val).digest('hex');
var usr_crt_key = _.template(this.key_map_crt, {signer:signer});
this._redisCmd('HSET', [ usr_crt_key, crt_id, crt_val ], function(err, created) {
// NOTE: created would be 0 if the field already existed, 1 otherwise
callback(err, crt_id);
});
};
// Remove an authorization certificate of a user, also removing
// any signature made with the certificate.
//
// @param signer a signer name (cartodb username)
// @param crt_id certificate identifier, as returned by addCertificate
// @param callback function(err)
//
o.delCertificate = function(signer, crt_id, callback) {
var db = this.db_signatures;
var crt_sig_key = _.template(this.key_crt_sig, {signer:signer, crt_id:crt_id});
var signed_map_list;
var redis_client;
var that = this;
Step (
function getRedisClient() {
that._acquireRedis(this);
},
function removeCertificate(err, data) {
if ( err ) throw err;
redis_client = data;
// Remove the certificate (would be enough to stop authorizing uses)
var usr_crt_key = _.template(that.key_map_crt, {signer:signer});
redis_client.HDEL(usr_crt_key, crt_id, this);
},
function getMapSignatures(err, deleted) {
if ( err ) throw err;
if ( ! deleted ) {
// debugging (how can this be possible?)
console.log("WARNING: authorization certificate '" + crt_id
+ "' by user '" + signer + "' did not exist on delete request");
}
// Get all signatures by this certificate
redis_client.SMEMBERS(crt_sig_key, this);
},
function delMapSignaturesReference(err, map_id_list) {
if ( err ) throw err;
signed_map_list = map_id_list;
if ( debug ) {
console.log("Certificate '" + crt_id + "' from user '" + signer
+ "' was used to sign " + signed_map_list.length + " maps");
}
redis_client.DEL(crt_sig_key, this);
},
function delMapSignatures(err) {
if ( err ) throw err;
var crt_sig_key = _.template(that.key_crt_sig, {signer:signer, crt_id:crt_id});
var tx = redis_client.MULTI();
for (var i=0; i<signed_map_list.length; ++i) {
var map_id = signed_map_list[i];
var map_sig_key = _.template(that.key_map_sig, {signer:signer, map_id:map_id});
//console.log("Queuing removal of '" + crt_id + "' from '" + map_sig_key + "'");
tx.SREM( map_sig_key, crt_id )
}
tx.EXEC(this);
},
function reportTransaction(err, rets) {
if ( err ) throw err;
if ( debug ) {
for (var i=0; i<signed_map_list.length; ++i) {
var ret = rets[i];
if ( ! ret ) {
console.log("No signature with certificate '" + crt_id
+ "' of user '" + signer + "' found in map '"
+ signed_map_list[i] + "'");
} else {
console.log("Signature with certificate '" + crt_id
+ "' of user '" + signer + "' removed from map '"
+ signed_map_list[i] + "'");
}
}
}
return null;
},
function finish(err) {
if ( ! _.isUndefined(redis_client) ) {
that._releaseRedis(redis_client);
}
callback(err);
}
);
};
// Sign a map with a certificate reference
//
// @param signer a signer name (cartodb username)
// @param map_id a layergroup_id
// @param crt_id signature certificate identifier
//
// @param callback function(Error)
//
o.signMap = function(signer, map_id, crt_id, callback) {
var that = this;
Step(
function addMapSignature() {
var map_sig_key = _.template(that.key_map_sig, {signer:signer, map_id:map_id});
if ( debug ) {
console.log("Adding " + crt_id + " to " + map_sig_key);
}
that._redisCmd('SADD', [ map_sig_key, crt_id ], this);
},
function addCertificateUsage(err) {
// Add the map to the set of maps signed by the given cert
if ( err ) throw err;
var crt_sig_key = _.template(that.key_crt_sig, {signer:signer, crt_id:crt_id});
that._redisCmd('SADD', [ crt_sig_key, map_id ], this);
},
function finish(err) {
callback(err);
}
);
};
// Sign a map with a full certificate
//
// @param signer a signer name (cartodb username)
// @param map_id a layergroup_id
// @param cert_id signature certificate identifier
//
// @param callback function(Error, String) return certificate id
//
o.addSignature = function(signer, map_id, cert, callback) {
var that = this;
var certificate_id;
Step(
function addCertificate() {
that.addCertificate(signer, cert, this);
},
function signMap(err, cert_id) {
if ( err ) throw err;
if ( ! cert_id ) throw new Error("addCertificate returned no certificate id");
certificate_id = cert_id;
that.signMap(signer, map_id, cert_id, this);
},
function finish(err) {
callback(err, certificate_id);
}
);
};
module.exports = SignedMaps;

View File

@@ -0,0 +1,615 @@
var crypto = require('crypto');
var Step = require('step');
var _ = require('underscore');
// Templates in this hash (keyed as <username>@<template_name>)
// are being worked on.
var user_template_locks = {};
// Class handling map templates
//
// See http://github.com/CartoDB/Windshaft-cartodb/wiki/Template-maps
//
// @param redis_pool an instance of a "redis-mpool"
// See https://github.com/CartoDB/node-redis-mpool
// Needs version 0.x.x of the API.
//
// @param signed_maps an instance of a "signed_maps" class,
// See signed_maps.js
//
// @param opts TemplateMap options. Supported elements:
// 'max_user_templates' limit on the number of per-user
//
//
function TemplateMaps(redis_pool, signed_maps, opts) {
this.redis_pool = redis_pool;
this.signed_maps = signed_maps;
this.opts = opts || {};
// Database containing templates
// TODO: allow configuring ?
// NOTE: currently it is the same as
// the one containing layergroups
this.db_signatures = 0;
//
// Map templates are owned by a user that specifies access permissions
// for their instances.
//
// We have the following datastores:
//
// 1. User teplates: set of per-user map templates
// NOTE: each template would have an associated auth
// reference, see signed_maps.js
// User templates (HASH:tpl_id->tpl_val)
this.key_usr_tpl = "map_tpl|<%= owner %>";
// User template locks (HASH:tpl_id->ctime)
this.key_usr_tpl_lck = "map_tpl|<%= owner %>|locks";
};
var o = TemplateMaps.prototype;
//--------------- PRIVATE METHODS --------------------------------
o._userTemplateLimit = function() {
return this.opts['max_user_templates'] || 0;
};
o._acquireRedis = function(callback) {
this.redis_pool.acquire(this.db_signatures, callback);
};
o._releaseRedis = function(client) {
this.redis_pool.release(this.db_signatures, client);
};
/**
* Internal function to communicate with redis
*
* @param redisFunc - the redis function to execute
* @param redisArgs - the arguments for the redis function in an array
* @param callback - function to pass results too.
*/
o._redisCmd = function(redisFunc, redisArgs, callback) {
var redisClient;
var that = this;
var db = that.db_signatures;
Step(
function getRedisClient() {
that.redis_pool.acquire(db, this);
},
function executeQuery(err, data) {
if ( err ) throw err;
redisClient = data;
redisArgs.push(this);
redisClient[redisFunc.toUpperCase()].apply(redisClient, redisArgs);
},
function releaseRedisClient(err, data) {
if ( ! _.isUndefined(redisClient) ) that.redis_pool.release(db, redisClient);
callback(err, data);
}
);
};
// @param callback function(err, obtained)
o._obtainTemplateLock = function(owner, tpl_id, callback) {
var usr_tpl_lck_key = _.template(this.key_usr_tpl_lck, {owner:owner});
var that = this;
var gotLock = false;
Step (
function obtainLock() {
var ctime = Date.now();
that._redisCmd('HSETNX', [usr_tpl_lck_key, tpl_id, ctime], this);
},
function checkLock(err, locked) {
if ( err ) throw err;
if ( ! locked ) {
// Already locked
// TODO: unlock if expired ?
throw new Error("Template '" + tpl_id + "' of user '" + owner + "' is locked");
}
return gotLock = true;
},
function finish(err) {
callback(err, gotLock);
}
);
};
// @param callback function(err, deleted)
o._releaseTemplateLock = function(owner, tpl_id, callback) {
var usr_tpl_lck_key = _.template(this.key_usr_tpl_lck, {owner:owner});
this._redisCmd('HDEL', [usr_tpl_lck_key, tpl_id], callback);
};
o._reValidIdentifier = /^[a-zA-Z][0-9a-zA-Z_]*$/;
o._checkInvalidTemplate = function(template) {
if ( template.version != '0.0.1' ) {
return new Error("Unsupported template version " + template.version);
}
var tplname = template.name;
if ( ! tplname ) {
return new Error("Missing template name");
}
if ( ! tplname.match(this._reValidIdentifier) ) {
return new Error("Invalid characters in template name '" + tplname + "'");
}
var phold = template.placeholders;
for (var k in phold) {
if ( ! k.match(this._reValidIdentifier) ) {
return new Error("Invalid characters in placeholder name '" + k + "'");
}
if ( ! phold[k].hasOwnProperty('default') ) {
return new Error("Missing default for placeholder '" + k + "'");
}
if ( ! phold[k].hasOwnProperty('type') ) {
return new Error("Missing type for placeholder '" + k + "'");
}
};
// Check certificate validity
var cert = this.getTemplateCertificate(template);
var err = this.signed_maps.checkInvalidCertificate(cert);
if ( err ) return err;
// TODO: run more checks over template format ?
};
//--------------- PUBLIC API -------------------------------------
// Extract a signature certificate from a template
//
// The certificate will be ready to be passed to
// SignedMaps.addCertificate or SignedMaps.authorizedByCert
//
o.getTemplateCertificate = function(template) {
var cert = {
version: '0.0.1',
template_id: template.name,
auth: template.auth
};
return cert;
};
// Add a template
//
// NOTE: locks user+template_name or fails
//
// @param owner cartodb username of the template owner
//
// @param template layergroup template, see
// http://github.com/CartoDB/Windshaft-cartodb/wiki/Template-maps#template-format
//
// @param callback function(err, tpl_id)
// Return template identifier (only valid for given user)
//
o.addTemplate = function(owner, template, callback) {
var invalidError = this._checkInvalidTemplate(template);
if ( invalidError ) {
callback(invalidError);
return;
}
var tplname = template.name;
// Procedure:
//
// - Check against limit
// 0. Obtain a lock for user+template_name, fail if impossible
// 1. Check no other template exists with the same name
// 2. Install certificate extracted from template, extending
// it to contain a name to properly salt things out.
// 3. Modify the template object to reference certificate by id
// 4. Install template
// 5. Release lock
//
//
var usr_tpl_key = _.template(this.key_usr_tpl, {owner:owner});
var gotLock = false;
var that = this;
var limit = that._userTemplateLimit();
Step(
function checkLimit() {
if ( ! limit ) return 0;
that._redisCmd('HLEN', [ usr_tpl_key ], this);
},
// try to obtain a lock
function obtainLock(err, len) {
if ( err ) throw err;
if ( limit && len >= limit ) {
throw new Error("User '" + owner + "' reached limit on number of templates (" + len + "/" + limit + ")");
}
that._obtainTemplateLock(owner, tplname, this);
},
function getExistingTemplate(err, locked) {
if ( err ) throw err;
if ( ! locked ) {
// Already locked
throw new Error("Template '" + tplname + "' of user '" + owner + "' is locked");
}
gotLock = true;
that._redisCmd('HEXISTS', [ usr_tpl_key, tplname ], this);
},
function installCertificate(err, exists) {
if ( err ) throw err;
if ( exists ) {
throw new Error("Template '" + tplname + "' of user '" + owner + "' already exists");
}
var cert = that.getTemplateCertificate(template);
that.signed_maps.addCertificate(owner, cert, this);
},
function installTemplate(err, crt_id) {
if ( err ) throw err;
template.auth_id = crt_id;
var tpl_val = JSON.stringify(template);
that._redisCmd('HSET', [ usr_tpl_key, tplname, tpl_val ], this);
},
function releaseLock(err, newfield) {
if ( ! err && ! newfield ) {
console.log("ERROR: addTemplate overridden existing template '"
+ tplname + "' of '" + owner
+ "' -- HSET returned " + overridden + ": someone added it without locking ?");
// TODO: how to recover this ?!
}
if ( err && ! gotLock ) throw err;
// release the lock
var next = this;
that._releaseTemplateLock(owner, tplname, function(e, d) {
if ( e ) {
console.log("Error removing lock on template '" + tplname
+ "' of user '" + owner + "': " + e);
} else if ( ! d ) {
console.log("ERROR: lock on template '" + tplname
+ "' of user '" + owner + "' externally removed during insert!");
}
next(err);
});
},
function finish(err) {
callback(err, tplname);
}
);
};
// Delete a template
//
// NOTE: locks user+template_name or fails
//
// Also deletes associated authentication certificate, which
// in turn deletes all instance signatures
//
// @param owner cartodb username of the template owner
//
// @param tpl_id template identifier as returned
// by addTemplate or listTemplates
//
// @param callback function(err)
//
o.delTemplate = function(owner, tpl_id, callback) {
var usr_tpl_key = _.template(this.key_usr_tpl, {owner:owner});
var gotLock = false;
var that = this;
Step(
// try to obtain a lock
function obtainLock() {
that._obtainTemplateLock(owner, tpl_id, this);
},
function getExistingTemplate(err, locked) {
if ( err ) throw err;
if ( ! locked ) {
// Already locked
throw new Error("Template '" + tpl_id + "' of user '" + owner + "' is locked");
}
gotLock = true;
that._redisCmd('HGET', [ usr_tpl_key, tpl_id ], this);
},
function delCertificate(err, tplval) {
if ( err ) throw err;
if ( ! tplval ) {
throw new Error("Template '" + tpl_id + "' of user '" + owner + "' does not exist");
}
var tpl = JSON.parse(tplval);
if ( ! tpl.auth_id ) {
// not sure this is an error, in case we'll ever
// allow unsigned templates...
console.log("ERROR: installed template '" + tpl_id
+ "' of user '" + owner + "' has no auth_id reference: "); console.dir(tpl);
return null;
}
var next = this;
that.signed_maps.delCertificate(owner, tpl.auth_id, function(err) {
if ( err ) {
var msg = "ERROR: could not delete certificate '"
+ tpl.auth_id + "' associated with template '"
+ tpl_id + "' of user '" + owner + "': " + err;
// I'm actually not sure we want this event to be fatal
// (avoiding a deletion of the template itself)
next(new Error(msg));
} else {
next();
}
});
},
function delTemplate(err) {
if ( err ) throw err;
that._redisCmd('HDEL', [ usr_tpl_key, tpl_id ], this);
},
function releaseLock(err, deleted) {
if ( ! err && ! deleted ) {
console.log("ERROR: template '" + tpl_id
+ "' of user '" + owner + "' externally removed during delete!");
}
if ( ! gotLock ) {
if ( err ) throw err;
return null;
}
// release the lock
var next = this;
that._releaseTemplateLock(owner, tpl_id, function(e, d) {
if ( e ) {
console.log("Error removing lock on template '" + tpl_id
+ "' of user '" + owner + "': " + e);
} else if ( ! d ) {
console.log("ERROR: lock on template '" + tpl_id
+ "' of user '" + owner + "' externally removed during delete!");
}
next(err);
});
},
function finish(err) {
callback(err);
}
);
};
// Update a template
//
// NOTE: locks user+template_name or fails
//
// Also deletes and re-creates associated authentication certificate,
// which in turn deletes all instance signatures
//
// @param owner cartodb username of the template owner
//
// @param tpl_id template identifier as returned by addTemplate
//
// @param template layergroup template, see
// http://github.com/CartoDB/Windshaft-cartodb/wiki/Template-maps#template-format
//
// @param callback function(err)
//
o.updTemplate = function(owner, tpl_id, template, callback) {
var invalidError = this._checkInvalidTemplate(template);
if ( invalidError ) {
callback(invalidError);
return;
}
var tplname = template.name;
if ( tpl_id != tplname ) {
callback(new Error("Cannot update name of a map template ('" + tpl_id + "' != '" + tplname + "')"));
return;
}
var usr_tpl_key = _.template(this.key_usr_tpl, {owner:owner});
var gotLock = false;
var that = this;
Step(
// try to obtain a lock
function obtainLock() {
that._obtainTemplateLock(owner, tpl_id, this);
},
function getExistingTemplate(err, locked) {
if ( err ) throw err;
if ( ! locked ) {
// Already locked
throw new Error("Template '" + tpl_id + "' of user '" + owner + "' is locked");
}
gotLock = true;
that._redisCmd('HGET', [ usr_tpl_key, tpl_id ], this);
},
function delOldCertificate(err, tplval) {
if ( err ) throw err;
if ( ! tplval ) {
throw new Error("Template '" + tpl_id + "' of user '"
+ owner +"' does not exist");
}
var tpl = JSON.parse(tplval);
if ( ! tpl.auth_id ) {
// not sure this is an error, in case we'll ever
// allow unsigned templates...
console.log("ERROR: installed template '" + tpl_id
+ "' of user '" + owner + "' has no auth_id reference: "); console.dir(tpl);
return null;
}
var next = this;
that.signed_maps.delCertificate(owner, tpl.auth_id, function(err) {
if ( err ) {
var msg = "ERROR: could not delete certificate '"
+ tpl.auth_id + "' associated with template '"
+ tpl_id + "' of user '" + owner + "': " + err;
// I'm actually not sure we want this event to be fatal
// (avoiding a deletion of the template itself)
next(new Error(msg));
} else {
next();
}
});
},
function installNewCertificate(err) {
if ( err ) throw err;
var cert = that.getTemplateCertificate(template);
that.signed_maps.addCertificate(owner, cert, this);
},
function updTemplate(err, crt_id) {
if ( err ) throw err;
template.auth_id = crt_id;
var tpl_val = JSON.stringify(template);
that._redisCmd('HSET', [ usr_tpl_key, tplname, tpl_val ], this);
},
function releaseLock(err, newfield) {
if ( ! err && newfield ) {
console.log("ERROR: template '" + tpl_id
+ "' of user '" + owner + "' externally removed during update!");
}
if ( ! gotLock ) {
if ( err ) throw err;
return null;
}
// release the lock
var next = this;
that._releaseTemplateLock(owner, tpl_id, function(e, d) {
if ( e ) {
console.log("Error removing lock on template '" + tpl_id
+ "' of user '" + owner + "': " + e);
} else if ( ! d ) {
console.log("ERROR: lock on template '" + tpl_id
+ "' of user '" + owner + "' externally removed during update!");
}
next(err);
});
},
function finish(err) {
callback(err);
}
);
};
// List user templates
//
// @param owner cartodb username of the templates owner
//
// @param callback function(err, tpl_id_list)
// Returns a list of template identifiers
//
o.listTemplates = function(owner, callback) {
var usr_tpl_key = _.template(this.key_usr_tpl, {owner:owner});
this._redisCmd('HKEYS', [ usr_tpl_key ], callback);
};
// Get a templates
//
// @param owner cartodb username of the template owner
//
// @param tpl_id template identifier as returned
// by addTemplate or listTemplates
//
// @param callback function(err, template)
// Return full template definition
//
o.getTemplate = function(owner, tpl_id, callback) {
var usr_tpl_key = _.template(this.key_usr_tpl, {owner:owner});
var that = this;
Step(
function getTemplate() {
that._redisCmd('HGET', [ usr_tpl_key, tpl_id ], this);
},
function parseTemplate(err, tpl_val) {
if ( err ) throw err;
var tpl = JSON.parse(tpl_val);
// Should we strip auth_id ?
return tpl;
},
function finish(err, tpl) {
callback(err, tpl);
}
);
};
// Perform placeholder substitutions on a template
//
// @param template a template object (will not be modified)
//
// @param params an object containing named subsitution parameters
// Only the ones found in the template's placeholders object
// will be used, with missing ones taking default values.
//
// @returns a layergroup configuration
//
// @throws Error on malformed template or parameter
//
o._reNumber = /^([-+]?[\d\.]?\d+([eE][+-]?\d+)?)$/;
o._reCSSColorName = /^[a-zA-Z]+$/;
o._reCSSColorVal = /^#[0-9a-fA-F]{3,6}$/;
o._replaceVars = function(str, params) {
//return _.template(str, params); // lazy way, possibly dangerous
// Construct regular expressions for each param
if ( ! params._re ) {
params._re = {};
for (var k in params) {
params._re[k] = RegExp("<%=\\s*" + k + "\\s*%>", "g");
}
}
for (var k in params) str = str.replace(params._re[k], params[k]);
return str;
};
o.instance = function(template, params) {
var all_params = {};
var phold = template.placeholders;
for (var k in phold) {
var val = params.hasOwnProperty(k) ? params[k] : phold[k].default;
var type = phold[k].type;
// properly escape
if ( type === 'sql_literal' ) {
// duplicate any single-quote
val = val.replace(/'/g, "''");
}
else if ( type === 'sql_ident' ) {
// duplicate any double-quote
val = val.replace(/"/g, '""');
}
else if ( type === 'number' ) {
// check it's a number
if ( typeof(val) !== 'number' && ! val.match(this._reNumber) ) {
throw new Error("Invalid number value for template parameter '"
+ k + "': " + val);
}
}
else if ( type === 'css_color' ) {
// check it only contains letters or
// starts with # and only contains hexdigits
if ( ! val.match(this._reCSSColorName) && ! val.match(this._reCSSColorVal) ) {
throw new Error("Invalid css_color value for template parameter '"
+ k + "': " + val);
}
}
else {
// NOTE: should be checked at template create/update time
throw new Error("Invalid placeholder type '" + type + "'");
}
all_params[k] = val;
}
// NOTE: we're deep-cloning the layergroup here
var layergroup = JSON.parse(JSON.stringify(template.layergroup));
for (var i=0; i<layergroup.layers.length; ++i) {
var lyropt = layergroup.layers[i].options;
if ( lyropt.cartocss ) lyropt.cartocss = this._replaceVars(lyropt.cartocss, all_params);
if ( lyropt.sql) lyropt.sql = this._replaceVars(lyropt.sql, all_params);
// Anything else ?
}
return layergroup;
};
// Return a fingerPrint of the object
o.fingerPrint = function(template) {
return crypto.createHash('md5')
.update(JSON.stringify(template))
.digest('hex')
;
};
module.exports = TemplateMaps;

685
npm-shrinkwrap.json generated
View File

@@ -1,261 +1,296 @@
{
"name": "windshaft-cartodb",
"version": "0.2.0-dev",
"version": "1.10.0",
"dependencies": {
"cluster2": {
"version": "0.3.5",
"node-varnish": {
"version": "0.3.0",
"from": "http://github.com/Vizzuality/node-varnish/tarball/0.3.0"
},
"underscore": {
"version": "1.3.3"
},
"windshaft": {
"version": "0.20.0",
"from": "http://github.com/CartoDB/Windshaft/tarball/0.20.0",
"dependencies": {
"express": {
"version": "2.5.11",
"grainstore": {
"version": "0.18.1",
"dependencies": {
"connect": {
"version": "1.9.2",
"carto": {
"version": "0.9.5-cdb2",
"from": "http://github.com/CartoDB/carto/tarball/0.9.5-cdb2",
"dependencies": {
"formidable": {
"version": "1.0.11"
}
}
},
"mime": {
"version": "1.2.4"
},
"qs": {
"version": "0.4.2"
},
"mkdirp": {
"version": "0.3.0"
}
}
},
"ejs": {
"version": "0.8.3"
},
"npm": {
"version": "1.1.62",
"dependencies": {
"semver": {
"version": "1.0.14"
},
"ini": {
"version": "1.0.4"
},
"slide": {
"version": "1.1.3"
},
"abbrev": {
"version": "1.0.3"
},
"graceful-fs": {
"version": "1.1.14"
},
"minimatch": {
"version": "0.2.6"
},
"nopt": {
"version": "2.0.0"
},
"rimraf": {
"version": "2.0.2"
},
"request": {
"version": "2.9.203",
"from": "git://github.com/isaacs/request"
},
"which": {
"version": "1.0.5"
},
"tar": {
"version": "0.1.13"
},
"fstream": {
"version": "0.1.19"
},
"block-stream": {
"version": "0.0.6"
},
"inherits": {
"version": "1.0.0",
"from": "git://github.com/isaacs/inherits"
},
"mkdirp": {
"version": "0.3.4"
},
"read": {
"version": "1.0.4",
"dependencies": {
"mute-stream": {
"version": "0.0.3"
}
}
},
"lru-cache": {
"version": "2.0.4"
},
"node-gyp": {
"version": "0.6.11"
},
"fstream-npm": {
"version": "0.1.2",
"dependencies": {
"fstream-ignore": {
"version": "0.0.5"
}
}
},
"uid-number": {
"version": "0.0.3"
},
"archy": {
"version": "0.0.2"
},
"chownr": {
"version": "0.0.1"
},
"npmlog": {
"version": "0.0.2"
},
"ansi": {
"version": "0.1.2"
},
"npm-registry-client": {
"version": "0.2.7"
},
"read-package-json": {
"version": "0.1.5"
},
"read-installed": {
"version": "0.0.2"
},
"glob": {
"version": "3.1.12"
},
"init-package-json": {
"version": "0.0.5",
"dependencies": {
"promzard": {
"version": "0.2.0"
}
}
},
"osenv": {
"version": "0.0.3"
},
"lockfile": {
"version": "0.2.1"
},
"retry": {
"version": "0.6.0"
},
"couch-login": {
"version": "0.1.12"
},
"once": {
"version": "1.1.1"
},
"npmconf": {
"version": "0.0.16",
"dependencies": {
"config-chain": {
"version": "1.1.2",
"underscore": {
"version": "1.4.4"
},
"xml2js": {
"version": "0.2.8",
"dependencies": {
"proto-list": {
"version": "1.2.2"
"sax": {
"version": "0.5.8"
}
}
},
"optimist": {
"version": "0.6.1",
"dependencies": {
"wordwrap": {
"version": "0.0.2"
},
"minimist": {
"version": "0.0.8"
}
}
}
}
},
"opener": {
"version": "1.3.0"
}
}
}
}
},
"node-varnish": {
"version": "0.1.1"
},
"underscore": {
"version": "1.1.7"
},
"grainstore": {
"version": "0.6.4",
"dependencies": {
"carto": {
"version": "0.8.2-cdb-dev-3",
"from": "git://github.com/CartoDB/carto.git#cdb-0.8",
"dependencies": {
"underscore": {
"version": "1.3.3"
},
"mapnik-reference": {
"version": "4.0.5"
"version": "5.0.7"
},
"xml2js": {
"version": "0.1.14",
"millstone": {
"version": "0.6.11",
"dependencies": {
"sax": {
"version": "0.4.2"
}
}
}
}
},
"millstone": {
"version": "0.5.10-cdb-01",
"from": "git://github.com/CartoDB/millstone.git#cdb-node04-devel",
"dependencies": {
"underscore": {
"version": "1.3.3"
},
"request": {
"version": "2.11.4",
"dependencies": {
"form-data": {
"version": "0.0.3",
"underscore": {
"version": "1.5.2"
},
"request": {
"version": "2.26.0",
"dependencies": {
"combined-stream": {
"version": "0.0.3",
"qs": {
"version": "0.6.6"
},
"json-stringify-safe": {
"version": "5.0.0"
},
"forever-agent": {
"version": "0.5.2"
},
"tunnel-agent": {
"version": "0.3.0"
},
"http-signature": {
"version": "0.10.0",
"dependencies": {
"delayed-stream": {
"version": "0.0.5"
"assert-plus": {
"version": "0.1.2"
},
"asn1": {
"version": "0.1.11"
},
"ctype": {
"version": "0.5.2"
}
}
},
"async": {
"version": "0.1.9"
"hawk": {
"version": "1.0.0",
"dependencies": {
"hoek": {
"version": "0.9.1"
},
"boom": {
"version": "0.4.2"
},
"cryptiles": {
"version": "0.2.2"
},
"sntp": {
"version": "0.2.4"
}
}
},
"aws-sign": {
"version": "0.3.0"
},
"oauth-sign": {
"version": "0.3.0"
},
"cookie-jar": {
"version": "0.3.0"
},
"node-uuid": {
"version": "1.4.1"
},
"form-data": {
"version": "0.1.2",
"dependencies": {
"combined-stream": {
"version": "0.0.4",
"dependencies": {
"delayed-stream": {
"version": "0.0.5"
}
}
},
"async": {
"version": "0.2.10"
}
}
}
}
},
"srs": {
"version": "0.3.11"
},
"zipfile": {
"version": "0.4.3"
},
"sqlite3": {
"version": "2.2.0",
"dependencies": {
"node-pre-gyp": {
"version": "0.2.6",
"dependencies": {
"nopt": {
"version": "2.1.2",
"dependencies": {
"abbrev": {
"version": "1.0.4"
}
}
},
"npmlog": {
"version": "0.0.6",
"dependencies": {
"ansi": {
"version": "0.2.1"
}
}
},
"semver": {
"version": "2.1.0"
},
"tar": {
"version": "0.1.19",
"dependencies": {
"inherits": {
"version": "2.0.1"
},
"block-stream": {
"version": "0.0.7"
},
"fstream": {
"version": "0.1.25",
"dependencies": {
"graceful-fs": {
"version": "2.0.3"
}
}
}
}
},
"tar-pack": {
"version": "2.0.0",
"dependencies": {
"uid-number": {
"version": "0.0.3"
},
"once": {
"version": "1.1.1"
},
"debug": {
"version": "0.7.4"
},
"fstream": {
"version": "0.1.25",
"dependencies": {
"graceful-fs": {
"version": "2.0.3"
},
"inherits": {
"version": "2.0.1"
}
}
},
"fstream-ignore": {
"version": "0.0.7",
"dependencies": {
"minimatch": {
"version": "0.2.14",
"dependencies": {
"sigmund": {
"version": "1.0.0"
}
}
},
"inherits": {
"version": "2.0.1"
}
}
},
"readable-stream": {
"version": "1.0.26-2",
"dependencies": {
"string_decoder": {
"version": "0.10.25-1"
}
}
},
"graceful-fs": {
"version": "1.2.3"
}
}
},
"aws-sdk": {
"version": "2.0.0-rc9",
"dependencies": {
"xml2js": {
"version": "0.2.4",
"dependencies": {
"sax": {
"version": "0.6.0"
}
}
},
"xmlbuilder": {
"version": "0.4.2"
}
}
},
"rc": {
"version": "0.3.4",
"dependencies": {
"minimist": {
"version": "0.0.8"
},
"deep-extend": {
"version": "0.2.8"
},
"ini": {
"version": "1.1.0"
}
}
},
"rimraf": {
"version": "2.2.6"
}
}
}
}
},
"mime": {
"version": "1.2.7"
"version": "1.2.11"
},
"mkdirp": {
"version": "0.3.5"
},
"optimist": {
"version": "0.6.1",
"dependencies": {
"wordwrap": {
"version": "0.0.2"
},
"minimist": {
"version": "0.0.8"
}
}
}
}
},
"srs": {
"version": "0.2.16"
},
"zipfile": {
"version": "0.3.2"
},
"sqlite3": {
"version": "2.1.5"
},
"mime": {
"version": "1.2.7"
},
"mkdirp": {
"version": "0.3.4"
}
}
}
}
},
"windshaft": {
"version": "0.5.8",
"dependencies": {
"underscore": {
"version": "1.3.3"
},
"generic-pool": {
"version": "2.0.4"
},
"express": {
"version": "2.5.11",
@@ -264,7 +299,7 @@
"version": "1.9.2",
"dependencies": {
"formidable": {
"version": "1.0.11"
"version": "1.0.14"
}
}
},
@@ -280,10 +315,10 @@
}
},
"tilelive": {
"version": "4.3.1",
"version": "4.4.3",
"dependencies": {
"optimist": {
"version": "0.3.4",
"version": "0.3.7",
"dependencies": {
"wordwrap": {
"version": "0.0.2"
@@ -296,59 +331,203 @@
}
},
"tilelive-mapnik": {
"version": "0.3.3-dev",
"from": "git://github.com/Vizzuality/tilelive-mapnik.git#7df70554",
"version": "0.6.8",
"dependencies": {
"mapnik": {
"version": "0.7.14"
},
"eio": {
"version": "0.1.0"
"version": "0.2.2"
},
"mime": {
"version": "1.2.11"
},
"sphericalmercator": {
"version": "1.0.2"
}
}
},
"lru-cache": {
"version": "2.3.1"
},
"carto": {
"version": "0.9.5-cdb3",
"from": "http://github.com/CartoDB/carto/tarball/0.9.5-cdb3",
"dependencies": {
"underscore": {
"version": "1.4.4"
},
"mapnik-reference": {
"version": "5.0.7"
},
"xml2js": {
"version": "0.2.8",
"dependencies": {
"sax": {
"version": "0.5.8"
}
}
},
"optimist": {
"version": "0.6.1",
"dependencies": {
"wordwrap": {
"version": "0.0.2"
},
"minimist": {
"version": "0.0.8"
}
}
}
}
},
"step-profiler": {
"version": "0.0.1",
"from": "git://github.com/CartoDB/node-step-profiler.git#0.0.1"
},
"underscore.string": {
"version": "1.1.6",
"dependencies": {
"underscore": {
"version": "1.1.7"
}
}
},
"pg": {
"version": "2.6.2",
"dependencies": {
"generic-pool": {
"version": "2.0.3"
},
"buffer-writer": {
"version": "1.0.0"
}
}
},
"torque.js": {
"version": "2.2.00"
},
"node-statsd": {
"version": "0.0.7"
}
}
},
"step": {
"version": "0.0.5"
},
"generic-pool": {
"version": "1.0.12"
},
"redis": {
"version": "0.7.2"
},
"hiredis": {
"version": "0.1.14"
},
"request": {
"version": "2.9.202"
},
"cartodb-redis": {
"version": "0.3.0"
},
"redis-mpool": {
"version": "0.0.4",
"from": "http://github.com/CartoDB/node-redis-mpool/tarball/0.0.4",
"dependencies": {
"generic-pool": {
"version": "2.0.4"
},
"hiredis": {
"version": "0.1.16",
"dependencies": {
"bindings": {
"version": "1.1.1"
}
}
}
}
},
"mapnik": {
"version": "0.7.26-cdb1",
"from": "http://github.com/Vizzuality/node-mapnik/tarball/0.7.26-cdb1"
},
"lzma": {
"version": "1.2.3"
},
"log4js": {
"version": "0.6.10",
"dependencies": {
"async": {
"version": "0.1.15"
},
"readable-stream": {
"version": "1.0.26",
"dependencies": {
"string_decoder": {
"version": "0.10.25-1"
}
}
}
}
},
"rollbar": {
"version": "0.3.1",
"dependencies": {
"node-uuid": {
"version": "1.4.1"
},
"lru-cache": {
"version": "2.2.4"
}
}
},
"strftime": {
"version": "0.6.2"
},
"semver": {
"version": "1.1.4"
},
"redis": {
"version": "0.8.6"
},
"mocha": {
"version": "1.2.1",
"version": "1.14.0",
"dependencies": {
"commander": {
"version": "0.6.1"
"version": "2.0.0"
},
"growl": {
"version": "1.5.1"
"version": "1.7.0"
},
"jade": {
"version": "0.26.3",
"dependencies": {
"commander": {
"version": "0.6.1"
},
"mkdirp": {
"version": "0.3.0"
}
}
},
"diff": {
"version": "1.0.2"
"version": "1.0.7"
},
"debug": {
"version": "0.7.0"
"version": "0.7.4"
},
"mkdirp": {
"version": "0.3.5"
},
"glob": {
"version": "3.2.3",
"dependencies": {
"minimatch": {
"version": "0.2.14",
"dependencies": {
"lru-cache": {
"version": "2.5.0"
},
"sigmund": {
"version": "1.0.0"
}
}
},
"graceful-fs": {
"version": "2.0.2"
},
"inherits": {
"version": "2.0.1"
}
}
}
}
}

View File

@@ -1,36 +1,44 @@
{
"private": true,
"name": "windshaft-cartodb",
"version": "1.0.1",
"version": "1.10.1",
"description": "A map tile server for CartoDB",
"url": "https://github.com/Vizzuality/Windshaft-cartodb",
"keywords": [
"cartodb"
],
"url": "https://github.com/CartoDB/Windshaft-cartodb",
"licenses": [{
"type": "BSD",
"url": "https://github.com/Vizzuality/Windshaft-cartodb/blob/master/LICENCE"
"url": "https://github.com/CartoDB/Windshaft-cartodb/blob/master/LICENCE"
}],
"repositories": [{
"repository": {
"type": "git",
"url": "git://github.com/Vizzuality/Windshaft-cartodb.git"
}],
"author": {
"name": "Simon Tokumine, Javi Santana, Vizzuality",
"url": "http://vizzuality.com",
"email": "simon@vizzuality.com"
"url": "git://github.com/CartoDB/Windshaft-cartodb.git"
},
"author": "Vizzuality <contact@vizzuality.com> (http://vizzuality.com)",
"contributors": [
"Simon Tokumine <simon@vizzuality.com>",
"Javi Santana <jsantana@vizzuality.com>",
"Sandro Santilli <strk@vizzuality.com>"
],
"dependencies": {
"cluster2": "git://github.com/CartoDB/cluster2.git#28cde11",
"node-varnish": "0.1.1",
"underscore" : "1.1.x",
"grainstore" : "~0.6.2",
"windshaft" : "~0.5.8",
"node-varnish": "http://github.com/Vizzuality/node-varnish/tarball/0.3.0",
"underscore" : "~1.3.3",
"windshaft" : "http://github.com/CartoDB/Windshaft/tarball/0.20.0",
"step": "0.0.x",
"generic-pool": "1.0.x",
"redis": "0.7.2",
"hiredis": "~0.1.14",
"request": "2.9.202"
"request": "2.9.202",
"cartodb-redis": "~0.3.0",
"redis-mpool": "http://github.com/CartoDB/node-redis-mpool/tarball/0.0.4",
"mapnik": "http://github.com/Vizzuality/node-mapnik/tarball/0.7.26-cdb1",
"lzma": "~1.2.3",
"log4js": "~0.6.10",
"rollbar": "~0.3.1"
},
"devDependencies": {
"mocha": "1.2.1"
"mocha": "1.14.0",
"redis": "~0.8.3",
"strftime": "~0.6.0",
"semver": "~1.1.0"
},
"scripts": {
"test": "make check"

View File

@@ -1,11 +1,33 @@
#!/bin/sh
# Must match config.redis_pool.port in test/support/config.js
REDIS_PORT=6333
OPT_CREATE_REDIS=yes # create the redis test environment
OPT_CREATE_PGSQL=yes # create the PostgreSQL test environment
OPT_DROP_REDIS=yes # drop the redis test environment
OPT_DROP_PGSQL=yes # drop the PostgreSQL test environment
cd $(dirname $0)
BASEDIR=$(pwd)
cd -
REDIS_PORT=`node -e "console.log(require('${BASEDIR}/config/environments/test.js').redis.port)"`
export REDIS_PORT
cleanup() {
echo "Cleaning up"
kill ${PID_REDIS}
if test x"$OPT_DROP_REDIS" = xyes; then
if test x"$PID_REDIS" = x; then
PID_REDIS=$(cat ${BASEDIR}/redis.pid)
if test x"$PID_REDIS" = x; then
echo "Could not find a test redis pid to kill it"
return;
fi
fi
echo "Killing test redis pid ${PID_REDIS}"
kill ${PID_REDIS}
fi
if test x"$OPT_DROP_PGSQL" = xyes; then
# TODO: drop postgresql ?
echo "Dropping PostgreSQL test database isn't implemented yet"
fi
}
cleanup_and_exit() {
@@ -22,21 +44,76 @@ die() {
trap 'cleanup_and_exit' 1 2 3 5 9 13
echo "Starting redis on port ${REDIS_PORT}"
echo "port ${REDIS_PORT}" | redis-server - > test.log &
PID_REDIS=$!
while [ -n "$1" ]; do
# This is kept for backward compatibility
if test "$1" = "--nodrop"; then
OPT_DROP_REDIS=no
OPT_DROP_PGSQL=no
shift
continue
elif test "$1" = "--nodrop-pg"; then
OPT_DROP_PGSQL=no
shift
continue
elif test "$1" = "--nodrop-redis"; then
OPT_DROP_REDIS=no
shift
continue
elif test "$1" = "--nocreate-pg"; then
OPT_CREATE_PGSQL=no
shift
continue
elif test "$1" = "--nocreate-redis"; then
OPT_CREATE_REDIS=no
shift
continue
# This is kept for backward compatibility
elif test "$1" = "--nocreate"; then
OPT_CREATE_REDIS=no
OPT_CREATE_PGSQL=no
shift
continue
else
break
fi
done
echo "Preparing the database"
cd test/support; sh prepare_db.sh >> test.log || die "database preparation failure (see test.log)"; cd -;
if [ -z "$1" ]; then
echo "Usage: $0 [<options>] <test> [<test>]" >&2
echo "Options:" >&2
echo " --nocreate do not create the test environment on start" >&2
echo " --nodrop do not drop the test environment on exit" >&2
exit 1
fi
TESTS=$@
if test x"$OPT_CREATE_REDIS" = xyes; then
echo "Starting redis on port ${REDIS_PORT}"
echo "port ${REDIS_PORT}" | redis-server - > ${BASEDIR}/test.log &
PID_REDIS=$!
echo ${PID_REDIS} > ${BASEDIR}/redis.pid
fi
PREPARE_DB_OPTS=
if test x"$OPT_CREATE_PGSQL" != xyes; then
PREPARE_DB_OPTS="$PREPARE_DB_OPTS --skip-pg"
fi
if test x"$OPT_CREATE_REDIS" != xyes; then
PREPARE_DB_OPTS="$PREPARE_DB_OPTS --skip-redis"
fi
echo "Preparing the environment"
cd ${BASEDIR}/test/support
sh prepare_db.sh ${PREPARE_DB_OPTS} || die "database preparation failure"
cd -
PATH=node_modules/.bin/:$PATH
echo "Running tests"
mocha -u tdd \
test/unit/cartodb/redis_pool.test.js \
test/unit/cartodb/req2params.test.js \
test/acceptance/cache_validator.js \
test/acceptance/server.js
mocha -t 10000 -u tdd ${MOCHA_OPTS} ${TESTS}
ret=$?
cleanup
exit $ret

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

1924
test/acceptance/templates.js Normal file

File diff suppressed because it is too large Load Diff

BIN
test/fixtures/blank.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 850 B

View File

@@ -0,0 +1 @@
{"grid":[" "," "," "," "," "," "," "," "," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"," !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"],"keys":["","1"],"data":{"1":{"cartodb_id":1}}}

BIN
test/fixtures/test_multilayer_bbox.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 KiB

View File

@@ -0,0 +1 @@
{"grid":[" "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," !!! "," !!!!!! "," !!!!!!! "," !!!!!!!! "," !!!!!!!! "," !!!!!!!! "," !!!!!!! "," !!!!!! "," !!! "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "],"keys":["","2"],"data":{"2":{"cartodb_id":2}}}

View File

@@ -0,0 +1 @@
{"grid":[" "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," !! "," !!! "," !! "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "," "],"keys":["","2"],"data":{"2":{"cartodb_id":4}}}

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.9 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 KiB

83
test/support/SQLAPIEmu.js Normal file
View File

@@ -0,0 +1,83 @@
var http = require('http');
var url = require('url');
var _ = require('underscore');
var o = function(port, cb) {
this.queries = [];
var that = this;
this.requests = [];
this.sqlapi_server = http.createServer(function(req,res) {
//console.log("server got request with method " + req.method);
var query;
that.requests.push(req);
if ( req.method == 'GET' ) {
query = url.parse(req.url, true).query;
that.handleQuery(query, res);
}
else if ( req.method == 'POST') {
var data = '';
req.on('data', function(chunk) {
//console.log("GOT Chunk " + chunk);
data += chunk;
});
req.on('end', function() {
//console.log("Data is: "); console.dir(data);
query = JSON.parse(data);
//console.log("handleQuery is " + that.handleQuery);
that.handleQuery(query, res);
});
}
else {
that.handleQuery('SQLAPIEmu does not support method' + req.method, res);
}
}).listen(port, cb);
};
o.prototype.handleQuery = function(query, res) {
this.queries.push(query);
if ( query.q.match('SQLAPIERROR') ) {
res.statusCode = 400;
res.write(JSON.stringify({'error':'Some error occurred'}));
} else if ( query.q.match('SQLAPINOANSWER') ) {
console.log("SQLAPIEmulator will never respond, on request");
return;
} else if ( query.q.match('EPOCH.* as max') ) {
// This is the structure of the known query sent by tiler
var row = {
'max': 1234567890.123
};
res.write(JSON.stringify({rows: [ row ]}));
} else {
if ( query.q.match('_private_') && query.api_key === undefined) {
res.statusCode = 403;
res.write(JSON.stringify({'error':'forbidden: ' + JSON.stringify(query)}));
} else {
var qs = JSON.stringify(query);
var row = {
// This is the structure of the known query sent by tiler
'cdb_querytables': '{' + qs + '}',
'max': qs
};
var out_obj = {rows: [ row ]};
var out = JSON.stringify(out_obj);
res.write(out);
}
}
res.end();
};
o.prototype.close = function(cb) {
this.sqlapi_server.close(cb);
};
o.prototype.getLastRequest = function() {
return this.requests.pop();
};
module.exports = o;

View File

@@ -8,29 +8,93 @@ var exec = require('child_process').exec;
var assert = module.exports = exports = require('assert');
assert.imageEqualsFile = function(buffer, file_b, callback) {
// @param tolerance number of tolerated grid cell differences
assert.utfgridEqualsFile = function(buffer, file_b, tolerance, callback) {
fs.writeFileSync('/tmp/grid.json', buffer, 'binary'); // <-- to debug/update
var expected_json = JSON.parse(fs.readFileSync(file_b, 'utf8'));
var err = null;
var Celldiff = function(x, y, ev, ov) {
this.x = x;
this.y = y;
this.ev = ev;
this.ov = ov;
};
Celldiff.prototype.toString = function() {
return '(' + this.x + ',' + this.y + ')["' + this.ev + '" != "' + this.ov + '"]';
};
try {
var obtained_json = JSON.parse(buffer);
// compare grid
var obtained_grid = obtained_json.grid;
var expected_grid = expected_json.grid;
var nrows = obtained_grid.length
if (nrows != expected_grid.length) {
throw new Error( "Obtained grid rows (" + nrows +
") != expected grid rows (" + expected_grid.length + ")" );
}
var celldiff = [];
for (var i=0; i<nrows; ++i) {
var ocols = obtained_grid[i];
var ecols = expected_grid[i];
var ncols = ocols.length;
if ( ncols != ecols.length ) {
throw new Error( "Obtained grid cols (" + ncols +
") != expected grid cols (" + ecols.length +
") on row " + i );
}
for (var j=0; j<ncols; ++j) {
var ocell = ocols[j];
var ecell = ecols[j];
if ( ocell !== ecell ) {
celldiff.push(new Celldiff(i, j, ecell, ocell));
}
}
}
if ( celldiff.length > tolerance ) {
throw new Error( celldiff.length + " cell differences: " + celldiff );
}
assert.deepEqual(obtained_json.keys, expected_json.keys);
} catch (e) { err = e; }
callback(err);
};
//
// @param tol tolerated color distance as a percent over max channel value
// by default this is zero. For meaningful values, see
// http://www.imagemagick.org/script/command-line-options.php#metric
//
assert.imageEqualsFile = function(buffer, file_b, tol, callback) {
if (!callback) callback = function(err) { if (err) throw err; };
file_b = path.resolve(file_b);
var file_a = '/tmp/' + (Math.random() * 1e16);
var file_a = '/tmp/windshaft-test-image-test.png'; // + (Math.random() * 1e16); // TODO: make predictable
var err = fs.writeFileSync(file_a, buffer, 'binary');
if (err) throw err;
exec('compare -metric PSNR "' + file_a + '" "' +
var fuzz = tol + '%';
exec('compare -fuzz ' + fuzz + ' -metric AE "' + file_a + '" "' +
file_b + '" /dev/null', function(err, stdout, stderr) {
if (err) {
fs.unlinkSync(file_a);
callback(err);
} else {
stderr = stderr.trim();
if (stderr === 'inf') {
fs.unlinkSync(file_a);
callback(null);
var similarity = parseFloat(stderr);
if ( similarity > 0 ) {
var err = new Error('Images not equal(' + similarity + '): ' +
file_a + ' ' + file_b);
err.similarity = similarity;
callback(err);
} else {
var similarity = parseFloat(stderr);
var err = new Error('Images not equal(' + similarity + '): ' +
file_a + ' ' + file_b);
err.similarity = similarity;
callback(err);
fs.unlinkSync(file_a);
callback(null);
}
}
});

View File

@@ -1,5 +1,8 @@
var _ = require('underscore');
require(__dirname + '/test_helper');
module.exports = function(opts) {
var config = {
@@ -7,7 +10,7 @@ module.exports = function(opts) {
max: 10,
idleTimeoutMillis: 1,
reapIntervalMillis: 1,
port: 6333 // TODO: read from test env ?
port: global.environment.redis.port
}
}

View File

@@ -10,27 +10,102 @@
# TODO: fix that
#
PREPARE_REDIS=yes
PREPARE_PGSQL=yes
while [ -n "$1" ]; do
if test "$1" = "--skip-pg"; then
PREPARE_PGSQL=no
shift; continue
elif test "$1" = "--skip-redis"; then
PREPARE_REDIS=no
shift; continue
fi
done
die() {
msg=$1
echo "${msg}" >&2
exit 1
}
TEST_DB="cartodb_test_user_1_db"
REDIS_PORT=6333
# This is where postgresql connection parameters are read from
TESTENV=../../config/environments/test.js
if [ \! -r ${TESTENV} ]; then
echo "Cannot read ${TESTENV}" >&2
exit 1
fi
echo "preparing postgres..."
dropdb "${TEST_DB}"
createdb -Ttemplate_postgis -EUTF8 "${TEST_DB}" || die "Could not create test database"
psql "${TEST_DB}" < ./sql/windshaft.test.sql
psql "${TEST_DB}" < ./sql/gadm4.sql
TESTUSERID=1
echo "preparing redis..."
echo "HSET rails:users:localhost id 1" | redis-cli -p ${REDIS_PORT} -n 5
echo 'HSET rails:users:localhost database_name "'"${TEST_DB}"'"' | redis-cli -p ${REDIS_PORT} -n 5
echo "HSET rails:users:localhost map_key 1234" | redis-cli -p ${REDIS_PORT} -n 5
echo "SADD rails:users:localhost:map_key 1235" | redis-cli -p ${REDIS_PORT} -n 5
echo 'HSET rails:'"${TEST_DB}"':my_table infowindow "this, that, the other"' | redis-cli -p ${REDIS_PORT} -n 0
echo 'HSET rails:'"${TEST_DB}"':test_table_private_1 privacy "0"' | redis-cli -p ${REDIS_PORT} -n 0
TESTUSER=`node -e "console.log(require('${TESTENV}').postgres_auth_user || '')"`
if test -z "$TESTUSER"; then
echo "Missing postgres_auth_user from ${TESTENV}" >&2
exit 1
fi
TESTUSER=`echo ${TESTUSER} | sed "s/<%= user_id %>/${TESTUSERID}/"`
TESTPASS=`node -e "console.log(require('${TESTENV}').postgres_auth_pass || 'test')"`
# TODO: should postgres_auth_pass be optional ?
if test -z "$TESTPASS"; then
echo "Missing postgres_auth_pass from ${TESTENV}" >&2
exit 1
fi
TESTPASS=`echo ${TESTPASS} | sed "s/<%= user_id %>/${TESTUSERID}/"`
TEST_DB="${TESTUSER}_db"
# NOTE: will be set by caller trough environment
if test -z "$REDIS_PORT"; then REDIS_PORT=6333; fi
PUBLICUSER=`node -e "console.log(require('${TESTENV}').postgres.user || 'xxx')"`
PUBLICPASS=`node -e "console.log(require('${TESTENV}').postgres.password || 'xxx')"`
echo "PUBLICUSER: ${PUBLICUSER}"
echo "PUBLICPASS: ${PUBLICPASS}"
echo "TESTUSER: ${TESTUSER}"
echo "TESTPASS: ${TESTPASS}"
if test x"$PREPARE_PGSQL" = xyes; then
echo "preparing postgres..."
dropdb "${TEST_DB}"
createdb -Ttemplate_postgis -EUTF8 "${TEST_DB}" || die "Could not create test database"
cat sql/windshaft.test.sql sql/gadm4.sql |
sed "s/:PUBLICUSER/${PUBLICUSER}/" |
sed "s/:PUBLICPASS/${PUBLICPASS}/" |
sed "s/:TESTUSER/${TESTUSER}/" |
sed "s/:TESTPASS/${TESTPASS}/" |
psql -v ON_ERROR_STOP=1 ${TEST_DB} || exit 1
fi
if test x"$PREPARE_REDIS" = xyes; then
echo "preparing redis..."
cat <<EOF | redis-cli -p ${REDIS_PORT} -n 5
HMSET rails:users:localhost id ${TESTUSERID} \
database_name '${TEST_DB}' \
map_key 1234
SADD rails:users:localhost:map_key 1235
EOF
# A user configured as with cartodb-2.5.0+
cat <<EOF | redis-cli -p ${REDIS_PORT} -n 5
HMSET rails:users:cartodb250user id ${TESTUSERID} \
database_name "${TEST_DB}" \
database_host "localhost" \
database_password "${TESTPASS}" \
map_key 4321
EOF
cat <<EOF | redis-cli -p ${REDIS_PORT} -n 0
HSET rails:${TEST_DB}:my_table infowindow "this, that, the other"
HSET rails:${TEST_DB}:test_table_private_1 privacy "0"
EOF
fi
echo "Finished preparing data. Ready to run tests"
echo "Finished preparing data. Run tests with expresso."

File diff suppressed because one or more lines are too long

View File

@@ -15,11 +15,13 @@ SET search_path = public, pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
-- publicuser role
CREATE USER publicuser;
-- public user role
DROP USER IF EXISTS :PUBLICUSER;
CREATE USER :PUBLICUSER WITH PASSWORD ':PUBLICPASS';
-- db owner role
CREATE USER test_cartodb_user_1;
DROP USER IF EXISTS :TESTUSER;
CREATE USER :TESTUSER WITH PASSWORD ':TESTPASS';
-- first table
CREATE TABLE test_table (
@@ -51,19 +53,20 @@ SELECT pg_catalog.setval('test_table_cartodb_id_seq', 60, true);
ALTER TABLE test_table ALTER COLUMN cartodb_id SET DEFAULT nextval('test_table_cartodb_id_seq'::regclass);
INSERT INTO test_table VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.314252', 1, 'Hawai', 'Calle de Pérez Galdós 9, Madrid, Spain', '0101000020E6100000A6B73F170D990DC064E8D84125364440', '0101000020110F000076491621312319C122D4663F1DCC5241');
INSERT INTO test_table VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.319101', 2, 'El Estocolmo', 'Calle de la Palma 72, Madrid, Spain', '0101000020E6100000C90567F0F7AB0DC0AB07CC43A6364440', '0101000020110F0000C4356B29423319C15DD1092DADCC5241');
INSERT INTO test_table VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.324', 3, 'El Rey del Tallarín', 'Plaza Conde de Toreno 2, Madrid, Spain', '0101000020E610000021C8410933AD0DC0CB0EF10F5B364440', '0101000020110F000053E71AC64D3419C10F664E4659CC5241');
INSERT INTO test_table VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.329509', 4, 'El Lacón', 'Manuel Fernández y González 8, Madrid, Spain', '0101000020E6100000BC5983F755990DC07D923B6C22354440', '0101000020110F00005DACDB056F2319C1EC41A980FCCA5241');
INSERT INTO test_table VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.334931', 5, 'El Pico', 'Calle Divino Pastor 12, Madrid, Spain', '0101000020E61000003B6D8D08C6A10DC0371B2B31CF364440', '0101000020110F00005F716E91992A19C17DAAA4D6DACC5241');
INSERT INTO test_table VALUES
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.314252', 1, 'Hawai', 'Calle de Pérez Galdós 9, Madrid, Spain', '0101000020E6100000A6B73F170D990DC064E8D84125364440', '0101000020110F000076491621312319C122D4663F1DCC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.319101', 2, 'El Estocolmo', 'Calle de la Palma 72, Madrid, Spain', '0101000020E6100000C90567F0F7AB0DC0AB07CC43A6364440', '0101000020110F0000C4356B29423319C15DD1092DADCC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.324', 3, 'El Rey del Tallarín', 'Plaza Conde de Toreno 2, Madrid, Spain', '0101000020E610000021C8410933AD0DC0CB0EF10F5B364440', '0101000020110F000053E71AC64D3419C10F664E4659CC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.329509', 4, 'El Lacón', 'Manuel Fernández y González 8, Madrid, Spain', '0101000020E6100000BC5983F755990DC07D923B6C22354440', '0101000020110F00005DACDB056F2319C1EC41A980FCCA5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.334931', 5, 'El Pico', 'Calle Divino Pastor 12, Madrid, Spain', '0101000020E61000003B6D8D08C6A10DC0371B2B31CF364440', '0101000020110F00005F716E91992A19C17DAAA4D6DACC5241');
ALTER TABLE ONLY test_table ADD CONSTRAINT test_table_pkey PRIMARY KEY (cartodb_id);
CREATE INDEX test_table_the_geom_idx ON test_table USING gist (the_geom);
CREATE INDEX test_table_the_geom_webmercator_idx ON test_table USING gist (the_geom_webmercator);
GRANT ALL ON TABLE test_table TO test_cartodb_user_1;
GRANT SELECT ON TABLE test_table TO publicuser;
GRANT ALL ON TABLE test_table TO :TESTUSER;
GRANT SELECT ON TABLE test_table TO :PUBLICUSER;
-- second table
CREATE TABLE test_table_2 (
@@ -95,19 +98,20 @@ SELECT pg_catalog.setval('test_table_2_cartodb_id_seq', 60, true);
ALTER TABLE test_table_2 ALTER COLUMN cartodb_id SET DEFAULT nextval('test_table_2_cartodb_id_seq'::regclass);
INSERT INTO test_table_2 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.314252', 1, 'Hawai', 'Calle de Pérez Galdós 9, Madrid, Spain', '0101000020E6100000A6B73F170D990DC064E8D84125364440', '0101000020110F000076491621312319C122D4663F1DCC5241');
INSERT INTO test_table_2 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.319101', 2, 'El Estocolmo', 'Calle de la Palma 72, Madrid, Spain', '0101000020E6100000C90567F0F7AB0DC0AB07CC43A6364440', '0101000020110F0000C4356B29423319C15DD1092DADCC5241');
INSERT INTO test_table_2 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.324', 3, 'El Rey del Tallarín', 'Plaza Conde de Toreno 2, Madrid, Spain', '0101000020E610000021C8410933AD0DC0CB0EF10F5B364440', '0101000020110F000053E71AC64D3419C10F664E4659CC5241');
INSERT INTO test_table_2 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.329509', 4, 'El Lacón', 'Manuel Fernández y González 8, Madrid, Spain', '0101000020E6100000BC5983F755990DC07D923B6C22354440', '0101000020110F00005DACDB056F2319C1EC41A980FCCA5241');
INSERT INTO test_table_2 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.334931', 5, 'El Pico', 'Calle Divino Pastor 12, Madrid, Spain', '0101000020E61000003B6D8D08C6A10DC0371B2B31CF364440', '0101000020110F00005F716E91992A19C17DAAA4D6DACC5241');
INSERT INTO test_table_2 VALUES
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.314252', 1, 'Hawai', 'Calle de Pérez Galdós 9, Madrid, Spain', '0101000020E6100000A6B73F170D990DC064E8D84125364440', '0101000020110F000076491621312319C122D4663F1DCC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.319101', 2, 'El Estocolmo', 'Calle de la Palma 72, Madrid, Spain', '0101000020E6100000C90567F0F7AB0DC0AB07CC43A6364440', '0101000020110F0000C4356B29423319C15DD1092DADCC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.324', 3, 'El Rey del Tallarín', 'Plaza Conde de Toreno 2, Madrid, Spain', '0101000020E610000021C8410933AD0DC0CB0EF10F5B364440', '0101000020110F000053E71AC64D3419C10F664E4659CC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.329509', 4, 'El Lacón', 'Manuel Fernández y González 8, Madrid, Spain', '0101000020E6100000BC5983F755990DC07D923B6C22354440', '0101000020110F00005DACDB056F2319C1EC41A980FCCA5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.334931', 5, 'El Pico', 'Calle Divino Pastor 12, Madrid, Spain', '0101000020E61000003B6D8D08C6A10DC0371B2B31CF364440', '0101000020110F00005F716E91992A19C17DAAA4D6DACC5241');
ALTER TABLE ONLY test_table_2 ADD CONSTRAINT test_table_2_pkey PRIMARY KEY (cartodb_id);
CREATE INDEX test_table_2_the_geom_idx ON test_table_2 USING gist (the_geom);
CREATE INDEX test_table_2_the_geom_webmercator_idx ON test_table_2 USING gist (the_geom_webmercator);
GRANT ALL ON TABLE test_table_2 TO test_cartodb_user_1;
GRANT SELECT ON TABLE test_table_2 TO publicuser;
GRANT ALL ON TABLE test_table_2 TO :TESTUSER;
GRANT SELECT ON TABLE test_table_2 TO :PUBLICUSER;
-- third table
CREATE TABLE test_table_3 (
@@ -139,19 +143,20 @@ SELECT pg_catalog.setval('test_table_3_cartodb_id_seq', 60, true);
ALTER TABLE test_table_3 ALTER COLUMN cartodb_id SET DEFAULT nextval('test_table_3_cartodb_id_seq'::regclass);
INSERT INTO test_table_3 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.314252', 1, 'Hawai', 'Calle de Pérez Galdós 9, Madrid, Spain', '0101000020E6100000A6B73F170D990DC064E8D84125364440', '0101000020110F000076491621312319C122D4663F1DCC5241');
INSERT INTO test_table_3 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.319101', 2, 'El Estocolmo', 'Calle de la Palma 72, Madrid, Spain', '0101000020E6100000C90567F0F7AB0DC0AB07CC43A6364440', '0101000020110F0000C4356B29423319C15DD1092DADCC5241');
INSERT INTO test_table_3 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.324', 3, 'El Rey del Tallarín', 'Plaza Conde de Toreno 2, Madrid, Spain', '0101000020E610000021C8410933AD0DC0CB0EF10F5B364440', '0101000020110F000053E71AC64D3419C10F664E4659CC5241');
INSERT INTO test_table_3 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.329509', 4, 'El Lacón', 'Manuel Fernández y González 8, Madrid, Spain', '0101000020E6100000BC5983F755990DC07D923B6C22354440', '0101000020110F00005DACDB056F2319C1EC41A980FCCA5241');
INSERT INTO test_table_3 VALUES ('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.334931', 5, 'El Pico', 'Calle Divino Pastor 12, Madrid, Spain', '0101000020E61000003B6D8D08C6A10DC0371B2B31CF364440', '0101000020110F00005F716E91992A19C17DAAA4D6DACC5241');
INSERT INTO test_table_3 VALUES
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.314252', 1, 'Hawai', 'Calle de Pérez Galdós 9, Madrid, Spain', '0101000020E6100000A6B73F170D990DC064E8D84125364440', '0101000020110F000076491621312319C122D4663F1DCC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.319101', 2, 'El Estocolmo', 'Calle de la Palma 72, Madrid, Spain', '0101000020E6100000C90567F0F7AB0DC0AB07CC43A6364440', '0101000020110F0000C4356B29423319C15DD1092DADCC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.324', 3, 'El Rey del Tallarín', 'Plaza Conde de Toreno 2, Madrid, Spain', '0101000020E610000021C8410933AD0DC0CB0EF10F5B364440', '0101000020110F000053E71AC64D3419C10F664E4659CC5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.329509', 4, 'El Lacón', 'Manuel Fernández y González 8, Madrid, Spain', '0101000020E6100000BC5983F755990DC07D923B6C22354440', '0101000020110F00005DACDB056F2319C1EC41A980FCCA5241'),
('2011-09-21 14:02:21.358706', '2011-09-21 14:02:21.334931', 5, 'El Pico', 'Calle Divino Pastor 12, Madrid, Spain', '0101000020E61000003B6D8D08C6A10DC0371B2B31CF364440', '0101000020110F00005F716E91992A19C17DAAA4D6DACC5241');
ALTER TABLE ONLY test_table_3 ADD CONSTRAINT test_table_3_pkey PRIMARY KEY (cartodb_id);
CREATE INDEX test_table_3_the_geom_idx ON test_table_3 USING gist (the_geom);
CREATE INDEX test_table_3_the_geom_webmercator_idx ON test_table_3 USING gist (the_geom_webmercator);
GRANT ALL ON TABLE test_table_3 TO test_cartodb_user_1;
GRANT SELECT ON TABLE test_table_3 TO publicuser;
GRANT ALL ON TABLE test_table_3 TO :TESTUSER;
GRANT SELECT ON TABLE test_table_3 TO :PUBLICUSER;
-- private table
CREATE TABLE test_table_private_1 (
@@ -169,5 +174,6 @@ CREATE TABLE test_table_private_1 (
CONSTRAINT enforce_srid_the_geom CHECK ((st_srid(the_geom) = 4326)),
CONSTRAINT enforce_srid_the_geom_webmercator CHECK ((st_srid(the_geom_webmercator) = 3857))
);
INSERT INTO test_table_private_1 SELECT * from test_table;
GRANT ALL ON TABLE test_table_private_1 TO test_cartodb_user_1;
GRANT ALL ON TABLE test_table_private_1 TO :TESTUSER;

View File

@@ -6,11 +6,41 @@
*/
var _ = require('underscore');
var assert = require('assert');
var LZMA = require('lzma/lzma_worker.js').LZMA;
// set environment specific variables
global.settings = require(__dirname + '/../../config/settings');
global.environment = require(__dirname + '/../../config/environments/test');
_.extend(global.settings, global.environment);
process.env.NODE_ENV = 'test';
// Utility function to compress & encode LZMA
function lzma_compress_to_base64(payload, mode, callback) {
LZMA.compress(payload, mode,
function(ints) {
ints = ints.map(function(c) { return String.fromCharCode(c + 128) }).join('')
var base64 = new Buffer(ints, 'binary').toString('base64');
callback(null, base64);
},
function(percent) {
//console.log("Compressing: " + percent + "%");
}
);
}
// Check that the response headers do not request caching
// Throws on failure
function checkNoCache(res) {
assert.ok(!res.headers.hasOwnProperty('x-cache-channel'));
assert.ok(!res.headers.hasOwnProperty('cache-control')); // is this correct ?
assert.ok(!res.headers.hasOwnProperty('last-modified')); // is this correct ?
}
module.exports = {
lzma_compress_to_base64: lzma_compress_to_base64,
checkNoCache: checkNoCache
}

View File

@@ -1,61 +0,0 @@
var assert = require('../../support/assert')
, _ = require('underscore')
, RedisPool = require('../../../lib/cartodb/redis_pool')
, tests = module.exports = {};
suite('redis_pool', function() {
// configure redis pool instance to use in tests
var test_opts = require('../../support/config').redis_pool;
var redis_pool = new RedisPool(test_opts);
test('RedisPool object exists', function(done){
assert.ok(RedisPool);
done();
});
test('RedisPool can create new redis_pool objects with default settings', function(done){
var redis_pool = new RedisPool();
done();
});
test('RedisPool can create new redis_pool objects with specific settings', function(done){
var redis_pool = new RedisPool(_.extend({host:'127.0.0.1', port: '6379'}, test_opts));
done();
});
test('pool object has an acquire function', function(done){
var found=false;
var functions = _.functions(redis_pool);
for (var i=0; i<functions.length; ++i) {
if ( functions[i] == 'acquire' ) { found=true; break; }
}
assert.ok(found);
done();
});
test('calling aquire returns a redis client object that can get/set', function(done){
redis_pool.acquire(0, function(err, client){
client.set("key","value");
client.get("key", function(err,data){
assert.equal(data, "value");
redis_pool.release(0, client); // needed to exit tests
done();
})
});
});
test('calling aquire on another DB returns a redis client object that can get/set', function(done){
redis_pool.acquire(2, function(err, client){
client.set("key","value");
client.get("key", function(err,data){
assert.equal(data, "value");
redis_pool.release(2, client); // needed to exit tests
done();
})
});
});
});

View File

@@ -7,7 +7,12 @@ var assert = require('assert')
suite('req2params', function() {
// configure redis pool instance to use in tests
var opts = require('../../../lib/cartodb/server_options');
var opts = require('../../../lib/cartodb/server_options')();
var test_user = _.template(global.environment.postgres_auth_user, {user_id:1});
var test_pubuser = global.environment.postgres.user;
var test_database = test_user + '_db';
test('can be found in server_options', function(){
assert.ok(_.isFunction(opts.req2params));
@@ -15,52 +20,68 @@ suite('req2params', function() {
test('cleans up request', function(done){
opts.req2params({headers: { host:'localhost' }, query: {dbuser:'hacker',dbname:'secret'}}, function(err, req) {
if ( err ) { console.log(err); throw new Error(err); }
if ( err ) { done(err); return; }
assert.ok(_.isObject(req.query), 'request has query');
assert.ok(!req.query.hasOwnProperty('dbuser'), 'dbuser was removed from query');
assert.ok(req.hasOwnProperty('params'), 'request has params');
assert.ok(req.params.hasOwnProperty('interactivity'), 'request params have interactivity');
assert.equal(req.params.dbname, 'cartodb_test_user_1_db', 'could forge dbname: '+ req.params.dbname);
assert.ok(!req.hasOwnProperty('dbuser'), 'could inject dbuser ('+req.params.dbuser+')');
assert.equal(req.params.dbname, test_database, 'could forge dbname: '+ req.params.dbname);
assert.ok(req.params.dbuser === test_pubuser, 'could inject dbuser ('+req.params.dbuser+')');
done();
});
});
test('sets dbname from redis metadata', function(done){
opts.req2params({headers: { host:'localhost' }, query: {} }, function(err, req) {
if ( err ) { console.log(err); throw new Error(err); }
if ( err ) { done(err); return; }
//console.dir(req);
assert.ok(_.isObject(req.query), 'request has query');
assert.ok(!req.query.hasOwnProperty('dbuser'), 'dbuser was removed from query');
assert.ok(req.hasOwnProperty('params'), 'request has params');
assert.ok(req.params.hasOwnProperty('interactivity'), 'request params have interactivity');
// database_name for user "localhost" (see test/support/prepare_db.sh)
assert.equal(req.params.dbname, 'cartodb_test_user_1_db');
// unauthenticated request gets no dbuser
assert.ok(!req.params.hasOwnProperty('dbuser'), 'could inject dbuser ('+req.params.dbuser+')');
assert.equal(req.params.dbname, test_database);
assert.ok(req.params.dbuser === test_pubuser, 'could inject dbuser ('+req.params.dbuser+')');
done();
});
});
test('sets also dbuser for authenticated requests', function(done){
opts.req2params({headers: { host:'localhost' }, query: {map_key: '1234'} }, function(err, req) {
if ( err ) { console.log(err); throw new Error(err); }
if ( err ) { done(err); return; }
//console.dir(req);
assert.ok(_.isObject(req.query), 'request has query');
assert.ok(!req.query.hasOwnProperty('dbuser'), 'dbuser was removed from query');
assert.ok(req.hasOwnProperty('params'), 'request has params');
assert.ok(req.params.hasOwnProperty('interactivity'), 'request params have interactivity');
// database_name for user "localhost" (see test/support/prepare_db.sh)
assert.equal(req.params.dbname, 'cartodb_test_user_1_db');
// id for user "localhost" (see test/support/prepare_db.sh)
assert.equal(req.dbuser, 'test_cartodb_user_1');
assert.equal(req.params.dbname, test_database);
assert.equal(req.params.dbuser, test_user);
opts.req2params({headers: { host:'localhost' }, query: {map_key: '1235'} }, function(err, req) {
// wrong key resets params to no user
assert.ok(!req.hasOwnProperty('dbuser'), 'could inject dbuser ('+req.params.dbuser+')');
assert.ok(req.params.dbuser === test_pubuser, 'could inject dbuser ('+req.params.dbuser+')');
done();
});
});
});
test('it should extend params with decoded lzma', function(done) {
var qo = {
style: 'test',
style_version: '2.1.0',
cache_buster: 5
};
test_helper.lzma_compress_to_base64(JSON.stringify(qo), 1, function(err, data) {
opts.req2params({ headers: { host:'localhost' }, query: { non_included: 'toberemoved', api_key: 'test', style: 'override', lzma: data }}, function(err, req) {
if ( err ) { done(err); return; }
var query = req.params
assert.equal(qo.style, query.style)
assert.equal(qo.style_version, query.style_version)
assert.equal(qo.cache_buster, query.cache_buster)
assert.equal('test', query.api_key)
assert.equal(undefined, query.non_included)
done();
});
});
});
});

View File

@@ -0,0 +1,109 @@
var assert = require('assert')
//, _ = require('underscore')
, RedisPool = require('redis-mpool')
, SignedMaps = require('../../../lib/cartodb/signed_maps.js')
, test_helper = require('../../support/test_helper')
, Step = require('step')
, tests = module.exports = {};
suite('signed_maps', function() {
// configure redis pool instance to use in tests
var redis_pool = RedisPool(global.environment.redis);
test('can sign map with open and token-based auth', function(done) {
var smap = new SignedMaps(redis_pool);
assert.ok(smap);
var sig = 'sig1';
var map = 'map1';
var tok = 'tok1';
var crt = {
version:'0.0.1',
layergroup_id:map,
auth: {}
};
var crt1_id; // by token
var crt2_id; // open
Step(
function() {
smap.isAuthorized(sig,map,tok,this);
},
function checkAuthFailure1(err, authorized) {
if ( err ) throw err;
assert.ok(!authorized, "unexpectedly authorized");
crt.auth.method = 'token';
crt.auth.valid_tokens = [tok];
smap.addSignature(sig, map, crt, this)
},
function getCert1(err, id) {
if ( err ) throw err;
assert.ok(id, "undefined signature id");
crt1_id = id; // keep note of it
//console.log("Certificate 1 is " + crt1_id);
smap.isAuthorized(sig,map,'',this);
},
function checkAuthFailure2(err, authorized) {
if ( err ) throw err;
assert.ok(!authorized, "unexpectedly authorized");
smap.isAuthorized(sig,map,tok,this);
},
function checkAuthSuccess1(err, authorized) {
if ( err ) throw err;
assert.ok(authorized, "unauthorized :(");
crt.auth.method = 'open';
delete crt.auth.valid_tokens;
smap.addSignature(sig, map, crt, this)
},
function getCert2(err, id) {
if ( err ) throw err;
assert.ok(id, "undefined signature id");
crt2_id = id; // keep note of it
//console.log("Certificate 2 is " + crt2_id);
smap.isAuthorized(sig,map,'arbitrary',this);
},
function checkAuthSuccess2_delCert2(err, authorized) {
if ( err ) throw err;
assert.ok(authorized, "unauthorized :(");
var next = this;
smap.delCertificate(sig, crt2_id, function(e) {
if (e) next(e);
else smap.isAuthorized(sig,map,'arbitrary',next);
});
},
function checkAuthFailure3_delCert2(err, authorized) {
if ( err ) throw err;
assert.ok(!authorized, "unexpectedly authorized");
smap.delCertificate(sig, crt1_id, this);
},
function finish(err) {
done(err);
}
);
});
test('can validate certificates', function(done) {
var smap = new SignedMaps(redis_pool);
assert.ok(smap);
Step(
function invalidVersion() {
var cert = { version: '-1' };
var err = smap.checkInvalidCertificate(cert);
assert.ok(err);
assert.equal(err.message, "Unsupported certificate version -1");
return null;
},
function invalidTokenAuth() {
var cert = { version: '0.0.1', auth: { method:'token', valid_token:[] } };
var err = smap.checkInvalidCertificate(cert);
assert.ok(err);
assert.equal(err.message, "Invalid 'token' authentication: missing valid_tokens");
return null;
},
function finish(err) {
done(err);
}
);
});
});

View File

@@ -0,0 +1,506 @@
var assert = require('assert')
//, _ = require('underscore')
, RedisPool = require('redis-mpool')
, SignedMaps = require('../../../lib/cartodb/signed_maps.js')
, TemplateMaps = require('../../../lib/cartodb/template_maps.js')
, test_helper = require('../../support/test_helper')
, Step = require('step')
, tests = module.exports = {};
suite('template_maps', function() {
// configure redis pool instance to use in tests
var redis_pool = RedisPool(global.environment.redis);
var signed_maps = new SignedMaps(redis_pool);
test('does not accept template with unsupported version', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'6.6.6',
name:'k', auth: {}, layergroup: {} };
Step(
function() {
tmap.addTemplate('me', tpl, this);
},
function checkFailed(err) {
assert.ok(err);
assert.ok(err.message.match(/unsupported.*version/i), err);
return null;
},
function finish(err) {
done(err);
}
);
});
test('does not accept template with missing name', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'0.0.1',
auth: {}, layergroup: {} };
Step(
function() {
tmap.addTemplate('me', tpl, this);
},
function checkFailed(err) {
assert.ok(err);
assert.ok(err.message.match(/missing.*name/i), err);
return null;
},
function finish(err) {
done(err);
}
);
});
test('does not accept template with invalid name', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'0.0.1',
auth: {}, layergroup: {} };
var invalidnames = [ "ab|", "a b", "a@b", "1ab", "_x", "", " x", "x " ];
var testNext = function() {
if ( ! invalidnames.length ) { done(); return; }
var n = invalidnames.pop();
tpl.name = n;
tmap.addTemplate('me', tpl, function(err) {
if ( ! err ) {
done(new Error("Unexpected success with invalid name '" + n + "'"));
}
else if ( ! err.message.match(/template.*name/i) ) {
done(new Error("Unexpected error message with invalid name '" + n
+ "': " + err));
}
else {
testNext();
}
});
};
testNext();
});
test('does not accept template with invalid placeholder name', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'0.0.1',
name: "valid", placeholders: {},
auth: {}, layergroup: {} };
var invalidnames = [ "ab|", "a b", "a@b", "1ab", "_x", "", " x", "x " ];
var testNext = function() {
if ( ! invalidnames.length ) { done(); return; }
var n = invalidnames.pop();
tpl.placeholders = {};
tpl.placeholders[n] = { type:'number', default:1 };
tmap.addTemplate('me', tpl, function(err) {
if ( ! err ) {
done(new Error("Unexpected success with invalid name '" + n + "'"));
}
else if ( ! err.message.match(/invalid.*name/i) ) {
done(new Error("Unexpected error message with invalid name '" + n
+ "': " + err));
}
else {
testNext();
}
});
};
testNext();
});
test('does not accept template with missing placeholder default', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'0.0.1',
name: "valid", placeholders: { v: {} },
auth: {}, layergroup: {} };
tmap.addTemplate('me', tpl, function(err) {
if ( ! err ) {
done(new Error("Unexpected success with missing placeholder default"));
}
else if ( ! err.message.match(/missing default/i) ) {
done(new Error("Unexpected error message with missing placeholder default: "
+ err));
}
else {
done();
}
});
});
test('does not accept template with missing placeholder type', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'0.0.1',
name: "valid", placeholders: { v: { default:1 } },
auth: {}, layergroup: {} };
tmap.addTemplate('me', tpl, function(err) {
if ( ! err ) {
done(new Error("Unexpected success with missing placeholder type"));
}
else if ( ! err.message.match(/missing type/i) ) {
done(new Error("Unexpected error message with missing placeholder default: "
+ err));
}
else {
done();
}
});
});
// See http://github.com/CartoDB/Windshaft-cartodb/issues/128
test('does not accept template with invalid token auth (undefined tokens)',
function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl = { version:'0.0.1',
name: "invalid_auth1", placeholders: { },
auth: { method: 'token' }, layergroup: {} };
tmap.addTemplate('me', tpl, function(err) {
if ( ! err ) {
done(new Error("Unexpected success with invalid token auth (undefined tokens)"));
}
else if ( ! err.message.match(/invalid 'token' authentication/i) ) {
done(new Error("Unexpected error message with invalid token auth (undefined tokens): "
+ err));
}
else {
done();
}
});
});
test('add, get and delete a valid template', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var expected_failure = false;
var tpl_id;
var tpl = { version:'0.0.1',
name: 'first', auth: {}, layergroup: {} };
Step(
function() {
tmap.addTemplate('me', tpl, this);
},
function addOmonimousTemplate(err, id) {
if ( err ) throw err;
tpl_id = id;
assert.equal(tpl_id, 'first');
expected_failure = true;
// should fail, as it already exists
tmap.addTemplate('me', tpl, this);
},
function getTemplate(err) {
if ( ! expected_failure && err ) throw err;
assert.ok(err);
assert.ok(err.message.match(/already exists/i), err);
tmap.getTemplate('me', tpl_id, this);
},
function delTemplate(err, got_tpl) {
if ( err ) throw err;
assert.deepEqual(got_tpl, tpl);
tmap.delTemplate('me', tpl_id, this);
},
function finish(err) {
done(err);
}
);
});
test('add multiple templates, list them', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var expected_failure = false;
var tpl1 = { version:'0.0.1', name: 'first', auth: {}, layergroup: {} };
var tpl1_id;
var tpl2 = { version:'0.0.1', name: 'second', auth: {}, layergroup: {} };
var tpl2_id;
Step(
function addTemplate1() {
tmap.addTemplate('me', tpl1, this);
},
function addTemplate2(err, id) {
if ( err ) throw err;
tpl1_id = id;
tmap.addTemplate('me', tpl2, this);
},
function listTemplates(err, id) {
if ( err ) throw err;
tpl2_id = id;
tmap.listTemplates('me', this);
},
function checkTemplates(err, ids) {
if ( err ) throw err;
assert.equal(ids.length, 2);
assert.ok(ids.indexOf(tpl1_id) != -1, ids.join(','));
assert.ok(ids.indexOf(tpl2_id) != -1, ids.join(','));
return null;
},
function delTemplate1(err) {
if ( tpl1_id ) {
var next = this;
tmap.delTemplate('me', tpl1_id, function(e) {
if ( err || e ) next(new Error(err + '; ' + e));
else next();
});
} else {
if ( err ) throw err;
return null;
}
},
function delTemplate2(err) {
if ( tpl2_id ) {
var next = this;
tmap.delTemplate('me', tpl2_id, function(e) {
if ( err || e ) next(new Error(err + '; ' + e));
else next();
});
} else {
if ( err ) throw err;
return null;
}
},
function finish(err) {
done(err);
}
);
});
test('update templates', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var expected_failure = false;
var owner = 'me';
var tpl = { version:'0.0.1',
name: 'first',
auth: { method: 'open' },
layergroup: {}
};
var tpl_id;
Step(
function addTemplate() {
tmap.addTemplate(owner, tpl, this);
},
// Updating template name should fail
function updateTemplateName(err, id) {
if ( err ) throw err;
tpl_id = id;
expected_failure = true;
tpl.name = 'second';
tmap.updTemplate(owner, tpl_id, tpl, this);
},
function updateTemplateAuth(err) {
if ( err && ! expected_failure) throw err;
expected_failure = false;
assert.ok(err);
tpl.name = 'first';
tpl.auth.method = 'token';
tpl.auth.valid_tokens = [ 'tok1' ];
tmap.updTemplate(owner, tpl_id, tpl, this);
},
function updateTemplateWithInvalid(err) {
if ( err ) throw err;
tpl.version = '999.999.999';
expected_failure = true;
tmap.updTemplate(owner, tpl_id, tpl, this);
},
function updateUnexistentTemplate(err) {
if ( err && ! expected_failure) throw err;
expected_failure = false;
assert.ok(err);
assert.ok(err.message.match(/unsupported.*version/i), err);
tpl.version = '0.0.1';
expected_failure = true;
tmap.updTemplate(owner, 'unexistent', tpl, this);
},
function delTemplate(err) {
if ( err && ! expected_failure) throw err;
expected_failure = false;
assert.ok(err);
assert.ok(err.message.match(/cannot update name/i), err);
tmap.delTemplate(owner, tpl_id, this);
},
function finish(err) {
done(err);
}
);
});
test('instanciate templates', function() {
var tmap = new TemplateMaps(redis_pool, signed_maps);
assert.ok(tmap);
var tpl1 = {
version: '0.0.1',
name: 'acceptance1',
auth: { method: 'open' },
placeholders: {
fill: { type: "css_color", default: "red" },
color: { type: "css_color", default: "#a0fF9A" },
name: { type: "sql_literal", default: "test" },
zoom: { type: "number", default: "0" },
test_number: { type: "number", default: 23 },
},
layergroup: {
version: '1.0.0',
global_cartocss_version: '2.0.2',
layers: [
{ options: {
sql: "select '<%=name %>' || id, g from t",
cartocss: '#layer { marker-fill:<%= fill %>; marker-width: <%=test_number %>; }'
} },
{ options: {
sql: "select fun('<%= name%>') g from x",
cartocss: '#layer { line-color:<%= color %>; marker-fill:<%= color %>; }'
} },
{ options: {
sql: "select g from x",
cartocss: '#layer[zoom=<%=zoom%>] { }'
} }
]
}
};
var inst = tmap.instance(tpl1, {});
var lyr = inst.layers[0].options;
assert.equal(lyr.sql, "select 'test' || id, g from t");
assert.equal(lyr.cartocss, '#layer { marker-fill:red; marker-width: 23; }');
lyr = inst.layers[1].options;
assert.equal(lyr.sql, "select fun('test') g from x");
assert.equal(lyr.cartocss, '#layer { line-color:#a0fF9A; marker-fill:#a0fF9A; }');
inst = tmap.instance(tpl1, {color:'yellow', name:"it's dangerous"});
lyr = inst.layers[0].options;
assert.equal(lyr.sql, "select 'it''s dangerous' || id, g from t");
assert.equal(lyr.cartocss, '#layer { marker-fill:red; marker-width: 23; }');
lyr = inst.layers[1].options;
assert.equal(lyr.sql, "select fun('it''s dangerous') g from x");
assert.equal(lyr.cartocss, '#layer { line-color:yellow; marker-fill:yellow; }');
// Invalid css_color
var err = null;
try { inst = tmap.instance(tpl1, {color:'##ff00ff'}); }
catch (e) { err = e; }
assert.ok(err);
assert.ok(err.message.match(/invalid css_color/i), err);
// Invalid css_color 2 (too few digits)
var err = null;
try { inst = tmap.instance(tpl1, {color:'#ff'}); }
catch (e) { err = e; }
assert.ok(err);
assert.ok(err.message.match(/invalid css_color/i), err);
// Invalid css_color 3 (too many digits)
var err = null;
try { inst = tmap.instance(tpl1, {color:'#1234567'}); }
catch (e) { err = e; }
assert.ok(err);
assert.ok(err.message.match(/invalid css_color/i), err);
// Invalid number
var err = null;
try { inst = tmap.instance(tpl1, {zoom:'#'}); }
catch (e) { err = e; }
assert.ok(err);
assert.ok(err.message.match(/invalid number/i), err);
// Invalid number 2
var err = null;
try { inst = tmap.instance(tpl1, {zoom:'23e'}); }
catch (e) { err = e; }
assert.ok(err);
assert.ok(err.message.match(/invalid number/i), err);
// Valid number
var err = null;
try { inst = tmap.instance(tpl1, {zoom:'-.23e10'}); }
catch (e) { err = e; }
assert.ok(!err);
});
// Can set a limit on the number of user templates
test('can limit number of user templates', function(done) {
var tmap = new TemplateMaps(redis_pool, signed_maps, {
max_user_templates: 2
});
assert.ok(tmap);
var tpl = { version:'0.0.1', auth: {}, layergroup: {} };
var expectErr = false;
var idMe = [];
var idYou = [];
Step(
function oneForMe() {
tpl.name = 'oneForMe';
tmap.addTemplate('me', tpl, this);
},
function twoForMe(err, id) {
if ( err ) throw err;
assert.ok(id);
idMe.push(id);
tpl.name = 'twoForMe';
tmap.addTemplate('me', tpl, this);
},
function threeForMe(err, id) {
if ( err ) throw err;
assert.ok(id);
idMe.push(id);
tpl.name = 'threeForMe';
expectErr = true;
tmap.addTemplate('me', tpl, this);
},
function errForMe(err, id) {
if ( err && ! expectErr ) throw err;
expectErr = false;
assert.ok(err);
assert.ok(err.message.match(/limit.*template/), err);
return null;
},
function delOneMe(err) {
if ( err ) throw err;
tmap.delTemplate('me', idMe.shift(), this);
},
function threeForMeRetry(err) {
if ( err ) throw err;
tpl.name = 'threeForMe';
tmap.addTemplate('me', tpl, this);
},
function oneForYou(err, id) {
if ( err ) throw err;
assert.ok(id);
idMe.push(id);
tpl.name = 'oneForYou';
tmap.addTemplate('you', tpl, this);
},
function twoForYou(err, id) {
if ( err ) throw err;
assert.ok(id);
idYou.push(id);
tpl.name = 'twoForYou';
tmap.addTemplate('you', tpl, this);
},
function threeForYou(err, id) {
if ( err ) throw err;
assert.ok(id);
idYou.push(id);
tpl.name = 'threeForYou';
expectErr = true;
tmap.addTemplate('you', tpl, this);
},
function errForYou(err, id) {
if ( err && ! expectErr ) throw err;
expectErr = false;
assert.ok(err);
assert.ok(err.message.match(/limit.*template/), err);
return null;
},
function finish(err) {
// TODO: delete all templates
done(err);
}
);
});
});

81
tools/convert_database_styles Executable file
View File

@@ -0,0 +1,81 @@
#!/usr/bin/env node
var path = require('path');
var grainstore = require('../node_modules/windshaft/node_modules/grainstore');
var mapnik = require('mapnik');
var redis = require('redis');
function usage(me, exitcode) {
console.log("Usage: " + me + " <database_name> <table_name> [<target_mapnik_version>]");
process.exit(exitcode);
}
var node_path = process.argv.shift();
var script_path = process.argv.shift();
var me = path.basename(script_path);
var database_name = process.argv.shift()
var table_name = process.argv.shift()
var MAPNIK_VERSION = process.argv.shift()
if ( ! MAPNIK_VERSION ) {
MAPNIK_VERSION = mapnik.versions.mapnik;
}
if ( ! database_name || ! table_name) {
usage(me, 1);
}
var REDIS_PORT = 6379; // TODO: make a command line parameter
var dbnum = 0;
var mml_store = new grainstore.MMLStore({port:REDIS_PORT}, {mapnik_version:MAPNIK_VERSION});
var failures = [];
var client = redis.createClient(REDIS_PORT, 'localhost');
client.on('connect', function() {
client.select(dbnum);
client.keys('map_style|' + database_name + '|' + table_name, function(err, matches) {
processNext = function() {
if ( ! matches.length ) process.exit(failures.length);
var k = matches.shift();
if ( /map_style\|.*\|.*\|/.test(k) ) {
//console.warn("Key " + k + " is EXTENDED, skipping");
processNext();
}
var out = 'map_style|' + database_name + '|' + table_name + ': ';
var mml_builder = mml_store.mml_builder({
dbname:database_name,
table:table_name},
function(err, payload) {
if ( err ) {
console.warn(out + err.message);
failures.push(k); processNext();
}
else {
mml_builder.resetStyle(function(err, data) {
if ( err ) {
console.warn(out + err.message);
failures.push(k);
}
else console.log(out + 'OK');
processNext();
}, true);
}
});
};
processNext();
});
});

40
tools/create_multilayer Executable file
View File

@@ -0,0 +1,40 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/layergroup
# This is for direct windshaft connection
#tiler_url=http://dev.localhost.lan:8083/database/cartodb_dev_user_1_db/layergroup
while test -n "$1"; do
if test "$1" = "-v"; then
verbose=yes
elif test -z "$cfg"; then
cfg="$1"
else
tiler_url="$1"
fi
shift
done
if test -z "$cfg"; then
echo "Usage: $0 [-v] <config_file> [<tiler_url>]" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
exit 1
fi
cmd="curl -skH Content-Type:application/json --data-binary @- ${tiler_url}"
if test x${verbose} = xyes; then
cmd="${cmd} -v"
fi
res=`cat ${cfg} | tr '\n' ' ' | ${cmd}`
if test $? -gt 0; then
echo "curl command failed: ${cmd}"
fi
if test x${verbose} = xyes; then
echo "${res}"
fi
tok=`echo "$res" | sed 's/.*"layergroupid":"\([^"]*\)".*/\1/'`
echo $tok

49
tools/create_template Executable file
View File

@@ -0,0 +1,49 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/template
apikey=${CDB_APIKEY}
while test -n "$1"; do
if test "$1" = "-v"; then
verbose=yes
elif test "$1" = "-k"; then
shift
apikey="$1"
elif test "$1" = "-u"; then
shift
tiler_url="$1"
elif test -z "$cfg"; then
cfg="$1"
else
echo "Unused parameter $1" >&2
fi
shift
done
if test -z "$cfg"; then
echo "Usage: $0 [-v] [-k <api_key>] [-u <tiler_url>] <template_config>" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
echo "Default <api_key> is read from CDB_APIKEY env variable" >&2
exit 1
fi
cmd="curl -skH Content-Type:application/json --data-binary @- ${tiler_url}?api_key=${apikey}"
if test x${verbose} = xyes; then
cmd="${cmd} -v"
fi
res=`cat ${cfg} | tr '\n' ' ' | ${cmd}`
if test $? -gt 0; then
echo "curl command failed: ${cmd}"
fi
if test x${verbose} = xyes; then
echo "${res}"
fi
# Successful response contains no space
echo "$res" | grep " " && { echo $res && exit 1; }
tok=`echo "$res" | sed 's/.*"template_id":"\([^"]*\)".*/\1/'`
echo $tok

45
tools/delete_template Executable file
View File

@@ -0,0 +1,45 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/template
apikey=${CDB_APIKEY}
while test -n "$1"; do
if test "$1" = "-v"; then
verbose=yes
elif test "$1" = "-k"; then
shift
apikey="$1"
elif test "$1" = "-u"; then
shift
tiler_url="$1"
elif test -z "$tpl"; then
tpl="$1"
else
echo "Unused parameter $1" >&2
fi
shift
done
if test -z "$tpl"; then
echo "Usage: $0 [-v] [-k <api_key>] [-u <tiler_url>] <template_id>" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
echo "Default <api_key> is read from CDB_APIKEY env variable" >&2
exit 1
fi
cmd="curl -X DELETE -skH Content-Type:application/json ${tiler_url}/${tpl}?api_key=${apikey}"
if test x${verbose} = xyes; then
cmd="${cmd} -v"
fi
res=`${cmd}`
if test $? -gt 0; then
echo "curl command failed: ${cmd}"
fi
if test x${verbose} = xyes; then
echo "${res}"
fi
tok=`echo "$res" | sed 's/.*"template_id":"\([^"]*\)".*/\1/'`
echo $tok

View File

@@ -0,0 +1,11 @@
{"version":"1.0.1",
"layers":[{
"type":"cartodb",
"options":{
"sql":"select 1 as id, ST_SetSRID(ST_MakePoint(0,0),3857) as the_geom_webmercator",
"cartocss":"#style{ marker-width: 12;}",
"cartocss_version":"2.1.1",
"Interactivity":"id"
}
}]
}

View File

@@ -0,0 +1,10 @@
{"version":"1.0.1",
"layers":[{
"type":"torque",
"options":{
"sql":"select 1 as id, ST_SetSRID(ST_MakePoint(0,0),3857) as the_geom_webmercator",
"cartocss":"Map{ -torque-time-attribute:'id'; -torque-aggregation-function:'count(id)'; -torque-frame-count:2; -torque-resolution:2}",
"cartocss_version": "2.1.1"
}
}]
}

View File

@@ -0,0 +1,17 @@
{
"version":"0.0.1",
"name":"simple",
"placeholders":{},
"auth":{ "method":"open" },
"layergroup":{
"version":"1.0.1",
"layers":[{
"type":"cartodb",
"options":{
"sql":"select ST_SetSRID(ST_MakePoint(0,0),3857) as the_geom_webmercator",
"cartocss":"#s{ marker-width: 12;}",
"cartocss_version":"2.1.1"
}
}]
}
}

59
tools/flush_cache Executable file
View File

@@ -0,0 +1,59 @@
#!/usr/bin/env node
var path = require('path');
var request = require('request');
function usage(me, exitcode) {
console.log("Usage: " + me + " [--env <environment>] <username> <tablename>");
process.exit(exitcode);
}
var node_path = process.argv.shift();
var script_path = process.argv.shift();
var basedir = path.dirname(script_path);
var me = path.basename(script_path);
var ENV = 'development.js';
var username, table;
var arg;
while ( arg = process.argv.shift() ) {
if ( arg == '--env' ) {
ENV = process.argv.shift();
}
else if ( ! username ) {
username = arg;
}
else if ( ! table ) {
table = arg;
}
else {
console.warn("Unused parameter " + arg);
}
}
if ( ! table ) {
usage(me, 1);
}
global.environment = require('../config/environments/' + ENV);
// _after_ setting global.environment
var serverOptions = require('../lib/cartodb/server_options');
var host = global.environment.host;
var port = global.environment.port;
var re = ''+serverOptions.re_userFromHost;
var hostname = re.replace(/^\/\^/, '')
.replace(/\/$$/, '')
.replace(/\\/g,'')
.replace(/\([^)]*\)/,username)
;
//console.log("re: " + re);
//console.log("hostname: " + hostname);
var url = 'http://' + host + ':' + port + '/tiles/' + table + '/flush_cache';
request.del({ url: url, headers: { host: hostname } },
function(err, res, body) {
if ( err ) throw err;
console.log(res.body);
});

53
tools/instanciate_template Executable file
View File

@@ -0,0 +1,53 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/template
apikey=${CDB_APIKEY}
while test -n "$1"; do
if test "$1" = "-v"; then
verbose=yes
elif test "$1" = "-k"; then
shift
apikey="$1"
elif test "$1" = "-u"; then
shift
tiler_url="$1"
elif test -z "$tpl"; then
tpl="$1"
elif test -z "$cfg"; then
cfg="$1"
else
echo "Unused parameter $1" >&2
fi
shift
done
if test -z "$tpl"; then
echo "Usage: $0 [-v] [-k <api_key>] [-u <tiler_url>] <template_id> [<template_params>]" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
echo "Default <api_key> is read from CDB_APIKEY env variable" >&2
exit 1
fi
if test -z "$cfg"; then
cfg="/dev/null"
fi
tiler_url="${tiler_url}/${tpl}"
cmd="curl -X POST -skH Content-Type:application/json --data-binary @- ${tiler_url}?api_key=${apikey}"
if test x${verbose} = xyes; then
cmd="${cmd} -v"
fi
res=`cat ${cfg} | tr '\n' ' ' | ${cmd}`
if test $? -gt 0; then
echo "curl command failed: ${cmd}"
fi
if test x${verbose} = xyes; then
echo "${res}"
fi
tok=`echo "$res" | sed 's/.*"layergroupid":"\([^"]*\)".*/\1/'`
echo $tok

45
tools/list_templates Executable file
View File

@@ -0,0 +1,45 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/template
apikey=${CDB_APIKEY}
while test -n "$1"; do
if test "$1" = "-v"; then
verbose=yes
elif test "$1" = "-k"; then
shift
apikey="$1"
elif test "$1" = "-u"; then
shift
tiler_url="$1"
elif test "$1" = "-h" -o "$1" = "-?"; then
echo "Usage: $0 [-v] [-k <api_key>] [-u <tiler_url>]" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
echo "Default <api_key> is read from CDB_APIKEY env variable" >&2
exit 0
else
echo "Unused parameter $1" >&2
fi
shift
done
cmd="curl -X GET -sk ${tiler_url}?api_key=${apikey}"
if test x${verbose} = xyes; then
cmd="${cmd} -v"
fi
res=`${cmd}`
if test $? -gt 0; then
echo "curl command failed: ${cmd}"
fi
if test x${verbose} = xyes; then
echo "${res}"
fi
node <<EOF
var parsed = JSON.parse('$res');
console.dir(parsed);
EOF

19
tools/munin/Makefile Normal file
View File

@@ -0,0 +1,19 @@
MUNIN_PLUGINS_DIR=/etc/munin/plugins
MUNIN_PLUGINS_CONFIG_DIR=/etc/munin/plugin-conf.d
PWD=$(shell pwd)
all: windshaft.conf
windshaft.conf: windshaft.conf.in
sed 's#@PWD@#$(PWD)#' < $< > $@
install-munin-plugin-conf: windshaft.conf
install -m 644 $< $(MUNIN_PLUGINS_CONFIG_DIR)/windshaft.conf
install-munin-plugin: windshaft
install -m 755 $< $(MUNIN_PLUGINS_DIR)/windshaft
install: install-munin-plugin install-munin-plugin-conf
clean:
rm -f windshaft.conf

87
tools/munin/windshaft Executable file
View File

@@ -0,0 +1,87 @@
#!/bin/sh
envnik=$(basename "${TILER_ENVIRONMENT}" .js)
if test "$1" = "config"; then
echo "graph_title fd usage (${envnik})"
cat <<"EOM"
graph_vlabel number of file descriptors
graph_category windshaft
graph_scale no
procs.label Number of tiler processes
pgsql.label PostgreSQL connections (max)
redis.label Redis connections (max)
http.label Incoming http requests (max)
caches.label Number of renderer caches (max)
nfd.label Total file descriptors (max)
EOM
exit 0
fi
if test x"$1" != x; then
TILER_ENVIRONMENT=$(cd $(dirname $0); pwd)/../../config/environments/${1}.js
fi
if test -z "$TILER_ENVIRONMENT"; then
echo "Usage: $0 [<environment>]" >&2
echo " or: [TILER_ENVIRONMENT=<environment>] $0" >&2
exit 1
fi
http_port=$(echo "console.log(require('${TILER_ENVIRONMENT}').port)" | node) || exit 1
pgsql_port=$(echo "console.log(require('${TILER_ENVIRONMENT}').postgres.port)" | node) || exit 1
redis_port=$(echo "console.log(require('${TILER_ENVIRONMENT}').redis.port)" | node) || exit 1
pids=$(lsof -i :${http_port} | grep LISTEN | awk '{print $2}')
nworkers=$(echo "${pids}" | wc -l)
pids=$(echo "${pids}" | paste -sd ' ')
if test -z "${pids}"; then
echo "No processes found listening on tcp port '${http_port}'" >&2
exit 1
fi
tmpreport="/tmp/checkfd.$$.txt"
lsof -Pp $(echo "${pids}" | tr ' ' ',') > "${tmpreport}"
maxdb=0
maxredis=0
maxhttp=0
maxtot=0
maxcache=0
for pid in ${pids}; do
cnt=$(grep "${pid}" "${tmpreport}" | grep ":${pgsql_port} " | wc -l);
if test $cnt -gt $maxdb; then maxdb=$cnt; fi
cnt=$(grep "${pid}" "${tmpreport}" | grep ":${redis_port} " | wc -l);
if test $cnt -gt $maxredis; then maxredis=$cnt; fi
cnt=$(grep "${pid}" "${tmpreport}" | grep ":${http_port}-" | grep -v "LISTEN" | wc -l);
if test $cnt -gt $maxhttp; then maxhttp=$cnt; fi
cnt=$(grep "${pid}" "${tmpreport}" | wc -l);
if test $cnt -gt $maxtot; then maxtot=$cnt; fi
log=$(grep "${pid}" "${tmpreport}" | grep -w 1w | awk '{print $9}')
if test -e "${log}"; then
kill -USR2 "${pid}"
cnt=$(tac ${log} | sed -n -e '/ItemKey/p;/ RenderCache /q' | wc -l)
if test $cnt -gt $maxcache; then maxcache=$cnt; fi
else
# report the error...
maxcache=-1
fi
done
echo "procs.value ${nworkers}"
echo "pgsql.value ${maxdb}"
echo "redis.value ${maxredis}"
echo "http.value ${maxhttp}"
echo "caches.value ${maxcache}"
echo "nfd.value ${maxtot}"
rm -f "${tmpreport}"

View File

@@ -0,0 +1,5 @@
# Configuration file for munin plugin
[windshaft]
user root
env.TILER_ENVIRONMENT @PWD@/../../config/environments/production.js

View File

@@ -0,0 +1,54 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/template
apikey=${CDB_APIKEY}
while test -n "$1"; do
if test "$1" = "-k"; then
shift
apikey="$1"
elif test "$1" = "-u"; then
shift
tiler_url="$1"
elif test -z "$tpl"; then
tpl="$1"
else
echo "Unused parameter $1" >&2
fi
shift
done
if test -z "$tpl"; then
echo "Usage: $0 [-v] [-k <api_key>] [-u <tiler_url>] <template_config>" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
echo "Default <api_key> is read from CDB_APIKEY env variable" >&2
exit 1
fi
basedir=$(cd $(dirname $0); cd ..; pwd)
export CDB_APIKEY=${apikey}
max=3000000
i=0
while test "$i" -le "$max"; do
tpln=`cat ${tpl} | sed "s/\"name\":\"\(.*\)\"/\"name\":\"\1${i}\"/"`
tpl_id=`echo ${tpln} | ${basedir}/create_template -u ${tiler_url} /dev/stdin`
if test $? -ne 0; then
echo $tpl_id >&2
break
fi
tpl_id=`echo ${tpln} | ${basedir}/update_template -u ${tiler_url} ${tpl_id} /dev/stdin`
if test $? -ne 0; then
echo $tpl_id >&2
break
fi
out=`${basedir}/delete_template -u ${tiler_url} ${tpl_id}`
if test $? -ne 0; then
echo $out >&2
break
fi
i=$((i+1))
if test `expr $i % 100` -eq 0; then
echo -n "."
fi
done

View File

@@ -1,35 +1,105 @@
#!/usr/bin/env node
// Reset redis-stored XML styles so that they are regenerated
// from CartoCSS on next tile request
/*
var redis = require('redis')
This scripts drops all extended map_style keys in redis and regenerates
the XML caches in all the base ones to target the configured mapnik_version.
var REDIS_PORT = 6379; // TODO: make a parameter
Optionally (with --convert) it also re-writes the CartoCSS if needed
to target the configured mapnik_version.
It is recommended to make a backup of the redis database before using
this script.
*/
var path = require('path');
// Reset all styles in the store
var grainstore = require('../node_modules/windshaft/node_modules/grainstore/lib/grainstore');
var mapnik = require('mapnik');
var redis = require('redis');
function usage(me, exitcode) {
console.log("Usage: " + me + " [--convert] <environment>");
process.exit(exitcode);
}
var doConvert = false;
var node_path = process.argv.shift();
var script_path = process.argv.shift();
var me = path.basename(script_path);
var ENV;
var arg;
while ( arg = process.argv.shift() ) {
if ( arg == '--convert' ) {
doConvert = true;
}
else if ( ! ENV ) {
ENV = arg;
}
else {
usage(me, 1);
}
}
if ( ! ENV ) usage(me, 1);
global.environment = require('../config/environments/' + ENV);
var serverOptions = require('../lib/cartodb/server_options'); // _after_ setting global.environment
var MAPNIK_VERSION = global.environment.mapnik_version || mapnik.versions.mapnik;
console.log( (doConvert ? "Converting" : "Resetting" ) + ' all styles to target ' + MAPNIK_VERSION);
var dbnum = 0;
var client = redis.createClient(REDIS_PORT, 'localhost');
var mml_store = new grainstore.MMLStore(serverOptions.redis, serverOptions.grainstore);
var failures = [];
var client = redis.createClient(serverOptions.redis.port, serverOptions.redis.host);
client.on('connect', function() {
client.select(dbnum);
client.keys('map_style|*', function(err, matches) {
processNext = function() {
if ( ! matches.length ) process.exit(0);
if ( ! matches.length ) process.exit(failures.length);
var k = matches.shift();
console.log("Resetting XML in key: " + k);
client.get(k, function(err, val) {
if ( err ) throw err;
val = JSON.parse(val);
delete val.xml;
client.set(k, JSON.stringify(val), function() {
console.log("done with style " + k);
if ( /map_style\|.*\|.*\|/.test(k) ) {
// See https://github.com/Vizzuality/Windshaft-cartodb/issues/58
//console.warn("Key " + k + " is EXTENDED, dropping");
client.del(k, function(err) {
if ( err ) console.warn("Error dropping key " + k);
processNext();
});
return;
}
var params = RegExp(/map_style\|(.*)\|(.*)/).exec(k);
var db = params[1];
var tab = params[2];
var out = 'map_style|' + db + '|' + tab + ': ';
var mml_builder = mml_store.mml_builder({dbname:db, table:tab},
function(err, payload) {
if ( err ) { console.warn(out + err.message); failures.push(k); processNext(); }
else {
mml_builder.resetStyle(function(err, data) {
if ( err ) { console.warn(out + err.message); failures.push(k); }
else console.log(out + 'OK' + ( doConvert ? ' (converted)' : '' ));
processNext();
}, doConvert);
}
});
}
};
processNext();
});
});

147
tools/show_style Executable file
View File

@@ -0,0 +1,147 @@
#!/usr/bin/env node
var path = require('path');
var redis = require('redis');
var Step = require('step');
function usage(me, exitcode) {
console.log("Usage: " + me + " [--env <environment>] <username> [<tablename>|~<token>]");
process.exit(exitcode);
}
var node_path = process.argv.shift();
var script_path = process.argv.shift();
var basedir = path.dirname(script_path);
var me = path.basename(script_path);
var ENV = 'development.js';
var username, token;
var arg;
while ( arg = process.argv.shift() ) {
if ( arg == '--env' ) {
ENV = process.argv.shift();
}
else if ( ! username ) {
username = arg;
}
else if ( ! token ) {
token = arg;
}
else {
console.warn("Unused parameter " + arg);
}
}
if ( ! username ) usage(me, 1);
console.log("Using environment " + ENV);
global.environment = require('../config/environments/' + ENV);
var serverOptions = require('../lib/cartodb/server_options'); // _after_ setting global.environment
var client;
var dbname;
Step(
function getClient() {
client = redis.createClient(serverOptions.redis.port, serverOptions.redis.host);
client.on('connect', this);
},
function getUserMeta(err) {
if ( err ) throw err;
client.select(5);
client.hgetall('rails:users:' + username, this);
},
function readDB(err, data) {
if ( err ) throw err;
if ( ! data )
throw new Error('Username ' + username + ' unknown by redis on port '
+ serverOptions.redis.port + ' (try CARTODB/script/restore_redis?)');
//console.log("Data:"); console.dir(data);
dbname = data['database_name'];
console.log("Database name for user " + username + ": " + dbname);
client.select(0);
return null;
},
function showTokens(err) {
if ( err ) throw err;
if ( token ) return null;
var next = this;
Step(
function getTokens() {
client.keys('map_style|' + dbname + '|*', this);
},
function showTokens(err, data) {
if (err) throw err;
if ( data ) console.log(data.join('\n'));
return null;
},
function showTokensFinish(err) {
next(err);
}
);
},
function showStyle(err) {
if ( err ) throw err;
if ( ! token ) return null;
var next = this;
Step(
function getStyle() {
client.get('map_style|' + dbname + '|' + token, this);
},
function showStyle(err, data) {
if ( err ) throw err;
if ( ! data ) {
throw new Error(token + ': no such map style known by redis on port '
+ serverOptions.redis.port);
}
//console.log("data: " + data);
var x=JSON.parse(data);
printMapnikStyle(x, this);
},
function showStyleFinish(err) {
next(err);
}
);
},
function finish(err) {
if ( err ) {
console.error(err.message)
process.exit(1);
}
process.exit(0);
}
);
function printMapnikStyle(x, callback) {
console.log('style: ' + x.style);
console.log('version: ' + x.version);
var grainstore = require(basedir + '/../node_modules/windshaft/node_modules/grainstore/lib/grainstore');
var mml_store = new grainstore.MMLStore(serverOptions.redis, serverOptions.grainstore);
var builderconfig = {dbname:dbname};
if ( token.match(/^~/) ) {
builderconfig.token = token.substring(1);
} else {
builderconfig.table = token;
}
var mml_builder;
Step(
function getBuilder() {
mml_builder = mml_store.mml_builder(builderconfig, this);
},
function getXML(err, builder) {
if ( err ) throw err;
mml_builder.toXML(this);
},
function showXML(err, xml) {
if ( err ) throw err;
console.log('- XML - ');
console.log(xml);
return null;
},
function finish(err) {
callback(err);
}
);
}

47
tools/update_template Executable file
View File

@@ -0,0 +1,47 @@
#!/bin/sh
verbose=no
tiler_url=http://dev.localhost.lan:8181/tiles/template
apikey=${CDB_APIKEY}
while test -n "$1"; do
if test "$1" = "-v"; then
verbose=yes
elif test "$1" = "-k"; then
shift
apikey="$1"
elif test "$1" = "-u"; then
shift
tiler_url="$1"
elif test -z "$tpl"; then
tpl="$1"
elif test -z "$cfg"; then
cfg="$1"
else
echo "Unused parameter $1" >&2
fi
shift
done
if test -z "$cfg"; then
echo "Usage: $0 [-v] [-k <api_key>] [-u <tiler_url>] <template_id> <template_config>" >&2
echo "Default <tiler_url> is ${tiler_url}" >&2
echo "Default <api_key> is read from CDB_APIKEY env variable" >&2
exit 1
fi
cmd="curl -X PUT -skH Content-Type:application/json --data-binary @- ${tiler_url}/${tpl}?api_key=${apikey}"
if test x${verbose} = xyes; then
cmd="${cmd} -v"
fi
res=`cat ${cfg} | tr '\n' ' ' | ${cmd}`
if test $? -gt 0; then
echo "curl command failed: ${cmd}"
fi
if test x${verbose} = xyes; then
echo "${res}"
fi
tok=`echo "$res" | sed 's/.*"template_id":"\([^"]*\)".*/\1/'`
echo $tok