Release 2.88.4

Merge pull request #636 from CartoDB/upgrade-camshaft
Upgrades camshaft to 0.50.3
2017-03-10 11:06:06 +01:00 · 2017-03-10 11:05:30 +01:00 · 2017-03-10 10:58:07 +01:00 · 2017-03-02 11:00:53 +01:00 · 2017-03-02 10:48:20 +01:00 · 2017-02-23 18:12:54 +01:00
8 changed files with 139 additions and 74 deletions
--- a/2
+++ b/2
@@ -1,7 +1,7 @@
 1. Test (make clean all check), fix if broken before proceeding
 2. Ensure proper version in package.json
 3. Ensure NEWS section exists for the new version, review it, add release date
-4. Recreate npm-shrinkwrap.json with: `npm install --no-shrinkwrap && npm shrinkwrap`
+4. Recreate npm-shrinkwrap.json with: `make shrinkwrap`
 5. Commit package.json, npm-shrinwrap.json, NEWS
 6. git tag -a Major.Minor.Patch # use NEWS section as content
 7. Stub NEWS/package for next version
--- a/8
+++ b/8
@@ -7,7 +7,13 @@ all:
 	@$(SHELL) ./scripts/install.sh

 clean:
-	rm -rf node_modules/*
+	rm -rf node_modules/
+
+shrinkwrap: clean
+	rm npm-shrinkwrap.json
+	npm install --no-shrinkwrap --production
+	npm prune
+	npm shrinkwrap

 distclean: clean
 	rm config.status*
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,5 +1,32 @@
 # Changelog

+## 2.88.4
+Released 2017-03-10
+
+Announcements:
+ - Upgrades camshaft to [0.50.3](https://github.com/CartoDB/camshaft/releases/tag/0.50.3).
+
+
+## 2.88.3
+Released 2017-03-02
+
+Bug fixes:
+- Category dataviews now uses the proper aggregation function for the 'Other' category. See https://github.com/CartoDB/Windshaft-cartodb/issues/628
+
+## 2.88.2
+Released 2017-02-23
+
+Announcements:
+ - Upgrades camshaft to [0.50.2](https://github.com/CartoDB/camshaft/releases/tag/0.50.2).
+
+
+## 2.88.1
+Released 2017-02-21
+
+Announcements:
+ - Upgrades camshaft to [0.50.1](https://github.com/CartoDB/camshaft/releases/tag/0.50.1)
+
+
 ## 2.88.0
 Released 2017-02-21

@@ -8,6 +35,7 @@ Announcements:
 - Upgrades cartodb-psql to [0.7.1](https://github.com/CartoDB/node-cartodb-psql/releases/tag/0.7.1).
 - Upgrades windshaft to [2.7.0](https://github.com/CartoDB/windshaft/releases/tag/2.7.0).

+
 ## 2.87.5
 Released 2017-02-02

@@ -21,17 +49,20 @@ Released 2017-01-20
 Bug fixes:
 - Be able to not compute NULL categories and null values  in aggregation dataviews #617.

+
 ## 2.87.3
 Released 2016-12-19

 Bug fixes:
 - Fix overviews-related dataviews problems. See https://github.com/CartoDB/Windshaft-cartodb/pull/604

+
 ## 2.87.2
 Released 2016-12-19

 - Use exception safe Dataservices API functions. See https://github.com/CartoDB/dataservices-api/issues/314 and https://github.com/CartoDB/camshaft/issues/242

+
 ## 2.87.1
 Released 2016-12-13

--- a/lib/cartodb/controllers/layergroup.js
+++ b/lib/cartodb/controllers/layergroup.js
@@ -338,6 +338,8 @@ LayergroupController.prototype.staticMap = function(req, res, width, height, zoo
 LayergroupController.prototype.sendResponse = function(req, res, body, status, headers) {
    var self = this;

+    req.profiler.done('res');
+
    res.set('Cache-Control', 'public,max-age=31536000');

    // Set Last-Modified header
--- a/lib/cartodb/models/dataview/aggregation.js
+++ b/lib/cartodb/models/dataview/aggregation.js
@@ -48,7 +48,8 @@ var rankedAggregationQueryTpl = dot.template([
    '  FROM categories, summary, categories_summary_min_max, categories_summary_count',
    '  WHERE rank < {{=it._limit}}',
    'UNION ALL',
-    'SELECT \'Other\' category, sum(value), true as agg, nulls_count, min_val, max_val, count, categories_count',
+    'SELECT \'Other\' category, {{=it._aggregationFn}}(value) as value, true as agg, nulls_count, min_val, max_val,',
+    '       count, categories_count',
    '  FROM categories, summary, categories_summary_min_max, categories_summary_count',
    '  WHERE rank >= {{=it._limit}}',
    'GROUP BY nulls_count, min_val, max_val, count, categories_count'
@@ -129,27 +130,7 @@ Aggregation.prototype.sql = function(psql, override, callback) {

    if (!!override.ownFilter) {
        aggregationSql = [
-            "WITH",
-            [
-                summaryQueryTpl({
-                    _query: _query,
-                    _column: this.column
-                }),
-                rankedCategoriesQueryTpl({
-                    _query: _query,
-                    _column: this.column,
-                    _aggregation: this.getAggregationSql(),
-                    _aggregationColumn: this.aggregation !== 'count' ? this.aggregationColumn : null
-                }),
-                categoriesSummaryMinMaxQueryTpl({
-                    _query: _query,
-                    _column: this.column
-                }),
-                categoriesSummaryCountQueryTpl({
-                    _query: _query,
-                    _column: this.column
-                })
-            ].join(',\n'),
+            this.getCategoriesCTESql(_query, this.column, this.aggregation, this.aggregationColumn),
            aggregationQueryTpl({
                _query: _query,
                _column: this.column,
@@ -159,30 +140,11 @@ Aggregation.prototype.sql = function(psql, override, callback) {
        ].join('\n');
    } else {
        aggregationSql = [
-            "WITH",
-            [
-                summaryQueryTpl({
-                    _query: _query,
-                    _column: this.column
-                }),
-                rankedCategoriesQueryTpl({
-                    _query: _query,
-                    _column: this.column,
-                    _aggregation: this.getAggregationSql(),
-                    _aggregationColumn: this.aggregation !== 'count' ? this.aggregationColumn : null
-                }),
-                categoriesSummaryMinMaxQueryTpl({
-                    _query: _query,
-                    _column: this.column
-                }),
-                categoriesSummaryCountQueryTpl({
-                    _query: _query,
-                    _column: this.column
-                })
-            ].join(',\n'),
+            this.getCategoriesCTESql(_query, this.column, this.aggregation, this.aggregationColumn),
            rankedAggregationQueryTpl({
                _query: _query,
                _column: this.column,
+                _aggregationFn: this.aggregation !== 'count' ? this.aggregation : 'sum',
                _limit: CATEGORIES_LIMIT
            })
        ].join('\n');
@@ -193,6 +155,32 @@ Aggregation.prototype.sql = function(psql, override, callback) {
    return callback(null, aggregationSql);
 };

+Aggregation.prototype.getCategoriesCTESql = function(query, column, aggregation, aggregationColumn) {
+    return [
+                "WITH",
+                [
+                    summaryQueryTpl({
+                        _query: query,
+                        _column: column
+                    }),
+                    rankedCategoriesQueryTpl({
+                        _query: query,
+                        _column: column,
+                        _aggregation: this.getAggregationSql(),
+                        _aggregationColumn: aggregation !== 'count' ? aggregationColumn : null
+                    }),
+                    categoriesSummaryMinMaxQueryTpl({
+                        _query: query,
+                        _column: column
+                    }),
+                    categoriesSummaryCountQueryTpl({
+                        _query: query,
+                        _column: column
+                    })
+                ].join(',\n')
+            ].join('\n');
+};
+
 var aggregationFnQueryTpl = dot.template('{{=it._aggregationFn}}({{=it._aggregationColumn}})');
 Aggregation.prototype.getAggregationSql = function() {
    return aggregationFnQueryTpl({
--- a/npm-shrinkwrap.json
+++ b/npm-shrinkwrap.json
@@ -1,6 +1,6 @@
 {
  "name": "windshaft-cartodb",
-  "version": "2.88.0",
+  "version": "2.88.4",
  "dependencies": {
    "body-parser": {
      "version": "1.14.2",
@@ -105,9 +105,9 @@
      }
    },
    "camshaft": {
-      "version": "0.50.0",
-      "from": "camshaft@0.50.0",
-      "resolved": "https://registry.npmjs.org/camshaft/-/camshaft-0.50.0.tgz",
+      "version": "0.50.3",
+      "from": "camshaft@0.50.3",
+      "resolved": "https://registry.npmjs.org/camshaft/-/camshaft-0.50.3.tgz",
      "dependencies": {
        "async": {
          "version": "1.5.2",
@@ -228,9 +228,9 @@
              }
            },
            "safe-json-stringify": {
-              "version": "1.0.3",
+              "version": "1.0.4",
              "from": "safe-json-stringify@>=1.0.0 <2.0.0",
-              "resolved": "https://registry.npmjs.org/safe-json-stringify/-/safe-json-stringify-1.0.3.tgz"
+              "resolved": "https://registry.npmjs.org/safe-json-stringify/-/safe-json-stringify-1.0.4.tgz"
            },
            "moment": {
              "version": "2.17.1",
@@ -869,9 +869,9 @@
              }
            },
            "is-my-json-valid": {
-              "version": "2.15.0",
+              "version": "2.16.0",
              "from": "is-my-json-valid@>=2.12.4 <3.0.0",
-              "resolved": "https://registry.npmjs.org/is-my-json-valid/-/is-my-json-valid-2.15.0.tgz",
+              "resolved": "https://registry.npmjs.org/is-my-json-valid/-/is-my-json-valid-2.16.0.tgz",
              "dependencies": {
                "generate-function": {
                  "version": "2.0.0",
@@ -976,9 +976,9 @@
              }
            },
            "sshpk": {
-              "version": "1.10.2",
+              "version": "1.11.0",
              "from": "sshpk@>=1.7.0 <2.0.0",
-              "resolved": "https://registry.npmjs.org/sshpk/-/sshpk-1.10.2.tgz",
+              "resolved": "https://registry.npmjs.org/sshpk/-/sshpk-1.11.0.tgz",
              "dependencies": {
                "asn1": {
                  "version": "0.2.3",
@@ -1062,9 +1062,9 @@
          "resolved": "https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.8.2.tgz"
        },
        "qs": {
-          "version": "6.3.1",
+          "version": "6.3.2",
          "from": "qs@>=6.3.0 <6.4.0",
-          "resolved": "https://registry.npmjs.org/qs/-/qs-6.3.1.tgz"
+          "resolved": "https://registry.npmjs.org/qs/-/qs-6.3.2.tgz"
        },
        "stringstream": {
          "version": "0.0.5",
@@ -1189,9 +1189,9 @@
          }
        },
        "carto": {
-          "version": "0.15.1-cdb1",
+          "version": "0.15.1-cdb2",
          "from": "cartodb/carto#0.15.1-cdb2",
-          "resolved": "git://github.com/cartodb/carto.git#673cf381215948269827052afa152e485b01e2f0",
+          "resolved": "git://github.com/cartodb/carto.git#8e7d8c299078104940b3beb70e7bdcf6a34c5784",
          "dependencies": {
            "mapnik-reference": {
              "version": "6.0.5",
@@ -3369,7 +3369,7 @@
          "dependencies": {
            "strip-ansi": {
              "version": "3.0.1",
-              "from": "strip-ansi@>=3.0.0 <4.0.0",
+              "from": "strip-ansi@>=3.0.1 <4.0.0",
              "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
              "dependencies": {
                "ansi-regex": {
@@ -3470,9 +3470,9 @@
                      "resolved": "https://registry.npmjs.org/parse-json/-/parse-json-2.2.0.tgz",
                      "dependencies": {
                        "error-ex": {
-                          "version": "1.3.0",
+                          "version": "1.3.1",
                          "from": "error-ex@>=1.2.0 <2.0.0",
-                          "resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.0.tgz",
+                          "resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.1.tgz",
                          "dependencies": {
                            "is-arrayish": {
                              "version": "0.2.1",
@@ -3515,9 +3515,9 @@
                  }
                },
                "normalize-package-data": {
-                  "version": "2.3.5",
+                  "version": "2.3.6",
                  "from": "normalize-package-data@>=2.3.2 <3.0.0",
-                  "resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-2.3.5.tgz",
+                  "resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-2.3.6.tgz",
                  "dependencies": {
                    "hosted-git-info": {
                      "version": "2.2.0",
@@ -3735,9 +3735,9 @@
                      "resolved": "https://registry.npmjs.org/delegates/-/delegates-1.0.0.tgz"
                    },
                    "readable-stream": {
-                      "version": "2.2.2",
+                      "version": "2.2.3",
                      "from": "readable-stream@>=2.0.0 <3.0.0||>=1.1.13 <2.0.0",
-                      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.2.2.tgz",
+                      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.2.3.tgz",
                      "dependencies": {
                        "buffer-shims": {
                          "version": "1.0.0",
@@ -3919,7 +3919,7 @@
                    },
                    "minimatch": {
                      "version": "3.0.3",
-                      "from": "minimatch@>=3.0.0 <4.0.0",
+                      "from": "minimatch@>=3.0.2 <4.0.0",
                      "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.3.tgz",
                      "dependencies": {
                        "brace-expansion": {
@@ -3978,9 +3978,9 @@
                  "resolved": "https://registry.npmjs.org/block-stream/-/block-stream-0.0.9.tgz"
                },
                "fstream": {
-                  "version": "1.0.10",
+                  "version": "1.0.11",
                  "from": "fstream@>=1.0.2 <2.0.0",
-                  "resolved": "https://registry.npmjs.org/fstream/-/fstream-1.0.10.tgz",
+                  "resolved": "https://registry.npmjs.org/fstream/-/fstream-1.0.11.tgz",
                  "dependencies": {
                    "graceful-fs": {
                      "version": "4.1.11",
@@ -4002,9 +4002,9 @@
              "resolved": "https://registry.npmjs.org/tar-pack/-/tar-pack-3.3.0.tgz",
              "dependencies": {
                "fstream": {
-                  "version": "1.0.10",
+                  "version": "1.0.11",
                  "from": "fstream@>=1.0.10 <1.1.0",
-                  "resolved": "https://registry.npmjs.org/fstream/-/fstream-1.0.10.tgz",
+                  "resolved": "https://registry.npmjs.org/fstream/-/fstream-1.0.11.tgz",
                  "dependencies": {
                    "graceful-fs": {
                      "version": "4.1.11",
--- a/package.json
+++ b/package.json
@@ -1,7 +1,7 @@
 {
  "private": true,
  "name": "windshaft-cartodb",
-  "version": "2.88.0",
+  "version": "2.88.4",
  "description": "A map tile server for CartoDB",
  "keywords": [
    "cartodb"
@@ -20,7 +20,7 @@
  ],
  "dependencies": {
    "body-parser": "~1.14.0",
-    "camshaft": "0.50.0",
+    "camshaft": "0.50.3",
    "cartodb-psql": "~0.7.1",
    "cartodb-query-tables": "~0.1.0",
    "cartodb-redis": "0.13.1",
--- a/test/acceptance/dataviews/aggregation.js
+++ b/test/acceptance/dataviews/aggregation.js
@@ -1,5 +1,4 @@
 require('../../support/test_helper');
-
 var assert = require('../../support/assert');
 var TestClient = require('../../support/test-client');

@@ -107,4 +106,43 @@ describe('aggregations happy cases', function() {
            });
        });
    });
+
+    var operations_and_values = {'count': 9, 'sum': 45, 'avg': 5, 'max': 9, 'min': 1};
+
+    var query_other = [
+        'select generate_series(1,3) as val, \'other_a\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(4,6) as val, \'other_b\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(7,9) as val, \'other_c\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(10,12) as val, \'category_1\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(10,12) as val, \'category_2\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(10,12) as val, \'category_3\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(10,12) as val, \'category_4\' as cat, NULL as the_geom_webmercator',
+        'select generate_series(10,12) as val, \'category_5\' as cat, NULL as the_geom_webmercator'
+    ].join(' UNION ALL ');
+
+    Object.keys(operations_and_values).forEach(function (operation) {
+        var description = 'should aggregate OTHER category using "' + operation + '"';
+
+        it(description, function (done) {
+            this.testClient = new TestClient(aggregationOperationMapConfig(operation, query_other, 'cat', 'val'));
+            this.testClient.getDataview('cat', { own_filter: 0 }, function (err, aggregation) {
+                assert.ifError(err);
+
+                assert.ok(aggregation);
+                assert.equal(aggregation.type, 'aggregation');
+                assert.ok(aggregation.categories);
+                assert.equal(aggregation.categoriesCount, 8);
+                assert.equal(aggregation.count, 24);
+                assert.equal(aggregation.nulls, 0);
+
+                var aggregated_categories = aggregation.categories.filter( function(category) {
+                    return category.agg === true;
+                });
+                assert.equal(aggregated_categories.length, 1);
+                assert.equal(aggregated_categories[0].value, operations_and_values[operation]);
+
+                done();
+            });
+        });
+    });
 });
Author	SHA1	Message	Date
Raul Ochoa	8789a959e5	Release 2.88.4	2017-03-10 11:06:06 +01:00
Raul Ochoa	5765ac59cc	Merge pull request #636 from CartoDB/upgrade-camshaft Upgrades camshaft to 0.50.3	2017-03-10 11:05:30 +01:00
Raul Ochoa	3b9bf96431	Upgrades camshaft to 0.50.3	2017-03-10 10:58:07 +01:00
Mario de Frutos	8868066445	Stubs next version	2017-03-02 11:00:53 +01:00
Mario de Frutos	b446c31cbc	Other category now uses the selected aggregated function (#633 ) * Other category in category widget uses selected aggregation function Fixes https://github.com/CartoDB/Windshaft-cartodb/issues/628	2017-03-02 10:48:20 +01:00
Raul Ochoa	2d6e7070a6	Stubs next version	2017-02-23 18:12:54 +01:00
Raul Ochoa	473e0cb902	Release 2.88.2	2017-02-23 18:12:14 +01:00
Raul Ochoa	1a8fca0534	Formatting	2017-02-23 18:11:39 +01:00
Raul Ochoa	e24bc12fc9	Merge pull request #632 from CartoDB/upgrade-camshaft Upgrades camshaft to 0.50.2	2017-02-23 18:10:22 +01:00
Raul Ochoa	e77c9141ed	Upgrades camshaft to 0.50.2	2017-02-23 18:00:55 +01:00
Raul Ochoa	321157b17b	Make target to generate shrinkwrap file applying prune It removes dev dependencies from the local node_modules and avoid getting those into shrinkwrap file.	2017-02-23 17:57:52 +01:00
Raul Ochoa	6ac6574b4c	clean target to completely delete node_modules dir	2017-02-23 17:57:44 +01:00
Raul Ochoa	933d486a57	Merge pull request #630 from CartoDB/close-response-timer Close timer for response preparation	2017-02-22 18:46:56 +01:00
Raul Ochoa	8a1c7f5b52	Close timer for response preparation Timer for affectedTables is taking everything from response timing, adding a tag to represent all the response preparation. That way affectedTables only represents the time for retrieving the affected tables themselves.	2017-02-22 18:38:25 +01:00
Rafa de la Torre	822954be5d	Stub next version	2017-02-21 11:06:54 +01:00
Rafa de la Torre	57dc17518c	Release 2.88.1	2017-02-21 11:01:15 +01:00
Mario de Frutos	3df8d4844e	Stubs next version	2017-02-21 08:17:09 +01:00