release artifact

clarification of docs for obs_getboundariesbygeometry function

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,6 +2,12 @@
 
 I'd like to request a new data observatory extension deploy: dump + extension
 
+## Performance comparison to last deployment
+
+Please include link here to comparison perftests:
+
+http://52.71.151.140/perftest/#oldsha..newsha
+
 ## Dump database id to be deployed
 
 Please put here the dump id to be deployed: <dump_id>

NEWS.md

@@ -1,3 +1,164 @@
+1.4.0 (2017-03-21)
+
+__API Changes__
+
+* Allow for override of `target_area` and `target_geoms` in `OBS_GetMeta`
+  ([#276](https://github.com/CartoDB/observatory-extension/pull/265)).  This
+  allows the interface to work with points and sparse areas much btter.
+* Allow for override of `max_timespan_rank` and `max_score_rank` on an
+  item-by-item basis for metadata.
+* `numer_description`, `geom_description`, `denom_description`,
+  `numer_t_description`, `denom_t_description` and `geom_t_description` now
+  returned as part of `OBS_GetMeta`.
+
+__Improvements__
+
+* Reduced amount of simplification done on input geometries (from 0.0001 above
+  500 points to 0.00001 above 1000 points).
+* Added tests to confirm that accurate results are returned from automatic
+  boundary selection
+
+1.3.5 (2017-03-15)
+
+No changes.  Artifact to allow for data update.
+
+1.3.4 (2017-03-10)
+
+__Bugfixes__
+
+* Remove erroneously committed `RAISE NOTICE` in `OBS_GetData`
+
+1.3.3 (2017-03-10)
+
+__Bugfixes__
+
+* Resolve divide-by-zero errors in cases where the intersection of an
+  Observatory geometry and user geometry has 0 area
+  ([#265](https://github.com/CartoDB/observatory-extension/pull/265))
+* Run MakeValid on geometry's when intersecting, if necessary
+  ([#268](https://github.com/CartoDB/observatory-extension/pull/268))
+
+__Improvements__
+
+* Add performance tests for multiple columns in `OBS_GetData`
+* Major performance boost for `autotest.py` through the use of multi-column
+  `OBS_GetData` instead of separate `OBS_GetMeasure` calls for every single
+  measurement.
+  ([#268](https://github.com/CartoDB/observatory-extension/pull/268))
+* Major performance boost for `OBS_GetData` in cases where multiple columns are
+  requested.  Previously, each additional column would result in a linear
+  slowdown, even if geometries could be reused.
+  ([#267](https://github.com/CartoDB/observatory-extension/pull/267))
+
+1.3.2 (2017-03-02)
+
+__Bugfixes__
+
+* Accept "prenormalized" as well as "predenominated" to bypass normalization.
+  This fixes issues with Camshaft.
+
+1.3.1 (2017-02-16)
+
+__Improvements__
+
+* It is now possible to obtain measures that are averages or medians over
+  arbitrary polygons ([#254](https://github.com/CartoDB/observatory-extension/pull/254).
+* Added test point for Australian data
+* `OBS_GetLegacyMetadata` now returns median and averages in cases where it is
+  called for measures for polygons
+
+1.3.0 (2017-01-17)
+
+__API Changes__
+
+* `OBS_GetMeasureDataMulti()` is now called `OBS_GetData()`
+* `OBS_GetMeasureMetaMulti()` is now called `OBS_GetMeta()`
+* Additional signature for `OBS_GetData` which can take an array of `TEXT`,
+  mimicking functionality of `OBS_GetMeasureByID`
+
+__Improvements__
+
+* Generate fixtures from `obs_meta`
+* Remove unused table-level code
+* Refactor all augmentation and geometry functions to obtain data from
+  `OBS_GetMeta()` and `OBS_GetData()`.
+* Improvements to `OBS_GetMeta()` so it can still fill in metadata in cases
+  where only a geometry is being requested.
+* `OBS_GetData()` returns two-column table instead of anonymous record.
+* `OBS_GetData()` can return categorical (text) and geometries
+
+__Bugfixes__
+
+* Remove unnecessary dependency on `postgres_fdw`
+* `OBS_GetData()` now aggregates measures with mixed geoms correctly
+
+__API Changes__
+
+1.2.1 (2017-01-17)
+
+__Improvements__
+
+* Support Point/LineString in responses from `OBS_GetBoundary`.
+  ([#243](https://github.com/CartoDB/observatory-extension/pull/233))
+
+1.2.0 (2016-12-28)
+
+__API Changes__
+
+* Added `OBS_GetMeasureDataMulti`, which takes an array of geomvals and
+  parameters as JSON, and returns a set of RECORDs keyed by the vals of the
+  geomvals.
+* Added `OBS_GetMeasureMetaMulti`, which takes sparse metadata as JSON (for
+  example, the measure ID) and returns a filled-out version of the metadata
+  sufficient for use with `OBS_GetMeasureDataMulti`.
+
+__Improvements__
+
+* Move tests to 2015
+* Fixes to `_OBS_GetGeometryScores` to avoid spamming NOTICEs about all pixels
+  for a band being NULL
+* Tests for `_OBS_GetGeometryScores` with complex geometries
+* Performance tests for `OBS_GetMeasureDataMulti`
+* Return both `table_id` and `column_id` from `_OBS_GetGeometryScores`
+
+1.1.7 (2016-12-15)
+
+__Improvements__
+
+* Use simpler raster table and simplified `_OBSGetGeometryScores` functions to
+  improve performance
+* In cases where geometry passed into geometry scoring function has greater
+  than 10K points, simply use its buffer instead
+* Add `IMMUTABLE` to `_OBSGetGeometryScores`
+* Add tests explicitly for `_OBSGetGeometryScores` in perftest.py
+* Yields a ~50% improvement in performance for `_OBSGetGeomeryScores`.
+
+1.1.6 (2016-12-08)
+
+__Bugfixes__
+
+* Fix divide by zero condition in "denominator" branch of `OBS_GetMeasure`
+  when passing in a polygon ([#233](https://github.com/CartoDB/observatory-extension/pull/233)).
+
+__Improvements__
+
+* Use `ST_Subdivide` to improve performance when functions are called on very
+  complex geometries (with many points) ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Improve raster scoring to more heavily weight boundaries with nearer to
+  correct number of points, and penalize boundaries with lots of blank space
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Remove some redundant area calculations in `OBS_GetMeasure`
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Replace use of `format('%L', var)` with proper use of `EXECUTE` and `$1` etc.
+  variables ([#231](https://github.com/CartoDB/observatory-extension/pull/231))
+* Add test point for Brazil
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+* Improvements to performance tests
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+  - Support simple and complex geometries
+  - Handle all code branches
+  - Add ability to persist results to JSON for graph visualization later
+
 1.1.5 (2016-11-29)
 
 __Bugfixes__

doc/boundary_functions.md

@@ -2,9 +2,9 @@
 
 Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
-## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
+## OBS_GetBoundariesByGeometry(geom geometry, geometry_id text)
 
 The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a set of boundary geometries that intersect a supplied geometry. This can be used to find all boundaries that are within or overlap a bounding box. You have the ability to choose whether to retrieve all boundaries that intersect your supplied bounding box or only those that fall entirely inside of your bounding box.
 
@@ -12,7 +12,7 @@ The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a se
 
 Name |Description
 --- | ---
-polygon | a bounding box or other WGS84 geometry
+geom | a WGS84 geometry
 geometry_id | a string identifier for a boundary geometry
 timespan (optional) | year(s) to request from ('NULL' (default) gives most recent)
 overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
@@ -26,7 +26,7 @@ Column Name | Description
 the_geom | a boundary geometry (e.g., US Census tract boundaries)
 geom_refs | a string identifier for the geometry (e.g., geoids of US Census tracts)
 
-If geometries are not found for the requested `polygon`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
+If geometries are not found for the requested `geom`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
 
 #### Example
 
@@ -44,7 +44,6 @@ FROM OBS_GetBoundariesByGeometry(
 
 #### Errors
 
-* If a geometry other than a point is passed as the first argument, an error is thrown: `Invalid geometry type (ST_Polygon), expecting 'ST_Point'`
 * If an `overlap_type` other than the valid ones listed above is entered, then an error is thrown
 
 ## OBS_GetPointsByGeometry(polygon geometry, geometry_id text)

doc/discovery_functions.md

@@ -56,3 +56,306 @@ time_span | the timespan attached the boundary. this does not mean that the boun
 ```SQL
 SELECT * FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.7, -73.9))
 ```
+
+## OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan)
+
+Return available numerators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the numerator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only numerators for which all of these apply are returned  `NULL` to ignore (optional)
+denom_id | Text | the ID of a denominator to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+numer_id | Text | The ID of the numerator
+numer_name | Text | A human readable name for the numerator
+numer_description | Text | Description of the numerator.  Is sometimes NULL
+numer_weight | Numeric | Numeric "weight" of the numerator.  Ignored.
+numer_license | Text | ID of the license for the numerator
+numer_source | Text | ID of the source for the numerator
+numer_type | Text | Postgres type of the numerator
+numer_aggregate | Text | Aggregate type of the numerator.  If `'SUM'`, this can be normalized by area
+numer_extra | JSONB | Extra information about the numerator column.  Ignored.
+numer_tags | Text[] | Array of all tags applying to this numerator
+valid_denom | Boolean | True if the `denom_id` argument is a valid denominator for this numerator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this numerator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this numerator, False otherwise
+
+#### Examples
+
+Obtain all numerators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326))
+```
+
+Obtain all numerators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+employment related for the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+related to both employment and age & gender for the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment, subsection/tags.age_gender}');
+```
+
+Obtain all numerators that work with US population (`us.census.acs.B01003001`)
+as a denominator.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_denom IS True;
+```
+
+Obtain all numerators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all numerators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan)
+
+Return available denominators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the denominator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only denominators for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+geom_id | Text | the ID of a geometry to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+denom_id | Text | The ID of the denominator
+denom_name | Text | A human readable name for the denominator
+denom_description | Text | Description of the denominator.  Is sometimes NULL
+denom_weight | Numeric | Numeric "weight" of the denominator.  Ignored.
+denom_license | Text | ID of the license for the denominator
+denom_source | Text | ID of the source for the denominator
+denom_type | Text | Postgres type of the denominator
+denom_aggregate | Text | Aggregate type of the denominator.  If `'SUM'`, this can be normalized by area
+denom_extra | JSONB | Extra information about the denominator column.  Ignored.
+denom_tags | Text[] | Array of all tags applying to this denominator
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this denominator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this denominator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this denominator, False otherwise
+
+#### Examples
+
+Obtain all denominators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all denominators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all denominators for male population (`us.census.acs.B01001002`).
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01001002')
+WHERE valid_numer IS True;
+```
+
+Obtain all denominators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all denominators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan)
+
+Return available geometries within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which must intersect the geometry
+filter_tags | Text[] | a list of filters.  Only geometries for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+timespan | Text | the ID of a timespan to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+geom_id | Text | The ID of the geometry
+geom_name | Text | A human readable name for the geometry
+geom_description | Text | Description of the geometry.  Is sometimes NULL
+geom_weight | Numeric | Numeric "weight" of the geometry.  Ignored.
+geom_aggregate | Text | Aggregate type of the geometry.  Ignored.
+geom_license | Text | ID of the license for the geometry
+geom_source | Text | ID of the source for the geometry
+geom_type | Text | Postgres type of the geometry
+geom_extra | JSONB | Extra information about the geometry column.  Ignored.
+geom_tags | Text[] | Array of all tags applying to this geometry
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this geometry, False otherwise
+valid_denom | Boolean | True if the `geom_id` argument is a valid geometry for this geometry, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this geometry, False otherwise
+score | Numeric | Score between 0 and 100 for this geometry, higher numbers mean that this geometry is a better choice for the passed extent
+numtiles | Numeric | How many raster tiles were read for score, numgeoms, and percentfill estimates
+numgeoms | Numeric | About how many of these geometries fit inside the passed extent
+percentfill | Numeric | About what percentage of the passed extent is filled with these geometries
+estnumgeoms | Numeric | Ignored
+meanmediansize | Numeric | Ignored
+
+#### Examples
+
+Obtain all geometries that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all geometries that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all geometries that work with total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all geometries with timespan  `2015`.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id)
+
+Return available timespans within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the timespan's data must intersect with
+filter_tags | Text[] | a list of filters.  Ignore
+numer_id | Text | the ID of a numerator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+timespan_id | Text | The ID of the timespan
+timespan_name | Text | A human readable name for the timespan
+timespan_description | Text | Ignored
+timespan_weight | Numeric | Ignored
+timespan_license | Text | Ignored
+timespan_source | Text | Ignored
+timespan_aggregate | Text | Ignored
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this timespan, False otherwise
+valid_denom | Boolean | True if the `timespan` argument is a valid timespan for this timespan, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this timespan, False otherwise
+
+#### Examples
+
+Obtain all timespans that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all timespans for total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all timespans that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```

doc/measures_functions.md

@@ -2,21 +2,21 @@
 
 [Data Observatory Measures](https://carto.com/docs/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
-There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
+There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
-You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 
-The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
+The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
 
 #### Arguments
 
 Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
-measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+measure_name | a human-readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -39,7 +39,7 @@ SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
 
 ## OBS_GetUSCensusMeasure(polygon geometry, measure_name text)
 
-The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetUSCensusMeasure``` function below.
+The ```OBS_GetUSCensusMeasure(polygon, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use the ```OBS_GetMeasure``` function below.
 
 #### Arguments
 
@@ -47,7 +47,7 @@ Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
 measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -70,7 +70,7 @@ SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
 
 ## OBS_GetMeasure(point geometry, measure_id text)
 
-The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
+The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
 #### Arguments
 
@@ -78,7 +78,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf). (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html). (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -109,7 +109,7 @@ Name |Description
 --- | ---
 polygon_geometry | a WGS84 polygon geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -132,7 +132,7 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
 
 #### Errors
 
-* If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
+* If an unrecognized normalization type is input, raises error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
 
 ## OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
 
@@ -170,7 +170,7 @@ SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001'
 
 ## OBS_GetCategory(point geometry, category_id text)
 
-The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) for more detail.
+The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/index.html) for more detail.
 
 #### Arguments
 
@@ -195,3 +195,295 @@ Add the Category to an empty column text column based on point locations in your
 UPDATE tablename
 SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
 ```
+
+## OBS_GetMeta(extent geometry, metadata json, max_timespan_rank, max_score_rank, target_geoms)
+
+The ```OBS_GetMeta(extent, metadata)``` function returns a completed Data
+Observatory metadata JSON Object for use in ```OBS_GetData(geomvals,
+metadata)``` or ```OBS_GetData(ids, metadata)```.  It is not possible to pass
+metadata to those functions if it is not processed by ```OBS_GetMeta(extent,
+metadata)``` first.
+
+`OBS_GetMeta` makes it possible to automatically select appropriate timespans
+and boundaries for the measurement you want.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+extent | A geometry of the extent of the input geometries
+metadata | A JSON array composed of metadata input objects.  Each indicates one desired measure for an output column, and optionally additional parameters about that column
+max_timespan_rank | How many historical time periods to include.  Defaults to 1
+max_boundary_score_rank | How many alternative boundary levels to include.  Defaults to 1
+target_geoms | Target number of geometries.  Boundaries with close to this many objects within `extent` will be ranked highest. 
+
+The schema of the metadata input objects are as follows:
+
+Metadata Input Key | Description
+--- | -----------
+numer_id | The identifier for the desired measurement.  If left blank, but a `geom_id` is specified, the column will return a geometry instead of a measurement.
+geom_id | Identifier for a desired geographic boundary level to use when calculating measures.  Will be automatically assigned if undefined.  If defined but `numer_id` is blank, then the column will return a geometry instead of a measurement.
+normalization | The desired normalization.  One of 'area', 'prenormalized', or 'denominated'.  'Area' will normalize the measure per square kilometer, 'prenormalized' will return the original value, and 'denominated' will normalize by a denominator.  Ignored if this metadata object specifies a geometry.
+denom_id | Identifier for a desired normalization column in case `normalization` is 'denominated'.  Will be automatically assigned if necessary.  Ignored if this metadata object specifies a geometry.
+numer_timespan | The desired timespan for the measurement.  Defaults to most recent timespan available if left unspecified.
+geom_timespan | The desired timespan for the geometry.  Defaults to timespan matching numer_timespan if left unspecified.
+target_area | Instead of aiming to have `target_geoms` in the area of the geometry passed as `extent`, fill this area.  Unit is square degrees WGS84.  Set this to `0` if you want to use the smallest source geometry for this element of metadata, for example if you're passing in points.
+target_geoms | Override global `target_geoms` for this element of metadata
+max_timespan_rank | Override global `max_timespan_rank` for this element of metadata
+max_score_rank | Override global `max_score_rank` for this element of metadata
+
+#### Returns
+
+A JSON array composed of metadata output objects.
+
+Key | Description
+--- | -----------
+meta | A JSON array with completed metadata for the requested data, including all keys below
+
+The schema of the metadata output objects are as follows.  You should pass this
+array as-is to ```OBS_GetData```.  If you modify any values the function will
+fail.
+
+Metadata Output Key | Description
+--- | -----------
+numer_id | Identifier for desired measurement
+numer_timespan | Timespan that will be used of the desired measurement
+numer_name | Human-readable name of desired measure
+numer_description | Long human-readable description of the desired measure
+numer_t_description | Further information about the source table
+numer_type | PostgreSQL/PostGIS type of desired measure
+numer_colname | Internal identifier for column name
+numer_tablename | Internal identifier for table
+numer_geomref_colname | Internal identifier for geomref column name
+denom_id | Identifier for desired normalization
+denom_timespan | Timespan that will be used of the desired normalization
+denom_name | Human-readable name of desired measure's normalization
+denom_description | Long human-readable description of the desired measure's normalization
+denom_t_description | Further information about the source table
+denom_type | PostgreSQL/PostGIS type of desired measure's normalization
+denom_colname | Internal identifier for normalization column name
+denom_tablename | Internal identifier for normalization table
+denom_geomref_colname | Internal identifier for normalization geomref column name
+geom_id | Identifier for desired boundary geometry
+geom_timespan | Timespan that will be used of the desired boundary geometry
+geom_name | Human-readable name of desired boundary geometry
+geom_description | Long human-readable description of the desired boundary geometry
+geom_t_description | Further information about the source table
+geom_type | PostgreSQL/PostGIS type of desired boundary geometry
+geom_colname | Internal identifier for boundary geometry column name
+geom_tablename | Internal identifier for boundary geometry table
+geom_geomref_colname | Internal identifier for boundary geometry ref column name
+timespan_rank | Ranking of this measurement by time, most recent is 1, second most recent 2, etc.
+score | The score of this measurement's boundary compared to the `extent` and `target_geoms` passed in.  Between 0 and 100.
+score_rank | The ranking of this measurement's boundary, highest ranked is 1, second is 2, etc.
+numer_aggregate | The aggregate type of the numerator, either `sum`, `average`, `median`, or blank
+denom_aggregate | The aggregate type of the denominator, either `sum`, `average`, `median`, or blank
+normalization | The sort of normalization that will be used for this measure, either `area`, `predenominated`, or `denominated`
+
+#### Examples
+
+Obtain metadata that can augment with one additional column of US population
+data, using a boundary relevant for the geometry provided and latest timespan.
+Limit to only the most recent column most relevant to the extent & density of
+input geometries in `tablename`.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with one additional column of US population
+data, using census tract boundaries.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with two additional columns, one for total
+population and one for male population.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+## OBS_GetData(geomvals array[geomval], metadata json)
+
+The ```OBS_GetData(geomvals, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every Geometry of
+the `geomval` element in the `geomvals` array. The metadata argument must be
+obtained from ```OBS_GetMeta(extent, metadata)```.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+geomvals | An array of `geomval` elements, which are obtained by casting together a `Geometry` and a `Numeric`.  This should be obtained by using `ARRAY_AGG((the_geom, cartodb_id)::geomval)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `geomvals`
+array corresponds to one row:
+
+Column | Type    | Description
+------ | ----    | -----------
+id     | Numeric | ID corresponding to the `val` component of an element of the input `geomvals` array
+data   | JSON    | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every geometry in a table, keyed by cartodb_id:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename)
+SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with a blank numeric column called `pop_density` with population
+densities:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename),
+data AS (
+  SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+Update a table with two measurements at once, population density and household
+density.  The table should already have a Numeric column `pop_density` and
+`household_density`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom),4326),
+    '[{"numer_id": "us.census.acs.B01003001"},{"numer_id": "us.census.acs.B11001001"}]',
+    1, 1, COUNT(*)
+) meta from tablename),
+data AS (
+  SELECT id,
+     data->0->>'value' AS pop_density,
+     data->1->>'value' AS household_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density,
+    household_density = data.household_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+## OBS_GetData(ids array[text], metadata json)
+
+The ```OBS_GetData(ids, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every id of
+the `ids` array. The metadata argument must be obtained from
+`OBS_GetMeta(extent, metadata)`.  When obtaining metadata, one must include
+the `geom_id` corresponding to the boundary that the `ids` refer to.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+ids | An array of `TEXT` elements.  This should be obtained by using `ARRAY_AGG(col_of_geom_refs)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+For this function to work, the `metadata` argument must include a `geom_id`
+that corresponds to the ids found in `col_of_geom_refs`.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `ids` array
+corresponds to one row:
+
+Column | Type | Description
+------ | ---- | -----------
+id     | Text | ID corresponding to an element of the input `ids` array
+data   | JSON | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every row of a table with FIPS code county IDs
+(USA).
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename)
+SELECT id AS fips, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((fips) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with population densities for every FIPS code county ID (USA).
+This table has a blank column called `pop_density` and fips codes stored in a
+column `fips`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename),
+data as (
+  SELECT id AS fips, (data->0->>'value') AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((fips) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE fips = data.id
+```

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.1.5'
-requires = 'postgis, postgres_fdw'
+default_version = '1.4.0'
+requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/generate_fixtures.py

@@ -38,30 +38,22 @@ def get_tablename_query(column_id, boundary_id, timespan):
         timespan, give back the current table hash from the data observatory
     """
     return """
-        SELECT t.tablename, geoid_ct.colname colname, t.id table_id
-        FROM observatory.obs_table t,
-             observatory.obs_column_table geoid_ct,
-             observatory.obs_column_table data_ct
-        WHERE
-             t.id = geoid_ct.table_id AND
-             t.id = data_ct.table_id AND
-             geoid_ct.column_id =
-        (SELECT source_id
-         FROM observatory.obs_column_to_column
-         WHERE target_id = '{boundary_id}'
-         AND reltype = 'geom_ref'
-        ) AND
-        data_ct.column_id = '{column_id}' AND
-        timespan = '{timespan}'
-    """.format(column_id=column_id,
-               boundary_id=boundary_id,
-               timespan=timespan)
+        SELECT numer_tablename, numer_geomref_colname, numer_tid,
+               geom_tablename, geom_geomref_colname, geom_tid
+        FROM observatory.obs_meta
+        WHERE numer_id = '{numer_id}' AND
+              geom_id = '{geom_id}' AND
+              numer_timespan = '{numer_timespan}'
+    """.format(numer_id=column_id,
+               geom_id=boundary_id,
+               numer_timespan=timespan)
 
 
 METADATA_TABLES = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
                    'obs_tag', 'obs_column_to_column', 'obs_dump_version', 'obs_meta',
                    'obs_meta_numer', 'obs_meta_denom', 'obs_meta_geom',
-                   'obs_meta_timespan', 'obs_column_table_tile', ]
+                   'obs_meta_timespan', 'obs_column_table_tile',
+                   'obs_column_table_tile_simple']
 
 FIXTURES = [
     ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
@@ -75,7 +67,7 @@ FIXTURES = [
     ('us.census.acs.B05001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.acs.B08006008_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08301010_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006009_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006011_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006015_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
@@ -129,7 +121,6 @@ FIXTURES = [
     ('us.census.acs.B15003022', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B15003023', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19013001', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B19083001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19301001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B25001001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B25002003', 'us.census.tiger.block_group', '2010 - 2014'),
@@ -142,15 +133,6 @@ FIXTURES = [
     ('us.census.acs.B25081002', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B08134001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B08134002', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134003', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134004', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134005', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134006', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134007', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134008', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134009', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134010', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08135001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001002', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001003', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001004', 'us.census.tiger.block_group', '2010 - 2014'),
@@ -167,24 +149,73 @@ FIXTURES = [
     ('us.census.acs.B19001015', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001016', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001017', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
-    ('us.census.tiger.block_group', 'us.census.tiger.block_group', '2014'),
-    ('us.census.tiger.zcta5', 'us.census.tiger.zcta5', '2014'),
-    ('us.census.tiger.county', 'us.census.tiger.county', '2014'),
     ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.spielman_singleton_segments.X2', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.spielman_singleton_segments.X31', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.spielman_singleton_segments.X55', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-06'),
-    ('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
-    ('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
-    ('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
-    ('us.census.tiger.census_tract_clipped', 'us.census.tiger.census_tract_clipped', '2014'),
+    ('whosonfirst.wof_country_name', 'whosonfirst.wof_country_geom', '2016'),
+    ('us.census.acs.B01003001', 'us.census.tiger.zcta5_clipped', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.block_group_clipped', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.census_tract_clipped', '2010 - 2014'),
+    ('us.census.tiger.fullname', 'us.census.tiger.pointlm_geom', '2016'),
+    ('us.census.tiger.fullname', 'us.census.tiger.prisecroads_geom', '2016'),
+    ('us.census.tiger.name', 'us.census.tiger.county', '2015'),
+    ('us.census.tiger.name', 'us.census.tiger.county_clipped', '2015'),
+    ('us.census.tiger.name', 'us.census.tiger.block_group', '2015'),
 ]
 
 OUTFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
@@ -203,6 +234,7 @@ def dump(cols, tablename, where=''):
                           ' | sed "s:SET search_path.*::" '
                           ' | sed "s:CREATE TABLE :CREATE TABLE observatory.:" '
                           ' | sed "s:ALTER TABLE.*OWNER.*::" '
+                          ' | sed "s:SET idle_in_transaction_session_timeout.*::" '
                           ' >> {outfile}'.format(
                               tablename=tablename,
                               outfile=OUTFILE_PATH,
@@ -233,15 +265,19 @@ def main():
         tablename_query = get_tablename_query(column_id, boundary_id, timespan)
         resp = query(tablename_query).fetchone()
         if resp:
-            tablename, colname, table_id = resp
+            numer_tablename, numer_colname, numer_table_id = resp[0:3]
+            geom_tablename, geom_colname, geom_table_id = resp[3:6]
         else:
-            print("Could not find table for {}, {}, {}".format(
+            raise Exception("Could not find table for {}, {}, {}".format(
                 column_id, boundary_id, timespan))
-            continue
-        table_colname = (tablename, colname, boundary_id, table_id, )
-        if table_colname not in unique_tables:
-            print(table_colname)
-            unique_tables.add(table_colname)
+        numer = (numer_tablename, numer_colname, numer_table_id, )
+        geom = (geom_tablename, geom_colname, geom_table_id, )
+        if numer not in unique_tables:
+            print(numer)
+            unique_tables.add(numer)
+        if geom not in unique_tables:
+            print(geom)
+            unique_tables.add(geom)
 
     print unique_tables
 
@@ -290,13 +326,14 @@ def main():
                 "column_id IN ('{}', '{}')".format(numer_id, geom_id)
                 for numer_id, geom_id, timespan in FIXTURES
             ])
-        elif tablename in ('obs_column_table', 'obs_column_table_tile', ):
-            where = 'WHERE column_id IN ({numer_ids}) ' \
-                    'OR column_id IN ({geom_ids}) ' \
-                    'OR table_id IN ({table_ids}) '.format(
+        elif tablename in ('obs_column_table', 'obs_column_table_tile',
+                           'obs_column_table_tile_simple'):
+            where = '''WHERE table_id IN ({table_ids}) AND
+              (column_id IN ({numer_ids}) OR column_id IN ({geom_ids}))
+                    '''.format(
                         numer_ids=','.join(["'{}'".format(x) for x, _, _ in FIXTURES]),
                         geom_ids=','.join(["'{}'".format(x) for _, x, _ in FIXTURES]),
-                        table_ids=','.join(["'{}'".format(x) for _, _, _, x in unique_tables])
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
                     )
         elif tablename == 'obs_column_to_column':
             where = "WHERE " + " OR ".join([
@@ -308,18 +345,24 @@ def main():
             where = 'WHERE timespan IN ({timespans}) ' \
                     'OR id IN ({table_ids}) '.format(
                         timespans=','.join(["'{}'".format(x) for _, _, x in FIXTURES]),
-                        table_ids=','.join(["'{}'".format(x) for _, _, _, x in unique_tables])
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
                     )
         else:
             where = ''
         dump('*', tablename, where)
 
-    for tablename, colname, boundary_id, table_id in unique_tables:
-        if 'zcta5' in boundary_id:
+    for tablename, colname, table_id in unique_tables:
+        if 'zcta5' in table_id or 'zillow_zip' in table_id:
             where = '\'11%\''
             compare = 'LIKE'
-        elif 'whosonfirst' in boundary_id:
-            where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
+        elif 'pri_sec_roads' in table_id or 'point_landmark' in table_id:
+            dump('*', tablename, 'WHERE geom && ST_MakeEnvelope(-74,40.69,-73.9,40.72, 4326)')
+            continue
+        elif 'whosonfirst' in table_id:
+            where = "('85632785','85633051','85633111','85633147','85633253','85633267')"
+            compare = 'IN'
+        elif 'county' in table_id and 'tiger' in table_id:
+            where = "('48061', '36047')"
             compare = 'IN'
         else:
             where = '\'36047%\''

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.1.5'
-requires = 'postgis, postgres_fdw'
+default_version = '1.4.0'
+requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/15_fdw_utilities.sql

@@ -1,67 +0,0 @@
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectRemoteTable(fdw_name text, schema_name text, user_dbname text, user_hostname text, username text, user_tablename text, user_schema text)
-RETURNS void
-AS $$
-DECLARE
-  row record;
-  option record;
-  connection_str json;
-BEGIN
-  -- Build connection string
-  connection_str := '{"server":{"extensions":"postgis", "dbname":"'
-    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"6432"}, "users":{"public"'
-    || ':{"user":"' || username ||'", "password":""} } }';
-
-  -- This function tries to be as idempotent as possible, by not creating anything more than once
-  -- (not even using IF NOT EXIST to avoid throwing warnings)
-  IF NOT EXISTS ( SELECT * FROM pg_extension WHERE extname = 'postgres_fdw') THEN
-    CREATE EXTENSION postgres_fdw;
-  END IF;
-  -- Create FDW first if it does not exist
-  IF NOT EXISTS ( SELECT * FROM pg_foreign_server WHERE srvname = fdw_name)
-    THEN
-    EXECUTE FORMAT('CREATE SERVER %I FOREIGN DATA WRAPPER postgres_fdw', fdw_name);
-  END IF;
-
-  -- Set FDW settings
-  FOR row IN SELECT p.key, p.value from lateral json_each_text(connection_str->'server') p
-    LOOP
-      IF NOT EXISTS (WITH a AS (select split_part(unnest(srvoptions), '=', 1) as options from pg_foreign_server where srvname=fdw_name) SELECT * from a where options = row.key)
-        THEN
-        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (ADD %I %L)', fdw_name, row.key, row.value);
-      ELSE
-        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (SET %I %L)', fdw_name, row.key, row.value);
-      END IF;
-    END LOOP;
-
-    -- Create user mappings
-    FOR row IN SELECT p.key, p.value from lateral json_each(connection_str->'users') p LOOP
-      -- Check if entry on pg_user_mappings exists
-      IF NOT EXISTS ( SELECT * FROM pg_user_mappings WHERE srvname = fdw_name AND usename = row.key ) THEN
-        EXECUTE FORMAT ('CREATE USER MAPPING FOR %I SERVER %I', row.key, fdw_name);
-      END IF;
-
-      -- Update user mapping settings
-      FOR option IN SELECT o.key, o.value from lateral json_each_text(row.value) o LOOP
-        IF NOT EXISTS (WITH a AS (select split_part(unnest(umoptions), '=', 1) as options from pg_user_mappings WHERE srvname = fdw_name AND usename = row.key) SELECT * from a where options = option.key) THEN
-          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (ADD %I %L)', row.key, fdw_name, option.key, option.value);
-        ELSE
-          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (SET %I %L)', row.key, fdw_name, option.key, option.value);
-        END IF;
-      END LOOP;
-    END LOOP;
-
-    -- Create schema if it does not exist.
-    IF NOT EXISTS ( SELECT * from pg_namespace WHERE nspname=fdw_name) THEN
-      EXECUTE FORMAT ('CREATE SCHEMA %I', fdw_name);
-    END IF;
-
-    -- Bring the remote cdb_tablemetadata
-    IF NOT EXISTS ( SELECT * FROM PG_CLASS WHERE relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=fdw_name) and relname='cdb_tablemetadata') THEN
-      EXECUTE FORMAT ('CREATE FOREIGN TABLE %I.cdb_tablemetadata (tabname text, updated_at timestamp with time zone) SERVER %I OPTIONS (table_name ''cdb_tablemetadata_text'', schema_name ''public'', updatable ''false'')', fdw_name, fdw_name);
-    END IF;
-
-    -- Import target table
-    EXECUTE FORMAT ('IMPORT FOREIGN SCHEMA %I LIMIT TO (%I) from SERVER %I INTO %I', user_schema, user_tablename, fdw_name, schema_name);
-
-END;
-$$ LANGUAGE PLPGSQL;

src/pg/sql/40_observatory_utility.sql

@@ -205,6 +205,18 @@ END;
 $$ LANGUAGE plpgsql;
 
 
+-- Function we can call to raise an exception in the midst of a SQL statement
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_RaiseNotice(
+    message TEXT
+) RETURNS TEXT
+AS $$
+BEGIN
+  RAISE NOTICE '%', message;
+  RETURN NULL;
+END;
+$$ LANGUAGE plpgsql;
+
+
 -- Create a function that always returns the first non-NULL item
 CREATE OR REPLACE FUNCTION cdb_observatory.first_agg ( anyelement, anyelement )
 RETURNS anyelement LANGUAGE SQL IMMUTABLE STRICT AS $$
@@ -220,3 +232,39 @@ CREATE AGGREGATE cdb_observatory.FIRST (
         stype    = anyelement
 );
 
+CREATE OR REPLACE FUNCTION cdb_observatory.isnumeric (
+  typename varchar
+)
+RETURNS BOOLEAN LANGUAGE SQL IMMUTABLE STRICT AS $$
+  SELECT LOWER(typename) IN (
+    'smallint',
+    'integer',
+    'bigint',
+    'decimal',
+    'numeric',
+    'real',
+    'double precision'
+  )
+$$;
+
+-- Attempt to perform intersection, if there's an exception then buffer
+-- https://gis.stackexchange.com/questions/50399/how-best-to-fix-a-non-noded-intersection-problem-in-postgis
+CREATE OR REPLACE FUNCTION cdb_observatory.safe_intersection(
+  geom_a Geometry(Geometry, 4326),
+  geom_b Geometry(Geometry, 4326)
+)
+RETURNS Geometry(Geometry, 4326) AS
+$$
+BEGIN
+    RETURN ST_MakeValid(ST_Intersection(geom_a, geom_b));
+    EXCEPTION
+        WHEN OTHERS THEN
+            BEGIN
+                RETURN ST_MakeValid(ST_Intersection(ST_Buffer(geom_a, 0.0000001), ST_Buffer(geom_b, 0.0000001)));
+                EXCEPTION
+                    WHEN OTHERS THEN
+                        RETURN NULL;
+    END;
+END
+$$
+LANGUAGE 'plpgsql' STABLE STRICT;

src/pg/sql/42_observatory_exploration.sql

@@ -298,7 +298,7 @@ BEGIN
     ) SELECT available_geoms.*, score, numtiles, notnull_percent, numgeoms,
              percentfill, estnumgeoms, meanmediansize
       FROM available_geoms, scores
-      WHERE available_geoms.geom_id = scores.geom_id
+      WHERE available_geoms.geom_id = scores.column_id
   $string$, geom_clause)
   USING numer_id, denom_id, timespan, filter_tags, bounds;
   RETURN;
@@ -369,8 +369,10 @@ RETURNS TABLE (
 DECLARE
   aggregate_condition TEXT DEFAULT '';
 BEGIN
-  IF aggregate_type IS NOT NULL THEN
-    aggregate_condition := format(' AND numer_aggregate = %L ', aggregate_type);
+  IF LOWER(aggregate_type) ILIKE 'sum' THEN
+    aggregate_condition := ' AND numer_aggregate IN (''sum'', ''median'', ''average'') ';
+  ELSIF aggregate_type IS NOT NULL THEN
+    aggregate_condition := format(' AND numer_aggregate ILIKE %L ', aggregate_type);
   END IF;
   RETURN QUERY
   EXECUTE format($string$
@@ -412,75 +414,95 @@ BEGIN
 END
 $$ LANGUAGE plpgsql;
 
+
 CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryScores(
   bounds Geometry(Geometry, 4326) DEFAULT NULL,
   filter_geom_ids TEXT[] DEFAULT NULL,
-  desired_num_geoms INTEGER DEFAULT 3000
+  desired_num_geoms INTEGER DEFAULT NULL,
+  desired_area NUMERIC DEFAULT NULL
 ) RETURNS TABLE (
   score NUMERIC,
   numtiles BIGINT,
-  geom_id TEXT,
+  table_id TEXT,
+  column_id TEXT,
   notnull_percent NUMERIC,
   numgeoms NUMERIC,
   percentfill NUMERIC,
   estnumgeoms NUMERIC,
   meanmediansize NUMERIC
 ) AS $$
+DECLARE
+  num_geoms_multiplier Numeric;
 BEGIN
+  IF desired_num_geoms IS NULL THEN
+    desired_num_geoms := 3000;
+  END IF;
   filter_geom_ids := COALESCE(filter_geom_ids, (ARRAY[])::TEXT[]);
+  -- Very complex geometries simply fail.  For a boundary check, we can
+  -- comfortably get away with the simplicity of an envelope
+  IF ST_Npoints(bounds) > 10000 THEN
+    bounds := ST_Envelope(bounds);
+  END IF;
+  IF desired_area IS NULL THEN
+    desired_area := ST_Area(bounds);
+  END IF;
+
+  -- In case of points, desired_area will be 0.  We still want an accurate
+  -- estimate of numgeoms in that case.
+  IF desired_area = 0 THEN
+    num_geoms_multiplier := 1;
+  ELSE
+    num_geoms_multiplier := Coalesce(desired_area / Nullif(ST_Area(bounds), 0), 1);
+  END IF;
+
   RETURN QUERY
-  EXECUTE format($string$
-    SELECT
-      (1 / (abs(numgeoms - $3)
-        --* (1 / Coalesce(NullIf(notnull_percent, 0), 1))
-        --* (1 / Coalesce(NullIf(percentfill, 0), 0.0001))
-      ))::Numeric
-      AS score, *
-    FROM (
-      WITH clipped_geom AS (
-        SELECT column_id, table_id
-          , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True)
-                 ELSE tile END clipped_tile
-          , tile
-        FROM observatory.obs_column_table_tile
-        WHERE ($1 IS NULL OR ST_Intersects($1, tile))
-          AND (column_id = ANY($2) OR cardinality($2) = 0)
-      ), clipped_geom_countagg AS (
-        SELECT column_id, table_id
-          , ST_CountAgg(clipped_tile, 2, True)::Numeric notnull_pixels
-          , ST_CountAgg(clipped_tile, 2, False)::Numeric pixels
-        FROM clipped_geom
-        GROUP BY column_id, table_id
-      ) SELECT
-        count(*)::BIGINT, a.column_id
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN cdb_observatory.FIRST(notnull_pixels) / cdb_observatory.FIRST(pixels)
-                ELSE 1
-          END)::Numeric AS notnull_percent
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN (ST_SummaryStatsAgg(clipped_tile, 2, True)).sum
-                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 2, ST_PointOnSurface($1)), 0) * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(cdb_observatory.FIRST(tile), 0, 0)) * cdb_observatory.FIRST(pixels))
-          END)::Numeric AS numgeoms
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN (ST_SummaryStatsAgg(clipped_tile, 3, True)).mean
-                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 3, ST_PointOnSurface($1)), 0)
-          END)::Numeric AS percentfill
-        , ((ST_Area(ST_Transform($1, 3857)) / 1000000) / NullIf(
-            CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                 THEN (ST_SummaryStatsAgg(clipped_tile, 1, True)).mean
-                 ELSE Coalesce(ST_Value(cdb_observatory.FIRST(tile), 1, ST_PointOnSurface($1)), 0)
-            END, 0))::Numeric AS estnumgeoms
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN (ST_SummaryStatsAgg(clipped_tile, 1, True)).mean
-                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 1, ST_PointOnSurface($1)), 0)
-          END)::Numeric AS meanmediansize
+  EXECUTE $string$
+    WITH clipped_geom AS (
+      SELECT column_id, table_id
+        , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True) -- -20
+               ELSE tile END clipped_tile
+        , tile
+      FROM observatory.obs_column_table_tile_simple
+      WHERE ($1 IS NULL OR ST_Intersects($1, tile))
+        AND (column_id = ANY($2) OR cardinality($2) = 0)
+    ), clipped_geom_countagg AS (
+      SELECT column_id, table_id
+        , BOOL_AND(ST_BandIsNoData(clipped_tile, 1)) nodata
+      FROM clipped_geom
+      GROUP BY column_id, table_id
+    ), clipped_geom_reagg AS (
+      SELECT COUNT(*)::BIGINT cnt, a.column_id, a.table_id,
+             cdb_observatory.FIRST(nodata) first_nodata,
+             cdb_observatory.FIRST(tile) first_tile,
+             (ST_SummaryStatsAgg(clipped_tile, 1, False)).sum::Numeric sum_geoms, -- ND
+             (ST_SummaryStatsAgg(clipped_tile, 2, False)).mean::Numeric / 255 mean_fill --ND
       FROM clipped_geom_countagg a, clipped_geom b
       WHERE a.table_id = b.table_id
         AND a.column_id = b.column_id
       GROUP BY a.column_id, a.table_id
-      ORDER BY a.column_id, a.table_id
-    ) foo
-  $string$) USING bounds, filter_geom_ids, desired_num_geoms;
+    ), final AS (
+      SELECT
+        cnt, table_id, column_id
+        , NULL::Numeric AS notnull_percent
+        , (CASE WHEN first_nodata IS FALSE
+                THEN sum_geoms
+                ELSE COALESCE(ST_Value(first_tile, 1, ST_PointOnSurface($1)), 0)
+                  * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(first_tile, 0, 0)))
+          END)::Numeric * $4
+        AS numgeoms
+        , (CASE WHEN first_nodata IS FALSE
+                THEN mean_fill
+                ELSE COALESCE(ST_Value(first_tile, 2, ST_PointOnSurface($1))::Numeric / 255, 0) -- -2
+          END)::Numeric
+        AS percentfill
+        , null::numeric estnumgeoms
+        , null::numeric meanmediansize
+      FROM clipped_geom_reagg
+    ) SELECT
+      ((100.0 / (1+abs(log(0.0001 + $3) - log(0.0001 + numgeoms::Numeric)))) * percentfill)::Numeric
+      AS score, *
+      FROM final
+  $string$ USING bounds, filter_geom_ids, desired_num_geoms, num_geoms_multiplier;
   RETURN;
 END
-$$ LANGUAGE plpgsql;
+$$ LANGUAGE plpgsql IMMUTABLE;

src/pg/sql/44_observatory_geometries.sql

@@ -40,44 +40,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
   END IF;
 
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                                           time_span) As x(target_tables,
-                                                           timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    -- TODO: modify for only one table returned instead of arbitrarily choosing
-    --       one with LIMIT 1 (could be conflict between clipped vs non-clipped
-    --       boundaries in the metadata tables)
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN NULL::geometry;
-  END IF;
-
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return the first boundary in intersections
-  EXECUTE format(
-    'SELECT the_geom
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', target_table)
-  INTO boundary
-  USING geom;
+  EXECUTE $query$
+    SELECT * FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2, $3) LIMIT 1
+  $query$ INTO boundary
+  USING geom, boundary_id, time_span;
 
   RETURN boundary;
 
@@ -111,67 +78,17 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryId(
 RETURNS text
 AS $$
 DECLARE
-  output_id text;
-  target_table text;
-  geoid_colname text;
+  result TEXT;
 BEGIN
 
-  -- If not point, raise error
-  IF ST_GeometryType(geom) != 'ST_Point'
-  THEN
-    RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
-  END IF;
-
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                                timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
-    RETURN NULL::text;
-  END IF;
-
-  EXECUTE
-    format('SELECT ct.colname
-              FROM observatory.obs_column_to_column c2c,
-                   observatory.obs_column_table ct,
-                   observatory.obs_table t
-             WHERE c2c.reltype = ''geom_ref''
-               AND ct.column_id = c2c.source_id
-               AND ct.table_id = t.id
-               AND t.tablename = %L'
-   , target_table)
-  INTO geoid_colname;
-
-  --RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
-
-  -- return geometry id column value
-  EXECUTE format(
-    'SELECT %I::text
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', geoid_colname, target_table)
-  INTO output_id
-  USING geom;
-
-  RETURN output_id;
+  EXECUTE $query$
+    SELECT geom_refs FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+      $1, $2, $3) LIMIT 1
+  $query$
+  INTO result
+  USING geom, boundary_id, time_span;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -203,35 +120,21 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryById(
 RETURNS geometry(geometry, 4326)
 AS $$
 DECLARE
-  boundary geometry(geometry, 4326);
-  target_table text;
-  geoid_colname text;
-  geom_colname text;
+  result GEOMETRY;
 BEGIN
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  --RAISE NOTICE '%', target_table;
-
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No geometries found';
-    RETURN NULL::geometry;
-  END IF;
-
-  -- retrieve boundary
-  EXECUTE
-    format(
-    'SELECT %I
-     FROM observatory.%I
-     WHERE %I = $1
-     LIMIT 1', geom_colname, target_table, geoid_colname)
-  INTO boundary
-  USING geometry_id;
-
-  RETURN boundary;
+  EXECUTE $query$
+    SELECT (data->0->>'value')::Geometry
+    FROM cdb_observatory.OBS_GetData(
+      ARRAY[$1],
+      cdb_observatory.OBS_GetMeta(
+        ST_MakeEnvelope(-180, -90, 180, 90, 4326),
+        ('[{"geom_id": "' || $2 || '"}]')::JSON))
+  $query$
+  INTO result
+  USING geometry_id, boundary_id;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -245,13 +148,12 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetBoundariesByGeometry(
   boundary_id text,
   time_span text DEFAULT NULL,
   overlap_type text DEFAULT NULL)
-RETURNS TABLE(the_geom geometry, geom_refs text)
-AS $$
+RETURNS TABLE (
+  the_geom geometry,
+  geom_refs text
+) AS $$
 DECLARE
-  boundary geometry(Geometry, 4326);
-  geom_colname text;
-  geoid_colname text;
-  target_table text;
+  meta JSON;
 BEGIN
   overlap_type := COALESCE(overlap_type, 'intersects');
   -- check inputs
@@ -259,34 +161,27 @@ BEGIN
   THEN
     -- recognized overlap type (map to ST_Contains, ST_Intersects, and ST_Within)
     RAISE EXCEPTION 'Overlap type ''%'' is not an accepted type (choose intersects, within, or contains)', overlap_type;
-  ELSIF ST_GeometryType(geom) NOT IN ('ST_Polygon', 'ST_MultiPolygon')
-  THEN
-      RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  -- TODO: add timespan in search
-  -- TODO: add overlap info in search
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
+  EXECUTE $query$
+    SELECT cdb_observatory.OBS_GetMeta($1, JSON_Build_Array(JSON_Build_Object(
+                'geom_id', $2, 'geom_timespan', $3)))
+  $query$
+  INTO meta
+  USING geom, boundary_id, time_span;
 
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
+  IF meta->0->>'geom_id' IS NULL THEN
+    RETURN QUERY EXECUTE 'SELECT NULL::Geometry, NULL::Text LIMIT 0';
     RETURN;
   END IF;
 
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
-  EXECUTE format(
-    'SELECT %I, %I::text
-     FROM observatory.%I
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
+  RETURN QUERY EXECUTE $query$
+    SELECT (data->0->>'value')::Geometry the_geom, data->0->>'geomref' geom_refs
+    FROM cdb_observatory.OBS_GetData(
+      ARRAY[($1, 1)::geomval], $2, False
+    )
+  $query$ USING geom, meta;
   RETURN;
 
 END;
@@ -414,27 +309,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
-    RETURN;
-  END IF;
-
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
-  EXECUTE format(
-    'SELECT ST_PointOnSurface(%I) As %s, %I::text
-     FROM observatory.%I
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
+  RETURN QUERY EXECUTE $query$
+    SELECT ST_PointOnSurface(the_geom), geom_refs
+    FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2)
+  $query$ USING geom, boundary_id;
   RETURN;
 
 END;
@@ -534,44 +413,3 @@ BEGIN
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
-
-
--- _OBS_GetGeometryMetadata()
--- TODO: add timespan in search
--- TODO: add choice of clipped versus not clipped
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryMetadata(boundary_id text)
-RETURNS table(geoid_colname text, target_table text, geom_colname text)
-AS $$
-BEGIN
-
-  RETURN QUERY
-  EXECUTE
-  format($string$
-    SELECT geoid_ct.colname::text As geoid_colname,
-           tablename::text,
-           geom_ct.colname::text As geom_colname
-    FROM observatory.obs_column_table As geoid_ct,
-         observatory.obs_table As geom_t,
-         observatory.obs_column_table As geom_ct,
-         observatory.obs_column As geom_c
-    WHERE geoid_ct.column_id
-       IN (
-         SELECT source_id
-         FROM observatory.obs_column_to_column
-         WHERE reltype = 'geom_ref'
-           AND target_id = '%s'
-         )
-      AND geoid_ct.table_id = geom_t.id AND
-          geom_t.id = geom_ct.table_id AND
-          geom_ct.column_id = geom_c.id AND
-          geom_c.type ILIKE 'geometry' AND
-          geom_c.id = '%s'
-    $string$, boundary_id, boundary_id);
-  RETURN;
-    --  AND geom_t.timespan = '%s' <-- put in requested year
-    -- TODO: filter by clipped vs. not so appropriate tablename are unique
-    --       so the limit 1 can be removed
-    RETURN;
-
-END;
-$$ LANGUAGE plpgsql;

src/pg/sql/50_table_level_framework.sql

@@ -1,82 +0,0 @@
-CREATE TYPE cdb_observatory.ds_fdw_metadata as (schemaname text, tabname text, servername text);
-CREATE TYPE cdb_observatory.ds_return_metadata as (colnames text[], coltypes text[]);
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectUserTable(username text, orgname text, user_db_role text, input_schema text, dbname text, host_addr text, table_name text)
-RETURNS cdb_observatory.ds_fdw_metadata
-AS $$
-DECLARE
-  fdw_server text;
-  fdw_import_schema text;
-  connection_str json;
-  import_foreign_schema_q text;
-  epoch_timestamp text;
-BEGIN
-
-  SELECT extract(epoch from now() at time zone 'utc')::int INTO epoch_timestamp;
-  fdw_server := 'fdw_server_' || username || '_' || epoch_timestamp;
-  fdw_import_schema:= fdw_server;
-
-  -- Import foreign table
-  EXECUTE FORMAT ('SELECT cdb_observatory._OBS_ConnectRemoteTable(%L, %L, %L, %L, %L, %L, %L)', fdw_server, fdw_import_schema, dbname, host_addr, user_db_role, table_name, input_schema);
-
-  RETURN (fdw_import_schema::text, table_name::text, fdw_server::text);
-
-EXCEPTION
-  WHEN others THEN
-    -- Disconnect user imported table. Delete schema and FDW server.
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || fdw_import_schema || '".' || table_name;
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || fdw_import_schema || '".cdb_tablemetadata';
-    EXECUTE 'DROP SCHEMA IF EXISTS "' || fdw_import_schema || '"';
-    EXECUTE 'DROP USER MAPPING IF EXISTS FOR public SERVER "' || fdw_server || '"';
-    EXECUTE 'DROP SERVER IF EXISTS "' || fdw_server || '"';
-
-    RETURN (null, null, null);
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetReturnMetadata(username text, orgname text, function_name text, params json)
-RETURNS cdb_observatory.ds_return_metadata
-AS $$
-DECLARE
-  colnames text[];
-  coltypes text[];
-BEGIN
-  EXECUTE FORMAT('SELECT r.colnames::text[], r.coltypes::text[] FROM cdb_observatory._%sResultMetadata(%L::json) r', function_name, params::text)
-  INTO colnames, coltypes;
-
-  RETURN (colnames::text[], coltypes::text[]);
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_FetchJoinFdwTableData(username text, orgname text, table_schema text, table_name text, function_name text, params json)
-RETURNS SETOF record
-AS $$
-DECLARE
-  data_query text;
-  rec RECORD;
-BEGIN
-
-    EXECUTE FORMAT('SELECT cdb_observatory._%sQuery(%L, %L, %L::json)', function_name, table_schema, table_name, params::text)
-    INTO data_query;
-
-    FOR rec IN EXECUTE data_query
-    LOOP
-        RETURN NEXT rec;
-    END LOOP;
-  RETURN;
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;
-
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_DisconnectUserTable(username text, orgname text, table_schema text, table_name text, servername text)
-RETURNS boolean
-AS $$
-BEGIN
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".' || table_name;
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".cdb_tablemetadata';
-    EXECUTE 'DROP SCHEMA IF EXISTS "' || table_schema || '"';
-    EXECUTE 'DROP USER MAPPING IF EXISTS FOR public SERVER "' || servername || '"';
-    EXECUTE 'DROP SERVER IF EXISTS "' || servername || '"';
-    RETURN true;
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;

src/pg/sql/51_table_level_functions.sql

@@ -1,79 +0,0 @@
---
---
--- OBS_GetMeasure
---
---
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetMeasureResultMetadata(params json)
-RETURNS cdb_observatory.ds_return_metadata
-AS $$
-DECLARE
-  colnames text[]; -- Array to store the name of the measures to be returned
-  coltypes text[]; -- Array to store the type of the measures to be returned
-  requested_measures text[];
-  measure_id text;
-BEGIN
-  -- By definition, all the measure results for the OBS_GetMeasure API are numeric values
-  SELECT ARRAY(SELECT json_array_elements_text(params->'measure_id'))::text[] INTO requested_measures;
-
-  FOREACH measure_id IN ARRAY requested_measures
-  LOOP
-    SELECT array_append(colnames, measure_id) INTO colnames;
-    SELECT array_append(coltypes, 'numeric'::text) INTO coltypes;
-  END LOOP;
-
-  RETURN (colnames::text[], coltypes::text[]);
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetMeasureQuery(table_schema text, table_name text, params json)
-RETURNS text
-AS $$
-DECLARE
-  data_query text;
-  measure_ids_arr text[];
-  measure_id text;
-  measures_list text;
-  measures_query text;
-  normalize text;
-  boundary_id text;
-  time_span text;
-  geom_table_name text;
-  data_table_name text;
-BEGIN
-    measures_query := '';
-    -- SELECT table_name from obs_meta WHERE boundary_id = {bound} AND [...] INTO geom_table_name
-    geom_table_name := 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308';
-    -- SELECT table_name from obs_meta WHERE time_span = {time} AND [...] INTO data_table_name
-    data_table_name := 'observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd';
-
-    -- Get measure_ids array from JSON
-    SELECT ARRAY(SELECT json_array_elements_text(params->'measure_id'))::text[] INTO measure_ids_arr;
-
-    -- Get a comma-separated list of measures ("total_pop, over_16_pop") to be used in SELECTs
-    SELECT array_to_string(measure_ids_arr, ',') INTO measures_list;
-
-    FOREACH measure_id IN ARRAY measure_ids_arr
-    LOOP
-      -- Build query to compute each value and normalize
-      -- Assumes the default normalization method, the normalize parameter given in the JSON
-      -- should be checked in order to build the final query
-      SELECT measures_query || ' sum(' || measure_id || '/fraction)::numeric as ' || measure_id || ', ' INTO measures_query;
-    END LOOP;
-
-    -- Data query should select the measures and the cartodb_id of the user table, in that order.
-    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
-        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
-        || geom_table_name || ' as a, '
-        || table_schema || '.' || table_name || ' AS b '
-        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
-        || measures_list
-        || ' FROM ' || data_table_name || ' ) '
-        || 'SELECT '
-        || measures_query
-        || ' cartodb_id::int FROM _areas, values '
-        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';
-    RETURN data_query;
-END;
-$$ LANGUAGE plpgsql;
-

src/pg/test/expected/01_install_test.out

@@ -1,5 +1,4 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION postgres_fdw;
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/expected/40_observatory_utility_test.out

@@ -21,3 +21,7 @@ t
 obs_dumpversion_notnull
 t
 (1 row)
+ERROR:  Error performing intersection: TopologyException: found non-noded intersection between LINESTRING (-97.1968 25.9574, -97.1971 25.9576) and LINESTRING (-97.197 25.9575, -97.1972 25.9576) at -97.19699802694231 25.957551976080605
+complex_safe_intersection_works
+t
+(1 row)

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -3,36 +3,12 @@
 obs_getdemographicsnapshot_test_no_returns
 t
 (1 row)
-obs_get_median_income_at_test_point
-t
-(1 row)
-obs_get_median_income_at_null_island
-t
-(1 row)
-obs_getpoints_for_test_point_value|obs_getpoints_for_test_point_name|obs_getpoints_for_test_point_tablename|obs_getpoints_for_test_point_aggregate|obs_getpoints_for_test_point_type|obs_getpoints_for_test_point_description
-t|t|t|t|t|t
-(1 row)
-obs_getpoints_for_null_island
-t
-(1 row)
-obs_getpolygons_for_test_point
-t
-(1 row)
-obs_getpolygons_for_null_island
-t
-(1 row)
 test_point_segmentation
 t
 (1 row)
 null_island_segmentation
 t
 (1 row)
-getcategories_at_test_point_1
-t
-(1 row)
-getcategories_at_null_island
-t
-(1 row)
 obs_getmeasure_zhvi_point_test
 t
 (1 row)
@@ -72,6 +48,15 @@ t
 obs_getmeasure_out_of_bounds_geometry
 t
 (1 row)
+obs_getmeasure_estimate_for_blank_aggregate
+t
+(1 row)
+obs_getmeasure_per_capita_income_average
+t
+(1 row)
+obs_getmeasure_median_capita_income_average
+t
+(1 row)
 obs_getcategory_point
 t
 (1 row)
@@ -129,3 +114,178 @@ t
 obs_getmeasurebyid_null_id
 t
 (1 row)
+obs_getmeta_null_null_is_null
+t
+(1 row)
+obs_getmeta_null_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_null_is_null
+t
+(1 row)
+obs_getmeta_nullisland_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_us_measure_is_null
+t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization|id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+obs_getmeta_conflicting_metadata
+t
+(1 row)
+obs_getdata_geomval_empty_null
+t
+(1 row)
+obs_getdata_text_empty_null
+t
+(1 row)
+obs_getdata_geomval_empty_one_measure
+t
+(1 row)
+id|data_point_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_prenormalized|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_prenormalized|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_one_null|data_polygon_measure_two_null
+t|t|t
+(1 row)
+id|data_polygon_measure_one_predenom|data_polygon_measure_two_predenom
+t|t|t
+(1 row)
+id|data_polygon_measure_one_area|data_polygon_measure_two_area
+t|t|t
+(1 row)
+id|data_polygon_measure_tract|data_polygon_measure_bg
+t|t|t
+(1 row)
+id|data_point_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|valcol
+t|t|t
+(1 row)
+id|correct_num_geoms
+t|t
+(1 row)
+id|correct_num_geoms|correct_pop
+t|t|t
+(1 row)
+id|correct_num_geoms|correct_pop|correct_bg_names
+t|t|t|t
+(1 row)
+id|correct_num_points
+t|t
+(1 row)
+id|correct_num_points|pointgeom_names
+t|t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_predenom
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null|obs_getdata_by_id_two_measure_null
+t|t|t
+(1 row)
+id|obs_getdata_by_id_categorical
+t|t
+(1 row)
+id|obs_getdata_by_id_geometry
+t|t
+(1 row)
+obs_getdata_api_geomvals_no_args
+t
+(1 row)
+ary_type|obs_getdata_api_geomvals_args_numer_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomvals_args_string_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomrefs_args_numer_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomrefs_args_string_return
+t|t
+(1 row)
+setseed
+
+(1 row)
+bg_sample|bg_max_error|bg_avg_error|bg_min_error
+1|t|t|t
+2|t|t|t
+3|t|t|t
+5|t|t|t
+10|t|t|t
+25|t|t|t
+50|t|t|t
+100|t|t|t
+2085|t|t|t
+(9 rows)
+tract_sample|tract_max_error|tract_avg_error|tract_min_error
+1|t|t|t
+2|t|t|t
+3|t|t|t
+5|t|t|t
+10|t|t|t
+25|t|t|t
+50|t|t|t
+100|t|t|t
+761|t|t|t
+(9 rows)
+no_bg_point_error
+t
+(1 row)

src/pg/test/expected/42_observatory_exploration_test.out

@@ -114,7 +114,7 @@ t
 _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg
 t
 (1 row)
-_obs_getavailablegeometries_bg_2014
+_obs_getavailablegeometries_bg_2015
 t
 (1 row)
 _obs_getavailablegeometries_bg_not_1996
@@ -159,21 +159,36 @@ t
 _obs_geometryscores_2500km_buffer
 t
 (1 row)
-_obs_geometryscores_numgeoms_500m_buffer
-t
-(1 row)
-_obs_geometryscores_numgeoms_5km_buffer
-t
-(1 row)
-_obs_geometryscores_numgeoms_50km_buffer
-t
-(1 row)
-_obs_geometryscores_numgeoms_500km_buffer
-t
-(1 row)
-_obs_geometryscores_numgeoms_2500km_buffer
-t
-(1 row)
+column_id|_obs_geometryscores_numgeoms_500m_buffer
+us.census.tiger.block_group|2
+us.census.tiger.census_tract|1
+us.census.tiger.zcta5|0
+us.census.tiger.county|0
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_5km_buffer
+us.census.tiger.block_group|244
+us.census.tiger.census_tract|78
+us.census.tiger.zcta5|9
+us.census.tiger.county|0
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_50km_buffer
+us.census.tiger.block_group|10817
+us.census.tiger.census_tract|3396
+us.census.tiger.zcta5|484
+us.census.tiger.county|11
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_500km_buffer
+us.census.tiger.block_group|48567
+us.census.tiger.census_tract|15823
+us.census.tiger.zcta5|6466
+us.census.tiger.county|295
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_2500km_buffer
+us.census.tiger.block_group|165852
+us.census.tiger.census_tract|55283
+us.census.tiger.zcta5|27046
+us.census.tiger.county|2551
+(4 rows)
 _obs_geometryscores_500km_buffer_50_geoms
 t
 (1 row)
@@ -186,16 +201,28 @@ t
 _obs_geometryscores_500km_buffer_25000_geoms
 t
 (1 row)
+testarea_uses_tract
+t
+(1 row)
+points_use_bg
+t
+(1 row)
 _total_pop_in_legacy_builder_metadata
 t
 (1 row)
 _median_income_in_legacy_builder_metadata
 t
 (1 row)
+_gini_in_legacy_builder_metadata
+t
+(1 row)
 _total_pop_in_legacy_builder_metadata_sums
 t
 (1 row)
-_median_income_not_in_legacy_builder_metadata_sums
+_median_income_in_legacy_builder_metadata_sums
+t
+(1 row)
+_gini_not_in_legacy_builder_metadata_sums
 t
 (1 row)
 _no_dupe_subsections_in_legacy_builder_metadata

src/pg/test/expected/44_observatory_geometries_test.out

@@ -42,6 +42,12 @@ t
 obs_getboundarybyid_boundary_id_mismatch_geom_id
 t
 (1 row)
+_obs_getboundariesbygeometry_roads_around_cartodb
+t
+(1 row)
+_obs_getboundariesbygeometry_points_around_cartodb
+t
+(1 row)
 _obs_getboundariesbygeometry_tracts_around_cartodb
 t
 (1 row)
@@ -87,9 +93,3 @@ t
 obs_getpointsbypointandradius_around_null_island
 t
 (1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)

src/pg/test/fixtures/drop_fixtures.sql

@@ -13,17 +13,22 @@ DROP TABLE IF EXISTS observatory.obs_meta_denom;
 DROP TABLE IF EXISTS observatory.obs_meta_geom;
 DROP TABLE IF EXISTS observatory.obs_meta_timespan;
 DROP TABLE IF EXISTS observatory.obs_column_table_tile;
-DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
-DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
-DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
-DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
-DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
-DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile_simple;
+DROP TABLE IF EXISTS observatory.obs_78fb6c1d6ff6505225175922c2c389ce48d7632c;
 DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
-DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
-DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
+DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
+DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
+DROP TABLE IF EXISTS observatory.obs_c4411eba732408d47d73281772dbf03d60645dec;
 DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
+DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
+DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
+DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
+DROP TABLE IF EXISTS observatory.obs_0310c639744a2014bb1af82709228f05b59e7d3d;
+DROP TABLE IF EXISTS observatory.obs_87a814e485deabe3b12545a537f693d16ca702c2;
+DROP TABLE IF EXISTS observatory.obs_e32f8e59c7c8861ee5ee4029b3ace2af9a5c9caf;
+DROP TABLE IF EXISTS observatory.obs_23cb5063486bd7cf36f17e89e5e65cd31b331f6e;
 DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
 DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
-DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
-DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
+DROP TABLE IF EXISTS observatory.obs_8e30e6b3792430b410ba5b9e49cdc6a0d404d48f;
+DROP TABLE IF EXISTS observatory.obs_08025e1287e3af2b5de571d06562ba8d3bdb48e9;
+DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;

src/pg/test/sql/01_install_test.sql

@@ -1,6 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION postgres_fdw;
 
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/sql/40_observatory_utility_test.sql

@@ -11,8 +11,8 @@ SELECT
   cdb_observatory._OBS_GeomTable(
     ST_SetSRID(ST_Point(-74.0059, 40.7128), 4326),
     'us.census.tiger.census_tract',
-    '2014'
-  ) = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As _obs_geomtable_with_returned_table;
+    '2015'
+  ) = 'obs_87a814e485deabe3b12545a537f693d16ca702c2' As _obs_geomtable_with_returned_table;
 
 -- get null for unknown geometry_id
 -- should give back null
@@ -47,3 +47,15 @@ SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') =
 
 SELECT cdb_observatory.OBS_DumpVersion()
   IS NOT NULL AS OBS_DumpVersion_notnull;
+
+-- Should fail to perform intersection
+SELECT ST_IsValid(ST_Intersection(
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county_clipped')
+)) AS complex_intersection_fails;
+
+-- Should succeed in intersecting
+SELECT ST_IsValid(cdb_observatory.safe_intersection(
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county_clipped')
+)) AS complex_safe_intersection_works;

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -5,128 +5,21 @@ SET client_min_messages TO WARNING;
 --
 WITH result as(
   Select count(coalesce(OBS_GetDemographicSnapshot->>'value', 'foo')) expected_columns
-  FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint())
+  FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint(), '2010 - 2014')
 ) select expected_columns = 52 as OBS_GetDemographicSnapshot_test_no_returns
 FROM result;
 
-WITH result as (
-  SELECT _OBS_Get::text as expected FROM
-  cdb_observatory._OBS_Get(
-    cdb_observatory._TestPoint(),
-    Array['us.census.acs.B19013001']::text[],
-    '2010 - 2014',
-    'us.census.tiger.block_group'
-  )
-) SELECT expected = '{"value":79292,"name":"Median Household Income in the past 12 Months","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"median","type":"Numeric","description":"Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc.  It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans'' (VA) payments, unemployment and/or worker''s compensation, child support, and alimony."}'
-  As OBS_Get_median_income_at_test_point
-  FROM result;
-
--- median income at null island
-WITH result as (
-  SELECT count(_OBS_Get) as expected FROM
-  cdb_observatory._OBS_Get(
-    ST_SetSRID(ST_Point(0, 0), 4326),
-    Array['us.census.acs.B19013001']::text[],
-    '2010 - 2014',
-    'us.census.tiger.block_group'
-  )
-) select expected = 0 as OBS_Get_median_income_at_null_island
-  from result;
-
--- OBS_GetPoints
--- obs_getpoints
--- --------------------
--- {4809.33511352425}
-
--- SELECT
---   (cdb_observatory._OBS_GetPoints(
---     cdb_observatory._TestPoint(),
---     'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- block groups (see _obs_geomtable)
---     (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
---   ))[1]::text = '{"value":10923.093200390833950,"name":"Total Population","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'
---   as OBS_GetPoints_for_test_point;
-WITH cte As (
-SELECT
-  (cdb_observatory._OBS_GetPoints(
-    cdb_observatory._TestPoint(),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- block groups (see _obs_geomtable)
-    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  ))[1]
-  as OBS_GetPoints_for_test_point)
-SELECT
-  (abs((OBS_GetPoints_for_test_point ->> 'value')::numeric - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPoints_for_test_point_value,
-  (OBS_GetPoints_for_test_point ->> 'name') = 'Total Population' As OBS_GetPoints_for_test_point_name,
-  (OBS_GetPoints_for_test_point ->> 'tablename') = 'obs_1a098da56badf5f32e336002b0a81708c40d29cd' As OBS_GetPoints_for_test_point_tablename,
-  (OBS_GetPoints_for_test_point ->> 'aggregate') = 'sum' As OBS_GetPoints_for_test_point_aggregate,
-  (OBS_GetPoints_for_test_point ->> 'type') = 'Numeric' As OBS_GetPoints_for_test_point_type,
-  (OBS_GetPoints_for_test_point ->> 'description') = 'The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.' As OBS_GetPoints_for_test_point_description
-FROM cte;
-
--- what happens at null island
-
-SELECT
-  (cdb_observatory._OBS_GetPoints(
-    ST_SetSRID(ST_Point(0, 0), 4326),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  ))[1]::text  is null
-  as OBS_GetPoints_for_null_island;
-
--- OBS_GetPolygons
---   obs_getpolygons
--- --------------------
---  {12996.8172420752}
-
-SELECT
-  (cdb_observatory._OBS_GetPolygons(
-    cdb_observatory._TestArea(),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json]
-))[1]::text = '{"value":12327.3133495107,"name":"Total Population","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'
-  as OBS_GetPolygons_for_test_point;
-
--- see what happens around null island
-SELECT
-  ((cdb_observatory._OBS_GetPolygons(
-    ST_Buffer(ST_SetSRID(ST_Point(0, 0), 4326)::geography, 500)::geometry,
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  )[1]->>'value') is null
-  as OBS_GetPolygons_for_null_island;
-
 SELECT cdb_observatory.OBS_GetSegmentSnapshot(
     cdb_observatory._TestPoint(),
     'us.census.tiger.census_tract'
-)::text =
- '{"x10_segment":"Wealthy, urban without Kids","x55_segment":"Wealthy transplants displacing long-term local residents","us.census.acs.B01003001_quantile":"0.3235","us.census.acs.B01001002_quantile":"0.494716216216216","us.census.acs.B01001026_quantile":"0.183756756756757","us.census.acs.B01002001_quantile":"0.0752837837837838","us.census.acs.B03002003_quantile":"0.293162162162162","us.census.acs.B03002004_quantile":"0.455527027027027","us.census.acs.B03002006_quantile":"0.656405405405405","us.census.acs.B03002012_quantile":"0.840081081081081","us.census.acs.B05001006_quantile":"0.727135135135135","us.census.acs.B08006001_quantile":"0.688635135135135","us.census.acs.B08006002_quantile":"0.0204459459459459","us.census.acs.B08006008_quantile":"0.679324324324324","us.census.acs.B08006009_quantile":"0.996716216216216","us.census.acs.B08006011_quantile":"0.967418918918919","us.census.acs.B08006015_quantile":"0.512945945945946","us.census.acs.B08006017_quantile":"0.0504864864864865","us.census.acs.B09001001_quantile":"0.192405405405405","us.census.acs.B11001001_quantile":"0.331702702702703","us.census.acs.B14001001_quantile":"0.296283783783784","us.census.acs.B14001002_quantile":"0.045472972972973","us.census.acs.B14001005_quantile":"0.0442702702702703","us.census.acs.B14001006_quantile":"0.0829054054054054","us.census.acs.B14001007_quantile":"0.701135135135135","us.census.acs.B14001008_quantile":"0.404527027027027","us.census.acs.B15003001_quantile":"0.191824324324324","us.census.acs.B15003017_quantile":"0.864162162162162","us.census.acs.B15003022_quantile":"0.754297297297297","us.census.acs.B15003023_quantile":"0.350054054054054","us.census.acs.B16001001_quantile":"0.217635135135135","us.census.acs.B16001002_quantile":"0.85972972972973","us.census.acs.B16001003_quantile":"0.342851351351351","us.census.acs.B17001001_quantile":"0.51204054054054","us.census.acs.B17001002_quantile":"0.813540540540541","us.census.acs.B19013001_quantile":"0.0948648648648649","us.census.acs.B19083001_quantile":"0.678351351351351","us.census.acs.B19301001_quantile":"0.146108108108108","us.census.acs.B25001001_quantile":"0.149067567567568","us.census.acs.B25002003_quantile":"0","us.census.acs.B25004002_quantile":"0","us.census.acs.B25004004_quantile":"0.944554054054054","us.census.acs.B25058001_quantile":"0.398040540540541","us.census.acs.B25071001_quantile":"0.0596081081081081","us.census.acs.B25075001_quantile":"0","us.census.acs.B25075025_quantile":null}' as test_point_segmentation;
+)::JSONB =
+ '{"x10_segment": "Wealthy, urban without Kids", "x55_segment": "Wealthy transplants displacing long-term local residents", "us.census.acs.B01001002_quantile": "0.494716216216216", "us.census.acs.B01001026_quantile": "0.183756756756757", "us.census.acs.B01002001_quantile": "0.0752837837837838", "us.census.acs.B01003001_quantile": "0.3235", "us.census.acs.B03002003_quantile": "0.293162162162162", "us.census.acs.B03002004_quantile": "0.455527027027027", "us.census.acs.B03002006_quantile": "0.656405405405405", "us.census.acs.B03002012_quantile": "0.840081081081081", "us.census.acs.B05001006_quantile": "0.727135135135135", "us.census.acs.B08006001_quantile": "0.688635135135135", "us.census.acs.B08006002_quantile": "0.0204459459459459", "us.census.acs.B08006009_quantile": "0.679324324324324", "us.census.acs.B08006011_quantile": "0.996716216216216", "us.census.acs.B08006015_quantile": "0.967418918918919", "us.census.acs.B08006017_quantile": "0.512945945945946", "us.census.acs.B08301010_quantile": "0.994743243243243", "us.census.acs.B09001001_quantile": "0.0504864864864865", "us.census.acs.B11001001_quantile": "0.192405405405405", "us.census.acs.B14001001_quantile": "0.331702702702703", "us.census.acs.B14001002_quantile": "0.296283783783784", "us.census.acs.B14001005_quantile": "0.045472972972973", "us.census.acs.B14001006_quantile": "0.0442702702702703", "us.census.acs.B14001007_quantile": "0.0829054054054054", "us.census.acs.B14001008_quantile": "0.701135135135135", "us.census.acs.B15003001_quantile": "0.404527027027027", "us.census.acs.B15003017_quantile": "0.191824324324324", "us.census.acs.B15003022_quantile": "0.864162162162162", "us.census.acs.B15003023_quantile": "0.754297297297297", "us.census.acs.B16001001_quantile": "0.350054054054054", "us.census.acs.B16001002_quantile": "0.217635135135135", "us.census.acs.B16001003_quantile": "0.85972972972973", "us.census.acs.B17001001_quantile": "0.342851351351351", "us.census.acs.B17001002_quantile": "0.51204054054054", "us.census.acs.B19013001_quantile": "0.813540540540541", "us.census.acs.B19083001_quantile": "0.0948648648648649", "us.census.acs.B19301001_quantile": "0.678351351351351", "us.census.acs.B25001001_quantile": "0.146108108108108", "us.census.acs.B25002003_quantile": "0.149067567567568", "us.census.acs.B25004002_quantile": "0", "us.census.acs.B25004004_quantile": "0", "us.census.acs.B25058001_quantile": "0.944554054054054", "us.census.acs.B25071001_quantile": "0.398040540540541", "us.census.acs.B25075001_quantile": "0.0596081081081081", "us.census.acs.B25075025_quantile": "0"}'::JSONB as test_point_segmentation;
 
 -- segmentation around null island
 SELECT cdb_observatory.OBS_GetSegmentSnapshot(
     ST_SetSRID(ST_Point(0, 0), 4326),
     'us.census.tiger.census_tract'
-)::text = '{"x10_segment":null,"x55_segment":null,"us.census.acs.B01003001_quantile":null,"us.census.acs.B01001002_quantile":null,"us.census.acs.B01001026_quantile":null,"us.census.acs.B01002001_quantile":null,"us.census.acs.B03002003_quantile":null,"us.census.acs.B03002004_quantile":null,"us.census.acs.B03002006_quantile":null,"us.census.acs.B03002012_quantile":null,"us.census.acs.B05001006_quantile":null,"us.census.acs.B08006001_quantile":null,"us.census.acs.B08006002_quantile":null,"us.census.acs.B08006008_quantile":null,"us.census.acs.B08006009_quantile":null,"us.census.acs.B08006011_quantile":null,"us.census.acs.B08006015_quantile":null,"us.census.acs.B08006017_quantile":null,"us.census.acs.B09001001_quantile":null,"us.census.acs.B11001001_quantile":null,"us.census.acs.B14001001_quantile":null,"us.census.acs.B14001002_quantile":null,"us.census.acs.B14001005_quantile":null,"us.census.acs.B14001006_quantile":null,"us.census.acs.B14001007_quantile":null,"us.census.acs.B14001008_quantile":null,"us.census.acs.B15003001_quantile":null,"us.census.acs.B15003017_quantile":null,"us.census.acs.B15003022_quantile":null,"us.census.acs.B15003023_quantile":null,"us.census.acs.B16001001_quantile":null,"us.census.acs.B16001002_quantile":null,"us.census.acs.B16001003_quantile":null,"us.census.acs.B17001001_quantile":null,"us.census.acs.B17001002_quantile":null,"us.census.acs.B19013001_quantile":null,"us.census.acs.B19083001_quantile":null,"us.census.acs.B19301001_quantile":null,"us.census.acs.B25001001_quantile":null,"us.census.acs.B25002003_quantile":null,"us.census.acs.B25004002_quantile":null,"us.census.acs.B25004004_quantile":null,"us.census.acs.B25058001_quantile":null,"us.census.acs.B25071001_quantile":null,"us.census.acs.B25075001_quantile":null,"us.census.acs.B25075025_quantile":null}' as null_island_segmentation;
-
-WITH result as (
-  SELECT array_agg(_OBS_GetCategories) as expected FROM
-    cdb_observatory._OBS_GetCategories(
-      cdb_observatory._TestPoint(),
-      Array['us.census.spielman_singleton_segments.X10'],
-      'us.census.tiger.census_tract'
-  )
-)
-  select (expected)[1]::text = '{"category":"Wealthy, urban without Kids","name":"Spielman-Singleton Segments: 10 Clusters","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 10 clusters"}' as GetCategories_at_test_point_1
-  from result;
-
-WITH result as (
-  SELECT array_agg(_OBS_GetCategories) as expected FROM
-    cdb_observatory._OBS_GetCategories(
-      ST_SetSRID(ST_Point(0,0), 4326),
-      Array['us.census.spielman_singleton_segments.X10'],
-      'us.census.tiger.census_tract'
-  )
-)
-  select expected[0] is NULL as GetCategories_at_null_island
-  from result;
+)::text is null as null_island_segmentation;
 
 -- Point-based OBS_GetMeasure with zillow
 SELECT abs(OBS_GetMeasure_zhvi_point - 597900) / 597900 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
@@ -196,7 +89,7 @@ SELECT (abs(cdb_observatory.OBS_GetMeasure(
 -- Poly-based OBS_GetMeasure with denominator normalization
 SELECT abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._TestArea(),
-  'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
+  'us.census.acs.B01001002', 'denominator', null, '2010 - 2014') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
 
 -- Poly-based OBS_GetMeasure with one very bad geom
 SELECT abs(cdb_observatory.OBS_GetMeasure(
@@ -213,13 +106,28 @@ SELECT cdb_observatory.OBS_GetMeasure(
   ST_SetSRID(st_point(0, 0), 4326),
   'us.census.acs.B01003001') IS NULL As OBS_GetMeasure_out_of_bounds_geometry;
 
+-- OBS_GetMeasure over arbitrary area for a measure we cannot estimate
+SELECT cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.1),
+  'us.census.acs.B19083001') IS NULL As OBS_GetMeasure_estimate_for_blank_aggregate;
+
+-- OBS_GetMeasure over arbitrary area for an average measure we can estimate
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.01),
+  'us.census.acs.B19301001') - 20025) / 20025 < 0.001 As OBS_GetMeasure_per_capita_income_average;
+
+-- OBS_GetMeasure over arbitrary area for a median measure we can estimate
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.01),
+  'us.census.acs.B19013001') - 39266) / 39266 < 0.001 As OBS_GetMeasure_median_capita_income_average;
+
 -- Point-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
   cdb_observatory._TestPoint(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As OBS_GetCategory_point;
 
 -- Poly-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
-  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;
+  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Hispanic and Young' As obs_getcategory_polygon;
 
 -- NULL Input OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
@@ -276,7 +184,7 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
 
 -- Area-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
-  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;
+  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Hispanic and Young' As OBS_GetUSCensusCategory_polygon;
 
 -- Null-input OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
@@ -323,3 +231,713 @@ SELECT cdb_observatory.OBS_GetMeasureById(
   'us.census.tiger.block_group',
   '2010 - 2014'
 ) IS NULL As OBS_GetMeasureById_null_id;
+
+-- OBS_GetMeta null/null
+SELECT cdb_observatory.OBS_GetMeta(NULL, NULL) IS NULL
+AS OBS_GetMeta_null_null_is_null;
+
+-- OBS_GetMeta null/empty array
+SELECT cdb_observatory.OBS_GetMeta(NULL, '[]') IS NULL
+AS OBS_GetMeta_null_empty_is_null;
+
+-- OBS_GetMeta nullisland/null
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0), NULL) IS NULL
+AS OBS_GetMeta_nullisland_null_is_null;
+
+-- OBS_GetMeta nullisland/empty array
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0), '[]') IS NULL
+AS OBS_GetMeta_nullisland_empty_is_null;
+
+-- OBS_GetMeta nullisland/us_measure data
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0),
+  '[{"numer_id": "us.census.acs.B01003001"}]') IS NULL
+AS OBS_GetMeta_nullisland_us_measure_is_null;
+
+-- OBS_GetMeta for point completes one partial measure with "best" metadata
+-- with no denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01003001' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'total_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Total Population' numer_name,
+(meta->0->>'denom_id') IS NULL denom_id,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes one partial measure with "best" metadata
+-- with a denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for polygon completes one partial measure with "best" metadata
+-- with no denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01003001' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'total_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Total Population' numer_name,
+(meta->0->>'denom_id') IS NULL denom_id,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for polygon completes one partial measure with "best" metadata
+-- with a denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01001002"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with "best"
+-- metadata, includes geom alternatives if asked
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002"}]', null, 2) meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') IS NULL normalization,
+(meta->1->>'id')::integer = 1 id,
+(meta->1->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->1->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->1->>'score_rank')::integer = 2 score_rank,
+(meta->1->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->1->>'numer_colname') = 'male_pop' numer_colname,
+(meta->1->>'numer_type') = 'Numeric' numer_type,
+(meta->1->>'numer_name') = 'Male Population' numer_name,
+(meta->1->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->1->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->1->>'denom_colname') = 'total_pop' denom_colname,
+(meta->1->>'denom_type') = 'Numeric' denom_type,
+(meta->1->>'denom_name') = 'Total Population' denom_name,
+(meta->1->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->1->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with "best" metadata
+-- with pre-computed geom
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "geom_id": "us.census.tiger.census_tract"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->0->>'normalization') IS NULL normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with conflicting
+-- metadata
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "denom_id": "us.census.acs.B01001002", "geom_id": "us.census.tiger.census_tract"}]') IS NULL
+AS obs_getmeta_conflicting_metadata;
+
+-- OBS_GetData/OBS_GetMeta by id with empty list/null
+WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::TEXT[], null))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_geomval_empty_null FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with empty list/null
+WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::GEOMVAL[], null))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_text_empty_null FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with empty list
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::GEOMVAL[],
+  (SELECT meta FROM meta)))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_geomval_empty_one_measure FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure NULL
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 10923) / 10923 < 0.001 data_point_measure_null,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure NULL
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_null,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure area
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 10923) / 10923 < 0.001 data_point_measure_area,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure area
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_area,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure predenom
+-- called "prednormalized"
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "prenormalized"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 1900) / 1900 < 0.001 data_point_measure_prenormalized,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure predenom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 1900) / 1900 < 0.001 data_point_measure_predenominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure predenom
+-- called "prenormalized"
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "prenormalized"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_prenormalized,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure predenom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_predenominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with impossible denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' IS NULL data_point_measure_impossible_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one impossible denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' IS NULL data_polygon_measure_impossible_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 0.6215) / 0.6215 < 0.001 data_point_measure_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one denom measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures NULL normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_one_null,
+       abs((data->1->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_two_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures predenom normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom"}, {"numer_id": "us.census.acs.B01001002", "normalization": "predenom"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_one_predenom,
+       abs((data->1->>'value')::Numeric - 6043) / 6043 < 0.001 data_polygon_measure_two_predenom
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures area normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}, {"numer_id": "us.census.acs.B01001002", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_one_area,
+       abs((data->1->>'value')::Numeric - 7739) / 7739 < 0.001 data_polygon_measure_two_area
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures different geoms
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 16960) / 16960 < 0.001 data_polygon_measure_tract,
+       abs((data->1->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_bg
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Wealthy transplants displacing long-term local residents' data_point_categorical,
+       data->1->>'value' IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
+       data->1->>'value' IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with one categorical and one measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}, {"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
+       abs((data->1->>'value')::Numeric - 15787) / 15787 < 0.0001 valcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
+       abs(sum((data->1->>'value')::numeric) - 15787) / 15787 < 0.001 correct_pop
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure + one text
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.tiger.name", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
+       abs(sum((data->1->>'value')::numeric) - 15787) / 15787 < 0.001 correct_pop,
+       array_agg(distinct data->2->>'value') = '{"Block Group 1","Block Group 2","Block Group 3","Block Group 4","Block Group 5"}' correct_bg_names
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with points inside a polygon
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.pointlm_geom"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) AS id,
+       count(distinct (data->0->>'value')::geometry(point, 4326)) = 3 correct_num_points
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with points inside a polygon + one text
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.pointlm_geom"}, {"geom_id": "us.census.tiger.pointlm_geom", "numer_id": "us.census.tiger.fullname"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) AS id,
+       count(distinct (data->0->>'value')::geometry(point, 4326)) = 3 correct_num_points,
+       array_agg(data->1->>'value') = '{"Bushwick Yards","Edward Block Square","Bushwick Houses"}' pointgeom_names
+FROM data;
+
+
+-- OBS_GetData by id with one standard measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 5578) / 5578 < 0.001 obs_getdata_by_id_one_measure_null
+FROM data;
+
+-- OBS_GetData by id with one standard measure, predenominated
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"normalization": "predenominated", "geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 3241) / 3241 < 0.001 obs_getdata_by_id_one_measure_predenom
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with two standard measures
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}, {"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 5578) / 5578 < 0.001 obs_getdata_by_id_one_measure_null,
+       (abs((data->1->>'value')::numeric) - 0.6053) / 0.6053 < 0.001 obs_getdata_by_id_two_measure_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       data->0->>'value' = 'Wealthy transplants displacing long-term local residents' obs_getdata_by_id_categorical
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with one geometry
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       ST_GeometryType((data->0->>'value')::geometry) = 'ST_MultiPolygon' obs_getdata_by_id_geometry
+FROM data;
+
+-- OBS_GetData with an API + geomvals, no args
+SELECT (SELECT array_agg(json_array_elements::text) @> array['"us.census.tiger.census_tract"']
+        FROM json_array_elements(data->0->'value'))
+AS OBS_GetData_API_geomvals_no_args
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+  '[{"numer_type": "text", "numer_colname": "boundary_id", "api_method": "obs_getavailableboundaries"}]');
+
+-- OBS_GetData with an API + geomvals, args, numeric
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'number'
+AS OBS_GetData_API_geomvals_args_numer_return
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+    '[{"numer_type": "numeric", "numer_colname": "obs_getmeasure", "api_method": "obs_getmeasure", "api_args": ["us.census.acs.B01003001"]}]');
+
+-- OBS_GetData with an API + geomvals, args, text
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'string'
+AS OBS_GetData_API_geomvals_args_string_return
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+    '[{"numer_type": "text", "numer_colname": "obs_getcategory", "api_method": "obs_getcategory", "api_args": ["us.census.spielman_singleton_segments.X55"]}]');
+
+-- OBS_GetData with an API + geomrefs, args, numeric
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'number'
+AS OBS_GetData_API_geomrefs_args_numer_return
+FROM cdb_observatory.obs_getdata(array['36047076200'],
+      '[{"numer_type": "numeric", "numer_colname": "obs_getmeasurebyid", "api_method": "obs_getmeasurebyid", "api_args": ["us.census.acs.B01003001", "us.census.tiger.census_tract"]}]');
+
+-- OBS_GetData with an API + geomrefs, args, text
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'string'
+AS OBS_GetData_API_geomrefs_args_string_return
+FROM cdb_observatory.obs_getdata(array['36047'],
+      '[{"numer_type": "text", "numer_colname": "obs_getboundarybyid", "api_method": "obs_getboundarybyid", "api_args": ["us.census.tiger.county"]}]');
+
+-- Ensure consistent results below.
+select setseed(0);
+
+-- Check that random assortment of block groups in Brooklyn return accurate data
+WITH _geoms AS (
+  SELECT
+    (data->0->>'value')::geometry the_geom,
+    data->0->>'geomref' geom_ref,
+    (data->1->>'value')::numeric total_pop
+  FROM cdb_observatory.OBS_GetData(
+    array[(st_buffer(cdb_observatory._testpoint(), 0.2), 1)::geomval],
+    (SELECT cdb_observatory.OBS_GetMeta(ST_MakeEnvelope(-179, 89, 179, -89, 4326),
+      '[{"geom_id": "us.census.tiger.block_group"},
+        {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group", "normalization": "predenom"}]')),
+    FALSE
+  )
+  WHERE data->0->>'geomref' LIKE '36047%'
+  ORDER BY RANDOM()
+), geoms AS (
+  SELECT *, row_number() OVER () cartodb_id FROM _geoms
+), samples AS (
+  SELECT COUNT(*) cnt, unnest(ARRAY[1, 2, 3, 5, 10, 25, 50, 100, COUNT(*)]) sample FROM geoms
+), filtered AS (
+  SELECT * FROM geoms, samples WHERE cartodb_id % (cnt / sample) = 0
+), summary AS (
+  SELECT sample, ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+    COUNT(*)::INT cnt,
+    ARRAY_AGG((the_geom, cartodb_id)::geomval) geomvals,
+    SUM(ST_Area(the_geom))::Numeric sumarea
+  FROM filtered
+  GROUP BY sample
+), meta AS (
+  SELECT sample, cdb_observatory.OBS_GetMeta(extent,
+    ('[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "target_area": ' || sumarea || '}]')::JSON,
+    1, 1, cnt) meta
+  FROM summary
+  GROUP BY sample, extent, cnt, sumarea
+), results AS (
+  SELECT summary.sample, id, meta->0->>'geom_id' geom_id, (data->0->>'value')::Numeric as val
+  FROM summary, meta, LATERAL cdb_observatory.OBS_GetData(geomvals, meta) data
+  WHERE summary.sample = meta.sample
+) SELECT sample bg_sample
+ , MAX(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 bg_max_error
+ , AVG(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 bg_avg_error
+ , MIN(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 bg_min_error
+FROM geoms, results
+WHERE cartodb_id = id
+GROUP BY sample
+ORDER BY sample
+;
+
+-- Check that random assortment of tracts in Brooklyn return accurate data
+WITH _geoms AS (
+  SELECT
+    (data->0->>'value')::geometry the_geom,
+    data->0->>'geomref' geom_ref,
+    (data->1->>'value')::numeric total_pop
+  FROM cdb_observatory.OBS_GetData(
+    array[(st_buffer(cdb_observatory._testpoint(), 0.2), 1)::geomval],
+    (SELECT cdb_observatory.OBS_GetMeta(ST_MakeEnvelope(-179, 89, 179, -89, 4326),
+      '[{"geom_id": "us.census.tiger.census_tract"},
+        {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract", "normalization": "predenom"}]')),
+    FALSE
+  )
+  WHERE data->0->>'geomref' LIKE '36047%'
+  ORDER BY RANDOM()
+), geoms AS (
+  SELECT *, row_number() OVER () cartodb_id FROM _geoms
+), samples AS (
+  SELECT COUNT(*) cnt, unnest(ARRAY[1, 2, 3, 5, 10, 25, 50, 100, COUNT(*)]) sample FROM geoms
+), filtered AS (
+  SELECT * FROM geoms, samples WHERE cartodb_id % (cnt / sample) = 0
+), summary AS (
+  SELECT sample, ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+    COUNT(*)::INT cnt,
+    ARRAY_AGG((the_geom, cartodb_id)::geomval) geomvals,
+    SUM(ST_Area(the_geom))::Numeric sumarea
+  FROM filtered
+  GROUP BY sample
+), meta AS (
+  SELECT sample, cdb_observatory.OBS_GetMeta(extent,
+    ('[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "target_area": ' || sumarea || '}]')::JSON,
+    1, 1, cnt) meta
+  FROM summary
+  GROUP BY sample, extent, cnt, sumarea
+), results AS (
+  SELECT summary.sample, id, meta->0->>'geom_id' geom_id, (data->0->>'value')::Numeric as val
+  FROM summary, meta, LATERAL cdb_observatory.OBS_GetData(geomvals, meta) data
+  WHERE summary.sample = meta.sample
+) SELECT sample tract_sample
+ , MAX(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 tract_max_error
+ , AVG(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 tract_avg_error
+ , MIN(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 tract_min_error
+FROM geoms, results
+WHERE cartodb_id = id
+GROUP BY sample
+ORDER BY sample
+;
+
+-- Check that random assortment of block group points in Brooklyn return accurate data
+WITH _geoms AS (
+  SELECT
+    ST_PointOnSurface((data->0->>'value')::geometry) the_geom,
+    data->0->>'geomref' geom_ref,
+    (data->1->>'value')::numeric total_pop
+  FROM cdb_observatory.OBS_GetData(
+    array[(st_buffer(cdb_observatory._testpoint(), 0.2), 1)::geomval],
+    (SELECT cdb_observatory.OBS_GetMeta(ST_MakeEnvelope(-179, 89, 179, -89, 4326),
+      '[{"geom_id": "us.census.tiger.block_group"},
+        {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group", "normalization": "predenom"}]')),
+    FALSE
+  )
+  WHERE data->0->>'geomref' LIKE '36047%'
+), geoms AS (
+  SELECT *, row_number() OVER () cartodb_id FROM _geoms
+), samples AS (
+  SELECT COUNT(*) cnt, unnest(ARRAY[1, 2, 3, 5, 10, 25, 50, 100, COUNT(*)]) sample FROM geoms
+), filtered AS (
+  SELECT * FROM geoms, samples WHERE cartodb_id % (cnt / sample) = 0
+), summary AS (
+  SELECT sample, ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+    COUNT(*)::INT cnt,
+    ARRAY_AGG((the_geom, cartodb_id)::geomval) geomvals,
+    SUM(ST_Area(the_geom))::Numeric sumarea
+  FROM filtered
+  GROUP BY sample
+), meta AS (
+  SELECT sample, cdb_observatory.OBS_GetMeta(extent,
+    ('[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "target_area": ' || sumarea || '}]')::JSON,
+    1, 1, cnt) meta
+  FROM summary
+  GROUP BY sample, extent, cnt, sumarea
+), results AS (
+  SELECT summary.sample, id, meta->0->>'geom_id' geom_id, (data->0->>'value')::Numeric as val
+  FROM summary, meta, LATERAL cdb_observatory.OBS_GetData(geomvals, meta) data
+  WHERE summary.sample = meta.sample
+) SELECT
+ BOOL_AND(abs((geoms.total_pop - val) /
+      Coalesce(NullIf(total_pop, 0), 1)) = 0) is True no_bg_point_error
+FROM geoms, results
+WHERE cartodb_id = id
+;

src/pg/test/sql/42_observatory_exploration_test.sql

@@ -10,11 +10,11 @@ SET client_min_messages TO WARNING;
 
 -- _OBS_SearchTables tests
 SELECT
-  t.table_name = 'obs_1babf5a26a1ecda5fb74963e88408f71d0364b81' As _OBS_SearchTables_tables_match,
-  t.timespan = '2014' As _OBS_SearchTables_timespan_matches
+  t.table_name = 'obs_0310c639744a2014bb1af82709228f05b59e7d3d' As _OBS_SearchTables_tables_match,
+  t.timespan = '2015' As _OBS_SearchTables_timespan_matches
 FROM cdb_observatory._OBS_SearchTables(
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) As t(table_name, timespan);
 
 -- _OBS_SearchTables tests
@@ -278,9 +278,9 @@ AS _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg;
 SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
 FROM cdb_observatory.OBS_GetAvailableGeometries(
   ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
-  NULL, NULL, NULL, '2014'
+  NULL, NULL, NULL, '2015'
 ) WHERE valid_timespan = True)
-AS _obs_getavailablegeometries_bg_2014;
+AS _obs_getavailablegeometries_bg_2015;
 
 SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
 FROM cdb_observatory.OBS_GetAvailableGeometries(
@@ -350,126 +350,173 @@ AS _obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014;
 -- _OBS_GetGeometryScores tests
 --
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
        ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-             'us.census.tiger.zcta5', 'us.census.tiger.county']
+             'us.census.tiger.county', 'us.census.tiger.zcta5']
        AS _obs_geometryscores_500m_buffer
        FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-             'us.census.tiger.zcta5', 'us.census.tiger.county']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+       = ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.county', 'us.census.tiger.zcta5']
        AS _obs_geometryscores_5km_buffer
        FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.zcta5',
-             'us.census.tiger.county', 'us.census.tiger.block_group']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
        AS _obs_geometryscores_50km_buffer
        FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
-       ARRAY[ 'us.census.tiger.county', 'us.census.tiger.zcta5',
-             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY[ 'us.census.tiger.zcta5', 'us.census.tiger.census_tract',
+              'us.census.tiger.county', 'us.census.tiger.block_group' ]
       AS _obs_geometryscores_500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
-             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+       = ARRAY['us.census.tiger.county', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.block_group']
       AS _obs_geometryscores_2500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
-  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+  ARRAY['us.census.tiger.county', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.block_group'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text
-      = '{ "us.census.tiger.block_group" : 9, "us.census.tiger.census_tract" : 3, "us.census.tiger.zcta5" : 0, "us.census.tiger.county" : 0 }'
-      AS _obs_geometryscores_numgeoms_500m_buffer
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_500m_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
-      '{ "us.census.tiger.block_group" : 899, "us.census.tiger.census_tract" : 328, "us.census.tiger.zcta5" : 45, "us.census.tiger.county" : 1 }'
-      AS _obs_geometryscores_numgeoms_5km_buffer
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_5km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
-      '{ "us.census.tiger.block_group" : 12112, "us.census.tiger.census_tract" : 3792, "us.census.tiger.zcta5" : 550, "us.census.tiger.county" : 13 }'
-      AS _obs_geometryscores_numgeoms_50km_buffer
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_50km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
-      '{ "us.census.tiger.block_group" : 48415, "us.census.tiger.census_tract" : 15776, "us.census.tiger.zcta5" : 6534, "us.census.tiger.county" : 295 }'
-      AS _obs_geometryscores_numgeoms_500km_buffer
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
-      '{ "us.census.tiger.block_group" : 165489, "us.census.tiger.census_tract" : 55152, "us.census.tiger.zcta5" : 26500, "us.census.tiger.county" : 2551 }'
-      AS _obs_geometryscores_numgeoms_2500km_buffer
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_2500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
        ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
              'us.census.tiger.census_tract', 'us.census.tiger.block_group']
       AS _obs_geometryscores_500km_buffer_50_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50)
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC)
-      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.county',
-              'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.census_tract',
+              'us.census.tiger.county', 'us.census.tiger.block_group']
       AS _obs_geometryscores_500km_buffer_500_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500)
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.zcta5',
-             'us.census.tiger.county', 'us.census.tiger.block_group']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
       AS _obs_geometryscores_500km_buffer_2500_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500)
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-             'us.census.tiger.zcta5', 'us.census.tiger.county']
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+       = ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+               'us.census.tiger.county', 'us.census.tiger.zcta5']
       AS _obs_geometryscores_500km_buffer_25000_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000)
+      WHERE table_id LIKE '%2015%';
+
+-- Check that one small geom approximates tract data
+WITH geoms AS (SELECT cdb_observatory._testarea() the_geom),
+summary AS (SELECT ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+                   COUNT(*)::INT cnt,
+                   SUM(ST_Area(the_geom))::Numeric sumarea
+            FROM geoms)
+SELECT column_id = 'us.census.tiger.census_tract' testarea_uses_tract
+FROM summary, LATERAL (
+  SELECT *
+  FROM cdb_observatory._OBS_GetGeometryScores(extent,
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'],
+        cnt, sumarea)) foo
+ORDER BY score DESC LIMIT 1;
+
+-- Check that randomly distributed points always use smallest geometry if we
+-- order by numgeoms desc
+WITH geoms as (SELECT UNNEST(ARRAY[
+    cdb_observatory._testpoint(),
+    st_translate(cdb_observatory._testpoint(), -0.003, 0),
+    st_translate(cdb_observatory._testpoint(), -0.006, 0)
+]) the_geom),
+summary as (SELECT
+  ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+  SUM(ST_Area(the_geom))::Numeric area,
+  COUNT(*)::INTEGER cnt
+  FROM geoms
+)
+SELECT column_id = 'us.census.tiger.block_group' points_use_bg
+      FROM summary, LATERAL (
+        SELECT * FROM cdb_observatory._OBS_GetGeometryScores(
+          extent,
+          ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+                'us.census.tiger.zcta5', 'us.census.tiger.county'],
+        cnt, area)) foo
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC LIMIT 1;
 
 --
 -- OBS_LegacyBuilderMetadata tests
@@ -485,15 +532,25 @@ SELECT 'us.census.acs.B19013001' IN (SELECT
   FROM cdb_observatory.OBS_LegacyBuilderMetadata()
 ) AS _median_income_in_legacy_builder_metadata;
 
+SELECT 'us.census.acs.B19083001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _gini_in_legacy_builder_metadata;
+
 SELECT 'us.census.acs.B01003001' IN (SELECT
   (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
   FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
 ) AS _total_pop_in_legacy_builder_metadata_sums;
 
-SELECT 'us.census.acs.B19013001' NOT IN (SELECT
+SELECT 'us.census.acs.B19013001' IN (SELECT
   (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
   FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
-) AS _median_income_not_in_legacy_builder_metadata_sums;
+) AS _median_income_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19083001' NOT IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _gini_not_in_legacy_builder_metadata_sums;
 
 SELECT COUNT(*) = 0 _no_dupe_subsections_in_legacy_builder_metadata FROM (
   SELECT name, subsection, count(*) FROM

src/pg/test/sql/44_observatory_geometries_test.sql

@@ -42,7 +42,7 @@ SELECT cdb_observatory.OBS_GetBoundary(
 SELECT cdb_observatory.OBS_GetBoundary(
   cdb_observatory._TestPoint(),
   'us.census.tiger.census_tract',
-  '2014'
+  '2015'
 ) = :'cartodb_census_tract_geometry' As OBS_GetBoundary_year_census_tract;
 
 -- should return null
@@ -65,7 +65,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 SELECT cdb_observatory.OBS_GetBoundaryId(
   cdb_observatory._TestPoint(),
   'us.census.tiger.census_tract',
-  '2014'
+  '2015'
 ) = '36047048500'::text As OBS_GetBoundaryId_cartodb_census_tract_with_year;
 
 -- should give back '36047', the geoid of cartodb's county (King's/
@@ -73,7 +73,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 SELECT cdb_observatory.OBS_GetBoundaryId(
   cdb_observatory._TestPoint(),
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) = '36047'::text As OBS_GetBoundaryId_cartodb_county_with_year;
 
 -- should give back null since there is not a census tract at null island
@@ -104,7 +104,7 @@ SELECT cdb_observatory.OBS_GetBoundaryById(
 SELECT cdb_observatory.OBS_GetBoundaryById(
   '36047',
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) = :'cartodb_county_geometry' OBS_GetBoundaryById_boundary_id_mismatch_geom_id;
 
 -- should give null since boundary_id does not match geometry reference id
@@ -115,6 +115,34 @@ SELECT cdb_observatory.OBS_GetBoundaryById(
 
 -- _OBS_GetBoundariesByGeometry
 
+SELECT array_agg(geom_refs) = Array[ '1104486618765', '1104486642837',
+                                '1104991798384', '1105044325367',
+                                '1105089330200', '1105089331758']
+As _OBS_GetBoundariesByGeometry_roads_around_cartodb
+FROM (
+  SELECT *
+  FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+    -- near CartoDB's office
+    ST_MakeEnvelope(-74,40.69,-73.99,40.7,
+                    4326),
+    'us.census.tiger.prisecroads_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
+SELECT
+  array_agg(geom_refs) = Array['1102654301684', '1102654307106',
+                               '1102654326686', '1102654351507' ]
+As _OBS_GetBoundariesByGeometry_points_around_cartodb
+FROM (
+  SELECT *
+  FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+    -- near CartoDB's office
+    ST_MakeEnvelope(-73.9452409744,40.6988851644,-73.9280319214,40.7101254524,
+                    4326),
+    'us.census.tiger.pointlm_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
 -- check that all census tracts intersecting with the geometry are returned
 -- order them to ensure that the same values are returned
 SELECT
@@ -270,7 +298,7 @@ FROM (
                     -73.9280319214,40.7101254524,
                     4326),
     'us.census.tiger.census_tract',
-    '2014')
+    '2015')
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
@@ -313,7 +341,7 @@ FROM (
     cdb_observatory._testpoint(),
     500,
     'us.census.tiger.census_tract',
-    '2014')
+    '2015')
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
@@ -330,21 +358,4 @@ FROM (
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
--- _OBS_GetGeometryMetadata
--- get metadata for census tracts
-
-SELECT
-  geoid_colname = 'geoid' As geoid_name_matches,
-  target_table = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As table_name_matches,
-  geom_colname = 'the_geom' As geom_name_matches
-FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract')
-     As m(geoid_colname, target_table, geom_colname);
-
--- get metadata for boundaries with clipped geometries
- SELECT
-   geoid_colname = 'geoid' As geoid_name_matches,
-   target_table = 'obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963' As table_name_matches,
-   geom_colname = 'the_geom' As geom_name_matches
- FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract_clipped') As m(geoid_colname, target_table, geom_colname);
-
 \i test/fixtures/drop_fixtures.sql

src/python/requirements.txt

@@ -1,3 +1,4 @@
 nose
+nose-timer
 nose_parameterized
 psycopg2

src/python/test/autotest.py

@@ -2,38 +2,21 @@ from nose.tools import assert_equal, assert_is_not_none
 from nose.plugins.skip import SkipTest
 from nose_parameterized import parameterized
 
+from itertools import izip_longest
 from util import query
+from collections import OrderedDict
+import json
+
+
+def grouper(iterable, n, fillvalue=None):
+    "Collect data into fixed-length chunks or blocks"
+    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
+    args = [iter(iterable)] * n
+    return izip_longest(fillvalue=fillvalue, *args)
+
 
 USE_SCHEMA = True
 
-MEASURE_COLUMNS = query('''
-SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
-FROM observatory.obs_meta
-WHERE numer_type ILIKE 'numeric'
-AND numer_weight > 0
-''').fetchall()
-
-CATEGORY_COLUMNS = query('''
-SELECT distinct numer_id
-FROM observatory.obs_meta
-WHERE numer_type ILIKE 'text'
-AND numer_weight > 0
-''').fetchall()
-
-BOUNDARY_COLUMNS = query('''
-SELECT id FROM observatory.obs_column
-WHERE type ILIKE 'geometry'
-AND weight > 0
-''').fetchall()
-
-US_CENSUS_MEASURE_COLUMNS = query('''
-SELECT distinct numer_name
-FROM observatory.obs_meta
-WHERE numer_type ILIKE 'numeric'
-AND 'us.census.acs.acs' = ANY (subsection_tags)
-AND numer_weight > 0
-''').fetchall()
-
 SKIP_COLUMNS = set([
     u'mx.inegi_columns.INDI18',
     u'mx.inegi_columns.ECO40',
@@ -61,8 +44,72 @@ SKIP_COLUMNS = set([
     u'mx.inegi_columns.POB33',
     u'mx.inegi_columns.POB58',
     u'mx.inegi_columns.DISC4',
+    u'mx.inegi_columns.VIV41',
+    u'mx.inegi_columns.VIV40',
+    u'mx.inegi_columns.VIV17',
+    u'mx.inegi_columns.VIV25',
+    u'mx.inegi_columns.EDU10',
+    u'whosonfirst.wof_disputed_name',
+    u'us.census.tiger.fullname',
+    u'whosonfirst.wof_marinearea_name',
+    u'us.census.tiger.mtfcc',
+    u'whosonfirst.wof_county_name',
+    u'whosonfirst.wof_region_name',
+    'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    , 'fr.insee.P12_ACTOCC15P_ILT45D'
+    , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    , 'fr.insee.P12_ACTOCC15P_ILT45D'
+    , 'uk.ons.LC3202WA0007'
+    , 'uk.ons.LC3202WA0010'
+    , 'uk.ons.LC3202WA0004'
+    , 'uk.ons.LC3204WA0004'
+    , 'uk.ons.LC3204WA0007'
+    , 'uk.ons.LC3204WA0010'
 ])
 
+MEASURE_COLUMNS = query('''
+SELECT ARRAY_AGG(DISTINCT numer_id) numer_ids,
+       numer_aggregate,
+       denom_reltype,
+       section_tags
+FROM observatory.obs_meta
+WHERE numer_weight > 0
+  AND numer_id NOT IN ('{skip}')
+  AND section_tags IS NOT NULL
+  AND subsection_tags IS NOT NULL
+GROUP BY numer_aggregate, section_tags, denom_reltype
+'''.format(skip="', '".join(SKIP_COLUMNS))).fetchall()
+
+#CATEGORY_COLUMNS = query('''
+#SELECT distinct numer_id
+#FROM observatory.obs_meta
+#WHERE numer_type ILIKE 'text'
+#AND numer_weight > 0
+#''').fetchall()
+#
+#BOUNDARY_COLUMNS = query('''
+#SELECT id FROM observatory.obs_column
+#WHERE type ILIKE 'geometry'
+#AND weight > 0
+#''').fetchall()
+#
+#US_CENSUS_MEASURE_COLUMNS = query('''
+#SELECT distinct numer_name
+#FROM observatory.obs_meta
+#WHERE numer_type ILIKE 'numeric'
+#AND 'us.census.acs' = ANY (subsection_tags)
+#AND numer_weight > 0
+#''').fetchall()
+
+
 #def default_geometry_id(column_id):
 #    '''
 #    Returns default test point for the column_id.
@@ -113,40 +160,43 @@ def default_lonlat(column_id):
     elif column_id.startswith('th.'):
         return (13.725377712079784, 100.49263000488281)
     # cols for French Guyana only
-    elif column_id in ('fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
-                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
-                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
-                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
-                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
-                       , 'fr.insee.P12_ACTOCC15P_ILT45D'
-                       , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
-                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
-                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
-                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
-                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
-                       , 'fr.insee.P12_ACTOCC15P_ILT45D'):
-        return (4.938408371206558, -52.32908248901367)
+    #elif column_id in ('fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    #                   , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    #                   , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    #                   , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    #                   , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    #                   , 'fr.insee.P12_ACTOCC15P_ILT45D'
+    #                   , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    #                   , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    #                   , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    #                   , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    #                   , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    #                   , 'fr.insee.P12_ACTOCC15P_ILT45D'):
+    #    return (4.938408371206558, -52.32908248901367)
     elif column_id.startswith('fr.'):
         return (48.860875144709475, 2.3613739013671875)
     elif column_id.startswith('ca.'):
         return (43.65594991256823, -79.37965393066406)
     elif column_id.startswith('us.census.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.dma.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.ihme.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.bls.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.qcew.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('whosonfirst.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.epa.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('eu.'):
         raise SkipTest('No tests for Eurostat!')
-        #return (52.52207036136366, 13.40606689453125)
+    elif column_id.startswith('br.'):
+        return (-23.53, -46.63)
+    elif column_id.startswith('au.'):
+        return (-33.8806, 151.2131)
     else:
         raise Exception('No catalog point set for {}'.format(column_id))
 
@@ -166,46 +216,74 @@ def default_area(column_id):
         point=point)
     return area
 
-@parameterized(US_CENSUS_MEASURE_COLUMNS)
-def test_get_us_census_measure_points(name):
-    resp = query('''
-SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
-                 '''.format(name=name.replace("'", "''"),
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point('')))
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
+#@parameterized(US_CENSUS_MEASURE_COLUMNS)
+#def test_get_us_census_measure_points(name):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
+#                 '''.format(name=name.replace("'", "''"),
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point('')))
+#    rows = resp.fetchall()
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
 
 
-@parameterized(MEASURE_COLUMNS)
-def test_get_measure_areas(column_id, point_only):
-    if column_id in SKIP_COLUMNS:
-        raise SkipTest('Column {} should be skipped'.format(column_id))
-    if point_only:
+def grouped_measure_columns():
+    for numer_ids, numer_aggregate, denom_reltype, section_tags in MEASURE_COLUMNS:
+        for colgroup in grouper(numer_ids, 50):
+            yield [c for c in colgroup if c], numer_aggregate, denom_reltype, section_tags
+
+
+@parameterized(grouped_measure_columns())
+def test_get_measure_points(numer_ids, numer_aggregate, denom_reltype, section_tags):
+    _test_measures(numer_ids, numer_aggregate, section_tags, denom_reltype, default_point(numer_ids[0]))
+
+
+@parameterized(grouped_measure_columns())
+def test_get_measure_areas(numer_ids, numer_aggregate, denom_reltype, section_tags):
+    if numer_aggregate is None or numer_aggregate.lower() not in ('sum', 'median', 'average'):
         return
-    resp = query('''
-SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            area=default_area(column_id)))
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
+    if numer_aggregate.lower() in ('median', 'average') \
+       and (denom_reltype is None \
+            or denom_reltype.lower() != 'universe'):
+        return
+    _test_measures(numer_ids, numer_aggregate, section_tags, denom_reltype, default_area(numer_ids[0]))
 
 
-@parameterized(MEASURE_COLUMNS)
-def test_get_measure_points(column_id, point_only):
-    if column_id in SKIP_COLUMNS:
-        raise SkipTest('Column {} should be skipped'.format(column_id))
-    resp = query('''
-SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
+def _test_measures(numer_ids, numer_aggregate, section_tags, denom_reltype, geom):
+    in_params = []
+    for numer_id in numer_ids:
+        in_params.append({
+            'numer_id': numer_id,
+            'normalization': 'predenominated'
+        })
+
+    params = query(u'''
+        SELECT {schema}OBS_GetMeta({geom}, '{in_params}')
+    '''.format(schema='cdb_observatory.' if USE_SCHEMA else '',
+               geom=geom,
+               in_params=json.dumps(in_params))).fetchone()[0]
+
+    # We can get duplicate IDs from multi-denominators, so for now we
+    # compress those measures into a single
+    params = OrderedDict([(p['id'], p) for p in params]).values()
+    assert_equal(len(params), len(in_params),
+                 'Inconsistent out and in params for {}'.format(in_params))
+
+    q = u'''
+    SELECT * FROM {schema}OBS_GetData(ARRAY[({geom}, 1)::geomval], '{params}')
+    '''.format(schema='cdb_observatory.' if USE_SCHEMA else '',
+               geom=geom,
+               params=json.dumps(params).replace(u"'", "''"))
+    resp = query(q).fetchone()
+    assert_is_not_none(resp, 'NULL returned for {}'.format(in_params))
+    rawvals = resp[1]
+    vals = [v['value'] for v in rawvals]
+
+    assert_equal(len(vals), len(in_params))
+    for i, val in enumerate(vals):
+        assert_is_not_none(val, 'NULL for {}'.format(in_params[i]['numer_id']))
+
 
 #@parameterized(CATEGORY_COLUMNS)
 #def test_get_category_areas(column_id):
@@ -219,18 +297,18 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
 #    assert_equal(1, len(rows))
 #    assert_is_not_none(rows[0][0])
 
-@parameterized(CATEGORY_COLUMNS)
-def test_get_category_points(column_id):
-    if column_id in SKIP_COLUMNS:
-        raise SkipTest('Column {} should be skipped'.format(column_id))
-    resp = query('''
-SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
+#@parameterized(CATEGORY_COLUMNS)
+#def test_get_category_points(column_id):
+#    if column_id in SKIP_COLUMNS:
+#        raise SkipTest('Column {} should be skipped'.format(column_id))
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    rows = resp.fetchall()
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
 
 #@parameterized(BOUNDARY_COLUMNS)
 #def test_get_boundaries_by_geometry(column_id):

src/python/test/perftest.py

@@ -5,58 +5,348 @@ from util import query, commit
 
 from time import time
 
+import json
+import os
+
 USE_SCHEMA = True
 
 for q in (
-        'DROP TABLE IF EXISTS obs_censustest',
-        '''CREATE TABLE obs_censustest (cartodb_id SERIAL PRIMARY KEY,
-           the_geom GEOMETRY, name TEXT, measure NUMERIC, category TEXT)''',
-        '''INSERT INTO obs_censustest (the_geom, name)
-           SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
-           st_makeenvelope(-74.05437469482422,40.66319159533881,
-                           -73.81885528564453,40.745696344339564, 4326),
-           'us.census.tiger.block_group_clipped') As m(the_geom, geoid)'''
+        'DROP TABLE IF EXISTS obs_perftest_simple',
+        '''CREATE TABLE obs_perftest_simple (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_simple (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-74.1, 40.5,
+                                 -73.8, 40.9, 4326),
+                 'us.census.tiger.census_tract_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 1000''',
+        'DROP TABLE IF EXISTS obs_perftest_complex',
+        '''CREATE TABLE obs_perftest_complex (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_complex (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-75.05437469482422,40.66319159533881,
+                                 -73.81885528564453,41.745696344339564, 4326),
+                 'us.census.tiger.county_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;''',
+        'DROP TABLE IF EXISTS obs_perftest_country_simple',
+        '''CREATE TABLE obs_perftest_country_simple (cartodb_id SERIAL PRIMARY KEY,
+           geom GEOMETRY,
+           name TEXT) ''',
+        '''INSERT INTO obs_perftest_country_simple (geom, name)
+           SELECT the_geom geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-179,-89, 179,89, 4326),
+                 'whosonfirst.wof_country_geom')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 50;''',
+        'DROP TABLE IF EXISTS obs_perftest_country_complex',
+        '''CREATE TABLE obs_perftest_country_complex (cartodb_id SERIAL PRIMARY KEY,
+           geom GEOMETRY,
+           name TEXT) ''',
+        '''INSERT INTO obs_perftest_country_complex (geom, name)
+           SELECT the_geom geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-179,-89, 179,89, 4326),
+                 'whosonfirst.wof_country_geom')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;''',
+        #'''SET statement_timeout = 5000;'''
 ):
-    query(q.format(
+    q_formatted = q.format(
         schema='cdb_observatory.' if USE_SCHEMA else '',
-    ))
+    )
+    start = time()
+    resp = query(q_formatted)
+    end = time()
+    print('{} for {}'.format(int(end - start), q_formatted))
+    if q.lower().startswith('insert'):
+        if resp.rowcount == 0:
+            raise Exception('''Performance fixture creation "{}" inserted 0 rows,
+                            this will break tests.  Check the query to determine
+                            what is going wrong.'''.format(q_formatted))
     commit()
 
 
 ARGS = {
-    'OBS_GetMeasureByID': "name, 'us.census.acs.B01001002', '{}'",
-    'OBS_GetMeasure': "{}, 'us.census.acs.B01001002'",
-    'OBS_GetCategory': "{}, 'us.census.spielman_singleton_segments.X10'",
+    ('OBS_GetMeasureByID', None): "name, 'us.census.acs.B01001002', '{}'",
+    ('OBS_GetMeasure', 'predenominated'): "{}, 'us.census.acs.B01003001', null, {}",
+    ('OBS_GetMeasure', 'area'): "{}, 'us.census.acs.B01001002', 'area', {}",
+    ('OBS_GetMeasure', 'denominator'): "{}, 'us.census.acs.B01001002', 'denominator', {}",
+    ('OBS_GetCategory', None): "{}, 'us.census.spielman_singleton_segments.X10', {}",
+    ('_OBS_GetGeometryScores', None): "{}, NULL"
 }
 
-GEOMS = {
-    'point': 'ST_PointOnSurface(the_geom)',
-    'polygon_match': 'the_geom',
-    'polygon_buffered': 'ST_Buffer(the_geom::GEOGRAPHY, 1000)::GEOMETRY(GEOMETRY, 4326)',
-}
+
+def record(params, results):
+    sha = os.environ['OBS_EXTENSION_SHA']
+    msg = os.environ.get('OBS_EXTENSION_MSG')
+    fpath = os.path.join(os.environ['OBS_PERFTEST_DIR'], sha + '.json')
+    if os.path.isfile(fpath):
+        tests = json.load(open(fpath, 'r'))
+    else:
+        tests = {}
+    with open(fpath, 'w') as fhandle:
+        tests[json.dumps(params)] = {
+            'params': params,
+            'results': results
+        }
+        json.dump(tests, fhandle)
+
+@parameterized([
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 5000),
+
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 5000),
+])
+def test_getgeometryscores_performance(geom_complexity, api_method, filters, target_geoms):
+    print api_method, geom_complexity, filters, target_geoms
+
+    rownums = (1, 5, 10, ) if 'complex' in geom_complexity else (5, 25, 50,)
+    results = []
+    for rows in rownums:
+        stmt = '''SELECT {schema}{api_method}(geom, {filters}, {target_geoms})
+                   FROM obs_perftest_{complexity}
+                   WHERE cartodb_id <= {n}'''.format(
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       filters=filters,
+                       target_geoms=target_geoms,
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'filters': filters,
+            'target_geoms': target_geoms
+        }, results)
+
+@parameterized([
+    ('simple', 'OBS_GetMeasureByID', None, 'us.census.tiger.census_tract', None),
+    ('complex', 'OBS_GetMeasureByID', None, 'us.census.tiger.county', None),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'point', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+])
+def test_getmeasure_performance(geom_complexity, api_method, normalization, geom, boundary):
+    print api_method, geom_complexity, normalization, geom, boundary
+    col = 'measure' if 'measure' in api_method.lower() else 'category'
+    results = []
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (5, 25, 50, )
+    for rows in rownums:
+        stmt = '''UPDATE obs_perftest_{complexity}
+                   SET {col} = {schema}{api_method}({args})
+                   WHERE cartodb_id <= {n}'''.format(
+                       col=col,
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       args=ARGS[api_method, normalization].format(geom, boundary),
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'normalization': normalization,
+            'boundary': boundary,
+            'geom': geom
+        }, results)
 
 
 @parameterized([
-    ('OBS_GetMeasureByID', 'us.census.tiger.block_group_clipped'),
-    ('OBS_GetMeasureByID', 'us.census.tiger.county'),
-    ('OBS_GetMeasure', GEOMS['point']),
-    ('OBS_GetMeasure', GEOMS['polygon_match']),
-    ('OBS_GetMeasure', GEOMS['polygon_buffered']),
-    ('OBS_GetCategory', GEOMS['point']),
-    ('OBS_GetCategory', GEOMS['polygon_match']),
-    ('OBS_GetCategory', GEOMS['polygon_buffered']),
+    ('simple', 'predenominated', 'point', 'null'),
+    ('simple', 'predenominated', 'geom', 'null'),
+    ('simple', 'predenominated', 'offset_geom', 'null'),
+    ('simple', 'area', 'point', 'null'),
+    ('simple', 'area', 'geom', 'null'),
+    ('simple', 'area', 'offset_geom', 'null'),
+    ('simple', 'denominator', 'point', 'null'),
+    ('simple', 'denominator', 'geom', 'null'),
+    ('simple', 'denominator', 'offset_geom', 'null'),
+
+    ('simple', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'predenominated', 'geom', 'null'),
+    ('complex', 'predenominated', 'offset_geom', 'null'),
+    ('complex', 'area', 'geom', 'null'),
+    ('complex', 'area', 'offset_geom', 'null'),
+    ('complex', 'denominator', 'geom', 'null'),
+    ('complex', 'denominator', 'offset_geom', 'null'),
+
+    ('complex', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
 ])
-def test_performance(api_method, arg):
-    print api_method, arg
-    col = 'measure' if 'measure' in api_method.lower() else 'category'
-    for rows in (1, 10, 50, 100):
-        q = 'UPDATE obs_censustest SET {col} = {schema}{api_method}({args}) WHERE cartodb_id < {n}'.format(
-            col=col,
-            schema='cdb_observatory.' if USE_SCHEMA else '',
-            api_method=api_method,
-            args=ARGS[api_method].format(arg),
-            n=rows+1)
-        start = time()
-        query(q)
-        end = time()
-        print rows, ': ', (rows / (end - start)), ' QPS'
+def test_getdata_performance(geom_complexity, normalization, geom, boundary):
+    print geom_complexity, normalization, geom, boundary
+
+    cols = ['us.census.acs.B01001002',
+            'us.census.acs.B01001003',
+            'us.census.acs.B01001004',
+            'us.census.acs.B01001005',
+            'us.census.acs.B01001006',
+            'us.census.acs.B01001007',
+            'us.census.acs.B01001008',
+            'us.census.acs.B01001009',
+            'us.census.acs.B01001010',
+            'us.census.acs.B01001011', ]
+    in_meta = [{"numer_id": col,
+                "normalization": normalization,
+                "geom_id": None if boundary.lower() == 'null' else boundary.replace("'", '')}
+               for col in cols]
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (10, 50, 100)
+
+    for num_meta in (1, 10, ):
+        results = []
+        for rows in rownums:
+            stmt = '''
+    with data as (
+      SELECT id, data FROM {schema}OBS_GetData(
+        (SELECT array_agg(({geom}, cartodb_id)::geomval)
+         FROM obs_perftest_{complexity}
+         WHERE cartodb_id <= {n}),
+        (SELECT {schema}OBS_GetMeta(
+          (SELECT st_setsrid(st_extent({geom}), 4326)
+           FROM obs_perftest_{complexity}
+           WHERE cartodb_id <= {n}),
+          '{in_meta}'::JSON
+        ))
+      ))
+    UPDATE obs_perftest_{complexity}
+    SET measure = (data->0->>'value')::Numeric
+    FROM data
+    WHERE obs_perftest_{complexity}.cartodb_id = data.id
+    ;
+            '''.format(
+                point_or_poly='point' if geom == 'point' else 'polygon',
+                complexity=geom_complexity,
+                schema='cdb_observatory.' if USE_SCHEMA else '',
+                geom=geom,
+                in_meta=json.dumps(in_meta[0:num_meta]),
+                n=rows)
+            start = time()
+            query(stmt)
+            end = time()
+            qps = (rows / (end - start))
+            results.append({
+                'rows': rows,
+                'qps': qps,
+                'stmt': stmt
+            })
+            print rows, ': ', qps, ' QPS'
+
+        if 'OBS_RECORD_TEST' in os.environ:
+            record({
+                'geom_complexity': geom_complexity,
+                'api_method': 'OBS_GetData',
+                'normalization': normalization,
+                'boundary': boundary,
+                'geom': geom,
+                'num_meta': str(num_meta)
+            }, results)