Merge pull request #301 from CartoDB/develop

Release 1.6.0

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,23 @@
+## Request for a new Data observatory extension deploy
+
+I'd like to request a new data observatory extension deploy: dump + extension
+
+## Performance comparison to last deployment
+
+Please include link here to comparison perftests:
+
+http://52.71.151.140/perftest/#oldsha..newsha
+
+## Dump database id to be deployed
+
+Please put here the dump id to be deployed: <dump_id>
+
+## Data Observatory extension PRs included.
+
+*Please update the NEWS.md*
+
+Add down here the PR links to be added and deployed:
+
+  - 
+
+// @CartoDB/dataservices

Makefile

@@ -18,7 +18,7 @@ test:   ## Run the tests for the development version of the extension
 	$(MAKE) -C $(EXT_DIR) test
 
 # Generate a new release into release
-release: ## Generate a new release of the extension. Only for telease manager
+release: ## Generate a new release of the extension. Only for release manager
 	$(MAKE) -C $(EXT_DIR) release
 
 # Install the current release.

NEWS.md

@@ -1,3 +1,408 @@
-0.0.1 (open date)
+1.6.0 (2017-07-20)
+
+__Improvements__
+
+* The current OBS_GetAvailableNumerators is not designed with our
+  UI in mind so it's causing a lot of troubles and we're doing so
+  many hacks to fit our UI needs and the interface of the function so this
+  function it's a better fit for our purposes. ([#300](https://github.com/CartoDB/observatory-extension/pull/300))
+* Now use the new meta table `obs_meta_geom_numer_timespan` to filter
+  the geometries by geometries timespan and/or numerator timespan (which
+  is what we get when we use the obs_getavailabletimespans) ([#302](https://github.com/CartoDB/observatory-extension/pull/302))
+
+__Bugfixes__
+
+* Right now we're doing INNER JOINS when we JOIN the `_procgeoms` and
+  the data so we end up with NULL value instead of id, NULL value. ([#298](https://github.com/CartoDB/observatory-extension/pull/298))
+
+
+1.5.1 (2017-05-16)
+
+__Improvements__
+
+* Much improved performance for `OBS_GetData` when augmenting with several
+  different geometries simultaneously ([#285](https://github.com/CartoDB/observatory-extension/pull/285))
+* Return the automatically assigned normalization type from `OBS_GetMeta`
+  ([#285](https://github.com/CartoDB/observatory-extension/pull/285))
+
+1.5.0 (2017-04-24)
+
+__API Changes__
+
+* Add `suggested_name` to `OBS_GetMeta` responses
+  ([#281](https://github.com/CartoDB/observatory-extension/pull/281))
+* Add `geom_type`, `geom_extra`, and `geom_tags` to
+  `OBS_GetAvailableGeometries`.  This brings it up to spec with existing docs.
+  ([#282](https://github.com/CartoDB/observatory-extension/pull/282))
+* Add `timespan_type`, `timespan_extra`, and `timespan_tags` to
+  `OBS_GetAvailableTimespans` for consistency.
+  ([#282](https://github.com/CartoDB/observatory-extension/pull/282))
+
+1.4.0 (2017-03-21)
+
+__API Changes__
+
+* Allow for override of `target_area` and `target_geoms` in `OBS_GetMeta`
+  ([#276](https://github.com/CartoDB/observatory-extension/pull/276)).  This
+  allows the interface to work with points and sparse areas much btter.
+* Allow for override of `max_timespan_rank` and `max_score_rank` on an
+  item-by-item basis for metadata.
+* `numer_description`, `geom_description`, `denom_description`,
+  `numer_t_description`, `denom_t_description` and `geom_t_description` now
+  returned as part of `OBS_GetMeta`.
+
+__Improvements__
+
+* Reduced amount of simplification done on input geometries (from 0.0001 above
+  500 points to 0.00001 above 1000 points).
+* Added tests to confirm that accurate results are returned from automatic
+  boundary selection
+
+1.3.5 (2017-03-15)
+
+No changes.  Artifact to allow for data update.
+
+1.3.4 (2017-03-10)
+
+__Bugfixes__
+
+* Remove erroneously committed `RAISE NOTICE` in `OBS_GetData`
+
+1.3.3 (2017-03-10)
+
+__Bugfixes__
+
+* Resolve divide-by-zero errors in cases where the intersection of an
+  Observatory geometry and user geometry has 0 area
+  ([#265](https://github.com/CartoDB/observatory-extension/pull/265))
+* Run MakeValid on geometry's when intersecting, if necessary
+  ([#268](https://github.com/CartoDB/observatory-extension/pull/268))
+
+__Improvements__
+
+* Add performance tests for multiple columns in `OBS_GetData`
+* Major performance boost for `autotest.py` through the use of multi-column
+  `OBS_GetData` instead of separate `OBS_GetMeasure` calls for every single
+  measurement.
+  ([#268](https://github.com/CartoDB/observatory-extension/pull/268))
+* Major performance boost for `OBS_GetData` in cases where multiple columns are
+  requested.  Previously, each additional column would result in a linear
+  slowdown, even if geometries could be reused.
+  ([#267](https://github.com/CartoDB/observatory-extension/pull/267))
+
+1.3.2 (2017-03-02)
+
+__Bugfixes__
+
+* Accept "prenormalized" as well as "predenominated" to bypass normalization.
+  This fixes issues with Camshaft.
+
+1.3.1 (2017-02-16)
+
+__Improvements__
+
+* It is now possible to obtain measures that are averages or medians over
+  arbitrary polygons ([#254](https://github.com/CartoDB/observatory-extension/pull/254).
+* Added test point for Australian data
+* `OBS_GetLegacyMetadata` now returns median and averages in cases where it is
+  called for measures for polygons
+
+1.3.0 (2017-01-17)
+
+__API Changes__
+
+* `OBS_GetMeasureDataMulti()` is now called `OBS_GetData()`
+* `OBS_GetMeasureMetaMulti()` is now called `OBS_GetMeta()`
+* Additional signature for `OBS_GetData` which can take an array of `TEXT`,
+  mimicking functionality of `OBS_GetMeasureByID`
+
+__Improvements__
+
+* Generate fixtures from `obs_meta`
+* Remove unused table-level code
+* Refactor all augmentation and geometry functions to obtain data from
+  `OBS_GetMeta()` and `OBS_GetData()`.
+* Improvements to `OBS_GetMeta()` so it can still fill in metadata in cases
+  where only a geometry is being requested.
+* `OBS_GetData()` returns two-column table instead of anonymous record.
+* `OBS_GetData()` can return categorical (text) and geometries
+
+__Bugfixes__
+
+* Remove unnecessary dependency on `postgres_fdw`
+* `OBS_GetData()` now aggregates measures with mixed geoms correctly
+
+__API Changes__
+
+1.2.1 (2017-01-17)
+
+__Improvements__
+
+* Support Point/LineString in responses from `OBS_GetBoundary`.
+  ([#243](https://github.com/CartoDB/observatory-extension/pull/233))
+
+1.2.0 (2016-12-28)
+
+__API Changes__
+
+* Added `OBS_GetMeasureDataMulti`, which takes an array of geomvals and
+  parameters as JSON, and returns a set of RECORDs keyed by the vals of the
+  geomvals.
+* Added `OBS_GetMeasureMetaMulti`, which takes sparse metadata as JSON (for
+  example, the measure ID) and returns a filled-out version of the metadata
+  sufficient for use with `OBS_GetMeasureDataMulti`.
+
+__Improvements__
+
+* Move tests to 2015
+* Fixes to `_OBS_GetGeometryScores` to avoid spamming NOTICEs about all pixels
+  for a band being NULL
+* Tests for `_OBS_GetGeometryScores` with complex geometries
+* Performance tests for `OBS_GetMeasureDataMulti`
+* Return both `table_id` and `column_id` from `_OBS_GetGeometryScores`
+
+1.1.7 (2016-12-15)
+
+__Improvements__
+
+* Use simpler raster table and simplified `_OBSGetGeometryScores` functions to
+  improve performance
+* In cases where geometry passed into geometry scoring function has greater
+  than 10K points, simply use its buffer instead
+* Add `IMMUTABLE` to `_OBSGetGeometryScores`
+* Add tests explicitly for `_OBSGetGeometryScores` in perftest.py
+* Yields a ~50% improvement in performance for `_OBSGetGeomeryScores`.
+
+1.1.6 (2016-12-08)
+
+__Bugfixes__
+
+* Fix divide by zero condition in "denominator" branch of `OBS_GetMeasure`
+  when passing in a polygon ([#233](https://github.com/CartoDB/observatory-extension/pull/233)).
+
+__Improvements__
+
+* Use `ST_Subdivide` to improve performance when functions are called on very
+  complex geometries (with many points) ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Improve raster scoring to more heavily weight boundaries with nearer to
+  correct number of points, and penalize boundaries with lots of blank space
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Remove some redundant area calculations in `OBS_GetMeasure`
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Replace use of `format('%L', var)` with proper use of `EXECUTE` and `$1` etc.
+  variables ([#231](https://github.com/CartoDB/observatory-extension/pull/231))
+* Add test point for Brazil
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+* Improvements to performance tests
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+  - Support simple and complex geometries
+  - Handle all code branches
+  - Add ability to persist results to JSON for graph visualization later
+
+1.1.5 (2016-11-29)
+
+__Bugfixes__
+
+* Return `NULL` instead of raising an exception when a measure is requested for
+  a geometry where it does not exist ([#220](https://github.com/CartoDB/observatory-extension/issues/220)).
+
+1.1.4 (2016-11-21)
+
+__Bugfixes__
+
+* Fix duplicate subsections with only a partial set of measures appearing from
+  `OBS_GetLegacyMetadata` ([#216](https://github.com/CartoDB/observatory-extension/issues/216)).
+
+1.1.3 (2016-11-15)
+
+* Temporarily ignore EU data for the sake of testing
+
+1.1.2 (2016-11-09)
+
+__Improvements__
+
+* Update public `OBS_GetMeasure` to use highest ranked boundary, aiming for 500
+  geoms. ([#190](https://github.com/CartoDB/observatory-extension/issues/190))
+* Update test generation to capture our raster tiles
+* Standardize the way we generate our test points for `autotest.py`
+* Add points for epa and eurostat
+* Should support database dump generated 20161109
+
+__API Changes (Internal)__
+
+* Add internal `_OBS_GetGeometryScores`
+
+1.1.1 (2016-10-14)
+
+__Improvements__
+
+* Test points for Canada and France ([#204](https://github.com/CartoDB/observatory-extension/issues/120))
+
+1.1.0 (2016-10-04)
+
+__Bugfixes__
+
+* Fixed some minor errors in test suite
+
+__Improvements__
+
+* We now generate test fixtures from local data instead of remote server
+  ([#120](https://github.com/CartoDB/observatory-extension/issues/120))
+
+__API Changes__
+
+* New function, `OBS_LegacyBuilderMetadata`, which resolves
+  ([#133]( https://github.com/CartoDB/observatory-extension/issues/133))
+* Creates "dimensional" metadata grabbing functions
+  (`OBS_GetAvailableNumerators`, `OBS_GetAvailableDenominators`,
+  `OBS_GetAvailableGeometries`, `OBS_GetAvailableTimespans`) which will be
+  used for obtaining metadata in the replacement for the Data Library
+  ([CartoDB/design#104](https://github.com/CartoDB/design/issues/104)). This
+  is also referred to here ([CartoDB/design#68](https://github.com/CartoDB/design/issues/68)).
+
+1.0.7 (2016-09-20)
+
+__Bugfixes__
+
+* `NULL` geometries or geometry IDs no longer result in an exception from any
+  augmentation functions ([#178](https://github.com/CartoDB/observatory-extension/issues/178))
+
+__Improvements__
+
+* Automatic tests work for Canada and Thailand
+
+1.0.6 (2016-09-08)
+
+__Improvements__
+
+* New function structure for Table-level functions which allows to separate the
+  framework logic from the observatory measure functions.
+
+1.0.5 (2016-08-12)
+
+__Improvements__
+
+* Integration tests moved to `src/python/test/`, and can be run without hitting
+  any HTTP SQL API.
+
+1.0.4 (2016-07-26)
+
+__Bugfixes__
+
+* Always default arguments to `NULL`, which prevents duplication & overwrite by
+  dataservices-api
+  ([#173](https://github.com/CartoDB/observatory-extension/issues/173))
+
+1.0.3 (2016-07-25)
+
+__Bugfixes__
+
+* Raise exception instead of crashing when `OBS_GetMeasure` is passed a polygon
+  in combination with a non-summable measure ([cartodb/issues
+  #9063](https://github.com/CartoDB/cartodb/issues/9063))
+* Unnecessary dependencies on cartodb and plpythonu removed
+  ([#161](https://github.com/CartoDB/observatory-extension/issues/161))
+* Tests forced to run in-order on all systems
+  ([#162](https://github.com/CartoDB/observatory-extension/issues/162))
+* Area normalization done by square kilometer instead of square meter for
+  polygons ([#158](https://github.com/CartoDB/observatory-extension/issues/158))
+* `postgres-fdw` installed as required in unit test environment
+  ([#166](https://github.com/CartoDB/observatory-extension/issues/166))
+
+__Improvements__
+
+* Added tests to make sure all functions can handle explicit NULL as default
+  ([#159](https://github.com/CartoDB/observatory-extension/issues/159))
+* Buffer and snaptogrid used to be far more liberal accepting problem geoms
+  ([#170](https://github.com/CartoDB/observatory-extension/issues/160))
+
+
+1.0.2 (2016-07-12)
+---
+
+__Bugfixes__
+
+* Fix for `OBS_GetCategory` outside the US ([#135](https://github.com/CartoDB/observatory-extension/pull/137))
+* `OBS_GetMeasure` now respects the `normalize` parameter even when passed
+  a multi/polygon. Previously, no normalization was erroneously assumed.
+
+__Improvements__
+
+* Automated tests cover Mexico data
+* `obs_meta` is now provisioned during unit tests
+* `obs_meta` is now used during end-to-end tests
+* `OBS_GetMeasureByID` uses `obs_meta` internally, which should help
+  performance
+* `OBS_GetCategory` uses `obs_meta` internally, which should help perfromance
+* `OBS_GetCategory` will pick the correct category for an arbitrary polygon
+  (the category covering the highest % of that polygon)
+* `OBS_GetMeasure` has been updated to use `obs_meta` internally, which should
+  help performance
+* `OBS_GetMeasure` now can be passed "none" and skip normalization by area or
+  denominator for points
+* Fixtures are only loaded at the start of the unit test suite, and dropped at the end,
+  instead of at the start/end of each individual test file
+* Comment noisy NOTICEs ([#73](https://github.com/CartoDB/observatory-extension/issues/73))
+
+1.0.1 (2016-07-01)
+---
+
+__Bugfixes__
+
+* Fix for ERROR:  Operation on mixed SRID geometries #130
+
+
+1.0.0 (6/27/2016)
+-----
+
+* Incremented to 1.0.0 to be in compliance with [SemVer](http://semver.org/),
+  which disallows use of 0.x.x versions.  This also reflects that we are
+  already in production.
+
+__API Changes__
+
+* Added `OBS_DumpVersion` to look up version data ([#118](https://github.com/CartoDB/observatory-extension/pull/118))
+
+__Improvements__
+
+* Whether data exists for a geom now determined by polygon intersection instead of
+  BBOX overlap ([#119](https://github.com/CartoDB/observatory-extension/pull/119))
+* Automated tests cover Spanish and UK data
+  ([#115](https://github.com/CartoDB/observatory-extension/pull/115))
+* Automated tests cover `OBS_GetUSCensusMeasure`
+  ([#105](https://github.com/CartoDB/observatory-extension/pull/105))
+
+__Bugfixes__
+
+* Geom table can have different `geomref_colname` than the data table
+  ([#123](https://github.com/CartoDB/observatory-extension/pull/123))
+
+
+0.0.5 (5/27/2016)
+-----
+* Adds new function `OBS_GetMeasureById` ([#96](https://github.com/CartoDB/observatory-extension/pull/96))
+
+0.0.4 (5/25/2016)
+-----
+* Updates queries involving US Census measure tags to be more generic ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
+* Fixes tests which relied on an erroneous subset of block groups ([#95](https://github.com/CartoDB/observatory-extension/pull/95))
+
+0.0.3 (5/24/2016)
+-----
+* Generalizes internal queries to properly pull from multiple named geometry references
+* Adds tests for Who's on First boundaries
+* Improves automatic fixtures testing script
+
+0.0.2 (5/19/2016)
+-----
+* Adds Data Observatory exploration functions
+* Adds Data Observatory boundary functions
+* Adds Data Observatory measure functions
+* Adds script to generate fixtures for tests
+* Adds script for the automatic testing of metadata
+* Adds full documentation for all included functions
+* removes `cartodb` extension dependency
+
+0.0.1 (5/19/2016)
 ------------------
-* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`;
+* First iteration of `OBS_GetDemographicSnapshot(location Geometry(Point,4326))`

RELEASE.md

@@ -20,12 +20,6 @@ script for the new release, `release/observatory--X.Y.Z.sql`:
 make release
 ```
 
-Then, the release manager shall produce upgrade and downgrade scripts
-to migrate to/from the previous release. In the case of minor/patch
-releases this simply consist in extracting the functions that have changed
-and placing them in the proper `release/observatory--X.Y.Z--A.B.C.sql`
-file.
-
 The new release can be deployed for staging/smoke tests with this command:
 
 ```

doc/API.md

@@ -1,11 +1,13 @@
 # Data Observatory Documentation
 
-This file is for reference purposes only. It is intended for tracking the Data Observatory functions that should be displayed from the live Docs site. Like all API doc, the golden source of this code will live in this observatory-extension repo, and will be edited in this repo.
+This file is for reference purposes only. It is intended for tracking the Data Observatory functions that should be pulled into the live Docs site. Like all API doc, the golden source of this code will live in this observatory-extension repo, and will be edited in this repo. Other non-code related content will live as a local file in the Docs repo.
 
 ## Documentation
 
+* Overview (local file in the Docs repo)
+* Accessing the Data Observatory (local file in the Docs repo)
 * [Measures Functions](measures_functions.md)
 * [Boundary Functions](boundary_functions.md)
 * [Discovery Functions](discovery_functions.md)
-* [Glossary](glossary.md)
-* [License](license.md)
+* [Glossary](local file in the Docs repo)
+* [License](local file in the Docs repo)

doc/boundary_functions.md

@@ -1,10 +1,10 @@
 # Boundary Functions
 
-Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
+Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
-## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
+## OBS_GetBoundariesByGeometry(geom geometry, geometry_id text)
 
 The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a set of boundary geometries that intersect a supplied geometry. This can be used to find all boundaries that are within or overlap a bounding box. You have the ability to choose whether to retrieve all boundaries that intersect your supplied bounding box or only those that fall entirely inside of your bounding box.
 
@@ -12,7 +12,7 @@ The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a se
 
 Name |Description
 --- | ---
-polygon | a bounding box or other WGS84 geometry
+geom | a WGS84 geometry
 geometry_id | a string identifier for a boundary geometry
 timespan (optional) | year(s) to request from ('NULL' (default) gives most recent)
 overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
@@ -26,7 +26,7 @@ Column Name | Description
 the_geom | a boundary geometry (e.g., US Census tract boundaries)
 geom_refs | a string identifier for the geometry (e.g., geoids of US Census tracts)
 
-If geometries are not found for the requested `polygon`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
+If geometries are not found for the requested `geom`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
 
 #### Example
 
@@ -44,7 +44,6 @@ FROM OBS_GetBoundariesByGeometry(
 
 #### Errors
 
-* If a geometry other than a point is passed as the first argument, an error is thrown: `Invalid geometry type (ST_Polygon), expecting 'ST_Point'`
 * If an `overlap_type` other than the valid ones listed above is entered, then an error is thrown
 
 ## OBS_GetPointsByGeometry(polygon geometry, geometry_id text)
@@ -91,14 +90,14 @@ FROM OBS_GetPointsByGeometry(
 
 ## OBS_GetBoundary(point_geometry, boundary_id)
 
-The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID glossary table below](below). This is a useful method for performing aggregations of points.
+The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids). This is a useful method for performing aggregations of points.
 
 #### Arguments
 
 Name | Description
 --- | ---
 point_geometry | a WGS84 polygon geometry (the_geom)
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
 
 #### Returns
@@ -124,14 +123,14 @@ SET the_geom = OBS_GetBoundary(the_geom, 'us.census.tiger.block_group')
 
 ## OBS_GetBoundaryId(point_geometry, boundary_id)
 
-The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID glossary table below](below). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
+The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID Glossary](http://docs/carto-engine/data/glossary/#boundary-ids). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
 
 #### Arguments
 
 Name |Description
 --- | ---
 point_geometry | a WGS84 point geometry (the_geom)
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
 
 #### Returns
@@ -164,7 +163,7 @@ The ```OBS_GetBoundaryById(geometry_id, boundary_id)``` returns the boundary geo
 Name | Description
 --- | ---
 geometry_id | a string identifier for a Boundary geometry
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
 
 #### Returns

doc/discovery_functions.md

@@ -1,6 +1,6 @@
 # Discovery Functions
 
-If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data.
+If you are using the [discovery methods](https://carto.com/docs/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) and [measures](https://carto.com/docs/carto-engine/data/overview/#measures-data) data.
 
 ## OBS_Search(search_term)
 
@@ -47,7 +47,7 @@ A TABLE containing the following properties
 
 Key | Description
 --- | ---
-boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 description | a brief description of the boundary dataset
 time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
 
@@ -56,3 +56,309 @@ time_span | the timespan attached the boundary. this does not mean that the boun
 ```SQL
 SELECT * FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.7, -73.9))
 ```
+
+## OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan)
+
+Return available numerators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the numerator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only numerators for which all of these apply are returned  `NULL` to ignore (optional)
+denom_id | Text | the ID of a denominator to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+numer_id | Text | The ID of the numerator
+numer_name | Text | A human readable name for the numerator
+numer_description | Text | Description of the numerator.  Is sometimes NULL
+numer_weight | Numeric | Numeric "weight" of the numerator.  Ignored.
+numer_license | Text | ID of the license for the numerator
+numer_source | Text | ID of the source for the numerator
+numer_type | Text | Postgres type of the numerator
+numer_aggregate | Text | Aggregate type of the numerator.  If `'SUM'`, this can be normalized by area
+numer_extra | JSONB | Extra information about the numerator column.  Ignored.
+numer_tags | Text[] | Array of all tags applying to this numerator
+valid_denom | Boolean | True if the `denom_id` argument is a valid denominator for this numerator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this numerator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this numerator, False otherwise
+
+#### Examples
+
+Obtain all numerators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326))
+```
+
+Obtain all numerators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+employment related for the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+related to both employment and age & gender for the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment, subsection/tags.age_gender}');
+```
+
+Obtain all numerators that work with US population (`us.census.acs.B01003001`)
+as a denominator.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_denom IS True;
+```
+
+Obtain all numerators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all numerators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan)
+
+Return available denominators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the denominator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only denominators for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+geom_id | Text | the ID of a geometry to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+denom_id | Text | The ID of the denominator
+denom_name | Text | A human readable name for the denominator
+denom_description | Text | Description of the denominator.  Is sometimes NULL
+denom_weight | Numeric | Numeric "weight" of the denominator.  Ignored.
+denom_license | Text | ID of the license for the denominator
+denom_source | Text | ID of the source for the denominator
+denom_type | Text | Postgres type of the denominator
+denom_aggregate | Text | Aggregate type of the denominator.  If `'SUM'`, this can be normalized by area
+denom_extra | JSONB | Extra information about the denominator column.  Ignored.
+denom_tags | Text[] | Array of all tags applying to this denominator
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this denominator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this denominator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this denominator, False otherwise
+
+#### Examples
+
+Obtain all denominators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all denominators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all denominators for male population (`us.census.acs.B01001002`).
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01001002')
+WHERE valid_numer IS True;
+```
+
+Obtain all denominators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all denominators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan)
+
+Return available geometries within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which must intersect the geometry
+filter_tags | Text[] | a list of filters.  Only geometries for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+timespan | Text | the ID of a timespan to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+geom_id | Text | The ID of the geometry
+geom_name | Text | A human readable name for the geometry
+geom_description | Text | Description of the geometry.  Is sometimes NULL
+geom_weight | Numeric | Numeric "weight" of the geometry.  Ignored.
+geom_aggregate | Text | Aggregate type of the geometry.  Ignored.
+geom_license | Text | ID of the license for the geometry
+geom_source | Text | ID of the source for the geometry
+geom_type | Text | Postgres type of the geometry
+geom_extra | JSONB | Extra information about the geometry column.  Ignored.
+geom_tags | Text[] | Array of all tags applying to this geometry
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this geometry, False otherwise
+valid_denom | Boolean | True if the `geom_id` argument is a valid geometry for this geometry, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this geometry, False otherwise
+score | Numeric | Score between 0 and 100 for this geometry, higher numbers mean that this geometry is a better choice for the passed extent
+numtiles | Numeric | How many raster tiles were read for score, numgeoms, and percentfill estimates
+numgeoms | Numeric | About how many of these geometries fit inside the passed extent
+percentfill | Numeric | About what percentage of the passed extent is filled with these geometries
+estnumgeoms | Numeric | Ignored
+meanmediansize | Numeric | Ignored
+
+#### Examples
+
+Obtain all geometries that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all geometries that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all geometries that work with total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all geometries with timespan  `2015`.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id)
+
+Return available timespans within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the timespan's data must intersect with
+filter_tags | Text[] | a list of filters.  Ignore
+numer_id | Text | the ID of a numerator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+timespan_id | Text | The ID of the timespan
+timespan_name | Text | A human readable name for the timespan
+timespan_description | Text | Ignored
+timespan_weight | Numeric | Ignored
+timespan_aggregate | Text | Ignored
+timespan_license | Text | Ignored
+timespan_source | Text | Ignored
+timespan_type | Text | Ignored
+timespan_extra | JSONB | Ignored
+timespan_tags | JSONB | Ignored
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this timespan, False otherwise
+valid_denom | Boolean | True if the `timespan` argument is a valid timespan for this timespan, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this timespan, False otherwise
+
+#### Examples
+
+Obtain all timespans that are available within a small rectangle.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all timespans for total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all timespans that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```

doc/glossary.md

@@ -1,126 +0,0 @@
-# Glossary
-
-A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
-
-## Boundary IDs
-
-Boundary name | Boundary ID | Shoreline Clipped Boundary ID
---------------------- | --------------------- | ---
-US States  |  us.census.tiger.state  |  us.census.tiger.state_clipped
-US County  |  us.census.tiger.county  |  us.census.tiger.county_clipped
-US Census Zip Code Tabulation Areas  |  us.census.tiger.zcta5  |  us.census.tiger.zcta5_clipped
-US Census Tracts  |  us.census.tiger.census_tract  |  us.census.tiger.census_tract_clipped
-US Elementary School District  |  us.census.tiger.school_district_elementary  |  us.census.tiger.school_district_elementary_clipped
-US Secondary School District  |  us.census.tiger.school_district_secondary  |  us.census.tiger.school_district_secondary_clipped
-US Unified School District  |  us.census.tiger.school_district_unified  |  us.census.tiger.school_district_unified_clipped
-US Congressional Districts  |  us.census.tiger.congressional_district  |  us.census.tiger.congressional_district_clipped
-US Census Blocks  |  us.census.tiger.block  |  us.census.tiger.block_clipped
-US Census Block Groups  |  us.census.tiger.block_group  |  us.census.tiger.block_group_clipped
-US Census PUMAs  |  us.census.tiger.puma  |  us.census.tiger.puma_clipped
-US Incorporated Places  |  us.census.tiger.place  |  us.census.tiger.place_clipped
-ES Sección Censal  |  es.ine.geom  |   none
-Regions (First-level Administrative)  |  whosonfirst.wof_region_geom  |   none
-Continents  |  whosonfirst.wof_continent_geom  |   none
-Countries  |  whosonfirst.wof_country_geom  |  none
-Marine Areas  |  whosonfirst.wof_marinearea_geom  |  none
-Disputed Areas  |  whosonfirst.wof_disputed_geom  |  none
-
-
-
-## OBS_GetUSCensusMeasure Names Table
-
-This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
-
-Measure name             |  Measure description
------------------------- | --------------------
-Male Population   |   The number of people within each geography who are male.
-Female Population   |   The number of people within each geography who are female.
-Median Age   |   The median age of all people in a given geographic area.
-Total Population   |   The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.
-Population not Hispanic   |   The number of people not identifying as Hispanic or Latino in each geography.
-White Population   |   The number of people identifying as white, non-Hispanic in each geography.
-Black or African American Population   |   The number of people identifying as black or African American, non-Hispanic in each geography.
-American Indian and Alaska Native Population   |   The number of people identifying as American Indian or Alaska native in each geography.
-Asian Population   |   The number of people identifying as Asian, non-Hispanic in each geography.
-Other Race population   |   The number of people identifying as another race in each geography.
-Two or more races population   |   The number of people identifying as two or more races in each geography.
-Hispanic Population   |   The number of people identifying as Hispanic or Latino in each geography.
-Not a U.S. Citizen Population   |   The number of people within each geography who indicated that they are not U.S. citizens.
-Workers over the Age of 16   |   The number of people in each geography who work. Workers include those employed at private for-profit companies, the self-employed, government workers and non-profit employees.
-Commuters by Car, Truck, or Van   |   The number of workers age 16 years and over within  a geographic area who primarily traveled to work by car, truck or  van.  This is the principal mode of travel or type of conveyance,  by distance rather than time, that the worker usually used to get  from home to work.
-Commuters who drove alone   |   The number of workers age 16 years and over within a geographic area who primarily traveled by car driving alone. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Commuters by Carpool   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by carpool.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Commuters by Bus   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by bus.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.  This is a subset of workers who commuted by public transport.
-Commuters by Subway or Elevated   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by subway or elevated train.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.  This is a subset of workers who commuted by public transport.
-Walked to Work   |   The number of workers age 16 years and over within a geographic area who primarily walked to work.  This would mean that of any way of getting to work, they travelled the most distance walking.
-Worked at Home   |   The count within a geographical area of workers over the age of 16 who worked at home.
-Workers age 16 and over who do not work from home   |   The number of workers over the age of 16 who do not work from home in a geographic area.
-Number of workers with less than 10 minute commute   |   The number of workers over the age of 16 who do not work from home and commute in less than 10 minutes in a geographic area.
-Number of workers with a commute between 35 and 44 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 35 and 44 minutes in a geographic area.
-Number of workers with a commute of over 60 minutes   |   The number of workers over the age of 16 who do not work from home and commute in over 60 minutes in a geographic area.
-Aggregate travel time to work   |   The total number of minutes every worker over the age of 16 who did not work from home spent spent commuting to work in one day in a geographic area.
-Commuters by Public Transportation   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by public transportation.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Number of workers with a commute between 10 and 14 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 10 and 14 minutes in a geographic area.
-Number of workers with a commute between 15 and 19 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 15 and 19 minutes in a geographic area.
-Number of workers with a commute between 20 and 24 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 20 and 24 minutes in a geographic area.
-Number of workers with a commute between 25 and 29 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 25 and 29 minutes in a geographic area.
-Number of workers with a commute between 30 and 34 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area.
-Number of workers with a commute between 45 and 59 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area.
-Children under 18 Years of Age   |   The number of people within each geography who are under 18 years of age.
-Households   |   A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
-Population 15 Years and Over   |   The number of people in a geographic area who are over the age of 15.  This is used mostly as a denominator of marital status.
-Never Married   |   The number of people in a geographic area who have never been married.
-Currently married   |   The number of people in a geographic area who are currently married.
-Married but separated   |   The number of people in a geographic area who are married but separated.
-Widowed   |   The number of people in a geographic area who are widowed.
-Divorced   |   The number of people in a geographic area who are divorced.
-Population 3 Years and Over   |   The total number of people in each geography age 3 years and over.  This denominator is mostly used to calculate rates of school enrollment.
-Students Enrolled in School   |   The total number of people in each geography currently enrolled at any level of school, from nursery or pre-school to advanced post-graduate education.  Only includes those over the age of 3.
-Students Enrolled in Grades 1 to 4   |   The total number of people in each geography currently enrolled in grades 1 through 4 inclusive.  This corresponds roughly to elementary school.
-Students Enrolled in Grades 5 to 8   |   The total number of people in each geography currently enrolled in grades 5 through 8 inclusive.  This corresponds roughly to middle school.
-Students Enrolled in Grades 9 to 12   |   The total number of people in each geography currently enrolled in grades 9 through 12 inclusive.  This corresponds roughly to high school.
-Students Enrolled as Undergraduate in College   |   The number of people in a geographic area who are enrolled in college at the undergraduate level. Enrollment refers to being registered or listed as a student in an educational program leading to a college degree. This may be a public school or college, a private school or college.
-Population 25 Years and Over   |   The number of people in a geographic area who are over the age of 25.  This is used mostly as a denominator of educational attainment.
-Population Completed High School   |   The number of people in a geographic area over the age of 25 who completed high school, and did not complete a more advanced degree.
-Population completed less than one year of college, no degree   |   The number of people in a geographic area over the age of 25 who attended college for less than one year and no further.
-Population completed more than one year of college, no degree   |   The number of people in a geographic area over the age of 25 who attended college for more than one year but did not obtain a degree.
-Population Completed Associate's Degree   |   The number of people in a geographic area over the age of 25 who obtained a associate's degree, and did not complete a more advanced degree.
-Population Completed Bachelor's Degree   |   The number of people in a geographic area over the age of 25 who obtained a bachelor's degree, and did not complete a more advanced degree.
-Population Completed Master's Degree   |   The number of people in a geographic area over the age of 25 who obtained a master's degree, but did not complete a more advanced degree.
-Population 5 Years and Over   |   The number of people in a geographic area who are over the age of 5.  This is primarily used as a denominator of measures of language spoken at home.
-Speaks only English at Home   |   The number of people in a geographic area over age 5 who speak only English at home.
-Speaks Spanish at Home   |   The number of people in a geographic area over age 5 who speak Spanish at home, possibly in addition to other languages.
-Population for Whom Poverty Status Determined   |   The number of people in each geography who could be identified as either living in poverty or not.  This should be used as the denominator when calculating poverty rates, as it excludes people for whom it was not possible to determine poverty.
-Income In The Past 12 Months Below Poverty Level   |   The number of people in a geographic area who are part of a family (which could be just them as an individual) determined to be in poverty following the Office of Management and Budget's Directive 14. (https://www.census.gov/hhes/povmeas/methodology/ombdir14.html)
-Households with income less than $10,000   |   The number of households in a geographic area whose annual income was less than $10,000.
-Households with income of $10,000 to $14,999   |   The number of households in a geographic area whose annual income was between $10,000 and $14,999.
-Households with income of $15,000 to $19,999   |   The number of households in a geographic area whose annual income was between $15,000 and $19,999.
-Households with income of $20,000 To $24,999   |   The number of households in a geographic area whose annual income was between $20,000 and $24,999.
-Households with income of $25,000 To $29,999   |   The number of households in a geographic area whose annual income was between $20,000 and $24,999.
-Households with income of $30,000 To $34,999   |   The number of households in a geographic area whose annual income was between $30,000 and $34,999.
-Households with income of $35,000 To $39,999   |   The number of households in a geographic area whose annual income was between $35,000 and $39,999.
-Households with income of $40,000 To $44,999   |   The number of households in a geographic area whose annual income was between $40,000 and $44,999.
-Households with income of $45,000 To $49,999   |   The number of households in a geographic area whose annual income was between $45,000 and $49,999.
-Households with income of $50,000 To $59,999   |   The number of households in a geographic area whose annual income was between $50,000 and $59,999.
-Households with income of $60,000 To $74,999   |   The number of households in a geographic area whose annual income was between $60,000 and $74,999.
-Households with income of $75,000 To $99,999   |   The number of households in a geographic area whose annual income was between $75,000 and $99,999.
-Households with income of $100,000 To $124,999   |   The number of households in a geographic area whose annual income was between $100,000 and $124,999.
-Households with income of $125,000 To $149,999   |   The number of households in a geographic area whose annual income was between $125,000 and $149,999.
-Households with income of $150,000 To $199,999   |   The number of households in a geographic area whose annual income was between $150,000 and $1999,999.
-Households with income of $200,000 Or More   |   The number of households in a geographic area whose annual income was more than $200,000.
-Median Household Income in the past 12 Months   |   Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc.  It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans' (VA) payments, unemployment and/or worker's compensation, child support, and alimony.
-Population age 16 and over   |   The number of people in each geography who are age 16 or over.
-Population in Labor Force   |   The number of people in each geography who are either in the civilian labor force or are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
-Population in Civilian Labor Force   |   The number of civilians 16 years and over in each geography who can be classified as either employed or unemployed below.
-Employed Population   |   The number of civilians 16 years old and over in each geography who either (1) were at work, that is, those who did any work at all during the reference week as paid employees, worked in their own business or profession, worked on their own farm, or worked 15 hours or more as unpaid workers on a family farm or in a family business; or (2) were with a job but not at work, that is, those who did not work during the reference week but had jobs or businesses from which they were temporarily absent due to illness, bad weather, industrial dispute, vacation, or other personal reasons. Excluded from the employed are people whose only activity consisted of work around the house or unpaid volunteer work for religious, charitable, and similar organizations; also excluded are all institutionalized people and people on active duty in the United States Armed Forces.
-Unemployed Population   |   The number of civilians in each geography who are 16 years old and over and are classified as unemployed.
-Population in Armed Forces   |   The number of people in each geography who are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
-Population Not in Labor Force   |   The number of people in each geography who are 16 years old and over who are not classified as members of the labor force. This category consists mainly of students, homemakers, retired workers, seasonal workers interviewed in an off season who were not looking for work, institutionalized people, and people doing only incidental unpaid family work.
-Housing Units   |   A count of housing units in each geography.  A housing unit is a house, an apartment, a mobile home or trailer, a group of rooms, or a single room occupied as separate living quarters, or if vacant, intended for occupancy as separate living quarters.
-Vacant Housing Units   |   The count of vacant housing units in a geographic area. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Vacant Housing Units for Rent   |   The count of vacant housing units in a geographic area that are for rent. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Vacant Housing Units for Sale   |   The count of vacant housing units in a geographic area that are for sale. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Median Rent   |   The median contract rent within a geographic area. The contract rent is the monthly rent agreed to or contracted for, regardless of any furnishings, utilities, fees, meals, or services that may be included. For vacant units, it is the monthly rent asked for the rental unit at the time of interview.
-Percent of Household Income Spent on Rent   |   Within a geographic area, the median percentage of household income which was spent on gross rent.  Gross rent is the amount of the contract rent plus the estimated average monthly cost of utilities (electricity, gas, water, sewer etc.) and fuels (oil, coal, wood, etc.) if these are paid by the renter.  Household income is the sum of the income of all people 15 years and older living in the household.
-Owner-occupied Housing Units valued at $1,000,000 or more.   |   The count of owner occupied housing units in a geographic area that are valued at $1,000,000 or more.  Value is the respondent's estimate of how much the property (house and lot, mobile home and lot, or condominium unit) would sell for if it were for sale.
-Owner-occupied Housing Units with a Mortgage   |   The count of housing units within a geographic area that are mortagaged. Mortgage refers to all forms of debt where the property is pledged as security for repayment of the debt, including deeds of trust, trust deed, contracts to purchase, land contracts, junior mortgages, and home equity loans.

doc/license.md

@@ -1,18 +0,0 @@
-# License
-
-The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. For competency, please still review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
-
-Name  | Terms link 
--------|---------
-ACS   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
-TIGER   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
-Zillow Home Value Index | This data is "Aggregate Data", per the Zillow Terms of Use<br /><br />[http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm)
-Who's on First |  [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License)
-GeoNames  |  [http://www.geonames.org/](http://www.geonames.org/)
-GeoPlanet  |  [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/)
-Natural Earth  |  [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/)
-Quattroshapes  |  [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
-Zetashapes  |  [http://zetashapes.com/license](http://zetashapes.com/license)
-Spielman & Singleton  |  [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
-Instituto Nacional de Estadistica  |  [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
-

doc/measures_functions.md

@@ -1,22 +1,22 @@
 # Measures Functions
 
-[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
+[Data Observatory Measures](https://carto.com/docs/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
-There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf).
+There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
-You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 
-The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
+The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
 
 #### Arguments
 
 Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
-measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://cartodb.github.io/bigmetadata/index.html) (optional)
+measure_name | a human-readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -39,15 +39,15 @@ SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
 
 ## OBS_GetUSCensusMeasure(polygon geometry, measure_name text)
 
-The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetUSCensusMeasure``` function below.
+The ```OBS_GetUSCensusMeasure(polygon, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use the ```OBS_GetMeasure``` function below.
 
 #### Arguments
 
 Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
-measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
+measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -70,7 +70,7 @@ SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
 
 ## OBS_GetMeasure(point geometry, measure_id text)
 
-The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf)).
+The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
 #### Arguments
 
@@ -78,7 +78,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html). (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -109,7 +109,7 @@ Name |Description
 --- | ---
 polygon_geometry | a WGS84 polygon geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -132,11 +132,45 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
 
 #### Errors
 
-* If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
+* If an unrecognized normalization type is input, raises error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
+
+## OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
+
+The ```OBS_GetMeasureById(geom_ref, measure_id, boundary_id)``` function returns any Data Observatory measure that corresponds to the boundary in ```boundary_id``` that has a geometry reference of ```geom_ref```.
+
+#### Arguments
+
+Name |Description
+--- | ---
+geom_ref | a geometry reference (e.g., a US Census geoid)
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span (optional) | time span of interest (e.g., 2010 - 2014). If `NULL` is passed, the measure from the most recent data will be used.
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw measure associated with `geom_ref`
+
+#### Example
+
+Add a measure to an empty column based on county geoids in your table
+
+```SQL
+UPDATE tablename
+SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001', 'us.census.tiger.county')
+```
+
+#### Errors
+
+* Returns `NULL` if there is a mismatch between the geometry reference and the boundary id such as using the geoid of a county with the boundary of block groups
 
 ## OBS_GetCategory(point geometry, category_id text)
 
-The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail.
+The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/index.html) for more detail.
 
 #### Arguments
 
@@ -161,3 +195,296 @@ Add the Category to an empty column text column based on point locations in your
 UPDATE tablename
 SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
 ```
+
+## OBS_GetMeta(extent geometry, metadata json, max_timespan_rank, max_score_rank, target_geoms)
+
+The ```OBS_GetMeta(extent, metadata)``` function returns a completed Data
+Observatory metadata JSON Object for use in ```OBS_GetData(geomvals,
+metadata)``` or ```OBS_GetData(ids, metadata)```.  It is not possible to pass
+metadata to those functions if it is not processed by ```OBS_GetMeta(extent,
+metadata)``` first.
+
+`OBS_GetMeta` makes it possible to automatically select appropriate timespans
+and boundaries for the measurement you want.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+extent | A geometry of the extent of the input geometries
+metadata | A JSON array composed of metadata input objects.  Each indicates one desired measure for an output column, and optionally additional parameters about that column
+num_timespan_options | How many historical time periods to include.  Defaults to 1
+num_score_options | How many alternative boundary levels to include.  Defaults to 1
+target_geoms | Target number of geometries.  Boundaries with close to this many objects within `extent` will be ranked highest. 
+
+The schema of the metadata input objects are as follows:
+
+Metadata Input Key | Description
+--- | -----------
+numer_id | The identifier for the desired measurement.  If left blank, but a `geom_id` is specified, the column will return a geometry instead of a measurement.
+geom_id | Identifier for a desired geographic boundary level to use when calculating measures.  Will be automatically assigned if undefined.  If defined but `numer_id` is blank, then the column will return a geometry instead of a measurement.
+normalization | The desired normalization.  One of 'area', 'prenormalized', or 'denominated'.  'Area' will normalize the measure per square kilometer, 'prenormalized' will return the original value, and 'denominated' will normalize by a denominator.  Ignored if this metadata object specifies a geometry.
+denom_id | Identifier for a desired normalization column in case `normalization` is 'denominated'.  Will be automatically assigned if necessary.  Ignored if this metadata object specifies a geometry.
+numer_timespan | The desired timespan for the measurement.  Defaults to most recent timespan available if left unspecified.
+geom_timespan | The desired timespan for the geometry.  Defaults to timespan matching numer_timespan if left unspecified.
+target_area | Instead of aiming to have `target_geoms` in the area of the geometry passed as `extent`, fill this area.  Unit is square degrees WGS84.  Set this to `0` if you want to use the smallest source geometry for this element of metadata, for example if you're passing in points.
+target_geoms | Override global `target_geoms` for this element of metadata
+max_timespan_rank | Only include timespans of this recency (for example, `1` is only the most recent timespan). No limit by default
+max_score_rank | Only include boundaries of this relevance (for example, `1` is the most relevant boundary).  Is `1` by default
+
+#### Returns
+
+A JSON array composed of metadata output objects.
+
+Key | Description
+--- | -----------
+meta | A JSON array with completed metadata for the requested data, including all keys below
+
+The schema of the metadata output objects are as follows.  You should pass this
+array as-is to ```OBS_GetData```.  If you modify any values the function will
+fail.
+
+Metadata Output Key | Description
+--- | -----------
+suggested_name | A suggested column name for adding this to an existing table
+numer_id | Identifier for desired measurement
+numer_timespan | Timespan that will be used of the desired measurement
+numer_name | Human-readable name of desired measure
+numer_description | Long human-readable description of the desired measure
+numer_t_description | Further information about the source table
+numer_type | PostgreSQL/PostGIS type of desired measure
+numer_colname | Internal identifier for column name
+numer_tablename | Internal identifier for table
+numer_geomref_colname | Internal identifier for geomref column name
+denom_id | Identifier for desired normalization
+denom_timespan | Timespan that will be used of the desired normalization
+denom_name | Human-readable name of desired measure's normalization
+denom_description | Long human-readable description of the desired measure's normalization
+denom_t_description | Further information about the source table
+denom_type | PostgreSQL/PostGIS type of desired measure's normalization
+denom_colname | Internal identifier for normalization column name
+denom_tablename | Internal identifier for normalization table
+denom_geomref_colname | Internal identifier for normalization geomref column name
+geom_id | Identifier for desired boundary geometry
+geom_timespan | Timespan that will be used of the desired boundary geometry
+geom_name | Human-readable name of desired boundary geometry
+geom_description | Long human-readable description of the desired boundary geometry
+geom_t_description | Further information about the source table
+geom_type | PostgreSQL/PostGIS type of desired boundary geometry
+geom_colname | Internal identifier for boundary geometry column name
+geom_tablename | Internal identifier for boundary geometry table
+geom_geomref_colname | Internal identifier for boundary geometry ref column name
+timespan_rank | Ranking of this measurement by time, most recent is 1, second most recent 2, etc.
+score | The score of this measurement's boundary compared to the `extent` and `target_geoms` passed in.  Between 0 and 100.
+score_rank | The ranking of this measurement's boundary, highest ranked is 1, second is 2, etc.
+numer_aggregate | The aggregate type of the numerator, either `sum`, `average`, `median`, or blank
+denom_aggregate | The aggregate type of the denominator, either `sum`, `average`, `median`, or blank
+normalization | The sort of normalization that will be used for this measure, either `area`, `predenominated`, or `denominated`
+
+#### Examples
+
+Obtain metadata that can augment with one additional column of US population
+data, using a boundary relevant for the geometry provided and latest timespan.
+Limit to only the most recent column most relevant to the extent & density of
+input geometries in `tablename`.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with one additional column of US population
+data, using census tract boundaries.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with two additional columns, one for total
+population and one for male population.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+## OBS_GetData(geomvals array[geomval], metadata json)
+
+The ```OBS_GetData(geomvals, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every Geometry of
+the `geomval` element in the `geomvals` array. The metadata argument must be
+obtained from ```OBS_GetMeta(extent, metadata)```.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+geomvals | An array of `geomval` elements, which are obtained by casting together a `Geometry` and a `Numeric`.  This should be obtained by using `ARRAY_AGG((the_geom, cartodb_id)::geomval)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `geomvals`
+array corresponds to one row:
+
+Column | Type    | Description
+------ | ----    | -----------
+id     | Numeric | ID corresponding to the `val` component of an element of the input `geomvals` array
+data   | JSON    | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every geometry in a table, keyed by cartodb_id:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename)
+SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with a blank numeric column called `pop_density` with population
+densities:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename),
+data AS (
+  SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+Update a table with two measurements at once, population density and household
+density.  The table should already have a Numeric column `pop_density` and
+`household_density`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom),4326),
+    '[{"numer_id": "us.census.acs.B01003001"},{"numer_id": "us.census.acs.B11001001"}]',
+    1, 1, COUNT(*)
+) meta from tablename),
+data AS (
+  SELECT id,
+     data->0->>'value' AS pop_density,
+     data->1->>'value' AS household_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density,
+    household_density = data.household_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+## OBS_GetData(ids array[text], metadata json)
+
+The ```OBS_GetData(ids, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every id of
+the `ids` array. The metadata argument must be obtained from
+`OBS_GetMeta(extent, metadata)`.  When obtaining metadata, one must include
+the `geom_id` corresponding to the boundary that the `ids` refer to.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+ids | An array of `TEXT` elements.  This should be obtained by using `ARRAY_AGG(col_of_geom_refs)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+For this function to work, the `metadata` argument must include a `geom_id`
+that corresponds to the ids found in `col_of_geom_refs`.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `ids` array
+corresponds to one row:
+
+Column | Type | Description
+------ | ---- | -----------
+id     | Text | ID corresponding to an element of the input `ids` array
+data   | JSON | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every row of a table with FIPS code county IDs
+(USA).
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename)
+SELECT id AS fips, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((fips) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with population densities for every FIPS code county ID (USA).
+This table has a blank column called `pop_density` and fips codes stored in a
+column `fips`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename),
+data as (
+  SELECT id AS fips, (data->0->>'value') AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((fips) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE fips = data.id
+```

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '0.0.4'
+default_version = '1.6.0'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/autotest.py

@@ -1,100 +0,0 @@
-from nose.tools import assert_equal, assert_is_not_none
-from nose_parameterized import parameterized
-
-import os
-import re
-import requests
-
-HOSTNAME = os.environ['OBS_HOSTNAME']
-API_KEY = os.environ['OBS_API_KEY']
-META_HOSTNAME = os.environ.get('OBS_META_HOSTNAME', HOSTNAME)
-META_API_KEY = os.environ.get('OBS_META_API_KEY', API_KEY)
-USE_SCHEMA = 'OBS_USE_SCHEMA' in os.environ
-
-
-def query(q, is_meta=False, **options):
-    '''
-    Query the account.  Returned is the response, wrapped by the requests
-    library.
-    '''
-    url = 'https://{hostname}/api/v2/sql'.format(
-        hostname=META_HOSTNAME if is_meta else HOSTNAME)
-    params = options.copy()
-    params['q'] = re.sub(r'\s+', ' ', q)
-    params['api_key'] = META_API_KEY if is_meta else API_KEY
-    return requests.get(url, params=params)
-
-MEASURE_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'numeric'
-AND weight > 0
-''', is_meta=True).json()['rows']]
-
-CATEGORY_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'text'
-AND weight > 0
-''', is_meta=True).json()['rows']]
-
-BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
-WHERE type ILIKE 'geometry'
-AND weight > 0
-''', is_meta=True).json()['rows']]
-
-def default_point(column_id):
-    '''
-    Returns default test point for the column_id.
-    '''
-    if column_id == 'whosonfirst.wof_disputed_geom':
-        return 'CDB_LatLng(33.78, 76.57)'
-    elif column_id == 'whosonfirst.wof_marinearea_geom':
-        return 'CDB_LatLng(43.33, -68.47)'
-    elif column_id in ('us.census.tiger.school_district_elementary',
-                       'us.census.tiger.school_district_secondary',
-                       'us.census.tiger.school_district_elementary_clipped',
-                       'us.census.tiger.school_district_secondary_clipped'):
-        return 'CDB_LatLng(40.7025, -73.7067)'
-    elif column_id.startswith('es.ine'):
-        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
-    elif column_id.startswith('us.zillow'):
-        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
-    else:
-        return 'CDB_LatLng(40.7, -73.9)'
-
-
-@parameterized(MEASURE_COLUMNS)
-def test_measure_points(column_id):
-    resp = query('''
-SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
-
-@parameterized(CATEGORY_COLUMNS)
-def test_category_points(column_id):
-    resp = query('''
-SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
-
-@parameterized(BOUNDARY_COLUMNS)
-def test_boundary_points(column_id):
-    resp = query('''
-SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])

scripts/generate_fixtures.py

@@ -1,91 +1,374 @@
+import os
+import psycopg2
+import subprocess
+
+DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
+    user=os.environ.get('PGUSER', 'postgres'),
+    password=os.environ.get('PGPASSWORD', ''),
+    host=os.environ.get('PGHOST', 'localhost'),
+    port=os.environ.get('PGPORT', '5432'),
+    database=os.environ.get('PGDATABASE', 'postgres'),
+))
+CURSOR = DB_CONN.cursor()
+
+
+def query(q):
+    '''
+    Query the database.
+    '''
+    try:
+        CURSOR.execute(q)
+        return CURSOR
+    except:
+        DB_CONN.rollback()
+        raise
+
+
+def commit():
+    try:
+        DB_CONN.commit()
+    except:
+        DB_CONN.rollback()
+        raise
 
-from sqldumpr import Dumpr
 
 def get_tablename_query(column_id, boundary_id, timespan):
     """
         given a column_id, boundary-id (us.census.tiger.block_group), and
         timespan, give back the current table hash from the data observatory
     """
-    q = """
-        SELECT t.tablename, geoid_ct.colname colname
-        FROM obs_table t,
-             obs_column_table geoid_ct,
-             obs_column_table data_ct
-        WHERE
-             t.id = geoid_ct.table_id AND
-             t.id = data_ct.table_id AND
-             geoid_ct.column_id =
-        (SELECT source_id
-         FROM obs_column_to_column
-         WHERE target_id = '{boundary_id}'
-         AND reltype = 'geom_ref'
-        ) AND
-        data_ct.column_id = '{column_id}' AND
-        timespan = '{timespan}'
-    """.replace('\n','')
+    return """
+        SELECT numer_tablename, numer_geomref_colname, numer_tid,
+               geom_tablename, geom_geomref_colname, geom_tid
+        FROM observatory.obs_meta
+        WHERE numer_id = '{numer_id}' AND
+              geom_id = '{geom_id}' AND
+              numer_timespan = '{numer_timespan}'
+    """.format(numer_id=column_id,
+               geom_id=boundary_id,
+               numer_timespan=timespan)
 
-    return q.format(column_id=column_id,
-                    boundary_id=boundary_id,
-                    timespan=timespan)
 
-def select_star(tablename):
-    return "SELECT * FROM {}".format(tablename)
+METADATA_TABLES = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
+                   'obs_tag', 'obs_column_to_column', 'obs_dump_version', 'obs_meta',
+                   'obs_meta_numer', 'obs_meta_denom', 'obs_meta_geom',
+                   'obs_meta_timespan', 'obs_column_table_tile',
+                   'obs_column_table_tile_simple']
 
-cdb = Dumpr('observatory.cartodb.com','')
-
-metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag', 'obs_tag', 'obs_column_to_column']
-
-fixtures = [
-    ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
-    ('us.census.tiger.block_group', 'us.census.tiger.block_group', '2014'),
-    ('us.census.tiger.zcta5', 'us.census.tiger.zcta5', '2014'),
-    ('us.census.tiger.county', 'us.census.tiger.county', '2014'),
-    ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+FIXTURES = [
     ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001026_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B05001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08301010_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006009_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006011_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006015_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006017_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B09001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001005_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001007_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001008_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B17001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B17001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.spielman_singleton_segments.X55', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
-    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-03'),
-    ('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
-    ('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
-    ('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
+    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-06'),
+    ('whosonfirst.wof_country_name', 'whosonfirst.wof_country_geom', '2016'),
+    ('us.census.acs.B01003001', 'us.census.tiger.zcta5_clipped', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.block_group_clipped', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.census_tract_clipped', '2010 - 2014'),
+    ('us.census.tiger.fullname', 'us.census.tiger.pointlm_geom', '2016'),
+    ('us.census.tiger.fullname', 'us.census.tiger.prisecroads_geom', '2016'),
+    ('us.census.tiger.name', 'us.census.tiger.county', '2015'),
+    ('us.census.tiger.name', 'us.census.tiger.county_clipped', '2015'),
+    ('us.census.tiger.name', 'us.census.tiger.block_group', '2015'),
 ]
 
-unique_tables = set()
+OUTFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
+                            'src/pg/test/fixtures/load_fixtures.sql')
+DROPFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
+                             'src/pg/test/fixtures/drop_fixtures.sql')
 
-for f in fixtures:
-    column_id, boundary_id, timespan = f
-    tablename_query = get_tablename_query(*f)
-    resp = cdb.query(tablename_query).json()['rows'][0]
-    tablename = resp['tablename']
-    colname = resp['colname']
-    table_colname = (tablename, colname, boundary_id, )
-    if table_colname not in unique_tables:
-        print table_colname
-        unique_tables.add(table_colname)
+def dump(cols, tablename, where=''):
 
-print unique_tables
+    with open(DROPFILE_PATH, 'a') as dropfile:
+        dropfile.write('DROP TABLE IF EXISTS observatory.{tablename};\n'.format(
+            tablename=tablename,
+        ))
 
-with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
-    with open('src/pg/test/fixtures/drop_fixtures.sql', 'w') as dropfiles:
-        outfile.write('SET client_min_messages TO WARNING;\n\set ECHO none\n')
-        dropfiles.write('SET client_min_messages TO WARNING;\n\set ECHO none\n')
-        for tablename in metadata:
-            cdb.dump(select_star(tablename), tablename, outfile, schema='observatory')
-            dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
-            print tablename
+    subprocess.check_call('pg_dump -x --section=pre-data -t observatory.{tablename} '
+                          ' | sed "s:SET search_path.*::" '
+                          ' | sed "s:CREATE TABLE :CREATE TABLE observatory.:" '
+                          ' | sed "s:ALTER TABLE.*OWNER.*::" '
+                          ' | sed "s:SET idle_in_transaction_session_timeout.*::" '
+                          ' >> {outfile}'.format(
+                              tablename=tablename,
+                              outfile=OUTFILE_PATH,
+                          ), shell=True)
 
-        for tablename, colname, boundary_id in unique_tables:
-            if 'zcta5' in boundary_id:
-                where = '\'11%\''
-                compare = 'LIKE'
-            elif 'whosonfirst' in boundary_id:
-                where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
-                compare = 'IN'
-            else:
-                where = '\'36047%\''
-                compare = 'LIKE'
-            print ' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)])
-            cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
-                     tablename, outfile, schema='observatory')
-            dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
+    with open(OUTFILE_PATH, 'a') as outfile:
+        outfile.write('COPY observatory."{}" FROM stdin WITH CSV HEADER;\n'.format(tablename))
+
+    subprocess.check_call('''
+      psql -c "COPY (SELECT {cols} \
+      FROM observatory.{tablename} {where}) \
+      TO STDOUT WITH CSV HEADER" >> {outfile}'''.format(
+          cols=cols,
+          tablename=tablename,
+          where=where,
+          outfile=OUTFILE_PATH,
+      ), shell=True)
+
+    with open(OUTFILE_PATH, 'a') as outfile:
+        outfile.write('\\.\n\n')
+
+
+def main():
+    unique_tables = set()
+
+    for f in FIXTURES:
+        column_id, boundary_id, timespan = f
+        tablename_query = get_tablename_query(column_id, boundary_id, timespan)
+        resp = query(tablename_query).fetchone()
+        if resp:
+            numer_tablename, numer_colname, numer_table_id = resp[0:3]
+            geom_tablename, geom_colname, geom_table_id = resp[3:6]
+        else:
+            raise Exception("Could not find table for {}, {}, {}".format(
+                column_id, boundary_id, timespan))
+        numer = (numer_tablename, numer_colname, numer_table_id, )
+        geom = (geom_tablename, geom_colname, geom_table_id, )
+        if numer not in unique_tables:
+            print(numer)
+            unique_tables.add(numer)
+        if geom not in unique_tables:
+            print(geom)
+            unique_tables.add(geom)
+
+    print unique_tables
+
+    with open(OUTFILE_PATH, 'w') as outfile:
+        outfile.write('SET client_min_messages TO WARNING;\n\\set ECHO none\n')
+        outfile.write('CREATE SCHEMA IF NOT EXISTS observatory;\n\n')
+
+    with open(DROPFILE_PATH, 'w') as dropfile:
+        dropfile.write('SET client_min_messages TO WARNING;\n\\set ECHO none\n')
+
+    for tablename in METADATA_TABLES:
+        print(tablename)
+        if tablename == 'obs_meta':
+            where = "WHERE " + " OR ".join([
+                "(numer_id, geom_id, numer_timespan) = ('{}', '{}', '{}')".format(
+                    numer_id, geom_id, timespan)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_numer':
+            where = "WHERE " + " OR ".join([
+                "numer_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_denom':
+            where = "WHERE " + " OR ".join([
+                "denom_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_geom':
+            where = "WHERE " + " OR ".join([
+                "geom_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_timespan':
+            where = "WHERE " + " OR ".join([
+                "timespan_id = ('{}')".format(timespan)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_column':
+            where = "WHERE " + " OR ".join([
+                "id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_column_tag':
+            where = "WHERE " + " OR ".join([
+                "column_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename in ('obs_column_table', 'obs_column_table_tile',
+                           'obs_column_table_tile_simple'):
+            where = '''WHERE table_id IN ({table_ids}) AND
+              (column_id IN ({numer_ids}) OR column_id IN ({geom_ids}))
+                    '''.format(
+                        numer_ids=','.join(["'{}'".format(x) for x, _, _ in FIXTURES]),
+                        geom_ids=','.join(["'{}'".format(x) for _, x, _ in FIXTURES]),
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
+                    )
+        elif tablename == 'obs_column_to_column':
+            where = "WHERE " + " OR ".join([
+                "source_id IN ('{}', '{}') OR target_id IN ('{}', '{}')".format(
+                    numer_id, geom_id, numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_table':
+            where = 'WHERE timespan IN ({timespans}) ' \
+                    'OR id IN ({table_ids}) '.format(
+                        timespans=','.join(["'{}'".format(x) for _, _, x in FIXTURES]),
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
+                    )
+        else:
+            where = ''
+        dump('*', tablename, where)
+
+    for tablename, colname, table_id in unique_tables:
+        if 'zcta5' in table_id or 'zillow_zip' in table_id:
+            where = '\'11%\''
+            compare = 'LIKE'
+        elif 'pri_sec_roads' in table_id or 'point_landmark' in table_id:
+            dump('*', tablename, 'WHERE geom && ST_MakeEnvelope(-74,40.69,-73.9,40.72, 4326)')
+            continue
+        elif 'whosonfirst' in table_id:
+            where = "('85632785','85633051','85633111','85633147','85633253','85633267')"
+            compare = 'IN'
+        elif 'county' in table_id and 'tiger' in table_id:
+            where = "('48061', '36047')"
+            compare = 'IN'
+        else:
+            where = '\'36047%\''
+            compare = 'LIKE'
+        print ' '.join(['*', tablename, "WHERE {}::text {} {}".format(colname, compare, where)])
+        dump('*', tablename, "WHERE {}::text {} {}".format(colname, compare, where))
+
+if __name__ == '__main__':
+    main()

src/pg/Makefile

@@ -24,7 +24,7 @@ $(DATA): $(SOURCES_DATA)
 	$(SED) $(REPLACEMENTS) $(SOURCES_DATA_DIR)/*.sql > $@
 
 TEST_DIR = test
-REGRESS = $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql)))
+REGRESS = $(sort $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql))))
 REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)'
 
 PG_CONFIG = pg_config

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '0.0.4'
+default_version = '1.6.0'
 requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/42_observatory_exploration.sql

@@ -1,4 +1,4 @@
--- return a table that contains a string match based on input
+
 -- TODO: implement search for timespan
 
 CREATE OR REPLACE FUNCTION cdb_observatory._OBS_SearchTables(
@@ -114,9 +114,499 @@ BEGIN
       AND
         observatory.OBS_column.type = 'Geometry'
       AND
-        $1 && bounds::box2d
+        ST_Intersects($1, st_setsrid(observatory.obs_table.the_geom, 4326))
   $string$ || timespan_query
   USING geom;
   RETURN;
 END
 $$ LANGUAGE plpgsql;
+
+-- Functions the interface works from to identify available numerators,
+-- denominators, geometries, and timespans
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableNumerators(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  numer_id TEXT,
+  numer_name TEXT,
+  numer_description TEXT,
+  numer_weight NUMERIC,
+  numer_license TEXT,
+  numer_source TEXT,
+  numer_type TEXT,
+  numer_aggregate TEXT,
+  numer_extra JSONB,
+  numer_tags JSONB,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  denom_id := COALESCE(denom_id, '');
+  geom_id := COALESCE(geom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT numer_id::TEXT,
+           numer_name::TEXT,
+           numer_description::TEXT,
+           numer_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           numer_type numer_type,
+           numer_aggregate numer_aggregate,
+           numer_extra::JSONB numer_extra,
+           numer_tags numer_tags,
+    $1 = ANY(denoms) valid_denom,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_numer
+    WHERE %s (numer_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING denom_id, geom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetNumerators(
+  bounds GEOMETRY DEFAULT NULL,
+  section_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  subsection_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  other_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  ids TEXT[] DEFAULT ARRAY[]::TEXT[],
+  name TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT '',
+  geom_id TEXT DEFAULT '',
+  timespan TEXT DEFAULT ''
+) RETURNS TABLE (
+  numer_id TEXT,
+  numer_name TEXT,
+  numer_description TEXT,
+  numer_weight NUMERIC,
+  numer_license TEXT,
+  numer_source TEXT,
+  numer_type TEXT,
+  numer_aggregate TEXT,
+  numer_extra JSONB,
+  numer_tags JSONB,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  where_clause_elements TEXT[];
+  geom_clause TEXT;
+  where_clause TEXT;
+BEGIN
+  where_clause_elements := (ARRAY[])::TEXT[];
+  where_clause := '';
+
+  IF bounds IS NOT NULL THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$ST_Intersects(the_geom, '%s'::geometry)$data$, bounds));
+  END IF;
+  IF cardinality(section_tags) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_tags ?| '%s'$data$, section_tags));
+  END IF;
+  IF cardinality(subsection_tags) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_tags ?| '%s'$data$, subsection_tags));
+  END IF;
+  IF cardinality(other_tags) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_tags ?| '%s'$data$, other_tags));
+  END IF;
+  IF cardinality(ids) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_id IN (array_to_string('%s'::text[], ','))$data$, ids));
+  END IF;
+  IF name IS NOT NULL AND name != '' THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_name ilike '%%%s%%'$data$, name));
+  END IF;
+  IF cardinality(where_clause_elements) > 0 THEN
+    where_clause := format($clause$WHERE %s$clause$, array_to_string(where_clause_elements, ' AND '));
+  END IF;
+  RAISE DEBUG '%', array_to_string(where_clause_elements, ' AND ');
+
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT numer_id::TEXT,
+           numer_name::TEXT,
+           numer_description::TEXT,
+           numer_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           numer_type numer_type,
+           numer_aggregate numer_aggregate,
+           numer_extra::JSONB numer_extra,
+           numer_tags numer_tags,
+    $1 = ANY(denoms) valid_denom,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_numer
+    %s
+  $string$, where_clause)
+  USING denom_id, geom_id, timespan;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableDenominators(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  denom_id TEXT,
+  denom_name TEXT,
+  denom_description TEXT,
+  denom_weight NUMERIC,
+  denom_license TEXT,
+  denom_source TEXT,
+  denom_type TEXT,
+  denom_aggregate TEXT,
+  denom_extra JSONB,
+  denom_tags JSONB,
+  valid_numer BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  geom_id := COALESCE(geom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT denom_id::TEXT,
+           denom_name::TEXT,
+           denom_description::TEXT,
+           denom_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           denom_type::TEXT,
+           denom_aggregate::TEXT,
+           denom_extra::JSONB,
+           denom_tags::JSONB,
+    $1 = ANY(numers) valid_numer,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_denom
+    WHERE %s (denom_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING numer_id, geom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableGeometries(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  geom_id TEXT,
+  geom_name TEXT,
+  geom_description TEXT,
+  geom_weight NUMERIC,
+  geom_aggregate TEXT,
+  geom_license TEXT,
+  geom_source TEXT,
+  geom_type TEXT,
+  geom_extra JSONB,
+  geom_tags JSONB,
+  valid_numer BOOLEAN,
+  valid_denom BOOLEAN,
+  valid_timespan BOOLEAN,
+  score NUMERIC,
+  numtiles BIGINT,
+  notnull_percent NUMERIC,
+  numgeoms NUMERIC,
+  percentfill NUMERIC,
+  estnumgeoms NUMERIC,
+  meanmediansize NUMERIC
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  denom_id := COALESCE(denom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    WITH available_geoms AS (
+      SELECT geom_id::TEXT,
+             geom_name::TEXT,
+             geom_description::TEXT,
+             geom_weight::NUMERIC,
+             NULL::TEXT geom_aggregate,
+             NULL::TEXT license,
+             NULL::TEXT source,
+             geom_type::TEXT,
+             geom_extra::JSONB,
+             geom_tags::JSONB,
+             $1 = ANY(numers) valid_numer,
+             $2 = ANY(denoms) valid_denom,
+             CASE WHEN $3 IS NOT NULL AND $3 != '' THEN
+                -- Here we are looking for geometries with: a) geometry timespan or b) numerators linked to that geometries that fit in the
+                -- timespan passed. For example it look for geometries with timespan '2015 - 2015' or numerators linked to that geometry that has
+                -- '2015 - 2015' as one of the valid timespans.
+                -- If we pass a numerator_id, we filter by that numerator
+                CASE WHEN $1 IS NOT NULL AND $1 != '' THEN
+                  EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND onu.numer_id = $1 AND ($3 = ANY(onu.timespans) OR $3 IN (select(unnest(o.timespans)))))
+                ELSE
+                  EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND ($3 = ANY(onu.timespans) OR $3 IN (select(unnest(o.timespans)))))
+                END
+             ELSE
+              false
+             END as valid_timespan
+      FROM observatory.obs_meta_geom o
+      WHERE %s (geom_tags ?& $4 OR CARDINALITY($4) = 0)
+    ), scores AS (
+      SELECT * FROM cdb_observatory._OBS_GetGeometryScores($5,
+        (SELECT ARRAY_AGG(geom_id) FROM available_geoms)
+      )
+    ) SELECT DISTINCT ON (geom_id) available_geoms.*, score, numtiles, notnull_percent, numgeoms,
+             percentfill, estnumgeoms, meanmediansize
+      FROM available_geoms, scores
+      WHERE available_geoms.geom_id = scores.column_id
+  $string$, geom_clause)
+  USING numer_id, denom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableTimespans(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL
+) RETURNS TABLE (
+  timespan_id TEXT,
+  timespan_name TEXT,
+  timespan_description TEXT,
+  timespan_weight NUMERIC,
+  timespan_aggregate TEXT,
+  timespan_license TEXT,
+  timespan_source TEXT,
+  timespan_type TEXT,
+  timespan_extra JSONB,
+  timespan_tags JSONB,
+  valid_numer BOOLEAN,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  denom_id := COALESCE(denom_id, '');
+  geom_id := COALESCE(geom_id, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT timespan_id::TEXT,
+           timespan_name::TEXT,
+           timespan_description::TEXT,
+           timespan_weight::NUMERIC,
+           NULL::TEXT timespan_aggregate,
+           NULL::TEXT timespan_license,
+           NULL::TEXT timespan_source,
+           NULL::TEXT timespan_type,
+           NULL::JSONB timespan_extra,
+           NULL::JSONB timespan_tags,
+    $1 = ANY(numers) valid_numer,
+    $2 = ANY(denoms) valid_denom,
+    $3 = ANY(geoms) valid_geom_id
+    FROM observatory.obs_meta_timespan
+    WHERE %s (timespan_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING numer_id, denom_id, geom_id, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+
+-- Function below should replace SQL in
+-- https://github.com/CartoDB/cartodb/blob/ab465cb2918c917940e955963b0cd8a050c06600/lib/assets/javascripts/cartodb3/editor/layers/layer-content-views/analyses/data-observatory-metadata.js
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_LegacyBuilderMetadata(
+  aggregate_type TEXT DEFAULT NULL
+)
+RETURNS TABLE (
+  name TEXT,
+  subsection JSONB
+) AS $$
+DECLARE
+  aggregate_condition TEXT DEFAULT '';
+BEGIN
+  IF LOWER(aggregate_type) ILIKE 'sum' THEN
+    aggregate_condition := ' AND numer_aggregate IN (''sum'', ''median'', ''average'') ';
+  ELSIF aggregate_type IS NOT NULL THEN
+    aggregate_condition := format(' AND numer_aggregate ILIKE %L ', aggregate_type);
+  END IF;
+  RETURN QUERY
+  EXECUTE format($string$
+    WITH expanded AS (
+        SELECT JSONB_Build_Object('id', numer_id, 'name', numer_name) "column",
+               SUBSTR((sections).key, 9) section_id, (sections).value section_name,
+               SUBSTR((subsections).key, 12) subsection_id, (subsections).value subsection_name
+        FROM (
+          SELECT numer_id, numer_name,
+                 jsonb_each_text(numer_tags) as sections,
+                 jsonb_each_text as subsections
+          FROM (SELECT numer_id, numer_name, numer_tags,
+                       jsonb_each_text(numer_tags)
+                FROM cdb_observatory.obs_getavailablenumerators()
+                WHERE numer_weight > 0 %s
+               ) foo
+        ) bar
+        WHERE (sections).key LIKE 'section/%%'
+          AND (subsections).key LIKE 'subsection/%%'
+      ), grouped_by_subsections AS (
+        SELECT JSONB_Agg(JSONB_Build_Object('f1', "column")) AS columns,
+               section_id, section_name, subsection_id, subsection_name
+        FROM expanded
+        GROUP BY section_id, section_name, subsection_id, subsection_name
+      )
+    SELECT section_name as name, JSONB_Agg(
+      JSONB_Build_Object(
+        'f1', JSONB_Build_Object(
+          'name', subsection_name,
+          'id', subsection_id,
+          'columns', columns
+        )
+      )
+    ) as subsection
+    FROM grouped_by_subsections
+    GROUP BY section_name
+  $string$, aggregate_condition);
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryScores(
+  bounds Geometry(Geometry, 4326) DEFAULT NULL,
+  filter_geom_ids TEXT[] DEFAULT NULL,
+  desired_num_geoms INTEGER DEFAULT NULL,
+  desired_area NUMERIC DEFAULT NULL
+) RETURNS TABLE (
+  score NUMERIC,
+  numtiles BIGINT,
+  table_id TEXT,
+  column_id TEXT,
+  notnull_percent NUMERIC,
+  numgeoms NUMERIC,
+  percentfill NUMERIC,
+  estnumgeoms NUMERIC,
+  meanmediansize NUMERIC
+) AS $$
+DECLARE
+  num_geoms_multiplier Numeric;
+BEGIN
+  IF desired_num_geoms IS NULL THEN
+    desired_num_geoms := 3000;
+  END IF;
+  filter_geom_ids := COALESCE(filter_geom_ids, (ARRAY[])::TEXT[]);
+  -- Very complex geometries simply fail.  For a boundary check, we can
+  -- comfortably get away with the simplicity of an envelope
+  IF ST_Npoints(bounds) > 10000 THEN
+    bounds := ST_Envelope(bounds);
+  END IF;
+  IF desired_area IS NULL THEN
+    desired_area := ST_Area(bounds);
+  END IF;
+
+  -- In case of points, desired_area will be 0.  We still want an accurate
+  -- estimate of numgeoms in that case.
+  IF desired_area = 0 THEN
+    num_geoms_multiplier := 1;
+  ELSE
+    num_geoms_multiplier := Coalesce(desired_area / Nullif(ST_Area(bounds), 0), 1);
+  END IF;
+
+  RETURN QUERY
+  EXECUTE $string$
+    WITH clipped_geom AS (
+      SELECT column_id, table_id
+        , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True) -- -20
+               ELSE tile END clipped_tile
+        , tile
+      FROM observatory.obs_column_table_tile_simple
+      WHERE ($1 IS NULL OR ST_Intersects($1, tile))
+        AND (column_id = ANY($2) OR cardinality($2) = 0)
+    ), clipped_geom_countagg AS (
+      SELECT column_id, table_id
+        , BOOL_AND(ST_BandIsNoData(clipped_tile, 1)) nodata
+      FROM clipped_geom
+      GROUP BY column_id, table_id
+    ), clipped_geom_reagg AS (
+      SELECT COUNT(*)::BIGINT cnt, a.column_id, a.table_id,
+             cdb_observatory.FIRST(nodata) first_nodata,
+             cdb_observatory.FIRST(tile) first_tile,
+             (ST_SummaryStatsAgg(clipped_tile, 1, False)).sum::Numeric sum_geoms, -- ND
+             (ST_SummaryStatsAgg(clipped_tile, 2, False)).mean::Numeric / 255 mean_fill --ND
+      FROM clipped_geom_countagg a, clipped_geom b
+      WHERE a.table_id = b.table_id
+        AND a.column_id = b.column_id
+      GROUP BY a.column_id, a.table_id
+    ), final AS (
+      SELECT
+        cnt, table_id, column_id
+        , NULL::Numeric AS notnull_percent
+        , (CASE WHEN first_nodata IS FALSE
+                THEN sum_geoms
+                ELSE COALESCE(ST_Value(first_tile, 1, ST_PointOnSurface($1)), 0)
+                  * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(first_tile, 0, 0)))
+          END)::Numeric * $4
+        AS numgeoms
+        , (CASE WHEN first_nodata IS FALSE
+                THEN mean_fill
+                ELSE COALESCE(ST_Value(first_tile, 2, ST_PointOnSurface($1))::Numeric / 255, 0) -- -2
+          END)::Numeric
+        AS percentfill
+        , null::numeric estnumgeoms
+        , null::numeric meanmediansize
+      FROM clipped_geom_reagg
+    ) SELECT
+      ((100.0 / (1+abs(log(0.0001 + $3) - log(0.0001 + numgeoms::Numeric)))) * percentfill)::Numeric
+      AS score, *
+      FROM final
+  $string$ USING bounds, filter_geom_ids, desired_num_geoms, num_geoms_multiplier;
+  RETURN;
+END
+$$ LANGUAGE plpgsql IMMUTABLE;

src/pg/sql/44_observatory_geometries.sql

@@ -40,44 +40,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
   END IF;
 
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                                           time_span) As x(target_tables,
-                                                           timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    -- TODO: modify for only one table returned instead of arbitrarily choosing
-    --       one with LIMIT 1 (could be conflict between clipped vs non-clipped
-    --       boundaries in the metadata tables)
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN NULL::geometry;
-  END IF;
-
-  RAISE NOTICE 'target_table: %', target_table;
-
   -- return the first boundary in intersections
-  EXECUTE format(
-    'SELECT the_geom
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', target_table)
-  INTO boundary
-  USING geom;
+  EXECUTE $query$
+    SELECT * FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2, $3) LIMIT 1
+  $query$ INTO boundary
+  USING geom, boundary_id, time_span;
 
   RETURN boundary;
 
@@ -111,67 +78,17 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryId(
 RETURNS text
 AS $$
 DECLARE
-  output_id text;
-  target_table text;
-  geoid_colname text;
+  result TEXT;
 BEGIN
 
-  -- If not point, raise error
-  IF ST_GeometryType(geom) != 'ST_Point'
-  THEN
-    RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
-  END IF;
-
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                                timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
-    RETURN NULL::text;
-  END IF;
-
-  EXECUTE
-    format('SELECT ct.colname
-              FROM observatory.obs_column_to_column c2c,
-                   observatory.obs_column_table ct,
-                   observatory.obs_table t
-             WHERE c2c.reltype = ''geom_ref''
-               AND ct.column_id = c2c.source_id
-               AND ct.table_id = t.id
-               AND t.tablename = %L'
-   , target_table)
-  INTO geoid_colname;
-
-  RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
-
-  -- return geometry id column value
-  EXECUTE format(
-    'SELECT %I::text
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', geoid_colname, target_table)
-  INTO output_id
-  USING geom;
-
-  RETURN output_id;
+  EXECUTE $query$
+    SELECT geom_refs FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+      $1, $2, $3) LIMIT 1
+  $query$
+  INTO result
+  USING geom, boundary_id, time_span;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -203,35 +120,21 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryById(
 RETURNS geometry(geometry, 4326)
 AS $$
 DECLARE
-  boundary geometry(geometry, 4326);
-  target_table text;
-  geoid_colname text;
-  geom_colname text;
+  result GEOMETRY;
 BEGIN
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  RAISE NOTICE '%', target_table;
-
-  IF target_table IS NULL
-  THEN
-    RAISE NOTICE 'No geometries found';
-    RETURN NULL::geometry;
-  END IF;
-
-  -- retrieve boundary
-  EXECUTE
-    format(
-    'SELECT %I
-     FROM observatory.%I
-     WHERE %I = $1
-     LIMIT 1', geom_colname, target_table, geoid_colname)
-  INTO boundary
-  USING geometry_id;
-
-  RETURN boundary;
+  EXECUTE $query$
+    SELECT (data->0->>'value')::Geometry
+    FROM cdb_observatory.OBS_GetData(
+      ARRAY[$1],
+      cdb_observatory.OBS_GetMeta(
+        ST_MakeEnvelope(-180, -90, 180, 90, 4326),
+        ('[{"geom_id": "' || $2 || '"}]')::JSON))
+  $query$
+  INTO result
+  USING geometry_id, boundary_id;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -244,49 +147,41 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetBoundariesByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
-RETURNS TABLE(the_geom geometry, geom_refs text)
-AS $$
+  overlap_type text DEFAULT NULL)
+RETURNS TABLE (
+  the_geom geometry,
+  geom_refs text
+) AS $$
 DECLARE
-  boundary geometry(Geometry, 4326);
-  geom_colname text;
-  geoid_colname text;
-  target_table text;
+  meta JSON;
 BEGIN
-
+  overlap_type := COALESCE(overlap_type, 'intersects');
   -- check inputs
   IF lower(overlap_type) NOT IN ('contains', 'intersects', 'within')
   THEN
     -- recognized overlap type (map to ST_Contains, ST_Intersects, and ST_Within)
     RAISE EXCEPTION 'Overlap type ''%'' is not an accepted type (choose intersects, within, or contains)', overlap_type;
-  ELSIF ST_GeometryType(geom) NOT IN ('ST_Polygon', 'ST_MultiPolygon')
-  THEN
-      RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  -- TODO: add timespan in search
-  -- TODO: add overlap info in search
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
+  EXECUTE $query$
+    SELECT cdb_observatory.OBS_GetMeta($1, JSON_Build_Array(JSON_Build_Object(
+                'geom_id', $2, 'geom_timespan', $3)))
+  $query$
+  INTO meta
+  USING geom, boundary_id, time_span;
 
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
+  IF meta->0->>'geom_id' IS NULL THEN
+    RETURN QUERY EXECUTE 'SELECT NULL::Geometry, NULL::Text LIMIT 0';
     RETURN;
   END IF;
 
-  RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
-  EXECUTE format(
-    'SELECT %I, %I::text
-     FROM observatory.%I
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
+  RETURN QUERY EXECUTE $query$
+    SELECT (data->0->>'value')::Geometry the_geom, data->0->>'geomref' geom_refs
+    FROM cdb_observatory.OBS_GetData(
+      ARRAY[($1, 1)::geomval], $2, False
+    )
+  $query$ USING geom, meta;
   RETURN;
 
 END;
@@ -318,7 +213,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -364,7 +259,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByPointAndRadius(
   radius numeric, -- radius in meters
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -382,7 +277,8 @@ BEGIN
                FROM cdb_observatory._OBS_GetBoundariesByGeometry(
                         circle_boundary,
                         boundary_id,
-                        time_span);
+                        time_span,
+                        overlap_type);
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
@@ -394,7 +290,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetPointsByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -403,6 +299,7 @@ DECLARE
   geoid_colname text;
   target_table text;
 BEGIN
+  overlap_type := COALESCE(overlap_type, 'intersects');
 
   IF lower(overlap_type) NOT IN ('contains', 'within', 'intersects')
   THEN
@@ -412,27 +309,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
-    RETURN;
-  END IF;
-
-  RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
-  EXECUTE format(
-    'SELECT ST_PointOnSurface(%I) As %s, %I::text
-     FROM observatory.%I
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
+  RETURN QUERY EXECUTE $query$
+    SELECT ST_PointOnSurface(the_geom), geom_refs
+    FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2)
+  $query$ USING geom, boundary_id;
   RETURN;
 
 END;
@@ -464,7 +345,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -509,7 +390,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByPointAndRadius(
   radius numeric, -- radius in meters
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -532,44 +413,3 @@ BEGIN
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
-
-
--- _OBS_GetGeometryMetadata()
--- TODO: add timespan in search
--- TODO: add choice of clipped versus not clipped
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryMetadata(boundary_id text)
-RETURNS table(geoid_colname text, target_table text, geom_colname text)
-AS $$
-BEGIN
-
-  RETURN QUERY
-  EXECUTE
-  format($string$
-    SELECT geoid_ct.colname::text As geoid_colname,
-           tablename::text,
-           geom_ct.colname::text As geom_colname
-    FROM observatory.obs_column_table As geoid_ct,
-         observatory.obs_table As geom_t,
-         observatory.obs_column_table As geom_ct,
-         observatory.obs_column As geom_c
-    WHERE geoid_ct.column_id
-       IN (
-         SELECT source_id
-         FROM observatory.obs_column_to_column
-         WHERE reltype = 'geom_ref'
-           AND target_id = '%s'
-         )
-      AND geoid_ct.table_id = geom_t.id AND
-          geom_t.id = geom_ct.table_id AND
-          geom_ct.column_id = geom_c.id AND
-          geom_c.type ILIKE 'geometry' AND
-          geom_c.id = '%s'
-    $string$, boundary_id, boundary_id);
-  RETURN;
-    --  AND geom_t.timespan = '%s' <-- put in requested year
-    -- TODO: filter by clipped vs. not so appropriate tablename are unique
-    --       so the limit 1 can be removed
-    RETURN;
-
-END;
-$$ LANGUAGE plpgsql;

src/pg/test/expected/01_install_test.out

@@ -1,6 +1,4 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION plpythonu;
-CREATE EXTENSION cartodb;
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/expected/40_observatory_utility_test.out

@@ -9,21 +9,19 @@ t
 _obs_geomtable_with_null_response
 t
 (1 row)
-test_get_obs_column_with_geoid_and_census_1|test_get_obs_column_with_geoid_and_census_2
-t|t
-(1 row)
-obs_getcolumndata_missing_measure
-t
-(1 row)
 _obs_buildsnapshotquery_test_1
 t
 (1 row)
 _obs_buildsnapshotquery_test_2
 t
 (1 row)
-_obs_getrelatedcolumn_test
-t
-(1 row)
 _obs_standardizemeasurename_test
 t
 (1 row)
+obs_dumpversion_notnull
+t
+(1 row)
+ERROR:  Error performing intersection: TopologyException: found non-noded intersection between LINESTRING (-97.1968 25.9574, -97.1971 25.9576) and LINESTRING (-97.197 25.9575, -97.1972 25.9576) at -97.19699802694231 25.957551976080605
+complex_safe_intersection_works
+t
+(1 row)

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -1,39 +1,14 @@
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
+\pset format unaligned
 \set ECHO none
 obs_getdemographicsnapshot_test_no_returns
 t
 (1 row)
-obs_get_median_income_at_test_point
-t
-(1 row)
-obs_get_median_income_at_null_island
-t
-(1 row)
-obs_getpoints_for_test_point_value|obs_getpoints_for_test_point_name|obs_getpoints_for_test_point_tablename|obs_getpoints_for_test_point_aggregate|obs_getpoints_for_test_point_type|obs_getpoints_for_test_point_description
-t|t|t|t|t|t
-(1 row)
-obs_getpoints_for_null_island
-t
-(1 row)
-obs_getpolygons_for_test_point
-t
-(1 row)
-obs_getpolygons_for_null_island
-t
-(1 row)
 test_point_segmentation
 t
 (1 row)
 null_island_segmentation
 t
 (1 row)
-getcategories_at_test_point_1
-t
-(1 row)
-getcategories_at_null_island
-t
-(1 row)
 obs_getmeasure_zhvi_point_test
 t
 (1 row)
@@ -43,36 +18,289 @@ t
 obs_getmeasure_total_pop_point_test
 t
 (1 row)
+obs_getmeasure_total_pop_point_null_normalization_test
+t
+(1 row)
+obs_getmeasure_total_pop_point_area_test
+t
+(1 row)
 obs_getmeasure_total_pop_polygon_test
 t
 (1 row)
+obs_getmeasure_total_pop_polygon_null_normalization_test
+t
+(1 row)
+obs_getmeasure_total_pop_polygon_area_test
+t
+(1 row)
 obs_getmeasure_total_male_point_denominator
 t
 (1 row)
 obs_getmeasure_total_male_poly_denominator
 t
 (1 row)
+obs_getmeasure_bad_geometry
+t
+(1 row)
+obs_getmeasure_null_geometry
+t
+(1 row)
+obs_getmeasure_out_of_bounds_geometry
+t
+(1 row)
+obs_getmeasure_estimate_for_blank_aggregate
+t
+(1 row)
+obs_getmeasure_per_capita_income_average
+t
+(1 row)
+obs_getmeasure_median_capita_income_average
+t
+(1 row)
 obs_getcategory_point
 t
 (1 row)
 obs_getcategory_polygon
 t
 (1 row)
+obs_getcategory_null
+t
+(1 row)
 obs_getpopulation
 t
 (1 row)
 obs_getpopulation_polygon_test
 t
 (1 row)
+obs_getpopulation_polygon_null_test
+t
+(1 row)
+obs_getpopulation_polygon_null_geom_test
+t
+(1 row)
 obs_getuscensusmeasure_point_male_pop
 t
 (1 row)
 obs_getuscensusmeasure
 t
 (1 row)
+obs_getuscensusmeasure_null
+t
+(1 row)
+obs_getuscensusmeasure_null_geom
+t
+(1 row)
 obs_getuscensuscategory_point
 t
 (1 row)
 obs_getuscensuscategory_polygon
 t
 (1 row)
+obs_getuscensuscategory_null
+t
+(1 row)
+obs_getmeasurebyid_cartodb_census_tract
+t
+(1 row)
+obs_getmeasurebyid_null_boundary_null_timespan
+t
+(1 row)
+obs_getmeasurebyid_cartodb_block_group
+t
+(1 row)
+obs_getmeasurebyid_nulls
+t
+(1 row)
+obs_getmeasurebyid_null_id
+t
+(1 row)
+obs_getmeta_null_null_is_null
+t
+(1 row)
+obs_getmeta_null_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_null_is_null
+t
+(1 row)
+obs_getmeta_nullisland_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_us_measure_is_null
+t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization|id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+obs_getmeta_conflicting_metadata
+t
+(1 row)
+obs_getmeta_suggested_name
+t
+(1 row)
+obs_getmeta_suggested_name_implicit_area
+t
+(1 row)
+obs_getmeta_suggested_name_area
+t
+(1 row)
+obs_getmeta_suggested_name_denom
+t
+(1 row)
+obs_getdata_geomval_empty_null
+t
+(1 row)
+obs_getdata_text_empty_null
+t
+(1 row)
+obs_getdata_geomval_empty_one_measure
+t
+(1 row)
+id|data_point_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_prenormalized|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_prenormalized|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_one_null|data_polygon_measure_two_null
+t|t|t
+(1 row)
+id|data_polygon_measure_one_null|data_polygon_measure_two_null
+t|t|t
+(1 row)
+id|data_polygon_measure_one_predenom|data_polygon_measure_two_predenom
+t|t|t
+(1 row)
+id|data_polygon_measure_one_area|data_polygon_measure_two_area
+t|t|t
+(1 row)
+id|data_polygon_measure_tract|data_polygon_measure_bg
+t|t|t
+(1 row)
+id|data_point_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|valcol
+t|t|t
+(1 row)
+id|correct_num_geoms
+t|t
+(1 row)
+id|correct_num_geoms|correct_pop
+t|t|t
+(1 row)
+id|correct_num_geoms|correct_pop|correct_bg_names
+t|t|t|t
+(1 row)
+id|correct_num_points
+t|t
+(1 row)
+id|correct_num_points|pointgeom_names
+t|t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_predenom
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null|obs_getdata_by_id_two_measure_null
+t|t|t
+(1 row)
+id|obs_getdata_by_id_categorical
+t|t
+(1 row)
+id|obs_getdata_by_id_geometry
+t|t
+(1 row)
+obs_getdata_api_geomvals_no_args
+t
+(1 row)
+ary_type|obs_getdata_api_geomvals_args_numer_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomvals_args_string_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomrefs_args_numer_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomrefs_args_string_return
+t|t
+(1 row)
+setseed
+
+(1 row)
+bg_sample|bg_max_error|bg_avg_error|bg_min_error
+1|t|t|t
+2|t|t|t
+3|t|t|t
+5|t|t|t
+10|t|t|t
+25|t|t|t
+50|t|t|t
+100|t|t|t
+2085|t|t|t
+(9 rows)
+tract_sample|tract_max_error|tract_avg_error|tract_min_error
+1|t|t|t
+2|t|t|t
+3|t|t|t
+5|t|t|t
+10|t|t|t
+25|t|t|t
+50|t|t|t
+100|t|t|t
+761|t|t|t
+(9 rows)
+no_bg_point_error
+t
+(1 row)

src/pg/test/expected/42_observatory_exploration_test.out

@@ -1,5 +1,4 @@
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
+\pset format unaligned
 \set ECHO none
 _obs_searchtables_tables_match|_obs_searchtables_timespan_matches
 t|t
@@ -13,3 +12,279 @@ t
 _obs_getavailableboundariesexist
 t
 (1 row)
+_obs_getavailablenumerators_usa_pop_in_all
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getavailablenumerators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getavailablenumerators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getavailablenumerators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getavailablenumerators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getavailablenumerators_zillow_at_zcta5
+t
+(1 row)
+_obs_getavailablenumerators_no_zillow_at_block_group
+t
+(1 row)
+_obs_getavailablenumerators_total_pop_2010_2014
+t
+(1 row)
+_obs_getavailablenumerators_no_total_pop_1996
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_all
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getnumerators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getnumerators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getnumerators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getnumerators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getnumerators_zillow_at_zcta5
+t
+(1 row)
+_obs_getnumerators_no_zillow_at_block_group
+t
+(1 row)
+_obs_getnumerators_total_pop_2010_2014
+t
+(1 row)
+_obs_getnumerators_no_total_pop_1996
+t
+(1 row)
+_obs_getnumerators_total_pop_by_name
+t
+(1 row)
+_obs_getnumerators_total_pop_by_section
+t
+(1 row)
+_obs_getnumerators_total_pop_not_in_canada
+t
+(1 row)
+_obs_getnumerators_total_pop_by_subsection
+t
+(1 row)
+_obs_getnumerators_total_pop_not_in_employment_subsection
+t
+(1 row)
+_obs_getnumerators_total_pop_by_id
+t
+(1 row)
+_obs_getnumerators_total_pop_not_with_other_id
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_all
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getavailabledenominators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getavailabledenominators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getavailabledenominators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getavailabledenominators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getavailabledenominators_at_zcta5
+t
+(1 row)
+_obs_getavailabledenominators_none_spanish_geom
+t
+(1 row)
+_obs_getavailabledenominators_total_pop_2010_2014
+t
+(1 row)
+_obs_getavailabledenominators_no_total_pop_1996
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_all
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_nyc_point
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_usa_extents
+t
+(1 row)
+_obs_getavailablegeometries_no_usa_bg_not_in_zero_point
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_boundary_subsection
+t
+(1 row)
+_obs_getavailablegeometries_no_bg_in_uk_section
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_not_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_denom_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_bg_2015
+t
+(1 row)
+_obs_getavailablegeometries_bg_not_1996
+t
+(1 row)
+_obs_getavailablegeometries_has_boundary_tag
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_all
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_nyc_point
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_usa_extents
+t
+(1 row)
+_obs_getavailabletimespans_no_usa_bg_not_in_zero_point
+t
+(1 row)
+_obs_getavailabletimespans_total_pop_in_2010_2014
+t
+(1 row)
+_obs_getavailabletimespans_foobarbaz_not_in_2010_2014
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_denom_in_2010_2014
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014
+t
+(1 row)
+_obs_geometryscores_500m_buffer
+t
+(1 row)
+_obs_geometryscores_5km_buffer
+t
+(1 row)
+_obs_geometryscores_50km_buffer
+t
+(1 row)
+_obs_geometryscores_500km_buffer
+t
+(1 row)
+_obs_geometryscores_2500km_buffer
+t
+(1 row)
+column_id|_obs_geometryscores_numgeoms_500m_buffer
+us.census.tiger.block_group|2
+us.census.tiger.census_tract|1
+us.census.tiger.zcta5|0
+us.census.tiger.county|0
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_5km_buffer
+us.census.tiger.block_group|244
+us.census.tiger.census_tract|78
+us.census.tiger.zcta5|9
+us.census.tiger.county|0
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_50km_buffer
+us.census.tiger.block_group|10817
+us.census.tiger.census_tract|3396
+us.census.tiger.zcta5|484
+us.census.tiger.county|11
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_500km_buffer
+us.census.tiger.block_group|48567
+us.census.tiger.census_tract|15823
+us.census.tiger.zcta5|6466
+us.census.tiger.county|295
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_2500km_buffer
+us.census.tiger.block_group|165852
+us.census.tiger.census_tract|55283
+us.census.tiger.zcta5|27046
+us.census.tiger.county|2551
+(4 rows)
+_obs_geometryscores_500km_buffer_50_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_500_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_2500_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_25000_geoms
+t
+(1 row)
+testarea_uses_tract
+t
+(1 row)
+points_use_bg
+t
+(1 row)
+_total_pop_in_legacy_builder_metadata
+t
+(1 row)
+_median_income_in_legacy_builder_metadata
+t
+(1 row)
+_gini_in_legacy_builder_metadata
+t
+(1 row)
+_total_pop_in_legacy_builder_metadata_sums
+t
+(1 row)
+_median_income_in_legacy_builder_metadata_sums
+t
+(1 row)
+_gini_not_in_legacy_builder_metadata_sums
+t
+(1 row)
+_no_dupe_subsections_in_legacy_builder_metadata
+t
+(1 row)

src/pg/test/expected/44_observatory_geometries_test.out

@@ -1,7 +1,4 @@
 \pset format unaligned
-\set ECHO all
-\i test/fixtures/load_fixtures.sql
-SET client_min_messages TO WARNING;
 \set ECHO none
 obs_getboundary_cartodb_census_tract
 t
@@ -45,6 +42,12 @@ t
 obs_getboundarybyid_boundary_id_mismatch_geom_id
 t
 (1 row)
+_obs_getboundariesbygeometry_roads_around_cartodb
+t
+(1 row)
+_obs_getboundariesbygeometry_points_around_cartodb
+t
+(1 row)
 _obs_getboundariesbygeometry_tracts_around_cartodb
 t
 (1 row)
@@ -90,9 +93,3 @@ t
 obs_getpointsbypointandradius_around_null_island
 t
 (1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)

src/pg/test/fixtures/drop_fixtures.sql

@@ -6,16 +6,29 @@ DROP TABLE IF EXISTS observatory.obs_column;
 DROP TABLE IF EXISTS observatory.obs_column_tag;
 DROP TABLE IF EXISTS observatory.obs_tag;
 DROP TABLE IF EXISTS observatory.obs_column_to_column;
+DROP TABLE IF EXISTS observatory.obs_dump_version;
+DROP TABLE IF EXISTS observatory.obs_meta;
+DROP TABLE IF EXISTS observatory.obs_meta_numer;
+DROP TABLE IF EXISTS observatory.obs_meta_denom;
+DROP TABLE IF EXISTS observatory.obs_meta_geom;
+DROP TABLE IF EXISTS observatory.obs_meta_timespan;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile_simple;
+DROP TABLE IF EXISTS observatory.obs_78fb6c1d6ff6505225175922c2c389ce48d7632c;
 DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
 DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
+DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
+DROP TABLE IF EXISTS observatory.obs_c4411eba732408d47d73281772dbf03d60645dec;
 DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
 DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
-DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
-DROP TABLE IF EXISTS observatory.obs_8764a6b439a4f8714f54d4b3a157bc5e36519066;
-DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
-DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
-DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
+DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
 DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
+DROP TABLE IF EXISTS observatory.obs_0310c639744a2014bb1af82709228f05b59e7d3d;
+DROP TABLE IF EXISTS observatory.obs_87a814e485deabe3b12545a537f693d16ca702c2;
+DROP TABLE IF EXISTS observatory.obs_e32f8e59c7c8861ee5ee4029b3ace2af9a5c9caf;
+DROP TABLE IF EXISTS observatory.obs_23cb5063486bd7cf36f17e89e5e65cd31b331f6e;
+DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
+DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
+DROP TABLE IF EXISTS observatory.obs_8e30e6b3792430b410ba5b9e49cdc6a0d404d48f;
+DROP TABLE IF EXISTS observatory.obs_08025e1287e3af2b5de571d06562ba8d3bdb48e9;
 DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
-DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
-DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;

src/pg/test/sql/01_install_test.sql

@@ -1,7 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION plpythonu;
-CREATE EXTENSION cartodb;
 
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/sql/40_observatory_utility_test.sql

@@ -1,6 +1,8 @@
 \pset format unaligned
 \set ECHO all
 \i test/fixtures/load_fixtures.sql
+SET client_min_messages TO WARNING;
+\set ECHO none
 
 -- OBS_GeomTable
 -- get table with known geometry_id
@@ -9,8 +11,8 @@ SELECT
   cdb_observatory._OBS_GeomTable(
     ST_SetSRID(ST_Point(-74.0059, 40.7128), 4326),
     'us.census.tiger.census_tract',
-    '2014'
-  ) = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As _obs_geomtable_with_returned_table;
+    '2015'
+  ) = 'obs_87a814e485deabe3b12545a537f693d16ca702c2' As _obs_geomtable_with_returned_table;
 
 -- get null for unknown geometry_id
 -- should give back null
@@ -27,29 +29,6 @@ SELECT
 --     'us.census.tiger.census_tract'
 --   );
 
-WITH result as (
-SELECT
-  array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
-    'us.census.tiger.census_tract',
-    Array['us.census.spielman_singleton_segments.X55', 'us.census.acs.B01003001'],
-         '2010 - 2014') a
-)
-select
-(expected)[1]::text  = '{"colname":"x55","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"name":"Spielman-Singleton Segments: 55 Clusters","type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 55 clusters","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_1,
-(expected)[2]::text  = '{"colname":"total_pop","tablename":"obs_b393b5b88c6adda634b2071a8005b03c551b609a","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.","boundary_id":"us.census.tiger.census_tract"}' as test_get_obs_column_with_geoid_and_census_2
-from result;
-
--- should be null-valued
-WITH result as (
-SELECT
-  array_agg(a) expected from cdb_observatory._OBS_GetColumnData(
-    'us.census.tiger.census_tract',
-    Array['us.census.tiger.baloney'],
-    '2010 - 2014') a
-)
-select expected is null as OBS_GetColumnData_missing_measure
-from result;
-
 -- OBS_BuildSnapshotQuery
 -- Should give back: SELECT  vals[1] As total_pop, vals[2] As male_pop, vals[3] As female_pop, vals[4] As median_age
 SELECT
@@ -63,16 +42,20 @@ SELECT
     Array['mandarin_orange']
   ) = 'SELECT  vals[1] As mandarin_orange' As _OBS_BuildSnapshotQuery_test_2;
 
-SELECT cdb_observatory._OBS_GetRelatedColumn(
-    Array[
-     'es.ine.t3_1',
-     'us.census.acs.B01003001',
-     'us.census.acs.B01001002'
-    ],
-     'denominator'
- ) = '{es.ine.t1_1,NULL,us.census.acs.B01003001}' As _OBS_GetRelatedColumn_test;
-
 -- should give back a standardized measure name
 SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') = 'test_343_2_qqq' As _OBS_StandardizeMeasureName_test;
 
-\i test/fixtures/drop_fixtures.sql
+SELECT cdb_observatory.OBS_DumpVersion()
+  IS NOT NULL AS OBS_DumpVersion_notnull;
+
+-- Should fail to perform intersection
+SELECT ST_IsValid(ST_Intersection(
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county_clipped')
+)) AS complex_intersection_fails;
+
+-- Should succeed in intersecting
+SELECT ST_IsValid(cdb_observatory.safe_intersection(
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county_clipped')
+)) AS complex_safe_intersection_works;

src/pg/test/sql/42_observatory_exploration_test.sql

@@ -1,5 +1,6 @@
-\i test/fixtures/load_fixtures.sql
 \pset format unaligned
+\set ECHO none
+SET client_min_messages TO WARNING;
 
 -- set up variables for use in testing
 
@@ -9,11 +10,11 @@
 
 -- _OBS_SearchTables tests
 SELECT
-  t.table_name = 'obs_1babf5a26a1ecda5fb74963e88408f71d0364b81' As _OBS_SearchTables_tables_match,
-  t.timespan = '2014' As _OBS_SearchTables_timespan_matches
+  t.table_name = 'obs_0310c639744a2014bb1af82709228f05b59e7d3d' As _OBS_SearchTables_tables_match,
+  t.timespan = '2015' As _OBS_SearchTables_timespan_matches
 FROM cdb_observatory._OBS_SearchTables(
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) As t(table_name, timespan);
 
 -- _OBS_SearchTables tests
@@ -33,4 +34,670 @@ FROM cdb_observatory.OBS_GetAvailableBoundaries(
   cdb_observatory._TestPoint()
 ) AS t(boundary_id, description, time_span, tablename);
 
-\i test/fixtures/drop_fixtures.sql
+--
+-- OBS_GetAvailableNumerators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators())
+AS _obs_getavailablenumerators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getavailablenumerators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.income']
+))
+AS _obs_getavailablenumerators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01001002' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablenumerators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B19013001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablenumerators_no_income_denom_by_total_pop;
+
+SELECT 'us.zillow.AllHomes_Zhvi' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getavailablenumerators_zillow_at_zcta5;
+
+SELECT 'us.zillow.AllHomes_Zhvi' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.block_group'
+) WHERE valid_geom = True)
+AS _obs_getavailablenumerators_no_zillow_at_block_group;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailablenumerators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailablenumerators_no_total_pop_1996;
+
+--
+-- _OBS_GetNumerators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators())
+AS _obs_getnumerators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
+)) AS _obs_getnumerators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
+)) AS _obs_getnumerators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
+)) AS _obs_getnumerators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  subsection_tags => ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getnumerators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  subsection_tags => ARRAY['subsection/tags.income']
+))
+AS _obs_getnumerators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01001002' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  denom_id => 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getnumerators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B19013001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  denom_id => 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getnumerators_no_income_denom_by_total_pop;
+
+SELECT 'us.zillow.AllHomes_Zhvi' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  geom_id => 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getnumerators_zillow_at_zcta5;
+
+SELECT 'us.zillow.AllHomes_Zhvi' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  geom_id => 'us.census.tiger.block_group'
+) WHERE valid_geom = True)
+AS _obs_getnumerators_no_zillow_at_block_group;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  timespan => '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getnumerators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  timespan => '1996'
+) WHERE valid_timespan = True)
+AS _obs_getnumerators_no_total_pop_1996;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  name => 'tot'
+))
+AS _obs_getnumerators_total_pop_by_name;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.united_states}'
+))
+AS _obs_getnumerators_total_pop_by_section;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.ca}'
+))
+AS _obs_getnumerators_total_pop_not_in_canada;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.united_states}',
+  subsection_tags => '{subsection/tags.age_gender}'
+))
+AS _obs_getnumerators_total_pop_by_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.united_states}',
+  subsection_tags => '{subsection/tags.employment}'
+))
+AS _obs_getnumerators_total_pop_not_in_employment_subsection;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ids => '{us.census.acs.B01003001}'
+))
+AS _obs_getnumerators_total_pop_by_id;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ids => '{us.census.acs.B01003002}'
+))
+AS _obs_getnumerators_total_pop_not_with_other_id;
+
+--
+-- OBS_GetAvailableDenominators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators())
+AS _obs_getavailabledenominators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getavailabledenominators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.income']
+))
+AS _obs_getavailabledenominators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01001002'
+) WHERE valid_numer = True)
+AS _obs_getavailabledenominators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B19013001'
+) WHERE valid_numer = True)
+AS _obs_getavailabledenominators_no_income_denom_by_total_pop;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getavailabledenominators_at_zcta5;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'es.ine.the_geom'
+) WHERE valid_geom = True)
+AS _obs_getavailabledenominators_none_spanish_geom;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailabledenominators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailabledenominators_no_total_pop_1996;
+
+--
+-- OBS_GetAvailableGeometries tests
+--
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries())
+AS _obs_getavailablegeometries_usa_bg_in_all;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_usa_bg_in_nyc_point;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_usa_bg_in_usa_extents;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_no_usa_bg_not_in_zero_point;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.boundary']
+))
+AS _obs_getavailablegeometries_usa_bg_in_boundary_subsection;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['section/tags.uk']
+))
+AS _obs_getavailablegeometries_no_bg_in_uk_section;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_numer = True)
+AS _obs_getavailablegeometries_total_pop_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'foo.bar.baz'
+) WHERE valid_numer = True)
+AS _obs_getavailablegeometries_foobarbaz_not_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_total_pop_denom_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'foo.bar.baz'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2015'
+) WHERE valid_timespan = True)
+AS _obs_getavailablegeometries_bg_2015;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailablegeometries_bg_not_1996;
+
+SELECT 'subsection/tags.boundary' IN (SELECT (Jsonb_Each(geom_tags)).key
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)
+)) AS _obs_getavailablegeometries_has_boundary_tag;
+
+--
+-- OBS_GetAvailableTimespans tests
+--
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans())
+AS _obs_getavailabletimespans_2010_2014_in_all;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_2010_2014_in_nyc_point;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_2010_2014_in_usa_extents;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_no_usa_bg_not_in_zero_point;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_numer = True)
+AS _obs_getavailabletimespans_total_pop_in_2010_2014;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'foo.bar.baz'
+) WHERE valid_numer = True)
+AS _obs_getavailabletimespans_foobarbaz_not_in_2010_2014;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_total_pop_denom_in_2010_2014;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'foo.bar.baz'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014;
+
+--
+-- _OBS_GetGeometryScores tests
+--
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.county', 'us.census.tiger.zcta5']
+       AS _obs_geometryscores_500m_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+       = ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.county', 'us.census.tiger.zcta5']
+       AS _obs_geometryscores_5km_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+       AS _obs_geometryscores_50km_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY[ 'us.census.tiger.zcta5', 'us.census.tiger.census_tract',
+              'us.census.tiger.county', 'us.census.tiger.block_group' ]
+      AS _obs_geometryscores_500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+       = ARRAY['us.census.tiger.county', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_2500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.county', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.block_group'])
+      WHERE table_id LIKE '%2015%';
+
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_500m_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
+
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_5km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
+
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_50km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
+
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
+
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_2500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
+             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_50_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50)
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.census_tract',
+              'us.census.tiger.county', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_500_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500)
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+      AS _obs_geometryscores_500km_buffer_2500_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500)
+      WHERE table_id LIKE '%2015%';
+
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
+       = ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+               'us.census.tiger.county', 'us.census.tiger.zcta5']
+      AS _obs_geometryscores_500km_buffer_25000_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000)
+      WHERE table_id LIKE '%2015%';
+
+-- Check that one small geom approximates tract data
+WITH geoms AS (SELECT cdb_observatory._testarea() the_geom),
+summary AS (SELECT ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+                   COUNT(*)::INT cnt,
+                   SUM(ST_Area(the_geom))::Numeric sumarea
+            FROM geoms)
+SELECT column_id = 'us.census.tiger.census_tract' testarea_uses_tract
+FROM summary, LATERAL (
+  SELECT *
+  FROM cdb_observatory._OBS_GetGeometryScores(extent,
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'],
+        cnt, sumarea)) foo
+ORDER BY score DESC LIMIT 1;
+
+-- Check that randomly distributed points always use smallest geometry if we
+-- order by numgeoms desc
+WITH geoms as (SELECT UNNEST(ARRAY[
+    cdb_observatory._testpoint(),
+    st_translate(cdb_observatory._testpoint(), -0.003, 0),
+    st_translate(cdb_observatory._testpoint(), -0.006, 0)
+]) the_geom),
+summary as (SELECT
+  ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+  SUM(ST_Area(the_geom))::Numeric area,
+  COUNT(*)::INTEGER cnt
+  FROM geoms
+)
+SELECT column_id = 'us.census.tiger.block_group' points_use_bg
+      FROM summary, LATERAL (
+        SELECT * FROM cdb_observatory._OBS_GetGeometryScores(
+          extent,
+          ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+                'us.census.tiger.zcta5', 'us.census.tiger.county'],
+        cnt, area)) foo
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC LIMIT 1;
+
+--
+-- OBS_LegacyBuilderMetadata tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _total_pop_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B19013001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _median_income_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B19083001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _gini_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _total_pop_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19013001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _median_income_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19083001' NOT IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _gini_not_in_legacy_builder_metadata_sums;
+
+SELECT COUNT(*) = 0 _no_dupe_subsections_in_legacy_builder_metadata FROM (
+  SELECT name, subsection, count(*) FROM
+    (SELECT name, ((JSONB_Array_Elements(subsection))->'f1')->>'id' subsection
+    FROM cdb_observatory.obs_legacybuildermetadata()) foo
+  GROUP BY name, subsection
+  HAVING count(*) > 1
+) bar;
+

src/pg/test/sql/44_observatory_geometries_test.sql

@@ -1,6 +1,6 @@
 \pset format unaligned
-\set ECHO all
-\i test/fixtures/load_fixtures.sql
+\set ECHO none
+SET client_min_messages TO WARNING;
 
 -- set up variables for use in testing
 
@@ -34,7 +34,7 @@ SELECT cdb_observatory.OBS_GetBoundary(
 -- expect null geometry since there are no census tracts at null island
 -- timespan implictly null
 SELECT cdb_observatory.OBS_GetBoundary(
-  CDB_LatLng(0, 0),
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
   'us.census.tiger.census_tract'
 ) IS NULL As OBS_GetBoundary_null_island_census_tract;
 
@@ -42,7 +42,7 @@ SELECT cdb_observatory.OBS_GetBoundary(
 SELECT cdb_observatory.OBS_GetBoundary(
   cdb_observatory._TestPoint(),
   'us.census.tiger.census_tract',
-  '2014'
+  '2015'
 ) = :'cartodb_census_tract_geometry' As OBS_GetBoundary_year_census_tract;
 
 -- should return null
@@ -65,7 +65,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 SELECT cdb_observatory.OBS_GetBoundaryId(
   cdb_observatory._TestPoint(),
   'us.census.tiger.census_tract',
-  '2014'
+  '2015'
 ) = '36047048500'::text As OBS_GetBoundaryId_cartodb_census_tract_with_year;
 
 -- should give back '36047', the geoid of cartodb's county (King's/
@@ -73,12 +73,12 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 SELECT cdb_observatory.OBS_GetBoundaryId(
   cdb_observatory._TestPoint(),
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) = '36047'::text As OBS_GetBoundaryId_cartodb_county_with_year;
 
 -- should give back null since there is not a census tract at null island
 SELECT cdb_observatory.OBS_GetBoundaryId(
-  CDB_LatLng(0, 0),
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
   'us.census.tiger.census_tract'
 ) IS NULL As OBS_GetBoundaryId_null_island;
 
@@ -104,7 +104,7 @@ SELECT cdb_observatory.OBS_GetBoundaryById(
 SELECT cdb_observatory.OBS_GetBoundaryById(
   '36047',
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) = :'cartodb_county_geometry' OBS_GetBoundaryById_boundary_id_mismatch_geom_id;
 
 -- should give null since boundary_id does not match geometry reference id
@@ -115,6 +115,34 @@ SELECT cdb_observatory.OBS_GetBoundaryById(
 
 -- _OBS_GetBoundariesByGeometry
 
+SELECT array_agg(geom_refs) = Array[ '1104486618765', '1104486642837',
+                                '1104991798384', '1105044325367',
+                                '1105089330200', '1105089331758']
+As _OBS_GetBoundariesByGeometry_roads_around_cartodb
+FROM (
+  SELECT *
+  FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+    -- near CartoDB's office
+    ST_MakeEnvelope(-74,40.69,-73.99,40.7,
+                    4326),
+    'us.census.tiger.prisecroads_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
+SELECT
+  array_agg(geom_refs) = Array['1102654301684', '1102654307106',
+                               '1102654326686', '1102654351507' ]
+As _OBS_GetBoundariesByGeometry_points_around_cartodb
+FROM (
+  SELECT *
+  FROM cdb_observatory._OBS_GetBoundariesByGeometry(
+    -- near CartoDB's office
+    ST_MakeEnvelope(-73.9452409744,40.6988851644,-73.9280319214,40.7101254524,
+                    4326),
+    'us.census.tiger.pointlm_geom')
+  ORDER BY geom_refs ASC
+) As m(the_geom, geom_refs);
+
 -- check that all census tracts intersecting with the geometry are returned
 -- order them to ensure that the same values are returned
 SELECT
@@ -270,7 +298,7 @@ FROM (
                     -73.9280319214,40.7101254524,
                     4326),
     'us.census.tiger.census_tract',
-    '2014')
+    '2015')
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
@@ -313,7 +341,7 @@ FROM (
     cdb_observatory._testpoint(),
     500,
     'us.census.tiger.census_tract',
-    '2014')
+    '2015')
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
@@ -330,21 +358,4 @@ FROM (
   ORDER BY geom_refs ASC
 ) As m(the_geom, geom_refs);
 
--- _OBS_GetGeometryMetadata
--- get metadata for census tracts
-
-SELECT
-  geoid_colname = 'geoid' As geoid_name_matches,
-  target_table = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As table_name_matches,
-  geom_colname = 'the_geom' As geom_name_matches
-FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract')
-     As m(geoid_colname, target_table, geom_colname);
-
--- get metadata for boundaries with clipped geometries
- SELECT
-   geoid_colname = 'geoid' As geoid_name_matches,
-   target_table = 'obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963' As table_name_matches,
-   geom_colname = 'the_geom' As geom_name_matches
- FROM cdb_observatory._OBS_GetGeometryMetadata('us.census.tiger.census_tract_clipped') As m(geoid_colname, target_table, geom_colname);
-
 \i test/fixtures/drop_fixtures.sql

src/python/requirements.txt

@@ -0,0 +1,4 @@
+nose
+nose-timer
+nose_parameterized
+psycopg2

src/python/test/README.md

@@ -0,0 +1,14 @@
+### Integration/performance tests
+
+Tests here are meant to be run on a box with an Observatory meta/data dump
+loaded and ready to be tested against the API.
+
+The local Python needs the requirements in `src/python/requirements.txt`.
+
+In order to find and access the correct database, the `PGUSER`, `PGPASSWORD`,
+`PGHOST`, `PGPORT` and `PGDATABASE` env variables should be set.
+
+Tests should be executed as follows:
+
+     nosetests test/autotest.py
+     nosetests -s test/perftest.py

src/python/test/autotest.py

@@ -0,0 +1,373 @@
+from nose.tools import assert_equal, assert_is_not_none
+from nose.plugins.skip import SkipTest
+from nose_parameterized import parameterized
+
+from itertools import izip_longest
+from util import query
+from collections import OrderedDict
+import json
+
+
+def grouper(iterable, n, fillvalue=None):
+    "Collect data into fixed-length chunks or blocks"
+    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
+    args = [iter(iterable)] * n
+    return izip_longest(fillvalue=fillvalue, *args)
+
+
+USE_SCHEMA = True
+
+SKIP_COLUMNS = set([
+    u'mx.inegi_columns.INDI18',
+    u'mx.inegi_columns.ECO40',
+    u'mx.inegi_columns.POB34',
+    u'mx.inegi_columns.POB63',
+    u'mx.inegi_columns.INDI7',
+    u'mx.inegi_columns.EDU28',
+    u'mx.inegi_columns.SCONY10',
+    u'mx.inegi_columns.EDU31',
+    u'mx.inegi_columns.POB7',
+    u'mx.inegi_columns.VIV30',
+    u'mx.inegi_columns.INDI12',
+    u'mx.inegi_columns.EDU13',
+    u'mx.inegi_columns.ECO43',
+    u'mx.inegi_columns.VIV9',
+    u'mx.inegi_columns.HOGAR25',
+    u'mx.inegi_columns.POB32',
+    u'mx.inegi_columns.ECO7',
+    u'mx.inegi_columns.INDI19',
+    u'mx.inegi_columns.INDI16',
+    u'mx.inegi_columns.POB65',
+    u'mx.inegi_columns.INDI3',
+    u'mx.inegi_columns.INDI9',
+    u'mx.inegi_columns.POB36',
+    u'mx.inegi_columns.POB33',
+    u'mx.inegi_columns.POB58',
+    u'mx.inegi_columns.DISC4',
+    u'mx.inegi_columns.VIV41',
+    u'mx.inegi_columns.VIV40',
+    u'mx.inegi_columns.VIV17',
+    u'mx.inegi_columns.VIV25',
+    u'mx.inegi_columns.EDU10',
+    u'whosonfirst.wof_disputed_name',
+    u'us.census.tiger.fullname',
+    u'whosonfirst.wof_marinearea_name',
+    u'us.census.tiger.mtfcc',
+    u'whosonfirst.wof_county_name',
+    u'whosonfirst.wof_region_name',
+    'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    , 'fr.insee.P12_ACTOCC15P_ILT45D'
+    , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    , 'fr.insee.P12_ACTOCC15P_ILT45D'
+    , 'uk.ons.LC3202WA0007'
+    , 'uk.ons.LC3202WA0010'
+    , 'uk.ons.LC3202WA0004'
+    , 'uk.ons.LC3204WA0004'
+    , 'uk.ons.LC3204WA0007'
+    , 'uk.ons.LC3204WA0010'
+    , 'br.geo.subdistritos_name'
+])
+
+MEASURE_COLUMNS = query('''
+SELECT ARRAY_AGG(DISTINCT numer_id) numer_ids,
+       numer_aggregate,
+       denom_reltype,
+       section_tags
+FROM observatory.obs_meta
+WHERE numer_weight > 0
+  AND numer_id NOT IN ('{skip}')
+  AND section_tags IS NOT NULL
+  AND subsection_tags IS NOT NULL
+GROUP BY numer_aggregate, section_tags, denom_reltype
+'''.format(skip="', '".join(SKIP_COLUMNS))).fetchall()
+
+#CATEGORY_COLUMNS = query('''
+#SELECT distinct numer_id
+#FROM observatory.obs_meta
+#WHERE numer_type ILIKE 'text'
+#AND numer_weight > 0
+#''').fetchall()
+#
+#BOUNDARY_COLUMNS = query('''
+#SELECT id FROM observatory.obs_column
+#WHERE type ILIKE 'geometry'
+#AND weight > 0
+#''').fetchall()
+#
+#US_CENSUS_MEASURE_COLUMNS = query('''
+#SELECT distinct numer_name
+#FROM observatory.obs_meta
+#WHERE numer_type ILIKE 'numeric'
+#AND 'us.census.acs' = ANY (subsection_tags)
+#AND numer_weight > 0
+#''').fetchall()
+
+
+#def default_geometry_id(column_id):
+#    '''
+#    Returns default test point for the column_id.
+#    '''
+#    if column_id == 'whosonfirst.wof_disputed_geom':
+#        return 'ST_SetSRID(ST_MakePoint(76.57, 33.78), 4326)'
+#    elif column_id == 'whosonfirst.wof_marinearea_geom':
+#        return 'ST_SetSRID(ST_MakePoint(-68.47, 43.33), 4326)'
+#    elif column_id in ('us.census.tiger.school_district_elementary',
+#                       'us.census.tiger.school_district_secondary',
+#                       'us.census.tiger.school_district_elementary_clipped',
+#                       'us.census.tiger.school_district_secondary_clipped'):
+#        return 'ST_SetSRID(ST_MakePoint(-73.7067, 40.7025), 4326)'
+#    elif column_id.startswith('es.ine'):
+#        return 'ST_SetSRID(ST_MakePoint(-2.51141249535454, 42.8226119029222), 4326)'
+#    elif column_id.startswith('us.zillow'):
+#        return 'ST_SetSRID(ST_MakePoint(-81.3544048197256, 28.3305906291771), 4326)'
+#    elif column_id.startswith('ca.'):
+#        return ''
+#    else:
+#        return 'ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)'
+
+
+def default_lonlat(column_id):
+    '''
+    Returns default test point for the column_id.
+    '''
+    if column_id == 'whosonfirst.wof_disputed_geom':
+        return (76.57, 33.78)
+    elif column_id == 'whosonfirst.wof_marinearea_geom':
+        return (-68.47, 43.33)
+    elif column_id in ('us.census.tiger.school_district_elementary',
+                       'us.census.tiger.school_district_secondary',
+                       'us.census.tiger.school_district_elementary_clipped',
+                       'us.census.tiger.school_district_secondary_clipped'):
+        return (40.7025, -73.7067)
+    elif column_id.startswith('uk'):
+        if 'WA' in column_id:
+            return (51.46844551219723, -3.184833526611328)
+        else:
+            return (51.51461834694225, -0.08883476257324219)
+    elif column_id.startswith('es'):
+        return (42.8226119029222, -2.51141249535454)
+    elif column_id.startswith('us.zillow'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('mx.'):
+        return (19.41347699386547, -99.17019367218018)
+    elif column_id.startswith('th.'):
+        return (13.725377712079784, 100.49263000488281)
+    # cols for French Guyana only
+    #elif column_id in ('fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    #                   , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    #                   , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    #                   , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    #                   , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    #                   , 'fr.insee.P12_ACTOCC15P_ILT45D'
+    #                   , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+    #                   , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+    #                   , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+    #                   , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+    #                   , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+    #                   , 'fr.insee.P12_ACTOCC15P_ILT45D'):
+    #    return (4.938408371206558, -52.32908248901367)
+    elif column_id.startswith('fr.'):
+        return (48.860875144709475, 2.3613739013671875)
+    elif column_id.startswith('ca.'):
+        return (43.65594991256823, -79.37965393066406)
+    elif column_id.startswith('us.census.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('us.dma.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('us.ihme.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('us.bls.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('us.qcew.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('whosonfirst.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('us.epa.'):
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('eu.'):
+        raise SkipTest('No tests for Eurostat!')
+    elif column_id.startswith('br.'):
+        return (-23.53, -46.63)
+    elif column_id.startswith('au.'):
+        return (-33.8806, 151.2131)
+    else:
+        raise Exception('No catalog point set for {}'.format(column_id))
+
+
+def default_point(column_id):
+    lat, lng = default_lonlat(column_id)
+    return 'ST_SetSRID(ST_MakePoint({lng}, {lat}), 4326)'.format(
+        lat=lat, lng=lng)
+
+
+def default_area(column_id):
+    '''
+    Returns default test area for the column_id
+    '''
+    point = default_point(column_id)
+    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 250), 4326)'.format(
+        point=point)
+    return area
+
+#@parameterized(US_CENSUS_MEASURE_COLUMNS)
+#def test_get_us_census_measure_points(name):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
+#                 '''.format(name=name.replace("'", "''"),
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point('')))
+#    rows = resp.fetchall()
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+
+def grouped_measure_columns():
+    for numer_ids, numer_aggregate, denom_reltype, section_tags in MEASURE_COLUMNS:
+        for colgroup in grouper(numer_ids, 50):
+            yield [c for c in colgroup if c], numer_aggregate, denom_reltype, section_tags
+
+
+@parameterized(grouped_measure_columns())
+def test_get_measure_points(numer_ids, numer_aggregate, denom_reltype, section_tags):
+    _test_measures(numer_ids, numer_aggregate, section_tags, denom_reltype, default_point(numer_ids[0]))
+
+
+@parameterized(grouped_measure_columns())
+def test_get_measure_areas(numer_ids, numer_aggregate, denom_reltype, section_tags):
+    if numer_aggregate is None or numer_aggregate.lower() not in ('sum', 'median', 'average'):
+        return
+    if numer_aggregate.lower() in ('median', 'average') \
+       and (denom_reltype is None \
+            or denom_reltype.lower() != 'universe'):
+        return
+    _test_measures(numer_ids, numer_aggregate, section_tags, denom_reltype, default_area(numer_ids[0]))
+
+
+def _test_measures(numer_ids, numer_aggregate, section_tags, denom_reltype, geom):
+    in_params = []
+    for numer_id in numer_ids:
+        in_params.append({
+            'numer_id': numer_id,
+            'normalization': 'predenominated'
+        })
+
+    params = query(u'''
+        SELECT {schema}OBS_GetMeta({geom}, '{in_params}')
+    '''.format(schema='cdb_observatory.' if USE_SCHEMA else '',
+               geom=geom,
+               in_params=json.dumps(in_params))).fetchone()[0]
+
+    # We can get duplicate IDs from multi-denominators, so for now we
+    # compress those measures into a single
+    params = OrderedDict([(p['id'], p) for p in params]).values()
+    assert_equal(len(params), len(in_params),
+                 'Inconsistent out and in params for {}'.format(in_params))
+
+    q = u'''
+    SELECT * FROM {schema}OBS_GetData(ARRAY[({geom}, 1)::geomval], '{params}')
+    '''.format(schema='cdb_observatory.' if USE_SCHEMA else '',
+               geom=geom,
+               params=json.dumps(params).replace(u"'", "''"))
+    resp = query(q).fetchone()
+    assert_is_not_none(resp, 'NULL returned for {}'.format(in_params))
+    rawvals = resp[1]
+    vals = [v['value'] for v in rawvals]
+
+    assert_equal(len(vals), len(in_params))
+    for i, val in enumerate(vals):
+        assert_is_not_none(val, 'NULL for {}'.format(in_params[i]['numer_id']))
+
+
+#@parameterized(CATEGORY_COLUMNS)
+#def test_get_category_areas(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(CATEGORY_COLUMNS)
+#def test_get_category_points(column_id):
+#    if column_id in SKIP_COLUMNS:
+#        raise SkipTest('Column {} should be skipped'.format(column_id))
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    rows = resp.fetchall()
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundaries_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_points_by_geometry(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            area=default_area(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_points(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            point=default_point(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+
+#@parameterized(BOUNDARY_COLUMNS)
+#def test_get_boundary_by_id(column_id):
+#    resp = query('''
+#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
+#                 '''.format(column_id=column_id,
+#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+#                            geometry_id=default_geometry_id(column_id)))
+#    assert_equal(resp.status_code, 200)
+#    rows = resp.json()['rows']
+#    assert_equal(1, len(rows))
+#    assert_is_not_none(rows[0][0])
+

src/python/test/perftest.py

@@ -0,0 +1,352 @@
+from nose.tools import assert_equal, assert_is_not_none
+from nose_parameterized import parameterized
+
+from util import query, commit
+
+from time import time
+
+import json
+import os
+
+USE_SCHEMA = True
+
+for q in (
+        'DROP TABLE IF EXISTS obs_perftest_simple',
+        '''CREATE TABLE obs_perftest_simple (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_simple (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-74.1, 40.5,
+                                 -73.8, 40.9, 4326),
+                 'us.census.tiger.census_tract_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 1000''',
+        'DROP TABLE IF EXISTS obs_perftest_complex',
+        '''CREATE TABLE obs_perftest_complex (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_complex (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-75.05437469482422,40.66319159533881,
+                                 -73.81885528564453,41.745696344339564, 4326),
+                 'us.census.tiger.county_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;''',
+        'DROP TABLE IF EXISTS obs_perftest_country_simple',
+        '''CREATE TABLE obs_perftest_country_simple (cartodb_id SERIAL PRIMARY KEY,
+           geom GEOMETRY,
+           name TEXT) ''',
+        '''INSERT INTO obs_perftest_country_simple (geom, name)
+           SELECT the_geom geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-179,-89, 179,89, 4326),
+                 'whosonfirst.wof_country_geom')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 50;''',
+        'DROP TABLE IF EXISTS obs_perftest_country_complex',
+        '''CREATE TABLE obs_perftest_country_complex (cartodb_id SERIAL PRIMARY KEY,
+           geom GEOMETRY,
+           name TEXT) ''',
+        '''INSERT INTO obs_perftest_country_complex (geom, name)
+           SELECT the_geom geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-179,-89, 179,89, 4326),
+                 'whosonfirst.wof_country_geom')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;''',
+        #'''SET statement_timeout = 5000;'''
+):
+    q_formatted = q.format(
+        schema='cdb_observatory.' if USE_SCHEMA else '',
+    )
+    start = time()
+    resp = query(q_formatted)
+    end = time()
+    print('{} for {}'.format(int(end - start), q_formatted))
+    if q.lower().startswith('insert'):
+        if resp.rowcount == 0:
+            raise Exception('''Performance fixture creation "{}" inserted 0 rows,
+                            this will break tests.  Check the query to determine
+                            what is going wrong.'''.format(q_formatted))
+    commit()
+
+
+ARGS = {
+    ('OBS_GetMeasureByID', None): "name, 'us.census.acs.B01001002', '{}'",
+    ('OBS_GetMeasure', 'predenominated'): "{}, 'us.census.acs.B01003001', null, {}",
+    ('OBS_GetMeasure', 'area'): "{}, 'us.census.acs.B01001002', 'area', {}",
+    ('OBS_GetMeasure', 'denominator'): "{}, 'us.census.acs.B01001002', 'denominator', {}",
+    ('OBS_GetCategory', None): "{}, 'us.census.spielman_singleton_segments.X10', {}",
+    ('_OBS_GetGeometryScores', None): "{}, NULL"
+}
+
+
+def record(params, results):
+    sha = os.environ['OBS_EXTENSION_SHA']
+    msg = os.environ.get('OBS_EXTENSION_MSG')
+    fpath = os.path.join(os.environ['OBS_PERFTEST_DIR'], sha + '.json')
+    if os.path.isfile(fpath):
+        tests = json.load(open(fpath, 'r'))
+    else:
+        tests = {}
+    with open(fpath, 'w') as fhandle:
+        tests[json.dumps(params)] = {
+            'params': params,
+            'results': results
+        }
+        json.dump(tests, fhandle)
+
+@parameterized([
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('country_simple', '_OBS_GetGeometryScores', 'NULL', 5000),
+
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('country_complex', '_OBS_GetGeometryScores', 'NULL', 5000),
+])
+def test_getgeometryscores_performance(geom_complexity, api_method, filters, target_geoms):
+    print api_method, geom_complexity, filters, target_geoms
+
+    rownums = (1, 5, 10, ) if 'complex' in geom_complexity else (5, 25, 50,)
+    results = []
+    for rows in rownums:
+        stmt = '''SELECT {schema}{api_method}(geom, {filters}, {target_geoms})
+                   FROM obs_perftest_{complexity}
+                   WHERE cartodb_id <= {n}'''.format(
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       filters=filters,
+                       target_geoms=target_geoms,
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'filters': filters,
+            'target_geoms': target_geoms
+        }, results)
+
+@parameterized([
+    ('simple', 'OBS_GetMeasureByID', None, 'us.census.tiger.census_tract', None),
+    ('complex', 'OBS_GetMeasureByID', None, 'us.census.tiger.county', None),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'point', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+])
+def test_getmeasure_performance(geom_complexity, api_method, normalization, geom, boundary):
+    print api_method, geom_complexity, normalization, geom, boundary
+    col = 'measure' if 'measure' in api_method.lower() else 'category'
+    results = []
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (5, 25, 50, )
+    for rows in rownums:
+        stmt = '''UPDATE obs_perftest_{complexity}
+                   SET {col} = {schema}{api_method}({args})
+                   WHERE cartodb_id <= {n}'''.format(
+                       col=col,
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       args=ARGS[api_method, normalization].format(geom, boundary),
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print rows, ': ', qps, ' QPS'
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'normalization': normalization,
+            'boundary': boundary,
+            'geom': geom
+        }, results)
+
+
+@parameterized([
+    ('simple', 'predenominated', 'point', 'null'),
+    ('simple', 'predenominated', 'geom', 'null'),
+    ('simple', 'predenominated', 'offset_geom', 'null'),
+    ('simple', 'area', 'point', 'null'),
+    ('simple', 'area', 'geom', 'null'),
+    ('simple', 'area', 'offset_geom', 'null'),
+    ('simple', 'denominator', 'point', 'null'),
+    ('simple', 'denominator', 'geom', 'null'),
+    ('simple', 'denominator', 'offset_geom', 'null'),
+
+    ('simple', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'predenominated', 'geom', 'null'),
+    ('complex', 'predenominated', 'offset_geom', 'null'),
+    ('complex', 'area', 'geom', 'null'),
+    ('complex', 'area', 'offset_geom', 'null'),
+    ('complex', 'denominator', 'geom', 'null'),
+    ('complex', 'denominator', 'offset_geom', 'null'),
+
+    ('complex', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
+])
+def test_getdata_performance(geom_complexity, normalization, geom, boundary):
+    print geom_complexity, normalization, geom, boundary
+
+    cols = ['us.census.acs.B01001002',
+            'us.census.acs.B01001003',
+            'us.census.acs.B01001004',
+            'us.census.acs.B01001005',
+            'us.census.acs.B01001006',
+            'us.census.acs.B01001007',
+            'us.census.acs.B01001008',
+            'us.census.acs.B01001009',
+            'us.census.acs.B01001010',
+            'us.census.acs.B01001011', ]
+    in_meta = [{"numer_id": col,
+                "normalization": normalization,
+                "geom_id": None if boundary.lower() == 'null' else boundary.replace("'", '')}
+               for col in cols]
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (10, 50, 100)
+
+    for num_meta in (1, 10, ):
+        results = []
+        for rows in rownums:
+            stmt = '''
+    with data as (
+      SELECT id, data FROM {schema}OBS_GetData(
+        (SELECT array_agg(({geom}, cartodb_id)::geomval)
+         FROM obs_perftest_{complexity}
+         WHERE cartodb_id <= {n}),
+        (SELECT {schema}OBS_GetMeta(
+          (SELECT st_setsrid(st_extent({geom}), 4326)
+           FROM obs_perftest_{complexity}
+           WHERE cartodb_id <= {n}),
+          '{in_meta}'::JSON
+        ))
+      ))
+    UPDATE obs_perftest_{complexity}
+    SET measure = (data->0->>'value')::Numeric
+    FROM data
+    WHERE obs_perftest_{complexity}.cartodb_id = data.id
+    ;
+            '''.format(
+                point_or_poly='point' if geom == 'point' else 'polygon',
+                complexity=geom_complexity,
+                schema='cdb_observatory.' if USE_SCHEMA else '',
+                geom=geom,
+                in_meta=json.dumps(in_meta[0:num_meta]),
+                n=rows)
+            start = time()
+            query(stmt)
+            end = time()
+            qps = (rows / (end - start))
+            results.append({
+                'rows': rows,
+                'qps': qps,
+                'stmt': stmt
+            })
+            print rows, ': ', qps, ' QPS'
+
+        if 'OBS_RECORD_TEST' in os.environ:
+            record({
+                'geom_complexity': geom_complexity,
+                'api_method': 'OBS_GetData',
+                'normalization': normalization,
+                'boundary': boundary,
+                'geom': geom,
+                'num_meta': str(num_meta)
+            }, results)

src/python/test/util.py

@@ -0,0 +1,31 @@
+import os
+import psycopg2
+
+DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
+    user=os.environ.get('PGUSER', 'postgres'),
+    password=os.environ.get('PGPASSWORD', ''),
+    host=os.environ.get('PGHOST', 'localhost'),
+    port=os.environ.get('PGPORT', '5432'),
+    database=os.environ.get('PGDATABASE', 'postgres'),
+))
+CURSOR = DB_CONN.cursor()
+
+
+def query(q):
+    '''
+    Query the database.
+    '''
+    try:
+        CURSOR.execute(q)
+        return CURSOR
+    except:
+        DB_CONN.rollback()
+        raise
+
+
+def commit():
+    try:
+        DB_CONN.commit()
+    except:
+        DB_CONN.rollback()
+        raise