Merge pull request #361 from CartoDB/chore/ch111328/remove-do-v1-api

Mark as deprecated

.github/PULL_REQUEST_TEMPLATE.md

@@ -2,6 +2,17 @@
 
 I'd like to request a new data observatory extension deploy: dump + extension
 
+**VERY IMPORTANT!!!**
+
+PLEASE USE `python scripts/generate_fixtures.py` TO GENERATE NEW FIXTURES FOR
+THE NEW DUMP AND OVERRIDE IT IN THIS PROJECT BEFORE PASS THE TESTS
+
+## Performance comparison to last deployment
+
+Please include link here to comparison perftests:
+
+http://52.71.151.140/perftest/#oldsha..newsha
+
 ## Dump database id to be deployed
 
 Please put here the dump id to be deployed: <dump_id>
@@ -12,6 +23,6 @@ Please put here the dump id to be deployed: <dump_id>
 
 Add down here the PR links to be added and deployed:
 
-  - 
+  -
 
-// @CartoDB/dataservices
+// @CartoDB/datateam

.travis.yml

@@ -0,0 +1,43 @@
+language: c
+sudo: required
+
+env:
+  global:
+    - PGUSER=postgres
+    - PGDATABASE=postgres
+    - PGOPTIONS='-c client_min_messages=NOTICE'
+
+jobs:
+  include:
+    - env: POSTGRESQL_VERSION="9.6" POSTGIS_VERSION="2.5"
+      dist: xenial
+    - env: POSTGRESQL_VERSION="10" POSTGIS_VERSION="2.5"
+      dist: xenial
+    - env: POSTGRESQL_VERSION="11" POSTGIS_VERSION="2.5"
+      dist: xenial
+    - env: POSTGRESQL_VERSION="12" POSTGIS_VERSION="2.5"
+      dist: bionic
+    - env: POSTGRESQL_VERSION="12" POSTGIS_VERSION="3"
+      dist: bionic
+
+script:
+  - sudo apt-get install -y --allow-unauthenticated --no-install-recommends --no-install-suggests postgresql-$POSTGRESQL_VERSION postgresql-client-$POSTGRESQL_VERSION postgresql-server-dev-$POSTGRESQL_VERSION postgresql-common
+  - if [[ $POSTGRESQL_VERSION == '9.6' ]]; then sudo apt-get install -y postgresql-contrib-9.6; fi;
+  - sudo apt-get install -y --allow-unauthenticated postgresql-$POSTGRESQL_VERSION-postgis-$POSTGIS_VERSION postgresql-$POSTGRESQL_VERSION-postgis-$POSTGIS_VERSION-scripts postgis
+  # For pre12, install plpython2. For PG12 install plpython3
+  - if [[ $POSTGRESQL_VERSION != '12' ]]; then sudo apt-get install -y postgresql-plpython-$POSTGRESQL_VERSION python python-redis; else sudo apt-get install -y postgresql-plpython3-12 python3 python3-redis; fi;
+  - sudo pg_dropcluster --stop $POSTGRESQL_VERSION main
+  - sudo rm -rf /etc/postgresql/$POSTGRESQL_VERSION /var/lib/postgresql/$POSTGRESQL_VERSION /var/ramfs/postgresql/$POSTGRESQL_VERSION
+  - sudo pg_createcluster -u postgres $POSTGRESQL_VERSION main --start -- --auth-local trust --auth-host password
+  - export PGPORT=$(pg_lsclusters | grep $POSTGRESQL_VERSION | awk '{print $3}')
+  - cd src/pg/
+  - make
+  - sudo make install
+  - make installcheck
+
+after_failure:
+  - pg_lsclusters
+  - cat test/regression.out
+  - cat test/regression.diffs
+  - echo $PGPORT
+  - sudo cat /var/log/postgresql/postgresql-$POSTGRESQL_VERSION-main.log

CONTRIBUTING.md

@@ -28,8 +28,8 @@ Run the tests with `make test`.
 
 Update extension in a working database with:
 ```
-ALTER EXTENSION observatory VERSION TO 'current';
+ALTER EXTENSION observatory UPDATE TO 'current';
-ALTER EXTENSION observatory VERSION TO 'dev';
+ALTER EXTENSION observatory UPDATE TO 'dev';
 ```
 
 Note: we keep the current development version install as 'dev' always;

NEWS.md

@@ -1,4 +1,260 @@
+1.10.0 (2018-07-??)
+-------------------
+
+__Improvements__
+
+* Updated for PostgreSQL 12 and PostGIS 3.0 compatibility.
+
+1.9.0 (2018-04-20)
+------------------
+
+__Improvements__
+
+* Improved `OBS_GetAvailableGeometries` for the DO Timespans project ([#325](https://github.com/CartoDB/observatory-extension/pull/325))
+* Improved `OBS_GetAvailableTimespans` for the DO Timespans project ([#324](https://github.com/CartoDB/bigmetadata/issues/324))
+* Modified the denominated suggested_name to mitigate collisions ([#327](https://github.com/CartoDB/observatory-extension/pull/327))
+* Fixed some errors so now the extension supports PostgreSQL 10 ([#329](https://github.com/CartoDB/observatory-extension/pull/329))
+* Fixed documentation
+* Add support for multiple PostgreSQL and Postgis versions in our travis script for test purposes
+
+1.8.0 (2017-10-18)
+------------------
+
+__Improvements__
+
+* Add `number_geometries` field to `OBS_GetAvailableGeometries` in order to provide the number of geometries from the source data to be used in the score calculation ([#313](https://github.com/CartoDB/observatory-extension/issues/313))
+
+1.7.0 (2017-08-18)
+------------------
+
+__Improvements__
+
+* Add Travis support to execute the extension tests ([#183](https://github.com/CartoDB/observatory-extension/issues/183))
+
+__API Changes__
+
+* Add new function `OBS_MetadataValidation` ([#303](https://github.com/CartoDB/observatory-extension/pull/303))
+
+__Bugfixes__
+
+* Fixed parentheses for obs_getdata with ids
+* Fixed failing tests due changes in the data dump for some TIGER geometries
+
+1.6.0 (2017-07-20)
+------------------
+
+__Improvements__
+
+* The current OBS_GetAvailableNumerators is not designed with our
+  UI in mind so it's causing a lot of troubles and we're doing so
+  many hacks to fit our UI needs and the interface of the function so this
+  function it's a better fit for our purposes. ([#300](https://github.com/CartoDB/observatory-extension/pull/300))
+* Now use the new meta table `obs_meta_geom_numer_timespan` to filter
+  the geometries by geometries timespan and/or numerator timespan (which
+  is what we get when we use the obs_getavailabletimespans) ([#302](https://github.com/CartoDB/observatory-extension/pull/302))
+
+__Bugfixes__
+
+* Right now we're doing INNER JOINS when we JOIN the `_procgeoms` and
+  the data so we end up with NULL value instead of id, NULL value. ([#298](https://github.com/CartoDB/observatory-extension/pull/298))
+
+
+1.5.1 (2017-05-16)
+------------------
+
+__Improvements__
+
+* Much improved performance for `OBS_GetData` when augmenting with several
+  different geometries simultaneously ([#285](https://github.com/CartoDB/observatory-extension/pull/285))
+* Return the automatically assigned normalization type from `OBS_GetMeta`
+  ([#285](https://github.com/CartoDB/observatory-extension/pull/285))
+
+1.5.0 (2017-04-24)
+------------------
+
+__API Changes__
+
+* Add `suggested_name` to `OBS_GetMeta` responses
+  ([#281](https://github.com/CartoDB/observatory-extension/pull/281))
+* Add `geom_type`, `geom_extra`, and `geom_tags` to
+  `OBS_GetAvailableGeometries`.  This brings it up to spec with existing docs.
+  ([#282](https://github.com/CartoDB/observatory-extension/pull/282))
+* Add `timespan_type`, `timespan_extra`, and `timespan_tags` to
+  `OBS_GetAvailableTimespans` for consistency.
+  ([#282](https://github.com/CartoDB/observatory-extension/pull/282))
+
+1.4.0 (2017-03-21)
+------------------
+
+__API Changes__
+
+* Allow for override of `target_area` and `target_geoms` in `OBS_GetMeta`
+  ([#276](https://github.com/CartoDB/observatory-extension/pull/276)).  This
+  allows the interface to work with points and sparse areas much btter.
+* Allow for override of `max_timespan_rank` and `max_score_rank` on an
+  item-by-item basis for metadata.
+* `numer_description`, `geom_description`, `denom_description`,
+  `numer_t_description`, `denom_t_description` and `geom_t_description` now
+  returned as part of `OBS_GetMeta`.
+
+__Improvements__
+
+* Reduced amount of simplification done on input geometries (from 0.0001 above
+  500 points to 0.00001 above 1000 points).
+* Added tests to confirm that accurate results are returned from automatic
+  boundary selection
+
+1.3.5 (2017-03-15)
+------------------
+
+No changes.  Artifact to allow for data update.
+
+1.3.4 (2017-03-10)
+------------------
+
+__Bugfixes__
+
+* Remove erroneously committed `RAISE NOTICE` in `OBS_GetData`
+
+1.3.3 (2017-03-10)
+------------------
+
+__Bugfixes__
+
+* Resolve divide-by-zero errors in cases where the intersection of an
+  Observatory geometry and user geometry has 0 area
+  ([#265](https://github.com/CartoDB/observatory-extension/pull/265))
+* Run MakeValid on geometry's when intersecting, if necessary
+  ([#268](https://github.com/CartoDB/observatory-extension/pull/268))
+
+__Improvements__
+
+* Add performance tests for multiple columns in `OBS_GetData`
+* Major performance boost for `autotest.py` through the use of multi-column
+  `OBS_GetData` instead of separate `OBS_GetMeasure` calls for every single
+  measurement.
+  ([#268](https://github.com/CartoDB/observatory-extension/pull/268))
+* Major performance boost for `OBS_GetData` in cases where multiple columns are
+  requested.  Previously, each additional column would result in a linear
+  slowdown, even if geometries could be reused.
+  ([#267](https://github.com/CartoDB/observatory-extension/pull/267))
+
+1.3.2 (2017-03-02)
+------------------
+
+__Bugfixes__
+
+* Accept "prenormalized" as well as "predenominated" to bypass normalization.
+  This fixes issues with Camshaft.
+
+1.3.1 (2017-02-16)
+------------------
+
+__Improvements__
+
+* It is now possible to obtain measures that are averages or medians over
+  arbitrary polygons ([#254](https://github.com/CartoDB/observatory-extension/pull/254).
+* Added test point for Australian data
+* `OBS_GetLegacyMetadata` now returns median and averages in cases where it is
+  called for measures for polygons
+
+1.3.0 (2017-01-17)
+------------------
+
+__API Changes__
+
+* `OBS_GetMeasureDataMulti()` is now called `OBS_GetData()`
+* `OBS_GetMeasureMetaMulti()` is now called `OBS_GetMeta()`
+* Additional signature for `OBS_GetData` which can take an array of `TEXT`,
+  mimicking functionality of `OBS_GetMeasureByID`
+
+__Improvements__
+
+* Generate fixtures from `obs_meta`
+* Remove unused table-level code
+* Refactor all augmentation and geometry functions to obtain data from
+  `OBS_GetMeta()` and `OBS_GetData()`.
+* Improvements to `OBS_GetMeta()` so it can still fill in metadata in cases
+  where only a geometry is being requested.
+* `OBS_GetData()` returns two-column table instead of anonymous record.
+* `OBS_GetData()` can return categorical (text) and geometries
+
+__Bugfixes__
+
+* Remove unnecessary dependency on `postgres_fdw`
+* `OBS_GetData()` now aggregates measures with mixed geoms correctly
+
+1.2.1 (2017-01-17)
+------------------
+
+__Improvements__
+
+* Support Point/LineString in responses from `OBS_GetBoundary`.
+  ([#243](https://github.com/CartoDB/observatory-extension/pull/233))
+
+1.2.0 (2016-12-28)
+------------------
+
+__API Changes__
+
+* Added `OBS_GetMeasureDataMulti`, which takes an array of geomvals and
+  parameters as JSON, and returns a set of RECORDs keyed by the vals of the
+  geomvals.
+* Added `OBS_GetMeasureMetaMulti`, which takes sparse metadata as JSON (for
+  example, the measure ID) and returns a filled-out version of the metadata
+  sufficient for use with `OBS_GetMeasureDataMulti`.
+
+__Improvements__
+
+* Move tests to 2015
+* Fixes to `_OBS_GetGeometryScores` to avoid spamming NOTICEs about all pixels
+  for a band being NULL
+* Tests for `_OBS_GetGeometryScores` with complex geometries
+* Performance tests for `OBS_GetMeasureDataMulti`
+* Return both `table_id` and `column_id` from `_OBS_GetGeometryScores`
+
+1.1.7 (2016-12-15)
+------------------
+
+__Improvements__
+
+* Use simpler raster table and simplified `_OBSGetGeometryScores` functions to
+  improve performance
+* In cases where geometry passed into geometry scoring function has greater
+  than 10K points, simply use its buffer instead
+* Add `IMMUTABLE` to `_OBSGetGeometryScores`
+* Add tests explicitly for `_OBSGetGeometryScores` in perftest.py
+* Yields a ~50% improvement in performance for `_OBSGetGeomeryScores`.
+
+1.1.6 (2016-12-08)
+------------------
+
+__Bugfixes__
+
+* Fix divide by zero condition in "denominator" branch of `OBS_GetMeasure`
+  when passing in a polygon ([#233](https://github.com/CartoDB/observatory-extension/pull/233)).
+
+__Improvements__
+
+* Use `ST_Subdivide` to improve performance when functions are called on very
+  complex geometries (with many points) ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Improve raster scoring to more heavily weight boundaries with nearer to
+  correct number of points, and penalize boundaries with lots of blank space
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Remove some redundant area calculations in `OBS_GetMeasure`
+  ([#232](https://github.com/CartoDB/observatory-extension/pull/232))
+* Replace use of `format('%L', var)` with proper use of `EXECUTE` and `$1` etc.
+  variables ([#231](https://github.com/CartoDB/observatory-extension/pull/231))
+* Add test point for Brazil
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+* Improvements to performance tests
+  ([#229](https://github.com/CartoDB/observatory-extension/pull/229))
+  - Support simple and complex geometries
+  - Handle all code branches
+  - Add ability to persist results to JSON for graph visualization later
+
 1.1.5 (2016-11-29)
+------------------
 
 __Bugfixes__
 
@@ -6,6 +262,7 @@ __Bugfixes__
   a geometry where it does not exist ([#220](https://github.com/CartoDB/observatory-extension/issues/220)).
 
 1.1.4 (2016-11-21)
+------------------
 
 __Bugfixes__
 
@@ -13,10 +270,12 @@ __Bugfixes__
   `OBS_GetLegacyMetadata` ([#216](https://github.com/CartoDB/observatory-extension/issues/216)).
 
 1.1.3 (2016-11-15)
+------------------
 
 * Temporarily ignore EU data for the sake of testing
 
 1.1.2 (2016-11-09)
+------------------
 
 __Improvements__
 
@@ -32,12 +291,14 @@ __API Changes (Internal)__
 * Add internal `_OBS_GetGeometryScores`
 
 1.1.1 (2016-10-14)
+------------------
 
 __Improvements__
 
 * Test points for Canada and France ([#204](https://github.com/CartoDB/observatory-extension/issues/120))
 
 1.1.0 (2016-10-04)
+------------------
 
 __Bugfixes__
 
@@ -60,6 +321,7 @@ __API Changes__
   is also referred to here ([CartoDB/design#68](https://github.com/CartoDB/design/issues/68)).
 
 1.0.7 (2016-09-20)
+------------------
 
 __Bugfixes__
 
@@ -71,6 +333,7 @@ __Improvements__
 * Automatic tests work for Canada and Thailand
 
 1.0.6 (2016-09-08)
+------------------
 
 __Improvements__
 
@@ -78,6 +341,7 @@ __Improvements__
   framework logic from the observatory measure functions.
 
 1.0.5 (2016-08-12)
+------------------
 
 __Improvements__
 
@@ -85,6 +349,7 @@ __Improvements__
   any HTTP SQL API.
 
 1.0.4 (2016-07-26)
+------------------
 
 __Bugfixes__
 
@@ -93,6 +358,7 @@ __Bugfixes__
   ([#173](https://github.com/CartoDB/observatory-extension/issues/173))
 
 1.0.3 (2016-07-25)
+------------------
 
 __Bugfixes__
 

README.md

@@ -1,64 +1,5 @@
 # Observatory extension
 
-CartoDB extension that implements the row-level functions needed by the Observatory Service.
+## :warning: Deprecated :warning:
 
-## Code organization
+This repository has been deprecated! No further maintenance or development will be done.
-
-```
-.
-├── doc     # documentation
-├── release # released versions
-└── src     # source code
-    └── pg
-        ├── sql
-        └── test
-            ├── expected
-            ├── fixtures
-            └── sql
-```
-
-# Development workflow
-
-We distinguish two roles regarding the development cycle:
-
-* *developers* will implement new functionality and bugfixes into
-  the codebase and will request for new releases of the extension.
-* A *release manager* will attend these requests and will handle
-  the release process. The release process is sequential:
-  no concurrent releases will ever be in the works.
-
-We use the default `develop` branch as the basis for development.
-The `master` branch is used to merge and tag releases to be
-deployed in production.
-
-Developers shall create a new topic branch from `develop` for any new feature
-or bugfix and commit their changes to it and eventually merge back into
-the `develop` branch. When a new release is required a Pull Request
-will be open against the `develop` branch.
-
-The `develop` pull requests will be handled by the release manage,
-who will merge into master where new releases are prepared and tagged.
-The `master` branch is the sole responsibility of the release masters
-and developers must not commit or merge into it.
-
-## Development Guidelines
-
-For a detailed description of the development process please see
-the [CONTRIBUTING.md](CONTRIBUTING.md) guide.
-
-Any modification to the source code
-shall always be done in a topic branch created from the `develop` branch.
-
-Tests, documentation and peer code reviews are required for all
-modifications.
-
-The tests are executed by running this from the top directory:
-```
-sudo make install
-make test
-```
-## Release
-
-The release and deployment process is described in the
-[RELEASE.md](RELEASE.md) guide and it is the responsibility of the designated
-release manager.

carto-package.json

@@ -0,0 +1,9 @@
+{
+  "name": "observatory-server-extension",
+  "current_version": {
+    "requires": {
+      "postgresql": "^10.0.0",
+      "postgis": "^2.4.0.0"
+    }
+  }
+}

doc/boundary_functions.md

@@ -2,9 +2,9 @@
 
 Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
-## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
+## OBS_GetBoundariesByGeometry(geom geometry, geometry_id text)
 
 The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a set of boundary geometries that intersect a supplied geometry. This can be used to find all boundaries that are within or overlap a bounding box. You have the ability to choose whether to retrieve all boundaries that intersect your supplied bounding box or only those that fall entirely inside of your bounding box.
 
@@ -12,7 +12,7 @@ The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a se
 
 Name |Description
 --- | ---
-polygon | a bounding box or other WGS84 geometry
+geom | a WGS84 geometry
 geometry_id | a string identifier for a boundary geometry
 timespan (optional) | year(s) to request from ('NULL' (default) gives most recent)
 overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
@@ -26,7 +26,7 @@ Column Name | Description
 the_geom | a boundary geometry (e.g., US Census tract boundaries)
 geom_refs | a string identifier for the geometry (e.g., geoids of US Census tracts)
 
-If geometries are not found for the requested `polygon`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
+If geometries are not found for the requested `geom`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
 
 #### Example
 
@@ -44,7 +44,6 @@ FROM OBS_GetBoundariesByGeometry(
 
 #### Errors
 
-* If a geometry other than a point is passed as the first argument, an error is thrown: `Invalid geometry type (ST_Polygon), expecting 'ST_Point'`
 * If an `overlap_type` other than the valid ones listed above is entered, then an error is thrown
 
 ## OBS_GetPointsByGeometry(polygon geometry, geometry_id text)
@@ -124,7 +123,7 @@ SET the_geom = OBS_GetBoundary(the_geom, 'us.census.tiger.block_group')
 
 ## OBS_GetBoundaryId(point_geometry, boundary_id)
 
-The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID Glossary](http://docs/carto-engine/data/glossary/#boundary-ids). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
+The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
 
 #### Arguments
 

doc/discovery_functions.md

@@ -56,3 +56,310 @@ time_span | the timespan attached the boundary. this does not mean that the boun
 ```SQL
 SELECT * FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.7, -73.9))
 ```
+
+## OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan)
+
+Return available numerators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the numerator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only numerators for which all of these apply are returned  `NULL` to ignore (optional)
+denom_id | Text | the ID of a denominator to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+numer_id | Text | The ID of the numerator
+numer_name | Text | A human readable name for the numerator
+numer_description | Text | Description of the numerator.  Is sometimes NULL
+numer_weight | Numeric | Numeric "weight" of the numerator.  Ignored.
+numer_license | Text | ID of the license for the numerator
+numer_source | Text | ID of the source for the numerator
+numer_type | Text | Postgres type of the numerator
+numer_aggregate | Text | Aggregate type of the numerator.  If `'SUM'`, this can be normalized by area
+numer_extra | JSONB | Extra information about the numerator column.  Ignored.
+numer_tags | Text[] | Array of all tags applying to this numerator
+valid_denom | Boolean | True if the `denom_id` argument is a valid denominator for this numerator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this numerator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this numerator, False otherwise
+
+#### Examples
+
+Obtain all numerators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326))
+```
+
+Obtain all numerators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+employment related for the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+related to both employment and age & gender for the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment, subsection/tags.age_gender}');
+```
+
+Obtain all numerators that work with US population (`us.census.acs.B01003001`)
+as a denominator.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_denom IS True;
+```
+
+Obtain all numerators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all numerators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan)
+
+Return available denominators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the denominator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only denominators for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+geom_id | Text | the ID of a geometry to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+denom_id | Text | The ID of the denominator
+denom_name | Text | A human readable name for the denominator
+denom_description | Text | Description of the denominator.  Is sometimes NULL
+denom_weight | Numeric | Numeric "weight" of the denominator.  Ignored.
+denom_license | Text | ID of the license for the denominator
+denom_source | Text | ID of the source for the denominator
+denom_type | Text | Postgres type of the denominator
+denom_aggregate | Text | Aggregate type of the denominator.  If `'SUM'`, this can be normalized by area
+denom_extra | JSONB | Extra information about the denominator column.  Ignored.
+denom_tags | Text[] | Array of all tags applying to this denominator
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this denominator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this denominator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this denominator, False otherwise
+
+#### Examples
+
+Obtain all denominators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all denominators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all denominators for male population (`us.census.acs.B01001002`).
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01001002')
+WHERE valid_numer IS True;
+```
+
+Obtain all denominators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all denominators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan, number_geometries)
+
+Return available geometries within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which must intersect the geometry
+filter_tags | Text[] | a list of filters.  Only geometries for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+timespan | Text | the ID of a timespan to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+number_geometries | Integer | an additional variable that is used to adjust the calculation of the [score](https://carto.com/docs/carto-engine/data/discovery-functions/#returns-4) (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+geom_id | Text | The ID of the geometry
+geom_name | Text | A human readable name for the geometry
+geom_description | Text | Description of the geometry.  Is sometimes NULL
+geom_weight | Numeric | Numeric "weight" of the geometry.  Ignored.
+geom_aggregate | Text | Aggregate type of the geometry.  Ignored.
+geom_license | Text | ID of the license for the geometry
+geom_source | Text | ID of the source for the geometry
+geom_type | Text | Postgres type of the geometry
+geom_extra | JSONB | Extra information about the geometry column.  Ignored.
+geom_tags | Text[] | Array of all tags applying to this geometry
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this geometry, False otherwise
+valid_denom | Boolean | True if the `geom_id` argument is a valid geometry for this geometry, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this geometry, False otherwise
+score | Numeric | Score between 0 and 100 for this geometry, higher numbers mean that this geometry is a better choice for the passed extent
+numtiles | Numeric | How many raster tiles were read for score, numgeoms, and percentfill estimates
+numgeoms | Numeric | About how many of these geometries fit inside the passed extent
+percentfill | Numeric | About what percentage of the passed extent is filled with these geometries
+estnumgeoms | Numeric | Ignored
+meanmediansize | Numeric | Ignored
+
+#### Examples
+
+Obtain all geometries that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all geometries that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all geometries that work with total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all geometries with timespan  `2015`.
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id)
+
+Return available timespans within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the timespan's data must intersect with
+filter_tags | Text[] | a list of filters.  Ignore
+numer_id | Text | the ID of a numerator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+timespan_id | Text | The ID of the timespan
+timespan_name | Text | A human readable name for the timespan
+timespan_description | Text | Ignored
+timespan_weight | Numeric | Ignored
+timespan_aggregate | Text | Ignored
+timespan_license | Text | Ignored
+timespan_source | Text | Ignored
+timespan_type | Text | Ignored
+timespan_extra | JSONB | Ignored
+timespan_tags | JSONB | Ignored
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this timespan, False otherwise
+valid_denom | Boolean | True if the `timespan` argument is a valid timespan for this timespan, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this timespan, False otherwise
+
+#### Examples
+
+Obtain all timespans that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all timespans for total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all timespans that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```

doc/measures_functions.md

@@ -2,21 +2,21 @@
 
 [Data Observatory Measures](https://carto.com/docs/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
-There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
+There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
-You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 
-The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
+The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
 
 #### Arguments
 
 Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
-measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
+measure_name | a human-readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -39,7 +39,7 @@ SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
 
 ## OBS_GetUSCensusMeasure(polygon geometry, measure_name text)
 
-The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetUSCensusMeasure``` function below.
+The ```OBS_GetUSCensusMeasure(polygon, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use the ```OBS_GetMeasure``` function below.
 
 #### Arguments
 
@@ -47,7 +47,7 @@ Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
 measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
-normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -70,7 +70,7 @@ SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
 
 ## OBS_GetMeasure(point geometry, measure_id text)
 
-The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
+The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/index.html).
 
 #### Arguments
 
@@ -78,7 +78,7 @@ Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
 measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf). (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html). (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -108,8 +108,8 @@ The ```OBS_GetMeasure(polygon, measure_id)``` function returns any Data Observat
 Name |Description
 --- | ---
 polygon_geometry | a WGS84 polygon geometry (the_geom)
-measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))
-normalize | for measures that are are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
 
@@ -132,7 +132,7 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
 
 #### Errors
 
-* If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
+* If an unrecognized normalization type is input, raises error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
 
 ## OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
 
@@ -143,7 +143,7 @@ The ```OBS_GetMeasureById(geom_ref, measure_id, boundary_id)``` function returns
 Name |Description
 --- | ---
 geom_ref | a geometry reference (e.g., a US Census geoid)
-measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))  
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span (optional) | time span of interest (e.g., 2010 - 2014). If `NULL` is passed, the measure from the most recent data will be used.
 
@@ -170,7 +170,7 @@ SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001'
 
 ## OBS_GetCategory(point geometry, category_id text)
 
-The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) for more detail.
+The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/index.html) for more detail.
 
 #### Arguments
 
@@ -195,3 +195,345 @@ Add the Category to an empty column text column based on point locations in your
 UPDATE tablename
 SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
 ```
+
+## OBS_GetMeta(extent geometry, metadata json, max_timespan_rank, max_score_rank, target_geoms)
+
+The ```OBS_GetMeta(extent, metadata)``` function returns a completed Data
+Observatory metadata JSON Object for use in ```OBS_GetData(geomvals,
+metadata)``` or ```OBS_GetData(ids, metadata)```.  It is not possible to pass
+metadata to those functions if it is not processed by ```OBS_GetMeta(extent,
+metadata)``` first.
+
+`OBS_GetMeta` makes it possible to automatically select appropriate timespans
+and boundaries for the measurement you want.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+extent | A geometry of the extent of the input geometries
+metadata | A JSON array composed of metadata input objects.  Each indicates one desired measure for an output column, and optionally additional parameters about that column
+num_timespan_options | How many historical time periods to include.  Defaults to 1
+num_score_options | How many alternative boundary levels to include.  Defaults to 1
+target_geoms | Target number of geometries.  Boundaries with close to this many objects within `extent` will be ranked highest.
+
+The schema of the metadata input objects are as follows:
+
+Metadata Input Key | Description
+--- | -----------
+numer_id | The identifier for the desired measurement.  If left blank, but a `geom_id` is specified, the column will return a geometry instead of a measurement.
+geom_id | Identifier for a desired geographic boundary level to use when calculating measures.  Will be automatically assigned if undefined.  If defined but `numer_id` is blank, then the column will return a geometry instead of a measurement.
+normalization | The desired normalization.  One of 'area', 'prenormalized', or 'denominated'.  'Area' will normalize the measure per square kilometer, 'prenormalized' will return the original value, and 'denominated' will normalize by a denominator.  Ignored if this metadata object specifies a geometry.
+denom_id | Identifier for a desired normalization column in case `normalization` is 'denominated'.  Will be automatically assigned if necessary.  Ignored if this metadata object specifies a geometry.
+numer_timespan | The desired timespan for the measurement.  Defaults to most recent timespan available if left unspecified.
+geom_timespan | The desired timespan for the geometry.  Defaults to timespan matching numer_timespan if left unspecified.
+target_area | Instead of aiming to have `target_geoms` in the area of the geometry passed as `extent`, fill this area.  Unit is square degrees WGS84.  Set this to `0` if you want to use the smallest source geometry for this element of metadata, for example if you're passing in points.
+target_geoms | Override global `target_geoms` for this element of metadata
+max_timespan_rank | Only include timespans of this recency (for example, `1` is only the most recent timespan). No limit by default
+max_score_rank | Only include boundaries of this relevance (for example, `1` is the most relevant boundary).  Is `1` by default
+
+#### Returns
+
+A JSON array composed of metadata output objects.
+
+Key | Description
+--- | -----------
+meta | A JSON array with completed metadata for the requested data, including all keys below
+
+The schema of the metadata output objects are as follows.  You should pass this
+array as-is to ```OBS_GetData```.  If you modify any values the function will
+fail.
+
+Metadata Output Key | Description
+--- | -----------
+suggested_name | A suggested column name for adding this to an existing table
+numer_id | Identifier for desired measurement
+numer_timespan | Timespan that will be used of the desired measurement
+numer_name | Human-readable name of desired measure
+numer_description | Long human-readable description of the desired measure
+numer_t_description | Further information about the source table
+numer_type | PostgreSQL/PostGIS type of desired measure
+numer_colname | Internal identifier for column name
+numer_tablename | Internal identifier for table
+numer_geomref_colname | Internal identifier for geomref column name
+denom_id | Identifier for desired normalization
+denom_timespan | Timespan that will be used of the desired normalization
+denom_name | Human-readable name of desired measure's normalization
+denom_description | Long human-readable description of the desired measure's normalization
+denom_t_description | Further information about the source table
+denom_type | PostgreSQL/PostGIS type of desired measure's normalization
+denom_colname | Internal identifier for normalization column name
+denom_tablename | Internal identifier for normalization table
+denom_geomref_colname | Internal identifier for normalization geomref column name
+geom_id | Identifier for desired boundary geometry
+geom_timespan | Timespan that will be used of the desired boundary geometry
+geom_name | Human-readable name of desired boundary geometry
+geom_description | Long human-readable description of the desired boundary geometry
+geom_t_description | Further information about the source table
+geom_type | PostgreSQL/PostGIS type of desired boundary geometry
+geom_colname | Internal identifier for boundary geometry column name
+geom_tablename | Internal identifier for boundary geometry table
+geom_geomref_colname | Internal identifier for boundary geometry ref column name
+timespan_rank | Ranking of this measurement by time, most recent is 1, second most recent 2, etc.
+score | The score of this measurement's boundary compared to the `extent` and `target_geoms` passed in.  Between 0 and 100.
+score_rank | The ranking of this measurement's boundary, highest ranked is 1, second is 2, etc.
+numer_aggregate | The aggregate type of the numerator, either `sum`, `average`, `median`, or blank
+denom_aggregate | The aggregate type of the denominator, either `sum`, `average`, `median`, or blank
+normalization | The sort of normalization that will be used for this measure, either `area`, `predenominated`, or `denominated`
+
+#### Examples
+
+Obtain metadata that can augment with one additional column of US population
+data, using a boundary relevant for the geometry provided and latest timespan.
+Limit to only the most recent column most relevant to the extent & density of
+input geometries in `tablename`.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with one additional column of US population
+data, using census tract boundaries.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with two additional columns, one for total
+population and one for male population.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+## OBS_MetadataValidation(extent geometry, geometry_type text, metadata json, target_geoms)
+
+The ```OBS_MetadataValidation``` function performs a validation check over the known issues using the extent, type of geometry, and metadata that is being used in the ```OBS_GetMeta``` function.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+extent | A geometry of the extent of the input geometries
+geometry_type | The geometry type of the source data
+metadata | A JSON array composed of metadata input objects. Each indicates one desired measure for an output column, and optional additional parameters about that column
+target_geoms | Target number of geometries. Boundaries with close to this many objects within `extent` will be ranked highest
+
+The schema of the metadata input objects are as follows:
+
+Metadata Input Key | Description
+--- | -----------
+numer_id | The identifier for the desired measurement. If left blank, a `geom_id` is specified and the column returns a geometry, instead of a measurement
+geom_id | Identifier for a desired geographic boundary level used to calculate measures. If undefined, this is automatically assigned. If defined, `numer_id` is blank and the column returns a geometry, instead of a measurement
+normalization | The desired normalization. One of 'area', 'prenormalized', or 'denominated'.  'Area' will normalize the measure per square kilometer, 'prenormalized' will return the original value, and 'denominated' will normalize by a denominator. If the metadata object specifies a geometry, this is ignored
+denom_id | When `normalization` is 'denominated', this is the identifier for a desired normalization column. This is automatically assigned. If the metadata object specifies a geometry, this is ignored
+numer_timespan | The desired timespan for the measurement. If left unspecified, it defaults to the most recent timespan available
+geom_timespan | The desired timespan for the geometry. If left unspecified, it defaults to the timespan matching `numer_timespan`
+target_area | Instead of aiming to have `target_geoms` in the area of the geometry passed as `extent`, fill this area. Unit is square degrees WGS84. Set this to `0` if you want to use the smallest source geometry for this element of metadata. For example, if you are passing in points
+target_geoms | Override global `target_geoms` for this element of metadata
+max_timespan_rank | Only include timespans of this recency (For example, `1` is only the most recent timespan). There is no limit by default
+max_score_rank | Only include boundaries of this relevance (for example, `1` is the most relevant boundary). The default is `1`
+
+#### Returns
+
+Key | Description
+--- | -----------
+valid | A boolean field that represents if the validation was successful or not
+errors | A text array with all possible errors
+
+#### Examples
+
+Validate metadata with two additional columns of US census data; using a boundary relevant for the geometry provided and the latest timespan. Limited to the most recent column, and the most relevant, based on the extent and density of input geometries in `tablename`.
+
+```SQL
+SELECT OBS_MetadataValidation(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  ST_GeometryType(the_geom),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  COUNT(*)::INTEGER
+) FROM tablename
+GROUP BY ST_GeometryType(the_geom)
+```
+
+## OBS_GetData(geomvals array[geomval], metadata json)
+
+The ```OBS_GetData(geomvals, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every Geometry of
+the `geomval` element in the `geomvals` array. The metadata argument must be
+obtained from ```OBS_GetMeta(extent, metadata)```.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+geomvals | An array of `geomval` elements, which are obtained by casting together a `Geometry` and a `Numeric`.  This should be obtained by using `ARRAY_AGG((the_geom, cartodb_id)::geomval)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `geomvals`
+array corresponds to one row:
+
+Column | Type    | Description
+------ | ----    | -----------
+id     | Numeric | ID corresponding to the `val` component of an element of the input `geomvals` array
+data   | JSON    | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every geometry in a table, keyed by cartodb_id:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename)
+SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with a blank numeric column called `pop_density` with population
+densities:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename),
+data AS (
+  SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+Update a table with two measurements at once, population density and household
+density.  The table should already have a Numeric column `pop_density` and
+`household_density`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom),4326),
+    '[{"numer_id": "us.census.acs.B01003001"},{"numer_id": "us.census.acs.B11001001"}]',
+    1, 1, COUNT(*)
+) meta from tablename),
+data AS (
+  SELECT id,
+     data->0->>'value' AS pop_density,
+     data->1->>'value' AS household_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density,
+    household_density = data.household_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+## OBS_GetData(ids array[text], metadata json)
+
+The ```OBS_GetData(ids, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every id of
+the `ids` array. The metadata argument must be obtained from
+`OBS_GetMeta(extent, metadata)`.  When obtaining metadata, one must include
+the `geom_id` corresponding to the boundary that the `ids` refer to.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+ids | An array of `TEXT` elements.  This should be obtained by using `ARRAY_AGG(col_of_geom_refs)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+For this function to work, the `metadata` argument must include a `geom_id`
+that corresponds to the ids found in `col_of_geom_refs`.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `ids` array
+corresponds to one row:
+
+Column | Type | Description
+------ | ---- | -----------
+id     | Text | ID corresponding to an element of the input `ids` array
+data   | JSON | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every row of a table with FIPS code county IDs
+(USA).
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename)
+SELECT id AS fips, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG(fips) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with population densities for every FIPS code county ID (USA).
+This table has a blank column called `pop_density` and fips codes stored in a
+column `fips`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename),
+data as (
+  SELECT id AS fips, (data->0->>'value') AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG(fips) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE fips = data.id
+```

docs/guides/01-overview.md

@@ -0,0 +1,8 @@
+## Overview
+
+Quick reference guides for learning how to use the Data Observatory features.
+
+- [Data discovery guide](https://carto.com/developers/cartoframes/guides/Data-discovery/)
+- [Data enrichment guide](https://carto.com/developers/cartoframes/guides/Data-enrichment/)
+
+Play with [real examples](https://carto.com/developers/cartoframes/examples/#example-data-observatory).

docs/reference/01-introduction.md

@@ -0,0 +1,5 @@
+## Introduction
+
+Browse the interactive API documentation to search for specific Data Observatory methods, arguments, and sample code that can be used to build your applications.
+
+[Check the reference](https://carto.com/developers/cartoframes/reference/#heading-Data-Observatory).

docs/v1/examples/01-measures-functions.md

@@ -0,0 +1,185 @@
+
+## Measures functions examples
+
+- Add a measure to an empty numeric column based on point locations in your table.
+
+```SQL
+UPDATE tablename
+SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
+
+
+- Add a measure to an empty numeric column based on polygons in your table
+
+```SQL
+UPDATE tablename
+SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
+```
+
+- Add a measure to an empty numeric column based on point locations in your table
+
+```SQL
+UPDATE tablename
+SET median_home_value_sqft = OBS_GetMeasure(the_geom, 'us.zillow.AllHomes_MedianValuePerSqft')
+```
+
+
+- Add a measure to an empty column based on polygons in your table
+
+```SQL
+UPDATE tablename
+SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
+```
+
+
+- Add the Category to an empty column text column based on point locations in your table
+
+```SQL
+UPDATE tablename
+SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
+```
+
+
+- Obtain metadata that can augment with one additional column of US population
+data, using a boundary relevant for the geometry provided and latest timespan.
+Limit to only the most recent column most relevant to the extent & density of
+input geometries in `tablename`.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+- Obtain metadata that can augment with one additional column of US population
+data, using census tract boundaries.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+- Obtain metadata that can augment with two additional columns, one for total
+population and one for male population.
+
+```SQL
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+
+- Validate metadata with two additional columns of US census data; using a boundary relevant for the geometry provided and the latest timespan. Limited to the most recent column, and the most relevant, based on the extent and density of input geometries in `tablename`.
+
+```SQL
+SELECT OBS_MetadataValidation(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  ST_GeometryType(the_geom),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  COUNT(*)::INTEGER
+) FROM tablename
+GROUP BY ST_GeometryType(the_geom)
+```
+
+
+- Obtain population densities for every geometry in a table, keyed by cartodb_id:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename)
+SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+- Update a table with a blank numeric column called `pop_density` with population
+densities:
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename),
+data AS (
+  SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+- Update a table with two measurements at once, population density and household
+density.  The table should already have a Numeric column `pop_density` and
+`household_density`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom),4326),
+    '[{"numer_id": "us.census.acs.B01003001"},{"numer_id": "us.census.acs.B11001001"}]',
+    1, 1, COUNT(*)
+) meta from tablename),
+data AS (
+  SELECT id,
+     data->0->>'value' AS pop_density,
+     data->1->>'value' AS household_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density,
+    household_density = data.household_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+
+- Obtain population densities for every row of a table with FIPS code county IDs
+(USA).
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename)
+SELECT id AS fips, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG(fips) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+- Update a table with population densities for every FIPS code county ID (USA).
+This table has a blank column called `pop_density` and fips codes stored in a
+column `fips`.
+
+```SQL
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename),
+data as (
+  SELECT id AS fips, (data->0->>'value') AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG(fips) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE fips = data.id
+```

docs/v1/examples/02-boundary-functions.md

@@ -0,0 +1,76 @@
+- Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geom_refs` (text).
+
+```sql
+INSERT INTO manhattan_census_tracts(the_geom, geom_refs)
+SELECT *
+FROM OBS_GetBoundariesByGeometry(
+       ST_MakeEnvelope(-74.0251922607,40.6945658517,
+                      -73.9651107788,40.7377626342,
+                      4326),
+       'us.census.tiger.census_tract')
+```
+
+- Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_tract_points` which has columns `the_geom` (geometry) and `geom_refs` (text).
+
+```sql
+INSERT INTO manhattan_tract_points (the_geom, geom_refs)
+SELECT *
+FROM OBS_GetPointsByGeometry(
+       ST_MakeEnvelope(-74.0251922607,40.6945658517,
+                       -73.9651107788,40.7377626342,
+                       4326),
+       'us.census.tiger.census_tract')
+```
+
+
+- Overwrite a point geometry with a boundary geometry that contains it in your table
+
+```SQL
+UPDATE tablename
+SET the_geom = OBS_GetBoundary(the_geom, 'us.census.tiger.block_group')
+```
+
+
+- Write the US Census block group geoid that contains the point geometry for every row as a new column in your table.
+
+```SQL
+UPDATE tablename
+SET geometry_id = OBS_GetBoundaryId(the_geom, 'us.census.tiger.block_group')
+```
+
+
+- Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to and insert into a table called, `overlapping_polygons`. This is a useful method for creating new choropleths of aggregate data.
+
+```SQL
+INSERT INTO overlapping_polygons (the_geom, geometry_id, point_count)
+SELECT
+  OBS_GetBoundaryById(geometry_id, 'us.census.tiger.county') As the_geom,
+  geometry_id,
+  count(*)
+FROM tablename
+GROUP BY geometry_id
+```
+
+
+- Insert into table `denver_census_tracts` the census tract boundaries and geom_refs of census tracts which intersect within 10 miles of downtown Denver, Colorado.
+
+```sql
+INSERT INTO denver_census_tracts(the_geom, geom_refs)
+SELECT *
+FROM OBS_GetBoundariesByPointAndRadius(
+  CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado
+  10000 * 1.609, -- 10 miles (10km * conversion to miles)
+  'us.census.tiger.census_tract')
+```
+
+
+- Insert into table `denver_tract_points` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA.
+
+```sql
+INSERT INTO denver_tract_points(the_geom, geom_refs)
+SELECT *
+FROM OBS_GetPointsByPointAndRadius(
+  CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado
+  10000 * 1.609, -- 10 miles (10km * conversion to miles)
+  'us.census.tiger.census_tract')
+```

docs/v1/examples/03-discovery-functions.md

@@ -0,0 +1,160 @@
+
+```SQL
+SELECT * FROM OBS_Search('home value')
+```
+
+```SQL
+SELECT * FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.7, -73.9))
+```
+
+- Obtain all numerators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326))
+```
+
+- Obtain all numerators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+- Obtain all numerators that are available within a small rectangle and are
+employment related for the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment}');
+```
+
+- Obtain all numerators that are available within a small rectangle and are
+related to both employment and age & gender for the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment, subsection/tags.age_gender}');
+```
+
+- Obtain all numerators that work with US population (`us.census.acs.B01003001`)
+as a denominator.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_denom IS True;
+```
+
+- Obtain all numerators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+- Obtain all numerators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+- Obtain all denominators that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+- Obtain all denominators that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+- Obtain all denominators for male population (`us.census.acs.B01001002`).
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01001002')
+WHERE valid_numer IS True;
+```
+
+- Obtain all denominators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+- Obtain all denominators available in the timespan `2011 - 2015`.
+
+```SQL
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+- Obtain all geometries that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+- Obtain all geometries that are available within a small rectangle and are for
+the United States only.
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+- Obtain all geometries that work with total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+- Obtain all geometries with timespan  `2015`.
+
+```SQL
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2015')
+WHERE valid_timespan IS True;
+```
+
+- Obtain all timespans that are available within a small rectangle.
+
+```SQL
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+- Obtain all timespans for total population (`us.census.acs.B01003001`).
+
+```SQL
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+- Obtain all timespans that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```SQL
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```

docs/v1/examples/examples.json

@@ -0,0 +1,107 @@
+{
+    "main": {
+        "file": "import/import-from-database.md"
+    },
+    "categories": [
+        {
+            "title": "Import",
+            "samples": [
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import standard table",
+                    "desc": "Import standard table into your CARTO account.",
+                    "file": "import/import-standard-table.md"
+                },
+                {
+                    "title": "Import sync table",
+                    "desc": "Import sync table into your CARTO account from database.",
+                    "file": "import/import-sync-table.md"
+                },
+                {
+                    "title": "Import sync table as dataset",
+                    "desc": "Import sync table as dataset into your CARTO account.",
+                    "file": "import/import-from-database.md"
+                }
+            ]
+        },
+        {
+            "title": "Export",
+            "samples": [
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                }
+            ]
+        },
+        {
+            "title": "Tables",
+            "samples": [
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                }
+            ]
+        },
+        {
+            "title": "Misc",
+            "samples": [
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                },
+                {
+                    "title": "Import from database",
+                    "desc": "Import data into your CARTO account from database.",
+                    "file": "import/import-from-database.md"
+                }
+            ]
+        }
+    ]
+}

docs/v1/guides/01-overview.md

@@ -0,0 +1,88 @@
+## Overview
+
+For Enterprise account plans, the [Data Observatory](https://carto.com/data) provides access to a searchable catalog of advanced location data, such as census block, population segments, boundaries and so on. A set of SQL functions allow you to augment your own data and broaden your analysis by discovering boundaries and measures of data from this catalog.
+
+This section describes the Data Observatory functions and the type of data that it returns.
+
+### Functions Overview
+
+There are several functions for accessing different categories of data into your visualizations. You can discover and retrieve data by requesting OBS functions from the Data Observatory. These Data Observatory functions are designed for specific, targeted methods of data analysis. The response for these functions are classified into two primary types of data results; measures and boundaries.
+
+- Boundaries are the geospatial boundaries you need to map or aggregate your data. Examples include Country Borders, Zip Code Tabulation Areas, and Counties
+
+- Measures are the various dimensions of information that CARTO can tell you about a place. Examples include, Population, Household Income, and Median Age
+
+Depending on the OBS function, you will get one, or both, types of data in your result. See [Measures and Boundary Data](#measures-and-boundary-results) for details about available data.
+
+#### Measures Functions
+
+Use location-based measures to analyze your data by accessing population and industry measurements at point locations, or within a region or polygon. These include variables for demographic, economic, and other types of information.
+
+- See [Measures Functions]({{ site.dataobservatory_docs }}/reference/#measures-functions) for specific OBS functions
+- Returns Measures data results
+
+#### Boundary Functions
+
+Use global boundaries to analyze your data by accessing multi-scaled geometries for visualizations. Examples include US Block Groups and Census Tracts. These enable you to aggregate your data into geometric polygons. You can also use your own data to query specific boundaries.
+
+- See [Boundary Functions]({{ site.dataobservatory_docs }}/reference/#boundary-functions) for specific OBS functions
+- Returns Boundary data results
+
+#### Discovery Functions
+
+Discovery Functions provide easier ways for you to find Measures and Boundaries of interest in the Data Observatory. The Discovery functions allow you to perform targeted searches for Measures, or use your own data to discover what is available at a given location. As this is a **retrieval tool** of the Data Observatory, the query results do not change your table. The response back displays one or more identifiers as matches to your search criteria. Each unique identifier can _then_ be used as part of other OBS functions to access any of the other Data Observatory functions.
+
+- See [Discovery Functions]({{ site.dataobservatory_docs }}/reference/#discovery-functions) for specific OBS functions
+- Returns Boundary or Measures matches for your data
+
+### Measures and Boundary Results
+
+The response from the Data Observatory functions are classified as either Measures or Boundary. Depending on your OBS function, you will get one, or both, types of data in your result.
+
+#### Measures Data
+
+Measures provide details about local populations, markets, industries and other dimensions. You can search for available Measures using the Discovery functions, or by viewing the Data Catalog. Measures can be requested for Point locations, or can be summarized for Polygons (regions). In general, Point location requests will return raw aggregate values (e.g. Median Rent), or will provide amounts per square kilometer (e.g. Population). The total square kilometers of the area searched will be returned, allowing you to get raw counts, if needed. Alternatively, if you search over a polygon, raw counts will be returned.
+
+The following table indicates where Measures data results are available. Measures can include raw measures and when indicated, can provide geometries.
+
+Data Category | Examples | Type of Data Response | Availability
+--- | ---
+Housing | Vacant Housing Units, Median Rent, Units for Sale, Mortgage Count | Point measurement, Area measurement, With Geo Border | United States
+Income | Median Household Income, Gini Index | Point measurement, Area measurement, With Geo Border | United States
+Education | Students Enrolled in School, Population Completed H.S | Point measurement, Area measurement, With Geo Border | United States
+Languages | Speaks Spanish at Home, Speaks only English at Home | Point measurement, Area measurement, With Geo Border | United States
+Employment | Workers over the Age of 16 | Point measurement, Area measurement, With Geo Border | United States
+Jobs and Workforce | Origin-Destination of Workforce, Job Wages by job type | Point measurement, Area measurement, With Geo Border | United States
+Transportation | Commuters by Public Transportation, Work at Home | Point measurement, Area measurement, With Geo Border | United States
+Race, Age and Gender | Asian Population, Median Age, Job wages by race | Point measurement, Area measurement, With Geo Border | United States, Spain
+Population | Population per Square Kilometer | Point measurement, Area measurement | United States, Spain
+
+#### Boundary Data
+
+The following table indicates where Boundary data results are available.
+
+Boundary Name | Availability
+--- | ---
+Countries | Global
+First-level administrative subdivisions | Global
+Second-level administrative subdivisions | United States
+Zip Code Tabulation Areas (ZCTA) | United States
+Congressional Districts | United States
+Digital Marketing Areas | United States
+Census Public Use Microdata Areas | United States
+Census Tracts |United States
+Census Block Groups | United States
+US Census Blocks | United States
+Disputed Areas | Global
+Marine Area | Global
+Oceans | Global
+Continents | Global
+Timezones | Global
+
+##### Water Clipping Levels
+
+Many geometries come with various degrees of water accuracy (how closely they follow features such as coastlines). Water clipping refers to how the level of accuracy is returned by the Data Observatory. Data results can either include no clip (no water areas are clipped in the geometry), or high clip (coastlines and inland waterways are clipped out of the final geometry). For example, US Census data might only show coastlines as a straight border line, and not as an inland water area. To find out which levels of water clipping are available for Boundary layers, refer to the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html).
+
+**Note:** While high clip water levels may be better for some kinds of maps and analysis, this type of data consumes more account storage space and may be subject to quota limitations.
+
+For details about how to access any of this data, see [Accessing the Data Observatory]({{ site.dataobservatory_docs }}/guides/accessing-the-data-observatory/).

docs/v1/guides/02-accessing-the-data-observatory.md

@@ -0,0 +1,121 @@
+## Accessing the Data Observatory
+
+The workflow for accessing the Data Observatory includes using a SQL query to apply a specific method of data enrichment or analysis to your data. You can access the Data Observatory by applying a custom query in CARTO Builder, or directly through the SQL API.
+
+#### Prerequisites
+
+You must have an Enterprise account and be familiar with using SQL requests.
+
+- The Data Observatory catalog includes data that is managed by CARTO, on a SaaS cloud platform. For Enterprise users, the Data Observatory can be enabled by contacting CARTO.
+
+- A set of Data Observatory functions (prefaced with "OBS" for Observatory), allow you to retrieve boundaries and measures data through a SQL request. These functions should be used with UPDATE and INSERT statements, not SELECT statements, as we are currently not supporting dynamic use of the Data Observatory 
+
+**Tip:** See the recommended [Best Practices](#best-practices) for using the Data Observatory.
+
+### Enrich from Data Observatory
+
+As an alternative to using SQL queries, you can apply the _Enrich from Data Observatory_ ANALYSIS to a selected map layer in CARTO Builder. This enables you add a new column with contextual demographic and economic measures, without having to apply the code yourself. For details, see the [Enrich from Data Observatory Guide](https://carto.com/learn/guides/analysis/enrich-from-data-observatory) in our Learn hub.
+
+### Apply OBS Functions to a Dataset
+
+This procedure describes how to access the Data Observatory functions by applying SQL queries in a selected dataset.
+
+1) Review the [prerequisites](#prerequisites) section before attempting to access any of the Data Observatory functions
+
+2) [View the Data Observatory Catalog](https://cartodb.github.io/bigmetadata/index.html)
+
+An overview for each of the analyzed functions of data appears, and indicates the unique function signature needed to access the catalog item. You can copy the OBS function from the Data Observatory catalog and modify the placeholder parameters shown in curly brackets (e.g. "{table_name}").
+
+3) From _Your datasets_ dashboard in CARTO, click _NEW DATASET_ and _CREATE EMPTY DATASET_. 
+
+This creates an untitled table. You can get population measurements from the Data Observatory to build your dataset and create a map.
+
+4) 	The SQL view is available when you are viewing your dataset in table view (Data View). Click the slider to switch between viewing your data by METADATA (table) to _SQL_ (opens the SQL view). 
+
+5) Apply the OBS function to modify your table. 
+
+For example, the following image displays a SQL query using the Boundary function, [`OBS_GetBoundariesByGeometry(geom geometry, geometry_id text)`](https://carto.com/docs/carto-engine/data/boundary-functions/#obsgetboundariesbygeometrygeom-geometry-geometryid-text) function. The SQL query inserts the boundary data as a single polygon geometry for each row of data. 
+
+![Query OBS Function in empty dataset](../img/obs_getboundary.jpg)
+
+
+**Tip:** Want to insert population data to create a dataset? Replace `{my table name}` with your dataset name, and apply the SQL query:
+
+```sql
+INSERT INTO {my table name} (the_geom, name)
+SELECT * 
+FROM OBS_GetBoundariesByGeometry(
+  st_makeenvelope(-73.97257804870605,40.671134192879286,-73.89052391052246,40.722868115036974, 4326),
+  'us.census.tiger.census_tract'
+) As m(the_geom, geoid);
+```
+
+Another example shows how to get the local male population into your dataset. Before applying the SQL query, click _ADD COLUMN_ to create and name a column to store the [`OBS_GetMeasure`]({{ site.dataobservatory_docs}}/reference/#obsgetmeasurepolygon-geometry-measureid-text) data.
+
+![Query local male population and apply to data](../img/local_male_pop.jpg)
+
+**Tip:** Want to update your dataset to include the local male population from the Data Observatory? Replace `{my table name}` with your dataset name, and apply the SQL query:
+
+```sql
+UPDATE {my table name}
+SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B01001002')
+```
+6) Click _CREATE MAP_ from your dataset, to visualize the Data Observatory results. You can add custom styling, and add widgets to better visualize your data
+
+![Visualize Data Observatory results](../img/visualize_obs_data.jpg)
+
+
+### SQL API and OBS Functions
+
+This procedure describes how to access the Data Observatory functions directly through the SQL API.
+
+1. In order to use the SQL API, you must be [authenticated]({{ site.bdataobservatory_docs }}/guides/authentication/#authentication) using API keys
+
+	**Note:** Review the [prerequisites](#prerequisites) section before attempting to access any of the Data Observatory functions and [view the Data Observatory Catalog](https://cartodb.github.io/bigmetadata/index.html) to identify the OBS function you are looking for.
+
+2. Query the Data Observatory directly with a specified `OBS` function to apply the results (Measures/Boundaries data) to your table, with the INSERT or UPDATE function
+
+```sql
+https://{username}.carto.com/api/v2/sql?q=UPDATE {tablename}
+SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B01001002')&api_key={api_key}
+```
+### Tips
+
+Other useful tips about OBS functions:
+
+- Some Data Observatory functions return geometries, enabling you to apply an UPDATE statement with an OBS function, to update `the_geom` column
+- To include [water clipping levels]({{ site.dataobservatory_docs }}/guides/overview/#water-clipping-levels) as part of your results, append `_clipped` as part of the OBS function. For example: 
+
+```sql
+UPDATE {tablename}
+SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B01001002','area','us.census.tiger.census_tract_clipped')
+```
+
+### Best Practices
+
+The following usage notes are recommended when using the Data Observatory functions in SQL queries:
+
+- It is discouraged to use the SELECT operation with the Data Observatory functions in your map layers. The results may be visible, but CARTO may not support dynamic rendering of the Data Observatory in the future, so your visualizations may break
+
+	The Data Observatory is **recommended** to be used with INSERT or UPDATE operations, for applying analyzed measures and boundaries data to your tables. While SELECT (retrieve) is standard for SQL API requests, be mindful of quota consumption and use INSERT (to insert a new record) or UPDATE (to update an existing record), for best practices. 
+
+	**Exception:** [Discovery Functions]({{ site.dataobservatory_docs }}/guides/overview/#discovery-functions) are the exception. You can use SELECT as these functions are not actually retrieving data, they are retrieving ids that you can use for other functions.
+
+- You can reduce storage space for unneeded geometries and optimize query optimizations by applying the PostGIS [`ST_Simplify`](http://www.postgis.org/docs/ST_Simplify.html) function. For example, you can simplify the `the_geom` for a large table of polygons and reduce the size of them for quicker rendering. For other tips, see the [most commonly used PostGIS functions](https://carto.com/docs/faqs/postgresql-and-postgis/#what-are-the-most-common-postgis-functions) that you can apply with CARTO
+
+- Only point or polygon geometries are supported for OBS functions. If you attempt to apply Measures or Boundary results to line geometries, an error appears
+
+- The Data Observatory is optimal for modifying existing tables with analytical results, not for building new tables of data
+
+	**Exception:** Exceptions apply for the following boundary functions, since they were designed to return multiple responses of geographical identifiers, as opposed to a single geometry. Create an empty dataset and build a new dataset from a SQL query, using any one of these boundary functions.
+
+	- [`OBS_GetBoundariesByGeometry(geom geometry, geometry_id text)`]({{ site.dataobservatory_docs }}/reference/#boundary-functions#obsgetboundariesbygeometrygeom-geometry-geometryid-text)
+	- [`OBS_GetPointsByGeometry(polygon geometry, geometry_id text)`]({{ site.dataobservatory_docs }}/reference/#boundary-functions#obsgetpointsbygeometrypolygon-geometry-geometryid-text)
+	- [`OBS_GetBoundariesByPointAndRadius(point geometry, radius numeric, boundary_id text`]({{ site.dataobservatory_docs }}/reference/#boundary-functions#obsgetboundariesbypointandradiuspoint-geometry-radius-numeric-boundaryid-text)
+	- [`OBS_GetPointsByPointAndRadius(point geometry, radius numeric, boundary_id text`]({{ site.dataobservatory_docs }}/reference/#boundary-functions#obsgetpointsbypointandradiuspoint-geometry-radius-numeric-boundaryid-text)
+
+- For optimal performance, each SQL request should not exceed 100 rows. As an alternative, you can use a [SQL Batch Query](/docs/carto-engine/sql-api/batch-queries) for queries with long-running CPU processing times
+
+### Examples
+
+View our [CARTO Blogs](https://carto.com/blog/categories/product/) for examples that highlight the benefits of using the Data Observatory.

docs/v1/guides/03-glossary.md

@@ -0,0 +1,126 @@
+## Glossary
+
+A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
+
+### Boundary IDs
+
+Boundary Name | Boundary ID | Shoreline Clipped Boundary ID
+--------------------- | --------------------- | ---
+US States  |  us.census.tiger.state  |  us.census.tiger.state_clipped
+US County  |  us.census.tiger.county  |  us.census.tiger.county_clipped
+US Census Zip Code Tabulation Areas  |  us.census.tiger.zcta5  |  us.census.tiger.zcta5_clipped
+US Census Tracts  |  us.census.tiger.census_tract  |  us.census.tiger.census_tract_clipped
+US Elementary School District  |  us.census.tiger.school_district_elementary  |  us.census.tiger.school_district_elementary_clipped
+US Secondary School District  |  us.census.tiger.school_district_secondary  |  us.census.tiger.school_district_secondary_clipped
+US Unified School District  |  us.census.tiger.school_district_unified  |  us.census.tiger.school_district_unified_clipped
+US Congressional Districts  |  us.census.tiger.congressional_district  |  us.census.tiger.congressional_district_clipped
+US Census Blocks  |  us.census.tiger.block  |  us.census.tiger.block_clipped
+US Census Block Groups  |  us.census.tiger.block_group  |  us.census.tiger.block_group_clipped
+US Census PUMAs  |  us.census.tiger.puma  |  us.census.tiger.puma_clipped
+US Incorporated Places  |  us.census.tiger.place  |  us.census.tiger.place_clipped
+ES Sección Censal  |  es.ine.geom  |   none
+Regions (First-level Administrative)  |  whosonfirst.wof_region_geom  |   none
+Continents  |  whosonfirst.wof_continent_geom  |   none
+Countries  |  whosonfirst.wof_country_geom  |  none
+Marine Areas  |  whosonfirst.wof_marinearea_geom  |  none
+Disputed Areas  |  whosonfirst.wof_disputed_geom  |  none
+
+
+
+### OBS_GetUSCensusMeasure Names Table
+
+This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](https://cartodb.github.io/bigmetadata/index.html).
+
+Measure ID | Measure Name | Measure Description
+--------------------- | --------------------- | ---
+us.census.acs.B01002001	  |  	Median Age	  |  	The median age of all people in a given geographic area.
+us.census.acs.B15003021	  |  	Population Completed Associate’s Degree	  |  	The number of people in a geographic area over the age of 25 who obtained a associate’s degree, and did not complete a more advanced degree.
+us.census.acs.B15003022	  |  	Population Completed Bachelor’s Degree	  |  	The number of people in a geographic area over the age of 25 who obtained a bachelor’s degree, and did not complete a more advanced degree.
+us.census.acs.B15003023	  |  	Population Completed Master’s Degree	  |  	The number of people in a geographic area over the age of 25 who obtained a master’s degree, but did not complete a more advanced degree.
+us.census.acs.B14001007	  |  	Students Enrolled in Grades 9 to 12	  |  	The total number of people in each geography currently enrolled in grades 9 through 12 inclusive. This corresponds roughly to high school.
+us.census.acs.B05001006	  |  	Not a U.S. Citizen Population	  |  	The number of people within each geography who indicated that they are not U.S. citizens.
+us.census.acs.B19001012	  |  	Households with income of $60,000 To $74,999	  |  	The number of households in a geographic area whose annual income was between $60,000 and $74,999.
+us.census.acs.B01003001	  |  	Total Population	  |  	The total number of all people living in a given geographic area. This is a very useful catch-all denominator when calculating rates.
+us.census.acs.B01001002	  |  	Male Population	  |  	The number of people within each geography who are male.
+us.census.acs.B01001026	  |  	Female Population	  |  	The number of people within each geography who are female.
+us.census.acs.B03002003	  |  	White Population	  |  	The number of people identifying as white, non-Hispanic in each geography.
+us.census.acs.B03002004	  |  	Black or African American Population	  |  	The number of people identifying as black or African American, non-Hispanic in each geography.
+us.census.acs.B03002006	  |  	Asian Population	  |  	The number of people identifying as Asian, non-Hispanic in each geography.
+us.census.acs.B03002012	  |  	Hispanic Population	  |  	The number of people identifying as Hispanic or Latino in each geography.
+us.census.acs.B03002005	  |  	American Indian and Alaska Native Population	  |  	The number of people identifying as American Indian or Alaska native in each geography.
+us.census.acs.B03002008	  |  	Other Race population	  |  	The number of people identifying as another race in each geography.
+us.census.acs.B03002009	  |  	Two or more races population	  |  	The number of people identifying as two or more races in each geography.
+us.census.acs.B03002002	  |  	Population not Hispanic	  |  	The number of people not identifying as Hispanic or Latino in each geography.
+us.census.acs.B23025001	  |  	Population age 16 and over	  |  	The number of people in each geography who are age 16 or over.
+us.census.acs.B08006001	  |  	Workers over the Age of 16	  |  	The number of people in each geography who work. Workers include those employed at private for-profit companies, the self-employed, government workers and non-profit employees.
+us.census.acs.B08006002	  |  	Commuters by Car, Truck, or Van	  |  	The number of workers age 16 years and over within a geographic area who primarily traveled to work by car, truck or van. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
+us.census.acs.B08006003	  |  	Commuters who drove alone	  |  	The number of workers age 16 years and over within a geographic area who primarily traveled by car driving alone. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
+us.census.acs.B11001001	  |  	Households	  |  	A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
+us.census.acs.B08006004	  |  	Commuters by Carpool	  |  	The number of workers age 16 years and over within a geographic area who primarily traveled to work by carpool. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
+us.census.acs.B08301010	  |  	Commuters by Public Transportation	  |  	The number of workers age 16 years and over within a geographic area who primarily traveled to work by public transportation. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
+us.census.acs.B08006009	  |  	Commuters by Bus	  |  	The number of workers age 16 years and over within a geographic area who primarily traveled to work by bus. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work. This is a subset of workers who commuted by public transport.
+us.census.acs.B08006011	  |  	Commuters by Subway or Elevated	  |  	The number of workers age 16 years and over within a geographic area who primarily traveled to work by subway or elevated train. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work. This is a subset of workers who commuted by public transport.
+us.census.acs.B08006015	  |  	Walked to Work	  |  	The number of workers age 16 years and over within a geographic area who primarily walked to work. This would mean that of any way of getting to work, they travelled the most distance walking.
+us.census.acs.B08006017	  |  	Worked at Home	  |  	The count within a geographical area of workers over the age of 16 who worked at home.
+us.census.acs.B09001001	  |  	Children under 18 Years of Age	  |  	The number of people within each geography who are under 18 years of age.
+us.census.acs.B14001001	  |  	Population 3 Years and Over	  |  	The total number of people in each geography age 3 years and over. This denominator is mostly used to calculate rates of school enrollment.
+us.census.acs.B14001002	  |  	Students Enrolled in School	  |  	The total number of people in each geography currently enrolled at any level of school, from nursery or pre-school to advanced post-graduate education. Only includes those over the age of 3.
+us.census.acs.B14001005	  |  	Students Enrolled in Grades 1 to 4	  |  	The total number of people in each geography currently enrolled in grades 1 through 4 inclusive. This corresponds roughly to elementary school.
+us.census.acs.B14001006	  |  	Students Enrolled in Grades 5 to 8	  |  	The total number of people in each geography currently enrolled in grades 5 through 8 inclusive. This corresponds roughly to middle school.
+us.census.acs.B14001008	  |  	Students Enrolled as Undergraduate in College	  |  	The number of people in a geographic area who are enrolled in college at the undergraduate level. Enrollment refers to being registered or listed as a student in an educational program leading to a college degree. This may be a public school or college, a private school or college.
+us.census.acs.B15003001	  |  	Population 25 Years and Over	  |  	The number of people in a geographic area who are over the age of 25. This is used mostly as a denominator of educational attainment.
+us.census.acs.B15003017	  |  	Population Completed High School	  |  	The number of people in a geographic area over the age of 25 who completed high school, and did not complete a more advanced degree.
+us.census.acs.B15003019	  |  	Population completed less than one year of college, no degree	  |  	The number of people in a geographic area over the age of 25 who attended college for less than one year and no further.
+us.census.acs.B15003020	  |  	Population completed more than one year of college, no degree	  |  	The number of people in a geographic area over the age of 25 who attended college for more than one year but did not obtain a degree.
+us.census.acs.B16001001	  |  	Population 5 Years and Over	  |  	The number of people in a geographic area who are over the age of 5. This is primarily used as a denominator of measures of language spoken at home.
+us.census.acs.B16001002	  |  	Speaks only English at Home	  |  	The number of people in a geographic area over age 5 who speak only English at home.
+us.census.acs.B16001003	  |  	Speaks Spanish at Home	  |  	The number of people in a geographic area over age 5 who speak Spanish at home, possibly in addition to other languages.
+us.census.acs.B17001001	  |  	Population for Whom Poverty Status Determined	  |  	The number of people in each geography who could be identified as either living in poverty or not. This should be used as the denominator when calculating poverty rates, as it excludes people for whom it was not possible to determine poverty.
+us.census.acs.B17001002	  |  	Income In The Past 12 Months Below Poverty Level	  |  	The number of people in a geographic area who are part of a family (which could be just them as an individual) determined to be in poverty following the Office of Management and Budget’s Directive 14. (https://www.census.gov/hhes/povmeas/methodology/ombdir14.html)
+us.census.acs.B08134010	  |  	Number of workers with a commute of over 60 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in over 60 minutes in a geographic area.
+us.census.acs.B12005002	  |  	Never Married	  |  	The number of people in a geographic area who have never been married.
+us.census.acs.B12005005	  |  	Currently married	  |  	The number of people in a geographic area who are currently married.
+us.census.acs.B12005008	  |  	Married but separated	  |  	The number of people in a geographic area who are married but separated.
+us.census.acs.B12005012	  |  	Widowed	  |  	The number of people in a geographic area who are widowed.
+us.census.acs.B12005015	  |  	Divorced	  |  	The number of people in a geographic area who are divorced.
+us.census.acs.B19013001	  |  	Median Household Income in the past 12 Months	  |  	Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc. It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans’ (VA) payments, unemployment and/or worker’s compensation, child support, and alimony.
+us.census.acs.B25001001	  |  	Housing Units	  |  	A count of housing units in each geography. A housing unit is a house, an apartment, a mobile home or trailer, a group of rooms, or a single room occupied as separate living quarters, or if vacant, intended for occupancy as separate living quarters.
+us.census.acs.B25002003	  |  	Vacant Housing Units	  |  	The count of vacant housing units in a geographic area. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
+us.census.acs.B25004002	  |  	Vacant Housing Units for Rent	  |  	The count of vacant housing units in a geographic area that are for rent. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
+us.census.acs.B19001013	  |  	Households with income of $75,000 To $99,999	  |  	The number of households in a geographic area whose annual income was between $75,000 and $99,999.
+us.census.acs.B19001014	  |  	Households with income of $100,000 To $124,999	  |  	The number of households in a geographic area whose annual income was between $100,000 and $124,999.
+us.census.acs.B25004004	  |  	Vacant Housing Units for Sale	  |  	The count of vacant housing units in a geographic area that are for sale. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
+us.census.acs.B25058001	  |  	Median Rent	  |  	The median contract rent within a geographic area. The contract rent is the monthly rent agreed to or contracted for, regardless of any furnishings, utilities, fees, meals, or services that may be included. For vacant units, it is the monthly rent asked for the rental unit at the time of interview.
+us.census.acs.B25071001	  |  	Percent of Household Income Spent on Rent	  |  	Within a geographic area, the median percentage of household income which was spent on gross rent. Gross rent is the amount of the contract rent plus the estimated average monthly cost of utilities (electricity, gas, water, sewer etc.) and fuels (oil, coal, wood, etc.) if these are paid by the renter. Household income is the sum of the income of all people 15 years and older living in the household.
+us.census.acs.B25075025	  |  	Owner-occupied Housing Units valued at $1,000,000 or more.	  |  	The count of owner occupied housing units in a geographic area that are valued at $1,000,000 or more. Value is the respondent’s estimate of how much the property (house and lot, mobile home and lot, or condominium unit) would sell for if it were for sale.
+us.census.acs.B25081002	  |  	Owner-occupied Housing Units with a Mortgage	  |  	The count of housing units within a geographic area that are mortagaged. Mortgage refers to all forms of debt where the property is pledged as security for repayment of the debt, including deeds of trust, trust deed, contracts to purchase, land contracts, junior mortgages, and home equity loans.
+us.census.acs.B23025002	  |  	Population in Labor Force	  |  	The number of people in each geography who are either in the civilian labor force or are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
+us.census.acs.B23025003	  |  	Population in Civilian Labor Force	  |  	The number of civilians 16 years and over in each geography who can be classified as either employed or unemployed below.
+us.census.acs.B08135001	  |  	Aggregate travel time to work	  |  	The total number of minutes every worker over the age of 16 who did not work from home spent spent commuting to work in one day in a geographic area.
+us.census.acs.B19001002	  |  	Households with income less than $10,000	  |  	The number of households in a geographic area whose annual income was less than $10,000.
+us.census.acs.B19001003	  |  	Households with income of $10,000 to $14,999	  |  	The number of households in a geographic area whose annual income was between $10,000 and $14,999.
+us.census.acs.B19001004	  |  	Households with income of $15,000 to $19,999	  |  	The number of households in a geographic area whose annual income was between $15,000 and $19,999.
+us.census.acs.B23025004	  |  	Employed Population	  |  	The number of civilians 16 years old and over in each geography who either (1) were at work, that is, those who did any work at all during the reference week as paid employees, worked in their own business or profession, worked on their own farm, or worked 15 hours or more as unpaid workers on a family farm or in a family business; or (2) were with a job but not at work, that is, those who did not work during the reference week but had jobs or businesses from which they were temporarily absent due to illness, bad weather, industrial dispute, vacation, or other personal reasons. Excluded from the employed are people whose only activity consisted of work around the house or unpaid volunteer work for religious, charitable, and similar organizations; also excluded are all institutionalized people and people on active duty in the United States Armed Forces.
+us.census.acs.B23025005	  |  	Unemployed Population	  |  	The number of civilians in each geography who are 16 years old and over and are classified as unemployed.
+us.census.acs.B23025006	  |  	Population in Armed Forces	  |  	The number of people in each geography who are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
+us.census.acs.B23025007	  |  	Population Not in Labor Force	  |  	The number of people in each geography who are 16 years old and over who are not classified as members of the labor force. This category consists mainly of students, homemakers, retired workers, seasonal workers interviewed in an off season who were not looking for work, institutionalized people, and people doing only incidental unpaid family work.
+us.census.acs.B12005001	  |  	Population 15 Years and Over	  |  	The number of people in a geographic area who are over the age of 15. This is used mostly as a denominator of marital status.
+us.census.acs.B08134001	  |  	Workers age 16 and over who do not work from home	  |  	The number of workers over the age of 16 who do not work from home in a geographic area.
+us.census.acs.B08134002	  |  	Number of workers with less than 10 minute commute	  |  	The number of workers over the age of 16 who do not work from home and commute in less than 10 minutes in a geographic area.
+us.census.acs.B08303004	  |  	Number of workers with a commute between 10 and 14 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 10 and 14 minutes in a geographic area.
+us.census.acs.B08303005	  |  	Number of workers with a commute between 15 and 19 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 15 and 19 minutes in a geographic area.
+us.census.acs.B08303006	  |  	Number of workers with a commute between 20 and 24 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 20 and 24 minutes in a geographic area.
+us.census.acs.B08303007	  |  	Number of workers with a commute between 25 and 29 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 25 and 29 minutes in a geographic area.
+us.census.acs.B08303008	  |  	Number of workers with a commute between 30 and 34 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area.
+us.census.acs.B08134008	  |  	Number of workers with a commute between 35 and 44 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 35 and 44 minutes in a geographic area.
+us.census.acs.B08303011	  |  	Number of workers with a commute between 45 and 59 minutes	  |  	The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area.
+us.census.acs.B19001005	  |  	Households with income of $20,000 To $24,999	  |  	The number of households in a geographic area whose annual income was between $20,000 and $24,999.
+us.census.acs.B19001006	  |  	Households with income of $25,000 To $29,999	  |  	The number of households in a geographic area whose annual income was between $20,000 and $24,999.
+us.census.acs.B19001007	  |  	Households with income of $30,000 To $34,999	  |  	The number of households in a geographic area whose annual income was between $30,000 and $34,999.
+us.census.acs.B19001008	  |  	Households with income of $35,000 To $39,999	  |  	The number of households in a geographic area whose annual income was between $35,000 and $39,999.
+us.census.acs.B19001009	  |  	Households with income of $40,000 To $44,999	  |  	The number of households in a geographic area whose annual income was between $40,000 and $44,999.
+us.census.acs.B19001010	  |  	Households with income of $45,000 To $49,999	  |  	The number of households in a geographic area whose annual income was between $45,000 and $49,999.
+us.census.acs.B19001011	  |  	Households with income of $50,000 To $59,999	  |  	The number of households in a geographic area whose annual income was between $50,000 and $59,999.
+us.census.acs.B19001015	  |  	Households with income of $125,000 To $149,999	  |  	The number of households in a geographic area whose annual income was between $125,000 and $149,999.
+us.census.acs.B19001016	  |  	Households with income of $150,000 To $199,999	  |  	The number of households in a geographic area whose annual income was between $150,000 and $1999,999.
+us.census.acs.B19001017	  |  	Households with income of $200,000 Or More	  |  	The number of households in a geographic area whose annual income was more than $200,000.

docs/v1/reference/01-introduction.md

@@ -0,0 +1,5 @@
+## Introduction
+
+The Data Observatory, available for Enterprise accounts, provides access to a catalog of analyzed data methods, and enables you to apply the results to your own datasets.
+
+The contents described in this document are subject to CARTO's [Terms of Service](https://carto.com/legal/)

docs/v1/reference/02-authentication.md

@@ -0,0 +1,9 @@
+## Authentication
+
+Data Observatory, like any other [CARTO platform's component]({{site.fundamental_docs}}/components/), requires using an API Key. From your CARTO dashboard, click _[Your API keys](https://carto.com/login)_ from the avatar drop-down menu to view your uniquely generated API Key for managing data with CARTO Engine.
+
+![Your API Keys](../img/avatar_do.gif)
+
+Learn more about the [basics of authorization]({{site.fundamental_docs}}/authorization/), or dig into the details of [Auth API]({{site.authapi_docs}}/), if you want to know more about this part of CARTO platform.
+
+The examples in this documentation may include a placeholder for the API Key. Ensure that you modify any placeholder parameters with your own credentials.

docs/v1/reference/03-versioning.md

@@ -0,0 +1,3 @@
+## Versioning
+
+Data Observartory uses [Semantic Versioning](http://semver.org/). View our Github repository to find tags for each [release](https://github.com/CartoDB/observatory-extension/releases).

docs/v1/reference/04-measures-functions.md

@@ -0,0 +1,539 @@
+## Measures Functions
+
+[Data Observatory Measures]({{site.dataobservatory_docs}}/guides/overview/#methods-overview) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
+
+There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/index.html).
+
+You can [access]({{site.dataobservatory_docs}}/guides/overview/accessing-the-data-observatory/) measures through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you]({{site.dataobservatory_docs}}/guides/overview/accessing-the-data-observatory/) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+
+### OBS_GetUSCensusMeasure(point geometry, measure_name text)
+
+The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use measure IDs with the ```OBS_GetMeasure``` function below.
+
+#### Arguments
+
+Name |Description
+--- | ---
+point | a WGS84 point geometry (the_geom)
+measure_name | a human-readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span | time span of interest (e.g., 2010 - 2014)
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw or normalized measure
+
+#### Example
+
+Add a measure to an empty numeric column based on point locations in your table.
+
+```sql
+UPDATE tablename
+SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population')
+```
+
+### OBS_GetUSCensusMeasure(polygon geometry, measure_name text)
+
+The ```OBS_GetUSCensusMeasure(polygon, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory. To access the full list, use the ```OBS_GetMeasure``` function below.
+
+#### Arguments
+
+Name |Description
+--- | ---
+polygon | a WGS84 polygon geometry (the_geom)
+measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span | time span of interest (e.g., 2010 - 2014)
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw or normalized measure
+
+#### Example
+
+Add a measure to an empty numeric column based on polygons in your table
+
+```sql
+UPDATE tablename
+SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population')
+```
+
+### OBS_GetMeasure(point geometry, measure_id text)
+
+The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/index.html).
+
+#### Arguments
+
+Name |Description
+--- | ---
+point | a WGS84 point geometry (the_geom)
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)). It is important to note that these are different than 'measure_name' used in the Census based functions above.
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. The other option is 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html). (optional)
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span | time span of interest (e.g., 2010 - 2014)
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw or normalized measure
+
+#### Example
+
+Add a measure to an empty numeric column based on point locations in your table
+
+```sql
+UPDATE tablename
+SET median_home_value_sqft = OBS_GetMeasure(the_geom, 'us.zillow.AllHomes_MedianValuePerSqft')
+```
+
+### OBS_GetMeasure(polygon geometry, measure_id text)
+
+The ```OBS_GetMeasure(polygon, measure_id)``` function returns any Data Observatory measure calculated within a polygon.
+
+#### Arguments
+
+Name |Description
+--- | ---
+polygon_geometry | a WGS84 polygon geometry (the_geom)
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))
+normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/index.html) (optional)
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span | time span of interest (e.g., 2010 - 2014)
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw or normalized measure
+
+#### Example
+
+Add a measure to an empty column based on polygons in your table
+
+```sql
+UPDATE tablename
+SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001')
+```
+
+#### Errors
+
+* If an unrecognized normalization type is input, raises error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".`
+
+### OBS_GetMeasureById(geom_ref text, measure_id text, boundary_id text)
+
+The ```OBS_GetMeasureById(geom_ref, measure_id, boundary_id)``` function returns any Data Observatory measure that corresponds to the boundary in ```boundary_id``` that has a geometry reference of ```geom_ref```.
+
+#### Arguments
+
+Name |Description
+--- | ---
+geom_ref | a geometry reference (e.g., a US Census geoid)
+measure_id | a measure identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf))
+boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
+time_span (optional) | time span of interest (e.g., 2010 - 2014). If `NULL` is passed, the measure from the most recent data will be used.
+
+#### Returns
+
+A NUMERIC value
+
+Key | Description
+--- | ---
+value | the raw measure associated with `geom_ref`
+
+#### Example
+
+Add a measure to an empty column based on county geoids in your table
+
+```sql
+UPDATE tablename
+SET household_count = OBS_GetMeasureById(geoid_column, 'us.census.acs.B11001001', 'us.census.tiger.county')
+```
+
+#### Errors
+
+* Returns `NULL` if there is a mismatch between the geometry reference and the boundary id such as using the geoid of a county with the boundary of block groups
+
+## OBS_GetCategory(point geometry, category_id text)
+
+The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/index.html) for more detail.
+
+#### Arguments
+
+Name |Description
+--- | ---
+point | a WGS84 point geometry (the_geom)
+category_id | a category identifier from the Data Observatory ([see available measures](https://cartodb.github.io/bigmetadata/observatory.pdf)).
+
+#### Returns
+
+A TEXT value
+
+Key | Description
+--- | ---
+value | a text based category found at the supplied point
+
+#### Example
+
+Add the Category to an empty column text column based on point locations in your table
+
+```sql
+UPDATE tablename
+SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55')
+```
+
+### OBS_GetMeta(extent geometry, metadata json, max_timespan_rank, max_score_rank, target_geoms)
+
+The ```OBS_GetMeta(extent, metadata)``` function returns a completed Data
+Observatory metadata JSON Object for use in ```OBS_GetData(geomvals,
+metadata)``` or ```OBS_GetData(ids, metadata)```.  It is not possible to pass
+metadata to those functions if it is not processed by ```OBS_GetMeta(extent,
+metadata)``` first.
+
+`OBS_GetMeta` makes it possible to automatically select appropriate timespans
+and boundaries for the measurement you want.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+extent | A geometry of the extent of the input geometries
+metadata | A JSON array composed of metadata input objects.  Each indicates one desired measure for an output column, and optionally additional parameters about that column
+num_timespan_options | How many historical time periods to include.  Defaults to 1
+num_score_options | How many alternative boundary levels to include.  Defaults to 1
+target_geoms | Target number of geometries.  Boundaries with close to this many objects within `extent` will be ranked highest.
+
+The schema of the metadata input objects are as follows:
+
+Metadata Input Key | Description
+--- | -----------
+numer_id | The identifier for the desired measurement.  If left blank, but a `geom_id` is specified, the column will return a geometry instead of a measurement.
+geom_id | Identifier for a desired geographic boundary level to use when calculating measures.  Will be automatically assigned if undefined.  If defined but `numer_id` is blank, then the column will return a geometry instead of a measurement.
+normalization | The desired normalization.  One of 'area', 'prenormalized', or 'denominated'.  'Area' will normalize the measure per square kilometer, 'prenormalized' will return the original value, and 'denominated' will normalize by a denominator.  Ignored if this metadata object specifies a geometry.
+denom_id | Identifier for a desired normalization column in case `normalization` is 'denominated'.  Will be automatically assigned if necessary.  Ignored if this metadata object specifies a geometry.
+numer_timespan | The desired timespan for the measurement.  Defaults to most recent timespan available if left unspecified.
+geom_timespan | The desired timespan for the geometry.  Defaults to timespan matching numer_timespan if left unspecified.
+target_area | Instead of aiming to have `target_geoms` in the area of the geometry passed as `extent`, fill this area.  Unit is square degrees WGS84.  Set this to `0` if you want to use the smallest source geometry for this element of metadata, for example if you're passing in points.
+target_geoms | Override global `target_geoms` for this element of metadata
+max_timespan_rank | Only include timespans of this recency (for example, `1` is only the most recent timespan). No limit by default
+max_score_rank | Only include boundaries of this relevance (for example, `1` is the most relevant boundary).  Is `1` by default
+
+#### Returns
+
+A JSON array composed of metadata output objects.
+
+Key | Description
+--- | -----------
+meta | A JSON array with completed metadata for the requested data, including all keys below
+
+The schema of the metadata output objects are as follows.  You should pass this
+array as-is to ```OBS_GetData```.  If you modify any values the function will
+fail.
+
+Metadata Output Key | Description
+--- | -----------
+suggested_name | A suggested column name for adding this to an existing table
+numer_id | Identifier for desired measurement
+numer_timespan | Timespan that will be used of the desired measurement
+numer_name | Human-readable name of desired measure
+numer_description | Long human-readable description of the desired measure
+numer_t_description | Further information about the source table
+numer_type | PostgreSQL/PostGIS type of desired measure
+numer_colname | Internal identifier for column name
+numer_tablename | Internal identifier for table
+numer_geomref_colname | Internal identifier for geomref column name
+denom_id | Identifier for desired normalization
+denom_timespan | Timespan that will be used of the desired normalization
+denom_name | Human-readable name of desired measure's normalization
+denom_description | Long human-readable description of the desired measure's normalization
+denom_t_description | Further information about the source table
+denom_type | PostgreSQL/PostGIS type of desired measure's normalization
+denom_colname | Internal identifier for normalization column name
+denom_tablename | Internal identifier for normalization table
+denom_geomref_colname | Internal identifier for normalization geomref column name
+geom_id | Identifier for desired boundary geometry
+geom_timespan | Timespan that will be used of the desired boundary geometry
+geom_name | Human-readable name of desired boundary geometry
+geom_description | Long human-readable description of the desired boundary geometry
+geom_t_description | Further information about the source table
+geom_type | PostgreSQL/PostGIS type of desired boundary geometry
+geom_colname | Internal identifier for boundary geometry column name
+geom_tablename | Internal identifier for boundary geometry table
+geom_geomref_colname | Internal identifier for boundary geometry ref column name
+timespan_rank | Ranking of this measurement by time, most recent is 1, second most recent 2, etc.
+score | The score of this measurement's boundary compared to the `extent` and `target_geoms` passed in.  Between 0 and 100.
+score_rank | The ranking of this measurement's boundary, highest ranked is 1, second is 2, etc.
+numer_aggregate | The aggregate type of the numerator, either `sum`, `average`, `median`, or blank
+denom_aggregate | The aggregate type of the denominator, either `sum`, `average`, `median`, or blank
+normalization | The sort of normalization that will be used for this measure, either `area`, `predenominated`, or `denominated`
+
+#### Examples
+
+Obtain metadata that can augment with one additional column of US population
+data, using a boundary relevant for the geometry provided and latest timespan.
+Limit to only the most recent column most relevant to the extent & density of
+input geometries in `tablename`.
+
+```sql
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with one additional column of US population
+data, using census tract boundaries.
+
+```sql
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+Obtain metadata that can augment with two additional columns, one for total
+population and one for male population.
+
+```sql
+SELECT OBS_GetMeta(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  1, 1,
+  COUNT(*)
+) FROM tablename
+```
+
+### OBS_MetadataValidation(extent geometry, geometry_type text, metadata json, target_geoms)
+
+The ```OBS_MetadataValidation``` function performs a validation check over the known issues using the extent, type of geometry, and metadata that is being used in the ```OBS_GetMeta``` function.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+extent | A geometry of the extent of the input geometries
+geometry_type | The geometry type of the source data
+metadata | A JSON array composed of metadata input objects. Each indicates one desired measure for an output column, and optional additional parameters about that column
+target_geoms | Target number of geometries. Boundaries with close to this many objects within `extent` will be ranked highest
+
+The schema of the metadata input objects are as follows:
+
+Metadata Input Key | Description
+--- | -----------
+numer_id | The identifier for the desired measurement. If left blank, a `geom_id` is specified and the column returns a geometry, instead of a measurement
+geom_id | Identifier for a desired geographic boundary level used to calculate measures. If undefined, this is automatically assigned. If defined, `numer_id` is blank and the column returns a geometry, instead of a measurement
+normalization | The desired normalization. One of 'area', 'prenormalized', or 'denominated'.  'Area' will normalize the measure per square kilometer, 'prenormalized' will return the original value, and 'denominated' will normalize by a denominator. If the metadata object specifies a geometry, this is ignored
+denom_id | When `normalization` is 'denominated', this is the identifier for a desired normalization column. This is automatically assigned. If the metadata object specifies a geometry, this is ignored
+numer_timespan | The desired timespan for the measurement. If left unspecified, it defaults to the most recent timespan available
+geom_timespan | The desired timespan for the geometry. If left unspecified, it defaults to the timespan matching `numer_timespan`
+target_area | Instead of aiming to have `target_geoms` in the area of the geometry passed as `extent`, fill this area. Unit is square degrees WGS84. Set this to `0` if you want to use the smallest source geometry for this element of metadata. For example, if you are passing in points
+target_geoms | Override global `target_geoms` for this element of metadata
+max_timespan_rank | Only include timespans of this recency (For example, `1` is only the most recent timespan). There is no limit by default
+max_score_rank | Only include boundaries of this relevance (for example, `1` is the most relevant boundary). The default is `1`
+
+#### Returns
+
+Key | Description
+--- | -----------
+valid | A boolean field that represents if the validation was successful or not
+errors | A text array with all possible errors
+
+#### Examples
+
+Validate metadata with two additional columns of US census data; using a boundary relevant for the geometry provided and the latest timespan. Limited to the most recent column, and the most relevant, based on the extent and density of input geometries in `tablename`.
+
+```sql
+SELECT OBS_MetadataValidation(
+  ST_SetSRID(ST_Extent(the_geom), 4326),
+  ST_GeometryType(the_geom),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]',
+  COUNT(*)::INTEGER
+) FROM tablename
+GROUP BY ST_GeometryType(the_geom)
+```
+
+### OBS_GetData(geomvals array[geomval], metadata json)
+
+The ```OBS_GetData(geomvals, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every Geometry of
+the `geomval` element in the `geomvals` array. The metadata argument must be
+obtained from ```OBS_GetMeta(extent, metadata)```.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+geomvals | An array of `geomval` elements, which are obtained by casting together a `Geometry` and a `Numeric`.  This should be obtained by using `ARRAY_AGG((the_geom, cartodb_id)::geomval)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `geomvals`
+array corresponds to one row:
+
+Column | Type    | Description
+------ | ----    | -----------
+id     | Numeric | ID corresponding to the `val` component of an element of the input `geomvals` array
+data   | JSON    | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every geometry in a table, keyed by cartodb_id:
+
+```sql
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename)
+SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with a blank numeric column called `pop_density` with population
+densities:
+
+```sql
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001"}]',
+    1, 1, COUNT(*)
+) meta FROM tablename),
+data AS (
+  SELECT id AS cartodb_id, (data->0->>'value')::Numeric AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+Update a table with two measurements at once, population density and household
+density.  The table should already have a Numeric column `pop_density` and
+`household_density`.
+
+```sql
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom),4326),
+    '[{"numer_id": "us.census.acs.B01003001"},{"numer_id": "us.census.acs.B11001001"}]',
+    1, 1, COUNT(*)
+) meta from tablename),
+data AS (
+  SELECT id,
+     data->0->>'value' AS pop_density,
+     data->1->>'value' AS household_density
+  FROM OBS_GetData((SELECT ARRAY_AGG((the_geom, cartodb_id)::geomval) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density,
+    household_density = data.household_density
+FROM data
+WHERE cartodb_id = data.id
+```
+
+## OBS_GetData(ids array[text], metadata json)
+
+The ```OBS_GetData(ids, metadata)``` function returns a measure and/or
+geometry corresponding to the `metadata` JSON array for each every id of
+the `ids` array. The metadata argument must be obtained from
+`OBS_GetMeta(extent, metadata)`.  When obtaining metadata, one must include
+the `geom_id` corresponding to the boundary that the `ids` refer to.
+
+#### Arguments
+
+Name | Description
+---- | -----------
+ids | An array of `TEXT` elements.  This should be obtained by using `ARRAY_AGG(col_of_geom_refs)` from the CARTO table one wishes to obtain data for.
+metadata | A JSON array composed of metadata output objects from ```OBS_GetMeta(extent, metadata)```.  The schema of the elements of the `metadata` JSON array corresponds to that of the output of ```OBS_GetMeta(extent, metadata)```, and this argument must be obtained from that function in order for the call to be valid.
+
+For this function to work, the `metadata` argument must include a `geom_id`
+that corresponds to the ids found in `col_of_geom_refs`.
+
+#### Returns
+
+A TABLE with the following schema, where each element of the input `ids` array
+corresponds to one row:
+
+Column | Type | Description
+------ | ---- | -----------
+id     | Text | ID corresponding to an element of the input `ids` array
+data   | JSON | A JSON array with elements corresponding to the input `metadata` JSON array
+
+Each `data` object has the following keys:
+
+Key   | Description
+---   | -----------
+value | The value of the measurement or geometry for the geometry corresponding to this row and measurement corresponding to this position in the `metadata` JSON array
+
+To determine the appropriate cast for `value`, one can use the `numer_type`
+or `geom_type` key corresponding to that value in the input `metadata` JSON
+array.
+
+#### Examples
+
+Obtain population densities for every row of a table with FIPS code county IDs
+(USA).
+
+```sql
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename)
+SELECT id AS fips, (data->0->>'value')::Numeric AS pop_density
+FROM OBS_GetData((SELECT ARRAY_AGG(fips) FROM tablename),
+                 (SELECT meta FROM meta))
+```
+
+Update a table with population densities for every FIPS code county ID (USA).
+This table has a blank column called `pop_density` and fips codes stored in a
+column `fips`.
+
+```sql
+WITH meta AS (
+  SELECT OBS_GetMeta(
+    ST_SetSRID(ST_Extent(the_geom), 4326),
+    '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.county"}]'
+) meta FROM tablename),
+data as (
+  SELECT id AS fips, (data->0->>'value') AS pop_density
+  FROM OBS_GetData((SELECT ARRAY_AGG(fips) FROM tablename),
+                   (SELECT meta FROM meta)))
+UPDATE tablename
+SET pop_density = data.pop_density
+FROM data
+WHERE fips = data.id
+```

docs/v1/reference/05-boundary-functions.md

@@ -0,0 +1,273 @@
+## Boundary Functions
+
+Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
+
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through CARTO Builder. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+
+### OBS_GetBoundariesByGeometry(geom geometry, geometry_id text)
+
+The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a set of boundary geometries that intersect a supplied geometry. This can be used to find all boundaries that are within or overlap a bounding box. You have the ability to choose whether to retrieve all boundaries that intersect your supplied bounding box or only those that fall entirely inside of your bounding box.
+
+#### Arguments
+
+Name |Description
+--- | ---
+geom | a WGS84 geometry
+geometry_id | a string identifier for a boundary geometry
+timespan (optional) | year(s) to request from ('NULL' (default) gives most recent)
+overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
+
+#### Returns
+
+A table with the following columns:
+
+Column Name | Description
+--- | ---
+the_geom | a boundary geometry (e.g., US Census tract boundaries)
+geom_refs | a string identifier for the geometry (e.g., geoids of US Census tracts)
+
+If geometries are not found for the requested `geom`, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
+
+#### Example
+
+Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geom_refs` (text).
+
+```sql
+INSERT INTO manhattan_census_tracts(the_geom, geom_refs)
+SELECT *
+FROM OBS_GetBoundariesByGeometry(
+       ST_MakeEnvelope(-74.0251922607,40.6945658517,
+                      -73.9651107788,40.7377626342,
+                      4326),
+       'us.census.tiger.census_tract')
+```
+
+#### Errors
+
+* If an `overlap_type` other than the valid ones listed above is entered, then an error is thrown
+
+## OBS_GetPointsByGeometry(polygon geometry, geometry_id text)
+
+The ```OBS_GetPointsByGeometry(polygon, geometry_id)``` method returns point geometries and their geographical identifiers that intersect (or are contained by) a bounding box polygon and lie on the surface of a boundary corresponding to the boundary with same geographical identifiers (e.g., a point that is on a census tract with the same geoid). This is a useful alternative to ```OBS_GetBoundariesByGeometry``` listed above because it returns much less data for each location.
+
+#### Arguments
+
+Name |Description
+--- | ---
+polygon | a bounding box or other geometry
+geometry_id | a string identifier for a boundary geometry
+timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
+overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
+
+#### Returns
+
+A table with the following columns:
+
+Column Name | Description
+--- | ---
+the_geom | a point geometry on a boundary (e.g., a point that lies on a US Census tract)
+geom_refs| a string identifier for the geometry (e.g., the geoid of a US Census tract)
+
+If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then NULL values are returned.
+
+#### Example
+
+Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_tract_points` which has columns `the_geom` (geometry) and `geom_refs` (text).
+
+```sql
+INSERT INTO manhattan_tract_points (the_geom, geom_refs)
+SELECT *
+FROM OBS_GetPointsByGeometry(
+       ST_MakeEnvelope(-74.0251922607,40.6945658517,
+                       -73.9651107788,40.7377626342,
+                       4326),
+       'us.census.tiger.census_tract')
+```
+
+#### Errors
+
+* If a geometry other than a point is passed as the first argument, an error is thrown: `Invalid geometry type (ST_Point), expecting 'ST_MultiPolygon' or 'ST_Polygon'`
+
+### OBS_GetBoundary(point_geometry, boundary_id)
+
+The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids). This is a useful method for performing aggregations of points.
+
+#### Arguments
+
+Name | Description
+--- | ---
+point_geometry | a WGS84 polygon geometry (the_geom)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
+timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
+
+#### Returns
+
+A boundary geometry. If no value is found at the requested `boundary_id` or `timespan`, a null value is returned.
+
+Value | Description
+--- | ---
+geom | WKB geometry
+
+#### Example
+
+Overwrite a point geometry with a boundary geometry that contains it in your table
+
+```sql
+UPDATE tablename
+SET the_geom = OBS_GetBoundary(the_geom, 'us.census.tiger.block_group')
+```
+
+#### Errors
+
+* If a geometry other than a point is passed, an error is thrown: `Invalid geometry type (ST_Line), expecting 'ST_Point'`
+
+### OBS_GetBoundaryId(point_geometry, boundary_id)
+
+The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
+
+#### Arguments
+
+Name |Description
+--- | ---
+point_geometry | a WGS84 point geometry (the_geom)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
+timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
+
+#### Returns
+
+A TEXT boundary geometry id. If no value is found at the requested `boundary_id` or `timespan`, a null value is returned.
+
+Value | Description
+--- | ---
+geometry_id | a string identifier of a geometry in the Boundaries
+
+#### Example
+
+Write the US Census block group geoid that contains the point geometry for every row as a new column in your table.
+
+```sql
+UPDATE tablename
+SET geometry_id = OBS_GetBoundaryId(the_geom, 'us.census.tiger.block_group')
+```
+
+#### Errors
+
+* If a geometry other than a point is passed, an error is thrown: `Invalid geometry type (ST_Line), expecting 'ST_Point'`
+
+### OBS_GetBoundaryById(geometry_id, boundary_id)
+
+The ```OBS_GetBoundaryById(geometry_id, boundary_id)``` returns the boundary geometry for a unique geometry_id. A geometry_id can be found using the ```OBS_GetBoundaryId(point_geometry, boundary_id)``` method described above.
+
+#### Arguments
+
+Name | Description
+--- | ---
+geometry_id | a string identifier for a Boundary geometry
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
+timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
+
+#### Returns
+
+A boundary geometry. If a geometry is not found for the requested `geometry_id`, `boundary_id`, or `timespan`, then a null value is returned.
+
+Key | Description
+--- | ---
+geom | a WGS84 polygon geometry
+
+#### Example
+
+Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to and insert into a table called, `overlapping_polygons`. This is a useful method for creating new choropleths of aggregate data.
+
+```sql
+INSERT INTO overlapping_polygons (the_geom, geometry_id, point_count)
+SELECT
+  OBS_GetBoundaryById(geometry_id, 'us.census.tiger.county') As the_geom,
+  geometry_id,
+  count(*)
+FROM tablename
+GROUP BY geometry_id
+```
+
+### OBS_GetBoundariesByPointAndRadius(point geometry, radius numeric, boundary_id text)
+
+The ```OBS_GetBoundariesByPointAndRadius(point, radius, boundary_id)``` method returns boundary geometries and their geographical identifiers that intersect (or are contained by) a circle centered on a point with a radius.
+
+#### Arguments
+
+Name |Description
+--- | ---
+point | a WGS84 point geometry
+radius | a radius (in meters) from the center point
+geometry_id | a string identifier for a boundary geometry
+timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
+overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
+
+#### Returns
+
+A table with the following columns:
+
+Column Name | Description
+--- | ---
+the_geom | a boundary geometry (e.g., a US Census tract)
+geom_refs| a string identifier for the geometry (e.g., the geoid of a US Census tract)
+
+If geometries are not found for the requested point and radius, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
+
+#### Example
+
+Insert into table `denver_census_tracts` the census tract boundaries and geom_refs of census tracts which intersect within 10 miles of downtown Denver, Colorado.
+
+```sql
+INSERT INTO denver_census_tracts(the_geom, geom_refs)
+SELECT *
+FROM OBS_GetBoundariesByPointAndRadius(
+  CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado
+  10000 * 1.609, -- 10 miles (10km * conversion to miles)
+  'us.census.tiger.census_tract')
+```
+
+#### Errors
+
+* If a geometry other than a point is passed, an error is thrown. E.g., `Invalid geometry type (ST_Line), expecting 'ST_Point'`
+
+### OBS_GetPointsByPointAndRadius(point geometry, radius numeric, boundary_id text)
+
+The ```OBS_GetPointsByPointAndRadius(point, radius, boundary_id)``` method returns point geometries on boundaries (e.g., a point that lies on a Census tract) and their geographical identifiers that intersect (or are contained by) a circle centered on a point with a radius.
+
+#### Arguments
+
+Name |Description
+--- | ---
+point | a WGS84 point geometry
+radius | radius (in meters)
+geometry_id | a string identifier for a boundary geometry
+timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
+overlap_type (optional) | one of '[intersects](http://postgis.net/docs/manual-2.2/ST_Intersects.html)' (default), '[contains](http://postgis.net/docs/manual-2.2/ST_Contains.html)', or '[within](http://postgis.net/docs/manual-2.2/ST_Within.html)'.
+
+#### Returns
+
+A table with the following columns:
+
+Column Name | Description
+--- | ---
+the_geom | a point geometry (e.g., a point on a US Census tract)
+geom_refs | a string identifier for the geometry (e.g., the geoid of a US Census tract)
+
+If geometries are not found for the requested point and radius, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned.
+
+#### Example
+
+Insert into table `denver_tract_points` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA.
+
+```sql
+INSERT INTO denver_tract_points(the_geom, geom_refs)
+SELECT *
+FROM OBS_GetPointsByPointAndRadius(
+  CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado
+  10000 * 1.609, -- 10 miles (10km * conversion to miles)
+  'us.census.tiger.census_tract')
+```
+
+#### Errors
+
+* If a geometry other than a point is passed, an error is thrown. E.g., `Invalid geometry type (ST_Line), expecting 'ST_Point'`

docs/v1/reference/06-discovery-functions.md

@@ -0,0 +1,365 @@
+## Discovery Functions
+
+If you are using the [discovery methods]({{ site.dataobservatory_docs}}/guides/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary]({{ site.dataobservatory_docs}}/guides/overview/#boundary-data) and [measures]({{ site.dataobservatory_docs}}/guides/overview/#measures-data) data.
+
+### OBS_Search(search_term)
+
+Use arbitrary text to search all available measures
+
+#### Arguments
+
+Name | Description
+--- | ---
+search_term | a string to search for available measures
+boundary_id | a string identifier for a boundary geometry (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Description
+--- | ---
+id | the unique id of the measure for use with the ```OBS_GetMeasure``` function
+name | the human readable name of the measure
+description | a brief description of the measure
+aggregate | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value)
+source | where the data came from (e.g. US Census Bureau)
+
+#### Example
+
+```sql
+SELECT * FROM OBS_Search('home value')
+```
+
+### OBS_GetAvailableBoundaries(point_geometry)
+
+Returns available `boundary_id`s at a given point geometry.
+
+#### Arguments
+
+Name | Description
+--- | ---
+point_geometry |  a WGS84 point geometry (e.g. the_geom)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Description
+--- | ---
+boundary_id | a boundary identifier from the [Boundary ID Glossary]({{ site.dataobservatory_docs}}/guides/glossary/#boundary-ids)
+description | a brief description of the boundary dataset
+time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
+
+#### Example
+
+```sql
+SELECT * FROM OBS_GetAvailableBoundaries(CDB_LatLng(40.7, -73.9))
+```
+
+### OBS_GetAvailableNumerators(bounds, filter_tags, denom_id, geom_id, timespan)
+
+Return available numerators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the numerator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only numerators for which all of these apply are returned  `NULL` to ignore (optional)
+denom_id | Text | the ID of a denominator to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the numerator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+numer_id | Text | The ID of the numerator
+numer_name | Text | A human readable name for the numerator
+numer_description | Text | Description of the numerator.  Is sometimes NULL
+numer_weight | Numeric | Numeric "weight" of the numerator.  Ignored.
+numer_license | Text | ID of the license for the numerator
+numer_source | Text | ID of the source for the numerator
+numer_type | Text | Postgres type of the numerator
+numer_aggregate | Text | Aggregate type of the numerator.  If `'SUM'`, this can be normalized by area
+numer_extra | JSONB | Extra information about the numerator column.  Ignored.
+numer_tags | Text[] | Array of all tags applying to this numerator
+valid_denom | Boolean | True if the `denom_id` argument is a valid denominator for this numerator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this numerator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this numerator, False otherwise
+
+#### Examples
+
+Obtain all numerators that are available within a small rectangle.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326))
+```
+
+Obtain all numerators that are available within a small rectangle and are for
+the United States only.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+employment related for the United States only.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment}');
+```
+
+Obtain all numerators that are available within a small rectangle and are
+related to both employment and age & gender for the United States only.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states, subsection/tags.employment, subsection/tags.age_gender}');
+```
+
+Obtain all numerators that work with US population (`us.census.acs.B01003001`)
+as a denominator.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_denom IS True;
+```
+
+Obtain all numerators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all numerators available in the timespan `2011 - 2015`.
+
+```sql
+SELECT * FROM OBS_GetAvailableNumerators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+### OBS_GetAvailableDenominators(bounds, filter_tags, numer_id, geom_id, timespan)
+
+Return available denominators within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the denominator's data must intersect with
+filter_tags | Text[] | a list of filters.  Only denominators for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+geom_id | Text | the ID of a geometry to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+timespan | Text | the ID of a timespan to check whether the denominator is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+denom_id | Text | The ID of the denominator
+denom_name | Text | A human readable name for the denominator
+denom_description | Text | Description of the denominator.  Is sometimes NULL
+denom_weight | Numeric | Numeric "weight" of the denominator.  Ignored.
+denom_license | Text | ID of the license for the denominator
+denom_source | Text | ID of the source for the denominator
+denom_type | Text | Postgres type of the denominator
+denom_aggregate | Text | Aggregate type of the denominator.  If `'SUM'`, this can be normalized by area
+denom_extra | JSONB | Extra information about the denominator column.  Ignored.
+denom_tags | Text[] | Array of all tags applying to this denominator
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this denominator, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this denominator, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this denominator, False otherwise
+
+#### Examples
+
+Obtain all denominators that are available within a small rectangle.
+
+```sql
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all denominators that are available within a small rectangle and are for
+the United States only.
+
+```sql
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all denominators for male population (`us.census.acs.B01001002`).
+
+```sql
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01001002')
+WHERE valid_numer IS True;
+```
+
+Obtain all denominators that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```sql
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```
+
+Obtain all denominators available in the timespan `2011 - 2015`.
+
+```sql
+SELECT * FROM OBS_GetAvailableDenominators(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2011 - 2015')
+WHERE valid_timespan IS True;
+```
+
+### OBS_GetAvailableGeometries(bounds, filter_tags, numer_id, denom_id, timespan, number_geometries)
+
+Return available geometries within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which must intersect the geometry
+filter_tags | Text[] | a list of filters.  Only geometries for which all of these apply are returned  `NULL` to ignore (optional)
+numer_id | Text | the ID of a numerator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+timespan | Text | the ID of a timespan to check whether the geometry is valid against.  Will not reduce length of returned table, but will change values for `valid_timespan` (optional)
+number_geometries | Integer | an additional variable that is used to adjust the calculation of the [score]({{ site.dataobservatory_docs}}/guides/discovery-functions/#returns-4) (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+geom_id | Text | The ID of the geometry
+geom_name | Text | A human readable name for the geometry
+geom_description | Text | Description of the geometry.  Is sometimes NULL
+geom_weight | Numeric | Numeric "weight" of the geometry.  Ignored.
+geom_aggregate | Text | Aggregate type of the geometry.  Ignored.
+geom_license | Text | ID of the license for the geometry
+geom_source | Text | ID of the source for the geometry
+geom_type | Text | Postgres type of the geometry
+geom_extra | JSONB | Extra information about the geometry column.  Ignored.
+geom_tags | Text[] | Array of all tags applying to this geometry
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this geometry, False otherwise
+valid_denom | Boolean | True if the `geom_id` argument is a valid geometry for this geometry, False otherwise
+valid_timespan | Boolean | True if the `timespan` argument is a valid timespan for this geometry, False otherwise
+score | Numeric | Score between 0 and 100 for this geometry, higher numbers mean that this geometry is a better choice for the passed extent
+numtiles | Numeric | How many raster tiles were read for score, numgeoms, and percentfill estimates
+numgeoms | Numeric | About how many of these geometries fit inside the passed extent
+percentfill | Numeric | About what percentage of the passed extent is filled with these geometries
+estnumgeoms | Numeric | Ignored
+meanmediansize | Numeric | Ignored
+
+#### Examples
+
+Obtain all geometries that are available within a small rectangle.
+
+```sql
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all geometries that are available within a small rectangle and are for
+the United States only.
+
+```sql
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), '{section/tags.united_states}');
+```
+
+Obtain all geometries that work with total population (`us.census.acs.B01003001`).
+
+```sql
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all geometries with timespan  `2015`.
+
+```sql
+SELECT * FROM OBS_GetAvailableGeometries(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, '2015')
+WHERE valid_timespan IS True;
+```
+
+## OBS_GetAvailableTimespans(bounds, filter_tags, numer_id, denom_id, geom_id)
+
+Return available timespans within a boundary and with the specified
+`filter_tags`.
+
+#### Arguments
+
+Name | Type | Description
+--- | --- | ---
+bounds | Geometry(Geometry, 4326) | a geometry which some of the timespan's data must intersect with
+filter_tags | Text[] | a list of filters.  Ignore
+numer_id | Text | the ID of a numerator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_numer` (optional)
+denom_id | Text | the ID of a denominator to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_denom` (optional)
+geom_id | Text | the ID of a geometry to check whether the timespans is valid against.  Will not reduce length of returned table, but will change values for `valid_geom` (optional)
+
+#### Returns
+
+A TABLE containing the following properties
+
+Key | Type | Description
+--- | ---- | -----------
+timespan_id | Text | The ID of the timespan
+timespan_name | Text | A human readable name for the timespan
+timespan_description | Text | Ignored
+timespan_weight | Numeric | Ignored
+timespan_aggregate | Text | Ignored
+timespan_license | Text | Ignored
+timespan_source | Text | Ignored
+timespan_type | Text | Ignored
+timespan_extra | JSONB | Ignored
+timespan_tags | JSONB | Ignored
+valid_numer | Boolean | True if the `numer_id` argument is a valid numerator for this timespan, False otherwise
+valid_denom | Boolean | True if the `timespan` argument is a valid timespan for this timespan, False otherwise
+valid_geom | Boolean | True if the `geom_id` argument is a valid geometry for this timespan, False otherwise
+
+#### Examples
+
+Obtain all timespans that are available within a small rectangle.
+
+```sql
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326));
+```
+
+Obtain all timespans for total population (`us.census.acs.B01003001`).
+
+```sql
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, 'us.census.acs.B01003001')
+WHERE valid_numer IS True;
+```
+
+Obtain all timespans that work with US states (`us.census.tiger.state`)
+as a geometry.
+
+```sql
+SELECT * FROM OBS_GetAvailableTimespans(
+  ST_MakeEnvelope(-74, 41, -73, 40, 4326), NULL, NULL, NULL, 'us.census.tiger.state')
+WHERE valid_geom IS True;
+```

docs/v1/support/01-support-options.md

@@ -0,0 +1,35 @@
+## Support Options
+
+Feeling stuck? There are many ways to find help.
+
+* Ask a question on [GIS StackExchange](https://gis.stackexchange.com/questions/tagged/carto) using the `CARTO` tag.
+* [Report an issue](https://github.com/CartoDB/cartodb.js/issues) in Github.
+* Engine Plan customers have additional access to enterprise-level support through CARTO's support representatives.
+
+If you just want to describe an issue or share an idea, just <a class="typeform-share" href="https://cartohq.typeform.com/to/mH6RRl" data-mode="popup" target="_blank"> send your feedback</a>.
+
+### Issues on Github
+
+If you think you may have found a bug, or if you have a feature request that you would like to share with the CARTO.js team, please [open an issue](https://github.com/cartodb/cartodb.js/issues/new). 
+
+Before opening an issue, review the [contributing guidelines](https://github.com/CartoDB/cartodb.js/blob/develop/CONTRIBUTING.md#filling-a-ticket).
+
+
+### Community support on GIS Stack Exchange
+
+GIS Stack Exchange is the most popular community in the geospatial industry. This is a collaboratively-edited question and answer site for geospatial programmers and technicians. It is a fantastic resource for asking technical questions about developing and maintaining your application.
+
+When posting a new question, please consider the following:
+
+* Read the GIS Stack Exchange [help](https://gis.stackexchange.com/help) and [how to ask](https://gis.stackexchange.com/help/how-to-ask) pages for guidelines and tips about posting questions.
+* Be very clear about your question in the subject. A clear explanation helps those trying to answer your question, as well as those who may be looking for information in the future.
+* Be informative in your post. Details, code snippets, logs, screenshots, etc. help others to understand your problem.
+* Use code that demonstrates the problem. It is very hard to debug errors without sample code to reproduce the problem.
+
+### Engine Plan Customers
+
+Engine Plan customers have additional support options beyond general community support. As per your account Terms of Service, you have access to enterprise-level support through CARTO's support representatives available at [enterprise-support@carto.com](mailto:enterprise-support@carto.com)
+
+In order to speed up the resolution of your issue, provide as much information as possible (even if it is a link from community support). This allows our engineers to investigate your problem as soon as possible.
+
+If you are not yet CARTO customer, browse our [plans & pricing](https://carto.com/pricing/) and find the right plan for you.

docs/v1/support/02-contribute.md

@@ -0,0 +1,36 @@
+## Contribute
+
+CARTO platform is an open-source ecosystem. You can read about the [fundamentals]({{site.fundamental_docs}}/components/) of CARTO architecture and its components.
+We are more than happy to receive your contributions to the code and the documentation as well.
+
+## Filling a ticket
+
+If you want to open a new issue in our repository, please follow these instructions:
+
+1. Descriptive title.
+2. Write a good description, it always helps.
+3. Specify the steps to reproduce the problem.
+4. Try to add an example showing the problem.
+
+## Contributing code
+
+Best part of open source, collaborate in Data Observatory code!. We like hearing from you, so if you have any bug fixed, or a new feature ready to be merged, those are the steps you should follow:
+
+1. Fork the repository.
+2. Create a new branch in your forked repository.
+3. Commit your changes. Add new tests if it is necessary.
+4. Open a pull request.
+5. Any of the maintainers will take a look.
+6. If everything works, it will merged and released \o/.
+
+If you want more detailed information, this [GitHub guide](https://guides.github.com/activities/contributing-to-open-source/) is a must.
+
+## Completing documentation
+
+Data Observatory documentation is located in ```docs/```. That folder is the content that appears in the [Developer Center](http://carto.com/developers/data-observatory/). Just follow the instructions described in [contributing code](#contributing-code) and after accepting your pull request, we will make it appear online :).
+
+**Tip:** A convenient, easy way of proposing changes in documentation is by using the GitHub editor directly on the web. You can easily create a branch with your changes and make a PR from there.
+
+## Submitting contributions
+
+You will need to sign a Contributor License Agreement (CLA) before making a submission. [Learn more here](https://carto.com/contributions).

docs/v1/support/03-license.md

@@ -0,0 +1,32 @@
+## License
+
+The Data Observatory is a collection of data sources with varying licenses and terms of use. We have endeavored to find you data that will work for the broadest set of use-cases. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
+
+_**Legal Note**: The Data Observatory makes use of a variety of third party data and databases (collectively, the “Data”). You acknowledge that the included Data, and the licenses and terms of use, may be amended from time to time. Whenever you use the Data, you agree to the current relevant terms or license. Some Data will require that you provide attribution to the data source. Other Data may be protected by US or international copyright laws, treaties, or conventions. The Data and associated metadata are provided 'as-is', without express or implied warranty of any kind, including, but not limited to, infringement, merchantability and fitness for a particular purpose. CartoDB is not responsible for the accuracy, completeness, timeliness or quality of the Data._
+
+Name  | Terms link 
+-------|---------
+ACS   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
+Australian Bureau of Statistics DataPacks   |  [https://creativecommons.org/licenses/by/2.5/au/](https://creativecommons.org/licenses/by/2.5/au/)
+Bureau of Labor Statistics Quarterly Census of Employment and Wages (QCEW) | [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
+Censo Demográfico of the Instituto Brasileiro de Geografia e Estatística (IBGE) | Statistics are provided by the federal Institute of Applied Economic Research (IPEA), many of which are reproduced from another source. Some series are regularly updated, others are not. Licensing information is similar to CC-BY, allowing copying and reuse, but requiring attribution.<br /><br />[http://www.ipeadata.gov.br/iframe_direitouso.aspx](http://www.ipeadata.gov.br/iframe_direitouso.aspx?width=1009&height=767)
+Consumer Data Research Centre | [http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/](http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/)
+El Instituto Nacional de Estadística (INE) | The National Statistics Institute (INE) of Spain includes data from multiple sources. If you are re-using their data, they explicitly require that you reference them accordingly<br /><br />[http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
+EuroGraphics EuroGlobalMap | [http://www.eurogeographics.org/content/eurogeographics-euroglobalmap-opendata](http://www.eurogeographics.org/content/eurogeographics-euroglobalmap-opendata)<br /><br />This product includes Intellectual Property from European National Mapping and Cadastral Authorities and is licensed on behalf of these by EuroGeographics. Original product is available for free at [www.eurogeographics.org](http://www.eurogeographics.org/). Terms of the license available at [http://www.eurogeographics.org/form/topographic-data-eurogeographics](http://www.eurogeographics.org/form/topographic-data-eurogeographics)
+GeoNames  |  [http://www.geonames.org/](http://www.geonames.org/)
+GeoPlanet  |  [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/)
+Instituto Nacional de Estadística y Geografía  |  The National Statistics and Geography Institute (INEGI) of Mexico requires credit be given to INEGI as an author<br /><br />[http://www.inegi.org.mx/terminos/terminos_info.aspx](http://www.inegi.org.mx/terminos/terminos_info.aspx)
+National Center for Geographic Information (CNIG) | [https://www.cnig.es/propiedadIntelectual.do](https://www.cnig.es/propiedadIntelectual.do)
+National Institute of Statistics and Economic Studies (INSEE) | [http://www.insee.fr/en/service/default.asp?page=rediffusion/copyright.htm](http://www.insee.fr/en/service/default.asp?page=rediffusion/copyright.htm)
+Natural Earth  |  [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/)
+Northern Ireland Statistics and Research Agency | [https://www.nisra.gov.uk/statistics/terms-and-conditions](https://www.nisra.gov.uk/statistics/terms-and-conditions)
+Office for National Statistics (ONS) | [http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/](http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/)
+Quattroshapes  |  [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
+Scotland's Census Data Warehouse by National Records of Scotland | [https://www.nrscotland.gov.uk/copyright-and-disclaimer](https://www.nrscotland.gov.uk/copyright-and-disclaimer)
+Spielman & Singleton  |  [https://www.openicpsr.org/openicpsr/project/100235/version/V5/view](https://www.openicpsr.org/openicpsr/project/100235/version/V5/view)
+Statistics Canada Census of Population 2011 | [http://www.statcan.gc.ca/eng/reference/licence](http://www.statcan.gc.ca/eng/reference/licence)
+Statistics Canada National Household Survey 2011 | [http://www.statcan.gc.ca/eng/reference/licence](http://www.statcan.gc.ca/eng/reference/licence)
+TIGER   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
+Who's on First |  [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License)
+Zetashapes  |  [http://zetashapes.com/license](http://zetashapes.com/license)
+Zillow Home Value Index | This data is "Aggregate Data", per the Zillow Terms of Use<br /><br />[http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm)

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.1.5'
+default_version = '1.9.0'
-requires = 'postgis, postgres_fdw'
+requires = 'postgis'
 superuser = true
 schema = cdb_observatory

scripts/ci/docker-test.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+docker run -e PGHOST=localhost -e PGPORT=5432 -v `pwd`:/srv --entrypoint="/bin/bash" ${1} /srv/scripts/ci/run_tests_docker.sh && \
+    docker ps --filter status=dead --filter status=exited -aq | xargs docker rm -v

scripts/ci/install_postgres.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+
+# echo commands
+set -x
+
+# exit on error
+set -e
+
+dpkg -l | grep postgresql
+
+# Add the PDGD repository
+apt-key adv --keyserver keys.gnupg.net --recv-keys 7FCC7D46ACCC4CF8
+add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
+apt-get update
+
+# Remove those all PgSQL versions except the one we're testing
+PGSQL_VERSIONS=(9.2 9.3 9.4 9.5 9.6 10)
+/etc/init.d/postgresql stop # stop travis default instance
+for V in "${PGSQL_VERSIONS[@]}"; do
+    if [ "$V" != "$PGSQL_VERSION" ]; then
+        apt-get -y remove --purge postgresql-${V} postgresql-client-${V} postgresql-contrib-${V} postgresql-${V}-postgis-2.3-scripts
+    else
+        apt-get -y remove --purge postgresql-${V}-postgis-2.3-scripts
+    fi
+done
+
+apt-get -y autoremove
+
+# Install PostgreSQL
+apt-get -y install postgresql-${PGSQL_VERSION} postgresql-${PGSQL_VERSION}-postgis-${POSTGIS_VERSION} postgresql-server-dev-${PGSQL_VERSION} postgresql-plpython-${PGSQL_VERSION}
+
+# Configure it to accept local connections from postgres
+echo -e "# TYPE  DATABASE        USER            ADDRESS                 METHOD \nlocal   all             postgres                                trust\nlocal   all             all                                     trust\nhost    all             all             127.0.0.1/32            trust" > /etc/postgresql/${PGSQL_VERSION}/main/pg_hba.conf
+
+# Restart PostgreSQL
+/etc/init.d/postgresql restart ${PGSQL_VERSION}
+
+dpkg -l | grep postgresql

scripts/ci/run_tests_docker.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+/etc/init.d/postgresql start
+
+cd /srv
+
+make clean-all
+make install
+
+cd /srv/src/pg
+
+make test || { cat /srv/src/pg/test/regression.diffs; false; }

scripts/generate_fixtures.py

@@ -2,12 +2,18 @@ import os
 import psycopg2
 import subprocess
 
+PGUSER = os.environ.get('PGUSER', 'postgres')
+PGPASSWORD = os.environ.get('PGPASSWORD', '')
+PGHOST=os.environ.get('PGHOST', 'localhost')
+PGPORT=os.environ.get('PGPORT', '5432')
+PGDATABASE=os.environ.get('PGDATABASE', 'postgres')
+
 DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
-    user=os.environ.get('PGUSER', 'postgres'),
+    user=PGUSER,
-    password=os.environ.get('PGPASSWORD', ''),
+    password=PGPASSWORD,
-    host=os.environ.get('PGHOST', 'localhost'),
+    host=PGHOST,
-    port=os.environ.get('PGPORT', '5432'),
+    port=PGPORT,
-    database=os.environ.get('PGDATABASE', 'postgres'),
+    database=PGDATABASE
 ))
 CURSOR = DB_CONN.cursor()
 
@@ -38,30 +44,22 @@ def get_tablename_query(column_id, boundary_id, timespan):
         timespan, give back the current table hash from the data observatory
     """
     return """
-        SELECT t.tablename, geoid_ct.colname colname, t.id table_id
+        SELECT numer_tablename, numer_geomref_colname, numer_tid,
-        FROM observatory.obs_table t,
+               geom_tablename, geom_geomref_colname, geom_tid
-             observatory.obs_column_table geoid_ct,
+        FROM observatory.obs_meta
-             observatory.obs_column_table data_ct
+        WHERE numer_id = '{numer_id}' AND
-        WHERE
+              geom_id = '{geom_id}' AND
-             t.id = geoid_ct.table_id AND
+              numer_timespan = '{numer_timespan}'
-             t.id = data_ct.table_id AND
+    """.format(numer_id=column_id,
-             geoid_ct.column_id =
+               geom_id=boundary_id,
-        (SELECT source_id
+               numer_timespan=timespan)
-         FROM observatory.obs_column_to_column
-         WHERE target_id = '{boundary_id}'
-         AND reltype = 'geom_ref'
-        ) AND
-        data_ct.column_id = '{column_id}' AND
-        timespan = '{timespan}'
-    """.format(column_id=column_id,
-               boundary_id=boundary_id,
-               timespan=timespan)
 
 
 METADATA_TABLES = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
                    'obs_tag', 'obs_column_to_column', 'obs_dump_version', 'obs_meta',
-                   'obs_meta_numer', 'obs_meta_denom', 'obs_meta_geom',
+                   'obs_table_to_table', 'obs_meta_numer', 'obs_meta_denom',
-                   'obs_meta_timespan', 'obs_column_table_tile', ]
+                   'obs_meta_geom', 'obs_meta_timespan', 'obs_meta_geom_numer_timespan',
+                   'obs_column_table_tile', 'obs_column_table_tile_simple']
 
 FIXTURES = [
     ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
@@ -75,7 +73,7 @@ FIXTURES = [
     ('us.census.acs.B05001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.acs.B08006008_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08301010_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006009_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006011_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.acs.B08006015_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
@@ -129,7 +127,6 @@ FIXTURES = [
     ('us.census.acs.B15003022', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B15003023', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19013001', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B19083001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19301001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B25001001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B25002003', 'us.census.tiger.block_group', '2010 - 2014'),
@@ -142,15 +139,6 @@ FIXTURES = [
     ('us.census.acs.B25081002', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B08134001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B08134002', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134003', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134004', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134005', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134006', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134007', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134008', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134009', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08134010', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.acs.B08135001', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001002', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001003', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001004', 'us.census.tiger.block_group', '2010 - 2014'),
@@ -167,24 +155,71 @@ FIXTURES = [
     ('us.census.acs.B19001015', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001016', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B19001017', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
-    ('us.census.tiger.block_group', 'us.census.tiger.block_group', '2014'),
-    ('us.census.tiger.zcta5', 'us.census.tiger.zcta5', '2014'),
-    ('us.census.tiger.county', 'us.census.tiger.county', '2014'),
     ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
     ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.spielman_singleton_segments.X2', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08134010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.spielman_singleton_segments.X31', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.census.spielman_singleton_segments.X55', 'us.census.tiger.census_tract', '2010 - 2014'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
     ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-06'),
-    ('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
+    ('us.census.acs.B01003001', 'us.census.tiger.zcta5', '2010 - 2014'),
-    ('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
-    ('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
-    ('us.census.tiger.census_tract_clipped', 'us.census.tiger.census_tract_clipped', '2014'),
+    ('us.census.tiger.place_geoname', 'us.census.tiger.place_clipped', '2015'),
+    ('us.census.tiger.county_geoname', 'us.census.tiger.county_clipped', '2015'),
+    ('us.census.tiger.county_geoname', 'us.census.tiger.county', '2015'),
+    ('us.census.tiger.block_group_geoname', 'us.census.tiger.block_group', '2015'),
 ]
 
 OUTFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
@@ -199,26 +234,35 @@ def dump(cols, tablename, where=''):
             tablename=tablename,
         ))
 
-    subprocess.check_call('pg_dump -x --section=pre-data -t observatory.{tablename} '
+    subprocess.check_call('PGPASSWORD={pgpassword} PGUSER={pguser} PGHOST={pghost} PGDATABASE={pgdb} '
+                          'pg_dump -x --section=pre-data -t observatory.{tablename} '
                           ' | sed "s:SET search_path.*::" '
-                          ' | sed "s:CREATE TABLE :CREATE TABLE observatory.:" '
                           ' | sed "s:ALTER TABLE.*OWNER.*::" '
+                          ' | sed "s:SET idle_in_transaction_session_timeout.*::" '
                           ' >> {outfile}'.format(
                               tablename=tablename,
                               outfile=OUTFILE_PATH,
+                              pgpassword=PGPASSWORD,
+                              pghost=PGHOST,
+                              pgdb=PGDATABASE,
+                              pguser=PGUSER
                           ), shell=True)
 
     with open(OUTFILE_PATH, 'a') as outfile:
         outfile.write('COPY observatory."{}" FROM stdin WITH CSV HEADER;\n'.format(tablename))
 
     subprocess.check_call('''
-      psql -c "COPY (SELECT {cols} \
+      PGPASSWORD={pgpassword} psql -U {pguser} -d {pgdb} -h {pghost} -c "COPY (SELECT {cols} \
       FROM observatory.{tablename} {where}) \
       TO STDOUT WITH CSV HEADER" >> {outfile}'''.format(
           cols=cols,
           tablename=tablename,
           where=where,
           outfile=OUTFILE_PATH,
+          pgpassword=PGPASSWORD,
+          pghost=PGHOST,
+          pgdb=PGDATABASE,
+          pguser=PGUSER
       ), shell=True)
 
     with open(OUTFILE_PATH, 'a') as outfile:
@@ -233,15 +277,19 @@ def main():
         tablename_query = get_tablename_query(column_id, boundary_id, timespan)
         resp = query(tablename_query).fetchone()
         if resp:
-            tablename, colname, table_id = resp
+            numer_tablename, numer_colname, numer_table_id = resp[0:3]
+            geom_tablename, geom_colname, geom_table_id = resp[3:6]
         else:
-            print("Could not find table for {}, {}, {}".format(
+            raise Exception("Could not find table for {}, {}, {}".format(
                 column_id, boundary_id, timespan))
-            continue
+        numer = (numer_tablename, numer_colname, numer_table_id, )
-        table_colname = (tablename, colname, boundary_id, table_id, )
+        geom = (geom_tablename, geom_colname, geom_table_id, )
-        if table_colname not in unique_tables:
+        if numer not in unique_tables:
-            print(table_colname)
+            print(numer)
-            unique_tables.add(table_colname)
+            unique_tables.add(numer)
+        if geom not in unique_tables:
+            print(geom)
+            unique_tables.add(geom)
 
     print unique_tables
 
@@ -290,13 +338,14 @@ def main():
                 "column_id IN ('{}', '{}')".format(numer_id, geom_id)
                 for numer_id, geom_id, timespan in FIXTURES
             ])
-        elif tablename in ('obs_column_table', 'obs_column_table_tile', ):
+        elif tablename in ('obs_column_table', 'obs_column_table_tile',
-            where = 'WHERE column_id IN ({numer_ids}) ' \
+                           'obs_column_table_tile_simple'):
-                    'OR column_id IN ({geom_ids}) ' \
+            where = '''WHERE table_id IN ({table_ids}) AND
-                    'OR table_id IN ({table_ids}) '.format(
+              (column_id IN ({numer_ids}) OR column_id IN ({geom_ids}))
+                    '''.format(
                         numer_ids=','.join(["'{}'".format(x) for x, _, _ in FIXTURES]),
                         geom_ids=','.join(["'{}'".format(x) for _, x, _ in FIXTURES]),
-                        table_ids=','.join(["'{}'".format(x) for _, _, _, x in unique_tables])
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
                     )
         elif tablename == 'obs_column_to_column':
             where = "WHERE " + " OR ".join([
@@ -308,18 +357,22 @@ def main():
             where = 'WHERE timespan IN ({timespans}) ' \
                     'OR id IN ({table_ids}) '.format(
                         timespans=','.join(["'{}'".format(x) for _, _, x in FIXTURES]),
-                        table_ids=','.join(["'{}'".format(x) for _, _, _, x in unique_tables])
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
+                    )
+        elif tablename in ('obs_table_to_table'):
+            where = '''WHERE source_id IN ({table_ids})'''.format(
+                        table_ids=','.join(["'{}'".format(x) for _, _, x in unique_tables])
                     )
         else:
             where = ''
         dump('*', tablename, where)
 
-    for tablename, colname, boundary_id, table_id in unique_tables:
+    for tablename, colname, table_id in unique_tables:
-        if 'zcta5' in boundary_id:
+        if 'zcta5' in table_id or 'zillow_zip' in table_id:
             where = '\'11%\''
             compare = 'LIKE'
-        elif 'whosonfirst' in boundary_id:
+        elif 'county' in table_id and 'tiger' in table_id:
-            where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
+            where = "('48061', '36047')"
             compare = 'IN'
         else:
             where = '\'36047%\''

scripts/requirements.txt

@@ -1,3 +1,4 @@
 requests
 nose
 nose_parameterized
+psycopg2

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.1.5'
+default_version = '1.9.0'
-requires = 'postgis, postgres_fdw'
+requires = 'postgis'
 superuser = true
 schema = cdb_observatory

src/pg/sql/15_fdw_utilities.sql

@@ -1,67 +0,0 @@
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectRemoteTable(fdw_name text, schema_name text, user_dbname text, user_hostname text, username text, user_tablename text, user_schema text)
-RETURNS void
-AS $$
-DECLARE
-  row record;
-  option record;
-  connection_str json;
-BEGIN
-  -- Build connection string
-  connection_str := '{"server":{"extensions":"postgis", "dbname":"'
-    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"6432"}, "users":{"public"'
-    || ':{"user":"' || username ||'", "password":""} } }';
-
-  -- This function tries to be as idempotent as possible, by not creating anything more than once
-  -- (not even using IF NOT EXIST to avoid throwing warnings)
-  IF NOT EXISTS ( SELECT * FROM pg_extension WHERE extname = 'postgres_fdw') THEN
-    CREATE EXTENSION postgres_fdw;
-  END IF;
-  -- Create FDW first if it does not exist
-  IF NOT EXISTS ( SELECT * FROM pg_foreign_server WHERE srvname = fdw_name)
-    THEN
-    EXECUTE FORMAT('CREATE SERVER %I FOREIGN DATA WRAPPER postgres_fdw', fdw_name);
-  END IF;
-
-  -- Set FDW settings
-  FOR row IN SELECT p.key, p.value from lateral json_each_text(connection_str->'server') p
-    LOOP
-      IF NOT EXISTS (WITH a AS (select split_part(unnest(srvoptions), '=', 1) as options from pg_foreign_server where srvname=fdw_name) SELECT * from a where options = row.key)
-        THEN
-        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (ADD %I %L)', fdw_name, row.key, row.value);
-      ELSE
-        EXECUTE FORMAT('ALTER SERVER %I OPTIONS (SET %I %L)', fdw_name, row.key, row.value);
-      END IF;
-    END LOOP;
-
-    -- Create user mappings
-    FOR row IN SELECT p.key, p.value from lateral json_each(connection_str->'users') p LOOP
-      -- Check if entry on pg_user_mappings exists
-      IF NOT EXISTS ( SELECT * FROM pg_user_mappings WHERE srvname = fdw_name AND usename = row.key ) THEN
-        EXECUTE FORMAT ('CREATE USER MAPPING FOR %I SERVER %I', row.key, fdw_name);
-      END IF;
-
-      -- Update user mapping settings
-      FOR option IN SELECT o.key, o.value from lateral json_each_text(row.value) o LOOP
-        IF NOT EXISTS (WITH a AS (select split_part(unnest(umoptions), '=', 1) as options from pg_user_mappings WHERE srvname = fdw_name AND usename = row.key) SELECT * from a where options = option.key) THEN
-          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (ADD %I %L)', row.key, fdw_name, option.key, option.value);
-        ELSE
-          EXECUTE FORMAT('ALTER USER MAPPING FOR %I SERVER %I OPTIONS (SET %I %L)', row.key, fdw_name, option.key, option.value);
-        END IF;
-      END LOOP;
-    END LOOP;
-
-    -- Create schema if it does not exist.
-    IF NOT EXISTS ( SELECT * from pg_namespace WHERE nspname=fdw_name) THEN
-      EXECUTE FORMAT ('CREATE SCHEMA %I', fdw_name);
-    END IF;
-
-    -- Bring the remote cdb_tablemetadata
-    IF NOT EXISTS ( SELECT * FROM PG_CLASS WHERE relnamespace = (SELECT oid FROM pg_namespace WHERE nspname=fdw_name) and relname='cdb_tablemetadata') THEN
-      EXECUTE FORMAT ('CREATE FOREIGN TABLE %I.cdb_tablemetadata (tabname text, updated_at timestamp with time zone) SERVER %I OPTIONS (table_name ''cdb_tablemetadata_text'', schema_name ''public'', updatable ''false'')', fdw_name, fdw_name);
-    END IF;
-
-    -- Import target table
-    EXECUTE FORMAT ('IMPORT FOREIGN SCHEMA %I LIMIT TO (%I) from SERVER %I INTO %I', user_schema, user_tablename, fdw_name, schema_name);
-
-END;
-$$ LANGUAGE PLPGSQL;

src/pg/sql/40_observatory_utility.sql

@@ -1,3 +1,16 @@
+-- In Postgis 3+, geomval is part of postgis_raster
+-- Trying to workaround it by creating the type ourselves leads to other issues:
+-- - If we create it under public, then if we try to create postgis_raster afterwards it will fail (ERROR:  type "geomval" already exists).
+-- - If we create it under cdb_observatory, then things work until we install postgis_raster. At that moment depending on how
+-- the search_path is set (per call, function, user...) we start getting random errors since it's mixing public.geomval and 
+-- cdb_observatory.geomval (function cdb_observatory.obs_getdata(geomval[], json) does not exist)
+DO $$
+BEGIN
+    IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'geomval') THEN
+        RAISE EXCEPTION 'Missing `geomval` type. Use `CREATE EXTENSION postgis_raster` to enable it.';
+    END IF;
+END$$;
+
 
 -- Returns the table name with geoms for the given geometry_id
 -- TODO probably needs to take in the column_id array to get the relevant
@@ -205,6 +218,18 @@ END;
 $$ LANGUAGE plpgsql;
 
 
+-- Function we can call to raise an exception in the midst of a SQL statement
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_RaiseNotice(
+    message TEXT
+) RETURNS TEXT
+AS $$
+BEGIN
+  RAISE NOTICE '%', message;
+  RETURN NULL;
+END;
+$$ LANGUAGE plpgsql;
+
+
 -- Create a function that always returns the first non-NULL item
 CREATE OR REPLACE FUNCTION cdb_observatory.first_agg ( anyelement, anyelement )
 RETURNS anyelement LANGUAGE SQL IMMUTABLE STRICT AS $$
@@ -220,3 +245,39 @@ CREATE AGGREGATE cdb_observatory.FIRST (
         stype    = anyelement
 );
 
+CREATE OR REPLACE FUNCTION cdb_observatory.isnumeric (
+  typename varchar
+)
+RETURNS BOOLEAN LANGUAGE SQL IMMUTABLE STRICT AS $$
+  SELECT LOWER(typename) IN (
+    'smallint',
+    'integer',
+    'bigint',
+    'decimal',
+    'numeric',
+    'real',
+    'double precision'
+  )
+$$;
+
+-- Attempt to perform intersection, if there's an exception then buffer
+-- https://gis.stackexchange.com/questions/50399/how-best-to-fix-a-non-noded-intersection-problem-in-postgis
+CREATE OR REPLACE FUNCTION cdb_observatory.safe_intersection(
+  geom_a Geometry(Geometry, 4326),
+  geom_b Geometry(Geometry, 4326)
+)
+RETURNS Geometry(Geometry, 4326) AS
+$$
+BEGIN
+    RETURN ST_MakeValid(ST_Intersection(geom_a, geom_b));
+    EXCEPTION
+        WHEN OTHERS THEN
+            BEGIN
+                RETURN ST_MakeValid(ST_Intersection(ST_Buffer(geom_a, 0.0000001), ST_Buffer(geom_b, 0.0000001)));
+                EXCEPTION
+                    WHEN OTHERS THEN
+                        RETURN NULL;
+    END;
+END
+$$
+LANGUAGE 'plpgsql' STABLE STRICT;

src/pg/sql/42_observatory_exploration.sql

@@ -181,6 +181,86 @@ BEGIN
 END
 $$ LANGUAGE plpgsql;
 
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetNumerators(
+  bounds GEOMETRY DEFAULT NULL,
+  section_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  subsection_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  other_tags TEXT[] DEFAULT ARRAY[]::TEXT[],
+  ids TEXT[] DEFAULT ARRAY[]::TEXT[],
+  name TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT '',
+  geom_id TEXT DEFAULT '',
+  timespan TEXT DEFAULT ''
+) RETURNS TABLE (
+  numer_id TEXT,
+  numer_name TEXT,
+  numer_description TEXT,
+  numer_weight NUMERIC,
+  numer_license TEXT,
+  numer_source TEXT,
+  numer_type TEXT,
+  numer_aggregate TEXT,
+  numer_extra JSONB,
+  numer_tags JSONB,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  where_clause_elements TEXT[];
+  geom_clause TEXT;
+  where_clause TEXT;
+BEGIN
+  where_clause_elements := (ARRAY[])::TEXT[];
+  where_clause := '';
+
+  IF bounds IS NOT NULL THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$ST_Intersects(the_geom, '%s'::geometry)$data$, bounds));
+  END IF;
+  IF cardinality(section_tags) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_tags ?| '%s'$data$, section_tags));
+  END IF;
+  IF cardinality(subsection_tags) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_tags ?| '%s'$data$, subsection_tags));
+  END IF;
+  IF cardinality(other_tags) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_tags ?| '%s'$data$, other_tags));
+  END IF;
+  IF cardinality(ids) > 0 THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_id IN (array_to_string('%s'::text[], ','))$data$, ids));
+  END IF;
+  IF name IS NOT NULL AND name != '' THEN
+    where_clause_elements := array_append(where_clause_elements, format($data$numer_name ilike '%%%s%%'$data$, name));
+  END IF;
+  IF cardinality(where_clause_elements) > 0 THEN
+    where_clause := format($clause$WHERE %s$clause$, array_to_string(where_clause_elements, ' AND '));
+  END IF;
+  RAISE DEBUG '%', array_to_string(where_clause_elements, ' AND ');
+
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT numer_id::TEXT,
+           numer_name::TEXT,
+           numer_description::TEXT,
+           numer_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           numer_type numer_type,
+           numer_aggregate numer_aggregate,
+           numer_extra::JSONB numer_extra,
+           numer_tags numer_tags,
+    $1 = ANY(denoms) valid_denom,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_numer
+    %s
+  $string$, where_clause)
+  USING denom_id, geom_id, timespan;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
 CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableDenominators(
   bounds GEOMETRY DEFAULT NULL,
   filter_tags TEXT[] DEFAULT NULL,
@@ -243,7 +323,8 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableGeometries(
   filter_tags TEXT[] DEFAULT NULL,
   numer_id TEXT DEFAULT NULL,
   denom_id TEXT DEFAULT NULL,
-  timespan TEXT DEFAULT NULL
+  timespan TEXT DEFAULT NULL,
+  number_geoms INTEGER DEFAULT NULL
 ) RETURNS TABLE (
   geom_id TEXT,
   geom_name TEXT,
@@ -252,6 +333,9 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableGeometries(
   geom_aggregate TEXT,
   geom_license TEXT,
   geom_source TEXT,
+  geom_type TEXT,
+  geom_extra JSONB,
+  geom_tags JSONB,
   valid_numer BOOLEAN,
   valid_denom BOOLEAN,
   valid_timespan BOOLEAN,
@@ -286,21 +370,37 @@ BEGIN
              NULL::TEXT geom_aggregate,
              NULL::TEXT license,
              NULL::TEXT source,
-      $1 = ANY(numers) valid_numer,
+             geom_type::TEXT,
-      $2 = ANY(denoms) valid_denom,
+             geom_extra::JSONB,
-      $3 = ANY(timespans) valid_timespan
+             geom_tags::JSONB,
-      FROM observatory.obs_meta_geom
+             $1 = ANY(numers) valid_numer,
+             $2 = ANY(denoms) valid_denom,
+             CASE WHEN $3 IS NOT NULL AND $3 != '' THEN
+                -- Here we are looking for geometries with: a) geometry timespan or b) numerators linked to that geometries that fit in the
+                -- timespan passed. For example it look for geometries with timespan '2015 - 2015' or numerators linked to that geometry that has
+                -- '2015 - 2015' as one of the valid timespans.
+                -- If we pass a numerator_id, we filter by that numerator
+                CASE WHEN $1 IS NOT NULL AND $1 != '' THEN
+                  EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND onu.numer_id = $1 AND ($3 = ANY(onu.timespans) OR $3 IN (select(unnest(o.timespans)))))
+                ELSE
+                  EXISTS (SELECT 1 FROM observatory.obs_meta_geom_numer_timespan onu WHERE o.geom_id = onu.geom_id AND ($3 = ANY(onu.geom_timespans) OR $3 IN (select(unnest(o.timespans)))))
+                END
+             ELSE
+              false
+             END as valid_timespan
+      FROM observatory.obs_meta_geom o
       WHERE %s (geom_tags ?& $4 OR CARDINALITY($4) = 0)
     ), scores AS (
-      SELECT * FROM cdb_observatory._OBS_GetGeometryScores($5,
+      SELECT * FROM cdb_observatory._OBS_GetGeometryScores(bounds => $5,
-        (SELECT ARRAY_AGG(geom_id) FROM available_geoms)
+        filter_geom_ids => (SELECT ARRAY_AGG(geom_id) FROM available_geoms),
+        desired_num_geoms => $6::integer
       )
-    ) SELECT available_geoms.*, score, numtiles, notnull_percent, numgeoms,
+    ) SELECT DISTINCT ON (geom_id) available_geoms.*, score, numtiles, notnull_percent, numgeoms,
              percentfill, estnumgeoms, meanmediansize
       FROM available_geoms, scores
-      WHERE available_geoms.geom_id = scores.geom_id
+      WHERE available_geoms.geom_id = scores.column_id
   $string$, geom_clause)
-  USING numer_id, denom_id, timespan, filter_tags, bounds;
+  USING numer_id, denom_id, timespan, filter_tags, bounds, number_geoms;
   RETURN;
 END
 $$ LANGUAGE plpgsql;
@@ -319,6 +419,9 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableTimespans(
   timespan_aggregate TEXT,
   timespan_license TEXT,
   timespan_source TEXT,
+  timespan_type TEXT,
+  timespan_extra JSONB,
+  timespan_tags JSONB,
   valid_numer BOOLEAN,
   valid_denom BOOLEAN,
   valid_geom BOOLEAN
@@ -343,11 +446,14 @@ BEGIN
            timespan_description::TEXT,
            timespan_weight::NUMERIC,
            NULL::TEXT timespan_aggregate,
-           NULL::TEXT license,
+           NULL::TEXT timespan_license,
-           NULL::TEXT source,
+           NULL::TEXT timespan_source,
-    $1 = ANY(numers) valid_numer,
+           timespan_type::TEXT,
-    $2 = ANY(denoms) valid_denom,
+           NULL::JSONB timespan_extra,
-    $3 = ANY(geoms) valid_geom_id
+           NULL::JSONB timespan_tags,
+    COALESCE($1 = ANY(numers), false) valid_numer,
+    COALESCE($2 = ANY(denoms), false) valid_denom,
+    COALESCE($3 = ANY(geoms), false) valid_geom_id
     FROM observatory.obs_meta_timespan
     WHERE %s (timespan_tags ?& $4 OR CARDINALITY($4) = 0)
   $string$, geom_clause)
@@ -369,8 +475,10 @@ RETURNS TABLE (
 DECLARE
   aggregate_condition TEXT DEFAULT '';
 BEGIN
-  IF aggregate_type IS NOT NULL THEN
+  IF LOWER(aggregate_type) ILIKE 'sum' THEN
-    aggregate_condition := format(' AND numer_aggregate = %L ', aggregate_type);
+    aggregate_condition := ' AND numer_aggregate IN (''sum'', ''median'', ''average'') ';
+  ELSIF aggregate_type IS NOT NULL THEN
+    aggregate_condition := format(' AND numer_aggregate ILIKE %L ', aggregate_type);
   END IF;
   RETURN QUERY
   EXECUTE format($string$
@@ -412,75 +520,95 @@ BEGIN
 END
 $$ LANGUAGE plpgsql;
 
+
 CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryScores(
   bounds Geometry(Geometry, 4326) DEFAULT NULL,
   filter_geom_ids TEXT[] DEFAULT NULL,
-  desired_num_geoms INTEGER DEFAULT 3000
+  desired_num_geoms INTEGER DEFAULT NULL,
+  desired_area NUMERIC DEFAULT NULL
 ) RETURNS TABLE (
   score NUMERIC,
   numtiles BIGINT,
-  geom_id TEXT,
+  table_id TEXT,
+  column_id TEXT,
   notnull_percent NUMERIC,
   numgeoms NUMERIC,
   percentfill NUMERIC,
   estnumgeoms NUMERIC,
   meanmediansize NUMERIC
 ) AS $$
+DECLARE
+  num_geoms_multiplier Numeric;
 BEGIN
+  IF desired_num_geoms IS NULL THEN
+    desired_num_geoms := 3000;
+  END IF;
   filter_geom_ids := COALESCE(filter_geom_ids, (ARRAY[])::TEXT[]);
+  -- Very complex geometries simply fail.  For a boundary check, we can
+  -- comfortably get away with the simplicity of an envelope
+  IF ST_Npoints(bounds) > 10000 THEN
+    bounds := ST_Envelope(bounds);
+  END IF;
+  IF desired_area IS NULL THEN
+    desired_area := ST_Area(bounds);
+  END IF;
+
+  -- In case of points, desired_area will be 0.  We still want an accurate
+  -- estimate of numgeoms in that case.
+  IF desired_area = 0 THEN
+    num_geoms_multiplier := 1;
+  ELSE
+    num_geoms_multiplier := Coalesce(desired_area / Nullif(ST_Area(bounds), 0), 1);
+  END IF;
+
   RETURN QUERY
-  EXECUTE format($string$
+  EXECUTE $string$
-    SELECT
+    WITH clipped_geom AS (
-      (1 / (abs(numgeoms - $3)
+      SELECT column_id, table_id
-        --* (1 / Coalesce(NullIf(notnull_percent, 0), 1))
+        , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True) -- -20
-        --* (1 / Coalesce(NullIf(percentfill, 0), 0.0001))
+               ELSE tile END clipped_tile
-      ))::Numeric
+        , tile
-      AS score, *
+      FROM observatory.obs_column_table_tile_simple
-    FROM (
+      WHERE ($1 IS NULL OR ST_Intersects($1, tile))
-      WITH clipped_geom AS (
+        AND (column_id = ANY($2) OR cardinality($2) = 0)
-        SELECT column_id, table_id
+    ), clipped_geom_countagg AS (
-          , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True)
+      SELECT column_id, table_id
-                 ELSE tile END clipped_tile
+        , BOOL_AND(ST_BandIsNoData(clipped_tile, 1)) nodata
-          , tile
+      FROM clipped_geom
-        FROM observatory.obs_column_table_tile
+      GROUP BY column_id, table_id
-        WHERE ($1 IS NULL OR ST_Intersects($1, tile))
+    ), clipped_geom_reagg AS (
-          AND (column_id = ANY($2) OR cardinality($2) = 0)
+      SELECT COUNT(*)::BIGINT cnt, a.column_id, a.table_id,
-      ), clipped_geom_countagg AS (
+             cdb_observatory.FIRST(nodata) first_nodata,
-        SELECT column_id, table_id
+             cdb_observatory.FIRST(tile) first_tile,
-          , ST_CountAgg(clipped_tile, 2, True)::Numeric notnull_pixels
+             (ST_SummaryStatsAgg(clipped_tile, 1, False)).sum::Numeric sum_geoms, -- ND
-          , ST_CountAgg(clipped_tile, 2, False)::Numeric pixels
+             (ST_SummaryStatsAgg(clipped_tile, 2, False)).mean::Numeric / 255 mean_fill --ND
-        FROM clipped_geom
-        GROUP BY column_id, table_id
-      ) SELECT
-        count(*)::BIGINT, a.column_id
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN cdb_observatory.FIRST(notnull_pixels) / cdb_observatory.FIRST(pixels)
-                ELSE 1
-          END)::Numeric AS notnull_percent
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN (ST_SummaryStatsAgg(clipped_tile, 2, True)).sum
-                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 2, ST_PointOnSurface($1)), 0) * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(cdb_observatory.FIRST(tile), 0, 0)) * cdb_observatory.FIRST(pixels))
-          END)::Numeric AS numgeoms
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN (ST_SummaryStatsAgg(clipped_tile, 3, True)).mean
-                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 3, ST_PointOnSurface($1)), 0)
-          END)::Numeric AS percentfill
-        , ((ST_Area(ST_Transform($1, 3857)) / 1000000) / NullIf(
-            CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                 THEN (ST_SummaryStatsAgg(clipped_tile, 1, True)).mean
-                 ELSE Coalesce(ST_Value(cdb_observatory.FIRST(tile), 1, ST_PointOnSurface($1)), 0)
-            END, 0))::Numeric AS estnumgeoms
-        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
-                THEN (ST_SummaryStatsAgg(clipped_tile, 1, True)).mean
-                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 1, ST_PointOnSurface($1)), 0)
-          END)::Numeric AS meanmediansize
       FROM clipped_geom_countagg a, clipped_geom b
       WHERE a.table_id = b.table_id
         AND a.column_id = b.column_id
       GROUP BY a.column_id, a.table_id
-      ORDER BY a.column_id, a.table_id
+    ), final AS (
-    ) foo
+      SELECT
-  $string$) USING bounds, filter_geom_ids, desired_num_geoms;
+        cnt, table_id, column_id
+        , NULL::Numeric AS notnull_percent
+        , (CASE WHEN first_nodata IS FALSE
+                THEN sum_geoms
+                ELSE COALESCE(ST_Value(first_tile, 1, ST_PointOnSurface($1)), 0)
+                  * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(first_tile, 0, 0)))
+          END)::Numeric * $4
+        AS numgeoms
+        , (CASE WHEN first_nodata IS FALSE
+                THEN mean_fill
+                ELSE COALESCE(ST_Value(first_tile, 2, ST_PointOnSurface($1))::Numeric / 255, 0) -- -2
+          END)::Numeric
+        AS percentfill
+        , null::numeric estnumgeoms
+        , null::numeric meanmediansize
+      FROM clipped_geom_reagg
+    ) SELECT
+      ((100.0 / (1+abs(log(0.0001 + $3) - log(0.0001 + numgeoms::Numeric)))) * percentfill)::Numeric
+      AS score, *
+      FROM final
+  $string$ USING bounds, filter_geom_ids, desired_num_geoms, num_geoms_multiplier;
   RETURN;
 END
-$$ LANGUAGE plpgsql;
+$$ LANGUAGE plpgsql IMMUTABLE;

src/pg/sql/44_observatory_geometries.sql

@@ -40,44 +40,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
   END IF;
 
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                                           time_span) As x(target_tables,
-                                                           timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    -- TODO: modify for only one table returned instead of arbitrarily choosing
-    --       one with LIMIT 1 (could be conflict between clipped vs non-clipped
-    --       boundaries in the metadata tables)
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN NULL::geometry;
-  END IF;
-
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return the first boundary in intersections
-  EXECUTE format(
+  EXECUTE $query$
-    'SELECT the_geom
+    SELECT * FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2, $3) LIMIT 1
-     FROM observatory.%I
+  $query$ INTO boundary
-     WHERE ST_Intersects($1, the_geom)
+  USING geom, boundary_id, time_span;
-     LIMIT 1', target_table)
-  INTO boundary
-  USING geom;
 
   RETURN boundary;
 
@@ -111,67 +78,17 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryId(
 RETURNS text
 AS $$
 DECLARE
-  output_id text;
+  result TEXT;
-  target_table text;
-  geoid_colname text;
 BEGIN
 
-  -- If not point, raise error
+  EXECUTE $query$
-  IF ST_GeometryType(geom) != 'ST_Point'
+    SELECT geom_refs FROM cdb_observatory._OBS_GetBoundariesByGeometry(
-  THEN
+      $1, $2, $3) LIMIT 1
-    RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_Point''', ST_GeometryType(geom);
+  $query$
-  END IF;
+  INTO result
-
+  USING geom, boundary_id, time_span;
-  -- choose appropriate table based on time_span
-  IF time_span IS NULL
-  THEN
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                                timespans)
-    ORDER BY x.timespans DESC
-    LIMIT 1;
-  ELSE
-    SELECT x.target_tables INTO target_table
-    FROM cdb_observatory._OBS_SearchTables(boundary_id,
-                            time_span) As x(target_tables,
-                                            timespans)
-    WHERE x.timespans = time_span
-    LIMIT 1;
-  END IF;
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'Warning: No boundaries found for ''%''', boundary_id;
-    RETURN NULL::text;
-  END IF;
-
-  EXECUTE
-    format('SELECT ct.colname
-              FROM observatory.obs_column_to_column c2c,
-                   observatory.obs_column_table ct,
-                   observatory.obs_table t
-             WHERE c2c.reltype = ''geom_ref''
-               AND ct.column_id = c2c.source_id
-               AND ct.table_id = t.id
-               AND t.tablename = %L'
-   , target_table)
-  INTO geoid_colname;
-
-  --RAISE NOTICE 'target_table: %, geoid_colname: %', target_table, geoid_colname;
-
-  -- return geometry id column value
-  EXECUTE format(
-    'SELECT %I::text
-     FROM observatory.%I
-     WHERE ST_Intersects($1, the_geom)
-     LIMIT 1', geoid_colname, target_table)
-  INTO output_id
-  USING geom;
-
-  RETURN output_id;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -203,35 +120,21 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundaryById(
 RETURNS geometry(geometry, 4326)
 AS $$
 DECLARE
-  boundary geometry(geometry, 4326);
+  result GEOMETRY;
-  target_table text;
-  geoid_colname text;
-  geom_colname text;
 BEGIN
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
+  EXECUTE $query$
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
+    SELECT (data->0->>'value')::Geometry
-
+    FROM cdb_observatory.OBS_GetData(
-  --RAISE NOTICE '%', target_table;
+      ARRAY[$1],
-
+      cdb_observatory.OBS_GetMeta(
-  IF target_table IS NULL
+        ST_MakeEnvelope(-180, -90, 180, 90, 4326),
-  THEN
+        ('[{"geom_id": "' || $2 || '"}]')::JSON))
-    --RAISE NOTICE 'No geometries found';
+  $query$
-    RETURN NULL::geometry;
+  INTO result
-  END IF;
+  USING geometry_id, boundary_id;
-
-  -- retrieve boundary
-  EXECUTE
-    format(
-    'SELECT %I
-     FROM observatory.%I
-     WHERE %I = $1
-     LIMIT 1', geom_colname, target_table, geoid_colname)
-  INTO boundary
-  USING geometry_id;
-
-  RETURN boundary;
 
+  RETURN result;
 END;
 $$ LANGUAGE plpgsql;
 
@@ -245,13 +148,12 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetBoundariesByGeometry(
   boundary_id text,
   time_span text DEFAULT NULL,
   overlap_type text DEFAULT NULL)
-RETURNS TABLE(the_geom geometry, geom_refs text)
+RETURNS TABLE (
-AS $$
+  the_geom geometry,
+  geom_refs text
+) AS $$
 DECLARE
-  boundary geometry(Geometry, 4326);
+  meta JSON;
-  geom_colname text;
-  geoid_colname text;
-  target_table text;
 BEGIN
   overlap_type := COALESCE(overlap_type, 'intersects');
   -- check inputs
@@ -259,34 +161,27 @@ BEGIN
   THEN
     -- recognized overlap type (map to ST_Contains, ST_Intersects, and ST_Within)
     RAISE EXCEPTION 'Overlap type ''%'' is not an accepted type (choose intersects, within, or contains)', overlap_type;
-  ELSIF ST_GeometryType(geom) NOT IN ('ST_Polygon', 'ST_MultiPolygon')
-  THEN
-      RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  -- TODO: add timespan in search
+  EXECUTE $query$
-  -- TODO: add overlap info in search
+    SELECT cdb_observatory.OBS_GetMeta($1, JSON_Build_Array(JSON_Build_Object(
-  SELECT * INTO geoid_colname, target_table, geom_colname
+                'geom_id', $2, 'geom_timespan', $3)))
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
+  $query$
+  INTO meta
+  USING geom, boundary_id, time_span;
 
-  -- if no tables are found, raise notice and return null
+  IF meta->0->>'geom_id' IS NULL THEN
-  IF target_table IS NULL
+    RETURN QUERY EXECUTE 'SELECT NULL::Geometry, NULL::Text LIMIT 0';
-  THEN
-    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
     RETURN;
   END IF;
 
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
+  RETURN QUERY EXECUTE $query$
-  EXECUTE format(
+    SELECT (data->0->>'value')::Geometry the_geom, data->0->>'geomref' geom_refs
-    'SELECT %I, %I::text
+    FROM cdb_observatory.OBS_GetData(
-     FROM observatory.%I
+      ARRAY[($1, 1)::geomval], $2, False
-     WHERE ST_%s($1, the_geom)
+    )
-     ', geom_colname, geoid_colname, target_table, overlap_type)
+  $query$ USING geom, meta;
-  USING geom;
   RETURN;
 
 END;
@@ -414,27 +309,11 @@ BEGIN
     RAISE EXCEPTION 'Invalid geometry type (%), expecting ''ST_MultiPolygon'' or ''ST_Polygon''', ST_GeometryType(geom);
   END IF;
 
-  SELECT * INTO geoid_colname, target_table, geom_colname
-  FROM cdb_observatory._OBS_GetGeometryMetadata(boundary_id);
-
-  -- if no tables are found, raise notice and return null
-  IF target_table IS NULL
-  THEN
-    --RAISE NOTICE 'No boundaries found for bounding box ''%'' in ''%''', ST_AsText(geom), boundary_id;
-    RETURN QUERY SELECT NULL::geometry, NULL::text;
-    RETURN;
-  END IF;
-
-  --RAISE NOTICE 'target_table: %', target_table;
-
   -- return first boundary in intersections
-  RETURN QUERY
+  RETURN QUERY EXECUTE $query$
-  EXECUTE format(
+    SELECT ST_PointOnSurface(the_geom), geom_refs
-    'SELECT ST_PointOnSurface(%I) As %s, %I::text
+    FROM cdb_observatory._OBS_GetBoundariesByGeometry($1, $2)
-     FROM observatory.%I
+  $query$ USING geom, boundary_id;
-     WHERE ST_%s($1, the_geom)
-     ', geom_colname, geom_colname, geoid_colname, target_table, overlap_type)
-  USING geom;
   RETURN;
 
 END;
@@ -534,44 +413,3 @@ BEGIN
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
-
-
--- _OBS_GetGeometryMetadata()
--- TODO: add timespan in search
--- TODO: add choice of clipped versus not clipped
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryMetadata(boundary_id text)
-RETURNS table(geoid_colname text, target_table text, geom_colname text)
-AS $$
-BEGIN
-
-  RETURN QUERY
-  EXECUTE
-  format($string$
-    SELECT geoid_ct.colname::text As geoid_colname,
-           tablename::text,
-           geom_ct.colname::text As geom_colname
-    FROM observatory.obs_column_table As geoid_ct,
-         observatory.obs_table As geom_t,
-         observatory.obs_column_table As geom_ct,
-         observatory.obs_column As geom_c
-    WHERE geoid_ct.column_id
-       IN (
-         SELECT source_id
-         FROM observatory.obs_column_to_column
-         WHERE reltype = 'geom_ref'
-           AND target_id = '%s'
-         )
-      AND geoid_ct.table_id = geom_t.id AND
-          geom_t.id = geom_ct.table_id AND
-          geom_ct.column_id = geom_c.id AND
-          geom_c.type ILIKE 'geometry' AND
-          geom_c.id = '%s'
-    $string$, boundary_id, boundary_id);
-  RETURN;
-    --  AND geom_t.timespan = '%s' <-- put in requested year
-    -- TODO: filter by clipped vs. not so appropriate tablename are unique
-    --       so the limit 1 can be removed
-    RETURN;
-
-END;
-$$ LANGUAGE plpgsql;

src/pg/sql/45_observatory_mvt.sql.disabled

@@ -0,0 +1,839 @@
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetTileBounds(z INTEGER, x INTEGER, y INTEGER)
+RETURNS NUMERIC[] AS $$
+import math
+def tile2lnglat(z, x, y):
+    n = 2.0 ** z
+    y = (1 << z) - y - 1
+
+    lon = x / n * 360.0 - 180.0
+    lat_rad = math.atan(math.sinh(math.pi * (1 - 2 * y / n)))
+    lat = - math.degrees(lat_rad)
+
+    return lon, lat
+
+lon0, lat0 = tile2lnglat(z, x, y)
+lon1, lat1 = tile2lnglat(z, x+1, y-1)
+
+return [lon0, lat0, lon1, lat1]
+$$ LANGUAGE plpythonu;
+
+DROP FUNCTION IF EXISTS cdb_observatory.OBS_GetMVT(z INTEGER, x INTEGER, y INTEGER, params JSONB);
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMVT(z INTEGER, x INTEGER, y INTEGER,
+params JSON DEFAULT NULL,
+extent INTEGER DEFAULT 4096, buf INTEGER DEFAULT 256, clip_geom BOOLEAN DEFAULT True)
+RETURNS TABLE (mvt BYTEA)
+AS $$
+DECLARE
+  bounds NUMERIC[];
+  geom GEOMETRY;
+  ext BOX2D;
+  meta JSON;
+
+  procgeom_clauses TEXT;
+  val_clauses TEXT;
+  json_clause TEXT;
+BEGIN
+  bounds := cdb_observatory.OBS_GetTileBounds(z, x, y);
+  geom := ST_MakeEnvelope(bounds[1], bounds[2], bounds[3], bounds[4], 4326);
+  ext := ST_MakeBox2D(ST_Point(bounds[1], bounds[2]), ST_Point(bounds[3], bounds[4]));
+  meta := cdb_observatory.obs_getmeta(geom, params::json, 1::integer, 1::integer, 1::integer);
+
+  /* Read metadata to generate clauses for query */
+  EXECUTE $query$
+    WITH _meta AS (SELECT
+      row_number() over () colid, *
+      FROM json_to_recordset($1)
+      AS x(id TEXT, numer_id TEXT, numer_aggregate TEXT, numer_colname TEXT,
+            numer_geomref_colname TEXT, numer_tablename TEXT, numer_type TEXT,
+            denom_id TEXT, denom_aggregate TEXT, denom_colname TEXT,
+            denom_geomref_colname TEXT, denom_tablename TEXT, denom_type TEXT,
+            denom_reltype TEXT, geom_id TEXT, geom_colname TEXT,
+            geom_geomref_colname TEXT, geom_tablename TEXT, geom_type TEXT,
+            numer_timespan TEXT, geom_timespan TEXT, normalization TEXT,
+            api_method TEXT, api_args JSON)
+    ),
+
+  -- Generate procgeom clauses.
+  -- These join the users' geoms to the relevant geometries for the
+  -- asked-for measures in the Observatory.
+  _procgeom_clauses AS (
+    SELECT
+      '_procgeoms_' || Coalesce(left(geom_tablename,40) || '_' || geom_geomref_colname, api_method) || ' AS (' ||
+          'SELECT ' ||
+            'st_intersection(' || geom_tablename || '.' || geom_colname || ', _geoms.geom) AS geom, ' ||
+            'ST_AsMVTGeom(st_intersection(' || geom_tablename || '.' || geom_colname || ', _geoms.geom), $2, $3, $4, $5) AS mvtgeom, ' ||
+            geom_tablename || '.' || geom_geomref_colname || ' AS geomref, ' ||
+            'CASE WHEN ST_Within(_geoms.geom, ' || geom_tablename || '.' || geom_colname || ')
+                    THEN ST_Area(_geoms.geom) / Nullif(ST_Area(' || geom_tablename || '.' || geom_colname || '), 0)
+                    WHEN ST_Within(' || geom_tablename || '.' || geom_colname || ', _geoms.geom)
+                    THEN 1
+                    ELSE ST_Area(cdb_observatory.safe_intersection(_geoms.geom, ' || geom_tablename || '.' || geom_colname || ')) /
+                      Nullif(ST_Area(' || geom_tablename || '.' || geom_colname || '), 0)
+              END pct_obs' || '
+          FROM _geoms, observatory.' || geom_tablename || '
+          WHERE ST_Intersects(_geoms.geom, ' || geom_tablename || '.' || geom_colname || ')'
+          || ')'
+      AS procgeom_clause
+    FROM _meta
+    GROUP BY api_method, geom_tablename, geom_geomref_colname, geom_colname
+  ),
+
+  -- Generate val clauses.
+  -- These perform interpolations or other necessary calculations to
+  -- provide values according to users geometries.
+  _val_clauses AS (
+    SELECT
+      '_vals_' || Coalesce(left(geom_tablename,40) || '_' || geom_geomref_colname, api_method) || ' AS (
+        SELECT _procgeoms.geomref, _procgeoms.mvtgeom, ' ||
+          String_Agg('json_build_object(' || CASE
+            -- api-delivered values
+            WHEN api_method IS NOT NULL THEN
+            '''' || numer_colname || ''', ' ||
+              'ARRAY_AGG( ' ||
+                api_method || '.' || numer_colname || ')::' || numer_type || '[]'
+            -- numeric internal values
+            WHEN cdb_observatory.isnumeric(numer_type) THEN
+            '''' || numer_colname || ''', ' || CASE
+              -- denominated
+              WHEN LOWER(normalization) LIKE 'denom%'
+                THEN CASE
+                WHEN denom_tablename IS NULL THEN ' NULL '
+                -- denominated polygon interpolation
+                ELSE
+                ' ROUND(CAST(SUM(' || numer_tablename || '.' || numer_colname || ' ' ||
+                ' * _procgeoms.pct_obs ' ||
+                ' ) / NULLIF(SUM(' || denom_tablename || '.' || denom_colname || ' ' ||
+                '            * _procgeoms.pct_obs), 0) AS NUMERIC), 4) '
+                END
+              -- areaNormalized
+              WHEN LOWER(normalization) LIKE 'area%'
+                THEN
+                -- areaNormalized polygon interpolation
+                ' ROUND(CAST(SUM(' || numer_tablename || '.' || numer_colname || ' ' ||
+                ' * _procgeoms.pct_obs' ||
+                ' ) / (Nullif(ST_Area(cdb_observatory.FIRST(_procgeoms.geom)::Geography), 0) / 1000000) AS NUMERIC), 4) '
+              -- median/average measures with universe
+              WHEN LOWER(numer_aggregate) IN ('median', 'average') AND
+                  denom_reltype ILIKE 'universe' AND LOWER(normalization) LIKE 'pre%'
+                THEN
+                -- predenominated polygon interpolation weighted by universe
+                ' ROUND(CAST(SUM(' || numer_tablename || '.' || numer_colname ||
+                ' * ' || denom_tablename || '.' || denom_colname ||
+                ' * _procgeoms.pct_obs ' ||
+                ' ) / Nullif(SUM(' || denom_tablename || '.' || denom_colname ||
+                ' * _procgeoms.pct_obs ' || '), 0)AS NUMERIC), 4) '
+              -- prenormalized for summable measures. point or summable only!
+              WHEN numer_aggregate ILIKE 'sum' AND LOWER(normalization) LIKE 'pre%'
+                THEN
+                -- predenominated polygon interpolation
+                ' ROUND(CAST(SUM(' || numer_tablename || '.' || numer_colname || ' ' ||
+                ' * _procgeoms.pct_obs) AS NUMERIC), 4) '
+              -- Everything else. Point only!
+              ELSE
+                ' cdb_observatory._OBS_RaiseNotice(''Cannot perform calculation over polygon for ' ||
+                    numer_id || '/' || coalesce(denom_id, '') || '/' || geom_id || '/' || numer_timespan || ''')::Numeric '
+              END || '::' || numer_type
+
+            -- categorical/text
+            WHEN LOWER(numer_type) LIKE 'text' THEN
+              '''' || numer_colname || ''', ' || 'MODE() WITHIN GROUP (ORDER BY ' || numer_tablename || '.' || numer_colname || ') '
+              -- geometry
+            WHEN numer_id IS NULL THEN
+              '''geomref'', _procgeoms.geomref, ' ||
+              '''' || numer_colname || ''', ' || 'cdb_observatory.FIRST(_procgeoms.mvtgeom)::TEXT'
+            ELSE ''
+            END
+          || ') val_' || colid, ', ')
+        || '
+        FROM _procgeoms_' || Coalesce(left(geom_tablename,40) || '_' || geom_geomref_colname, api_method) || ' _procgeoms ' ||
+          Coalesce(String_Agg(DISTINCT
+              Coalesce('LEFT JOIN observatory.' || numer_tablename || ' ON _procgeoms.geomref = observatory.' || numer_tablename || '.' || numer_geomref_colname,
+                ', LATERAL (SELECT * FROM cdb_observatory.' || api_method || '(_procgeoms.mvtgeom' || Coalesce(', ' ||
+                    (SELECT STRING_AGG(REPLACE(val::text, '"', ''''), ', ')
+                      FROM (SELECT JSON_Array_Elements(api_args) as val) as vals),
+                    '') || ')) AS ' || api_method)
+          , ' '), '') ||
+                                E'\n GROUP BY _procgeoms.geomref, _procgeoms.mvtgeom
+                                    ORDER BY _procgeoms.geomref'
+      || ')'
+      AS val_clause,
+      '_vals_' || Coalesce(left(geom_tablename, 40) || '_' || geom_geomref_colname, api_method) AS cte_name
+    FROM _meta
+    GROUP BY geom_tablename, geom_geomref_colname, geom_colname, api_method
+  ),
+
+  -- Generate clauses necessary to join together val_clauses
+  _val_joins AS (
+    SELECT String_Agg(a.cte_name || '.geomref = ' || b.cte_name || '.geomref ', ' AND ') val_joins
+    FROM _val_clauses a, _val_clauses b
+    WHERE a.cte_name != b.cte_name
+      AND a.cte_name < b.cte_name
+  ),
+
+  -- Generate JSON clause.  This puts together vals from val_clauses
+  _json_clause AS (SELECT
+    'SELECT ST_AsMVT(q, ''data'', $3) FROM (' ||
+    'SELECT ' || cdb_observatory.FIRST(cte_name) || '.mvtgeom geom,
+        replace(' || (SELECT String_Agg('val_' || colid, '::TEXT || ') FROM _meta) || ', ''}{'', '', '')::jsonb
+      FROM ' || String_Agg(cte_name, ', ') ||
+    ' WHERE ST_Area(' || cdb_observatory.FIRST(cte_name) || '.mvtgeom) > 0' ||
+    Coalesce(' AND ' || val_joins, ') q')
+    AS json_clause
+    FROM _val_clauses, _val_joins
+    GROUP BY val_joins
+  )
+
+  SELECT (SELECT String_Agg(procgeom_clause, E',\n     ') FROM _procgeom_clauses),
+          (SELECT String_Agg(val_clause, E',\n     ') FROM _val_clauses),
+          json_clause
+    FROM _json_clause
+  $query$ INTO
+    procgeom_clauses,
+    val_clauses,
+    json_clause
+  USING meta;
+
+  IF procgeom_clauses IS NULL OR val_clauses IS NULL OR json_clause IS NULL THEN
+    RETURN;
+  END IF;
+
+  /* Execute query */
+  RETURN QUERY EXECUTE format($query$
+    WITH _geoms AS (%s),
+    -- procgeom_clauses
+    %s,
+
+    -- val_clauses
+    %s
+
+    -- json_clause
+    %s
+  $query$, 'SELECT $1::geometry as geom',
+            String_Agg(procgeom_clauses, E',\n       '),
+            String_Agg(val_clauses, E',\n       '),
+            json_clause)
+  USING geom, ext, extent, buf, clip_geom;
+  RETURN;
+
+END
+$$ LANGUAGE plpgsql;
+
+DROP TABLE IF EXISTS cdb_observatory.OBS_CachedMeta;
+CREATE TABLE cdb_observatory.OBS_CachedMeta(
+  z INTEGER,
+  parameters TEXT,
+  num_timespans INTEGER,
+  num_scores INTEGER,
+  num_target_geoms INTEGER,
+  result JSON,
+  PRIMARY KEY (z, parameters, num_timespans, num_scores, num_target_geoms)
+);
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_RetrieveMeta(
+  zoom INTEGER,
+  geom geometry(Geometry, 4326),
+  getmeta_parameters JSON,
+  num_timespan_options INTEGER DEFAULT NULL,
+  num_score_options INTEGER DEFAULT NULL,
+  target_geoms INTEGER DEFAULT NULL)
+RETURNS JSON
+AS $$
+DECLARE
+  result JSON;
+BEGIN
+  SELECT c.result
+    INTO result
+    FROM cdb_observatory.OBS_CachedMeta c
+   WHERE c.z = zoom
+     AND c.parameters = getmeta_parameters::TEXT
+     AND c.num_timespans = num_timespan_options
+     AND c.num_scores = num_score_options
+     AND c.num_target_geoms = target_geoms;
+
+  IF result IS NULL THEN
+    result := cdb_observatory.obs_getmeta(geom, getmeta_parameters, num_timespan_options, num_score_options, target_geoms);
+
+    INSERT INTO cdb_observatory.OBS_CachedMeta(z, parameters, num_timespans, num_scores, num_target_geoms, result)
+    SELECT zoom, getmeta_parameters::TEXT, num_timespan_options, num_score_options, target_geoms, result
+    ON CONFLICT (z, parameters, num_timespans, num_scores, num_target_geoms) 
+    DO UPDATE SET result = EXCLUDED.result;
+  END IF;
+
+  return result;
+END
+$$ LANGUAGE plpgsql PARALLEL RESTRICTED;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMCDates(
+  mc_schema TEXT,
+  geo_level TEXT,
+  month_no TEXT DEFAULT NULL)
+RETURNS TEXT[]
+AS $$
+DECLARE
+  mc_table TEXT;
+  where_clause TEXT DEFAULT '';
+  dates TEXT[];
+BEGIN
+  mc_table := cdb_observatory.OBS_GetMCTable(mc_schema, geo_level);
+
+  IF month_no IS NOT NULL THEN
+    where_clause := format(
+      $query$
+      WHERE month LIKE '%1$s/__/____'
+      $query$, LPAD(month_no, 2, '0'));
+  END IF;
+
+  EXECUTE
+  format(
+    $query$
+    SELECT ARRAY_AGG(DISTINCT month) dates
+      FROM "%1$s".%2$s
+      %3$s
+    $query$, mc_schema, mc_table, where_clause)
+  INTO dates;
+
+  RETURN dates;
+END
+$$ LANGUAGE plpgsql PARALLEL RESTRICTED;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMCTable(mc_schema TEXT, geo_level TEXT)
+RETURNS TEXT
+AS $$
+DECLARE
+  mc_table TEXT;
+BEGIN
+--  SELECT tablename from pg_tables
+--    INTO mc_table
+--   WHERE schemaname = mc_schema
+--     AND tablename LIKE '%'||geo_level||'%';
+  mc_table := 'mc_' || geo_level;
+  RETURN mc_table;
+END
+$$ LANGUAGE plpgsql PARALLEL RESTRICTED;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMCDOMVT(
+  z INTEGER, x INTEGER, y INTEGER,
+  geography_level TEXT,
+  do_measurements TEXT[],
+  mc_measurements TEXT[],
+  mc_categories TEXT[] DEFAULT ARRAY['TR']::TEXT[],
+  mc_months TEXT[] DEFAULT ARRAY['2018-02-01']::TEXT[],
+  use_meta_cache BOOLEAN DEFAULT True,
+  shoreline_clipped BOOLEAN DEFAULT True,
+  optimize_clipping BOOLEAN DEFAULT False,
+  simplify_geometries BOOLEAN DEFAULT False,
+  area_normalized BOOLEAN DEFAULT False,
+  extent INTEGER DEFAULT 4096,
+  buf INTEGER DEFAULT 256,
+  clip_geom BOOLEAN DEFAULT True)
+RETURNS TABLE (
+  mvtgeom GEOMETRY,
+  mvtdata JSONB
+)
+AS $$
+DECLARE
+  state_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.state';
+  county_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.county';
+  tract_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.census_tract';
+  blockgroup_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.block_group';
+  block_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.block';
+
+  mc_schema CONSTANT TEXT DEFAULT 'us.mastercard';
+  mc_geoid CONSTANT TEXT DEFAULT 'region_id';
+  mc_category_column CONSTANT TEXT DEFAULT 'category';
+  mc_month_column CONSTANT TEXT DEFAULT 'month';
+  mc_table TEXT;
+  mc_category TEXT;
+  mc_table_categories TEXT DEFAULT '';
+  mc_month TEXT;
+  mc_month_slug TEXT;
+  mc_measurements_categories TEXT[];
+  mc_measurement TEXT;
+
+  bounds NUMERIC[];
+  geom GEOMETRY;
+  ext BOX2D;
+
+  measurement TEXT;
+  getmeta_parameters TEXT;
+  meta JSON;
+  mc_geography_level TEXT;
+
+  numer_tablename_do TEXT DEFAULT '';
+  numer_tablenames_do TEXT[] DEFAULT ARRAY['']::TEXT[];
+  numer_tablenames_do_outer TEXT DEFAULT '';
+  numer_tablenames_mc TEXT DEFAULT '';
+  numer_colnames_do TEXT DEFAULT '';
+  numer_colnames_do_qualified TEXT DEFAULT '';
+  numer_colnames_do_normalized TEXT DEFAULT '';
+  numer_colnames_mc TEXT DEFAULT '';
+  numer_colnames_mc_current TEXT DEFAULT '';
+  numer_colnames_mc_qualified TEXT DEFAULT '';
+  numer_colnames_mc_qualified_current TEXT DEFAULT '';
+  numer_colnames_mc_normalized TEXT DEFAULT '';
+  numer_colnames_mc_normalized_current TEXT DEFAULT '';
+  geom_tablenames TEXT;
+  geom_colnames TEXT;
+  geom_geomref_colnames TEXT;
+  geom_geomref_colnames_qualified TEXT;
+  geom_relations_do TEXT[] DEFAULT ARRAY['']::TEXT[];
+  geom_relations_mc TEXT DEFAULT '';
+  geom_mc_outerjoins TEXT DEFAULT '';
+
+  simplification_tolerance NUMERIC DEFAULT 0;
+  area_normalization TEXT DEFAULT '';
+  i INTEGER DEFAULT 0;
+  clipped TEXT default '';
+BEGIN
+  IF area_normalized THEN
+    area_normalization := '/area_ratio';
+  END IF;
+
+  IF shoreline_clipped THEN
+    clipped := '_clipped';
+  END IF;
+
+  CASE
+    WHEN geography_level = state_geoname THEN
+      simplification_tolerance := 0.1;
+      IF optimize_clipping THEN
+        clipped := '';
+      END IF;
+    WHEN geography_level = county_geoname THEN
+      simplification_tolerance := 0.01;
+    WHEN geography_level = tract_geoname THEN
+      simplification_tolerance := 0.001;
+    WHEN geography_level = blockgroup_geoname THEN
+      simplification_tolerance := 0.0001;
+    WHEN geography_level = block_geoname THEN
+      simplification_tolerance := 0.0001;
+    ELSE
+      simplification_tolerance := 0;
+  END CASE;
+
+  IF NOT simplify_geometries THEN
+    simplification_tolerance := 0;
+  END IF;
+
+  bounds := cdb_observatory.OBS_GetTileBounds(z, x, y);
+  geom := ST_MakeEnvelope(bounds[1], bounds[2], bounds[3], bounds[4], 4326);
+  ext := ST_MakeBox2D(ST_Transform(ST_SetSRID(ST_Point(bounds[1], bounds[2]), 4326), 3857),
+                      ST_Transform(ST_SetSRID(ST_Point(bounds[3], bounds[4]), 4326), 3857));
+
+  ---------DO---------
+  getmeta_parameters := '[ ';
+  FOREACH measurement IN ARRAY do_measurements LOOP
+    getmeta_parameters := getmeta_parameters || '{"numer_id":"' || measurement || '","geom_id":"' || geography_level || clipped ||'"},';
+  END LOOP;
+  getmeta_parameters := substring(getmeta_parameters from 1 for length(getmeta_parameters) - 1) || ' ]';
+
+  IF use_meta_cache THEN
+    meta := cdb_observatory.OBS_RetrieveMeta(z, geom, getmeta_parameters::json, 1::integer, 1::integer, 1::integer);
+  ELSE
+    meta := cdb_observatory.obs_getmeta(geom, getmeta_parameters::json, 1::integer, 1::integer, 1::integer);
+  END IF;
+
+  IF meta IS NOT NULL THEN
+    SELECT  array_agg(distinct 'observatory.'||numer_tablename) numer_tablenames,
+            string_agg(distinct numer_colname, ',')||',' numer_colnames,
+            string_agg(distinct numer_tablename||'.'||numer_colname, ',')||',' numer_colnames_qualified,
+            string_agg(distinct numer_colname||area_normalization||' '||numer_colname, ',')||',' numer_colnames_normalized,
+            (array_agg(distinct 'observatory.'||geom_tablename))[1] geom_tablenames,
+            (array_agg(distinct geom_colname))[1] geom_colnames,
+            (array_agg(distinct geom_geomref_colname))[1] geom_geomref_colnames,
+            (array_agg(distinct geom_tablename||'.'||geom_geomref_colname))[1] geom_geomref_colnames_qualified,
+            array_agg(distinct numer_tablename||'.'||numer_geomref_colname||'='||geom_tablename||'.'||geom_geomref_colname) geom_relations
+      INTO numer_tablenames_do, numer_colnames_do, numer_colnames_do_qualified, numer_colnames_do_normalized, geom_tablenames, geom_colnames,
+          geom_geomref_colnames, geom_geomref_colnames_qualified, geom_relations_do
+      FROM json_to_recordset(meta)
+        AS x(id TEXT, numer_id TEXT, numer_aggregate TEXT, numer_colname TEXT, numer_geomref_colname TEXT, numer_tablename TEXT,
+            numer_type TEXT, denom_id TEXT, denom_aggregate TEXT, denom_colname TEXT, denom_geomref_colname TEXT, denom_tablename TEXT,
+            denom_type TEXT, denom_reltype TEXT, geom_id TEXT, geom_colname TEXT, geom_geomref_colname TEXT, geom_tablename TEXT,
+            geom_type TEXT, numer_timespan TEXT, geom_timespan TEXT, normalization TEXT, api_method TEXT, api_args JSON);
+
+    IF numer_tablenames_do IS NULL OR numer_colnames_do IS NULL OR numer_colnames_do_qualified IS NULL OR numer_colnames_do_normalized IS NULL
+      OR geom_tablenames IS NULL OR geom_colnames IS NULL OR geom_geomref_colnames IS NULL OR geom_geomref_colnames_qualified IS NULL
+      OR geom_relations_do IS NULL THEN
+      RETURN;
+    END IF;
+
+    i := 0;
+    FOREACH numer_tablename_do IN ARRAY numer_tablenames_do LOOP
+      i := i + 1;
+      numer_tablenames_do_outer := numer_tablenames_do_outer || 'LEFT OUTER JOIN ' || numer_tablename_do || ' ON ' || geom_relations_do[i] || ' ';
+    END LOOP;
+  ELSE
+    getmeta_parameters := '[{"geom_id":"' || geography_level || clipped ||'"}]';
+    meta := cdb_observatory.obs_getmeta(geom, getmeta_parameters::json, 1::integer, 1::integer, 1::integer);
+
+    IF meta IS NULL THEN
+      RETURN;
+    END IF;
+
+    SELECT  (array_agg(distinct 'observatory.'||geom_tablename))[1] geom_tablenames,
+            (array_agg(distinct geom_colname))[1] geom_colnames,
+            (array_agg(distinct geom_geomref_colname))[1] geom_geomref_colnames,
+            (array_agg(distinct geom_tablename||'.'||geom_geomref_colname))[1] geom_geomref_colnames_qualified
+      FROM json_to_recordset(meta)
+      INTO geom_tablenames, geom_colnames, geom_geomref_colnames, geom_geomref_colnames_qualified
+        AS x(id TEXT, numer_id TEXT, numer_aggregate TEXT, numer_colname TEXT, numer_geomref_colname TEXT, numer_tablename TEXT,
+            numer_type TEXT, denom_id TEXT, denom_aggregate TEXT, denom_colname TEXT, denom_geomref_colname TEXT, denom_tablename TEXT,
+            denom_type TEXT, denom_reltype TEXT, geom_id TEXT, geom_colname TEXT, geom_geomref_colname TEXT, geom_tablename TEXT,
+            geom_type TEXT, numer_timespan TEXT, geom_timespan TEXT, normalization TEXT, api_method TEXT, api_args JSON);
+  END IF;
+
+  ---------MC---------
+  IF geography_level = 'us.census.tiger.census_tract' THEN
+    mc_geography_level := 'tract';
+  ELSE
+    mc_geography_level := (string_to_array(geography_level, '.'))[array_length(string_to_array(geography_level, '.'), 1)];
+  END IF;
+
+  mc_table := cdb_observatory.OBS_GetMCTable(mc_schema, mc_geography_level);
+
+  FOREACH mc_month IN ARRAY mc_months LOOP
+    mc_month_slug := replace(mc_month, '/', '');
+    FOREACH mc_category IN ARRAY mc_categories LOOP
+      mc_category := lower(mc_category);
+      mc_measurements_categories := ARRAY['']::TEXT[];
+
+      FOREACH mc_measurement IN ARRAY mc_measurements LOOP
+        mc_measurements_categories := array_append(mc_measurements_categories, mc_measurement||'_'||mc_category);
+      END LOOP;
+
+      SELECT string_agg(column_name||'_'||mc_month_slug, ','),
+            string_agg(mc_table||'_'||mc_month_slug||'.'||column_name||' '||column_name||'_'||mc_month_slug, ','),
+            string_agg(distinct column_name||'_'||mc_month_slug||area_normalization||' '||column_name||'_'||mc_month_slug, ',')
+        INTO numer_colnames_mc_current, numer_colnames_mc_qualified_current, numer_colnames_mc_normalized_current
+        FROM information_schema.columns
+      WHERE table_schema = mc_schema
+        AND table_name = mc_table
+        AND column_name = ANY(mc_measurements_categories);
+
+      IF numer_colnames_mc_current IS NOT NULL THEN
+        numer_colnames_mc := coalesce(numer_colnames_mc, '')||numer_colnames_mc_current||',';
+      END IF;
+      IF numer_colnames_mc_qualified_current IS NOT NULL THEN
+        numer_colnames_mc_qualified := coalesce(numer_colnames_mc_qualified, '')||numer_colnames_mc_qualified_current||',';
+      END IF;
+      IF numer_colnames_mc_normalized_current IS NOT NULL THEN
+        numer_colnames_mc_normalized := coalesce(numer_colnames_mc_normalized, '')||numer_colnames_mc_normalized_current||',';
+      END IF;
+    END LOOP;
+
+    IF mc_table IS NOT NULL THEN
+      numer_tablenames_mc := '"'||mc_schema||'".'||mc_table||' '||mc_table||'_'||mc_month_slug;
+      geom_relations_mc := mc_table||'_'||mc_month_slug||'.'||mc_geoid||'='||geom_geomref_colnames_qualified;
+      mc_table_categories := mc_table||'_'||mc_month_slug||'.'||mc_month_column||'='''||mc_month||'''';
+
+      geom_mc_outerjoins := coalesce(geom_mc_outerjoins, '')||' LEFT OUTER JOIN '||numer_tablenames_mc||' ON '||geom_relations_mc||' AND '||mc_table_categories;
+    END IF;
+  END LOOP;
+
+  ---------Query build and execution---------
+  RETURN QUERY EXECUTE format(
+    $query$
+    SELECT  mvtgeom,
+            (select row_to_json(_)::jsonb from (select id, %9$s %3$s area_ratio, area) as _) as mvtdata
+          FROM (
+      SELECT ST_AsMVTGeom(ST_Transform(the_geom, 3857), $1, $2, $3, $4) AS mvtgeom, %8$s as id, %6$s %7$s area_ratio, area FROM (
+        SELECT  %1$s the_geom, %8$s, %2$s %10$s
+                CASE  WHEN ST_Within($5, %1$s)
+                        THEN ST_Area($5) / Nullif(ST_Area(%1$s), 0)
+                      WHEN ST_Within(%1$s, $5)
+                        THEN 1
+                      ELSE ST_Area(ST_Intersection(st_simplifyvw(%1$s, $6), $5)) / Nullif(ST_Area(%1$s), 0)
+                END area_ratio,
+                ROUND(ST_Area(ST_Transform(the_geom,3857))::NUMERIC, 2) area
+          FROM %5$s
+               %4$s
+               %11$s
+        WHERE st_intersects(%1$s, $5)
+      ) p
+    ) q
+    $query$,
+    geom_colnames, numer_colnames_do_qualified, numer_colnames_mc, numer_tablenames_do_outer, geom_tablenames, numer_colnames_do_normalized,
+    numer_colnames_mc_normalized, geom_geomref_colnames, numer_colnames_do, numer_colnames_mc_qualified, geom_mc_outerjoins)
+  USING ext, extent, buf, clip_geom, geom, simplification_tolerance
+  RETURN;
+END
+$$ LANGUAGE plpgsql PARALLEL RESTRICTED;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetMCDOMVT(
+  z INTEGER,
+  geography_level TEXT,
+  do_measurements TEXT[],
+  mc_measurements TEXT[],
+  mc_categories TEXT[] DEFAULT ARRAY['TR']::TEXT[],
+  mc_months TEXT[] DEFAULT ARRAY['2018-02-01']::TEXT[],
+  use_meta_cache BOOLEAN DEFAULT True,
+  shoreline_clipped BOOLEAN DEFAULT True,
+  optimize_clipping BOOLEAN DEFAULT False,
+  simplify_geometries BOOLEAN DEFAULT False,
+  area_normalized BOOLEAN DEFAULT False,
+  extent INTEGER DEFAULT 4096,
+  buf INTEGER DEFAULT 256,
+  clip_geom BOOLEAN DEFAULT True)
+RETURNS TABLE (
+  x INTEGER,
+  y INTEGER,
+  zoom INTEGER,
+  mvtgeom GEOMETRY,
+  mvtdata JSONB
+)
+AS $$
+DECLARE
+  state_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.state';
+  county_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.county';
+  tract_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.census_tract';
+  blockgroup_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.block_group';
+  block_geoname CONSTANT TEXT DEFAULT 'us.census.tiger.block';
+
+  tiler_table_prefix CONSTANT TEXT DEFAULT 'tiler.xyz_us_do_geoms_tiles_temp_';
+  avg_x INTEGER;
+  avg_y INTEGER;
+
+  mc_schema CONSTANT TEXT DEFAULT 'us.mastercard';
+  mc_geoid CONSTANT TEXT DEFAULT 'region_id';
+  mc_category_column CONSTANT TEXT DEFAULT 'category';
+  mc_month_column CONSTANT TEXT DEFAULT 'month';
+  mc_table TEXT;
+  mc_category TEXT;
+  mc_category_name TEXT;
+  mc_table_categories TEXT DEFAULT '';
+  mc_month TEXT;
+  mc_month_slug TEXT;
+  mc_measurements_categories TEXT[];
+  mc_measurement TEXT;
+
+  measurement TEXT;
+  getmeta_parameters TEXT;
+  meta JSON;
+  mc_geography_level TEXT;
+
+  simplification_tolerance NUMERIC DEFAULT 0;
+  area_normalization TEXT DEFAULT '';
+  clipped TEXT default '';
+  i INTEGER DEFAULT 0;
+
+  bounds NUMERIC[];
+  geom GEOMETRY;
+  ext BOX2D;
+
+  numer_tablename_do TEXT DEFAULT '';
+  numer_tablenames_do TEXT[] DEFAULT ARRAY['']::TEXT[];
+  numer_tablenames_do_outer TEXT DEFAULT '';
+  numer_tablenames_mc TEXT DEFAULT '';
+  numer_colnames_do TEXT DEFAULT '';
+  numer_colnames_do_qualified TEXT DEFAULT '';
+  numer_colnames_do_normalized TEXT DEFAULT '';
+  numer_colnames_mc TEXT DEFAULT '';
+  numer_colnames_mc_current TEXT DEFAULT '';
+  numer_colnames_mc_qualified TEXT DEFAULT '';
+  numer_colnames_mc_qualified_current TEXT DEFAULT '';
+  numer_colnames_mc_normalized TEXT DEFAULT '';
+  numer_colnames_mc_normalized_current TEXT DEFAULT '';
+  geom_tablenames TEXT;
+  geom_colnames TEXT;
+  geom_geomref_colnames TEXT;
+  geom_geomref_colnames_qualified TEXT;
+  geom_relations_do TEXT[] DEFAULT ARRAY['']::TEXT[];
+  geom_relations_mc TEXT DEFAULT '';
+  geom_mc_outerjoins TEXT DEFAULT '';
+BEGIN
+  IF geography_level = 'us.census.tiger.census_tract' THEN
+    mc_geography_level := 'tract';
+  ELSE
+    mc_geography_level := (string_to_array(geography_level, '.'))[array_length(string_to_array(geography_level, '.'), 1)];
+  END IF;
+
+  -- Get the average x and y (in the middle of the BBox)
+  EXECUTE
+  format(
+    $query$
+    SELECT ROUND(AVG(x)) AS x, ROUND(AVG(y)) as y
+    FROM %3$s%1$s_%2$s
+    $query$, mc_geography_level, z, tiler_table_prefix)
+  INTO avg_x, avg_y;
+
+  IF area_normalized THEN
+    area_normalization := '/area_ratio';
+  END IF;
+
+  IF shoreline_clipped THEN
+    clipped := '_clipped';
+  END IF;
+
+  CASE
+    WHEN geography_level = state_geoname THEN
+      simplification_tolerance := 0.1;
+      IF optimize_clipping THEN
+        clipped := '';
+      END IF;
+    WHEN geography_level = county_geoname THEN
+      simplification_tolerance := 0.01;
+    WHEN geography_level = tract_geoname THEN
+      simplification_tolerance := 0.001;
+    WHEN geography_level = blockgroup_geoname THEN
+      simplification_tolerance := 0.0001;
+    WHEN geography_level = block_geoname THEN
+      simplification_tolerance := 0.0001;
+    ELSE
+      simplification_tolerance := 0;
+  END CASE;
+
+  IF NOT simplify_geometries THEN
+    simplification_tolerance := 0;
+  END IF;
+
+  bounds := cdb_observatory.OBS_GetTileBounds(z, avg_x, avg_y);
+  geom := ST_MakeEnvelope(bounds[1], bounds[2], bounds[3], bounds[4], 4326);
+  ext := ST_MakeBox2D(ST_Transform(ST_SetSRID(ST_Point(bounds[1], bounds[2]), 4326), 3857),
+                      ST_Transform(ST_SetSRID(ST_Point(bounds[3], bounds[4]), 4326), 3857));
+
+  ---------DO---------
+  getmeta_parameters := '[ ';
+  FOREACH measurement IN ARRAY do_measurements LOOP
+    getmeta_parameters := getmeta_parameters || '{"numer_id":"' || measurement || '","geom_id":"' || geography_level || clipped ||'"},';
+  END LOOP;
+  getmeta_parameters := substring(getmeta_parameters from 1 for length(getmeta_parameters) - 1) || ' ]';
+
+  IF use_meta_cache THEN
+    meta := cdb_observatory.OBS_RetrieveMeta(z, geom, getmeta_parameters::json, 1::integer, 1::integer, 1::integer);
+  ELSE
+    meta := cdb_observatory.obs_getmeta(geom, getmeta_parameters::json, 1::integer, 1::integer, 1::integer);
+  END IF;
+
+  IF meta IS NOT NULL THEN
+    SELECT  array_agg(distinct 'observatory.'||numer_tablename) numer_tablenames,
+            string_agg(distinct numer_colname, ',')||',' numer_colnames,
+            string_agg(distinct numer_tablename||'.'||numer_colname, ',')||',' numer_colnames_qualified,
+            string_agg(distinct numer_colname||area_normalization||' '||numer_colname, ',')||',' numer_colnames_normalized,
+            (array_agg(distinct 'observatory.'||geom_tablename))[1] geom_tablenames,
+            (array_agg(distinct geom_colname))[1] geom_colnames,
+            (array_agg(distinct geom_geomref_colname))[1] geom_geomref_colnames,
+            (array_agg(distinct geom_tablename||'.'||geom_geomref_colname))[1] geom_geomref_colnames_qualified,
+            array_agg(distinct numer_tablename||'.'||numer_geomref_colname||'='||geom_tablename||'.'||geom_geomref_colname) geom_relations
+      INTO numer_tablenames_do, numer_colnames_do, numer_colnames_do_qualified, numer_colnames_do_normalized, geom_tablenames, geom_colnames,
+          geom_geomref_colnames, geom_geomref_colnames_qualified, geom_relations_do
+      FROM json_to_recordset(meta)
+        AS x(id TEXT, numer_id TEXT, numer_aggregate TEXT, numer_colname TEXT, numer_geomref_colname TEXT, numer_tablename TEXT,
+            numer_type TEXT, denom_id TEXT, denom_aggregate TEXT, denom_colname TEXT, denom_geomref_colname TEXT, denom_tablename TEXT,
+            denom_type TEXT, denom_reltype TEXT, geom_id TEXT, geom_colname TEXT, geom_geomref_colname TEXT, geom_tablename TEXT,
+            geom_type TEXT, numer_timespan TEXT, geom_timespan TEXT, normalization TEXT, api_method TEXT, api_args JSON);
+
+    IF numer_tablenames_do IS NULL OR numer_colnames_do IS NULL OR numer_colnames_do_qualified IS NULL OR numer_colnames_do_normalized IS NULL
+      OR geom_tablenames IS NULL OR geom_colnames IS NULL OR geom_geomref_colnames IS NULL OR geom_geomref_colnames_qualified IS NULL
+      OR geom_relations_do IS NULL THEN
+      RETURN;
+    END IF;
+
+    i := 0;
+    FOREACH numer_tablename_do IN ARRAY numer_tablenames_do LOOP
+      i := i + 1;
+      numer_tablenames_do_outer := numer_tablenames_do_outer || 'LEFT OUTER JOIN ' || numer_tablename_do || ' ON ' || geom_relations_do[i] || ' ';
+    END LOOP;
+  ELSE
+    getmeta_parameters := '[{"geom_id":"' || geography_level || clipped ||'"}]';
+    meta := cdb_observatory.obs_getmeta(geom, getmeta_parameters::json, 1::integer, 1::integer, 1::integer);
+
+    IF meta IS NULL THEN
+      RETURN;
+    END IF;
+
+    SELECT  (array_agg(distinct 'observatory.'||geom_tablename))[1] geom_tablenames,
+            (array_agg(distinct geom_colname))[1] geom_colnames,
+            (array_agg(distinct geom_geomref_colname))[1] geom_geomref_colnames,
+            (array_agg(distinct geom_tablename||'.'||geom_geomref_colname))[1] geom_geomref_colnames_qualified
+      FROM json_to_recordset(meta)
+      INTO geom_tablenames, geom_colnames, geom_geomref_colnames, geom_geomref_colnames_qualified
+        AS x(id TEXT, numer_id TEXT, numer_aggregate TEXT, numer_colname TEXT, numer_geomref_colname TEXT, numer_tablename TEXT,
+            numer_type TEXT, denom_id TEXT, denom_aggregate TEXT, denom_colname TEXT, denom_geomref_colname TEXT, denom_tablename TEXT,
+            denom_type TEXT, denom_reltype TEXT, geom_id TEXT, geom_colname TEXT, geom_geomref_colname TEXT, geom_tablename TEXT,
+            geom_type TEXT, numer_timespan TEXT, geom_timespan TEXT, normalization TEXT, api_method TEXT, api_args JSON);
+  END IF;
+
+  ---------MC---------
+  IF geography_level = 'us.census.tiger.census_tract' THEN
+    mc_geography_level := 'tract';
+  ELSE
+    mc_geography_level := (string_to_array(geography_level, '.'))[array_length(string_to_array(geography_level, '.'), 1)];
+  END IF;
+
+  mc_table := cdb_observatory.OBS_GetMCTable(mc_schema, mc_geography_level);
+
+  FOREACH mc_month IN ARRAY mc_months LOOP
+    mc_month_slug := replace(mc_month, '/', '');
+    FOREACH mc_category IN ARRAY mc_categories LOOP
+      mc_category := lower(mc_category);
+      mc_measurements_categories := ARRAY['']::TEXT[];
+
+      FOREACH mc_measurement IN ARRAY mc_measurements LOOP
+        mc_measurements_categories := array_append(mc_measurements_categories, mc_measurement||'_'||mc_category);
+      END LOOP;
+
+      SELECT string_agg(column_name||'_'||mc_month_slug, ','),
+            string_agg(mc_table||'_'||mc_month_slug||'.'||column_name||' '||column_name||'_'||mc_month_slug, ','),
+            string_agg(distinct column_name||'_'||mc_month_slug||area_normalization||' '||column_name||'_'||mc_month_slug, ',')
+        INTO numer_colnames_mc_current, numer_colnames_mc_qualified_current, numer_colnames_mc_normalized_current
+        FROM information_schema.columns
+      WHERE table_schema = mc_schema
+        AND table_name = mc_table
+        AND column_name = ANY(mc_measurements_categories);
+
+      IF numer_colnames_mc_current IS NOT NULL THEN
+        numer_colnames_mc := coalesce(numer_colnames_mc, '')||numer_colnames_mc_current||',';
+      END IF;
+      IF numer_colnames_mc_qualified_current IS NOT NULL THEN
+        numer_colnames_mc_qualified := coalesce(numer_colnames_mc_qualified, '')||numer_colnames_mc_qualified_current||',';
+      END IF;
+      IF numer_colnames_mc_normalized_current IS NOT NULL THEN
+        numer_colnames_mc_normalized := coalesce(numer_colnames_mc_normalized, '')||numer_colnames_mc_normalized_current||',';
+      END IF;
+    END LOOP;
+
+    IF mc_table IS NOT NULL THEN
+      numer_tablenames_mc := '"'||mc_schema||'".'||mc_table||' '||mc_table||'_'||mc_month_slug;
+      geom_relations_mc := mc_table||'_'||mc_month_slug||'.'||mc_geoid||'='||geom_geomref_colnames_qualified;
+      mc_table_categories := mc_table||'_'||mc_month_slug||'.'||mc_month_column||'='''||mc_month||'''';
+
+      geom_mc_outerjoins := coalesce(geom_mc_outerjoins, '')||' LEFT OUTER JOIN '||numer_tablenames_mc||' ON '||geom_relations_mc||' AND '||mc_table_categories;
+    END IF;
+  END LOOP;
+
+  ---------Query build and execution---------
+  RETURN QUERY EXECUTE format(
+    $query$
+    SELECT  x, y, z,
+            mvtgeom,
+            (select row_to_json(_)::jsonb from (select id, %9$s %3$s area_ratio, area) as _) as mvtdata
+          FROM (
+      SELECT x, y, z,
+             ST_AsMVTGeom(ST_Transform(the_geom, 3857),
+                          bbox2d, $1, $2, $3) AS mvtgeom, %8$s as id, %6$s %7$s area_ratio, area FROM (
+        SELECT  tx.x, tx.y, tx.z,
+                %1$s the_geom, %8$s, %2$s %10$s
+                CASE  WHEN ST_Within(tx.envelope, %1$s)
+                        THEN ST_Area(tx.envelope) / Nullif(ST_Area(%1$s), 0)
+                      WHEN ST_Within(%1$s, tx.envelope)
+                        THEN 1
+                      ELSE ST_Area(ST_Intersection(st_simplifyvw(%1$s, $4), tx.envelope)) / Nullif(ST_Area(%1$s), 0)
+                END area_ratio,
+                ROUND(ST_Area(ST_Transform(the_geom,3857))::NUMERIC, 2) area,
+                ST_MakeBox2D(ST_Transform(ST_SetSRID(ST_Point(tx.bounds[1], tx.bounds[2]), 4326), 3857),
+                             ST_Transform(ST_SetSRID(ST_Point(tx.bounds[3], tx.bounds[4]), 4326), 3857)) bbox2d
+          FROM tiler.xyz_us_mc_tiles_temp_%12$s_%13$s tx,
+               %5$s
+               %4$s
+               %11$s
+        WHERE st_intersects(%1$s, tx.envelope)
+      ) p
+    ) q
+    $query$,
+    geom_colnames, numer_colnames_do_qualified, numer_colnames_mc, numer_tablenames_do_outer, geom_tablenames, numer_colnames_do_normalized,
+    numer_colnames_mc_normalized, geom_geomref_colnames, numer_colnames_do, numer_colnames_mc_qualified, geom_mc_outerjoins,
+    mc_geography_level, z)
+  USING extent, buf, clip_geom, simplification_tolerance
+  RETURN;
+END
+$$ LANGUAGE plpgsql PARALLEL RESTRICTED;

src/pg/sql/50_table_level_framework.sql

@@ -1,82 +0,0 @@
-CREATE TYPE cdb_observatory.ds_fdw_metadata as (schemaname text, tabname text, servername text);
-CREATE TYPE cdb_observatory.ds_return_metadata as (colnames text[], coltypes text[]);
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_ConnectUserTable(username text, orgname text, user_db_role text, input_schema text, dbname text, host_addr text, table_name text)
-RETURNS cdb_observatory.ds_fdw_metadata
-AS $$
-DECLARE
-  fdw_server text;
-  fdw_import_schema text;
-  connection_str json;
-  import_foreign_schema_q text;
-  epoch_timestamp text;
-BEGIN
-
-  SELECT extract(epoch from now() at time zone 'utc')::int INTO epoch_timestamp;
-  fdw_server := 'fdw_server_' || username || '_' || epoch_timestamp;
-  fdw_import_schema:= fdw_server;
-
-  -- Import foreign table
-  EXECUTE FORMAT ('SELECT cdb_observatory._OBS_ConnectRemoteTable(%L, %L, %L, %L, %L, %L, %L)', fdw_server, fdw_import_schema, dbname, host_addr, user_db_role, table_name, input_schema);
-
-  RETURN (fdw_import_schema::text, table_name::text, fdw_server::text);
-
-EXCEPTION
-  WHEN others THEN
-    -- Disconnect user imported table. Delete schema and FDW server.
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || fdw_import_schema || '".' || table_name;
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || fdw_import_schema || '".cdb_tablemetadata';
-    EXECUTE 'DROP SCHEMA IF EXISTS "' || fdw_import_schema || '"';
-    EXECUTE 'DROP USER MAPPING IF EXISTS FOR public SERVER "' || fdw_server || '"';
-    EXECUTE 'DROP SERVER IF EXISTS "' || fdw_server || '"';
-
-    RETURN (null, null, null);
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetReturnMetadata(username text, orgname text, function_name text, params json)
-RETURNS cdb_observatory.ds_return_metadata
-AS $$
-DECLARE
-  colnames text[];
-  coltypes text[];
-BEGIN
-  EXECUTE FORMAT('SELECT r.colnames::text[], r.coltypes::text[] FROM cdb_observatory._%sResultMetadata(%L::json) r', function_name, params::text)
-  INTO colnames, coltypes;
-
-  RETURN (colnames::text[], coltypes::text[]);
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_FetchJoinFdwTableData(username text, orgname text, table_schema text, table_name text, function_name text, params json)
-RETURNS SETOF record
-AS $$
-DECLARE
-  data_query text;
-  rec RECORD;
-BEGIN
-
-    EXECUTE FORMAT('SELECT cdb_observatory._%sQuery(%L, %L, %L::json)', function_name, table_schema, table_name, params::text)
-    INTO data_query;
-
-    FOR rec IN EXECUTE data_query
-    LOOP
-        RETURN NEXT rec;
-    END LOOP;
-  RETURN;
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;
-
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_DisconnectUserTable(username text, orgname text, table_schema text, table_name text, servername text)
-RETURNS boolean
-AS $$
-BEGIN
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".' || table_name;
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".cdb_tablemetadata';
-    EXECUTE 'DROP SCHEMA IF EXISTS "' || table_schema || '"';
-    EXECUTE 'DROP USER MAPPING IF EXISTS FOR public SERVER "' || servername || '"';
-    EXECUTE 'DROP SERVER IF EXISTS "' || servername || '"';
-    RETURN true;
-END;
-$$ LANGUAGE plpgsql SECURITY DEFINER;

src/pg/sql/51_table_level_functions.sql

@@ -1,79 +0,0 @@
---
---
--- OBS_GetMeasure
---
---
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetMeasureResultMetadata(params json)
-RETURNS cdb_observatory.ds_return_metadata
-AS $$
-DECLARE
-  colnames text[]; -- Array to store the name of the measures to be returned
-  coltypes text[]; -- Array to store the type of the measures to be returned
-  requested_measures text[];
-  measure_id text;
-BEGIN
-  -- By definition, all the measure results for the OBS_GetMeasure API are numeric values
-  SELECT ARRAY(SELECT json_array_elements_text(params->'measure_id'))::text[] INTO requested_measures;
-
-  FOREACH measure_id IN ARRAY requested_measures
-  LOOP
-    SELECT array_append(colnames, measure_id) INTO colnames;
-    SELECT array_append(coltypes, 'numeric'::text) INTO coltypes;
-  END LOOP;
-
-  RETURN (colnames::text[], coltypes::text[]);
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetMeasureQuery(table_schema text, table_name text, params json)
-RETURNS text
-AS $$
-DECLARE
-  data_query text;
-  measure_ids_arr text[];
-  measure_id text;
-  measures_list text;
-  measures_query text;
-  normalize text;
-  boundary_id text;
-  time_span text;
-  geom_table_name text;
-  data_table_name text;
-BEGIN
-    measures_query := '';
-    -- SELECT table_name from obs_meta WHERE boundary_id = {bound} AND [...] INTO geom_table_name
-    geom_table_name := 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308';
-    -- SELECT table_name from obs_meta WHERE time_span = {time} AND [...] INTO data_table_name
-    data_table_name := 'observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd';
-
-    -- Get measure_ids array from JSON
-    SELECT ARRAY(SELECT json_array_elements_text(params->'measure_id'))::text[] INTO measure_ids_arr;
-
-    -- Get a comma-separated list of measures ("total_pop, over_16_pop") to be used in SELECTs
-    SELECT array_to_string(measure_ids_arr, ',') INTO measures_list;
-
-    FOREACH measure_id IN ARRAY measure_ids_arr
-    LOOP
-      -- Build query to compute each value and normalize
-      -- Assumes the default normalization method, the normalize parameter given in the JSON
-      -- should be checked in order to build the final query
-      SELECT measures_query || ' sum(' || measure_id || '/fraction)::numeric as ' || measure_id || ', ' INTO measures_query;
-    END LOOP;
-
-    -- Data query should select the measures and the cartodb_id of the user table, in that order.
-    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
-        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
-        || geom_table_name || ' as a, '
-        || table_schema || '.' || table_name || ' AS b '
-        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
-        || measures_list
-        || ' FROM ' || data_table_name || ' ) '
-        || 'SELECT '
-        || measures_query
-        || ' cartodb_id::int FROM _areas, values '
-        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';
-    RETURN data_query;
-END;
-$$ LANGUAGE plpgsql;
-

src/pg/test/expected/01_install_test.out

@@ -1,5 +1,167 @@
--- Install dependencies
-CREATE EXTENSION postgis;
-CREATE EXTENSION postgres_fdw;
 -- Install the extension
-CREATE EXTENSION observatory VERSION 'dev';
+\set ECHO none
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+
+ set_config 
+------------
+ 
+(1 row)
+

src/pg/test/expected/40_observatory_utility_test.out

@@ -1,6 +1,5 @@
 \pset format unaligned
 \set ECHO all
-\i test/fixtures/load_fixtures.sql
 SET client_min_messages TO WARNING;
 \set ECHO none
 _obs_geomtable_with_returned_table
@@ -21,3 +20,6 @@ t
 obs_dumpversion_notnull
 t
 (1 row)
+complex_safe_intersection_works
+t
+(1 row)

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -3,36 +3,12 @@
 obs_getdemographicsnapshot_test_no_returns
 t
 (1 row)
-obs_get_median_income_at_test_point
-t
-(1 row)
-obs_get_median_income_at_null_island
-t
-(1 row)
-obs_getpoints_for_test_point_value|obs_getpoints_for_test_point_name|obs_getpoints_for_test_point_tablename|obs_getpoints_for_test_point_aggregate|obs_getpoints_for_test_point_type|obs_getpoints_for_test_point_description
-t|t|t|t|t|t
-(1 row)
-obs_getpoints_for_null_island
-t
-(1 row)
-obs_getpolygons_for_test_point
-t
-(1 row)
-obs_getpolygons_for_null_island
-t
-(1 row)
 test_point_segmentation
 t
 (1 row)
 null_island_segmentation
 t
 (1 row)
-getcategories_at_test_point_1
-t
-(1 row)
-getcategories_at_null_island
-t
-(1 row)
 obs_getmeasure_zhvi_point_test
 t
 (1 row)
@@ -72,6 +48,15 @@ t
 obs_getmeasure_out_of_bounds_geometry
 t
 (1 row)
+obs_getmeasure_estimate_for_blank_aggregate
+t
+(1 row)
+obs_getmeasure_per_capita_income_average
+t
+(1 row)
+obs_getmeasure_median_capita_income_average
+t
+(1 row)
 obs_getcategory_point
 t
 (1 row)
@@ -129,3 +114,196 @@ t
 obs_getmeasurebyid_null_id
 t
 (1 row)
+obs_getmeta_null_null_is_null
+t
+(1 row)
+obs_getmeta_null_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_null_is_null
+t
+(1 row)
+obs_getmeta_nullisland_empty_is_null
+t
+(1 row)
+obs_getmeta_nullisland_us_measure_is_null
+t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization|id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+id|numer_id|timespan_rank|score_rank|numer_aggregate|numer_colname|numer_type|numer_name|denom_id|denom_aggregate|denom_colname|denom_type|denom_name|geom_id|normalization
+t|t|t|t|t|t|t|t|t|t|t|t|t|t|t
+(1 row)
+obs_getmeta_conflicting_metadata
+t
+(1 row)
+obs_getmeta_suggested_name
+t
+(1 row)
+obs_getmeta_suggested_name_implicit_area
+t
+(1 row)
+obs_getmeta_suggested_name_area
+t
+(1 row)
+obs_getmeta_suggested_name_denom
+t
+(1 row)
+obs_getdata_geomval_empty_null
+t
+(1 row)
+obs_getdata_text_empty_null
+t
+(1 row)
+obs_getdata_geomval_empty_one_measure
+t
+(1 row)
+id|data_point_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_null|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_area|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_prenormalized|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_prenormalized|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_predenominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_impossible_denominated|nullcol
+t|t|t
+(1 row)
+id|data_point_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_denominated|nullcol
+t|t|t
+(1 row)
+id|data_polygon_measure_one_null|data_polygon_measure_two_null
+t|t|t
+(1 row)
+id|data_polygon_measure_one_null|data_polygon_measure_two_null
+t|t|t
+(1 row)
+id|data_polygon_measure_one_predenom|data_polygon_measure_two_predenom
+t|t|t
+(1 row)
+id|data_polygon_measure_one_area|data_polygon_measure_two_area
+t|t|t
+(1 row)
+id|data_polygon_measure_tract|data_polygon_measure_bg
+t|t|t
+(1 row)
+id|data_point_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|nullcol
+t|t|t
+(1 row)
+id|data_poly_categorical|valcol
+t|t|t
+(1 row)
+id|correct_num_geoms
+t|t
+(1 row)
+id|correct_num_geoms|correct_pop
+t|t|t
+(1 row)
+id|correct_num_geoms|correct_pop|correct_bg_names
+t|t|t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_predenom
+t|t
+(1 row)
+id|obs_getdata_by_id_one_measure_null|obs_getdata_by_id_two_measure_null
+t|t|t
+(1 row)
+id|obs_getdata_by_id_categorical
+t|t
+(1 row)
+id|obs_getdata_by_id_geometry
+t|t
+(1 row)
+obs_getdata_api_geomvals_no_args
+t
+(1 row)
+ary_type|obs_getdata_api_geomvals_args_numer_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomvals_args_string_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomrefs_args_numer_return
+t|t
+(1 row)
+ary_type|obs_getdata_api_geomrefs_args_string_return
+t|t
+(1 row)
+setseed
+
+(1 row)
+bg_sample|bg_max_error|bg_avg_error|bg_min_error
+1|t|t|t
+2|t|t|t
+3|t|t|t
+5|t|t|t
+10|t|t|t
+25|t|t|t
+50|t|t|t
+100|t|t|t
+2085|t|t|t
+(9 rows)
+tract_sample|tract_max_error|tract_avg_error|tract_min_error
+1|t|t|t
+2|t|t|t
+3|t|t|t
+5|t|t|t
+10|t|t|t
+25|t|t|t
+50|t|t|t
+100|t|t|t
+741|t|t|t
+(9 rows)
+no_bg_point_error
+t
+(1 row)
+valid|errors
+t|{}
+(1 row)
+valid|errors
+f|{"Median or average aggregation only supports prenormalized normalization, denominated passed. Please review the provided options"}
+(1 row)
+valid|errors
+f|{"Normalizated measure should have a numerator and a denominator. Please review the provided options."}
+(1 row)

src/pg/test/expected/42_observatory_exploration_test.out

@@ -48,6 +48,63 @@ t
 _obs_getavailablenumerators_no_total_pop_1996
 t
 (1 row)
+_obs_getnumerators_usa_pop_in_all
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getnumerators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getnumerators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getnumerators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getnumerators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getnumerators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getnumerators_zillow_at_zcta5
+t
+(1 row)
+_obs_getnumerators_no_zillow_at_block_group
+t
+(1 row)
+_obs_getnumerators_total_pop_2010_2014
+t
+(1 row)
+_obs_getnumerators_no_total_pop_1996
+t
+(1 row)
+_obs_getnumerators_total_pop_by_name
+t
+(1 row)
+_obs_getnumerators_total_pop_by_section
+t
+(1 row)
+_obs_getnumerators_total_pop_not_in_canada
+t
+(1 row)
+_obs_getnumerators_total_pop_by_subsection
+t
+(1 row)
+_obs_getnumerators_total_pop_not_in_employment_subsection
+t
+(1 row)
+_obs_getnumerators_total_pop_by_id
+t
+(1 row)
+_obs_getnumerators_total_pop_not_with_other_id
+t
+(1 row)
 _obs_getavailabledenominators_usa_pop_in_all
 t
 (1 row)
@@ -114,12 +171,15 @@ t
 _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg
 t
 (1 row)
-_obs_getavailablegeometries_bg_2014
+_obs_getavailablegeometries_bg_2015
 t
 (1 row)
 _obs_getavailablegeometries_bg_not_1996
 t
 (1 row)
+_obs_getavailablegeometries_has_boundary_tag
+t
+(1 row)
 _obs_getavailabletimespans_2010_2014_in_all
 t
 (1 row)
@@ -159,21 +219,36 @@ t
 _obs_geometryscores_2500km_buffer
 t
 (1 row)
-_obs_geometryscores_numgeoms_500m_buffer
+column_id|_obs_geometryscores_numgeoms_500m_buffer
-t
+us.census.tiger.block_group|2
-(1 row)
+us.census.tiger.census_tract|1
-_obs_geometryscores_numgeoms_5km_buffer
+us.census.tiger.zcta5|0
-t
+us.census.tiger.county|0
-(1 row)
+(4 rows)
-_obs_geometryscores_numgeoms_50km_buffer
+column_id|_obs_geometryscores_numgeoms_5km_buffer
-t
+us.census.tiger.block_group|244
-(1 row)
+us.census.tiger.census_tract|78
-_obs_geometryscores_numgeoms_500km_buffer
+us.census.tiger.zcta5|9
-t
+us.census.tiger.county|0
-(1 row)
+(4 rows)
-_obs_geometryscores_numgeoms_2500km_buffer
+column_id|_obs_geometryscores_numgeoms_50km_buffer
-t
+us.census.tiger.block_group|10818
-(1 row)
+us.census.tiger.census_tract|3396
+us.census.tiger.zcta5|483
+us.census.tiger.county|11
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_500km_buffer
+us.census.tiger.block_group|48569
+us.census.tiger.census_tract|15825
+us.census.tiger.zcta5|6465
+us.census.tiger.county|295
+(4 rows)
+column_id|_obs_geometryscores_numgeoms_2500km_buffer
+us.census.tiger.block_group|165852
+us.census.tiger.census_tract|55283
+us.census.tiger.zcta5|26529
+us.census.tiger.county|2551
+(4 rows)
 _obs_geometryscores_500km_buffer_50_geoms
 t
 (1 row)
@@ -186,16 +261,28 @@ t
 _obs_geometryscores_500km_buffer_25000_geoms
 t
 (1 row)
+testarea_uses_tract
+t
+(1 row)
+points_use_bg
+t
+(1 row)
 _total_pop_in_legacy_builder_metadata
 t
 (1 row)
 _median_income_in_legacy_builder_metadata
 t
 (1 row)
+_gini_in_legacy_builder_metadata
+t
+(1 row)
 _total_pop_in_legacy_builder_metadata_sums
 t
 (1 row)
-_median_income_not_in_legacy_builder_metadata_sums
+_median_income_in_legacy_builder_metadata_sums
+t
+(1 row)
+_gini_not_in_legacy_builder_metadata_sums
 t
 (1 row)
 _no_dupe_subsections_in_legacy_builder_metadata

src/pg/test/expected/44_observatory_geometries_test.out

@@ -54,9 +54,6 @@ t
 obs_getboundariesbygeometry_tracts_around_null_island
 t
 (1 row)
-obs_getboundariesbygeometry_wof
-t
-(1 row)
 obs_getboundariesbypointandradius_around_cartodb
 t
 (1 row)
@@ -87,9 +84,3 @@ t
 obs_getpointsbypointandradius_around_null_island
 t
 (1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)
-geoid_name_matches|table_name_matches|geom_name_matches
-t|t|t
-(1 row)

src/pg/test/fixtures/drop_fixtures.sql

@@ -8,22 +8,28 @@ DROP TABLE IF EXISTS observatory.obs_tag;
 DROP TABLE IF EXISTS observatory.obs_column_to_column;
 DROP TABLE IF EXISTS observatory.obs_dump_version;
 DROP TABLE IF EXISTS observatory.obs_meta;
+DROP TABLE IF EXISTS observatory.obs_table_to_table;
 DROP TABLE IF EXISTS observatory.obs_meta_numer;
 DROP TABLE IF EXISTS observatory.obs_meta_denom;
 DROP TABLE IF EXISTS observatory.obs_meta_geom;
 DROP TABLE IF EXISTS observatory.obs_meta_timespan;
+DROP TABLE IF EXISTS observatory.obs_meta_geom_numer_timespan;
 DROP TABLE IF EXISTS observatory.obs_column_table_tile;
-DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile_simple;
-DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
+DROP TABLE IF EXISTS observatory.obs_78fb6c1d6ff6505225175922c2c389ce48d7632c;
-DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
+DROP TABLE IF EXISTS observatory.obs_fae094ddb7157380e2495b9867e1f067fdbdf288;
-DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
+DROP TABLE IF EXISTS observatory.obs_d03c931c9b7f9df54c3fae95bb7f958fe3187c71;
+DROP TABLE IF EXISTS observatory.obs_a6811c89ed79ab4339d89a86907b586439cc74df;
 DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
-DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
+DROP TABLE IF EXISTS observatory.obs_3b537fe9a1dcdd3be4a53f64429e30a836ecb6ee;
-DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
+DROP TABLE IF EXISTS observatory.obs_c4411eba732408d47d73281772dbf03d60645dec;
-DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
-DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
-DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
-DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
-DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
-DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
 DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
+DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
+DROP TABLE IF EXISTS observatory.obs_8e30e6b3792430b410ba5b9e49cdc6a0d404d48f;
+DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
+DROP TABLE IF EXISTS observatory.obs_87a814e485deabe3b12545a537f693d16ca702c2;
+DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
+DROP TABLE IF EXISTS observatory.obs_9b319c207dfa600c2296a6d46055e54a4c00f646;
+DROP TABLE IF EXISTS observatory.obs_9b319c207dfa600c2296a6d46055e54a4c00f646;
+DROP TABLE IF EXISTS observatory.obs_0310c639744a2014bb1af82709228f05b59e7d3d;
+DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;

src/pg/test/sql/01_install_test.sql

@@ -1,6 +1,16 @@
--- Install dependencies
-CREATE EXTENSION postgis;
-CREATE EXTENSION postgres_fdw;
-
 -- Install the extension
-CREATE EXTENSION observatory VERSION 'dev';
+\set ECHO none
+\set QUIET on
+SET client_min_messages TO ERROR;
+
+-- For Postgis 3+ install postgis_raster. Otherwise observatory will fail to install
+DO $$
+BEGIN
+    IF EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'postgis_raster') THEN
+        CREATE EXTENSION postgis_raster WITH SCHEMA public CASCADE;
+    END IF;
+END$$;
+
+CREATE EXTENSION observatory VERSION 'dev' CASCADE;
+
+\i test/fixtures/load_fixtures.sql

src/pg/test/sql/40_observatory_utility_test.sql

@@ -1,6 +1,5 @@
 \pset format unaligned
 \set ECHO all
-\i test/fixtures/load_fixtures.sql
 SET client_min_messages TO WARNING;
 \set ECHO none
 
@@ -11,8 +10,8 @@ SELECT
   cdb_observatory._OBS_GeomTable(
     ST_SetSRID(ST_Point(-74.0059, 40.7128), 4326),
     'us.census.tiger.census_tract',
-    '2014'
+    '2015'
-  ) = 'obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7' As _obs_geomtable_with_returned_table;
+  ) = 'obs_87a814e485deabe3b12545a537f693d16ca702c2' As _obs_geomtable_with_returned_table;
 
 -- get null for unknown geometry_id
 -- should give back null
@@ -47,3 +46,9 @@ SELECT cdb_observatory._OBS_StandardizeMeasureName('test 343 %% 2 qqq }}{{}}') =
 
 SELECT cdb_observatory.OBS_DumpVersion()
   IS NOT NULL AS OBS_DumpVersion_notnull;
+
+-- Should succeed in intersecting
+SELECT ST_IsValid(cdb_observatory.safe_intersection(
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county'),
+        cdb_observatory.OBS_GetBoundaryByID('48061', 'us.census.tiger.county_clipped')
+)) AS complex_safe_intersection_works;

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -5,138 +5,31 @@ SET client_min_messages TO WARNING;
 --
 WITH result as(
   Select count(coalesce(OBS_GetDemographicSnapshot->>'value', 'foo')) expected_columns
-  FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint())
+  FROM cdb_observatory.OBS_GetDemographicSnapshot(cdb_observatory._TestPoint(), '2010 - 2014')
 ) select expected_columns = 52 as OBS_GetDemographicSnapshot_test_no_returns
 FROM result;
 
-WITH result as (
-  SELECT _OBS_Get::text as expected FROM
-  cdb_observatory._OBS_Get(
-    cdb_observatory._TestPoint(),
-    Array['us.census.acs.B19013001']::text[],
-    '2010 - 2014',
-    'us.census.tiger.block_group'
-  )
-) SELECT expected = '{"value":79292,"name":"Median Household Income in the past 12 Months","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"median","type":"Numeric","description":"Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc.  It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans'' (VA) payments, unemployment and/or worker''s compensation, child support, and alimony."}'
-  As OBS_Get_median_income_at_test_point
-  FROM result;
-
--- median income at null island
-WITH result as (
-  SELECT count(_OBS_Get) as expected FROM
-  cdb_observatory._OBS_Get(
-    ST_SetSRID(ST_Point(0, 0), 4326),
-    Array['us.census.acs.B19013001']::text[],
-    '2010 - 2014',
-    'us.census.tiger.block_group'
-  )
-) select expected = 0 as OBS_Get_median_income_at_null_island
-  from result;
-
--- OBS_GetPoints
--- obs_getpoints
--- --------------------
--- {4809.33511352425}
-
--- SELECT
---   (cdb_observatory._OBS_GetPoints(
---     cdb_observatory._TestPoint(),
---     'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- block groups (see _obs_geomtable)
---     (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
---   ))[1]::text = '{"value":10923.093200390833950,"name":"Total Population","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'
---   as OBS_GetPoints_for_test_point;
-WITH cte As (
-SELECT
-  (cdb_observatory._OBS_GetPoints(
-    cdb_observatory._TestPoint(),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- block groups (see _obs_geomtable)
-    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  ))[1]
-  as OBS_GetPoints_for_test_point)
-SELECT
-  (abs((OBS_GetPoints_for_test_point ->> 'value')::numeric - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPoints_for_test_point_value,
-  (OBS_GetPoints_for_test_point ->> 'name') = 'Total Population' As OBS_GetPoints_for_test_point_name,
-  (OBS_GetPoints_for_test_point ->> 'tablename') = 'obs_1a098da56badf5f32e336002b0a81708c40d29cd' As OBS_GetPoints_for_test_point_tablename,
-  (OBS_GetPoints_for_test_point ->> 'aggregate') = 'sum' As OBS_GetPoints_for_test_point_aggregate,
-  (OBS_GetPoints_for_test_point ->> 'type') = 'Numeric' As OBS_GetPoints_for_test_point_type,
-  (OBS_GetPoints_for_test_point ->> 'description') = 'The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.' As OBS_GetPoints_for_test_point_description
-FROM cte;
-
--- what happens at null island
-
-SELECT
-  (cdb_observatory._OBS_GetPoints(
-    ST_SetSRID(ST_Point(0, 0), 4326),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  ))[1]::text  is null
-  as OBS_GetPoints_for_null_island;
-
--- OBS_GetPolygons
---   obs_getpolygons
--- --------------------
---  {12996.8172420752}
-
-SELECT
-  (cdb_observatory._OBS_GetPolygons(
-    cdb_observatory._TestArea(),
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json]
-))[1]::text = '{"value":12327.3133495107,"name":"Total Population","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'
-  as OBS_GetPolygons_for_test_point;
-
--- see what happens around null island
-SELECT
-  ((cdb_observatory._OBS_GetPolygons(
-    ST_Buffer(ST_SetSRID(ST_Point(0, 0), 4326)::geography, 500)::geometry,
-    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
-    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
-  )[1]->>'value') is null
-  as OBS_GetPolygons_for_null_island;
-
 SELECT cdb_observatory.OBS_GetSegmentSnapshot(
     cdb_observatory._TestPoint(),
     'us.census.tiger.census_tract'
-)::text =
+)::JSONB =
- '{"x10_segment":"Wealthy, urban without Kids","x55_segment":"Wealthy transplants displacing long-term local residents","us.census.acs.B01003001_quantile":"0.3235","us.census.acs.B01001002_quantile":"0.494716216216216","us.census.acs.B01001026_quantile":"0.183756756756757","us.census.acs.B01002001_quantile":"0.0752837837837838","us.census.acs.B03002003_quantile":"0.293162162162162","us.census.acs.B03002004_quantile":"0.455527027027027","us.census.acs.B03002006_quantile":"0.656405405405405","us.census.acs.B03002012_quantile":"0.840081081081081","us.census.acs.B05001006_quantile":"0.727135135135135","us.census.acs.B08006001_quantile":"0.688635135135135","us.census.acs.B08006002_quantile":"0.0204459459459459","us.census.acs.B08006008_quantile":"0.679324324324324","us.census.acs.B08006009_quantile":"0.996716216216216","us.census.acs.B08006011_quantile":"0.967418918918919","us.census.acs.B08006015_quantile":"0.512945945945946","us.census.acs.B08006017_quantile":"0.0504864864864865","us.census.acs.B09001001_quantile":"0.192405405405405","us.census.acs.B11001001_quantile":"0.331702702702703","us.census.acs.B14001001_quantile":"0.296283783783784","us.census.acs.B14001002_quantile":"0.045472972972973","us.census.acs.B14001005_quantile":"0.0442702702702703","us.census.acs.B14001006_quantile":"0.0829054054054054","us.census.acs.B14001007_quantile":"0.701135135135135","us.census.acs.B14001008_quantile":"0.404527027027027","us.census.acs.B15003001_quantile":"0.191824324324324","us.census.acs.B15003017_quantile":"0.864162162162162","us.census.acs.B15003022_quantile":"0.754297297297297","us.census.acs.B15003023_quantile":"0.350054054054054","us.census.acs.B16001001_quantile":"0.217635135135135","us.census.acs.B16001002_quantile":"0.85972972972973","us.census.acs.B16001003_quantile":"0.342851351351351","us.census.acs.B17001001_quantile":"0.51204054054054","us.census.acs.B17001002_quantile":"0.813540540540541","us.census.acs.B19013001_quantile":"0.0948648648648649","us.census.acs.B19083001_quantile":"0.678351351351351","us.census.acs.B19301001_quantile":"0.146108108108108","us.census.acs.B25001001_quantile":"0.149067567567568","us.census.acs.B25002003_quantile":"0","us.census.acs.B25004002_quantile":"0","us.census.acs.B25004004_quantile":"0.944554054054054","us.census.acs.B25058001_quantile":"0.398040540540541","us.census.acs.B25071001_quantile":"0.0596081081081081","us.census.acs.B25075001_quantile":"0","us.census.acs.B25075025_quantile":null}' as test_point_segmentation;
+ '{"x10_segment": "Wealthy, urban without Kids", "x55_segment": "Wealthy transplants displacing long-term local residents", "us.census.acs.B01001002_quantile": "0.494716216216216", "us.census.acs.B01001026_quantile": "0.183756756756757", "us.census.acs.B01002001_quantile": "0.0752837837837838", "us.census.acs.B01003001_quantile": "0.3235", "us.census.acs.B03002003_quantile": "0.293162162162162", "us.census.acs.B03002004_quantile": "0.455527027027027", "us.census.acs.B03002006_quantile": "0.656405405405405", "us.census.acs.B03002012_quantile": "0.840081081081081", "us.census.acs.B05001006_quantile": "0.727135135135135", "us.census.acs.B08006001_quantile": "0.688635135135135", "us.census.acs.B08006002_quantile": "0.0204459459459459", "us.census.acs.B08006009_quantile": "0.679324324324324", "us.census.acs.B08006011_quantile": "0.996716216216216", "us.census.acs.B08006015_quantile": "0.967418918918919", "us.census.acs.B08006017_quantile": "0.512945945945946", "us.census.acs.B08301010_quantile": "0.994743243243243", "us.census.acs.B09001001_quantile": "0.0504864864864865", "us.census.acs.B11001001_quantile": "0.192405405405405", "us.census.acs.B14001001_quantile": "0.331702702702703", "us.census.acs.B14001002_quantile": "0.296283783783784", "us.census.acs.B14001005_quantile": "0.045472972972973", "us.census.acs.B14001006_quantile": "0.0442702702702703", "us.census.acs.B14001007_quantile": "0.0829054054054054", "us.census.acs.B14001008_quantile": "0.701135135135135", "us.census.acs.B15003001_quantile": "0.404527027027027", "us.census.acs.B15003017_quantile": "0.191824324324324", "us.census.acs.B15003022_quantile": "0.864162162162162", "us.census.acs.B15003023_quantile": "0.754297297297297", "us.census.acs.B16001001_quantile": "0.350054054054054", "us.census.acs.B16001002_quantile": "0.217635135135135", "us.census.acs.B16001003_quantile": "0.85972972972973", "us.census.acs.B17001001_quantile": "0.342851351351351", "us.census.acs.B17001002_quantile": "0.51204054054054", "us.census.acs.B19013001_quantile": "0.813540540540541", "us.census.acs.B19083001_quantile": "0.0948648648648649", "us.census.acs.B19301001_quantile": "0.678351351351351", "us.census.acs.B25001001_quantile": "0.146108108108108", "us.census.acs.B25002003_quantile": "0.149067567567568", "us.census.acs.B25004002_quantile": "0", "us.census.acs.B25004004_quantile": "0", "us.census.acs.B25058001_quantile": "0.944554054054054", "us.census.acs.B25071001_quantile": "0.398040540540541", "us.census.acs.B25075001_quantile": "0.0596081081081081", "us.census.acs.B25075025_quantile": "0"}'::JSONB as test_point_segmentation;
 
 -- segmentation around null island
 SELECT cdb_observatory.OBS_GetSegmentSnapshot(
     ST_SetSRID(ST_Point(0, 0), 4326),
     'us.census.tiger.census_tract'
-)::text = '{"x10_segment":null,"x55_segment":null,"us.census.acs.B01003001_quantile":null,"us.census.acs.B01001002_quantile":null,"us.census.acs.B01001026_quantile":null,"us.census.acs.B01002001_quantile":null,"us.census.acs.B03002003_quantile":null,"us.census.acs.B03002004_quantile":null,"us.census.acs.B03002006_quantile":null,"us.census.acs.B03002012_quantile":null,"us.census.acs.B05001006_quantile":null,"us.census.acs.B08006001_quantile":null,"us.census.acs.B08006002_quantile":null,"us.census.acs.B08006008_quantile":null,"us.census.acs.B08006009_quantile":null,"us.census.acs.B08006011_quantile":null,"us.census.acs.B08006015_quantile":null,"us.census.acs.B08006017_quantile":null,"us.census.acs.B09001001_quantile":null,"us.census.acs.B11001001_quantile":null,"us.census.acs.B14001001_quantile":null,"us.census.acs.B14001002_quantile":null,"us.census.acs.B14001005_quantile":null,"us.census.acs.B14001006_quantile":null,"us.census.acs.B14001007_quantile":null,"us.census.acs.B14001008_quantile":null,"us.census.acs.B15003001_quantile":null,"us.census.acs.B15003017_quantile":null,"us.census.acs.B15003022_quantile":null,"us.census.acs.B15003023_quantile":null,"us.census.acs.B16001001_quantile":null,"us.census.acs.B16001002_quantile":null,"us.census.acs.B16001003_quantile":null,"us.census.acs.B17001001_quantile":null,"us.census.acs.B17001002_quantile":null,"us.census.acs.B19013001_quantile":null,"us.census.acs.B19083001_quantile":null,"us.census.acs.B19301001_quantile":null,"us.census.acs.B25001001_quantile":null,"us.census.acs.B25002003_quantile":null,"us.census.acs.B25004002_quantile":null,"us.census.acs.B25004004_quantile":null,"us.census.acs.B25058001_quantile":null,"us.census.acs.B25071001_quantile":null,"us.census.acs.B25075001_quantile":null,"us.census.acs.B25075025_quantile":null}' as null_island_segmentation;
+)::text is null as null_island_segmentation;
-
-WITH result as (
-  SELECT array_agg(_OBS_GetCategories) as expected FROM
-    cdb_observatory._OBS_GetCategories(
-      cdb_observatory._TestPoint(),
-      Array['us.census.spielman_singleton_segments.X10'],
-      'us.census.tiger.census_tract'
-  )
-)
-  select (expected)[1]::text = '{"category":"Wealthy, urban without Kids","name":"Spielman-Singleton Segments: 10 Clusters","tablename":"obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d","aggregate":null,"type":"Text","description":"Sociodemographic classes from Spielman and Singleton 2015, 10 clusters"}' as GetCategories_at_test_point_1
-  from result;
-
-WITH result as (
-  SELECT array_agg(_OBS_GetCategories) as expected FROM
-    cdb_observatory._OBS_GetCategories(
-      ST_SetSRID(ST_Point(0,0), 4326),
-      Array['us.census.spielman_singleton_segments.X10'],
-      'us.census.tiger.census_tract'
-  )
-)
-  select expected[0] is NULL as GetCategories_at_null_island
-  from result;
 
 -- Point-based OBS_GetMeasure with zillow
-SELECT abs(OBS_GetMeasure_zhvi_point - 597900) / 597900 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
+SELECT abs(OBS_GetMeasure_zhvi_point - 446000) / 446000 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
-  ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
+  ST_SetSRID(ST_Point(-73.90820503234865, 40.69469600456701), 4326),
   'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
 ) As t(OBS_GetMeasure_zhvi_point);
 
 -- Point-based OBS_GetMeasure with later measure
-SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 995400) / 995400 < 5.0 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
+SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 701400) / 701400 < 5.0 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
-  ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
+  ST_SetSRID(ST_Point(-73.90820503234865, 40.69469600456701), 4326),
   'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2016-06'
 ) As t(OBS_GetMeasure_zhvi_point_default_latest);
 
@@ -196,7 +89,7 @@ SELECT (abs(cdb_observatory.OBS_GetMeasure(
 -- Poly-based OBS_GetMeasure with denominator normalization
 SELECT abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._TestArea(),
-  'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
+  'us.census.acs.B01001002', 'denominator', null, '2010 - 2014') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
 
 -- Poly-based OBS_GetMeasure with one very bad geom
 SELECT abs(cdb_observatory.OBS_GetMeasure(
@@ -213,13 +106,28 @@ SELECT cdb_observatory.OBS_GetMeasure(
   ST_SetSRID(st_point(0, 0), 4326),
   'us.census.acs.B01003001') IS NULL As OBS_GetMeasure_out_of_bounds_geometry;
 
+-- OBS_GetMeasure over arbitrary area for a measure we cannot estimate
+SELECT cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.1),
+  'us.census.acs.B19083001') IS NULL As OBS_GetMeasure_estimate_for_blank_aggregate;
+
+-- OBS_GetMeasure over arbitrary area for an average measure we can estimate
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.01),
+  'us.census.acs.B19301001') - 20025) / 20025 < 0.001 As OBS_GetMeasure_per_capita_income_average;
+
+-- OBS_GetMeasure over arbitrary area for a median measure we can estimate
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  ST_Buffer(cdb_observatory._testpoint(), 0.01),
+  'us.census.acs.B19013001') - 39266) / 39266 < 0.001 As OBS_GetMeasure_median_capita_income_average;
+
 -- Point-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
   cdb_observatory._TestPoint(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As OBS_GetCategory_point;
 
 -- Poly-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
-  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;
+  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Hispanic and Young' As obs_getcategory_polygon;
 
 -- NULL Input OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
@@ -276,7 +184,7 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
 
 -- Area-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
-  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;
+  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Hispanic and Young' As OBS_GetUSCensusCategory_polygon;
 
 -- Null-input OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
@@ -323,3 +231,728 @@ SELECT cdb_observatory.OBS_GetMeasureById(
   'us.census.tiger.block_group',
   '2010 - 2014'
 ) IS NULL As OBS_GetMeasureById_null_id;
+
+-- OBS_GetMeta null/null
+SELECT cdb_observatory.OBS_GetMeta(NULL, NULL) IS NULL
+AS OBS_GetMeta_null_null_is_null;
+
+-- OBS_GetMeta null/empty array
+SELECT cdb_observatory.OBS_GetMeta(NULL, '[]') IS NULL
+AS OBS_GetMeta_null_empty_is_null;
+
+-- OBS_GetMeta nullisland/null
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0), NULL) IS NULL
+AS OBS_GetMeta_nullisland_null_is_null;
+
+-- OBS_GetMeta nullisland/empty array
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0), '[]') IS NULL
+AS OBS_GetMeta_nullisland_empty_is_null;
+
+-- OBS_GetMeta nullisland/us_measure data
+SELECT cdb_observatory.OBS_GetMeta(ST_Point(0, 0),
+  '[{"numer_id": "us.census.acs.B01003001"}]') IS NULL
+AS OBS_GetMeta_nullisland_us_measure_is_null;
+
+-- OBS_GetMeta for point completes one partial measure with "best" metadata
+-- with no denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01003001' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'total_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Total Population' numer_name,
+(meta->0->>'denom_id') IS NULL denom_id,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') = 'area' normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes one partial measure with "best" metadata
+-- with a denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') = 'denominated' normalization
+FROM meta;
+
+-- OBS_GetMeta for polygon completes one partial measure with "best" metadata
+-- with no denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01003001' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'total_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Total Population' numer_name,
+(meta->0->>'denom_id') IS NULL denom_id,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') = 'area' normalization
+FROM meta;
+
+-- OBS_GetMeta for polygon completes one partial measure with "best" metadata
+-- with a denominator
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01001002"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' geom_id,
+(meta->0->>'normalization') = 'denominated' normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with "best"
+-- metadata, includes geom alternatives if asked
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "max_score_rank": 2}]', null, 2) meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 OR (meta->0->>'score_rank')::integer = 2 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.block_group' OR (meta->0->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->0->>'normalization') = 'denominated' normalization,
+(meta->1->>'id')::integer = 1 id,
+(meta->1->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->1->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->1->>'score_rank')::integer = 1 OR (meta->1->>'score_rank')::integer = 2 score_rank,
+(meta->1->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->1->>'numer_colname') = 'male_pop' numer_colname,
+(meta->1->>'numer_type') = 'Numeric' numer_type,
+(meta->1->>'numer_name') = 'Male Population' numer_name,
+(meta->1->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->1->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->1->>'denom_colname') = 'total_pop' denom_colname,
+(meta->1->>'denom_type') = 'Numeric' denom_type,
+(meta->1->>'denom_name') = 'Total Population' denom_name,
+(meta->1->>'geom_id') = 'us.census.tiger.block_group' OR (meta->1->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->1->>'normalization') = 'denominated' normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with "best" metadata
+-- with pre-computed geom
+WITH meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "geom_id": "us.census.tiger.census_tract"}]') meta)
+SELECT
+(meta->0->>'id')::integer = 1 id,
+(meta->0->>'numer_id') = 'us.census.acs.B01001002' numer_id,
+(meta->0->>'timespan_rank')::integer = 1 timespan_rank,
+(meta->0->>'score_rank')::integer = 1 score_rank,
+(meta->0->>'numer_aggregate') = 'sum' numer_aggregate,
+(meta->0->>'numer_colname') = 'male_pop' numer_colname,
+(meta->0->>'numer_type') = 'Numeric' numer_type,
+(meta->0->>'numer_name') = 'Male Population' numer_name,
+(meta->0->>'denom_id') = 'us.census.acs.B01003001' denom_id,
+(meta->0->>'denom_aggregate') = 'sum' denom_aggregate,
+(meta->0->>'denom_colname') = 'total_pop' denom_colname,
+(meta->0->>'denom_type') = 'Numeric' denom_type,
+(meta->0->>'denom_name') = 'Total Population' denom_name,
+(meta->0->>'geom_id') = 'us.census.tiger.census_tract' geom_id,
+(meta->0->>'normalization') = 'denominated' normalization
+FROM meta;
+
+-- OBS_GetMeta for point completes several partial measures with conflicting
+-- metadata
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "denom_id": "us.census.acs.B01001002", "geom_id": "us.census.tiger.census_tract"}]') IS NULL
+AS obs_getmeta_conflicting_metadata;
+
+-- OBS_GetMeta provides suggested name for simple meta request
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom"}]'
+)->0->>'suggested_name' = 'total_pop_2010_2014' obs_getmeta_suggested_name;
+
+-- OBS_GetMeta provides suggested name for simple meta request with area norm
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]'
+)->0->>'suggested_name' = 'total_pop_per_sq_km_2010_2014' obs_getmeta_suggested_name_implicit_area;
+
+-- OBS_GetMeta provides suggested name for simple meta request with area norm
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]'
+)->0->>'suggested_name' = 'total_pop_per_sq_km_2010_2014' obs_getmeta_suggested_name_area;
+
+-- OBS_GetMeta provides suggested name for simple meta request with denom
+SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denom"}]'
+)->0->>'suggested_name' = 'male_pop_2010_2014_by_total_pop' obs_getmeta_suggested_name_denom;
+
+-- OBS_GetData/OBS_GetMeta by id with empty list/null
+WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::TEXT[], null))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_geomval_empty_null FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with empty list/null
+WITH data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::GEOMVAL[], null))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_text_empty_null FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with empty list
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(ARRAY[]::GEOMVAL[],
+  (SELECT meta FROM meta)))
+SELECT ARRAY_AGG(data) IS NULL AS obs_getdata_geomval_empty_one_measure FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure NULL
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 10923) / 10923 < 0.001 data_point_measure_null,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure NULL
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_null,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure area
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 10923) / 10923 < 0.001 data_point_measure_area,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure area
+-- normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_area,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure predenom
+-- called "prednormalized"
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "prenormalized"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 1900) / 1900 < 0.001 data_point_measure_prenormalized,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one standard measure predenom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 1900) / 1900 < 0.001 data_point_measure_predenominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure predenom
+-- called "prenormalized"
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "prenormalized"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_prenormalized,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one standard measure predenom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_predenominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with impossible denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' IS NULL data_point_measure_impossible_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one impossible denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' IS NULL data_polygon_measure_impossible_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with denom
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 0.6215) / 0.6215 < 0.001 data_point_measure_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one denom measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01001002", "normalization": "denominated"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_denominated,
+       data->1 IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures NULL normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001"}, {"numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_one_null,
+       abs((data->1->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_two_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two measures and one return null
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B19013001_quantile"}, {"numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       (data->0->>'value') is NULL data_polygon_measure_one_null,
+       abs((data->1->>'value')::Numeric - 0.4902) / 0.4902 < 0.001 data_polygon_measure_two_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures predenom normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom"}, {"numer_id": "us.census.acs.B01001002", "normalization": "predenom"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 12327) / 12327 < 0.001 data_polygon_measure_one_predenom,
+       abs((data->1->>'value')::Numeric - 6043) / 6043 < 0.001 data_polygon_measure_two_predenom
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures area normalization
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "normalization": "area"}, {"numer_id": "us.census.acs.B01001002", "normalization": "area"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_one_area,
+       abs((data->1->>'value')::Numeric - 7739) / 7739 < 0.001 data_polygon_measure_two_area
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with two standard measures different geoms
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract"}, {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       abs((data->0->>'value')::Numeric - 16960) / 16960 < 0.001 data_polygon_measure_tract,
+       abs((data->1->>'value')::Numeric - 15787) / 15787 < 0.001 data_polygon_measure_bg
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by point geom with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestPoint(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestPoint(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Wealthy transplants displacing long-term local residents' data_point_categorical,
+       data->1->>'value' IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by polygon geom with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
+       data->1->>'value' IS NULL nullcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with one categorical and one measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"numer_id": "us.census.spielman_singleton_segments.X55"}, {"numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta)))
+SELECT id = 1 id,
+       data->0->>'value' = 'Hispanic Black mix multilingual, high poverty, renters, uses public transport' data_poly_categorical,
+       abs((data->1->>'value')::Numeric - 15790) / 15790 < 0.0001 valcol
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
+       abs(sum((data->1->>'value')::numeric) - 12329) / 12329 < 0.001 correct_pop
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by geom with polygons inside a polygon + one measure + one text
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "geom_id": "us.census.tiger.block_group"}, {"numer_id": "us.census.tiger.block_group_geoname", "geom_id": "us.census.tiger.block_group"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY[(cdb_observatory._TestArea(), 1)::geomval],
+  (SELECT meta FROM meta), false))
+SELECT every(id = 1) is TRUE id,
+       count(distinct (data->0->>'value')::geometry) = 16 correct_num_geoms,
+       abs(sum((data->1->>'value')::numeric) - 12329) / 12329 < 0.001 correct_pop,
+       array_agg(distinct data->2->>'value') = '{"Block Group 1","Block Group 2","Block Group 3","Block Group 4","Block Group 5"}' correct_bg_names
+FROM data;
+
+-- OBS_GetData by id with one standard measure
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 5578) / 5578 < 0.001 obs_getdata_by_id_one_measure_null
+FROM data;
+
+-- OBS_GetData by id with one standard measure, predenominated
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"normalization": "predenominated", "geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 3241) / 3241 < 0.001 obs_getdata_by_id_one_measure_predenom
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with two standard measures
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01003001"}, {"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.acs.B01001002"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       (abs((data->0->>'value')::numeric) - 5578) / 5578 < 0.001 obs_getdata_by_id_one_measure_null,
+       (abs((data->1->>'value')::numeric) - 0.6053) / 0.6053 < 0.001 obs_getdata_by_id_two_measure_null
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with one categorical
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract", "numer_id": "us.census.spielman_singleton_segments.X55"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       data->0->>'value' = 'Wealthy transplants displacing long-term local residents' obs_getdata_by_id_categorical
+FROM data;
+
+-- OBS_GetData/OBS_GetMeta by id with one geometry
+WITH
+meta AS (SELECT cdb_observatory.OBS_GetMeta(cdb_observatory._TestArea(),
+  '[{"geom_id": "us.census.tiger.census_tract"}]') meta),
+data AS (SELECT * FROM cdb_observatory.OBS_GetData(
+    ARRAY['36047048500'],
+    (SELECT meta FROM meta)))
+SELECT id = '36047048500' AS id,
+       ST_GeometryType((data->0->>'value')::geometry) = 'ST_MultiPolygon' obs_getdata_by_id_geometry
+FROM data;
+
+-- OBS_GetData with an API + geomvals, no args
+SELECT (SELECT array_agg(json_array_elements::text) @> array['"us.census.tiger.census_tract"']
+        FROM json_array_elements(data->0->'value'))
+AS OBS_GetData_API_geomvals_no_args
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+  '[{"numer_type": "text", "numer_colname": "boundary_id", "api_method": "obs_getavailableboundaries"}]');
+
+-- OBS_GetData with an API + geomvals, args, numeric
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'number'
+AS OBS_GetData_API_geomvals_args_numer_return
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+    '[{"numer_type": "numeric", "numer_colname": "obs_getmeasure", "api_method": "obs_getmeasure", "api_args": ["us.census.acs.B01003001"]}]');
+
+-- OBS_GetData with an API + geomvals, args, text
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'string'
+AS OBS_GetData_API_geomvals_args_string_return
+FROM cdb_observatory.obs_getdata(array[(cdb_observatory._testarea(), 1)::geomval],
+    '[{"numer_type": "text", "numer_colname": "obs_getcategory", "api_method": "obs_getcategory", "api_args": ["us.census.spielman_singleton_segments.X55"]}]');
+
+-- OBS_GetData with an API + geomrefs, args, numeric
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'number'
+AS OBS_GetData_API_geomrefs_args_numer_return
+FROM cdb_observatory.obs_getdata(array['36047076200'],
+      '[{"numer_type": "numeric", "numer_colname": "obs_getmeasurebyid", "api_method": "obs_getmeasurebyid", "api_args": ["us.census.acs.B01003001", "us.census.tiger.census_tract"]}]');
+
+-- OBS_GetData with an API + geomrefs, args, text
+SELECT json_typeof(data->0->'value') = 'array' ary_type,
+       json_typeof(data->0->'value'->0) = 'string'
+AS OBS_GetData_API_geomrefs_args_string_return
+FROM cdb_observatory.obs_getdata(array['36047'],
+      '[{"numer_type": "text", "numer_colname": "obs_getboundarybyid", "api_method": "obs_getboundarybyid", "api_args": ["us.census.tiger.county"]}]');
+
+-- Ensure consistent results below.
+select setseed(0);
+
+-- Check that random assortment of block groups in Brooklyn return accurate data
+WITH _geoms AS (
+  SELECT
+    (data->0->>'value')::geometry the_geom,
+    data->0->>'geomref' geom_ref,
+    (data->1->>'value')::numeric total_pop
+  FROM cdb_observatory.OBS_GetData(
+    array[(st_buffer(cdb_observatory._testpoint(), 0.2), 1)::geomval],
+    (SELECT cdb_observatory.OBS_GetMeta(ST_MakeEnvelope(-179, 89, 179, -89, 4326),
+      '[{"geom_id": "us.census.tiger.block_group"},
+        {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group", "normalization": "predenom"}]')),
+    FALSE
+  )
+  WHERE data->0->>'geomref' LIKE '36047%'
+  ORDER BY RANDOM()
+), geoms AS (
+  SELECT *, row_number() OVER () cartodb_id FROM _geoms
+), samples AS (
+  SELECT COUNT(*) cnt, unnest(ARRAY[1, 2, 3, 5, 10, 25, 50, 100, COUNT(*)]) sample FROM geoms
+), filtered AS (
+  SELECT * FROM geoms, samples WHERE cartodb_id % (cnt / sample) = 0
+), summary AS (
+  SELECT sample, ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+    COUNT(*)::INT cnt,
+    ARRAY_AGG((the_geom, cartodb_id)::geomval) geomvals,
+    SUM(ST_Area(the_geom))::Numeric sumarea
+  FROM filtered
+  GROUP BY sample
+), meta AS (
+  SELECT sample, cdb_observatory.OBS_GetMeta(extent,
+    ('[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "target_area": ' || sumarea || '}]')::JSON,
+    1, 1, cnt) meta
+  FROM summary
+  GROUP BY sample, extent, cnt, sumarea
+), results AS (
+  SELECT summary.sample, id, meta->0->>'geom_id' geom_id, (data->0->>'value')::Numeric as val
+  FROM summary, meta, LATERAL cdb_observatory.OBS_GetData(geomvals, meta) data
+  WHERE summary.sample = meta.sample
+) SELECT sample bg_sample
+ , MAX(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 bg_max_error
+ , AVG(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 bg_avg_error
+ , MIN(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 bg_min_error
+FROM geoms, results
+WHERE cartodb_id = id
+GROUP BY sample
+ORDER BY sample
+;
+
+-- Check that random assortment of tracts in Brooklyn return accurate data
+WITH _geoms AS (
+  SELECT
+    (data->0->>'value')::geometry the_geom,
+    data->0->>'geomref' geom_ref,
+    (data->1->>'value')::numeric total_pop
+  FROM cdb_observatory.OBS_GetData(
+    array[(st_buffer(cdb_observatory._testpoint(), 0.2), 1)::geomval],
+    (SELECT cdb_observatory.OBS_GetMeta(ST_MakeEnvelope(-179, 89, 179, -89, 4326),
+      '[{"geom_id": "us.census.tiger.census_tract"},
+        {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.census_tract", "normalization": "predenom"}]')),
+    FALSE
+  )
+  WHERE data->0->>'geomref' LIKE '36047%'
+  and (data->1->>'value')::numeric > 1000
+  ORDER BY geom_ref
+), geoms AS (
+  SELECT *, row_number() OVER () cartodb_id FROM _geoms
+), samples AS (
+  SELECT COUNT(*) cnt, unnest(ARRAY[1, 2, 3, 5, 10, 25, 50, 100, COUNT(*)]) sample FROM geoms
+), filtered AS (
+  SELECT * FROM geoms, samples WHERE cartodb_id % (cnt / sample) = 0
+), summary AS (
+  SELECT sample, ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+    COUNT(*)::INT cnt,
+    ARRAY_AGG((the_geom, cartodb_id)::geomval) geomvals,
+    SUM(ST_Area(the_geom))::Numeric sumarea
+  FROM filtered
+  GROUP BY sample
+), meta AS (
+  SELECT sample, cdb_observatory.OBS_GetMeta(extent,
+    ('[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "target_area": ' || sumarea || '}]')::JSON,
+    1, 1, cnt) meta
+  FROM summary
+  GROUP BY sample, extent, cnt, sumarea
+), results AS (
+  SELECT summary.sample, id, meta->0->>'geom_id' geom_id, (data->0->>'value')::Numeric as val
+  FROM summary, meta, LATERAL cdb_observatory.OBS_GetData(geomvals, meta) data
+  WHERE summary.sample = meta.sample
+) SELECT sample tract_sample
+ , MAX(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 tract_max_error
+ , AVG(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 tract_avg_error
+ , MIN(100 * abs((geoms.total_pop - val) / Coalesce(NullIf(total_pop, 0), NULL)))::Numeric(10, 2) < 10 tract_min_error
+FROM geoms, results
+WHERE cartodb_id = id
+GROUP BY sample
+ORDER BY sample
+;
+
+-- Check that random assortment of block group points in Brooklyn return accurate data
+WITH _geoms AS (
+  SELECT
+    ST_PointOnSurface((data->0->>'value')::geometry) the_geom,
+    data->0->>'geomref' geom_ref,
+    (data->1->>'value')::numeric total_pop
+  FROM cdb_observatory.OBS_GetData(
+    array[(st_buffer(cdb_observatory._testpoint(), 0.2), 1)::geomval],
+    (SELECT cdb_observatory.OBS_GetMeta(ST_MakeEnvelope(-179, 89, 179, -89, 4326),
+      '[{"geom_id": "us.census.tiger.block_group"},
+        {"numer_id": "us.census.acs.B01003001", "geom_id": "us.census.tiger.block_group", "normalization": "predenom"}]')),
+    FALSE
+  )
+  WHERE data->0->>'geomref' LIKE '36047%'
+), geoms AS (
+  SELECT *, row_number() OVER () cartodb_id FROM _geoms
+), samples AS (
+  SELECT COUNT(*) cnt, unnest(ARRAY[1, 2, 3, 5, 10, 25, 50, 100, COUNT(*)]) sample FROM geoms
+), filtered AS (
+  SELECT * FROM geoms, samples WHERE cartodb_id % (cnt / sample) = 0
+), summary AS (
+  SELECT sample, ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+    COUNT(*)::INT cnt,
+    ARRAY_AGG((the_geom, cartodb_id)::geomval) geomvals,
+    SUM(ST_Area(the_geom))::Numeric sumarea
+  FROM filtered
+  GROUP BY sample
+), meta AS (
+  SELECT sample, cdb_observatory.OBS_GetMeta(extent,
+    ('[{"numer_id": "us.census.acs.B01003001", "normalization": "predenom", "target_area": ' || sumarea || '}]')::JSON,
+    1, 1, cnt) meta
+  FROM summary
+  GROUP BY sample, extent, cnt, sumarea
+), results AS (
+  SELECT summary.sample, id, meta->0->>'geom_id' geom_id, (data->0->>'value')::Numeric as val
+  FROM summary, meta, LATERAL cdb_observatory.OBS_GetData(geomvals, meta) data
+  WHERE summary.sample = meta.sample
+) SELECT
+ BOOL_AND(abs((geoms.total_pop - val) /
+      Coalesce(NullIf(total_pop, 0), 1)) = 0) is True no_bg_point_error
+FROM geoms, results
+WHERE cartodb_id = id
+;
+
+-- OBS_MetadataValidation
+
+SELECT * FROM cdb_observatory.OBS_MetadataValidation(NULL, 'ST_Polygon', '[{"numer_id": "us.census.acs.B01003001","denom_id": null,"normalization": "prenormalized","geom_id": null,"numer_timespan": "2010 - 2014"}]'::json, 500);
+SELECT * FROM cdb_observatory.OBS_MetadataValidation(NULL, 'ST_Polygon', '[{"numer_id": "us.census.acs.B25058001","denom_id": null,"normalization": "denominated","geom_id": null,"numer_timespan": "2010 - 2014"}]'::json, 500);
+SELECT * FROM cdb_observatory.OBS_MetadataValidation(NULL, 'ST_Polygon', '[{"numer_id": "us.census.acs.B15003001","denom_id": null,"normalization": "denominated","geom_id": null,"numer_timespan": "2010 - 2014"}]'::json, 500);

src/pg/test/sql/42_observatory_exploration_test.sql

@@ -10,11 +10,11 @@ SET client_min_messages TO WARNING;
 
 -- _OBS_SearchTables tests
 SELECT
-  t.table_name = 'obs_1babf5a26a1ecda5fb74963e88408f71d0364b81' As _OBS_SearchTables_tables_match,
+  t.table_name = 'obs_0310c639744a2014bb1af82709228f05b59e7d3d' As _OBS_SearchTables_tables_match,
-  t.timespan = '2014' As _OBS_SearchTables_timespan_matches
+  t.timespan = '2015' As _OBS_SearchTables_timespan_matches
 FROM cdb_observatory._OBS_SearchTables(
   'us.census.tiger.county',
-  '2014'
+  '2015'
 ) As t(table_name, timespan);
 
 -- _OBS_SearchTables tests
@@ -119,6 +119,142 @@ FROM cdb_observatory.OBS_GetAvailableNumerators(
 ) WHERE valid_timespan = True)
 AS _obs_getavailablenumerators_no_total_pop_1996;
 
+--
+-- _OBS_GetNumerators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators())
+AS _obs_getnumerators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
+)) AS _obs_getnumerators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
+)) AS _obs_getnumerators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
+)) AS _obs_getnumerators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  subsection_tags => ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getnumerators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  subsection_tags => ARRAY['subsection/tags.income']
+))
+AS _obs_getnumerators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01001002' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  denom_id => 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getnumerators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B19013001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  denom_id => 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getnumerators_no_income_denom_by_total_pop;
+
+SELECT 'us.zillow.AllHomes_Zhvi' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  geom_id => 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getnumerators_zillow_at_zcta5;
+
+SELECT 'us.zillow.AllHomes_Zhvi' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  geom_id => 'us.census.tiger.block_group'
+) WHERE valid_geom = True)
+AS _obs_getnumerators_no_zillow_at_block_group;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  timespan => '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getnumerators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  timespan => '1996'
+) WHERE valid_timespan = True)
+AS _obs_getnumerators_no_total_pop_1996;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  name => 'tot'
+))
+AS _obs_getnumerators_total_pop_by_name;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.united_states}'
+))
+AS _obs_getnumerators_total_pop_by_section;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.ca}'
+))
+AS _obs_getnumerators_total_pop_not_in_canada;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.united_states}',
+  subsection_tags => '{subsection/tags.age_gender}'
+))
+AS _obs_getnumerators_total_pop_by_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  section_tags => '{section/tags.united_states}',
+  subsection_tags => '{subsection/tags.employment}'
+))
+AS _obs_getnumerators_total_pop_not_in_employment_subsection;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ids => '{us.census.acs.B01003001}'
+))
+AS _obs_getnumerators_total_pop_by_id;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory._OBS_GetNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ids => '{us.census.acs.B01003002}'
+))
+AS _obs_getnumerators_total_pop_not_with_other_id;
+
 --
 -- OBS_GetAvailableDenominators tests
 --
@@ -278,9 +414,9 @@ AS _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg;
 SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
 FROM cdb_observatory.OBS_GetAvailableGeometries(
   ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
-  NULL, NULL, NULL, '2014'
+  NULL, NULL, NULL, '2015'
 ) WHERE valid_timespan = True)
-AS _obs_getavailablegeometries_bg_2014;
+AS _obs_getavailablegeometries_bg_2015;
 
 SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
 FROM cdb_observatory.OBS_GetAvailableGeometries(
@@ -289,6 +425,11 @@ FROM cdb_observatory.OBS_GetAvailableGeometries(
 ) WHERE valid_timespan = True)
 AS _obs_getavailablegeometries_bg_not_1996;
 
+SELECT 'subsection/tags.boundary' IN (SELECT (Jsonb_Each(geom_tags)).key
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)
+)) AS _obs_getavailablegeometries_has_boundary_tag;
+
 --
 -- OBS_GetAvailableTimespans tests
 --
@@ -350,126 +491,173 @@ AS _obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014;
 -- _OBS_GetGeometryScores tests
 --
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
        ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-             'us.census.tiger.zcta5', 'us.census.tiger.county']
+             'us.census.tiger.county', 'us.census.tiger.zcta5']
        AS _obs_geometryscores_500m_buffer
        FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
-       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+       = ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-             'us.census.tiger.zcta5', 'us.census.tiger.county']
+               'us.census.tiger.zcta5', 'us.census.tiger.county']
        AS _obs_geometryscores_5km_buffer
        FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.county', 'us.census.tiger.zcta5'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.zcta5',
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
-             'us.census.tiger.county', 'us.census.tiger.block_group']
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
        AS _obs_geometryscores_50km_buffer
        FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
-       ARRAY[ 'us.census.tiger.county', 'us.census.tiger.zcta5',
+       ARRAY[ 'us.census.tiger.zcta5', 'us.census.tiger.census_tract',
-             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+              'us.census.tiger.county', 'us.census.tiger.block_group' ]
       AS _obs_geometryscores_500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
-       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
+       = ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
              'us.census.tiger.census_tract', 'us.census.tiger.block_group']
       AS _obs_geometryscores_2500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
-  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+  ARRAY['us.census.tiger.county', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.block_group'])
+      WHERE table_id LIKE '%2015%';
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_500m_buffer
-      = '{ "us.census.tiger.block_group" : 9, "us.census.tiger.census_tract" : 3, "us.census.tiger.zcta5" : 0, "us.census.tiger.county" : 0 }'
-      AS _obs_geometryscores_numgeoms_500m_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_5km_buffer
-      '{ "us.census.tiger.block_group" : 899, "us.census.tiger.census_tract" : 328, "us.census.tiger.zcta5" : 45, "us.census.tiger.county" : 1 }'
-      AS _obs_geometryscores_numgeoms_5km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_50km_buffer
-      '{ "us.census.tiger.block_group" : 12112, "us.census.tiger.census_tract" : 3792, "us.census.tiger.zcta5" : 550, "us.census.tiger.county" : 13 }'
-      AS _obs_geometryscores_numgeoms_50km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_500km_buffer
-      '{ "us.census.tiger.block_group" : 48415, "us.census.tiger.census_tract" : 15776, "us.census.tiger.zcta5" : 6534, "us.census.tiger.county" : 295 }'
-      AS _obs_geometryscores_numgeoms_500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+SELECT column_id, numgeoms::int AS _obs_geometryscores_numgeoms_2500km_buffer
-      '{ "us.census.tiger.block_group" : 165489, "us.census.tiger.census_tract" : 55152, "us.census.tiger.zcta5" : 26500, "us.census.tiger.county" : 2551 }'
-      AS _obs_geometryscores_numgeoms_2500km_buffer
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'])
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC;
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
        ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
              'us.census.tiger.census_tract', 'us.census.tiger.block_group']
       AS _obs_geometryscores_500km_buffer_50_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50)
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC)
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
-      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.county',
+      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.census_tract',
-              'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+              'us.census.tiger.block_group', 'us.census.tiger.county']
       AS _obs_geometryscores_500km_buffer_500_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500)
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC) =
-       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.zcta5',
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.block_group',
-             'us.census.tiger.county', 'us.census.tiger.block_group']
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
       AS _obs_geometryscores_500km_buffer_2500_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500)
+      WHERE table_id LIKE '%2015%';
 
-SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+SELECT ARRAY_AGG(column_id ORDER BY score DESC)
-       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+       = ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-             'us.census.tiger.zcta5', 'us.census.tiger.county']
+               'us.census.tiger.zcta5', 'us.census.tiger.county']
       AS _obs_geometryscores_500km_buffer_25000_geoms
       FROM cdb_observatory._OBS_GetGeometryScores(
   ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
   ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
-        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000);
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000)
+      WHERE table_id LIKE '%2015%';
+
+-- Check that one small geom approximates tract data
+WITH geoms AS (SELECT cdb_observatory._testarea() the_geom),
+summary AS (SELECT ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+                   COUNT(*)::INT cnt,
+                   SUM(ST_Area(the_geom))::Numeric sumarea
+            FROM geoms)
+SELECT column_id = 'us.census.tiger.census_tract' testarea_uses_tract
+FROM summary, LATERAL (
+  SELECT *
+  FROM cdb_observatory._OBS_GetGeometryScores(extent,
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'],
+        cnt, sumarea)) foo
+ORDER BY score DESC LIMIT 1;
+
+-- Check that randomly distributed points always use smallest geometry if we
+-- order by numgeoms desc
+WITH geoms as (SELECT UNNEST(ARRAY[
+    cdb_observatory._testpoint(),
+    st_translate(cdb_observatory._testpoint(), -0.003, 0),
+    st_translate(cdb_observatory._testpoint(), -0.006, 0)
+]) the_geom),
+summary as (SELECT
+  ST_SetSRID(ST_Extent(the_geom), 4326) extent,
+  SUM(ST_Area(the_geom))::Numeric area,
+  COUNT(*)::INTEGER cnt
+  FROM geoms
+)
+SELECT column_id = 'us.census.tiger.block_group' points_use_bg
+      FROM summary, LATERAL (
+        SELECT * FROM cdb_observatory._OBS_GetGeometryScores(
+          extent,
+          ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+                'us.census.tiger.zcta5', 'us.census.tiger.county'],
+        cnt, area)) foo
+      WHERE table_id LIKE '%2015%'
+      ORDER BY numgeoms DESC LIMIT 1;
 
 --
 -- OBS_LegacyBuilderMetadata tests
@@ -485,15 +673,25 @@ SELECT 'us.census.acs.B19013001' IN (SELECT
   FROM cdb_observatory.OBS_LegacyBuilderMetadata()
 ) AS _median_income_in_legacy_builder_metadata;
 
+SELECT 'us.census.acs.B19083001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _gini_in_legacy_builder_metadata;
+
 SELECT 'us.census.acs.B01003001' IN (SELECT
   (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
   FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
 ) AS _total_pop_in_legacy_builder_metadata_sums;
 
-SELECT 'us.census.acs.B19013001' NOT IN (SELECT
+SELECT 'us.census.acs.B19013001' IN (SELECT
   (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
   FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
-) AS _median_income_not_in_legacy_builder_metadata_sums;
+) AS _median_income_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19083001' NOT IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _gini_not_in_legacy_builder_metadata_sums;
 
 SELECT COUNT(*) = 0 _no_dupe_subsections_in_legacy_builder_metadata FROM (
   SELECT name, subsection, count(*) FROM

src/python/requirements.txt

@@ -1,3 +1,4 @@
 nose
+nose-timer
 nose_parameterized
 psycopg2

src/python/test/autotest.py

@@ -1,89 +1,135 @@
 from nose.tools import assert_equal, assert_is_not_none
-from nose.plugins.skip import SkipTest
 from nose_parameterized import parameterized
 
+from itertools import zip_longest
 from util import query
+from collections import OrderedDict
+import json
+
+
+def grouper(iterable, n, fillvalue=None):
+    "Collect data into fixed-length chunks or blocks"
+    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
+    args = [iter(iterable)] * n
+    return zip_longest(fillvalue=fillvalue, *args)
+
 
 USE_SCHEMA = True
 
-MEASURE_COLUMNS = query('''
-SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
-FROM observatory.obs_meta
-WHERE numer_type ILIKE 'numeric'
-AND numer_weight > 0
-''').fetchall()
-
-CATEGORY_COLUMNS = query('''
-SELECT distinct numer_id
-FROM observatory.obs_meta
-WHERE numer_type ILIKE 'text'
-AND numer_weight > 0
-''').fetchall()
-
-BOUNDARY_COLUMNS = query('''
-SELECT id FROM observatory.obs_column
-WHERE type ILIKE 'geometry'
-AND weight > 0
-''').fetchall()
-
-US_CENSUS_MEASURE_COLUMNS = query('''
-SELECT distinct numer_name
-FROM observatory.obs_meta
-WHERE numer_type ILIKE 'numeric'
-AND 'us.census.acs.acs' = ANY (subsection_tags)
-AND numer_weight > 0
-''').fetchall()
-
 SKIP_COLUMNS = set([
-    u'mx.inegi_columns.INDI18',
+    'mx.inegi_columns.INDI18',
-    u'mx.inegi_columns.ECO40',
+    'mx.inegi_columns.ECO40',
-    u'mx.inegi_columns.POB34',
+    'mx.inegi_columns.POB34',
-    u'mx.inegi_columns.POB63',
+    'mx.inegi_columns.POB63',
-    u'mx.inegi_columns.INDI7',
+    'mx.inegi_columns.INDI7',
-    u'mx.inegi_columns.EDU28',
+    'mx.inegi_columns.EDU28',
-    u'mx.inegi_columns.SCONY10',
+    'mx.inegi_columns.SCONY10',
-    u'mx.inegi_columns.EDU31',
+    'mx.inegi_columns.EDU31',
-    u'mx.inegi_columns.POB7',
+    'mx.inegi_columns.POB7',
-    u'mx.inegi_columns.VIV30',
+    'mx.inegi_columns.VIV30',
-    u'mx.inegi_columns.INDI12',
+    'mx.inegi_columns.INDI12',
-    u'mx.inegi_columns.EDU13',
+    'mx.inegi_columns.EDU13',
-    u'mx.inegi_columns.ECO43',
+    'mx.inegi_columns.ECO43',
-    u'mx.inegi_columns.VIV9',
+    'mx.inegi_columns.VIV9',
-    u'mx.inegi_columns.HOGAR25',
+    'mx.inegi_columns.HOGAR25',
-    u'mx.inegi_columns.POB32',
+    'mx.inegi_columns.POB32',
-    u'mx.inegi_columns.ECO7',
+    'mx.inegi_columns.ECO7',
-    u'mx.inegi_columns.INDI19',
+    'mx.inegi_columns.INDI19',
-    u'mx.inegi_columns.INDI16',
+    'mx.inegi_columns.INDI16',
-    u'mx.inegi_columns.POB65',
+    'mx.inegi_columns.POB65',
-    u'mx.inegi_columns.INDI3',
+    'mx.inegi_columns.INDI3',
-    u'mx.inegi_columns.INDI9',
+    'mx.inegi_columns.INDI9',
-    u'mx.inegi_columns.POB36',
+    'mx.inegi_columns.POB36',
-    u'mx.inegi_columns.POB33',
+    'mx.inegi_columns.POB33',
-    u'mx.inegi_columns.POB58',
+    'mx.inegi_columns.POB58',
-    u'mx.inegi_columns.DISC4',
+    'mx.inegi_columns.DISC4',
+    'mx.inegi_columns.VIV41',
+    'mx.inegi_columns.VIV40',
+    'mx.inegi_columns.VIV17',
+    'mx.inegi_columns.VIV25',
+    'mx.inegi_columns.EDU10',
+    'whosonfirst.wof_disputed_name',
+    'us.census.tiger.fullname',
+    'whosonfirst.wof_marinearea_name',
+    'us.census.tiger.mtfcc',
+    'whosonfirst.wof_county_name',
+    'whosonfirst.wof_region_name',
+    'fr.insee.P12_RP_CHOS',
+    'fr.insee.P12_RP_HABFOR',
+    'fr.insee.P12_RP_EAUCH',
+    'fr.insee.P12_RP_BDWC',
+    'fr.insee.P12_RP_MIDUR',
+    'fr.insee.P12_RP_CLIM',
+    'fr.insee.P12_RP_MIBOIS',
+    'fr.insee.P12_RP_CASE',
+    'fr.insee.P12_RP_TTEGOU',
+    'fr.insee.P12_RP_ELEC',
+    'fr.insee.P12_ACTOCC15P_ILT45D',
+    'fr.insee.P12_RP_CHOS',
+    'fr.insee.P12_RP_HABFOR',
+    'fr.insee.P12_RP_EAUCH',
+    'fr.insee.P12_RP_BDWC',
+    'fr.insee.P12_RP_MIDUR',
+    'fr.insee.P12_RP_CLIM',
+    'fr.insee.P12_RP_MIBOIS',
+    'fr.insee.P12_RP_CASE',
+    'fr.insee.P12_RP_TTEGOU',
+    'fr.insee.P12_RP_ELEC',
+    'fr.insee.P12_ACTOCC15P_ILT45D',
+    'uk.ons.LC3202WA0007',
+    'uk.ons.LC3202WA0010',
+    'uk.ons.LC3202WA0004',
+    'uk.ons.LC3204WA0004',
+    'uk.ons.LC3204WA0007',
+    'uk.ons.LC3204WA0010',
+    'br.geo.subdistritos_name',
+    # Problems with universe (denominator is zero in test area)
+    'ca.statcan.census2016.cols_census.c0058_t',
+    'ca.statcan.census2016.cols_census.c1878_f',
+    'ca.statcan.census2016.cols_census.c1878_m',
+    'ca.statcan.census2016.cols_census.c1674_t',
+    'ca.statcan.census2016.cols_census.c1675_t',
+    'ca.statcan.census2016.cols_census.c1676_t',
+    'ca.statcan.census2016.cols_census.c1677_t',
+    'ca.statcan.census2016.cols_census.c0801_t',
+    'ca.statcan.census2016.cols_census.c0802_t',
+    'ca.statcan.census2016.cols_census.c0803_t',
+    'ca.statcan.census2016.cols_census.c0805_t',
+    'ca.statcan.census2016.cols_census.c0806_t',
+    'ca.statcan.census2016.cols_census.c0807_t',
+    'ca.statcan.census2016.cols_census.c0809_t',
+    'ca.statcan.census2016.cols_census.c0810_t',
+    'ca.statcan.census2016.cols_census.c0811_t',
+    'ca.statcan.census2016.cols_census.c0813_t',
+    'ca.statcan.census2016.cols_census.c0814_t',
+    'ca.statcan.census2016.cols_census.c0815_t',
+    'ca.statcan.census2016.cols_census.c0817_t',
+    'ca.statcan.census2016.cols_census.c0818_t',
+    'ca.statcan.census2016.cols_census.c0820_t',
+    'ca.statcan.census2016.cols_census.c0821_t',
+    'ca.statcan.census2016.cols_census.c0821_t',
+    'ca.statcan.census2016.cols_census.c0823_t',
+    'ca.statcan.census2016.cols_census.c0824_t',
+    'ca.statcan.census2016.cols_census.c0826_t',
+    'ca.statcan.census2016.cols_census.c0827_t',
+    'ca.statcan.census2016.cols_census.c0829_t',
+    'ca.statcan.census2016.cols_census.c0830_t',
+    'ca.statcan.census2016.cols_census.c0832_t',
+    'ca.statcan.census2016.cols_census.c0833_t'
 ])
 
-#def default_geometry_id(column_id):
+MEASURE_COLUMNS = query('''
-#    '''
+SELECT cdb_observatory.FIRST(distinct numer_id) numer_ids,
-#    Returns default test point for the column_id.
+       numer_aggregate,
-#    '''
+       denom_reltype
-#    if column_id == 'whosonfirst.wof_disputed_geom':
+FROM observatory.obs_meta
-#        return 'ST_SetSRID(ST_MakePoint(76.57, 33.78), 4326)'
+WHERE numer_weight > 0
-#    elif column_id == 'whosonfirst.wof_marinearea_geom':
+  AND numer_id NOT IN ('{skip}')
-#        return 'ST_SetSRID(ST_MakePoint(-68.47, 43.33), 4326)'
+  AND numer_id NOT LIKE 'eu.%' --Skipping Eurostat
-#    elif column_id in ('us.census.tiger.school_district_elementary',
+  AND section_tags IS NOT NULL
-#                       'us.census.tiger.school_district_secondary',
+  AND subsection_tags IS NOT NULL
-#                       'us.census.tiger.school_district_elementary_clipped',
+GROUP BY numer_id, numer_aggregate, denom_reltype
-#                       'us.census.tiger.school_district_secondary_clipped'):
+'''.format(skip="', '".join(SKIP_COLUMNS))).fetchall()
-#        return 'ST_SetSRID(ST_MakePoint(-73.7067, 40.7025), 4326)'
-#    elif column_id.startswith('es.ine'):
-#        return 'ST_SetSRID(ST_MakePoint(-2.51141249535454, 42.8226119029222), 4326)'
-#    elif column_id.startswith('us.zillow'):
-#        return 'ST_SetSRID(ST_MakePoint(-81.3544048197256, 28.3305906291771), 4326)'
-#    elif column_id.startswith('ca.'):
-#        return ''
-#    else:
-#        return 'ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)'
 
 
 def default_lonlat(column_id):
@@ -94,11 +140,6 @@ def default_lonlat(column_id):
         return (76.57, 33.78)
     elif column_id == 'whosonfirst.wof_marinearea_geom':
         return (-68.47, 43.33)
-    elif column_id in ('us.census.tiger.school_district_elementary',
-                       'us.census.tiger.school_district_secondary',
-                       'us.census.tiger.school_district_elementary_clipped',
-                       'us.census.tiger.school_district_secondary_clipped'):
-        return (40.7025, -73.7067)
     elif column_id.startswith('uk'):
         if 'WA' in column_id:
             return (51.46844551219723, -3.184833526611328)
@@ -110,185 +151,121 @@ def default_lonlat(column_id):
         return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('mx.'):
         return (19.41347699386547, -99.17019367218018)
-    elif column_id.startswith('th.'):
-        return (13.725377712079784, 100.49263000488281)
-    # cols for French Guyana only
-    elif column_id in ('fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
-                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
-                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
-                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
-                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
-                       , 'fr.insee.P12_ACTOCC15P_ILT45D'
-                       , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
-                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
-                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
-                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
-                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
-                       , 'fr.insee.P12_ACTOCC15P_ILT45D'):
-        return (4.938408371206558, -52.32908248901367)
     elif column_id.startswith('fr.'):
         return (48.860875144709475, 2.3613739013671875)
     elif column_id.startswith('ca.'):
         return (43.65594991256823, -79.37965393066406)
+    elif (column_id.startswith('us.census.tiger.school_district_elementary') or
+          column_id.startswith('us.census.tiger.school_district_secondary')):
+        return (40.7025, -73.7067)
     elif column_id.startswith('us.census.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
-    elif column_id.startswith('us.dma.'):
-        return (40.7, -73.9)
     elif column_id.startswith('us.ihme.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.bls.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.qcew.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('whosonfirst.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
     elif column_id.startswith('us.epa.'):
-        return (40.7, -73.9)
+        return (28.3305906291771, -81.3544048197256)
-    elif column_id.startswith('eu.'):
+    elif column_id.startswith('br.'):
-        raise SkipTest('No tests for Eurostat!')
+        return (-23.53, -46.63)
-        #return (52.52207036136366, 13.40606689453125)
+    elif column_id.startswith('au.'):
+        return (-33.8806, 151.2131)
     else:
         raise Exception('No catalog point set for {}'.format(column_id))
 
 
-def default_point(column_id):
+def default_point(test_point):
-    lat, lng = default_lonlat(column_id)
+    lat, lng = test_point
     return 'ST_SetSRID(ST_MakePoint({lng}, {lat}), 4326)'.format(
         lat=lat, lng=lng)
 
 
-def default_area(column_id):
+def default_area(test_point):
     '''
     Returns default test area for the column_id
     '''
-    point = default_point(column_id)
+    point = default_point(test_point)
     area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 250), 4326)'.format(
         point=point)
     return area
 
-@parameterized(US_CENSUS_MEASURE_COLUMNS)
+
-def test_get_us_census_measure_points(name):
+def filter_points():
-    resp = query('''
+    return MEASURE_COLUMNS
-SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
-                 '''.format(name=name.replace("'", "''"),
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point('')))
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
 
 
-@parameterized(MEASURE_COLUMNS)
+def filter_areas():
-def test_get_measure_areas(column_id, point_only):
+    filtered = []
-    if column_id in SKIP_COLUMNS:
+    for numer_ids, numer_aggregate, denom_reltype in MEASURE_COLUMNS:
-        raise SkipTest('Column {} should be skipped'.format(column_id))
+        if numer_aggregate is None or numer_aggregate.lower() not in ('sum', 'median', 'average'):
-    if point_only:
+            continue
-        return
+        if numer_aggregate.lower() in ('median', 'average') \
-    resp = query('''
+                and (denom_reltype is None or denom_reltype.lower() != 'universe'):
-SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
+            continue
-                 '''.format(column_id=column_id,
+        filtered.append((numer_ids, numer_aggregate, denom_reltype))
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
+
-                            area=default_area(column_id)))
+    return filtered
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
 
 
-@parameterized(MEASURE_COLUMNS)
+def grouped_measure_columns(filtered_columns):
-def test_get_measure_points(column_id, point_only):
+    groupbypoint = dict()
-    if column_id in SKIP_COLUMNS:
+    for row in filtered_columns:
-        raise SkipTest('Column {} should be skipped'.format(column_id))
+        numer_ids = row[0]
-    resp = query('''
+        point = default_lonlat(numer_ids)
-SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
+        if point in groupbypoint:
-                 '''.format(column_id=column_id,
+            groupbypoint[point].append(numer_ids)
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
+        else:
-                            point=default_point(column_id)))
+            groupbypoint[point] = [numer_ids]
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
 
-#@parameterized(CATEGORY_COLUMNS)
+    for point, numer_ids in groupbypoint.items():
-#def test_get_category_areas(column_id):
+        for colgroup in grouper(numer_ids, 50):
-#    resp = query('''
+            yield point, [c for c in colgroup if c]
-#SELECT * FROM {schema}OBS_GetCategory({area}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            area=default_area(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0][0])
 
-@parameterized(CATEGORY_COLUMNS)
-def test_get_category_points(column_id):
-    if column_id in SKIP_COLUMNS:
-        raise SkipTest('Column {} should be skipped'.format(column_id))
-    resp = query('''
-SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
-                 '''.format(column_id=column_id,
-                            schema='cdb_observatory.' if USE_SCHEMA else '',
-                            point=default_point(column_id)))
-    rows = resp.fetchall()
-    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0][0])
 
-#@parameterized(BOUNDARY_COLUMNS)
+@parameterized(grouped_measure_columns(filter_points()))
-#def test_get_boundaries_by_geometry(column_id):
+def test_get_measure_points(point, numer_ids):
-#    resp = query('''
+    _test_measures(numer_ids, default_point(point))
-#SELECT * FROM {schema}OBS_GetBoundariesByGeometry({area}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            area=default_area(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0][0])
 
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_points_by_geometry(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetPointsByGeometry({area}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            area=default_area(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0][0])
 
-#@parameterized(BOUNDARY_COLUMNS)
+@parameterized(grouped_measure_columns(filter_areas()))
-#def test_get_boundary_points(column_id):
+def test_get_measure_areas(point, numer_ids):
-#    resp = query('''
+    _test_measures(numer_ids, default_area(point))
-#SELECT * FROM {schema}OBS_GetBoundary({point}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            point=default_point(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0][0])
 
-#@parameterized(BOUNDARY_COLUMNS)
-#def test_get_boundary_id(column_id):
-#    resp = query('''
-#SELECT * FROM {schema}OBS_GetBoundaryId({point}, '{column_id}')
-#                 '''.format(column_id=column_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
-#                            point=default_point(column_id)))
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0][0])
 
-#@parameterized(BOUNDARY_COLUMNS)
+def _test_measures(numer_ids, geom):
-#def test_get_boundary_by_id(column_id):
+    in_params = []
-#    resp = query('''
+    for numer_id in numer_ids:
-#SELECT * FROM {schema}OBS_GetBoundaryById({geometry_id}, '{column_id}')
+        in_params.append({
-#                 '''.format(column_id=column_id,
+            'numer_id': numer_id,
-#                            schema='cdb_observatory.' if USE_SCHEMA else '',
+            'normalization': 'predenominated'
-#                            geometry_id=default_geometry_id(column_id)))
+        })
-#    assert_equal(resp.status_code, 200)
-#    rows = resp.json()['rows']
-#    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0][0])
 
+    params = query('''
+        SELECT {schema}OBS_GetMeta({geom}, '{in_params}')
+    '''.format(schema='cdb_observatory.' if USE_SCHEMA else '',
+               geom=geom,
+               in_params=json.dumps(in_params))).fetchone()[0]
+
+    # We can get duplicate IDs from multi-denominators, so for now we
+    # compress those measures into a single
+    params = list(OrderedDict([(p['id'], p) for p in params]).values())
+    assert_equal(len(params), len(in_params),
+                 'Inconsistent out and in params for {}'.format(in_params))
+
+    q = '''
+    SELECT * FROM {schema}OBS_GetData(ARRAY[({geom}, 1)::geomval], '{params}')
+    '''.format(schema='cdb_observatory.' if USE_SCHEMA else '',
+               geom=geom,
+               params=json.dumps(params).replace("'", "''"))
+    resp = query(q).fetchone()
+    assert_is_not_none(resp, 'NULL returned for {}'.format(in_params))
+    rawvals = resp[1]
+    vals = [v['value'] for v in rawvals]
+
+    assert_equal(len(vals), len(in_params))
+    for i, val in enumerate(vals):
+        assert_is_not_none(val, 'NULL for {}'.format(in_params[i]['numer_id']))

src/python/test/perftest.py

@@ -5,58 +5,314 @@ from util import query, commit
 
 from time import time
 
+import json
+import os
+
 USE_SCHEMA = True
 
 for q in (
-        'DROP TABLE IF EXISTS obs_censustest',
+        'DROP TABLE IF EXISTS obs_perftest_simple',
-        '''CREATE TABLE obs_censustest (cartodb_id SERIAL PRIMARY KEY,
+        '''CREATE TABLE obs_perftest_simple (cartodb_id SERIAL PRIMARY KEY,
-           the_geom GEOMETRY, name TEXT, measure NUMERIC, category TEXT)''',
+           point GEOMETRY,
-        '''INSERT INTO obs_censustest (the_geom, name)
+           geom GEOMETRY,
-           SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+           offset_geom GEOMETRY,
-           st_makeenvelope(-74.05437469482422,40.66319159533881,
+           name TEXT, measure NUMERIC, category TEXT)''',
-                           -73.81885528564453,40.745696344339564, 4326),
+        '''INSERT INTO obs_perftest_simple (point, geom, offset_geom, name)
-           'us.census.tiger.block_group_clipped') As m(the_geom, geoid)'''
+           SELECT ST_PointOnSurface(the_geom) point,
-):
+                  the_geom geom,
-    query(q.format(
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-74.1, 40.5,
+                                 -73.8, 40.9, 4326),
+                 'us.census.tiger.census_tract_2015_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) ASC
+           LIMIT 1000''',
+        'DROP TABLE IF EXISTS obs_perftest_complex',
+        '''CREATE TABLE obs_perftest_complex (cartodb_id SERIAL PRIMARY KEY,
+           point GEOMETRY,
+           geom GEOMETRY,
+           offset_geom GEOMETRY,
+           name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_perftest_complex (point, geom, offset_geom, name)
+           SELECT ST_PointOnSurface(the_geom) point,
+                  the_geom geom,
+                  ST_Translate(the_geom, -0.1, 0.1) offset_geom,
+                  geom_refs AS name
+           FROM (SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+                 st_makeenvelope(-75.05437469482422,40.66319159533881,
+                                 -73.81885528564453,41.745696344339564, 4326),
+                 'us.census.tiger.county_2015_clipped')) foo
+           ORDER BY ST_NPoints(the_geom) DESC
+           LIMIT 50;'''):
+    q_formatted = q.format(
         schema='cdb_observatory.' if USE_SCHEMA else '',
-    ))
+    )
+    start = time()
+    resp = query(q_formatted)
+    end = time()
+    print('{} for {}'.format(int(end - start), q_formatted))
+    if q.lower().startswith('insert'):
+        if resp.rowcount == 0:
+            raise Exception('''Performance fixture creation "{}" inserted 0 rows,
+                            this will break tests.  Check the query to determine
+                            what is going wrong.'''.format(q_formatted))
     commit()
 
 
 ARGS = {
-    'OBS_GetMeasureByID': "name, 'us.census.acs.B01001002', '{}'",
+    ('OBS_GetMeasureByID', None): "name, 'us.census.acs.B01001002', '{}'",
-    'OBS_GetMeasure': "{}, 'us.census.acs.B01001002'",
+    ('OBS_GetMeasure', 'predenominated'): "{}, 'us.census.acs.B01003001', null, {}",
-    'OBS_GetCategory': "{}, 'us.census.spielman_singleton_segments.X10'",
+    ('OBS_GetMeasure', 'area'): "{}, 'us.census.acs.B01001002', 'area', {}",
+    ('OBS_GetMeasure', 'denominator'): "{}, 'us.census.acs.B01001002', 'denominator', {}",
+    ('OBS_GetCategory', None): "{}, 'us.census.spielman_singleton_segments.X10', {}",
+    ('_OBS_GetGeometryScores', None): "{}, NULL"
 }
 
-GEOMS = {
+
-    'point': 'ST_PointOnSurface(the_geom)',
+def record(params, results):
-    'polygon_match': 'the_geom',
+    sha = os.environ['OBS_EXTENSION_SHA']
-    'polygon_buffered': 'ST_Buffer(the_geom::GEOGRAPHY, 1000)::GEOMETRY(GEOMETRY, 4326)',
+    msg = os.environ.get('OBS_EXTENSION_MSG')
-}
+    fpath = os.path.join(os.environ['OBS_PERFTEST_DIR'], sha + '.json')
+    if os.path.isfile(fpath):
+        tests = json.load(open(fpath, 'r'))
+    else:
+        tests = {}
+    with open(fpath, 'w') as fhandle:
+        tests[json.dumps(params)] = {
+            'params': params,
+            'results': results
+        }
+        json.dump(tests, fhandle)
+
+@parameterized([
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('simple', '_OBS_GetGeometryScores', 'NULL', 3000),
+
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 1),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 500),
+    ('complex', '_OBS_GetGeometryScores', 'NULL', 3000)
+])
+def test_getgeometryscores_performance(geom_complexity, api_method, filters, target_geoms):
+    print(api_method, geom_complexity, filters, target_geoms)
+
+    rownums = (1, 5, 10, ) if 'complex' in geom_complexity else (5, 25, 50,)
+    results = []
+    for rows in rownums:
+        stmt = '''SELECT {schema}{api_method}(geom, {filters}, {target_geoms})
+                   FROM obs_perftest_{complexity}
+                   WHERE cartodb_id <= {n}'''.format(
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       filters=filters,
+                       target_geoms=target_geoms,
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print(rows, ': ', qps, ' QPS')
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'filters': filters,
+            'target_geoms': target_geoms
+        }, results)
+
+@parameterized([
+    ('simple', 'OBS_GetMeasureByID', None, 'us.census.tiger.census_tract', None),
+    ('complex', 'OBS_GetMeasureByID', None, 'us.census.tiger.county', None),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'point', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', 'NULL'),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'geom', 'NULL'),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', 'NULL'),
+
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetMeasure', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'OBS_GetCategory', None, 'geom', "'us.census.tiger.census_tract'"),
+    ('complex', 'OBS_GetCategory', None, 'offset_geom', "'us.census.tiger.census_tract'"),
+])
+def test_getmeasure_performance(geom_complexity, api_method, normalization, geom, boundary):
+    print(api_method, geom_complexity, normalization, geom, boundary)
+    col = 'measure' if 'measure' in api_method.lower() else 'category'
+    results = []
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (5, 25, 50, )
+    for rows in rownums:
+        stmt = '''UPDATE obs_perftest_{complexity}
+                   SET {col} = {schema}{api_method}({args})
+                   WHERE cartodb_id <= {n}'''.format(
+                       col=col,
+                       complexity=geom_complexity,
+                       schema='cdb_observatory.' if USE_SCHEMA else '',
+                       api_method=api_method,
+                       args=ARGS[api_method, normalization].format(geom, boundary),
+                       n=rows)
+        start = time()
+        query(stmt)
+        end = time()
+        qps = (rows / (end - start))
+        results.append({
+            'rows': rows,
+            'qps': qps,
+            'stmt': stmt
+        })
+        print(rows, ': ', qps, ' QPS')
+
+    if 'OBS_RECORD_TEST' in os.environ:
+        record({
+            'geom_complexity': geom_complexity,
+            'api_method': api_method,
+            'normalization': normalization,
+            'boundary': boundary,
+            'geom': geom
+        }, results)
 
 
 @parameterized([
-    ('OBS_GetMeasureByID', 'us.census.tiger.block_group_clipped'),
+    ('simple', 'predenominated', 'point', 'null'),
-    ('OBS_GetMeasureByID', 'us.census.tiger.county'),
+    ('simple', 'predenominated', 'geom', 'null'),
-    ('OBS_GetMeasure', GEOMS['point']),
+    ('simple', 'predenominated', 'offset_geom', 'null'),
-    ('OBS_GetMeasure', GEOMS['polygon_match']),
+    ('simple', 'area', 'point', 'null'),
-    ('OBS_GetMeasure', GEOMS['polygon_buffered']),
+    ('simple', 'area', 'geom', 'null'),
-    ('OBS_GetCategory', GEOMS['point']),
+    ('simple', 'area', 'offset_geom', 'null'),
-    ('OBS_GetCategory', GEOMS['polygon_match']),
+    ('simple', 'denominator', 'point', 'null'),
-    ('OBS_GetCategory', GEOMS['polygon_buffered']),
+    ('simple', 'denominator', 'geom', 'null'),
+    ('simple', 'denominator', 'offset_geom', 'null'),
+
+    ('simple', 'predenominated', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'predenominated', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'area', 'offset_geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'point', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'geom', "'us.census.tiger.census_tract'"),
+    ('simple', 'denominator', 'offset_geom', "'us.census.tiger.census_tract'"),
+
+    ('complex', 'predenominated', 'geom', 'null'),
+    ('complex', 'predenominated', 'offset_geom', 'null'),
+    ('complex', 'area', 'geom', 'null'),
+    ('complex', 'area', 'offset_geom', 'null'),
+    ('complex', 'denominator', 'geom', 'null'),
+    ('complex', 'denominator', 'offset_geom', 'null'),
+
+    ('complex', 'predenominated', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'predenominated', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'area', 'offset_geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'geom', "'us.census.tiger.county'"),
+    ('complex', 'denominator', 'offset_geom', "'us.census.tiger.county'"),
 ])
-def test_performance(api_method, arg):
+def test_getdata_performance(geom_complexity, normalization, geom, boundary):
-    print api_method, arg
+    print(geom_complexity, normalization, geom, boundary)
-    col = 'measure' if 'measure' in api_method.lower() else 'category'
+
-    for rows in (1, 10, 50, 100):
+    cols = ['us.census.acs.B01001002',
-        q = 'UPDATE obs_censustest SET {col} = {schema}{api_method}({args}) WHERE cartodb_id < {n}'.format(
+            'us.census.acs.B01001003',
-            col=col,
+            'us.census.acs.B01001004',
-            schema='cdb_observatory.' if USE_SCHEMA else '',
+            'us.census.acs.B01001005',
-            api_method=api_method,
+            'us.census.acs.B01001006',
-            args=ARGS[api_method].format(arg),
+            'us.census.acs.B01001007',
-            n=rows+1)
+            'us.census.acs.B01001008',
-        start = time()
+            'us.census.acs.B01001009',
-        query(q)
+            'us.census.acs.B01001010',
-        end = time()
+            'us.census.acs.B01001011', ]
-        print rows, ': ', (rows / (end - start)), ' QPS'
+    in_meta = [{"numer_id": col,
+                "normalization": normalization,
+                "geom_id": None if boundary.lower() == 'null' else boundary.replace("'", '')}
+               for col in cols]
+
+    rownums = (1, 5, 10, ) if geom_complexity == 'complex' else (10, 50, 100)
+
+    for num_meta in (1, 10, ):
+        results = []
+        for rows in rownums:
+            stmt = '''
+    with data as (
+      SELECT id, data FROM {schema}OBS_GetData(
+        (SELECT array_agg(({geom}, cartodb_id)::geomval)
+         FROM obs_perftest_{complexity}
+         WHERE cartodb_id <= {n}),
+        (SELECT {schema}OBS_GetMeta(
+          (SELECT st_setsrid(st_extent({geom}), 4326)
+           FROM obs_perftest_{complexity}
+           WHERE cartodb_id <= {n}),
+          '{in_meta}'::JSON
+        ))
+      ))
+    UPDATE obs_perftest_{complexity}
+    SET measure = (data->0->>'value')::Numeric
+    FROM data
+    WHERE obs_perftest_{complexity}.cartodb_id = data.id
+    ;
+            '''.format(
+                point_or_poly='point' if geom == 'point' else 'polygon',
+                complexity=geom_complexity,
+                schema='cdb_observatory.' if USE_SCHEMA else '',
+                geom=geom,
+                in_meta=json.dumps(in_meta[0:num_meta]),
+                n=rows)
+            start = time()
+            query(stmt)
+            end = time()
+            qps = (rows / (end - start))
+            results.append({
+                'rows': rows,
+                'qps': qps,
+                'stmt': stmt
+            })
+            print(rows, ': ', qps, ' QPS')
+
+        if 'OBS_RECORD_TEST' in os.environ:
+            record({
+                'geom_complexity': geom_complexity,
+                'api_method': 'OBS_GetData',
+                'normalization': normalization,
+                'boundary': boundary,
+                'geom': geom,
+                'num_meta': str(num_meta)
+            }, results)