Merge pull request #225 from CartoDB/develop

Release 1.1.5
Version 1.1.5 artifacts
2016-11-29 17:58:08 +01:00 · 2016-11-29 17:43:28 +01:00 · 2016-11-29 17:40:53 +01:00 · 2016-11-29 16:45:10 +00:00 · 2016-11-29 11:35:19 -05:00 · 2016-11-29 16:41:44 +00:00
45 changed files with 45618 additions and 25756 deletions
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,17 @@
+## Request for a new Data observatory extension deploy
+
+I'd like to request a new data observatory extension deploy: dump + extension
+
+## Dump database id to be deployed
+
+Please put here the dump id to be deployed: <dump_id>
+
+## Data Observatory extension PRs included.
+
+*Please update the NEWS.md*
+
+Add down here the PR links to be added and deployed:
+
+  - 
+
+// @CartoDB/dataservices
--- a/2
+++ b/2
@@ -18,7 +18,7 @@ test:   ## Run the tests for the development version of the extension
 	$(MAKE) -C $(EXT_DIR) test

 # Generate a new release into release
-release: ## Generate a new release of the extension. Only for telease manager
+release: ## Generate a new release of the extension. Only for release manager
 	$(MAKE) -C $(EXT_DIR) release

 # Install the current release.
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,3 +1,121 @@
+1.1.5 (2016-11-29)
+
+__Bugfixes__
+
+* Return `NULL` instead of raising an exception when a measure is requested for
+  a geometry where it does not exist ([#220](https://github.com/CartoDB/observatory-extension/issues/220)).
+
+1.1.4 (2016-11-21)
+
+__Bugfixes__
+
+* Fix duplicate subsections with only a partial set of measures appearing from
+  `OBS_GetLegacyMetadata` ([#216](https://github.com/CartoDB/observatory-extension/issues/216)).
+
+1.1.3 (2016-11-15)
+
+* Temporarily ignore EU data for the sake of testing
+
+1.1.2 (2016-11-09)
+
+__Improvements__
+
+* Update public `OBS_GetMeasure` to use highest ranked boundary, aiming for 500
+  geoms. ([#190](https://github.com/CartoDB/observatory-extension/issues/190))
+* Update test generation to capture our raster tiles
+* Standardize the way we generate our test points for `autotest.py`
+* Add points for epa and eurostat
+* Should support database dump generated 20161109
+
+__API Changes (Internal)__
+
+* Add internal `_OBS_GetGeometryScores`
+
+1.1.1 (2016-10-14)
+
+__Improvements__
+
+* Test points for Canada and France ([#204](https://github.com/CartoDB/observatory-extension/issues/120))
+
+1.1.0 (2016-10-04)
+
+__Bugfixes__
+
+* Fixed some minor errors in test suite
+
+__Improvements__
+
+* We now generate test fixtures from local data instead of remote server
+  ([#120](https://github.com/CartoDB/observatory-extension/issues/120))
+
+__API Changes__
+
+* New function, `OBS_LegacyBuilderMetadata`, which resolves
+  ([#133]( https://github.com/CartoDB/observatory-extension/issues/133))
+* Creates "dimensional" metadata grabbing functions
+  (`OBS_GetAvailableNumerators`, `OBS_GetAvailableDenominators`,
+  `OBS_GetAvailableGeometries`, `OBS_GetAvailableTimespans`) which will be
+  used for obtaining metadata in the replacement for the Data Library
+  ([CartoDB/design#104](https://github.com/CartoDB/design/issues/104)). This
+  is also referred to here ([CartoDB/design#68](https://github.com/CartoDB/design/issues/68)).
+
+1.0.7 (2016-09-20)
+
+__Bugfixes__
+
+* `NULL` geometries or geometry IDs no longer result in an exception from any
+  augmentation functions ([#178](https://github.com/CartoDB/observatory-extension/issues/178))
+
+__Improvements__
+
+* Automatic tests work for Canada and Thailand
+
+1.0.6 (2016-09-08)
+
+__Improvements__
+
+* New function structure for Table-level functions which allows to separate the
+  framework logic from the observatory measure functions.
+
+1.0.5 (2016-08-12)
+
+__Improvements__
+
+* Integration tests moved to `src/python/test/`, and can be run without hitting
+  any HTTP SQL API.
+
+1.0.4 (2016-07-26)
+
+__Bugfixes__
+
+* Always default arguments to `NULL`, which prevents duplication & overwrite by
+  dataservices-api
+  ([#173](https://github.com/CartoDB/observatory-extension/issues/173))
+
+1.0.3 (2016-07-25)
+
+__Bugfixes__
+
+* Raise exception instead of crashing when `OBS_GetMeasure` is passed a polygon
+  in combination with a non-summable measure ([cartodb/issues
+  #9063](https://github.com/CartoDB/cartodb/issues/9063))
+* Unnecessary dependencies on cartodb and plpythonu removed
+  ([#161](https://github.com/CartoDB/observatory-extension/issues/161))
+* Tests forced to run in-order on all systems
+  ([#162](https://github.com/CartoDB/observatory-extension/issues/162))
+* Area normalization done by square kilometer instead of square meter for
+  polygons ([#158](https://github.com/CartoDB/observatory-extension/issues/158))
+* `postgres-fdw` installed as required in unit test environment
+  ([#166](https://github.com/CartoDB/observatory-extension/issues/166))
+
+__Improvements__
+
+* Added tests to make sure all functions can handle explicit NULL as default
+  ([#159](https://github.com/CartoDB/observatory-extension/issues/159))
+* Buffer and snaptogrid used to be far more liberal accepting problem geoms
+  ([#170](https://github.com/CartoDB/observatory-extension/issues/160))
+
+
 1.0.2 (2016-07-12)
 ---

--- a/doc/API.md
+++ b/doc/API.md
@@ -1,6 +1,6 @@
 # Data Observatory Documentation

-This file is for reference purposes only. It is intended for tracking the Data Observatory functions that should be displayed from the live Docs site. Like all API doc, the golden source of this code will live in this observatory-extension repo, and will be edited in this repo.
+This file is for reference purposes only. It is intended for tracking the Data Observatory functions that should be pulled into the live Docs site. Like all API doc, the golden source of this code will live in this observatory-extension repo, and will be edited in this repo. Other non-code related content will live as a local file in the Docs repo.

 ## Documentation

@@ -9,5 +9,5 @@ This file is for reference purposes only. It is intended for tracking the Data O
 * [Measures Functions](measures_functions.md)
 * [Boundary Functions](boundary_functions.md)
 * [Discovery Functions](discovery_functions.md)
-* [Glossary](glossary.md)
-* [License](license.md)
+* [Glossary](local file in the Docs repo)
+* [License](local file in the Docs repo)
--- a/doc/boundary_functions.md
+++ b/doc/boundary_functions.md
@@ -1,8 +1,8 @@
 # Boundary Functions

-Use the following functions to retrieve [Boundary](/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
+Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.

-You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).

 ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)

@@ -91,14 +91,14 @@ FROM OBS_GetPointsByGeometry(

 ## OBS_GetBoundary(point_geometry, boundary_id)

-The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID glossary table below](below). This is a useful method for performing aggregations of points.
+The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids). This is a useful method for performing aggregations of points.

 #### Arguments

 Name | Description
 --- | ---
 point_geometry | a WGS84 polygon geometry (the_geom)
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)

 #### Returns
@@ -124,14 +124,14 @@ SET the_geom = OBS_GetBoundary(the_geom, 'us.census.tiger.block_group')

 ## OBS_GetBoundaryId(point_geometry, boundary_id)

-The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID glossary table below](below). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
+The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID Glossary](http://docs/carto-engine/data/glossary/#boundary-ids). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.

 #### Arguments

 Name |Description
 --- | ---
 point_geometry | a WGS84 point geometry (the_geom)
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)

 #### Returns
@@ -164,7 +164,7 @@ The ```OBS_GetBoundaryById(geometry_id, boundary_id)``` returns the boundary geo
 Name | Description
 --- | ---
 geometry_id | a string identifier for a Boundary geometry
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)

 #### Returns
--- a/doc/discovery_functions.md
+++ b/doc/discovery_functions.md
@@ -1,6 +1,6 @@
 # Discovery Functions

-If you are using the [discovery methods](/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/carto-engine/data/overview/#boundary-data) and [measures](/carto-engine/data/overview/#measures-data) data.
+If you are using the [discovery methods](https://carto.com/docs/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) and [measures](https://carto.com/docs/carto-engine/data/overview/#measures-data) data.

 ## OBS_Search(search_term)

@@ -47,7 +47,7 @@ A TABLE containing the following properties

 Key | Description
 --- | ---
-boundary_id | a boundary identifier from the [boundary ID glossary](/carto-engine/data/glossary/#boundary-ids)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 description | a brief description of the boundary dataset
 time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.

--- a/doc/glossary.md
+++ b/doc/glossary.md
@@ -1,126 +0,0 @@
-# Glossary
-
-A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
-
-## Boundary IDs
-
-Boundary name | Boundary ID | Shoreline Clipped Boundary ID
--------------------- | --------------------- | ---
-US States  |  us.census.tiger.state  |  us.census.tiger.state_clipped
-US County  |  us.census.tiger.county  |  us.census.tiger.county_clipped
-US Census Zip Code Tabulation Areas  |  us.census.tiger.zcta5  |  us.census.tiger.zcta5_clipped
-US Census Tracts  |  us.census.tiger.census_tract  |  us.census.tiger.census_tract_clipped
-US Elementary School District  |  us.census.tiger.school_district_elementary  |  us.census.tiger.school_district_elementary_clipped
-US Secondary School District  |  us.census.tiger.school_district_secondary  |  us.census.tiger.school_district_secondary_clipped
-US Unified School District  |  us.census.tiger.school_district_unified  |  us.census.tiger.school_district_unified_clipped
-US Congressional Districts  |  us.census.tiger.congressional_district  |  us.census.tiger.congressional_district_clipped
-US Census Blocks  |  us.census.tiger.block  |  us.census.tiger.block_clipped
-US Census Block Groups  |  us.census.tiger.block_group  |  us.census.tiger.block_group_clipped
-US Census PUMAs  |  us.census.tiger.puma  |  us.census.tiger.puma_clipped
-US Incorporated Places  |  us.census.tiger.place  |  us.census.tiger.place_clipped
-ES Sección Censal  |  es.ine.geom  |   none
-Regions (First-level Administrative)  |  whosonfirst.wof_region_geom  |   none
-Continents  |  whosonfirst.wof_continent_geom  |   none
-Countries  |  whosonfirst.wof_country_geom  |  none
-Marine Areas  |  whosonfirst.wof_marinearea_geom  |  none
-Disputed Areas  |  whosonfirst.wof_disputed_geom  |  none
-
-
-
-## OBS_GetUSCensusMeasure Names Table
-
-This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
-
-Measure name             |  Measure description
------------------------ | --------------------
-Male Population   |   The number of people within each geography who are male.
-Female Population   |   The number of people within each geography who are female.
-Median Age   |   The median age of all people in a given geographic area.
-Total Population   |   The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.
-Population not Hispanic   |   The number of people not identifying as Hispanic or Latino in each geography.
-White Population   |   The number of people identifying as white, non-Hispanic in each geography.
-Black or African American Population   |   The number of people identifying as black or African American, non-Hispanic in each geography.
-American Indian and Alaska Native Population   |   The number of people identifying as American Indian or Alaska native in each geography.
-Asian Population   |   The number of people identifying as Asian, non-Hispanic in each geography.
-Other Race population   |   The number of people identifying as another race in each geography.
-Two or more races population   |   The number of people identifying as two or more races in each geography.
-Hispanic Population   |   The number of people identifying as Hispanic or Latino in each geography.
-Not a U.S. Citizen Population   |   The number of people within each geography who indicated that they are not U.S. citizens.
-Workers over the Age of 16   |   The number of people in each geography who work. Workers include those employed at private for-profit companies, the self-employed, government workers and non-profit employees.
-Commuters by Car, Truck, or Van   |   The number of workers age 16 years and over within  a geographic area who primarily traveled to work by car, truck or  van.  This is the principal mode of travel or type of conveyance,  by distance rather than time, that the worker usually used to get  from home to work.
-Commuters who drove alone   |   The number of workers age 16 years and over within a geographic area who primarily traveled by car driving alone. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Commuters by Carpool   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by carpool.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Commuters by Bus   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by bus.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.  This is a subset of workers who commuted by public transport.
-Commuters by Subway or Elevated   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by subway or elevated train.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.  This is a subset of workers who commuted by public transport.
-Walked to Work   |   The number of workers age 16 years and over within a geographic area who primarily walked to work.  This would mean that of any way of getting to work, they travelled the most distance walking.
-Worked at Home   |   The count within a geographical area of workers over the age of 16 who worked at home.
-Workers age 16 and over who do not work from home   |   The number of workers over the age of 16 who do not work from home in a geographic area.
-Number of workers with less than 10 minute commute   |   The number of workers over the age of 16 who do not work from home and commute in less than 10 minutes in a geographic area.
-Number of workers with a commute between 35 and 44 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 35 and 44 minutes in a geographic area.
-Number of workers with a commute of over 60 minutes   |   The number of workers over the age of 16 who do not work from home and commute in over 60 minutes in a geographic area.
-Aggregate travel time to work   |   The total number of minutes every worker over the age of 16 who did not work from home spent spent commuting to work in one day in a geographic area.
-Commuters by Public Transportation   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by public transportation.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Number of workers with a commute between 10 and 14 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 10 and 14 minutes in a geographic area.
-Number of workers with a commute between 15 and 19 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 15 and 19 minutes in a geographic area.
-Number of workers with a commute between 20 and 24 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 20 and 24 minutes in a geographic area.
-Number of workers with a commute between 25 and 29 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 25 and 29 minutes in a geographic area.
-Number of workers with a commute between 30 and 34 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area.
-Number of workers with a commute between 45 and 59 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area.
-Children under 18 Years of Age   |   The number of people within each geography who are under 18 years of age.
-Households   |   A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
-Population 15 Years and Over   |   The number of people in a geographic area who are over the age of 15.  This is used mostly as a denominator of marital status.
-Never Married   |   The number of people in a geographic area who have never been married.
-Currently married   |   The number of people in a geographic area who are currently married.
-Married but separated   |   The number of people in a geographic area who are married but separated.
-Widowed   |   The number of people in a geographic area who are widowed.
-Divorced   |   The number of people in a geographic area who are divorced.
-Population 3 Years and Over   |   The total number of people in each geography age 3 years and over.  This denominator is mostly used to calculate rates of school enrollment.
-Students Enrolled in School   |   The total number of people in each geography currently enrolled at any level of school, from nursery or pre-school to advanced post-graduate education.  Only includes those over the age of 3.
-Students Enrolled in Grades 1 to 4   |   The total number of people in each geography currently enrolled in grades 1 through 4 inclusive.  This corresponds roughly to elementary school.
-Students Enrolled in Grades 5 to 8   |   The total number of people in each geography currently enrolled in grades 5 through 8 inclusive.  This corresponds roughly to middle school.
-Students Enrolled in Grades 9 to 12   |   The total number of people in each geography currently enrolled in grades 9 through 12 inclusive.  This corresponds roughly to high school.
-Students Enrolled as Undergraduate in College   |   The number of people in a geographic area who are enrolled in college at the undergraduate level. Enrollment refers to being registered or listed as a student in an educational program leading to a college degree. This may be a public school or college, a private school or college.
-Population 25 Years and Over   |   The number of people in a geographic area who are over the age of 25.  This is used mostly as a denominator of educational attainment.
-Population Completed High School   |   The number of people in a geographic area over the age of 25 who completed high school, and did not complete a more advanced degree.
-Population completed less than one year of college, no degree   |   The number of people in a geographic area over the age of 25 who attended college for less than one year and no further.
-Population completed more than one year of college, no degree   |   The number of people in a geographic area over the age of 25 who attended college for more than one year but did not obtain a degree.
-Population Completed Associate's Degree   |   The number of people in a geographic area over the age of 25 who obtained a associate's degree, and did not complete a more advanced degree.
-Population Completed Bachelor's Degree   |   The number of people in a geographic area over the age of 25 who obtained a bachelor's degree, and did not complete a more advanced degree.
-Population Completed Master's Degree   |   The number of people in a geographic area over the age of 25 who obtained a master's degree, but did not complete a more advanced degree.
-Population 5 Years and Over   |   The number of people in a geographic area who are over the age of 5.  This is primarily used as a denominator of measures of language spoken at home.
-Speaks only English at Home   |   The number of people in a geographic area over age 5 who speak only English at home.
-Speaks Spanish at Home   |   The number of people in a geographic area over age 5 who speak Spanish at home, possibly in addition to other languages.
-Population for Whom Poverty Status Determined   |   The number of people in each geography who could be identified as either living in poverty or not.  This should be used as the denominator when calculating poverty rates, as it excludes people for whom it was not possible to determine poverty.
-Income In The Past 12 Months Below Poverty Level   |   The number of people in a geographic area who are part of a family (which could be just them as an individual) determined to be in poverty following the Office of Management and Budget's Directive 14. (https://www.census.gov/hhes/povmeas/methodology/ombdir14.html)
-Households with income less than $10,000   |   The number of households in a geographic area whose annual income was less than $10,000.
-Households with income of $10,000 to $14,999   |   The number of households in a geographic area whose annual income was between $10,000 and $14,999.
-Households with income of $15,000 to $19,999   |   The number of households in a geographic area whose annual income was between $15,000 and $19,999.
-Households with income of $20,000 To $24,999   |   The number of households in a geographic area whose annual income was between $20,000 and $24,999.
-Households with income of $25,000 To $29,999   |   The number of households in a geographic area whose annual income was between $20,000 and $24,999.
-Households with income of $30,000 To $34,999   |   The number of households in a geographic area whose annual income was between $30,000 and $34,999.
-Households with income of $35,000 To $39,999   |   The number of households in a geographic area whose annual income was between $35,000 and $39,999.
-Households with income of $40,000 To $44,999   |   The number of households in a geographic area whose annual income was between $40,000 and $44,999.
-Households with income of $45,000 To $49,999   |   The number of households in a geographic area whose annual income was between $45,000 and $49,999.
-Households with income of $50,000 To $59,999   |   The number of households in a geographic area whose annual income was between $50,000 and $59,999.
-Households with income of $60,000 To $74,999   |   The number of households in a geographic area whose annual income was between $60,000 and $74,999.
-Households with income of $75,000 To $99,999   |   The number of households in a geographic area whose annual income was between $75,000 and $99,999.
-Households with income of $100,000 To $124,999   |   The number of households in a geographic area whose annual income was between $100,000 and $124,999.
-Households with income of $125,000 To $149,999   |   The number of households in a geographic area whose annual income was between $125,000 and $149,999.
-Households with income of $150,000 To $199,999   |   The number of households in a geographic area whose annual income was between $150,000 and $1999,999.
-Households with income of $200,000 Or More   |   The number of households in a geographic area whose annual income was more than $200,000.
-Median Household Income in the past 12 Months   |   Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc.  It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans' (VA) payments, unemployment and/or worker's compensation, child support, and alimony.
-Population age 16 and over   |   The number of people in each geography who are age 16 or over.
-Population in Labor Force   |   The number of people in each geography who are either in the civilian labor force or are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
-Population in Civilian Labor Force   |   The number of civilians 16 years and over in each geography who can be classified as either employed or unemployed below.
-Employed Population   |   The number of civilians 16 years old and over in each geography who either (1) were at work, that is, those who did any work at all during the reference week as paid employees, worked in their own business or profession, worked on their own farm, or worked 15 hours or more as unpaid workers on a family farm or in a family business; or (2) were with a job but not at work, that is, those who did not work during the reference week but had jobs or businesses from which they were temporarily absent due to illness, bad weather, industrial dispute, vacation, or other personal reasons. Excluded from the employed are people whose only activity consisted of work around the house or unpaid volunteer work for religious, charitable, and similar organizations; also excluded are all institutionalized people and people on active duty in the United States Armed Forces.
-Unemployed Population   |   The number of civilians in each geography who are 16 years old and over and are classified as unemployed.
-Population in Armed Forces   |   The number of people in each geography who are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
-Population Not in Labor Force   |   The number of people in each geography who are 16 years old and over who are not classified as members of the labor force. This category consists mainly of students, homemakers, retired workers, seasonal workers interviewed in an off season who were not looking for work, institutionalized people, and people doing only incidental unpaid family work.
-Housing Units   |   A count of housing units in each geography.  A housing unit is a house, an apartment, a mobile home or trailer, a group of rooms, or a single room occupied as separate living quarters, or if vacant, intended for occupancy as separate living quarters.
-Vacant Housing Units   |   The count of vacant housing units in a geographic area. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Vacant Housing Units for Rent   |   The count of vacant housing units in a geographic area that are for rent. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Vacant Housing Units for Sale   |   The count of vacant housing units in a geographic area that are for sale. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Median Rent   |   The median contract rent within a geographic area. The contract rent is the monthly rent agreed to or contracted for, regardless of any furnishings, utilities, fees, meals, or services that may be included. For vacant units, it is the monthly rent asked for the rental unit at the time of interview.
-Percent of Household Income Spent on Rent   |   Within a geographic area, the median percentage of household income which was spent on gross rent.  Gross rent is the amount of the contract rent plus the estimated average monthly cost of utilities (electricity, gas, water, sewer etc.) and fuels (oil, coal, wood, etc.) if these are paid by the renter.  Household income is the sum of the income of all people 15 years and older living in the household.
-Owner-occupied Housing Units valued at $1,000,000 or more.   |   The count of owner occupied housing units in a geographic area that are valued at $1,000,000 or more.  Value is the respondent's estimate of how much the property (house and lot, mobile home and lot, or condominium unit) would sell for if it were for sale.
-Owner-occupied Housing Units with a Mortgage   |   The count of housing units within a geographic area that are mortagaged. Mortgage refers to all forms of debt where the property is pledged as security for repayment of the debt, including deeds of trust, trust deed, contracts to purchase, land contracts, junior mortgages, and home equity loans.
--- a/doc/license.md
+++ b/doc/license.md
@@ -1,20 +0,0 @@
-# License
-
-The Data Observatory is a collection of data sources with varying licenses and terms of use. We have endeavored to find you data that will work for the broadest set of use-cases. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
-
-_**Legal Note**: The Data Observatory makes use of a variety of third party data and databases (collectively, the “Data”). You acknowledge that the included Data, and the licenses and terms of use, may be amended from time to time. Whenever you use the Data, you agree to the current relevant terms or license. Some Data will require that you provide attribution to the data source. Other Data may be protected by US or international copyright laws, treaties, or conventions. The Data and associated metadata are provided 'as-is', without express or implied warranty of any kind, including, but not limited to, infringement, merchantability and fitness for a particular purpose. CartoDB is not responsible for the accuracy, completeness, timeliness or quality of the Data._
-
-Name  | Terms link 
-------|---------
-ACS   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
-TIGER   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
-Zillow Home Value Index | This data is "Aggregate Data", per the Zillow Terms of Use<br /><br />[http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm)
-Who's on First |  [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License)
-GeoNames  |  [http://www.geonames.org/](http://www.geonames.org/)
-GeoPlanet  |  [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/)
-Natural Earth  |  [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/)
-Quattroshapes  |  [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
-Zetashapes  |  [http://zetashapes.com/license](http://zetashapes.com/license)
-Spielman & Singleton  |  [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
-El Instituto Nacional de Estadística (INE) | The National Statistics Institute (INE) of Spain includes data from multiple sources. If you are re-using their data, they explicitly require that you reference them accordingly<br /><br />[http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
-
--- a/doc/measures_functions.md
+++ b/doc/measures_functions.md
@@ -1,10 +1,10 @@
 # Measures Functions

-[Data Observatory Measures](/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
+[Data Observatory Measures](https://carto.com/docs/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.

 There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).

-You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).

 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)

@@ -15,7 +15,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
 Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
-measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
+measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
 normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
@@ -46,7 +46,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
 Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
-measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
+measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
 normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
--- a/release/observatory--1.0.3.sql
+++ b/release/observatory--1.0.3.sql
--- a/release/observatory--1.0.4.sql
+++ b/release/observatory--1.0.4.sql
--- a/release/observatory--1.0.5.sql
+++ b/release/observatory--1.0.5.sql
--- a/release/observatory--1.0.6.sql
+++ b/release/observatory--1.0.6.sql
--- a/release/observatory--1.0.7.sql
+++ b/release/observatory--1.0.7.sql
--- a/release/observatory--1.1.0.sql
+++ b/release/observatory--1.1.0.sql
--- a/release/observatory--1.1.1.sql
+++ b/release/observatory--1.1.1.sql
--- a/release/observatory--1.1.2.sql
+++ b/release/observatory--1.1.2.sql
--- a/release/observatory--1.1.3.sql
+++ b/release/observatory--1.1.3.sql
--- a/release/observatory--1.1.4.sql
+++ b/release/observatory--1.1.4.sql
--- a/release/observatory--1.1.5.sql
+++ b/release/observatory--1.1.5.sql
--- a/release/observatory.control
+++ b/release/observatory.control
@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.2'
-requires = 'postgis'
+default_version = '1.1.5'
+requires = 'postgis, postgres_fdw'
 superuser = true
 schema = cdb_observatory
--- a/scripts/generate_fixtures.py
+++ b/scripts/generate_fixtures.py
@@ -1,213 +1,331 @@
+import os
+import psycopg2
+import subprocess
+
+DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
+    user=os.environ.get('PGUSER', 'postgres'),
+    password=os.environ.get('PGPASSWORD', ''),
+    host=os.environ.get('PGHOST', 'localhost'),
+    port=os.environ.get('PGPORT', '5432'),
+    database=os.environ.get('PGDATABASE', 'postgres'),
+))
+CURSOR = DB_CONN.cursor()
+
+
+def query(q):
+    '''
+    Query the database.
+    '''
+    try:
+        CURSOR.execute(q)
+        return CURSOR
+    except:
+        DB_CONN.rollback()
+        raise
+
+
+def commit():
+    try:
+        DB_CONN.commit()
+    except:
+        DB_CONN.rollback()
+        raise

-from sqldumpr import Dumpr

 def get_tablename_query(column_id, boundary_id, timespan):
    """
        given a column_id, boundary-id (us.census.tiger.block_group), and
        timespan, give back the current table hash from the data observatory
    """
-    q = """
-        SELECT t.tablename, geoid_ct.colname colname
-        FROM obs_table t,
-             obs_column_table geoid_ct,
-             obs_column_table data_ct
+    return """
+        SELECT t.tablename, geoid_ct.colname colname, t.id table_id
+        FROM observatory.obs_table t,
+             observatory.obs_column_table geoid_ct,
+             observatory.obs_column_table data_ct
        WHERE
             t.id = geoid_ct.table_id AND
             t.id = data_ct.table_id AND
             geoid_ct.column_id =
        (SELECT source_id
-         FROM obs_column_to_column
+         FROM observatory.obs_column_to_column
         WHERE target_id = '{boundary_id}'
         AND reltype = 'geom_ref'
        ) AND
        data_ct.column_id = '{column_id}' AND
        timespan = '{timespan}'
-    """.replace('\n','')
+    """.format(column_id=column_id,
+               boundary_id=boundary_id,
+               timespan=timespan)

-    return q.format(column_id=column_id,
-                    boundary_id=boundary_id,
-                    timespan=timespan)

-def select_star(tablename):
-    return "SELECT * FROM {}".format(tablename)
+METADATA_TABLES = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
+                   'obs_tag', 'obs_column_to_column', 'obs_dump_version', 'obs_meta',
+                   'obs_meta_numer', 'obs_meta_denom', 'obs_meta_geom',
+                   'obs_meta_timespan', 'obs_column_table_tile', ]

-cdb = Dumpr('observatory.cartodb.com','')
-
-metadata = ['obs_table', 'obs_column_table', 'obs_column', 'obs_column_tag',
-            'obs_tag', 'obs_column_to_column', 'obs_dump_version', ]
-
-fixtures = [
+FIXTURES = [
+    ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01001026_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01002001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002004_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B03002012_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B05001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006008_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006009_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006011_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006015_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B08006017_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B09001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B11001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001005_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001006_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001007_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B14001008_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003017_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003022_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B15003023_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B16001003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B17001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B17001002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19013001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19083001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B19301001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25001001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25002003_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004002_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25004004_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25058001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25071001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B25075025_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01001026', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B01002001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002012', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B03002002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B11001001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003017', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003019', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003020', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003021', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003022', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B15003023', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19013001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19083001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19301001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25001001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25002003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25004002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25004004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25058001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25071001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25075001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25075025', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B25081002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134007', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08134010', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B08135001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001002', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001003', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001004', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001005', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001006', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001007', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001008', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001009', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001010', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001011', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001012', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001013', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001014', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001015', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001016', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.acs.B19001017', 'us.census.tiger.block_group', '2010 - 2014'),
    ('us.census.tiger.census_tract', 'us.census.tiger.census_tract', '2014'),
    ('us.census.tiger.block_group', 'us.census.tiger.block_group', '2014'),
    ('us.census.tiger.zcta5', 'us.census.tiger.zcta5', '2014'),
    ('us.census.tiger.county', 'us.census.tiger.county', '2014'),
+    ('us.census.acs.B01001002', 'us.census.tiger.block_group', '2010 - 2014'),
    ('us.census.acs.B01003001', 'us.census.tiger.census_tract', '2010 - 2014'),
    ('us.census.acs.B01003001_quantile', 'us.census.tiger.census_tract', '2010 - 2014'),
    ('us.census.acs.B01003001', 'us.census.tiger.block_group', '2010 - 2014'),
+    ('us.census.spielman_singleton_segments.X2', 'us.census.tiger.census_tract', '2010 - 2014'),
    ('us.census.spielman_singleton_segments.X10', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.spielman_singleton_segments.X31', 'us.census.tiger.census_tract', '2010 - 2014'),
+    ('us.census.spielman_singleton_segments.X55', 'us.census.tiger.census_tract', '2010 - 2014'),
    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2014-01'),
-    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-03'),
+    ('us.zillow.AllHomes_Zhvi', 'us.census.tiger.zcta5', '2016-06'),
    ('whosonfirst.wof_country_geom', 'whosonfirst.wof_country_geom', '2016'),
    ('us.census.tiger.zcta5_clipped', 'us.census.tiger.zcta5_clipped', '2014'),
    ('us.census.tiger.block_group_clipped', 'us.census.tiger.block_group_clipped', '2014'),
+    ('us.census.tiger.census_tract_clipped', 'us.census.tiger.census_tract_clipped', '2014'),
 ]

-unique_tables = set()
+OUTFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
+                            'src/pg/test/fixtures/load_fixtures.sql')
+DROPFILE_PATH = os.path.join(os.path.dirname(__file__), '..',
+                             'src/pg/test/fixtures/drop_fixtures.sql')

-for f in fixtures:
-    column_id, boundary_id, timespan = f
-    tablename_query = get_tablename_query(*f)
-    resp = cdb.query(tablename_query).json()['rows'][0]
-    tablename = resp['tablename']
-    colname = resp['colname']
-    table_colname = (tablename, colname, boundary_id, )
-    if table_colname not in unique_tables:
-        print table_colname
-        unique_tables.add(table_colname)
+def dump(cols, tablename, where=''):

-print unique_tables
+    with open(DROPFILE_PATH, 'a') as dropfile:
+        dropfile.write('DROP TABLE IF EXISTS observatory.{tablename};\n'.format(
+            tablename=tablename,
+        ))

-with open('src/pg/test/fixtures/load_fixtures.sql', 'w') as outfile:
-    with open('src/pg/test/fixtures/drop_fixtures.sql', 'w') as dropfiles:
-        outfile.write('SET client_min_messages TO WARNING;\n\set ECHO none\n')
-        dropfiles.write('SET client_min_messages TO WARNING;\n\set ECHO none\n')
-        for tablename in metadata:
-            cdb.dump(select_star(tablename), tablename, outfile, schema='observatory')
-            dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
-            print tablename
+    subprocess.check_call('pg_dump -x --section=pre-data -t observatory.{tablename} '
+                          ' | sed "s:SET search_path.*::" '
+                          ' | sed "s:CREATE TABLE :CREATE TABLE observatory.:" '
+                          ' | sed "s:ALTER TABLE.*OWNER.*::" '
+                          ' >> {outfile}'.format(
+                              tablename=tablename,
+                              outfile=OUTFILE_PATH,
+                          ), shell=True)

-        for tablename, colname, boundary_id in unique_tables:
-            if 'zcta5' in boundary_id:
-                where = '\'11%\''
-                compare = 'LIKE'
-            elif 'whosonfirst' in boundary_id:
-                where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
-                compare = 'IN'
-            else:
-                where = '\'36047%\''
-                compare = 'LIKE'
-            print ' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)])
-            cdb.dump(' '.join([select_star(tablename), "WHERE {}::text {} {}".format(colname, compare, where)]),
-                     tablename, outfile, schema='observatory')
-            dropfiles.write('DROP TABLE IF EXISTS observatory.{};\n'.format(tablename))
+    with open(OUTFILE_PATH, 'a') as outfile:
+        outfile.write('COPY observatory."{}" FROM stdin WITH CSV HEADER;\n'.format(tablename))
+
+    subprocess.check_call('''
+      psql -c "COPY (SELECT {cols} \
+      FROM observatory.{tablename} {where}) \
+      TO STDOUT WITH CSV HEADER" >> {outfile}'''.format(
+          cols=cols,
+          tablename=tablename,
+          where=where,
+          outfile=OUTFILE_PATH,
+      ), shell=True)
+
+    with open(OUTFILE_PATH, 'a') as outfile:
+        outfile.write('\\.\n\n')


-        outfile.write('''
-ALTER TABLE observatory.obs_table
-  ADD PRIMARY KEY (id);
-ALTER TABLE observatory.obs_column_table
-  ADD PRIMARY KEY (column_id, table_id);
-CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, column_id);
-CREATE UNIQUE INDEX ON observatory.obs_column_table (table_id, colname);
-ALTER TABLE observatory.obs_column
-  ADD PRIMARY KEY (id);
-ALTER TABLE observatory.obs_column_to_column
-  ADD PRIMARY KEY (source_id, target_id, reltype);
-CREATE UNIQUE INDEX ON observatory.obs_column_to_column (target_id, source_id, reltype);
-CREATE INDEX ON observatory.obs_column_to_column (reltype);
-ALTER TABLE observatory.obs_column_tag
-  ADD PRIMARY KEY (column_id, tag_id);
-CREATE UNIQUE INDEX ON observatory.obs_column_tag (tag_id, column_id);
-ALTER TABLE observatory.obs_tag
-  ADD PRIMARY KEY (id);
-CREATE INDEX ON observatory.obs_tag (type);
+def main():
+    unique_tables = set()

-VACUUM ANALYZE observatory.obs_table;
-VACUUM ANALYZE observatory.obs_column_table;
-VACUUM ANALYZE observatory.obs_column;
-VACUUM ANALYZE observatory.obs_column_to_column;
-VACUUM ANALYZE observatory.obs_column_tag;
-VACUUM ANALYZE observatory.obs_tag;
+    for f in FIXTURES:
+        column_id, boundary_id, timespan = f
+        tablename_query = get_tablename_query(column_id, boundary_id, timespan)
+        resp = query(tablename_query).fetchone()
+        if resp:
+            tablename, colname, table_id = resp
+        else:
+            print("Could not find table for {}, {}, {}".format(
+                column_id, boundary_id, timespan))
+            continue
+        table_colname = (tablename, colname, boundary_id, table_id, )
+        if table_colname not in unique_tables:
+            print(table_colname)
+            unique_tables.add(table_colname)

-CREATE TABLE observatory.obs_meta AS
-SELECT numer_c.id numer_id,
-       denom_c.id denom_id,
-       geom_c.id geom_id,
-       MAX(numer_c.name) numer_name,
-       MAX(denom_c.name) denom_name,
-       MAX(geom_c.name) geom_name,
-       MAX(numer_c.description) numer_description,
-       MAX(denom_c.description) denom_description,
-       MAX(geom_c.description) geom_description,
-       MAX(numer_c.aggregate) numer_aggregate,
-       MAX(denom_c.aggregate) denom_aggregate,
-       MAX(geom_c.aggregate) geom_aggregate,
-       MAX(numer_c.type) numer_type,
-       MAX(denom_c.type) denom_type,
-       MAX(geom_c.type) geom_type,
-       MAX(numer_data_ct.colname) numer_colname,
-       MAX(denom_data_ct.colname) denom_colname,
-       MAX(geom_geom_ct.colname) geom_colname,
-       MAX(numer_geomref_ct.colname) numer_geomref_colname,
-       MAX(denom_geomref_ct.colname) denom_geomref_colname,
-       MAX(geom_geomref_ct.colname) geom_geomref_colname,
-       MAX(numer_t.tablename) numer_tablename,
-       MAX(denom_t.tablename) denom_tablename,
-       MAX(geom_t.tablename) geom_tablename,
-       MAX(numer_t.timespan) numer_timespan,
-       MAX(denom_t.timespan) denom_timespan,
-       MAX(numer_c.weight) numer_weight,
-       MAX(denom_c.weight) denom_weight,
-       MAX(geom_c.weight) geom_weight,
-       MAX(geom_t.timespan) geom_timespan,
-       MAX(geom_t.the_geom_webmercator)::geometry AS the_geom_webmercator,
-       ARRAY_AGG(DISTINCT s_tag.id) section_tags,
-       ARRAY_AGG(DISTINCT ss_tag.id) subsection_tags,
-       ARRAY_AGG(DISTINCT unit_tag.id) unit_tags
-FROM observatory.obs_column_table numer_data_ct,
-     observatory.obs_table numer_t,
-     observatory.obs_column_table numer_geomref_ct,
-     observatory.obs_column geomref_c,
-     observatory.obs_column_to_column geomref_c2c,
-     observatory.obs_column geom_c,
-     observatory.obs_column_table geom_geom_ct,
-     observatory.obs_column_table geom_geomref_ct,
-     observatory.obs_table geom_t,
-     observatory.obs_column_tag ss_ctag,
-     observatory.obs_tag ss_tag,
-     observatory.obs_column_tag s_ctag,
-     observatory.obs_tag s_tag,
-     observatory.obs_column_tag unit_ctag,
-     observatory.obs_tag unit_tag,
-     observatory.obs_column numer_c
-  LEFT JOIN (
-    observatory.obs_column_to_column denom_c2c
-    JOIN observatory.obs_column denom_c ON denom_c2c.target_id = denom_c.id
-    JOIN observatory.obs_column_table denom_data_ct ON denom_data_ct.column_id = denom_c.id
-    JOIN observatory.obs_table denom_t ON denom_data_ct.table_id = denom_t.id
-    JOIN observatory.obs_column_table denom_geomref_ct ON denom_geomref_ct.table_id = denom_t.id
-  ) ON denom_c2c.source_id = numer_c.id
-WHERE numer_c.id = numer_data_ct.column_id
-  AND numer_data_ct.table_id = numer_t.id
-  AND numer_t.id = numer_geomref_ct.table_id
-  AND numer_geomref_ct.column_id = geomref_c.id
-  AND geomref_c2c.reltype = 'geom_ref'
-  AND geomref_c.id = geomref_c2c.source_id
-  AND geom_c.id = geomref_c2c.target_id
-  AND geom_geomref_ct.column_id = geomref_c.id
-  AND geom_geomref_ct.table_id = geom_t.id
-  AND geom_geom_ct.column_id = geom_c.id
-  AND geom_geom_ct.table_id = geom_t.id
-  AND geom_c.type ILIKE 'geometry'
-  AND numer_c.type NOT ILIKE 'geometry'
-  AND numer_t.id != geom_t.id
-  AND numer_c.id != geomref_c.id
-  AND unit_tag.type = 'unit'
-  AND ss_tag.type = 'subsection'
-  AND s_tag.type = 'section'
-  AND unit_ctag.column_id = numer_c.id
-  AND unit_ctag.tag_id = unit_tag.id
-  AND ss_ctag.column_id = numer_c.id
-  AND ss_ctag.tag_id = ss_tag.id
-  AND s_ctag.column_id = numer_c.id
-  AND s_ctag.tag_id = s_tag.id
-  AND (denom_c2c.reltype = 'denominator' OR denom_c2c.reltype IS NULL)
-  AND (denom_geomref_ct.column_id = geomref_c.id OR denom_geomref_ct.column_id IS NULL)
-  AND (denom_t.timespan = numer_t.timespan OR denom_t.timespan IS NULL)
-GROUP BY numer_c.id, denom_c.id, geom_c.id,
-         numer_t.id, denom_t.id, geom_t.id;
-        ''')
+    print unique_tables

-        dropfiles.write('''
-DROP TABLE IF EXISTS observatory.obs_meta;
-                        ''')
+    with open(OUTFILE_PATH, 'w') as outfile:
+        outfile.write('SET client_min_messages TO WARNING;\n\\set ECHO none\n')
+        outfile.write('CREATE SCHEMA IF NOT EXISTS observatory;\n\n')
+
+    with open(DROPFILE_PATH, 'w') as dropfile:
+        dropfile.write('SET client_min_messages TO WARNING;\n\\set ECHO none\n')
+
+    for tablename in METADATA_TABLES:
+        print(tablename)
+        if tablename == 'obs_meta':
+            where = "WHERE " + " OR ".join([
+                "(numer_id, geom_id, numer_timespan) = ('{}', '{}', '{}')".format(
+                    numer_id, geom_id, timespan)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_numer':
+            where = "WHERE " + " OR ".join([
+                "numer_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_denom':
+            where = "WHERE " + " OR ".join([
+                "denom_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_geom':
+            where = "WHERE " + " OR ".join([
+                "geom_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_meta_timespan':
+            where = "WHERE " + " OR ".join([
+                "timespan_id = ('{}')".format(timespan)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_column':
+            where = "WHERE " + " OR ".join([
+                "id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_column_tag':
+            where = "WHERE " + " OR ".join([
+                "column_id IN ('{}', '{}')".format(numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename in ('obs_column_table', 'obs_column_table_tile', ):
+            where = 'WHERE column_id IN ({numer_ids}) ' \
+                    'OR column_id IN ({geom_ids}) ' \
+                    'OR table_id IN ({table_ids}) '.format(
+                        numer_ids=','.join(["'{}'".format(x) for x, _, _ in FIXTURES]),
+                        geom_ids=','.join(["'{}'".format(x) for _, x, _ in FIXTURES]),
+                        table_ids=','.join(["'{}'".format(x) for _, _, _, x in unique_tables])
+                    )
+        elif tablename == 'obs_column_to_column':
+            where = "WHERE " + " OR ".join([
+                "source_id IN ('{}', '{}') OR target_id IN ('{}', '{}')".format(
+                    numer_id, geom_id, numer_id, geom_id)
+                for numer_id, geom_id, timespan in FIXTURES
+            ])
+        elif tablename == 'obs_table':
+            where = 'WHERE timespan IN ({timespans}) ' \
+                    'OR id IN ({table_ids}) '.format(
+                        timespans=','.join(["'{}'".format(x) for _, _, x in FIXTURES]),
+                        table_ids=','.join(["'{}'".format(x) for _, _, _, x in unique_tables])
+                    )
+        else:
+            where = ''
+        dump('*', tablename, where)
+
+    for tablename, colname, boundary_id, table_id in unique_tables:
+        if 'zcta5' in boundary_id:
+            where = '\'11%\''
+            compare = 'LIKE'
+        elif 'whosonfirst' in boundary_id:
+            where = '(\'85632785\',\'85633051\',\'85633111\',\'85633147\',\'85633253\',\'85633267\')'
+            compare = 'IN'
+        else:
+            where = '\'36047%\''
+            compare = 'LIKE'
+        print ' '.join(['*', tablename, "WHERE {}::text {} {}".format(colname, compare, where)])
+        dump('*', tablename, "WHERE {}::text {} {}".format(colname, compare, where))
+
+if __name__ == '__main__':
+    main()
--- a/src/pg/Makefile
+++ b/src/pg/Makefile
@@ -24,7 +24,7 @@ $(DATA): $(SOURCES_DATA)
 	$(SED) $(REPLACEMENTS) $(SOURCES_DATA_DIR)/*.sql > $@

 TEST_DIR = test
-REGRESS = $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql)))
+REGRESS = $(sort $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql))))
 REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)'

 PG_CONFIG = pg_config
--- a/src/pg/observatory.control
+++ b/src/pg/observatory.control
@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.2'
-requires = 'postgis'
+default_version = '1.1.5'
+requires = 'postgis, postgres_fdw'
 superuser = true
 schema = cdb_observatory
--- a/src/pg/sql/15_fdw_utilities.sql
+++ b/src/pg/sql/15_fdw_utilities.sql
@@ -8,7 +8,7 @@ DECLARE
 BEGIN
  -- Build connection string
  connection_str := '{"server":{"extensions":"postgis", "dbname":"'
-    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"5432"}, "users":{"public"'
+    || user_dbname ||'", "host":"' || user_hostname ||'", "port":"6432"}, "users":{"public"'
    || ':{"user":"' || username ||'", "password":""} } }';

  -- This function tries to be as idempotent as possible, by not creating anything more than once
--- a/src/pg/sql/40_observatory_utility.sql
+++ b/src/pg/sql/40_observatory_utility.sql
--- a/src/pg/sql/41_observatory_augmentation.sql
+++ b/src/pg/sql/41_observatory_augmentation.sql
@@ -179,7 +179,10 @@ BEGIN
    --raise notice 'Cannot find data table for boundary ID %, column_ids %, and time_span %', geometry_level, column_ids, time_span;
  END IF;

-  IF ST_GeometryType(geom) = 'ST_Point'
+  IF geom IS NULL
+  THEN
+    results := NULL;
+  ELSIF ST_GeometryType(geom) = 'ST_Point'
  THEN
    --raise notice 'geom_table_name %, data_table_info %', geom_table_name, data_table_info::json[];
    results := cdb_observatory._OBS_GetPoints(geom,
@@ -357,37 +360,63 @@ DECLARE
  geom_colname TEXT;
  geom_geomref_colname TEXT;
  geom_tablename TEXT;
+  geom_id TEXT;
  result NUMERIC;
  sql TEXT;
  numer_name TEXT;
 BEGIN
+  IF geom IS NULL THEN
+    RETURN NULL;
+  END IF;

-  EXECUTE
-     $query$
-     SELECT numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
-            denom_colname, denom_geomref_colname, denom_tablename,
-            geom_colname, geom_geomref_colname, geom_tablename, numer_name
-             FROM observatory.obs_meta
-             WHERE (geom_id = $1 OR ($1 = ''))
-               AND numer_id = $2
-               AND (numer_timespan = $3 OR ($3 = ''))
-             ORDER BY geom_weight DESC, numer_timespan DESC
-             LIMIT 1
-     $query$
-    INTO numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
-         denom_colname, denom_geomref_colname, denom_tablename,
-         geom_colname, geom_geomref_colname, geom_tablename, numer_name
-    USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, '');
+  geom := ST_SnapToGrid(geom, 0.000001);

  IF ST_GeometryType(geom) = 'ST_Point' THEN
    geom_type := 'point';
  ELSIF ST_GeometryType(geom) IN ('ST_Polygon', 'ST_MultiPolygon') THEN
    geom_type := 'polygon';
+    geom := ST_Buffer(geom, 0.000001);
  ELSE
    RAISE EXCEPTION 'Invalid geometry type (%), can only handle ''ST_Point'', ''ST_Polygon'', and ''ST_MultiPolygon''',
                    ST_GeometryType(geom);
  END IF;

+
+  EXECUTE
+     $query$
+      WITH meta AS (SELECT numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
+                    denom_colname, denom_geomref_colname, denom_tablename,
+                    geom_colname, geom_geomref_colname, geom_tablename,
+                    numer_name, geom_id
+                     FROM observatory.obs_meta
+                     WHERE (geom_id = $1 OR ($1 = ''))
+                       AND numer_id = $2
+                       AND (numer_timespan = $3 OR ($3 = ''))),
+           scores AS (SELECT *
+                      FROM cdb_observatory._OBS_GetGeometryScores($4,
+                           (SELECT Array_Agg(geom_id) FROM meta), 500))
+     SELECT meta.*
+     FROM meta, scores
+     WHERE meta.geom_id = scores.geom_id
+     ORDER BY score DESC
+     LIMIT 1
+    $query$
+    INTO numer_aggregate, numer_colname, numer_geomref_colname, numer_tablename,
+         denom_colname, denom_geomref_colname, denom_tablename,
+         geom_colname, geom_geomref_colname, geom_tablename, numer_name, geom_id
+    USING COALESCE(boundary_id, ''), measure_id, COALESCE(time_span, ''),
+      CASE WHEN ST_GeometryType(geom) = 'ST_Point' THEN
+                st_buffer(geom::geography, 10)::geometry(geometry, 4326)
+           ELSE geom
+      END;
+
+  IF geom_id IS NULL THEN
+    RAISE NOTICE 'No boundary found for geom';
+    RETURN NULL;
+  ELSE
+    RAISE NOTICE 'Using boundary %', geom_id;
+  END IF;
+
  IF normalize ILIKE 'area' AND numer_aggregate ILIKE 'sum' THEN
    map_type := 'areaNormalized';
  ELSIF normalize ILIKE 'denominator' THEN
@@ -439,7 +468,7 @@ BEGIN
                                   WHERE ST_Intersects(%L, geom.%I)
                                     AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
                    SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
-                           ST_Area(%L::Geography)
+                           (ST_Area(%L::Geography) / 1000000)
                    FROM observatory.%I numer
                    WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
                geom, geom_colname, geom_colname,
@@ -481,7 +510,7 @@ BEGIN
    ELSIF map_type = 'predenominated' THEN
      IF numer_aggregate NOT ILIKE 'sum' THEN
        RAISE EXCEPTION 'Cannot calculate "%" (%) for custom area as it cannot be summed, use ST_PointOnSurface instead',
-                        numer_name, numer_id;
+                        numer_name, measure_id;
      ELSE
        sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
                                          / ST_Area(geom.%I) overlap, geom.%I geom_ref
@@ -523,6 +552,9 @@ DECLARE
  measure_val NUMERIC;
  data_geoid_colname TEXT;
 BEGIN
+  IF geom_ref IS NULL THEN
+    RETURN NULL;
+  END IF;

  EXECUTE
     $query$
@@ -571,6 +603,9 @@ DECLARE
  category_val TEXT;
  category_share NUMERIC;
 BEGIN
+  IF geom IS NULL THEN
+    RETURN NULL;
+  END IF;

  EXECUTE
     $query$
--- a/src/pg/sql/42_observatory_exploration.sql
+++ b/src/pg/sql/42_observatory_exploration.sql
@@ -1,4 +1,4 @@
-- return a table that contains a string match based on input
+
 -- TODO: implement search for timespan

 CREATE OR REPLACE FUNCTION cdb_observatory._OBS_SearchTables(
@@ -120,3 +120,367 @@ BEGIN
  RETURN;
 END
 $$ LANGUAGE plpgsql;
+
+-- Functions the interface works from to identify available numerators,
+-- denominators, geometries, and timespans
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableNumerators(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  numer_id TEXT,
+  numer_name TEXT,
+  numer_description TEXT,
+  numer_weight NUMERIC,
+  numer_license TEXT,
+  numer_source TEXT,
+  numer_type TEXT,
+  numer_aggregate TEXT,
+  numer_extra JSONB,
+  numer_tags JSONB,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  denom_id := COALESCE(denom_id, '');
+  geom_id := COALESCE(geom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT numer_id::TEXT,
+           numer_name::TEXT,
+           numer_description::TEXT,
+           numer_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           numer_type numer_type,
+           numer_aggregate numer_aggregate,
+           numer_extra::JSONB numer_extra,
+           numer_tags numer_tags,
+    $1 = ANY(denoms) valid_denom,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_numer
+    WHERE %s (numer_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING denom_id, geom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableDenominators(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  denom_id TEXT,
+  denom_name TEXT,
+  denom_description TEXT,
+  denom_weight NUMERIC,
+  denom_license TEXT,
+  denom_source TEXT,
+  denom_type TEXT,
+  denom_aggregate TEXT,
+  denom_extra JSONB,
+  denom_tags JSONB,
+  valid_numer BOOLEAN,
+  valid_geom BOOLEAN,
+  valid_timespan BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  geom_id := COALESCE(geom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT denom_id::TEXT,
+           denom_name::TEXT,
+           denom_description::TEXT,
+           denom_weight::NUMERIC,
+           NULL::TEXT license,
+           NULL::TEXT source,
+           denom_type::TEXT,
+           denom_aggregate::TEXT,
+           denom_extra::JSONB,
+           denom_tags::JSONB,
+    $1 = ANY(numers) valid_numer,
+    $2 = ANY(geoms) valid_geom,
+    $3 = ANY(timespans) valid_timespan
+    FROM observatory.obs_meta_denom
+    WHERE %s (denom_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING numer_id, geom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableGeometries(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  timespan TEXT DEFAULT NULL
+) RETURNS TABLE (
+  geom_id TEXT,
+  geom_name TEXT,
+  geom_description TEXT,
+  geom_weight NUMERIC,
+  geom_aggregate TEXT,
+  geom_license TEXT,
+  geom_source TEXT,
+  valid_numer BOOLEAN,
+  valid_denom BOOLEAN,
+  valid_timespan BOOLEAN,
+  score NUMERIC,
+  numtiles BIGINT,
+  notnull_percent NUMERIC,
+  numgeoms NUMERIC,
+  percentfill NUMERIC,
+  estnumgeoms NUMERIC,
+  meanmediansize NUMERIC
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  denom_id := COALESCE(denom_id, '');
+  timespan := COALESCE(timespan, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    WITH available_geoms AS (
+      SELECT geom_id::TEXT,
+             geom_name::TEXT,
+             geom_description::TEXT,
+             geom_weight::NUMERIC,
+             NULL::TEXT geom_aggregate,
+             NULL::TEXT license,
+             NULL::TEXT source,
+      $1 = ANY(numers) valid_numer,
+      $2 = ANY(denoms) valid_denom,
+      $3 = ANY(timespans) valid_timespan
+      FROM observatory.obs_meta_geom
+      WHERE %s (geom_tags ?& $4 OR CARDINALITY($4) = 0)
+    ), scores AS (
+      SELECT * FROM cdb_observatory._OBS_GetGeometryScores($5,
+        (SELECT ARRAY_AGG(geom_id) FROM available_geoms)
+      )
+    ) SELECT available_geoms.*, score, numtiles, notnull_percent, numgeoms,
+             percentfill, estnumgeoms, meanmediansize
+      FROM available_geoms, scores
+      WHERE available_geoms.geom_id = scores.geom_id
+  $string$, geom_clause)
+  USING numer_id, denom_id, timespan, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetAvailableTimespans(
+  bounds GEOMETRY DEFAULT NULL,
+  filter_tags TEXT[] DEFAULT NULL,
+  numer_id TEXT DEFAULT NULL,
+  denom_id TEXT DEFAULT NULL,
+  geom_id TEXT DEFAULT NULL
+) RETURNS TABLE (
+  timespan_id TEXT,
+  timespan_name TEXT,
+  timespan_description TEXT,
+  timespan_weight NUMERIC,
+  timespan_aggregate TEXT,
+  timespan_license TEXT,
+  timespan_source TEXT,
+  valid_numer BOOLEAN,
+  valid_denom BOOLEAN,
+  valid_geom BOOLEAN
+) AS $$
+DECLARE
+  geom_clause TEXT;
+BEGIN
+  filter_tags := COALESCE(filter_tags, (ARRAY[])::TEXT[]);
+  numer_id := COALESCE(numer_id, '');
+  denom_id := COALESCE(denom_id, '');
+  geom_id := COALESCE(geom_id, '');
+  IF bounds IS NULL THEN
+    geom_clause := '';
+  ELSE
+    geom_clause := 'ST_Intersects(the_geom, $5) AND';
+  END IF;
+  RETURN QUERY
+  EXECUTE
+  format($string$
+    SELECT timespan_id::TEXT,
+           timespan_name::TEXT,
+           timespan_description::TEXT,
+           timespan_weight::NUMERIC,
+           NULL::TEXT timespan_aggregate,
+           NULL::TEXT license,
+           NULL::TEXT source,
+    $1 = ANY(numers) valid_numer,
+    $2 = ANY(denoms) valid_denom,
+    $3 = ANY(geoms) valid_geom_id
+    FROM observatory.obs_meta_timespan
+    WHERE %s (timespan_tags ?& $4 OR CARDINALITY($4) = 0)
+  $string$, geom_clause)
+  USING numer_id, denom_id, geom_id, filter_tags, bounds;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+
+-- Function below should replace SQL in
+-- https://github.com/CartoDB/cartodb/blob/ab465cb2918c917940e955963b0cd8a050c06600/lib/assets/javascripts/cartodb3/editor/layers/layer-content-views/analyses/data-observatory-metadata.js
+CREATE OR REPLACE FUNCTION cdb_observatory.OBS_LegacyBuilderMetadata(
+  aggregate_type TEXT DEFAULT NULL
+)
+RETURNS TABLE (
+  name TEXT,
+  subsection JSONB
+) AS $$
+DECLARE
+  aggregate_condition TEXT DEFAULT '';
+BEGIN
+  IF aggregate_type IS NOT NULL THEN
+    aggregate_condition := format(' AND numer_aggregate = %L ', aggregate_type);
+  END IF;
+  RETURN QUERY
+  EXECUTE format($string$
+    WITH expanded AS (
+        SELECT JSONB_Build_Object('id', numer_id, 'name', numer_name) "column",
+               SUBSTR((sections).key, 9) section_id, (sections).value section_name,
+               SUBSTR((subsections).key, 12) subsection_id, (subsections).value subsection_name
+        FROM (
+          SELECT numer_id, numer_name,
+                 jsonb_each_text(numer_tags) as sections,
+                 jsonb_each_text as subsections
+          FROM (SELECT numer_id, numer_name, numer_tags,
+                       jsonb_each_text(numer_tags)
+                FROM cdb_observatory.obs_getavailablenumerators()
+                WHERE numer_weight > 0 %s
+               ) foo
+        ) bar
+        WHERE (sections).key LIKE 'section/%%'
+          AND (subsections).key LIKE 'subsection/%%'
+      ), grouped_by_subsections AS (
+        SELECT JSONB_Agg(JSONB_Build_Object('f1', "column")) AS columns,
+               section_id, section_name, subsection_id, subsection_name
+        FROM expanded
+        GROUP BY section_id, section_name, subsection_id, subsection_name
+      )
+    SELECT section_name as name, JSONB_Agg(
+      JSONB_Build_Object(
+        'f1', JSONB_Build_Object(
+          'name', subsection_name,
+          'id', subsection_id,
+          'columns', columns
+        )
+      )
+    ) as subsection
+    FROM grouped_by_subsections
+    GROUP BY section_name
+  $string$, aggregate_condition);
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetGeometryScores(
+  bounds Geometry(Geometry, 4326) DEFAULT NULL,
+  filter_geom_ids TEXT[] DEFAULT NULL,
+  desired_num_geoms INTEGER DEFAULT 3000
+) RETURNS TABLE (
+  score NUMERIC,
+  numtiles BIGINT,
+  geom_id TEXT,
+  notnull_percent NUMERIC,
+  numgeoms NUMERIC,
+  percentfill NUMERIC,
+  estnumgeoms NUMERIC,
+  meanmediansize NUMERIC
+) AS $$
+BEGIN
+  filter_geom_ids := COALESCE(filter_geom_ids, (ARRAY[])::TEXT[]);
+  RETURN QUERY
+  EXECUTE format($string$
+    SELECT
+      (1 / (abs(numgeoms - $3)
+        --* (1 / Coalesce(NullIf(notnull_percent, 0), 1))
+        --* (1 / Coalesce(NullIf(percentfill, 0), 0.0001))
+      ))::Numeric
+      AS score, *
+    FROM (
+      WITH clipped_geom AS (
+        SELECT column_id, table_id
+          , CASE WHEN $1 IS NOT NULL THEN ST_Clip(tile, $1, True)
+                 ELSE tile END clipped_tile
+          , tile
+        FROM observatory.obs_column_table_tile
+        WHERE ($1 IS NULL OR ST_Intersects($1, tile))
+          AND (column_id = ANY($2) OR cardinality($2) = 0)
+      ), clipped_geom_countagg AS (
+        SELECT column_id, table_id
+          , ST_CountAgg(clipped_tile, 2, True)::Numeric notnull_pixels
+          , ST_CountAgg(clipped_tile, 2, False)::Numeric pixels
+        FROM clipped_geom
+        GROUP BY column_id, table_id
+      ) SELECT
+        count(*)::BIGINT, a.column_id
+        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
+                THEN cdb_observatory.FIRST(notnull_pixels) / cdb_observatory.FIRST(pixels)
+                ELSE 1
+          END)::Numeric AS notnull_percent
+        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
+                THEN (ST_SummaryStatsAgg(clipped_tile, 2, True)).sum
+                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 2, ST_PointOnSurface($1)), 0) * (ST_Area($1) / ST_Area(ST_PixelAsPolygon(cdb_observatory.FIRST(tile), 0, 0)) * cdb_observatory.FIRST(pixels))
+          END)::Numeric AS numgeoms
+        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
+                THEN (ST_SummaryStatsAgg(clipped_tile, 3, True)).mean
+                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 3, ST_PointOnSurface($1)), 0)
+          END)::Numeric AS percentfill
+        , ((ST_Area(ST_Transform($1, 3857)) / 1000000) / NullIf(
+            CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
+                 THEN (ST_SummaryStatsAgg(clipped_tile, 1, True)).mean
+                 ELSE Coalesce(ST_Value(cdb_observatory.FIRST(tile), 1, ST_PointOnSurface($1)), 0)
+            END, 0))::Numeric AS estnumgeoms
+        , (CASE WHEN cdb_observatory.FIRST(notnull_pixels) > 0
+                THEN (ST_SummaryStatsAgg(clipped_tile, 1, True)).mean
+                ELSE COALESCE(ST_Value(cdb_observatory.FIRST(tile), 1, ST_PointOnSurface($1)), 0)
+          END)::Numeric AS meanmediansize
+      FROM clipped_geom_countagg a, clipped_geom b
+      WHERE a.table_id = b.table_id
+        AND a.column_id = b.column_id
+      GROUP BY a.column_id, a.table_id
+      ORDER BY a.column_id, a.table_id
+    ) foo
+  $string$) USING bounds, filter_geom_ids, desired_num_geoms;
+  RETURN;
+END
+$$ LANGUAGE plpgsql;
--- a/src/pg/sql/44_observatory_geometries.sql
+++ b/src/pg/sql/44_observatory_geometries.sql
@@ -244,7 +244,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetBoundariesByGeometry(
  geom geometry(Geometry, 4326),
  boundary_id text,
  time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -253,7 +253,7 @@ DECLARE
  geoid_colname text;
  target_table text;
 BEGIN
-
+  overlap_type := COALESCE(overlap_type, 'intersects');
  -- check inputs
  IF lower(overlap_type) NOT IN ('contains', 'intersects', 'within')
  THEN
@@ -318,7 +318,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByGeometry(
  geom geometry(Geometry, 4326),
  boundary_id text,
  time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -364,7 +364,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByPointAndRadius(
  radius numeric, -- radius in meters
  boundary_id text,
  time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -382,7 +382,8 @@ BEGIN
               FROM cdb_observatory._OBS_GetBoundariesByGeometry(
                        circle_boundary,
                        boundary_id,
-                        time_span);
+                        time_span,
+                        overlap_type);
  RETURN;
 END;
 $$ LANGUAGE plpgsql;
@@ -394,7 +395,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetPointsByGeometry(
  geom geometry(Geometry, 4326),
  boundary_id text,
  time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -403,6 +404,7 @@ DECLARE
  geoid_colname text;
  target_table text;
 BEGIN
+  overlap_type := COALESCE(overlap_type, 'intersects');

  IF lower(overlap_type) NOT IN ('contains', 'within', 'intersects')
  THEN
@@ -464,7 +466,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByGeometry(
  geom geometry(Geometry, 4326),
  boundary_id text,
  time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -509,7 +511,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByPointAndRadius(
  radius numeric, -- radius in meters
  boundary_id text,
  time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
--- a/src/pg/sql/50_table_level_framework.sql
+++ b/src/pg/sql/50_table_level_framework.sql
@@ -24,9 +24,12 @@ BEGIN
 EXCEPTION
  WHEN others THEN
    -- Disconnect user imported table. Delete schema and FDW server.
-    EXECUTE 'DROP FOREIGN TABLE IF EXISTS ' || fdw_import_schema || '.' || table_name;
-    EXECUTE 'DROP SCHEMA IF EXISTS ' || fdw_import_schema || ' CASCADE';
-    EXECUTE 'DROP SERVER IF EXISTS ' || fdw_server || ' CASCADE;';
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || fdw_import_schema || '".' || table_name;
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || fdw_import_schema || '".cdb_tablemetadata';
+    EXECUTE 'DROP SCHEMA IF EXISTS "' || fdw_import_schema || '"';
+    EXECUTE 'DROP USER MAPPING IF EXISTS FOR public SERVER "' || fdw_server || '"';
+    EXECUTE 'DROP SERVER IF EXISTS "' || fdw_server || '"';
+
    RETURN (null, null, null);
 END;
 $$ LANGUAGE plpgsql SECURITY DEFINER;
@@ -37,27 +40,9 @@ AS $$
 DECLARE
  colnames text[];
  coltypes text[];
-  requested_measures text[];
-  measure text;
 BEGIN
-
-  -- Simple mock, there should be real logic in here.
-
-  IF $3 NOT ILIKE 'GetMeasure' OR $3 IS NULL THEN
-    RAISE 'This function is not supported yet: %', $3;
-  END IF;
-
-  SELECT translate($4::json->>'tag_name','[]', '{}')::text[] INTO requested_measures;
-
-  FOREACH measure IN ARRAY requested_measures
-  LOOP
-    IF NOT measure ILIKE ANY (Array['total_pop', 'pop_16_over']::text[]) THEN
-      RAISE 'This measure is not supported yet: %', measure;
-    END IF;
-  SELECT array_append(colnames, measure) INTO colnames;
-  SELECT array_append(coltypes, 'double precision'::text) INTO coltypes;
-
-  END LOOP;
+  EXECUTE FORMAT('SELECT r.colnames::text[], r.coltypes::text[] FROM cdb_observatory._%sResultMetadata(%L::json) r', function_name, params::text)
+  INTO colnames, coltypes;

  RETURN (colnames::text[], coltypes::text[]);
 END;
@@ -68,41 +53,17 @@ RETURNS SETOF record
 AS $$
 DECLARE
  data_query text;
-  tag_name text[];
-  tag text;
-  tags_list text;
-  tags_query text;
  rec RECORD;
 BEGIN
-    SELECT translate($6::json->>'tag_name','[]', '{}')::text[] INTO tag_name;
-    SELECT array_to_string(tag_name, ',') INTO tags_list;
-    tags_query := '';
-
-    FOREACH tag IN ARRAY tag_name
-    LOOP
-      SELECT tags_query || ' sum(' || tag || '/fraction)::double precision as ' || tag || ', ' INTO tags_query;
-
-    END LOOP;
-
-    -- Simple mock, there should be real logic in here.
-    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
-        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
-        || 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308 as a, '
-        || table_schema || '.' || table_name || ' AS b '
-        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
-        || tags_list
-        || ' FROM observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd ) '
-        || 'SELECT '
-        || tags_query
-        || ' cartodb_id::int FROM _areas, values '
-        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';

+    EXECUTE FORMAT('SELECT cdb_observatory._%sQuery(%L, %L, %L::json)', function_name, table_schema, table_name, params::text)
+    INTO data_query;

    FOR rec IN EXECUTE data_query
    LOOP
        RETURN NEXT rec;
    END LOOP;
-    RETURN;
+  RETURN;
 END;
 $$ LANGUAGE plpgsql SECURITY DEFINER;

@@ -112,8 +73,10 @@ RETURNS boolean
 AS $$
 BEGIN
    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".' || table_name;
-    EXECUTE 'DROP SCHEMA IF EXISTS ' || table_schema || ' CASCADE';
-    EXECUTE 'DROP SERVER IF EXISTS ' || servername || ' CASCADE;';
+    EXECUTE 'DROP FOREIGN TABLE IF EXISTS "' || table_schema || '".cdb_tablemetadata';
+    EXECUTE 'DROP SCHEMA IF EXISTS "' || table_schema || '"';
+    EXECUTE 'DROP USER MAPPING IF EXISTS FOR public SERVER "' || servername || '"';
+    EXECUTE 'DROP SERVER IF EXISTS "' || servername || '"';
    RETURN true;
 END;
 $$ LANGUAGE plpgsql SECURITY DEFINER;
--- a/src/pg/sql/51_table_level_functions.sql
+++ b/src/pg/sql/51_table_level_functions.sql
@@ -0,0 +1,79 @@
+--
+--
+-- OBS_GetMeasure
+--
+--
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetMeasureResultMetadata(params json)
+RETURNS cdb_observatory.ds_return_metadata
+AS $$
+DECLARE
+  colnames text[]; -- Array to store the name of the measures to be returned
+  coltypes text[]; -- Array to store the type of the measures to be returned
+  requested_measures text[];
+  measure_id text;
+BEGIN
+  -- By definition, all the measure results for the OBS_GetMeasure API are numeric values
+  SELECT ARRAY(SELECT json_array_elements_text(params->'measure_id'))::text[] INTO requested_measures;
+
+  FOREACH measure_id IN ARRAY requested_measures
+  LOOP
+    SELECT array_append(colnames, measure_id) INTO colnames;
+    SELECT array_append(coltypes, 'numeric'::text) INTO coltypes;
+  END LOOP;
+
+  RETURN (colnames::text[], coltypes::text[]);
+END;
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetMeasureQuery(table_schema text, table_name text, params json)
+RETURNS text
+AS $$
+DECLARE
+  data_query text;
+  measure_ids_arr text[];
+  measure_id text;
+  measures_list text;
+  measures_query text;
+  normalize text;
+  boundary_id text;
+  time_span text;
+  geom_table_name text;
+  data_table_name text;
+BEGIN
+    measures_query := '';
+    -- SELECT table_name from obs_meta WHERE boundary_id = {bound} AND [...] INTO geom_table_name
+    geom_table_name := 'observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308';
+    -- SELECT table_name from obs_meta WHERE time_span = {time} AND [...] INTO data_table_name
+    data_table_name := 'observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd';
+
+    -- Get measure_ids array from JSON
+    SELECT ARRAY(SELECT json_array_elements_text(params->'measure_id'))::text[] INTO measure_ids_arr;
+
+    -- Get a comma-separated list of measures ("total_pop, over_16_pop") to be used in SELECTs
+    SELECT array_to_string(measure_ids_arr, ',') INTO measures_list;
+
+    FOREACH measure_id IN ARRAY measure_ids_arr
+    LOOP
+      -- Build query to compute each value and normalize
+      -- Assumes the default normalization method, the normalize parameter given in the JSON
+      -- should be checked in order to build the final query
+      SELECT measures_query || ' sum(' || measure_id || '/fraction)::numeric as ' || measure_id || ', ' INTO measures_query;
+    END LOOP;
+
+    -- Data query should select the measures and the cartodb_id of the user table, in that order.
+    data_query := '(WITH _areas AS(SELECT ST_Area(a.the_geom::geography)'
+        || '/ (1000 * 1000) as fraction, a.geoid, b.cartodb_id FROM '
+        || geom_table_name || ' as a, '
+        || table_schema || '.' || table_name || ' AS b '
+        || 'WHERE b.the_geom && a.the_geom ), values AS (SELECT geoid, '
+        || measures_list
+        || ' FROM ' || data_table_name || ' ) '
+        || 'SELECT '
+        || measures_query
+        || ' cartodb_id::int FROM _areas, values '
+        || 'WHERE values.geoid = _areas.geoid GROUP BY cartodb_id);';
+    RETURN data_query;
+END;
+$$ LANGUAGE plpgsql;
+
--- a/src/pg/test/expected/01_install_test.out
+++ b/src/pg/test/expected/01_install_test.out
@@ -1,6 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION plpythonu;
-CREATE EXTENSION cartodb;
+CREATE EXTENSION postgres_fdw;
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';
--- a/src/pg/test/expected/41_observatory_augmentation_test.out
+++ b/src/pg/test/expected/41_observatory_augmentation_test.out
@@ -42,39 +42,78 @@ t
 obs_getmeasure_total_pop_point_test
 t
 (1 row)
+obs_getmeasure_total_pop_point_null_normalization_test
+t
+(1 row)
+obs_getmeasure_total_pop_point_area_test
+t
+(1 row)
 obs_getmeasure_total_pop_polygon_test
 t
 (1 row)
+obs_getmeasure_total_pop_polygon_null_normalization_test
+t
+(1 row)
+obs_getmeasure_total_pop_polygon_area_test
+t
+(1 row)
 obs_getmeasure_total_male_point_denominator
 t
 (1 row)
 obs_getmeasure_total_male_poly_denominator
 t
 (1 row)
+obs_getmeasure_bad_geometry
+t
+(1 row)
+obs_getmeasure_null_geometry
+t
+(1 row)
+obs_getmeasure_out_of_bounds_geometry
+t
+(1 row)
 obs_getcategory_point
 t
 (1 row)
 obs_getcategory_polygon
 t
 (1 row)
+obs_getcategory_null
+t
+(1 row)
 obs_getpopulation
 t
 (1 row)
 obs_getpopulation_polygon_test
 t
 (1 row)
+obs_getpopulation_polygon_null_test
+t
+(1 row)
+obs_getpopulation_polygon_null_geom_test
+t
+(1 row)
 obs_getuscensusmeasure_point_male_pop
 t
 (1 row)
 obs_getuscensusmeasure
 t
 (1 row)
+obs_getuscensusmeasure_null
+t
+(1 row)
+obs_getuscensusmeasure_null_geom
+t
+(1 row)
 obs_getuscensuscategory_point
 t
 (1 row)
 obs_getuscensuscategory_polygon
 t
 (1 row)
+obs_getuscensuscategory_null
+t
+(1 row)
 obs_getmeasurebyid_cartodb_census_tract
 t
 (1 row)
@@ -87,3 +126,6 @@ t
 obs_getmeasurebyid_nulls
 t
 (1 row)
+obs_getmeasurebyid_null_id
+t
+(1 row)
--- a/src/pg/test/expected/42_observatory_exploration_test.out
+++ b/src/pg/test/expected/42_observatory_exploration_test.out
@@ -12,3 +12,192 @@ t
 _obs_getavailableboundariesexist
 t
 (1 row)
+_obs_getavailablenumerators_usa_pop_in_all
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getavailablenumerators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getavailablenumerators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getavailablenumerators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getavailablenumerators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getavailablenumerators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getavailablenumerators_zillow_at_zcta5
+t
+(1 row)
+_obs_getavailablenumerators_no_zillow_at_block_group
+t
+(1 row)
+_obs_getavailablenumerators_total_pop_2010_2014
+t
+(1 row)
+_obs_getavailablenumerators_no_total_pop_1996
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_all
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_nyc_point
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_usa_extents
+t
+(1 row)
+_obs_getavailabledenominators_no_usa_pop_not_in_zero_point
+t
+(1 row)
+_obs_getavailabledenominators_usa_pop_in_age_gender_subsection
+t
+(1 row)
+_obs_getavailabledenominators_no_pop_in_income_subsection
+t
+(1 row)
+_obs_getavailabledenominators_male_pop_denom_by_total_pop
+t
+(1 row)
+_obs_getavailabledenominators_no_income_denom_by_total_pop
+t
+(1 row)
+_obs_getavailabledenominators_at_zcta5
+t
+(1 row)
+_obs_getavailabledenominators_none_spanish_geom
+t
+(1 row)
+_obs_getavailabledenominators_total_pop_2010_2014
+t
+(1 row)
+_obs_getavailabledenominators_no_total_pop_1996
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_all
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_nyc_point
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_usa_extents
+t
+(1 row)
+_obs_getavailablegeometries_no_usa_bg_not_in_zero_point
+t
+(1 row)
+_obs_getavailablegeometries_usa_bg_in_boundary_subsection
+t
+(1 row)
+_obs_getavailablegeometries_no_bg_in_uk_section
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_not_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_denom_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg
+t
+(1 row)
+_obs_getavailablegeometries_bg_2014
+t
+(1 row)
+_obs_getavailablegeometries_bg_not_1996
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_all
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_nyc_point
+t
+(1 row)
+_obs_getavailabletimespans_2010_2014_in_usa_extents
+t
+(1 row)
+_obs_getavailabletimespans_no_usa_bg_not_in_zero_point
+t
+(1 row)
+_obs_getavailabletimespans_total_pop_in_2010_2014
+t
+(1 row)
+_obs_getavailabletimespans_foobarbaz_not_in_2010_2014
+t
+(1 row)
+_obs_getavailablegeometries_total_pop_denom_in_2010_2014
+t
+(1 row)
+_obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014
+t
+(1 row)
+_obs_geometryscores_500m_buffer
+t
+(1 row)
+_obs_geometryscores_5km_buffer
+t
+(1 row)
+_obs_geometryscores_50km_buffer
+t
+(1 row)
+_obs_geometryscores_500km_buffer
+t
+(1 row)
+_obs_geometryscores_2500km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_500m_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_5km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_50km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_500km_buffer
+t
+(1 row)
+_obs_geometryscores_numgeoms_2500km_buffer
+t
+(1 row)
+_obs_geometryscores_500km_buffer_50_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_500_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_2500_geoms
+t
+(1 row)
+_obs_geometryscores_500km_buffer_25000_geoms
+t
+(1 row)
+_total_pop_in_legacy_builder_metadata
+t
+(1 row)
+_median_income_in_legacy_builder_metadata
+t
+(1 row)
+_total_pop_in_legacy_builder_metadata_sums
+t
+(1 row)
+_median_income_not_in_legacy_builder_metadata_sums
+t
+(1 row)
+_no_dupe_subsections_in_legacy_builder_metadata
+t
+(1 row)
--- a/src/pg/test/fixtures/drop_fixtures.sql
+++ b/src/pg/test/fixtures/drop_fixtures.sql
@@ -7,18 +7,23 @@ DROP TABLE IF EXISTS observatory.obs_column_tag;
 DROP TABLE IF EXISTS observatory.obs_tag;
 DROP TABLE IF EXISTS observatory.obs_column_to_column;
 DROP TABLE IF EXISTS observatory.obs_dump_version;
-DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
-DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
-DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
-DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
-DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
-DROP TABLE IF EXISTS observatory.obs_8764a6b439a4f8714f54d4b3a157bc5e36519066;
-DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
-DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
-DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
-DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
-DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
-DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
-DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
-
 DROP TABLE IF EXISTS observatory.obs_meta;
+DROP TABLE IF EXISTS observatory.obs_meta_numer;
+DROP TABLE IF EXISTS observatory.obs_meta_denom;
+DROP TABLE IF EXISTS observatory.obs_meta_geom;
+DROP TABLE IF EXISTS observatory.obs_meta_timespan;
+DROP TABLE IF EXISTS observatory.obs_column_table_tile;
+DROP TABLE IF EXISTS observatory.obs_fcd4e4f5610f6764973ef8c0c215b2e80bec8963;
+DROP TABLE IF EXISTS observatory.obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308;
+DROP TABLE IF EXISTS observatory.obs_6c1309a64d8f3e6986061f4d1ca7b57743e75e74;
+DROP TABLE IF EXISTS observatory.obs_7615e8622a68bfc5fe37c69c9880edfb40250103;
+DROP TABLE IF EXISTS observatory.obs_d39f7fe5959891c8296490d83c22ded31c54af13;
+DROP TABLE IF EXISTS observatory.obs_1babf5a26a1ecda5fb74963e88408f71d0364b81;
+DROP TABLE IF EXISTS observatory.obs_65f29658e096ca1485bf683f65fdbc9f05ec3c5d;
+DROP TABLE IF EXISTS observatory.obs_144e8b4f906885b2e057ac4842644a553ae49c6e;
+DROP TABLE IF EXISTS observatory.obs_fc050f0b8673cfe3c6aa1040f749eb40975691b7;
+DROP TABLE IF EXISTS observatory.obs_1a098da56badf5f32e336002b0a81708c40d29cd;
+DROP TABLE IF EXISTS observatory.obs_1ea93bbc109c87c676b3270789dacf7a1430db6c;
+DROP TABLE IF EXISTS observatory.obs_b393b5b88c6adda634b2071a8005b03c551b609a;
+DROP TABLE IF EXISTS observatory.obs_1746e37b7cd28cb131971ea4187d42d71f09c5f3;
+DROP TABLE IF EXISTS observatory.obs_a01cd5d8ccaa6531cef715071e9307e6b1987ec3;
--- a/src/pg/test/fixtures/load_fixtures.sql
+++ b/src/pg/test/fixtures/load_fixtures.sql
--- a/src/pg/test/sql/01_install_test.sql
+++ b/src/pg/test/sql/01_install_test.sql
@@ -1,7 +1,6 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION plpythonu;
-CREATE EXTENSION cartodb;
+CREATE EXTENSION postgres_fdw;

 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';
--- a/src/pg/test/sql/41_observatory_augmentation_test.sql
+++ b/src/pg/test/sql/41_observatory_augmentation_test.sql
@@ -67,7 +67,7 @@ FROM cte;
 SELECT
  (cdb_observatory._OBS_GetPoints(
    ST_SetSRID(ST_Point(0, 0), 4326),
-    'obs_1a098da56badf5f32e336002b0a81708c40d29cd'::text, -- see example in obs_geomtable
+    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
    (Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
  ))[1]::text  is null
  as OBS_GetPoints_for_null_island;
@@ -89,7 +89,7 @@ SELECT
 SELECT
  ((cdb_observatory._OBS_GetPolygons(
    ST_Buffer(ST_SetSRID(ST_Point(0, 0), 4326)::geography, 500)::geometry,
-    'obs_1a098da56badf5f32e336002b0a81708c40d29cd'::text, -- see example in obs_geomtable
+    'obs_c6fb99c47d61289fbb8e561ff7773799d3fcc308'::text, -- see example in obs_geomtable
    Array['{"colname":"total_pop","tablename":"obs_1a098da56badf5f32e336002b0a81708c40d29cd","aggregate":"sum","name":"Total Population","type":"Numeric","description":"The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates."}'::json])
  )[1]->>'value') is null
  as OBS_GetPolygons_for_null_island;
@@ -129,15 +129,15 @@ WITH result as (
  from result;

 -- Point-based OBS_GetMeasure with zillow
-SELECT abs(OBS_GetMeasure_zhvi_point - 583600) / 583600 < 0.001 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
+SELECT abs(OBS_GetMeasure_zhvi_point - 597900) / 597900 < 5.0 AS OBS_GetMeasure_zhvi_point_test FROM cdb_observatory.OBS_GetMeasure(
  ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
  'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2014-01'
 ) As t(OBS_GetMeasure_zhvi_point);

-- Point-based OBS_GetMeasure with zillow default to latest
-SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 972900) / 972900 < 0.001 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
+-- Point-based OBS_GetMeasure with later measure
+SELECT abs(OBS_GetMeasure_zhvi_point_default_latest - 995400) / 995400 < 5.0 AS OBS_GetMeasure_zhvi_point_default_latest_test FROM cdb_observatory.OBS_GetMeasure(
  ST_SetSRID(ST_Point(-73.94602417945862, 40.6768220087458), 4326),
-  'us.zillow.AllHomes_Zhvi'
+  'us.zillow.AllHomes_Zhvi', null, 'us.census.tiger.zcta5', '2016-06'
 ) As t(OBS_GetMeasure_zhvi_point_default_latest);

 -- Point-based OBS_GetMeasure, default normalization (area)
@@ -148,6 +148,22 @@ SELECT abs(OBS_GetMeasure_total_pop_point - 10923.093200390833950) / 10923.09320
    'us.census.acs.B01003001'
 ) As t(OBS_GetMeasure_total_pop_point);

+-- Point-based OBS_GetMeasure, default normalization by NULL (area)
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_point_null_normalization - 10923.093200390833950) / 10923.093200390833950 < 0.001 As OBS_GetMeasure_total_pop_point_null_normalization_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestPoint(),
+    'us.census.acs.B01003001', NULL
+) As t(OBS_GetMeasure_total_pop_point_null_normalization);
+
+-- Point-based OBS_GetMeasure, explicit area normalization area
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_point_area - 10923.093200390833950) / 10923.093200390833950 < 0.001 As OBS_GetMeasure_total_pop_point_area_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestPoint(),
+    'us.census.acs.B01003001', 'area'
+) As t(OBS_GetMeasure_total_pop_point_area);
+
 -- Poly-based OBS_GetMeasure, default normalization (none)
 -- is result within 0.1% of expected
 SELECT abs(OBS_GetMeasure_total_pop_polygon - 12327.3133495107) / 12327.3133495107 < 0.001 As OBS_GetMeasure_total_pop_polygon_test FROM
@@ -156,6 +172,22 @@ SELECT abs(OBS_GetMeasure_total_pop_polygon - 12327.3133495107) / 12327.31334951
    'us.census.acs.B01003001'
 ) As t(OBS_GetMeasure_total_pop_polygon);

+-- Poly-based OBS_GetMeasure, default normalization by NULL (none)
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_polygon_null_normalization - 12327.3133495107) / 12327.3133495107 < 0.001 As OBS_GetMeasure_total_pop_polygon_null_normalization_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestArea(),
+    'us.census.acs.B01003001', NULL
+) As t(OBS_GetMeasure_total_pop_polygon_null_normalization);
+
+-- Poly-based OBS_GetMeasure, explicit area normalization
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_polygon_area - 15787.4325563538) / 15787.4325563538 < 0.001 As OBS_GetMeasure_total_pop_polygon_area_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestArea(),
+    'us.census.acs.B01003001', 'area'
+) As t(OBS_GetMeasure_total_pop_polygon_area);
+
 -- Point-based OBS_GetMeasure with denominator normalization
 SELECT (abs(cdb_observatory.OBS_GetMeasure(
  cdb_observatory._TestPoint(),
@@ -166,6 +198,21 @@ SELECT abs(cdb_observatory.OBS_GetMeasure(
  cdb_observatory._TestArea(),
  'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;

+-- Poly-based OBS_GetMeasure with one very bad geom
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  cdb_observatory._ProblemTestArea(),
+  'us.census.acs.B01003001') - 96230.2929825897) / 96230.2929825897 < 0.001 As OBS_GetMeasure_bad_geometry;
+
+-- OBS_GetMeasure with NULL Input geometry
+SELECT cdb_observatory.OBS_GetMeasure(
+  NULL,
+  'us.census.acs.B01003001') IS NULL As OBS_GetMeasure_null_geometry;
+
+-- OBS_GetMeasure where there is no data
+SELECT cdb_observatory.OBS_GetMeasure(
+  ST_SetSRID(st_point(0, 0), 4326),
+  'us.census.acs.B01003001') IS NULL As OBS_GetMeasure_out_of_bounds_geometry;
+
 -- Point-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
  cdb_observatory._TestPoint(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As OBS_GetCategory_point;
@@ -174,6 +221,10 @@ SELECT cdb_observatory.OBS_GetCategory(
 SELECT cdb_observatory.OBS_GetCategory(
  cdb_observatory._TestArea(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As obs_getcategory_polygon;

+-- NULL Input OBS_GetCategory
+SELECT cdb_observatory.OBS_GetCategory(
+  NULL, 'us.census.spielman_singleton_segments.X10') IS NULL As obs_getcategory_null;
+
 -- Point-based OBS_GetPopulation, default normalization (area)
 SELECT (abs(OBS_GetPopulation - 10923.093200390833950) / 10923.093200390833950) < 0.001 As OBS_GetPopulation FROM
  cdb_observatory.OBS_GetPopulation(
@@ -187,6 +238,20 @@ FROM
    cdb_observatory._TestArea()
  ) As m(obs_getpopulation_polygon);

+-- Poly-based OBS_GetPopulation, default normalization (none) specified as NULL
+SELECT (abs(obs_getpopulation_polygon_null - 12327.3133495107) / 12327.3133495107) < 0.001 As obs_getpopulation_polygon_null_test
+FROM
+  cdb_observatory.OBS_GetPopulation(
+    cdb_observatory._TestArea(), NULL
+  ) As m(obs_getpopulation_polygon_null);
+
+-- Null input OBS_GetPopulation
+SELECT obs_getpopulation_polygon_null_geom IS NULL As obs_getpopulation_polygon_null_geom_test
+FROM
+  cdb_observatory.OBS_GetPopulation(
+    NULL, NULL
+  ) As m(obs_getpopulation_polygon_null_geom);
+
 -- Point-based OBS_GetUSCensusMeasure, default normalization (area)
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
  cdb_observatory._testpoint(), 'male population') - 6789.5647735060920500) / 6789.5647735060920500) < 0.001 As obs_getuscensusmeasure_point_male_pop;
@@ -195,6 +260,16 @@ SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
  cdb_observatory._testarea(), 'male population') - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure;

+-- Poly-based OBS_GetUSCensusMeasure, default normalization (none) specified
+-- with NULL
+SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
+  cdb_observatory._testarea(), 'male population', NULL) - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure_null;
+
+-- Poly-based OBS_GetUSCensusMeasure, Null input geom
+SELECT cdb_observatory.obs_getuscensusmeasure(
+  NULL, 'male population', NULL) IS NULL As obs_getuscensusmeasure_null_geom;
+
+
 -- Point-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
  cdb_observatory._testpoint(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_point;
@@ -203,6 +278,10 @@ SELECT cdb_observatory.OBS_GetUSCensusCategory(
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
  cdb_observatory._testarea(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_polygon;

+-- Null-input OBS_GetUSCensusCategory
+SELECT cdb_observatory.OBS_GetUSCensusCategory(
+  NULL, 'Spielman-Singleton Segments: 10 Clusters') IS NULL As OBS_GetUSCensusCategory_null;
+

 -- OBS_GetMeasureById tests
 -- typical query
@@ -236,3 +315,11 @@ SELECT cdb_observatory.OBS_GetMeasureById(
  'us.census.tiger.block_group',
  '2010 - 2014'
 ) IS NULL As OBS_GetMeasureById_nulls;
+
+-- NULL input id
+SELECT cdb_observatory.OBS_GetMeasureById(
+  NULL,
+  'us.census.acs.B01003001',
+  'us.census.tiger.block_group',
+  '2010 - 2014'
+) IS NULL As OBS_GetMeasureById_null_id;
--- a/src/pg/test/sql/42_observatory_exploration_test.sql
+++ b/src/pg/test/sql/42_observatory_exploration_test.sql
@@ -33,3 +33,473 @@ SELECT COUNT(*) > 0 AS _OBS_GetAvailableBoundariesExist
 FROM cdb_observatory.OBS_GetAvailableBoundaries(
  cdb_observatory._TestPoint()
 ) AS t(boundary_id, description, time_span, tablename);
+
+--
+-- OBS_GetAvailableNumerators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators())
+AS _obs_getavailablenumerators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablenumerators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getavailablenumerators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.income']
+))
+AS _obs_getavailablenumerators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01001002' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablenumerators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B19013001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablenumerators_no_income_denom_by_total_pop;
+
+SELECT 'us.zillow.AllHomes_Zhvi' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getavailablenumerators_zillow_at_zcta5;
+
+SELECT 'us.zillow.AllHomes_Zhvi' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.block_group'
+) WHERE valid_geom = True)
+AS _obs_getavailablenumerators_no_zillow_at_block_group;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailablenumerators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT numer_id
+FROM cdb_observatory.OBS_GetAvailableNumerators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailablenumerators_no_total_pop_1996;
+
+--
+-- OBS_GetAvailableDenominators tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators())
+AS _obs_getavailabledenominators_usa_pop_in_all;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_usa_pop_in_nyc_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_usa_pop_in_usa_extents;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabledenominators_no_usa_pop_not_in_zero_point;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.age_gender']
+))
+AS _obs_getavailabledenominators_usa_pop_in_age_gender_subsection;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.income']
+))
+AS _obs_getavailabledenominators_no_pop_in_income_subsection;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01001002'
+) WHERE valid_numer = True)
+AS _obs_getavailabledenominators_male_pop_denom_by_total_pop;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B19013001'
+) WHERE valid_numer = True)
+AS _obs_getavailabledenominators_no_income_denom_by_total_pop;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.tiger.zcta5'
+) WHERE valid_geom = True)
+AS _obs_getavailabledenominators_at_zcta5;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'es.ine.the_geom'
+) WHERE valid_geom = True)
+AS _obs_getavailabledenominators_none_spanish_geom;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2010 - 2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailabledenominators_total_pop_2010_2014;
+
+SELECT 'us.census.acs.B01003001' NOT IN (SELECT denom_id
+FROM cdb_observatory.OBS_GetAvailableDenominators(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailabledenominators_no_total_pop_1996;
+
+--
+-- OBS_GetAvailableGeometries tests
+--
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries())
+AS _obs_getavailablegeometries_usa_bg_in_all;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_usa_bg_in_nyc_point;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_usa_bg_in_usa_extents;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailablegeometries_no_usa_bg_not_in_zero_point;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['subsection/tags.boundary']
+))
+AS _obs_getavailablegeometries_usa_bg_in_boundary_subsection;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  ARRAY['section/tags.uk']
+))
+AS _obs_getavailablegeometries_no_bg_in_uk_section;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_numer = True)
+AS _obs_getavailablegeometries_total_pop_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'foo.bar.baz'
+) WHERE valid_numer = True)
+AS _obs_getavailablegeometries_foobarbaz_not_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_total_pop_denom_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'foo.bar.baz'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_foobarbaz_denom_not_in_usa_bg;
+
+SELECT 'us.census.tiger.block_group' IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '2014'
+) WHERE valid_timespan = True)
+AS _obs_getavailablegeometries_bg_2014;
+
+SELECT 'us.census.tiger.block_group' NOT IN (SELECT geom_id
+FROM cdb_observatory.OBS_GetAvailableGeometries(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, '1996'
+) WHERE valid_timespan = True)
+AS _obs_getavailablegeometries_bg_not_1996;
+
+--
+-- OBS_GetAvailableTimespans tests
+--
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans())
+AS _obs_getavailabletimespans_2010_2014_in_all;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_2010_2014_in_nyc_point;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakeEnvelope(
+      -169.8046875, 21.289374355860424,
+      -47.4609375, 72.0739114882038
+  ), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_2010_2014_in_usa_extents;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
+  NULL, NULL, NULL, NULL
+)) AS _obs_getavailabletimespans_no_usa_bg_not_in_zero_point;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'us.census.acs.B01003001'
+) WHERE valid_numer = True)
+AS _obs_getavailabletimespans_total_pop_in_2010_2014;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, 'foo.bar.baz'
+) WHERE valid_numer = True)
+AS _obs_getavailabletimespans_foobarbaz_not_in_2010_2014;
+
+SELECT '2010 - 2014' IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'us.census.acs.B01003001'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_total_pop_denom_in_2010_2014;
+
+SELECT '2010 - 2014' NOT IN (SELECT timespan_id
+FROM cdb_observatory.OBS_GetAvailableTimespans(
+  ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326),
+  NULL, NULL, 'foo.bar.baz'
+) WHERE valid_denom = True)
+AS _obs_getavailablegeometries_foobarbaz_denom_not_in_2010_2014;
+
+--
+-- _OBS_GetGeometryScores tests
+--
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+       AS _obs_geometryscores_500m_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+       AS _obs_geometryscores_5km_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.zcta5',
+             'us.census.tiger.county', 'us.census.tiger.block_group']
+       AS _obs_geometryscores_50km_buffer
+       FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY[ 'us.census.tiger.county', 'us.census.tiger.zcta5',
+             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
+             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_2500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text
+      = '{ "us.census.tiger.block_group" : 9, "us.census.tiger.census_tract" : 3, "us.census.tiger.zcta5" : 0, "us.census.tiger.county" : 0 }'
+      AS _obs_geometryscores_numgeoms_500m_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 899, "us.census.tiger.census_tract" : 328, "us.census.tiger.zcta5" : 45, "us.census.tiger.county" : 1 }'
+      AS _obs_geometryscores_numgeoms_5km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 5000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 12112, "us.census.tiger.census_tract" : 3792, "us.census.tiger.zcta5" : 550, "us.census.tiger.county" : 13 }'
+      AS _obs_geometryscores_numgeoms_50km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 48415, "us.census.tiger.census_tract" : 15776, "us.census.tiger.zcta5" : 6534, "us.census.tiger.county" : 295 }'
+      AS _obs_geometryscores_numgeoms_500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT JSON_Object_Agg(geom_id, numgeoms::int ORDER BY numgeoms DESC)::Text =
+      '{ "us.census.tiger.block_group" : 165489, "us.census.tiger.census_tract" : 55152, "us.census.tiger.zcta5" : 26500, "us.census.tiger.county" : 2551 }'
+      AS _obs_geometryscores_numgeoms_2500km_buffer
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 2500000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county']);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.county', 'us.census.tiger.zcta5',
+             'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_50_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 50);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC)
+      = ARRAY['us.census.tiger.zcta5', 'us.census.tiger.county',
+              'us.census.tiger.census_tract', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_500_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 500);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.census_tract', 'us.census.tiger.zcta5',
+             'us.census.tiger.county', 'us.census.tiger.block_group']
+      AS _obs_geometryscores_500km_buffer_2500_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 2500);
+
+SELECT ARRAY_AGG(geom_id ORDER BY score DESC) =
+       ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+             'us.census.tiger.zcta5', 'us.census.tiger.county']
+      AS _obs_geometryscores_500km_buffer_25000_geoms
+      FROM cdb_observatory._OBS_GetGeometryScores(
+  ST_Buffer(ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)::Geography, 50000)::Geometry(Geometry, 4326),
+  ARRAY['us.census.tiger.block_group', 'us.census.tiger.census_tract',
+        'us.census.tiger.zcta5', 'us.census.tiger.county'], 25000);
+
+--
+-- OBS_LegacyBuilderMetadata tests
+--
+
+SELECT 'us.census.acs.B01003001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _total_pop_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B19013001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata()
+) AS _median_income_in_legacy_builder_metadata;
+
+SELECT 'us.census.acs.B01003001' IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _total_pop_in_legacy_builder_metadata_sums;
+
+SELECT 'us.census.acs.B19013001' NOT IN (SELECT
+  (jsonb_array_elements(((jsonb_array_elements(subsection))->'f1')->'columns')->'f1')->>'id' AS id
+  FROM cdb_observatory.OBS_LegacyBuilderMetadata('sum')
+) AS _median_income_not_in_legacy_builder_metadata_sums;
+
+SELECT COUNT(*) = 0 _no_dupe_subsections_in_legacy_builder_metadata FROM (
+  SELECT name, subsection, count(*) FROM
+    (SELECT name, ((JSONB_Array_Elements(subsection))->'f1')->>'id' subsection
+    FROM cdb_observatory.obs_legacybuildermetadata()) foo
+  GROUP BY name, subsection
+  HAVING count(*) > 1
+) bar;
+
--- a/src/pg/test/sql/44_observatory_geometries_test.sql
+++ b/src/pg/test/sql/44_observatory_geometries_test.sql
@@ -34,7 +34,7 @@ SELECT cdb_observatory.OBS_GetBoundary(
 -- expect null geometry since there are no census tracts at null island
 -- timespan implictly null
 SELECT cdb_observatory.OBS_GetBoundary(
-  CDB_LatLng(0, 0),
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
  'us.census.tiger.census_tract'
 ) IS NULL As OBS_GetBoundary_null_island_census_tract;

@@ -78,7 +78,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(

 -- should give back null since there is not a census tract at null island
 SELECT cdb_observatory.OBS_GetBoundaryId(
-  CDB_LatLng(0, 0),
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
  'us.census.tiger.census_tract'
 ) IS NULL As OBS_GetBoundaryId_null_island;

--- a/src/python/requirements.txt
+++ b/src/python/requirements.txt
@@ -0,0 +1,3 @@
+nose
+nose_parameterized
+psycopg2
--- a/src/python/test/README.md
+++ b/src/python/test/README.md
@@ -0,0 +1,14 @@
+### Integration/performance tests
+
+Tests here are meant to be run on a box with an Observatory meta/data dump
+loaded and ready to be tested against the API.
+
+The local Python needs the requirements in `src/python/requirements.txt`.
+
+In order to find and access the correct database, the `PGUSER`, `PGPASSWORD`,
+`PGHOST`, `PGPORT` and `PGDATABASE` env variables should be set.
+
+Tests should be executed as follows:
+
+     nosetests test/autotest.py
+     nosetests -s test/perftest.py
--- a/src/python/test/autotest.py
+++ b/src/python/test/autotest.py
@@ -1,105 +1,160 @@
 from nose.tools import assert_equal, assert_is_not_none
+from nose.plugins.skip import SkipTest
 from nose_parameterized import parameterized

-import os
-import re
-import requests
+from util import query

-HOSTNAME = os.environ['OBS_HOSTNAME']
-API_KEY = os.environ['OBS_API_KEY']
-META_HOSTNAME = os.environ.get('OBS_META_HOSTNAME', HOSTNAME)
-META_API_KEY = os.environ.get('OBS_META_API_KEY', API_KEY)
-USE_SCHEMA = 'OBS_USE_SCHEMA' in os.environ
+USE_SCHEMA = True

-
-def query(q, is_meta=False, **options):
-    '''
-    Query the account.  Returned is the response, wrapped by the requests
-    library.
-    '''
-    url = 'https://{hostname}/api/v2/sql'.format(
-        hostname=META_HOSTNAME if is_meta else HOSTNAME)
-    params = options.copy()
-    params['q'] = re.sub(r'\s+', ' ', q)
-    params['api_key'] = META_API_KEY if is_meta else API_KEY
-    return requests.get(url, params=params)
-
-MEASURE_COLUMNS = [(r['numer_id'], r['point_only'], ) for r in query('''
+MEASURE_COLUMNS = query('''
 SELECT distinct numer_id, numer_aggregate NOT ILIKE 'sum' as point_only
-FROM obs_meta
+FROM observatory.obs_meta
 WHERE numer_type ILIKE 'numeric'
 AND numer_weight > 0
-''', is_meta=True).json()['rows']]
+''').fetchall()

-CATEGORY_COLUMNS = [(r['numer_id'], ) for r in query('''
+CATEGORY_COLUMNS = query('''
 SELECT distinct numer_id
-FROM obs_meta
+FROM observatory.obs_meta
 WHERE numer_type ILIKE 'text'
 AND numer_weight > 0
-''', is_meta=True).json()['rows']]
+''').fetchall()

-BOUNDARY_COLUMNS = [(r['id'], ) for r in query('''
-SELECT id FROM obs_column
+BOUNDARY_COLUMNS = query('''
+SELECT id FROM observatory.obs_column
 WHERE type ILIKE 'geometry'
 AND weight > 0
-''', is_meta=True).json()['rows']]
+''').fetchall()

-US_CENSUS_MEASURE_COLUMNS = [(r['numer_name'], ) for r in query('''
+US_CENSUS_MEASURE_COLUMNS = query('''
 SELECT distinct numer_name
-FROM obs_meta
+FROM observatory.obs_meta
 WHERE numer_type ILIKE 'numeric'
 AND 'us.census.acs.acs' = ANY (subsection_tags)
 AND numer_weight > 0
-''', is_meta=True).json()['rows']]
+''').fetchall()
+
+SKIP_COLUMNS = set([
+    u'mx.inegi_columns.INDI18',
+    u'mx.inegi_columns.ECO40',
+    u'mx.inegi_columns.POB34',
+    u'mx.inegi_columns.POB63',
+    u'mx.inegi_columns.INDI7',
+    u'mx.inegi_columns.EDU28',
+    u'mx.inegi_columns.SCONY10',
+    u'mx.inegi_columns.EDU31',
+    u'mx.inegi_columns.POB7',
+    u'mx.inegi_columns.VIV30',
+    u'mx.inegi_columns.INDI12',
+    u'mx.inegi_columns.EDU13',
+    u'mx.inegi_columns.ECO43',
+    u'mx.inegi_columns.VIV9',
+    u'mx.inegi_columns.HOGAR25',
+    u'mx.inegi_columns.POB32',
+    u'mx.inegi_columns.ECO7',
+    u'mx.inegi_columns.INDI19',
+    u'mx.inegi_columns.INDI16',
+    u'mx.inegi_columns.POB65',
+    u'mx.inegi_columns.INDI3',
+    u'mx.inegi_columns.INDI9',
+    u'mx.inegi_columns.POB36',
+    u'mx.inegi_columns.POB33',
+    u'mx.inegi_columns.POB58',
+    u'mx.inegi_columns.DISC4',
+])
+
+#def default_geometry_id(column_id):
+#    '''
+#    Returns default test point for the column_id.
+#    '''
+#    if column_id == 'whosonfirst.wof_disputed_geom':
+#        return 'ST_SetSRID(ST_MakePoint(76.57, 33.78), 4326)'
+#    elif column_id == 'whosonfirst.wof_marinearea_geom':
+#        return 'ST_SetSRID(ST_MakePoint(-68.47, 43.33), 4326)'
+#    elif column_id in ('us.census.tiger.school_district_elementary',
+#                       'us.census.tiger.school_district_secondary',
+#                       'us.census.tiger.school_district_elementary_clipped',
+#                       'us.census.tiger.school_district_secondary_clipped'):
+#        return 'ST_SetSRID(ST_MakePoint(-73.7067, 40.7025), 4326)'
+#    elif column_id.startswith('es.ine'):
+#        return 'ST_SetSRID(ST_MakePoint(-2.51141249535454, 42.8226119029222), 4326)'
+#    elif column_id.startswith('us.zillow'):
+#        return 'ST_SetSRID(ST_MakePoint(-81.3544048197256, 28.3305906291771), 4326)'
+#    elif column_id.startswith('ca.'):
+#        return ''
+#    else:
+#        return 'ST_SetSRID(ST_MakePoint(-73.9, 40.7), 4326)'


-def default_geometry_id(column_id):
+def default_lonlat(column_id):
    '''
    Returns default test point for the column_id.
    '''
    if column_id == 'whosonfirst.wof_disputed_geom':
-        return 'CDB_LatLng(33.78, 76.57)'
+        return (76.57, 33.78)
    elif column_id == 'whosonfirst.wof_marinearea_geom':
-        return 'CDB_LatLng(43.33, -68.47)'
+        return (-68.47, 43.33)
    elif column_id in ('us.census.tiger.school_district_elementary',
                       'us.census.tiger.school_district_secondary',
                       'us.census.tiger.school_district_elementary_clipped',
                       'us.census.tiger.school_district_secondary_clipped'):
-        return 'CDB_LatLng(40.7025, -73.7067)'
-    elif column_id.startswith('es.ine'):
-        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
+        return (40.7025, -73.7067)
+    elif column_id.startswith('uk'):
+        if 'WA' in column_id:
+            return (51.46844551219723, -3.184833526611328)
+        else:
+            return (51.51461834694225, -0.08883476257324219)
+    elif column_id.startswith('es'):
+        return (42.8226119029222, -2.51141249535454)
    elif column_id.startswith('us.zillow'):
-        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
+        return (28.3305906291771, -81.3544048197256)
+    elif column_id.startswith('mx.'):
+        return (19.41347699386547, -99.17019367218018)
+    elif column_id.startswith('th.'):
+        return (13.725377712079784, 100.49263000488281)
+    # cols for French Guyana only
+    elif column_id in ('fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+                       , 'fr.insee.P12_ACTOCC15P_ILT45D'
+                       , 'fr.insee.P12_RP_CHOS', 'fr.insee.P12_RP_HABFOR'
+                       , 'fr.insee.P12_RP_EAUCH', 'fr.insee.P12_RP_BDWC'
+                       , 'fr.insee.P12_RP_MIDUR', 'fr.insee.P12_RP_CLIM'
+                       , 'fr.insee.P12_RP_MIBOIS', 'fr.insee.P12_RP_CASE'
+                       , 'fr.insee.P12_RP_TTEGOU', 'fr.insee.P12_RP_ELEC'
+                       , 'fr.insee.P12_ACTOCC15P_ILT45D'):
+        return (4.938408371206558, -52.32908248901367)
+    elif column_id.startswith('fr.'):
+        return (48.860875144709475, 2.3613739013671875)
+    elif column_id.startswith('ca.'):
+        return (43.65594991256823, -79.37965393066406)
+    elif column_id.startswith('us.census.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.dma.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.ihme.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.bls.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.qcew.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('whosonfirst.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('us.epa.'):
+        return (40.7, -73.9)
+    elif column_id.startswith('eu.'):
+        raise SkipTest('No tests for Eurostat!')
+        #return (52.52207036136366, 13.40606689453125)
    else:
-        return 'CDB_LatLng(40.7, -73.9)'
+        raise Exception('No catalog point set for {}'.format(column_id))


 def default_point(column_id):
-    '''
-    Returns default test point for the column_id.
-    '''
-    if column_id == 'whosonfirst.wof_disputed_geom':
-        return 'CDB_LatLng(33.78, 76.57)'
-    elif column_id == 'whosonfirst.wof_marinearea_geom':
-        return 'CDB_LatLng(43.33, -68.47)'
-    elif column_id in ('us.census.tiger.school_district_elementary',
-                       'us.census.tiger.school_district_secondary',
-                       'us.census.tiger.school_district_elementary_clipped',
-                       'us.census.tiger.school_district_secondary_clipped'):
-        return 'CDB_LatLng(40.7025, -73.7067)'
-    elif column_id.startswith('uk'):
-        if 'WA' in column_id:
-            return 'CDB_LatLng(51.46844551219723, -3.184833526611328)'
-        else:
-            return 'CDB_LatLng(51.51461834694225, -0.08883476257324219)'
-    elif column_id.startswith('es'):
-        return 'CDB_LatLng(42.8226119029222, -2.51141249535454)'
-    elif column_id.startswith('us.zillow'):
-        return 'CDB_LatLng(28.3305906291771, -81.3544048197256)'
-    elif column_id.startswith('mx.'):
-        return 'CDB_LatLng(19.41347699386547, -99.17019367218018)'
-    else:
-        return 'CDB_LatLng(40.7, -73.9)'
+    lat, lng = default_lonlat(column_id)
+    return 'ST_SetSRID(ST_MakePoint({lng}, {lat}), 4326)'.format(
+        lat=lat, lng=lng)


 def default_area(column_id):
@@ -107,27 +162,26 @@ def default_area(column_id):
    Returns default test area for the column_id
    '''
    point = default_point(column_id)
-    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 1000), 4326)'.format(
+    area = 'ST_Transform(ST_Buffer(ST_Transform({point}, 3857), 250), 4326)'.format(
        point=point)
    return area

@parameterized(US_CENSUS_MEASURE_COLUMNS)
 def test_get_us_census_measure_points(name):
-    print 'test_get_us_census_measure_points, ', name
    resp = query('''
 SELECT * FROM {schema}OBS_GetUSCensusMeasure({point}, '{name}')
                 '''.format(name=name.replace("'", "''"),
                            schema='cdb_observatory.' if USE_SCHEMA else '',
                            point=default_point('')))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
+    rows = resp.fetchall()
    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
+    assert_is_not_none(rows[0][0])


@parameterized(MEASURE_COLUMNS)
 def test_get_measure_areas(column_id, point_only):
-    print 'test_get_measure_areas, ', column_id, point_only
+    if column_id in SKIP_COLUMNS:
+        raise SkipTest('Column {} should be skipped'.format(column_id))
    if point_only:
        return
    resp = query('''
@@ -135,24 +189,23 @@ SELECT * FROM {schema}OBS_GetMeasure({area}, '{column_id}')
                 '''.format(column_id=column_id,
                            schema='cdb_observatory.' if USE_SCHEMA else '',
                            area=default_area(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
+    rows = resp.fetchall()
    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
+    assert_is_not_none(rows[0][0])


@parameterized(MEASURE_COLUMNS)
 def test_get_measure_points(column_id, point_only):
-    print 'test_get_measure_points, ', column_id, point_only
+    if column_id in SKIP_COLUMNS:
+        raise SkipTest('Column {} should be skipped'.format(column_id))
    resp = query('''
 SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
                 '''.format(column_id=column_id,
                            schema='cdb_observatory.' if USE_SCHEMA else '',
                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
+    rows = resp.fetchall()
    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
+    assert_is_not_none(rows[0][0])

 #@parameterized(CATEGORY_COLUMNS)
 #def test_get_category_areas(column_id):
@@ -164,20 +217,20 @@ SELECT * FROM {schema}OBS_GetMeasure({point}, '{column_id}')
 #    assert_equal(resp.status_code, 200)
 #    rows = resp.json()['rows']
 #    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
+#    assert_is_not_none(rows[0][0])

@parameterized(CATEGORY_COLUMNS)
 def test_get_category_points(column_id):
-    print 'test_get_category_points, ', column_id
+    if column_id in SKIP_COLUMNS:
+        raise SkipTest('Column {} should be skipped'.format(column_id))
    resp = query('''
 SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
                 '''.format(column_id=column_id,
                            schema='cdb_observatory.' if USE_SCHEMA else '',
                            point=default_point(column_id)))
-    assert_equal(resp.status_code, 200)
-    rows = resp.json()['rows']
+    rows = resp.fetchall()
    assert_equal(1, len(rows))
-    assert_is_not_none(rows[0].values()[0])
+    assert_is_not_none(rows[0][0])

 #@parameterized(BOUNDARY_COLUMNS)
 #def test_get_boundaries_by_geometry(column_id):
@@ -189,7 +242,7 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
 #    assert_equal(resp.status_code, 200)
 #    rows = resp.json()['rows']
 #    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
+#    assert_is_not_none(rows[0][0])

 #@parameterized(BOUNDARY_COLUMNS)
 #def test_get_points_by_geometry(column_id):
@@ -201,7 +254,7 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
 #    assert_equal(resp.status_code, 200)
 #    rows = resp.json()['rows']
 #    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
+#    assert_is_not_none(rows[0][0])

 #@parameterized(BOUNDARY_COLUMNS)
 #def test_get_boundary_points(column_id):
@@ -213,7 +266,7 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
 #    assert_equal(resp.status_code, 200)
 #    rows = resp.json()['rows']
 #    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
+#    assert_is_not_none(rows[0][0])

 #@parameterized(BOUNDARY_COLUMNS)
 #def test_get_boundary_id(column_id):
@@ -225,7 +278,7 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
 #    assert_equal(resp.status_code, 200)
 #    rows = resp.json()['rows']
 #    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
+#    assert_is_not_none(rows[0][0])

 #@parameterized(BOUNDARY_COLUMNS)
 #def test_get_boundary_by_id(column_id):
@@ -237,4 +290,5 @@ SELECT * FROM {schema}OBS_GetCategory({point}, '{column_id}')
 #    assert_equal(resp.status_code, 200)
 #    rows = resp.json()['rows']
 #    assert_equal(1, len(rows))
-#    assert_is_not_none(rows[0].values()[0])
+#    assert_is_not_none(rows[0][0])
+
--- a/src/python/test/perftest.py
+++ b/src/python/test/perftest.py
@@ -0,0 +1,62 @@
+from nose.tools import assert_equal, assert_is_not_none
+from nose_parameterized import parameterized
+
+from util import query, commit
+
+from time import time
+
+USE_SCHEMA = True
+
+for q in (
+        'DROP TABLE IF EXISTS obs_censustest',
+        '''CREATE TABLE obs_censustest (cartodb_id SERIAL PRIMARY KEY,
+           the_geom GEOMETRY, name TEXT, measure NUMERIC, category TEXT)''',
+        '''INSERT INTO obs_censustest (the_geom, name)
+           SELECT * FROM {schema}OBS_GetBoundariesByGeometry(
+           st_makeenvelope(-74.05437469482422,40.66319159533881,
+                           -73.81885528564453,40.745696344339564, 4326),
+           'us.census.tiger.block_group_clipped') As m(the_geom, geoid)'''
+):
+    query(q.format(
+        schema='cdb_observatory.' if USE_SCHEMA else '',
+    ))
+    commit()
+
+
+ARGS = {
+    'OBS_GetMeasureByID': "name, 'us.census.acs.B01001002', '{}'",
+    'OBS_GetMeasure': "{}, 'us.census.acs.B01001002'",
+    'OBS_GetCategory': "{}, 'us.census.spielman_singleton_segments.X10'",
+}
+
+GEOMS = {
+    'point': 'ST_PointOnSurface(the_geom)',
+    'polygon_match': 'the_geom',
+    'polygon_buffered': 'ST_Buffer(the_geom::GEOGRAPHY, 1000)::GEOMETRY(GEOMETRY, 4326)',
+}
+
+
+@parameterized([
+    ('OBS_GetMeasureByID', 'us.census.tiger.block_group_clipped'),
+    ('OBS_GetMeasureByID', 'us.census.tiger.county'),
+    ('OBS_GetMeasure', GEOMS['point']),
+    ('OBS_GetMeasure', GEOMS['polygon_match']),
+    ('OBS_GetMeasure', GEOMS['polygon_buffered']),
+    ('OBS_GetCategory', GEOMS['point']),
+    ('OBS_GetCategory', GEOMS['polygon_match']),
+    ('OBS_GetCategory', GEOMS['polygon_buffered']),
+])
+def test_performance(api_method, arg):
+    print api_method, arg
+    col = 'measure' if 'measure' in api_method.lower() else 'category'
+    for rows in (1, 10, 50, 100):
+        q = 'UPDATE obs_censustest SET {col} = {schema}{api_method}({args}) WHERE cartodb_id < {n}'.format(
+            col=col,
+            schema='cdb_observatory.' if USE_SCHEMA else '',
+            api_method=api_method,
+            args=ARGS[api_method].format(arg),
+            n=rows+1)
+        start = time()
+        query(q)
+        end = time()
+        print rows, ': ', (rows / (end - start)), ' QPS'
--- a/src/python/test/util.py
+++ b/src/python/test/util.py
@@ -0,0 +1,31 @@
+import os
+import psycopg2
+
+DB_CONN = psycopg2.connect('postgres://{user}:{password}@{host}:{port}/{database}'.format(
+    user=os.environ.get('PGUSER', 'postgres'),
+    password=os.environ.get('PGPASSWORD', ''),
+    host=os.environ.get('PGHOST', 'localhost'),
+    port=os.environ.get('PGPORT', '5432'),
+    database=os.environ.get('PGDATABASE', 'postgres'),
+))
+CURSOR = DB_CONN.cursor()
+
+
+def query(q):
+    '''
+    Query the database.
+    '''
+    try:
+        CURSOR.execute(q)
+        return CURSOR
+    except:
+        DB_CONN.rollback()
+        raise
+
+
+def commit():
+    try:
+        DB_CONN.commit()
+    except:
+        DB_CONN.rollback()
+        raise
Author	SHA1	Message	Date
Mario de Frutos	cbe7b6dd15	Merge pull request #225 from CartoDB/develop Release 1.1.5	2016-11-29 17:58:08 +01:00
Mario de Frutos	603d26c674	Version 1.1.5 artifacts	2016-11-29 17:43:28 +01:00
Mario de Frutos	355f6281e5	Merge pull request #224 from CartoDB/release-v-1.1.5 Release v 1.1.5	2016-11-29 17:40:53 +01:00
John Krauss	70f4807139	update NEWS	2016-11-29 16:45:10 +00:00
john krauss	84794124fd	Merge pull request #223 from CartoDB/fix-getmeasure-exc-out-of-bounds return NULL when there is no data for a measure at a geometry according to raster	2016-11-29 11:35:19 -05:00
John Krauss	6c08681446	return NULL when there is no data for a measure at a geometry according to our raster. Fixes #220	2016-11-29 16:41:44 +00:00
Mario de Frutos	e5e0b39595	Merge pull request #219 from CartoDB/develop Release 1.1.4	2016-11-22 11:11:17 +01:00
Mario de Frutos	713aacf535	Version 1.1.4 artifact	2016-11-22 10:03:22 +01:00
Mario de Frutos	9bf4b07be7	Merge pull request #218 from CartoDB/release-v-1.1.4 Release v 1.1.4	2016-11-22 10:01:35 +01:00
John Krauss	aaf580baca	update NEWS	2016-11-21 22:32:14 +00:00
john krauss	6845d4361d	Merge pull request #217 from CartoDB/fix-legacy-metadata-dupes Fix legacy metadata dupes	2016-11-21 16:59:05 -05:00
John Krauss	fa778f4eb0	test for #216	2016-11-21 22:03:31 +00:00
John Krauss	22a413102b	Fixes bug where multiple subsections returned from `OBS_LegacyBuilderMetadata`, #216	2016-11-21 21:50:56 +00:00
Mario de Frutos	08980f47a7	Merge pull request #215 from CartoDB/develop Release 1.1.3	2016-11-17 20:08:00 +01:00
Mario de Frutos	54d512d4fb	Release 1.1.3 artifact	2016-11-17 20:03:50 +01:00
Mario de Frutos	2a0ff6a541	Merge pull request #214 from CartoDB/release-v-1.1.3 release v1.1.3	2016-11-17 19:58:56 +01:00
John Krauss	62e13086e1	release v1.1.3	2016-11-15 18:36:53 +00:00
Mario de Frutos	60b723de92	Merge pull request #212 from CartoDB/develop Release 1.1.2	2016-11-11 17:09:56 +01:00
Mario de Frutos	7e04c38c3a	Release 1.1.2 artifact	2016-11-11 17:08:04 +01:00
Mario de Frutos	45dea25ec0	Merge pull request #211 from CartoDB/release-v-1.1.2 Release v 1.1.2	2016-11-11 17:05:28 +01:00
John Krauss	39836ea321	update NEWS.md	2016-11-09 22:11:20 +00:00
john krauss	17d343a756	Merge pull request #209 from CartoDB/use-rasters Use rasters	2016-11-09 16:58:25 -05:00
john krauss	c7c8a6676a	Merge pull request #210 from CartoDB/eu-epa-testpoints add test points for EU and EPA, make it easier to work with meta.py	2016-11-09 16:47:14 -05:00
John Krauss	be4b5abbfa	use highest ranked geom for obs_getmeasure, simplify scoring	2016-11-07 23:57:33 +00:00
John Krauss	8dad88a6b3	fix minor bug in _obs_getgeometryscores with FIRST, add tests	2016-11-07 21:26:44 +00:00
John Krauss	7e6489f2a1	add test points for EU and EPA, make it easier to work with meta.py	2016-11-07 16:50:00 +00:00
John Krauss	9fdca9161c	minor stylistic fix	2016-11-04 15:32:25 +00:00
John Krauss	785a5eed29	obs_getgeometryscores and usage by obs_getavailablegeometries	2016-11-02 21:11:38 +00:00
Mario de Frutos	c91fcab28c	Merge pull request #208 from CartoDB/develop Release 1.1.1	2016-10-21 12:09:59 +02:00
Mario de Frutos	174ee65f46	Release 1.1.1 artifact	2016-10-21 12:08:49 +02:00
Mario de Frutos	4aac696963	Merge pull request #205 from CartoDB/release-v-1.1.1 Release v 1.1.1	2016-10-21 12:06:01 +02:00
John Krauss	5c5b587495	Merge remote-tracking branch 'origin/release-v-1.1.1' into release-v-1.1.1	2016-10-14 20:18:01 +00:00
John Krauss	dccae1ed8b	NEWS for 1.1.1	2016-10-14 20:17:46 +00:00
john krauss	1e02593fae	Merge pull request #204 from CartoDB/fr-ca-testpoints adding testpoints for FR, Guayane, and CA	2016-10-14 16:09:39 -04:00
John Krauss	89d10ff993	do not skip canada tests	2016-10-14 18:39:47 +00:00
John Krauss	e35b7825ce	adding testpoints for FR, Guayane, and CA	2016-10-07 20:27:04 +00:00
Javier Goizueta	ff613f7c12	Merge pull request #203 from CartoDB/develop Release v1.1.0	2016-10-05 16:48:44 +02:00
Javier Goizueta	06e0b5bcf8	Release 1.1.0	2016-10-05 16:24:55 +02:00
Javier Goizueta	efae735324	Merge pull request #202 from CartoDB/release-v-1.1.0 Release v 1.1.0	2016-10-05 16:16:36 +02:00
John Krauss	7bf87faba1	adding NEWS for 1.1.0	2016-10-04 22:35:48 +00:00
john krauss	0b7e794fb9	Merge pull request #201 from CartoDB/builder-api-func Builder api func	2016-10-04 18:29:43 -04:00
John Krauss	017b404264	make bounds optional for dimensional queries, add all tests	2016-10-04 22:21:05 +00:00
John Krauss	50b745227b	working obs_getavailablenumerators tests	2016-10-04 20:10:24 +00:00
John Krauss	2171cb83c7	add tests for builder legacy func	2016-10-04 19:46:37 +00:00
John Krauss	0d9f0e4996	allow null geom to be passed in for the obs_get* functions, add in convenience legacy builder metadata function	2016-10-04 19:16:32 +00:00
John Krauss	b473ffe307	updated fixtures generation from local postgres, fixed a few tests that broke	2016-10-03 20:36:14 +00:00
John Krauss	2a1598d491	first pass on generating new metadata from local	2016-09-30 20:44:03 +00:00
John Krauss	827104756e	another test stub	2016-09-30 17:39:25 +00:00
John Krauss	3602aab804	remove table defintions, stub in tests	2016-09-29 20:53:12 +00:00
John Krauss	48221fc358	Merge branch 'develop' into builder-api-func	2016-09-29 20:23:08 +00:00
Carla	5629bdf035	Merge pull request #197 from CartoDB/develop Release v1.0.7	2016-09-21 12:02:04 +02:00
Carla Iriberri	f4113eaea3	Release 1.0.7	2016-09-21 11:24:29 +02:00
Carla	86fac2a600	Merge pull request #196 from CartoDB/release-v-1.0.7 Release v 1.0.7	2016-09-21 11:12:22 +02:00
John Krauss	2d753cd758	Skip bad MX measure, smaller buffer for faster tests, updated NEWS.md	2016-09-20 17:56:23 +00:00
john krauss	96a98c3bce	Merge pull request #194 from CartoDB/null-resilience Resolve #178	2016-09-20 13:38:11 -04:00
john krauss	d58263935d	Merge pull request #195 from CartoDB/ca-testing Add point to make sure CA data is present	2016-09-20 12:27:02 -04:00
John Krauss	104608c6d3	Add point to make sure CA data is present	2016-09-20 16:31:15 +00:00
John Krauss	c67fe12111	return NULL in cases when NULL is passed as input geometry or geometry ID. resolves #178	2016-09-20 16:26:13 +00:00
John Krauss	18cfdc60d0	tmp commit	2016-09-19 16:08:37 +00:00
Carla	d63934bfc5	Merge pull request #191 from CartoDB/develop Release 1.0.6 with table level framework improvements	2016-09-08 13:52:36 +02:00
Carla Iriberri	860290595c	Release 1.0.6	2016-09-08 10:37:37 +02:00
Carla	bf4ade2fa0	Merge pull request #186 from CartoDB/measure_release Use explicit functions for query construction and metadata	2016-09-08 09:58:25 +02:00
Carla	32d37a74b3	Remove cascades and quote conveniently	2016-09-02 12:04:03 +02:00
Mario de Frutos	da877e4ef0	Modify PR template to include the update of NEWS.md	2016-08-25 14:36:07 +02:00
Mario de Frutos	15de07ca33	Modify PR template	2016-08-25 14:30:42 +02:00
Mario de Frutos	8af3e22661	Merge pull request #188 from CartoDB/pr_template Added PR template	2016-08-25 14:27:14 +02:00
Mario de Frutos	fdd591b159	Added PR template	2016-08-25 11:28:01 +02:00
Carla Iriberri	5eb4ede219	Fix	2016-08-23 17:20:48 +02:00
Carla Iriberri	dd5f560359	Separate functions between files	2016-08-19 16:39:30 +02:00
Carla Iriberri	62c2693553	Avoid function check to dispatch	2016-08-19 13:04:54 +02:00
Carla Iriberri	48d1bfdb13	Remove JSON manipulation to use json functions	2016-08-19 12:45:38 +02:00
Carla Iriberri	30f27e5b58	Check function name and use param names instead of	2016-08-18 15:43:03 +02:00
Carla Iriberri	26b22a9bf4	Use explicit functions for query construction and metadata	2016-08-18 15:36:32 +02:00
Mario de Frutos	c9e809c061	Merge pull request #185 from CartoDB/develop Release 1.0.5	2016-08-18 15:06:50 +02:00
Mario de Frutos	43e83751ae	Release 1.0.5 artifact	2016-08-18 15:05:38 +02:00
Mario de Frutos	4c13434b9a	Merge pull request #182 from CartoDB/sql-tests SQL Integration and Performance Tests	2016-08-18 14:54:53 +02:00
Mario de Frutos	8785639ece	Merge pull request #154 from CartoDB/iriberri-patch-1 Use 6432 for connections from server	2016-08-18 11:10:02 +02:00
John Krauss	f991f5a1e6	docs and NEWS for the new tests	2016-08-12 18:56:06 +00:00
John Krauss	e4b4ebf72d	Adapted autotest to to work with SQL directly instead of over HTTP SQL API	2016-08-12 18:48:31 +00:00
Mario de Frutos	20f56c98de	Merge pull request #179 from CartoDB/develop Release 1.0.4	2016-08-10 16:20:00 +02:00
Mario de Frutos	e4ea90835a	Release 1.0.4 artifact	2016-08-10 16:18:59 +02:00
Mario de Frutos	8f2c8f571c	Merge pull request #175 from CartoDB/release-v-1.0.4 Release v 1.0.4	2016-08-10 16:15:56 +02:00
John Krauss	e9857e89fb	release-v-1.0.4 increment and news	2016-07-26 13:08:28 +00:00
john krauss	8ed2135a7f	Merge pull request #174 from CartoDB/all-null-defaults Always default to NULL, fixes #173	2016-07-26 09:03:53 -04:00
John Krauss	af69b44f25	Always default to NULL, fixes #173	2016-07-26 13:05:40 +00:00
Mario de Frutos	e12b729c51	Release 1.0.3 artifact	2016-07-25 16:44:05 +02:00
Mario de Frutos	a42827a3c9	Merge pull request #172 from CartoDB/develop Release 1.0.3	2016-07-25 16:11:28 +02:00
Mario de Frutos	948cdbff19	Merge pull request #171 from CartoDB/release-v-1.0.3 Release v 1.0.3	2016-07-25 16:09:35 +02:00
John Krauss	1c7c73f948	release candidate 1.0.3	2016-07-25 13:20:09 +00:00
john krauss	360adc47df	Merge pull request #170 from CartoDB/handle-bad-geoms Handle bad geoms	2016-07-25 09:15:09 -04:00
john krauss	571f1f343a	Merge pull request #168 from CartoDB/fix-per-sq-m-obs-getmeasure-area Fix per sq m obs getmeasure area	2016-07-25 09:14:08 -04:00
john krauss	186c57efbd	Merge pull request #167 from CartoDB/null-defaults Null defaults	2016-07-25 09:12:51 -04:00
john krauss	b139a24012	Merge pull request #165 from CartoDB/fix-required-libs Fix required libs	2016-07-25 09:09:21 -04:00
john krauss	260704327e	Merge pull request #164 from CartoDB/hotfix-error-on-exception in this function, its "measure_id" not "numer_id"	2016-07-25 09:06:26 -04:00
John Krauss	252610673a	handle difficult geometries more gracefully. fixes #160	2016-07-25 13:05:36 +00:00
John Krauss	d054f37528	fixes #160 : snaptogrid then buffer input polygons	2016-07-22 21:47:06 +00:00
John Krauss	3d58fd284a	fix #159 ensure getuscensusmeasure and getpopulation work as expected with NULL passed explicitly as normalization	2016-07-22 19:24:36 +00:00
John Krauss	efcea9be7b	Merge branch 'fix-required-libs' into fix-per-sq-m-obs-getmeasure-area	2016-07-22 18:59:42 +00:00
John Krauss	cf242515e3	install postgres_fdw in test setup. fixes #166	2016-07-22 18:59:11 +00:00
John Krauss	4c434f5448	tests doublechecking NULL default handled correctly, and that area normalization for polygon is per square kilometer	2016-07-22 18:56:27 +00:00
John Krauss	59dd09c554	Merge branch 'fix-required-libs' into fix-per-sq-m-obs-getmeasure-area	2016-07-22 18:17:54 +00:00
John Krauss	8187ab4bbe	ensure tests run in order. Fixes #162	2016-07-22 17:44:16 +00:00
John Krauss	e54d95fa8f	remove unused plpythonu and cartodb dependencies Fixes #161	2016-07-22 17:43:40 +00:00
John Krauss	d766f08b03	calculate area normalization of a polygon by square kilometer, not square meter. fixes #158	2016-07-22 15:18:43 +00:00
John Krauss	8f345fd508	in this function, its "measure_id" not "numer_id"	2016-07-21 15:35:09 -04:00
csobier	383c3eb6ec	Merge pull request #155 from CartoDB/148-move-glossary-and-license-files added absolutel urls for doc links as weird redirects are happening f…	2016-07-19 11:58:59 -04:00
csobier	798c0a73a1	added absolutel urls for doc links as weird redirects are happening for relative links	2016-07-19 11:57:40 -04:00
Carla	bfa57f4971	Use 6432 for connections from server	2016-07-19 17:54:08 +02:00
csobier	e7a16f4b4d	Merge pull request #153 from CartoDB/148-move-glossary-and-license-files fixing hyperlinks to live docs	2016-07-19 11:43:23 -04:00
csobier	540ff68a90	fixing hyperlinks to live docs	2016-07-19 11:42:19 -04:00
csobier	6a1df2abd1	Merge pull request #149 from CartoDB/148-move-glossary-and-license-files removed glossary and license files, updated any hyperlinks to these f…	2016-07-19 11:32:25 -04:00
csobier	dc9ed2de33	fixes issue 148	2016-07-19 11:30:09 -04:00
Carla	acaa434118	Merge pull request #152 from CartoDB/obs_fdw_dependency Add postgres_fdw as a dependency of observatory	2016-07-19 15:58:05 +02:00
Carla	173d7c0aec	Add postgres_fdw as a dependency of observatory	2016-07-19 15:56:25 +02:00
Belén Achaerandio	970d5d2119	Merge pull request #151 from CartoDB/add-docs-url CR Update measures_functions.md	2016-07-19 15:06:32 +02:00
Belén Achaerandio	a3681062cb	second-fix	2016-07-19 12:28:57 +02:00
Belén Achaerandio	a179a46b86	Update measures_functions.md	2016-07-19 12:15:58 +02:00
csobier	1054443117	removed glossary and license files, updated any hyperlinks to these files	2016-07-15 09:05:49 -04:00