Merge pull request #179 from CartoDB/develop

Release 1.0.4

NEWS.md

@@ -1,3 +1,35 @@
+1.0.4 (2016-07-26)
+
+__Bugfixes__
+
+* Always default arguments to `NULL`, which prevents duplication & overwrite by
+  dataservices-api
+  ([#173](https://github.com/CartoDB/observatory-extension/issues/173))
+
+1.0.3 (2016-07-25)
+
+__Bugfixes__
+
+* Raise exception instead of crashing when `OBS_GetMeasure` is passed a polygon
+  in combination with a non-summable measure ([cartodb/issues
+  #9063](https://github.com/CartoDB/cartodb/issues/9063))
+* Unnecessary dependencies on cartodb and plpythonu removed
+  ([#161](https://github.com/CartoDB/observatory-extension/issues/161))
+* Tests forced to run in-order on all systems
+  ([#162](https://github.com/CartoDB/observatory-extension/issues/162))
+* Area normalization done by square kilometer instead of square meter for
+  polygons ([#158](https://github.com/CartoDB/observatory-extension/issues/158))
+* `postgres-fdw` installed as required in unit test environment
+  ([#166](https://github.com/CartoDB/observatory-extension/issues/166))
+
+__Improvements__
+
+* Added tests to make sure all functions can handle explicit NULL as default
+  ([#159](https://github.com/CartoDB/observatory-extension/issues/159))
+* Buffer and snaptogrid used to be far more liberal accepting problem geoms
+  ([#170](https://github.com/CartoDB/observatory-extension/issues/160))
+
+
 1.0.2 (2016-07-12)
 ---
 

doc/API.md

@@ -1,6 +1,6 @@
 # Data Observatory Documentation
 
-This file is for reference purposes only. It is intended for tracking the Data Observatory functions that should be displayed from the live Docs site. Like all API doc, the golden source of this code will live in this observatory-extension repo, and will be edited in this repo.
+This file is for reference purposes only. It is intended for tracking the Data Observatory functions that should be pulled into the live Docs site. Like all API doc, the golden source of this code will live in this observatory-extension repo, and will be edited in this repo. Other non-code related content will live as a local file in the Docs repo.
 
 ## Documentation
 
@@ -9,5 +9,5 @@ This file is for reference purposes only. It is intended for tracking the Data O
 * [Measures Functions](measures_functions.md)
 * [Boundary Functions](boundary_functions.md)
 * [Discovery Functions](discovery_functions.md)
-* [Glossary](glossary.md)
-* [License](license.md)
+* [Glossary](local file in the Docs repo)
+* [License](local file in the Docs repo)

doc/boundary_functions.md

@@ -1,8 +1,8 @@
 # Boundary Functions
 
-Use the following functions to retrieve [Boundary](/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
+Use the following functions to retrieve [Boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below.
 
-You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) boundaries through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](http://docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text)
 
@@ -91,14 +91,14 @@ FROM OBS_GetPointsByGeometry(
 
 ## OBS_GetBoundary(point_geometry, boundary_id)
 
-The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID glossary table below](below). This is a useful method for performing aggregations of points.
+The ```OBS_GetBoundary(point_geometry, boundary_id)``` method returns a boundary geometry defined as overlapping the point geometry and from the desired boundary set (e.g. Census Tracts). See the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids). This is a useful method for performing aggregations of points.
 
 #### Arguments
 
 Name | Description
 --- | ---
 point_geometry | a WGS84 polygon geometry (the_geom)
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
 
 #### Returns
@@ -124,14 +124,14 @@ SET the_geom = OBS_GetBoundary(the_geom, 'us.census.tiger.block_group')
 
 ## OBS_GetBoundaryId(point_geometry, boundary_id)
 
-The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID glossary table below](below). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
+The ```OBS_GetBoundaryId(point_geometry, boundary_id)``` returns a unique geometry_id for the boundary geometry that contains a given point geometry. See the [Boundary ID Glossary](http://docs/carto-engine/data/glossary/#boundary-ids). The method can be combined with ```OBS_GetBoundaryById(geometry_id)``` to create a point aggregation workflow.
 
 #### Arguments
 
 Name |Description
 --- | ---
 point_geometry | a WGS84 point geometry (the_geom)
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
 
 #### Returns
@@ -164,7 +164,7 @@ The ```OBS_GetBoundaryById(geometry_id, boundary_id)``` returns the boundary geo
 Name | Description
 --- | ---
 geometry_id | a string identifier for a Boundary geometry
-boundary_id | a boundary identifier from the [Boundary ID glossary table below](below)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 timespan (optional) | year(s) to request from (`NULL` (default) gives most recent)
 
 #### Returns

doc/discovery_functions.md

@@ -1,6 +1,6 @@
 # Discovery Functions
 
-If you are using the [discovery methods](/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/carto-engine/data/overview/#boundary-data) and [measures](/carto-engine/data/overview/#measures-data) data.
+If you are using the [discovery methods](https://carto.com/docs/carto-engine/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](https://carto.com/docs/carto-engine/data/overview/#boundary-data) and [measures](https://carto.com/docs/carto-engine/data/overview/#measures-data) data.
 
 ## OBS_Search(search_term)
 
@@ -47,7 +47,7 @@ A TABLE containing the following properties
 
 Key | Description
 --- | ---
-boundary_id | a boundary identifier from the [boundary ID glossary](/carto-engine/data/glossary/#boundary-ids)
+boundary_id | a boundary identifier from the [Boundary ID Glossary](https://carto.com/docs/carto-engine/data/glossary/#boundary-ids)
 description | a brief description of the boundary dataset
 time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry.
 

doc/glossary.md

@@ -1,126 +0,0 @@
-# Glossary
-
-A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses.
-
-## Boundary IDs
-
-Boundary name | Boundary ID | Shoreline Clipped Boundary ID
---------------------- | --------------------- | ---
-US States  |  us.census.tiger.state  |  us.census.tiger.state_clipped
-US County  |  us.census.tiger.county  |  us.census.tiger.county_clipped
-US Census Zip Code Tabulation Areas  |  us.census.tiger.zcta5  |  us.census.tiger.zcta5_clipped
-US Census Tracts  |  us.census.tiger.census_tract  |  us.census.tiger.census_tract_clipped
-US Elementary School District  |  us.census.tiger.school_district_elementary  |  us.census.tiger.school_district_elementary_clipped
-US Secondary School District  |  us.census.tiger.school_district_secondary  |  us.census.tiger.school_district_secondary_clipped
-US Unified School District  |  us.census.tiger.school_district_unified  |  us.census.tiger.school_district_unified_clipped
-US Congressional Districts  |  us.census.tiger.congressional_district  |  us.census.tiger.congressional_district_clipped
-US Census Blocks  |  us.census.tiger.block  |  us.census.tiger.block_clipped
-US Census Block Groups  |  us.census.tiger.block_group  |  us.census.tiger.block_group_clipped
-US Census PUMAs  |  us.census.tiger.puma  |  us.census.tiger.puma_clipped
-US Incorporated Places  |  us.census.tiger.place  |  us.census.tiger.place_clipped
-ES Sección Censal  |  es.ine.geom  |   none
-Regions (First-level Administrative)  |  whosonfirst.wof_region_geom  |   none
-Continents  |  whosonfirst.wof_continent_geom  |   none
-Countries  |  whosonfirst.wof_country_geom  |  none
-Marine Areas  |  whosonfirst.wof_marinearea_geom  |  none
-Disputed Areas  |  whosonfirst.wof_disputed_geom  |  none
-
-
-
-## OBS_GetUSCensusMeasure Names Table
-
-This list contains human readable names accepted in the ```OBS_GetUSCensusMeasure``` function. For the more comprehensive list of columns available to the ```OBS_GetMeasure``` function, see the [Data Observatory Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
-
-Measure name             |  Measure description
------------------------- | --------------------
-Male Population   |   The number of people within each geography who are male.
-Female Population   |   The number of people within each geography who are female.
-Median Age   |   The median age of all people in a given geographic area.
-Total Population   |   The total number of all people living in a given geographic area.  This is a very useful catch-all denominator when calculating rates.
-Population not Hispanic   |   The number of people not identifying as Hispanic or Latino in each geography.
-White Population   |   The number of people identifying as white, non-Hispanic in each geography.
-Black or African American Population   |   The number of people identifying as black or African American, non-Hispanic in each geography.
-American Indian and Alaska Native Population   |   The number of people identifying as American Indian or Alaska native in each geography.
-Asian Population   |   The number of people identifying as Asian, non-Hispanic in each geography.
-Other Race population   |   The number of people identifying as another race in each geography.
-Two or more races population   |   The number of people identifying as two or more races in each geography.
-Hispanic Population   |   The number of people identifying as Hispanic or Latino in each geography.
-Not a U.S. Citizen Population   |   The number of people within each geography who indicated that they are not U.S. citizens.
-Workers over the Age of 16   |   The number of people in each geography who work. Workers include those employed at private for-profit companies, the self-employed, government workers and non-profit employees.
-Commuters by Car, Truck, or Van   |   The number of workers age 16 years and over within  a geographic area who primarily traveled to work by car, truck or  van.  This is the principal mode of travel or type of conveyance,  by distance rather than time, that the worker usually used to get  from home to work.
-Commuters who drove alone   |   The number of workers age 16 years and over within a geographic area who primarily traveled by car driving alone. This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Commuters by Carpool   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by carpool.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Commuters by Bus   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by bus.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.  This is a subset of workers who commuted by public transport.
-Commuters by Subway or Elevated   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by subway or elevated train.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.  This is a subset of workers who commuted by public transport.
-Walked to Work   |   The number of workers age 16 years and over within a geographic area who primarily walked to work.  This would mean that of any way of getting to work, they travelled the most distance walking.
-Worked at Home   |   The count within a geographical area of workers over the age of 16 who worked at home.
-Workers age 16 and over who do not work from home   |   The number of workers over the age of 16 who do not work from home in a geographic area.
-Number of workers with less than 10 minute commute   |   The number of workers over the age of 16 who do not work from home and commute in less than 10 minutes in a geographic area.
-Number of workers with a commute between 35 and 44 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 35 and 44 minutes in a geographic area.
-Number of workers with a commute of over 60 minutes   |   The number of workers over the age of 16 who do not work from home and commute in over 60 minutes in a geographic area.
-Aggregate travel time to work   |   The total number of minutes every worker over the age of 16 who did not work from home spent spent commuting to work in one day in a geographic area.
-Commuters by Public Transportation   |   The number of workers age 16 years and over within a geographic area who primarily traveled to work by public transportation.  This is the principal mode of travel or type of conveyance, by distance rather than time, that the worker usually used to get from home to work.
-Number of workers with a commute between 10 and 14 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 10 and 14 minutes in a geographic area.
-Number of workers with a commute between 15 and 19 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 15 and 19 minutes in a geographic area.
-Number of workers with a commute between 20 and 24 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 20 and 24 minutes in a geographic area.
-Number of workers with a commute between 25 and 29 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 25 and 29 minutes in a geographic area.
-Number of workers with a commute between 30 and 34 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area.
-Number of workers with a commute between 45 and 59 minutes   |   The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area.
-Children under 18 Years of Age   |   The number of people within each geography who are under 18 years of age.
-Households   |   A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people.
-Population 15 Years and Over   |   The number of people in a geographic area who are over the age of 15.  This is used mostly as a denominator of marital status.
-Never Married   |   The number of people in a geographic area who have never been married.
-Currently married   |   The number of people in a geographic area who are currently married.
-Married but separated   |   The number of people in a geographic area who are married but separated.
-Widowed   |   The number of people in a geographic area who are widowed.
-Divorced   |   The number of people in a geographic area who are divorced.
-Population 3 Years and Over   |   The total number of people in each geography age 3 years and over.  This denominator is mostly used to calculate rates of school enrollment.
-Students Enrolled in School   |   The total number of people in each geography currently enrolled at any level of school, from nursery or pre-school to advanced post-graduate education.  Only includes those over the age of 3.
-Students Enrolled in Grades 1 to 4   |   The total number of people in each geography currently enrolled in grades 1 through 4 inclusive.  This corresponds roughly to elementary school.
-Students Enrolled in Grades 5 to 8   |   The total number of people in each geography currently enrolled in grades 5 through 8 inclusive.  This corresponds roughly to middle school.
-Students Enrolled in Grades 9 to 12   |   The total number of people in each geography currently enrolled in grades 9 through 12 inclusive.  This corresponds roughly to high school.
-Students Enrolled as Undergraduate in College   |   The number of people in a geographic area who are enrolled in college at the undergraduate level. Enrollment refers to being registered or listed as a student in an educational program leading to a college degree. This may be a public school or college, a private school or college.
-Population 25 Years and Over   |   The number of people in a geographic area who are over the age of 25.  This is used mostly as a denominator of educational attainment.
-Population Completed High School   |   The number of people in a geographic area over the age of 25 who completed high school, and did not complete a more advanced degree.
-Population completed less than one year of college, no degree   |   The number of people in a geographic area over the age of 25 who attended college for less than one year and no further.
-Population completed more than one year of college, no degree   |   The number of people in a geographic area over the age of 25 who attended college for more than one year but did not obtain a degree.
-Population Completed Associate's Degree   |   The number of people in a geographic area over the age of 25 who obtained a associate's degree, and did not complete a more advanced degree.
-Population Completed Bachelor's Degree   |   The number of people in a geographic area over the age of 25 who obtained a bachelor's degree, and did not complete a more advanced degree.
-Population Completed Master's Degree   |   The number of people in a geographic area over the age of 25 who obtained a master's degree, but did not complete a more advanced degree.
-Population 5 Years and Over   |   The number of people in a geographic area who are over the age of 5.  This is primarily used as a denominator of measures of language spoken at home.
-Speaks only English at Home   |   The number of people in a geographic area over age 5 who speak only English at home.
-Speaks Spanish at Home   |   The number of people in a geographic area over age 5 who speak Spanish at home, possibly in addition to other languages.
-Population for Whom Poverty Status Determined   |   The number of people in each geography who could be identified as either living in poverty or not.  This should be used as the denominator when calculating poverty rates, as it excludes people for whom it was not possible to determine poverty.
-Income In The Past 12 Months Below Poverty Level   |   The number of people in a geographic area who are part of a family (which could be just them as an individual) determined to be in poverty following the Office of Management and Budget's Directive 14. (https://www.census.gov/hhes/povmeas/methodology/ombdir14.html)
-Households with income less than $10,000   |   The number of households in a geographic area whose annual income was less than $10,000.
-Households with income of $10,000 to $14,999   |   The number of households in a geographic area whose annual income was between $10,000 and $14,999.
-Households with income of $15,000 to $19,999   |   The number of households in a geographic area whose annual income was between $15,000 and $19,999.
-Households with income of $20,000 To $24,999   |   The number of households in a geographic area whose annual income was between $20,000 and $24,999.
-Households with income of $25,000 To $29,999   |   The number of households in a geographic area whose annual income was between $20,000 and $24,999.
-Households with income of $30,000 To $34,999   |   The number of households in a geographic area whose annual income was between $30,000 and $34,999.
-Households with income of $35,000 To $39,999   |   The number of households in a geographic area whose annual income was between $35,000 and $39,999.
-Households with income of $40,000 To $44,999   |   The number of households in a geographic area whose annual income was between $40,000 and $44,999.
-Households with income of $45,000 To $49,999   |   The number of households in a geographic area whose annual income was between $45,000 and $49,999.
-Households with income of $50,000 To $59,999   |   The number of households in a geographic area whose annual income was between $50,000 and $59,999.
-Households with income of $60,000 To $74,999   |   The number of households in a geographic area whose annual income was between $60,000 and $74,999.
-Households with income of $75,000 To $99,999   |   The number of households in a geographic area whose annual income was between $75,000 and $99,999.
-Households with income of $100,000 To $124,999   |   The number of households in a geographic area whose annual income was between $100,000 and $124,999.
-Households with income of $125,000 To $149,999   |   The number of households in a geographic area whose annual income was between $125,000 and $149,999.
-Households with income of $150,000 To $199,999   |   The number of households in a geographic area whose annual income was between $150,000 and $1999,999.
-Households with income of $200,000 Or More   |   The number of households in a geographic area whose annual income was more than $200,000.
-Median Household Income in the past 12 Months   |   Within a geographic area, the median income received by every household on a regular basis before payments for personal income taxes, social security, union dues, medicare deductions, etc.  It includes income received from wages, salary, commissions, bonuses, and tips; self-employment income from own nonfarm or farm businesses, including proprietorships and partnerships; interest, dividends, net rental income, royalty income, or income from estates and trusts; Social Security or Railroad Retirement income; Supplemental Security Income (SSI); any cash public assistance or welfare payments from the state or local welfare office; retirement, survivor, or disability benefits; and any other sources of income received regularly such as Veterans' (VA) payments, unemployment and/or worker's compensation, child support, and alimony.
-Population age 16 and over   |   The number of people in each geography who are age 16 or over.
-Population in Labor Force   |   The number of people in each geography who are either in the civilian labor force or are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
-Population in Civilian Labor Force   |   The number of civilians 16 years and over in each geography who can be classified as either employed or unemployed below.
-Employed Population   |   The number of civilians 16 years old and over in each geography who either (1) were at work, that is, those who did any work at all during the reference week as paid employees, worked in their own business or profession, worked on their own farm, or worked 15 hours or more as unpaid workers on a family farm or in a family business; or (2) were with a job but not at work, that is, those who did not work during the reference week but had jobs or businesses from which they were temporarily absent due to illness, bad weather, industrial dispute, vacation, or other personal reasons. Excluded from the employed are people whose only activity consisted of work around the house or unpaid volunteer work for religious, charitable, and similar organizations; also excluded are all institutionalized people and people on active duty in the United States Armed Forces.
-Unemployed Population   |   The number of civilians in each geography who are 16 years old and over and are classified as unemployed.
-Population in Armed Forces   |   The number of people in each geography who are members of the U.S. Armed Forces (people on active duty with the United States Army, Air Force, Navy, Marine Corps, or Coast Guard).
-Population Not in Labor Force   |   The number of people in each geography who are 16 years old and over who are not classified as members of the labor force. This category consists mainly of students, homemakers, retired workers, seasonal workers interviewed in an off season who were not looking for work, institutionalized people, and people doing only incidental unpaid family work.
-Housing Units   |   A count of housing units in each geography.  A housing unit is a house, an apartment, a mobile home or trailer, a group of rooms, or a single room occupied as separate living quarters, or if vacant, intended for occupancy as separate living quarters.
-Vacant Housing Units   |   The count of vacant housing units in a geographic area. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Vacant Housing Units for Rent   |   The count of vacant housing units in a geographic area that are for rent. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Vacant Housing Units for Sale   |   The count of vacant housing units in a geographic area that are for sale. A housing unit is vacant if no one is living in it at the time of enumeration, unless its occupants are only temporarily absent. Units temporarily occupied at the time of enumeration entirely by people who have a usual residence elsewhere are also classified as vacant.
-Median Rent   |   The median contract rent within a geographic area. The contract rent is the monthly rent agreed to or contracted for, regardless of any furnishings, utilities, fees, meals, or services that may be included. For vacant units, it is the monthly rent asked for the rental unit at the time of interview.
-Percent of Household Income Spent on Rent   |   Within a geographic area, the median percentage of household income which was spent on gross rent.  Gross rent is the amount of the contract rent plus the estimated average monthly cost of utilities (electricity, gas, water, sewer etc.) and fuels (oil, coal, wood, etc.) if these are paid by the renter.  Household income is the sum of the income of all people 15 years and older living in the household.
-Owner-occupied Housing Units valued at $1,000,000 or more.   |   The count of owner occupied housing units in a geographic area that are valued at $1,000,000 or more.  Value is the respondent's estimate of how much the property (house and lot, mobile home and lot, or condominium unit) would sell for if it were for sale.
-Owner-occupied Housing Units with a Mortgage   |   The count of housing units within a geographic area that are mortagaged. Mortgage refers to all forms of debt where the property is pledged as security for repayment of the debt, including deeds of trust, trust deed, contracts to purchase, land contracts, junior mortgages, and home equity loans.

doc/license.md

@@ -1,20 +0,0 @@
-# License
-
-The Data Observatory is a collection of data sources with varying licenses and terms of use. We have endeavored to find you data that will work for the broadest set of use-cases. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use.
-
-_**Legal Note**: The Data Observatory makes use of a variety of third party data and databases (collectively, the “Data”). You acknowledge that the included Data, and the licenses and terms of use, may be amended from time to time. Whenever you use the Data, you agree to the current relevant terms or license. Some Data will require that you provide attribution to the data source. Other Data may be protected by US or international copyright laws, treaties, or conventions. The Data and associated metadata are provided 'as-is', without express or implied warranty of any kind, including, but not limited to, infringement, merchantability and fitness for a particular purpose. CartoDB is not responsible for the accuracy, completeness, timeliness or quality of the Data._
-
-Name  | Terms link 
--------|---------
-ACS   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
-TIGER   |  [https://www.usa.gov/government-works](https://www.usa.gov/government-works)
-Zillow Home Value Index | This data is "Aggregate Data", per the Zillow Terms of Use<br /><br />[http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm)
-Who's on First |  [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License)
-GeoNames  |  [http://www.geonames.org/](http://www.geonames.org/)
-GeoPlanet  |  [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/)
-Natural Earth  |  [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/)
-Quattroshapes  |  [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md)
-Zetashapes  |  [http://zetashapes.com/license](http://zetashapes.com/license)
-Spielman & Singleton  |  [https://www.openicpsr.org/repoEntity/show/41329](https://www.openicpsr.org/repoEntity/show/41329)
-El Instituto Nacional de Estadística (INE) | The National Statistics Institute (INE) of Spain includes data from multiple sources. If you are re-using their data, they explicitly require that you reference them accordingly<br /><br />[http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout)
-

doc/measures_functions.md

@@ -1,10 +1,10 @@
 # Measures Functions
 
-[Data Observatory Measures](/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
+[Data Observatory Measures](https://carto.com/docs/carto-engine/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location.
 
 There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf).
 
-You can [access](/carto-engine/data/accessing/#accessing-the-data-observatory) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
+You can [access](https://carto.com/docs/carto-engine/data/accessing) measures through the CARTO Editor. The same methods will work if you are using the CARTO Engine to develop your application. We [encourage you](https://carto.com/docs/carto-engine/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT).
 
 ## OBS_GetUSCensusMeasure(point geometry, measure_name text)
 
@@ -15,7 +15,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
 Name |Description
 --- | ---
 point | a WGS84 point geometry (the_geom)
-measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
+measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
 normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)
@@ -46,7 +46,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure
 Name |Description
 --- | ---
 polygon | a WGS84 polygon geometry (the_geom)
-measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table).
+measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the Glossary](https://carto.com/docs/carto-engine/data/glossary/#obsgetuscensusmeasure-names-table).
 normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://data-observatory.s3.amazonaws.com/observatory.pdf) (optional)
 boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract')
 time_span | time span of interest (e.g., 2010 - 2014)

release/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.2'
-requires = 'postgis'
+default_version = '1.0.4'
+requires = 'postgis, postgres_fdw'
 superuser = true
 schema = cdb_observatory

src/pg/Makefile

@@ -24,7 +24,7 @@ $(DATA): $(SOURCES_DATA)
 	$(SED) $(REPLACEMENTS) $(SOURCES_DATA_DIR)/*.sql > $@
 
 TEST_DIR = test
-REGRESS = $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql)))
+REGRESS = $(sort $(notdir $(basename $(wildcard $(TEST_DIR)/sql/*test.sql))))
 REGRESS_OPTS = --inputdir='$(TEST_DIR)' --outputdir='$(TEST_DIR)'
 
 PG_CONFIG = pg_config

src/pg/observatory.control

@@ -1,5 +1,5 @@
 comment = 'CartoDB Observatory backend extension'
-default_version = '1.0.2'
-requires = 'postgis'
+default_version = '1.0.4'
+requires = 'postgis, postgres_fdw'
 superuser = true
 schema = cdb_observatory

src/pg/sql/41_observatory_augmentation.sql

@@ -361,6 +361,7 @@ DECLARE
   sql TEXT;
   numer_name TEXT;
 BEGIN
+  geom := ST_SnapToGrid(geom, 0.000001);
 
   EXECUTE
      $query$
@@ -383,6 +384,7 @@ BEGIN
     geom_type := 'point';
   ELSIF ST_GeometryType(geom) IN ('ST_Polygon', 'ST_MultiPolygon') THEN
     geom_type := 'polygon';
+    geom := ST_Buffer(geom, 0.000001);
   ELSE
     RAISE EXCEPTION 'Invalid geometry type (%), can only handle ''ST_Point'', ''ST_Polygon'', and ''ST_MultiPolygon''',
                     ST_GeometryType(geom);
@@ -439,7 +441,7 @@ BEGIN
                                    WHERE ST_Intersects(%L, geom.%I)
                                      AND ST_Area(ST_Intersection(%L, geom.%I)) / ST_Area(geom.%I) > 0)
                     SELECT SUM(numer.%I * (SELECT _geom.overlap FROM _geom WHERE _geom.geom_ref = numer.%I)) /
-                           ST_Area(%L::Geography)
+                           (ST_Area(%L::Geography) / 1000000)
                     FROM observatory.%I numer
                     WHERE numer.%I = ANY ((SELECT ARRAY_AGG(geom_ref) FROM _geom)::TEXT[])',
                 geom, geom_colname, geom_colname,
@@ -481,7 +483,7 @@ BEGIN
     ELSIF map_type = 'predenominated' THEN
       IF numer_aggregate NOT ILIKE 'sum' THEN
         RAISE EXCEPTION 'Cannot calculate "%" (%) for custom area as it cannot be summed, use ST_PointOnSurface instead',
-                        numer_name, numer_id;
+                        numer_name, measure_id;
       ELSE
         sql = format('WITH _geom AS (SELECT ST_Area(ST_Intersection(%L, geom.%I))
                                           / ST_Area(geom.%I) overlap, geom.%I geom_ref

src/pg/sql/44_observatory_geometries.sql

@@ -244,7 +244,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetBoundariesByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -253,7 +253,7 @@ DECLARE
   geoid_colname text;
   target_table text;
 BEGIN
-
+  overlap_type := COALESCE(overlap_type, 'intersects');
   -- check inputs
   IF lower(overlap_type) NOT IN ('contains', 'intersects', 'within')
   THEN
@@ -318,7 +318,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -364,7 +364,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetBoundariesByPointAndRadius(
   radius numeric, -- radius in meters
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -382,7 +382,8 @@ BEGIN
                FROM cdb_observatory._OBS_GetBoundariesByGeometry(
                         circle_boundary,
                         boundary_id,
-                        time_span);
+                        time_span,
+                        overlap_type);
   RETURN;
 END;
 $$ LANGUAGE plpgsql;
@@ -394,7 +395,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory._OBS_GetPointsByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE
@@ -403,6 +404,7 @@ DECLARE
   geoid_colname text;
   target_table text;
 BEGIN
+  overlap_type := COALESCE(overlap_type, 'intersects');
 
   IF lower(overlap_type) NOT IN ('contains', 'within', 'intersects')
   THEN
@@ -464,7 +466,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByGeometry(
   geom geometry(Geometry, 4326),
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 BEGIN
@@ -509,7 +511,7 @@ CREATE OR REPLACE FUNCTION cdb_observatory.OBS_GetPointsByPointAndRadius(
   radius numeric, -- radius in meters
   boundary_id text,
   time_span text DEFAULT NULL,
-  overlap_type text DEFAULT 'intersects')
+  overlap_type text DEFAULT NULL)
 RETURNS TABLE(the_geom geometry, geom_refs text)
 AS $$
 DECLARE

src/pg/test/expected/01_install_test.out

@@ -1,6 +1,5 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION plpythonu;
-CREATE EXTENSION cartodb;
+CREATE EXTENSION postgres_fdw;
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/expected/41_observatory_augmentation_test.out

@@ -42,15 +42,30 @@ t
 obs_getmeasure_total_pop_point_test
 t
 (1 row)
+obs_getmeasure_total_pop_point_null_normalization_test
+t
+(1 row)
+obs_getmeasure_total_pop_point_area_test
+t
+(1 row)
 obs_getmeasure_total_pop_polygon_test
 t
 (1 row)
+obs_getmeasure_total_pop_polygon_null_normalization_test
+t
+(1 row)
+obs_getmeasure_total_pop_polygon_area_test
+t
+(1 row)
 obs_getmeasure_total_male_point_denominator
 t
 (1 row)
 obs_getmeasure_total_male_poly_denominator
 t
 (1 row)
+obs_getmeasure_bad_geometry
+t
+(1 row)
 obs_getcategory_point
 t
 (1 row)
@@ -63,12 +78,18 @@ t
 obs_getpopulation_polygon_test
 t
 (1 row)
+obs_getpopulation_polygon_null_test
+t
+(1 row)
 obs_getuscensusmeasure_point_male_pop
 t
 (1 row)
 obs_getuscensusmeasure
 t
 (1 row)
+obs_getuscensusmeasure_null
+t
+(1 row)
 obs_getuscensuscategory_point
 t
 (1 row)

src/pg/test/sql/01_install_test.sql

@@ -1,7 +1,6 @@
 -- Install dependencies
 CREATE EXTENSION postgis;
-CREATE EXTENSION plpythonu;
-CREATE EXTENSION cartodb;
+CREATE EXTENSION postgres_fdw;
 
 -- Install the extension
 CREATE EXTENSION observatory VERSION 'dev';

src/pg/test/sql/41_observatory_augmentation_test.sql

@@ -148,6 +148,22 @@ SELECT abs(OBS_GetMeasure_total_pop_point - 10923.093200390833950) / 10923.09320
     'us.census.acs.B01003001'
 ) As t(OBS_GetMeasure_total_pop_point);
 
+-- Point-based OBS_GetMeasure, default normalization by NULL (area)
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_point_null_normalization - 10923.093200390833950) / 10923.093200390833950 < 0.001 As OBS_GetMeasure_total_pop_point_null_normalization_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestPoint(),
+    'us.census.acs.B01003001', NULL
+) As t(OBS_GetMeasure_total_pop_point_null_normalization);
+
+-- Point-based OBS_GetMeasure, explicit area normalization area
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_point_area - 10923.093200390833950) / 10923.093200390833950 < 0.001 As OBS_GetMeasure_total_pop_point_area_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestPoint(),
+    'us.census.acs.B01003001', 'area'
+) As t(OBS_GetMeasure_total_pop_point_area);
+
 -- Poly-based OBS_GetMeasure, default normalization (none)
 -- is result within 0.1% of expected
 SELECT abs(OBS_GetMeasure_total_pop_polygon - 12327.3133495107) / 12327.3133495107 < 0.001 As OBS_GetMeasure_total_pop_polygon_test FROM
@@ -156,6 +172,22 @@ SELECT abs(OBS_GetMeasure_total_pop_polygon - 12327.3133495107) / 12327.31334951
     'us.census.acs.B01003001'
 ) As t(OBS_GetMeasure_total_pop_polygon);
 
+-- Poly-based OBS_GetMeasure, default normalization by NULL (none)
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_polygon_null_normalization - 12327.3133495107) / 12327.3133495107 < 0.001 As OBS_GetMeasure_total_pop_polygon_null_normalization_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestArea(),
+    'us.census.acs.B01003001', NULL
+) As t(OBS_GetMeasure_total_pop_polygon_null_normalization);
+
+-- Poly-based OBS_GetMeasure, explicit area normalization
+-- is result within 0.1% of expected
+SELECT abs(OBS_GetMeasure_total_pop_polygon_area - 15787.4325563538) / 15787.4325563538 < 0.001 As OBS_GetMeasure_total_pop_polygon_area_test FROM
+  cdb_observatory.OBS_GetMeasure(
+    cdb_observatory._TestArea(),
+    'us.census.acs.B01003001', 'area'
+) As t(OBS_GetMeasure_total_pop_polygon_area);
+
 -- Point-based OBS_GetMeasure with denominator normalization
 SELECT (abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._TestPoint(),
@@ -166,6 +198,11 @@ SELECT abs(cdb_observatory.OBS_GetMeasure(
   cdb_observatory._TestArea(),
   'us.census.acs.B01001002', 'denominator') - 0.49026340444793965457) / 0.49026340444793965457 < 0.001 As OBS_GetMeasure_total_male_poly_denominator;
 
+-- Poly-based OBS_GetMeasure with one very bad geom
+SELECT abs(cdb_observatory.OBS_GetMeasure(
+  cdb_observatory._ProblemTestArea(),
+  'us.census.acs.B01003001') - 96230.2929825897) / 96230.2929825897 < 0.001 As OBS_GetMeasure_bad_geometry;
+
 -- Point-based OBS_GetCategory
 SELECT cdb_observatory.OBS_GetCategory(
   cdb_observatory._TestPoint(), 'us.census.spielman_singleton_segments.X10') = 'Wealthy, urban without Kids' As OBS_GetCategory_point;
@@ -187,6 +224,13 @@ FROM
     cdb_observatory._TestArea()
   ) As m(obs_getpopulation_polygon);
 
+-- Poly-based OBS_GetPopulation, default normalization (none) specified as NULL
+SELECT (abs(obs_getpopulation_polygon_null - 12327.3133495107) / 12327.3133495107) < 0.001 As obs_getpopulation_polygon_null_test
+FROM
+  cdb_observatory.OBS_GetPopulation(
+    cdb_observatory._TestArea(), NULL
+  ) As m(obs_getpopulation_polygon_null);
+
 -- Point-based OBS_GetUSCensusMeasure, default normalization (area)
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
   cdb_observatory._testpoint(), 'male population') - 6789.5647735060920500) / 6789.5647735060920500) < 0.001 As obs_getuscensusmeasure_point_male_pop;
@@ -195,6 +239,11 @@ SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
 SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
   cdb_observatory._testarea(), 'male population') - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure;
 
+-- Poly-based OBS_GetUSCensusMeasure, default normalization (none) specified
+-- with NULL
+SELECT (abs(cdb_observatory.obs_getuscensusmeasure(
+  cdb_observatory._testarea(), 'male population', NULL) - 6043.63061042765) / 6043.63061042765) < 0.001 As obs_getuscensusmeasure_null;
+
 -- Point-based OBS_GetUSCensusCategory
 SELECT cdb_observatory.OBS_GetUSCensusCategory(
   cdb_observatory._testpoint(), 'Spielman-Singleton Segments: 10 Clusters') = 'Wealthy, urban without Kids' As OBS_GetUSCensusCategory_point;

src/pg/test/sql/44_observatory_geometries_test.sql

@@ -34,7 +34,7 @@ SELECT cdb_observatory.OBS_GetBoundary(
 -- expect null geometry since there are no census tracts at null island
 -- timespan implictly null
 SELECT cdb_observatory.OBS_GetBoundary(
-  CDB_LatLng(0, 0),
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
   'us.census.tiger.census_tract'
 ) IS NULL As OBS_GetBoundary_null_island_census_tract;
 
@@ -78,7 +78,7 @@ SELECT cdb_observatory.OBS_GetBoundaryId(
 
 -- should give back null since there is not a census tract at null island
 SELECT cdb_observatory.OBS_GetBoundaryId(
-  CDB_LatLng(0, 0),
+  ST_SetSRID(ST_MakePoint(0, 0), 4326),
   'us.census.tiger.census_tract'
 ) IS NULL As OBS_GetBoundaryId_null_island;