From e78b9f603ceec24be8dd10189d436080e1ee3e77 Mon Sep 17 00:00:00 2001 From: csobier Date: Mon, 16 May 2016 14:04:40 -0400 Subject: [PATCH 01/16] added new doc topic, license --- doc/API.md | 1 + doc/license.md | 2 ++ 2 files changed, 3 insertions(+) create mode 100644 doc/license.md diff --git a/doc/API.md b/doc/API.md index 64ce16e..81d9a5b 100644 --- a/doc/API.md +++ b/doc/API.md @@ -8,3 +8,4 @@ This file is for reference purposes only. It is intended for tracking the Data O * [Boundary Functions](boundary_functions.md) * [Discovery Functions](discovery_functions.md) * [Glossary](glossary.md) +* [License](license.md) diff --git a/doc/license.md b/doc/license.md new file mode 100644 index 0000000..0c89292 --- /dev/null +++ b/doc/license.md @@ -0,0 +1,2 @@ +# License + From c4b45606a837acb6744b7422f2a49b49dda51643 Mon Sep 17 00:00:00 2001 From: "Andrew W. Hill" Date: Mon, 16 May 2016 14:42:43 -0400 Subject: [PATCH 02/16] Update license.md updated --- doc/license.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/doc/license.md b/doc/license.md index 0c89292..6ecaeec 100644 --- a/doc/license.md +++ b/doc/license.md @@ -1,2 +1,18 @@ # License +The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broades set of use-cases. Please still review the below terms for any dataset you use and please respect the rights of the owners of each dataset you use. We list below the third-party datasets that are used in the Data Observatory, along with the terms governing the use of each. + +Name | Terms link +-------|--------- +ACS | https://www.usa.gov/government-works +TIGER | https://www.usa.gov/government-works +Zillow Home Value Index | http://www.zillow.com/corp/Terms.htm +Who's on First | http://whosonfirst.mapzen.com#License +GeoNames | http://www.geonames.org/ +GeoPlanet | https://developer.yahoo.com/geo/geoplanet/ +Natural Earth | http://www.naturalearthdata.com/about/terms-of-use/ +Quattroshapes | https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md +Zetashapes | http://zetashapes.com/license +Spielmana & Singleton | http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335 +Instituto Nacional de Estadistica | http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout + From 5d9a12dac0daedba7ae45a18dc8ba0f508ef66da Mon Sep 17 00:00:00 2001 From: "Andrew W. Hill" Date: Mon, 16 May 2016 14:42:59 -0400 Subject: [PATCH 03/16] Update license.md --- doc/license.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/license.md b/doc/license.md index 6ecaeec..58be5d9 100644 --- a/doc/license.md +++ b/doc/license.md @@ -13,6 +13,6 @@ GeoPlanet | https://developer.yahoo.com/geo/geoplanet/ Natural Earth | http://www.naturalearthdata.com/about/terms-of-use/ Quattroshapes | https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md Zetashapes | http://zetashapes.com/license -Spielmana & Singleton | http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335 +Spielman & Singleton | http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335 Instituto Nacional de Estadistica | http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout From 7205676913d097b80385af18b3c7b55e43cb8e1b Mon Sep 17 00:00:00 2001 From: "Andrew W. Hill" Date: Mon, 16 May 2016 15:03:46 -0400 Subject: [PATCH 04/16] Update license.md --- doc/license.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/license.md b/doc/license.md index 58be5d9..9ac8bc6 100644 --- a/doc/license.md +++ b/doc/license.md @@ -1,6 +1,6 @@ # License -The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broades set of use-cases. Please still review the below terms for any dataset you use and please respect the rights of the owners of each dataset you use. We list below the third-party datasets that are used in the Data Observatory, along with the terms governing the use of each. +The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. Please still review the below terms for any dataset you use and please respect the rights of the owners of each dataset. We list below the third-party data sources that are used in the Data Observatory, along with the terms governing the use of each. Name | Terms link -------|--------- From b19e774604a3193023ae5bdf40e78cabe7513205 Mon Sep 17 00:00:00 2001 From: "Andrew W. Hill" Date: Mon, 16 May 2016 15:04:16 -0400 Subject: [PATCH 05/16] Update license.md --- doc/license.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/license.md b/doc/license.md index 9ac8bc6..ef8a45f 100644 --- a/doc/license.md +++ b/doc/license.md @@ -1,6 +1,6 @@ # License -The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. Please still review the below terms for any dataset you use and please respect the rights of the owners of each dataset. We list below the third-party data sources that are used in the Data Observatory, along with the terms governing the use of each. +The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. Please still review the below terms for any dataset you use and please respect the rights of the owners of each dataset. We list below the third-party data sources that are used in the Data Observatory, along with links to the terms governing the use of each. Name | Terms link -------|--------- From 323435fcfa4eb8ede1522a24d694f46826b222db Mon Sep 17 00:00:00 2001 From: csobier Date: Tue, 17 May 2016 11:38:40 -0400 Subject: [PATCH 06/16] updated all urls to point to changed permalinks --- doc/boundary_functions.md | 2 +- doc/discovery_functions.md | 4 ++-- doc/measures_functions.md | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/boundary_functions.md b/doc/boundary_functions.md index 9ba1c63..3a289ce 100644 --- a/doc/boundary_functions.md +++ b/doc/boundary_functions.md @@ -1,6 +1,6 @@ # Boundary Functions -Use the following functions to retrieve [Boundary](/cartodb-platform/dataobservatory/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below. +Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below. ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text) diff --git a/doc/discovery_functions.md b/doc/discovery_functions.md index 1bdf158..797b41c 100644 --- a/doc/discovery_functions.md +++ b/doc/discovery_functions.md @@ -1,6 +1,6 @@ # Discovery Functions -If you are using the [discovery methods](/cartodb-platform/dataobservatory/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/dataobservatory/overview/#boundary-data) and [measures](/cartodb-platform/dataobservatory/overview/#measures-data) data. +If you are using the [discovery methods](/cartodb-platform/data/overview/#discovery-methods) from the Data Observatory, use the following functions to retrieve [boundary](/cartodb-platform/data/overview/#boundary-data) and [measures](/cartodb-platform/data/overview/#measures-data) data. ## OBS_Search(search_term) @@ -47,7 +47,7 @@ A TABLE containing the following properties Key | Description --- | --- -boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/dataobservatory/glossary/#boundary-ids) +boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids) description | a brief description of the boundary dataset timespan | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry. diff --git a/doc/measures_functions.md b/doc/measures_functions.md index 217aa5e..2ad261e 100644 --- a/doc/measures_functions.md +++ b/doc/measures_functions.md @@ -1,6 +1,6 @@ # Measures Functions -[Data Observatory Measures](/cartodb-platform/dataobservatory/overview/#measures-methods) are the numerical location data you can access. The measure Functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (Point or Polygon) and receiving back a measure (an absolute value) for that location. +[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure Functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (Point or Polygon) and receiving back a measure (an absolute value) for that location. There are hundreds of Measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). @@ -15,7 +15,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure Name |Description --- | --- point | a WGS84 point geometry (the_geom) -measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/dataobservatory/glossary/#obsgetuscensusmeasure-names-table). +measure_name | a human readable name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table). normalize | for measures that are are **sums** (e.g. population) the default normalization is 'area' and response comes back as a rate per square kilometer. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](http://cartodb.github.io/bigmetadata/index.html) (optional) boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract') time_span | time span of interest (e.g., 2010 - 2014) @@ -46,7 +46,7 @@ The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure Name |Description --- | --- polygon | a WGS84 polygon geometry (the_geom) -measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/dataobservatory/glossary/#obsgetuscensusmeasure-names-table). +measure_name | a human readable string name of a US Census variable. The list of measure_names is [available in the glossary](/cartodb-platform/data/glossary/#obsgetuscensusmeasure-names-table). normalize | for measures that are **sums** (e.g. population) the default normalization is 'none' and response comes back as a raw value. Other options are 'denominator', which will use the denominator specified in the [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) (optional) boundary_id | source of geometries to pull measure from (e.g., 'us.census.tiger.census_tract') time_span | time span of interest (e.g., 2010 - 2014) From 49d3962a541fbe58e78be4ce067ea3489ca40a5c Mon Sep 17 00:00:00 2001 From: John Krauss Date: Tue, 17 May 2016 12:25:44 -0400 Subject: [PATCH 07/16] better readme for scripts/ --- scripts/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scripts/README.md b/scripts/README.md index 16e0803..3127ea6 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -10,6 +10,10 @@ Python 2.7 should cover you. Virtualenv recommended. ### Execution +Currently, we don't have direct access to the metadata end-to-end. This only +affects the generation of tests. As a stopgap, we have to define a connection +to the test Observatory account. + Run automated tests against a hostname: - (venv) OBS_HOSTNAME= OBS_API_KEY=foobar nosetests autotest.py + (venv) OBS_HOSTNAME= OBS_API_KEY=foobar OBS_META_HOSTNAME= OBS_META_API_KEY= nosetests scripts/autotest.py From 73200df38bd4ccc832654f50e051d291cf2651f3 Mon Sep 17 00:00:00 2001 From: John Krauss Date: Tue, 17 May 2016 13:32:46 -0400 Subject: [PATCH 08/16] simpler access to metadata via observatory.cartodb.com --- scripts/README.md | 2 +- scripts/autotest.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/README.md b/scripts/README.md index 3127ea6..64e1000 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -16,4 +16,4 @@ to the test Observatory account. Run automated tests against a hostname: - (venv) OBS_HOSTNAME= OBS_API_KEY=foobar OBS_META_HOSTNAME= OBS_META_API_KEY= nosetests scripts/autotest.py + (venv) OBS_HOSTNAME= OBS_API_KEY= OBS_META_HOSTNAME=observatory.cartodb.com OBS_META_API_KEY= nosetests scripts/autotest.py diff --git a/scripts/autotest.py b/scripts/autotest.py index 622686d..9c2105e 100644 --- a/scripts/autotest.py +++ b/scripts/autotest.py @@ -25,19 +25,19 @@ def query(q, is_meta=False, **options): return requests.get(url, params=params) MEASURE_COLUMNS = [(r['id'], ) for r in query(''' -SELECT id FROM observatory.obs_column +SELECT id FROM obs_column WHERE type ILIKE 'numeric' AND weight > 0 ''', is_meta=True).json()['rows']] CATEGORY_COLUMNS = [(r['id'], ) for r in query(''' -SELECT id FROM observatory.obs_column +SELECT id FROM obs_column WHERE type ILIKE 'text' AND weight > 0 ''', is_meta=True).json()['rows']] BOUNDARY_COLUMNS = [(r['id'], ) for r in query(''' -SELECT id FROM observatory.obs_column +SELECT id FROM obs_column WHERE type ILIKE 'geometry' AND weight > 0 ''', is_meta=True).json()['rows']] From a49be9506f0bf22e1d5cb67049408a3ba3e91ee2 Mon Sep 17 00:00:00 2001 From: andrewxhill Date: Tue, 17 May 2016 14:42:55 -0400 Subject: [PATCH 09/16] search updated --- doc/discovery_functions.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/discovery_functions.md b/doc/discovery_functions.md index 797b41c..406886e 100644 --- a/doc/discovery_functions.md +++ b/doc/discovery_functions.md @@ -19,16 +19,16 @@ A TABLE containing the following properties Key | Description --- | --- -measure_id | the unique id of the measure for use with the ```OBS_GetMeasure``` function +id | the unique id of the measure for use with the ```OBS_GetMeasure``` function name | the human readable name of the measure description | a brief description of the measure -aggregate_type | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value) -sources | where the data came from (e.g. US Census Bureau) +aggregate | **sum** are raw count values, **median** are statistical medians, **average** are statistical averages, **undefined** other (e.g. an index value) +source | where the data came from (e.g. US Census Bureau) #### Example ```SQL -SELECT * FROM OBS_Search('inequality') +SELECT * FROM OBS_Search('home value') ``` ## OBS_GetAvailableBoundaries(point_geometry) From bb78ea7fe5ed71dfc5a64a8cdb1407a54affb2d4 Mon Sep 17 00:00:00 2001 From: andrewxhill Date: Tue, 17 May 2016 15:02:14 -0400 Subject: [PATCH 10/16] updated bounary functions --- doc/boundary_functions.md | 21 +++++++++++---------- doc/discovery_functions.md | 2 +- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/doc/boundary_functions.md b/doc/boundary_functions.md index 3a289ce..66283db 100644 --- a/doc/boundary_functions.md +++ b/doc/boundary_functions.md @@ -28,10 +28,10 @@ If geometries are not found for the requested `polygon`, `geometry_id`, `timespa #### Example -Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geoid` (text). +Insert all Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geom_refs` (text). ```sql -INSERT INTO manhattan_census_tracts(the_geom, geoid) +INSERT INTO manhattan_census_tracts(the_geom, geom_refs) SELECT * FROM OBS_GetBoundariesByGeometry( ST_MakeEnvelope(-74.0251922607,40.6945658517, @@ -67,14 +67,14 @@ Column Name | Description the_geom | a point geometry on a boundary (e.g., a point that lies on a US Census tract) geom_refs| a string identifier for the geometry (e.g., the geoid of a US Census tract) -If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then null values are returned. +If geometries are not found for the requested geometry, `geometry_id`, `timespan`, or `overlap_type`, then NULL values are returned. #### Example -Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_census_tracts` which has columns `the_geom` (geometry) and `geoid` (text). +Insert points that lie on Census Tracts from Lower Manhattan and nearby areas within the supplied bounding box to a table named `manhattan_tract_points` which has columns `the_geom` (geometry) and `geom_refs` (text). ```sql -INSERT INTO manhattan_census_tract_points(the_geom, geoid) +INSERT INTO manhattan_tract_points (the_geom, geom_refs) SELECT * FROM OBS_GetPointsByGeometry( ST_MakeEnvelope(-74.0251922607,40.6945658517, @@ -175,9 +175,10 @@ geom | a WGS84 polygon geometry #### Example -Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to. +Use a table of `geometry_id`s (e.g., geoid from the U.S. Census) to select the unique boundaries that they correspond to and insert into a table called, `overlapping_polygons`. This is a useful method for creating new choropleths of aggregate data. ```SQL +INSERT INTO overlapping_polygons (the_geom, geometry_id, point_count) SELECT OBS_GetBoundaryById(geometry_id, 'us.census.tiger.county') As the_geom, geometry_id, @@ -213,10 +214,10 @@ If geometries are not found for the requested point and radius, `geometry_id`, ` #### Example -Insert into table `denver_census_tracts` the census tract boundaries and geoids of census tracts which intersect within 10 miles of downtown Denver, Colorado. +Insert into table `denver_census_tracts` the census tract boundaries and geom_refs of census tracts which intersect within 10 miles of downtown Denver, Colorado. ```sql -INSERT INTO denver_census_tracts(the_geom, geoid) +INSERT INTO denver_census_tracts(the_geom, geom_refs) SELECT * FROM OBS_GetBoundariesByPointAndRadius( CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado @@ -255,10 +256,10 @@ If geometries are not found for the requested point and radius, `geometry_id`, ` #### Example -Insert into table `denver_census_tracts` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA. +Insert into table `denver_tract_points` points on US census tracts and their corresponding geoids for census tracts which intersect within 10 miles of downtown Denver, Colorado, USA. ```sql -INSERT INTO denver_census_tracts(the_geom, geoid) +INSERT INTO denver_tract_points(the_geom, geom_refs) SELECT * FROM OBS_GetPointsByPointAndRadius( CDB_LatLng(39.7392, -104.9903), -- Denver, Colorado diff --git a/doc/discovery_functions.md b/doc/discovery_functions.md index 406886e..9b23589 100644 --- a/doc/discovery_functions.md +++ b/doc/discovery_functions.md @@ -49,7 +49,7 @@ Key | Description --- | --- boundary_id | a boundary identifier from the [boundary ID glossary](/cartodb-platform/data/glossary/#boundary-ids) description | a brief description of the boundary dataset -timespan | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry. +time_span | the timespan attached the boundary. this does not mean that the boundary is invalid outside of the timespan, but is the explicit timespan published with the geometry. #### Example From 54d0a6319c594abca6dfa4904d3cb404b80e1577 Mon Sep 17 00:00:00 2001 From: andrewxhill Date: Tue, 17 May 2016 15:12:05 -0400 Subject: [PATCH 11/16] updated measures doc --- doc/measures_functions.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/measures_functions.md b/doc/measures_functions.md index 2ad261e..4596c1f 100644 --- a/doc/measures_functions.md +++ b/doc/measures_functions.md @@ -1,14 +1,14 @@ # Measures Functions -[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure Functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (Point or Polygon) and receiving back a measure (an absolute value) for that location. +[Data Observatory Measures](/cartodb-platform/data/overview/#measures-methods) are the numerical location data you can access. The measure functions allow you to access individual measures to augment your own data or integrate in your analysis workflows. Measures are used by sending an identifier or a geometry (point or polygon) and receiving back a measure (an absolute value) for that location. -There are hundreds of Measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). +There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). -We show here how you can access Measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We encourage you to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT). +We show here how you can access measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We encourage you to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT). ## OBS_GetUSCensusMeasure(point geometry, measure_name text); -The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all Measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetMeasure``` function below. +The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below. #### Arguments @@ -33,8 +33,8 @@ value | the raw or normalized measure Add a measure to an empty numeric column based on point locations in your table. ```SQL -UPDATE tablename -SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population') +UPDATE tablename +SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population') ``` ## OBS_GetUSCensusMeasure(polygon geometry, measure_name text); @@ -96,7 +96,7 @@ Add a measure to an empty numeric column based on point locations in your table ```SQL UPDATE tablename -SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B08134006') +SET median_home_value_sqft = OBS_GetMeasure(the_geom, 'us.zillow.AllHomes_MedianValuePerSqft') ``` ## OBS_GetMeasure(polygon geometry, measure_id text); @@ -126,8 +126,8 @@ value | the raw or normalized measure Add a measure to an empty column based on polygons in your table ```SQL -UPDATE tablename -SET local_male_population = OBS_GetMeasure(the_geom, 'us.census.acs.B08134006') +UPDATE tablename +SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001') ``` #### Errors @@ -158,6 +158,6 @@ value | a text based category found at the supplied point Add the Category to an empty column text column based on point locations in your table ```SQL -UPDATE tablename +UPDATE tablename SET segmentation = OBS_GetCategory(the_geom, 'us.census.spielman_singleton_segments.X55') ``` From 0848d5aee47723373cb6b444914ed770220724a8 Mon Sep 17 00:00:00 2001 From: andrewxhill Date: Tue, 17 May 2016 15:37:13 -0400 Subject: [PATCH 12/16] remove ; --- doc/measures_functions.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/measures_functions.md b/doc/measures_functions.md index 4596c1f..6eb0a9c 100644 --- a/doc/measures_functions.md +++ b/doc/measures_functions.md @@ -6,7 +6,7 @@ There are hundreds of measures and the list is growing with each release. You ca We show here how you can access measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We encourage you to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT). -## OBS_GetUSCensusMeasure(point geometry, measure_name text); +## OBS_GetUSCensusMeasure(point geometry, measure_name text) The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables at a point location. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use measure IDs with the ```OBS_GetMeasure``` function below. @@ -37,7 +37,7 @@ UPDATE tablename SET total_population = OBS_GetUSCensusMeasure(the_geom, 'Total Population') ``` -## OBS_GetUSCensusMeasure(polygon geometry, measure_name text); +## OBS_GetUSCensusMeasure(polygon geometry, measure_name text) The ```OBS_GetUSCensusMeasure(point, measure_name)``` function returns a measure based on a subset of the US Census variables within a given polygon. The ```OBS_GetUSCensusMeasure``` function is limited to only a subset of all measures that are available in the Data Observatory, to access the full list, use the ```OBS_GetUSCensusMeasure``` function below. @@ -68,7 +68,7 @@ UPDATE tablename SET local_male_population = OBS_GetUSCensusMeasure(the_geom, 'Male Population') ``` -## OBS_GetMeasure(point geometry, measure_id text); +## OBS_GetMeasure(point geometry, measure_id text) The ```OBS_GetMeasure(point, measure_id)``` function returns any Data Observatory measure at a point location. You can browse all available Measures in the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf)). @@ -99,7 +99,7 @@ UPDATE tablename SET median_home_value_sqft = OBS_GetMeasure(the_geom, 'us.zillow.AllHomes_MedianValuePerSqft') ``` -## OBS_GetMeasure(polygon geometry, measure_id text); +## OBS_GetMeasure(polygon geometry, measure_id text) The ```OBS_GetMeasure(polygon, measure_id)``` function returns any Data Observatory measure calculated within a polygon. @@ -134,7 +134,7 @@ SET household_count = OBS_GetMeasure(the_geom, 'us.census.acs.B11001001') * If an unrecognized normalization type is input, raise an error: `'Only valid inputs for "normalize" are "area" (default) and "denominator".` -## OBS_GetCategory(point geometry, category_id text); +## OBS_GetCategory(point geometry, category_id text) The ```OBS_GetCategory(point, category_id)``` function returns any Data Observatory Category value at a point location. The Categories available are currently limited to Segmentation categories. See the Segmentation section of the [Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf) for more detail. From fdfab967af91dfdb2d4a5838a8a687b29aa007b1 Mon Sep 17 00:00:00 2001 From: andrewxhill Date: Tue, 17 May 2016 15:38:19 -0400 Subject: [PATCH 13/16] updated with none in boundary table --- doc/glossary.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/glossary.md b/doc/glossary.md index 982e8d2..3b5d071 100644 --- a/doc/glossary.md +++ b/doc/glossary.md @@ -1,6 +1,6 @@ # Glossary -A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses. +A list of boundary ids and measure_names for Data Observatory functions. For US based boundaries, the Shoreline Clipped version provides a high-quality shoreline clipping for mapping uses. ## Boundary IDs @@ -18,12 +18,12 @@ US Census Blocks | us.census.tiger.block | us.census.tiger.block_clipped US Census Block Groups | us.census.tiger.block_group | us.census.tiger.block_group_clipped US Census PUMAs | us.census.tiger.puma | us.census.tiger.puma_clipped US Incorporated Places | us.census.tiger.place | us.census.tiger.place_clipped -ES Sección Censal | es.ine.geom | -Regions (First-level Administrative) | whosonfirst.wof_region_geom | -Continents | whosonfirst.wof_continent_geom | -Countries | whosonfirst.wof_country_geom | -Marine Areas | whosonfirst.wof_marinearea_geom | -Disputed Areas | whosonfirst.wof_disputed_geom | +ES Sección Censal | es.ine.geom | none +Regions (First-level Administrative) | whosonfirst.wof_region_geom | none +Continents | whosonfirst.wof_continent_geom | none +Countries | whosonfirst.wof_country_geom | none +Marine Areas | whosonfirst.wof_marinearea_geom | none +Disputed Areas | whosonfirst.wof_disputed_geom | none @@ -67,7 +67,7 @@ Number of workers with a commute between 25 and 29 minutes | The number of w Number of workers with a commute between 30 and 34 minutes | The number of workers over the age of 16 who do not work from home and commute in between 30 and 34 minutes in a geographic area. Number of workers with a commute between 45 and 59 minutes | The number of workers over the age of 16 who do not work from home and commute in between 45 and 59 minutes in a geographic area. Children under 18 Years of Age | The number of people within each geography who are under 18 years of age. -Households | A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people. +Households | A count of the number of households in each geography. A household consists of one or more people who live in the same dwelling and also share at meals or living accommodation, and may consist of a single family or some other grouping of people. Population 15 Years and Over | The number of people in a geographic area who are over the age of 15. This is used mostly as a denominator of marital status. Never Married | The number of people in a geographic area who have never been married. Currently married | The number of people in a geographic area who are currently married. From 7f34c5e0e93ea6623d0c0f649523201eee41447b Mon Sep 17 00:00:00 2001 From: csobier Date: Tue, 17 May 2016 16:42:20 -0400 Subject: [PATCH 14/16] converted license links to workable hyperlinks --- doc/license.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/doc/license.md b/doc/license.md index ef8a45f..738271e 100644 --- a/doc/license.md +++ b/doc/license.md @@ -1,18 +1,18 @@ # License -The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. Please still review the below terms for any dataset you use and please respect the rights of the owners of each dataset. We list below the third-party data sources that are used in the Data Observatory, along with links to the terms governing the use of each. +The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. Please **still** review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use. Name | Terms link -------|--------- -ACS | https://www.usa.gov/government-works -TIGER | https://www.usa.gov/government-works -Zillow Home Value Index | http://www.zillow.com/corp/Terms.htm -Who's on First | http://whosonfirst.mapzen.com#License -GeoNames | http://www.geonames.org/ -GeoPlanet | https://developer.yahoo.com/geo/geoplanet/ -Natural Earth | http://www.naturalearthdata.com/about/terms-of-use/ -Quattroshapes | https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md -Zetashapes | http://zetashapes.com/license -Spielman & Singleton | http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335 -Instituto Nacional de Estadistica | http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout +ACS | [https://www.usa.gov/government-works](https://www.usa.gov/government-works) +TIGER | [https://www.usa.gov/government-works](https://www.usa.gov/government-works) +Zillow Home Value Index | [http://www.zillow.com/corp/Terms.htm](http://www.zillow.com/corp/Terms.htm) +Who's on First | [http://whosonfirst.mapzen.com#License](http://whosonfirst.mapzen.com#License) +GeoNames | [http://www.geonames.org/](http://www.geonames.org/) +GeoPlanet | [https://developer.yahoo.com/geo/geoplanet/](https://developer.yahoo.com/geo/geoplanet/) +Natural Earth | [http://www.naturalearthdata.com/about/terms-of-use/](http://www.naturalearthdata.com/about/terms-of-use/) +Quattroshapes | [https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md](https://github.com/foursquare/quattroshapes/blob/master/LICENSE.md) +Zetashapes | [http://zetashapes.com/license](http://zetashapes.com/license) +Spielman & Singleton | [http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335](http://www.tandfonline.com/doi/full/10.1080/00045608.2015.1052335) +Instituto Nacional de Estadistica | [http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout](http://www.ine.es/ss/Satellite?L=0&c=Page&cid=1254735849170&p=1254735849170&pagename=Ayuda%2FINELayout) From eb5bcb2f18a7ff6e7e44200d91a3bbe9b2d70aa5 Mon Sep 17 00:00:00 2001 From: csobier Date: Tue, 17 May 2016 17:13:30 -0400 Subject: [PATCH 15/16] wordsmithed --- doc/license.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/license.md b/doc/license.md index 738271e..8984f3d 100644 --- a/doc/license.md +++ b/doc/license.md @@ -1,6 +1,6 @@ # License -The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. Please **still** review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use. +The Data Observatory is a collection of various sources of data with varying licenses. We have worked hard to find you data that will work for the broadest set of use-cases. For competency, please still review the terms for any dataset you use and respect the rights of the owners for each dataset. The following third-party data sources are used in the Data Observatory, and we have included the links to the terms governing their use. Name | Terms link -------|--------- From 51705eb411754a789f3e6675fe209f84ca81ad8d Mon Sep 17 00:00:00 2001 From: csobier Date: Tue, 17 May 2016 18:26:45 -0400 Subject: [PATCH 16/16] added hyperlinks to best practices section --- doc/boundary_functions.md | 2 ++ doc/measures_functions.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/boundary_functions.md b/doc/boundary_functions.md index 66283db..8b614f1 100644 --- a/doc/boundary_functions.md +++ b/doc/boundary_functions.md @@ -2,6 +2,8 @@ Use the following functions to retrieve [Boundary](/cartodb-platform/data/overview/#boundary-data) data. Data ranges from small areas (e.g. US Census Block Groups) to large areas (e.g. Countries). You can access boundaries by point location lookup, bounding box lookup, direct ID access and several other methods described below. +You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) boundaries through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT). + ## OBS_GetBoundariesByGeometry(polygon geometry, geometry_id text) The ```OBS_GetBoundariesByGeometry(geometry, geometry_id)``` method returns a set of boundary geometries that intersect a supplied geometry. This can be used to find all boundaries that are within or overlap a bounding box. You have the ability to choose whether to retrieve all boundaries that intersect your supplied bounding box or only those that fall entirely inside of your bounding box. diff --git a/doc/measures_functions.md b/doc/measures_functions.md index 6eb0a9c..a07ad93 100644 --- a/doc/measures_functions.md +++ b/doc/measures_functions.md @@ -4,7 +4,7 @@ There are hundreds of measures and the list is growing with each release. You can currently discover and learn about measures contained in the Data Observatory by downloading our [Data Catalog](https://cartodb.github.io/bigmetadata/observatory.pdf). -We show here how you can access measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We encourage you to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT). +You can [access](/cartodb-platform/data/accessing/#accessing-the-data-observatory) measures through the CartoDB Editor. The same methods will work if you are using the CartoDB Platform to develop your application. We [encourage you](/cartodb-platform/data/accessing/#best-practices) to use table modifying methods (UPDATE and INSERT) over dynamic methods (SELECT). ## OBS_GetUSCensusMeasure(point geometry, measure_name text)