Skip to content

Commit cfbb473

Browse files
FilFil
authored
US Census Bureau shapefiles example (#1496)
* US Census Bureau shapefiles example * Update examples/loader-census/src/data/ca.json.sh * Apply suggestions from code review
1 parent 831ec8a commit cfbb473

File tree

8 files changed

+111
-0
lines changed

8 files changed

+111
-0
lines changed

‎examples/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454

5555
- [`loader-airtable`](https://observablehq.observablehq.cloud/framework-example-loader-airtable/) - Loading data from Airtable
5656
- [`loader-arrow`](https://observablehq.observablehq.cloud/framework-example-loader-arrow/) - Generating Apache Arrow IPC files
57+
- [`loader-census`](https://observablehq.observablehq.cloud/framework-example-loader-census/) - Loading and transform shapefiles from the U.S. Census Bureau
5758
- [`loader-databricks`](https://observablehq.observablehq.cloud/framework-example-loader-databricks/) - Loading data from Databricks
5859
- [`loader-duckdb`](https://observablehq.observablehq.cloud/framework-example-loader-duckdb/) - Processing data with DuckDB
5960
- [`loader-elasticsearch`](https://observablehq.observablehq.cloud/framework-example-loader-elasticsearch/) - Loading data from Elasticsearch

‎examples/loader-census/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.DS_Store
2+
/dist/
3+
node_modules/
4+
yarn-error.log

‎examples/loader-census/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[Framework examples →](../)
2+
3+
# U.S. Census Bureau data loader
4+
5+
View live: <https://observablehq.observablehq.cloud/framework-example-loader-census/>
6+
7+
This Observable Framework example demonstrates how to write a bash data loader that downloads high-resolution shapefiles from the US Census Bureau, and converts them into a web-friendly format.

‎examples/loader-census/observablehq.config.js‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
export default {
2+
root: "src"
3+
};

‎examples/loader-census/package.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"type": "module",
3+
"private": true,
4+
"scripts": {
5+
"clean": "rimraf src/.observablehq/cache",
6+
"build": "rimraf dist && observable build",
7+
"dev": "observable preview",
8+
"deploy": "observable deploy",
9+
"observable": "observable"
10+
},
11+
"dependencies": {
12+
"@observablehq/framework": "latest",
13+
"shapefile": "^0.6.6",
14+
"topojson-client": "^3.1.0",
15+
"topojson-server": "^3.0.1",
16+
"topojson-simplify": "^3.0.3"
17+
},
18+
"devDependencies": {
19+
"rimraf": "^5.0.5"
20+
},
21+
"engines": {
22+
"node": ">=18"
23+
}
24+
}

‎examples/loader-census/src/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/.observablehq/cache/

‎examples/loader-census/src/data/ca.json.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Download the ZIP archive from the Census Bureau (if needed).
2+
if [ ! -f src/.observablehq/cache/cb_2023_06_cousub_500k.zip ]; then
3+
curl -o src/.observablehq/cache/cb_2023_06_cousub_500k.zip 'https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_06_cousub_500k.zip'
4+
fi
5+
6+
# Unzip the ZIP archive to extract the shapefile.
7+
unzip -oqd src/.observablehq/cache src/.observablehq/cache/cb_2023_06_cousub_500k.zip
8+
9+
# Convert the shapefile to TopoJSON, simplify, and merge counties.
10+
shp2json --encoding utf-8 -n src/.observablehq/cache/cb_2023_06_cousub_500k.shp > src/.observablehq/cache/cb_2023_06_cousub_500k.ndjson
11+
12+
geo2topo -q 1e5 -n counties=src/.observablehq/cache/cb_2023_06_cousub_500k.ndjson \
13+
| toposimplify -f -s 1e-7 \
14+
| topomerge state=counties

‎examples/loader-census/src/index.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Census boundaries
2+
3+
Here’s how to fetch high-resolution shapefiles from the [U.S. Census Bureau](https://www.census.gov/geographies/mapping-files/time-series/geo/cartographic-boundary.html) and convert them into a web-friendly format in a bash data loader.
4+
5+
First, you’ll need to install a few packages:
6+
7+
```sh
8+
npm install shapefile topojson-client topojson-server topojson-simplify
9+
```
10+
11+
Next, here’s a bash script, `ca.json.sh`:
12+
13+
```bash
14+
# Download the ZIP archive from the Census Bureau (if needed).
15+
if [ ! -f src/.observablehq/cache/cb_2023_06_cousub_500k.zip ]; then
16+
curl -o src/.observablehq/cache/cb_2023_06_cousub_500k.zip 'https://www2.census.gov/geo/tiger/GENZ2023/shp/cb_2023_06_cousub_500k.zip'
17+
fi
18+
19+
# Unzip the ZIP archive to extract the shapefile.
20+
unzip -oqd src/.observablehq/cache src/.observablehq/cache/cb_2023_06_cousub_500k.zip
21+
22+
# Convert the shapefile to GeoJSON, then to TopoJSON, simplify, and merge counties.
23+
shp2json --encoding utf-8 -n src/.observablehq/cache/cb_2023_06_cousub_500k.shp > src/.observablehq/cache/cb_2023_06_cousub_500k.ndjson
24+
geo2topo -q 1e5 -n counties=src/.observablehq/cache/cb_2023_06_cousub_500k.ndjson \
25+
| toposimplify -f -s 1e-7 \
26+
| topomerge state=counties
27+
```
28+
29+
The census.gov URL comes from the Census Bureau page linked above. Here “06” is the state of California’s FIPS code, and “cousub” means county subdivisions.
30+
31+
And here’s the result displayed with Plot:
32+
33+
```js echo
34+
Plot.plot({
35+
width: 640,
36+
height: 720,
37+
projection: {
38+
type: "conic-conformal",
39+
parallels: [37 + 4 / 60, 38 + 26 / 60],
40+
rotate: [120 + 30 / 60, 0],
41+
domain: castate
42+
},
43+
marks: [
44+
Plot.geo(castate),
45+
Plot.geo(cacounties, {strokeOpacity: 0.2})
46+
]
47+
})
48+
```
49+
50+
```js echo
51+
const ca = FileAttachment("data/ca.json").json();
52+
```
53+
54+
```js echo
55+
const castate = topojson.feature(ca, ca.objects.state);
56+
const cacounties = topojson.mesh(ca, ca.objects.counties, (a, b) => a !== b);
57+
```

0 commit comments

Comments
 (0)