Audit inequality observations and their years

Goal and how to run

This lesson assumes basic Python knowledge. Read each method and caption before changing parameters. AI exercises are optional; no AI account is needed for the core workflow. Data lives in notebook metadata, so there is no companion data download. Keep the notebook saved in the active working folder.

Your goal is to understand the research question, execute transparent calculations, check the results, and state the limits of the conclusion.

Setup and parameters

from pathlib import Path
import base64, gzip, hashlib, json, platform
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, HTML
from IPython.display import display as _display
def display(value):
    if isinstance(value,pd.DataFrame):
        table=value.to_html(max_rows=12,escape=True)
        style="<style>.vibeit-readable-table{max-width:100%;overflow-x:auto}.vibeit-readable-table table{width:max-content;max-width:none;border-collapse:collapse}.vibeit-readable-table td,.vibeit-readable-table th{white-space:nowrap!important;word-break:normal!important;overflow-wrap:normal!important}</style>"
        return _display(HTML(style+'<div class="vibeit-readable-table">'+table+'</div>'))
    return _display(value)
RECIPE_ID='ec03-inequality'
OUTPUT_DIR=Path.cwd()/"results"/RECIPE_ID
plt.rcParams.update({"figure.dpi":120,"font.size":10,"axes.spines.top":False,"axes.spines.right":False,
                      "axes.prop_cycle":plt.cycler(color=["#18785f","#416aa6","#bc6541","#9974a5"])})
print("Python:",platform.python_version(),"; NumPy:",np.__version__,"; Pandas:",pd.__version__)
print("Working folder:",Path.cwd())
print("Core data mode: embedded snapshot; no network requests")

Python: 3.13.13 ; NumPy: 2.5.3 ; Pandas: 3.0.6
Working folder: /Users/zhiluo/Developer/vibeit-cookbook/cookbook/downloads
Core data mode: embedded snapshot; no network requests

Load the embedded snapshot

def load_snapshot(recipe_id, expected_sha256, snapshot_key):
    """Read embedded data from this notebook, including a renamed copy."""
    for path in sorted(Path.cwd().glob("*.ipynb")):
        try:
            notebook = json.loads(path.read_text(encoding="utf-8"))
            metadata = notebook.get("metadata", {}).get("vibeit_cookbook", {})
            blob = metadata.get("snapshots", {}).get(snapshot_key)
            if metadata.get("recipe_id") != recipe_id or not blob:
                continue
            if blob.get("sha256") != expected_sha256:
                continue
            raw = gzip.decompress(base64.b64decode(blob["gzip_base64"], validate=True))
            if hashlib.sha256(raw).hexdigest() != expected_sha256:
                raise ValueError("Embedded snapshot checksum mismatch: " + snapshot_key)
            return json.loads(raw)
        except (OSError, json.JSONDecodeError):
            continue
    raise FileNotFoundError(
        "Save/open the complete .ipynb in the active working folder. "
        "The embedded snapshot could not be located for " + recipe_id)

SNAPSHOT_HASHES={'worldbank': 'd45779250297fb9371748b85b81321b2ae88f93ecf70cfb88256647a68e54376'}
def snapshot(key):
    return load_snapshot(RECIPE_ID,SNAPSHOT_HASHES[key],key)
print("Embedded snapshots:",list(SNAPSHOT_HASHES))

Embedded snapshots: ['worldbank']

Read the method functions

def worldbank_panel(payload):
    """Preserve missing observations; official economy metadata excludes aggregates."""
    economies=pd.DataFrame(payload['economies']).set_index('iso3')
    observations=pd.DataFrame(payload['observations'])
    observations=observations[observations.iso3.isin(economies.index)].copy()
    if observations.duplicated(['iso3','year','indicator']).any():raise ValueError('Duplicate economy/year/indicator observations')
    observations['value']=pd.to_numeric(observations.value,errors='raise')
    wide=observations.pivot(index=['iso3','year'],columns='indicator',values='value').sort_index()
    return wide,economies,observations

print("Method ready: worldbank_panel")

Method ready: worldbank_panel

Data and model boundary

World Bank records retain definitions, missing observations and reference years. The six economies are declared examples, not a random global sample. Country averages, price-basket conversions and Gini indices describe different quantities; survey concepts and coverage can differ.

Steps

1. Preserve the observation year

SI.POV.GINI is an index on 0–100. Indicator-reference years, income/consumption concepts and population coverage differ. Missing observations must remain missing.

wb=snapshot("worldbank");panel,economies,observations=worldbank_panel(wb)
countries=["CHN","USA","DEU","BRA","IND","ZAF"]
assert set(countries)<=set(economies.index)
print("Economy/year records:",len(panel),"; official aggregates excluded; missing values preserved")
display(economies.loc[countries,["name","region","income_group"]])
gini=panel["SI.POV.GINI"].unstack("iso3").reindex(index=range(2010,2025),columns=countries)
observed=gini.stack().dropna().rename("Gini_index").reset_index()
assert observed.Gini_index.between(0,100).all();display(observed.head(8))

Economy/year records: 3255 ; official aggregates excluded; missing values preserved

2. Display actual reported values

Markers represent observations. An absent year is not zero or unchanged inequality. We avoid interpolation-like connecting lines.

In [5] · Python

fig,ax=plt.subplots(figsize=(9,5))
for code in countries:ax.plot(gini.index,gini[code],"o",label=economies.loc[code,"name"])
ax.set(xlabel="Indicator-reference year",ylabel="Gini index (0–100)",title="Reported values without imputation");ax.legend(fontsize=8);plt.tight_layout();plt.show()

3. Date the last available values

A fixed economy order with explicit years shows availability. Different vintages are not a synchronized global ranking. Economies lacking observations stay empty.

In [6] · Python

latest=observed.sort_values("year").groupby("iso3").tail(1).set_index("iso3").reindex(countries)
display(latest)
labels=[economies.loc[c,"name"]+" ("+(str(int(latest.loc[c,"year"])) if pd.notna(latest.loc[c,"year"]) else "no observation")+")" for c in countries]
fig,ax=plt.subplots(figsize=(9,4));ax.barh(labels,latest.Gini_index)
ax.set(xlabel="Reported Gini index (0–100)",title="Last available 2010–2024 values; dates differ");plt.tight_layout();plt.show()

	year	Gini_index
iso3
CHN	2022	36.0
USA	2024	41.8
DEU	2022	33.7
BRA	2024	50.3
IND	2023	23.0
ZAF	2022	54.1

4. Audit the availability matrix

Binary color says whether a source observation exists, not what the absent value would be. Common-year and latest-value selections answer different questions.

In [7] · Python

fig,ax=plt.subplots(figsize=(9,4));im=ax.imshow(gini.notna().T,aspect="auto",cmap="Blues",vmin=0,vmax=1)
ax.set_xticks(range(15),range(2010,2025),rotation=45);ax.set_yticks(range(6),[economies.loc[c,"name"] for c in countries])
ax.set(xlabel="Indicator year",title="Coverage: 1=reported, 0=missing");fig.colorbar(im,ax=ax,ticks=[0,1]);plt.tight_layout();plt.show()

5. Reconcile dates and export complete coverage

Count observed cells directly and check the chosen years against source availability. Empty selections are explicit; no years are artificially synchronized.

assert len(observed)==int(gini.notna().to_numpy().sum())
for code in countries:
 available=gini[code].dropna()
 if not available.empty:assert latest.loc[code,"year"]==available.index.max()
OUTPUT_DIR.mkdir(parents=True,exist_ok=True);gini.to_csv(OUTPUT_DIR/"Gini-with-missing-years.csv")
latest.to_csv(OUTPUT_DIR/"dated-last-observations.csv")
print("Source count/year selection reconciled:",len(observed),"observations")

Source count/year selection reconciled: 61 observations

Exercises and reference explanation

1. Why show each last-value year?
2. Can one Gini reconstruct a full distribution?

Reference explanation. Years and concepts differ. One Gini does not uniquely identify a Lorenz curve or income/consumption distribution.

Keep a copy before changing parameters. Compare old and new figures and record which inputs, calculations and conclusions changed.

Optional AI coding exercises

Prompt 1

Using NumPy/Pandas/Matplotlib, restrict to a common observed year and report exclusions.

Prompt 2

With the same packages, count available/missing years without imputation.

Human acceptance. Retain original reference years and 0–100 index units; explicitly report empty selections.

Review the edited source, then manually run affected cells and subsequent checks. Ask the assistant to explain the purpose, packages and data provenance of its changes, and reconcile that explanation with actual output.

Optional online extension

RUN_ONLINE_UPDATE=False
if RUN_ONLINE_UPDATE:
    import requests
    try:
        response=requests.get('https://api.worldbank.org/v2/indicator/NY.GDP.PCAP.KD?format=json',timeout=20)
        response.raise_for_status()
        print("Source:",response.url)
        print(response.text[:700])
    except requests.RequestException as error:
        print("Online extension unavailable:",type(error).__name__,str(error)[:160])
else:
    print("Online extension skipped; embedded core data unchanged")

Online extension skipped; embedded core data unchanged

Sources, snapshots and next steps

• World Bank economy metadata — World Bank CC BY 4.0 with additional terms; retain original indicator attribution; retrieved 2026-10-09. SHA-256 c9acf3932785191a649cbc04c3056af4cf64a236e5c3b49d49be25ddb0440bce. Preserve economy IDs, region and income-group metadata; exclude aggregates in student analyses.

• World Bank NY.GDP.PCAP.KD — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 59a9efaf20ab359c7832f24da45dda2282511c0eaf927d044583f47d08e02cda. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata NY.GDP.PCAP.KD — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 46689488cd435920647264ede1bfb2d4b51b09800fb36f3d3b1422a0b11ea49d. Preserve indicator definition, source organization and source note.

• World Bank NY.GDP.MKTP.KD — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 466f4e3176b509613f3df99acffce89b5f1ea14472c376dd10d8b8f67f6172b0. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata NY.GDP.MKTP.KD — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 bc4bc0ffec1755410f52b1091c6b1f46917c7417977102350d3d1102b4659216. Preserve indicator definition, source organization and source note.

• World Bank SP.POP.TOTL — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 7ecf8d749dbddd87a269f2019e6d42ed32875c57565cf5c5af097c6e6c9af528. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata SP.POP.TOTL — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 2f8491efc00bf2c0995112ff8e87aba69e5e9d22d0880b22f7e3a4ec757ba05c. Preserve indicator definition, source organization and source note.

• World Bank SP.DYN.LE00.IN — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 d8b5c6bd7fecaf8cee0e422444719ba1f42ec70996308bc74afd14ee48799055. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata SP.DYN.LE00.IN — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 a39cb6fddfbbb953053690366243dffe77a266c4deae79c46ce5a06da9f5fc31. Preserve indicator definition, source organization and source note.

• World Bank FP.CPI.TOTL — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 55c3b71ad4dd8c92c15fa7dc2b502a3d60f26cf47ae9a24e384bc2ff6108ff56. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata FP.CPI.TOTL — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 13b1e3c4b2b72f7fb3a630b4dde37f0e68e6d5e78dc06b2437dfea0c0b93af46. Preserve indicator definition, source organization and source note.

• World Bank FP.CPI.TOTL.ZG — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 849b6a88a84a7efdeb92d89e8db1db36f86543e28ff8eb645e0feaa8c634def7. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata FP.CPI.TOTL.ZG — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 df9371a3b6c01e55a3f28991e62e1b1ebbe0cd2a7da94456be9938c7a2962233. Preserve indicator definition, source organization and source note.

• World Bank SI.POV.GINI — World Bank CC BY 4.0 with additional terms; original sources retained in indicator metadata; retrieved 2026-10-09. SHA-256 4919413d3b9ef06b7445c3fd42d47b8a09e9836bbab7be84e4993099ffdc7ab2. Retain official observations including null values and footnotes, 2010–2024; no interpolation; no aggregate-country mixing.

• World Bank indicator metadata SI.POV.GINI — World Bank CC BY 4.0 with additional terms; retrieved 2026-10-09. SHA-256 e377bf82b38024748153880ff90c0106a0ef0fd8cb04761146cc60a2609cab3d. Preserve indicator definition, source organization and source note.

Next steps. Transfer these methods to your own public or authorized data while retaining input checks, recorded parameters and result validation. Document the new experimental design and method limits; this example does not validate a new experiment. Full data licenses and generation instructions accompany the cookbook.