Audit a real measurement table

Trace 150 UCI Iris observations, inspect repeat measurements and units, and build an honest exploratory report.

Data science & machine learning · Lesson 01🔋 Offline core analysis⏱ 50 minEmbedded real data📦 numpy · pandas · matplotlib · IPython

Goal and how to run

This lesson assumes basic Python knowledge. Read each method and caption before changing parameters. AI exercises are optional; no AI account is needed for the core workflow. Data lives in notebook metadata, so there is no companion data download. Keep the notebook saved in the active working folder.

Your goal is to understand the research question, execute transparent calculations, check the results, and state the limits of the conclusion.

Setup and parameters

In [1] · Python

from pathlib import Path
import base64, gzip, hashlib, json, platform
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, HTML
from IPython.display import display as _display
def display(value):
    if isinstance(value,pd.DataFrame):
        table=value.to_html(max_rows=12,escape=True)
        style="<style>.vibeit-readable-table{max-width:100%;overflow-x:auto}.vibeit-readable-table table{width:max-content;max-width:none;border-collapse:collapse}.vibeit-readable-table td,.vibeit-readable-table th{white-space:nowrap!important;word-break:normal!important;overflow-wrap:normal!important}</style>"
        return _display(HTML(style+'<div class="vibeit-readable-table">'+table+'</div>'))
    return _display(value)
RECIPE_ID='ds01-iris-audit'
OUTPUT_DIR=Path.cwd()/"results"/RECIPE_ID
plt.rcParams.update({"figure.dpi":120,"font.size":10,"axes.spines.top":False,"axes.spines.right":False,
                      "axes.prop_cycle":plt.cycler(color=["#18785f","#416aa6","#bc6541","#9974a5"])})
print("Python:",platform.python_version(),"; NumPy:",np.__version__,"; Pandas:",pd.__version__)
print("Working folder:",Path.cwd())
print("Core data mode: embedded snapshot; no network requests")

Python: 3.13.13 ; NumPy: 2.5.3 ; Pandas: 3.0.6
Working folder: /Users/zhiluo/Developer/vibeit-cookbook/cookbook/downloads
Core data mode: embedded snapshot; no network requests

Load the embedded snapshot

In [2] · Python

def load_snapshot(recipe_id, expected_sha256, snapshot_key):
    """Read embedded data from this notebook, including a renamed copy."""
    for path in sorted(Path.cwd().glob("*.ipynb")):
        try:
            notebook = json.loads(path.read_text(encoding="utf-8"))
            metadata = notebook.get("metadata", {}).get("vibeit_cookbook", {})
            blob = metadata.get("snapshots", {}).get(snapshot_key)
            if metadata.get("recipe_id") != recipe_id or not blob:
                continue
            if blob.get("sha256") != expected_sha256:
                continue
            raw = gzip.decompress(base64.b64decode(blob["gzip_base64"], validate=True))
            if hashlib.sha256(raw).hexdigest() != expected_sha256:
                raise ValueError("Embedded snapshot checksum mismatch: " + snapshot_key)
            return json.loads(raw)
        except (OSError, json.JSONDecodeError):
            continue
    raise FileNotFoundError(
        "Save/open the complete .ipynb in the active working folder. "
        "The embedded snapshot could not be located for " + recipe_id)

SNAPSHOT_HASHES={'iris': 'cb2512f34911979eef3a42b7a9e6fbc6ca566fd2c0eaa7bd86daefd7ac1cf5ad'}
def snapshot(key):
    return load_snapshot(RECIPE_ID,SNAPSHOT_HASHES[key],key)
print("Embedded snapshots:",list(SNAPSHOT_HASHES))

Embedded snapshots: ['iris']

Read the method functions

In [3] · Python

def audit_numeric_table(frame,columns):
    """Audit observed rows; never silently impute or remove them."""
    if frame.empty or frame.index.has_duplicates:
        raise ValueError('Expected a nonempty table with unique row IDs')
    missing=[name for name in columns if name not in frame.columns]
    if missing:raise ValueError('Missing required columns: '+str(missing))
    values=frame[columns].to_numpy(dtype=float)
    if not np.isfinite(values).all():raise ValueError('Non-finite measurements; inspect missing values explicitly')
    return {'rows':len(frame),'features':len(columns),'finite':True,'duplicate_measurement_rows':int(frame[columns].duplicated().sum())}

print("Method ready: audit_numeric_table")

Method ready: audit_numeric_table

Data and model boundary

These are frozen public measurements, not a new random population sample. The original UCI iris.data is retained, including its two source-documented transcription discrepancies. Repeated rounded feature values do not by themselves establish that two rows are the same plant.

Steps

1. Load and audit the original records

Use row IDs, not feature values, as observation identity. Repeated feature vectors can occur at the reported measurement precision. Keep them visible; deleting them silently changes the data. The audit rejects non-finite numeric values and duplicate row IDs.

In [4] · Python

iris=snapshot("iris")
frame=pd.DataFrame(iris["rows"],columns=iris["columns"])
features=list(frame.columns[:4])
audit=audit_numeric_table(frame,features)
assert len(frame)==150 and frame["species"].nunique()==3
assert (frame[features]>0).all().all()
print("Audit:",audit,"; measurement unit: cm")
print("Source variant:",iris["variant"])
display(frame.head(8))

Audit: {'rows': 150, 'features': 4, 'finite': True, 'duplicate_measurement_rows': 3} ; measurement unit: cm
Source variant: Original UCI iris.data; no silent correction of the two documented transcription discrepancies

	sepal_length_cm	sepal_width_cm	petal_length_cm	petal_width_cm	species
0	5.1	3.5	1.4	0.2	Iris-setosa
1	4.9	3.0	1.4	0.2	Iris-setosa
2	4.7	3.2	1.3	0.2	Iris-setosa
3	4.6	3.1	1.5	0.2	Iris-setosa
4	5.0	3.6	1.4	0.2	Iris-setosa
5	5.4	3.9	1.7	0.4	Iris-setosa
6	4.6	3.4	1.4	0.3	Iris-setosa
7	5.0	3.4	1.5	0.2	Iris-setosa

2. Compare the class counts and feature distributions

Each box summarizes a measurement in centimetres: median, quartiles and conventional 1.5-IQR whiskers. Source class balance is a property of this dataset; it is not the prevalence of these species in nature.

In [5] · Python

display(frame.groupby("species").size().rename("observations").to_frame())
fig,ax=plt.subplots(figsize=(8,4))
ax.boxplot([frame[c] for c in features],tick_labels=features)
ax.set_ylabel("Measurement (cm)");ax.tick_params(axis="x",rotation=20)
ax.set_title("Four observed Iris measurements");plt.tight_layout();plt.show()

	observations
species
Iris-setosa	50
Iris-versicolor	50
Iris-virginica	50

3. Look at a pair of measurements

Color identifies the source species code, not a learned cluster. The horizontal and vertical axes are petal length and width in centimetres. Separation in this projection motivates later modelling; it does not measure test-set accuracy.

In [6] · Python

fig,ax=plt.subplots(figsize=(7,5))
for label,part in frame.groupby("species"):
 ax.scatter(part[features[2]],part[features[3]],label=label,s=28,alpha=.75)
ax.set(xlabel="Petal length (cm)",ylabel="Petal width (cm)",title="Observed petal measurements")
ax.legend(fontsize=8);plt.tight_layout();plt.show()

4. Inspect pairwise association

Pearson correlation is dimensionless and ranges from −1 to +1. It summarizes pooled linear association. Because the three groups have different locations, pooled correlation need not describe an association within each species.

In [7] · Python

correlation=frame[features].corr()
fig,ax=plt.subplots(figsize=(6,5));im=ax.imshow(correlation,vmin=-1,vmax=1,cmap="coolwarm")
ax.set_xticks(range(4),features,rotation=30,ha="right");ax.set_yticks(range(4),features)
for i in range(4):
 for j in range(4):ax.text(j,i,f"{correlation.iloc[i,j]:.2f}",ha="center",va="center",fontsize=9)
fig.colorbar(im,ax=ax,label="Pearson correlation");plt.tight_layout();plt.show()

5. Reconcile and export the report

Confirm that category counts recover every row and correlation is symmetric. Export observed records separately from summaries so later students can reproduce the distinction.

In [8] · Python

assert frame.groupby("species").size().sum()==len(frame)
assert np.allclose(correlation,correlation.T) and np.allclose(np.diag(correlation),1)
OUTPUT_DIR.mkdir(parents=True,exist_ok=True)
frame.to_csv(OUTPUT_DIR/"iris-observations.csv",index=False)
frame.groupby("species")[features].agg(["mean","std","min","max"]).to_csv(OUTPUT_DIR/"measurement-summary.csv")
correlation.to_csv(OUTPUT_DIR/"correlation.csv")
(OUTPUT_DIR/"audit.json").write_text(json.dumps(audit,indent=2))
print("Checked",len(frame),"observations; exports:",OUTPUT_DIR)

Checked 150 observations; exports: /Users/zhiluo/Developer/vibeit-cookbook/cookbook/downloads/results/ds01-iris-audit

Exercises and reference explanation

How many source observations belong to each species?
Compare pooled correlation with within-species correlation.

Reference explanation. There are 50 observations per source class. Grouped correlations can differ because pooling combines within-group variation and between-group mean differences.

Keep a copy before changing parameters. Compare old and new figures and record which inputs, calculations and conclusions changed.

Optional AI coding exercises

Prompt 1

Using only NumPy, Pandas and Matplotlib, add separate within-species correlation heatmaps without deleting repeated measurement vectors.

Prompt 2

Using the same allowed packages, flag duplicate rounded measurement vectors and list their row IDs; do not remove them automatically.

Human acceptance. Check totals remain 150, every original row ID survives and each grouped matrix is symmetric.

Review the edited source, then manually run affected cells and subsequent checks. Ask the assistant to explain the purpose, packages and data provenance of its changes, and reconcile that explanation with actual output.

Optional online extension

In [9] · Python

RUN_ONLINE_UPDATE=False
if RUN_ONLINE_UPDATE:
    import requests
    try:
        response=requests.get('https://archive.ics.uci.edu/dataset/53',timeout=20)
        response.raise_for_status()
        print("Source:",response.url)
        print(response.text[:700])
    except requests.RequestException as error:
        print("Online extension unavailable:",type(error).__name__,str(error)[:160])
else:
    print("Online extension skipped; embedded core data unchanged")

Online extension skipped; embedded core data unchanged

Sources, snapshots and next steps

UCI iris — CC BY 4.0; credit UCI and original dataset authors; retrieved 2026-10-09. SHA-256 d11fe30213d36434a0879aab7cb00ce3c812eb7ba2495874438abff7b7b762e9. Extract iris.data from original download; preserve all rows and declared units; no imputation. Fisher, R. (1936). Iris. UCI Machine Learning Repository. DOI: 10.24432/C56C76.

Next steps. Transfer these methods to your own public or authorized data while retaining input checks, recorded parameters and result validation. Document the new experimental design and method limits; this example does not validate a new experiment. Full data licenses and generation instructions accompany the cookbook.