API reference¶
The full public API, generated from the in-source NumPy-style docstrings, is at the bottom of this page. The hand-written sections below cover the parts of the contract that the auto-generated reference cannot show at a glance — most importantly Phase 11, the loading layer, which is the pipeline's entry point.
Phase 11 — the loader (entry layer)¶
dextra begins one step earlier than most analysis toolkits: at the raw,
untrusted file. load turns an external source into a faithful, correctly-typed
DataFrame and discloses exactly how it was parsed and what was uncertain,
emitting a JSON-safe record that reproduces the frame on demand. It is
disclosure-first: the value is not "we parse better than pandas" — it is "we
parse transparently, safely, and reproducibly, with a safety net."
import dextra as dx
df = dx.load('sales.csv') # raw, messy file -> typed DataFrame + full disclosure
plan = dx.peek('sales.csv') # look before you load: plan + small preview, no commit
Short aliases dload / dpeek are identical to load / peek.
Five source families¶
load auto-detects the source family from the extension (override with
kind=), and each family is parsed and disclosed under the same contract:
| Family | Extensions | What is detected / disclosed |
|---|---|---|
| Delimited text | .csv, .tsv, and unknown text |
Encoding, delimiter/dialect, the real header row (junk preambles skipped), per-column dtype, locale numerics and dates — quote-aware and preamble-safe |
| Excel | .xlsx, .xlsm |
Sheet selection, the data block within a sheet, single- and multi-row headers (combined into top_bottom names), native cell types (values, never formulas). Legacy .xls is refused with guidance |
| Parquet | .parquet, .pq |
Typed pass-through; needs a lazy engine such as pyarrow |
| JSON | .json, .jsonl, .ndjson |
Records array or one object per line; nested values are serialised and the nesting is disclosed; bad NDJSON lines are skipped and flagged |
| SQL | .db, .sqlite, .sqlite3, or an open DB-API connection |
One parametrised SELECT via sql= + sql_params=; SQLite files are opened read-only; results are row-capped and truncation disclosed |
on_ambiguous — transparency scales with uncertainty¶
Confident parses load in a single line. Ambiguous decisions (an uncertain header row, several Excel sheets, an ID-like column that looks numeric, a single JSON object instead of an array) are surfaced according to one policy:
| Value | Behaviour |
|---|---|
"warn" (default for load) |
Load the frame and emit a DextraLoaderWarning for each ambiguous decision |
"raise" |
Raise LoaderAmbiguityError instead of guessing |
"plan" (default for peek) |
Return the load plan without loading the full frame |
Every load also prints a disclosure report and a one-line Decision: sentence
unless show=False.
The replayable load plan¶
When return_params=True, load also returns a JSON-safe load plan — the
complete, serialisable record of every decision it made. Passing it back
reproduces the frame deterministically:
df, plan = dx.load('messy.csv', return_params=True)
import json; json.dumps(plan) # the plan is always JSON-serialisable
df2 = dx.load('messy.csv', params=plan) # exact, decision-for-decision replay
The plan stores the source sha256. If the source has changed since the plan
was created, replay still proceeds but the mismatch is flagged (and raised under
on_ambiguous='raise'), so a silently-edited file can never masquerade as the
original.
Security model¶
The loader is the most security-sensitive entry point in the library, so it is conservative by default:
- No code execution on load. Pickle sources are refused unless
allow_pickle=Trueis passed explicitly. - SQL is parametrised only. Exactly one statement is allowed; stacked
statements are refused, and values bind through
sql_params=— never string formatting — so injection is structurally impossible. SQLite files open read-only; for a live connection read-only cannot be enforced and that fact is disclosed in the plan. - Bounded reads.
max_rows(and a default 1,000,000-row SQL guard) cap how much is read; truncation is disclosed and flagged. - Immutability. The source is never modified, and an in-memory
DataFramepassed toloadis returned as a typed copy, never mutated in place.
Audit trail¶
Like every dextra function, load records its decision to
df.attrs["dextra_audit"], including the load plan, so the provenance of a
frame travels with it through the rest of the pipeline.
Full generated reference¶
dextra - lightweight exploratory-data-analysis helpers.
Quick start
import dextra as dx dx.audit(df) # master audit dx.na_show(df) # show missing dx.impute(df) # fix missing
Phase 2: 22 statistical helpers. Phase 3 v2: 10 cleaning helpers (3 inspectors + 7 actors, short names).
dload = load
module-attribute
¶
dpeek = peek
module-attribute
¶
DEFAULT_BOX_COLORS = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd', '#8c564b', '#e377c2', '#7f7f7f', '#bcbd22', '#17becf']
module-attribute
¶
__version__ = '0.6.1'
module-attribute
¶
audit = clean_report
module-attribute
¶
dedup = dedupe
module-attribute
¶
dupscan = dup_show
module-attribute
¶
impute = handle_missing
module-attribute
¶
nascan = na_show
module-attribute
¶
outscan = out_show
module-attribute
¶
recast = cast_types
module-attribute
¶
tidycols = standardize_columns
module-attribute
¶
verify = validate_rules
module-attribute
¶
winsor = clip_outliers
module-attribute
¶
dashapp = dash
module-attribute
¶
confrep = confusion_report
module-attribute
¶
learncv = learning_curves
module-attribute
¶
residan = residual_analysis
module-attribute
¶
rocpr = roc_pr
module-attribute
¶
binize = bin
module-attribute
¶
clf = classify
module-attribute
¶
clus = cluster
module-attribute
¶
reg = regress
module-attribute
¶
boxpl = plot_boxplots
module-attribute
¶
hister = plot_histograms
module-attribute
¶
edarep = edareport
module-attribute
¶
imps = importance
module-attribute
¶
redun = redundancy
module-attribute
¶
relev = relevance
module-attribute
¶
selpipe = selectpipe
module-attribute
¶
numdesc = describe_numeric
module-attribute
¶
aov1 = anova_oneway
module-attribute
¶
chi2ind = chi_square_independence
module-attribute
¶
cim = confidence_interval_mean
module-attribute
¶
cip = confidence_interval_proportion
module-attribute
¶
corrmat = correlation_matrix
module-attribute
¶
emprule = empirical_rule_check
module-attribute
¶
freqtab = frequency_table
module-attribute
¶
gcmp = group_compare
module-attribute
¶
imbalance = class_imbalance
module-attribute
¶
missrep = missing_report
module-attribute
¶
normtest = normality_test
module-attribute
¶
outrep = outliers_report
module-attribute
¶
pskew = pearson_skewness
module-attribute
¶
slr = simple_linear_regression
module-attribute
¶
ssm = sample_size_mean
module-attribute
¶
ssp = sample_size_proportion
module-attribute
¶
t1 = t_test_one_sample
module-attribute
¶
t2 = t_test_two_sample
module-attribute
¶
tpair = t_test_paired
module-attribute
¶
vif = vif_scores
module-attribute
¶
xtab = cross_tab
module-attribute
¶
zsc = z_scores
module-attribute
¶
_PHASE_LABELS = {'dextra._loader': 'Phase 11 - loader (entry layer)', 'dextra.stats': 'Phase 1 - EDA', 'dextra.plots': 'Phase 1 - EDA', 'dextra.stats_advanced': 'Phase 2 - statistics', 'dextra.cleaning': 'Phase 3 - cleaning', 'dextra._features_numeric': 'Phase 4 - features', 'dextra._features_discretize': 'Phase 4 - features', 'dextra._features_derive': 'Phase 4 - features', 'dextra._features_pipeline': 'Phase 4 - features', 'dextra.features': 'Phase 4 - features', 'dextra.selection': 'Phase 5 - selection', 'dextra.modeling': 'Phase 6 - modeling', 'dextra.evaluation': 'Phase 7 - evaluation', 'dextra.timeseries': 'Phase 8 - timeseries', 'dextra.report': 'Phase 9 - report', 'dextra.dashboard': 'Phase 10 - dashboard', 'dextra.compat': 'scikit-learn compat'}
module-attribute
¶
_PHASE_ORDER = ['Phase 11 - loader (entry layer)', 'Phase 1 - EDA', 'Phase 2 - statistics', 'Phase 3 - cleaning', 'Phase 4 - features', 'Phase 5 - selection', 'Phase 6 - modeling', 'Phase 7 - evaluation', 'Phase 8 - timeseries', 'Phase 9 - report', 'Phase 10 - dashboard', 'scikit-learn compat', 'other']
module-attribute
¶
__all__ = ['__version__', 'DEFAULT_BOX_COLORS', 'functions', 'load', 'dload', 'peek', 'dpeek', 'describe_numeric', 'plot_histograms', 'plot_boxplots', 'numdesc', 'hister', 'boxpl', 'z_scores', 'pearson_skewness', 'empirical_rule_check', 'outliers_report', 'correlation_matrix', 'simple_linear_regression', 'missing_report', 'frequency_table', 'cross_tab', 'group_compare', 'confidence_interval_mean', 'confidence_interval_proportion', 'sample_size_mean', 'sample_size_proportion', 'normality_test', 't_test_one_sample', 't_test_two_sample', 't_test_paired', 'anova_oneway', 'chi_square_independence', 'vif_scores', 'class_imbalance', 'clean_report', 'standardize_columns', 'cast_types', 'validate_rules', 'handle_missing', 'dedupe', 'clip_outliers', 'na_show', 'dup_show', 'out_show', 'zsc', 'pskew', 'emprule', 'outrep', 'corrmat', 'slr', 'missrep', 'freqtab', 'xtab', 'gcmp', 'cim', 'cip', 'ssm', 'ssp', 'normtest', 't1', 't2', 'tpair', 'aov1', 'chi2ind', 'vif', 'imbalance', 'dedup', 'audit', 'nascan', 'dupscan', 'outscan', 'tidycols', 'recast', 'impute', 'winsor', 'verify', 'transform', 'scale', 'bin', 'binize', 'encode', 'dtfeats', 'cross', 'aggfeat', 'featpipe', 'redundancy', 'relevance', 'importance', 'rfe', 'selectpipe', 'redun', 'relev', 'imps', 'selpipe', 'regress', 'reg', 'classify', 'clf', 'cluster', 'clus', 'confusion_report', 'roc_pr', 'residual_analysis', 'learning_curves', 'confrep', 'rocpr', 'residan', 'learncv', 'tsdecomp', 'tsstat', 'tsfcast', 'edareport', 'edarep', 'dash', 'dashapp', 'DextraFeaturePipeline', 'DextraSelectPipeline', 'DextraRegressor', 'DextraClassifier', 'DextraClusterer']
module-attribute
¶
DextraClassifier
¶
Bases: ClassifierMixin, _BaseModelWrapper
sklearn classifier wrapping :func:dextra.classify (one baseline algorithm).
Fit learns a dextra params artifact; predict / predict_proba
reuse the fitted estimator -- drop-in for sklearn Pipelines.
Examples:
>>> from dextra.compat import DextraClassifier
>>> DextraClassifier(method="logistic").fit(X, y).predict(X_new)
DextraClusterer
¶
Bases: ClusterMixin, _BaseModelWrapper
sklearn clusterer wrapping :func:dextra.cluster (unsupervised, no y).
labels_ holds the labels assigned by the fit itself (sklearn
convention); predict assigns data via the persisted estimator (for
agglomerative that is a nearest-centroid rule, which may disagree
with the fit labels on boundary points).
Examples:
>>> from dextra.compat import DextraClusterer
>>> DextraClusterer(method="kmeans", k=4).fit(X).labels_
DextraFeaturePipeline
¶
Bases: TransformerMixin, BaseEstimator
sklearn transformer wrapping :func:dextra.featpipe (fit/apply, leak-safe).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
sequence of dict
|
A |
None
|
Examples:
>>> from dextra.compat import DextraFeaturePipeline
>>> DextraFeaturePipeline(steps=recipe).fit_transform(X)
DextraRegressor
¶
Bases: RegressorMixin, _BaseModelWrapper
sklearn regressor wrapping :func:dextra.regress (one baseline algorithm).
Fit learns a dextra params artifact; predict reuses the fitted
estimator -- drop-in for sklearn Pipelines and cross_val_score.
Examples:
>>> from dextra.compat import DextraRegressor
>>> DextraRegressor(method="forest").fit(X, y).predict(X_new)
DextraSelectPipeline
¶
Bases: TransformerMixin, BaseEstimator
sklearn transformer wrapping :func:dextra.selectpipe (fit/apply, leak-safe).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
sequence of dict
|
A |
None
|
Examples:
>>> from dextra.compat import DextraSelectPipeline
>>> DextraSelectPipeline(steps=recipe).fit_transform(X, y)
load(source, *, kind='auto', params=None, on_ambiguous='warn', encoding=None, sep=None, header_row=None, sheet=None, header_rows=None, sql=None, sql_params=None, parse_dates=True, decimal=None, thousands=None, na_values=None, max_rows=None, low_memory=False, sample_bytes=262144, return_params=False, show=True, decimals=4, df_name=None, interactive=False)
¶
Load a messy source into a typed DataFrame, disclosing every decision.
Auto-detects encoding, delimiter and the real header row of a delimited-text
source, then infers per-column types and measures how many cells parsed.
Transparency scales with uncertainty: confident parses load in one line;
ambiguous decisions are flagged and handled per on_ambiguous
('warn' -> load + warn; 'raise' -> raise; 'plan' -> return the
plan without loading). Every load emits a JSON-safe, replayable load plan
(the params artifact); load(source, params=plan) reproduces the frame
exactly. The source is never modified.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | PathLike | file - like | DataFrame
|
A path, an open binary/text stream, or an in-memory frame (typed
pass-through). Excel = |
required |
kind
|
('auto', 'csv', 'tsv', 'excel', 'parquet', 'json')
|
Source kind; inferred from the extension when |
"auto"
|
params
|
dict
|
A previously returned load plan to replay deterministically. |
None
|
on_ambiguous
|
('warn', 'raise', 'plan')
|
Policy when a decision is ambiguous (see above). |
"warn"
|
encoding
|
optional
|
Force a decision instead of detecting it. |
None
|
sep
|
optional
|
Force a decision instead of detecting it. |
None
|
header_row
|
optional
|
Force a decision instead of detecting it. |
None
|
decimal
|
optional
|
Force a decision instead of detecting it. |
None
|
thousands
|
optional
|
Force a decision instead of detecting it. |
None
|
sheet
|
str | int
|
Excel only: sheet name or 0-based index. Default: the single sheet, or the first visible one (flagged ambiguous when several exist). |
None
|
header_rows
|
int
|
Excel only: force the number of header rows (multi-row headers are
otherwise detected and combined into |
None
|
sql
|
str
|
SQL only: ONE parametrised statement (stacked statements are
refused). Bind values with |
None
|
sql_params
|
dict | tuple
|
Named ( |
None
|
parse_dates
|
bool
|
Attempt safe datetime inference. |
True
|
na_values
|
list
|
Extra NA tokens added to the pandas defaults. |
None
|
max_rows
|
int
|
Safety cap on the number of rows read. |
None
|
low_memory
|
bool
|
Delimited text only: stream the source once for header / ragged-row detection instead of materialising every cell as a Python list-of-lists, cutting the transient peak memory of a large CSV load (about a third on a million rows). Returns an identical frame and plan -- per-column type inference is unchanged. A no-op for already-typed sources (Excel / parquet / JSON / SQL). |
False
|
return_params
|
bool
|
Also return the load plan. |
False
|
show
|
bool
|
Print the disclosure report + the |
True
|
decimals
|
int
|
Numeric precision in the printed report. |
4
|
df_name
|
str
|
Name used in the audit / decision (inferred when omitted). |
None
|
interactive
|
bool
|
Prompt to confirm before loading (classroom use; never during
composition and never when |
False
|
Returns:
| Type | Description |
|---|---|
pandas.DataFrame, or (DataFrame, plan) when ``return_params=True``; the plan
|
|
alone when ``on_ambiguous='plan'``.
|
|
Examples:
>>> df = dx.load('messy.csv')
>>> df, plan = dx.load('messy.csv', return_params=True)
>>> df = dx.load('messy.csv', params=plan) # deterministic replay
>>> df = dx.load('book.xlsx', sheet='Q1') # Excel: pick a sheet
>>> df = dx.load('app.db', sql='SELECT * FROM t WHERE d = :d',
... sql_params={'d': '2026-01-01'}) # safe SQL
peek(source, *, kind='auto', on_ambiguous='plan', show=True, n_preview=10, df_name=None, **load_kwargs)
¶
Propose a load plan + preview WITHOUT committing a full load.
Returns the load plan (dict); loads at most n_preview rows for the sample.
The teaching / inspection entry point ("look before you load").
The plan carries plan_scope='preview': replaying it with
load(source, params=plan) loads ALL rows (the preview cap is
not part of the recipe) and warns once about the origin.
df_name= labels the source explicitly in the plan / audit (audit #11);
when omitted it is inferred as a last resort.
Examples:
>>> plan = dx.peek("messy.csv")
>>> df = dx.load("messy.csv", params=plan) # full load, all rows
cast_types(df, schema=None, auto_categorical=True, categorical_threshold=50, categorical_ratio=0.5, parse_threshold=0.9, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, fig_width=12.0, fig_height=5.5, dpi=110)
¶
Stage 2: smart dtype coercion.
If schema is provided, casts each named column to the requested dtype.
If schema is None, infers the best dtype per column:
Detection order (for object / str columns only): 1. Boolean - every value matches a recognized true/false token. 2. Datetime - >= parse_threshold parses with pd.to_datetime. 3. Numeric - >= parse_threshold parses with pd.to_numeric (with currency / comma stripping as a fallback). 4. Category - cardinality < categorical_threshold AND cardinality / n_rows < categorical_ratio.
Returns a NEW DataFrame; original is untouched. Memory before/after is reported.
DAMA dimension: Validity.
Examples:
>>> dx.cast_types(df)
>>> dx.cast_types(df, schema={'price': 'float64', 'date': 'datetime64[ns]'})
clean_report(df, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, fig_width=14.0, fig_height=8.0, dpi=110)
¶
Stage 0 / Stage 7: comprehensive data-quality audit.
Produces a per-column profile + global quality scores + cleaning trail review. Does NOT modify the input DataFrame.
For each column reports: dtype, n_missing, pct_missing, n_unique, cardinality_pct, memory_kb, sample_value, suggested_action.
Global metrics: n_rows, n_cols, n_complete_rows, n_duplicate_rows, memory_mb, completeness_score, uniqueness_score, consistency_score, quality_score (weighted 0-100).
Visual: 4-panel figure (missing per column, type composition, cardinality histogram, quality score gauge).
If df was previously cleaned by dextra (carries dextra_audit in
df.attrs), the cleaning trail is rendered at the bottom.
Examples:
>>> dx.clean_report(df)
clip_outliers(df, cols=None, method='iqr', k=1.5, z_threshold=3.0, action='clip', dry_run=False, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, fig_width=14.0, fig_height=6.0, dpi=110, params=None, return_params=False)
¶
Stage 5: outlier treatment.
Detection methods: 'iqr' - Tukey fence: LB = Q1 - kIQR, UB = Q3 + kIQR. 'zscore' - |Z| > z_threshold.
Actions: 'clip' - winsorization (replace with bound). Default; no row loss. 'drop' - drop rows where ANY analysed column is outlying.
Leakage-safe fit/apply: in FIT mode (params=None) pass
return_params=True to also get a replayable params dict with
the per-column bounds computed from THIS data; in APPLY mode
(params=<dict>) held-out data is clipped/dropped at those
fit-time bounds verbatim -- never recomputed. Columns whose
bounds could not be fitted (all-NaN, zero spread) are excluded
from the params and left untouched on apply.
DAMA dimension: Accuracy (correcting suspect values without inventing them).
Examples:
>>> dx.clip_outliers(df) # IQR, k=1.5, clip
>>> dx.clip_outliers(df, method='zscore', z_threshold=3)
>>> dx.clip_outliers(df, action='drop')
>>> tr, p = dx.clip_outliers(train, return_params=True)
>>> te = dx.clip_outliers(test, params=p) # train bounds, no re-fit
dedupe(df, subset=None, keep='first', drop_indices=None, dry_run=False, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, return_dropped=False, fig_width=12.0, fig_height=4.5, dpi=110)
¶
Stage 4: remove duplicate rows.
subset is the list of columns considered when defining a duplicate.
Default (None) uses all columns.
keep follows pandas' convention:
'first' - keep the first occurrence (default)
'last' - keep the last occurrence
False - drop all duplicates (keep none)
DAMA dimension: Uniqueness.
Examples:
>>> dx.dedupe(df)
>>> dx.dedupe(df, subset=['customer_id'])
>>> dropped = dx.dedupe(df, return_dropped=True)
dup_show(df, subset=None, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, max_rows_shown=20, fig_width=12.0, fig_height=5.0, dpi=110)
¶
Inspector: show duplicate rows grouped side-by-side.
Pure / read-only. Returns the duplicate rows with three extra columns:
- dup_group_id: integer ID per duplicate group.
- group_size: number of rows in the group.
- is_first_in_group: True for the first occurrence.
The result is sorted by group so the user can decide which rows to
drop and call dx.dedup(df, drop_indices=[...]).
DAMA dimension: Uniqueness.
Examples:
>>> view = dx.dup_show(df, subset=['customer_id'])
>>> # inspect, decide, then:
>>> df_clean = dx.dedup(df, drop_indices=view.index[view['is_first_in_group']==False])
handle_missing(df, strategy='auto', drop_threshold=0.6, fill_value=None, random_state=None, dry_run=False, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, fig_width=14.0, fig_height=5.5, dpi=110, params=None, return_params=False)
¶
Stage 3: handle missing values.
Strategies (string, or dict for per-column):
'auto' - smart per-column: median for skewed numeric, mean for
symmetric numeric, mode for categorical, ffill for
datetime, mode for boolean.
'mean' - mean (numeric only).
'median' - median (numeric only).
'mode' - most frequent (any dtype).
'ffill' - forward fill then back fill.
'bfill' - back fill then forward fill.
'constant' - fill with fill_value.
'drop_rows' - drop rows containing ANY NaN.
'drop_cols' - drop columns with > drop_threshold missing.
dict - per-column: {'colname': strategy_str}.
Leakage-safe fit/apply: in FIT mode (params=None) pass
return_params=True to also get a replayable params dict
holding the fill values learned from THIS data; in APPLY mode
(params=<dict>) those values are applied verbatim -- never
recomputed -- so held-out data is filled with train statistics.
mean/median/mode/constant freeze the fill value;
random_uniform / random_normal freeze the train distribution;
ffill/bfill/interpolate are order-based re-runs (no train
statistic exists); random_sample re-samples from the apply-side
data (warned). 'drop_cols' replays the fitted column drop;
'drop_rows' re-runs (no statistic).
Columns that are entirely missing have no statistic to impute from and are left unchanged (no RuntimeWarning is emitted).
DAMA dimension: Completeness.
Examples:
>>> dx.handle_missing(df) # auto
>>> dx.handle_missing(df, strategy={'price': 'median', 'name': 'mode'})
>>> dx.handle_missing(df, strategy='drop_cols', drop_threshold=0.5)
>>> tr, p = dx.handle_missing(train, strategy='mean', return_params=True)
>>> te = dx.handle_missing(test, params=p) # train means, no re-fit
na_show(df, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, max_rows_shown=10, fig_width=14.0, fig_height=5.5, dpi=110)
¶
Inspector: show rows with missing values without modifying the data.
Pure / read-only. Returns the rows that contain at least one NaN with
two extra columns:
- which_cols_missing: comma-separated list of columns with NaN.
- n_missing_in_row: count of NaN cells in the row.
Also prints a per-column profile with a recommended strategy.
DAMA dimension: Completeness.
Examples:
>>> view = dx.na_show(df)
>>> # decide a strategy from the suggestions, then:
>>> df_clean = dx.impute(df, strategy='median')
out_show(df, cols=None, method='iqr', k=1.5, z_threshold=3.0, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, max_rows_shown=10, fig_width=14.0, fig_height=5.5, dpi=110)
¶
Inspector: show rows containing outlying values.
Pure / read-only. Returns the outlying rows with three extra columns:
- outlier_in_columns: comma-separated list of columns where the
row is outlying.
- severity_z: maximum |z| across analysed columns.
- severity_iqr: maximum normalised distance beyond the IQR fence.
Sorted by severity_z descending so the user sees the worst offenders
first.
DAMA dimension: Accuracy.
Examples:
>>> view = dx.out_show(df, cols=['price', 'age'])
>>> # decide and apply:
>>> df_clean = dx.clip_outliers(df, cols=['price'], action='clip')
standardize_columns(df, lowercase=True, strip_cells=True, deduplicate_names=True, name_map=None, dry_run=False, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, fig_width=12.0, fig_height=5.0, dpi=110)
¶
Stage 1: Structural cleanup.
Operations applied (returns a NEW DataFrame; original is untouched):
1. Column names are NFKC-normalised and re-spelled as snake_case-ish:
every non-alphanumeric char becomes '_', multiple underscores
collapse, leading/trailing underscores stripped, optional lowercase.
2. If two columns would collide after normalisation, the second
receives a numeric suffix (_1, _2, ...).
3. Optional explicit name_map overrides the auto-naming on a per-key
basis.
4. For string/object columns: leading/trailing whitespace is stripped
from every cell (if strip_cells=True).
DAMA dimension: Consistency.
Examples:
>>> dx.standardize_columns(df)
>>> dx.standardize_columns(df, name_map={"CustID": "customer_id"})
validate_rules(df, rules, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, return_violations=False, fig_width=12.0, fig_height=5.0, dpi=110)
¶
Stage 6: business / consistency rules.
Each rule is a dict with these keys: name: str - unique identifier. check: str expression (eval'd via df.eval) OR callable taking df and returning a boolean Series. True = passing row. description: optional human-readable description (str). severity: optional 'error'|'warning' (default 'error').
Trust assumption: string check rules are executed with df.eval and
callables receive the full DataFrame -- rules are code, not data. Only
use rules from a trusted source (your own code or configuration); never
build them from untrusted end-user input.
For each rule we report n_violations, pct_violations, status.
return_violations=True returns the rows that failed at least one rule
plus a violated_rules column listing the rule names.
DAMA dimensions: Consistency, Validity.
Examples:
>>> rules = [
... {"name": "price_positive", "check": "price >= 0"},
... {"name": "age_valid", "check": "age.between(0, 120)"},
... {"name": "valid_email", "check": lambda d: d['email'].str.contains('@')},
... ]
>>> dx.validate_rules(df, rules)
dash(df, *, out='dashboard_app.py', output_dir=None, target=None, include_model=False, task='auto', launch=False, data_format='auto', max_hist=24, top_cat=10, theme='light', return_params=False, show=True, df_name=None)
¶
Generate a self-contained interactive Streamlit dashboard in one call.
Writes a thin dashboard_app.py, a sidecar data file (parquet when
an engine is available, else csv; pickle only on request), and a
*_meta.json
reproducibility manifest (dextra / Python / pandas versions, settings).
Running streamlit run dashboard_app.py renders the dextra dashboard: the
same sections as :func:edareport (Overview, Data quality, Univariate,
Bivariate, and an optional Model tab), made interactive via a sidebar. It
computes nothing new -- it reuses the neutral _compose section builders.
By default dash only generates and returns the app path
(launch=False). Pass output_dir= to collect every output in one
folder. Streamlit is the optional dash extra, imported lazily; the input
DataFrame is never mutated (the sidecar is a copy).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data to explore. Never mutated. |
required |
out
|
str
|
App file name (or path). With |
'dashboard_app.py'
|
output_dir
|
str
|
Directory to write the app, data and metadata into (created if needed). |
None
|
target
|
str
|
Default target column for the bivariate / model tabs. |
None
|
include_model
|
bool
|
Default state of the include-model toggle (the model tab needs a target and scikit-learn). |
False
|
task
|
('auto', 'regression', 'classification')
|
Default of the model-task sidebar control. |
"auto"
|
launch
|
bool
|
Also run |
False
|
data_format
|
('auto', 'parquet', 'csv', 'pickle')
|
Sidecar format. |
"auto"
|
max_hist
|
int
|
Defaults for the histogram-column and categorical-column caps. |
24
|
top_cat
|
int
|
Defaults for the histogram-column and categorical-column caps. |
24
|
theme
|
str
|
Visual theme hint ( |
'light'
|
return_params
|
bool
|
Return the JSON-safe build manifest (including the |
False
|
show
|
bool
|
Print the one-line |
True
|
df_name
|
str
|
Name used in the title / audit (inferred when omitted). |
None
|
Returns:
| Type | Description |
|---|---|
str or dict
|
The app path, or -- when |
Examples:
>>> dx.dash(df) # writes dashboard_app.py
>>> dx.dash(df, output_dir='dashboard', target='churn', include_model=True)
>>> dx.dash(df, data_format='parquet', launch=True)
confusion_report(df, y_true=None, y_pred=None, *, params=None, labels=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=15.0, fig_height=4.6, dpi=110)
¶
Per-class confusion diagnostics for a classifier in one line.
Two input modes. In LABEL mode pass y_true and y_pred (column names
or array-likes). In ARTIFACT mode pass params (a Phase-6 classify
artifact) and the data to judge on; the truth is derived from it. Returns a
dense per-class precision / recall / F1 / support table (plus accuracy and
macro / weighted averages), a three-panel figure, and a one-line decision.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y_true
|
str or array - like
|
True and predicted labels (label mode). A column name or a
Series/array aligned to |
None
|
y_pred
|
str or array - like
|
True and predicted labels (label mode). A column name or a
Series/array aligned to |
None
|
params
|
dict
|
A Phase-6 |
None
|
labels
|
sequence
|
Explicit class order. Defaults to the sorted union of the labels seen. |
None
|
return_params
|
bool
|
Standard dextra flags. |
False
|
show
|
bool
|
Standard dextra flags. |
False
|
plot
|
bool
|
Standard dextra flags. |
False
|
return_df
|
bool
|
Standard dextra flags. |
False
|
return_fig
|
bool
|
Standard dextra flags. |
False
|
decimals
|
bool
|
Standard dextra flags. |
False
|
df_name
|
bool
|
Standard dextra flags. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The per-class metrics table, and -- when requested -- the report descriptor and/or the matplotlib figure. |
Examples:
>>> dx.confusion_report(df, y_true='churn', y_pred='churn_pred')
>>> _, p = dx.classify(tr, y='churn', method='forest', return_params=True)
>>> dx.confusion_report(te, params=p) # artifact mode
learning_curves(df, y=None, cols=None, *, params=None, estimator=None, task=None, scoring=None, cv=5, train_sizes=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=5.4, dpi=110)
¶
Learning curves (train vs cross-validated score) in one line.
Re-fits an estimator on progressively larger subsets and plots the training
score against the cross-validated score -- the canonical bias / variance
diagnostic. In ARTIFACT mode pass params (any supervised Phase-6
artifact); the estimator, features and target are read from it. Otherwise
pass estimator= together with y and cols. Returns a per-train-
size score table, a two-panel figure (curve + train-CV gap), and a
one-line decision naming the likely regime.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y
|
str or array - like
|
Target (when not using an artifact). Column name or aligned array. |
None
|
cols
|
sequence of str
|
Feature columns (when not using an artifact). Defaults to every numeric non-boolean column except the target. |
None
|
params
|
dict
|
A Phase-6 supervised artifact ( |
None
|
estimator
|
sklearn estimator
|
An estimator to evaluate (when not using an artifact). |
None
|
task
|
('regression', 'classification')
|
Defaults to the artifact's task, else inferred from |
'regression'
|
scoring
|
str
|
sklearn scoring name. Defaults to |
None
|
cv
|
int
|
Cross-validation folds. |
5
|
train_sizes
|
array - like
|
Fractions of the training set. Defaults to |
None
|
return_params
|
bool
|
Standard dextra flags. |
False
|
show
|
bool
|
Standard dextra flags. |
False
|
plot
|
bool
|
Standard dextra flags. |
False
|
return_df
|
bool
|
Standard dextra flags. |
False
|
return_fig
|
bool
|
Standard dextra flags. |
False
|
decimals
|
bool
|
Standard dextra flags. |
False
|
df_name
|
bool
|
Standard dextra flags. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The per-train-size score table, and -- when requested -- the report descriptor and/or the matplotlib figure. |
Examples:
>>> _, p = dx.classify(tr, y='churn', method='forest', return_params=True)
>>> dx.learning_curves(tr, params=p) # artifact mode
>>> from sklearn.ensemble import RandomForestRegressor
>>> dx.learning_curves(tr, y='price', estimator=RandomForestRegressor())
residual_analysis(df, y_true=None, y_pred=None, *, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=9.0, dpi=110)
¶
Residual diagnostics for a regression model in one line.
Two input modes (LABEL: y_true + y_pred; ARTIFACT: a Phase-6
regress artifact via params). Returns a dense diagnostics table
(residual mean / std / skew / kurtosis, R2 / RMSE / MAE, Durbin-Watson, a
heteroscedasticity hint, and a Jarque-Bera normality p-value), a four-panel
figure (residual-vs-fitted, residual distribution, Normal Q-Q,
scale-location), and a one-line decision on the residual assumptions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y_true
|
str or array - like
|
True and predicted numeric values (label mode). |
None
|
y_pred
|
str or array - like
|
True and predicted numeric values (label mode). |
None
|
params
|
dict
|
A Phase-6 |
None
|
return_params
|
bool
|
Standard dextra flags. |
False
|
show
|
bool
|
Standard dextra flags. |
False
|
plot
|
bool
|
Standard dextra flags. |
False
|
return_df
|
bool
|
Standard dextra flags. |
False
|
return_fig
|
bool
|
Standard dextra flags. |
False
|
decimals
|
bool
|
Standard dextra flags. |
False
|
df_name
|
bool
|
Standard dextra flags. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The diagnostics table, and -- when requested -- the report descriptor and/or the matplotlib figure. |
Examples:
>>> dx.residual_analysis(df, y_true='price', y_pred='price_pred')
>>> _, p = dx.regress(tr, y='price', method='forest', return_params=True)
>>> dx.residual_analysis(te, params=p) # artifact mode
roc_pr(df, y_true=None, scores=None, *, params=None, pos_label=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=5.6, dpi=110)
¶
ROC and Precision-Recall curves for a classifier in one line.
Two input modes. In LABEL mode pass y_true and scores (predicted
probabilities / decision scores: a 1-D positive-class vector for binary, or
an (n, n_classes) matrix for multiclass one-vs-rest). In ARTIFACT mode
pass params (a Phase-6 classify artifact carrying a fitted
estimator) and the data; scores come from predict_proba /
decision_function. Returns a per-class ROC-AUC / average-precision
table (plus macro), a two-panel ROC + PR figure, and a one-line decision.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y_true
|
str or array - like
|
True labels (label mode). |
None
|
scores
|
array - like
|
Predicted scores (label mode): shape |
None
|
params
|
dict
|
A Phase-6 |
None
|
pos_label
|
optional
|
The positive class for a binary 1-D |
None
|
return_params
|
bool
|
Standard dextra flags. |
False
|
show
|
bool
|
Standard dextra flags. |
False
|
plot
|
bool
|
Standard dextra flags. |
False
|
return_df
|
bool
|
Standard dextra flags. |
False
|
return_fig
|
bool
|
Standard dextra flags. |
False
|
decimals
|
bool
|
Standard dextra flags. |
False
|
df_name
|
bool
|
Standard dextra flags. |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The per-class AUC / AP table, and -- when requested -- the report descriptor and/or the matplotlib figure. |
Examples:
>>> dx.roc_pr(df, y_true='churn', scores='churn_proba')
>>> _, p = dx.classify(tr, y='churn', method='forest', return_params=True)
>>> dx.roc_pr(te, params=p) # artifact mode
aggfeat(df, group=None, value=None, agg='mean', *, as_of=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Build group-aggregation features, with a temporal-leakage guard.
Two modes. In FIT mode (params=None) the per-group statistics are
learned from df. In APPLY mode (params supplied) they are applied
verbatim with no re-fitting.
Two aggregation modes:
- STATIC (
as_of=None) -- one statistic per group computed over all rows, joined back to every row. Leakage-safe across a train/test split because the statistics are learned on the training data only. - AS-OF (
as_ofgiven) -- an expanding window: each row sees only rows strictly earlier in theas_ofcolumn. This refuses the 'lag feature that peeks at the future' anti-pattern in FEATURES_PHILOSOPHY.md.
When as_of is omitted but the data contains datetime columns, a
warning is emitted (the rows may be time-ordered).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
group
|
str or sequence of str
|
Column(s) to group by. |
None
|
value
|
str or sequence of str
|
Numeric column(s) to aggregate (any dtype for count / nunique). |
None
|
agg
|
('mean', 'median', 'sum', 'std', 'min', 'max', 'count', 'nunique', 'compare')
|
The aggregation. 'compare' writes nothing and reports every option. |
'mean','median','sum','std','min','max','count','nunique','compare'
|
as_of
|
str
|
A datetime or numeric column. When given, switches to the expanding (past-only) aggregation that prevents temporal leakage. |
None
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.aggfeat(df_train, group='city', value='price',
... agg='mean', return_params=True)
>>> df_te = dx.aggfeat(df_test, params=p) # apply, no re-fit
>>> dx.aggfeat(df, group='city', value='price', agg='compare')
bin(df, cols=None, method='equal_width', n_bins=5, *, labels=None, inplace=False, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Discretise continuous numeric columns into ordered bins.
Two modes. In FIT mode (params=None) the bin edges are learned from
df. In APPLY mode (params supplied) the saved edges are applied
verbatim with no re-fitting -- the safeguard against the 'bin edges chosen
on the full dataset' anti-pattern in FEATURES_PHILOSOPHY.md section 5.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns to bin. If |
None
|
method
|
('equal_width', 'quantile', 'kmeans', 'compare')
|
'equal_width' -> edges equally spaced between min and max. 'quantile' -> edges at quantiles (each bin holds ~equal counts). 'kmeans' -> edges from 1-D k-means cluster centres. 'compare' -> writes nothing; reports how balanced each method's bins would be so you can choose. |
'equal_width'
|
n_bins
|
int
|
Desired number of bins (>= 2). Ties may yield fewer; a warning is emitted when that happens. |
5
|
labels
|
sequence
|
Custom ordered bin labels. Default labels are |
None
|
inplace
|
bool
|
If False a new column |
False
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure.
The binned column has an ordered |
Examples:
>>> df_tr, p = dx.bin(df_train, cols=['price'], method='quantile',
... n_bins=4, return_params=True)
>>> df_te = dx.bin(df_test, params=p) # apply, no re-fit
>>> dx.bin(df, cols=['price'], method='compare') # explore options
cross(df, pairs=None, method='product', *, cols=None, degree=2, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Build interaction features by crossing numeric columns.
Two modes. In FIT mode (params=None) the recipe (which columns to
cross and how) is recorded. In APPLY mode (params supplied) the exact
same interactions are re-created on new data.
This function learns no statistics; interactions are deterministic
arithmetic. The params dict is a reproducible recipe.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
pairs
|
sequence of (str, str)
|
Explicit column pairs to cross. If |
None
|
method
|
('ratio', 'product', 'diff', 'polynomial', 'compare')
|
'ratio' -> a / b (division by zero becomes NaN, never Inf). 'product' -> a * b. 'diff' -> a - b. 'polynomial' -> powers (x2 .. xdegree) plus pairwise products. 'compare' -> writes nothing; reports the spread of each candidate. |
'ratio'
|
cols
|
sequence of str
|
Numeric columns used when |
None
|
degree
|
int
|
Highest power for 'polynomial'. |
2
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the recipe dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.cross(df_train, pairs=[('price', 'area')],
... method='ratio', return_params=True)
>>> df_te = dx.cross(df_test, params=p) # apply, same recipe
>>> dx.cross(df, cols=['price', 'area'], method='compare')
dtfeats(df, cols=None, method='both', *, features=None, drop_original=False, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Extract calendar and cyclical features from datetime columns.
Two modes. In FIT mode (params=None) the recipe (which features to
extract) is recorded. In APPLY mode (params supplied) the exact same
feature set is re-created on new data -- guaranteeing identical columns
between train and test.
This function learns no statistics; extraction is deterministic. The
params dict is a reproducible recipe, not a fitted transformer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Datetime columns. If |
None
|
method
|
('calendar', 'cyclical', 'both', 'compare')
|
'calendar' -> integer parts (year, month, dayofweek, is_weekend ...). 'cyclical' -> sin/cos pairs so the model sees December next to January. 'both' -> calendar + cyclical. 'compare' -> writes nothing; reports how many features each produces. |
'calendar'
|
features
|
sequence of str
|
Subset of calendar features to extract. Default is a 10-feature set. When given, exactly these calendar features are produced and the default cyclical sin/cos pairs are suppressed (issue #7). |
None
|
drop_original
|
bool
|
If True the source datetime column is dropped after extraction. |
False
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the recipe dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.dtfeats(df_train, cols=['signup'], method='both',
... return_params=True)
>>> df_te = dx.dtfeats(df_test, params=p) # apply, same columns
>>> dx.dtfeats(df, cols=['signup'], method='compare')
encode(df, cols=None, method='onehot', *, y=None, n_folds=5, order=None, drop_first=False, handle_unknown='ignore', inplace=False, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Encode categorical columns into model-ready numeric features.
Two modes. In FIT mode (params=None) the encoding maps are learned
from df. In APPLY mode (params supplied) the saved maps are applied
verbatim with no re-fitting -- the safeguard against leakage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Categorical columns to encode. If |
None
|
method
|
('onehot', 'ordinal', 'target', 'frequency', 'compare')
|
'onehot' -> one 0/1 column per category ( |
'onehot'
|
y
|
Series, array, or str
|
The target. Required for method='target' (a column name is accepted). |
None
|
n_folds
|
int
|
Number of out-of-fold folds for target encoding. |
5
|
order
|
list or dict
|
Explicit category order for ordinal encoding. A flat list for a single
column, or |
None
|
drop_first
|
bool
|
For one-hot, drop the first category to avoid the dummy-variable trap. |
False
|
handle_unknown
|
('ignore', 'error')
|
Apply-mode behaviour for categories unseen during fit. 'ignore' maps them to the default (one-hot: all-zero row); 'error' raises. |
'ignore'
|
inplace
|
bool
|
If False new column(s) are added and the source kept. If True the source column is replaced (one-hot drops it and adds the dummies). |
False
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.encode(df_train, cols=['city'], method='target',
... y=df_train['churn'], return_params=True)
>>> df_te = dx.encode(df_test, params=p) # apply, no re-fit
>>> dx.encode(df, cols=['city'], method='compare') # explore options
featpipe(df, steps=None, params=None, *, save_path=None, load_path=None, protect=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.6, dpi=110)
¶
Chain dextra's feature-engineering functions -- plus two leakage-prone
cleaning steps (handle_missing / clip_outliers) -- into one pipeline.
featpipe is the Stage 4.4 convenience wrapper. It runs transform,
scale, bin, encode, dtfeats, cross and aggfeat in
sequence, threading the transformed DataFrame from one step to the next,
and collects every step's params dict into a single combined,
versioned, JSON-serialisable artifact -- a lightweight feature store.
Two modes mirror the per-function contract in FEATURES_PHILOSOPHY.md.
In FIT mode (steps supplied) each step is fitted on df and its
params recorded. In APPLY mode (params or load_path supplied) the
saved per-step params are replayed verbatim, in order, with no re-fitting
-- the safeguard against leakage across a train/test boundary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
steps
|
sequence of dict
|
Fit-mode recipe. Inside the pipeline, steps that support
|
None
|
params
|
dict
|
Apply-mode artifact: a combined dict returned by an earlier fit.
Triggers apply mode; |
None
|
save_path
|
str
|
Fit mode only. After fitting, the combined params dict is written to this path as indented JSON. |
None
|
load_path
|
str
|
Apply-mode shortcut. The combined params dict is read from this JSON
file, then applied. Mutually exclusive with |
None
|
protect
|
sequence of str
|
Fit mode only. Columns isolated from EVERY step: steps that
auto-select their columns (e.g. a bare |
None
|
return_params
|
bool
|
If True the combined params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The fully transformed DataFrame, and -- when requested -- the combined params dict and/or the matplotlib figure. |
Notes
The combined params dict has the shape::
{"function": "featpipe", "version": ..., "fit_at": ...,
"steps": [<params of step 0>, <params of step 1>, ...],
"metadata": {"n_steps": ..., "step_summary": [...],
"input_shape": [...], "output_shape": [...]}}
Each element of steps is exactly the JSON-serialisable params dict the
corresponding function already returns, so the whole artifact survives
json.dump / json.load and reproduces the transform on another
machine or day.
Examples:
>>> recipe = [
... {'fn': 'transform', 'cols': ['income'], 'method': 'log1p'},
... {'fn': 'scale', 'cols': ['income_log1p', 'age'], 'method': 'robust'},
... {'fn': 'encode', 'cols': ['city'], 'method': 'onehot'},
... ]
>>> df_tr, p = dx.featpipe(df_train, steps=recipe, return_params=True,
... save_path='pipeline.json')
>>> df_te = dx.featpipe(df_test, params=p) # apply, no re-fit
>>> df_te2 = dx.featpipe(df_test, load_path='pipeline.json') # same result
scale(df, cols=None, method='standard', *, inplace=False, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Place numeric columns on a common scale.
Two modes. In FIT mode (params=None) the scaler statistics are learned
from df. In APPLY mode (params supplied) the saved statistics are
applied verbatim with no re-fitting -- the safeguard against statistic
leakage (computing the mean on train+test together).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns to scale. If |
None
|
method
|
('standard', 'minmax', 'robust', 'maxabs', 'compare')
|
'standard' -> (x - mean) / std (z-score) 'minmax' -> (x - min) / (max - min) (0..1) 'robust' -> (x - median) / IQR (outlier-resistant) 'maxabs' -> x / max(|x|) (-1..1, sign preserved) 'compare' -> writes nothing; reports the resulting range/spread of every candidate scaler so you can choose. |
'standard'
|
inplace
|
bool
|
If False a new column |
False
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.scale(df_train, cols=['price', 'age'], method='robust',
... return_params=True)
>>> df_te = dx.scale(df_test, params=p) # apply, no re-fit
>>> dx.scale(df, cols=['price'], method='compare') # explore options
transform(df, cols=None, method='log', *, inplace=False, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.2, dpi=110)
¶
Reshape numeric distributions toward symmetry / normality.
Two modes. In FIT mode (params=None) the transform is learned from
df and a reproducible params dict is produced. In APPLY mode
(params supplied) the saved transform is applied verbatim with no
re-fitting -- the safeguard against statistic leakage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns to transform. If |
None
|
method
|
('log', 'log1p', 'sqrt', 'boxcox', 'yeojohnson', 'compare')
|
|
'log'
|
inplace
|
bool
|
If False a new column |
False
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
And, when requested, the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.transform(df_train, cols=['price'], method='boxcox',
... return_params=True)
>>> df_te = dx.transform(df_test, params=p) # apply, no re-fit
>>> dx.transform(df, cols=['price'], method='compare') # explore options
classify(df, y=None, cols=None, method='forest', *, cv=5, standardize=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=5.0, dpi=110)
¶
Fit an instant, cross-validated classification baseline in one line.
Mirrors :func:regress exactly (fit / apply / compare, hybrid artifact)
for a categorical target. In FIT mode the predicted labels are appended as
"<target>_pred"; APPLY mode predicts with the saved estimator (no
re-fit); COMPARE mode cross-validates every candidate and writes nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y
|
str or array - like
|
The categorical target (a column name, or a Series/array aligned to
|
None
|
cols
|
sequence of str
|
Feature columns. Defaults to every numeric (non-boolean) column except the target. Ignored in apply mode. |
None
|
method
|
('logistic', 'tree', 'forest', 'knn', 'compare')
|
The baseline classifier, or 'compare' to rank them all. |
'logistic'
|
cv
|
int
|
Stratified cross-validation folds (clamped to the rarest class count). |
5
|
standardize
|
bool
|
Default: |
None
|
params
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
return_params
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
show
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
plot
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
return_df
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
return_fig
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
decimals
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
df_name
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The input frame plus a predicted-label column, and -- when requested --
the hybrid |
Examples:
>>> out, p = dx.classify(df_train, y='churn', method='forest', return_params=True)
>>> preds = dx.classify(df_test, params=p) # apply, no re-fit
>>> dx.classify(df_train, y='churn', method='compare') # rank baselines
cluster(df, cols=None, method='kmeans', *, k=None, k_range=(2, 10), standardize=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=5.0, dpi=110)
¶
Fit an instant clustering baseline in one line (unsupervised; no target).
Mirrors :func:regress / :func:classify (fit / apply / compare, hybrid
artifact) but takes no y -- clustering finds the data's own
structure. In FIT mode the cluster label of each row is appended as
"cluster"; if k is None the number of clusters is chosen
automatically by maximising the silhouette over k_range. APPLY mode
assigns clusters to new data with the saved estimator (no re-fit); COMPARE
mode evaluates every candidate over k_range and writes nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Feature columns. Defaults to every numeric (non-boolean) column. Ignored in apply mode. |
None
|
method
|
('kmeans', 'agglomerative', 'compare')
|
The baseline clusterer, or 'compare' to rank them all. |
'kmeans'
|
k
|
int
|
Fixed number of clusters. If |
None
|
k_range
|
(int, int)
|
Inclusive search range for k when |
(2, 10)
|
standardize
|
bool
|
Whether to z-score features before clustering. Default |
None
|
params
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
return_params
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
show
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
plot
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
return_df
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
return_fig
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
decimals
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
df_name
|
Optional[dict]
|
Standard dextra flags (see :func: |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The input frame plus a |
Examples:
>>> out, p = dx.cluster(df, method='kmeans', return_params=True) # auto-k
>>> labels = dx.cluster(df_new, params=p) # apply, no re-fit
>>> dx.cluster(df, method='compare') # rank baselines
regress(df, y=None, cols=None, method='forest', *, cv=5, standardize=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=5.0, dpi=110)
¶
Fit an instant, cross-validated regression baseline in one line.
Three modes. In FIT mode (params=None and a concrete method) one
baseline regressor is trained on df and its in-sample predictions are
appended as "<target>_pred". In APPLY mode (params supplied) the
saved fitted estimator predicts on new data with no re-fit -- the safeguard
against modeling leakage. In COMPARE mode (method='compare') every
candidate regressor is cross-validated and ranked, but nothing is written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y
|
str or array - like
|
The numeric target. A column name (fit mode) or a Series/array aligned
to |
None
|
cols
|
sequence of str
|
Feature columns. If |
None
|
method
|
('linear', 'ridge', 'lasso', 'tree', 'forest', 'compare')
|
The baseline algorithm, or 'compare' to rank them all. |
'linear'
|
cv
|
int
|
Number of cross-validation folds (clamped to |
5
|
standardize
|
bool
|
Whether to z-score features. Default: |
None
|
params
|
dict
|
A hybrid artifact from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the hybrid |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The input frame plus a prediction column, and -- when requested -- the
hybrid |
Examples:
>>> out, p = dx.regress(df_train, y='price', method='forest', return_params=True)
>>> preds = dx.regress(df_test, params=p) # apply, no re-fit
>>> dx.regress(df_train, y='price', method='compare') # rank baselines
plot_boxplots(df, cols=None, decimals=2, iqr_multiplier=1.5, width=1400, row_height=350, opacity=0.7, line_color='orange', template='plotly_white', show_grid=True, title='Boxplots', colors=None, df_name=None, show=True, plot=True, return_fig=False, return_df=False, return_params=False, params=None)
¶
Stacked horizontal box-plots with annotated statistics (Plotly).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. |
required |
cols
|
sequence of str
|
Columns to plot. Defaults to every numeric column in |
None
|
decimals
|
int
|
Fractional digits used in hover text and annotation labels. |
``2``
|
iqr_multiplier
|
float
|
Multiplier used to derive the dashed outlier bounds. |
``1.5``
|
width
|
int
|
Figure dimensions, in pixels. Total height = |
1400
|
row_height
|
int
|
Figure dimensions, in pixels. Total height = |
1400
|
opacity
|
float
|
Marker / box opacity. |
0.7
|
line_color
|
str
|
Colour of the dashed outlier-bound lines. |
'orange'
|
template
|
str
|
Any Plotly figure template (e.g. |
'plotly_white'
|
show_grid
|
bool
|
Whether to draw the x-axis grid. |
True
|
title
|
str
|
Top-level figure title. |
'Boxplots'
|
colors
|
sequence or mapping of colour strings
|
Either a list that will be zipped against |
None
|
show
|
bool
|
Same semantics as :func: |
True
|
return_fig
|
bool
|
Same semantics as :func: |
True
|
return_df
|
bool
|
Same semantics as :func: |
True
|
Returns:
| Type | Description |
|---|---|
Figure, DataFrame, tuple of (Figure, DataFrame), or None
|
|
Examples:
>>> dx.plot_boxplots(df)
>>> fig, stats = dx.boxpl(df, return_fig=True, return_df=True, show=False)
plot_histograms(df, cols=None, bins='auto', decimals=2, iqr_multiplier=1.5, fig_width=17.0, fig_row_height=4.8, width_ratios=(3, 1), dpi=120, hist_color='skyblue', hist_edgecolor='black', alpha=0.85, kde=True, kde_color='blue', kde_linewidth=2.2, title='Histograms with adjacent statistical summary', save=False, output_dir='plots', filename='histograms_with_summary.png', df_name=None, show=True, plot=True, return_fig=False, return_df=False, return_params=False, params=None)
¶
Draw a histogram + KDE for each selected column with a side-panel summary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. |
required |
cols
|
sequence of str
|
Columns to plot. Defaults to every numeric column in |
None
|
bins
|
int
|
Number of histogram bins. |
``20``
|
decimals
|
int
|
Number of fractional digits in the side-panel summary. |
``2``
|
iqr_multiplier
|
float
|
Multiplier on IQR for the outlier bounds reported in the side panel. |
``1.5``
|
fig_width
|
float
|
Figure dimensions, in inches. Total height = |
17.0
|
fig_row_height
|
float
|
Figure dimensions, in inches. Total height = |
17.0
|
width_ratios
|
sequence of float
|
Relative widths of the plot column and the text column. |
``(3, 1)``
|
dpi
|
int
|
Figure resolution. |
``120``
|
hist_color
|
(str, str, float)
|
Histogram style. |
'skyblue'
|
hist_edgecolor
|
(str, str, float)
|
Histogram style. |
'skyblue'
|
alpha
|
(str, str, float)
|
Histogram style. |
'skyblue'
|
kde
|
bool
|
Whether to overlay a kernel-density estimate. |
``True``
|
kde_color
|
(str, float)
|
KDE style. |
'blue'
|
kde_linewidth
|
(str, float)
|
KDE style. |
'blue'
|
title
|
str
|
Figure-level super-title. |
'Histograms with adjacent statistical summary'
|
save
|
bool
|
If |
``False``
|
output_dir
|
str
|
Path used when |
'plots'
|
filename
|
str
|
Path used when |
'plots'
|
show
|
bool
|
If |
``True``
|
return_fig
|
bool
|
Whether to return the Matplotlib |
``False``
|
return_df
|
bool
|
Whether to return the Matplotlib |
``False``
|
Returns:
| Type | Description |
|---|---|
Figure, DataFrame, tuple of (Figure, DataFrame), or None
|
Depends on |
Examples:
>>> dx.plot_histograms(df)
>>> dx.hister(df, cols=["income"], bins=30)
edareport(df, *, out='report.html', target=None, title=None, sections=None, include_model=False, task='auto', max_hist=24, top_cat=10, theme='light', return_params=False, show=True, decimals=4, df_name=None)
¶
Compose a single self-contained HTML EDA report in one call.
Orchestrates the tested functions of Phases 1-8 -- it computes nothing new.
Builds, in order: an Overview, a Data-quality section (missing / duplicates
/ outliers), a Univariate section (numeric summary + histograms + top
categorical frequencies), a Bivariate section (correlation, plus class
balance when a categorical target is given), and -- when
include_model=True and a target is supplied -- a Baseline model &
evaluation section (a random-forest baseline trained on a split and judged
on the held-out test split, via Phases 6-7). Every figure is embedded as a
base64 PNG and every table inline, so the output is one portable file.
Sections are isolated: any that cannot run is skipped with a recorded reason and the rest of the report is still written. The input DataFrame is never mutated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data to report on. Never mutated. |
required |
out
|
str
|
Path of the HTML file to write. |
'report.html'
|
target
|
str
|
A column used by the class-balance and (optional) model sections. |
None
|
title
|
str
|
Report title (defaults to |
None
|
sections
|
sequence of str
|
Subset of |
None
|
include_model
|
bool
|
Build the optional baseline model / evaluation section (needs |
False
|
task
|
('auto', 'regression', 'classification')
|
Task of the model section. |
"auto"
|
max_hist
|
int
|
Cap on the number of numeric columns plotted as histograms. |
24
|
top_cat
|
int
|
Number of categorical columns to tabulate as frequency tables. |
10
|
theme
|
str
|
Visual theme ( |
'light'
|
return_params
|
bool
|
Return the JSON-safe build manifest (including the |
False
|
show
|
bool
|
Print the one-line |
True
|
decimals
|
int
|
Numeric formatting precision in the embedded tables. |
4
|
df_name
|
str
|
Name used in the title / audit (inferred when omitted). |
None
|
Returns:
| Type | Description |
|---|---|
str or dict
|
The output path, or -- when |
Examples:
>>> dx.edareport(df) # writes report.html
>>> dx.edareport(df, target='churn', include_model=True)
>>> manifest = dx.edareport(df, return_params=True)
importance(df, y=None, cols=None, method='tree', *, keep=10, threshold=None, task=None, protect=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.6, dpi=110)
¶
Rank features by a trained model's importances, keep the strongest.
An Embedded-family selector: it trains one model and reads the selection that model implies. Requires scikit-learn (imported lazily).
Two modes. In FIT mode (params=None) a model is trained on df /
y and its importances ranked. In APPLY mode (params supplied) the
saved decision is replayed verbatim -- the DataFrame is subset to the kept
columns with no re-training -- the safeguard against leakage. y is not
needed in apply mode.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y
|
str or array - like
|
The target. A column name or an array-like aligned to |
None
|
cols
|
sequence of str
|
The candidate-feature pool (numeric). If |
None
|
method
|
('tree', 'l1', 'linear', 'compare')
|
'tree' -> random-forest |
'tree'
|
keep
|
int
|
Keep the top- |
10
|
threshold
|
float
|
Used only when |
None
|
task
|
('classification', 'regression')
|
Force the task type; by default it is inferred from |
'classification'
|
protect
|
sequence of str
|
Columns that must never be dropped. |
None
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The DataFrame with low-importance candidate columns removed, and -- when requested -- the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.importance(df_train, y='churn', method='tree', keep=8,
... return_params=True)
>>> df_te = dx.importance(df_test, params=p) # apply, no re-train
>>> dx.importance(df, y='churn', method='compare') # explore
redundancy(df, cols=None, method='variance', *, threshold=None, protect=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.6, dpi=110)
¶
Drop features that carry little or duplicated information (no target).
A Filter-family selector. It judges columns purely on their intrinsic statistics -- never looking at a target -- and removes the redundant ones.
Two modes. In FIT mode (params=None) the candidates are scored on
df. In APPLY mode (params supplied) the saved decision is replayed
verbatim: the DataFrame is subset to the kept columns with no re-scoring
-- the safeguard against selection leakage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
The candidate-feature pool. If |
None
|
method
|
('variance', 'correlation', 'vif', 'compare')
|
'variance' -> drop features whose variance <= |
'variance'
|
threshold
|
float
|
The cut value; its meaning depends on |
None
|
protect
|
sequence of str
|
Columns that must never be dropped even if flagged. |
None
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The DataFrame with redundant candidate columns removed, and -- when requested -- the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.redundancy(df_train, method='correlation',
... threshold=0.9, return_params=True)
>>> df_te = dx.redundancy(df_test, params=p) # apply, no re-score
>>> dx.redundancy(df, method='compare') # explore
relevance(df, y=None, cols=None, method='anova', *, keep=10, threshold=None, protect=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.6, dpi=110)
¶
Rank features by their univariate association with the target, keep top.
A Filter-family selector. Each candidate feature is scored against the
target y by a univariate criterion; the strongest are kept.
Two modes. In FIT mode (params=None) the candidates are scored on
df / y. In APPLY mode (params supplied) the saved decision is
replayed verbatim -- the DataFrame is subset to the kept columns with no
re-scoring -- the safeguard against target leakage. y is not needed
in apply mode.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y
|
str or array - like
|
The target. A column name in |
None
|
cols
|
sequence of str
|
The candidate-feature pool (numeric). If |
None
|
method
|
('anova', 'chi2', 'mutualinfo', 'compare')
|
'anova' -> ANOVA F-test (classification) or regression F-test. 'chi2' -> chi-squared (non-negative features, classification). 'mutualinfo' -> mutual information (captures non-linear dependence; lazily imports scikit-learn). 'compare' -> rank by all three criteria; keep / drop nothing. |
'anova'
|
keep
|
int
|
Keep the top- |
10
|
threshold
|
float
|
Used only when |
None
|
protect
|
sequence of str
|
Columns that must never be dropped. |
None
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The DataFrame with low-relevance candidate columns removed, and -- when requested -- the params dict and/or the matplotlib figure. |
Notes
Scoring uses rows that are complete across the candidate features
and the target. On wide-missing data impute first --
dx.handle_missing(df) (or a handle_missing step before this
one in a featpipe recipe) -- otherwise the scorer raises with
this exact remedy. Per-feature scoring on incomplete rows is
declared debt (tracked in the project issues), not in 0.6.0.
Examples:
>>> df_tr, p = dx.relevance(df_train, y='churn', method='anova', keep=8,
... return_params=True)
>>> df_te = dx.relevance(df_test, params=p) # apply, no re-score
>>> dx.relevance(df, y='churn', method='compare') # explore
rfe(df, y=None, cols=None, *, keep=10, estimator='tree', step=1, task=None, protect=None, params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.6, dpi=110)
¶
Recursive Feature Elimination -- the wrapper-family selector.
Fits a model, drops the weakest feature(s), refits, and repeats until only
keep features remain. Requires scikit-learn (imported lazily).
Two modes. In FIT mode (params=None) the elimination runs on df /
y. In APPLY mode (params supplied) the saved decision is replayed
verbatim -- the DataFrame is subset to the kept columns with no re-fitting.
y is not needed in apply mode.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
y
|
str or array - like
|
The target. Required in fit mode; ignored in apply mode. Auto-protected. |
None
|
cols
|
sequence of str
|
The candidate-feature pool (numeric). If |
None
|
keep
|
int
|
Number of features RFE should keep. |
10
|
estimator
|
('tree', 'linear', 'compare')
|
The model RFE recurses on: a random forest, a linear/logistic model, or 'compare' to run both and report without dropping anything. |
'tree'
|
step
|
int
|
How many features to eliminate per iteration. |
1
|
task
|
('classification', 'regression')
|
Force the task type; by default it is inferred from |
'classification'
|
protect
|
sequence of str
|
Columns that must never be dropped. |
None
|
params
|
dict
|
A params dict from an earlier fit. Triggers apply mode. |
None
|
return_params
|
bool
|
If True the learned params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The DataFrame reduced to the kept features, and -- when requested -- the params dict and/or the matplotlib figure. |
Examples:
>>> df_tr, p = dx.rfe(df_train, y='churn', keep=8, estimator='tree',
... return_params=True)
>>> df_te = dx.rfe(df_test, params=p) # apply, no re-fit
>>> dx.rfe(df, y='churn', estimator='compare') # explore
selectpipe(df, steps=None, params=None, *, y=None, save_path=None, load_path=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.6, dpi=110)
¶
Chain the four dextra feature-selection functions into one pipeline.
selectpipe is the Stage 5.3 convenience wrapper. It runs redundancy,
relevance, importance and rfe in sequence, threading the
progressively narrower DataFrame from one step to the next, and collects
every step's params dict into a single combined, versioned,
JSON-serialisable artifact -- a lightweight selection record.
Two modes. In FIT mode (steps supplied) each step is fitted in order;
a step's candidate pool is whatever survived the previous steps. In APPLY
mode (params or load_path supplied) the saved per-step decisions
are replayed verbatim -- each step only subsets to its kept columns, no
re-scoring -- the safeguard against selection leakage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
steps
|
sequence of dict
|
Fit-mode recipe. Each dict has a |
None
|
params
|
dict
|
Apply-mode artifact: a combined dict returned by an earlier fit.
Triggers apply mode; |
None
|
y
|
str or array - like
|
The shared target. Auto-injected into every |
None
|
save_path
|
str
|
Fit mode only. Writes the combined params dict to this path as JSON. |
None
|
load_path
|
str
|
Apply-mode shortcut. Reads the combined params from this JSON file,
then applies it. Mutually exclusive with |
None
|
return_params
|
bool
|
If True the combined params dict is returned alongside the DataFrame. |
False
|
show
|
standard dextra flags.
|
|
True
|
plot
|
standard dextra flags.
|
|
True
|
return_df
|
standard dextra flags.
|
|
True
|
return_fig
|
standard dextra flags.
|
|
True
|
decimals
|
standard dextra flags.
|
|
True
|
df_name
|
standard dextra flags.
|
|
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The DataFrame reduced to the surviving features, and -- when requested -- the combined params dict and/or the matplotlib figure. |
Examples:
>>> recipe = [
... {'fn': 'redundancy', 'method': 'correlation', 'threshold': 0.9},
... {'fn': 'relevance', 'method': 'anova', 'keep': 15},
... {'fn': 'importance', 'method': 'tree', 'keep': 8},
... ]
>>> df_tr, p = dx.selectpipe(df_train, steps=recipe, y='churn',
... return_params=True, save_path='select.json')
>>> df_te = dx.selectpipe(df_test, params=p) # apply, no re-score
>>> df_te2 = dx.selectpipe(df_test, load_path='select.json') # same result
describe_numeric(df, cols=None, decimals=2, df_name=None, iqr_multiplier=1.5, ddof=1, metrics_as_rows=True, show=True, return_df=False, return_fig=False, return_params=False, params=None, plot=False, raw=False)
¶
Return a rich numeric summary of df.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Non-numeric values in selected columns are coerced to NaN. |
required |
cols
|
sequence of str
|
Columns to summarise. Defaults to every numeric column in |
None
|
decimals
|
int
|
Number of fractional digits used for formatting. |
``2``
|
df_name
|
str
|
Name shown in the summary header. If omitted, the caller-side variable name is inferred when possible. |
None
|
iqr_multiplier
|
float
|
Multiplier on the IQR when deriving lower / upper bounds for
outlier detection. Tukey's classical value is |
``1.5``
|
metrics_as_rows
|
bool
|
If |
``True``
|
show
|
bool
|
If |
``True``
|
return_df
|
bool
|
If |
``False``
|
raw
|
bool
|
When returning a DataFrame, controls whether numbers are kept as
|
``False``
|
Returns:
| Type | Description |
|---|---|
DataFrame or None
|
The summary frame, or |
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
KeyError
|
If any entry of |
ValueError
|
If the resolved column set contains no numeric data. |
Examples:
>>> import pandas as pd, numpy as np
>>> df = pd.DataFrame({'a': np.random.randn(100), 'b': np.random.randn(100)})
>>> summary = describe_numeric(df, return_df=True, raw=True, show=False)
>>> summary.loc['mean']
anova_oneway(df, group_col, value_col, alpha=0.05, decimals=4, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=14.0, fig_height=4.8, dpi=110)
¶
One-way ANOVA: compare means of >= 2 groups.
H0: all group means are equal. p < alpha => at least one differs. Source: F-M05-L08-01.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Tidy input data. Never mutated. |
required |
group_col
|
str
|
Categorical column defining the groups. |
required |
value_col
|
str
|
Numeric column whose means are compared. |
required |
alpha
|
float
|
Significance threshold. |
0.05
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.anova_oneway(df, "city", "income")
>>> dx.aov1(df, "plan", "monthly_charge")
chi_square_independence(df, row, col, alpha=0.05, decimals=4, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=12.0, fig_height=4.8, dpi=110)
¶
Chi-square test of independence between two categoricals.
H0: the two variables are independent. Expected counts come from the contingency table. Source: F-M05-L09-01.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
row
|
str
|
The two categorical columns to cross-tabulate. |
required |
col
|
str
|
The two categorical columns to cross-tabulate. |
required |
alpha
|
float
|
Significance threshold. |
0.05
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.chi_square_independence(df, "city", "churn")
>>> dx.chi2ind(df, "plan", "churn")
class_imbalance(target, col=None, decimals=2, name=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=12.0, fig_height=4.5, dpi=110)
¶
Class balance report for a classification target.
Computes count, percentage, and imbalance ratio (majority / minority). Severity classification: ratio < 2 -> 'balanced' 2-5 -> 'mild' 5-10 -> 'moderate' 10-50 -> 'severe' >= 50 -> 'extreme'
Examples:
>>> dx.class_imbalance(df['target'])
>>> dx.class_imbalance(df, 'target') # (df, col) form
confidence_interval_mean(data, confidence=0.95, decimals=4, name=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=10.0, fig_height=3.0, dpi=110)
¶
t-based confidence interval for the population mean.
Uses the Student-t distribution with n-1 degrees of freedom. CI = mean +/- t_{alpha/2, n-1} * SE, SE = s / sqrt(n). Standard error formula: F-M04-L09-02.
Examples:
>>> dx.confidence_interval_mean([12, 14, 11, 15, 13, 16])
confidence_interval_proportion(successes, n, confidence=0.95, method='wilson', decimals=4, name=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=10.0, fig_height=3.0, dpi=110)
¶
Confidence interval for a population proportion.
Methods: 'wald' - normal approximation, classical. Fails for small n / extreme p_hat. 'wilson' - score interval. Recommended for small n or extreme p_hat.
Examples:
>>> dx.confidence_interval_proportion(successes=42, n=200)
>>> dx.cip(successes=3, n=25, method="wilson")
correlation_matrix(df, cols=None, method='pearson', decimals=2, alpha=0.05, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, return_p=False, annot=True, mask_lower=False, cmap='RdBu_r', fig_width=12.0, fig_height=8.5, dpi=110)
¶
Correlation matrix with p-values and a labelled heatmap.
Pearson source: F-M03-L10-01. Spearman/Kendall are rank-based robust alternatives.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
('pearson', 'spearman', 'kendall')
|
|
'pearson'
|
alpha
|
float
|
|
0.05
|
return_p
|
if True, return (r_matrix, p_matrix) instead of r alone.
|
|
False
|
Examples:
>>> dx.correlation_matrix(df)
>>> dx.correlation_matrix(df, method='spearman')
cross_tab(df, row, col, normalize=None, alpha=0.05, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, return_test=False, cmap='Blues', fig_width=12.0, fig_height=6.0, dpi=110)
¶
Bivariate categorical: contingency table + chi-square + Cramér's V.
Cramér's V interpretation (Cohen): < 0.1 : negligible 0.1 - 0.3 : weak 0.3 - 0.5 : moderate >= 0.5 : strong
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
normalize
|
('index', 'columns', 'all', None)
|
Forwarded to |
'index'
|
return_test
|
bool
|
If True, return a dict with chi2, dof, p, cramers_v alongside the table. |
False
|
Examples:
>>> dx.cross_tab(df, row='gender', col='product')
empirical_rule_check(df, cols=None, decimals=2, df_name=None, tolerance=3.0, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=14.0, fig_row_height=3.8, dpi=110)
¶
Check actual vs theoretical 68/95/99.7 coverage.
Compares the share of values inside mean +/- 1/2/3 sigma with the Normal expectation -- a quick empirical normality sanity check.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns. Default: every numeric column. |
None
|
tolerance
|
float
|
Allowed deviation (in percentage points) before a band is flagged as off-Normal. |
3.0
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.empirical_rule_check(df)
>>> dx.emprule(df, return_df=True)
frequency_table(df, col, top_n=None, ascending=False, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=12.0, fig_height=5.5, dpi=110)
¶
Frequency table for a categorical column with Pareto chart.
Columns returned: category, count, pct, cumulative_count, cumulative_pct.
Examples:
>>> dx.frequency_table(df, 'category')
>>> dx.frequency_table(df, 'category', top_n=10)
group_compare(df, group_col, value_cols=None, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=14.0, fig_row_height=4.0, dpi=110)
¶
Compare numeric metrics across groups defined by a categorical column.
For each (group, value_col) pair, computes: n, mean, std, median, min, max, IQR.
Examples:
>>> dx.group_compare(df, group_col='region', value_cols='sales')
missing_report(df, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=14.0, fig_height=6.5, dpi=110)
¶
Comprehensive missing-values report per column.
Returns a per-column summary with: dtype, n_total, n_missing, pct_missing,
sample_value, and a heuristic recommendation based on pct_missing:
0% -> 'OK'
< 5% -> 'impute_mean/median/mode' (depending on dtype)
5 - 30% -> 'review'
30 - 60% -> 'consider_drop'
> 60% -> 'drop_column'
Visual: bar chart of pct_missing per column + missingness pattern heatmap.
Examples:
>>> dx.missing_report(df)
normality_test(data, method='auto', alpha=0.05, decimals=4, name=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=12.0, fig_height=4.5, dpi=110)
¶
Test whether data come from a Normal distribution.
H0: data are Normally distributed. p < alpha => Reject H0 (data are NOT Normal).
Methods: 'auto' : Shapiro-Wilk if n < 5000, else D'Agostino-Pearson. 'shapiro' : Shapiro-Wilk. 'normaltest' : D'Agostino-Pearson omnibus. 'jarque_bera' : Jarque-Bera (based on skew/kurt).
Examples:
>>> dx.normality_test(df["income"])
>>> dx.normtest(df["income"], method="jarque_bera")
outliers_report(df, cols=None, method='iqr', k=1.5, z_threshold=3.0, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, return_rows=False, fig_width=14.0, fig_row_height=3.5, dpi=110)
¶
Detect outlier rows by IQR fence or Z-score.
method='iqr' flags values outside Q1/Q3 -/+ multiplier*IQR; method='zscore' flags |Z| > threshold. Report only -- nothing is removed (use dx.clip_outliers to treat them).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns. Default: every numeric column. |
None
|
method
|
('iqr', 'zscore')
|
Detection rule (see above). |
'iqr'
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.outliers_report(df)
>>> dx.outrep(df, method="zscore", return_df=True)
pearson_skewness(df, cols=None, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=14.0, fig_row_height=3.5, dpi=110)
¶
Karl Pearson's skewness coefficient: 3*(mean - median)/sigma.
Sign says direction (positive = right tail), magnitude says how far mean and median disagree in sigma units. Source: F-M03-L06-01.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns. Default: every numeric column. |
None
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.pearson_skewness(df)
>>> dx.pskew(df, cols=["income"], return_df=True)
sample_size_mean(margin_error, std, confidence=0.95, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=10.0, fig_height=4.5, dpi=110)
¶
Required sample size to estimate a mean within +/- margin_error.
Formula: n = (z_{alpha/2} * sigma / E)^2
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
margin_error
|
float
|
Acceptable half-width E of the interval. |
required |
std
|
float
|
Population (or pilot) standard deviation. |
required |
confidence
|
float
|
Confidence level. |
0.95
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.sample_size_mean(margin_error=2.0, std=12.0)
>>> dx.ssm(margin_error=1.5, std=8.0, confidence=0.99)
sample_size_proportion(margin_error, p=0.5, confidence=0.95, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=12.0, fig_height=4.5, dpi=110)
¶
Required sample size to estimate a proportion within +/- margin_error.
Formula (F-M04-L07-01): n = z^2 * p * (1 - p) / E^2
Worst-case p = 0.5 gives the largest n. Always reported alongside the user-specified p so survey planners see both.
Examples:
>>> dx.sample_size_proportion(margin_error=0.03)
>>> dx.ssp(p=0.2, margin_error=0.05, confidence=0.99)
simple_linear_regression(df, x, y, alpha=0.05, decimals=4, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, return_residuals=False, ci_band=True, fig_width=16.0, fig_height=5.0, dpi=110)
¶
Simple linear regression Y = m*X + b with full diagnostics.
Source: course formulas F-M03-L11-01 : Y = m*X + b F-M03-L11-02 : m = r * (s_y / s_x) F-M03-L11-03 : b = mean(y) - m * mean(x)
Visual: 3-panel plot (1) scatter + regression line + (1-alpha) CI band (2) residuals vs fitted (homoscedasticity check) (3) Q-Q plot of residuals (normality check)
Examples:
>>> dx.simple_linear_regression(df, x='age', y='income')
t_test_one_sample(data, popmean, alternative='two-sided', alpha=0.05, decimals=4, name=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=10.0, fig_height=4.5, dpi=110)
¶
One-sample t-test against a hypothesized population mean.
H0: mu == popmean. p < alpha rejects H0. Source: F-M05-L05-01.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
array - like or Series
|
Sample observations. |
required |
popmean
|
float
|
Hypothesized population mean under H0. |
required |
alternative
|
('two-sided', 'less', 'greater')
|
Test direction. |
'two-sided'
|
alpha
|
float
|
Significance threshold. |
0.05
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.t_test_one_sample(df["weight"], popmean=70)
>>> dx.t1(df["weight"], 70, alternative="greater")
t_test_paired(before, after, alternative='two-sided', alpha=0.05, decimals=4, name_before=None, name_after=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=12.0, fig_height=4.5, dpi=110)
¶
Paired t-test (before vs after, matched pairs).
H0: mean difference == 0. Pairs with a missing side are dropped. Source: F-M05-L07-01.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
before
|
array - like or Series
|
Matched measurements, same length and order. |
required |
after
|
array - like or Series
|
Matched measurements, same length and order. |
required |
alternative
|
('two-sided', 'less', 'greater')
|
Test direction. |
'two-sided'
|
alpha
|
float
|
Significance threshold. |
0.05
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.t_test_paired(df["before"], df["after"])
>>> dx.tpair(pre, post, alternative="less")
t_test_two_sample(group1, group2, alternative='two-sided', alpha=0.05, equal_var=False, decimals=4, name1=None, name2=None, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=14.0, fig_height=4.5, dpi=110)
¶
Independent two-sample t-test. Welch's (equal_var=False) by default.
Source: F-M05-L06-01. H0: mu_a == mu_b.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
group1
|
array - like or Series
|
The two independent samples. |
required |
group2
|
array - like or Series
|
The two independent samples. |
required |
equal_var
|
bool
|
False = Welch's test (robust to unequal variances). |
False
|
alternative
|
('two-sided', 'less', 'greater')
|
Test direction. |
'two-sided'
|
alpha
|
float
|
Significance threshold. |
0.05
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.t_test_two_sample(ctrl["y"], treat["y"])
>>> dx.t2(ctrl["y"], treat["y"], equal_var=True)
vif_scores(df, cols=None, threshold=10.0, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, fig_width=11.0, fig_height=5.0, dpi=110)
¶
Variance Inflation Factor for every numeric column.
VIF_i = 1 / (1 - R^2_i) where R^2_i is the R^2 of regressing column i on all other selected numeric columns.
Rule of thumb: VIF < 5 -> no multicollinearity concern 5 <= VIF < threshold -> review VIF >= threshold (default 10) -> drop or combine the feature
Used before fitting any linear / logistic regression.
Examples:
>>> dx.vif_scores(df[['x1', 'x2', 'x3']])
z_scores(df, cols=None, threshold=3.0, decimals=2, df_name=None, show=True, plot=True, return_df=False, return_fig=False, params=None, return_params=False, return_zscores=False, fig_width=14.0, fig_row_height=3.5, dpi=110)
¶
Compute Z-scores per column and report extreme-value counts.
Z = (x - mu) / sigma (source: F-M03-L05-02).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
Input data. Never mutated. |
required |
cols
|
sequence of str
|
Numeric columns to score. Default: every numeric column. |
None
|
threshold
|
float
|
|Z| above this counts as an extreme value. |
3.0
|
return_zscores
|
bool
|
Also return the full Z-score frame. |
False
|
show
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
plot
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_df
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
return_fig
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
decimals
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
df_name
|
standard
|
dextra flags (print report / draw figure / return objects). |
True
|
Examples:
>>> dx.z_scores(df)
>>> dx.zsc(df, threshold=2.5, return_df=True)
tsdecomp(df, value=None, *, time=None, period=None, model='additive', method='classical', params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=9.0, dpi=110)
¶
Decompose a time series into trend / seasonal / residual in one line.
Two input modes. In SERIES mode pass value (a column name or array-like)
and, ideally, time (a datetime column used for the x-axis and to infer
the seasonal period). In ARTIFACT mode pass params (a descriptor from
a previous tsdecomp call) and the data; the resolved value / time /
period / model / method are read back from it. Returns a dense components
frame (observed / trend / seasonal / resid), a four-panel
figure, and a one-line decision naming the trend and seasonal strengths.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data holding the series. Never mutated. |
required |
value
|
str or array - like
|
The observed series (column name or values). Required in series mode. |
None
|
time
|
str or array - like
|
A datetime column / values for the x-axis and period inference. |
None
|
period
|
int
|
The seasonal period (e.g. 12 for monthly). Inferred from |
None
|
model
|
('additive', 'multiplicative')
|
The decomposition model. |
"additive"
|
method
|
('classical', 'stl')
|
|
"classical"
|
params
|
dict
|
A Phase-8 descriptor for artifact mode. |
None
|
return_params
|
bool
|
Also return the JSON-safe descriptor (no estimator). |
False
|
show
|
see the dextra
|
standard flags. |
True
|
plot
|
see the dextra
|
standard flags. |
True
|
return_df
|
see the dextra
|
standard flags. |
True
|
return_fig
|
see the dextra
|
standard flags. |
True
|
decimals
|
see the dextra
|
standard flags. |
True
|
df_name
|
see the dextra
|
standard flags. |
True
|
fig_width
|
figure geometry.
|
|
13.0
|
fig_height
|
figure geometry.
|
|
13.0
|
dpi
|
figure geometry.
|
|
13.0
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The components frame, and -- when requested -- the descriptor and / or the matplotlib figure. |
Examples:
>>> dx.tsdecomp(df, value='sales', time='month', period=12)
>>> res, p = dx.tsdecomp(df, value='sales', time='month',
... return_params=True)
>>> dx.tsdecomp(df_new, params=p) # artifact mode
tsfcast(df, value=None, *, time=None, horizon=12, valid=None, period=None, method='auto', params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=13.0, fig_height=5.6, dpi=110)
¶
Baseline forecast for a series, validated on a held-out tail, in one line.
Trains a simple baseline on every observation before the last valid
points, scores it on that untouched tail (no look-ahead), then re-fits on the
full series to project horizon steps ahead. Baselines: naive (last
value), snaive (last season), drift (last value + average slope),
mean. method="auto" picks snaive when a seasonal period is
available else naive. method="compare" ranks every baseline on the
validation window (by MASE) and writes no artifact.
Two input modes: SERIES (value (+ time)) and ARTIFACT (params
from a prior call -- settings are replayed on the new data). Dependency-free.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data holding the series. Never mutated. |
required |
value
|
str or array - like
|
The observed series. Required in series mode. |
None
|
time
|
str or array - like
|
A datetime column / values for ordering, the x-axis and the future index. |
None
|
horizon
|
int
|
Number of steps to forecast ahead. |
12
|
valid
|
int
|
Length of the held-out validation tail (defaults to |
None
|
period
|
int
|
Seasonal period for |
None
|
method
|
('auto', 'naive', 'snaive', 'drift', 'mean', 'compare')
|
The baseline (or |
"auto"
|
params
|
Optional[dict]
|
see the dextra standard flags. |
None
|
return_params
|
Optional[dict]
|
see the dextra standard flags. |
None
|
show
|
Optional[dict]
|
see the dextra standard flags. |
None
|
plot
|
Optional[dict]
|
see the dextra standard flags. |
None
|
return_df
|
Optional[dict]
|
see the dextra standard flags. |
None
|
return_fig
|
Optional[dict]
|
see the dextra standard flags. |
None
|
decimals
|
Optional[dict]
|
see the dextra standard flags. |
None
|
df_name
|
Optional[dict]
|
see the dextra standard flags. |
None
|
fig_width
|
figure geometry.
|
|
13.0
|
fig_height
|
figure geometry.
|
|
13.0
|
dpi
|
figure geometry.
|
|
13.0
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
For a named method: a forward-forecast frame ( |
Examples:
>>> dx.tsfcast(df, value='sales', time='month', horizon=12)
>>> dx.tsfcast(df, value='sales', method='compare') # leaderboard
>>> fc, p = dx.tsfcast(df, value='sales', method='drift', return_params=True)
>>> dx.tsfcast(df_new, params=p) # artifact mode
tsstat(df, value=None, *, time=None, max_diff=2, alpha=0.05, regression='c', params=None, return_params=False, show=True, plot=True, return_df=True, return_fig=False, decimals=4, df_name=None, fig_width=14.0, fig_height=4.8, dpi=110)
¶
Test a series for stationarity (ADF + KPSS) and suggest differencing.
Two input modes. In SERIES mode pass value (a column name or array-like)
and, optionally, time. In ARTIFACT mode pass params (a descriptor
from a previous tsstat call) and the data; value / time /
alpha / regression / max_diff are read back from it. Runs the
Augmented Dickey-Fuller test (null: unit root) and the KPSS test (null:
stationarity) -- complementary nulls -- reports both, gives the classic
four-case verdict, and suggests a differencing order d by differencing
until ADF rejects a unit root AND KPSS fails to reject stationarity (capped
at max_diff). Returns a two-row test table, a three-panel figure
(series + rolling mean, rolling std, ACF) and a one-line decision.
Requires statsmodels (the optional ts extra), imported lazily.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame
|
The data holding the series. Never mutated. |
required |
value
|
str or array - like
|
The observed series. Required in series mode. |
None
|
time
|
str or array - like
|
A datetime column / values for ordering and the x-axis. |
None
|
max_diff
|
int
|
The cap on the suggested differencing order. |
2
|
alpha
|
float
|
Significance level for both tests and the verdict. |
0.05
|
regression
|
('c', 'ct')
|
Deterministic term: |
"c"
|
params
|
dict
|
A Phase-8 descriptor for artifact mode. |
None
|
return_params
|
see the
|
dextra standard flags. |
False
|
show
|
see the
|
dextra standard flags. |
False
|
plot
|
see the
|
dextra standard flags. |
False
|
return_df
|
see the
|
dextra standard flags. |
False
|
return_fig
|
see the
|
dextra standard flags. |
False
|
decimals
|
see the
|
dextra standard flags. |
False
|
df_name
|
see the
|
dextra standard flags. |
False
|
fig_width
|
figure geometry.
|
|
14.0
|
fig_height
|
figure geometry.
|
|
14.0
|
dpi
|
figure geometry.
|
|
14.0
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The ADF / KPSS test table, and -- when requested -- the descriptor and / or the matplotlib figure. |
Examples:
>>> dx.tsstat(df, value='sales', time='month')
>>> tbl, p = dx.tsstat(df, value='sales', return_params=True)
>>> dx.tsstat(df_new, params=p) # artifact mode
functions()
¶
Print every public dextra function, grouped by phase, with a summary.
A zero-dependency discoverability aid: import dextra as dx;
dx.functions() lists the whole public API (functions and their short
aliases) organised by phase, each with its one-line docstring summary.
Examples:
>>> import dextra as dx
>>> dx.functions()