Skip to content

Batch

Run a set of cells as one job: build a journal, load/update the cells, and aggregate their summaries.

cellpy.batch is the 2.1 batch subsystem (it replaced the cellpy.utils.batch_tools machinery). cellpy.utils.batch remains as a thin re-export of the entry points below.

Entry points

batch

cellpy.batch -- the batch subsystem.

A boring, standard architecture for batch processing. cellpy.utils.batch is a thin re-export/shim.

Modules:

  • layout -- BatchPaths –

    pure path computation + ensure_dirs

Batch 

Batch(journal: Journal, policy: LoadPolicy | None = None, _db: dict | None = None)

A batch of cells: a journal, a lazy cell store, and derived frames.

experiment property 

experiment: '_LegacyExperimentAdapter'

Backward-compat view for legacy consumers (helpers/collectors).

cellpy.utils.helpers and cellpy.utils.collectors still reach into b.experiment.{cell_names,data,journal.pages,summary_frames}. This adapter keeps them working against the new Batch until they are migrated (Epic B/C); it is not part of the blessed API.

tests property 

tests: DataFrame

Per-test metadata across the batch (tidy long-format).

One row per (cell, test_id) with the native TestMeta fields plus cell/group/sub_group keys -- the per-test records a merged (campaign) cell carries. Empty frame if no cell exposes test metadata.

create_journal 

create_journal(**kwargs) -> Journal

Populate the journal from the configured database (if any).

Mirrors the legacy init() -> create_journal() flow: init stores the db config, create_journal performs the read.

from_cells classmethod 

from_cells(cells: Mapping[str, Any] | Sequence[Any], *, groups: Mapping[str, Any] | None = None, sub_groups: Mapping[str, Any] | None = None, group_labels: Mapping[Any, str] | None = None, selected: Mapping[str, bool] | None = None, name: str = 'in_memory', project: str = 'in_memory', policy: LoadPolicy | None = None) -> 'Batch'

Build a batch from already-loaded CellpyCell objects.

The pieces a batch needs -- a journal pages frame (polars, keyed by filename) plus a populated cell store -- are constructed here, so a GUI/notebook holding cells in memory can feed them straight to :func:cellpy.collect.collect_summaries / collect_cycles or batch.plot() without writing a journal to disk.

Parameters:

  • cells (Mapping[str, Any] | Sequence[Any]) –

    {label: CellpyCell} or a sequence of cells (labels are taken from cell.cell_name, falling back to cell_001 ..., de-duplicated).

  • groups / sub_groups –

    optional {label: value} maps (defaults: group 1 for all; sub_group 1..N).

  • group_labels (Mapping[Any, str] | None, default: None ) –

    optional {group: display label} map.

  • selected (Mapping[str, bool] | None, default: None ) –

    optional {label: bool} (defaults True) for the journal selected column.

  • name / project –

    journal name/project.

from_db classmethod 

from_db(name: str, project: str, policy: LoadPolicy | None = None, **db_kwargs) -> 'Batch'

Build a batch by reading a database (Excel or JSON).

link() -> 'Batch'

No-op in batch v3 (the store loads lazily); kept for the surface.

load 

load(**overrides) -> BatchResult

Load cells (alias of :meth:update, kept for the legacy surface).

update 

update(on_progress=None, executor: str = 'serial', **overrides) -> BatchResult

Load every cell, caching them in the store.

executor is "serial" (default), "threads" or "processes". Known :class:LoadPolicy fields in overrides update the policy; unknown (legacy) kwargs like testing are forwarded to the loader (cellpy.get) via loader_kwargs.

BatchLoadError

Bases: CellpyError

Raised by :meth:BatchResult.raise_if_failed when cells failed.

BatchPaths dataclass 

BatchPaths(name: str, project: str, project_dir: Path)

Computed, immutable folder layout for one batch.

Nothing here creates directories -- constructing a BatchPaths and reading its properties is free of side effects. Call :func:ensure_dirs to materialise the folders.

batch_dir property 

batch_dir: Path

The dump directory for this batch (<project_dir>/dump).

raw_dir property 

raw_dir: Path

Where exported raw data lives (<batch_dir>/raw_data).

all_dirs 

all_dirs() -> tuple[Path, ...]

Every directory this layout owns, parents first.

create classmethod 

create(name: str, project: str, project_dir: Path | str | None = None) -> 'BatchPaths'

Build a layout; project_dir defaults to the current directory.

journal_file 

journal_file(suffix: str = '.json') -> Path

Path to the journal file for this batch (not created here).

BatchResult dataclass 

BatchResult(results: list[CellResult] = list())

The outcome of a batch run: one :class:CellResult per cell.

cells 

cells() -> dict[str, Any]

Mapping of label -> loaded cell, successful cells only.

raise_if_failed 

raise_if_failed() -> 'BatchResult'

Strict mode: raise if any cell failed; otherwise return self.

report 

report() -> pl.DataFrame

A tidy per-cell outcome frame (the dataframe errors only hinted at).

CellResult dataclass 

CellResult(label: str, outcome: CellOutcome, cell: Any | None = None, source: str | None = None, seconds: float = 0.0, error: BaseException | None = None)

The outcome of loading one cell.

CellSpec dataclass 

CellSpec(label: str, raw_files: list = list(), cellpy_file: Any | None = None, instrument: str | None = None, model: str | None = None, mass: float | None = None, nom_cap: float | None = None, area: float | None = None, cycle_mode: str | None = None, overrides: dict = dict())

Fully resolved per-cell loading instructions.

CellStore 

CellStore(loaders: Mapping[str, Callable[[], Any]] | None = None, cells: Mapping[str, Any] | None = None)

Bases: Mapping

A lazy Mapping[str, CellpyCell].

Construct with per-label zero-argument loaders (called on first access) or with already-loaded cells (:meth:from_cells). Loaded cells are cached.

first 

first() -> Any

Load and return the first cell.

from_cells classmethod 

from_cells(cells: Mapping[str, Any]) -> 'CellStore'

Build a store over already-loaded cells (e.g. from a BatchResult).

sample 

sample() -> Any

Alias for :meth:first (a representative cell).

unload 

unload(label: str) -> None

Drop a loaded cell from the cache (explicit memory management).

Journal dataclass 

Journal(name: str | None = None, project: str | None = None, pages: DataFrame = pl.DataFrame(), session: dict = _empty_session(), meta: dict = dict())

A batch journal: the cells of an experiment plus session/meta state.

Attributes:

  • name (str | None) –

    batch name.

  • project (str | None) –

    project name.

  • pages (DataFrame) –

    one row per cell; filename is a column (keys-in-columns).

  • session (dict) –

    mutable session state (starred/bad_cells/bad_cycles/notes).

  • meta (dict) –

    free-form metadata carried through save/load (name, project, time_stamp, project_dir, ...).

cell_names property 

cell_names: list[str]

The cell labels, in page order.

LoadPolicy dataclass 

LoadPolicy(source: SourcePreference = SourcePreference.AUTO, recalc: bool = False, max_cycle: int | None = None, accept_errors: bool = True, all_in_memory: bool = False, skip_bad_cells: bool = False, selector: dict | None = None, loader_kwargs: dict = dict(), overrides: dict = dict())

Batch-wide loading options (one object instead of a kwargs tunnel).

SourcePreference

Bases: str, Enum

Which source a cell is loaded from.

combine_summaries 

combine_summaries(cells: Mapping[str, Any], journal: Journal | None = None) -> pl.DataFrame

Concatenate per-cell summaries into one tidy long-format frame.

Each row keeps its cell's summary columns plus cell/group/ sub_group keys. Cells without a summary are skipped. Returns an empty frame when nothing has a summary.

combine_tests 

combine_tests(cells: Mapping[str, Any], journal: Journal | None = None) -> pl.DataFrame

Per-test metadata across the batch as one tidy long-format frame.

One row per (cell, test_id) carrying the native TestMeta fields plus cell/group/sub_group keys. Surfaces the per-test records that a merged (campaign) cell holds in Data.tests at the batch level. Returns an empty frame when no cell exposes test metadata.

ensure_dirs 

ensure_dirs(paths: BatchPaths) -> tuple[Path, ...]

Create every directory in paths (idempotent). The only mkdir.

Returns the directories that were ensured (parents first).

from_cells 

from_cells(cells, **kwargs) -> Batch

Build a :class:Batch from already-loaded cells (see :meth:Batch.from_cells) -- feed it to collect_summaries / collect_cycles or call batch.plot().

from_journal 

from_journal(journal_file: Path | str, policy: LoadPolicy | None = None, **_kwargs) -> Batch

Build a :class:Batch from a journal file (.json or .xlsx).

journal_from_custom_json 

journal_from_custom_json(path: Path | str, column_map: Mapping[str, str], name: str | None = None, project: str | None = None) -> Journal

Build a :class:Journal from an arbitrary JSON file.

journal_from_frame 

journal_from_frame(frame: DataFrame | DataFrame, name: str | None = None, project: str | None = None) -> Journal

Build a journal from a dataframe of pages (polars or pandas).

The cell label must be available as a filename column (or the pandas index, which is promoted to one).

load 

load(name: str | None = None, project: str | None = None, *, journal: Journal | None = None, journal_file: Path | str | None = None, frame: Any | None = None, db: str | bool | None = None, policy: LoadPolicy | None = None, **kwargs) -> Batch

Build a :class:Batch from a journal source.

Source precedence: explicit journal model, journal_file, frame, then a database read (db reader name, or when only name/project are given). Falls back to an empty named journal.

load_cell 

load_cell(spec: CellSpec, policy: LoadPolicy | None = None) -> CellResult

Load one cell from its resolved spec. Pure-ish: no prints, no mutation.

Returns a :class:CellResult carrying the cell or the exception; only re-raises when policy.accept_errors is False.

parse_argument 

parse_argument(argument: Any) -> dict

Parse a journal argument cell into a dict of coerced values.

Accepts a dict ({"recalc": "False"}), the compact string form ("recalc=False;data_points=(1, 10000)"), or null-ish -> {}.

read_custom_json 

read_custom_json(path: Path | str, column_map: Mapping[str, str]) -> pl.DataFrame

Read an arbitrary JSON file into journal pages via a column map.

column_map maps source JSON keys to cellpy journal keys, e.g. {"cell_id": "filename", "mass_mg": "mass", "instrument_name": "instrument"}. The JSON may be a dict of columns ({key: [values]}) or a list of records. At least one source key must map to filename.

read_journal 

read_journal(path: Path | str) -> Journal

Load a journal into the :class:Journal model.

.json is the native, round-trippable format. .xlsx is supported read-only (a lab convenience); writing Excel is intentionally not supported in batch v3 (see :func:write_journal).

resolve_specs 

resolve_specs(journal: Journal, policy: LoadPolicy | None = None, per_cell: Mapping[str, Mapping[str, Any]] | None = None) -> list[CellSpec]

Resolve one :class:CellSpec per cell in journal.

Precedence (later wins): journal columns < journal argument < policy.overrides < per_cell[label].

run 

run(journal: Journal, policy: LoadPolicy | None = None, per_cell: dict | None = None, on_progress: ProgressHook | None = None, executor: str = 'serial') -> BatchResult

Load every cell in journal, returning a :class:BatchResult.

executor chooses "serial" (default), "threads" or "processes" -- all reuse :func:load_cell. Progress is reported via the on_progress callback; the runner never imports tqdm or prints.

write_journal 

write_journal(journal: Journal, path: Path | str) -> Path

Write a :class:Journal to path in the compatible JSON format.

Only .json is written. Excel journals are read-only in batch v3 (metadata plan Step 4); export a report frame instead of a journal.

Facade

facade

Batch facade.

The thin, notebook-friendly Batch class that ties the pieces together: journal + policy/resolve_specs + runner/result/store + aggregate/qc/outputs. Keeps the beloved surface the characterization net pinned -- pages, cells, summaries, update, report, save, mark_as_bad, drop -- while everything underneath is the new package.

Plot wiring: plot() delegates to the plotting layer via a small legacy adapter; the full tidy-frame plot path lands with the collectors redesign (Epic B, batch plan section 4.7).

Batch 

Batch(journal: Journal, policy: LoadPolicy | None = None, _db: dict | None = None)

A batch of cells: a journal, a lazy cell store, and derived frames.

experiment property 

experiment: '_LegacyExperimentAdapter'

Backward-compat view for legacy consumers (helpers/collectors).

cellpy.utils.helpers and cellpy.utils.collectors still reach into b.experiment.{cell_names,data,journal.pages,summary_frames}. This adapter keeps them working against the new Batch until they are migrated (Epic B/C); it is not part of the blessed API.

tests property 

tests: DataFrame

Per-test metadata across the batch (tidy long-format).

One row per (cell, test_id) with the native TestMeta fields plus cell/group/sub_group keys -- the per-test records a merged (campaign) cell carries. Empty frame if no cell exposes test metadata.

create_journal 

create_journal(**kwargs) -> Journal

Populate the journal from the configured database (if any).

Mirrors the legacy init() -> create_journal() flow: init stores the db config, create_journal performs the read.

from_cells classmethod 

from_cells(cells: Mapping[str, Any] | Sequence[Any], *, groups: Mapping[str, Any] | None = None, sub_groups: Mapping[str, Any] | None = None, group_labels: Mapping[Any, str] | None = None, selected: Mapping[str, bool] | None = None, name: str = 'in_memory', project: str = 'in_memory', policy: LoadPolicy | None = None) -> 'Batch'

Build a batch from already-loaded CellpyCell objects.

The pieces a batch needs -- a journal pages frame (polars, keyed by filename) plus a populated cell store -- are constructed here, so a GUI/notebook holding cells in memory can feed them straight to :func:cellpy.collect.collect_summaries / collect_cycles or batch.plot() without writing a journal to disk.

Parameters:

  • cells (Mapping[str, Any] | Sequence[Any]) –

    {label: CellpyCell} or a sequence of cells (labels are taken from cell.cell_name, falling back to cell_001 ..., de-duplicated).

  • groups / sub_groups –

    optional {label: value} maps (defaults: group 1 for all; sub_group 1..N).

  • group_labels (Mapping[Any, str] | None, default: None ) –

    optional {group: display label} map.

  • selected (Mapping[str, bool] | None, default: None ) –

    optional {label: bool} (defaults True) for the journal selected column.

  • name / project –

    journal name/project.

from_db classmethod 

from_db(name: str, project: str, policy: LoadPolicy | None = None, **db_kwargs) -> 'Batch'

Build a batch by reading a database (Excel or JSON).

link() -> 'Batch'

No-op in batch v3 (the store loads lazily); kept for the surface.

load 

load(**overrides) -> BatchResult

Load cells (alias of :meth:update, kept for the legacy surface).

update 

update(on_progress=None, executor: str = 'serial', **overrides) -> BatchResult

Load every cell, caching them in the store.

executor is "serial" (default), "threads" or "processes". Known :class:LoadPolicy fields in overrides update the policy; unknown (legacy) kwargs like testing are forwarded to the loader (cellpy.get) via loader_kwargs.

from_cells 

from_cells(cells, **kwargs) -> Batch

Build a :class:Batch from already-loaded cells (see :meth:Batch.from_cells) -- feed it to collect_summaries / collect_cycles or call batch.plot().

from_journal 

from_journal(journal_file: Path | str, policy: LoadPolicy | None = None, **_kwargs) -> Batch

Build a :class:Batch from a journal file (.json or .xlsx).

load 

load(name: str | None = None, project: str | None = None, *, journal: Journal | None = None, journal_file: Path | str | None = None, frame: Any | None = None, db: str | bool | None = None, policy: LoadPolicy | None = None, **kwargs) -> Batch

Build a :class:Batch from a journal source.

Source precedence: explicit journal model, journal_file, frame, then a database read (db reader name, or when only name/project are given). Falls back to an empty named journal.

Journal

journal

Batch journal model + JSON IO.

A journal is a document, not an actor: reading one never touches the filesystem layout, and the data model is separated from serialisation. This is the successor of utils/batch_tools/batch_journals.LabJournal (a ~1100-line class mixing data model, three file formats, path fixing, selection state and folder generation). Folder layout lives in :mod:cellpy.batch.layout.

The on-disk JSON format is preserved for compatibility: a top-level object with info_df (pages, pandas to_json "columns" orient), metadata and session. Pages follow the keys-in-columns law (polars report section 1.3): the cell label lives in the filename column, never in an index.

Journal dataclass 

Journal(name: str | None = None, project: str | None = None, pages: DataFrame = pl.DataFrame(), session: dict = _empty_session(), meta: dict = dict())

A batch journal: the cells of an experiment plus session/meta state.

Attributes:

  • name (str | None) –

    batch name.

  • project (str | None) –

    project name.

  • pages (DataFrame) –

    one row per cell; filename is a column (keys-in-columns).

  • session (dict) –

    mutable session state (starred/bad_cells/bad_cycles/notes).

  • meta (dict) –

    free-form metadata carried through save/load (name, project, time_stamp, project_dir, ...).

cell_names property 

cell_names: list[str]

The cell labels, in page order.

journal_from_custom_json 

journal_from_custom_json(path: Path | str, column_map: Mapping[str, str], name: str | None = None, project: str | None = None) -> Journal

Build a :class:Journal from an arbitrary JSON file.

journal_from_frame 

journal_from_frame(frame: DataFrame | DataFrame, name: str | None = None, project: str | None = None) -> Journal

Build a journal from a dataframe of pages (polars or pandas).

The cell label must be available as a filename column (or the pandas index, which is promoted to one).

read_custom_json 

read_custom_json(path: Path | str, column_map: Mapping[str, str]) -> pl.DataFrame

Read an arbitrary JSON file into journal pages via a column map.

column_map maps source JSON keys to cellpy journal keys, e.g. {"cell_id": "filename", "mass_mg": "mass", "instrument_name": "instrument"}. The JSON may be a dict of columns ({key: [values]}) or a list of records. At least one source key must map to filename.

read_journal 

read_journal(path: Path | str) -> Journal

Load a journal into the :class:Journal model.

.json is the native, round-trippable format. .xlsx is supported read-only (a lab convenience); writing Excel is intentionally not supported in batch v3 (see :func:write_journal).

write_journal 

write_journal(journal: Journal, path: Path | str) -> Path

Write a :class:Journal to path in the compatible JSON format.

Only .json is written. Excel journals are read-only in batch v3 (metadata plan Step 4); export a report frame instead of a journal.

Aggregation

aggregate

Batch aggregation.

Turns a set of loaded cells into one tidy, long-format frame with cell / group / sub_group key columns -- replacing the legacy wide/multiindex join_summaries machinery. This is the frame the collectors redesign (Epic B) builds on, so it lands here as batch.aggregate.combine_summaries.

combine_summaries 

combine_summaries(cells: Mapping[str, Any], journal: Journal | None = None) -> pl.DataFrame

Concatenate per-cell summaries into one tidy long-format frame.

Each row keeps its cell's summary columns plus cell/group/ sub_group keys. Cells without a summary are skipped. Returns an empty frame when nothing has a summary.

combine_tests 

combine_tests(cells: Mapping[str, Any], journal: Journal | None = None) -> pl.DataFrame

Per-test metadata across the batch as one tidy long-format frame.

One row per (cell, test_id) carrying the native TestMeta fields plus cell/group/sub_group keys. Surfaces the per-test records that a merged (campaign) cell holds in Data.tests at the batch level. Returns an empty frame when no cell exposes test metadata.