_input_ids extracts ids from a BatchEncoding (5.x) or bare list/tensor (4.x); render_prompt_ids builds the generation prompt and extracts the tensor; eval.py + resident.py operators use it (no more BatchEncoding to generate); 12 tests pass (2 new); mypy strict clean. Box eval verification follows this release. No submodule changes.
The transformers 5.x return-type change behind #15 also breaks generation:
apply_chat_template(add_generation_prompt=True, return_tensors="pt") returns a
BatchEncoding, and eval.py + resident.py passed it to model.generate, which does
inputs.shape[0] -> AttributeError (the holdout eval crashed on scenario 1). #15
fixed only the trainer. Factor a shared _input_ids helper and a render_prompt_ids
function; both operators use it. Tests cover _input_ids for both shapes and
render_prompt_ids.
A shared _input_ids helper extracts the id sequence from a BatchEncoding (5.x) or bare list/tensor (4.x); _render_ids uses it; a new render_prompt_ids(tokenizer, messages, device) builds the generation prompt and extracts input_ids; eval.py and resident.py operators use render_prompt_ids instead of passing a BatchEncoding to generate; unit test covers _input_ids for both shapes; existing tests pass; mypy strict clean; holdout eval runs on the box without the AttributeError.
_render_ids extracts input_ids from a BatchEncoding (5.x) or passes a list through (4.x); regression test asserts the BatchEncoding path yields the same mask; 10 tests pass; mypy strict clean. End-to-end box verification of the correct mask against Mistral done before this release. No submodule changes.
apply_chat_template returns a BatchEncoding ({input_ids: [...]}) on transformers
>= 5 where 4.x returned a bare list[int]. build_masked_example treated the render
as a dict, so len/slicing were wrong and the prefix-differencing spuriously
raised "chat template is not additive" on every real model. Extract the id
sequence via a _render_ids helper; verified the assistant-only mask against
mistralai/Mistral-7B-Instruct-v0.2. The fake tokenizer returned a bare list and
missed this, so a BatchEncoding-returning variant now guards it.
build_masked_example normalises apply_chat_template's BatchEncoding (transformers 5.x) and list[int] (4.x) returns via a _render_ids helper; a BatchEncoding-returning fake tokenizer produces the same mask as the list-returning one (regression test added); existing tests pass; mypy strict clean; verified end-to-end on the box that sekft-train --inspect produces a correct assistant-only mask against Mistral.
First release: the trainer (assistant-only mask, train=serve render), the
three-source data loader (raw dir / curated jsonl / Hub), eval, resident, and
structured logging. Packaged, typed, [gpu] extra, depends on posix-sdc[hub].
Pipfile.lock pins posix-sdc ==1.2.2 from the private index in default + develop sections with huggingface_hub and the [hub] closure present; committed; tree clean. No submodule changes.
The lock committed with the triplet (#13) predated the published wheel. Now that
posix-sdc 1.2.2 is on the private index, re-lock: it pins ==1.2.2 from the index
in both the default and develop sections, and the [hub] extra pulls
huggingface_hub and its transitive deps into the lock, so the next machine
installs the published wheel with the Hub path available.
Pipfile.lock pins tiararodney.posix-sdc ==1.2.2 from the private index in both default and develop sections, with huggingface_hub and its [hub] transitive deps present; committed on develop; tree clean after.
Wire the posix-sdc dependency as a triplet:
- pyproject declares the abstract posix-sdc[hub], so the trainer's --hub path can
reach the Hub (huggingface_hub) wherever sekft is installed;
- Pipfile [packages] pulls the published wheel from the private index;
- Pipfile [dev-packages] pulls the git develop branch for develop-time work.
Commit Pipfile.lock so the dependency surface and its lock land together.
pyproject declares posix-sdc[hub]; Pipfile [packages] pulls posix-sdc from the private index and [dev-packages] from git develop; Pipfile.lock committed and consistent; tree clean after commit.
sft.load_turns(data, hub, revision) yields turns from a raw rollout dir (keep-filtered), a curated .jsonl file, or the published corpus via posix-sdc load_trajectories (lazy import; Hub fallback) - verified the hub path yields all 787 trajectories via the editable checkout's data/; sekft-train gains --hub/--revision and dispatches --data by dir-vs-.jsonl; train()+inspect() use it; 9 tests pass (3 new for raw-dir/jsonl/reject); mypy strict clean (5 files). No submodule changes.
iter_keepers read only raw per-trajectory .json -- one of three input shapes.
Add load_turns(data, hub, revision) yielding assistant-bearing turns from a raw
rollout dir (keep-filtered), a curated .jsonl corpus (one record per line), or
the published corpus via posix-sdc's load_trajectories (the in-repo data/ of a
checkout, else the Hugging Face Hub). sekft-train gains --hub and --revision and
dispatches --data by dir-vs-.jsonl; train() and inspect() use it.
Raw-rollout reading stays sekft-local; curated + Hub reuse posix-sdc's loader,
imported lazily so the raw/jsonl paths need neither posix-sdc nor huggingface_hub
installed. Unit tests cover the raw-dir and jsonl dispatch.
sft.load_turns(data, hub, revision) yields turns from a raw rollout dir (keep-filtered), a curated .jsonl file, or the published corpus via posix-sdc load_trajectories (Hub fallback), imported lazily; sekft-train gains --hub and --revision and dispatches --data by dir-vs-.jsonl; train() and inspect() use it; unit tests cover the raw-dir and jsonl paths; existing tests stay green; mypy strict clean.
bool() wraps the operate_rate predicate in eval.py and resident.py so the sum counts cleanly-operated trajectories without summing None; mypy strict clean across the whole package (5 files); 6 tests pass.
sum(t.steps > 0 and t.meta.get("clean") for t in rows) yields the right operand
of `and` when steps>0, so a trajectory whose meta lacks the "clean" key
contributes None and sum() raises TypeError. Wrap the predicate in bool() so it
counts trajectories that operated and are clean. Surfaced by mypy once posix-sdc
began shipping py.typed (Trajectory is now typed).
operate_rate in eval.py and resident.py wraps the steps>0 and meta.get('clean') predicate in bool() so the sum counts cleanly-operated trajectories without summing None; mypy strict clean across the whole package (5 files); tests pass.
sekft-train logs run config and each phase (tokenizer/model load, build, training, save) via a sekft.train logger to stderr; -v/--verbose and -q/--quiet added; dataset accounting reports keepers->usable with over-length/empty-mask drop counts and a warning; build loop logs per-trajectory (debug) and every 100 (info); transformers verbosity raised so the per-step curve shows; inspect() logs likewise; 6 tests pass; mypy strict clean on sft.py + _log.py (verified loop logging fires with FakeTok). No submodule changes.
The trainer was nearly silent: outside an example count and a save line it
printed nothing through tokenizer load, the base-model load, example building, or
the training loop, and trajectories dropped for length or empty mask vanished
without a trace.
Add _log.py (a shared stderr logging setup so stdout stays clean for results)
and a module logger. sekft-train gains -v/--verbose and -q/--quiet. Log the run
config and each phase; report dataset accounting (keepers -> usable, with counts
dropped for over-length and empty-mask and a warning when any are dropped);
inside the build loop, a per-trajectory debug line and a progress line every 100;
raise transformers' verbosity during training so the per-step curve shows. Prints
in train() and inspect() are routed through the logger.
sekft-train logs run config and each phase (tokenizer load, model load, training, save) via a module logger to stderr; -v/--verbose and -q/--quiet control level; dataset accounting reports keepers->usable with counts dropped for length and empty-mask and warns when any are dropped; transformers verbosity raised so the per-step curve shows during training; inspect() logs likewise; existing sft tests stay green; mypy strict clean.
mypy --strict src tests passes (no issues, 6 files); ML/posix-sdc imports ignored via overrides; sft/eval/resident/tests fully annotated; mypy is a declared dev dep; py.typed ships in the package.
