Code Documentation

htr2hpc

API Client

exception htr2hpc.api_client.NotAllowed

exception htr2hpc.api_client.NotFound

class htr2hpc.api_client.OCRModel(pk: int, name: str, file: str, file_size: int, job: str, owner: str, training: bool, versions: list, documents: list[int], accuracy_percent: float, training_accuracy: float, rights: str, can_share: bool, parent: str | None = None)

accuracy_percent: float

model accuracy

can_share: bool

whether model is sharable

documents: list[int]

list of related documents by document id

file: str

url to model file

file_size: int

file size

job: str

training job (Segment or Recognize)

name: str

model name

owner: str

owner username

parent: str | None = None

parent model if this model is finetuned from another model; not always supplied

pk: int

model id in escriptorium

rights: str

model permissions

training: bool

boolean indicating whether or not this model is currently training

training_accuracy: float

training accuracy - local override to allow write access

versions: list

list of versions

class htr2hpc.api_client.ResultsList(api: eScriptoriumAPIClient, result_type: str, count: int, next: str, previous: str, results: list)

API list response.

class htr2hpc.api_client.Task(pk: int, document: int, document_part: int, workflow_state: int, label: str, messages: str, queued_at: datetime, started_at: datetime, done_at: datetime, method: str, user: int)

API response for a task result.

duration() → timedelta | None

how long the task took to complete

class htr2hpc.api_client.Workflow(convert: str | None = None, segment: str | None = None, transcribe: str | None = None)

htr2hpc.api_client.get_model_accuracy(model_file: Path)

Get kraken model accuracy from metadata in the model file

htr2hpc.api_client.to_namedtuple(name: str, data: Any)

convenience method to convert API response data into namedtuple objects for easier attribute access

Tasks

Views

Users

htr2hpc.users.init_user(user: AbstractUser, user_info: Any) → None

pucas EXTRA_USER_INIT hook: make new CAS accounts inactive by default.

Existing accounts are left unchanged to preserve any intentional changes (e.g. is_active set by an admin). Admin and staff accounts created via the createcasuser command are activated by that command after this hook runs.

Context Processors

htr2hpc.context_processors.CPU_COUNT = 2

number of logical CPUs in the system

htr2hpc.context_processors.HOSTNAME = 'build-33801908-project-1186947-htr2hpc'

hostname for the local VM

htr2hpc.context_processors.htr2hpc_version(request)

Custom context processor to expose the htr2hpc package version.

htr2hpc.context_processors.site_domain(request)

Custom context processor to expose the current site from the Django sites framework. Used to display environment-specific values (e.g. SSH key labels) without hardcoding hostnames in templates.

htr2hpc.context_processors.vm_status(request)

Custom context processor to return information about VM configuration and resources.

Management Commands

class htr2hpc.management.commands.cleanup_exports.Command(stdout=None, stderr=None, no_color=False, force_color=False)

add_arguments(parser)

Entry point for subclassed commands to add custom arguments.

delete_old_exports(export_dir: Path, retention_hours: int) → tuple[int, int]

Delete export files in export_dir older than retention_hours.

Returns a (count, total_bytes) tuple of files deleted (or that would be deleted when dry_run=True).

handle(*args, **kwargs)

The actual logic of the command. Subclasses must implement this method.

htr2hpc.management.commands.cleanup_exports.get_old_exports(users_dir: Path, cutoff: datetime) → Generator[tuple[Path, int], None, None]

Yield (path, size_in_bytes) for export files older than cutoff.

Training

exception htr2hpc.train.run.JobCancelled

Custom exception for when slurm job was cancelled

class htr2hpc.train.run.TrainingManager(base_url: str, api_token: str, work_dir: pathlib.Path, document_id: int, training_mode: str, model_name: str, num_workers: int, parts: intspan.core.intspan | None = None, model_id: int | None = None, task_report_id: int | None = None, update: htr2hpc.train.run.UpdateMode = <UpdateMode.NEVER: 0>, transcription_id: int | None = None, existing_data: bool = False, show_progress: bool = True, model_file: pathlib.Path = None, training_data_counts: dict | None = None, slurm_output: str = '', job_stats: str = '')

class htr2hpc.train.run.UpdateMode(value)

class htr2hpc.train.data.TrainingDataCounts(parts: int = 0, lines: int = 0, regions: int = 0)

htr2hpc.train.data.compile_data(segmentations, output_dir)

Compile a list of kraken segmentation objects into a binary file for recognition training.

htr2hpc.train.data.get_best_model(model_dir: Path, original_model: Path | None = None) → Path | None

Find the best model in the specified model_dir directory. By default, looks for a file named *_best.mlmodel. If no best model is found by filename, looks for best model based on accuracy score in kraken metadata. When original_model is specified, accuracy must be better than the original to be considered ‘best’.

htr2hpc.train.data.get_model_file(api, model_id, training_type, output_dir)

Download a model file from the eScriptorium and save it to the specified directory. Raises a ValueError if the model is not the specified training type. Returns a pathlib.Path to the downloaded file.

htr2hpc.train.data.get_prelim_model(input_model: Path)

Copies the input model to a file with suffix _prelim.mlmodel, then returns the path to that newly created file.

htr2hpc.train.data.get_segmentation_data(api, document_details, part_id, image_dir, transcription_id=None) → tuple[Segmentation, tuple]

Get a single document part from the eScriptorium API and generate a kraken segmentation object.

Returns a tuple of the segmentation object and the part details from the API, which includes image size needed for serialization. Includes transcription text when a transcription_id is specified.

htr2hpc.train.data.serialize_segmentation(segmentation: Segmentation, part)

Serialize a segmentation object as ALTO XML for use as training data. Requires kraken Segmentation and part details returned by eScriptorum API.

htr2hpc.train.data.split_segmentation(training_data_dir)

Takes as input directory containing ALTO XML files and creates a train.txt and validate.txt file which define the train/validation split. This allows consistency across the multiple train tasks.

htr2hpc.train.data.upload_best_model(api, model_dir: Path, model_type: str, model_id: int | None = None, original_model: Path | None = None) → Path | None

Upload the best model in the specified model directory to eScriptorium with the specified job type (Segment/Recognize). If a model id is specified, updates that model; otherwise creates a new model. Returns pathlib.Path object for best model if found and successfully uploaded; otherwise returns None.

htr2hpc.train.data.upload_models(api, model_dir: Path, model_type: str, show_progress=True) → int

Upload all model files in the specified model directory to eScriptorum with the specified job type (Segment/Recognize). Returns a count of the number of models created.

htr2hpc.train.slurm.recognition_train(input_data_dir: Path, output_model: Path, input_model: Path | None = None, num_workers: int = 8, mem_per_cpu: str = '2G', training_time: timedelta = datetime.timedelta(seconds=900), epochs: int | None = None) → int

Run ketos recognition training as a slurm job. Returns the slurm job id for the queued job.

htr2hpc.train.slurm.segtrain(input_data_dir: Path, output_model: Path, input_model: Path, num_workers: int = 8, mem_per_cpu: str = '4G', training_time: timedelta = datetime.timedelta(seconds=900), epochs: int | None = None) → int

Run ketos segmentation training as a slurm job. Returns the slurm job id for the queued job.

htr2hpc.train.slurm.slurm_job_queue_status(job_id: int) → str

Use squeue to get the full-word status (i.e., PENDING or RUNNING) for a queued slurm job.

htr2hpc.train.slurm.slurm_job_stats(job_id: int) → str

Use jobstats to get Slurm Job Statistics, to track resource usage

htr2hpc.train.slurm.slurm_job_status(job_id: int) → set

Use sacct to get the status of a slurm job that is no longer queued. Returns a set of unique full-word statuses, reporting across all tasks for the job.

htr2hpc.train.calculate.calc_cpu_mem(job_stats)

Given a preliminary job_stats output, return recommended mem per cpu.

htr2hpc.train.calculate.calc_full_duration(slurm_output, job_stats)

Given a preliminary slurm job output, return duration estimate: Setup time, plus N times the average epoch plus 10% for wiggle room. Assumes the train task will take 50 epochs. N = 50 - count of completed epochs from prelim train task.

htr2hpc.train.calculate.estimate_cpu_mem(training_data_size, training_mode)

Use files in input data dir to come up with estimate of prelim mem per cpu.

htr2hpc.train.calculate.estimate_duration(training_data_size, training_mode)

Use files in input data dir to come up with estimate of prelim train duration.

htr2hpc.train.calculate.slurm_count_epoch(slurm_output)

Return count of epochs

htr2hpc.train.calculate.slurm_get_avg_epoch(slurm_output)

Return average epoch duration, in seconds

htr2hpc.train.calculate.slurm_get_max_acc(slurm_output, training_mode)

Return a tuple of (epoch #, accuracy) for the epoch with highest accuracy

htr2hpc.train.calculate.stats_get_max_cpu(job_stats)

Return max CPU usage

Utility script to delete models from eScriptorim API, for easily cleaning up models created and uploaded for testing.

Requires an eScriptorium API token set as an environment variable in ESCRIPTORIUM_API_TOKEN.

Takes a base url for the eScriptorium instance and a model name prefix; will delete all models that start with the specified model name prefix.

usage:

python src/htr2hpc/train/rm_models.py https://test-htr.lib.princeton.edu/ model_prefix