Batch hashing¶

Many files or buffers in one call, on a pool of worker threads.

Types¶

ph_batch_item_t¶

One entry in a ph_hash_files() batch: a file path in, hashes and status out.

Explained in Hashing many files

ph_batch_buffer_item_t¶

One entry in a ph_hash_buffers() batch: an in-memory image buffer in, hashes and status out.

Explained in Hashing many files

ph_batch_continue_fn¶

typedef int (*ph_batch_continue_fn)(void *user_data);

Called before a batch worker starts each item; return 0 to stop the batch.

Called from worker threads – possibly several at once – so it must be thread-safe; typically it reads an atomic flag or a deadline. After it returns 0 once, no worker starts another item, items already in progress finish normally, and every item not started gets PH_ERR_CANCELLED. It may still be called a few more times by workers that had not yet seen the stop.

ph_batch_progress_fn¶

typedef void (*ph_batch_progress_fn)(size_t done, size_t total, void *user_data);

Called after a batch worker finishes an item, successfully or not.

done counts finished items: each value from 1 up to the number of items finished is passed exactly once, but calls come from worker threads, possibly concurrently and not in order, so it must be thread-safe. total is the batch size. Not called for items skipped by a cancellation.

ph_batch_options_t¶

Options for ph_hash_files_ex() and ph_hash_buffers_ex().

Initialize with ph_batch_options_init(), then set the fields you need: that fills in struct_size and the defaults, and keeps working when later versions append fields. A caller that fills the struct by hand must set struct_size to sizeof of the struct it was compiled with.

Explained in Hashing many files

Constants¶

PH_BATCH_HASHES_CAPACITY¶

#define PH_BATCH_HASHES_CAPACITY 8

Number of hash slots in each batch item's hashes[] array.

Deliberately separate from PH_HASH_FLAGS_COUNT, which only says how many algorithms are defined today. The array size is part of the batch structs' layout, and so of the ABI; tying it to the flag count would mean a new uint64_t algorithm moves status and changes the struct size. The spare slots are that headroom: up to four new uint64_t algorithms can be added without an ABI break. On a per-item success the first popcount(flags) slots are written and the rest are left untouched; on a per-item failure every slot is zeroed.

Explained in Hashing many files

Functions¶

ph_hash_files()¶

ph_error_t ph_hash_files(ph_batch_item_t *items, size_t n, uint32_t flags, int threads);

nodiscard · returns ph_error_t

Hashes a batch of image files, optionally across a pool of internal threads.

Each item is loaded with ph_load_from_file() and hashed with ph_compute_multi() on a context of its worker's own. A failure on one item is recorded in that item's status and does not stop the rest of the batch.

This is ph_hash_files_ex() with every option at its default:

• Default configuration. Every item is hashed as by a freshly created context, default max_pixels limit included: nothing set with a ph_context_set_* function applies. Pass a configured context through ph_hash_files_ex() instead.
• Blocking and not cancellable. The call returns after the last item and reports no progress; ph_hash_files_ex() adds both.
• Peak memory grows with the thread count. Each worker holds one decoded image at a time, about 4 bytes per pixel, so threads = 0 on a machine with many CPUs can hold many large images at once. Bound it with an explicit thread count, a lower max_pixels through ph_hash_files_ex(), or both.

Return contract – the overall return value reports only failures that stopped the batch from being worked on at all; anything that happened to an individual image is in that item's status and never in the return value:

• PH_ERR_INVALID_ARGUMENT – malformed call, nothing was written. flags is zero or contains an unknown bit, threads is negative, or items is NULL with n > 0. Validated before n is looked at, so an empty batch does not excuse a bad argument: ph_hash_files(NULL, 0, 0, -5) returns PH_ERR_INVALID_ARGUMENT, not PH_SUCCESS.
• PH_SUCCESS with n == 0 – no-op. items may be NULL in this case.
• PH_ERR_ALLOCATION_FAILED – the batch could not be worked on: no internal context or worker could be created, so not one item was looked at and all of them are left at PH_ERR_ALLOCATION_FAILED. Partial degradation is not reported here: if even one worker starts it processes the whole batch by itself and the call returns PH_SUCCESS.
• PH_SUCCESS otherwise – the batch was processed to the end. This says nothing about whether any individual image succeeded; inspect every items[i].status.

See Hashing many files.

Parameters

Returns PH_SUCCESS once the batch has been processed (regardless of per-item outcomes), PH_ERR_INVALID_ARGUMENT for a malformed call, or PH_ERR_ALLOCATION_FAILED if no item could be worked on at all. See the return contract above.

Used in batch_hash.c · find_duplicates.c

Explained in Hashing many files · Configuring a context · How libphash works · Loading images · Performance

ph_hash_buffers()¶

ph_error_t ph_hash_buffers(
    ph_batch_buffer_item_t *items,
    size_t n,
    uint32_t flags,
    int threads);

nodiscard · returns ph_error_t

Same as ph_hash_files(), but for already-in-memory encoded image buffers (e.g. downloaded bytes) instead of file paths.

Identical semantics and identical return contract, including validation running before the n == 0 shortcut: ph_hash_buffers(NULL, 0, 0, -5) returns PH_ERR_INVALID_ARGUMENT. An entry whose buffer is NULL or whose length is 0 is a per-item PH_ERR_INVALID_ARGUMENT in status, not an overall failure. The notes of ph_hash_files() on threads and on its defaults – default configuration, no cancellation, memory per worker – apply unchanged, and ph_hash_buffers_ex() is the way around the defaults. The encoded buffers themselves are the caller's and are not copied.

Returns As ph_hash_files(): PH_SUCCESS, PH_ERR_INVALID_ARGUMENT or PH_ERR_ALLOCATION_FAILED.

Explained in Hashing many files · Configuring a context

ph_batch_options_init()¶

ph_error_t ph_batch_options_init(ph_batch_options_t *options);

returns ph_error_t

Fills options with the defaults and the right struct_size.

Returns PH_SUCCESS, or PH_ERR_INVALID_ARGUMENT for a NULL options.

Used in batch_hash.c

Explained in Hashing many files

ph_hash_files_ex()¶

ph_error_t ph_hash_files_ex(
    ph_batch_item_t *items,
    size_t n,
    uint32_t flags,
    const ph_batch_options_t *options);

nodiscard · returns ph_error_t

ph_hash_files() with options: a configuration template, cancellation and progress.

Same items, flags, per-item statuses and validation as ph_hash_files(). options may be NULL, which means every default – the call is then ph_hash_files() with threads = 0. With a configuration template the peak-memory note on ph_hash_files() applies with that template's max_pixels as the bound.

Returns

• PH_ERR_INVALID_ARGUMENT: as ph_hash_files(), or struct_size smaller than any version of the struct, or a negative threads; nothing was written.
• PH_ERR_ALLOCATION_FAILED: as ph_hash_files() – no item was worked on.
• PH_ERR_CANCELLED: should_continue stopped the batch before every item was started. The items that were started have their real status; the rest have PH_ERR_CANCELLED and zeroed hashes.
• PH_SUCCESS: every item was processed; inspect each status.

Used in batch_hash.c

Explained in Hashing many files · Configuring a context · Handling errors

ph_hash_buffers_ex()¶

ph_error_t ph_hash_buffers_ex(
    ph_batch_buffer_item_t *items,
    size_t n,
    uint32_t flags,
    const ph_batch_options_t *options);

nodiscard · returns ph_error_t

ph_hash_buffers() with options; see ph_hash_files_ex().

Returns As ph_hash_files_ex(): PH_SUCCESS, PH_ERR_INVALID_ARGUMENT, PH_ERR_ALLOCATION_FAILED or PH_ERR_CANCELLED.

Explained in Hashing many files · Configuring a context · Handling errors