mirror of
https://github.com/ciromattia/kcc
synced 2026-09-01 08:27:06 +00:00
huge speed optimization on HDD by removing md5 (#845)
* Eliminate unnecessary use of MD5 checksum md5checksum() computes the actual checksum of a specified file, which is appropriately expensive, but the code seemed to be using the checksum result as a key into the imgMetadata dictionary to avoid handling image files being renamed during processing steps. This seems like a very expensive way to handle the rename so instead, I now update the imgMetadata keys with the new filename in the one place that the rename happens, and avoid MD5 checksums entirely. * merge conflicts * Add missing handling for image path renames due to nested chapter folder name * merge conflicts * merge conflicts * add perf_counters * imgFileProcessing perf_counter * use startswith and removeprefix --------- Co-authored-by: utopiafallen <utopiafallen@gmail.com>
This commit is contained in:
@@ -75,16 +75,6 @@ def walkLevel(some_dir, level=1):
|
||||
del dirs[:]
|
||||
|
||||
|
||||
def md5Checksum(fpath):
|
||||
with open(fpath, 'rb') as fh:
|
||||
m = md5()
|
||||
while True:
|
||||
data = fh.read(8192)
|
||||
if not data:
|
||||
break
|
||||
m.update(data)
|
||||
return m.hexdigest()
|
||||
|
||||
|
||||
def sanitizeTrace(traceback):
|
||||
return ''.join(format_tb(traceback))\
|
||||
|
||||
Reference in New Issue
Block a user