huge speed optimization on HDD by removing md5 (#845)

* Eliminate unnecessary use of MD5 checksum

md5checksum() computes the actual checksum of a specified file, which is appropriately expensive, but the code seemed to be using the checksum result as a key into the imgMetadata dictionary to avoid handling image files being renamed during processing steps. This seems like a very expensive way to handle the rename so instead, I now update the imgMetadata keys with the new filename in the one place that the rename happens, and avoid MD5 checksums entirely.

* merge conflicts

* Add missing handling for image path renames due to nested chapter folder name

* merge conflicts

* merge conflicts

* add perf_counters

* imgFileProcessing perf_counter

* use startswith and removeprefix

---------

Co-authored-by: utopiafallen <utopiafallen@gmail.com>
This commit is contained in:
Alex Xu
2025-03-04 11:28:23 -08:00
committed by GitHub
co-authored by utopiafallen
parent 5f8526da44
commit 01625904d1
3 changed files with 15 additions and 17 deletions
-10
View File
@@ -75,16 +75,6 @@ def walkLevel(some_dir, level=1):
del dirs[:]
def md5Checksum(fpath):
with open(fpath, 'rb') as fh:
m = md5()
while True:
data = fh.read(8192)
if not data:
break
m.update(data)
return m.hexdigest()
def sanitizeTrace(traceback):
return ''.join(format_tb(traceback))\