create-squashfs-snapshot: major speedups20240926T054112Z

The old code was slowly building up more and more files, and re-running
checksums of ALL files every pass.

As of 2024/09/25 the full runtime was approaching 134 seconds of which
113 seconds was just the SHA512 re-running over ALL files.

Instead of re-running over all files, produce and KEEP a dated signed
checksum for each day's binary artifacts - then at the end, combine them
with signature verification into the old style of file, so that existing
tools keep working.

Old runtime: 134 seconds
New runtime: 21 seconds

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>

1 files changed, 18 insertions, 17 deletions

diff --git a/create-squashfs-snapshot b/create-squashfs-snapshot
index 5e72e1e..2095ebf 100755
--- a/create-squashfs-snapshot
+++ b/create-squashfs-snapshot
@@ -163,21 +163,22 @@ for algo in "${algo_LIST[@]}" ; do
 done
 
 # create checksums for snapshot and deltas
-# OLD LOGIC, that scans entire 18GB
-date +ts-old-checksum-start=%s.%N
-ls -d -- *.sqfs *.sqdelta \
-| xargs sha512sum -- \
-| sort -k +2 \
-| gpg \
-	--batch \
-	--yes \
-	-u "${signkeyid}" \
-	--clearsign \
-	--comment "Current: gentoo-${today}" \
-	--output sha512sum.txt.tmp \
-	/dev/stdin
-mv sha512sum.txt.tmp sha512sum.txt
-date +ts-old-checksum-end=%s.%N
+# OLD LOGIC, that scans entire 18GB; with 18GB of data this added 2 minutes of
+# runtime onto a script that is otherwise under 30 seconds.
+#date +ts-old-checksum-start=%s.%N
+#ls -d -- *.sqfs *.sqdelta \
+#| xargs sha512sum -- \
+#| sort -k +2 \
+#| gpg \
+#	--batch \
+#	--yes \
+#	-u "${signkeyid}" \
+#	--clearsign \
+#	--comment "Current: gentoo-${today}" \
+#	--output sha512sum.txt.tmp \
+#	/dev/stdin
+#mv sha512sum.txt.tmp sha512sum.txt
+#date +ts-old-checksum-end=%s.%N
 
 # NEW LOGIC, that tries to re-use signed checksums
 # Helper func for signing.
@@ -251,7 +252,7 @@ find "$tempdir" -name 'gentoo-*combine-verified' \
 	--comment "Current: gentoo-${today}" \
 	--output "${tempdir}"/sha512sum.txt.tmp \
 	/dev/stdin
-mv "${tempdir}"/{sha512sum.txt.tmp,sha512sum.txt}
-diff -Nuar sha512sum.txt "${tempdir}"/sha512sum.txt || true
+# Move the final file into place
+mv "${tempdir}"/sha512sum.txt.tmp sha512sum.txt
 date +ts-new-checksum-done=%s.%N
 date +ts-last=%s.%N