Clean up use of shifts beyond composite types

GCC with the -fsanitize=undefined flag issues complaints like:

runtime error: left shift of 232 by 24 places cannot be represented in
type 'int'

due to the fact we shift the byte values into a 64 bit value. Rather
than adding lots of casts split out the formation into a set of steps
that doesn't end up shifting the byte value, just the final 64 bit
value.