Instead of partitioning over YYYY/mm/dd, take the SHA256 of the URL and
partition over the first, second and tail bytes of it. It has the upside
of being more rsync-friendly and there is no longer a need to keep a
different file, "everything", to look up already retrieved URLs. Also
replace the "url" file with a "meta" file, organized as key=value,
currently holding URL and retrieval date.
Lucas
147695edc4