dpaste and the wayback machine

[Wyatt via Digitalmars-d]

On Sunday, 7 February 2016 at 21:59:00 UTC, Andrei Alexandrescu 
wrote:
> Dpaste currently does not expire pastes by default. I was 
> thinking it would be nice if it saved them in the Wayback 
> Machine such that they are archived redundantly.
>
> I'm not sure what's the way to do it - probably linking the 
> newly-generated paste URLs from a page that the Wayback Machine 
> already knows of.
>
> I just saved this by hand: http://dpaste.dzfl.pl/2012caf872ec 
> (when the WM does not see a link that is search for, it offers 
> the option to archive it) obtaining 
> https://web.archive.org/web/20160207215546/http://dpaste.dzfl.pl/2012caf872ec.
>
>
> Thoughts?
>
You want it in Wayback?  Sounds like you need some WARC [0].  
Since anyone can upload to IA (using a nice S3-like API, even 
[1]), this should be pretty uncomplicated.  If you can get a list 
of all the paste URLs, you can use wget [2] to build the WARC 
fairly trivially. [3]  Then I'd suggest getting a dlang account 
and make an item [4] out of it.  Just make sure it's set to 
mediatype:web and it should get ingested by Wayback.

After that?  Generate a WARC when a paste is made and use the 
dlang S3 keys to add it to the previous item (or maybe just do it 
daily or weekly so as to not stress the derive queue too much).  
I'm pretty sure that's all that's needed.

-Wyatt

[0] http://fileformats.archiveteam.org/wiki/WARC
[1] https://archive.org/help/abouts3.txt
[2] -i,  --input-file=FILE   download URLs found in local or 
external FILE.
[3] 
http://www.archiveteam.org/index.php?title=Wget#Creating_WARC_with_wget
[4] 
https://blog.archive.org/2011/03/31/how-archive-org-items-are-structured/