To say that I’m paranoid about backups is like saying…well, it’s hard to find a point of comparison that’s fulsome enough. I keep everything. I have archives of old MacOS installers on a RAID drive at home. Yes, I’m aware you can get them from Apple, the question being how long that’ll last. I have TimeMachine Backups going back to ~2009.

Why? As another post I’m working on will suggest, it’s about privacy and about control. Privacy in the sense that I don’t feel like it’s possible to trust the platforms that the present age seems to mandate we rely on. And, for the same reasons the good folks at ArchiveTeam have outlined:

1. Because they don’t give a fuck about your data. Except insofar as they can monetize it.
2. Because they will delete your wedding photos. This is a reference to Anil Dash’s The Web We Lost which everyone should watch. Do it. Right now. Then read the rest of this, please.

My interest in preserving my stuff—no matter how worthless it may seem—also dovetails nicely with the web archiving projects I have going at work, e.g., the blog of our long time provost, Peter Stearns.

Web archiving as I practice it, the process of making a snapshot of a website in a format that preserves its accessibility and usability. The folks at ArchiveTeam have done a lot on this front, e.g., recent projects to grab as much of Yahoo! Groups as possible before it was too late. Verizon blocked them, just as they did with Tumblr. Just thought I’d post a little note about this for convenience sake.

I use wget to do this because it’s super simple and baked right in to standard Linux/MacOS. It does have its limitations. These wget commands will produce a warc file for which you’ll need a warc player. You can grab something like Webrecorder or Webrecorder Player to browse the captured site. You can also browse the site files directly: wget will capture the individual files in addition to making the warc.

It’ll take however long it takes, obviously it just depends on how big the site is. Something like this site mostly text and built with a static site generator like Hugo takes less than 30 seconds. Database driven sites like those which use Wordpress can be much larger. There’s also the speed of your internet connection and computer to factor in to the mix. One recent example—the Washington Metro’s site took about a week and came out to ~36GB. And that was running 24/7 on a 1 Tbps connection on a pretty snappy machine at work.

Using wget for web archiving

Generally, you can use:
wget -pkrm --warc-cdx --warc-file=foo -e robots=off https://foo.org

-p collects the prerequisites for the page/site

-k converts the links to make them work locally

-r captures the site recursively
this is especially useful when you have a site with a funky configuration that throws wget off. Not thinking of any Netscape founders/famous devs, JAMIE. Should that happen while using foo.org might not work, but using foo.org/bar/ might.

-m mirrors the site/structure etc.

-e executes an option as if it was in your .wgetrc

--warc-cdx creates an index file for the WARC file

You can also add this stuff to your .wgetrc in your home folder.

progress=bar
robots=off
random_wait=on
mirror=on
recursive=on
verbose=on
user_agent=Mozilla

If you’re using Windows for this, I’ll refrain from giving you the JWZ treatment but I’ve got no fucking clue how to use wget on Windows. Sorry. Maybe switch to something that’s not as fucking creepy as Windows has become.¹

CW: brief allusions to violent crimes near the bottom of the post

“What does it matter if I have your location pings?! We delete them after 24 hours!”

privacy. fucking. matters.

We live in an age where privacy should be more important than ever. An age when we have the technologies to track and surveil with alarming precision. And that all exists in a regulatory framework that is—to put it as mildly as possible—totally fucking incapable of protecting the privacy of the average citizen. We should all be furious. Instead, we’re on Facebook.

As I noted in my first post, the Overton Window has shifted dramatically since 2006 with the emergence of social media sites as a daily feature of our lives. The defensive posture that most people held on this topic—or claimed to hold, at least—melted away.

There is no more dangerous development and no area in need of a strict legal framework that protects end user privacy, especially in the US.

More than perhaps any other factor, the rise of the smartphone may be the tool that made this all possible.

the spy in your pocket

I opened my first checking account at the US Employees Oklahoma City Federal Credit Union a few days after my 18th birthday. One of the first purchases I made after receiving my debit card a few weeks later was a new wallet. The guy who sold it to me at Penn Square Mall was someone who was “Myspace famous.” For some reason, I remember his posting about going to the Apple Store in the mall to check his Myspace. But smartphones changed that. We could be connected to these networks all the time and fill the gaps with them. The biggest thing that Myspace lacked wasn’t something that Myspace had much control over. Myspace lacked a way to become a constant part of your life. Facebook (which opened itself to mass registration in 2006) was rising just as the iPhone was announced in 2007. Facebook was able to become an early and significant part of the mobile landscape. It was always right there, whenever you wanted to “connect” with your friends.

At the same time, these devices have become effectively essential to most peoples’ lives. During the week of 13 January, I’m planning to make an effort to use my phone less than 30 minutes each day. But I’m not like most people. Most people don’t think about it or maybe they just don’t care, even if they do think about it. Maybe it just never occurs to them.

going off to/at #transpo20

The thing that finally got me to write this post was a conversation I had at Transportation Camp.

Some of the richest and most dangerous data your phone can provide is your location. It’s also a permission that’s frequently requested by apps that shouldn’t really need it. My credit union’s mobile banking app does not need to access to my location. If I really need to find the nearest branch or ATM, I’m perfectly capable of putting in a city name or postal code, thanks! The same for almost every other app that asks for it. None of them really need unrestricted access to this information.

As it happens, it’s also a rich source of data for transit software. And while I imagine most of the people in this field who want access to this information have the best intentions to only use it for arrival time calculations or similar, their attitudes discussing it betray a fundamental lack of concern for user privacy: it’s not the fact that you claim to delete the data after 24hrs, no exceptions. It’s the fact that the data exists. It’s the fact that it is created. Because you’re not the only one trying to get it.

How do I know that if I give you this data, it stays with you? Or that there’s not another app also leaking my location to somewhere else? How do I know that you’re actually using the data they way your privacy policy says you do? I don’t. I can’t know for sure. And I have no good reason to trust you. Or, in a rather disgustingly capitalistic view of things, support your efforts to make money by giving you my data for free.

During a tense session hosted by Transit (it’s an app) and Lyft, ominously titled “We’re Watching You, and That’s a Good Thing” (you really can’t make this shite up, folks) this all kind of came to the fore.

One of the Transit folks started asking some questions about how many people wouldn’t trust them with their data. 80% of people put as hand up. Apple? About the same. Google? 90% plus me putting both hands up.

This obviously took them by surprise.

One person whom I only know as “Kevin,” was incredulous that people wouldn’t share their data to improve transit ETAs:

What does it matter if I have your location pings?! They’re deleted them after 24 hours!

Because, Kevin, the issue isn’t that you may delete it. It’s that I can’t trust you or any other app on my phone to truly protect my privacy. It’s that you’re asking for this data. And for me to give it to you, it must first come into existence. I’d really just prefer it never came into existence in the first place. I want to be able to exist in the world without the feeling that I’m being tracked. Not least by a $500 metal and glass brick in my back pocket. The obvious solution “smash your phone and throw it into the Potomac at high tide” is one that few people would dare consider and fewer still would carry it out.

During the Lyft/Transit session, they passed around the microphone to folks who wanted to explain why they wouldn’t trust Transit and Lyft with their data. I was one of the last folks to speak. I’m sorry to say, I don’t remember what precisely I said. I just remember that Kevin didn’t like it.

Not just smartphone pings but SmarTrip taps

DC’s regional transit agency (WMATA) also collects significant amounts of granular data from every customer who uses a registered card to pay for their journeys. As WMATA Metrorail charges fares based on the length of a journey and when it is taken, an enormous amount can be discerned from that data alone.

As a case in point: a customer taps into Vienna-Fairfax every weekday at 8:30am and taps out at Union Station some time later. Then, they tap in at Union Station at 5:45pm and tap out at Vienna-Fairfax. Given that data alone, you can already begin to form a judgement about who they might be, what profile they fit, etc. If you toss in a bus trip on either or both of those rail trips, you can at least get an idea of where they live based on the AM bus trip from the stop close to their house to Vienna-Fairfax. See? Don’t even need the address. But drawing on demographic data from near the bus stop, you can figure out even more about them. Are they a cashier at Sweetgreen? Or are they the mid-30s Biglaw associate who orders falafel every Wednesday? Now, buy some data from some shady data broker and toss it all in a Blendtec. Boom. Figured you out Josh McMahan, Associate at White Shoe Legal, LLP. Enjoy falafel day tomorrow, sign up for our emails and we’ll send you a coupon for 50¢ off your order!

You probably think this song is about you, Kevin (it is, lol)

At the end of the day, the folks from Transportation Camp all migrated around the corner to a bar for drinks/dinner. I had resigned myself to the fact that my sparring with Kevin had concluded, but fate had another thing yet in store.

After dinner and a few beers with some old and new friends, I spotted Kevin making the same case to a delightfully skeptical looking urban planner from a major planning firm. At last, we could argue less than 20 feet apart.

Standing there, listening to him run through the same hackneyed “We all want better transit arrival data, so lemme have yours!” that he’d made several times during several sessions. It dawned on me: “this dude has no empathy.”

So I tried a different tact. I tried to explain that someone who is the victim of abuse or sexual assault may have issues with just anyone taking their data. Then I moved on to the fact that at least three trans women of color were murdered in the DC area last year. Given that, maybe members of the Trans or the broader LGBTQ community might not want to just give up their location or any other data. Because, Kev, people are different from you. The calculus around public safety is different. Who these people can trust is different. They are not a self described “normal, 50-something, straight white dude.”

mumble “Thank you for educating me” mumble, he said, as he turned away to the person on his other side.

Big Data, Little Ethics?

I used to know someone who worked at a software company, and through them, a fair number of data scientists. I have the privilege of working with a great data scientist on a number of projects at work.

The thing that both my friend at the software company (but not their coworkers) and my colleague have mentioned is something that always bounces into my consciousness when issues like this come up:

Most data science people don’t remember that the data represents people. It’s just numbers. It’s just something to manipulate with their slick R or Stata skillz.

And this is the problem in a nutshell.

You might be trustworthy, Kevin, but not everyone is. And I resent that you think I should trust you by default. I resent that about any of these companies.

The default has become to share far more than anyone ever needs to know. But Kevin, be happy! We’ll know with 0.2% more certainty that the bus will arrive only 28 seconds behind schedule.

What the fuck have we already given up? And where are we going? Nowhere good. This bus goes straight to a hell of our own devices.

Life in the Devonian Era

I am a data hoarder. I have personal emails going back to 2004. Some papers from undergrad, lots of stuff from my time in Austria, and more stuff besides. This is all to say nothing of my grad school work, thesis, etc. etc. etc. There’s a reason my 8TB NAS only has 1.5TB still free. I also have a lot of current stuff to manage and in a growing number of text and non-text formats.

Until recently, I relied pretty heavily on a 200gb Google Drive for storage of “current” materials. For notes I used Joplin or Markor as appropriate. But my increasing determination to liberate myself from the FAANGS meant that it was time shake things up.¹

So I moved my cloud storage to Tresorit and my email to Fastmail (referral link). But this still left something to be desired.

Enter DevonThink. DevonThink is a personal database system akin to Yojimbo or Symphytum, etc.. Except on steroids and a liter of RedBull.

DevonThink has some powerful tools baked in, e.g., ABBYY FineReader for OCR. It does a really good job indexing and identifying connections between documents and emails, between tags and keywords. It can capture local copies of web pages, RSS feeds, etc. Neat, right?

But there’s something missing: web history.

Send in the [History] Hounds

Web history is a funny thing: we don’t want anyone else to have it (hi Google) but it can be extraordinarily useful for keeping track of things you forgot to bookmark or send to Pinboard or whatever.

History Hound creates a searchable index of your web history across multiple browsers. If you sync this stuff e.g., via Firefox Sync, it gets that stuff too.² It also integrates with NetNewsWire to index your RSS feeds! This is a brilliant idea.

But if you’ve got DevonThink on one hand and HistoryHound on the other, how can you make them talk to one another? Well, a quick email to History Hound’s developer reveals there is a way to get it to dump out your web history index. Hooray! But wait, the next issue is that History Hound’s output is in UTF-16LE and the resulting file doesn’t render well. My first export was after a month of using it and the file was ~50mb of text. Three or so months in and the export was 120mb. Yikes!

So I noticed two things that should’ve been obvious at first, 1. HH exports to HTML and 2. the file encoding is wrong or at least it’s not UTF-8. But there’s a solution.

Bless This Mess

Here’s how to export then cleanup your History Hound index for archiving:

1. Window>Index Status (or just ⌘3)
2. Option+click Clear Index
3. Save the file as foo.html
4. Running file foo.html should produce something like HTML document text, Little-endian UTF-16 Unicode text, with very long lines
5. Then iconv -f UTF-16LE -t UTF-8 foo.html >> foo_utf8.html

Et voila! Dump into DevonThink. The resulting file should be ~50% smaller and opening it in DevonThink (or a plain old browser) will give you a delightful and searchable plain-text index you can file away in DevonThink.

It’s a nice way to make two great pieces of software work together even better.³ And another way to add to my growing Memex.

2020-01-22 Update: UTF-8!

As noted by @StClairSoft after this was posted a few weeks ago, the new version now uses UTF-8 so you can save a bunch of steps above. Give their stuff a look. It’s good stuff. And indie developers like them need/deserve all the support they can get.