I recently found myself on a last minute business trip to Vietnam where I didn't have enough time to go through the regular consulate/embassy process (4-day turnaround) and had to apply online for an unofficial (albeit faster) "visa on arrival".
TL;DR. It works! And it worked for all of the friends I asked too!
Here's what I found out in my research on the visa on arrival program:
It's not 100% officially approved by the international consulate system. Every official government website I found told me to go through the longer, more expensive process (choice #1 above).
It's brokered by travel agencies in Vietnam. Some online services take place on a travel agency website, others are part of or sister companies of a travel agency.
By default, your visa application will be grouped together with 12-30 other people; this means that the approval letter you receive (a scanned online copy) will include the names and passport numbers of the 12-30 other people on the same page.
Certain online services offer a private approval letter service for an extra fee. Sometimes you can get it for free by specifically asking customer service. If you go private, the transaction time might be delayed.
A few online services also offer a fast-track service where you don't have to line up one extra time at the airport on arrival. (I don't know anyone who's tested this service so I can't say much about it.)
Here's what I found out about the various online services. As some online services have their staff pose as users to write and reply to messages on internet forums, I sought out real-life friends to see which service they went with.
My Vietnam Visa is the one I went with; their testimonial section has real, working links to New York Times and Timeout Dubai articles that cite them. And I contacted customer support to get a private letter without any extra charge, though it took an extra day.
Vietnam-Visa.com was used by one of my good friends from the US in 2012.
And here's a rundown of my experience after I received my approval letter:
Before leaving: I printed the approval letter I got from My Vietnam Visa in color. (I read that black and white is fine too.)
Before leaving: I filled out two copies of the arrival form. (Most services have links to or will email you the arrival form to fill out.) It turns out only one is needed, and if you forget you can find the form and fill it out at the arrival airport.
Before leaving: prepare 45 USD and two 4x6cm passport photos.
Upon arrival: I darted off the plane and towards customs at Ho Chi Minh airport. Off to the side there's a special small office and (unordered) line for visas on arrival. Friends have warned that the wait here can be anything from 30 minutes to 3 hours. Luckily, mine only took 30 minutes, (though the waiting process and lack of commmunication made me feel like I was in a game of Papers, Please).
Upon arrival: after getting my passport back from the office with a visa attached to a page, I went to line up for customs. (This is the same line that you would have to go through if you went through the official consulate process and got the visa attached to your passport pre-departure.)
A final note of caution: the visa on arrival service in Vietnam is relatively new and the rules may change any time. So be sure to research any new developments before applying!