For A Decent CSV Spreadsheet App

All I want is a decent spreadsheet app that does not insist on mangling my CSV files, which often have ID numbers in them which I might want to view as text and not numbers. Apple's Numbers is maddening (you have to export to CSV, extra steps, and it has a relatively low row limit, 65,535 I believe) and Microsoft's Excel is a little better but I'll use it as an example here of What You See Is Not What You Get.

I am doing some work on cities and (county-level) FIPS codes (so, in the US, FIPS codes are Federal level identifiers useful for a lot of things, they identify counties). Some cities are large and lie in more than one county. Some of the data I have deals with cities, and the income data is on the county level, so I need to map from cities to county FIPS.

Excel did not make this easy.

The file I grabbed off the net to help me map cities to FIPS (counties) quite correctly listed all the appropriate FIPS codes for each city. I needed to narrow this down to one (Wikipedia helped a lot, the geopolitical Wikipedians are nitpickers).

FIPS codes for counties have two parts, two leading digits for the state and then three digits for the county. So all FIPS codes that start with 36, for instance, are counties in New York state.

The format from my source file looked like this:

Raleigh, NC:    37063,183
Birmingham, AL: 01073,117
New York, NY:   36005,047,061,081,085

(I am pretty sure those 5 numbers for NYC are the 5 boroughs, I know Brooklyn is its own county, Kings county.)

Excel, however, would show the following in the main view, interpreting these IDs as numbers--errors are in the parentheses, A, B, and C:

Raleigh, NC:    37,063,183 (A)
Birmingham, AL: 1,073,117 (A,B)
New York, NY:   36,005,047,061,081,000 (A,C)

Errors:

1. Added a comma that isn't there.
2. Dropped leading zero.
3. Rounded rightside digits.

So there are at least three issues there, but the most difficult one is that it put a comma in after the two digits for the state, initially making me think that indeed the source file had a comma after the state component of the FIPS code. It did not. Parsing the file did not work.

That was all extremely infuriating, and reminded me of Microsoft's Clippy, where the coders thought they always knew better than you. Granted, a lot of apps and even programming language packages try to be smart and guess formats, and yes this can be useful. But if there are leading zeros and commas in odd places (or not) and it's a CSV (text) file, there could be a default "read CSV as text". Of course it seems that neither of these two programs have been coded to play nice with CSV files.

As such, they are not overly useful data science tools.

Microsoft's Epic Twitterbot Fail

If you read this blog, you've read about the rather hilarious failure of Microsoft's experiment with a learning Twitter bot. Trolls gave it so much input it started turning out hateful, sexist, racist tweets.

So we really have to wonder...

1. Why are Microsoft engineers so ignorant of Internet culture?
2. Why Microsoft engineers who program text-based bots have no idea about the range of text available?

Because these are epic failures. Epic. No wonder there are jokes about engineers being completely socially inept.

Blue Background, White Text

Microsoft Word used to have a fantastic option, making the background blue (instead of white) and the text white (instead of black). I and many people liked the change of contrast. I first remember falling in love with this feature in the much-loved Word 5.1 a long, long time ago (circa 1992).

I use it on my home machine with Word 2011. But with my new laptop, Word 2011 was not an option, I had to use Word 2016, which I rather like so far (despite initially causing massive problems for my citation management software). And, the option for blue background, white text is gone. And that's disappointing and problematic.

Having most of the screen be white (the background) makes the screen very bright. It's like staring into a light, albeit a dim one. You want to keep the contrast between the text and the background, but you don't need black on white to do that. A lot of interfaces do that and I think it is stupid. Even this Blogger editor is doing that (but notice what I've chosen for my blog layout). This is a blog, it isn't ink on paper, it's way beyond that.

Which is another part of the issue: the paradigm. This is a computer, it's not ink on paper, which is a whole other technology. Yes, writing papers on the computer stems from typing in black ink on white paper on a typewriter, but this isn't a typewriter. You can change the writing in your document to two columns, add images, add footnotes, move anything anywhere, add page numbers, make sections, change something to italics after you write it, have hyperlinks.... You know. Computer word processing is based on typing on a typewriter, but it is light years beyond even an IBM Selectrix II with correctable ribbon. The computer can spellcheck. You can edit on the page and it will shuffle the text around. You can justify the text after you type it and change all the margins, then undo and redo all of that. You can repaginate on the fly (they actually just do this these days). I could list probably hundreds of ways in which a word processor is different from the black ink on white paper typewriter experience. You can change the typeface and font size after you have typed the words--try that on a typewriter. Yet, the product managers for Word at Microsoft have decided that this is the right way, and the only way, to do it. It's an outdated paradigm, and it sucks for my eyes.

Feature creep is one thing. Removing a useful feature that's been around for over 20 years is another.

And I loved the file icon:

Excel and Date Formats

I hereby hate the people who programmed Excel and how it deals with date formats.

This error warning makes absolutely no sense. Sure, there are a ton of date formats. But the source file, which I made myself, is just a CSV file. The destination file is a brand new empty Excel file. Four years? You are kidding me. Approximately? What? This is not acceptable.

Edit: Aha! So this is what is going on. Not at all acceptable, since it means that Excel is interpreting the text as dates when I just want it to passively see everything as text. Microsoft has made Excel overdo it here, and it is not helping.

Word (Mac) and Removing Field Codes

So, if you're using Mendeley and you have all of your citations (references) in a Word document on your Mac, and you are ready to submit your Word document, you need to change all the field codes from field codes into regular old text showing the result of the field codes.

Word help does not help (bastards). The Internets assumes you are not using an awesome Unix-based OS, and tells you to select all the text and then use control-shift-F9. This does not work on a Mac, even though there is a control key. It's close, though:

Command-A (select all the text in the doc).
Command-Shift-F9 (changes all the field codes to text of just the results).

Better on a Mac:

Command-A (select all the text in the doc).
Command-6 (changes all the field codes to text of just the results).

C/o Adept Scientific from three years ago, and it still works today with Word Mac 2011.

Microsoft's Approach?

From a current NYTimes article by Nick Wingfield about Microsoft:

The [Microsoft] executives were stunned by how deeply Apple was willing to reach into the global supply chain to secure innovative materials for the iPad...

I read that as, "Microsoft executives don't give a damn about quality or innovation." However, neither quality nor innovation have ever been Microsoft hallmarks. Zune?

And, later in the article:

In a nod to Apple’s work with aluminum, Microsoft began to closely study materials that could be used to create a distinctive case for a tablet. Members of the Windows team gravitated toward magnesium...

Except of course, Steve Jobs knew about magnesium. I owned a Jobsian computer made of magnesium for some time: a NeXT cube.

Disappointing Analysis from Pogue

Usually, I like David Pogue's posts on technology, as he has a bit of humor and humanity that some other reviewers lack. But today Pogue lacks decent analysis in a review of the Lumia 900 phone. His presentation of pricing and screen resolution do not meet the basic standards of reporting. Recall I used to work at Ziff-Davis many years ago, where we did computer reviews.

Pogue:

It’s the Lumia 900. It’s beautiful, fast and powerful, and it’s only $100 (with a two-year AT&T contract). That’s half the price of an iPhone or a comparable Android phone — but you’re still getting a top-of-the-line machine.

The problem is that it is not at all half the price, thus his mention of the two-year contract, but that's a weak analysis and not at all true. Granted, different calling plans cost different amounts, but let's look at some hypothesized numbers. Let's call the cost of the phone the "up front" cost, since that's more accurate.

Phone	"Monthly"	Up-Front	Annual	Total
Lumia 900	$50	$100	$600	$700
iPhone 4S	$50	$200	$600	$800
Compared		50%		87.5%

87.5%, for this theorized monthly, is not half.
But many calling plans cost more. What if the theorized monthly were $100?

Phone	"Monthly"	Up-Front	Annual	Total
Lumia 900	$100	$100	$1,200	$1,300
iPhone 4S	$100	$200	$1,200	$1,400
Compared		50%		92.8%

92.8% is not half.
(And, you can get the iPhone 4 currently for $100, and the 3GS is free -- of course, like Pogue mentions but doesn't deal with, that's with a contract. Given that you can't divide by zero, the 3GS is unspeakably amazing.)

But that's not the only problem with Pogue's analysis.

Then again, the Lumia actually shows you a larger area, but in less detail. Its resolution is 800 by 480. The iPhone’s 3.5-incher has 960 by 640 pixels, so Apple’s screen is far sharper.

The Lumia 900 has a 4.3 inch (diagonal) screen. The problem is that Pogue gives us two sets of metrics: resolution and size; but these are not directly comparable. Resolutions are comparable, but only at the same size. We don't have the same size, so Pogue should tell us the dpi -- dots per inch (in this case, pixels). You can't exactly calculate the dpi from these numbers, because the diagonal measure does not give you the area in terms of square inches, so you can't calculate the pixels per (square) inch.

So, yes, the iPhone 4S has more pixels, but not 960x640 versus 800x480 more, because the screen sizes (not just resolutions) are different. Pogue addresses this, but not at all adequately.

"I Told You So"

About five years ago I told a major market research firm the following:

Currently the Zune is too problematic to be part of the digital near-future.

I said that Microsoft had to "fix it." I don't think the market research firm liked that I said that, since they blew me off after that. Something happened earlier this month that was so barely noted I missed it until earlier this week: Microsoft cancelled the Zune. It is actually difficult for me to find a news outlet that I am used to using which reported it (but there is always Wikipedia).

I could say that it feels good to be right, but I've been right the entire last five years and I've always known it. As for the market research firm in question, well, they don't know what they are doing.

If You Follow, You Will Never Lead: Microsoft's Business Non-Strategy

The title of this post comes from my years growing up as a sailor in Nantucket, Massachusetts, on the wall of the old sail room in the Nantucket Yacht Club. The idea was that you'll get the same wind as the boat ahead of you, and, unless you are a greatly better sailor or have a vastly better boat, you will never overtake the leading boat. If you were that great, you probably wouldn't be behind in the first place.

This is not completely true in business, despite the myth of "first mover advantage", which is only true for successful products (so, not true in the long-term for Palm, nor for Apple's Newton, nor Atari in video games, to name a few). If you can copy products but improve on them without getting sued for patent violation, that may be viable, albeit difficult, and, honestly, foolish.

But it is Microsoft's strategy. I'll have to point out ahead of time that although they are ahead in market share in terms of operating systems if you group all the different versions of Windows, they are not ahead in some equally valid areas of measurement, such as smart phones (iPhone versus Windows Phone -- and they're not phones, of course, they're minis, as in, mini computers), digital music devices (iPod versus Zune), and, another measure, market capitalization -- Apple closed at an all-time high on Monday ($342.45), for a market cap of $314 billion, second only to Exxon Mobile in American companies. (The most recent number I can get for Microsoft's market cap is $240.5 billion. Not bad, but less than $314 billion, clearly.)

I would include digital music sales, but I don't actually think Microsoft really does that in any workable manner (iTunes Music Store versus what?). I'll have to check. I guess I could mention PlaysForSure. Oh I guess it's Zune Marketplace, which seems like a copy of... the iTunes Music Store. Well that continues to back my point. (I'll place it in the table below.)

The point is, Microsoft loves to copy, or on occasion purchase outright, what others are doing. And they usually do it badly, in terms of design and usability, despite market success over the long term, so far. Let's look at some examples.

Microsoft	Bought/Copied
Bing	Copied Google (see note)
DOS	Licensed another form of DOS (86-DOS)
Halo	Bought Bungie, for the Xbox
Hotmail	Bought, not as good as Gmail
Windows	Copied Apple's Mac OS
Windows Phone	Copied Apple's iOS and iPhone for the most part
Xbox	Copied Sony to challenge the PS platform
Zune	Copied Apple's iPod
Zune Marketplace	Copied Apple's iTunes Music Store

(I corrected/clarified the copying/licensing for DOS, I didn't like my initial explanation and it turned out not to be very accurate.)

The point is, Microsoft does not lead, it follows. I admit one interpretation is that I am saying that first-mover, which I just derided in a previous paragraph, is a strong effect, but that is not what I am saying. What I am saying is that Microsoft does not innovate, nor does it really micro-innovate, copying a product and then improving it as much as they can (incrementally). They buy products when they can't make a good competitor, and they copy products and they still can't make a good competitor (Apple's OSes have always been better, don't compare market share unproblematically since you have to deal with the hardware side of the equation, where Microsoft was not making the computers).

Apple tends to make products (and services) that make new markets (or, do so successfully). They are, to some extent, based on pre-existing forms, but not ones that are market successes.

Category	Apple's Move
Digital Music Downloads	iTunes Music Store (paid, not early free Napster)
Digital Music Player	iPod (a digital Sony Walkman)
GUI OS/Mouse	Original Mac OS (from work at Xerox PARC)
Smart Phone	iPhone (a mini computer, really)
Unix	OSX (based on NeXTSTEP, from Unix)

I'm avoiding the iPad as I think we need to wait another year to see how the "tablet" market pans out in the initial phase of the iPad era. Like NFL football before Thanksgiving, it's just too soon to tell. Apple's iPad has generated the most media coverage of recent tablets, which could be an honest indicator of quality, but I think there is some effect from journalists (not incorrectly) thinking they need to cover the iPad. There are probably some other items I could place in the list, but those are the most defensible and explainable that I can think of. (So, I'm avoiding USB and floppy disks.)

As I pointed out, it could be argued that none of these were new products either, but the thing is that, unlike with the Microsoft examples, none of these Apple products went up against anything similar that was successful in the market. (RIM's Blackberry series of phones, although cool, were still in the interface and paradigm of cool handheld phone devices, like Palm to some extent, the iPhone is a different device, it is not really a phone in the full sense, it is a computer that also has phone capability -- did you ever call your desktop computer a phone when it had a modem?) OSX didn't face a mass market Unix, Unix is mainly in labs, corporate back offices, and the homes of Linux geeks. (I've had... two? At least two, Linux boxes, so far.) There was an MP3 player before the iPod (I can't recall the name offhand), but it was not widely successful. Apple made a better interface (the wheel, which has now been replaced with touch screens for the most part).

This is also not to say that everything Apple does is gold (the Newton didn't succeed in the market, Apple's computer offerings were overpriced, underpowered, and muddled in the early 1990s, and there have been occasional but odd AppleTV offerings--usually a Mac with a TV tuner card--which seem to have worked themselves out finally), nor that Microsoft isn't a market success (despite, or perhaps because of, monopolistic practices, and who can resist taking a shot at Microsoft's Bob?).

Leading, and innovating, versus following. As others have pointed out, innovation is risky, but not innovating is moreso.

Update: One area I avoided, unintentionally, was web browsers. Sure, both Apple and Microsoft have web browsers, neither are original, and both are bundled with their OSes. I am pretty sure MSIE came before OSX's Safari, so it doesn't fit my pattern, but Safari wasn't establishing a new market. Not every product for both companies work as examples, but more than enough do that we can look at the overall pattern.

Note: When I said Bing was "copying" Google, I didn't mean literally copying the search results, I just meant the idea. Apparently, according to this report, Bing is indeed directly copying Google search results. Amazing.

Microsoft's Kinect

Penny Arcade has had quite a bit of writing about the Kinect, they don't particularly like it, but others are fascinated by it. Whatever their take, I am profoundly disturbed by Jenna Wortham's writeup in the New York Times (indeed where one tends to find her writeups), especially the sentence that says how hackers are "getting the Kinect to do things it was not really meant to do," because this is not at all true (besides the "not really..." part, which any good Wikicultist would flag as "weasel words" and actually be correct about it).

The Kinect was not designed to be a motion sensing device that is inherently and only part of the Xbox 360, if it were, it would have been built in. It is not. It is a motion sensing device that you can connect to something with the right connector, with Microsoft hoping that would be the Xbox 360. And if you know anything about people, you know we like to play with things, especially things we like.

“Anytime there is engagement and excitement around our technology, we see that as a good thing,” said Craig Davidson, senior director for Xbox Live at Microsoft. “It’s naïve to think that any new technology that comes out won’t have a group that tinkers with it.”

Except of course Microsoft, or the people at it, were extremely naïve, because earlier...

A [Microsoft] representative said that it did not “condone the modification of its products” and that it would “work closely with law enforcement and product safety groups to keep Kinect tamper-resistant.”

Microsoft's model has typically been one of control. Control over Windows, control over the Xbox, control over Microsoft Office, and so on. It was Sony that made it easy to load Linux onto their PS3, not Microsoft and its Xbox 360, although Sony later took away this capability (I am not sure of the politics behind that one, it may be interesting). Note that hackers have adapted Linux for both platforms regardless.

But we've seen so many instances where people do like to play with things (it's a part of who we are). For instance, Bethesda's line of games, such as Oblivion, which is available for both the Xbox 360 and Windows. There are no mods for games or anything on the Xbox, it's not part of the business model. (Mods, made by players, opposed to patches and DLC, by the company.) On the PC, however, there is a thriving mod scene (which I have written about). Bethesda supports the modders, gives them forum space, and interviews them (here is one example, and you can check out their posts tagged "modding"). The people at Bethesda know we like to play games and play with games, and we will do so whether they want us to or not. Mods can, and do, fix bugs, add new maps, zones, characters, quests, and everything: for the game producer, your customers can be developers who make the game better, for free. It's not just win-win, it's win-win-win (producer, modder, players).

Here's a recent Ten Best Oblivion Mods list from PC Gamer. Keep in mind Oblivion is over four years old already. In part because it's a great game, but in part because of the mod scene, people are still playing it.

I'm not sure, definitely, how old the modding scene is: the Internet itself is essentially a giant mod, so, 40 years. It depends on your definition. The Flight Sim mod scene is pretty old, dating back to at least 1990. That's 20 years (and Flight Sim is now, or was for a long time, a Microsoft product!). One would think that everyone would have noticed this long-standing given (I resist the word "trend" there, this is a not a trend, it is a constant).

Blast From The Past: 1997, WebTV

12 years certainly does allow for some perspective, although people were insane in 1997 before the dot-con bubble burst (no "con" was not a typo). 

From Microsoft Took WebTV Risk, Despite Loss, by Steve Lohr, May 5, 1997.

WebTV's under-the-hood technology was probably the real lure for Microsoft, says Roger McNamee of Integral Capital Partners, an investment firm. Imagine the day when HDTV, he adds, becomes affordable and popular, with Microsoft charging the manufacturers a license fee of, say, $50 a set for the software that brings the Internet to those souped-up sets.

Well, HDTV is affordable and popular. Microsoft has the Xbox360... Sony's PS3, Apple's Mac TV approach, we have Boxee, Tivo, DVRs, even open source DVRs (Myth). I could go on about other parts of the ecosystem (like, Hulu). The internet is not currently the best delivery system for HDTV (digital cable, satellite, blu-ray...), even though I get both my internet and TV over the same cable.

I hate when analysts say dumb things.

Viruses, Botnets, Compromised Machines

What articles almost never mention is that these computers all run Microsoft Windows. Global Chinese spy net? Attacking Windows. What you should do? Not run Windows at your business. 

NXE Immediate Experience

Like many, I was given.... or forced to take... the New Xbox Experience (NXE) a bit early. There are at least two massive problems. One is that it forces you to have an avatar. I do not want an avatar. Or, a Mii, since that's what it looks exactly like, a Nintendo Mii. I absolutely despise the fact that some idiots at Microsoft think I have to have a Mii to use my Xbox, because it serves no functional purpose (and besides the selections of clothes are ridiculously limited, as are hair styles and colors).

But you can mostly ignore your Mii once you make it. Far more problematic is the interface. You really have to wonder how they achieve such heights of incompetence. Although the previous interface was dismal due to the lack of any sense as to where things were, this one suffers some obvious problems. One is that it wastes a lot of space. Yes, white space (unused space) is useful in some situations (such as buffer areas between different fields), but you don't need a ton of it. The other problem I have noticed so far is with the design of the interface in terms of the gestalt. There is a text list on the left hand side, each of which is a category of something or other. Each category then opens up a series of visual panes across the middle of the interface. You can scroll up and down through the list, and left and right across the panes. But why do they mix the text and visual gestalts? The big problem here, though, is that the panes cover up the list below the current selection. So, if you want to find something in the text list, you can read the list above where you are (although the text gets smaller and harder to read), but if what you are looking for is below your current selection in the list, you can't see it. (The list does wrap, though.) Unbelievable.

So far the look and feel is more like Windows. I wish they would put more effort into getting the hardware working, although my 4th Xbox is doing alright to date. If the hardware doesn't work, it doesn't matter how cool the OS is, I will never see it because my Xbox will be back at the repair center. (Yes, of course I mean Xbox 360. Whatever. I wonder what they will name the 3rd gen. Xbox Sphere?)

Update: I am not the only one who hates NXE (although I generally like NME), so does Gabe at PA: "Gabriel hates the New Xbox Experience". Oh and look a RRoD that isn't mine. Amazing.

Computer Systems

The XBox came back today! That's only 11 days. We'll see if it works later. Also, been having trouble getting my G4 Powerbook onto the net wirelessly with WPA and there is conflicting info online about whether it will work or not (Apple says it can, but is vague about it), and I tried to get Galactic Civilizations II running under Parallels but there is conflicting information about that as well (and it isn't working). Everything is uncertain!

Update: They sent me a different XBox (new? refurb?). Seems to work so far, although there is a loose part inside. Think they would have noticed that. I have it hooked up to my Apple Airport Express, which is rather amusing, although it means I don't have to adjust any wireless settings on the XBox (since it's all done to the AAE on my iMac, much better interface). 

Returning The XBox, Again

It is true, I have just returned my third XBox 360 to the mothership via UPS in the nice white box. (This is the one that RRoD'ed on me immediately when I got it back last time. Well done, Microsoft.) You'd think they could make better technology. Oh wait, how foolish of me, these are the people that make Windows and the Zune, what am I thinking? Off to Mesquite, TX, with it.

Microsoft Advertisements

This post is not about those completely nonsensical Seinfeld and Gates ads. And no, talking about the ads in this case really isn't a good thing, since we question the sanity of Microsoft's decisions.

This post is about the more recent "I'm a PC" advertisements from Microsoft. These ads remind viewers of the Apple ads. We think about Apple the entire time. We think how seriously Microsoft must take these Apple ads and the products behind them. Microsoft wants us to take Microsoft seriously, so, we in turn have to take seriously the things Microsoft takes seriously, and that's Apple. 

Great when your competitors are paying for ads that make consumers think about you as something to be reckoned with.

At the Sheraton

Yes it is. One of those Microsoft Table things. Unfortunately it is more like a lame version of a giant iPhone. The interface is gimmicky, less functional: sometimes it zooms and pivots when I am only moving it, not zooming or pivoting it. They didn't get it right. And sadly these ones aren't Internet capable. Really it's just a big, poorly done iPhone. Don't need it. If they did it right, and made them affordable, then maybe we could all have cricks in our necks from looking at the floor - oh bad ergonomics. But the interface really isn't new, although many features are not widely available they've been around in academic labs for quite some time.

Microsoft and the Digital Flag

Hope you've seen this, but apparently Microsoft is a bunch of idiots again (surprise surprise). The Digital Broadcast Flag was struck down, but Microsoft built it into Vista anyways, and now people can't record some shows (but come on, American Gladiator?). Unbelievable. So much for the personal in personal computers. If PCs were like this (locked down and burdened with DRM) when Bill Gates was at Harvard, we would never have had Microsoft. Ok maybe that's a good thing, but the reason is not so good.

Microsoft, the Zune, and People

Two days ago, the NYT bits blog ran a small piece on Microsoft's Zune and possible future DRM issues (and NBC and NBC's content). What is amazing is not so much the piece (and the stupidity of the people at Microsoft and NBC), but instead that there are currently 195 comments and that there is so much hatred for Microsoft in them. Amazing branding. How could people hate a music player? (I leave that obvious exercise up to you, but if Microsoft can't figure it out...) What is astonishing is that sometimes people at Microsoft actually say the right things (but we have so many phrases about the gulf between intent and action... do as I say, not as I do... easier said than done...).

Played For Sure, You Were!

Although the story is a bit old at this point, TidBITS has a nice writeup about how Microsoft is... (how to put it politely?) completely betraying their customers with their cancellation of their "Plays For Sure" DRM music system (great name, the humor is incredible, especially since they were sincere about the name -- but probably it was just marketing, and who trusts that?). The TidBITS article also points to an amusing analysis of Microsoft's "we are screwing you over but continue to trust us" letter.

I am reminded of the Microsoft (re)makes the iPod packaging video from a few years ago. This is why we don't want Microsoft to buy Yahoo; Yahoo would be destroyed (in several ways).