I recently was assigned a project at work. At its core, it’s software that processes incoming documents and creates another document (very vague, I know). It outputs a document in the .docx format.
We were in the middle of QA testing for a new version of this software. The QA tells me there’s a bug with the formatting. “No worries,” I say, as I open up Word and modify the template. They get back to me to tell me that there’s an issue on page 17 of the document.
I open up the document in the web preview of our software’s UI. I scroll down to page 17-wait, why are there only 16 pages? I download the document and open it with Word. 17 pages. What.
QA gets back to me again, telling me there’s some formatting issues. “But I fixed those,” I say, confused. I open the document in the web preview, and lo and behold, the indentation errors I JUST FIXED. Right there.
Now I am confused. I download the document and open it with Word. The formatting is… correct? I pull up my Word window and the web preview window side-by-side. I spot several differences. Some very minor, some that place the text onto another page altogether. “Oh well, there’s no issue with the document itself, it’s just the web preview,” I say. They accept it, and we move on.
A day later, QA tells me the PDF generated by the e-signature provider is extremely inconsistent. I take a look at the PDF. Weird fonts. Weird indentation. Nothing dysfunctional or broken beyond recognition, but small things here and there that just make it look… weird. I tell them it’s out of my control. Other component developers bring forward the same issues. QA throws their hands up, tells us to mark it as a known bug, and we move on.
why word, why?
After I was done with the component development and had time to think, I wondered why I faced the issues that occurred. It’s the same document. Why were the web view and PDF parser showing me two completely different documents? Why the inconsistency? Why did I spend half a week scratching my head over this before realizing it was out of my control?
The first thought that came to mind was, “Oh well, DOCX, PPTX, XLSX and all those Office formats are proprietary. It makes sense why other document readers cannot parse them correctly.”
The second thought that came to mind was, “Wait a minute. Can’t Google Docs read it? Isn’t the whole Office file format open source?”
I remembered reading about the Office Open XML (OOXML) standard, so I decided to do some digging. That took me down a rabbit hole of absolutely stupid decisions that led to this point. I hope you’re ready for story time. I swear it’s interesting.
the office file formats
Back in ye old days (the 1990s), Microsoft Office burst onto the scene with a word processor, a spreadsheet tool, and presentation software. I think we all know how that went, so instead, I will focus on the formats.
Back then, each product in the Office suite stored its files in binary form (Word had .doc, Excel had .xls, PowerPoint had .ppt). This was, of course, not very open, and not very fun to work with. Microsoft needed a better format.
Starting in 2000, they released some new formats called the Microsoft Office XML formats (not the same as OOXML; I’ll get to it). Office XP used this format for Excel spreadsheets, while Word documents would use their own XML format in Office 2003.
Now, the file was no longer a binary blob. It was a readable file in a new markup language format called XML, which had been published in 1998. This was something of a transitional period for Microsoft, since all the programs used different formats. Also, the data was now stored in one uncompressed XML file, which was not space-efficient.
Starting with Office 2007, Microsoft unveiled the “one format to rule them all”: Office Open XML (OOXML). This was an advancement over both the binary and older XML formats, in that it was now a compressed ZIP package, containing text, formatting, font data, embedded images, and everything needed to render a document. Speaking of .zip, here’s a fun game to play: try renaming a Word .docx file’s extension to .zip and extracting it.
This format was used by Word (.docx), PowerPoint (.pptx), and Excel (.xlsx). On paper, it seemed like a great improvement, and it actually was. Markup languages had been the way to go ever since HTML came onto the scene. They were just better at conveying information for formatted documents like the aforementioned. So far, everything seemed good. They had future-proofed their documents, safeguarding their products from obsolescence.
Then came the European Union.
bonjour
I must admit, that last sentence was purely for dramatic effect. It’s not wrong, though; the EU does have a role to play in this story.
Back in the early 2000s, governments were using these products to create documents; Microsoft Office and Adobe Acrobat PDF, to name the biggest examples. Slowly, they started to realize what was happening. The products they were using were mostly proprietary, meaning the fate of their super-extremely-important documents rested in the hands (and profit margins) of Big Corporation. Not a very ideal situation to be in, considering the kinds of documents you’re saving.
And so, in 2004, the EU asked Microsoft, “Hey, we’re using your document formats. How about you make them an open standard so we know you won’t shut them down any time soon?”
I mean, Microsoft didn’t have to do that, did they? What were the EU gonna do otherwise, stop using MS Office?
Yes. Yes they would.
Unlike my parents (who somehow read my silly blogs, hi!), who have a TON of important work documents in Office, for whom it would be a hassle and an impracticality to change document formats for no reason, the EU does not give a shit. They would much rather like their documents to not be tied to a corporation, as future-proofing. And it’s not like they were starved for options either; the OpenDocument Format (ODF) was already an open standard, having been created and supported by Sun Microsystems. The EU could very well switch to it if Microsoft didn’t comply. (Three years later, Adobe would start the process of opening up the PDF standard for similar reasons.)
Microsoft said, “fineeeeee,” and started work on opening up their OOXML standard. Without it, they would lose a major market in European governments.
But what does it mean to create an “open standard”?
standards, standards, standards
An open standard is a standard that is accessible and usable by anyone, without any payments of any kind. It is in the public domain and is usually maintained by a standards organization.
For example, if I woke up tomorrow and said, “I want to create my own 5G modem,” I would be made fun of, but also, I could theoretically do it if I had the expertise and knowledge. That is because the 5G standards are open, which means that if I made my modem to conform with a set specification, it would (theoretically) work on a 5G network.
The specification lays down the blueprint for how a particular standard works. It’s like a source and destination; you could take any route to get there, but if you follow it correctly, you will reach the destination.
Another advantage of an open standard, as mentioned previously, is that all you need to implement that particular standard is its documentation. You do not need any external undocumented tools or a particular company’s proprietary implementation of the standard as a base (massive foreshadowing). The standards documentation is ideally enough for you to implement it.
the big leagues
In 2006, Microsoft approached Ecma International (known for such hits as the JavaScript standard) with around 2,000 pages of documentation and asked them to submit their OOXML format as an open standard.
I uh, spent the last week writing about the history and entire timeline of what actually happened, and it just kept growing and growing into the thousands of words, so I will spare you the chronology. All you need to know is:
- Microsoft wanted its format standardized
- Ecma (European standards body) published it
- Ecma used their “close buddy status” with ISO to fast-track the publishing of the standard to make it an official internationally-accepted standard
- What followed was a lot of documentation, a lot of protesting, some borderline illegal stuff that M$ did to get the standard passed
- The OOXML standard finally became an official ISO standard, in 2008, as ISO/IEC 29500
Off-topic but i do not have anywhere else to put this, so here is an image of members of the Free Software Foundation in Bengaluru protesting against the ISO standardization, circa 2008. How cool is that?
So, what was the problem?
Now, even if you ignore all the borderline illegal things that M$ did to pass OOXML, it still is an overall terrible open standard.
Before that, let’s quickly understand what the OOXML format was created to do. Officially, the OOXML standard was a “formal standard for office productivity applications that is fully compatible with the Office Open XML Formats, submitted by Microsoft.”
Fair, I guess. Until you realize that the OOXML formats were meant to be compatible with basically EVERY version of Word, PowerPoint, and Excel, dating back decades. Which means every quirk, every workaround, every small feature introduced in one version of Office, never to be introduced again, would have to be supported.
Fun fact: the OOXML standard came out to be around 6,000 pages long. For comparison, the other document standard, ODF, is around 800 pages. This is where a lot of the problems lie: an open standard is meant to be compatible with every decision made by a single corporation. This did raise questions over what it means to be an “open” standard when most of the effort is spent preserving Microsoft-specific legacy behaviour.
Now, MS realised this (seems like they aren’t stupid after all) and deliberately split the standard into two parts: Transitional and Strict. Transitional OOXML would retain all the compatibility with older versions of Office. Strict, by comparison, was easier to implement, as it did not have to retain all those quirks.
Transitional was meant to be just that: a transitional format until the whole world adopted the strict, modern format. But how would they?
Microsoft added the ability to save Office documents in the Strict format in Office 2013, and yet, to this very day, the default format for saving Office documents is the Transitional format. I bet none of you knew there is an option to change this. Giving people an option is not enough if nobody knows what the hell it actually does. Microsoft never really understood the concept of “sane defaults.”
just follow the standard, then
“Arnav, you just said that you can implement a standard by following its documentation. Why do companies not just follow the OOXML standard, so that they can make software compatible with Office documents?” you may ask.
And to that, I respond, “Wait, you actually read what I wrote?”
But the point is, you are correct in assuming that if you simply followed a standard to the letter, your implementation would be perfect. However, you missed a key thing: This is Microsoft we’re talking about.
Of course they were not gonna make it easy for you. First off, remember when I said the standard is 6,000 pages long? Yeah, that would be the first deterrent. Then, even IF you followed it correctly, there is still one final hurdle to cross.
Microsoft’s own implementation of the standard.
Word for Office implements the OOXML standard, but with some tweaks. Tweaks here and there, a different algorithm to render page borders, maybe a half-point difference there, and so on. It seems minute, except when you open a Word-created DOCX in another word processor that follows OOXML, and then things seem off here and there.
Even though the standard was followed correctly, it was not done “how Microsoft does it.” Which wouldn’t matter for most standards. I mean, not every implementation is perfect.
Except Microsoft’s “imperfect” implementation is the baseline for how all Word documents are written, seen, and printed. So every company, no matter how closely they follow the official written standard, needs to also take into account how Microsoft did it.
Which defeats the ENTIRE PURPOSE of what an open standard is supposed to do. Eugh.
what now?
I don’t really know how to end this, so yeah. An “open standard” usually comes with some expectations, that anyone with the technical know-how would be able to successfully make a fully-compliant implementation. Which is somewhat true in this case, but as you probably understood through this whole post, the reality is a bit more complicated. Anyways, I don’t see a competitor coming anytime soon, so we are probably stuck with whatever stupid shit MS thinks of, for the next couple of decades at the very least. Something to think about the next time you open Word or Google Docs or whatever obscure word processor you’re using, idk. Maybe we should all use markdown, that would be fun.
Anyways, that’s all I got. Thanks for reading lol.