and I'm all out of bubble gum…


So… a bit of background: Outlook is a terrible program, and reasonable people might choose to use something better to interact with their emails. In so doing, one often has to fall back on the IMAP protocol to interact with one’s Exchange server.

And, if one’s Exchange server is hosted on Office 365, it appears that all of your data is served up via IMAP. By “all”, I mean not just email (which you expect), but also contacts and calendar events. In general, these should show up on the eponymous folders within your directory (“Contacts” and “Calendars”, and various subfolders within that). This means that your email program can, potentially, allow you to do things with your contacts and calendar events as though they were emails… things that Exchange/Office 365 might not necessarily recognize as “valid” and could result in serious data corruption and deletion. This. Is. Bad.

The Problem

Suppose one connected a client such as MailSpring via IMAP to Office 365. Hypothetically. Not that anyone near and dear to me might have done such a thing, but let’s suppose. It turns out that MailSpring will (correctly) recognize that the calendar events and contacts are not mail messages, and then fail to download them. In fact, the various folders would be filled with messages (one per contact or calendar event) that read:

Retrieval using the IMAP4 protocol failed

This looks like junk. So one might be forgiven for, say, deleting the message. Curiously, MailSpring then deletes the original (unretrieved) file from the server… deleting the contact or calendar event.

As an aside: I observe that many other mail clients (Outlook,, etc.) choose to simply hide these files, rather than presenting this confusing message.

As a further aside: these IMAP-error-message messages, in our scenario, confusingly appeared in the Sent Items folder, rather than a calendar folder. No explanation has thus far been found for this…

So. Maybe a few hundred or thousand calendar events get deleted from an Office 365 account. How to get them back?

The “Solution” (incomplete, unfortunately)

In the end, it was a multi-step process:

  1. Undelete the deleted messages. We did this via the the Office 365 web client for Outlook. We went into the Deleted Items folder, followed the link to recover deleted items, selected everything, and recovered it. This recovered about 700 of the deleted events (not all, but some). At this point, all the previously deleted events are now in the Deleted Items folder, but not fully deleted. (I’m not 100% confident, but I believe we could have done this step in either the old or the new web interface for Outlook.
  2. Restore the messages to the calendar itself. This was harder, because it turns out that — since most email clients, including the new Office 365 Outlook web interface correctly don’t display calendar events in their mail interface — this could only be done in the old Office 365 Outlook web interface. Which means this solution may not last forever. In the old interface, we were able to select batches of items and then, via the “Move…” menu icon, move them back to the calendar. We couldn’t move them all at once, since the web interface couldn’t track selections larger than about 30-40 items at once. At this point, all recovered calendar events now exist in the default calendar.
  3. Clean up by hand. Since multiple calendars were actually involved, we had to go through the recovered items by hand, deleting them from the default calendar.

Further Work

This takes us through midweek. At this point, it appears that this process is not 100% successful: new edits to the calendar are not always “sticking”. Our surmise is that Outlook views the calendar files as corrupted. This is further complexified by the fact that not one, but several calendars were involved. The subscribed calendars come from a separate account (not Office 365) and changes (including the original deletions) seem not to have transfered consistently upstream (although some new additions and moves from the default calendar have).

This is a pain.

Our next step, we think, will be to export the default calendar to an ICS file, unshare the files with the Office 365 account, and basically purge all calendar data from the Office 365 account. Then we’ll reimport the ICS file of exported items to the calendar, and reshare the calendars from

Confidence is not super-high. But I figured that posting this process could be helpful to others in our boat.

May 10th, 2019

Posted In: How To

Tags: , , , , ,

I’m working at a school using Office 365, and OneNote is the main event. I notice, however, that there don’t seem to be many clear, declarative answers to questions that I have about OneNote Class Notebooks, so I did some fiddling to find out.

As of the date of this post, the following questions and answers are empirically true of OneNote 2016 and the most current version of the Class Notebook plugin:

Can I create a class notebook in a custom location?

No (mostly). The motivation for this was that it would be really keen for class notebooks to live in our departmental shared OneDrive folder, so that we can both archive old classes and peek into each other’s notebooks with ease. It turns out that class notebooks can only be created in your personal OneDrive, in the Class Notebooks directory.

If you create the class notebook and then move it to a more desireable location (via OneNote’s ability to sync a notebook to a different location), it will totally make a copy of your notebook in that new location, but all of the students will still be accessing the original location, so it’s not gonna work for actually moving the OneNote.

If you move the class notebook (i.e. sync it to a new location and delete the original), you can no longer manage the notebook via the web interface that you need to use to add/remove students, lock the Collaboration Space, generate parent/guardian links, etc., etc. Essentially, it is dead to the web management interface. (But this does seem like a good way of archiving old class notebooks to a shared space at the end of the year.)

If you move your class notebook back to the Class Notebooks directory where it is supposed to be, the web management interface will not recognize it as a class notebook. And there is no path for (re)converting a seemingly normal OneNote notebook into a class notebook (again).

So, basically: no.

Can you rename the default underscore section groups?

Sort of, but it’s probably not a good idea.

I really don’t like “_Collaboration Space” and “_Content Library” (or “_Teacher Only”), which are the default names of those sections in class notebooks (and for which there is no management interface for renaming them).

You can, of course, just rename them in OneNote. And they’ll get re-alphabetized into the mix with the students’ section groups. And they seem to continue to behave with the proper permissions. Which is good. However, the Class Notebook plugin will now interpret those sections as being additional students (who seem to not have the right sections within their section group, to boot), so that when you try to distribute a page to your students, you need to be careful to not distribute it into your renamed Collaboration Space or Content Library. This is a little confusing.

If you rename them back to their original names, the Class Notebook plugin reverts to treating them as “special” section groups that are not representative of students.

Probably better not to rename them.

Can I put student collaboration sections into a section group after the fact?


You can now create these nifty little collaboration sections within your Collaboration Space that are limited to a specific group of students. You have to create these sections through the web interface. By default, these new collaboration sections are created at the root of the Collaboration Space.

If you move these sections into a new section group (e.g. “Last Week’s Groups”), the permissions appear to carry over into their new location.

In fact, if there are section groups created inside the Collaboration Space, you will have the option of creating new collaboration sections directly inside those section groups, through the web interface. W00t!

What does a parent/guardian link to a student section give access to?

These are created through the web interface.

It legit only gives access to that particular student’s section group. None of the shared section groups (Collaboration Space, Content Library) are visible through that link.

I did not explore what happens if you post a link into one of those section groups into a student section group… I doubt the link would be follow-able.

What does a link to the Content Library give access to?

Also created through the web interface.

This is also tightly restricted: just the Content Library. There doesn’t seem to be a built-in mechanism for sharing access to the Collaboration Space.

August 31st, 2018

Posted In: How To

Tags: , , , ,

For the past few years (well, since I’ve had an iPad), I’ve been exasperated by the time it takes whichever app Apple has streaming video from my iTunes library to my iPad to load my iTunes library. In the past, it has taken anywhere from 20-90 minutes to load the library each time I access it. Clearly, this isn’t really viable as a “streaming TV in bed” approach. I did some digging around, and it seems like the root issue is that my iTunes library is too big, so the iPad app can’t process the XML file it has to download (aside: this is ridiculous, of course).

Long story short, at the start of the summer, being blissfully unemployed, I took a moment to split my iTunes library in twain: one library of music and audiobooks to sync to my phone (not that that integration has been getting any better, mind you) and one library of movies and TV shows to stream to the AppleTV and my iPad and whatnot. (Note: this whole time the AppleTV could load my gigundous library… but not the iPad. Oy.)

Much metadata cleaning later (because it turns out that iTunes doesn’t actually write all the fields it should write to video files, because, well… it’s lousy), I now have two libraries. Most of the time I want the movies and TV library open, but I want to regularly open the music and audiobooks to sync podcasts (not that I trust Apple for that either: greg is in charge of syncing, iTunes just adds the new files to the library).

In the past, I’ve noted that I’ve found CLI defaults tool to be useful for managing a remote iTunes library. This proves not to be the case here — in fact, the solution is one of simple symlinking. I’ve written three scripts: a “master” script that just toggles between libraries and two “client” scripts that just pass a specific parameter to the master script (depending on which library should be opened).

August 12, 2017 edit: I have found that AppleScript is just too flaky for this operation, and have switched to an end run around iTunes using bash scripting. The original AppleScripts are still available, but I recommend the bash approach below at this point:

It’s worth noting a few things here:

  1. This script assumes that there is no actual valid iTunes library (or even a directory) at ~/Music/iTunes — the script operates by swapping out symlinks at that location to other libraries. The script does do a sanity check before trying to effect that swap, of course.
  2. I initially tried to do this using system accessibility scripting but found that to be unreliable (pasting the path into the “go to folder” dialog as iTunes opened was flaking reliably).
  3. Along the way, while I could have used the CLI osascript command to open the “master” script with appropriate parameters in the crontab… I couldn’t figure out how to give osascript appropriate permissions in the accessibility section of the Security & Privacy system preferences. Frustrating. [August 12, 2017 edit: …and I no longer both to schedule switching libraries with crontab anyway.]

July 30th, 2017

Posted In: How To

Tags: , , , ,

There seems to be some discussion online, but no actual, y’know… documentation regarding these two server variables.

Apparently they were introduced in Apache 2.3, and they are wicked useful, it turns out. At least, wicked useful if what you want to do is figure out what the URL of a to a specific file on the server is, assuming that you have only its path and some other file’s URL.

What follows seems to be empirically correct, but I welcome corrections:

DOCUMENT_ROOT is the system file path to the root of the web server’s document directory (usually something like '/var/www/')

But, imagine that per-user web directories have been enabled on your server. That is, if my user is foo, then there is a directory /home/foo/public_html in which all of foo’s web documents are stored (and served from). This is (probably) outside of the main server document root directory. The URL of foo’s web files would be something like

And this is where our variables enter the story.

The Empirically-based Theory

CONTEXT_PREFIX is the portion of the REQUEST_URI that triggered the server to serve up a file outside of the document root. And CONTEXT_DOCUMENT_ROOT is the root directory that that particular CONTEXT_PREFIX is rooted at. Nota bene: conveniently CONTEXT_DOCUMENT_ROOT contains the path the document root for whichever context this particular script was accessed through (so it would be the same as DOCUMENT_ROOT within the server’s document root, while CONTEXT_PREFIX would be an empty string, since there is no context prefix within the server’s document root.

A file in a user web directory

Consider this extended example:

User foo has placed files in that user’s public_html directory. There are nested directories within that public_html directory, and we are accessing a file at this URL:

This file is stored at the following path in the file system:


Thus, a sampling of relevant server variables (presented in $_SERVER for the PHP users amongst us) would be:

REQUEST_URI is /~foo/dir1/dir2/dir3/file.php?paramA

SCRIPT_NAME is /~foo/dir1/dir2/dir3/file.php


CONTEXT_DOCUMENT_ROOT is /home/foo/public_html

A file within server’s document root

Alternatively, a similar example within the server’s document root:

is a URL that access a file at:


This results in the following values:

REQUEST_URI is /dir1/dir2/dir3/file.php?paramA

SCRIPT_NAME is /dir1/dir2/dir3/file.php

CONTEXT_PREFIX is (the empty string)


For giggles, I cooked up some code to generate the URL of particular directory based on this information. I used the following little snippet to do this in DataUtilities::overlap() in battis/data-utilities:

March 30th, 2016

Posted In: How To

Tags: ,

So, my students in Advanced CS are working on their second semester projects. One project group is working on mastering Unity to build an iPhone game, and they just spent several days struggling with getting Unity Remote to work with their iPhones.

After much Googling and exasperation, they figured out the answer (which was not, prior to this post.

They had been able to see the iPhone in the Unity editor. They had been able to set up the remote to use the iPhone in the Unity editor in project settings. But when they “played” their game in the editor, it just sat there inertly, waiting for a connection. Consternation. Rending of garments. So forth and so on.

The trick, it turns out, is that when one installs the Unity editor, one must install iOS development support. Not a shocker. But certainly worth mentioning in an error message somewhere.

March 11th, 2016

Posted In: How To

Tags: , , ,

I just mentioned that my wife switches between two different iTunes libraries, including one that lives on a shared volume on our home server. There’s a great tip on Stack Overflow for writing a script to automate that swap between iTunes libraries (rather than having to hold down option while iTunes loads and then click around to find the library you want). But it seemed like it might be nice to document for posterity the full script that we’ve built, since it handles not just the library swapping, but also the mounting of the shared volume as well.

This script takes a little prep work, both generating the library location property and storing the authentication information in the user keychain. The Stack Overflow tip explains the former, and the latter is as easy as connecting to the server and remembering to check the “Save Authentication” checkbox during the authentication process.

March 18th, 2014

Posted In: How To

Tags: , ,

A bit of background: my wife has an iTunes library that is too large to fit comfortably on her MacBook Air (in all fairness: I have an iTunes library that is too large to fit on her MacBook Air, my MacBook Pro and several other laptops as well). She maintains a local library on her laptop for use at school (useful movies, songs, etc.) and stores her “real” library on our home server. Which involves a bit of fiddling to get it to work. The key issue is that she needs iTunes to load her library off of a shared disk. And, periodically (maybe every couple of months), something goes haywire and she gets this error:

The iTunes library Library file is locked, on a locked disk, or you do not have write permission for this file.

Needless to say, this is exasperating.

There are a number of suggested solutions out there, and I keep forgetting to bookmark “the one” that works. On my last search, I realized that this was because none of them are really the answer that works in our situation. Here’s what works (and I’m posting this as a note for self):

  1. Disconnect the laptop from the server (quitting iTunes first, of course).
  2. On the server, unset the nouchg flag (not 100% that this is necessary 100% of the time, but it’s necessary at least some of the time):
    sudo chflags -R nouchg
  3. Reboot the server.
  4. Reconnect from the laptop and rejoice.

At least, that’s what worked this evening, and it felt really familiar. (There’s an unspoken step 2a, of course: cancel the multi-hour process on the server that you had queued up moments before iTunes started misbehaving. Argh.)

March 18th, 2014

Posted In: How To

Tags: , , ,

So many of these entries are really just an attempt to make sure that the next time I go searching for an answer to some question, by gum there will be a search result. That is: I write up things that I couldn’t find an easy answer to, so that someone else reaps the benefit of my suffering.

But, sometimes, it’s just about admitting that I’m dumb.

Case in point: I’ve been grousing on Twitter about a number of things in the past week, one of which was that, for much of the fall, many of my Google calendars have been showing up in iCal doubled. Every event shows up twice. Sometimes more than that, when Google has a hiccup. And birthdays from my contacts have shown up as many as 34 times — no joke: I counted. But the core problem has been that I’ve been seeing many of my Google calendars twice in iCal. It makes me feel busy, but otherwise it serves no purpose.

And then it hit me this evening, as I was looking at the list of calendars in iCal: with the launch of Mountain Lion, Apple revised how it handled Mail, Contacts & Calendars — so much so, in fact, that they created a separate prefpane for it. That connects to iCal, Mail and the Address Book pretty transparently. Before that upgrade, to see secondary Google calendars (the ones other than your primary calendar — a distinction about which I have some more grousing to do at a later date), you had to enable calendar delegates in the Accounts section of the iCal preferences. It was messy, and ugly (you had both the delegate and the calendar nested in the delegate… for each and every calendar). But it worked.

I never turned off the delegates when Apple upgraded to Mail, Contacts & Calendars. So I was seeing both the delegate calendar and the calendar associated with my Google account in Mail, Contacts & Calendars.

For dumb.

I just turned off delegates and everything got better. Except maybe those duplicated birthdays… we’ll see what happens with them.

December 9th, 2012

Posted In: How To

Tags: , , , , ,

YouTube has changed their embed codes (several years ago) to IFRAMEs, replacing the old object embed codes. Turns out that on older web-based WYSIWYG-editors, such as Blackboard 8, the IFRAME embed codes get munged up, often with deleterious effects. In Blackboard, for example, you might lose the ability to see, edit or use a content area in which an IFRAME has been embedded. This. Is. Bad.

YouTube lets you switch to their old embed code for individual videos, but no so to embed a playlist. I wanted to embed a playlist, so I took a look at the object tag specs and the IFRAME embed code and came up with a simple script that generates an old-style object embed code for a YouTube playlist IFRAME embed code, thus:

Paste a YouTube playlist embed code below.

November 30th, 2012

Posted In: How To

Tags: , , , , , , , ,

I’m about to embark on a group video project in my New Media class. To that end, it’s a hassle when the video (and script!) exist only in the account of one of the group members. I saw one suggestion for how to store the iMovie events and projects on an external drive, which seemed deeply convoluted and, well… a hassle. I’m not interested in purchasing or setting up external drives, or in going through that rigamarole to set it up.

So, it occurred to me that /Users/Shared/ is a pretty fine place to store shared files. And that the modern Mac OS X seems to honor symlinks pretty regularly (something I use all the time to sync my life across machines using dropbox — symlinking whole directories to sync preferences, settings and documents that the developers haven’t — yet — moved to the cloud.)

It seems to work for iMovie as well. Here’s how I did it (stay tuned to see if it blows up on my students!):

  1. Trash the iMovie Events folder in the user’s Movies folder. (This assumes that this is, essentially, a clean install of iMovie — no previous events or projects to worry about. That’s my situation).
  2. Create a new iMovie Events folder in /Users/Shared/ (note that it is created there, not moved or copied there, so that it will inherit permissions from the Shared directory).
  3. Set the permissions on iMovie Events to let everyone read and write. I like the terminal for this:

    chmod -R a+rw /Users/Shared/iMovie\ Events
  4. In the terminal, create the symlink:

    ln -s /Users/Shared/iMovie\ Events ~/Movies/iMovie\ Events

    (Note that the space in “iMovie Events” needs to be escaped with a backslash!)

  5. In iMovie, create new projects on the hard drive (probably called Macintosh HD), rather than the user’s home folder. This will put the projects in /User/Shared/ (I didn’t know that until this experiment — cool!)

In my limited testing, this seems to work transparently. The biggest caveat is the permissions change. When creating events, iMovie seems to strip the write permissions off of the inherited permissions for the iMovie Events folder (but other users can still read and execute, which should be fine). Similarly, it’s possible that other users can only read (but not edit) the shared project.

We shall see.

October 31st, 2012

Posted In: How To

Tags: , , ,

Next Page »