Winter term: Goal setting and structure

After spending over a year holding on for dear life while running as fast as I can on a hyperspeed treadmill and juggling flaming chainsaws, I find myself in the enviable position of having A LIGHT TEACHING TERM. I’m advising two capstone “Comps” groups, running a one-credit seminar for a research cohort program, and working with research students.

To be fair, this is my heavy term for my administrative role, which means I’ll be racing on a treadmill of a different sort. But at least I won’t be juggling a heavy administrative load with a heavy teaching load, like I did last winter. (Do. Not. Recommend.) The workload is still significant, but the rhythm is completely different. More meetings, less rushing to post things on Moodle. More strategic planning, less specifications grading. More reporting, less recording video lectures and demonstrations.

More control over my time, less I-need-to-be-in-front-of-students time.

Knowing myself — and recognizing that the events of the past year, and the past week, have pushed my anxiety and depression into overdrive — I know that if I’m not careful, I can easily fall into a black hole of despair. The lack of a strict structure and schedule is not my friend in this regard.

One thing that does help? It’s the start of the year, which means it’s goal-setting time! And even Depressed and Anxious Me loooooooves a good goal-setting session. So I’ve tried to use this to my advantage — leveraging my goals to set up systems and a structure that should hopefully keep me on track this term, or at least keep me from falling too deeply into the abyss.

Goals

I decided to have my #21for2021 list serve as my goal list for the year, and tried to structure it accordingly, with work, home, and personal goals.

In Week 1 of 2021, I did manage to do (7) (call Mom), but failed to do (9) (reach out to someone in my work network). Whoops.

I usually set monthly goals, goals for each academic term (plus the summer), and weekly priorities (which I set at my Sunday Meeting). This year I’m still doing the monthly goal-setting and weekly priorities lists, but I’m experimenting with true quarterly goals (January-March, April-June, July-September, October-December). Quarters mostly overlap with academic terms, so it’s not a huge departure. It’s in these monthly and quarterly goals where I’ll get SMART (specific, measurable, achievable, realistic, time-bound) and detailed. If 2021 is anything like 2020 (or, er, even worse than 2020), these semi-frequent checkins will allow me to reevaluate and pivot if a goal just doesn’t make sense given the circumstances.

Notebook listing goals for the first quarter of 2021.
My first quarter goals for 2021. “Sustainability” seems to be the theme.

One thing I would like to be better about is rewards! I tend to finish something big and move on, without marking the achievement. (I still haven’t celebrated my promotion to full professor, or being named to my administrative position!) This is…not healthy. And probably not the best example for my kiddos. One of my January goals is to submit a paper for review — so maybe I should start by celebrating that win when it happens!

Crafting a routine

With lots of open time and lots of tasks to fill that time, it’s easy to get overwhelmed, or move from thing to thing without making much progress on anything. I’ve had some luck with “theming” my days in the past, so I’m using that this term. I give each day a “theme” that defines the type of work I’ll focus on for that day. For example, here are my themes for this term:

  • Leadership Mondays. I focus mainly on tasks associated with my administrative role. I meet with my dean and my program manager, work through project to-do lists, and do some long-term and short-term planning. I also set aside some time to do some leadership role-related reading — right now, for instance, I’m reading From Equity Talk to Equity Walk, which is useful to approximately 5673 parts of my job.
  • Deep Work Tuesdays. I always reserve Tuesdays for deep work, since I rarely teach on Tuesdays and I try hard to protect the day from meetings. I work on research and writing projects. If there are any big leadership things I didn’t finish on Monday, I’ll work on those on Tuesdays, too.
  • Research / Writing Wednesdays. Wednesdays are a bit more fragmented, with more meetings, so I use them to finish up writing and research tasks from Tuesday, and do research and writing tasks that are a bit clearer and more focused. If I happen to have a light meeting day, I’ll take advantage of that to do more deep work.
  • Meeting Thursdays. Thursdays are my heavy meeting days. This is where I put all those small tasks that I can do between meetings and / or when my brain is fried from peopling.
  • Career Planning Fridays. This ends up being more aspirational, because Fridays also become a dumping ground for everything that didn’t get finished Monday through Thursday and / or tasks from all of those Thursday meetings. But since I do have the luxury of devoting Mondays to administrative tasks, I’m really trying hard to reserve part of Fridays for long-term career planning — touching base with mentors, putting together materials for an administrative job search, figuring out what roles I might want to pursue, career-focused reading, etc. (Right now I’m reading How to Be a Dean.)

I’m also taking advantage of the flexibility to incorporate reading for work into my daily routine, something which all too often gets pushed off of the to-do list. I now read for 15 minutes right after I meditate each morning, so I can check it off the list right away. I’m hoping I can make this enough of a habit that I’ll continue it in the spring, when I have a more traditional teaching schedule.


However you’re approaching goal-setting and routine establishment this year, whether you’re going all-in or stepping back in the name of self-care, I hope this year is starting off well for you. And I’d love to hear your goals and strategies for approaching what looks like another uncertain year.

5 Lessons from Fall Term

Winter Term is underway (more on that next week)! Yet I still find myself processing and attempting to make sense of Fall Term. To be honest, I find myself dealing with what I can only describe as lingering and persistent trauma — not just over Fall Term, but over the state of the world more generally. It’s hard to process and analyze when everything feels so uncertain and impossibly hard.

On balance, Fall Term went…surprisingly well, given the circumstances. I had a small, engaged class of 15 students. No one unexpectedly disappeared, and everyone passed the course. My course revisions mostly worked, save for a project that went off the rails due to undocumented conflicts in different minor versions of Python. And I managed to make some forward progress on research and various other projects.

In thinking about the term, I found myself returning to five lessons I learned, or re-learned, over the course of the term.

Lesson 1: Everyone is struggling. And it’s ok to acknowledge that publicly.

Fall Term was hard for lots of us, for a variety of reasons. Time zone differences. Health, including mental health, issues. Worries over the election. Concern over the risk-taking behavior of other students. Racial trauma. Isolation and loneliness. Caregiving responsibilities. And while Carleton was not fully online, many of its courses were at least partially online, which meant everyone (students, faculty, and staff) spent much of their days interacting online — difficult even in the best of circumstances. In short, no one’s at their best.

I made checking in with my students a priority. I borrowed an idea from a staff colleague and started each synchronous class meeting with the same anonymous poll, asking them how they were doing. Originally I just summarized the responses, but as the term went on I started displaying the results as a percentage of respondents. I commented briefly, adding (truthfully) where I fell among the options, acknowledging the mindspace we collectively occupied that day, and reminded those who were struggling of various ways to reach out and seek help. Students indicated that they found this helpful — both to see that they were not alone wherever they fell on the continuum that week, and that I was honest about my own struggles. This is definitely something I will continue, including whenever we return to in-person instruction.

Poll window asking "how are you doing today?" with multiple choice options
Zoom editor view of the check-in poll I used at each synchronous class meeting.

Lesson 2: Teaching online is easier the second time around

Don’t get me wrong: Teaching online still feels unnatural, weird, and hard. But it felt way less so than it did in the spring. I was able to tap into the lessons I learned about organizing a week, a lesson, a class meeting, an explanation, and apply them to a very different course. Everything seemed to flow much better — even the project that went off the rails. It also helped that students had a term of online learning under their belts, and knew what to expect — from the modality, from each other, and from me.

I also appreciated even more all of the pedagogical work I put in this summer, and the pedagogical workshops I attended. It was time very much well spent and definitely made a huge difference in how the class ran, and worked.

Lesson 3: Specifications grading helped…a lot

Based on my reading of Specifications Grading and Grading for Equity, I completely revamped my course grading. Did it work? Hell yes!

I found this new-to-me style of grading freeing. Rather than agonizing over “is this exam answer worth 4 points or 5 points?”, I only had to ask “does this answer meet the expectations for the learning objective or concept?” Turns out, in most cases that’s a much easier question to answer. And knowing that students could revise and resubmit any summative work, I found it easier to make these judgment calls. Weirdly, I actually kind of enjoyed grading!

Most students took advantage of the revision opportunities — some multiple times. I found that a subset of the students were really invested in improving their learning through the revision process — and that this freed up some of them to take risks they might not have normally taken. Which, of course, is exactly what I want to happen in my courses! That said, from a grading management perspective, in the future I will likely limit the number of revisions, probably through some kind of token system, to prevent my workload from spiraling out of control.

I never quite figured out how to get Moodle to play nicely with this grading system. I ended up converting the expectations scale to a 4.0 scale and averaging things within categories to calculate the course grades. It was hard for students to figure out their own course grades because the averaging was somewhat opaque and was done outside of Moodle. In the future, I will invest the time to bake this into Moodle so that students have a better sense of how they’re doing in the course.

Lesson 4: Online pedagogy allows for some new collaborative learning opportunities

Computer Networks (the course I taught this fall) is conceptually tricky and often dense. In an in-person class, I make heavy use of office hours and class time to help students extract the important points of a concept, technique, protocol specification, or algorithm from the seemingly overwhelming details. After some success using Hypothes.is, an online annotation tool, in the spring, I experimented with Hypothes.is for some of the denser readings in the course. For a few of the daily targeted readings, I had students answer the reading questions in their small groups by annotating the reading with their answers. For a few others, I pre-annotated the reading to focus their attention on the main points, and had students comment on the annotations and/or add their own. I really liked how this worked out overall, and I think the students got more out of those readings. I plan to continue this practice in the spring and likely beyond. I could see it working really effectively for Intro CS and for Data Structures (our CS 2), where it’s really easy for students to get lost in the details of a reading.

Moving things online wasn’t always neat and contained, but sometimes that’s ok. I usually run an in-class simulation of Internet routing, where students act as autonomous systems in small teams: creating routing tables, entering into peering agreements with each other, and ultimately attempting to “route” data. What normally takes one class period in person spread over several days online. It was messy, and chaotic — and probably taught students about the messiness of real-world Internet routing more effectively and deeply than anything else I have attempted in my 17 years of teaching this topic.

Lesson 5: It’s really, really hard to troubleshoot virtually

When I can’t figure something out, I need to sit down and play around with it. When teaching in person, I spend a lot of time running to the computer lab so that I can see what the students see, in their coding environment, with the same tools and version of Python and all that good stuff.

When the rogue project went off the rails, I found myself flying blind. What version of Python were the students using? Were they all using the same version of Python? Is it the same as mine? Why does the code sometimes work when we ssh in to one of the servers, but not consistently? How do I help Windows users — most of my students — when I have a Mac? Ultimately, I was limited in how much troubleshooting I could do. I’m still not sure what I could have done better, other than perhaps requiring the students to run and develop the code within a virtual machine of some sort. But it’s something I continue to reflect upon how to improve — and particularly, how to better support Windows users in my courses.


I have a lighter teaching load this term — just our capstone, to make room for the heavier workload of my administrative role this term. But I’m already thinking ahead to how I can capture, consolidate, and integrate these lessons into my spring term course — the same course I taught last spring. I look forward to seeing how much better and more effective I can make that course based on what I learned this term.

Winter…break?

Carleton’s Fall Term ended, mercifully, over 2 weeks ago (end of classes, finals, grades submitted, the whole enchilada). Because of the way our calendar works, nothing changed from the way Fall Term usually works — we’re always fully done by Thanksgiving, with grades due the following week. Of course, that was really the only thing that remained “the same” about Fall Term. I’m still trying to wrap my head around the term (and tie up a few loose ends), and once I do I’ll have a post about that wild adventure.

But for now, it’s Winter Break, that glorious 6 week stretch between Fall and Winter Terms. This aspect of our calendar is definitely a huge perk of this particular job. (We pay for it, dearly, later in the year, with a very short turnaround between Winter and Spring Terms and a Spring Term that lasts into June.)

And every Winter Break brings the urge to…schedule the hell out of the time available. Believe that I will, in fact, complete approximately 20 projects during that time, write for uninterrupted hours each day, and finally catch up on All The Things! Why yes, I will attend that pedagogical workshop! And review that article! And completely overhaul my professional web page!*

This year, I allowed myself an hour of Fantasy Winter Break Planning, where I listed out all of the glorious things I would do. I made myself write this out on a very large piece of paper. I filled up that whole piece of paper.

And then I told myself to pick 3 things. Not 3 categories of things. 3 things.

After I stopped bargaining with myself (“how about 4? 4 is close to 3. ok, well, what if the category is small?”), I picked 3 things, and except for one (which I haven’t started yet because these first 2 weeks have been more meeting-heavy than I anticipated and something had to give), I’ve been making consistent progress. And not stressing (too much) about all of the things I’m not doing. And, most importantly, not working every night and every weekend. (Weekends off! It’s been a while.)

So, what 3 things that are my priorities for Winter Break?

  1. Complete a draft of an article about civic engagement in computer science. This is actually an item on my #20for2020 list that was going to be my main focus during my lighter Spring Term last year. (Thanks, global pandemic.) I’ve chipped away at it here and there, and I’d like to get it, if not out for review by the end of the month, then at least in good shape to submit somewhere in early January.
  2. Clean up the dataset we’re analyzing in our current project. There’s some information in the dataset that really shouldn’t be in there (it wasn’t cleaned as thoroughly as we expected), and we’ve been removing it piecemeal, but now we’re left with the things that are trickier to remove. Normally I wouldn’t put a task like this as a major priority, but cleaning this properly is going to take some sustained time and attention — and I think the techniques will come in handy with some other research tasks down the road.
  3. Mid- and long-term STEM planning. I did some of this in the fall, but honestly I mostly operated in triage mode. My goal is to move the STEM Board from “mostly reactive” to “mostly proactive” and from “here are the tasks we do” to “here’s how we plan for the future”. I also may need to finish up some reporting from, um, the previous year….

Of course things are not all smooth sailing, because another opportunity just came to my attention that I think is worth making room for. The good news is that if I do decide to do this, it will be off my plate by next Friday and I may be able to get most of it done in a 3-4 hour block. The bad news is that I’d probably need to give up one of my weekend afternoons to make this happen. Not ideal, but I think the payoff will be worth it if it’s accepted.**

On balance, I feel less frantic than I usually do. I’m not trying to do All The Things, and I’m (mostly) comfortable with that. I know that I have room in January for some of those things, and that the world won’t end if some things happen in January instead of December. I also know that I don’t have to completely finish every single aspect of those 3 things for my Winter Break to be successful. Good, solid, consistent progress is plenty, especially for a year like 2020. And, one could argue, should be plenty for any year, even years that are not complete dumpster fires.


* this item legitimately makes it onto my December to-do list every single year. Have I actually done this? No, I have not.

**so, I guess my bargaining with myself did work, because I did manage to sneak a 4th thing into the mix!

What a difference a year makes

This morning, I took a deep dive into my research notebooks from last fall to the present. I am trying to finish up a particular article by the end of the year, and wanted to remind myself where I left off with it at the end of the summer. (I worked with students this past term and prioritized projects that involved them over this particular paper.) I knew I’d started brainstorming and sketching out ideas sometime last fall, but I couldn’t remember exactly when.

I found those notes….and a whole set of notes on a whole bunch of other projects / articles in progress that I’d completely forgotten about.

My first reaction, of course, was to beat myself up over my lack of productivity and what I took as proof of what a lousy hack of a researcher I am. Because of course I did. With the exception of our 2019 CHI Work in Progress poster, I’ve been in a publishing slump for a few years. Which, in my better moments, I recognize as part of the normal cycle of research. More often, though, this becomes proof that I Do Not Know What I Am Doing and that It Was A Mistake Awarding Me Tenure and, my personal favorite, I Am A Deadbeat Professor.

(My head is a fun place, sometimes.)

But then I reflected back to November, 2019, and everything that’s happened since. The mental health crisis we faced with one of the kiddos, which took almost a year to properly get a handle on. The increased struggles helping the neurodiverse kiddo navigate daily life and school — which leveled up once the pandemic hit and have really leveled up this school year. The overly scheduled Winter Term 2020, for which my mantra for survival became “Spring will be more manageable.” (Ha!) And of course, the pandemic, with its ever-present firehose of work that hasn’t let up since March. Not to mention all of the chaos and stress and worry involved in managing the day-to-day, and the news cycle, and the election, and and and…

And that’s when I decided to reframe my thinking:

  • I managed to make forward progress on several different projects in a year filled with chaos.
  • I managed to launch a new line of inquiry.
  • I managed to collaborate with 4 brand-new-to-research students this year.
  • I have several articles in progress that I can, with some work, get out into the world in the next 6 months. Particularly since I am only teaching sections of our capstone course in the Winter.
  • I have a number of projects in progress. If some of them fail, that’s ok, because I have plenty of other ideas to pursue.
  • I could make more progress on some of these projects if I bring in collaborators, so perhaps that should be a goal in 2021.

As we head into the final part of 2020 (motto: The Gift That Keeps On Giving), let’s all remember to extend ourselves some grace, maybe even the same grace we regularly extend others. We’re all doing the best we can, with the resources and energy we have available. And that is more than enough.

#ScholarStrike

On Tuesday and Wednesday, September 8-9, I am participating in the Scholar Strike for Racial Justice, a mass action of higher education professionals protesting racist policing, state violence against communities of color, mass incarceration and other manifestations of racism.

Due to my participation, my email response may be delayed. Let me encourage you to follow #SCHOLARSTRIKE on Facebook, Instagram, and Twitter, and to engage with the teach-in occurring on the Scholar Strike YouTube channel. I’d also encourage you to read any of the following books, which delve more deeply into the intersection of race and technology:

Algorithms of Oppression: How Search Engines Reinforce Racism, by Safiya Umoja Noble

Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor, by Virginia Eubanks

Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech, by Sara Wachter-Boettcher

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, by Cathy O’Neil

Scholar Strike is meant to disrupt the everyday routines of academia, and to show solidarity with other workers striking for Black lives, particularly the athletes of the WNBA and NBA. On these two days, in a variety of ways, academic professionals are disrupting the status quo and refusing to stand by while racism and violence devastate Black Americans’ lives and communities.

— my email away message

When I originally conceived my #ScholarStrike participation plans, I expected I’d write a blog post in the “teach-in” spirit of the strike, discussing some of the many ways that white supremacy and technology intersect. Instead, I found myself tweeting a bit about educational disparities (as this video highlights, the connections between racist policing and the education system are iron-clad) and amplifying some of the excellent content on the ScholarStrike YouTube channel.

Here are some other ways I’m participating in this two-day action:

  • Donating 2 days’ (gross) pay to organizations working on hunger, educational justice, and development. “Hunger” might seem out of place on this list, but I wanted to do something to address immediate and basic needs in my local community as well as address bigger, systemic changes.
  • Finishing a working draft of our department’s first broadening participation in computing plan. A colleague and I participated in a workshop this summer to help departments develop plans to broaden participation in computing in meaningful ways. As a result of some sustained efforts yesterday, we now have a concrete set of steps to move forward. Our goal is to address a particular course that sets the tone for the major and where we lose students from particular demographics disproportionately. We’ve got a ton of work to do to bring our plans to fruition, but at the same time I’m eager to put this plan into action!
  • Centering anti-racism in my STEM leadership. As STEM Director, I set the agenda and focus for our STEM Board meetings and activities for the year. I’m eager to move some great conversations we started last year into tangible actions to broaden participation in STEM at Carleton, and more importantly to ensure STEM at Carleton is a space where all can thrive. I’ll spend a good chunk of today finalizing our agenda for the year — which is really the culmination of a lot of hard work that the STEM Program Manager and I put in this summer to get to this point.

Admittedly those last two activities don’t exactly scream “strike”. But I see them as foundational to the work I want to do this year to change the systems I participate in. And as someone with some power and privilege in those spaces, I want to be clear in how my intentions manifest into actions. In that respect, the activities fit the spirit of the strike, and thus I feel comfortable spending time doing that necessary work.

If you haven’t been participating in, or paying attention to, the #ScholarStrike, I encourage you to do so today. Take advantage of the incredible educational videos made by academics. Follow along on your favorite (or most tolerated) social media platform. Follow the links, watch, listen, and learn. Educate yourself on what’s happening in your local communities around race and policing. And then, find a way to take action, to put into practice what you’ve learned, to make our systems and our spaces anti-racist.

Planning for Fall (a story in pictures)

I’m overwhelmed.

Fall term classes don’t start for another week and a half, and I’m already at the stage where I’m semi-catatonic by the end of the work day. (Yesterday I gave up and took a nap. At 4:30pm.)

This time of year is usually full to the brim anyway — the mad rush to finalize the syllabi, helping advisees navigate changes in their schedules, setting priorities for the year for STEM at Carleton, meetings meetings meetings (and, hey, more meetings!), … the list goes on. This year, it’s that … times a thousand.

Yesterday as I navigated through various windows and apps on my laptop, I marveled at the juxtaposition between my “normal” workflow of preparing for the term and the additional preparations for a pandemic term. At the end of the day, I took some screenshots of some of the apps and sites I used throughout the day, to put together a mini photo-essay highlighting a “day in the life” of a professor preparing for the upcoming COVID-influenced term.

  • Checklist containing items to complete for preparing a course for the start of the term.
  • Mind map of a computer networks course.
  • Backward design worksheet with learning outcomes and evidence.
  • Class meeting times schedule.
  • Moodle landing page with course listings
  • Screen shot of part of the faculty COVID-19 FAQs
  • Teaching toolkit, pandemic edition: iPad, pencil, headset

(I did, however, spare you the screenshots of the multiple Zoom meetings I’ve participated in over the past few days. And of the firehose of emails. And of the various ways my family interrupted me mid-meetings. You’re welcome.)

Looking at these pictures, it strikes me that even though everything seems completely out of whack, the basic things I do to prepare for a term — wrangle with Moodle, finalize my learning outcomes, assemble my teaching toolkit — remain largely unchanged. The details may look different, but the broader strokes resemble what used to pass for normal. And that provides me with a teeny bit of comfort as I head into what promises to be a strange and stressful term.

How does preparing for the upcoming term/semester look for you? What new things are juxtaposed into your normal workflow?

Breaks are not optional

View of trees and a distant river from a ridge in Whitewater State Park, MN
View from atop a ridge on one of the hikes we did last week, in Whitewater State Park in Minnesota.

This summer has been difficult for many reasons, not least because, unlike most summers, there hasn’t been any sort of let-up in the workload. We rushed right from the end of the academic year into discussions and debates about Fall Term, and from there into preparing for a very complicated Fall Term. Many of us juggled this with mentoring research students and trying to get some semblance of scholarship done. We count on the firehose of work in early June to subside to a more steady gardenhose flow from mid-June through at least the first half of August, but this summer has been one long firehose of work.

Not to mention, of course, the underlying mid- to high-level stress of living through a pandemic.

Taking a true and sometimes extended break from work at some point in the summer is non-negotiable for me. I’ve learned over the years that I need to take at least a week, if not longer, to completely unplug and detach from work (and from social media). Otherwise, I enter the new academic year burned out, and that almost always spells disaster by Spring Term, if not sooner.

This break often entails packing up the car and the family and hitting the road for some quality time with Mother Nature. My family loves camping, and loves national and state parks, and it’s a rare year when our break doesn’t feature some or all of these. We usually take off in August, although last year’s epic romp through Colorado and parts of Utah happened in June, just to mix things up a bit.

We’d just started kicking around ideas for this summer’s adventure when everything shut down. As the spring wore on, we resigned ourselves to the fact that there would be no epic road trip this summer, and likely no getaway at all. I started thinking about what a “break” might look like in a summer with no child care and no place to go.

Fortunately, things are under enough control in Minnesota that in-state camping seemed to be a relatively safe option for a getaway. So we scaled back our epic road trip aspirations, picked a state park we’ve been wanting to visit, and made camping reservations.

My partner and I suffer from Cram-Vacation-Too-Full-Itis, as our kids like to point out. So this year, we worked hard to unschedule our trip. (And yes, I realize how ridiculous that sentence sounds.) Only one park over the 4 days, not several. Only one campsite, no moving around from park to park. We hiked in the (late) morning, once everyone was up and fed; took a long break back at the campsite for lunch, board games, and naps/reading; went swimming/fishing in the late afternoon; and relaxed around the fire in the evening. I started and got through half of a novel that’s been on my reading list for a while (which I’ve since finished) and worked on two small crocheting projects.

Yes, there were a couple of times where I found myself thinking, “should we get out and explore the area some more?” I am, after all, a textbook Type A personality. But for the most part I relaxed into the un-schedule. And I made sure to take the rest of the week off once we got back, instead of diving back into work.

I started this week fresh and able to work on some longer-term vision-y stuff I was blocked on pre-trip. Things still feel hard, but they feel more manageable. And that’s why taking a break is so valuable, and so non-negotiable for me.

Have you been able to take a break this summer? How are you rejuvenating yourself this weird summer?

Rethinking assessment

When we moved online and moved to mandatory S/Cr/NC grading (basically, Carleton’s version of pass-fail) this spring, I vastly simplified the way I graded projects and other major course assessments. Here’s what I wrote in my syllabus:

Each assessment that you hand in will be evaluated against a checklist related to one or more of the course learning objectives. I will rank each learning objective, and the overall submission, according to a three-point scale: Does not meet expectations; Meets expectations; Exceeds expectations. If an assignment does not meet expectations overall, you (and your team, where applicable) will have the opportunity to revise and resubmit it to be re-evaluated.

… You will earn an S in the course if at least 70% of your evaluated work (after revision, if applicable) is marked as “Meets expectations” or “Exceeds expectations.” You will earn a Cr in the course if between 60 and 70% of your evaluated work (after revision, if applicable) is marked as “Meets expectations.” You will earn an NC in the course if less than 60% of your evaluated work is marked as “Meets expectations.”

CS 257, Spring 2020 syllabus

I’d heard the term “specifications grading” and I knew that what I’d be doing in the spring was in the general spirit of specifications grading (if you squint hard enough). And it worked surprisingly well. Students knew exactly what they had to do to earn a particular grade in the course, and on individual assignments thanks to targeted rubrics. Allowing revision on any major assessment meant that students could recover from the inevitable hiccups during a pandemic term (and a term marked by grief, loss, and protests over George Floyd’s murder). And at the end of the term, when some students could just not give any more to their studies due to all that was happening around them, the system extended some much-needed grace — if they’d already met the threshold for an S, they could bow out or step back from the final assessment, assuming their teams were on board with their decisions.

I wondered: what would it take to do something like this during a graded academic term? I wanted some more guidance.

And so, I did what I always do when I want to learn more about something: I hit the books. Two books, in particular: Specifications Grading: Restoring Rigor, Motivating Students, and Saving Faculty Time, by Linda B. Nilson (2014); and Grading for Equity: What It Is, Why It Matters, and How It Can Transform Schools and Classrooms, by Joe Feldman (2018).

Both books tackle the same general problem: grades and grading are imperfect, biased, and measure lots of things other than how well students achieved learning outcomes. Nilson solves the problem with a straightforward up-or-out approach: work is either acceptable, or it’s not. Feldman’s solution is a bit more nuanced: work is somewhere on a (short!) continuum between “insufficient evidence” and “exceeds learning targets”, but nothing resembling a formative assessment and/or “life skills” gets a grade.

Specifications Grading is faculty-centered in its approach, at its heart. A key goal of specifications grading is to save faculty time while still maintaining high quality feedback to students. The specifications grading approach is two pronged. First, all individual assessment grades are pass-fail. The assignment either meets the standard of acceptability, or it does not. No partial credit, no wrangling over how many points something is worth. The standards of acceptability are spelled out in a detailed rubric, or checklist, so that students know exactly what constitutes an acceptable submission. Second, a student achieves a particular course grade (A, B, etc) by completing either a specified set of activities (“bundles” or “modules”), or by demonstrating more advanced mastery of the course learning outcomes (“jumping higher hurdles”). The bundles/modules/hurdles are spelled out in detail in the syllabus, so that it’s crystal clear how a student earns a particular grade. In my spring course, for example, the bundles were simply percentages of course assessments acceptably completed. In a graded course, a bundle is often more complex: extra assessments, for instance, or more challenging assignments. While setting up the bundles/modules/hurdles seems like a really time consuming process, it is front-loaded, done before the course starts, so that the grading itself during the term is more streamlined. Basically, the instructor decides what constitutes meeting learning outcomes, and constructs the assessments and bundles/modules/hurdles accordingly. At the end of the course, then, the grade more closely indicates the level of mastery of course learning outcomes than a traditional partial-credit focused grade.

Grading for Equity is, I would say, more student focused. (And more K-12 focused, although I certainly found enough in the book worthwhile to consider for the college context.) Grading for equity is based on three pillars. First, grades should accurately reflect student achievement towards learning outcomes. This means grades should not include things like formative assessments (homework, in-class activities), extra credit, behavior, or “soft skills” — they should only reflect the results of summative assessments, and only the most recent result of a summative assessment. Feldman also cautions against using the typical 100-point scale, which is skewed towards failure, in favor of more compact scales (a 4 point scale, for instance, or a minimum score). Second, grades should be bias-resistant. They should not reflect a teacher’s impression of student behavior, which is flawed for many reasons, nor reflect a student’s life circumstances (for instance, their ability to complete homework outside of school hours). Third, grades should be motivational. It should be transparent to students what counts as mastery of a learning objective and how to achieve a particular grade. Formative feedback should not penalize mistakes, because this promotes a fixed mindset rather than a growth mindset. For the former, Feldman is a fan of detailed rubrics and the four-point scale (something like “Exceeds expectations”, “Meets expectations”, “Partially meets expectations”, “Insufficient evidence”).

Both books agree that students learn at different rates, and any summative assessment should take this into account. Both systems, thus, allow for retakes and redos. Specifications grading puts some limits around redos to make things easier on the professor, recommending some kind of “token” system where students have some limited number of redos/late passes over the course of the term/semester. Grading for equity favors as many retakes (up to the end of the term/semester) as a student needs or wants in order for them to meet learning targets. (Theoretically, anyway; the book acknowledges that there could be a snowball effect particularly when later work depends on earlier work, and suggests that time limits on retakes would be appropriate in this context.) Grading for equity argues that later assessments should replace earlier grades rather than, say, averaging them, It gets into the weeds a bit on the freedom of faculty to count anything that demonstrates a learning objective as an appropriate assessment of that objective, including things like discussions in office hours. I get the spirit of this, but it seems like something like this would be ripe for bias.

So, how am I using what I learned from these books about assessment as I plan my fall course, a CS elective? And what am I struggling with?

Plan: Retain the meets/exceeds expectations scales, with minor changes. I really liked the ease and clarity of the three-point scale in the spring. Grading for equity makes a compelling argument for the inclusion of a “not yet demonstrated” category, allowing teachers to differentiate between “handed in and not sufficient” and “not handed in”. So I may move to a 4-point scale for some assessments (“insufficient evidence”, “partially meets expectations”, “meets expectations”, “exceeds expectations”). Roughly, “partially meets” in my head equates to C-level work, “meets” to B-level work, and “exceeds” to A-level work. Moodle likes to convert everything to percentages, which is not as useful for this type of grading. I need to figure out how to hack Moodle to show students something closer to this scale rather than “you’ve met expectations so you’ve earned 50% on this assignment”.

Plan: Allow for revisions and be flexible with deadlines. We’re still in the midst of a pandemic. We still live in a white supremacist society. And the 2020 elections….well, need I say more? Fall will be tough emotionally and mentally for many of us. Extending flexibility and grace to my students, being willing to meet them where they are, is the least I can do. And while I’ve used revisions on exams previously with pretty good results, I’m eager to extend that to all major assessments, as I did in the spring, with later grades replacing earlier grades. I still need to figure out what revision looks like for my “deconstructed exam questions”, though.

Struggle: Not grading homework. While specifications grading allows forms of preparing for class to count towards a bundle, grading for equity adamantly opposes the idea — even just giving points for students completing the homework before class, regardless of correctness. (Which has been my policy for years.) Again, the argument is compelling — homework is formative, not summative; grading homework for correctness penalizes making mistakes in the learning process; there are many good reasons students can’t complete homework outside of class. But I still want students to take preparing for class seriously, so that once we get into class we can be an effective learning community, ready to engage with ideas. Feldman indicates late in the book that keeping track of homework submission can be valuable in pointing out effective and less effective learning strategies to students, and that awarding a small percentage of the overall grade to homework is not horrible. So I think I will compromise and continue to grade homework for completion only, but reduce the percentage it’s worth (maybe from 10% to 5%?).

Struggle: Bundles and modules and objectives, oh my! I’ll admit that I had to put specifications grading down for a bit and come back to it later once it got into specific examples of bundles. Ditto when I tried to wrap my head around deconstructing my usual assessments around course learning outcomes, as grading for equity describes. I got stuck on how I’d translate this to my elective. On reflection, it will take a lot of up front work, but the increased transparency will be worth it. I plan to use the “higher hurdles” approach from specifications grading, measuring hurdle height with the 4 point scale from grading for equity. I’m still not sure if I can achieve complete separation of learning objectives when an assessment covers several of them, grade book wise, so I may have to let that go for now and try separating those out more cleanly in the grade book in a future term.

There are other struggles, of course — grading for equity has me puzzling over my approach to grading group work, for instance — but these are the key ones on my mind as I piece my course together. I’m eager to continue my experiments with assessment and curious to apply what I’ve learned from my spring experiences and from these two books.

Small gratitudes, summer edition

Any hopes I had for a bit of a respite between the end of my students’ summer of research and going full steam ahead with planning for fall term evaporated more quickly than you can say “can you make room on your calendar for….”. This week features All Of The Meetings, anti-racism education, two pedagogical webinars/conversations, and a two day virtual workshop. All good stuff, to be sure, but not at all conducive to catching one’s breath. Not to mention the ever-present and very real stress about what the school year will look like for the kiddos — the school district’s decision doesn’t come out until August 14 — and whether we should wait and see what the plan is or just go ahead and enroll them in the district’s online education program.

In the midst of a heavy week in the middle of spring term, I shared some small gratitudes, little things getting me through a difficult time. This week, when life feels overwhelming, I find myself reflecting on the small things that are getting me through the difficult times this summer:

Kayaking. I think in a past life I was a fish. I grew up swimming every day in the summer, and I worked as a lifeguard and swim instructor for half of my teens and 20s. I love everything about the water; it calms and centers me. Bonus for introverts: on a kayak, it’s easy to get far far away from people and close up to birds, muskrats, and turtles.

My blood pressure is dropping just looking at this picture…

Family food traditions. The formerly-7th-grader-now-8th-grader and I continue our Friday morning drive-through coffee tradition, although we have a new rule that she must be up by 9am or it can’t happen. The best weeks are the ones where she plugs in her phone to the car stereo and shares the music she’s listening to with me. I’ve discovered some new music through her! We also started, back in the spring, ordering takeout once a week on Tuesdays to support local restaurants, and we’ve continued it ever since. We take turns, so that everyone gets their favorites once a month.

I still dream about this meal.

Disc golf. The formerly-3rd-grader-now-4th-grader participated in a socially distanced “frisbee camp” one week in July where he learned how to play disc golf and ultimate frisbee. Long story short, we now own 2 sets of discs and we’ve been out a few times on one of the local disc golf courses. Neither of us are particularly good at it, but we have fun and we laugh at each other’s bad throws. And it’s easy enough to get in a quick 9 holes as a mid-afternoon break (well, at least on days that don’t feature wall-to-wall Zoom meetings).

My new home office. Despite the sometimes spotty Wi-Fi (I think we’ve finally pinpointed the cause and have a possible solution), I am loving my new office space. I love that I’m surrounded by my favorite color on the walls all day long. I love the view of the garden on the side of the house from my window. I love being close enough to the kitchen that I can get a quick coffee refill in the middle of a Zoom meeting. (This proximity also makes getting snacks more convenient, which is both a blessing and a curse.) And surprisingly, I love that I’m closer to what’s happening in the rest of the house (and have the option to close myself off from the chaos, too).

Not pictured: the crochet project in progress, which I now also keep in the office to work on during meetings where I’m listening and not taking notes.

What are you grateful for this summer?

My checklist for wrapping up summer research with students

This week marks the end of my summer research collaboration with my students. I usually wrap up research by the end of July/first week of August so that I can spend some quality time with my family and particularly the kiddos before school starts back up. Granted, this is more of a perk when we haven’t been cooped up together for months….

I digress.

The last research week is always hectic. No matter how on track we’ve been all summer, there’s always a lot to do to wrap things up. Finish the analyses. Make sure all the code is in the repo. Make sure all of the code is commented. Get a rough draft of the eventual conference paper to some stage of “completion”. And so on. A million little details, some of which inevitably slip through the cracks.

Every summer I tell myself I will make a checklist of what needs to be done. Every summer I fail to do so. Maybe it’s because I see my students every day, or almost every day, so part of me assumes that it will come up during a meeting, or that I’ll pop into the lab and remind them to do whatever just popped into my head.

But this summer, we’re all online, in 3 different cities in 2 different time zones. I’ve gotten in the habit of putting more things in writing, more formally. More lists, more systems in place. More structure.

Turns out, this put me in the perfect mindset to finally write that checklist.

Here’s my checklist for this summer. I suspect that in the future, particular details might change based on the nature of the project, but that the overall categories and most of the items will largely stay the same, or at least very similar.

I. Complete project writeup. (I always have students write something up about the project, no matter where we ended up. I think it’s important for students to get some practice writing for a technical audience.)

  •  Write up the methodology for each of the analyses you completed.
  •  Write up the results for each of the analyses you completed. Include graphs/tables.
  •  Write up the takeaway points for each of the analyses you completed. What did you learn? What do you think the results indicate? What are the next steps that should be done?

II. Check in and clean up all code. (I’ve learned the hard way over the years that students need to be reminded of this, and also of specifically what I mean by “clean up”.)

  •  Make sure all code is commented. Think of You, Six Months From Now. What does You, Six Months From Now need to know/remember about what’s in this code?
  •  Write a TODO list for each of the (major) scripts you wrote. (You can put this at the top of the file, in the comments.) What’s not working that needs to be fixed? What’s working imperfectly that needs to be fixed? What are the things you hoped to get to, but ran out of time?
  •  Write up how to execute each script. (You can put this at the top of the file, in the comments.) What data files does this operate on? Where are they specified in the code? Are there command line arguments? Any other assumptions that you made that others should know when running the script?

III. Write up onboarding docs and next steps for next set of students. (This is still a work in progress. Students, understandably, find it difficult to anticipate what others will struggle with, and invariably forget what they struggled with early on in the project.)

  •  Make sure all README files are up to date.
  •  Write up a “Start Here” document that describes what students starting on the project should know about the project, the code, and the data. (I ended up outlining this document for them, because they were really struggling with what to include.)
  •  Make sure all metadata documents (on all datasets) are updated, correct, and easy to find.

IV. Write up a short reflection for me about your experience. (You can defer this until next week if you’re overwhelmed!) Email is fine for this. (I don’t always remember to ask for this, and I always regret when I forget! I learn so much from these reflections.)

  •  What were you hoping to get out of this experience?
  •  How much of that do you feel you accomplished this summer?
  •  What, if anything, surprised you about your experience?
  •  What were you hoping to accomplish/get out of this experience that you did not?
  •  What work are you most proud of, and why?
  •  When/if I write letters of recommendation for you, what parts of your contributions to this project would you like me to emphasize?
  •  What advice would you give to future students on this project?
  •  What advice would you give to me to help me better mentor future students on this project?

V. Celebrate a job well done! (Admittedly, this is trickier to do in the time of Covid. Usually I take them out for lunch, but that doesn’t work when we’re all in different locales. I will likely send my students a little gift of appreciation and a note, but I’m still trying to figure out what to send.)


Do you use a checklist with your research students to keep track of end-of-the-project todos, or at other stages of your research project? I’d love to hear your experiences.