How to QA

2017-07-03 10:37:02 UTC

In my day job, I’m a QA (quality assurance) engineer for a Large Software Company. Today I’ll tell you how to prepare yourself to do a job like mine.

Fair warning: “QA” is a big tent; the details of a QA job vary widely, and your QA job may be very different from mine.

File bugs

(Ira Glass voice) File a lot of bugs. File a huge volume of bugs.

The ultimate responsibility of a QA team overall, and of QA engineers individually, is to produce actionable bug reports that can and do get fixed. The details vary widely, but everything comes back to that.

A good bug report says:

  • what you did
  • what you expected to happen
  • what actually happened, and how that differed

That’s the minimum, actually. That’s the least that’s needed for a bug report to be actionable.

Ideally, a bug report should also include needed diagnostic info (such as a sysdiagnose on Apple platforms), screenshots/video if applicable—as much info as possible, at least at first, to be sorted through for the gems of info that illuminate the actual problem. As a QA engineer, providing as much of this info you can gather is Actually Your Job.

Ah, but what to file? If you’re QAing something manually, you’re looking for:

  • any sort of friction (possible interface design issues)
  • any sort of fault (crashes, hangs, data loss)
  • anything in between (anything that did not do what you expected it to do)
  • anything you don’t know what you expect it to do (can be a design/empathy-for-the-user issue)

Also test anything that had been broken before. Look for regressions (previous problem returned) or new problems (“OK, you fixed X, but now it has problem Y”).

Practice, practice, practice

For filing formal bug reports, two good ways are filing Apple developer bugs and filing bugs with open-source projects.

A few caveats re open-source:

  • Many open-source projects need fixes more than they need more bug reports, so don’t be surprised if folks don’t rush to thank you for adding to the pile. Concentrate on high-value bugs (security vulnerabilities, crashes, providing desperately-needed steps to reproduce) rather than nitpicks. (Making this judgment call is itself something that’s valuable to practice.)
  • Search for duplicates first. Apple actually generally likes duplicates and uses them to influence triage and prioritization decisions, but open-source projects may resent the extra scut work of duping bugs together. (Finding existing bug reports is an underrated skill that is also valuable to practice.)
  • I’m a cishet white guy. Your mileage may vary a lot regarding how you and your contributions are received if you are not—anything from well-intentioned over-helpfulness (mansplaining, assumed noviceness) to outright misogyny/racism/anti-trans assholery.

Some open-source projects do code review and/or API review, which can be good practice for spotting designs that invite bugs (random example: “you’re taking a pointer but not the size of the buffer, so the API can’t check that it won’t go out of bounds”).

In that sort of setting, practice asking questions. These could be clarifying (“what does it mean if this returns nil/0?” “This is typed as a String. Are there constants, or what sort of strings should folks pass here?”) or more Socratic (“what does this API do if I pass a pointer to a buffer that’s too small? how does it detect that?”). Even if the immediate response is “that can’t happen” or “don’t do that”, established members of the community/team may back up your question with pressure to resolve the issue (“this should take a length with the buffer, or better yet return a Data”).

Lastly, volunteer for beta tests. I’ve beta-tested Flying Meat’s Acorn once or twice, and I think Panic have also solicited beta testers in the past. Beta-testing is extremely good practice for exploring an in-development product looking for friction and faults and writing up your findings.

As a beta tester, write up everything. Don’t be shy—if it looks wrong, write it up. I have sent in multiple pages of issues and you know what? That makes you worth your weight in gold. (90% of volunteer beta testers are just there for the possibility of a free license. Any developer who’s running a beta test program wants bug reports. They want them urgently—preferably while the product is still in beta.) Even if half the stuff on my list is stuff they meant to do, it’s still worth a second look if a beta tester (or more than one) objects to it, and the other half of the stuff on my list is stuff they may not want to ship with.

The QA mindset

Use your imagination. Try things you wouldn’t ordinarily. Follow the “what does that do?” impulse. Find the cold paths.

Everything that the product engineers thought of is probably really well-tested, and any issues that remain have turned into invisible corners. Your job is to remind them of the invisible corners and tell them about the things they haven’t encountered at all.

Question everything. Is that design optimal? Is the UI copy clear? Could a new user understand both? Was that actually the correct result? Was it the best result?

If you find yourself asking “What does that do?”, that can be a sign of inaccessibility to novices. Inversely, you should learn the product’s domain so you can spot things that don’t make sense in the domain. Try to develop the ability to do both: know the product’s domain and simultaneously spot barriers in the product to users entering that domain.

Be an advocate for users

Make sure the interface is accessible (overly-similar names can cause problems for dyslexic folks; reliance on color or images can cause problems for color-blind or visually-impaired folks).

Familiarize yourself with accessibility tools (on the Mac: Accessibility Inspector, everything in the Accessibility control panel, SimDaltonism) and use them on the product, and file bugs when you get stymied.

Challenge ableist/misogynistic/gender-binarist/racist/otherwise-problematic copy or artwork, using the dual justifications of “this is the right thing to do” and “not doing this is going to repel users/customers and/or cause PR trouble”.

Your QA job may vary

As I mentioned above, “QA” is a big tent. What I’ve described is an important baseline mindset and skillset to a QA engineer, but major portions of an individual QA job could be more sysadmin-oriented (e.g., administering an automated testing or CI fleet) and/or include work with specialized, company-specific internal tools that you’ll have no way to practice the use of outside of that department.

Know at least one scripting language such as Python or Ruby, whether for use in your own QA work (e.g., generating test data, filtering logs) or for working on automation systems (e.g., maintaining Python or JavaScript scripts that drive UI tests). If you’re starting from scratch (and absent any particular job-specific requirements), I’d recommend Python, which you can pick up in a few months, and from which you can learn similar languages like Ruby afterward.

Especially as more automated QA (unit testing, UI automation) becomes more the norm, I’d strongly recommend developing your system administration skills on any platforms you may end up working on. It’s been awhile since I had to start from scratch on this, so I invite you to suggest resources in the comments for learning system administration (on the Mac or otherwise) in 2017.

Leave a Reply

Do not delete the second sentence.