The Take-Home Test

Aug 05, 2019

work interview

Recently I was approached by an internal recruiter looking for someone who dabbles in Cloud, software development, infrastructure and security (so-called "Cloud DevSecOps"). The initial conversations went extremely well and I was moved onto the next stage where I was told my AWS skills would be "assessed."

The Setup

To my surprise, the assessment meant that I would be given a take-home test, which is an odd way to test someone's Cloud security skillset. After all, what could they do, grant me access to their AWS account and start racking up charges as I harden the system? All I was given in the preparatory materials was the idea that I would be integrating a third-party SSO provider into a Web application. Okay, this sounds oddly vague but, in a way, it may have a bit to do with Cloud Security. I set aside what limited free time I have these days and submitted a form that informs me that the test will begin and a timer will start.

Oh great, now I'm under time pressure and I don't yet know what I'm even building. Through some automated process, I receive the test package in a Github repository. I take 30 minutes to read and understand the instructions and project specification. It is there I find out that I have four hours (?!?!) to create a Web application, from scratch, that integrates an SSO provider for login which then provisions an AWS IAM user and then allows that person to generate AWS IAM credentials for CLI access. Plus audit logging and Web UI.

In 4 hours.

They were clear that any work performed after the 4 hour mark would be nice, but would not count toward my test score. Now I'll mention that when I was first notified about this test the recruiter said I could take as much time as I wanted, but the instructions were very clear about the merits of finishing within 4 hours (bonus points are awarded if I finish early).

The Work

I'm not going to bore you with what I did exactly, but here's a rundown of the things I worked on in just 4 hours, in the correct order. I wrote it in Go since that is what my development environment was setup for (they said I could use any language but I know they wanted Python since they mentioned that the code should adhere to PEP8 :facepalm:).

1. Create the server CLI
2. Create graceful shutdown
3. Create the routes and handlers
4. Create the Makefile for automating tedious processes (build, dist, test, formatting)
5. Create some test files (but didn't have time to populate them with testing code)
6. Create the front-end UI
7. Add handler code for redirecting a user to the SSO provider
8. Add handler code for generating new IAM credentials

I know from experience that what they really wanted was for me to use Rails or Django to speed up my work. That will only show I know how to use a framework and not how to code against the underlying HTTP spec, so what good would that do me or them?

The Result

Rather than draw out the conclusion, I'll simply state that I did not complete the test. It compiles and a server serves a page that redirects to the SSO provider but, functionally, nothing else exists. I added the IAM provisioning and other routes, but I ran out of time wiring everything up and writing tests. I was able to create some quick automation for compiling the final code, producing an artifact for Linux, MacOS, and Windows, and format the code according to common style guidelines.

Overall I'm not happy submitting an incomplete test, but I am happy with how far I got after four hours. Breaking down my time, I spent it fairly wisely:

• 30 min: read and understand instructions
• 30 min: design workflow (i.e. sequence diagrams) and structure time
• 2 hours 45 min: write code
• 15 min: polish and clean up

I stuck true to how I write software, and it remains to be seen if my principles align with the company's principles. I made sure I looked at the entire SDLC when creating full-fledged Web application and, more importantly for this position, incorporated security into each phase. But this result doesn't change my view that this type of skills assessment doesn't work.

Why This Doesn't Work

I'll get straight to the point: this type of skills assessment - that is, timed take-home test - is a poor gauge of a person's ability or skillset. Worse, it reflects poorly on the company, the team, and the individuals that believe in these types of assessments. I breakdown my reasoning into the following components.

It's not an objectively-scored test.

Talk with any formally-trained teacher and they will tell you that a test can only be scored (and, ultimately, useful) if the same test score can be reached by any examiner. If a test relies or subjectivity for scoring, then it's not a valid test. This is why tests are difficult to create. This test, in particular, was supposedly being scored on things like readability of the code, which is both subjective and has no unit of measure.

It had nothing to do with Cloud Security.

Building a Web application to interact with an SSO provider and provision IAM resources is just a business process. It admittedly sounds like Cloud Security because SSO and IAM services are being used, but replace them with any other service and it doesn't change the fact that it has nothing to do with Cloud Security. So the test isn't even an effective measure of the main skill for the job. Cloud Security is concerned with the security of data flowing in the Cloud, as well as storage. This test is connecting two services together, which is akin to software plumbing. Ironically, Cloud Security is the stuff around the Web application (third-party libraries, transport security, system security, network security, etc.).

Forced focus on time versus quality.

Unless you're in a competition, there is no excuse for prioritizing coding time over quality. After all, software developers are artisans, a mix of art and science; we are not athletes. No company would place code written under time pressure into production; it is one of the motivating factors behind secure development controls in many regulatory frameworks. These controls ensure due care is taken to safeguard data, they never allow racing of code writing.

Mushroom Programming.

This is from the adage that mushrooms are kept in the dark and thrown work (manure) at them in some insane sense of sustenance. Interviewing and hiring is a difficult process where an employer needs to know in the short-term if they can work with the candidate for the long-term. Then why, when in the courting phase of the relationship, do companies throw work at the candidate and then ignore them until its done? It's jarring for the candidate to be treated like just a resource and not a human being. And this leads to…

Moral Hazard

A word from the financial world, moral hazard is the concept where the risk of an action is placed upon someone else, leading the initiator to take more risks. By providing a take-home test, the employer doesn't feel the weight of the interview process. The extra 4 hours is borne by the candidate. If a company proposed spending 4 hours of employee time per candidate, it would reconsider its worth, but by placing that cost on the candidate it bears no weight to its actions. Put simply, the company is saying to the candidate, "my time is valuable, your's is not."

No Technical Guidance

Another jarring aspect of this particular test was that, other than the instructions file, literally no other files were provided. No starter application. No template.

Nothing.

I began the test with what amounts to a blank canvas. In some ways this can be freeing, but not under time pressure and not when a complete Web application is being requested. It may help if the time constraint was removed, but then now the company could be accused of abusing people's time ¹. Without some technical guidance, it is difficult to know how the test is scored or what the reviewers are looking for. The power is unsettingly placed further on their side instead of being shared equally.

Soup-to-Nuts in 4 Hours

Related to the previous concern, developers rarely start from scratch and even if they do, nobody expects them to produce a full Web application in 4 hours. If you were applying to be a chef, the assignment would not be "produce a 10-course presidential dinner in 4 hours," it would be either (a) plan out that type of dinner in 4 hours or (b) produce a single dish fit for a presidential dinner in 4 hours. There are myriad activities that happen outside of producing things (cooking, writing code) that are required if you are producing a complete packaged product for someone (e.g. planning, testing, presentation, etc.).

Inconsistent Instructions

If one chooses to place candidates under time pressure, it is vitally important that all instructions are consistent and understandable. The candidate is being forced to rush and any inconsistencies will rob time from performing the actual test. In this case, there were two major inconsistencies:

1. I had either unlimited time or 4 hours to finish the test; and
2. I could use any language I wanted but the code had to satisfy PEP8 style guidelines (a style checker exclusively for Python code).

Confusion wastes time and sows the seeds of doubt. From a security standpoint, these are things that one doesn't want in any process because it leads to errors.

Insecure Specifications

This is where I was starting to wonder if the company hid a real test (a security assessment) inside a test. In two instances I wondered why proper application security and product security weren't being followed.

The former because I was asked to download and run an unofficial (and untrusted to me) docker image that mocked AWS calls. This docker image required me to provide it with my .aws/config file… you know, the one that has the credentials to my personal and corporate AWS accounts? So I had to shuffle files around my system in order to provide it with a dummy configuration file.

The latter because allowing users to generate credentials and download them as a plaintext TXT file opens up the threat that the TXT file will reside unsecured in their Downloads folder (something I have witnessed multiple times, even in a DFIR context). As a security person applying for a security position, these issues cause me to pause and do risk analysis, robbing me of more test time.

Real Work

It's well-known that some unscrupulous companies use hiring and take-home assignments to gain free work (or spec work) from eager candidates. I have been a victim of this abuse in the past.

So my spidey-senses tingle when I see a test that looks like it could be a real application. And while I don't like creating toy applications for assignments either, there is a happy middleground somewhere. Tests in this middleground are incredibly difficult to produce, but opting for an easier path opens up the risk for candidates feeling manipulated and abused. This company chose the path of abuse.

That Not-So-Fresh Feeling

I'll admit that I didn't like doing a take-home test at all: before the test, once the test started, as well as afterward. I was left feeling alone, ignored, and inferior. It's not a great way to start a relationship. Just imagine if, two dates into a new relationship, your partner sends you off to get a thorough medical examination by yourself to see if you are worthy to continue dating.

While testing of any sort is part of due diligence, the manner in which one treats the person being tested matters just as much, since you will be with that person for a long time and people's memories surrounding abuse and mistreatment are surprisingly long.

SDLC? What SDLC?

I alluded to this earlier, but a test should not focus solely on the implementation phase (writing code) of building products. There are many other tasks in a complete SDLC that should not be forgotten by a candidate, because one does not write code in a vaccuum nor does one throw it over the fence for someone else to worry about. In fact, it is exactly this critical thinking I use to determine if someone is eligible for a "Senior" prefix in their job title when hiring. Only focusing on the implementation phase of SDLC tells me about where a company or team resides in software development maturity.

No Testing Criteria Given

In addition to having an objective scoring model, a test also needs to provide the candidate with a sense of what is being tested. If the candidate is asked to produce a complete Web application, in 4 hours no less, are you testing for the candidate's ability to:

• write code?
• read code?
• work with others?
• complete end-to-end delivery of a product?
• perform testing
• work in a fast-paced environment?
• work with chaotic, vague requirements?
• question authority?
• be a soldier and follow orders?

Combined with the score, the completed test will tell the examiner, hiring manager, auditor, company, or lawyer that "We are confident this candidate knows X." Allowing the candidate to know the testing criteria allows them to focus on flexing the right muscle, instead of half-flexing all the muscles at once.

Unrealistic Working Conditions

During the hiring process, one of the things that is crucial to determine is whether the candidate can work within the company's working conditions. Thus, any assignments, interviews, or tests must reflect real working conditions the candidate may face. The take-home test I completed could not reflect the company's working conditions or else that would mean they often provide vague requirements and then send you off alone to finish everything until you magically emerge with a completed product. No company operates like this nor would want to, especially if there is regulatory compliance to satisfy.

Allez Cuisine!

One of the annoying aspects of this test was that I was unprepared for the parameters of the test because they weren't announced until the timer started. In a way, this is akin to keeping contestants of a game show in the dark (often blindfolded) until the show producers do a reveal of the secret ingredient. But this isn't a game show, it's real life and your candidates are not contestants, they are human beings that deserve empathy and respect. I have been in this industry for over 20 years and I know that empathy is in short supply with many developers but, my gawd, a company that has empathy as a core value should at least show some of it.

Read Lots, Write Little

Any assignment, test or whiteboard exercise that focuses on writing code to assess a developer's skill is missing (my estimate) about 90% of what a software developer does. For the uninformed, the majority of a software developer's job is to read and understand application code. For junior developers starting their career, they will write more code than a senior or lead developer, but they will still maintain the axiom that developers read more than they write. Yet a take-home test that requires the candidate to start from scratch is overplaying the importance of writing code.

Assessing Minor Job Responsibilities

Surprisingly, the test I was given had little to do with my actual role. In other words, out of all the responsibilities in the job description, writing full Web applications wasn't listed, thought writing code was listed. But then what about the other responsibilities? A company must ensure most or all of the job requirements are assessed if due diligence is to be considered complete.

Previous Experience Means Nothing

Every employer has one thing in common: their hiring practice assumes that the candidate knows absolutely nothing about how to do the job for which they are applying. After 22 years of employment, several public work samples available, scads of references, and a public blog with 100s of posts spanning almost 10 years, I still get asked to prove that I know how to write software. No degree or certification or work experience matters, companies display a high level of contempt for candidates. This doesn't come from a place of ego, but rather from frustration as nothing has proven anything to employers in 22 years. It is my belief that this contemptuous attitude is the root cause for the shockingly high levels of imposter syndrome, anxiety and depression in our industry.

So…

After reading all of that you may walk away with the belief that I'm angry, difficult to work with, or don't understand something critical to why your company needs to operate this way. You'd be wrong in this regard. I haven't succeeded in the tech industry for this long, through various positions, with my eyes closed or only making enemies. I'm not even that angry, but instead more saddened that this is the state of hiring in our industry. If more people stood up, said "no" to interviewing practices like take-home tests, and banded together in solidarity, we might be able to make hiring fun. Candidates would feel like equal participants and there would be more empathy and respect flowing ².

A Better Approach

"So, smart guy, what would you do?" I hear you say. Here's what I have done in the past and would plan to do in just about every situation.

Talk.

Seriously. The entire test would be replaced with several discussion spanning 4-5 hours in total (about 1 hour per session). I want to feel the same time burden as the candidate. I want the candidate to understand that, in my theoretical company, we all share burdens equally, we never leave people alone (as in, lonely), and that their time is just as important as ours.

The topic of each discussion would be prepared in advance but left free-flowing to let the candidate lead. I look for confidence with humility, and the ability to take initiative in the absence of direction. The topics would be:

• Provide a work sample of your code. It could be previously written by you (verifiable) or written last night. We review your intentions, motivations, style, approach, future improvements, etc.
• Review a sample of code provided by me. This assesses your code-reading and peer-reviewing skills. You talk about what it does, note any bugs, and how you would improve it.
• I present a situation for a new service and we (mostly you) design it from infrastructure to UI. I look for secure infrastructure, design faults, and ingenuity.
• In a conversational tone (as opposed to interrogative), I ask about the candidate's experience with various technologies, likes and dislikes of everything from infrastructure to security to programming languages. I'm looking for breadth and depth of knowledge as well as curiosity, but I also want to get to know you. We don't need to be friends, but we do need to converse professionally.
• I ask the candidate to teach me something (pertaining to their role) that I may not be aware of (it doesn't matter if I am) and that I may find interesting (it doesn't matter if I don't).

And that's it. The testing is baked into the interviews and conversations. Everything is kept casual. In fact, if you look closely, these topics mimic the majority of a candidate's work day: teaming up to solve problems or design new things, reading other people's code, writing some code, and learning and teaching new topics.

Anyone reading this that interviews candidates, please consider dropping take-home tests and adopting my approach to skills assessment. Tweak it to your liking but remember that candidates are people too and interviewing is not a game show.

Footnotes