What I learned from my job interview with Facebook

From The Bit Source | By An Anonymous Sys Admin (Unedited)

Did you know that Facebook approximately receives 20,000 job applications a month?  An astounding number!  One would think that the HR department would have their own Hadoop clustered-system just to manage the volume of people that want to work at Facebook.  Pound per pound, it’s one of the top Internet companies with around a thousand full-time employees.

I became a Facebook-member approximately 1.5 years ago, and I have seen the reaches of Facebook expand.  From hosting couchsurfers (www.couchsurfing.org) to traveling around South America – everyone seems to have a Facebook account.  I often ask potential couchsurfing guests to add me as their friend on Facebook.  You could learn a lot from a person’s Facebook profile – what they like to do, people they hang out with, and their general personality.  You can place whatever you want on your profile, such as music you like, your employee, philosophy – you can create whatever story you like of yourself, however your friends keep you true to who you are.  It’s quite difficult to deceive.

As of this writing, a year has passed since I was first laid off (reduction in workforce is the politically correct saying).  For the past year I have travelled,  attended classes at the local community college, consulted on the side, and  I went on job interviews at tech shops that interested me.  I was a Sr. Systems Engineer with 10 years of experience that was picky about where I wanted to work, and I was in no rush in finding a job that I would not enjoy.  I wanted a career, not a job.  Then one day, a message came to me, not by email, but by Facebook message.

It stated, “I found your profile on Linkedin and was wondering if you would be interested in chatting about openings here at Facebook. I have the perfect position for you..now how do you say no to that? We are located in Palo Alto, CA. Let me know!”

I just stared at the message for a while.  I never thought I would relocate myself for a job, however for a company such as Facebook – I considered the migration.  For a computer geek, the Facebook site/system resembled the UNIX system.  And I just love *NIX.  Wall == wall command on unix.  Message == mail.  Photo and video viewing permissions was dependent upon  groups or individual USER_ID rights.  The thousands of external sites using an API to connect to the Facebook internals, reminded me of service ports on a given system.  Why wouldn’t one want to know the inner workings of a technically enriched environment, and at the same time being part of a greater movement of a social network.  “Hello China.  I’m in America.”  I’m all in; resume sent.

After a week of of FB message exchanges with the Facebook inside-recruiter, I finally talked with her on the phone – she asked me about my background in technology,  the past companies that I worked with, why I wanted to work for Facebook, and then she asked me if I was ready for 10 questions.  “Hold on one sec”, I thought.  I paused, and said, “Well, I am ready for your questions, however I would like some time to prep.  Could you call me within a week and ask me those questions?”  I was ready for any technical questions or conversations, but with HR recruiters, it’s not usually a conversation on technology pieces; it’s usually single answer questions to minute details.  I doubt this woman on the other line would know how to read a tcpdump, though I would not bet my life on the it.  SHE does work for technical recruiting for Facebook.  I opted for the safe bet.

I studied for a week.  I referenced O Reilly books up the wazoo, such as “TCP/IP Network Administration”,” Essential SNMP”,  “Unix Power Tools”, “Essential System Administration”, and the ever so classic, “Unix System Administration Handbook”.  *NIX material that I usually use as reference, now I was remembering odds and ends of UNIX/Network tidbits.  I was a regular 12 set-volume encyclopedia of UNIX fun!  If you were on the “Who Wants To Be A Millionaire”, and you got stumped on a question regarding *NIX, call me – I could help.
A week passed, and the 10 questions were asked.  “What does MTU stand for?”, “What’s the default MTU?”, “What is the TCP/IP handshake?”  Etc.

Most of the questions could have been answered within a breath, as I suspected.   I passed the first interview with one slight error.  I really dislike those types of interviews where they just ask a question in which you can search the answer within a simple google search.  Where’s the thinking involved?  However you have to understand within those 20,000 applications a month, there must be 300-400 applicants that want to develop within the Facebook IDE, and the only way to screen the ignorant is to ask these silly, one-off questions.  The recruiters are an ACL to a more expensive commodity, the time of a Facebook engineer.

The second interview was scheduled.  The Facebook recruiter recommended two things to review, 1) know some theory and practice behind scaling system environments, and 2) scripting [any language is fine - Perl, Python, or Bourne].  She said, “There is a possibility that the interviewer might ask you to script something, so be prepared to write code in real time.  We use collabedit.com.  He’ll call you within a week.”

The two recommendations made sense to me, what would be the major issue with Facebook currently?  Scale.  And how do you deal with millions of people, tens of thousands connections, with the possible peak of hundreds of thousands connections coming through the front-end?  What challenges does a infrastructure face with over 200 billion page views a month?  To my understanding there are two Data Centers currently for Facebook, one in Virginia and another in Silicon Valley.  A third is being built in Oregon.  [My guess would be some Global Traffic Manager was in place, with traffic going to an active/active site.  I would love to know.]  With the exponential growth in traffic, another Data Center in Europe would be the next move.  Physical high availability is one thing, however it is the application layer within those Data Centers that intrigues me the most.  There is already a model for hardware architecture, however from the perspective of software architecture – Facebook is actively reshaping the industry’s application stack.  At 40million new photos each day (~2,000 photos every second) – you better know scalability.  Delivering the goods is key.

The second recommendation to brush up on code was key.  Scripting is essential.  For any type of Computer Engineer – from Desktop to Systems Architect – you have to know how to script – it will make your life easier.  Even with scale being a major challenge in an expanding infrastructure, you will need some code to scale out (horizontally).  Scripting is the basic building block to automation.  To be honest, to be requested to code in real time during an interview was going to be my first.  I chose to sharpen my BASH scripting, since I used the interpreted statements the most frequently in the UNIX environment.  It was the most comfortable.  I usually write scripts in Bourne shell, however there are a few advantages to BASH, such as how the interpreter handles arrays and integers.  I haven’t scripted in a year, so I was pretty rusty.  Like any language, similar to Spanish or French, if you don’t use it – you lose it.  The good thing is – if you know how a sentence is formed, if you know how a program is formed (i.e. flow control, process handling, functions) – you will have the mindset of picking up any language.  Creating a program in real time also shows the interviewer how you think – how you can break a complex problem into smaller reasonable easy-to-chew chunks and placing everything together in one coherent solution.  Code has the possibility of being poetry, laced with verses of logic.

So how did I study for interview #2?  Like a madman.  I read web articles on scalability, wishing I had access to one of Yahoo’s Hadoop implementations.  I read web articles on the back-end db migrations from Oracle/MySQL to a NoSQL solution – twitter and digg.  I read up on authors such as John Allspaw (“The Art of Capacity Planning”, “Web Operations”), Steve Souders (“Even Faster Web Sites”), and Cal Henderson (“Building Scalable Web Sites”).  I felt I had some scalability concepts down.  For scripting, I referenced books such as, “Unix Power Tools” and “Learning the Bash”.  If you just read code, you can understand the language, however it’s best to practice.  Think of solutions to your everyday issues that can be automated, or close to automation.  So I contacted friends and asked them if they needed any coding done on their servers, or write code for my own self just to make my life easier (automation is a great thing).

Another great resource is YouTube – surprising as that can be.  You can view some of the programs from O’Reilly (Velocity Web Performance and Operations Conference, OSCON), Google I/O 2010, and other independent video productions.  If you can’t afford to go to the conferences [either time or money is the constraint], one of the greatest ways to get some exposure of new technology is YouTube recordings.  It was a bonus for me to find an O’Reilly Velocity conference recording of “Tom Cook, “A Day in the life of Facebook Operations”.
Watching tech videos is an entertaining, geeky way to eat your tuna sandwich while taking a break from coding.

I was sharpening my saw in all directions – books, friends, websites, and from my actual creations.  I was ready for the interview, so I thought… scalability – check.  Scripting – check.  The time came and the Facebook Engineer called me.  I recognized his voice, and his name was Tom.  I just had to ask, “Excuse me, but are you the same Tom that just did a conference talk at O’Reilly Velocity?”  He replied, “Why yes I am.  That video has had some traction.”  I was in awe, and in some degree – starstruck – I felt like I was talking to a celebrity or a technical god.  After some pleasantries the discussion of technology started and of course a few questions here and there.  What is load exactly?  What does it mean?  Discussions on threads and processes.  How can two processes communicate with one another?  What I slowly understood while I was talking with Tom Cook was that this was not a discussion on scalability on a macro scale, however it was it was discussion of scalability on a micro-scale.  I was not prepared for some of these questions, since some of these questions were Computer Science fundamental.  It took me a short while to re-calibrate myself, and try not to sound like I was bullshitting my way through an answer, and began some of my retorts with , “From my experience from C programming…”, “From my observations with this Unix utility…” – the interview turned from a conversation, to a Q/A type of interview, with the flow of a two-way exchange of information to reached-out answers.

I did not get the job, and I knew this by the end of the conversation.  I was not even asked to demonstrate my expertise on shell scripting.  I received the notification from Facebook that I did not received the job.

I have to admit I was disappointed that I wasn’t able to have a face-to-face interview with the company, it is still one of those companies where I would feel privileged to work for and a place where I feel I can contribute.  Even though I failed with the interview with Facebook, I learned something from the experience – my philosophy is ‘it’s never a total failure if you can learn from a situation’.  What I learned is: if you want to work at a high-caliber Technology Kung Fu shop, such as Facebook – you have to have some chops, not only on the practical, but also on the theoretical.  You need to understand the small particles of computer science.  To truly master scalability, performance, and availability – you have to know the fundamentals.

Moving forward with my own Kung Fu.  I have leaned a bit more closer to the Computer Science side by, 1) enrolling in a community college class in “Computer Architecture”, 2) reading “Practical Programming: An introduction to Computer Science Using Python”, and 3) and seeing where I can apply my free time in the open source community (via helping friends with their Tech Meetup groups and conferences).  Even writing this article helps me, by helping you – from one technologist to another – I say “learn each bit, each byte, challenge yourself, and if you love technology – learn some computer science”.  That’s my opinion.  At the end of this Facebook journey – I realized where I should be and where I’m need to be.  I love technology.  I am a scientist, and i should know the elements.  Know your Kung Fu.

References:
http://cns.ucsd.edu/lecturearchive09.shtml#Roth
http://www.Facebook.com/prinevilledatacenter
http://www.datacenterknowledge.com/archives/2009/04/17/a-look-inside-Facebooks-data-center/
http://www.YouTube.com/watch?v=T-Xr_PJdNmQ