I: The Mashup Artist
"What Descartes did was a good step. You have added much several ways, and especially in taking the colours of thin plates into philosophical consideration. If I have seen a little further it is by standing on the shoulders of Giants."
-- Isaac Newton, in a letter to his rival Robert Hooke. Feb. 5, 1676.
Sophomore year, I spent a significant amount of time producing mashups. You know, the genre of music in which the artist reappropriates existing work. The type of stuff you hear on the Hype Machine, in which the artist samples an existing piece of art to create something new. There's a deep question within the study of mashup music about the nature of art. What is original creation? Is mashup truly art? Is the artist not just stealing the work of others for self-profit?
Questions of authenticity and originality are a difficult topic. Without spending too much time exploring the depths of these questions and ending up half-drowned in a sea of existentialism, let me suggest a simple theory. Even though I no longer produce mashup music, I am nothing more than a mashup artist. And you're a mashup artist too.
What does it mean to be a mashup artist? I currently work in the field of computer science, training to be a software engineer. When I'm up against a problem, I usually turn to the web as my first resource. There, I find an extensive wealth of knowledge, and a community of people willing to provide support, advice, and code samples. Quite often, I can directly find the solution to my problem by browsing the web. On countless occasion, I've directly copied a code excerpt from StackOverflow.com and made the changes necessary to interoperate with the program I'm writing. What is this if not taking an existing artistic creation and modifying it to create something new? This, I argue, is why I'm a mashup artist.
Consider how much of what you give to other people is a function of what you have been given. I'm sure you can come up with examples of reappropriation in whatever line of work you pursue. I think American culture has a tendency to overemphasize the self. If I obtain success, it's because I am smart, because I worked hard and because I had the insight to get ahead in this system. I think this is a terribly flawed way to think about the world. The people who I have had the great pleasure of knowing in this life have given me gifts beyond their comprehension. If I am successful, it is primarily the work of others, not the work of myself. If I am someday able to have an impact on the world, it is because the world invested in me. If I am able to write a useful piece of software, it is because a magnificently complex system, built by countless others, exists beneath me. If I am able to build something new, it is because I was afforded the tools with which to create by the world around me.
The capitalistic structures we buy into make a lot of sense. They really do. They keep us motivated and working toward the illusory reward at the end of the path. Copyright is broken in many ways, and the lines of intellectual property are blurry at best. But let's face it, copyright exists for a reason. Still, the next time you accomplish something or the next time you create something of your "own", I'd like you to remember how much of that creation belongs to the world around you.
I'd like you to imagine yourself as a mashup artist. I'd like you to imagine yourself as sampling the brilliance already existing in the world and adding your own special touch. I'd like you to imagine yourself as giving the world around you tools with which to build even more spectacular things. There's a whole host of problems in the world that need talented people to solve them. Utilize the resources of the world around you, ask people for help and tackle them with all your energy. But take a break to look around every once in a while and realize how truly magnificent the world is. Don't believe me? When you go to google.com to type in a search query, you're employing hundreds of technologies that thousands of people have spent their entire lives developing. Just for you. Just so that you can go look at a funny picture of a cat. If that's not brilliant, then I don't know what is.
Self-abuse, the coolest thing on campus
Lately, I've been a bit bitter about college. Brown has been an absolutely wonderful place to spend my undergraduate career, and I'm certain I'll miss it as soon as I'm gone. I've had the chance to meet and interact with incredible people. By and large, I'm quite satisfied with my experience. But lately, I've been looking forward to leaving. I think part of this stems from my desire to produce real products for real people in the real world. I spend a significant amount of time on my academic work, completing projects for the sake of learning and earning a grade. I feel I could be spending that time (learning just as much) working on projects with a more tangible impact on the world around me. But I don't think that's the only reason I'm feeling burned on college. The primary reason may very well be the culture and climate of college.
This blog post may be surprising to people who know me well, because I'm about to criticize the way I've lived my life for the past two years.
I saw a Red Bull ad at a local bagel shop reading "Nobody ever wishes they'd slept more in college" and I think it summarizes the college mindset pretty well. There is a strong mentality that these four years you spend in college will be the most important four years of your life. As such, there's a pressure to make every single moment count. Sleep is for the weak. Proper nutrition and a cultivation of mental and physical well-being are completely neglected. Instead, there is a culture of excess. "Work hard, play hard" is the mantra of this life.
On campus, it is seen as "cool" to brag about how little sleep you got last night and how overcommitted your schedule is. Everyone at Brown does too much. I recently read Cal Newport's Zen Valedictorian Strategy, and I think he's on to something. If we have a reasonable number of tasks on our plate, we can focus on each one and devote the proper attention to it. In effect, we will get more out of each task. When we're swamped with more than we can handle, we deliver mediocre results across the board rather than delivering great results on anything.
This is coming from someone who is a serial over-committer. Last semester, I went weeks on end where I slept ~4 hours per night. I consumed energy drinks and an abundance of Starbucks coffee to keep myself going. I ate the quickest food I could grab and often neglected my friends to focus on my work. In retrospect, I probably wouldn't change things. I learned my limits last semester and I learned the importance of balance. I performed well academically and continued engaging in my extracurriculars. But looking back on it now, I'm not sure how much of it I actually internalized. Sleep is important for your long-term memory, and I'm certain I could have gotten more out of last semester by taking on less.
I think the cause of this overcommitting can partially be attributed to a fear of missing out (see Matt Swanson's opinions on the matter). If I don't seize an opportunity presented to me, I may miss out on all the benefits of exploring that opportunity. Likewise, if I stay at home with friends to watch a movie, or decide to catch up on much needed sleep, I'm missing out on a plethora of other events taking place on campus. The fact that there's too much going on all the time makes it difficult to say "no" and opt out. We undervalue the importance of downtime and taking breaks from the fast-paced life that is college.
If college pressures students into taking on too much, it also pressures them into partying too intensely. In the same way it is "cool" to push yourself past your natural sleep requirements, it is "cool" to push yourself past your substance limits. In the same way there is a fear of missing out in not taking on another commitment, there is a fear of missing out in staying in on a Sat. night. Somehow we've gotten to a point where our abilities to push ourselves beyond healthy limits is seen as strength. We need to have better respect for those limits, I think, to truly be happy.
To revisit the ad from above: I think it's important to remember that Red Bull is trying to sell you a product. They want to make sleepless nights in which you overextend your body "cool", so that they can collect more of your money. I've been a victim to this mentality. I've spent my fair share of money on energy drinks and put more of those unhealthy chemicals in my body than I'd care to remember. I know I'll continue to struggle with this culture for the time I have left at Brown. It's hard to not feel the "carpe diem" mentality when you're surrounded by it and your colleagues all argue in favor of it. The worst part is it's incredibly difficult to see yourself being sucked into that current. I would consider several of my friends to be workaholics. There's something admirable about their drive and their work ethic, but I also think in the long run, it's an unhealthy lifestyle to be leading. Believe it or not, there's a life after college. A life full of continued learning, opportunity and growth. You'll want to be healthy for the years after college, trust me. There are plenty of resources to learn the things you want to learn in the "real world", and there are plenty of great people out there too.
I need to remind myself of the importance of moderation, the importance of balance, and the importance of taking proper care of myself. College will tell you that balance is "uncool." Not only that, it will tell you that balance is "lazy". If you're taking care of yourself, you're "wasting" time that could better be spent on productive tasks. If you go to bed early on a Sat. night, you're "wasting" your time because you're not out with your friends indulging in the party scene (never mind that drinking too much is called getting "wasted" for a reason). Don't fall victim to these beliefs like I did. Proper sleep and nutrition will enhance your cognitive abilities, making up for the time you "wasted" taking care of yourself.
You only get one body; respect it.
A Final Grade Notifier in Two Nights
A few days ago, my friend Matt Patenaude and I undertook a project to build a system for Brown University students that would automatically notify you when new final grades have been posted to your account. Rather than frantically refreshing your grades, you can relax and know you'll be notified as soon as they're available.
The final product is simple: you register for the application, and whenever new grades are posted to your account, you get a text message and an email. You can even reply to the text message, and the app will call you and read your grades to you (for privacy reasons, we don’t put your grades in the text message). Despite the simplicity of the application, the architecture behind it is quite interesting. This blog post explores the architecture of the system and outlines some of the security features we put in place.
We had two major goals when implementing the system: Fort Knox-level security, and a distributed, scalable workflow. For those interested in learning more about the internals, follow along with the code on GitHub.
The Workflow
Brown uses Banner for its grades, and as anyone who’s ever used Banner can readily tell you, it’s… slow. And very broken, in many ways. For that reason, among others, we decided early on that having a single server querying Banner for grade information wouldn’t suffice, especially since we wanted to give Banner a little breathing room between requests from the same IP address so as not to frighten it. We decided, therefore, to use a distributed architecture that looks a little bit like this:
How it works in a nutshell: a user visits our website to sign up. When they do, the web server makes a short request to Banner to verify that they entered a valid username and password, then stores their credentials and contact information in the database. It then kicks our internal manager application to tell it that a new user has joined and should be inserted at the front of a worker queue. Finally, it displays a “Congratulations” message to the user to indicate registration is complete.
The manager application is where the action really happens. At server start-up, it creates a queue of all of the users registered for our application, sorted from least- to most-recently-updated, which is automatically restacked every hour (users will also be inserted on an ad-hoc basis immediately after sign-up). Worker nodes, running on separate machines (we’re currently utilizing a separate web server, a Mac Pro, and a laptop on the side), then ask the manager for a new user every 10 seconds or so. If there are still any users pending in the queue, the server peels them off and hands the username and password to the worker.
It’s then the job of that individual worker to talk to Banner, check if there are any new grades published, and alert the manager of the results. The manager then updates the database, and if there are new grades, emails the user with a grade report, and makes a call to the fantastic Twilio API to send the user a text message. If at any point a registered user texts something beginning with “g” to the assigned phone number, Twilio will inform the manager, and the manager will then instruct Twilio to call the user and read them their grades.
The beauty of this architecture is that any number of worker nodes can be running on any number of machines in unison, all pulling users from the queue. With a single worker, we can handle a load of about 120 users and still provide hourly updates to all of them without ruffling Banner’s feathers. With just 5 workers, some of which can be running on the same machines, we can handle up to 600 users, and it scales easily from there. Five lightweight virtual machines each running 10 worker nodes could theoretically gather grades for the entire Brown undergraduate population once each hour.
Making it Secure
We knew that if we wanted to legitimately ask our users to give us their Banner login information, we needed to make the system secure. Like, really secure. And really secure we made it. Let’s walk through the workflow again, but this time with security details.
When someone visits our website to sign-up, they’re switched over to SSL right away. At the advice of our friendly security guru and cohort Neal Poole, we sucked it up and spent the money for a real, honest-to-goodness SSL certificate from GeoTrust, not just a cheap-o self-signed one. The benefits: a little green lock icon in Chrome, a blue badge in Firefox, and a grey lock in the top-right corner of Safari. Win.
Beyond the added user comfort, this ensures that all traffic between the user and our front-facing web server is encrypted end to end, which means it’s very, very difficult for anyone to eavesdrop on the connection. AES-256-CBC with a 2048-bit key. We use the same SSL encryption scheme when the worker nodes phone in to the manager application, and also use SSL when communicating with Banner and Twilio.
Once a user’s username and password have been validated during the registration process, the password is encrypted using the server’s private key before being stored in the database. In order for a worker node to be able to use the credentials it receives from the server, it must be manually bestowed with the server’s public key. This also means that we (as administrators) can look at the database, and never actually see anyone’s password in plain text.
For added security, in order for a worker node to be allowed to request credentials from the manager, it must identify itself using a mechanism very similar to SSH public key authentication. Whenever we setup a new worker node, we generate a public/private key pair specifically for that worker. We then store a copy of the worker’s public key on the manager server, and associate it with the worker’s unique ID number. Each request from the worker to the server must be signed using the worker’s private key, and is verified by the manager before being processed. The requests and signatures are also timestamped, which helps to prevent replay attacks (which would be useless anyway, because the credentials are encrypted, and therefore useless to anyone but a valid worker).
So let’s sum it up: public/private key encryption on the passwords, public/private key signing and authentication with each worker, replay attack resistance, and it’s all done over an encrypted SSL pipeline. Not to mention a randomized password on our database, which itself is behind a firewall, and not listening on any Internet-facing interfaces.
Needless to say, we put a lot of work into the security.
Sidebar: Caller ID Spoofing
Since the application has the capability of communicating via text message and phone call, we also had to take a number of other security issues into account. For one, the “phone on the desk” scenario. We decided not to put your grades in the notification text message itself, because we didn’t want a situation where your phone goes off across the room, your friend leans over to look, and just says “man, sorry about chemistry dude.”
Thus, in order to get your grades by phone, you can have the system read them to you, privately. That does raise the issue of caller ID spoofing, however.
One of the original models we explored was one in which you could simply call the phone number dedicated to the application, and it would fetch and read your grades to you. However, this would be vulnerable to caller ID spoofing: if I knew you had an account on our app, I could spoof your caller ID, and have your grades read to me.
We solved the problem by only providing meaningful information in outgoing communications. When you text “g” to the application, it will call you, and read you your grades. This is immune to caller ID spoofing: let’s say I’m being nefarious, and I spoof Josiah Carberry's caller ID on a text to the application. The application will call that number back (if they’re a registered user) and read the grades over the phone, as expected. The caveat is that that call — regardless of the origin of the text — will go to Josiah Carberry, because it’s his phone number that made the request! Foiled again, caller ID spoofers.
What We Won’t Do
I feel like this is an appropriate time to make a statement about the future direction of this project, since people have made a number of suggestions already: we will not, under any circumstances, at any time adapt the tool to allow the pushing of information to Banner. That means that we will not turn this into an automated course registration system. This tool is a convenience, and nothing more; giving it more capabilities could give its users a competitive advantage over other students, which is not only against Brown’s Acceptable Use Policy, but very much unethical.
We also at this time do not have any intentions of deploying this tool for other schools that use Banner. Our code is, however, open source, so if someone at another school was interested, they could use it for inspiration. It will certainly need some modification, though: our source is not published for the purpose of use by others, but rather as an act of transparency to make our users even more comfortable.
Conclusion
As side projects go, this was a fun one. It was great collaborating with Matt for the first time, and I look forward to working on future projects with him. Brown students should check out Monocle, the application he's currently developing as a one-stop source for news, weather, events and more.
BannerGradeScraper: Automatic Grade Notifications
At the end of the semester, I always find myself repeatedly logging into Banner, navigating to the "Final Grades" page, and being disappointed when my professors have yet to submit my grades. In effect, my life starts looking like this:
Rather than allowing myself to continue living like this, I thought I'd try cooking up a solution to the problem. I've written a Python utility called "BannerGradeScraper" to automatically log in to Banner, check for updates on the "Final Grades" page, and send an email notification when a new grade comes in. You can download BannerGradeScraper here.
In order to use BannerGradeScraper, you need to have the following installed:
- Python 2.x. Download and installation instructions for Python 2.6.6 can be found here.
- The BeautifulSoup module, which can be downloaded here.
- The Mechanize module, which can be downloaded here.
To install BeautifulSoup, untar the file I've linked to above, then run 'python setup.py install'. You can use the same procedure to install Mechanize.
Once you have everything you need, open up UserCredentials.py, and fill in your login information for both Banner and Brown Gmail.
The last thing to do is run 'python GradeScraper.py'. BannerGradeScraper will check Banner every 30 minutes for updated grade information and email you when new grades come in. Note: If you close the terminal from which you launched GradeScraper.py, this continuous polling will stop.
In the next few days, I'll be looking for ways to make the setup procedure more intuitive. For now, if you're comfortable with basic Python, you should have no problem setting this up. Let me know if you have any questions!
UPDATE: Matt Patenaude and I have turned this project into a web application. Register on the site and we'll handle the rest. You'll receive automatic email/SMS notifications when new grades are available on Banner.
UPDATE 2: I've written a blog post explaining the architecture of the web application. You can find details on how the system is set up here.
Chrome Extension: Facebook Wall Search
Last week, I posted a rant of a note on Facebook, complaining about the limited search options available to Facebook users. I wanted to share a video with a friend. I knew my friend Chris Novello had posted it to my wall in spring 2010, and I knew the post contained the word "rhetoric." Facebook's limited search options gave me little to no option. The only thing I could do was scroll to the bottom of my profile page, request more content, and repeat until I found the post. Facebook does provide a Search API, but from what I can tell, it only goes back 30 days. So it couldn't really help me out with my problem.
This past weekend, I decided I'd take this frustration and channel it into building a Google Chrome extension. I've wanted to experiment with browser extensions for a while now, and I also wanted to learn javascript a bit better. After a long all-nigher Friday night and some continued hacking Sunday night, I'm proud to release a very primitive version of Facebook Wall Search.
Facebook Wall Search is a Google Chrome extension that builds a local cache of all the posts visible on your profile feed. Here's how it works. Once you've installed the browser extension, navigate to your profile page. Facebook Wall Search will start fetching as much of your profile content as possible. Essentially, it will simulate the act of repeatedly pressing the "Older Posts" button at the bottom of the page (much like I did when I was originally searching for Chris' video). Once it's loaded all the content possible - everything from the day you joined Facebook - it will parse the HTML contents of the page. This means it will build up a big database of all the visible posts, comments, etc. The next time you come back to your profile page, Facebook Wall Search will only look at the newest posts - the ones that aren't in the database yet. This way, you won't have to wait for Facebook Wall Search to fetch all the content every time you log into Facebook.
When Facebook Wall Search has finished archiving your entire history, it will notify you with an alert. At that point, you can start searching! Just click the Facebook Wall Search icon and type your search query. Here's the result of searching for the original video I was looking for: 
If you too are annoyed with the limited search features of Facebook, download Facebook Wall Search and give it a try. Keep in mind this is a very early release after only a weekend worth of investment. There are lots of opportunities to improve it - make a smarter, more flexible search, etc. If you have any specific ideas, or you'd like to report any bugs you're encountering, feel free to contact me.
The Jargon of Computer Science
I can't tell you the number of times I've heard the phrase, "I could never study computer science," or better yet, "I'm not smart enough for computer science." Every time I hear one of these phrases, I die a little bit inside. I've heard very intelligent, very capable people say these words to me. What are we telling people about computer science if they believe it's reserved for some nerdy technical elite? Where are they getting the idea they're not "good enough" to study CS?
The fact of the matter is that anyone is capable of understanding the basics of computer science. Yes - I really do mean anyone. I'm a strong proponent of adding introductory programming to high school curricula across the country. Computer science has many direct parallels with the real world. When we model a problem, we build a system of communicating pieces, where each piece is responsible for some small subtask. This is no different than building a hierarchy of people to run an organization. Or understanding how a specific component of a car helps the car run. Having knowledge of how systems work within a computer can actually help us understand how systems work outside of the computer, in the real world. The problem is not in the underlying concepts, but the way the concepts are explained to people.
How many times have you heard a technical person utter the phrase, "I'm using jQuery as AJAX to inject a new div into the DOM," or "I'm using PHP to parse the JSON feed given back by the Facebook API?" Ever had problems connecting to a local Wi-Fi network and gone into your system settings? DHCP? DNS? What are these cryptic acronyms?
The underlying concepts that would allow you to understand what the person is talking about are actually quite simple. But if you've never programmed before, you hear the first few words and your brain goes somewhere else. You can't understand what they're talking about because they're speaking in a foreign language. Instead, you jump to the conclusion - "That sounds hard. I must not be smart enough to understand what they're saying."
This is aggravating on so many levels. It's a breakdown in communication as bad as two people trying to talk in different languages. Only it's worse than that - because the non-technical person walks away from the conversation with lower self-esteem, feeling like they're not capable of understanding technology.
Before the criticisms start pouring in, let me go ahead and state: There is value in talking to other computer scientists in dense technical language. But that's assuming you both have a common understanding of what you're talking about. The problem is the disconnect between an arrogant person with experience and a fragile beginner.
Here are my suggestions for how to address the problem:
- If you're a technical person: Stop using unnecessary sophisticated technical language to make yourself sound smart. Maybe it will make the people around you think you're smart. But is it worth damaging the self-esteem of people around you? The reason people don't understand computer science is because they haven't had it explained to them in a way they can relate to. They've only heard technical abstractions built on top of technical abstractions. Take a second and try to explain what you do in a way they can understand. Encourage them to ask questions. Be patient. Teach them something.
- If you're not a technical person: Don't get turned off to computer science because of the dense technical language. If you find yourself in a conversation where you don't follow what they're talking about, stop and ask them to explain. If they don't have the patience to sit down and explain it to you in a way you can understand, don't blame yourself. Odds are that they're a bad teacher. Or that they're too impatient or arrogant to take the time and explain it well. Whether you believe it or not, you are capable of understanding these concepts. I encourage you to take a computer science course at some point in your life. It will be a struggle at first. It's a different way of thinking than you're used to.
Two quick stories:
- When I was a freshman at Brown University, I took an introductory computer science class called CS15. I had never programmed in my life and hadn't spent too much time around computers. Was I intimidated? Absolutely. The first programming assignment took me five hours to complete. By the time I had finished, I had written 5 lines of code. 5 lines of code in 5 hours? Wow. I did not know what I was doing then. I continued to struggle through the start of the semester. Luckily there was a network of fantastic TAs to help me through my problems. They never made me feel stupid. They assured me that they too had faced the same struggles I did. How was that possible? These TAs were all-star coders. But it took time for them to get there. If they hadn't been patient with me and receptive to my foolish misunderstandings, I probably would have dropped CS and found something else. I'm so grateful they stuck with me and encouraged me. Because in CS, I've found my true passion.
- Junior spring I took a course in web development called "Building Modern Web Apps." I had never built a web page in my life. The programming I had done was for building applications to run on your desktop, not in the browser. I was at the activities fair trying to recruit students to work at the on-campus radio station, WBRU-FM. I was talking about my web development course with my friends from the station. Someone in the room overheard me and asked what we were working on right now. "The current assignment is to build a simple website in HTML and style it with CSS," I said. The person responded, "Are you kidding me? That's a joke. That's not an assignment. That's something you code up in five minutes." How did I feel at this point? I had just spent an hour trying to learn how HTML and CSS work and had been making very simple mistakes that prevented me from making progress. "For you, maybe. I'm new to this. It's not so easy when you've never done it before," I replied. He rolled his eyes at me. And I felt stupid. For no reason. The truth: I was not incapable of being a successful web developer. I had just never done it before. This kind of arrogance and elitism is the kind that prevents people from reaching their full potential.
During my first CS16 lecture, professor John Hughes (known by his nickname "Spike") said one of the best things I've heard a professor say: "I'm going to make mistakes up here - and when I do, I want you to call me out on them. I'm no smarter than you. In fact, a lot of you in this room are probably smarter than me. I've just been doing this longer and I have more experience with it."
Don't be intimidated by people who are "smarter" than you.
