You can find part one of my semester project rundown here.
Purdue's course is regarded as one of the most project-intensive courses that an undergraduate can take. Over the course of a semester, a team of students must design and implement a real-world project proposed by a corporate partner. This year sponsored two projects: a mapping application tailored to building-to-building campus navigation*; and a social networking and data visualization application built on top of . There were four teams, two competing on each project. I was the team leader for one of the Del.icio.us teams.
As indicated by the length of this post, this project became my primary concern this semester. I could never have foreseen the hours and hours that I willingly poured into my team's application. We spent many evenings, nights and weekends putting together the largest project any of us have ever seen to completion.
The Team
My team consisted of myself and six other CS students: , , Kai, , , and . I knew I wanted to be the team leader before the class even started. I hope to make software engineering my career one day, so I wanted to gain experience leading team through the entire development process.
I am very pleased with how the team performed. Everyone worked hard throughout the semester and contributed positively to the entire project. Once builds got underway, each person adopted a particular part of the application. I was the jack-of-all-trades, laying the groundwork for future builds and helping in current builds wherever I could. This organization worked out incredibly well. We communicated well, and the individual parts of the application came together without too many problems.
It was amazing the project went as smoothly as it did considering...
- Three team members had part-time jobs
- Two team members were married
- One had a daughter and did not have a computer at home
- One was in the marching band (which is a huge time commitment)
- One was taking compilers (also a huge time commitment)
- None but myself had any experience with .NET programming or large application design.
Design and Implementation
The main idea of the project proposal was that Del.icio.us' existing tag and bookmark "space" defines social relationships between users. For example, two users that have many tags or bookmarks in common probably share similar interests. The project needed to make these implicit relationships explicit through more social networking and data mining functionality. The project would need to display these social and data relationships in some form of enhanced visual frontend. Because the project was sponsored by Motorola, we also needed some kind of mobile device interface to the application.
We spent the first half of the semester designing the application. The course required a formal requirements document, user manual, and design specification. The purposefully vague project description gave us a great deal of freedom to design creatively. Even so, it was difficult to know how the application would function before we had written any code.
It was during this design phase that Brandon thought of naming the application "Scrumptious". I thought this name fit perfectly because it implies Del.icio.us-ness, with a little bit extra added on. Brandon even registered the domain sc.rumptio.us** which I thought was a nice touch.
We had seven weeks during the second half of the semester in which to build the application we designed. Unlike most CS classes, we, rather than the teachers, determined exactly what we needed to turn in and when. I planned the sequence and duration of each "build module" before we started developing. Prior to each module, we had to turn in a build plan to specify what we were planning to develop, a test plan to show that the module met the functional requirements, and an inspection plan to verify that the code and tests were correct. We usually had three build modules under development at any given point. I am pleased, as the team leader, to say that we only had to modify our build schedule twice and made only minor changes to our design document.
It was not until we started building that we realized how large the project would be. We managed to meet all of our build plan goals, but it was not without a great deal of hard work and long nights. We had a short status meeting every Tuesday night to plan the week's tasks. Each team member did his part during the week in preparation for the build demo every Monday. Around the fifth build we also started meeting on Saturdays in the lab*** for long team development sessions. These long meetings made it feel like we were a small web startup getting ready to roll out our flagship product.
The Application
Despite having just seven builds, our application had a very long list of features. The following is a very short list of the application's core functionality.
Social Networking
Like other social networking applications, Scrumptious users could create personal profiles, form "buddy" links, and send public or private messages to other users. Pretty standard stuff.
Del.icio.us Integration
Users could do pretty much everything on Scrumptious that they could do in Del.icio.us. When a user added or modified a bookmark in Scrumptious, the changes were automatically mirrored back to Del.icio.us via the . Internally we had a DeliciousInterface class that was powerful enough to spin off into a standalone .NET library if we wanted.
The Scrumptious API
We tried to emulate Del.icio.us' open nature by creating a Scrumptious API similar to the Del.icio.us API. Like Del.icio.us, our API accepted requests and returned responses. It provided an interface to almost all of the functionality within the core system and drove both the visual and mobile interfaces (see below).
Data Relationships and Visualization
This was probably Scrumptious' killer feature. Internally, we wrote logic to retrieve users related by common tags and tags related by common bookmarks. Externally, we displayed these relationships in an interactive Flash interface.
The visual interface displayed a network of boxes connected by lines. The boxes could represent either users or tags depending on what the user wanted to browse. When a user clicked a box, a cloud of other boxes all related to the clicked box would spring outward and form links to the clicked box as well as related boxes already displayed on the network. When the user clicked the line connecting two boxes, he or she could see what made the two boxes related. For example, while browsing the tag network, clicking a line would display a list of bookmarks on which the two tags were used.
The boxes were positioned dynamically using a spring physics engine that I wrote. Related boxes would attract down to a certain threshold while all nodes would repel when below the threshold. It was engrossing to click a box and watch the network reach a new equilibrium.
Internally, the visual interface just pulled XML from the Scrumptious API and displayed the results.
I really wish the application was still running so I could link to a demo; words do not do the visual interface justice.
Mobile Interface
Kai became our "mobile guru". He designed and implemented a mobile thin client written in . Because it was driven completely by the Scrumptious API, the mobile interface only needed to handle the UI, sending the GET requests, and parsing the XML responses. Thanks to the power of the API, the mobile interface supported almost all of the functionality of the core system but on a tiny phone screen. Pretty impressive.
The big selling point of the mobile interface was that it illustrated how a third-party application could piggyback on Scrumptious in the same way that we piggybacked on Del.icio.us. I love how applications are starting to become more open and distributed making architectures like this possible.
Supporting Projects
In addition to the scheduled builds, the team put together several side projects to help support various parts of the application.
Database Refresher
Han became the "database guy" during the course of development. He wrote a quick app that would remove all of our test data from the database and refresh it to a normalized state.
.NET Mobile Interface
Before we had decided on an architecture for the mobile interface, Matt played around with the . We did not get much farther than screen prototypes, but it did give us an indication of what to expect on other mobile architectures. The reasons behind our choice of J2ME are too convoluted to go through here.
Documentation Generator
To make build plans less painful, I wrote a quick application that generated the required documentation from the output after compilation.
Del.icio.us Scraper
In order to have meaningful data for testing and for displaying on the visual interface, I wrote a scraper that spidered across and saved the data directly back to our database. It may bend Del.icio.us' a bit, but it gave us some great data with which to demo the application.
Unit Tests
Our test plans for each build module simply consisted of the code for unit tests. It was a perfect way to demo the functionality of each module; when everything was green, we got the points.
Hardware Headaches
As is the case with any software project, we had our share of problems during the course of development. The most frustrating by far involved our server.
Brandon had a dual-processor, quadruple-RAID, redundant power supply, rackmount server lying unused in his apartment. This was extremely fortunate because Purdue does not offer any way to host .NET applications. He agreed to donate it to the project for the semester and installed everything we needed.
At first we were planning to run the server in my apartment. This proved infeasible because the redundant power supplies produced redundant noise. It sounded like a miniature jet everywhere in the apartment even with all the doors closed. Fortunately, the TA allowed us to run the server in his office, but we soon found this setup had its own problems.
We thought the first crash was a fluke. All of us were in the lab developing at the time, so Jon and I trudged over to TA's office to restart the server. The second crash, not 24 hours later, made us worry a bit more. We were again in the lab, so Brett A. and I dejectedly walked over to the TA's office. To our horror, the door was locked! As it was dawning on me that we might have a serious problem, the door opened and the graduate student that shared the office with the TA ushered us in. With a sigh of relief, I restarted the server. The graduate student, hearing our complaints, gave us his number for if we ever found the door locked again. We copied it down, naively expecting that we would never need to call it.
The third crash was the worst. It was the day before a build module was due and key parts were not yet complete. The server went down at about 10 PM, so I knew as soon as I got the "Unable to connect" error that the office would be locked. I drove over anyway and valiantly jiggled the handle. I reluctantly called the graduate student. By some miracle he was home and thinking about heading into the office anyway. I agreed to pick him up.
The drive over was epic. I drove up and down the street—the very street I live on—several times but was unable to find the grad student's apartment. I stopped at my apartment and sprinted up the stairs to look at an online map. It was no help. I called him again, apologized for the delay, and got better directions. A few minutes later I found him shivering at the entrance to his apartment complex. We drove back to the office, and I restarted the server.
The team and I speculated long and hard about what could be causing the crashes. Perhaps we had too many connections to the database; perhaps we had some critical bug in our application; perhaps Windows was being characteristically flaky... who knows? After Thanksgiving break, Brandon installed a remote power management card and set it to restart the server every night. That prevented further crashes. In the end we fixed the symptom without ever knowing the true cause of the problem.
The Final Presentation
My team gave our final presentation on Saturday, December 3 to the course administrators and a Motorola representative. Each team had to give a 25-minute "sales presentation" followed later in the day by a 45-minute "poster session" in which we demoed the application. Our presentation went very well, but the server had one final laugh during our demo: we got a full-page server error that none of us had seen before while showing the Scrumptious API to the professor. Oh well.
I consider the project a great success. We got an A on our presentation, an A on our documentation and builds, and we won the competition against the other Del.icio.us team. In addition, I learned how to better gauge the scope of projects and how to coordinate a team of developers.
* The winning mapping team's project can be found at . They did a superb job.
** Right now http://sc.rumptio.us does not resolve to anything. We had to take down the server after the final presentation.
*** Live webcam pictures of the lab can be found and .