US vs Europe

Being in Europe now, I am starting to realize the colossal differences between these two continents. This is no surprise, as the same cultural gap exists between Asia and Africa, the other two continents I have visited, however it's still impressive...
First things first, I find that Europeans are relaxed. I walk in the street and feel like i'm jogging compared to the rest of the walkers...This might have to do with the fact that it's August and everyone is either on vacation or working in an almost idle state, waiting to leave for vacations =)
Then, the food. I am losing weight by eating more. I don't know what's wrong with food in the US but I am definitely eating more and exercising less but still I lose weight. This could be due to preservatives or jet lag so I will give it some time!
Next thing is the traffic lights and driving in general. I was surprised to see the traffic light across the intersection but it actually makes sense...Today I was waiting at the traffic light and a bus stopped right next to me blocking me the view of whether the light is green or red...And of course you can't see the perpendicular's street light either!
Mass transportation. That's a huge difference. Even though San Francisco admittedly has "some" means of public transportation it's still worse than a middle sized town in any small sized city in Europe. The reasons are well known like spread out suburban areas etc, however this is definitely one of the most amazing differences!
Attitude about drinking/smoking. In the US noone can buy drinks if she is under 21 and yet you see people drinking out of semi hidden beer cans in the street and every single sorority/fraternity person being drunk near lethal dose in parties...In Europe ( minus UK I guess but that's a different story :) ) everyone can buy drinks and yet you don't see people drunk on the streets ( except for the same people from UK mostly! ) and you don't see pretty much anyone drinking on the streets or being drunk in a club. I don't know what's about the culture that is different but I definitely can't help but pointing this out...
Regarding smoking, in the US you see people not smoking in general and even people who smoke do it sparely and you feel like they are feeling guilty for smoking. They have a quick bum and get back to continue with their lives. In Europe you see people not only smoking like this is their last cigarette but also integrating smoking with their everyday lives. For example in Amsterdam you see people sitting in a bench and smoking while tanning.
These are the first ones i pointed out up until now, i will try to keep this post updated with anything else that makes me think like "this is interesting" !

agile development and sprints

As anyone who knows the real me knows, I am deeply interested in all the agile development methodologies. Since I was back in school I wrote a paper about a variation of scrum that received great acceptance and reviews. While at work I got certified in scrum technologies and I try to be the agile evangelist in every project I work in. Doing so isn't easy, as old fashioned teams don't generally like getting introduced to a new way of doing things. People resist change and tend to be pessimistic about new processes =)
However, the way that agile development is being pushed in some big companies leaves a lot to be desired. In one of my past few weeks, I had to attend a meeting regarding how did the past two sprints go. This in principal is a great idea. You all meet in a big conference room, everyone writes down in notes what went well and what went wrong, some people organize them in clusters and then the management team goes back and works on the feedback. Be it good or bad, the management team has a clear view of how the project is going.
That's the theory...In practice, the dynamics of a huge development group are way more complex than that. When you get in the same room technical people and the business stakeholders, along with all the upper level management, hilarity ensues....The actual developers of the system don't wanna appear harsh in front of their managers and the broader team that can't understand the technical challenges of the project. The upper level management doesn't want to admit when there are serious flaws in the execution of the project as this might mean refactoring processes and this would mean a further delay in the release of it. And the last thing upper level technical management wants is to push a delay in the release and try to explain to the business people that this is because of "fixing technical/project management processes". This will appear like gibberish to the business people who don't care about how the system works, just that...it works!
The business people on the other hand, deserve a paragraph of their own! The whole meeting was a "pat each other on the back" fest...Junior product managers appraised how great their senior product managers are, senior product managers appraised project management and project management appraised back product management...All this tells me that they either have no clue how the project is going and where is it headed towards, or they are trying to prepare for the thunderstorm that is coming once the EVP's understand the full extent of issues that are dragging the project behind...
However, this was a great experience. Teached me how great agile processes are but how dangerous can they be when put in the wrong hands...
As for me? I wrote down my concerns about how code reviews are being held and that the "daily status standup" is a parody since you have so many people that people can't give any status but just bitch about blockers. But why would you wait till the standup to bitch about blockers and not just go and unblock them yourself by talking to the right people and of course why should I care about the blocker of person X with person Y who both are working in something that is totally irrelevant to my part of the platform? It makes for good work gossip though :p lol!

google maps api and hosting your pet projects

These last days I have started working in my new pet project. It's driven mostly by the personal need to fulfill my own problem and wondering if it can benefit other users as well.
For this project, I need to use some mapping servcie that provides map display, directions and rich javascript control. I was looking around the two major maps api's and couldn't but agree with Carol Bartz 's view that Yahoo! Maps can't even compare to the feature set of the Google Maps API. I was truly amazed by the number and quality of features the gmaps API provides and the rich integration with everything google. From analytics to adsense to the Google AJAX API among others...
The other dilemma I am facing this period is how to architect the system. In one sense it could be a javascript only solution but this would severely limit the feature set of the system. One of the features I would like to have is obviously as a platforms guy to be able to collect visitor's raw web server log data and analyze them. This seems to be impossible with google analytics, at least up to the point of an apache web log record can provide you.
Also, I evaluated the usage of google app engine for hosting this project. For my personal pet projects I am usually either storing them on my own git server along with documentation in my twiki server that comes from the same apache server from my old netbook style dell laptop =) But for a project like this, I would like to be potentially able to serve more than just me. Even if it's just friends that find this service as useful as I do.
Google app engine sounds like a good candidate. You can write your code in python or java ( both good options for my style ), you can server enough requests per day for a pet project and if needed you can scale out by paying a (small?) price.
I am still undecided which way I should go. I have started hacking on my dev box but I might soon move over to app engine.
Another nifty feature of the app engine is that it accepts java bytecode instead of source code. This means that pretty much every language that has bindings into jvm can run in this environment.
All in all, google app engine is great but it still lacks some features to make it flexible enough for me. Nonetheless, it provides a turn key solution that alleviates you from many problems. I will post sometime later regarding how did it go...:)

#IranElection

Throughout these days there is this debate about the Iranian elections. Twitter' role into that has been amazingly helpful...Iranian people are informing the rest of the world throughout small txt messages from twitter. The interesting part is that twitter can't be censored like google, yahoo or facebook can. Even if you block twitter.com, people can still receive updates throughout the twitter clients. This is a massively powerful platform for people that want to bypass dictatorship like censorship measures.
Hopefully the situation in Iran gets better rather than worse...Up until then and until the next situation like Iran's one, I can't but applaud the fact that the Internet is getting more and more difficult to suppress and censor and that people still find ways to bypass censorship.

GUI development

In the past couple of week, I have been working in a java project in my 20% time over my normal 100% job time...
It's nothing extraordinary, some xml files holding the data, that I have to parse and present to the user in some "quiz format". The motivation was mainly the fact that I can't memorize facts, mostly in my short to medium term memory. I don't know if it's due to getting older or using computers ( with all the convenience of textpads, copy paste, multiple screens etc ) all the time, but the net effect is the same.
So as I needed to memorize drink recipes for one of my hobbies, I decided to have some fun out of it by writing an application to quiz myself over it and hopefully learn the recipes =) I decided to use Java for the sole purpose that I haven't written Java GUI's in years ( I think it has been since junior year undergrad that I haven't written anything java gui related ) and I was wondering how has the technology evolved.
After some online searching, I found out that thankfully AWT isn't mentioned anywhere anymore, which was a huge relief. I don't think anyone will miss AWT's awkwardness....Unfortunately though, Swing and SWT are in the same sad state that I left them.
I am no front end expert ( sadly, quite the opposite ) but I can't but applaud Apple's approach to designing GUI's through Xcode. It's simple, intuitive and totally WYSIWYG oriented. Of course, iphone's 1 screen size contributes to that, but hell I don't care. I want something that works...
I decided to use SWT, coz it looked more powerful and let's face it, I found more suitable tutorials and books related to it =) After sketching out the initial paper/more like napkin prototype, I started implementing the basic API's. The rest of the application was pretty straightforward, but the GUI part was a hell and mess. I ended up hacking the listener to imitate the flow I wanted to have. Aren't there MVC model for Java? Of course there are. Are there any MVC models that Sun promotes or dictates the way apple does? Of course not. This was one of my best lessons earned to this exercise in Java GUI's....
So now I am liking Apple a little bit more ( and not only because my brand new macbook is totally awesome ) for all the dictatorship like rules it imposes to developers. Maybe I am converting from Perl's "there are 1000 ways to do sth and we won't tell you which one to follow" to Python's "you *have* to write code using indentation and that's an order" mentality....
In any case, if someone wants to have some fun with my java hacking on a Friday late night purple rain powered session, here it is...

hadoop summit 2009

Last week was the second hadoop summit. Being local and really interested in the technologies I decided to visit. I have mixed thoughts about the whole event that I will try to lay down in this post.
First of all, I was pleasantly surprised by the participants and the problems they try to tackle using map reduce related technologies. It's amazing to listen about how every different company from the smallest ones to the big ones ( Yahoo / Amazon ) are using these technologies to process data and extract analytics.
The talks were really informative for someone who is dealing with this stuff on a day to day basis. I got to learn how twitter / linkedin / facebook are trying to extract more data from their users and what are their plans for future expansion. I was amazed by the fact that many of these companies aren't having the huge datasets or big analytics groups that one would think they have. I think it was linkedin claiming to have 9PB ( that's just 9000 GB ) of data in their hadoop cluster. That's less than any one of my typical job submits...Other companies had a handful of engineers working in problems with great breadth. This was a great surprise for me, as I thought that every company dealing with map reduce problems has multi thousand nodes clusters. Apparently, there are companies dealing with mapreduce problems that can be solved in a hundreds and even tens of nodes.
An unexpected surprise was amazon's talk. These people are really amazing in terms of engineering and they are addressing the long tail of the mapreduce problem really effectively. These companies needing a couple of tens of nodes in a regular or adhoc basis are way better off renting infrastructure from amazon than having their own infrastructure, ops people and all the costs associated with that. I was really interested in how they build their infrastructure and especially with all the security implications that sharing a node with other users has...Maybe in the next hadoop summit I will ask them more about their security plan ;):)
All in all, my overall impression about the summit was really positive. I learned a whole ton of stuff and I feel that I can understand better how do different organizations use map reduce frameworks for fun and profit.
I will definitely follow up with a post about the scale camp the night before the summit which was really interesting as well...

requirements and implementation

One of the things I can never understand about Software Engineering is why requirements is so hard to define. I have been in numerous projects and requirements were always at best vaguely defined. Even when they were solidly defined, they weren't realistic in terms of high level goals. I tend to believe that the issue is that the people defining the requirements have no clue about how to implement them.
My proposal is that product/project managers get assigned to implement at least 10% of the required features. This way they can get a gist of what they are asking =)
Of course I don't believe my proposal will get adopted by nasdaq 50 companies tomorrow but I would love it if some companies would jump on this train and see what happens!

code reviews, standups, meetings and other ways to spend your work time

   Eveything else except for writing actual code...

   The biggest surprise for me when I had my first internship was how much time was spent doing everything else apart from writing actual code. When I moved on to a full time job, it only got worse. I can understand that a huge amount of time will be spent in requirements and design so that when you get to the actual code you know what you are doing and how.

   What I don't understand is the meetings madness. A prime example is code reviews. I held a code review this week and attended 3 or 4 more. The code review I held had two goals. 

   First, I wanted to get input regarding the code I wrote. I invited a really senior guy who has pretty much written the map reduce framework that I felt the least comfortable with using. His input was invaluable. He identified in a couple of minutes performance issues that I hadn't faced but could come down the road. 

   Secondly, I wanted to show to a junior hire what have I done and how. He is in the baby steps phase of writing map reduce code so he is not yet 101% into the map-combine-reduce mentality...

   From my point of view, the meeting was successful. I got my input and improved the performance and robustness of my code and the junior guy learned a ton from watching a 1 hour conversation and participating in it.

   Both guys had the actual code beforehand and pointers to the documentation of my code that explains what have I done, how and why.

   Then come all the other code reviews I attended this week. I attended both local code reviews and some with remote groups. In all of them, I had no code beforehand and no exact agenda. Agenda was sth like "code review for the gmail application" where gmail is of course millions of loc and we weren't going to review all this in an hour =) Really vague and not making much sense.

   In the meeting, we were going line to line over code without any kind of introduction for the big picture. The result was that all I got from these code reviews is a line by line low level understanding of the code and how to point out correct intendation ! This is not what I wanted to get out of it. 

   So why are code reviews so hard? What are the rules we need to establish for anyone who want to hold a code review meeting or even a meeting? Im not sure yet but hopefully one day I will....

writing code for the iphone

   Apart from writing mapreduce code, I like writing code for the iphone. I know it sounds totally incompatible, writing code for some distributed computing framework and for a really small handheld device but I really enjoy both of them...

   It might be because I started coding in GW-Basic back in the MS-DOS 3.3 era. I would write some code and the result was less than aesthetically appealing, but the fact that you had made sth by yourself, that you had the feeling of instructing this weird machine with the noisy fan what to do; that was amazing...

   Maybe the iphone reminds me of the constraints that we used to have in desktop programming. Nowadays developing for a desktop isn't that much about performance as it was some years or a decade ago. 

   You don't have coders writing things like Quake 3's Fast InvSqrt() ;) Or articles trying to find out who was the actual coder behind it:p Nowadays you have API's over API's, VM's, dynamic languages etcetc...

   The iphone is one of the last resorts for people who care about performance. About "The Art Of Computer Programming" rather than how to call a Foo foo = new Foo() and call the appropriate method for this foo object to do whatever needs to be done. In iphone programming you have to care about battery life. You have to care about memory consumption and you know that memory leaks will get your app killed sooner rather than later. You have to do tricks and custom tweak every little aspect of the UI for performance and battery life.

   I guess this post is getting too lengthy so I will stop here. I will write a followup post where I will talk about the actual experience of writing code for the iphone !

i hate pig

well, since i want to blog on mapreduce, I decided to spend some time for my second post about it. The title might be a bit misleading. I love the idea around PIG. PIG is a great abstraction around the Map Reduce framework and I'm lucky to have attended the class of one of the guys behind it. I'm talking of course about Chris Olston . 

However, there are several things I hate about pig:

• Lazy evaluation. Actually the fact that there is only one mode. The lazy evaluation mode...:(
  
• Errors that make no sense. "Can't cast bytearray into bytearray". Why would I want to turn an apple into an apple?!...Another one of my favorites is sth like "Internal error occured" and then a bunch of exceptions that point to the pig source code. no information in the error message and absolutely no indication of where did the error occur in the pig script.
  
• kinda slow. Well this isn't a big deal any more, plus most of the times I prefer the fact that I get my sh*t done way faster and in a cleaner way than in Java or streaming in cpp/python but still it's kinda annoying when you end up with a workflow that you could easily save a couple of hours or a couple of nodes if it was faster =)
  
• Adding in the previous argument **corner cases**...*sigh* I can't emphasize enough on this. In general, pig is not that much slower than writing in java map reduce for hadoop. But in some cases, you end up with a job that should take 5mins taking 105. I have a job that just finished that demonstrates exactly this "feature". 100 nodes, 105 minutes for some terabytes of data processing...
  
• It's not complex enough. I guess it's because of the whole "pig philosophy" but it's still not complex enough. I find myself crawling through UDF's or writing my own to do sth that is really super simple in Java.

I know it looks like pure pig bashing but it's more of a hate-love relationship. I hate pig for all of these but once I write a script in a couple of hours that I know that it's super easy to maintain and document, I feel so much happiness that I just can't resist coming back to her...:)

And of course with that comes the fact that when you assign a heir to this code, it's really easy to explain to him how does the code work. Try explaining complex map reduce code to someone that has no clue about MR heh...!

no on prop8

Yeah I know, I should probably have a mapreduce post as my first one but then the title contains "and random stuff" =)

Well, I am witnessing a series of protests regarding uphelding prop 8 in the CA supreme court. I live in downtown SF so it's hard to miss them! 

I used to be undecided about this issue. I don't have any personal interest in the issue plus I can't vote anyway, however I want to be as politically active as possible, so I was at least thinking about it. On one hand "no on prop 8" has a set of really good argument. On the other hand "yes on prop 8" commercials around the time of elections were all about "teaching kids to be gay" and "allowing gay people to adopt kids". I dont really mind about the first one, plus it sounds ridiculous anyway. I would guess that prop 8 at worst would mean "teach kids to be whatever they want"...

Allowing homosexuals to adopt however, I dont know. It's a tough call....

However screw it, I think that equal rights are above all the other arguments and I am def supporting prop 8 for the past months. Today the protest did nothing else but reinstating my belief.

I hope that gays will manage to pass a new prop or convince the governor and state to allow once again the gay marriages.