akrish

I just finished taking a compilers class (well except for the final), and it’s been the single hardest class I’ve taken in college. Homeworks were non-trivial, exams were very difficult, and the class project was to write a compiler for a subset of python. As far as class projects go, this one has been completely ridiculous, my project partners and I have devoted weeks to building this compiler and fortunately we managed to finish it, but many groups I believe did not.

Anyway, in designing the compiler, I developed a new understanding of python features, despite having programmed in python for over a year.  My group and I found a lot of really crazy features of python that lead to problems when designing our compiler. I’m not going to talk about all the cool things that everyone knows about python, like lambdas, function values, nested functions and so on. I thought it would be fun to mention some of the really cool stuff here.

Dynamically Adding to Classes

Python basically implements classes as dictionaries. Instances are simply copies of the dictionary (well technically they are copied lazily, using copy on write), but instances naturally contain all of the mappings that the class contains (as they should). However, you can dynamically add mappings to classes so I can do something like this:

class A(object):
    x = 3
a = A()
print a.x ==> 3
print a.y ==> AttributeError
A.y = 7
print a.y ==>7

And I can even do this for methods:

class A(object):
    x = 3
A.foo = (lambda self,y: (self.x, y))
a = A()
print a.foo(6) ==> (3, 6)

When we discovered this, my project partners and I were amazed. It’s a really cool feature but I can hardly see a use for it (if you do see one, please let me know). Further, it makes implementing classes pretty inefficient since you can’t use a virtual table approach that C++ and Java can use.

Differentiating functions and methods within classes

We submitted our project after thinking we were done and “thoroughly” testing everything. A couple of hours later, we got feedback from the auto-grading system saying we failed (among a couple others) a test case that did something like this:

class A(object):
    x = 4
def g(n):
    print n
a = A()
a.x = g
a.x(5) ==> 5

We failed this test case because we assumed that any call to an attribute reference was a method call, and so we implicitly pushed on a as the first argument. This isn’t the correct behavior as we then pushed on one more argument than we should have. When we saw this, we were again amazed. Python somehow knows that a.x is a function and not a method, so it knows when it should implicitly push on self.

We hacked together a fix that I thought was pretty cool. We were representing all functions as boxed 12 byte function values, where the first byte pointed to a dummy virtual table, the second byte pointed to the code for the function and the 3rd byte was the static link for that function. Since from this perspective, functions and methods were exactly the same thing, we had no way to tell whether the object we had was a function value or a method value, so we were pretty stuck. Instead, we added a byte to all function values that kept track of whether they were functions or methods (yeah this is a huge waste of space, but performance wasn’t our priority at this point). Then at runtime, we are forced to check whether the value we’re trying to call is a function or a method, and behave appropriately in each case.

Unfortunately there’s a huge increase in code size. Now every call in python requires the check mentioned above, and two distinct call instructions, of which only one would get executed (because we were required to push on the number of arguments as a parameter to the callee function). We discovered this bug like 2 hours before the deadline, and we weren’t being graded on performance, so we didn’t really care about this issue. I am interested to know how python does it. My guess is that in the class dictionary, methods aren’t stored as “function values” so you would be able to tell right away, but I haven’t looked it up. If you know how python does it, let me know.

If-Expressions

I didn’t really use If Expressions before this class, and now that I’m more familiar with them, I think they are actually quite useful in writing concise code. I’m not talking about the standard if-elif-else clauses, but more of the analogue to x ? y : z syntax in C. And this feature isn’t that crazy, in fact it’s what I expect python would do. So an if expression looks like x if y else z and the semantics are that this statement returns x if y evaluates to True and otherwise it returns z. So it’s just like the ?: construct in many other languages. It’s cool when you use it with function values. For example:

def foo(x): print x
def bar(x): print -x
y = True
(foo if y else bar)(4) ==> 4
z = False
(foo if y else bar)(4) ==> -4

And that’s really cool. I can really concisely choose what function I want to call at runtime. I haven’t thought of a lot of uses for this exactly, but I can definitely see myself taking advantage of it’s presence.

The global keyword
There are some really cool features with global that I think are pretty standard in most languages but that I’m not used to seeing. For example:

def g():
    global a
    a = 3
print a ==> NameError
g()
print a ==> 3

This means that the when you declare a as global, if it’s not already bound in the global environment, you make a binding for it. I’m not exactly sure how this works in other languages, but I can totally see a language throwing an error if a is not bound in the global environment. Here though, python is smart enough to make the binding for you, and then you can use it after the call to g. This means that there are some really interesting ways that you can dynamically add names to the global environment.

So I’m sure there are more really interesting features of python that I haven’t discovered, and I’m pretty sure there are some that I have found that I’m just no remembering right now but these are some of the coolest language features I’ve seen. As I mentioned throughout, some are definitely more useful than others and I’m sure that in some cases these may be undesirable. If you know of any crazy python features that I didn’t highlight, I’d be interested in hearing about them.

No this isn’t about learning a programming language. This is about learning spoken languages…

My family hails from South India, and for both my parents and both sets of grandparents Tamil is the native language. And for as long as I can remember, I’ve been able to understand Tamil, but have been very inept at speaking (meaning pretty much that I can’t say a full sentence but can answer questions with a couple of words on occasion). This weekend my brother and I were struggling to speak Tamil to our mom, dad, aunt and uncle, and my friend Vivek who is pretty fluent was just breezing along. We all started thinking about why it’s so hard to speak when both of us can understand so easily.

One of the main reasons that we discovered was that verb conjugations are really challenging in Tamil. Both of us are fluent in spanish (after learning it for several years in school) and understand a lot about verb tenses and conjugations but we are still unable to understand the tamil conjugations. Part of it may be that when learning spanish, we are taught all the conjugations and them seem to be similar for pretty much all the verbs, whereas the way that we learn tamil is just from hearing bits and pieces every now and then, so we never have a chance to fully learn all of the conjugations. At the same time, from what I hear it seems that there are so many special case verbs that it wouldn’t really matter if I knew all the conjugations anyway.

Another reason was that Tamil is such an old language that there are so many different dialects and my mom, dad, Vivek, and my aunt all had different ways of saying the exact same thing. Somehow they all were able to understand each other and I was able to understand all of them as well, but this completely confused my brother and I when it came to trying to speak.

Also, similar to Spanish, there are different ways of saying things depending on who you’re addressing. For example, depending on who I’m talking to, the command sit down can be okachi, okazhu, or okazhungo if the person is younger than me, my peer, or my elder respectively. (Note: the “zh” is like a really strange sound that I’ve heard only in tamil. It doesn’t have a good translation to the English alphabet but most Tamil written in english does employ that to represent the character. It’s kind of a mix between and ‘l’ and and ‘r’ sound that both my and my brother find kind of hard to say. Read this for a bit more information). And using the first two constructs is very disrespectful if you’re talking to an elder, so you have to be careful.

As an example. We were trying to say something like “when I was young, I spoke tamil well.” I think my first take was “this (item) is a boy, he understands tamil.” I used 3rd person instead of first, used the wrong pronoun and of course the wrong verb tenses all over the place. I guess that’s not too bad because probably a lot of my family would understand what I’m trying to say (mostly because they can tell that I can’t speak well so they’ll pay more attention and think a little harder about finding some meaning). Eventually we were taught how to say the sentence, but my dad had a different way of conjugation “was” then Vivek did, so we pretty much stayed thoroughly confused.

As an interesting programming parallel. Our difficulty in learning Tamil can be compared to the difficulty someone has one first trying to write programs in a new language. Despite having read all the tutorials and seen a lot of code in the language, one still struggles to write programs until a firm grasp of writing the language has been acquired. Whenever I’m learning a new programming language, I always have to glance back at tutorials, look at reference books, and read function documentation for the first couple of programs that I write, even though I’m able to read and understand code in that language. When I’m writing code, I have to pay more attention to all the quirks of the language that I can just neglect without losing the meaning while I’m reading. Similarly with Tamil, I can understand very well because I ignore the quirks (verb conjugations, articles and pronouns, dialect-ual differences, etc.) but when I try to speak, I do have to get those things right, and I don’t know them well until I’ve spoken a lot.

So haven’t written a techy/nerdy post in awhile and I presume that’s because my life outside of work hasn’t been that nerdy during the summer. But I thought it’s time to return to my normal self and so… lets talk about python.

This article is titled the way it is because I’ve heard that phrase uttered so many times in the past year or two that it’s insanse. So I spent a little bit of time last summer learning python and the interns and I worked on a project in python, but it wasn’t too involved and I promptly forgot a lot of it in the months to come. I still used php or perl for a lot of my scripting purposes because I was familiar with them and my friends would continuously nag me to make the move over to python, but I never really got around to it.

So this summer, I was initially given the option to write in whatever language I wanted and thought that it would be a good time to learn python, especially because it’s used in a class I’m taking next semester anyway. I took about a week in the beginning of my internship to learn about python and ported some of my old code over to python. I already knew that python was pretty cool, but it seriously makes programming much much faster. And even though I ended up using perl for my intern project, I’m pretty happy that I learned python and I think I’ll be using it more in the years to come.

Ok so why is python so cool. First there are the reasons that everyone says, it’s like writing pseudo-code, but then it actually works. That’s pretty cool because programmers learn to think in pseudo-code but then have to translate that to whatever language. The closer the language is to pseudo-code the better.

It’s interpreted, which has pros and cons. Interpreted languages are generally slower than compiled languages, and yes I think python is slower than C, but for internal scripts where performance isn’t that critical, python being interpreted is pretty helpful.

Python is a lot like scheme/lisp with lambda expressions and functions as first-class data. I was disappointed that you can only have 1 line lambdas, but they’re still pretty useful. And passing functions around is amazing and extremely useful. Also has built in functions like map() and filter() which just make list manipulation so much easier.

Consider this, lets say I have a list of numbers and I want to make a sorted list of all the even numbers. In python…
sorted ( filter (lambda x: return x % 2 == 0, myList))
In most other languages (except lisp)… I’m pretty sure it would be a lot less concise. And yes the same sort of thing can be done in scheme, but I like scheme a lot too.

I like the way that python allows me to include my test cases of a module within that actual module. The if __name__='__main__': feature is pretty cool. You can’t really do that in perl and it’s annoying because I need to have a testModule.pl script for every Module.pm. With python I pretty much reduce the number of files I have to worry about by a factor of 2.

Also python is pretty easy to read and that makes your programs easier to maintain. I’ve been experiencing this problem where perl isn’t that easy to read (especially if you use a lot of the uncommon features) and now that I have to go back and make changes to my code it’s pretty hard to figure out where I’m supposed to look and how I’m supposed to make the change. And not that I’ve done this, but I think it’d be a lot easier if I wrote in python.

Anyway, I’m not really an expert on python or programming languages in general. In fact I haven’t even used python for a substantial project or anything, but I have been reading about it and playing with it so I found some things that I like. And I think I’ll be using python more in the future so I’ll become more and more familiar with it and probably find more things I like and also some things that I can’t stand.

Disclaimer: If my facts are wrong here please feel free to correct me because these were the impressions that I got and it would be good if I got them corrected.

I learned PHP in January and February of this year, using it for a couple of websites that I’ve written (including this blog). Over the course of several projects, I’ve become pretty familiar with the language, the online documentation, and all the in’s and out’s of PHP. I also first touched perl at the beginning of this year, but only for a short little project. This summer, most of the programming I’ve done at work has been in perl, so I’ve gotten quite familiar with it as well. Now that I know both of these pretty well, I have a little more flexibility in designing and implementing my ideas. That being said, every time I start a project, I have to think about which language is ideal. Now, I’ve gotten pretty good at this, and I know when to use perl over PHP and vice versa.

For web scripting, I prefer PHP. I really like that PHP can be embedded into html. It’s really quick and easy to write some pretty powerful web pages with PHP, and I’m a lot more comfortable with web forms in php over perl. I also have an incentive to write web sites in php, because my local server doesn’t actually execute any perl scripts called by the browser. So for most of my web stuff, I use PHP.

On the other hand, shell scripts are a lot easier to write in Perl. With perl, it’s really easy to interact with the shell and execute shell commands, and I also like that you can easily pass in arguments when you call a perl script from the command line. What’s more, with the default file handlers in perl, it’s easy to take inputs and have a more interactive shell script in perl. Although both languages have the capabilities to run through the shell, I just find it a lot easier in perl.

I also prefer perl for text-processing, most because it’s a lot easier to use regular expressions in perl. They “~” syntax is very simple, and a lot more readable than the “preg” functions in php. I also find it a lot easier to work with files; the perl file-handle data type makes reading and writing to files really clean. Again, PHP has all the same functionality as perl, it’s just a lot easier in perl, which is why I prefer it for text-processing.

Along the same lines, XML parsing is a lot easier to do in perl. Actually that’s the reason why I decided to write this article. I used to be parsing my xml feeds to this site using PHP’s xml parser, but have recently switched to a perl parser that I wrote yesterday. Because perl has such extensive libraries (i.e. CPAN) there’s less need to re-invent the wheel in perl than there is in PHP. PHP is a pretty powerful language, but because perl has a huge developer community that contributes to CPAN, a lot of functionality is constantly being added to Perl, while PHP isn’t such a dynamic language. In that respect, I prefer to use perl for a lot of things.

So why use php at all? For one thing, php is a higher level language. Most of my code remains pretty clean and tidy, and I don’t have to worry about a lot of minor issues, like variable declarations, that I do have to worry about in perl. I also like how php function declarations include the input parameters, whereas in perl they’re passed in to the special @_ array. PHP is quite a bit simpler to use than perl, so I prefer it when I don’t need the additional power of perl.

Recently, I’ve taken to using perl over PHP because I’ve been using perl a lot more recently and I find that I can do everything that I want to in perl. In PHP, I still can do almost everything that I need to, but sometimes it’s a little syntactically awkward and messy, which is why I prefer perl. I’ve started to figure out which language is ideal for the task at hand. As I learn more languages (I’ve started looking at python), I’ll naturally be better equipped with solutions to a given problem, but it’ll be harder to decide which language to use. Personally, I see this as a good thing, because it’ll be a lot easier complete my tasks if I choose the correct tools.

I’ve been playing around with scheme a lot in the past couple of days and it’s gotten me thinking about design strategies and how to use less code to get things done. Although I haven’t been doing much interesting stuff with scheme since I re-started using it, I have been reading about some of the more complex functionality of the language, and I hope to start writing some productive code pretty soon. Already I’m noticing tons of things that can be simplified through the use of scheme, and hopefully I can take advantage of this with my projects. I’m not gonna explain what scheme is so if you’re interested visit the link above.

My introduction to scheme came about a year ago, in an introductory programming class that was taught entirely in scheme. At first I hated the language because it was so different from Java and everything else I knew (most of the other people in the class felt the same way), but the language really grew on me as I realized how elegant it is. The course didn’t cover all of the amazing things about the language (i.e. macros, I/O, and using scheme as a scripting language) but we did hit some of them like higher-order procedures, scoping, and different types of recursion. Learning these concepts last year made it a lot easier for me to go back to scheme and pick things up where I left off, so that I can now focus on the stuff I haven’t already learned.

So why is scheme so amazing? First off, higher-order procedures are awesome. I know other languages have them (python), but they’re still really cool and very useful. There are so many situations where I’d want to pass in procedures as arguments, or return procedures from some other procedure and a lot of modern scripting languages are really restricting because they don’t allow you to do that. Secondly, everything can be written in very few lines of code, so that you’re programs stay readable. Most functions that I’ve written don’t exceed 20 lines, and readability is key on larger projects where you’re going to come back and modify stuff. Functions stay small because recursion is the norm in the language. Rather than iterate through things with a looping construct, I’d just use recursion and save a couple of lines of code. Finally, macros. These guys are pretty amazing. They let me write/modify my code at runtime. They let me create special forms and new syntactical structures that I can use later on (like if I need to use a certain structure several times I’d just write a macro for it to keep my code simple). Macros let me customize the language so that it’s perfect for whatever project I need it for. With all of these cool and very useful features, it’s amazing that so few people actually use scheme (or Lisp for that matter) on a daily basis.

There are a couple of ways I would use scheme on my current projects that I think would make my life a lot easier. One definite way would be to interface with mysql and really cut down the number of functions I have that query my database. I’m thinking I would have only 2 functions that generate Insert and Select statements (or one for each type of statement) and then one function that gets the statement and then executes it. Then I can have some other functions that modify/manipulate the mysql results the way I want them to. What’s cool is that I can pass in all the manipulator functions to the general mysql-interaction function so that all of the handling of mysql resources occurs in the same place. Then what I can do is build a macro that performs the mysql command so that I don’t have to make all these function calls every time I want to query mysql; I’ll just have to write one word. This idea hasn’t been fully thought out, nor have I even started to implement it, and it may very well be as lengthy as writing tons of mysql interaction functions, but it seems like it would make interfacing with mysql a lot easier.

Using macros would also be really helpful when getting data from forms vie POST or GET. I tend to store all that data in temp variables, and it happens on pretty much every page of my website. So rather than writing very similar code on each page, I could just make a macro that does that for me and call it on every page. I could have a macro that just takes a hash, and stores all the data in properly scoped variables for the remainder of that page. I hope this idea ends up working because I don’t like to have a lot of scripting code on my content-pages (I believe they should be mostly HTML) and this would take like 5-10 lines off of those pages and move them somewhere else on the back-end.

Unfortunately, there are some problems with scheme. First off, it’s very hard to find a hosting company that’ll have a scheme interpreter for your use, so you won’t be able to use it for web scripting unless you build/maintain your own server(s). I also don’t know how (if at all possible) to interface with mysql, which could threaten the feasibility of the first idea. Finally it may not be easy to embed in HTML (a real plus for PHP), which may be a turn off to a lot of people. I don’t really know if the pro’s outweigh the cons with scheme, but I hope to play around with it and come a conclusion about it in the near future.

Scheme’s obviously a really cool language, but that fact that it doesn’t interface well with other things (mysql, html, other web stuff) gives me doubt to it’s usability. I was thinking that I could use it as a meta-back-end, making it spit out PHP code that I then evaluate at runtime, but this may not work out as planned if I can’t interface scheme with PHP. Even just by itself, scheme is really powerful and can do a lot, but as I’m trying to build a website, it’s not much use to me on it’s own. I’m gonna be spending the next couple of days trying to get scheme to interface with other stuff, and after that, I’ll have a better feeling for how useful scheme will actually be to me.