akrish

With the wide-spread adoption of the internet, the traditional academic publishing system has become somewhat antiquated. This has caused a lot of uproar within the academic communities, and many prominent researchers have been thinking about alternative publishing systems. There’s a lot of material about this floating around the internet, but in this article I will outline some of my thoughts and ideas.

The Problem
If you are familiar with the problem, you might want to skip down to the next section where I talk about some proposals that I and others have thought about.

To start out: What is wrong with the current system? It is actually quite complicated, but the main idea is that publishers (Elsevier, Springer, etc.) no longer seem to be adding any value while continuing to exorbitantly charge both authors and readers. Traditionally, the role of the publisher was to aid in distribution of academic material, and when this was legitimately a service, I completely understand them charging for it. However, now that almost all content can be obtained electronically, the role of distributor is no longer necessary. Yet publishers continue to charge ridiculous fees for journal subscriptions, which are required for an institution to obtain even electronic access to journal articles. I remember reading somewhere that university libraries spend the majority of their budget on journal subscriptions.

So why do researchers continue to publish in these journals? Well it is well known that academia is instilled with a publish-or-perish mentality, and moreover the specific venue in which you publish influences how your peers regard your work. Journals are scored by impact factor and publishing in journals with high-impact factor indicates that I am a good researcher. The quality of journal in which I publish plays a significant role in hiring decisions and other career opportunities and this, at least to me, is the primary reason why researchers continue to submit to these closed journals. There are some other factors, that motivate researchers to publish in journals, such as the peer-review system and the fact that publication is a sanity check that the work is correct and reasonable. However, I think the main motivation is to demonstrate one’s research ability. Noam Nisan talks about some other reasons and more details about this problem here.

To summarize, as it stands, the publishers provide no real value, but they restrict access to the elite journals. This motivates researchers to stick with their clearly flawed system. If we could introduce an open system to score and critique papers, with a mechanism for recognizing outstanding papers, it seems like we could break free of the existing system.

A Popular Solution
One popular solution to do this is a combination of Reddit and the ArXiv for academics. Researchers can post their papers online and then other people can leave comments and reviews of the paper. Everyone has a reputation score and the influence of one’s comments depends on their reputation. Maybe papers can get assigned scores, so anyone can score the paper, but the weight of their score depends on their reputation. That way, on my CV I can write down all of my papers along with their scores, so that others can quickly glance at my CV and get an idea of how important my research is. This is the basic idea but obviously there are a lot of details so that one cannot game the system. I’ve spent some time thinking about this and I think that if you implement it carefully in enough you can make it work. Timothy Gowers also seems to think so and he has thought about many of the details. If you are interested, please read his blog post, here.

One of the comments left on the Timothy Gowers’ blog post is that we might not want to turn life into a game, where reputation points mean everything. I really agree with this; some black-box is calculating my reputation on this website and the score output by this has serious consequences on my life in terms of career opportunities, etc. It makes academic life too much like a game, where everything I am trying to do revolves around increasing scores on my papers and increasing my reputation. So while I still think the system could work, it may not be what all academics want.

A less popular proposal
Gowers briefly talks about another idea, or at least an extension to his existing proposal that I think merits some additional discussion. The idea is this: anyone can start, edit, and curate their own online journal about whatever they want. They assemble a team of reviewers, who could be peers, friends, collaborators, or really anyone else they know. The editor of a journal and the team of reviewers is public information, and their reputation (not necessarily based on a scoring system) is what helps determine the quality of the journal. When I write a paper, I can submit it to one of these online journals, where it will go through the peer-review process, and possibly be accepted. Submissions and reviews can potentially be done anonymously, to allow for double-blind reviewing. Acceptance into someone’s online journal is a stamp of approval of a paper, and on my CV I would list which online journals my papers were accepted into. As in the other system, once a paper is accepted somewhere, maybe anyone should be free to comment on and score it.

There are several ways this system can account for journal quality/impact factor. A simple one is to use the editor’s and reviewers’ reputations as a proxy for the quality of the journal. Another is to allow journals to have reputation scores, based on the scores of that journal’s papers. This second solutions presents a startup problem, but I think you could bootstrap by using the first solution until the journal has a substantial number of articles. Also note that this same problem arises when I want to start a real journal. Again there are some details that need to be worked out but I do believe this sort of system could be made to work.

As a sort of aside, Journal of Machine Learning Research (JMLR) is an example of some of these ideas at work. The journal is open, providing free online access of all of its articles, and it still has a fairly high impact factor. In 2004, apparently it had the second ISI impact factor of any computer science journal (source). This small-scale experiment suggests that this sort of idea might actually work.

In Conclusion
If anyone reads this, I’d be interested to know what you think about these proposals. Do you see any serious complications/problems? Do you have any alternative proposals?

I just submitted a research paper to a conference (INFOCOM) a couple of weeks ago and have been recently trying to figure out what I want take on as my next research project. It’s been a pretty rough couple of weeks, where I wasn’t really sure what I should be doing and I realized that this phase is what makes doing research challenging. In this post I wanted to talk a bit about my experiences over the last two weeks and some of my other thoughts on how to do research effectively. Since I work in the more theoretical side of machine learning, I realize that most of my accounts will be tailored to research in this field, and after thinking about this a bit more, some parts of this certainly won’t apply to other research areas (even within computer science).

Over the course of my last project, we came up with a bunch of ideas for small projects that we could work on after the submission, and the first place I went for inspiration was these projects. Unfortunately, many of these ideas were incremental changes to my existing project, i.e. ways to relax certain assumptions, ways to make the guarantees slightly stronger, etc. and I didn’t think they would make for substantial projects. I also had bunch of ideas that I had come up with when thinking about class project ideas for both of the machine learning classes I took last year. Some of these were terrible, some were interesting, but I didn’t really find anything that I was really excited about working on. I will emphasize here that it is a really good idea to write all of these things down when they come up, so that you don’t forget about things you want to do later on. The fact that I wrote down almost all of the ideas I had come up with over the last year was really helpful in finding a new project.

I’ve been working on problems related to network tomography, and to look for some inspiration I read a bunch of the newer papers in that field, along with some papers discussing problems in statistical learning, which is another area that I find really interesting. One of the ideas I had on my list was related to sparse coding, which is a pretty new idea to find simple representations of signals. I started reading a bunch of papers on sparse coding and kept my idea in the back of my head while reading, and eventually I figured out that my idea, as I had originally framed it months earlier, would not work. This was discouraging, but with some more thinking I found that a related formulation looked more promising. By promising I mean that I thought it would be interesting, and that I believe it solves an open problem. I decided that I might want to start working on this idea.

Before diving into the theoretical side of any problem, I like to run some simple simulations to see if the idea makes sense. In this context, I found an implementation of a related algorithm and modified it to see if my idea would work on some fairly trivial inputs. I like to do this as a sanity check, so I don’t waste my time proving things about algorithms that don’t work in practice. This I think is quite unique to machine learning, where the theory is mostly motivated by practice, meaning that no one will really care about how great your guarantees are if your method does not perform well at least on simulated data. At any rate, I didn’t want to start thinking about the guarantees of my algorithm without making sure that the it would work, and thats why I ran some simply simulations.

This is more or less where I am right now with my project, and the next step is the hard part: a theoretical analysis of the method. I don’t really have any clue about how to do this and honestly I find that I make progress in this direction at the most random times, such as when I’m in the shower. This reminds me of something that Manuel Blum told my algorithms class that I think is really interesting. He said, “I’m interested in where ideas come from?” I, for one, am also interested in this, but have no idea about the answer.

I think I kind of lost track of what I wanted to say in this article so I will summarize, and possibly add some new stuff here. When I’m evaluating whether a research idea is worthwhile or not I tend to ask the following questions (in order).

1. Is it well-motivated? Are there real-world applications where you would want to do this?
2. Is it novel? Has anyone else done it or related things before? (literature search)
3. Is it interesting to me?
4. Does it work, at least on simple examples? (simulations)
5. Can you prove that it works? What are the assumptions/restrictions and what guarantees can you make? How do these compare to related work?

Usually, if I can answer affirmatively to the first 4 questions, then I am willing to spend some time and figure out answers to the fifth question. I went through these steps for most of the ideas I had written down over the course of the last year. Many of them didn’t make it passed the first question, some involved a little more thought and a perusing through related work. This last idea so far has made it passed the first four barriers; the next step is the analysis.

Ok so as usual I haven’t written in here in awhile, but I thought it would be a good idea to (attempt to) start something that I’m going to call “Week in Review.” Essentially, the point is for me to write just a small post about whatever I did last week, and the point is that I do this every week, hopefully every Sunday evening. So without beating around the bush, here we go:

I kind of think about my life in three separate categories: school/work, ultimate, and the rest so I’ll break these into those three categories as well.

School
First, last week a spent a bunch of time thinking about a course project for my statistical machine learning class. I read a couple of papers about matrix completion, clustering and related topics, and eventually settled on a project about multidimensional scaling, which is a dimensionality reduction technique. At a high level, I plan to study MDS in noisy conditions, and hopefully I’ll find some interesting results about when MDS works and when it doesn’t. I also volunteered to present a paper at the statistical machine learning reading group, so I spent a bunch of time early in the week preparing for that presentation. I presented a paper by Tsybakov (You can find it here) which studies nuclear norm penalization for matrix completion, and gives both upper and lower bounds for the problem. The paper is pretty technical and I didn’t understand all of it, but I thought I did a good job of figuring out what was important and presenting it to the group. I was relatively happy with my presentation, shows what you can do with a little preparation.

In classes, we talked about minimax theory, which I think is really interesting, but also kind of a black art. I borrowed Tsybakov’s book from the library and I want to read it to learn more about these lower bounds. Hopefully I can find some time to read it. In algorithms we spent the week talking about the Planar Separator theorem, but for the most part I was thoroughly confused. I also made “reasonable” progress on my research, and I’m starting to get a much better idea of what is actually going on.

Ultimate
Last week was a pretty great week for ultimate. We had a couple of warm days and finally managed to do an outdoor track workout and have outdoor practices which was amazing. These were basically my first full practices in over a year, and it felt really great. On Wednesday we ran a pretty brutal track workout (8×150, 8×100, 8×50 with 60/45/30s rest between reps) and somehow I managed to keep up my speed through the 100s and the 50s, which made me happy. On Thursday we practiced outside and while things were a little rusty, I was pretty happy with my downfield defense, although my flick hucks were really terrible. Saturday was super windy, and we had another outdoor practice which was quite challenging. We spent most of the time working on zone-O/D. On the defensive side I mostly played in the middle of the cup and was relatively happy with my performance. On offense I was really disappointed. My flicks are terrible in the wind and this is completely inexcusable.

On saturday night we had two winter league games against the other two CMU teams. Winter league has been really fun and both games we played were pretty competitive. In the first, we scored a buzzer beater to take the game into overtime but then lost. The other game we lost by a couple. After getting home at 2AM on saturday night, we had another practice at noon on Sunday, and I was already pretty exhausted. To top it off, a couple of Corey’s buddies (who happen to be really good at ultimate) were coming to guest coach. We spent most of the time working on defensive positioning and physicality; most of the time focusing on shutting down the under cut while being able to catch up on the deep shot. I felt like I did a good job in the drills, but didn’t carry that over to when we scrimmaged. I guess part of it was that I was super tired, but I definitely did not play well when we scrimmaged. I thought the practice was really productive and felt good (albeit extremely tired) afterwards. I need to learn how to lay out and do it more.

Life
Life was pretty eventful this week. Ravi is still really excited about Settler’s of Catan, so we played two pretty epic games on Thursday and Sunday respectively. Both games had several people at 9 points and I’m happy to say that I was really close to winning in both games. In one I actually had 10 points but it didn’t get around to my turn, which is really annoying. On Friday, Google hosted a TG (happy hour thing) at their campus, and I won a Nexus One in their raffle, which I was excited about. Unfortunately the phone doesn’t work on my provider, so I think I’m going to sell it.

My parents visited this weekend and my mom cooked a bunch of food for my roommates and my brother. This was really great, especially since Robbie and I were really exhausted from practices. On Sunday we came home and practically ate everything my mom had made. My family went to the Strip District on Saturday to check out the market, where there is actually a lot of really good stuff. I tried some really tasty jams and we also bought some pretty good donuts. I should go there more. It was pretty unfortunate that while my parents were here I had some much ultimate stuff going on because I didn’t really spend too much time with them. On the plus side, my freezer is filled with leftovers so I won’t have to cook for the week!

So we’re going to skip forward quite a bit in my israel trip to today. I promise to go back and write about the rest of my trip, which was arguably the best part. But now, lets jump to today…

I have 7 days left (by now it’s more like 6). I generally have very mixed feelings about leaving Israel. Part of me really likes everything about being here, except for my dormitory. I have a solid group of friends, I have a great, stimulating job, I get to play ultimate, go to the beach, eat amazing food, you name it. This part of me is really sad to leave.

The other part of me is really excited to go home. I haven’t seen my parents for almost 9 weeks and I haven’t seen my brother for even longer (I think I last saw him in January… ridiculous). Of course I also miss a lot of my friends from home and from college and I’m excited to see everyone. My parents would love to hear me admit this, but I actually miss home-made indian food, though I’m sure I’ll still complain about it when I get home.

So last night I took part in a ultimate game between the two teams in Tel Aviv. In my time here, I’ve gotten to know many players from both teams pretty well, and yesterday I had to say goodbye to many of them, because I’m not sure if I’ll be able to play ultimate again before I leave. And it made me pretty sad. I really liked a lot of the people who I played ultimate with, and unless I come back to Israel or they come to the US, it’s very unlikely that we’ll see each other again.

Fortunately, I did get contact information for many of them, and I plan to keep in touch, but it’s definitely not the same as hanging out or playing ultimate together. I do have a reason to come back though.

And as my time here quickly runs out, I’ll have to say more and more goodbyes to some really great people. I really wish I could spend more time with them before we have to part ways…

Ok wow this post and the previous one were probably really depressing. Don’t get me wrong, I’m not moping around and wasting my last days, I’m “living it up”: doing as much as I can, eating great food, hanging out with my friends and making sure I take advantage of being here. But I do think about the depressing stuff every once in awhile.

Oh yeah, one more reason I’m excited to go home. I’m going almost directly to Bali for a solid week and a half of vacation!

I started reading the news recently (yes I used to not read the news or really care what was going on in the world) and it’s making me really depressed. I subscribed to the New York Times’ RSS feed and here’s what I read about today.

• 16 Dead in Afghan Helicopter Crash – Last week there was the Iranian plane crash that killed over 100 people and now this. The middle east is still as depressing as ever.
• Explosion Kills 9 in Afghanistan – These I.E.Ds are still littering afghani streets and pretty much just wreaking havoc. not good.
• Chinese Question Police Absence in Ethnic Riots – First of all there were massive ethnic riots in China between Uighurs and the Han, and second of all, the police really didn’t do a good job of stopping the initial riots of the Uighurs and the ensuing retaliation of the Han.
• Explosions at 2 Hotels in Indonesia Kill at least 9 – Yup, and I’ll be in Bali in just a couple of days…
• 5 Officers Shot and 2 Suspects Killed in N.J. Melee – And even at home things aren’t pretty. This one is a pretty violent gunfight.

I guess there is some light in all of this but it doesn’t seem like much consolation

• Mauritians See Chance to Break Cycle of Coups – After years of living under military regimes, the Mauritians appear to be having a legitimate election. Of course, the current ruler is a pretty strong favorite to win, and he first came to power via a coup, so while this is good, it’s not all that amazing.
• Signs of Hope Emerge in West Bank – The Palestians, no longer under Hamas, is experience some economic growth and some form of peace. There has been a lot less violence between Israel and Palestine over the last year or so and I personally am pretty hopeful for some sort of peace agreement.

And while most of the world is as chaotic as ever, Israel seems to be relatively peaceful. Of course it’s a pretty precarious peace here, but there haven’t been any commotion since I’ve been here, so that’s pretty good.

I think this is partly why I don’t like to read the news. I guess it’s good to know what’s going on in the world, but when the world is as messed up as it is, it just makes me depressed.