akrish

The fight against academic publishers is heating up as Tyler Neylon’s website continues to gain support against Elsevier. If you haven’t heard, the website is a place where you can publicly declare that you will boycott Elsevier, one of the academic publishers with particularly terrible practices. It may have been created in response to Timothy Gowers’ public boycott declaration, and it is supported by him and many other famous scientists and mathematicians. As of today there are 3867 total signatories, and 544 signatories in computer science alone.

First off, why is Elsevier (and most other academic publishers) so evil? In a nutshell, they exploit the work of academics (funded by taxpayers) to turn incredible profits without adding much value. Journals are run by volunteer editors (academics), papers are reviewed by volunteer reviewers (academics) and papers are written by academic researchers. Moreover, researchers are expected to prepare “camera-ready” versions of their papers, which makes the paper almost entirely ready for publication. Publishers charge exorbitant subscription fees for their journals, but their costs are minimal and their value-add is effectively non-existent.

But publishing companies effectively have a monopoly on the top journals that academics need to publish in to advance their careers. Alternative publishing venues haven’t caught on because publishing there doesn’t carry the same weight as publishing in elite journals like Nature and Science. The fact is that academia, as it pushes the boundaries of knowledge, is very conservative about accepting change. Thus, despite the fact that several alternatives have been proposed (i.e. this, this and maybe even the Arxiv along with several more abstract proposals), academia has been slow to adopt alternative venues/media for publication.

Movements like “The Cost of Knowledge” are designed to combat the inherent inertia in academia, in hopes that we can converge on a better method of publication. Once academics realize that the many of their colleagues are boycotting Elsevier/Springer/etc, it will become much more reasonable for them to boycott as well. And once the majority of a field boycotts one of these companies, either alternative publishing venues will gain credibility, or the company will be forced to change its policies/pricing/etc or risk going under.

To me, the only issue is that this movement has to involve academic institutions as a whole in addition to individual researchers. Institutions use impact factor of journals as a surrogate for research quality and use this metric in hiring and tenure decisions. Until this changes, young, untenured researchers are going to be reluctant to boycott publishing companies that run elite journals because of the career implications that boycott has. This is probably one of the primary reasons why I haven’t joined the boycott yet.

The public boycott does have some interesting side-effects. First, the fact that the boycott is public and supported by top researchers means that it is more likely to gain traction. The fact that there is a list of elite researchers who are boycotting may influence how institutions make hiring decisions, which could kick start a positive feedback loop resulting in a much more powerful boycott. A more indirect effect is that top researchers are now boycotting elite journals, meaning that the quality of those journals will decline. This might force institutions to rethink how they make hiring decisions while also enabling alternative publishing media to flourish.

Whether the boycott is successful or not, enough people are up in arms (in the blogosphere, etc.) about publishing that it finally seems that academics have enough traction to prompt some sort of change in the academic publishing system. Hopefully we’ll see some positive changes in the next couple of years.

With the wide-spread adoption of the internet, the traditional academic publishing system has become somewhat antiquated. This has caused a lot of uproar within the academic communities, and many prominent researchers have been thinking about alternative publishing systems. There’s a lot of material about this floating around the internet, but in this article I will outline some of my thoughts and ideas.

The Problem
If you are familiar with the problem, you might want to skip down to the next section where I talk about some proposals that I and others have thought about.

To start out: What is wrong with the current system? It is actually quite complicated, but the main idea is that publishers (Elsevier, Springer, etc.) no longer seem to be adding any value while continuing to exorbitantly charge both authors and readers. Traditionally, the role of the publisher was to aid in distribution of academic material, and when this was legitimately a service, I completely understand them charging for it. However, now that almost all content can be obtained electronically, the role of distributor is no longer necessary. Yet publishers continue to charge ridiculous fees for journal subscriptions, which are required for an institution to obtain even electronic access to journal articles. I remember reading somewhere that university libraries spend the majority of their budget on journal subscriptions.

So why do researchers continue to publish in these journals? Well it is well known that academia is instilled with a publish-or-perish mentality, and moreover the specific venue in which you publish influences how your peers regard your work. Journals are scored by impact factor and publishing in journals with high-impact factor indicates that I am a good researcher. The quality of journal in which I publish plays a significant role in hiring decisions and other career opportunities and this, at least to me, is the primary reason why researchers continue to submit to these closed journals. There are some other factors, that motivate researchers to publish in journals, such as the peer-review system and the fact that publication is a sanity check that the work is correct and reasonable. However, I think the main motivation is to demonstrate one’s research ability. Noam Nisan talks about some other reasons and more details about this problem here.

To summarize, as it stands, the publishers provide no real value, but they restrict access to the elite journals. This motivates researchers to stick with their clearly flawed system. If we could introduce an open system to score and critique papers, with a mechanism for recognizing outstanding papers, it seems like we could break free of the existing system.

A Popular Solution
One popular solution to do this is a combination of Reddit and the ArXiv for academics. Researchers can post their papers online and then other people can leave comments and reviews of the paper. Everyone has a reputation score and the influence of one’s comments depends on their reputation. Maybe papers can get assigned scores, so anyone can score the paper, but the weight of their score depends on their reputation. That way, on my CV I can write down all of my papers along with their scores, so that others can quickly glance at my CV and get an idea of how important my research is. This is the basic idea but obviously there are a lot of details so that one cannot game the system. I’ve spent some time thinking about this and I think that if you implement it carefully in enough you can make it work. Timothy Gowers also seems to think so and he has thought about many of the details. If you are interested, please read his blog post, here.

One of the comments left on the Timothy Gowers’ blog post is that we might not want to turn life into a game, where reputation points mean everything. I really agree with this; some black-box is calculating my reputation on this website and the score output by this has serious consequences on my life in terms of career opportunities, etc. It makes academic life too much like a game, where everything I am trying to do revolves around increasing scores on my papers and increasing my reputation. So while I still think the system could work, it may not be what all academics want.

A less popular proposal
Gowers briefly talks about another idea, or at least an extension to his existing proposal that I think merits some additional discussion. The idea is this: anyone can start, edit, and curate their own online journal about whatever they want. They assemble a team of reviewers, who could be peers, friends, collaborators, or really anyone else they know. The editor of a journal and the team of reviewers is public information, and their reputation (not necessarily based on a scoring system) is what helps determine the quality of the journal. When I write a paper, I can submit it to one of these online journals, where it will go through the peer-review process, and possibly be accepted. Submissions and reviews can potentially be done anonymously, to allow for double-blind reviewing. Acceptance into someone’s online journal is a stamp of approval of a paper, and on my CV I would list which online journals my papers were accepted into. As in the other system, once a paper is accepted somewhere, maybe anyone should be free to comment on and score it.

There are several ways this system can account for journal quality/impact factor. A simple one is to use the editor’s and reviewers’ reputations as a proxy for the quality of the journal. Another is to allow journals to have reputation scores, based on the scores of that journal’s papers. This second solutions presents a startup problem, but I think you could bootstrap by using the first solution until the journal has a substantial number of articles. Also note that this same problem arises when I want to start a real journal. Again there are some details that need to be worked out but I do believe this sort of system could be made to work.

As a sort of aside, Journal of Machine Learning Research (JMLR) is an example of some of these ideas at work. The journal is open, providing free online access of all of its articles, and it still has a fairly high impact factor. In 2004, apparently it had the second ISI impact factor of any computer science journal (source). This small-scale experiment suggests that this sort of idea might actually work.

In Conclusion
If anyone reads this, I’d be interested to know what you think about these proposals. Do you see any serious complications/problems? Do you have any alternative proposals?

Yesterday I had a discussion with two of my friends about why we, as researchers, choose to work on the problems we work on. In statistics and machine learning, and maybe more generally in computer science, this can be a pretty interesting question. Do we work on problems purely for personal interest? Or do we require that problems have some well-thought-out practical application?

This is a philosophical question that I think is fairly unique to computer science and machine learning. In disciplines like biology, there is no question that applications are necessary motivation for a research problem. On the other hand, in pure math, it seems like applications are independent of problem choice; it’s nice if research is practice in nature, but it’s not necessary. CS and ML research is interesting, because in many respects it is only loosely motivated by application, and often it is more about developing theoretical results rather than demonstrating practicality. This is particularly true of the research that I have been doing recently in statistical machine learning.

The discussion went something like this: person A advocates working on things because they are interesting to him, while person B advocates having some grounding in reality before delving into a research problem. Person B at first claimed that the problems are interesting to the academic community only if they are practically motivated. As a counter, person B mentions the centuries of mathematical results (i.e. number theory, etc.) that only years later become “useful.” Clearly, these researchers were motivated by interest. Person B then asked why person A worked at all, and person B responded that he enjoys the problems he works on, mostly because they are interesting. The three of us then started talking about this in more detail, with focus on statistical machine learning problems.

Yesterday, we didn’t reach a conclusion on what was the right thing to do. I was tending toward choose problems based on interest. Assuming you have some sense of what has been going on in the community, your view on what is interesting and what isn’t should at least somewhat match the view of the community. If you feel a problem is interesting, my bets are that many other people feel that way, and for this reason it is a reasonable problem to work on.

I chatted with person B again today about this subject, and he brought up two interesting points. First, it is unlikely that you will get funding to work on problems that are not practically motivated. While this is probably true later in life, graduate school is exactly for this purpose. You get paid to work on the things that you find interesting, it doesn’t matter whether they’re well-motivated or not. Even later on, I think there is an art to making your work sound relevant and convincing people that it is worth their funding, and even with grants, I believe you have some flexibility to work on problems that interest you.

His second point was that working on obscure problems that no one cares about is not productive, even if you find it personally interesting. For example, if you take a well-known problem, modify it slightly, and rederive results for the modification, it isn’t interesting unless the modification you made is well motivated. I agreed with him on this point; I don’t really consider it research to walk through calculations for subtly different problems. There has to be something novel to the modified problem. On the other hand, if you took an obscure problem, and came up with some elegant, novel solution, that would be interesting and useful. Additionally, if you worked on an obscure problem, but came up with intermediary results that could be applicable elsewhere (for example new concentration inequalities), I would also say that is useful.

So after thinking about this for a couple of days here is where I stand. A research problem is worth working on (read: I would work on a problem) if: 1) you find it interesting, 2) it is practically motivated, 3) you believe that other researchers will be interested in your results (whether or not is it practically motivated), 4) You believe there are intermediary consequences, lemmas, or results that will be widely useful, or 5) You believe there is something elegant about the problem and its solution (i.e. the solution is not mechanical). To me (1) is necessary, and at least one of two through five are required.

I just submitted a research paper to a conference (INFOCOM) a couple of weeks ago and have been recently trying to figure out what I want take on as my next research project. It’s been a pretty rough couple of weeks, where I wasn’t really sure what I should be doing and I realized that this phase is what makes doing research challenging. In this post I wanted to talk a bit about my experiences over the last two weeks and some of my other thoughts on how to do research effectively. Since I work in the more theoretical side of machine learning, I realize that most of my accounts will be tailored to research in this field, and after thinking about this a bit more, some parts of this certainly won’t apply to other research areas (even within computer science).

Over the course of my last project, we came up with a bunch of ideas for small projects that we could work on after the submission, and the first place I went for inspiration was these projects. Unfortunately, many of these ideas were incremental changes to my existing project, i.e. ways to relax certain assumptions, ways to make the guarantees slightly stronger, etc. and I didn’t think they would make for substantial projects. I also had bunch of ideas that I had come up with when thinking about class project ideas for both of the machine learning classes I took last year. Some of these were terrible, some were interesting, but I didn’t really find anything that I was really excited about working on. I will emphasize here that it is a really good idea to write all of these things down when they come up, so that you don’t forget about things you want to do later on. The fact that I wrote down almost all of the ideas I had come up with over the last year was really helpful in finding a new project.

I’ve been working on problems related to network tomography, and to look for some inspiration I read a bunch of the newer papers in that field, along with some papers discussing problems in statistical learning, which is another area that I find really interesting. One of the ideas I had on my list was related to sparse coding, which is a pretty new idea to find simple representations of signals. I started reading a bunch of papers on sparse coding and kept my idea in the back of my head while reading, and eventually I figured out that my idea, as I had originally framed it months earlier, would not work. This was discouraging, but with some more thinking I found that a related formulation looked more promising. By promising I mean that I thought it would be interesting, and that I believe it solves an open problem. I decided that I might want to start working on this idea.

Before diving into the theoretical side of any problem, I like to run some simple simulations to see if the idea makes sense. In this context, I found an implementation of a related algorithm and modified it to see if my idea would work on some fairly trivial inputs. I like to do this as a sanity check, so I don’t waste my time proving things about algorithms that don’t work in practice. This I think is quite unique to machine learning, where the theory is mostly motivated by practice, meaning that no one will really care about how great your guarantees are if your method does not perform well at least on simulated data. At any rate, I didn’t want to start thinking about the guarantees of my algorithm without making sure that the it would work, and thats why I ran some simply simulations.

This is more or less where I am right now with my project, and the next step is the hard part: a theoretical analysis of the method. I don’t really have any clue about how to do this and honestly I find that I make progress in this direction at the most random times, such as when I’m in the shower. This reminds me of something that Manuel Blum told my algorithms class that I think is really interesting. He said, “I’m interested in where ideas come from?” I, for one, am also interested in this, but have no idea about the answer.

I think I kind of lost track of what I wanted to say in this article so I will summarize, and possibly add some new stuff here. When I’m evaluating whether a research idea is worthwhile or not I tend to ask the following questions (in order).

1. Is it well-motivated? Are there real-world applications where you would want to do this?
2. Is it novel? Has anyone else done it or related things before? (literature search)
3. Is it interesting to me?
4. Does it work, at least on simple examples? (simulations)
5. Can you prove that it works? What are the assumptions/restrictions and what guarantees can you make? How do these compare to related work?

Usually, if I can answer affirmatively to the first 4 questions, then I am willing to spend some time and figure out answers to the fifth question. I went through these steps for most of the ideas I had written down over the course of the last year. Many of them didn’t make it passed the first question, some involved a little more thought and a perusing through related work. This last idea so far has made it passed the first four barriers; the next step is the analysis.

This week was marked by two momentous events: 1. My department had our admit visit weekend and 2. I played in my first college ultimate tournament since January 2010.

The two events clashed a bit, so I ended up missing about half of the visit weekend so that I could go to Virginia for the tournament. The parts I was around for (Thursday night and Friday during the day) went smoothly and were very fun. The department held a reception for the prospectives on Thursday night so that the new students could meet each other and also meet some faculty, staff and current students. I got to meet a bunch of prospectives who seemed pretty excited about CMU and my department. On Friday I had to do a bunch of homework, so I didn’t spend too much time with the prospectives but I ended up meeting a couple others and also presented a poster at our department poster session. I presented some work I did last semester about high dimensional clustering using Gaussian Mixture Models with sparse inverse covariance matrices (See here for details).

After the poster session I met up with some of my teammates and we headed to the ultimate tournament, Hellfish Bonanza. As I said, this was my first time playing real competitive ultimate since getting injured last February, so I was pretty excited. In terms of training, my team had a pretty decent winter, but I think we were all itching to get outside and play. We came into the tournament seeded 6th (second in our pool) and played Richmond, Georgetown, and James Madison (the hosts) in our pool on the first day. We came out a little soft and shaky, but beat Richmond. For the Georgetown game, the wind started to pick up and we played really miserably, costing us the game. We also lost to JMU in another really windy game. We played a really good crossover game against Maryland, squeaking away with a universe point win to put us into the championship bracket on Sunday. We came out pretty strong on Sunday, beating Towsen to make it to semi’s, but then we lost to Virginia Tech and JMU to end up taking fourth overall in the tournament. I thought we played pretty well all day on Sunday, just that we tired out towards the end of the day and didn’t have enough juice to really bring it in the third place game.

Personally, I was relatively happy with my performance, especially considering I hadn’t played competitively in awhile. That being said, there are definitely things I need to work on. I’m feeling really unconfident about my flicks, especially in the wind (because I don’t grip the disc firmly enough), and this did show up a couple of times during the weekend. I did feel like my marks, and dump defense were pretty solid, which is the thing I’ve been focusing on in practices, but as always, there is room for improvement. We’re going to another tournament this week (in Georgia!), so I’ll be looking to improve on some of these facets of my game then.