akrish

With the wide-spread adoption of the internet, the traditional academic publishing system has become somewhat antiquated. This has caused a lot of uproar within the academic communities, and many prominent researchers have been thinking about alternative publishing systems. There’s a lot of material about this floating around the internet, but in this article I will outline some of my thoughts and ideas.

The Problem
If you are familiar with the problem, you might want to skip down to the next section where I talk about some proposals that I and others have thought about.

To start out: What is wrong with the current system? It is actually quite complicated, but the main idea is that publishers (Elsevier, Springer, etc.) no longer seem to be adding any value while continuing to exorbitantly charge both authors and readers. Traditionally, the role of the publisher was to aid in distribution of academic material, and when this was legitimately a service, I completely understand them charging for it. However, now that almost all content can be obtained electronically, the role of distributor is no longer necessary. Yet publishers continue to charge ridiculous fees for journal subscriptions, which are required for an institution to obtain even electronic access to journal articles. I remember reading somewhere that university libraries spend the majority of their budget on journal subscriptions.

So why do researchers continue to publish in these journals? Well it is well known that academia is instilled with a publish-or-perish mentality, and moreover the specific venue in which you publish influences how your peers regard your work. Journals are scored by impact factor and publishing in journals with high-impact factor indicates that I am a good researcher. The quality of journal in which I publish plays a significant role in hiring decisions and other career opportunities and this, at least to me, is the primary reason why researchers continue to submit to these closed journals. There are some other factors, that motivate researchers to publish in journals, such as the peer-review system and the fact that publication is a sanity check that the work is correct and reasonable. However, I think the main motivation is to demonstrate one’s research ability. Noam Nisan talks about some other reasons and more details about this problem here.

To summarize, as it stands, the publishers provide no real value, but they restrict access to the elite journals. This motivates researchers to stick with their clearly flawed system. If we could introduce an open system to score and critique papers, with a mechanism for recognizing outstanding papers, it seems like we could break free of the existing system.

A Popular Solution
One popular solution to do this is a combination of Reddit and the ArXiv for academics. Researchers can post their papers online and then other people can leave comments and reviews of the paper. Everyone has a reputation score and the influence of one’s comments depends on their reputation. Maybe papers can get assigned scores, so anyone can score the paper, but the weight of their score depends on their reputation. That way, on my CV I can write down all of my papers along with their scores, so that others can quickly glance at my CV and get an idea of how important my research is. This is the basic idea but obviously there are a lot of details so that one cannot game the system. I’ve spent some time thinking about this and I think that if you implement it carefully in enough you can make it work. Timothy Gowers also seems to think so and he has thought about many of the details. If you are interested, please read his blog post, here.

One of the comments left on the Timothy Gowers’ blog post is that we might not want to turn life into a game, where reputation points mean everything. I really agree with this; some black-box is calculating my reputation on this website and the score output by this has serious consequences on my life in terms of career opportunities, etc. It makes academic life too much like a game, where everything I am trying to do revolves around increasing scores on my papers and increasing my reputation. So while I still think the system could work, it may not be what all academics want.

A less popular proposal
Gowers briefly talks about another idea, or at least an extension to his existing proposal that I think merits some additional discussion. The idea is this: anyone can start, edit, and curate their own online journal about whatever they want. They assemble a team of reviewers, who could be peers, friends, collaborators, or really anyone else they know. The editor of a journal and the team of reviewers is public information, and their reputation (not necessarily based on a scoring system) is what helps determine the quality of the journal. When I write a paper, I can submit it to one of these online journals, where it will go through the peer-review process, and possibly be accepted. Submissions and reviews can potentially be done anonymously, to allow for double-blind reviewing. Acceptance into someone’s online journal is a stamp of approval of a paper, and on my CV I would list which online journals my papers were accepted into. As in the other system, once a paper is accepted somewhere, maybe anyone should be free to comment on and score it.

There are several ways this system can account for journal quality/impact factor. A simple one is to use the editor’s and reviewers’ reputations as a proxy for the quality of the journal. Another is to allow journals to have reputation scores, based on the scores of that journal’s papers. This second solutions presents a startup problem, but I think you could bootstrap by using the first solution until the journal has a substantial number of articles. Also note that this same problem arises when I want to start a real journal. Again there are some details that need to be worked out but I do believe this sort of system could be made to work.

As a sort of aside, Journal of Machine Learning Research (JMLR) is an example of some of these ideas at work. The journal is open, providing free online access of all of its articles, and it still has a fairly high impact factor. In 2004, apparently it had the second ISI impact factor of any computer science journal (source). This small-scale experiment suggests that this sort of idea might actually work.

In Conclusion
If anyone reads this, I’d be interested to know what you think about these proposals. Do you see any serious complications/problems? Do you have any alternative proposals?

I’m a little behind on this one but a couple of weeks ago Mr. Yuk made the 11 hour drive down to Wilmington, North Carolina to compete in Fall Easterns, a highly competitive tournament featuring some of the best teams in the Atlantic Coast and Ohio Valley regions. The tournament got a lot of coverage on skydmagazine (see here, here and here) and here is the official results from score reporter.

We came in seeded somewhere in the bottom 4 out of 16 teams (I heard that they didn’t snake the last round of the pools, so it’s not obvious exactly what seed we are) and finished 4th over all, losing the 3rd place game by forfeit only because we had to start driving home. Here’s a quick game by game recap:

1. University of Delaware — After getting into Wilmington after 3am the night before, we clearly weren’t ready to start playing at 9am and this showed as we quickly went down 7-1 against Delaware. It was incredibly windy (20-30 mph winds) and our offense couldn’t seem to break their zone. Meanwhile they adopted a “punt and play-D” strategy that seemed to work quite well as we gave them great field position over and over, allowing them to convert easy scores. We came out a lot more fired up in the second half, but had already dug ourselves into too deep of a hole to win this one. Delaware wins 15-7.
2. University of Virginia — If it’s at all possible, winds seemed to increase for the second round of the day, and this lead to a really sloppy game. However we came out strong. We adapted to the wind, playing the same style of field position game that was successful for Delaware against us and this allowed our O-line to convert. At the same time our D-line’s zone generated turn after turn, and we accumulated a couple of breaks. CMU wins 6-10 on hard cap. Some video footage of this game is here.
3. William and Mary — We played W&M at Hucks of the Hilltops a few weeks before Easterns (I wasn’t there hence no recap here), and lost to them on universe point after being up for most of the game. Coming off a big win against UVa, we were hungry to bring it to these guys. We started off really well, converting our O-points and punching in a couple of breaks, eventually taking half, but afterwards we started to slip. If I remember correctly soft-cap went on when we were up 12-9 and W&M broke over and over again to tie it at 13s and bring it to universe. They would come out ahead again, leaving us on one hand dejected, but on the other, excited to play regional rival Ohio State in our cross over game. William and Mary wins 14-13.
4. Ohio State — OSU knocked us out of regionals last year in a very close game, and our veterans were looking forward to facing them in the crossover game. To start out, I think they were playing a lot of their tryout players, but we took a pretty substantial lead, forcing them to put in their veterans for the second half. We kept bringing the pressure and came out with a comfortable win and a berth into the Championship bracket on sunday. CMU wins 14-8. Footage of this game is here
5. University of North Carolina (Chapel Hill) — Our first game on Sunday was the quarterfinal matchup against UNC. As we did on Saturday morning, we came out slow, allowing UNC to take half 8-3. The second half however, was a completely different story. We generated break after break, with big defensive plays, winning the half 10-7, but losing the game 15-13. UNC is a very good team (they ended up winning the tournament) and I was very impressed with how we played, particularly in the second half. Footage of this game is here
6. North Carolina State — Losing in quarterfinals, we dropped into the 3rd place bracket. I don’t remember too much of this game, except that we had a lot of momentum from the UNC game and kept rolling. CMU wins 13-9.
7. University of South Carolina — The Skyd report from day 1 suggested that USC was the story of the tournament, because of their unexpected success, despite our similarly unexpected success. As a result, we thought of this matchup as a way to prove that we were just as memorable as they were. I don’t remember too many details of this game either, except that it wasn’t ever particularly close. CMU wins 15-7, earning us a berth in the 3rd place game.
8. University of North Carolina Wilmington — The third place game was set to start at 3:45. With a 11 hour car ride in front of us, we simply couldn’t push off leaving for a couple more hours and were forced to forfeit to UNCW.

For the most part, we played much better than I expected. This made me really excited to be a part of Mr. Yuk this year and I’m hopeful that we make some noise in the spring and going into the college series. The win against OSU was especially satisfying because we’re going to see them again at regionals and it shows that we can compete against some of the better teams in our region. Our success has also motivated me to do work over the winter, so that I can contribute in the spring.

On the positive side, for the most part our offense was silky-smooth. In the USC game they only had three turnovers and gave up no breaks. I, in part, expected this, as we only graduated one O-line player, and two of our starting handlers have returned from serious injuries, but it was definitely reassuring to see. It’s really nice to play for a team where you can count on the O-line to score. Our D-line, after losing several seniors, also impressed me for the most part. Holes were filled by rookies who really stepped up, and despite our lack of offensive-minded players, we managed to convert after generating turnovers. We can certainly improve in this area, because it was clear that our defensive offense was run by three or four players, and most everyone else didn’t really have an idea of what to do after the turnover.

Of course there are many things we need to work on. Most patently obvious is that we need to start playing when games actually start. We can’t afford to give up insane leads in the first half as we did against Delaware and UNC. I don’t really know how to fix this, because it is mostly a mental thing and something that I don’t know how to teach people. More tactically, William and Mary discovered that they could keep hitting up-the-line dump cuts, and this helped the go on their run towards the end of the game. UNC broke our marks over and over again, resulting in a lot of fairly easy scores. These are two sort of little details that will haunt us in the spring if we don’t fix them.

Fall Easterns wrapped up our fall season and hopefully gives us a lot of momentum as we go indoors for the winter. Our next tournament will be sometime in late February or March. I’m really looking forward to an exciting spring season.

This past weekend, Mr. Yuk travelled to Gambier, Ohio for our first tournament of the 2011-2012 season. With just a few practices under our belts and since we hadn’t made any cuts yet, we went as a split squad, with two fairly balanced teams. I’m still injured, but I hung out mostly with CMU X, so I didn’t get to see too much of the other half of our team. This means that this recap will be predominantly about the squad I watched.

Results-wise, both teams did really well. Both teams went 4-3 on the weekend and X took 3rd place only behind the two Penn State teams. Y did a bit worse on Saturday, so they ended up in the 7th place bracket on Sunday. They won out the bracket to finish 7th out of 12 teams.

As I mentioned, this was a tryout tournament, so we focused on the basics, namely cutting, dump-swing, and hard man defense. While there were a lot of things that needed fixing, I was very impressed with how the rookies and younger players performed and, in particular, how so many players improved tremendously over the course of the weekend.

My squad did several things really well. Specifically, I liked how many of our rookies played within themselves on offense, and decided to dump the disc to veteran handlers rather than make tougher upfield throws. I also really liked many aspects of our handler defense; we earned several turnovers because we prevented easy resets. Lastly, our upfield cutting looked pretty decent (especially for a first tournament) with good timing on the cuts and good cutting structure (hard deep, hard underneath). This was especially true of our veteran players although our rookies did a decent job of this as well.

That being said, there were definitely many things that we need to work on. Most salient was conditioning. We played five games on Saturday and by the third, it was obvious that we were getting fatigued. Cuts were slower and less frequent, defense let up, and we made more and more mental mistakes. This is a huge problem, especially as we have been a team that wins games on defense with our legs. Of course it is early on the season so I’m confident that we’ll build our endurance and be prepared come spring. Other things that could be worked on include: defending up-line dump cuts, general downfield defense (positioning, footwork, intensity), reading discs (especially deep throws), and as always throwing and catching.

Lastly, I was really happy that we had many close games that we managed to win. X had close games against both Case Western teams (one of them ended in universe point) and another universe point win against Michigan State while Y had a u-point win against Akron and a u-point loss to Case. We also had a couple of come-from-behind wins that were really exciting. Last year, we weren’t very good at finishing close games or fighting while behind, so I was really excited about these results.

We’ll be making our first round of cuts soon, and then taking the A-team tryouts to Hucks off the Hilltop in October. I’m looking forward to a great (hopefully successful) season!

Yesterday I had a discussion with two of my friends about why we, as researchers, choose to work on the problems we work on. In statistics and machine learning, and maybe more generally in computer science, this can be a pretty interesting question. Do we work on problems purely for personal interest? Or do we require that problems have some well-thought-out practical application?

This is a philosophical question that I think is fairly unique to computer science and machine learning. In disciplines like biology, there is no question that applications are necessary motivation for a research problem. On the other hand, in pure math, it seems like applications are independent of problem choice; it’s nice if research is practice in nature, but it’s not necessary. CS and ML research is interesting, because in many respects it is only loosely motivated by application, and often it is more about developing theoretical results rather than demonstrating practicality. This is particularly true of the research that I have been doing recently in statistical machine learning.

The discussion went something like this: person A advocates working on things because they are interesting to him, while person B advocates having some grounding in reality before delving into a research problem. Person B at first claimed that the problems are interesting to the academic community only if they are practically motivated. As a counter, person B mentions the centuries of mathematical results (i.e. number theory, etc.) that only years later become “useful.” Clearly, these researchers were motivated by interest. Person B then asked why person A worked at all, and person B responded that he enjoys the problems he works on, mostly because they are interesting. The three of us then started talking about this in more detail, with focus on statistical machine learning problems.

Yesterday, we didn’t reach a conclusion on what was the right thing to do. I was tending toward choose problems based on interest. Assuming you have some sense of what has been going on in the community, your view on what is interesting and what isn’t should at least somewhat match the view of the community. If you feel a problem is interesting, my bets are that many other people feel that way, and for this reason it is a reasonable problem to work on.

I chatted with person B again today about this subject, and he brought up two interesting points. First, it is unlikely that you will get funding to work on problems that are not practically motivated. While this is probably true later in life, graduate school is exactly for this purpose. You get paid to work on the things that you find interesting, it doesn’t matter whether they’re well-motivated or not. Even later on, I think there is an art to making your work sound relevant and convincing people that it is worth their funding, and even with grants, I believe you have some flexibility to work on problems that interest you.

His second point was that working on obscure problems that no one cares about is not productive, even if you find it personally interesting. For example, if you take a well-known problem, modify it slightly, and rederive results for the modification, it isn’t interesting unless the modification you made is well motivated. I agreed with him on this point; I don’t really consider it research to walk through calculations for subtly different problems. There has to be something novel to the modified problem. On the other hand, if you took an obscure problem, and came up with some elegant, novel solution, that would be interesting and useful. Additionally, if you worked on an obscure problem, but came up with intermediary results that could be applicable elsewhere (for example new concentration inequalities), I would also say that is useful.

So after thinking about this for a couple of days here is where I stand. A research problem is worth working on (read: I would work on a problem) if: 1) you find it interesting, 2) it is practically motivated, 3) you believe that other researchers will be interested in your results (whether or not is it practically motivated), 4) You believe there are intermediary consequences, lemmas, or results that will be widely useful, or 5) You believe there is something elegant about the problem and its solution (i.e. the solution is not mechanical). To me (1) is necessary, and at least one of two through five are required.

My first trip to the club series (on a real club team) started this weekend as Dire Wolf travelled to New Jersey for the sectional tournament. Just before leaving on Friday afternoon, I saw a physical therapist and was diagnosed with shoulder impingement, so I didn’t end up playing, which was unfortunate. Nevertheless, Dire Wolf took care of business, finishing 3rd overall and qualifying for regionals.

We came into the tournament seeded second in a 5-team pool, behind Oakland. We were really looking forward to games against Oakland (Pittsburgh rivals), Southpaw (Philly elite team) and Bear Proof, who beat us in a close game at NY invite. Southpaw and Bear Proof were both in the other pool, so on saturday we took care of business with the lower seeded teams (beating Clockwork Orange 13-4, Misogyny 13-6, and Hypnotoad 13-4). For the last round of the game we faced Oakland for the fourth time this season.

Oakland (Game 1): Dire Wolf pulled to start, and immediately broke to take a quick 1-0 lead. We traded points to 4s, where oakland eventually broke back, and finally took half 7-5. In the second half, Oakland opened with a string of breaks and eventually beat us convincingly, with a score of 13-6. Wolf played really well in the first half, forcing errant throws and a lot of 50-50 balls from Oakland, but we had a hard time earning turnovers off of these (they had at least two forced throws that could have easily resulted in turnovers). Even still, we generated turnovers, but failed to convert once we got the disc. After Oakland ironed out their kinks, they played a much more consistent game, and we started making more mistakes on offense, allowing them to run away with the game. With this loss, we finished second in our pool and qualified for semifinals on sunday.

Southpaw: Again we came out strong, earning a quick break and taking a 2-1 lead. Southpaw broke back after a couple of traded points to tie the game at 4s with Dire Wolf on D. Stifling dump defense by Greeno and Skylar resulting in a high-stall forced throw and a turnover and we broke again with a big huck from Tad to Karl for the score. Southpaw came back strong and a couple of miscues by our offense resulted in two breaks for Southpaw, to take half 8-6. After half, though we didn’t roll over like we did in the Oakland game, but managed to get back on serve at 9s capitalizing on some of their miscues. Unfortunately, that’s where the journey ends, as Southpaw got their act together and broke several times to win 15-10.

Southpaw is supposed to be one of the stronger teams in our region, and while we knew we could compete with them, I think we were all pretty excited about this result. I hope we play them again at regionals.

Bear Proof: We dropped into the third place game where we played Bear Proof, from central PA. I don’t have too much to say about this game. Hungry for revenge from NY invite, we came out really strong and went up 5-1 right away. We took half 8-2 and eventually took the game to 13-4, game to 15. At this point we knew we had pretty much won the game but it would be nice to bring it home convincingly. Unfortunately, we let bear proof run off a string of breaks and bring the game to 13-7, with the soft cap on. The last point was epically long with a slew of turnovers by both teams that ended in a bear proof score, but the hard cap gave us the win 13-8. Again we came out really strong, but slowed down in the second half, and if the game were closer to begin with, we would have let Bear Proof back into the game towards the end. Going from 13-4 to 13-8 is pretty much unacceptable, and a better team would have punished us for slipping in the second half. At any rate, with this victory, we proceeded to the second place game, a rematch against Oakland Ultimate.

Oakland (Game 2): The theme of the weekend for Dire Wolf was a strong start to close games. The same is true of this game, with us earning a break and making it 2-1 right away. We traded to 4-3, when Oakland ran off a string of breaks to take half at 5-8. Unlike the last time, we didn’t roll over after half, score our O point and the next to D points to make it 9-8. However, once Oakland converted their O point, the rolled off a couple of breaks and eventually won 15-11. This game got pretty out of hand, with a lot of arguing, aggressive marks and fouls, and a lot of disc spiking. However, it was also really exciting to watch, as it stayed pretty close throughout, and the lead shifted back and forth a couple of times. Both teams played well, and if we can stay focused for the entire game, I think we could come away with a win against these guys.

Closing Thoughts: I think the theme of this weekend is focus and consistency. In all of the games I talked about, we gave up a lot of points (and often the game) really late in the game. We’ve proven that we can play with some of the best teams in our region, but we need to be able to do that for the whole game, not just for the first half or first three quarters. I don’t believe this is an endurance issue, but rather an lack of mental focus towards the end of the game, as a lot of our turnovers seem to come from bad decisions and mistakes rather than fatigue. If we can figure out how to stay consistent for a duration of a game (and really the entire tournament) then I truly believe that we have a shot at nationals. It’s going to be tough, but I definitely think it’s possible.