SESSION 3 BUILDING YOUR RESEARCH PROGRAM Nancy Leveson, University of California at Irvine; Mary Vernon, University of Wisconsin; Vijaya Ramachandran, University of Texas. [Note: the introduction of the panel was not recorded.] We are all going to talk about building a research program because we are all in different areas. I'm going to take the lead on publications because I am editor-in-chief of a journal and have a lot of opinions about it.---Laughter--. Mary is going to talk a little bit about balancing theory experimentation, and Vijaya is going to take the lead on discussing getting good students and working with them. Mary is first. My name is Mary Vernon and I'm at the University of Wisconsin - Madison. I started as an Assistant Professor 1983. I was promoted to Associate Professor with tenure in 1989. My research area is computer system performance modeling and the community that I identify with professionally is the ACM Sigmetrics community. I will address the issue of building your research program in five parts: 1) developing a research theme, 2) selecting research problems, 3) general tips, 4) collaborating, and 5) balancing theory vs. experiment. 1) Developing a Research Theme: Looking back, I think that the most important difference between doing research as a graduate student and doing research as a faculty member is thinking in a broader scope about what you are trying to accomplish with your research. In this regard, I recommend identifying a theme for the research that you are interested in doing and/or some guiding principles that will distinguish your work in some area. Perhaps other people will also following similar guiding principles but it's good to have some angle that you feel is important, that will guide your contributions, and that later others can point to and say your work was important partly because it followed those guiding principles. I don't know exactly how to characterize the process of developing a theme but it should involve building on your expertise and developing a vision for the future. When I began my acacemic career, I had expertise in performance modeling and I was interested in performance evaluation of parallel architectures. At the time, analytic modeling techniques had been applied extensively to performance issues in operating systems and computer networks, but not to design trade-offs in computer architecture. The theme that I chose, that worked for me and still guides some of the research problems I work on today, was the theme of developing analytical modeling techniques that are effective for analysis of parallel system design trade-offs. This theme guided the research problems I chose and the kinds of research collaborations I became involved in. The theme was also broad enough to allow me to work on many specific research problems in performance modeling and parallel architectures in a unified way. Another example of a broad theme with an important guiding principle would perhaps be the development of shared-memory parallel architectures that are based on simple hardware. In summary, what you need is a topic and something guiding your work that in the end should increase its impact. Another important point is to make sure you have all the advantages in terms of background and experience for the topic you choose. -- If you are producing good results others may want to also pursue the topic; make sure you can stay at the forefront and produce your share of the key results rapidly. 2) Selecting research problems: On the topic of choosing specific research problems to work on, this is very similar to what you have done as a graduate student, only you can pick more of them because you can pick some that you will work on together with your own graduate students. A good rule of thumb is to choose non-trivial "chapter size" problems. (The results will usually become a chapter in a Ph.D. dissertation and/or a paper to submit for publication.) Perhaps when you are initially developing your your theme, you'll have an initial paper that's a substantial effort -- more than just a chapter, to get the ball rolling. For example, when I started working on analytical modeling techniques the first technique that I developed together with a graduate student was quite an involved effort -- more than just a chapter size problem. But once you get rolling, you can continue to pick the non trivial chapter-size problems that are the important next step in developing the results. In any case, you should always be working on problems that you consider to be important. In my opinion, this is the way to keep up your enthusiasm for the research you are doing. You also want to be sure that others will also care about the results, so it's a good idea to periodically seek feedback about how others perceive the results. It's not necessary to continuously ask people whether what you are working on is important; hopefully you have developed some sense about good research by now. However, if you find that others don't view your results as significant, then you should take a step back and think about the problems you are addressing and/or whether you are explaining the context and results in a way that clearly conveys the important points. Perhaps you can reorient the problems you are working on, or explain the work more clearly. Perhaps you will need to wrap up the topic and choose the next problems more carefully. One way to select problems that are important is to choose problems that are in vogue that other people are also working on and/or care very much about. Alternatively, you can look for new directions -- something that is missing from what's currently being studied -- if you produce results that show it is important, other people will find this very interesting and worthwhile. Yet again, you might find contradictory results in the literature and instead of arguing the issue, do some analysis that shows why the contradictions exist -- perhaps there are subtle or not-so-subtle differences in the underlying assumptions that you can expose to explain the contradictions, and this can lead to a very important contribution. These are some angles to look for. I think you probably already know this from doing your dissertation research, so it's probably more useful to focus on what I said above about developing a theme to unify the problems you will work on -- and to keep the bigger picture in mind as you do your research. 3) General Tips: First, good papers have impact and get recognized. (This wasn't so obvious to me when I started out.) The first implication is that it's fine to spend quite a while working on something important. If it takes you a year to develop something important it's ok because once the results are out there your effort will be widely recognized. Also, it's good to wait until you've written the paper before you do too much advertising of what you are working on. The long term impact comes from the papers you produce. Second, it's important to publicize your work. One way to do this is to develop a mailing list of colleagues who may be interested in your work, and to mail them your technical reports as you produce them. Frequently giving talks at other research institutions is also a good idea. This echoes what you heard in the first panel about how to get tenure, but it is also part of a longer term goal of developing your research visibility. Starting in your second and third year you can develop a habit of giving talks at other places, particularly when you are going to be in the area and it doesn't cost much for the other institution to invite you, or when it's inexpensive enough for you to cover the costs yourself. It's useful both for the feedback you get and for publicizing your research results. A colleague of mine had a good way of looking at this, which is that in the end your reputation is the product of the significance of the work you do and the amount of hype you give it. Both factors in this product increase your impact. So you don't want to do one without the other; you need both to succeed. I think this is an important model to keep in mind. Third, it doesn't hurt to publicize your institution. For example, if you develop a system that you are going to publicize or distribute, give it a name that includes your institution's name. It doesn't hurt when you do something that has impact to help some of the recognition reflect on your institution. When you come up for merit reviews and promotions, if you have visibly increased the reputation of your department or university that's definitely not going to hurt. It's also not going to carry you by itself, but it certainly is a useful thing to do along the way. Finally, it will help your own research if you look for the positive in other people's research. I think this is a point that frequently gets missed. If you first look at a paper and you don't think it has much to offer you, or you think there are lots of flaws in it, looking at what it does have to offer you is important. Also, acknowledging other peoples good work always improves your own reputation in my opinion. 4) Something brief comments about collaboration: I think both collaboration and individual effort are important. You should look for opportunities to do both. When collaborating with colleagues, written notes about ideas that are discussed help everyone remember what the key points are and make sure that contributions by individual members of the group don't get overlooked. So it's good to develop a habit of individuals writing up their key ideas after meetings. If there are any questions later about who deserves credit for various ideas, everyone can look back at the notes and remember. Don't forget to put your name on your notes, and encourage others to write up their ideas as well. It's also good to discuss coauthorship and lead authorship issues up front. Even when you can't settle all of the points up front, you ought to establish a mechanism for how it's going to be decided. This maybe feels a little uncomfortable the first time you do it but it's generally a reasonable thing to do. Along the same lines, once the work is out you should have some discussion about how all the authors will help to publicize the work. You all benefit from having it publicized and you each benefit from participating in that. So again it's just a good general rule of thumb helps everybody. 5) My last remarks concern balancing theory and experiment: I don't have a lot to say on this topic. If you have specific questions I will be happy to take a stab at answering them. Very briefly, I think you should work on what you are good at. I also think that if you are doing experimental work it's really important to use and/or develop abstractions that help you interpret your results. In fact, the abstractions you develop could be important in their own right. In any case, I see lots of experimental work where the results are interpreted in a particular way without any data to back up the claims. Perhaps it's just the author's intuition that led to the interpretation; perhaps the interpretation is incorrect. Thoroughly questioning the reasons for the results you obtain will help make sure your interpretation is solid. On the other hand, if your work is more theoretical, then I think you always need to be clear about the origins of the problems you work on and about the applications of your results. Test out whether or not the results you get really have some practical applicability every once in a while. Finally, I think some of the nicest research involves developing formal methods that apply to practical problems in any sub-area of computer science. If you can do this a few times over your career it's really nice. I'm setting my clock -- I talk forever if I don't discipline myself. Let me first tell you about myself. My name is Nancy Leveson. I'm currently at the University of California, Irvine but I'm going up to the University of Washington in one month. Let me tell you about some of the things that I think are important in becoming successful. These are my own tips --- each of us speaking is from our own viewpoints because we are all in different fields and different fields work differently. [Transparency bullet: Start with problems, not solutions] First of all, one of the most important things is to start with problems, not solutions. Too often people read journals, they see solutions, and they try to improve them. I went to a conference once where a man from NASA said: "I've been reading a lot of papers on the Byzantine General's Problem, and they are all very brilliant and very creative but, you know, in 20 years we have never had a Byzantine General's failure at NASA." ---Laughter--- "Could some of you work on the problems we do have?" It seems to me that sometimes we are like the Greek philosopher, Diogenes, who took a lamp in search of truth. Only we take a solution in search of a problem. For example, there are certain people that you know beforehand, for whatever problem, they are going to apply a Petri net to it. Whether it's applicable or not, that's what they are going to use. Another example is that now, for every problem, "object oriented" must solve it -- just stick that title on your solution, and the problem is solved. That's OK if you want to be a pedestrian researcher. But if you want to be a super star, I guess in this context I should say if you want to be the Queen Bee, then what you want is to be the person who comes up with new solutions. The worker bees just keep reusing and refining things. [Transparency: Pick problems because they are important, not because they are easy to solve] My second point is something a famous astrophysicist used to ask his students: Do you choose problems because they are important or do you choose them because you can solve them? I've always picked very hard problems that everyone else ignored. They were too busy copying each other or trying to make epsilon improvements in what others had done. Even if you make only a little headway on a hard problem, you've made a major contribution. For example, there is a technique, called a recovery block, that has been around for 20 years that nobody has ever used because it requires writing an assertion. You also need a recovery cache, but building a recovery cache is simple. Everyone can come up with a recovery cache. There are 227 papers on recovery caches, each one a little bit different and not really much better than the other. Everyone knows we can build a recovery cache. I could assign an undergraduate student to build a recovery cache. But you can't use the technique because you can't write an assertion, so what's the point? We wrote a paper on how you write assertions. It didn't solve the whole problem but at least it attempted to make the technique useable. I have a colleague who, when serving on program committees, talks about "boutique papers." A boutique paper is a paper that does a really nice job solving some really unimportant problem. --Laughter--. We accept them only if we are short of papers. And so remember: Small steps on important problems are better than "yet another way to do X". [Transparency: Understand your problem first] Third, understand your problem first. Determine what are the most important aspects of your problem, not just what everyone else is doing. Read what's been done before -- there is too much reinventing with new names in our field. The latest thing in my field now is "domain analysis." If people called it "application analysis," they would all see that this was trivial stuff we did a long time ago. [Transparency: Determine: (1) what need to know to solve problem, (2) how information can be obtained, (3) how you will determine the problem has been solved] An important point is that the research contribution may be in getting data to understand the problem better. We need to understand our problems first. One of my most widely cited papers contains no original solutions. It's just a detailed and clear exposition of a problem and possible solutions, whether they might work or not, and what are the remaining important problems. Everyone tells me they think it's a wonderful paper. So when you are going to start on a problem, first determine what you need to know to solve the problem and how you can get the information to solve it. Just last week a student came up to me and said she had some problem reports that somebody else collected. She said she wanted to do something with them for her dissertation, and wanted to know what to do with it. That's not the way to work. Collect your own data. Start with a problem, collect the data that solves the problem, don't try and go backward -- you are not going to end up with anything significant otherwise. Next, you need to determine how you are going to show that the problem was actually solved. Too many people forget that last part. They stop with proposing a solution, and they never show that their solution is really any better. Plan a way to validate what you have done in some way other than handwaving. [Transparency: Don't jump on bandwagons] And finally, question what has been done before -- the underlying assumptions that everyone accepts. Another paper of mine is widely cited by people who love it and by those who hate it (which is just fine as long as they cite it) ---Laughter---. Basically it is on a technique that people for 20 years have been promoting and saying was going to make ultra-high reliability software. It was being used in nuclear power plants and commercial aircraft all over the world. The justification of its used was based on an assumption that I thought was loony tunes. A colleague and I looked at each other and said these people can't be serious. So we did a carefully designed experiment that showed that the assumptions just didn't hold. Don't jump on band wagons. Don't assume because everyone is doing something one way that it's the only way. Don't follow the crowd. It's sort of like the gold rush --- once you've heard that gold has been discovered in the Yukon and you get up there, it's all been panned out. Dare to strike out on your own on problems that you think are important. Remember, chose problems because they are important, try solutions because they fit the problem not the opposite. Don't define the problem to fit the solution. And also question the assumptions in your field, they may not be right. It's amazing how many times people just follow each and just keep making the same assumptions. The people who come up with new paradigms and new ways of looking at things are the people who challenge and question the underlying assumptions that everyone has accepted. There are just a couple of more topics that I will cover very quickly. The first is whether you should work in several areas or just one before tenure. It's OK to change area after your dissertation, but do so early or late in the tenure period. Don't do it in the middle. Try to concentrate and don't dabble in a lot of areas. We see people who don't get tenure because they have one paper in one field that's nice and one paper in a totally different field that's nice. We get letters that say "This person wrote one paper in this field and it is good, but I never saw anything else by him or her." That's what each letter says and unfortunately that's not very helpful. So it's OK to change fields, I changed fields completely. I was in programming languages and formal semantics for my dissertation, which was plagiarized immediately. I never got any credit for it, and I was so disgusted with the whole thing I totally changed fields. Then I changed into a different field because I liked the people better in it. But the fields were related enough that everyone in the second field could appreciate the work I had done in the other one. What you want to do is choose projects, as Mary said, that build on each other so that letter writers can say you have a series if results that build on each other and that advance the field. One last point about going beyond your thesis. Too many people spend 20 years, or until they stop working entirely and become deadwood, working on minor variations of their dissertation. I happen to think that's incredibly boring. I can't imagine doing that, but a lot of people do. It's OK to extend what you are doing but you need to make sure that you are not just regurgitating or making epsilon changes in what you have done. And finally (this is a really weird suggestion, this is my wierdo one of the day), don't name your technique or language. I must admit that I finally have done this because I just wrote a paper and couldn't keep referring throughout the paper to "the language" --- it got so awkward that I finally put a name on it. The problem is you fall in love with something you name. I've got this theory that you will never stop working on it -- you'll end up making tools for it and selling it forever. So even though I finally did name something, to counteract that I made all my graduate students work on how to make a better language and to determine what is wrong with this one. They are all writing dissertations on how to do much better than the thing we just did. I also gave it a horrible name that no one will ever remember -- you know, sort of some letters put together, totally non-mnemonic and --Laughter--- And so be careful don't don't end up falling in love with your creations because your job as a researcher is to continue to improve your solutions and to come up with new ones. Our job is to take small steps, determine what we have learned from each one, determine what are the good things about it that work and what the things that need to be improved. Then go on the the next problem and the next solution. APPLAUSE , APPLAUSE I am Vijaya Ramachandran and I am an associate professor at the University of Texas at Austin. I graduated in 1983 from Princeton University and I was an assistant professor from '83 to '88 at the University of Illinois at Urbana. Since '89 I have been an associate professor at the University of Texas at Austin. My area of research is theoretical computer science, especially algorithm design and parallel computation. One advantage of being the third person to talk is that most of the topics have been covered by the other two speakers. So I will be fairly brief and get us back on track in terms of time. On the choice of a research program, I think it is wise to use your thesis work as your starting point when you are getting started on your research. You spent the last several years just immersing yourself in your thesis work and you have made yourself an expert in this area. So you want to build from your strength. It will take you considerably more time to build a track record if you make a dramatic switch and change to a different area of research. So I think it is very wise to use your thesis as a starting point. But do not just build on top of your thesis work. You are probably quite sick of the thesis work anyway --- Laughter-- but you do want to build on techniques and insights into the area that you acquired during your thesis research. You should make use of these and branch out using perhaps the same techniques in a different field or in a related field. Hence you can quickly build on what you already have as a base. Now I move to a second point which I think has been addressed by many people both in the first session and this session. It is wise to concentrate on one area before tenure because you will be evaluated by senior people who will compare you with the best people in the field and you want to make a favorable impression on them. You can do that better if you have a solid record of accomplishment in one area. It also allows you to gain visibility and it gives you the opportunity to come up with collaborators. Let me talk a little bit about collaborators since the other two speakers did not speak too much on this topic. I think having collaborators is very useful. Talking and working with other people gives you the opportunity to come up with new ideas that you probably would not have come up on your own. It is always nice to have collaborators and whenever the opportunity presents itself you should try to make use of it. Especially useful are collaborators who are in other institutions because they also give you valuable contacts, allow you to gain visibility and help you at the time when tenure letters are written. If you have collaborators in other institutions it will also reflect well on your stature in the field. However I should also point out that it is very very nice to have some singly-authored papers when you come up for tenure. When you have collaborators one thing that is always a little sticky is the question of how much of the work was done by each of the contributing authors. If some of your collaborators are writing your tenure letters they will indicate clearly if you made a substantial contribution to the paper. But quite often, when a person comes up for tenure and does not have any singly-authored paper, the question comes up on how much of the research in multi-authored papers was actually performed by the person coming up for tenure. So it is very helpful to have some singly-authored papers. If you do not have singly-authored papers it is wise to have a variety of collaborators. And avoid having your thesis advisor as a dominant collaborator after you graduate. You can have certainly a few co-authored papers if you have a good working relationship with your thesis advisor and you can certainly continue that collaboration but do not do that to the exclusion of other collaborators. It is important to establish a research record independent of your advisor. Quality before quantity,-- I think everyone has mentioned that, and also moving conference papers to journals, so I will not talk about these topics. Let me talk about the last topic on my list, which is the question of picking the area of research to branch into. Of course this is largely determinedby your own interest but you can decide to pursue one of two tracks. One possible track is to get into a "hot" area. Most research areas have hot areas that change fairly frequently. For instance, in theory at the time when I was graduating, VLSI was a hot area, and sometime back on-line algorithms was a hot area. This is not just a fad. Usually these hot areas are motivated by current issues of practical importance and so it is worthwhile if you can make a contribution in such an area. Now one `plus' about working in a hot area is that it is possible that you may come up with a significant result fairly quickly. This is because, as a new area, many topics are unresearched or at least not very widely researched, so it is possible that a dramatic breakthrough could be accomplished merely because other people have just not got to it. And in that sense there is a potential for great rewards in working in a hot area. However you have to be careful and make sure that you are well-connected within the research community, because typically there are a lot of people working in such an area and the same result is likely to get discovered independently by several groups of people. If you are not tied into the community you may not know that other people have come up with the same result until you see it published in a conference, and by then it may be too late to get credit for your work. So if you work in a hot area be sure that you know the people who are working in the area, and that you are in touch with them so that you are aware of results as and when they appear. Now the other track to take is to work in a well-established area and I think Nancy talked about that. Even a little improvement in a well-established area could reflect well on your research capabilities because many people have looked at problems in this area and so any improvement is worthwhile. But it is likely to be harder to get results in such an area. So which way you go depends on your own inclinations but keep in mind the pluses and minuses of these two tracks. I think I will stop here. Thank you. APPLAUSE, APPLAUSE, APPLAUSE. In order to have time left for questions, what we are going to do now is each of us is going to say a few words about each of the other three topics and then the others will just comment on what we may have left out. I am currently Editor-in-Chief of IEEE Transactions on Software Engineering. This is not a job I would wish on anyone, but I have learned a lot about being an author. First of all, where should you be publishing? Publish in the best professional journals and conferences, especially IEEE and ACM, that you can. Avoid private journals with low readership and poorly refereed conferences -- you are going to be wasting your time. People will discount your papers in them. There may be reasons to go to workshops and meet people; we will talk about networking later. But don't waste your best papers on them. Magazines are less prestigious than research journals but they are good for broader circulations. Save your best stuff for the research journals. Now, how much and what in each paper. You wouldn't believe it, but people send me their whole master's theses, signature page and all. The signature page is a dead give away that you didn't put a lot of effort into rewriting this --Laughter-- for the journal. Maybe they just think that it proves that somebody has read it and liked it. I don't know. My algorithm is to take the parts of your work that are most innovative and also the parts most in need of feedback. Submit these to a rigorously refereed conference first, the very best one in your field. Use the feedback to improve your work. Add what you have left out and submit it to a journal. Don't submit the same thing; extend it but submit it to a journal as soon as possible. Do not do concurrent submissions --- NO NO NO -- very bad. Odds are you are going to have the same referee on both papers, and they are going to get furious. Let me tell you, they really hate it. Both journals will automatically reject the paper. And the referees are still going to be angry the next time they see the paper. Don't do it, it's not worth it. Also, don't take little pieces and try to get lots of papers, or try to publish epsilon changes in your work; that annoys people too. Err on the side of putting too much in a paper if you have to err. Having a classic paper that's widely cited is better than a bunch of small and, by themselves, insufficient papers. Now, what do refereers look for? I feel like an expert on this because when I started, everything I wrote was rejected. Now I rarely have papers rejected. It may differ in your field so you are going to have to adjust for your own field, but they want you to explain what the problem is and what your approach to the problem is and how it differs from previous solutions. One of the tips that Mary gave me last night in the bar, of course we were drinking coke ---Laughter--- is to look at the papers that you like --- those that you really respect--- that are important in your field and use them as a model of how to write a paper. If you liked it, other people are going to like it. Don't fill your papers with the latest buzzwords. And don't make up new terminology unless it's absolutely needed. It seems to me sometimes that new terminology is escalating faster than new ideas. We don't need new terminology, we need new ideas. Don't oversell. Explain the limitations of your work. I think that's one of the reasons people like my papers --- I say where it's not going to work and that all I've done is make some small steps. They are so relieved from the tedium of hearing everyone say that they have discovered the silver bullet that's going to solve all of our problems that I think they just accept it out of relief. --Laughter-- Finally, go carefully through a realistic example if it's appropriate. Style issues are very important. There are people who submit papers, and we can't get any referees to agree to review them because they are hard to read and take forever to read. They start the paper, and then they put it on their desk and a year later it's still sitting on their desk. Other people submit papers and they are always returned in a few weeks, So don't just blame the referees for slow reviewing -- it may be your fault. So make sure your paper is readable. Don't try to snow people. Easily understood papers are much more likely to be accepted. I've also found the people who have really significant contributions do their very best to explain them in the simplest way possible. The people who really don't have anything significant try and snow people by making it obscure. Don't use non-standard notations if the usual notations will work. It's too hard for people to read. If you use mathematical equations, as most of us do, explain them in English --- do not expect people to get your meaning from a mathematical notation alone. The math notation is there to make any ambiguity in the English part clear. If the referee has to go through complicated mathematical expressions and spend 20 minutes on one line, they will not be favorably disposed to your paper and they will not return it quickly. Organization, readability, and understandability are important. Spend time making your paper as readable and understandable as possible. Ask people to read and comment on it. Volunteer to do the same for them. Even if they are not in your area -- that may be even better since if they can understand it ... Don't submit the paper to a journal until it is the best you can do. If you don't, you are wasting your time -- it will take a long time to be reviewed and it will be rejected, so why bother? Use reviews to your advantage. Maria said "don't be afraid to fail". I would add "learn from your failures." There is nothing wrong with failing as long as you learn from the failures. It hurts to get criticism but it's the only way you learn. I sometimes get reviews back and I read them and say "OH THOSE IDIOTS HOW COULD THEY SAY SUCH STUPID THINGS." I throw them down on the desk and then a few days later when I've calmed down, I go back and say "OK, what did I do wrong?" -----Laughter--- If you assume the referees don't know what they are talking about, you are never going to learn. I had one author recently who had four referees tell him his paper was awful. he came back and he said those referees didn't know what they are talking about, they obviously didn't read the paper carefully. He was so persistent that I sent it out to three more referees who came back with the exact same answer. This time he complained that the referees obviously had never read the references he cited and that they didn't know anything about the topic. Now I happen to know that those referees wrote the papers he cited and that they definitely understood them--Laughter--. There is a Yiddish saying that translates something like: If one person calls thee an ass. ignore him. If two people call thee an ass, consider it carefully. If three people call thee an ass, get thee a saddle. --Laughter, laughter, laughter. Some people are not very tactful in the way that they phrase criticism. It's a lot better to criticize the paper then to criticize the author. It hurts less. I've gotten some reviews back that were just awful, such as "this is the worst trash I've ever seen and the author should be embarrassed to have ever written this." Just ignore the vicious part and learn what you can from the review. If they misunderstood what you said, then you probably weren't clear enough. You can use their misunderstanding to your advantage to improve your paper. Figure out why they misunderstood it and how you can present it differently to make that less likely. Sometimes reviewers will try and impose their views on the subject. They shouldn't be telling you what to write just because your approach to the problem is different than theirs. You can write a rebuttal. Phrase it nicely and say something like: referee B had some very interesting points that were very helpful and useful and I thank him/her very much for that but I don't believe that it is really proper for me to change this section because ... Write it nicely and explain and usually even if the referee does not agree, the editor will. Finally, who should be included as an author. I err on the side of inclusion --- I think that it's better to be known as generous than to have people think you've stolen their ideas or their work. It just doesn't hurt to include people. I have a colleague who describes the problem of who is included as an author in terms of an archaeological dig. When the archeologist goes and gets the money to pay you to dig and he tells you where to dig and then you find something important, his name goes first on the paper. All you were doing was following directions. If he is paying you and he told you how and where to dig but you did it a little differently and in a little different place but in his dig, your name goes first. When you start your own dig without help from him or her, then it's your paper alone. We want to leave time for questions so the only point I think I would like to add is that for your first couple of papers after your dissertation it may be helpful to have a colleague read the paper before you submit it. The colleague might be at your own institution or at another institution -- somebody that you feel comfortable with asking and someone you believe will give you really good feedback on it. This can really be useful. I also have just one comment on what Nancy said regarding the convention of not having simultaneous journal and conference submissions of the same paper. This convention can vary across sub-disciplines. For instance in the theory area, one of the main reasons why people send papers to conferences is the quick turn-around that you get. Your paper gets published in six months and it is usually a rough draft, while papers that go through journal refereeing normally take or two to three years to appear. So it is not uncommon for those people who have a journal version of a paper ready at the time of conference submission to submit it both to a conference and a journal. I'll ask Maria Klawe to reconfirm that. (Maria Klawe confirms this forcefully.) If you are neither in Nancy's area or in theory you may want to ask around to find out what the convention is. (Voices again) The reason we don't like that in my field and in other fields is that you waste the referee's time and the program committees hate it. If they decide they are going to accept 20 papers, and they reject the others, and then all of a sudden you pull yours out, someone else's paper might have been rejected when it mightn't have been. It doesn't seem fair to me to do to the editors, referees, or people running the conference. Let's get on to the last topic we want to talk about before opening this forum for questions -- getting students and working with them. To summarize this discussion, it appears that the convention on simultaneous journal and conference submissions varies across sub-disciplines, and this is something to keep in mind if you are considering this. I will now move on to the last point we want to talk before opening this forum for questions. That is the question of getting good students and working with them. What I have to say is really my personal viewpoint. Here is what I do about getting students and working with them. I usually find the students who do research with me through the graduate courses I teach. During such a course the student has a good opportunity to see whether this is a research area the student wants to work in, and whether the student would be comfortable working with me. And although the latter point is not usually addressed from the faculty member's point of view, I, personally, also like to make sure that the student and I are comfortable working together. I know of several colleagues who do not go through this approach. To recruit students they look at the applications of prospective students. If they find someone who has expressed interest in the research area they work in, they offer them an RA even before the student has arrived on campus. That way they have a student working with them right from the start. That is also a way to get students but I personally do not like this method because it is possible that the student in question turns out to a person who has some pre-conceived notion of what a CS professor should look like ---Laughter---. I do feel that, by and large, people are quite open but you do find a few people, including students, who assume that women are just not capable of doing research in computer science. With my name, especially, people often do not know that I am a female. They just look at my research interests and feel that it is an interesting area, and when they finally see me in person, they look at me as if I am from some other planet __Laughter-- and it is clear that I made a mistake in choosing that student. So for me personally I find that the nicest way to get a student is through a course. That way I am also able to evaluate the student and know f he or she is of high caliber and so on. I should warn you that in the initial stages of working with a student it is a lot of work on your part as a faculty member. Of course it is a lot of work for the student, too, but it is also a lot of work for you because you are training the student in the art of doing research. The student is probably very enthusiastic and bright, but the student still needs to learn how to do research and how to write papers. The first paper you write with the student will be an enormous amount of work on your part because the student really is not familiar with technical writing. It is going to be harder if the student is a foreign student whose is not fluent in English because there is also gong to be the problem of correcting the grammar. So you will be investing a lot of time and effort on the student. You will also invest a lot of work if your joint paper gets accepted and the student is going to make the presentation. You will probably have to review the student's presentation two or three times the first time around to make sure that the presentation is a credit to your work and gets the information across. Although all of this sounds very hard, the reward comes in later on when the student matures and becomes more or less your colleague and your collaborator. Then your rewards are quite substantial. The choice of research topics again I think varies widely across advisors. What I typically do is suggest a few areas and hope the student expresses an interest in one of them in which case I give pointers to the literature. Then again, students vary. Some students are very independent and this is more than enough. They will go around and look at the literature and then come up come back with possible topics and you can pick up from there. Now if it looks as if the student is floundering after several months then I give the student a more concrete problem to work on and then proceed from there. I tend to have meetings in a regular basis because even if we do not have anything tangible going on, simply to monitor the student's interests and progress. Typically I meet with students once a week but if we are working on something exciting I may meet with the student daily. Coauthorship on publications: Initially when the student and I work together, I typically put in a tremendous amount of work and the papers we produce become coauthored papers, with the student and myself as the coauthors. Now as the student gets more senior the student would have learned the ropes and I am basically there to guide the student. The student does the work substantially on her or his own, and at that point I often have the student publish the papers as a singly-authored paper. That is great for the student because she or he has some singly-authored papers on graduation, and this can help quite a bit in getting a good job. This is a somewhat debatable point because even with a senior student I , as an advisor, have certainly contributed to the paper, and could justify my name as a coauthor. This has just been the way that I have operated. I think I will stop here and let the other add these comments. Ok, we should have some time for questions. I'm going to put some comments I have about experimentation in the transcript later. [Added comments: Building does not necessarily equal experimentation. Experimentation implies that a hypothesis is being tested. Too much of so-called experimental research is development. There is nothing wrong with development, but don't delude yourself that it is research. I believe the difference is that with experimentation you are buildng something not so others can use it (that is development), but so that people can use the ideas that you validated by building it.] Mary wanted to quickly say a couple of things about students. I'll just make a few more points about getting and working with good students. I feel very fortunate that I've had a number of very good students over the past several years. In my experience it really helps to give informal seminars periodically at your institution --at least once or twice a year, about the research problems you are working on and your resent results. Also, if you have an opportunity to lead an informal weekly seminar in your area, this can also be a very good way to interact with students that you work with or might work with in the future. In terms of working with students, it's very helpful to think about several stages that you will go through with a student, especially if you haven't done this before. Initially, there is a period where you and the student need to get to know each other and where you need to critically evaluate their research strengths (and weaknesses) so that you can help them select research problems that match their strengths. Even if you've already known the student in a class setting, getting to know them in a research setting takes some time. During this time you need to think carefully about what kinds of problems they can work on productively. You should get them involved in something interesting and worthwhile, but also something that you're fairly certain they can contribute to. Then I think you move into a phase where they are taking more of the initiative and are developing results that will clearly be part of their dissertation research, but they are still getting a lot of direction from you on it. Finally, they gradually require less direction (but still some feedback on what they are doing.) It's nice, for you as well as for the student, to produce publishable results all the way through this process, but you can't always count on it early on. Thus, thinking about the initial stage is really important. You (and the student) should also be willing to cut your losses early if the research relationship is not productive. You're not doing anybody a favor if you try to continue an unproductive relationship too long. Finally, something that colleagues have mentioned to me and I certainly agree with, is that it's important to remember that not all students are as good as you are. It's hard to realize this when you are just starting out your career as a faculty member, but it takes a lot of talents to do research. It's not just a matter of how smart you are. Not all students you meet will have all of the requisite capabilities so this is again part of evaluating the research capabilities in the beginning. My name is ?????? I just graduated from the University of Illinois and accepted a position at Wayne State. A couple of questions that I have you have already addressed but I would like to ask for a more detailed answer. One has to do with coauthoring papers with students. In some cases, you have the idea. This is the simple case. Also cases when students have the idea and you are the sounding board for the ideas are simple. But all the other ones, you know, when you decide whether your name really should go first or second ... I have heard some stories like: never ever put your name first on a paper with a student because it's a bad form in the field. So I'd like to hear about that. And, second, a more serious problem that I have is that on one hand all this tenure evaluation procedures seem to be about how people like your ideas. If you don't get good letters, then you won't get tenure. They need to say that this is a wonderful piece of work, I love it, it's a great contribution. On the other hand, if you are attacking some serious problems, you also get feedback of the kind this is wild stuff, who knows what that really is or I'm really upset about those results, meaning, it's obvious that the research that you did is very significant in some sense but it's not that obvious that everybody agrees that's its a great contribution. So what I am saying is that I see some at least contradiction between the really great ideas that you are addressing in the pre-tenure years and trying to get tenure. The question of the order of names in a coauthored paper, again, is a function of the subarea in which you work. For instance, in the theory area in which I work, the norm is to list your names alphabetically. It is somewhat unusual to have an inversion and if you do have an inversion it is because the lead author did most of the work. So that, in some sense, reduces the problem and people do not normally fight too much about it (though sometimes they do). Now on the other question about whether it is a risk that you take if you work in a controversial area during your tenure-track stage: With this type of research, you may generate work that is both very much appreciated by some people and very much condemned by others. In this case you do run the risk of not having a clear-cut, strong record when you come up for tenure. So I would suggest that it is a somewhat a dangerous track to take before you are tenured. You may want to differ that type of work until after you have received tenure. But if you are absolutely confident that what you are doing is really terrific stuff, then you should go for it. On the topic of ordering of authors, I would echo that alphabetical ordering is a reasonable default rule to work from when all authors have contributed significantly to a paper. However, I also think a lot of credit goes to the faculty member(s) when a paper is jointly authored by students and faculty, so its also worthwhile to consider putting the student name(s) ahead of the faculty name(s) if there's a case to be made that the student's role in the research was significant. (Note that I have a name that's late in the alphabet so alphabetical ordering often puts the student(s) first anyway.) Particularly when a paper contains the results of a student's thesis research and/or they are close to finishing their doctorate, it could be appropriate and important for the student's name to go first, even if this does not follow the alphabetical order and even if the advisor has made a significant contribution in supervising or directing the research. Particularly since significant credit often goes to the advisor, I think it's a good idea to be somewhat generous on this point. You want your students to get deserved credit in the long run. On the second topic I'm not sure I've had any real experience in working in a "controversial area". However, I've worked in an area that people weren't really looking at and didn't necessarily believe in, i.e., that analytical models could produce practically significant results for evaluating system design trade-offs. I produced results that gave new insights and/or uncovered problems that people were previously unaware of, and these research results were viewed as significant. Thus, I think as long as you can demonstrate in some real way that you have produced useful results, it's ok to work on a problem that isn't popular or take a controversial approach. Two questions if there is time. As advisors, do you ever ask for feedback from your students about well you are doing as an advisor? And the second question is I hope that if I take a professors job that I will have bunch students that are a lot smarter than I am. Are there any special problems dealing with students who are a lot smarter than you are?---- Laughter, Laughter, Laughter I got handed this microphone... let's see, do I ever solicit feedback from students on how well I'm advising them? For students who have been out for a few years, I've had some feedback on a few tips that they wish that I had told them when they were starting out. Those were things I didn't know in my first few years as a faculty member were important to tell students; yet somehow I knew them, perhaps from my advisor. And I've used that feedback to give more complete advice to my students as they finish their degrees. [Perhaps the transcript of this workshop will provide much of that information.] In terms of the daily advising and so forth, I think feedback comes just from observing how well the students are doing. I adjust if I don't think I'm giving them enough advice and/or help for them to do well. So, I don't explicitly ask for feedback. The other question -- students who are a lot smarter than me -- this was another point I'd like to make. Your goal should be to have students who are better than you are, and advising such students is terrific. I don't think there's any special problem with doing this. Just give them good problems to work on, good references to start from, and your expertise. Well thanks very much to the panel.