Identifying Research Value

[justinhart]

This year, a few times I have been approached to talk about the research value of RoboCup@Home. In particular, I have been asked to explain the research value in various venues, and to try to attract HRI researchers to our league.

I think that the best way to do this is to identify the concrete research problems that are addressed in each task. Therefore, I would propose that on each task in the rulebook, we identify the core research problem that is addressed (which should be a current, open problem). This should be a problem that we could easily envision a resulting publication for, if the team comes up with a novel solution. The research problem would be placed in the introduction of each task, in bold, in a black outlined box.

For the "I Want This" task, it would look something like this.

Research Problem: Generation of legible gaze and pointing gestures, interpretation of human gaze and pointing gestures.

[mnegretev]

I agree with @justinhart, but I think this is already done (may be not well done) since every task says the focus of the task, which is supposed to be the open problem to solve. Also teams are encouraged to include in their TDP the novelty of their approach.

I think what can be done to increase the scientific relevance of the league is to evaluate TDPs as the thing they are: papers. Usually, when a reviewer evaluates a paper focuses a lot in the contribution considering the current state of the art.
Thus, may be should be mandatory for teams to include in the TDP a good review of the literature and emphasize the contribution with respect to the state of the art, not only with respect to the previous team work. If this is done, it could ease the writing of a future paper.

[justinhart]

I don't think that this is already accomplished. Could you, right now, identify an appropriate conference to publish results for every task in this rulebook?

[mnegretev]

I think there are a lot of possible conferences and journals (ICRA, IROS, Autonomous Robots and Systems, Intelligent Service Robotics), but ok, may be the focus of each task should be rephrased.
Anyway, reviewing the state of the art and emphasizing contribution is something that is done in every scientific paper. Making mandatory (not only encouraged) for all teams to include the contribution with respect to the state of the art could also help to increase the scientific relevance. Together with the rephrasing of the task focus to better state the open problem to solve.

[justinhart]

I can totally agree with putting some sort of statement of the research plan in the TDP, but I think we're in agreement here. You could go to each task and say:

Task: I want this
Problem: Generation & Interpretation of Gaze & Point Gestures
Venue: Ro-Man, HRI, IJSR

The idea here is to drive a metric where participation in the league has a clear connection to papers. What I'd really like to see out of this is something where maybe we all agree to submit to the same conference and/or write a paper as a league about the task solved in each stage. For instance, I'd really like to propose that we get a group who collaborate on a paper that goes to Ro-Man or HRI this year for "I Want This."

[kyordhel]

Some comments here

• Messing with TDPs is extremely complex. Turn them into actual papers (we did) and you'll get a lot of teams with awesome research that can't get a robot going, while good teams (finalists) with more technical approach will be sorted out. This also impacts directly teams formed mostly with undergrads. Peek a look to the winners in last 3 years and they aren't necessarily the tip of the spear in SotA research.

• Till 2018 all tests had a foci, although nobody really paid attention to that. We tried to standardize testing to produce benchmarking and the outcome wasn't good. We are changing our approach to a goal-driven one and for 2019 we care little on how people solved the tasks as long as the task is solved. IMHO, that's terrible in terms of producing data for scientific publishing and team collaboration in research. However, I'm quite sure it will push innovation.

• The scientific community values little to nothing application. Most of the techniques and skills we use are considered as solved or surpassed since a decade ago in lab conditions. Mount that thing into a robot and coordinate everything and reviewers will tell you: that's an engineering problem, is not science or buy better hardware or your experiments are not repeatable. In all cases your paper is most likely to get rejected. We try to deal with real-world condition as much as we can. In this aspect, I think @Home is closer to industry than to academia.

• A robot can only run algorithms that would fit in a laptop. That prevents us from using the most advanced techniques until they are mature enough to fit. We already tried external computing and found it unreliable. Further, our people needs expertise in several areas to merge everything together, meaning that the learning curves are stepper and much more effort is required to keep updated.

With all that in mind, and after several years of witnessing failed attempts of taking @Home closer to the conference world, I'd say: pork it! The only viable solution I see is to build our own community of experts, stop depending on Elsevier and Springer, and peer-review our real-world solutions aiming to take science out of its golden cage of uselessness and start presenting it as the world-changing and goal-driven gorgon it is.

[justinhart]

Just today the comment was made on the thread about the laptop backpacks that the higher-ups would like to see more cooperation between teams, and more papers coming out of the league. This could become a route to making that happen.

What if we just give it a try with the "I Want That" task. We get volunteers from the league and try to do a league-wide publication. I think that this could be a vector to the kind of research ROI that they're looking for.

[MatthijsBurgh]

In my opinion to get research and RoboCup more aligned is to have a road map. Because if we keep jumping from left to right, how should anyone incorporate his/hers research with RoboCup.

[justinhart]

Propose tasks that your group finds interesting, but where you're not so far ahead as to make the competition unfair.

[MatthijsBurgh]

What about having a roadmap is unfair for the competition?

[justinhart]

You misread my comment. I agreed with you and suggested that a short- term solution is for teams to suggest tasks.

[kyordhel]

@justinhart I've this paper this paper with a roadmap for NLP targeting GPSR. Please give it a look.

@mnegretev Please check with Jesus if you can come up with something similar for Navigation. It would be nice if you share the outcome with us and send it to the RoboCup symposium.

@MatthijsBurgh TU/e unified world mode looks like a good cornerstone for a roadmap on common knowledge sharing. There is a good advance for manipulation with KnowRob on the how to, but there is no deep Analysis on what is important for a robot to know in order to understand its environment. This roadmap should be related with the progress of the others but on the side of abstraction. Please give it a thought.

We require volunteers for a roadmap on Manipulation and all branches of Computer Vision (objects, people, poses, activities, reading, etc)