MASH logo
Login | Register

Factory Contest

From MashWiki

Jump to: navigation, search

Contents

Summary

The MASH Factory is an on-line interface on the MASH project website that allows the development and testing of heuristics which help a simulated robot solve goal-planing tasks. The robot perceives its environment as an image generated subjectively from its location, and can at any moment perform one of the three following actions: turn left, turn right, and go forward.

The MASH Factory Contest will award three Nexus 7 Android Tablets (32Gb of memory and 3G connectivity) to the participants who have developed the heuristics leading to the highest performance on a series of reference tasks. These prizes will be awarded at the end of the contest, on Jun 7th, 2013.

You can get additional information about the Factory in the main Wiki of the project, or check the Frequently Asked Questions about the contest.

Contact us by mail at info [at] mash-project.eu for more information.

Tasks and teachers

Several tasks have been defined for the contest, all of which can be solved by appropriately navigating the environment. You can find the list of tasks below.

For each task, we have designed a teacher, which is a hand-crafted strategy which at any moment chooses an action to perform. This teacher is virtually perfect: In each task, it is able to reach the goal without mistake. However, it has access to more information than the images alone. In particular, it knows, precisely, the shape and size of the room, the location of the robot, the location of the flags and obstacles, etc.

Goal of the contest

The goal of the contest is not to write a program able to select the proper action, but to write algorithms that can extract meaningful information from the images perceived by the robot, which in turn may be used by the learning component. More precisely, each participant must write heuristics, each a piece of C++ code, which take as input a series of images in bitmap format, and produces as output a series of values or features.

A learning algorithm, which we have developed, takes all these values as input, and automatically combines them to mimic the teacher's behaviour.

The contestants' goal is to write heuristics such that this learning system performs as correctly as possible.

Performance evaluation

A contestant's heuristics are stored on the MASH platform, and she/he can assess their performance on the tasks without installing any software on her/his computer.

For every task, we have recorded 200 runs. Each one is initialized by randomizing the environment (room size, flag and obstacle locations, etc.) and placing the robot at a random location. Then, the robot performs a mix of random actions and actions selected by the teacher. This ensures that it will visit more areas in the environment than if it was only trying to reach the goal as quickly as possible.

Over such a trajectory, we record the actions chosen by the teacher, even when the robot opts to move randomly.

Half of these runs are used to train the robot, using the contestant's heuristics, and the other half is used to assess the accuracy of the resulting intelligent system. More precisely, the platform computes for each of the three actions how frequently they were correctly chosen by the trained robot. The average of these three, action specific, accuracies is used as the overall performance measure on the task.

Hence, making always the same action will produce the worst possible accuracy of 33%.

Ranking of contestants

All the contestants will be ranked on each task individually using a third set of 100 runs, according to the accuracy the system reached using her/his heuristics.

Three prizes will be awarded:

  • Two for Best Accuracies. The contestants will be ranked for each task according to the accuracy they obtained, and the winners will be the two contestants with the best average ranks.
  • One for Best Contribution. The heuristics of the contestants ranked 3nd to 12th according to their average ranks will be judged by a jury composed of the researchers involved in the project, which will award the prize according to the generality and originality of the heuristics.


Each one of the three winners will be offered a Nexus 7 32Gb with 3G connectivity.

List of Tasks

follow_the_light.png
Task #1: Follow the light

In a rectangular room, the ambient light colour indicates directly which action should be performed: red to turn left, green to turn right, white to go forward.

Teacher strategy:

The teacher simply performs the action according to the color.

Example video


reach_1_flag__SingleRoom_1.png
Task #2: Reach the flag

In a rectangular room, without obstacles, reach a red flag.

Teacher strategy:

If the flag is visible, go toward it, otherwise turn left.

Example video


reach_1_flag__TwoRooms_1.png
Task #3: One more room

Reach a red flag in an environment of two rooms.

Teacher strategy:

If the flag is visible, go toward it

else if the door is visible, go toward it

else turn left.

Example video


reach_1_flag__LShapedCorridor_1.png
Task #4: The corridor

Reach a red flag in a L-shaped corridor.

Teacher strategy:

If the flag is visible, go toward it

else if the corner is visible, go toward it

else turn left.

Example video


follow_the_arrow.png
Task #5: Follow the arrow

Reach the flag indicated by the arrow.

Teacher strategy:

  1. Go toward the arrow, until close enough
  2. Turn in the direction indicated by the arrow, until the flag is visible
  3. go toward the flag
Example video


follow_the_line.png
Task #6: Follow the line

Follow the coloured line on the floor to reach the flag. The task is failed if the robot leaves the line.

Teacher strategy:

Move forward and turn left and right when necessary to remains on the floor path.

Example video


reach_correct_pillar.png
Task #7: Reach the correct pillar

Reach the pillar painted with diagonal lines.

Teacher strategy:

If the correct pillar is visible, go toward it, else turn left.

Example video


reach_correct_object.png
Task #8: Reach the trophy

Reach the pedestal with the trophy on it.

Teacher strategy:

If the trophy is visible, go toward it, else turn left.

Example video


Frequently Asked Questions (FAQ)

The FAQ can be found here.

Personal