Adds mapf benchmark tools to this repo#41
Conversation
I think the root cause of #27 is this. I'm not sure why it was not being triggered earlier, but seems like we are accessing an 8 bit bitfield but limiting the index to `u8::MAX`instead of 7. Signed-off-by: Arjo Chakravarty <arjo129@gmail.com> Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Luca Della Vedova <lucadv@intrinsic.ai> Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Michael X. Grey <greyxmike@gmail.com> Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Michael X. Grey <mxgrey@intrinsic.ai> Co-authored-by: Michael X. Grey <mxgrey@intrinsic.ai> Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
The project now includes: - mapf-bench crate: A tool for running multi-agent negotiation benchmarks on Moving AI grid maps. - scripts/benchmark.py: An automated script that downloads maps/scenarios, runs benchmarks for 2-50 agents, and generates reports with strict timeouts. - .gitignore update: The cache/ directory used for benchmarking data is now ignored. Generated-by: Gemini-CLI Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
arjo129
force-pushed
the
arjoc/feat/benchmark-infra
branch
from
d5ed740 to
04e67d5
Compare
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
mxgrey
left a comment
mxgrey
left a comment
There was a problem hiding this comment.
I have a few very minor comments, but the main thing that should be done is adding a README for the mapf-bench module to explain what this benchmarking tool is, what it does, and how to use it correctly. It's especially important to document the relationship between mapf-bench and scripts/benchmark.py.
| pub struct Map { | ||
| pub width: usize, | ||
| pub height: usize, | ||
| pub grid: Vec<Vec<char>>, |
There was a problem hiding this comment.
I was very surprised to see that "grid" is a matrix of characters instead of something like an integer cost. Later in the code I found that it consists of special characters like G, which I assume is a goal, S which I assume is a starting point, and . which I assume is an obstacle.
Let's add some documentation here to explain what characters are expected for this map format.
There was a problem hiding this comment.
I meant to add documentation directly to this grid: Vec<Vec<char>> field. Maybe a link to some legend that explains what each character means. Or if there isn't a legend we can reference, then provide a list of each expected character and its meaning.
| pub map_file: String, | ||
| pub map_width: usize, | ||
| pub map_height: usize, | ||
| pub start_x: usize, |
There was a problem hiding this comment.
Minor nitpick, but should we consider start: [usize; 2] and goal: [usize; 2]? Or maybe [i64; 2] to match the later conversion?
| } | ||
|
|
||
| def main(): | ||
| parser = argparse.ArgumentParser() |
There was a problem hiding this comment.
Let's add some documentation about what this script is for and when/how to use it. I'm noticing some subprocess.run of cargo build, which adds all sort of questions, like .. is this a meta-build script..?
There was a problem hiding this comment.
Sorry, I should've been more explicit: I meant to add documentation to the argparse description. That way it's visible both for people who read the code and also anyone calling the command from the terminal.
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
|
|
||
| #[derive(Parser, Debug)] | ||
| #[command(author, version, about, long_about = None)] | ||
| struct Args { |
There was a problem hiding this comment.
Let's add a docstring to this so that people calling the program from the command line know what it's for.
| seen as the defacto "mapf" banchmark. This crate contains abinary to load the benchmark files into memory and run | ||
| a single benchmark. | ||
|
|
||
| It is advised to use the python benchmark script located in the `scripts/` folder tot automate the downloading and |
There was a problem hiding this comment.
| It is advised to use the python benchmark script located in the `scripts/` folder tot automate the downloading and | |
| It is advised to use the python benchmark script located in the `scripts/` folder to automate the downloading and |
There was a problem hiding this comment.
Besides recommending to people that they use the script, this readme should explain what the intended workflow is for carrying out a benchmark.
The comments you added to benchmark.py would probably be more helpful to have in this README.
Signed-off-by: Arjo Chakravarty <arjoc@intrinsic.ai>
4 participants
New feature implementation
Implemented feature
This adds the standard MAPF benchmarks as infrastructure.
The key idea is if we have new techniques that are faster like #40 . we should be able to test their improvement rapidly.
Implementation description
We added a new crate which converts the mapf benchmark into a negotiation
Scenario. We also have a python script that automatically downloads the relevant benchmarks and runs them. To use it run:GenAI Use
We follow OSRA's policy on GenAI tools
Generated-by: Gemini-CLI