experitur automates machine learning and other computer science experiments and stores the results in an easily accessible format. It includes grid search, random search, parameter substitution, inheritance and resuming aborted experiments.
Read the documentation!
Every experiment is described in a regular python file. The @Experiment decorator is used to mark experiment entry-points.
By default, parameters are defined as a parameter grid where each parameter has a list of values that it can take. A number of trials is generated from the cross product of the values of each parameter.
(So it works like sklearn.model_selection.ParameterGrid.)
An experiment is a regular function that is decorated with @Experiment (unless it is abstract or derived). Upon execution, this function gets called with the current trial's parameters. It may return a result dictionary.
Signature: (trial: Trial) -> Optional[dict]
from experitur import Experiment, Trial
@Experiment(
parameters={
"parameter_1": [1,2,3],
"parameter_2": ["a", "b", "c"],
})
def example_experiment(trial: Trial):
"""This is an example experiment."""
print("parameter_1:", trial["parameter_1"])
print("parameter_2:", trial["parameter_2"])
return {}You can run the experiment using experitur run example.py and example_experiment will be called six times with every combination of [1,2] x [a,b,c].
The Python file can contain multiple experiments:
from experitur import Experiment, Trial
@Experiment(...)
def example1(trial: Trial):
...
@Experiment(...)
def example2(trial: Trial):
...One experiment may inherit the settings of another, using the parent parameter.
from experitur import experiment
@Experiment(...)
def example1(trial):
...
# Derived with own entry point:
@Experiment(parent=example1)
def example2(trial):
...
# Derived with inherited entry point:
example3 = experiment("example3", parent=example2)Every experiment receives a Trial instance that allows access to the parameters and meta-data of the trial.
Parameters are accessed with the [] operator (e.g. trial["a"]), meta-data is accessed with the . operator (e.g. trial.wdir).
When experitur executes a script, it creates the following file structure in the directory where the experiment file is located:
/
+- script.py
+- script/
| +- experiment_id/
| | +- trial_id/
| | | +- experitur.yaml
| | ...
| ...
<script>/<experiment_id>/<trial_id>/experitur.yaml contains the parameters and the results from a trial, e.g.:
experiment:
func: simple.simple
meta: null
name: simple
parent: null
id: simple/a-1_b-3
parameters:
a: 1
b: 3
result: null
success: true
time_end: 2020-03-26 21:01:51.648282
time_start: 2020-03-26 21:01:51.147210
wdir: simple/simple/a-1_b-3
Most items should be self-explanatory. parameters are the parameters passed to the entry point. id is derived from the parameters that are varied in the parameter grid. This way, you can easily interpret the file structure.
experitur is packaged on PyPI.
pip install experiturBe warned that this package is currently under heavy development and anything might change any time!
To install the development version, do:
pip install -U git+https://github.com/moi90/experitur.git- examples/example.py: A very basic example showing the workings of
set_default_parametersandapply_parameters. - examples/classifier.py: Try different parameters of
sklearn.svm.SVCto classify handwritten digits (the MNIST test set). Run the example, add more parameter values and see howexperiturskips already existing configurations during the next run.
experitur is under active development, so any user feedback, bug reports, comments, suggestions, or pull requests are highly appreciated. Please use the bug tracker and fork the repository.
experitur is tested with Python 3.7, 3.8, 3.9, and 3.10.