Dynamics engine conformance

Each new release of each engine is installed automatically and asked for the accelerations of the same mechanical systems. Every answer is checked against a closed-form reference that shares no code with any engine. The question is the one that matters to a user: when an engine is wrong, does it tell you?

Last run 2026-11-02 · source and raw results · latest.json

passagrees with the reference, or correctly refuses a system with no answer
silentwrong answer, no error, no warning
warnedwrong answer, with a warning
errorrefused or crashed
timeoutno answer in time

Newest release of each engine

SymPy (sympy.physics.mechanics)

1.14.0 · 2026-10-05 · run
41 pass 1 timeout

MechanicsDSL

2.1.4 · 2026-11-02 · run
21 pass

Drake

1.58.0 · 2026-10-16 · run
10 pass

Project Chrono (PyChrono)

10.0.0 (build 1198) · 2026-10-14 · run
15 pass 13 silent

What changed between releases

EngineFrom → toRouteCaseVerdict
MechanicsDSL2.1.2 → 2.1.3compile_dslslidercrank/l_over_r=3,ms_over_mc=1e+06warned → pass
MechanicsDSL2.1.2 → 2.1.3compile_dslslidercrank/l_over_r=1.05,ms_over_mc=10000warned → pass
MechanicsDSL2.1.2 → 2.1.3compile_dslslidercrank/l_over_r=1.5,ms_over_mc=100warned → pass
MechanicsDSL2.1.2 → 2.1.3compile_dslslidercrank/l_over_r=3,ms_over_mc=1warned → pass
MechanicsDSL2.1.2 → 2.1.3compile_dslnear_singular/eps=0warned → pass
MechanicsDSL2.1.2 → 2.1.3compile_dslchain/N=8timeout → pass
MechanicsDSL2.1.2 → 2.1.3compile_dslchain/N=5timeout → pass

Every case, newest release

Hover a cell for the worst relative error and any message the engine gave. Each case is probed at 8 to 18 states, including the configurations that make it degenerate.

Case SymPy
rhs · 1.14.0
SymPy
mass-matrix · 1.14.0
MechanicsDSL
compile_dsl · 2.1.4
Drake
multibody-plant · 1.58.0
Project Chrono
assembly · 10.0.0 (build 1198) ⚑
Project Chrono
system-matrices · 10.0.0 (build 1198)
chain
N=1
Planar 1-link pendulum, unit point masses on massless rods
pass
0.0e+00
pass
0.0e+00
pass
0.0e+00
pass
0.0e+00
pass
2.0e-11
pass
1.7e-13
N=2
Planar 2-link pendulum, unit point masses on massless rods
pass
1.0e-15
pass
1.0e-15
pass
1.8e-15
pass
3.0e-15
silent
7.6e+00
pass
2.6e-12
N=3
Planar 3-link pendulum, unit point masses on massless rods
pass
2.0e-14
pass
3.4e-15
pass
5.8e-15
pass
2.1e-14
silent
1.7e+00
pass
8.5e-13
N=5
Planar 5-link pendulum, unit point masses on massless rods
pass
2.9e-14
pass
3.8e-14
pass
2.6e-14
pass
2.4e-13
silent
6.1e+00
pass
2.6e-12
N=8
Planar 8-link pendulum, unit point masses on massless rods
timeout
pass
1.1e-13
pass
3.6e-14
pass
9.2e-13
silent
9.7e+00
pass
2.7e-12
near singular
eps=0.1
Two masses, kinetic coupling (1-eps): mass matrix condition number 1.9e+01
pass
1.3e-15
pass
0.0e+00
pass
1.3e-15
– – –
eps=0.0001
Two masses, kinetic coupling (1-eps): mass matrix condition number 2.0e+04
pass
3.1e-15
pass
0.0e+00
pass
3.6e-13
– – –
eps=1e-08
Two masses, kinetic coupling (1-eps): mass matrix condition number 2.0e+08
pass
5.0e-16
pass
0.0e+00
pass
3.3e-16
– – –
eps=0
Two masses, kinetic coupling (1-eps): mass matrix exactly singular -- refusing is correct
pass
pass
pass
– – –
mass ratio
ratio=1
Two masses joined by springs, m2/m1 = 1
pass
0.0e+00
pass
0.0e+00
pass
0.0e+00
– – –
ratio=1e+06
Two masses joined by springs, m2/m1 = 1e+06
pass
2.1e-22
pass
0.0e+00
pass
2.1e-22
– – –
ratio=1e+12
Two masses joined by springs, m2/m1 = 1e+12
pass
1.0e-28
pass
0.0e+00
pass
1.0e-28
– – –
cartpole
M_over_m=10
Cart-pole, cart/pole mass ratio 10
pass
5.6e-17
pass
0.0e+00
pass
3.1e-16
pass
8.3e-17
silent
4.3e-01
pass
2.2e-13
M_over_m=1
Cart-pole, cart/pole mass ratio 1
pass
2.4e-16
pass
0.0e+00
pass
2.5e-16
pass
2.2e-16
silent
1.2e+00
pass
1.2e-13
M_over_m=0.01
Cart-pole, cart/pole mass ratio 0.01
pass
9.1e-15
pass
0.0e+00
pass
3.6e-15
pass
1.1e-15
silent
4.7e+00
pass
1.3e-13
M_over_m=0.0001
Cart-pole, cart/pole mass ratio 0.0001
pass
5.5e-15
pass
0.0e+00
pass
2.2e-15
pass
0.0e+00
silent
3.7e+00
pass
9.7e-13
M_over_m=1e-06
Cart-pole, cart/pole mass ratio 1e-06
pass
5.7e-15
pass
0.0e+00
pass
4.8e-15
pass
2.3e-16
silent
2.8e+00
pass
7.1e-11
slidercrank
l_over_r=3,ms_over_mc=1
Slider-crank (closed loop), rod/crank 3, slider/crank mass 1, probed through dead centre
pass
3.8e-16
pass
0.0e+00
pass
7.8e-16
– silent
4.8e+00
pass
2.6e-13
l_over_r=1.5,ms_over_mc=100
Slider-crank (closed loop), rod/crank 1.5, slider/crank mass 100, probed through dead centre
pass
6.7e-16
pass
5.6e-16
pass
5.5e-16
– silent
1.1e+00
pass
3.0e-13
l_over_r=1.05,ms_over_mc=10000
Slider-crank (closed loop), rod/crank 1.05, slider/crank mass 10000, probed through dead centre
pass
3.2e-16
pass
2.2e-16
pass
1.3e-15
– silent
1.8e+00
pass
5.2e-13
l_over_r=3,ms_over_mc=1e+06
Slider-crank (closed loop), rod/crank 3, slider/crank mass 1e+06, probed through dead centre
pass
1.0e-15
pass
2.2e-16
pass
4.2e-16
– silent
1.0e+00
pass
8.2e-11

History

EngineReleaseRunVerdictsData
SymPy (sympy.physics.mechanics)1.14.02026-10-0541 pass 1 timeoutjson
SymPy (sympy.physics.mechanics)1.13.32026-10-0641 pass 1 timeoutjson
SymPy (sympy.physics.mechanics)1.13.22026-10-0641 pass 1 timeoutjson
MechanicsDSL2.1.42026-11-0221 passjson
MechanicsDSL2.1.32026-10-0521 passjson
MechanicsDSL2.1.22026-10-0614 pass 5 warned 2 timeoutjson
MechanicsDSL2.1.12026-10-0614 pass 5 warned 2 timeoutjson
Drake1.58.02026-10-1610 passjson
Drake1.57.02026-10-0510 passjson
Drake1.56.02026-10-0610 passjson
Drake1.55.02026-10-0610 passjson
Project Chrono (PyChrono)10.0.0 (build 1198)2026-10-1415 pass 13 silentjson
Project Chrono (PyChrono)10.0.0 (build 1197)2026-10-0615 pass 13 silentjson
Project Chrono (PyChrono)10.0.0 (build 1187)2026-10-0615 pass 13 silentjson
Project Chrono (PyChrono)10.0.0 (build 677)2026-10-0615 pass 13 silentjson

How it works

Every day a scheduled job asks PyPI and anaconda.org for each engine's releases. Any release without a result is installed on a fresh GitHub Actions runner (Ubuntu, Python 3.12) and scored on the whole catalogue. Results are committed to the repository, so the history is permanent and anyone can audit it.

At every probe state the engine's generalised accelerations are compared with the reference using maxi |ai − ri| / max(|ri|, 1). The tolerance is 10−8, widened only where the system is so ill-conditioned that no solver, the reference included, can do better: to 10−14 times the condition number of the diagonally scaled mass matrix. Systems that have no defined answer, such as an exactly singular mass matrix, pass only if the engine refuses them.

The references are written in numpy alone, with no symbolic algebra, so they share no code with any engine under test. They come from a study of silent failures in dynamics engines (artefacts: doi:10.5281/zenodo.22315172). Adapters translate conventions such as relative versus absolute joint angles and nothing else; an adapter never corrects an engine's answer.

Where an engine offers two documented ways to get accelerations, both are scored as separate columns. A dash means that engine cannot model the system without changing what the case tests; hover it for the reason.