How to measure performance?

Hello CheckiO Friends! We have been talking about the performance testing in the neighbour post. Now we would like to discuss how to measure this performance with you.

So far, we have two variants: The first is to use the fixed size virtual machine (CPU, Memory limited) and a simple timeout for the given dataset.

The second is to measure the complexity of the algorithms. We use the dedicated machine and run your solution several times with different by sizes datasets. For example: N=100, N=1000, N=10000, N=20000. Then we compare these results and get the complexity of the algorithm. For example: N=100 => t=0.001; N=1000 => t=0.11; N=2000 => t=0.43, so the result looks like O(N^2).

The first variant requires an in depth understanding of languages in great detail. The second is language independent and relies on the algorithm.

Now that you've heard our ideas, what do you think?