Evaluation

We have extensively tested our port to ensure that it can get exactly the same results than the original RSIM. Also, the purpose of our evaluation is to check if using the ported version of RSIM is a cost-effective solution to perform the great number of long running simulations needed for our research.

Firstly, we compare the execution speed of RSIM running in several different architectures. Secondly, since the execution time of a single benchmark is not the most valuable metric for our purposes, we define a better indicator of the usefulness of each simulation platform and version of RSIM. We will measure which version allows us to utilize our computing resources more efficiently in terms of hardware cost and execution time. We will use a metric based in the normalized number of simulations per hour per thousand euros.

We have measured the impact that the actual benchmark being simulated has in the speedup obtained by our port and have found that it is small, but not inexistent. Hence, we have chosen a small set of representative benchmarks from the SPLASH suite to perform our experiments.

We have also measured the impact that varying the problem size of the simulated benchmarks has in the achieved speedup and have found that it is very small once a certain threshold is reached. Other simulator parameters, like the number of processors, have a very small influence too. Hence, we have chosen medium problem sizes and we use two processors and default values for the rest of the parameters for our experiments.

We have evaluated the speed of running our port of RSIM in the following machines:

• A high-end Solaris/SPARC Sunblade-2000 system: SPARC-1.
• A low-end Solaris/SPARC Sunblade-100 system: SPARC-2.
• A high-end Linux/Athlon64 SMP system (running in legacy IA-32 mode): X86-64.
• A high-end Linux/Xeon SMP system: XEON.
• A low-end Linux/Pentium-IV system: P-IV.

The relevant characteristics and price of each machine is shown in the following table. The prices indicated for the machines are necessarily approximate. These are the approximate prices that those systems would cost as of January 2005 in Spain. Table 1: Characteristics of evaluated configurations.

In figure 1 we show the normalized throughput time of our set of benchmarks for each architecture. In other words, we show the throughput speedup of each machine compared with the slowest one (SPARC-2).

Figure 1: Normalized throughput per processors for each architecture. Normalized throughput per processors for each architecture

As some of our machines are SMP systems with two processors. In those cases, we can run two instances of RSIM simultaneously effectively doubling the throughput. Since the simulation work is CPU limited with very little IO and modest memory requirements there is no contention between the two independent processes.

In figure 2 we show the average number of simulations per hour per thousand euros achieved for each platform. When we account for the price of each machine and the number of processors, we see that the cheapest platform is the best option for efficiently take advantage of any given budget. Also, the easy availability of these kind of machines make them an even more attractive alternative to the expensive Solaris/SPARC machines used until now to run simulations based on RSIM. Figure 2: Average number of simulations per hour per thousand euro. Average number of simulations per hour per thousand euro