Operating system and compiler influences on benchmarking

This page is about running the same benchmark on the same hardware with different operating systems and, more importantly, different compiler choices. The machine used is a PC clone with an AMD 486DX4/133 processor (also known as 5x86P75) with 256 KB L2-cache and a Biostar motherboard with zero wait-state DRAM timing. The benchmark used is BYTEmark, with some binaries compiled by me and others downloaded from BYTE's www site. This synthetic benchmark is intended to measure single-process single-threaded CPU/FPU/cache/memory performance, and nothing else. This in particular means that all other variables (C compiler, C and math libraries, OS, memory management, for example) should be left constant. The results below will show you what happens when you do not keep them equal.

If you need an unzip utility for MS-DOS, here is unzip.exe.

  1. MS-DOS 6.22 with a 16-bit executable compiled with the Watcom version 10.5 C-compiler, as downloaded from BYTE's www site.
    Also available here: byte16.zip
    A sample output of bytecpu -ccom.dat : bytecpu.16bit.txt
  2. MS-DOS with a 32-bit executable compiled with the Watcom version 10.5 C-compiler, and running under DOS/4GW Run-time Protection Mode, as downloaded from BYTE's www site.
    Also available here: byte32.zip
    A sample output of bytecpu -ccom.dat : bytecpu.32bit.txt
  3. MS-Windows 3.1 on top of MS-DOS 6.22, the executable was compiled with the Watcom version 10.0a C-compiler, as downloaded from BYTE's www site.
    Also available here: bytewin.zip
    A sample output of bytecpu -ccom.dat : bytecpu.windows.txt
  4. MS-DOS 6.22 with a 32-bit executable compiled with gcc version 2.7.2 (see http://www.delorie.com/djgpp/ for the DOS port of gcc). The executable runs under protected mode using DPMI. The sources used are my Linux/Unix port version 2.1 of BYTEmark, which also compile on MS-DOS with only minor changes to the Makefile.
    Available here (includes an executable): nbench21.zip
    A sample output of nbench -v : nbench21.dos.txt
  5. Linux 2.0.32 with gcc-2.7.2.3, my Linux/Unix port version 2.1 of BYTEmark.
    Available here (includes a binary): nbench-byte-2.1.tar.gz
    A sample output of nbench -v : nbench-byte-2.1.gcc.txt
  6. Linux 2.0.32 with egcs-1.00, my Linux/Unix port version 2.1 of BYTEmark.
    These are the same sources as the ones used for gcc.
    A sample output of nbench -v : nbench-byte-2.1.egcs.txt

All the numbers listed below after the names of the tests are raw numbers, to be interpreted as arrays sorted per second, or bits flipped per seconds, or assignment problems solved per second, etc. The flag -ccom.dat in the commands listed above causes the executable to read in the file com.dat which is part of the distribution, and it simply sets verbose output mode. For the Linux port the same can be achieved by specifying the -v flag.

Finally, all the memory, integer/arithmetic, and floating-point indices listed below are with respect to a PC clone with an AMD K6/233 with 512 KB L2-cache running Linux 2.0.32 and using gcc version 2.7.2.3 and libc version 5.4.38. The memory index is the geometric mean of the tests String Sort, Bitfield, and Assignment (as compared to the baseline, of course). The integer/arithmetic index is the geometric mean of the tests Numeric Sort, FP Emulation, IDEA, and Huffman. The floating-point index is the geometric mean of the tests Fourier, Neural Net, and LU Decomposition. The complete documentation is included in the files nbench21.zip or nbench-byte-2.1.tar.gz. Here are the results. 

OS:         DOS 16bit   DOS 32bit  Windows3.1         DOS       Linux       Linux
Compiler: Watcom-10.5 Watcom-10.5 Watcom-10.0   gcc-2.7.2 gcc-2.7.2.3   egcs-1.00
---------------------------------------------------------------------------------
NUMERIC SO      3.792      30.723      27.701      26.703      30.900      30.956
STRING SOR      0.463       2.035       1.837       1.501       3.146       3.315
BITFIELD   409105.340 5030838.820 4845797.540 4630800.000 6042100.000 5906900.000
FP EMULATI      0.837       2.096       2.092       3.389       3.511       3.331
FOURIER       451.840     499.653     499.344     243.780     440.350     435.700
ASSIGNMENT      0.036       0.193       0.194       0.237       0.275       0.310
IDEA           23.224      51.922      50.444      47.811      49.751      50.641
HUFFMAN         7.499      17.083      33.877       9.064      32.712      31.516
NEURAL NET      0.030       0.250       0.250       0.263       0.273       0.273
LU DECOMPO      0.764       8.452       9.349       6.924       8.173       9.364
---------------------------------------------------------------------------------
MEM INDEX       0.025       0.169       0.162       0.159       0.234       0.246
INT INDEX       0.067       0.215       0.247       0.196       0.285       0.280
FP INDEX        0.055       0.258       0.266       0.193       0.251       0.262
---------------------------------------------------------------------------------
Baseline: AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38, Linux 2.0.32

Conclusion: The effect of different choices of C-compiler, C and math libraries, and OS cannot be ignored.

Send comments but not flames to mayer@math.utah.edu.

[leftarrow] Back to the benchmarking page.

First posted: Dec/17/1997
Last updated: Mon Jan 2 12:24:29 PST 2017