I found this on the MicroSquish Knowledge Base:
Computer May Reboot Continuously with More Than 1.5 GB of RAM
The information in this article applies to:
* Microsoft Windows Millennium Edition
* Microsoft Windows 98 Second Edition
* Microsoft Windows 98
This article was previously published under Q304943
SYMPTOMS
If your computer has more than 1.5 gigabytes (GB) of memory (RAM), the computer may reboot continuously when you try to start Windows Millennium Edition (Me) or Windows 98.
Or, when you try to install Windows Me or Windows 98 with more than 1.5 GB of RAM installed, Setup may stop responding (hang) or reboot continuously.
CAUSE
Windows Me and Windows 98 are not designed to handle more than 1 GB of RAM. More than 1 GB can lead to potential system instability.
WORKAROUND
To work around this issue, add the following line to the [386enh] section of the System.ini file:
MaxPhysPage=40000
This limits the amount of physical RAM that Windows can access to 1 GB. To add this line, use the following steps:
1. Use any text editor (such as Notepad) to open the System.ini file in the Windows folder.
2. Add the following line in the [386Enh] section of the file:
MaxPhysPage=40000
3. Save the file, and then restart your computer.
For additional information about the MaxPhysPage entry, click the article number below to view the article in the Microsoft Knowledge Base:
181862 Specifying Amount of RAM Available to Windows Using MaxPhysPage
STATUS
This behavior is by design.
MORE INFORMATION
For additional information about memory limitations in Windows Me and Windows 98, click the article numbers below to view the articles in the Microsoft Knowledge Base:
253912 "Out of Memory" Error Messages with Large Amounts of RAM Installed
184447 Error Message: Insufficient Memory to Initialize Windows
Note that decreasing the size of VCache as mentioned in Q253912 does not resolve this particular issue.