[southlander]

I would suspect RAM is the issue. The work rendering at a pixel level when combining all those images is huge. There would be some massive temporary files being created while the stitching is being done and before the final file is written, and there may well be multiple working files being held in RAM simultaneously. I couldn't generate large panorama files while using ICE on a 32 bit machine with 4Gb of RAM - I had to specify the output file as a JPEG of only around 5000 pixels on the long edge to get it to complete a pano stitch successfully.

Don't forget JPG will only be 8 bit colour whereas TIFF will likely be 16 bit - so there is a significant file size increase using TIFF rather than JPG, even for the same number of pixels -compressed or not.

However, I haven't done any panos to date on my updated rig which is 8GB RAM so I can't offer any advice based on experience. It's somewhere on the "to do" list, as is painting the verandah, fixing split doors, you get the picture!

[RGlasel]

then complains when it tries to stitch them together

The same program is compiled for 32 bit Windows, which can't address more than 4GB of RAM anyway. The real limitation is with CPU intensive image processing, and to deal with that you need to be a Hugin programmer, or get support from someone who is. Being open source software, no one is obliged to provide support, but your "edge" case might be of interest to people working on the project.