Have to remember, people were a little smaller back then, but still would have been crowded.
The Fort was only the main structure. It had many smaller auxiliary buildings around it. If I recall Margaret Harradine's book Port Elizabeth I think the Royal Garrison Company was 250 men strong.The Fort would make a lovely little military museum. If one stacks a few mock barrels and cannon balls in the Arsenal (with the pitched roof) and cover the roof of the blockhouse with perspex or something you can display uniforms, weaponry and portraits of their history up to 1856, Captain Francis Evatt, etc. Oh, if only we could find a benevolent benefactor!