Sounds like a caching problem of some kind to me, since it is recognizing the zone wrong I would guess IE. Hopefully its current on all its patches, make sure there aren't any 'recommended' ones that might apply that might not have gotten installed by windows update automatically, etc.
Might be able to clean up by: internet options - content tab - clear ssl state button. Might also need to clean temp internet files and restart the browser too. If you had already tried that, I'm not really sure what to say.
Its possible the site may have a 4 hour SSL session timeout, although it is more common to have shorter than that like 2 hours. Maybe have them log out of the application when they go on breaks and see if that helps - if it requires a smart card for logon they probably should be doing that anyways... It may be that doing this might keep it from getting into that state so it doesn't cache it for subsequant issues.
Not likely, but a small possibility that pulling and reinserting the card may help. Do you know if this site occasionally polls to see if the card is still inserted, or is it just used for a client authentication for an SSL session (so card would not need to be in after they logged into the site the first time)? If it seems to be related to the card, try contacting the smartcard middleward vendor.