I just found a way to retrieve the old forums. Here is the original bug report from 2/21/2006.
Current Status: Blizzard is not aware of the bug.
Bug Description: Non-binary spells get partially resisted by mobs of level higher than 60 even when resistances on the mob are brought down to 0 by -resist mechanics (Curse of Shadows, Curse of Elements and items with "Decreases the magical resistances of your spell targets by X" mechanics). This has been confirmed for Arcane Missiles, Fireball and Shadowbolt. This problem is not existent against level 60 and lower targets (either PVE or PVP).
Experiments:
Before 1.9 Onyxia had vulnerabilities when casting Arcane Missiles and COS was applied. This means that she had less than 75 arcane resistance. With patch 1.9 missiles are getting partial resists at the same rate as in 1.8 without COS. The same holds for most MC trash mobs and bosses.
When casting Arcane Missiles at Druid with 48 arcane resists almost every second missile was partially resisted with no -resist gear. With -50 resist gear there were no partial resists.
Data collected on Ebonroc. In both cases there was Curse of Shadows on him. The tables show in percentages how much damage hit the target (0 = full resist, 0.25 means 75% damage was resisted and so on)
1. case:
-0 resist
7% to hit
658 samples
0.00 : 0.0%
0.25 : 1.0%
0.50 : 3.3%
0.75 : 12.5%
1.00 : 83.1%
2. case:
-40 resist
5% to hit
849 samples
0.00 : 2.0%
0.25 : 0.6%
0.50 : 4.2%
0.75 : 10.2%
1.00 : 82.8%
Comparing both cases first one can see that +to hit gear is performing as expected (+7% to hit and arcane focus results in no misses, +5% to hit results in 2% of white resists).
The partial resists in both cases however are the same, worth around 30 arcane resist. Distribution of partial resists is the same up to the small difference due to randomness. -40 resist gear should result in substantial drop in partial resists if not eliminate them completely.
Additional information:
http://forums-en.wow-europe.com/thread.aspx?FN=wow-mage-en&T=132218&P=1
http://forums.subcreation.net/viewtopic.php?t=482
EDIT:
Note that the test were done before 1.11. Post 1.11 Arcane Focus no longer guarantess 100% hit.
Further experiments detailed on the above link have confirmed that indeed the resistances that we are encountering here are level based, 8 resist per level difference, equaling 2% damage mitigation.
Based on the information understood about resist calculations it would seem that the change in code that prevents resistances to go below zero was placed in wrong place, before the level based resistances are added. I'm concluding this is a bug, because binary spells which use a separate calculation do not exhibit this problem (i.e. penetration can counter level based resistances for binary spells).