c# internet protocol instantiating packets the fastest?

c# internet protocol instantiating packets the fastest?

I have a C# game server that creates a new object for every packet it
receives that it then fills from the datasteam. After recently
benchmarking it, I notice the line that does
var packettype = Opcodes[opcode];
var packet = (Packet)Activator.CreateInstance(packettype.Type);
is consuming the most resources of the whole application after profiling.
Changing everything to struct and creating a cached table of the empty
packet bodies seems like it might work but a lot of redesign might be
needed.
Anyone know a good way to really reduce the CPU usage of this, I was
thinking keeping a instantiated object in the Opcodes table and then
somehow copying it instead of instantiating a new object for each packet?
A lot of the packets are lightweight, but some are really large, having
multiple fields and lists.
Note: Each packet is created 100% exactly the same before being filled.