Security:Counter Attack?

It is illegal to do any of the things that will be mentioned in this post. But to spice things up, let us assume that the law is somehow magically waived in this instance. What would be a ‘fair’ thing to do in the instance below:

As an administrator, you get several warning emails from other administrators that one of your servers may be compromised. It is now sending out a lot of probe and attack packets. You do an intial forensic and have been able to detect:

a. the server is indeed compromised.
b. the apache user is being used by x program.
c. the x program is connecting out to some irc channel
d. you take hold of the offending series of x programs that are downloaded into the /var/temp and dissect them
e. You now know who the attacker is, what channel the bots are connecting to etc. ie. enough data to mount a counter attack…(this is of course illegal and may land you in jail)

At this stage, if you were in this position, what would you likely be doing as a counter-measure? or counter-attack perhaps?

Some objective of ones action for counter measures would be:
1. Find out how the server was compromised and limit future similar intrusions/compromise.
2. Make it ‘painful’ for the hacker to do so in the future. (aka. teach Cosm** a lesson).
3. Not to go overboard with the counter-attack so as not to cause harm on person or family of the attacker.