Someone is asserting a right to payment. That's the only person the
consumer is bothered with.
That's an oversimplified view of the way it works.
There are three parties to the transaction. The merchant (who is the
party asserting the right to be paid), the network and the acquirer.
The two major CC issuers operate the network. They make a percentage of
the transaction in both charge and chargeback situations and
irrespective whether it is fraudulent or not. The acquirer (usually a
bank, but can be other companies) takes the credit risk and mostly
profits from the high interest rate on credit. However it's a high risk
unsecured credit transaction. The merchant must take all reasonable
steps to ensure that the identification of the customer and
authentication of the card. Failure to do this will render the merchant
liable for the fraudulent transaction (this is mostly the case). If the
merchant can show that all reasonable steps were taken to ensure the
integrity of the transaction then the acquirer takes the loss. Notice
how the network has absolved itself from the credit risk? There is very
little incentive for the network (who own the infrastructure and issue
the cards) to remedy the situation.