2022–01–11 Incident Report

Updates

This article has been updated for the 2022–01–17 incident when raydium api.raydium.io was returning abnormal lp_price. Due to recent instability related to the pricing report of Raydium LP pools, we will pause on adding new Raydium LP pools

Impacted users may refer to the additional list of wallet addresses below and reach out to us on Discord for any additional questions. All funds are safe, and we have already pursued the same compensation scheme for all wallets that experienced greater than $1 in liquidation value.

What took place

At around 1530pm 11th Jan 2022 (UTC), we upgraded our pricing mechanism for SOL-USDC and SOL-USDT Raydium LP. This upgraded pricing mechanism deviated from Raydium’s official API pricing data as it contained a synchronization issue.

While the upgraded pricing mechanism behaved normally most of the time, it also occasionally led to unstable prices being recorded in Apricot. This issue existed for the SOL-USDC, SOL-USDT, and RAY-USDC Raydium LP. Prices for Orca LPs and all other tokens were not affected.

It was identified and fixed at around 0230am 12th Jan 2022 (UTC). The prices of affected LP tokens are back to normal and stable since then.

How users were affected

This instability in the LP token prices affected our users who had significant positions in the SOL-USDC, SOL-USDT, and RAY-USDC Raydium LPs. It affected these users in two ways:

1. Some users had their LP tokens forcibly redeemed by Apricot. This happened because Apricot’s liquidation engine, using incorrect LP prices, reached the conclusion that these users were near liquidation, and as part of the liquidation procedure forcibly redeemed some of these users’ LP tokens. For this group of users, this forced redemption actually returned their account status to normal, and stopped actual liquidation (which carries a 4% liquidation penalty) from taking place.
2. Some users had their positions partially liquidated. This resulted in some of their collateral being sold and used to repay their debt. As a result, for this group of users there is a direct 4% liquidation penalty incurred on them.

Compensation for affected users

For users who were erroneously liquidated and who experienced a 4% liquidation penalty as a consequence, we will return 5% of the liquidation value in USDC directly to the affected wallets. The 25% increase is an expression of our sincere apology.

Transfer to the liquidated wallets will be done within 24 hrs (only listing wallets that experienced > $1 liquidation value):

BqfLWMq…MssfetW  4CmdnnA…of4CyEQ  6VGcEyf…GjXefPN  E7FbMzq…GqHPX8v
Fu8tW3K…CSHqTzo  9EuPejL…9yHzZMN  BFAEoGb…w1zuFX8  5w81nKC…qXDD7bh
HDFigdD…pWe1PBi  GoSAGHm…oRhEdn6  8oDBAVj…Cuy7Nib  2RXQv4v…Dcp6n9w
4d4HJoH…j9SjVpN  8t2r2pa…iNDnCfA  12DMsVw…RecF43M  ESSFrGB…9EYd1DW
Ebwnz3i…o4PA9tb  EmEia62…27YME3X  Br6E8VR…cq6eAKe  9QiFNdb…d7zqdt1

Updated wallet list for 2022–01–17

For users whose LP tokens were forcibly redeemed but who were otherwise not liquidated, as it strictly did not affect their Net Deposit, we do not have additional compensation planned.

Last, but not least

Again, we sincerely apologize for the confusion and inconvenience caused by this incident. We will use this as a lesson to further strengthen our testing requirement and risk management practice. We look forward to your continued support as we continue to improve all aspects of Apricot. 🍑