[Bug]: Recover Data From Server When Client Is Missing Data And Throws Timestamp Counter Overflow

by ADMIN 98 views

Introduction

In this article, we will discuss a bug that occurs when a client is missing data and throws a timestamp counter overflow error. This issue is particularly relevant for users who have self-hosted instances of Actual and have experienced data loss due to various reasons such as rebooting their local browser session or exceeding the number of rows in the messages_binary table.

Context

Repeated clearing and importing via API during development can lead to huge message tables in the SQLite database. This can result in a situation where the client is unable to sync with the server, throwing a timestamp counter overflow error. In this scenario, the client-side data is lost, and only the data on the server remains.

Issue Details

When the client attempts to sync with the server, it fails with a thrown OverflowError message: timestamp counter overflow. The console log before the failure indicates that the client has received 178,059 messages from the server. This error occurs because there is a limit to the number of messages permitted during sync, which is set by a constant MAX_COUNTER in /actual/packages/crdt/src/crdt/timestamp.ts.

Solution

The 'Reset sync' feature under 'Advanced Settings' in the Actual web interface can be utilized to reset the huge message tables causing the OverflowError. This feature is designed to resolve issues related to data loss and sync failures.

Additional Considerations

If you are unable to access the 'Reset sync' function in the web interface, there is a more laborious way to recover the data from the server, bypassing the OverflowError. This involves cloning the Actual repo, increasing the MAX_COUNTER limit in the /actual/packages/crdt/src/crdt/timestamp.ts file, building from source, copying the server-files and user-files directories from your original server to your local temporary development server, and finally running the development server. This will permit you to load the budget with sync messages exceeding the normal constant, access the 'Reset sync' function in the web interface settings, and then create a backup of the recovered budget (now with the message tables cleared by 'Reset sync'). Finally, the backup .zip files can be imported into your permanent server.

Reproducing the Issue

To reproduce the issue, perform enough transactions with the API to exceed the number of rows in the messages_binary table past 65535.

Hosting Actual

The self-hosted instance of Actual is running on version v25.4.0.

Browsers

The issue is observed on LibreWolf, which may be responsible for wiping local data after reboot due to strict privacy settings.

Operating System

The operating system is not applicable in this scenario.

Conclusion

In conclusion, the bug discussed in this article occurs when a client is missing data and throws a timestamp counter overflow error. The solution involves utilizing the 'Reset sync' feature under 'Advanced Settings' in the Actual web interface or using a more laborious method to recover the data from the server. By following these steps, users can resolve issues related to data loss and sync failures and recover their data.

Recommendations

To avoid this issue in the future, users can take the following precautions* Utilize the runImport method in custom importers as explained in the writing data importers docs.

  • Regularly back up their data to prevent data loss in case of a reboot or other issues.
  • Monitor their messages_binary table to prevent it from exceeding the normal constant.

Q&A

Q: What is the cause of the timestamp counter overflow error?

A: The timestamp counter overflow error occurs because there is a limit to the number of messages permitted during sync, which is set by a constant MAX_COUNTER in /actual/packages/crdt/src/crdt/timestamp.ts. When the number of messages exceeds this limit, the client throws an OverflowError.

Q: How can I recover my data from the server when I encounter a timestamp counter overflow error?

A: You can recover your data from the server by utilizing the 'Reset sync' feature under 'Advanced Settings' in the Actual web interface. This feature is designed to resolve issues related to data loss and sync failures.

Q: What if I am unable to access the 'Reset sync' function in the web interface?

A: If you are unable to access the 'Reset sync' function in the web interface, you can use a more laborious method to recover your data from the server. This involves cloning the Actual repo, increasing the MAX_COUNTER limit in the /actual/packages/crdt/src/crdt/timestamp.ts file, building from source, copying the server-files and user-files directories from your original server to your local temporary development server, and finally running the development server.

Q: How can I prevent this issue from occurring in the future?

A: To prevent this issue from occurring in the future, you can take the following precautions:

  • Utilize the runImport method in custom importers as explained in the writing data importers docs.
  • Regularly back up your data to prevent data loss in case of a reboot or other issues.
  • Monitor your messages_binary table to prevent it from exceeding the normal constant.

Q: What is the recommended version of Actual to use?

A: The recommended version of Actual to use is v25.4.0.

Q: Which browsers are affected by this issue?

A: The issue is observed on LibreWolf, which may be responsible for wiping local data after reboot due to strict privacy settings.

Q: What is the operating system used for this issue?

A: The operating system used for this issue is not applicable.

Q: Can I reproduce this issue?

A: Yes, you can reproduce this issue by performing enough transactions with the API to exceed the number of rows in the messages_binary table past 65535.

Q: What are the steps to reproduce this issue?

A: The steps to reproduce this issue are as follows:

  1. Perform enough transactions with the API to exceed the number of rows in the messages_binary table past 65535.
  2. Observe the client throwing an OverflowError due to the timestamp counter overflow.

Conclusion

In conclusion, the timestamp counter overflow error is a common issue that can occur when using Actual. By understanding the cause of the issue and following the recommended steps to recover your data, you can minimize the risk of encountering this issue and ensure the integrity of your data.