Data Integrity Issues
This document is mainly used to record common data integrity issues encountered during the use of Doris. It will be updated periodically.
| Issue Description | Occurrence Conditions | Affected Versions | Fix Versions | Impacted Scope | Fix PR |
|---|---|---|---|---|---|
| Partial column updates in merge-on-write Unique table reintroduce previously deleted data | A __DORIS_DELETE_SIGN__ column is specified during partial column updates, and the historical data contains rows marked as deleted by the __DORIS_DELETE_SIGN__ column | <2.1.8, <3.0.4 | >=2.1.8, >=3.0.4 | Compute-storage coupled mode, compute-storage decoupled mode, partial column updates | #46194 |
| Duplicate key data in the table | Concurrent imports in merge-on-write Unique table in compute-storage decoupled mode | <3.0.4 | >=3.0.4 | Compute-storage decoupled mode | #46039 |
| Duplicate key data in the table | Concurrent imports between imports and between import and compaction in merge-on-write Unique table in compute-storage decoupled mode | <3.0.4 | >=3.0.4 | Compute-storage decoupled mode | #44975 |
| System-generated values for auto-increment column are 0 or duplicate | Network anomalies between BE and FE | <2.1.8, <3.0.3 | >=2.1.8, >=3.0.3 | Compute-storage coupled mode, compute-storage decoupled mode, auto-increment column | #43774 |
Stream Load import does not delete data that meets the delete condition in merge-on-write Unique table | Stream Load import with merge_type: MERGE, partial_columns: true, and delete parameters | <2.0.15, <2.17, <3.0.3 | >=2.0.15, >=2.17, >=3.0.3 | Compute-storage coupled mode, compute-storage decoupled mode, partial column updates | #40730 |
| Partial column updates result in unintended update of auto-increment column values | The table has auto-increment column on the Value column, and the partial column update import does not specify values for those auto-increment column | <2.1.6, <3.0.2 | >=2.1.6, >=3.0.2 | Compute-storage coupled mode, compute-storage decoupled mode, auto-increment column | #39996 |
| Duplicate key data in the table | User adds sequence column functionality to a merge-on-write Unique table that does not support sequence columns using ALTER TABLE tbl ENABLE FEATURE "SEQUENCE_LOAD" WITH ..., followed by new imports | <2.0.15, <2.1.6, <3.0.2 | >=2.0.15, >=2.1.6, >=3.0.2 | Compute-storage coupled mode, compute-storage decoupled mode | #39958 |
| Duplicate key data in the table | Concurrent imports or concurrent imports and compactions in merge-on-write Unique table in compute-storage decoupled mode | <3.0.1 | >=3.0.1 | Compute-storage decoupled mode | #39018 |
| Partial column update import results in data corruption in merge-on-write Unique table | Concurrent partial column updates in merge-on-write Unique table, with BE restart during the import process | <2.0.15, <2.1.6, <3.0.2 | >=2.0.15, >=2.1.6, >=3.0.2 | Compute-storage coupled mode, compute-storage decoupled mode, partial column updates | #38331 |
| Duplicate key data in the table | Concurrent imports and compaction in merge-on-write Unique table in compute-storage decoupled mode | <3.0.2 | >=3.0.2 | Compute-storage decoupled mode | #37670, #41309, #39791 |
| Duplicate key data in the table | Large single import in merge-on-write Unique table with sequence column triggers segment compaction | <2.0.15, <2.1.6, <3.0.2 | >=2.0.15, >=2.1.6, >=3.0.2 | Compute-storage coupled mode, compute-storage decoupled mode | #38369 |
| Duplicate key data in the table | Failed full clone on merge-on-write Unique table in compute-storage coupled mode | <2.0.13, <2.1.5, <3.0.0 | >=2.0.13, >=2.1.5, >=3.0.0 | Compute-storage coupled mode | #37001 |
| Duplicate key data in the table | Stream Load imports with internal retry process in merge-on-write Unique table in compute-storage decoupled mode | <3.0.0 | >=3.0.0 | Compute-storage decoupled mode | #36670 |
| Inconsistent multi-replica data in merge-on-write Unique table | Partial column update import with __DORIS_DELETE_SIGN__ column on merge-on-write Unique table, and inconsistent Base Compaction progress across replicas during import | <2.0.15, <2.1.5, <3.0.0 | >=2.0.15, >=2.1.5, >=3.0.0 | Compute-storage coupled mode, compute-storage decoupled mode, partial column updates | #36210 |
| Duplicate key data in the table | Concurrent partial column updates and BE restart during imports on merge-on-write Unique table | <2.0.11, <2.1.4, <3.0.0 | >=2.0.11, >=2.1.4, >=3.0.0 | Compute-storage coupled mode, compute-storage decoupled mode, partial column updates | #35739 |