Possible db corruption issue: IllegalArgumentException: Could not deserialize

[angelix]

In a production app, we have many exception like this:

java.lang.IllegalArgumentException: Could not deserialize [-84, -19, 0, 5, 115, 114, 0, 26, 111, 114, 103, 46, 100, 105, 122, 105, 116, 97, 114, 116, 46, 110, 111, 50, 46, 78, 105, 116, 114, 105, 116, 101, 73, 100, -18, 115, 1, -74, 97, -1, 36, -103, 2, 0, 1, 76, 0, 7, 105, 100, 86, 97, 108, 117, 101, 116, 0, 16, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 76, 111, 110, 103, 59, 120, 112, 115, 114, 0, 14, 106, 97, 118, 97, 46, 108, 97, 110, 103, 46, 76, 111, 110, 103, 59, -117, -28, -112, -52, -113, 35, -33, 2, 0, 1, 74, 0, 5, 118, 97, 108, 117, 101, 120, 114, 0, 16, 106, 97, 118, 97, 46, 108, 97, 110, 103, 46, 78, 117, 109, 98, 101, 114, -122, -84, -107, 29, 11, -108, -32, -117, 2, 0, 0, 120, 112, 0, 0, 0, 65, 24, 4, -100, -122] [1.4.196/0] at org.h2.mvstore.DataUtils.newIllegalArgumentException(DataUtils.java:728) at org.h2.mvstore.type.ObjectDataType.deserialize(ObjectDataType.java:377) at org.h2.mvstore.type.ObjectDataType$SerializedObjectType.read(ObjectDataType.java:1556) at org.h2.mvstore.type.ObjectDataType.read(ObjectDataType.java:232) at org.h2.mvstore.type.ObjectDataType.read(ObjectDataType.java:115) at org.h2.mvstore.Page.read(Page.java:708) at org.h2.mvstore.Page.read(Page.java:195) at org.h2.mvstore.MVStore.readPage(MVStore.java:1952) at org.h2.mvstore.MVMap.readPage(MVMap.java:741) at org.h2.mvstore.MVMap.setRootPos(MVMap.java:751) at org.h2.mvstore.MVStore.openMap(MVStore.java:476) at org.h2.mvstore.MVStore.openMap(MVStore.java:427) at org.dizitart.no2.store.NitriteMVStore.openMap(NitriteMVStore.java:89) at org.dizitart.no2.Nitrite.getCollection(Nitrite.java:111)

Any ideas how can i debug this kind of crashes?

[anidotnet]

I am not sure what causing this. If you have any means of reproducing it via a simple unit test, I can try to debug it. It might be due to an issue in h2 mvstore. Can you upgrade to latest release version of nitrite 3.1.0 and try? I have upgraded the mvstore version where they have fixed lots of bugs.

There is also another possibility, you are trying to save an object in a Document which does not implement Serializable.

[anidotnet]

@angelix have you seen any improvement after upgrading to 3.1.0?

[angelix]

@anidotnet it is not easily reproducible. I'll have to see crash reports from production in a week or so.

[anidotnet]

Ok. Let me know here what you see.

[anidotnet]

@angelix have you encountered this issue lately?

[angelix]

I couldn't update yet to 3.1.0 cause of the D8 issue.

[anidotnet]

@angelix Have you looked into the example android project - https://github.com/dizitart/nitrite-database/tree/master/example-android-app ? Use the android tools plugin mentioned there and you will be good to go.

[ArthurSav]

Sadly this is a very common issue that i think nitrite should address.

The cause of this is usually when serializable object repositories have been changed while not having a stable serialVersionUID.

My solution is check if repos are corrupted and then delete restore what's possible. Not perfect since users might lose local data.

i.e

public void recreateIfNeeded(){
    if (isCorrupt()) {
      // recreate
    }
  }

  public boolean isCorrupt(){
    try {
      getNitrite().getRepository(User.class).find().firstOrDefault();
      // do this for every repo
    } catch (IllegalArgumentException e) {
      return true;
    }
    return false;
  }

[anidotnet]

I would very much like to see what types of code cause this. Every pojo used in ObjectRepository gets converted into Document object. So serialVersionUID is really not required for the object. Document has a fixed uid so it gets deserialized without any problem.

The only case it can happen if someone insert such an object in a document direcrly, like while implementing Mappable, if someone insert an arbitrary object in a document. The right course of action for this is to implement Mappable for that object too.

If you are still unable to fix this issue, provide me some code which will reproduce this issue.

[ArthurSav]

Can't run this example at the moment but it should crash if you follow the steps.

1. Create pojo that implements both Serializable and Mappable.

public class User implements java.io.Serializable, Mappable {
  @Id private Long id;
  private String name;

  @Override public Document write(NitriteMapper mapper) {
    Document document = new Document();
    document.put("id", id);
    document.put("name", name);
    return document;
  }

  @Override public void read(NitriteMapper mapper, Document document) {
    if (document != null) {
      this.id = (Long) document.get("id");
      this.name = (String) document.get("name"));
    }
  }
}

2. Save this pojo as an object repo in nitrite

3. Modify pojo

public class User implements java.io.Serializable, Mappable {
  @Id private Long id;
  private String name;
  private String email;

  @Override public Document write(NitriteMapper mapper) {
    Document document = new Document();
    document.put("id", id);
    document.put("name", name);
    document.put("email", email);
    return document;
  }

  @Override public void read(NitriteMapper mapper, Document document) {
    if (document != null) {
      this.id = (Long) document.get("id");
      this.name = (String) document.get("name"));
      this.email = (String) document.get("email")
    }
  }
}

4. Re-open app and try to fetch the saved pojo

nitrite.getRepository(User.class).find().firstOrDefault()

I believe this is when it crashes

[anidotnet]

I tried to follow exactly as you said. But could not reproduce this issue. Here is my test file

public class GitHubIssueTest {

    @Test
    public void testSerializability() {
        Nitrite db = Nitrite.builder().filePath("/tmp/serial.db").openOrCreate();
        ObjectRepository<User> repository = db.getRepository(User.class);
//        User user = new User();
//        user.id = 1L;
//        user.name = "Nitrite";
//
//        repository.insert(user);

        User user = repository.find().firstOrDefault();
        assertNotNull(user);
		assertEquals(user.name, "Nitrite");
    }

    public class User implements java.io.Serializable, Mappable {
        @Id private Long id;
        private String name;
        private String email; // added this after storing into the repo

        @Override public Document write(NitriteMapper mapper) {
            Document document = new Document();
            document.put("id", id);
            document.put("name", name);
            document.put("email", email); // added this after storing into the repo
            return document;
        }

        @Override public void read(NitriteMapper mapper, Document document) {
            if (document != null) {
                this.id = (Long) document.get("id");
                this.name = (String) document.get("name");
                this.email = (String) document.get("email"); // added this after storing into the repo
            }
        }
    }
}

[vbecas]

I debugged the issue and noticed that the seliarVersionUID for NitriteId was different. ProGuard was stripping the Serializable interface from the NitriteId class. The issue was fixed by adding the following rules to proguard:

-keepnames class * implements java.io.Serializable
-keepclassmembers class * implements java.io.Serializable {
static final long serialVersionUID;
private static final java.io.ObjectStreamField[] serialPersistentFields;
!static !transient ;
private void writeObject(java.io.ObjectOutputStream);
private void readObject(java.io.ObjectInputStream);
java.lang.Object writeReplace();
java.lang.Object readResolve();
}

For more info:

https://www.guardsquare.com/en/products/proguard/manual/examples#serializable

[anidotnet]

@ArthurSav I have tested the same thing on android, but still I am unable to reproduce it. But if the suggestion of @vbecas solves the problem for you then it's good. I am closing the issue for now.

I am adding these pro-guard rules in the example android app for future reference.