dbapi interface

[remdragon]

#I found sqlite3worker extremely helpful to me, and made a number of enhancements that I wanted to share back with the project:

dbapi compatible interface that implements enough features to be a drop-in replacement for 3 different database abstraction libraries that I use. This can be used by replacing "import sqlite3" with "import sqlite3worker as sqlite3"

use Queue objects in place of uuid tokens, this simplifies several code constructs including the back-off delay while waiting for a response.

eliminate auto-commit and support commit calls directly giving user complete control of how often they want to commit.

catch exceptions and throw in caller's thread

eliminate need for "thread_running" attribute by using threading.Thread.join()

Try to protect the user from accidentally opening two different worker objects on the same database.

[dashawn888]

Great. Thank you for your help. =) I'm running into a couple issues and I think we can work through it. The unit tests are failing now. If you look at https://travis-ci.org/dashawn888/sqlite3worker/requests you will see they are failing. If you call python sqlite3worker_test.py this is what I see. https://gist.github.com/anonymous/18d5abc56f3084dd14463844b845aa6f This is what I can do to help. If you give me some example code that you use this library with I can write some unit tests. But because it's failing tests I'm afraid that if I merge in the code it'll break other people that use this library. I think we can get this working. Shoot me some code that uses this library and we can fix the unit tests. Thanks, Shawn

…

On Thu, Aug 10, 2017 at 9:00 PM, Royce Mitchell ***@***.***> wrote: I found sqlite3worker extremely helpful to me, and made a number of enhancements that I wanted to share back with the project: dbapi compatible interface that implements enough features to be a drop-in replacement for 3 different database abstraction libraries that I use. This can be dropped by replacing "import sqlite3" with "import sqlite3worker as sqlite3" use Queue objects in place of uuid tokens, this simplifies several code constructs including the back-off delay while waiting for a response. eliminate auto-commit and support commit calls directly giving user complete control of how often they want to commit. catch exceptions and throw in caller's thread eliminate need for "thread_running" attribute by using threading.Thread.join() Try to protect the user from accidentally opening two different worker objects on the same database. ------------------------------ You can view, comment on, or merge this pull request online at: #3 Commit Summary - Update sqlite3worker.py File Changes - *M* sqlite3worker.py <https://github.com/dashawn888/sqlite3worker/pull/3/files#diff-0> (497) Patch Links: - https://github.com/dashawn888/sqlite3worker/pull/3.patch - https://github.com/dashawn888/sqlite3worker/pull/3.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABE8dL365s9tF5ktFj8208pGjPyuqDgPks5sW9HcgaJpZM4O0Ocj> .

[remdragon]

I'm so sorry I didn't look for tests. I have made a couple fixes to my version of sqlite3worker based on the tests and adjusted the tests to match the new way that sqlite3worker generates errors. All tests now pass. Should I generate a new pull or is there a way to update the existing pull? None of my code that uses sqlite3worker is in github. I'm probably going to head to bed shortly, so here is a dropbox link to a zip file that contains the updated sqlite3worker and test as well as the other code files that have been modified to use it: https://www.dropbox.com/s/bsjh3ay4jok7ys3/dashawn888.zip?dl=0 shelve_sqlite.py is a library that mimics the shelve interface but uses sqlite as the storage backend and adds some features as well. dbms_sqlite.py is a database abstraction library I built a year ago. sqlite is one of the supported database engines. dbms_base.py is common support code for dbms_sqlite.py worm_sqlite.py is a new database abstraction library / ORM I built a couple months ago after learning about peewee, reading it's documentation, and realizing that my dbms library was more laborious than I liked. There are limitations in peewee that make it unsuitable for my purposes, so I built my own ORM. As you can see, all 3 of these libraries have test code and all tests are passing ( dbms_sqlite's test code is embedded in the file - I haven't moved it's test code to a separate test file yet ). I didn't try to make sure you have all the files you need to run these tests because you only asked to see the code, but if there are files missing and you want to run them to confirm for yourself, let me know. I haven't put sqlite3worker in production anywhere yet, but the 3 libraries above are all in production and some of the applications are experiencing database locked errors that I'm hoping your sqlite3worker will be able to fix. I will be putting it in production possibly tomorrow, but definitely next week. Royce Mitchell, IT Consultant ITAS Solutions royce3@itas-solutions.com On Fri, Aug 11, 2017 at 12:06 AM, Shawn Lee <notifications@github.com> wrote:

…

Great. Thank you for your help. =) I'm running into a couple issues and I think we can work through it. The unit tests are failing now. If you look at https://travis-ci.org/dashawn888/sqlite3worker/requests you will see they are failing. If you call python sqlite3worker_test.py this is what I see. https://gist.github.com/anonymous/18d5abc56f3084dd14463844b845aa6f This is what I can do to help. If you give me some example code that you use this library with I can write some unit tests. But because it's failing tests I'm afraid that if I merge in the code it'll break other people that use this library. I think we can get this working. Shoot me some code that uses this library and we can fix the unit tests. Thanks, Shawn On Thu, Aug 10, 2017 at 9:00 PM, Royce Mitchell ***@***.***> wrote: > I found sqlite3worker extremely helpful to me, and made a number of > enhancements that I wanted to share back with the project: > > dbapi compatible interface that implements enough features to be a drop-in > replacement for 3 different database abstraction libraries that I use. This > can be dropped by replacing "import sqlite3" with "import sqlite3worker as > sqlite3" > > use Queue objects in place of uuid tokens, this simplifies several code > constructs including the back-off delay while waiting for a response. > > eliminate auto-commit and support commit calls directly giving user > complete control of how often they want to commit. > > catch exceptions and throw in caller's thread > > eliminate need for "thread_running" attribute by using > threading.Thread.join() > > Try to protect the user from accidentally opening two different worker > objects on the same database. > ------------------------------ > You can view, comment on, or merge this pull request online at: > > #3 > Commit Summary > > - Update sqlite3worker.py > > File Changes > > - *M* sqlite3worker.py > <https://github.com/dashawn888/sqlite3worker/pull/3/files#diff-0> > (497) > > Patch Links: > > - https://github.com/dashawn888/sqlite3worker/pull/3.patch > - https://github.com/dashawn888/sqlite3worker/pull/3.diff > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#3>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/ ABE8dL365s9tF5ktFj8208pGjPyuqDgPks5sW9HcgaJpZM4O0Ocj> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADka1yJ_f83t_-y7fMsPS6ny8YahxUajks5sW-FNgaJpZM4O0Ocj> .

[remdragon]

Shawn,

It looks like you have to manually merge the pull because of the merge conflicts caused by the fact that I forked from palantir instead of yours. Your auto-builder appears to be refusing to test because of the merge conflict if I'm reading that error message correctly.

[remdragon]

All checks are passing. I had to change the tests because they were written to return strings in case of error and I changed the behavior to throw exceptions when errors occur. This may require changes to code that uses your library, but it will make error handling much cleaner and more pythonic.

[remdragon]

It took longer than I had expected, but the version of the code I just committed is now in production ( for the last 24 hours ) as a drop-in replacement for sqlite3 using the dbapi interface in a multi-threaded application that was experiencing database locked errors. I will update here if I run into any further issues.