80 Commits

Author SHA1 Message Date
Luiz Carvalho
c09532864d feat: better database errors (#2109)
* feat(core): create error kind enum

* feat(core): add error kind for postgres

* feat(core): add error kind for sqlite

* feat(core): add error kind for mysql

* test(postgres): add error tests

* test(sqlite): add error tests

* test(mysql): add error tests

* fix(tests): fix tests rebasing

* refac(errors): add `ErrorKind::Other` variant
2023-02-21 13:25:25 -08:00
tyrelr
5378dea6af Sqlite describe fixes (#2253)
* add failing test for nested orderby

* log query paths which were abandoned due to invalid state or looping.  Allow instructions to be executed a small number of times to fix nested order by query

* add failing testcase using nested orderby

* fix handling of sequence/offset and rewind

* fix handling when sqlite nests records inside of records

* add test of temporary table handling

* WIP add test failure for temp table access

* fix support for temp tables

* add tests for sqlite datetime functions

* add basic date and time function support

* handle gosub opcode correctly

* add group by test

* fix group by handling

* add additional passing group by test

* add test case for simple limit query

* fix IfPos & If touching wrong branches state, fix IfPos using wrong branch criteria

* add test for large offsets

* add short-circuit for possible query offset loops

* add groupby query that is predicted incorrectly

* fix handling of integer cast failures

* add tests for single-row aggregate results

* fix handling of null-based branching

* add test for coercion of text by sum

* fix calculation of sum value coercion

* add failing test for recursive with query

* add logic for delete operation to fix queries grouping by columns from a recursive query
2023-02-21 13:25:25 -08:00
Austin Bonander
b5312c3b6f Break drivers out into separate crates, clean up some technical debt (#2039)
* WIP rt refactors

* refactor: break drivers out into separate crates

also cleans up significant technical debt
2023-02-21 13:25:25 -08:00
tyrelr
d5b8c66e24 Fix sqlite update return and order by type inference (#1960)
* add failing test cases for update/delete return into

* fix regression in null tracking by improving tracking of cursor empty/full state

* add failing test case for order by column types

* Add support for SorterOpen,SorterInsert,SorterData

* add failing test case for unions

* fix range copy/move implementation

* fix wrong copy/move range

* remove calls to dbg!
2023-02-21 13:25:25 -08:00
Marco Neumann
5e56da87e0
fix: ensure migration progress is not lost for PG, mysql and sqlite (#1991)
* fix: ensure migration progress is not lost for PG

Fixes #1966.

* fix: ensure migration progress is not lost for sqlite

This is similar to #1966.

* fix: ensure reverse migration progress is not lost for PG

See #1966.

* fix: ensure reverse migration progress is not lost for sqlite

See #1966.

* fix: ensure migration progress is not lost for mysql

This is similar to #1966.

* fix: ensure reverse migration progress is not lost for mysql

See #1966.

* test: check migration type as well

* test: extend migrations testing

* fix: work around MySQL implicit commits

* refactor: simplify migration testing
2022-09-12 17:52:04 -07:00
Richard Bradfield
20877d83fd
Add extension support for SQLite (#2062)
* Add extension support for SQLite

While SQLite supports loading extensions at run-time via either the C
API or the SQL interface, they strongly recommend [1] only enabling the C
API so that SQL injections don't allow attackers to run arbitrary
extension code.

Here we take the most conservative approach, we enable only the C
function, and then only when the user requests extensions be loaded in
their `SqliteConnectOptions`, and disable it again once we're done
loading those requested modules. We don't add any support for loading
extensions via environment variables or connection strings.

Extensions in the options are stored as an IndexMap as the load order
can have side effects, they will be loaded in the order they are
supplied by the caller.

Extensions with custom entry points are supported, but a default API
is exposed as most users will interact with extensions using the
defaults.

[1]: https://sqlite.org/c3ref/enable_load_extension.html

* Add extension testing for SQlite

Extends x.py to download an appropriate shared object file for supported
operating systems, and uses wget to fetch one into the GitHub Actions
context for use by CI.

Overriding LD_LIBRARY_PATH for only this specific DB minimises the
impact on the rest of the suite.
2022-09-01 15:03:27 -07:00
szymek156
c931cab95f
Szymek156/issue2009 (#2014)
* SQLite: Execute SQLCipher pragmas as very first operations on the database

SQLCipher requires, apart from 'key' pragma also other cipher-related
to be executed before read/write to the database.

* Added tests for SQLCipher functionality

* remove default-features from libsqlite3-sys when building from dev-dependencies

Co-authored-by: Szymon Zimnowoda <szimnowoda.memri@gmail.com>
2022-08-05 12:20:14 -07:00
Austin Bonander
054f61980a
feat: implement testing utilities (#2001) 2022-08-02 14:38:12 -07:00
Austin Bonander
a2eceec33b
chore: replace dotenv with dotenvy (#2003)
* chore: replace `dotenv` with `dotenvy`

The former appears to be unmaintained and the latter is a drop-in replacement.

* chore: fix all warnings
2022-07-28 14:33:44 -07:00
Austin Bonander
bc3e70545b
sqlite improvements (#1965)
* use direct blocking calls for SQLite in `sqlx_macros`
    * this also ensures the database is closed properly, cleaning up tempfiles
* don't send `PRAGMA journal_mode` unless set
    * this previously defaulted to WAL mode which is a permanent setting
      on databases which doesn't necessarily apply to all use-cases
    * changing into or out of WAL mode acquires an exclusive lock on the database
      that can't be waited on by `sqlite3_busy_timeout()`
    * for consistency, `sqlx-cli` commands that create databases will still
      create SQLite databases in WAL mode; added a flag to disable this.
* in general, don't send `PRAGMA`s unless different than default
    * we were sending a bunch of `PRAGMA`s with their default values just to enforce
      an execution order on them, but we can also do this by inserting empty slots
      for their keys into the `IndexMap`
* add error code to `SqliteError` printout
* document why `u64` is not supported
2022-07-12 13:59:37 -07:00
John B Codes
cfef70a796
Add Sqlite support for the time crate (#1865)
* feat(sqlite): Add 'time' crate support for date/time types
docs(sqlite): Update types module docs for JSON and Chrono
docs(mysql): Update types module docs for JSON

* More efficient time crate decoding with FormatItem::First and hand-crafting of format descriptions

* Replace temporary testing code with original intention

* Replace duplicated formatting test with intended test

* Performance improvements to decoding OffsetDateTime, PrimitiveDateTime, and Time

* Use correct iteration for OffsetDateTime

* Reduce visibility of format constants

Co-authored-by: John B Codes <johnbcodes@users.noreply.github.com>
2022-07-08 16:51:50 -07:00
Kian-Meng Ang
d52f301a94
Fix typos (#1894)
* Fix typos

* Update CHANGELOG.md

Co-authored-by: Austin Bonander <austin.bonander@gmail.com>

Co-authored-by: Austin Bonander <austin.bonander@gmail.com>
2022-06-08 14:56:56 -07:00
tyrelr
ed56622526
Improve Sqlite support for sub-queries and CTE's (#1816)
* reproduce incorrect nullability for materialized views

* split ephemeral/index-only table handling from real table handling

* add test for literal null, expect nullability to be identified from table information

* gather interpreter state into a struct, no change in behaviour

* prevent infinite loops that could arise once branching is supported

* track nullability alongside the datatype instead of in a separate lookup

* implement basic comprehension of branching opcodes

* fix datatype calculation of aggregates which are never 'stepped' through

* implement coroutine and return operations, including tracking of 'actual' integer value stored in the register by Integer/InitCoroutine/Yield operations.

* strip unnecessary history field out

* Modify variable test to expect bind-variable outputs to be nullable, rather than unknown

* add partially commented-out union tests, simplify code to satisfy simplest union case

* fix unit test incorrectly expecting primary keys to be implicitly not-null

* add failing test for recursive tables

* add logging of query explain plan

* track explain plan execution history

* broken RowData implementation (doesn't alias)

* Implement OpenPseudo tables as an alias of a register value

* fix comment

* clean up logging code warnings

* use cfg to omit QueryPlanLogger unless sqlite feature is used
2022-06-07 14:24:08 -07:00
Paolo Barbolini
ec15f6b30c
Update uuid crate to v1 (#1821) 2022-04-20 12:48:29 -07:00
05storm26
f328cc15d8
Sqlite chrono::DateTime<FixedOffset> timezone fix (#1618) 2022-04-14 15:06:55 -07:00
tyrelr
217dc55062
Fix #1249 Left joins in SQLite can break the query macros (#1789)
* Reproduce github issue#1249: Left joins in sqlite can break the query macros

* Fix panic caused by unknown cursor columns when executing NullRow command. Fixes #1249
2022-04-07 13:18:37 -07:00
Adam Cigánek
347374b94b
SQLite unlock notification (#1658)
* sqlite: add test for concurrent table access (failing)

* sqlite: implement unlock notification
2022-02-15 20:10:36 -08:00
Austin Bonander
fdbfc5dfc3
Prepare 0.5.10 release (#1603)
* fix(cli): change new `rustls` and `native-tls` features to use correct runtime feature

* chore: upgrade SQLx crates to 0.5.10, upgrade all dependencies to latest versions

chore(cli): upgraded `clap` to `3.0.0-rc.9`

* fix(tests/sqlite): ignore `issue_1467()` as spuriously failing

I'm well aware of the principle that a spuriously failing test is a failing test, but even though I have it outputting the seed used with a reproducible PRNG, I can't reproduce the failures locally, so 🤷.

* chore: add CHANGELOG entry for 0.5.10
2021-12-29 17:25:49 -08:00
Austin Bonander
63ca2ccc6c
refactor(sqlite): make background thread responsible for all FFI calls (#1551) 2021-12-29 15:23:02 -08:00
Austin Bonander
1b5dd6514b
preparing 0.5.8 release (#1466)
* preparing 0.5.8 release

* fix warnings before release
2021-10-01 14:45:25 -07:00
Andrew Whitehead
ba3e373b7e
Shut down statement worker in Sqlite Connection::close (#1453)
* add explicit shutdown of sqlite statement worker in Connection::close()

Signed-off-by: Andrew Whitehead <cywolf@gmail.com>

* test sqlite database close method

Signed-off-by: Andrew Whitehead <cywolf@gmail.com>

* await worker shutdown after dropping SqliteConnection

Signed-off-by: Andrew Whitehead <cywolf@gmail.com>

* restore explicit drop

Signed-off-by: Andrew Whitehead <cywolf@gmail.com>
2021-09-30 18:14:30 -07:00
Austin Bonander
8b30f3059b
Fix a panic in the worker thread when dropping the connection while SqliteRows still exist (#1450)
* chore(sqlite): add repro for #1419

* fix(sqlite): hold a reference to the connection in `SqliteRow`

fixes #1419
2021-09-22 16:55:22 -07:00
Montana Low
ec510b37e7
Finish support for Postgres COPY (#1345)
* feat(postgres): WIP implement `COPY FROM/TO STDIN`

Signed-off-by: Austin Bonander <austin@launchbadge.com>

* feat(postgres): WIP implement `COPY FROM/TO STDIN`

Signed-off-by: Austin Bonander <austin@launchbadge.com>

* test and complete support for postgres copy

Co-authored-by: Austin Bonander <austin@launchbadge.com>
2021-09-13 13:03:38 -07:00
Austin Bonander
71388a7ef2
sqlite: fix a couple segfaults (#1351)
* sqlite: use Arc instead of Copy-able StatementHandle

This guarantees that StatementHandle is never used after calling
`sqlite3_finalize`. Now `sqlite3_finalize` is only called when
StatementHandle is dropped.

(cherry picked from commit 5eebc05dc371512bae14cf94498087bdadeddec0)

* sqlite: use Weak poiter to StatementHandle in the worker

Otherwise some tests fail to close connection.

(cherry picked from commit 5461eeeee30772e54e8874f60805b04bdc989278)

* Fix segfault due to race condition in sqlite (#1300)

(cherry picked from commit bb62cf767e3e44896bf4607da8e18237241ed170)

* fix(sqlite): run `sqlite3_reset()` in `StatementWorker`

this avoids possible race conditions without using a mutex

* fix(sqlite): have `StatementWorker` keep a strong ref to `ConnectionHandle`

this should prevent the database handle from being finalized before all statement handles
have been finalized

* fix(sqlite/test): make `concurrent_resets_dont_segfault` runtime-agnostic

Co-authored-by: link2xt <link2xt@testrun.org>
Co-authored-by: Adam Cigánek <adam.ciganek@gmail.com>
2021-08-16 14:39:45 -07:00
Atkins
9f7205e80f
Fix GitHub Actions and integration test (#1346)
* fix test suite

* rustfmt

* need Row

* test: fix integration test scripts and update the upstream supported databases

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* ci(actions): update supported databases

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* ci(actions): use `pg_isready` instead of `sleep` to avoid error cause by database not ready

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* feat(core): add `trait PgConnectionInfo` for connection parameter status from server

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* test(postgres): fix integration test for postgres

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* test(mysql): fix integration tests

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* ci(actions): test database against the oldest and newest supported versions

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* docs(core): document `trait PgConnectionInfo`

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

Co-authored-by: Montana Low <montanalow@gmail.com>
2021-07-28 14:00:34 -07:00
marshoepial
cb3ff28721
Keep track of column typing in SQLite EXPLAIN parsing (#1323)
* NewRowid, Column opcodes, better pointer handling

* Implement tracking of column typing on sqlite explain parser

* fmt for sqlite column typing for explain parsing

Co-authored-by: marshoepial <marshoepial@gmail.com>
2021-07-21 16:27:47 -07:00
link2xt
bd06836fba test(sqlite): extend it_caches_statements() test
Test that initial PRAGMA statements and &str statements are not
cached, but sqlx::query() statements are cached by default.
2021-04-15 18:26:06 -07:00
Alexander Krotov
78656eb469 fix(sqlite): reset the statement when fetch_many() stream is dropped
Unlike `Executor.fetch_optional()`, `Executor.fetch_many()` does not
have a single exit.  The stream can be dropped at any time.  To catch
this event, we create a `StatementResetter` structure inside the stream
loop and reset the statement when it is dropped.

A test case `it_resets_prepared_statement_after_fetch_many` is
similar to `it_resets_prepared_statement_after_fetch_one` which tests
`Executor.fetch_optional()`.
2021-04-09 00:23:17 -07:00
crajcan
469c4ac130
Correct Postgres' Length Limit for Bind Parameters (#1141)
* Bump limit for postgres' bind parameters

* Fix lingering fmt error
2021-03-29 19:39:27 -07:00
Ryan Leckey
b6e1275617
fix: derive(Type) should emit a forwarded impl for the compatible function 2021-03-18 16:12:00 -07:00
Austin Bonander
6b036055e5
format: run rustfmt 2021-01-20 22:26:48 -08:00
Austin Bonander
7c32928ebc
feat(macros): implement query_scalar!() and variants
Signed-off-by: Austin Bonander <austin@launchbadge.com>
2021-01-20 22:23:04 -08:00
Ryan Leckey
b0c430ed18
feat(sqlite): track nullable through left joins 2021-01-20 22:22:26 -08:00
Marcin Kaźmierczak
8328f893e4 sqlite test - prepared statements are reset after fetch_one 2021-01-20 22:10:59 -08:00
Jonas Platte
a1d562f04b
Remove the Done trait 2021-01-12 14:37:44 +01:00
Ryan Leckey
e9e1dcf78e
style: rustfmt 2020-12-19 00:08:29 -08:00
Ryan Leckey
4f261b7346
test: added an explicit cast for bstr test where sqlite failed to do an implicit cast 2020-12-18 21:56:50 -08:00
James Tomlinson
a959451e85
Add Sqlite Uuid type tests. 2020-12-18 21:33:49 -08:00
Josh Triplett
603c1950fe Add tests for git2 2020-11-21 14:32:29 -08:00
Josh Triplett
d396e5a948 Add tests for bstr 2020-11-21 14:32:29 -08:00
David Yamnitsky
12b4250454 support sqlite bind parameters of the form $NNN 2020-11-06 15:08:08 -08:00
Peter Maatman
5b7f3ab0e3 sqlite: Add test for JSON functions 2020-10-20 20:08:20 -07:00
Peter Maatman
fae72200e5 sqlite: Add basic json implementation
Fixes #608
2020-10-20 20:08:20 -07:00
Raphaël Thériault
ced09e0545
Support using both nullability and type overrides (#549)
* Make it possible to use both nullability and type overrides

* Fix override parsing lookahead logic

* Update column override tests

* Support nullability overrides with wildcard type overrides

* Fix tests

* Update query! overrides docs

* Remove last bits of macro_result!

* rustfmt
2020-07-27 00:43:35 -07:00
Ryan Leckey
e575501a39 feat: add Executor::prepare, a hook into the automatic statement preparation life-cycle 2020-07-24 07:24:23 -07:00
Julius de Bruijn
e8a4c54ac7 Allow setting caching per-query 2020-07-24 07:24:23 -07:00
Julius de Bruijn
590f97df4a Caching describe 2020-07-24 07:24:23 -07:00
Ryan Leckey
96b76dc737 update todo examples to 0.4 and use migrations
* update tests/x.py

 * add examples/x.py
2020-07-18 05:43:16 -07:00
Ryan Leckey
f4eb968c8d test(sqlite): add DB parameter to Decode invocation 2020-07-17 03:22:40 -07:00
Ryan Leckey
6949b84e8c style: rustfmt 2020-07-15 02:31:57 -07:00