diff --git a/.github/workflows/development.yaml b/.github/workflows/development.yaml index 6eef3970..39d2d61c 100644 --- a/.github/workflows/development.yaml +++ b/.github/workflows/development.yaml @@ -37,3 +37,22 @@ jobs: COMPOSE_HTTP_TIMEOUT: "120" run: | docker-compose -f LNX-docker-compose.yaml up --build --exit-code-from app + publish-docs: + if: | + github.event_name == 'push' && + startsWith(github.ref, 'refs/tags') + needs: test + runs-on: ubuntu-latest + env: + DOCKER_CLIENT_TIMEOUT: "120" + COMPOSE_HTTP_TIMEOUT: "120" + steps: + - uses: actions/checkout@v2 + - name: Deploy docs + run: | + export MODE=BUILD + export PACKAGE=datajoint + export UPSTREAM_REPO=https://github.com/${GITHUB_REPOSITORY}.git + export HOST_UID=$(id -u) + docker compose -f docs/docker-compose.yaml up --exit-code-from docs --build + git push origin gh-pages diff --git a/.github/workflows/publish-docs.yaml b/.github/workflows/publish-docs.yaml new file mode 100644 index 00000000..7f39245e --- /dev/null +++ b/.github/workflows/publish-docs.yaml @@ -0,0 +1,22 @@ +name: Publish Docs +on: + push: + tags: + - 'test*.*.*' + workflow_dispatch: +jobs: + publish-docs: + runs-on: ubuntu-latest + env: + DOCKER_CLIENT_TIMEOUT: "120" + COMPOSE_HTTP_TIMEOUT: "120" + steps: + - uses: actions/checkout@v2 + - name: Deploy docs + run: | + export MODE=BUILD + export PACKAGE=datajoint + export UPSTREAM_REPO=https://github.com/${GITHUB_REPOSITORY}.git + export HOST_UID=$(id -u) + docker compose -f docs/docker-compose.yaml up --exit-code-from docs --build + git push origin gh-pages diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..aa03919b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,12 @@ +# Changelog + +Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and +[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention. + +## [3.5.0] - 2022-03-21 + ++ Bugfix: Cascading delete for renamed foreign keys (#379) PR #386 ++ Minor: Add renaming the same attribute multiple times within a single projection PR #386 ++ Minor: Add config for reading values with 32-bit dimensions (datajoint/mym#86) PR #395 + +[3.5.0]: https://github.com/datajoint/element-deeplabcut/releases/tag/3.5.0 diff --git a/README.md b/README.md index c6c54f1a..df30eacb 100644 --- a/README.md +++ b/README.md @@ -1,68 +1,28 @@ [![View DataJoint on File Exchange](https://www.mathworks.com/matlabcentral/images/matlab-file-exchange.svg)](https://www.mathworks.com/matlabcentral/fileexchange/63218-datajoint) # Welcome to DataJoint for MATLAB! -DataJoint for MATLAB is a high-level programming interface for relational databases designed to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data. - -DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers. -Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com). Related resources are listed at https://datajoint.org. - -## Installation -
-Click to expand details - -### (Recommended) Greater than R2016b - -1. Utilize MATLAB built-in GUI i.e. *Top Ribbon -> Add-Ons -> Get Add-Ons* -2. Search and Select `DataJoint` -3. Select *Add from GitHub* - -### Using GHToolbox (FileExchange Community Toolbox) - -1. Install *GHToolbox* using using an appropriate method in https://github.com/datajoint/GHToolbox -2. run: `ghtb.install('datajoint/datajoint-matlab')` - -### Less than R2016b - -1. Utilize MATLAB built-in GUI i.e. *Top Ribbon -> Add-Ons -> Get Add-Ons* -2. Search and Select `DataJoint` -3. Select *Download from GitHub* -4. Save `DataJoint.mltbx` locally -5. Navigate in MATLAB tree browser to saved toolbox file -6. Right-Click and Select *Install* -7. Select *Install* - -### From Source - -1. Download `DataJoint.mltbx` locally -2. Navigate in MATLAB tree browser to saved toolbox file -3. Right-Click and Select *Install* -4. Select *Install* -
- -## Config -For help in utilizing `dj.config` (added in `3.4.0`), you may access the help via `help('dj.config')` or review it online [here](https://github.com/datajoint/datajoint-matlab/blob/c2bd6b3e195dfeef773d4e12bad5573c461193b0/%2Bdj/config.m#L2-L27). Formal documentation to follow. - -## Documentation and Tutorials +DataJoint for MATLAB is a high-level programming interface for relational databases designed to support data processing chains in science labs. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, and querying data. -* https://datajoint.org -- start page -* https://docs.datajoint.org -- up-to-date documentation -* https://tutorials.datajoint.io -- step-by-step tutorials -* https://elements.datajoint.org -- catalog of example pipelines -* https://codebook.datajoint.io -- interactive online tutorials +For more information, see our +[general](https://datajoint.com/docs/welcome/) and +[MATLAB](https://datajoint.com/docs/core/datajoint-matlab/) documentation pages. ## Citation + + If your work uses DataJoint for MATLAB, please cite the following Research Resource Identifier (RRID) and manuscript. + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for MATLAB (version ``) + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658 -## Running Tests Locally +## For Developers: Running Tests Locally +
Click to expand details -* Create an `.env` with desired development environment values e.g. ++ Create an `.env` with desired development environment values e.g. + ``` sh MATLAB_USER=rguzman MATLAB_LICENSE=IyBCRUd... # For image usage instructions see https://github.com/guzman-raphael/matlab, https://hub.docker.com/r/raphaelguzman/matlab @@ -73,11 +33,12 @@ MATLAB_GID=1000 MYSQL_TAG=5.7 MINIO_VER=RELEASE.2022-01-03T18-22-58Z ``` -* `cp local-docker-compose.yaml docker-compose.yaml` -* `docker-compose up` (Note configured `JUPYTER_PASSWORD`) -* Select a means of running MATLAB e.g. Jupyter Notebook, GUI, or Terminal (see bottom) -* Add `tests` directory to path e.g. in MATLAB, `addpath('tests')` -* Run desired tests. Some examples are as follows: + ++ `cp local-docker-compose.yaml docker-compose.yaml` ++ `docker-compose up` (Note configured `JUPYTER_PASSWORD`) ++ Select a means of running MATLAB e.g. Jupyter Notebook, GUI, or Terminal (see bottom) ++ Add `tests` directory to path e.g. in MATLAB, `addpath('tests')` ++ Run desired tests. Some examples are as follows: | Use Case | MATLAB Code | | ---------------------------- | ------------------------------------------------------------------------------ | @@ -86,23 +47,20 @@ MINIO_VER=RELEASE.2022-01-03T18-22-58Z | Run one specific test | `runtests('TestTls/TestTls_testInsecureConn')` | | Run tests based on test name | `import matlab.unittest.TestSuite;`
`import matlab.unittest.selectors.HasName;`
`import matlab.unittest.constraints.ContainsSubstring;`
`suite = TestSuite.fromClass(?Main, ... `
    `HasName(ContainsSubstring('Conn')));`
`run(suite)`| - ### Launch Jupyter Notebook -* Navigate to `localhost:8888` -* Input Jupyter password -* Launch a notebook i.e. `New > MATLAB` - ++ Navigate to `localhost:8888` ++ Input Jupyter password ++ Launch a notebook i.e. `New > MATLAB` ### Launch MATLAB GUI (supports remote interactive debugger) -* Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash` -* Launch Matlab by runnning command `matlab` - ++ Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash` ++ Launch Matlab by running command `matlab` ### Launch MATLAB Terminal -* Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash` -* Launch Matlab with no GUI by runnning command `matlab -nodisplay` ++ Shell into `datajoint-matlab_app_1` i.e. `docker exec -it datajoint-matlab_app_1 bash` ++ Launch Matlab with no GUI by running command `matlab -nodisplay` -
\ No newline at end of file + diff --git a/docs-parts/admin/5-blob-config_lang1.rst b/docs-parts/admin/5-blob-config_lang1.rst deleted file mode 100644 index 15bf13b7..00000000 --- a/docs-parts/admin/5-blob-config_lang1.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. note:: - - External storage is not yet implemented in MATLAB. - The feature will be added in an upcoming release: https://github.com/datajoint/datajoint-matlab/issues/143 diff --git a/docs-parts/admin/5-blob-config_lang2.rst b/docs-parts/admin/5-blob-config_lang2.rst deleted file mode 100644 index efae4392..00000000 --- a/docs-parts/admin/5-blob-config_lang2.rst +++ /dev/null @@ -1 +0,0 @@ -Use ``dj.config`` for configuration. diff --git a/docs-parts/admin/5-blob-config_lang3.rst b/docs-parts/admin/5-blob-config_lang3.rst deleted file mode 100644 index 974c5f07..00000000 --- a/docs-parts/admin/5-blob-config_lang3.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. note:: - - The cache folder is not yet implemented in MATLAB. - The feature will be added in an upcoming release: https://github.com/datajoint/datajoint-matlab/issues/143 diff --git a/docs-parts/admin/5-blob-config_lang4.rst b/docs-parts/admin/5-blob-config_lang4.rst deleted file mode 100644 index 15bf13b7..00000000 --- a/docs-parts/admin/5-blob-config_lang4.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. note:: - - External storage is not yet implemented in MATLAB. - The feature will be added in an upcoming release: https://github.com/datajoint/datajoint-matlab/issues/143 diff --git a/docs-parts/admin/5-blob-config_lang5.rst b/docs-parts/admin/5-blob-config_lang5.rst deleted file mode 100644 index 15bf13b7..00000000 --- a/docs-parts/admin/5-blob-config_lang5.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. note:: - - External storage is not yet implemented in MATLAB. - The feature will be added in an upcoming release: https://github.com/datajoint/datajoint-matlab/issues/143 diff --git a/docs-parts/computation/01-autopopulate_lang1.rst b/docs-parts/computation/01-autopopulate_lang1.rst deleted file mode 100644 index 6ecbf940..00000000 --- a/docs-parts/computation/01-autopopulate_lang1.rst +++ /dev/null @@ -1,23 +0,0 @@ - -.. code-block:: MATLAB - - %{ - # Filtered image - -> test.Image - --- - filtered_image : longblob - %} - - classdef FilteredImage < dj.Computed - methods(Access=protected) - function make(self, key) - img = fetch1(test.Image & key, 'image'); - key.filtered_image = myfilter(img); - self.insert(key) - end - end - end - -.. note:: Currently matlab uses ``makeTuples`` rather than ``make``. This will be fixed in an upcoming release: https://github.com/datajoint/datajoint-matlab/issues/141 - -The ``make`` method receives one argument: the struct ``key`` containing the primary key value of an element of :ref:`key source ` to be worked on. diff --git a/docs-parts/computation/01-autopopulate_lang2.rst b/docs-parts/computation/01-autopopulate_lang2.rst deleted file mode 100644 index ea6e781e..00000000 --- a/docs-parts/computation/01-autopopulate_lang2.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - populate(test.FilteredImage) diff --git a/docs-parts/computation/01-autopopulate_lang3.rst b/docs-parts/computation/01-autopopulate_lang3.rst deleted file mode 100644 index 3f25bb5f..00000000 --- a/docs-parts/computation/01-autopopulate_lang3.rst +++ /dev/null @@ -1,3 +0,0 @@ -Behavior of the ``populate`` method depends on the number of output arguments requested in the function call. -When no output arguments are requested, errors will halt population. -With two output arguments (``failedKeys`` and ``errors``), ``populate`` will catch any encountered errors and return them along with the offending keys. diff --git a/docs-parts/computation/01-autopopulate_lang4.rst b/docs-parts/computation/01-autopopulate_lang4.rst deleted file mode 100644 index 851fed25..00000000 --- a/docs-parts/computation/01-autopopulate_lang4.rst +++ /dev/null @@ -1,28 +0,0 @@ -The function ``parpopulate`` works identically to ``populate`` except that it uses a job reservation mechanism to allow multiple processes to populate the same table in parallel without collision. -When running ``parpopulate`` for the first time, DataJoint will create a job reservation table and its class ``.Jobs`` with the following declaration: - -.. code-block:: matlab - - {% - # the job reservation table - table_name : varchar(255) # className of the table - key_hash : char(32) # key hash - --- - status : enum('reserved','error','ignore')# if tuple is missing, the job is available - key=null : blob # structure containing the key - error_message="" : varchar(1023) # error message returned if failed - error_stack=null : blob # error stack if failed - host="" : varchar(255) # system hostname - pid=0 : int unsigned # system process id - timestamp=CURRENT_TIMESTAMP : timestamp # automatic timestamp - %} - -A job is considered to be available when ``.Jobs`` contains no matching entry. - -For each ``make`` call, ``parpopulate`` sets the job status to ``reserved``. -When the job is completed, the record is removed. -If the job results in error, the job record is left in place with the status set to ``error`` and the error message and error stacks saved. -Consequently, jobs that ended in error during the last execution will not be attempted again until you delete the corresponding entities from ``.Jobs``. - -The primary key of the jobs table comprises the name of the class and a 32-character hash of the job's primary key. -However, the key is saved in a separate field for error debugging purposes. diff --git a/docs-parts/computation/02-keysource_lang1.rst b/docs-parts/computation/02-keysource_lang1.rst deleted file mode 100644 index 6db0f642..00000000 --- a/docs-parts/computation/02-keysource_lang1.rst +++ /dev/null @@ -1 +0,0 @@ -A custom key source can be configured by setting the ``keySource`` property within a table's ``classdef`` block, using MATLAB's `dependent properties `_ syntax. diff --git a/docs-parts/computation/02-keysource_lang2.rst b/docs-parts/computation/02-keysource_lang2.rst deleted file mode 100644 index e9d53ce0..00000000 --- a/docs-parts/computation/02-keysource_lang2.rst +++ /dev/null @@ -1,17 +0,0 @@ -.. code-block:: matlab - - %{ - -> Recording - --- - sample_rate : float - eeg_data : longblob - %} - classdef EEG < dj.Imported - - methods - function q = get.keySource(self) - q = ephys.Recording & 'recording_type = "EEG"' - end - end - - end diff --git a/docs-parts/computation/04-master-part_lang1.rst b/docs-parts/computation/04-master-part_lang1.rst deleted file mode 100644 index 0a28aea6..00000000 --- a/docs-parts/computation/04-master-part_lang1.rst +++ /dev/null @@ -1,52 +0,0 @@ - -In MATLAB, the master and part tables are declared in a separate ``classdef`` file. -The name of the part table must begin with the name of the master table. -The part table must declare the property ``master`` containing an object of the master. - -``+test/Segmentation.m`` - -.. code-block:: matlab - - %{ - # image segmentation - -> test.Image - %} - classdef Segmentation < dj.Computed - methods(Access=protected) - function make(self, key) - self.insert(key) - make(test.SegmentationRoi, key) - end - end - end - -``+test/SegmentationROI.m`` - -.. code-block:: matlab - - %{ - # Region of interest resulting from segmentation - -> test.Segmentation - roi : smallint # roi number - --- - roi_pixels : longblob # indices of pixels - roi_weights : longblob # weights of pixels - %} - - classdef SegmentationROI < dj.Part - properties(SetAccess=protected) - master = test.Segmentation - end - methods - function make(self, key) - image = fetch1(test.Image & key, 'image'); - [roi_pixels, roi_weighs] = mylib.segment(image); - for roi=1:length(roi_pixels) - entity = key; - entity.roi_pixels = roi_pixels{roi}; - entity.roi_weights = roi_weights{roi}; - self.insert(entity) - end - end - end - end diff --git a/docs-parts/computation/04-master-part_lang2.rst b/docs-parts/computation/04-master-part_lang2.rst deleted file mode 100644 index e0a49073..00000000 --- a/docs-parts/computation/04-master-part_lang2.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - populate(Segmentation) diff --git a/docs-parts/computation/04-master-part_lang3.rst b/docs-parts/computation/04-master-part_lang3.rst deleted file mode 100644 index 468e2d6d..00000000 --- a/docs-parts/computation/04-master-part_lang3.rst +++ /dev/null @@ -1,51 +0,0 @@ - -``+test/ArrayResponse.m`` - -.. code-block:: matlab - - %{ - -> Probe - array: int - %} - classdef ArrayResponse < dj.Computed - methods(Access=protected) - function make(self, key) - self.insert(key) - make(test.ArrayResponseElectrodeResponse, key) - end - end - end - -``+test/ArrayResponseElectrodeResponse.m`` - -.. code-block:: matlab - - %{ - -> test.ArrayResponse - electrode : int % electrode number on the probe - %} - classdef ArrayResponseElectrodeResponse < dj.Part - methods(SetAccess=protected) - function make(self, key) - self.insert(key) - end - end - end - -``+test/ArrayResponseChannelResponse.m`` - -.. code-block:: matlab - - %{ - -> test.ArrayResponseElectrodeResponse - channel: int - --- - response: longblob % response of a channel - %} - classdef ArrayResponseChannelResponse < dj.Part - methods(SetAccess=protected) - function make(self, key) - self.insert(key) - end - end - end diff --git a/docs-parts/computation/06-distributed-computing_kill_order_by.rst b/docs-parts/computation/06-distributed-computing_kill_order_by.rst deleted file mode 100644 index f9aed23a..00000000 --- a/docs-parts/computation/06-distributed-computing_kill_order_by.rst +++ /dev/null @@ -1,14 +0,0 @@ - -For example, to sort the output by hostname in descending order: - -.. code-block:: matlab - - - dj.kill('', dj.conn, 'host desc'); - - ID USER HOST DB COMMAND TIME STATE INFO TIME_MS ROWS_SENT ROWS_EXAMINED - +--+ +----+ +---------+ +--+ +-------+ +----+ +-----+ +----+ +-------+ +---------+ +-------------+ - 35 cat localhost:38772 Sleep 94 94040 0 0 - 36 cat localhost:36543 Sleep 68 68421 1 0 - - process to kill ('q'-quit, 'a'-all) > q diff --git a/docs-parts/computation/06-distributed-computing_lang1.rst b/docs-parts/computation/06-distributed-computing_lang1.rst deleted file mode 100644 index d1e5cae6..00000000 --- a/docs-parts/computation/06-distributed-computing_lang1.rst +++ /dev/null @@ -1,2 +0,0 @@ - -Job reservations are activated by replacing ``populate`` calls with identical ``parpopulate`` calls. diff --git a/docs-parts/computation/06-distributed-computing_lang2.rst b/docs-parts/computation/06-distributed-computing_lang2.rst deleted file mode 100644 index 0dbd34a4..00000000 --- a/docs-parts/computation/06-distributed-computing_lang2.rst +++ /dev/null @@ -1,3 +0,0 @@ - -.. todo: matlab - diff --git a/docs-parts/computation/06-distributed-computing_lang3.rst b/docs-parts/computation/06-distributed-computing_lang3.rst deleted file mode 100644 index 3eb23a59..00000000 --- a/docs-parts/computation/06-distributed-computing_lang3.rst +++ /dev/null @@ -1,3 +0,0 @@ - -.. todo: similarly, in matlab (blah) - diff --git a/docs-parts/computation/06-distributed-computing_lang4.rst b/docs-parts/computation/06-distributed-computing_lang4.rst deleted file mode 100644 index 3eb23a59..00000000 --- a/docs-parts/computation/06-distributed-computing_lang4.rst +++ /dev/null @@ -1,3 +0,0 @@ - -.. todo: similarly, in matlab (blah) - diff --git a/docs-parts/computation/06-distributed-computing_lang5.rst b/docs-parts/computation/06-distributed-computing_lang5.rst deleted file mode 100644 index 3eb23a59..00000000 --- a/docs-parts/computation/06-distributed-computing_lang5.rst +++ /dev/null @@ -1,3 +0,0 @@ - -.. todo: similarly, in matlab (blah) - diff --git a/docs-parts/concepts/04-Integrity_lang1.rst b/docs-parts/concepts/04-Integrity_lang1.rst deleted file mode 100644 index d3d4d495..00000000 --- a/docs-parts/concepts/04-Integrity_lang1.rst +++ /dev/null @@ -1,25 +0,0 @@ -``+test/Mouse.m`` - -.. code-block:: matlab - - %{ - mouse_name : varchar(64) - --- - mouse_dob : datetime - %} - - classdef Mouse < dj.Manual - end - -``+test/MouseDeath.m`` - -.. code-block:: matlab - - %{ - -> test.Mouse - --- - death_date : datetime - %} - - classdef MouseDeath < dj.Manual - end diff --git a/docs-parts/concepts/04-Integrity_lang2.rst b/docs-parts/concepts/04-Integrity_lang2.rst deleted file mode 100644 index ae0abdeb..00000000 --- a/docs-parts/concepts/04-Integrity_lang2.rst +++ /dev/null @@ -1,28 +0,0 @@ -``+test/EEGRecording.m`` - -.. code-block:: matlab - - %{ - -> test.Session - eeg_recording_id : int - --- - eeg_system : varchar(64) - num_channels : int - %} - - classdef EEGRecording < dj.Manual - end - -``+test/ChannelData.m`` - -.. code-block:: matlab - - %{ - -> test.EEGRecording - channel_idx : int - --- - channel_data : longblob - %} - - classdef ChannelData < dj.Imported - end diff --git a/docs-parts/concepts/04-Integrity_lang3.rst b/docs-parts/concepts/04-Integrity_lang3.rst deleted file mode 100644 index 065616c3..00000000 --- a/docs-parts/concepts/04-Integrity_lang3.rst +++ /dev/null @@ -1,37 +0,0 @@ -``+test/Mouse.m`` - -.. code-block:: matlab - - %{ - mouse_name : varchar(64) - --- - mouse_dob : datetime - %} - - classdef Mouse < dj.Manual - end - -``+test/SubjectGroup.m`` - -.. code-block:: matlab - - %{ - group_number : int - --- - group_name : varchar(64) - %} - - classdef SubjectGroup < dj.Manual - end - -``+test/SubjectGroupGroupMember.m`` - -.. code-block:: matlab - - %{ - -> test.SubjectGroup - -> test.Mouse - %} - - classdef SubjectGroupGroupMember < dj.Part - end diff --git a/docs-parts/concepts/04-Integrity_lang4.rst b/docs-parts/concepts/04-Integrity_lang4.rst deleted file mode 100644 index 104f5ec6..00000000 --- a/docs-parts/concepts/04-Integrity_lang4.rst +++ /dev/null @@ -1,34 +0,0 @@ -``+test/RecordingModality.m`` - -.. code-block:: matlab - - %{ - modality : varchar(64) - %} - - classdef RecordingModality < dj.Lookup - end - -``+test/MultimodalSession.m`` - -.. code-block:: matlab - - %{ - -> test.Session - modes : int - %} - - classdef MultimodalSession < dj.Manual - end - -``+test/MultimodalSessionSessionMode.m`` - -.. code-block:: matlab - - %{ - -> test.MultimodalSession - -> test.RecordingModality - %} - - classdef MultimodalSessionSessionMode < dj.Part - end diff --git a/docs-parts/definition/01-Creating-Schemas_lang1.rst b/docs-parts/definition/01-Creating-Schemas_lang1.rst deleted file mode 100644 index 2a8d91d2..00000000 --- a/docs-parts/definition/01-Creating-Schemas_lang1.rst +++ /dev/null @@ -1,60 +0,0 @@ - -A schema can be created either automatically using the ``dj.createSchema`` script or manually. -While ``dj.createSchema`` simplifies the process, the manual approach yields a better understanding of what actually takes place, so both approaches are listed below. - -Manual -^^^^^^^^^^^^ -**Step 1.** Create the database schema - -Use the following command to create a new schema on the database server: - -.. code-block:: matlab - - query(dj.conn, 'CREATE SCHEMA `alice_experiment`') - -Note that you must have create privileges for the schema name pattern (as described in :ref:`hosting`). -It is a common practice to grant all privileges to users for schemas that begin with the username, in addition to some shared schemas. -Thus the user ``alice`` would be able to perform any work in any schema that begins with ``alice_``. - -**Step 2.** Create the MATLAB package - -DataJoint organizes schemas as MATLAB **packages**. -If you are not familiar with packages, please review: - -* `How to work with MATLAB packages `_ -* `How to manage MATLAB's search paths `_ - -In your project directory, create the package folder, which must begin with a ``+`` sign. -For example, for the schema called ``experiment``, you would create the folder ``+experiment``. -Make sure that your project directory (the parent directory of your package folder) is added to the MATLAB search path. - -**Step 3.** Associate the package with the database schema - -This step tells DataJoint that all classes in the package folder ``+experiment`` will work with tables in the database schema ``alice_experiment``. -Each package corresponds to exactly one schema. -In some special cases, multiple packages may all relate to a single database schema, but in most cases there will be a one-to-one relationship between packages and schemas. - -In the ``+experiment`` folder, create the file ``getSchema.m`` with the following contents: - -.. code-block:: matlab - - function obj = getSchema - persistent OBJ - if isempty(OBJ) - OBJ = dj.Schema(dj.conn, 'experiment', 'alice_experiment'); - end - obj = OBJ; - end - -This function returns a persistent object of type ``dj.Schema``, establishing the link between the ``experiment`` package in MATLAB and the schema ``alice_experiment`` on the database server. - -Automatic -^^^^^^^^^^^^^ - -Alternatively, you can execute - -.. code-block:: matlab - - >> dj.createSchema - -This automated script will walk you through the steps 1--3 above and will create the schema, the package folder, and the ``getSchema`` function in that folder. diff --git a/docs-parts/definition/02-Creating-Tables_lang1.rst b/docs-parts/definition/02-Creating-Tables_lang1.rst deleted file mode 100644 index 6b46f467..00000000 --- a/docs-parts/definition/02-Creating-Tables_lang1.rst +++ /dev/null @@ -1,42 +0,0 @@ - -DataJoint provides the interactive script ``dj.new`` for creating a new table. -It will prompt to enter the new table's class name in the form ``package.ClassName``. -This will create the file ``+package/ClassName.m``. - -For example, define the table ``experiment.Person`` - -.. code-block:: matlab - - >> dj.new - Enter .: experiment.Person - - Choose table tier: - L=lookup - M=manual - I=imported - C=computed - P=part - (L/M/I/C/P) > M - -This will create the file ``+experiment/Person.m`` with the following contents: - -.. code-block:: matlab - - %{ - # my newest table - # add primary key here - ----- - # add additional attributes - %} - - classdef Person < dj.Manual - end - -While ``dj.new`` adds a little bit of convenience, some users may create the classes from scratch manually. - -Each newly created class must inherit from the DataJoint class corresponding to the correct :ref:`data tier `: ``dj.Lookup``, ``dj.Manual``, ``dj.Imported`` or ``dj.Computed``. - -The most important part of the table definition is the comment preceding the ``classdef``. -DataJoint will parse this comment to define the table. - -The class will become usable after you edit this comment as described in :ref:`definitions`. diff --git a/docs-parts/definition/03-Table-Definition_lang1.rst b/docs-parts/definition/03-Table-Definition_lang1.rst deleted file mode 100644 index 8016a84f..00000000 --- a/docs-parts/definition/03-Table-Definition_lang1.rst +++ /dev/null @@ -1,17 +0,0 @@ - -The table definition is contained in the first block comment in the class definition file. -Note that although it looks like a mere comment, the table definition is parsed by DataJoint. -This solution is thought to be convenient since MATLAB does not provide convenient syntax for multiline strings. - -.. code-block:: matlab - - %{ - # database users - username : varchar(20) # unique user name - --- - first_name : varchar(30) - last_name : varchar(30) - role : enum('admin', 'contributor', 'viewer') - %} - classdef User < dj.Manual - end diff --git a/docs-parts/definition/03-Table-Definition_lang2.rst b/docs-parts/definition/03-Table-Definition_lang2.rst deleted file mode 100644 index 760e1190..00000000 --- a/docs-parts/definition/03-Table-Definition_lang2.rst +++ /dev/null @@ -1,2 +0,0 @@ -Users do not need to do anything special to have the table created in the database. -The table is created upon the first attempt to use the class for manipulating its data (e.g. inserting or fetching entities). diff --git a/docs-parts/definition/03-Table-Definition_lang3.rst b/docs-parts/definition/03-Table-Definition_lang3.rst deleted file mode 100644 index f55743ac..00000000 --- a/docs-parts/definition/03-Table-Definition_lang3.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - s = describe(lab.User) diff --git a/docs-parts/definition/03-Table-Definition_lang4.rst b/docs-parts/definition/03-Table-Definition_lang4.rst deleted file mode 100644 index 7e92829a..00000000 --- a/docs-parts/definition/03-Table-Definition_lang4.rst +++ /dev/null @@ -1,6 +0,0 @@ - -Furthermore, DataJoint provides the ``syncDef`` method to update the ``classdef`` file definition string for the table with the definition in the actual table: - -.. code-block:: matlab - - syncDef(lab.User) % updates the table definition in file +lab/User.m diff --git a/docs-parts/definition/07-Primary-Key_lang1.rst b/docs-parts/definition/07-Primary-Key_lang1.rst deleted file mode 100644 index 9c358779..00000000 --- a/docs-parts/definition/07-Primary-Key_lang1.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. code-block:: matlab - - key.scan_idx = fetch1(Scan & key, 'max(scan_idx)+1 -> next') - diff --git a/docs-parts/definition/10-Dependencies_lang1.rst b/docs-parts/definition/10-Dependencies_lang1.rst deleted file mode 100644 index 16a39c0f..00000000 --- a/docs-parts/definition/10-Dependencies_lang1.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - show(mp.BrainSlice) diff --git a/docs-parts/definition/11-ERD_lang1.rst b/docs-parts/definition/11-ERD_lang1.rst deleted file mode 100644 index 61f7e6e9..00000000 --- a/docs-parts/definition/11-ERD_lang1.rst +++ /dev/null @@ -1,20 +0,0 @@ - -The schema object for a package can be obtained using its ``getSchema`` function. -(See :ref:`schema`.) - -.. code-block:: matlab - - draw(dj.ERD(seq.getSchema)) % draw the ERD - -DataJoint provides shortcuts to plot ERD of a table neighborhood or a schema using the ``erd`` command: - -.. code-block:: matlab - - % plot the ERD of the stimulus schema - erd stimulus - - % plot the neighborhood of the stimulus.Trial table - erd stimulus.Trial - - % plot the stimulus and experiment schemas and the neighborhood of preprocess.Sync - erd stimulus experiment preprocess.Sync diff --git a/docs-parts/definition/11-ERD_lang2.rst b/docs-parts/definition/11-ERD_lang2.rst deleted file mode 100644 index e3174b1a..00000000 --- a/docs-parts/definition/11-ERD_lang2.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - draw(dj.ERD(seq.Genome)) diff --git a/docs-parts/definition/11-ERD_lang3.rst b/docs-parts/definition/11-ERD_lang3.rst deleted file mode 100644 index d286eeca..00000000 --- a/docs-parts/definition/11-ERD_lang3.rst +++ /dev/null @@ -1,5 +0,0 @@ - -.. code-block:: matlab - - % plot the ERD with tables Genome and Species from package +seq. - draw(dj.ERD(seq.Genome) + dj.ERD(seq.Species)) diff --git a/docs-parts/definition/11-ERD_lang4.rst b/docs-parts/definition/11-ERD_lang4.rst deleted file mode 100644 index cadca52d..00000000 --- a/docs-parts/definition/11-ERD_lang4.rst +++ /dev/null @@ -1,15 +0,0 @@ - -.. code-block:: matlab - - % Plot all the tables directly downstream from ``seq.Genome``: - draw(dj.ERD(seq.Genome)+1) - -.. code-block:: matlab - - % Plot all the tables directly upstream from ``seq.Genome``: - draw(dj.ERD(seq.Genome)-1) - -.. code-block:: matlab - - % Plot the local neighborhood of ``seq.Genome`` - draw(dj.ERD(seq.Genome)+1-1+1-1) diff --git a/docs-parts/definition/12-Example_lang1.rst b/docs-parts/definition/12-Example_lang1.rst deleted file mode 100644 index e0873eae..00000000 --- a/docs-parts/definition/12-Example_lang1.rst +++ /dev/null @@ -1,48 +0,0 @@ - -File ``+experiment/Animal.m`` - -.. code-block:: matlab - - %{ - # information about animal - animal_id : int # animal id assigned by the lab - --- - -> experiment.Species - date_of_birth=null : date # YYYY-MM-DD optional - sex='' : enum('M', 'F', '') # leave empty if unspecified - %} - classdef Animal < dj.Manual - end - -File ``+experiment/Session.m`` - -.. code-block:: matlab - - %{ - # Experiment Session - -> experiment.Animal - session : smallint # session number for the animal - --- - session_date : date # YYYY-MM-DD - -> experiment.User - -> experiment.Anesthesia - -> experiment.Rig - %} - classdef Session < dj.Manual - end - -File ``+experiment/Scan.m`` - -.. code-block:: matlab - - %{ - # Two-photon imaging scan - -> experiment.Session - scan : smallint # scan number within the session - --- - -> experiment.Lens - laser_wavelength : decimal(5,1) # um - laser_power : decimal(4,1) # mW - %} - classdef Scan < dj.Manual - end diff --git a/docs-parts/definition/13-Lookup-Tables_lang1.rst b/docs-parts/definition/13-Lookup-Tables_lang1.rst deleted file mode 100644 index a0bf40fb..00000000 --- a/docs-parts/definition/13-Lookup-Tables_lang1.rst +++ /dev/null @@ -1,21 +0,0 @@ - -File ``+lab/User.m`` - -.. code-block:: matlab - - %{ - # users in the lab - username : varchar(20) # user in the lab - --- - first_name : varchar(20) # user first name - last_name : varchar(20) # user last name - %} - classdef User < dj.Lookup - properties - contents = { - 'cajal' 'Santiago' 'Cajal' - 'hubel' 'David' 'Hubel' - 'wiesel' 'Torsten' 'Wiesel' - } - end - end diff --git a/docs-parts/definition/14-Drop_lang1.rst b/docs-parts/definition/14-Drop_lang1.rst deleted file mode 100644 index bad038b9..00000000 --- a/docs-parts/definition/14-Drop_lang1.rst +++ /dev/null @@ -1,5 +0,0 @@ - -.. code-block:: matlab - - % drop the Person table from the lab schema - drop(lab.Person) diff --git a/docs-parts/definition/14-Drop_lang2.rst b/docs-parts/definition/14-Drop_lang2.rst deleted file mode 100644 index 83bf221b..00000000 --- a/docs-parts/definition/14-Drop_lang2.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. note:: - - This rule is currently not enforced in MATLAB, but calling ``drop`` directly on a part table will produce an error in the future. - See `issue #125 `_ on ``datajoint-matlab`` for more information. diff --git a/docs-parts/existing/1-Loading-Classes_lang1.rst b/docs-parts/existing/1-Loading-Classes_lang1.rst deleted file mode 100644 index f20fac2c..00000000 --- a/docs-parts/existing/1-Loading-Classes_lang1.rst +++ /dev/null @@ -1,18 +0,0 @@ - -Creating a virtual class -~~~~~~~~~~~~~~~~~~~~~~~~ - -To facilitate working with existing pipelines, DataJoint MATLAB creates a ``TableAccessor`` property in each schema object. -The ``TableAccessor`` property, a *virtual class generator*, is available as ``schema.v``, and allows listing and querying of the tables defined on the server without needing to create the MATLAB table definitions locally. -For example, creating a scratch ``experiment`` schema package and querying an existing ``my_experiment.Session`` table on the server can be done as follows: - -.. code-block:: matlab - - dj.createSchema('experiment', '/scratch', 'my_experiment') - addpath('/scratch') - experiment_schema = experiment.getSchema(); - experiment_schema.v.Session() & 'session_id=1234'; - -.. note:: - - You can view the available tables in a schema by using tab completion on the ``schema.v`` property. diff --git a/docs-parts/index_lang1.rst b/docs-parts/index_lang1.rst deleted file mode 100644 index b9609d41..00000000 --- a/docs-parts/index_lang1.rst +++ /dev/null @@ -1 +0,0 @@ -This is a detailed manual for active users of DataJoint in MATLAB. diff --git a/docs-parts/intro/Releases_lang1.rst b/docs-parts/intro/Releases_lang1.rst deleted file mode 100644 index da5c02cf..00000000 --- a/docs-parts/intro/Releases_lang1.rst +++ /dev/null @@ -1,73 +0,0 @@ -3.5.0 -- March 21, 2022 --------------------------- -* Bugfix: Cascading delete for renamed foreign keys (#379) PR #386 -* Minor: Add renaming the same attribute multiple times within a single projection PR #386 -* Minor: Add config for reading values with 32-bit dimensions (datajoint/mym#86) PR #395 - -3.4.3 -- May 28, 2021 --------------------------- -* Bugfix: `dj.config` omits default values when loading new config immediately after MATLAB boot (#359) PR #369 -* Bugfix: Regression error when using fetchn with a join (#361) PR #369 -* Bugfix: Cascading delete not functioning properly when using renamed foreign keys (#362) PR #369 -* Update NGINX reverse-proxy image use PR #369 -* Bugfix: Add support to curly brackets in comments (#365) PR #373 - -3.4.2 -- March 16, 2021 --------------------------- -* Bugfix: Fetchn with zero results throws an error (#353) PR #355 -* Bugfix: Syntax error in documentation for next auto_increment value (#352) PR #355 - -3.4.1 -- December 18, 2020 --------------------------- -* Bugfix: Error on accessing unmanaged Imported/Computed tables (#336) PR #338 -* Bugfix: Certain characters in attribute comment not escaped properly (#210, #335) PR #338 -* Bugfix: `dj.config.load(...)` after initial MATLAB boot throws invalid input error. PR #338 - -3.4.0 -- December 11, 2020 --------------------------- -* Minor: Add dj.config to be compatible with dj-python and removed dj.set (#186) #188 -* Minor: Add UUID DataJoint datatype (#180) PR #194 -* Minor: Add file external storage (#143) PR #197 -* Minor: Add S3 external storage (#88) PR #207 -* Minor: Improve dependency version compatibility handling (#228) PR #285 -* Minor: Add unique and nullable options for foreign keys (#110) PR #303 -* Minor: Add non-interactive option for dj.new (#69) #317 -* Minor: Add dj.kill_quick (#251) PR #314 -* Minor: Log connection ID, user in jobs table (#87, #275) PR #314 -* Bugfix: Handle empty password (#250) PR #279, #292 -* Bugfix: Disable GUI password if running headless (#278) PR #280, #292 -* Bugfix: Add order_by option to dj.kill output (#229) PR #248, #292 -* Bugfix: erd function missing from package (#307) PR #310 -* Bugfix: Error on extremely short table names (#311) PR #317 -* Bugfix: Incorrect return when fetchn of an external field (#269) PR #274 -* Bugfix: MATLAB crashes randomly on insert 8-byte string (#255) PR #257 -* Bugfix: Errors thrown when seeing unsupported DataJoint types (#254) PR #265 -* Bugfix: Fix SQL argument growth condition on blobs (#217) PR #220 -* Tests: Add R2016b tests (#233) PR #235 -* Tests: Convert testing framework from TravisCI to GitHub Actions (#320) PR #317 -* Tests: Increase test coverage - -3.3.2 -- October 15, 2020 -------------------------- -* Bugfix: Add blob validation for insert/update regarding sparse matrices which are not yet supported (#238) PR #241 -* Bugfix: Modify update to allow nullable updates for strings/date (#211) PR #213 -* Bugfix: createSchema had some issues with MySQL8 PR #213 -* Update tests -* Docs: Update example related to virtual class (#199) PR #261 -* Docs: Fix typos (#150, #151) PR #263, PR #262 -* Upgrade packaging and installation to utilize MATLAB Toolbox i.e. `DataJoint.mltbx` PR #285 - -3.3.1 -- October 31, 2019 -------------------------- -* Ability to create schema without GUI PR #155 -* Support secure connections with TLS (aka SSL) (#103) PR #157, mym-PR #11, #12, #13 -* Allow GUI-based password entry to avoid cleartext password from being captured in MATLAB log PR #159 -* Add detailed error message if DJ012 Python-native blobs detected (#170) mYm-PR #16 -* Add support for PAM connections via MariaDB's Dialog plugin (#168, #169) mYm-PR #14, #15 -* Minor improvements to reuse of connection if applicable PR #166, #167 -* Bugfixes (#152) - -3.2.2 -- February 5, 2019 -------------------------- - -`Previous release notes TBD` diff --git a/docs-parts/manipulation/1-Insert_lang1.rst b/docs-parts/manipulation/1-Insert_lang1.rst deleted file mode 100644 index 898abb23..00000000 --- a/docs-parts/manipulation/1-Insert_lang1.rst +++ /dev/null @@ -1,26 +0,0 @@ - -The ``insert`` method inserts any number of entities in the form of a structure array with field names corresponding to the attribute names. - -For example - -.. code-block:: matlab - - s.username = 'alice'; - s.first_name = 'Alice'; - s.last_name = 'Cooper'; - insert(lab.Person, s) - -Quick entry of multiple entities takes advantage of MATLAB's cell array notation: - -.. code-block:: matlab - - insert(lab.Person, { - 'alice' 'Alice' 'Cooper' - 'bob' 'Bob' 'Dylan' - 'carol' 'Carol' 'Douglas' - }) - -In this case, the values must match the order of the attributes in the table. - -The optional parameter ``command`` can be either ``'IGNORE'`` or ``'REPLACE'``. -Duplicates, unmatched attributes, or missing required attributes will cause insert errors, unless ``command`` is specified. diff --git a/docs-parts/manipulation/1-Insert_lang2.rst b/docs-parts/manipulation/1-Insert_lang2.rst deleted file mode 100644 index bdc12baf..00000000 --- a/docs-parts/manipulation/1-Insert_lang2.rst +++ /dev/null @@ -1,9 +0,0 @@ - -.. code-block:: matlab - - % Server-side inserts are faster... - phase_two.Protocol.insert(phase_one.Protocol) - - % ...than fetching before inserting - protocols = phase_one.Protocol.fetch(); - phase_two.Protocol.insert(protocols) diff --git a/docs-parts/manipulation/2-Delete_lang1.rst b/docs-parts/manipulation/2-Delete_lang1.rst deleted file mode 100644 index 37f65317..00000000 --- a/docs-parts/manipulation/2-Delete_lang1.rst +++ /dev/null @@ -1 +0,0 @@ -The ``del`` method deletes entities from a table and all dependent entries in dependent tables. diff --git a/docs-parts/manipulation/2-Delete_lang2.rst b/docs-parts/manipulation/2-Delete_lang2.rst deleted file mode 100644 index 6572d9d6..00000000 --- a/docs-parts/manipulation/2-Delete_lang2.rst +++ /dev/null @@ -1,13 +0,0 @@ - -Delete the entire contents of the table ``tuning.VonMises`` and all its dependents: - -.. code-block:: matlab - - % delete all entries from tuning.VonMises - del(tuning.VonMises) - - % delete entries from tuning.VonMises for mouse 1010 - del(tuning.VonMises & 'mouse=1010') - - % delete entries from tuning.VonMises except mouse 1010 - del(tuning.VonMises - 'mouse=1010') diff --git a/docs-parts/manipulation/2-Delete_lang3.rst b/docs-parts/manipulation/2-Delete_lang3.rst deleted file mode 100644 index 9def73be..00000000 --- a/docs-parts/manipulation/2-Delete_lang3.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. note:: - - This rule is currently not enforced in MATLAB, but calling ``del`` directly on a part table will produce an error in the future. - See `issue #193 `_ on ``datajoint-matlab`` for more information. diff --git a/docs-parts/manipulation/3-Transactions_lang1.rst b/docs-parts/manipulation/3-Transactions_lang1.rst deleted file mode 100644 index 655f9741..00000000 --- a/docs-parts/manipulation/3-Transactions_lang1.rst +++ /dev/null @@ -1,33 +0,0 @@ -Transactions are formed using the methods ``startTransaction``, ``cancelTransaction``, and ``commitTransaction`` of a connection object. -A connection object may obtained from any table object. - -For example, the following code inserts matching entries for the master table ``Session`` and its part table ``SessionExperimenter``. - -.. code-block:: matlab - - % get the connection object - session = Session - connection = session.conn - - % insert Session and Session.Experimenter entries in a transaction - connection.startTransaction - try - key.subject_id = animal_id; - key.session_time = session_time; - - session_entry = key; - session_entry.brain_region = region; - insert(Session, session_entry) - - experimenter_entry = key; - experimenter_entry.experimenter = username; - insert(SessionExperimenter, experiment_entry) - connection.commitTransaction - catch - connection.cancelTransaction - end - - -Here, to external observers, both inserts will take effect together only upon exiting from the ``try-catch`` block or will not have any effect at all. -For example, if the second insert fails due to an error, the first insert will be rolled back. - diff --git a/docs-parts/queries/01-Queries_lang1.rst b/docs-parts/queries/01-Queries_lang1.rst deleted file mode 100644 index 3c193a26..00000000 --- a/docs-parts/queries/01-Queries_lang1.rst +++ /dev/null @@ -1,5 +0,0 @@ - -.. code-block:: matlab - - query = experiment.Session; - diff --git a/docs-parts/queries/01-Queries_lang2.rst b/docs-parts/queries/01-Queries_lang2.rst deleted file mode 100644 index 6d247c5b..00000000 --- a/docs-parts/queries/01-Queries_lang2.rst +++ /dev/null @@ -1,5 +0,0 @@ - -.. code-block:: matlab - - query = experiment.Session * experiment.Scan & 'animal_id = 102'; - diff --git a/docs-parts/queries/01-Queries_lang3.rst b/docs-parts/queries/01-Queries_lang3.rst deleted file mode 100644 index 1b81e5bd..00000000 --- a/docs-parts/queries/01-Queries_lang3.rst +++ /dev/null @@ -1,7 +0,0 @@ - -.. code-block:: matlab - - s = query.fetch() - -Here fetching from the ``query`` object produces the struct array ``s`` of the queried data. - diff --git a/docs-parts/queries/01-Queries_lang4.rst b/docs-parts/queries/01-Queries_lang4.rst deleted file mode 100644 index 8f3b7ee9..00000000 --- a/docs-parts/queries/01-Queries_lang4.rst +++ /dev/null @@ -1,10 +0,0 @@ - -The ``exists`` method applied to a query object evaluates to ``true`` if the query returns any entities and to ``false`` if the query result is empty. - -The ``count`` method applied to a query object determines the number of entities returned by the query. - -.. code-block:: matlab - - % number of ephys sessions since the start of 2018. - n = count(ephys.Session & 'session_date >= "2018-01-01"') - diff --git a/docs-parts/queries/02-Example-Schema_lang1.rst b/docs-parts/queries/02-Example-Schema_lang1.rst deleted file mode 100644 index 0945cef0..00000000 --- a/docs-parts/queries/02-Example-Schema_lang1.rst +++ /dev/null @@ -1,138 +0,0 @@ - -.. warning:: - Empty primary keys, such as in the ``CurrentTerm`` table, are not yet supported by DataJoint. - This feature will become available in a future release. - See `Issue #127 `_ for more information. - -File ``+university/Student.m`` - -.. code-block:: matlab - - %{ - student_id : int unsigned # university ID - --- - first_name : varchar(40) - last_name : varchar(40) - sex : enum('F', 'M', 'U') - date_of_birth : date - home_address : varchar(200) # street address - home_city : varchar(30) - home_state : char(2) # two-letter abbreviation - home_zipcode : char(10) - home_phone : varchar(14) - %} - classdef Student < dj.Manual - end - -File ``+university/Department.m`` - -.. code-block:: matlab - - %{ - dept : char(6) # abbreviated department name, e.g. BIOL - --- - dept_name : varchar(200) # full department name - dept_address : varchar(200) # mailing address - dept_phone : varchar(14) - %} - classdef Department < dj.Manual - end - -File ``+university/StudentMajor.m`` - -.. code-block:: matlab - - %{ - -> university.Student - --- - -> university.Department - declare_date : date # when student declared her major - %} - classdef StudentMajor < dj.Manual - end - -File ``+university/Course.m`` - -.. code-block:: matlab - - %{ - -> university.Department - course : int unsigned # course number, e.g. 1010 - --- - course_name : varchar(200) # e.g. "Cell Biology" - credits : decimal(3,1) # number of credits earned by completing the course - %} - classdef Course < dj.Manual - end - -File ``+university/Term.m`` - -.. code-block:: matlab - - %{ - term_year : year - term : enum('Spring', 'Summer', 'Fall') - %} - classdef Term < dj.Manual - end - -File ``+university/Section.m`` - -.. code-block:: matlab - - %{ - -> university.Course - -> university.Term - section : char(1) - --- - room : varchar(12) # building and room code - %} - classdef Section < dj.Manual - end - -File ``+university/CurrentTerm.m`` - -.. code-block:: matlab - - %{ - --- - -> university.Term - %} - classdef CurrentTerm < dj.Manual - end - -File ``+university/Enroll.m`` - -.. code-block:: matlab - - %{ - -> university.Section - -> university.Student - %} - classdef Enroll < dj.Manual - end - -File ``+university/LetterGrade.m`` - -.. code-block:: matlab - - %{ - grade : char(2) - --- - points : decimal(3,2) - %} - classdef LetterGrade < dj.Manual - end - -File ``+university/Grade.m`` - -.. code-block:: matlab - - %{ - -> university.Enroll - --- - -> university.LetterGrade - %} - classdef Grade < dj.Manual - end - diff --git a/docs-parts/queries/03-Fetch_lang1.rst b/docs-parts/queries/03-Fetch_lang1.rst deleted file mode 100644 index c1c0f857..00000000 --- a/docs-parts/queries/03-Fetch_lang1.rst +++ /dev/null @@ -1,130 +0,0 @@ - -DataJoint for MATLAB provides three distinct fetch methods: ``fetch``, ``fetch1``, and ``fetchn``. -The three methods differ by the type and number of their returned variables. - -``query.fetch`` returns the result in the form of an *n* ⨉ 1 `struct array `_ where *n* is the number of records matching the query expression. - -``query.fetch1`` and ``query.fetchn`` split the result into separate output arguments, one for each attribute of the query. - -The types of the variables returned by ``fetch1`` and ``fetchn`` depend on the :ref:`datatypes ` of the attributes. -``query.fetchn`` will enclose any attributes of char and blob types in `cell arrays `_ whereas ``query.fetch1`` will unpack them. - -MATLAB has two alternative forms of invoking a method on an object: using the dot notation or passing the object as the first argument. -The following two notations produce an equivalent result: - -.. code-block:: matlab - - result = query.fetch(query, 'attr1') - result = fetch(query, 'attr1') - -However, the dot syntax only works when the query object is already assigned to a variable. -The second syntax is more commonly used to avoid extra variables. - -For example, the two methods below are equivalent although the second method creates an extra variable. - -.. code-block:: matlab - - # Method 1 - result = fetch(university.Student, '*'); - - # Method 2 - query = university.Student; - result = query.fetch() - - -Fetch the primary key -~~~~~~~~~~~~~~~~~~~~~ - -Without any arguments, the ``fetch`` method retrieves the primary key values of the table in the form of a single column ``struct``. -The attribute names become the fieldnames of the ``struct``. - -.. code-block:: matlab - - keys = query.fetch; - keys = fetch(university.Student & university.StudentMajor); - -Note that MATLAB allows calling functions without the parentheses ``()``. - - -Fetch entire query -~~~~~~~~~~~~~~~~~~ - -With a single-quoted asterisk (``'*'``) as the input argument, the ``fetch`` command retrieves the entire result as a struct array. - -.. code-block:: matlab - - data = query.fetch('*'); - - data = fetch(university.Student & university.StudentMajor, '*'); - -In some cases, the amount of data returned by fetch can be quite large. -When ``query`` is a table object rather than a query expression, ``query.sizeOnDisk()`` reports the estimated size of the entire table. -It can be used to assess whether running ``query.fetch('*')`` would be wise. -Please note that it is only currently possible to query the size of entire tables stored directly in the database . - -As separate variables -~~~~~~~~~~~~~~~~~~~~~ - -The ``fetch1`` and ``fetchn`` methods are used to retrieve each attribute into a separate variable. -DataJoint needs two different methods to tell MATLAB whether the result should be in array or scalar form; for numerical fields it does not matter (because scalars are still matrices in MATLAB) but non-uniform collections of values must be enclosed in cell arrays. - -``query.fetch1`` is used when ``query`` contains exactly one entity, otherwise ``fetch1`` will raise an error. - -``query.fetchn`` returns an arbitrary number of elements with character arrays and blobs returned in the form of cell arrays, even when ``query`` happens to contain a single entity. - -.. code-block:: matlab - - % when tab has exactly one entity: - [name, img] = query.fetch1('name', 'image'); - - % when tab has any number of entities: - [names, imgs] = query.fetchn('name', 'image'); - - -Obtaining the primary key along with individual values -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -It is often convenient to know the primary key values corresponding to attribute values retrieved by ``fetchn``. -This can be done by adding a special input argument indicating the request and another output argument to receive the key values: - -.. code-block:: matlab - - % retrieve names, images, and corresponding primary key values: - [names, imgs, keys] = query.fetchn('name', 'image', 'KEY'); - -The resulting value of ``keys`` will be a column array of type ``struct``. -This mechanism is only implemented for ``fetchn``. - -Rename and calculate -~~~~~~~~~~~~~~~~~~~~ - -In DataJoint for MATLAB, all ``fetch`` methods have all the same capability as the :ref:`proj ` operator. -For example, renaming an attribute can be accomplished using the syntax below. - -.. code-block:: matlab - - [names, BMIs] = query.fetchn('name', 'weight/height/height -> bmi'); - -See :ref:`proj` for an in-depth description of projection. - -Sorting and limiting the results -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To sort the result, add the additional ``ORDER BY`` argument in ``fetch`` and ``fetchn`` methods as the last argument. - -.. code-block:: matlab - - % retrieve field ``course_name`` from courses - % in the biology department, sorted by course number - notes = fetchn(university.Course & 'dept="BIOL"', 'course_name', ... - 'ORDER BY course'); - -The ORDER BY argument is passed directly to SQL and follows the same syntax as the `ORDER BY clause `_ - -Similarly, the LIMIT and OFFSET clauses can be used to limit the result to a subset of entities. -For example, to return the most advanced courses, one could do the following: - -.. code-block:: matlab - - s = fetch(university.Course, '*', 'ORDER BY course DESC LIMIT 5') - -The limit clause is passed directly to SQL and follows the same `rules `_ diff --git a/docs-parts/queries/04-Iteration_lang1.rst b/docs-parts/queries/04-Iteration_lang1.rst deleted file mode 100644 index b87045a9..00000000 --- a/docs-parts/queries/04-Iteration_lang1.rst +++ /dev/null @@ -1,11 +0,0 @@ - -In the simple example below, iteration is used to display the names and values of the primary key attributes of each entity in the simple table or table expression ``tab``. - -.. code-block:: matlab - - for key = tab.fetch()' - disp(key) - end - -Note that the results returned by ``fetch`` must be transposed. -MATLAB iterates across columns, so the single column ``struct`` returned by ``fetch`` must be transposed into a single row. diff --git a/docs-parts/queries/06-Restriction_lang1.rst b/docs-parts/queries/06-Restriction_lang1.rst deleted file mode 100644 index ba86fe7a..00000000 --- a/docs-parts/queries/06-Restriction_lang1.rst +++ /dev/null @@ -1,7 +0,0 @@ - -* another table -* a mapping, or ``struct`` -* an expression in a character string -* a collection of conditions as a ``struct`` or cell array -* a Boolean expression (``true`` or ``false``) -* a query expression diff --git a/docs-parts/queries/06-Restriction_lang2.rst b/docs-parts/queries/06-Restriction_lang2.rst deleted file mode 100644 index 40480683..00000000 --- a/docs-parts/queries/06-Restriction_lang2.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - ephys.Session & struct('session_dat', '2018-01-01') diff --git a/docs-parts/queries/06-Restriction_lang3.rst b/docs-parts/queries/06-Restriction_lang3.rst deleted file mode 100644 index 16b0238d..00000000 --- a/docs-parts/queries/06-Restriction_lang3.rst +++ /dev/null @@ -1,8 +0,0 @@ - -.. code-block:: matlab - - % All the sessions performed by Alice - ephys.Session & 'user = "Alice"' - - % All the experiments at least one minute long - ephys.Experiment & 'duration >= 60' diff --git a/docs-parts/queries/06-Restriction_lang4.rst b/docs-parts/queries/06-Restriction_lang4.rst deleted file mode 100644 index c8612401..00000000 --- a/docs-parts/queries/06-Restriction_lang4.rst +++ /dev/null @@ -1,17 +0,0 @@ - -.. warning:: - This section documents future intended behavior in MATLAB, which is contrary to current behavior. - DataJoint for MATLAB has an open `issue `_ tracking this change. - -A collection can be a cell array or structure array. -Cell arrays can contain collections of arbitrary restriction conditions. -Structure arrays are limited to collections of mappings, each having the same attributes. - -.. code-block:: matlab - - % a cell aray: - cond_cell = {'first_name = "Aaron"', 'last_name = "Aaronson"'} - - % a structure array: - cond_struct = struct('first_name', 'Aaron', 'last_name', 'Paul') - cond_struct(2) = struct('first_name', 'Rosie', 'last_name', 'Aaronson') diff --git a/docs-parts/queries/06-Restriction_lang5.rst b/docs-parts/queries/06-Restriction_lang5.rst deleted file mode 100644 index c715f6e7..00000000 --- a/docs-parts/queries/06-Restriction_lang5.rst +++ /dev/null @@ -1,11 +0,0 @@ - -.. code-block:: matlab - - university.Student() & {'first_name = "Aaron"', 'last_name = "Aaronson"'} - -.. figure:: ../_static/img/matlab_collection.png - :align: center - :alt: restriction by collection - - Restriction by a collection, returning any entities matching any condition in the collection. - diff --git a/docs-parts/queries/06-Restriction_lang6.rst b/docs-parts/queries/06-Restriction_lang6.rst deleted file mode 100644 index cdc65e77..00000000 --- a/docs-parts/queries/06-Restriction_lang6.rst +++ /dev/null @@ -1,4 +0,0 @@ - -``A & true`` and ``A - false`` are equivalent to ``A``. - -``A & false`` and ``A - true`` are empty. diff --git a/docs-parts/queries/06-Restriction_lang7.rst b/docs-parts/queries/06-Restriction_lang7.rst deleted file mode 100644 index e69de29b..00000000 diff --git a/docs-parts/queries/06-Restriction_lang8.rst b/docs-parts/queries/06-Restriction_lang8.rst deleted file mode 100644 index 5c581581..00000000 --- a/docs-parts/queries/06-Restriction_lang8.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. code-block:: matlab - - query = ephys.Session & 'user = "Alice"' - ephys.Experiment & query diff --git a/docs-parts/queries/08-Proj_lang1.rst b/docs-parts/queries/08-Proj_lang1.rst deleted file mode 100644 index fa4005df..00000000 --- a/docs-parts/queries/08-Proj_lang1.rst +++ /dev/null @@ -1,3 +0,0 @@ - -Renaming is done using a string: -``tab('old_attr->new_attr')``. diff --git a/docs-parts/queries/08-Proj_lang2.rst b/docs-parts/queries/08-Proj_lang2.rst deleted file mode 100644 index aaa21642..00000000 --- a/docs-parts/queries/08-Proj_lang2.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - tab.proj('mouse->animal', 'stimulus') diff --git a/docs-parts/queries/08-Proj_lang3.rst b/docs-parts/queries/08-Proj_lang3.rst deleted file mode 100644 index 5f278d1d..00000000 --- a/docs-parts/queries/08-Proj_lang3.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - tab * tab.proj('cell->other') diff --git a/docs-parts/queries/08-Proj_lang4.rst b/docs-parts/queries/08-Proj_lang4.rst deleted file mode 100644 index 3dccf35a..00000000 --- a/docs-parts/queries/08-Proj_lang4.rst +++ /dev/null @@ -1,4 +0,0 @@ - -.. code-block:: matlab - - tab.proj('scan_z-surface_z -> depth') & 'depth > 500' diff --git a/docs-parts/queries/09-Aggr_lang1.rst b/docs-parts/queries/09-Aggr_lang1.rst deleted file mode 100644 index f522b52c..00000000 --- a/docs-parts/queries/09-Aggr_lang1.rst +++ /dev/null @@ -1,7 +0,0 @@ - -.. code-block:: matlab - - % Number of students in each course section - university.Section.aggr(university.Enroll, 'count(*)->n') - % Average grade in each course - university.Course.aggr(university.Grade * university.LetterGrade, 'avg(points)->avg_grade') diff --git a/docs-parts/queries/11-Universal-Sets_lang1.rst b/docs-parts/queries/11-Universal-Sets_lang1.rst deleted file mode 100644 index 1d6e0011..00000000 --- a/docs-parts/queries/11-Universal-Sets_lang1.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. note:: - - ``dj.U`` is not yet implemented in MATLAB. - The feature will be added in an upcoming release: https://github.com/datajoint/datajoint-matlab/issues/144 - -.. code-block:: matlab - - % All home cities of students - dj.U('home_city', 'home_state') & university.Student - - % Total number of students from each city - aggr(dj.U('home_city', 'home_state'), university.Student, 'count(*)->n') - - % Total number of students from each state - aggr(U('home_state'), university.Student, 'count(*)->n') - - % Total number of students in the database - aggr(U(), university.Student, 'count(*)->n') diff --git a/docs-parts/setup/01-Install-and-Connect_lang1.rst b/docs-parts/setup/01-Install-and-Connect_lang1.rst deleted file mode 100644 index a4a6d5ce..00000000 --- a/docs-parts/setup/01-Install-and-Connect_lang1.rst +++ /dev/null @@ -1,29 +0,0 @@ - -1. Download the DataJoint MATLAB Toolbox from the `MATLAB Central FileExchange `_. -2. Open ``DataJoint.mltbx`` and follow installation instructions. -3. After installation, verify from MATLAB that you have the latest version of DataJoint (3.0.0 or above): - :: - - >> dj.version - DataJoint version 3.0.0 -4. At the MATLAB command prompt, assign the environment variables with the database credentials. - For example, if you are connection to the server ``alicelab.datajoint.io`` with username ``alice`` and password ``haha not my real password``, execute the following commands: - :: - - setenv DJ_USER alice - setenv DJ_HOST alicelab.datajoint.io - setenv DJ_PASS 'haha not my real password' - -You will need to execute these commands at the beginning of each DataJoint work session. -To automate this process, you might like to use the `startup.m `_ script. - -However, be careful not to share this file or commit it to a public directory (a common mistake), as it contains a your login credentials in plain text. -If you are not sure, it is better not to set ``DJ_PASS``, in which case DataJoint will prompt to enter the password when connecting to the database. - -To change the database password, use the following command - -:: - - >> dj.setPassword('my#cool!new*psswrd') - -And update your credentials in your startup script for the next session. diff --git a/docs-parts/version_common.json b/docs-parts/version_common.json deleted file mode 100644 index 5bef3068..00000000 --- a/docs-parts/version_common.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "comm_version": "v0.1" -} \ No newline at end of file diff --git a/docs/.docker/Dockerfile b/docs/.docker/Dockerfile new file mode 100644 index 00000000..649d256d --- /dev/null +++ b/docs/.docker/Dockerfile @@ -0,0 +1,15 @@ +FROM datajoint/miniconda3:4.10.3-py3.9-alpine +ARG PACKAGE +WORKDIR /main +COPY --chown=anaconda:anaconda ./docs/.docker/apk_requirements.txt ${APK_REQUIREMENTS} +COPY --chown=anaconda:anaconda ./docs/.docker/pip_requirements.txt ${PIP_REQUIREMENTS} +RUN \ + /entrypoint.sh echo "Dependencies installed" && \ + git config --global user.name "GitHub Action" && \ + git config --global user.email "action@github.com"&& \ + git config --global pull.rebase false && \ + git init +# COPY --chown=anaconda:anaconda ./${PACKAGE} /main/${PACKAGE} +COPY --chown=anaconda:anaconda ./docs/mkdocs.yaml /main/docs/mkdocs.yaml +COPY --chown=anaconda:anaconda ./docs/src /main/docs/src +COPY --chown=anaconda:anaconda ./CHANGELOG.md /main/docs/src/about/changelog.md \ No newline at end of file diff --git a/docs/.docker/apk_requirements.txt b/docs/.docker/apk_requirements.txt new file mode 100644 index 00000000..5664e303 --- /dev/null +++ b/docs/.docker/apk_requirements.txt @@ -0,0 +1 @@ +git diff --git a/docs/.docker/pip_requirements.txt b/docs/.docker/pip_requirements.txt new file mode 100644 index 00000000..3e98f0c5 --- /dev/null +++ b/docs/.docker/pip_requirements.txt @@ -0,0 +1,6 @@ +mkdocs-material +mkdocs-redirects +mike +mdx-truly-sane-lists +mkdocs-literate-nav +mkdocs-exclude-search diff --git a/docs/.markdownlint.yaml b/docs/.markdownlint.yaml new file mode 100644 index 00000000..593bb1d7 --- /dev/null +++ b/docs/.markdownlint.yaml @@ -0,0 +1,20 @@ +# https://github.com/DavidAnson/markdownlint +# https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md +MD009: false # permit trailing spaces +MD013: + line_length: "88" # Line length limits + tables: false # disable for tables + headings: false # disable for headings + code_blocks: false # disable for code blocks +MD030: false # Number of spaces after a list +MD033: # HTML elements allowed + allowed_elements: + - "div" + - "span" + - "a" + - "br" + - "sup" + - "figure" +MD034: false # Bare URLs OK +MD031: false # Spacing w/code blocks. Conflicts with `??? Note` and code tab styling +MD046: false # Spacing w/code blocks. Conflicts with `??? Note` and code tab styling diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..dd9689db --- /dev/null +++ b/docs/README.md @@ -0,0 +1,36 @@ +# Docs Contributions + +Docs contributors should be aware of the following extensions, with the corresponding +settings files, that were used in developing these docs: + +- [MarkdownLinter](https://github.com/DavidAnson/markdownlint): + - `.markdownlint.yaml` establishes settings for various + [linter rules](https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md) + - `.vscode/settings.json` formatting settings + + ```json + { + "[markdown]" : { + "editor.rulers": [88], + "editor.formatOnPaste": true, + "editor.formatOnSave": true, + // https://github.com/stkb/Rewrap/ + // Toggle via command prompt per file + // Default paragraph rewrap key: alt+q or option+q + "rewrap.autoWrap.enabled": true, + "rewrap.wrappingColumn": 88 + }, + // https://github.com/DavidAnson/markdownlint + "editor.codeActionsOnSave": { + "source.fixAll.markdownlint":true + }, + "markdownlint.focusMode": 5, // ignore issues around the cursor + } + ``` + +- [CSpell](https://github.com/streetsidesoftware/vscode-spell-checker): `cspell.json` +has various ignored words. + +- [ReWrap](https://github.com/stkb/Rewrap/): `.vscode/settings.json` allows toggling +automated hard wrapping for files at 88 characters. This can also be keymapped to be +performed on individual paragraphs, see documentation. diff --git a/docs/cspell.json b/docs/cspell.json new file mode 100644 index 00000000..cb29bedd --- /dev/null +++ b/docs/cspell.json @@ -0,0 +1,38 @@ +// cSpell Settings +//https://github.com/streetsidesoftware/vscode-spell-checker +{ + "version": "0.2", // Version of the setting file. Always 0.2 + "language": "en", // language - current active spelling language + "enabledLanguageIds": [ + "markdown" + ], + // flagWords - list of words to be always considered incorrect + // This is useful for offensive words and common spelling errors. + // For example "hte" should be "the" + "flagWords": [], + "allowCompoundWords": true, + "ignorePaths": [ + "src/archive" + ], + "words": [ + "ghtb", + "disp", + "mltbx", + "setenv", + "isempty", + "fetchn", + "struct", + "linenums", + "repr", + "conda", + "numpy", + "sess", + "cond", + "aggr", + "Aggr", + "cajal", + "Cajal", + "hubel", + "Hubel" + ] +} diff --git a/docs/mkdocs.yaml b/docs/mkdocs.yaml new file mode 100644 index 00000000..9bc27006 --- /dev/null +++ b/docs/mkdocs.yaml @@ -0,0 +1,114 @@ +# ---------------------- PROJECT SPECIFIC --------------------------- + +site_name: DataJoint Matlab +repo_url: https://github.com/datajoint/datajoint-matlab +repo_name: datajoint/datajoint-matlab +nav: + - DataJoint Matlab: getting-started/index.md + - Getting Started: getting-started/index.md + - Existing Pipelines: concepts/existing-pipelines.md + - Query Language: + - Common Commands: query-lang/common-commands.md + - Operators: query-lang/operators.md + - Iteration: query-lang/iteration.md + - Reproducibility: + - Table Tiers: reproduce/table-tiers.md + - Make Method: reproduce/make-method.md + # - Tutorials: tutorials.md # Commented out pending viable draft + - Changelog: changelog.md + +# ---------------------------- STANDARD ----------------------------- + +edit_uri: ./edit/master/docs/src +docs_dir: ./src +theme: + font: + text: Roboto Slab + code: Source Code Pro + name: material + custom_dir: src/.overrides + icon: + logo: main/company-logo + favicon: assets/images/company-logo-blue.png + features: + - toc.integrate + - content.code.annotate # Add codeblock annotations + palette: + - media: "(prefers-color-scheme: light)" + scheme: datajoint + toggle: + icon: material/brightness-7 + name: Switch to dark mode + - media: "(prefers-color-scheme: dark)" + scheme: slate + toggle: + icon: material/brightness-4 + name: Switch to light mode +plugins: + - search + - redirects: + redirect_maps: + "index.md": "getting-started/index.md" + - literate-nav: + nav_file: navigation.md + - exclude-search: + exclude: + - "*/navigation.md" + - "*/archive/*md" +markdown_extensions: + - attr_list + - toc: + permalink: true + - pymdownx.emoji: + emoji_index: !!python/name:materialx.emoji.twemoji + emoji_generator: !!python/name:materialx.emoji.to_svg + options: + custom_icons: + - .overrides/.icons + - mdx_truly_sane_lists + - pymdownx.tabbed: + alternate_style: true + - admonition + - pymdownx.details + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format +extra: + generator: false # Disable watermark + analytics: + provider: google + property: !ENV GOOGLE_ANALYTICS_KEY + version: + provider: mike + social: + - icon: main/company-logo + link: https://www.datajoint.com + name: DataJoint + - icon: fontawesome/brands/slack + link: https://datajoint.slack.com + name: Slack + - icon: fontawesome/brands/linkedin + link: https://www.linkedin.com/company/datajoint + name: LinkedIn + - icon: fontawesome/brands/twitter + link: https://twitter.com/datajoint + name: Twitter + - icon: fontawesome/brands/github + link: https://github.com/datajoint + name: GitHub + - icon: fontawesome/brands/docker + link: https://hub.docker.com/u/datajoint + name: DockerHub + - icon: fontawesome/brands/python + link: https://pypi.org/user/datajointbot + name: PyPI + - icon: fontawesome/brands/stack-overflow + link: https://stackoverflow.com/questions/tagged/datajoint + name: StackOverflow + - icon: fontawesome/brands/youtube + link: https://www.youtube.com/channel/UCdeCuFOTCXlVMRzh6Wk-lGg + name: YouTube +extra_css: + - assets/stylesheets/extra.css diff --git a/docs/src/.overrides/.icons/main/company-logo.svg b/docs/src/.overrides/.icons/main/company-logo.svg new file mode 100644 index 00000000..e876313c --- /dev/null +++ b/docs/src/.overrides/.icons/main/company-logo.svg @@ -0,0 +1,11 @@ + + + + + + + + diff --git a/docs/src/.overrides/assets/images/company-logo-blue.png b/docs/src/.overrides/assets/images/company-logo-blue.png new file mode 100644 index 00000000..d15194b8 Binary files /dev/null and b/docs/src/.overrides/assets/images/company-logo-blue.png differ diff --git a/docs/src/.overrides/assets/stylesheets/extra.css b/docs/src/.overrides/assets/stylesheets/extra.css new file mode 100644 index 00000000..46b6aa59 --- /dev/null +++ b/docs/src/.overrides/assets/stylesheets/extra.css @@ -0,0 +1,93 @@ +:root { + --dj-primary: #00a0df; + --dj-secondary: #ff5113; + --dj-background: #808285; + --dj-black: #000000; + --dj-white: #ffffff; +} + +/* footer previous/next navigation */ +.md-footer__inner:not([hidden]) { + display: none +} + +/* footer social icons */ +html a[title="DataJoint"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="Slack"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="LinkedIn"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="Twitter"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="GitHub"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="DockerHub"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="PyPI"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="StackOverflow"].md-social__link svg { + color: var(--dj-primary); +} +html a[title="YouTube"].md-social__link svg { + color: var(--dj-primary); +} + +[data-md-color-scheme="datajoint"] { + /* ribbon */ + /* ribbon + markdown heading expansion */ + --md-primary-fg-color: var(--dj-black); + /* ribbon text */ + --md-primary-bg-color: var(--dj-primary); + + /* navigation */ + /* navigation header + links */ + --md-typeset-a-color: var(--dj-primary); + /* navigation on hover + diagram outline */ + --md-accent-fg-color: var(--dj-secondary); + + /* main */ + /* main header + already viewed*/ + --md-default-fg-color--light: var(--dj-background); + /* primary text */ + --md-typeset-color: var(--dj-black); + /* code comments + diagram text */ + --md-code-fg-color: var(--dj-primary); + + /* footer */ + /* previous/next text */ + /* --md-footer-fg-color: var(--dj-primary); */ +} + +[data-md-color-scheme="slate"] { + /* ribbon */ + /* ribbon + markdown heading expansion */ + --md-primary-fg-color: var(--dj-primary); + /* ribbon text */ + --md-primary-bg-color: var(--dj-white); + + /* navigation */ + /* navigation header + links */ + --md-typeset-a-color: var(--dj-primary); + /* navigation on hover + diagram outline */ + --md-accent-fg-color: var(--dj-secondary); + + /* main */ + /* main header + already viewed*/ + /* --md-default-fg-color--light: var(--dj-background); */ + /* primary text */ + --md-typeset-color: var(--dj-white); + /* code comments + diagram text */ + --md-code-fg-color: var(--dj-primary); + + /* footer */ + /* previous/next text */ + /* --md-footer-fg-color: var(--dj-white); */ +} diff --git a/docs/src/.overrides/partials/nav.html b/docs/src/.overrides/partials/nav.html new file mode 100644 index 00000000..5c090954 --- /dev/null +++ b/docs/src/.overrides/partials/nav.html @@ -0,0 +1,33 @@ +{% set class = "md-nav md-nav--primary" %} +{% if "navigation.tabs" in features %} +{% set class = class ~ " md-nav--lifted" %} +{% endif %} +{% if "toc.integrate" in features %} +{% set class = class ~ " md-nav--integrated" %} +{% endif %} + \ No newline at end of file diff --git a/docs/src/changelog.md b/docs/src/changelog.md new file mode 120000 index 00000000..699cc9e7 --- /dev/null +++ b/docs/src/changelog.md @@ -0,0 +1 @@ +../../CHANGELOG.md \ No newline at end of file diff --git a/docs/src/concepts/existing-pipelines.md b/docs/src/concepts/existing-pipelines.md new file mode 100644 index 00000000..8f98a178 --- /dev/null +++ b/docs/src/concepts/existing-pipelines.md @@ -0,0 +1,29 @@ +# Existing Pipelines + +This section describes how to work with database schemas without access to the original +code that generated the schema. These situations often arise when the database is +created by another user who has not shared the generating code yet or when the database +schema is created from a programming language other than Matlab. + +## Creating a virtual class + +DataJoint MATLAB creates a `TableAccessor` property in each schema object. The +`TableAccessor` property, a *virtual class generator*, is available as `schema.v`, and +allows listing and querying of the tables defined on the server without needing to +create the MATLAB table definitions locally. For example, creating a scratch +`experiment` schema package and querying an existing `my_experiment.Session` table on +the server can be done as follows: + +``` matlab +dj.createSchema('experiment', '/scratch', 'my_experiment') +addpath('/scratch') +experiment_schema = experiment.getSchema(); +experiment_schema.v.Session() & 'session_id=1234'; +``` + +???+ Note + + You can view the available tables in a schema by using tab completion on + the `schema.v` property. + +To visualize an unfamiliar schema, see commands for generating [diagrams](../../getting-started/#diagram). diff --git a/docs/src/getting-started/index.md b/docs/src/getting-started/index.md new file mode 100644 index 00000000..7514fc17 --- /dev/null +++ b/docs/src/getting-started/index.md @@ -0,0 +1,300 @@ +# Getting Started + +## Installation + +First, please install DataJoint via one of the following: + +=== "Matlab ≥ R2016b (recommended)" + + 1. Utilize MATLAB built-in GUI i.e. *Top Ribbon -> Add-Ons -> Get Add-Ons* + 2. Search and Select `DataJoint` + 3. Select *Add from GitHub* + +=== "Matlab < R2016" + + 1. Utilize MATLAB built-in GUI i.e. *Top Ribbon -> Add-Ons -> Get Add-Ons* + 2. Search and Select `DataJoint` + 3. Select *Download from GitHub* + 4. Save `DataJoint.mltbx` locally + 5. Navigate in MATLAB tree browser to saved toolbox file + 6. Right-Click and Select *Install* + 7. Select *Install* + +=== "GHToolbox" + + 1. Install *GHToolbox* using using an appropriate method in https://github.com/datajoint/GHToolbox + 2. run: `ghtb.install('datajoint/datajoint-matlab')` + +=== "From source" + + 1. Download `DataJoint.mltbx` locally + 2. Navigate in MATLAB tree browser to saved toolbox file + 3. Right-Click and Select *Install* + 4. Select *Install* + +After installing, that you have the latest version from Matlab: + +```matlab +>> dj.version +DataJoint version 3.5.0 +``` + +## Connection + +At the MATLAB command prompt, assign the environment variables with +the database credentials. For example, if you are connection to the +server `tutorial-db.datajoint.io` with username `alice` and password +`fake-password`, execute the following commands: + +```matlab +setenv DJ_USER alice +setenv DJ_HOST tutorial-db.datajoint.io +setenv DJ_PASS 'fake-password' +``` + +!!! note + + Although you may connect to any MySQL server of your choice, the DataJoint company + offers an online tutorial environment at `tutorial-db.datajoint.io`. Simply sign up + for a free [DataJoint account](https://accounts.datajoint.io). You will be granted + privileges to create schemas that are prefixed as `{user}_`. + +You will need to execute these commands at the beginning of each DataJoint work session. +To automate this process, you might add these items to the Matlab +[startup.m](https://www.mathworks.com/help/matlab/ref/startup.html) script. + +However, be careful not to share this file or commit it to a public directory (a common +mistake), as it contains a your login credentials in plain text. If you are not sure, +it is better not to set `DJ_PASS`, in which case DataJoint will prompt to enter the +password when connecting to the database. + +To change the database password, use the following command + +```matlab +>> dj.setPassword('my#cool!new*password') +``` + +And update your credentials in your startup script for the next session. + +For more information on various settings, access help via `help('dj.config')` or review +it online +[here](https://github.com/datajoint/datajoint-matlab/blob/c2bd6b3e195dfeef773d4e12bad5573c461193b0/%2Bdj/config.m#L2-L27). + +## Creating Schemas + +A schema can be created either automatically using the `dj.createSchema` +script or manually. While `dj.createSchema` simplifies the process, we'll also highlight +the manual approach to demonstrate each step. + +### Manual + +We can create the database schema using the following command: + +``` matlab +query(dj.conn, 'CREATE SCHEMA `{user}_my_schema`') +``` + +??? Note "Server privileges" + + You must have create privileges for the schema name pattern. It is a common practice + to grant all privileges to users for schemas that begin with the username, in + addition to some shared schemas. Thus the user `alice` would be able to perform any + work in any schema that begins with `alice_`. + +Next, we can create the MATLAB package. + +DataJoint organizes schemas as MATLAB **packages**. If you are not +familiar with packages, please review: + +- [How to work with MATLAB packages](https://www.mathworks.com/help/matlab/matlab_oop/scoping-classes-with-packages.html) +- [How to manage MATLAB's search paths](https://www.mathworks.com/help/matlab/search-path.html) + +In your project directory, create the package folder, which must begin with a `+` sign. +For example, for the schema called `my_schema`, you would create the folder +`+my_schema`. Make sure that your project directory (the parent directory of your +package folder) is added to the MATLAB search path. + +Finally, we'll associate the package with the database schema. + +This step tells DataJoint that all classes in the package folder `+my_schema` will work +with tables in the database schema `{user}_my_schema`. Each package corresponds to +exactly one schema. In some special cases, multiple packages may all relate to a single +database schema, but in most cases there will be a one-to-one relationship between +packages and schemas. + +In the `+my_schema` folder, create the file `getSchema.m` with the +following contents: + +``` matlab +function obj = getSchema +persistent OBJ +if isempty(OBJ) + OBJ = dj.Schema(dj.conn, 'experiment', 'alice_experiment'); +end +obj = OBJ; +end +``` + +This function returns a persistent object of type `dj.Schema`, +establishing the link between the `my_schema` package in MATLAB and the +schema `{user}_my_schema` on the database server. + +## Automatic + +Alternatively, we can execute + +``` matlab +>> dj.createSchema +``` + +This automated script will walk you through the steps above and will create the schema, +the package folder, and the `getSchema` function in that folder. + +## Defining Tables + +DataJoint provides the interactive script `dj.new` for creating a new table. It will +prompt to enter the new table's class name in the form `package.ClassName`. This will +create the file `+package/ClassName.m`. + +For example, define the table `my_schema.Rectangle` + +``` matlab + >> dj.new + Enter .: my_schema.Rectangle + + Choose table tier: + L=lookup + M=manual + I=imported + C=computed + P=part + (L/M/I/C/P) > M +``` + +This will create the file `+my_schema.Rectangle.m` with the following +contents: + +``` matlab +%{ +# my newest table +# add primary key here +----- +# add additional attributes +%} +classdef Rectangle < dj.Manual +end +``` + +While `dj.new` adds a little bit of convenience, we can also create the classes from +scratch manually. Each newly created class must inherit from the DataJoint class +corresponding to the correct [data tier](../../reproduce/table-tiers): `dj.Lookup`, +`dj.Manual`, `dj.Imported` or `dj.Computed`. + +The most important part of the table definition is the comment preceding the `classdef`. +DataJoint will parse this comment to define the table. The class will become usable +after you edit this comment. For example: + +File `+my_schema/Rectangle.m` + +```matlab +%{ + shape_id: int + --- + shape_height: float + shape_width: float +%} +classdef Rectangle < dj.Manual +end +``` + +File `+my_schema/Area.m` + +```matlab +%{ + -> my_schema.Rectangle + --- + shape_area: float +%} +classdef Area < dj.Computed +end +``` + +The table definition is contained in the first block comment in the class definition +file. Note that although it looks like a mere comment, the table definition is parsed +by DataJoint. + +Users do not need to do anything special to have the table created in the database. The +table is created upon the first attempt to use the class for manipulating its data +(e.g. inserting or fetching entities). + +Furthermore, DataJoint provides the `syncDef` method to update the `classdef` file +definition string for the table with the definition in the actual table: + +``` matlab + syncDef(my_schema.Area) % updates the table definition in file +my_schema/Area.m +``` + +## Diagram + +### Display + +The diagram displays the relationship of the data model in the data pipeline. + +This can be done for an entire schema, or multiple schema: + +``` matlab +draw(dj.ERD(my_schema.getSchema)) +% OR +erd my_schema +erd my_schema my_other_schema +``` + +Or for individual or sets of tables: +```python +erd my_schema.Rectangle +draw(dj.ERD(my_schema.Rectangle) + dj.ERD(my_schema.Area)) +``` + +![Shapes Pipeline](../images/shapes_pipeline.svg) + +### Customize + +Adding or substracting a number to a diagram object adds nodes downstream or upstream, +respectively, in the pipeline. + +``` python +draw(dj.ERD(my_schema.Rectangle)+1) # (1) +``` + +1. Plot all the tables directly downstream from `my_schema.Rectangle` + +```python +draw(dj.ERD(my_schema)-1+1) # (1) +``` + +1. Plot all tables directly downstream of those directly upstream of this schema. + +## Add data + +Let's add data for a rectangle: + +```matlab +insert(my_schema.Rectangle, {1, 2, 4}) +insert(my_schema.Rectangle, [{2, 2, 3},{3, 4, 2}]) +``` + +## Run computation + +Let's start the computations on our entity: `Area`. + +```python +populate(my_schema.Rectangle) +``` + +## Query + +Let's inspect the results. + +```python +Area & 'shape_area >= 8' +``` diff --git a/docs/src/images/shapes_pipeline.svg b/docs/src/images/shapes_pipeline.svg new file mode 100644 index 00000000..203b7b47 --- /dev/null +++ b/docs/src/images/shapes_pipeline.svg @@ -0,0 +1,36 @@ + + +%3 + + + +Area + + +Area + + + + + +Rectangle + + +Rectangle + + + + + +Rectangle->Area + + + + \ No newline at end of file diff --git a/docs/src/query-lang/common-commands.md b/docs/src/query-lang/common-commands.md new file mode 100644 index 00000000..b7916cdd --- /dev/null +++ b/docs/src/query-lang/common-commands.md @@ -0,0 +1,154 @@ +# Common Commands + + + +## Make + +See the article on [`make` methods](../../reproduce/make-method/) + +## Fetch + +DataJoint for MATLAB provides three distinct fetch methods: `fetch`, `fetch1`, and +`fetchn`. The three methods differ by the type and number of their returned variables. + +- `query.fetch` returns the result in the form of an *n* ⨉ 1 [struct array](https://www.mathworks.com/help/matlab/ref/struct.html) + where *n* is the number of records matching the query expression. + +- `query.fetch1` and `query.fetchn` split the result into separate output arguments, one + for each attribute of the query. + +The types of the variables returned by `fetch1` and `fetchn` depend on the datatypes of +the attributes. `query.fetchn` will enclose any attributes of char and blob types in +[cell arrays](https://www.mathworks.com/help/matlab/cell-arrays.html) whereas +`query.fetch1` will unpack them. + +MATLAB has two alternative forms of invoking a method on an object: using the dot +notation or passing the object as the first argument. The following two notations +produce an equivalent result: + +``` matlab +result = query.fetch(query, 'attr1') +result = fetch(query, 'attr1') +``` + +However, the dot syntax only works when the query object is already assigned to a +variable. The second syntax is more commonly used to avoid extra variables. + +For example, the two methods below are equivalent although the second +method creates an extra variable. + +``` matlab +result = fetch(my_schema.Rectangle, '*'); + +query = my_schema.Rectangle; +result = query.fetch() +``` + +### The primary key + +Without any arguments, the `fetch` method retrieves the primary key values of the table +in the form of a single column `struct`. The attribute names become the fieldnames of +the `struct`. + +``` matlab +keys = query.fetch; +keys = fetch(my_schema.Rectangle & my_schema.Area); +``` + +Note that MATLAB allows calling functions without the parentheses `()`. + +### An entire query + +With a single-quoted asterisk (`'*'`) as the input argument, the `fetch` command +retrieves the entire result as a struct array. + +``` matlab +data = query.fetch('*'); +data = fetch(my_schema.Rectangle & my_schema.Area, '*'); +``` + +??? Note "For very large tables..." + + In some cases, the amount of data returned by fetch can be quite large. When `query` + is a table object rather than a query expression, `query.sizeOnDisk()` reports the + estimated size of the entire table. It can be used to assess whether running + `query.fetch('*')` would be wise. Please note that it is only currently possible to + query the size of entire tables stored directly in the database . + +### Separate variables + +The `fetch1` and `fetchn` methods are used to retrieve each attribute into a separate +variable. DataJoint needs two different methods to tell MATLAB whether the result +should be in array or scalar form; for numerical fields it does not matter +(because scalars are still matrices in MATLAB) but non-uniform collections of values +must be enclosed in cell arrays. + +`query.fetch1` is used when `query` contains exactly one entity, otherwise `fetch1` will +raise an error. + +`query.fetchn` returns an arbitrary number of elements with character arrays and blobs +returned in the form of cell arrays, even when `query` happens to contain a single +entity. + +``` matlab +[shape_id, height] = query.fetch1('shape_id', 'shape_height'); % (1) +[shape_ids, heights] = query.fetchn('shape_id', 'shape_height'); % (2) +``` + +1. When the table has exactly one entity. +2. When the table has any number of entities: + +### The primary key and individual values + +It is often convenient to know the primary key values corresponding to attribute values +retrieved by `fetchn`. This can be done by adding a special input argument indicating +the request and another output argument to receive the key values: + +``` matlab +[shape_ids, heights, keys] = query.fetchn('shape_id', 'shape_height', 'KEY'); +``` + +The resulting value of `keys` will be a column array of type `struct`. +This mechanism is only implemented for `fetchn`. + +### Rename and calculate + +In DataJoint for MATLAB, all `fetch` methods have all the same capability as the +[`proj` operator](../operators#proj). For example, renaming an attribute can be +accomplished using the syntax below. + +``` matlab +[shape_ids, perimeter] = query.fetchn('shape_id', ... + '2*(shape_height+shape_width) -> perimeter'); +``` + +See [`proj`](../operators#proj) for an in-depth description of projection. + +### Sorting results + +To sort the result, add the additional `ORDER BY` argument in `fetch` and `fetchn` +methods as the last argument. + +The following command retrieves the field `shape_id` from rectangles with height greater +than 2, sorted by width. + +``` matlab +notes = fetchn(my_schema.Rectangle & 'shape_height>2"', 'shape_id' ... + 'ORDER BY shape_width'); +``` + +The ORDER BY argument is passed directly to SQL and follows the same syntax as the +[ORDER BY clause](https://dev.mysql.com/doc/refman/5.7/en/order-by-optimization.html) + +Similarly, the LIMIT and OFFSET clauses can be used to limit the result to a subset of +entities. For example, to return the five rectangles with largest area, one could do the +following: + +``` matlab +s = fetch(my_schema.Area, '*', 'ORDER BY shape_area DESC LIMIT 5') +``` + +The limit clause is passed directly to SQL and follows the same +[rules](https://dev.mysql.com/doc/refman/5.7/en/select.html) + + diff --git a/docs/src/query-lang/iteration.md b/docs/src/query-lang/iteration.md new file mode 100644 index 00000000..ef1c602a --- /dev/null +++ b/docs/src/query-lang/iteration.md @@ -0,0 +1,18 @@ +# Iteration + +The DataJoint model primarily handles data as sets, in the form of tables. However, it +can sometimes be useful to access or to perform actions such as visualization upon +individual entities sequentially. In DataJoint this is accomplished through iteration. + +In the simple example below, iteration is used to display the names and values of the +primary key attributes of each entity in the simple table or table expression `tab`. + +``` matlab +for key = tab.fetch()' + disp(key) +end +``` + +Note that the results returned by `fetch` must be transposed. MATLAB iterates across +columns, so the single column `struct` returned by `fetch` must be transposed into a +single row. diff --git a/docs/src/query-lang/operators.md b/docs/src/query-lang/operators.md new file mode 100644 index 00000000..fd785e74 --- /dev/null +++ b/docs/src/query-lang/operators.md @@ -0,0 +1,147 @@ +# Operators + +The examples below will use the table definitions in [table tiers](../../reproduce/table-tiers). + + + +## Restriction + +`&` and `-` operators permit restriction. + +### By a mapping + +For a [Session table](../../reproduce/table-tiers#manual-tables), that has the attribute +`session_date`, we can restrict to sessions from January 1st, 2022: + +```matlab +Session & struct('session_date', '2018-01-01') +``` + +If there were any typos (e.g., using `sess_date` instead of `session_date`), our query +will return all of the entities of `Session`. + +### By a string + +Conditions may include arithmetic operations, functions, range tests, etc. Restriction +of table `A` by a string containing an attribute not found in table `A` produces an +error. + +```matlab +Session & 'user = "Alice"' % (1) +Session & 'session_date >= "2022-01-01"' % (2) +``` + +1. All the sessions performed by Alice +2. All of the sessions on or after January 1st, 2022 + +### By a collection + +When `cond` is a collection of conditions, the conditions are applied by logical +disjunction (logical OR). Restricting a table by a collection will return all entities +that meet *any* of the conditions in the collection. + +For example, if we restrict the `Session` table by a collection containing two +conditions, one for user and one for date, the query will return any sessions with a +matching user *or* date. + +```matlab +cond_cell = {'user = "Alice"', 'session_date = "2022-01-01"'} % (1) +cond_struct = struct('user', 'Alice', 'session_date', '2022-01-01') % (2) +cond_struct(2) = struct('user', 'Jerry', 'session_date', '2022-01-01') + +Session() & cond_struct % (3) +``` + +1. A cell array +2. A structure array +3. This command will show all the sessions that either Alice or Jerry conducted on the + given day. + +### By a query + +Restriction by a query object is a generalization of restriction by a table. The example +below creates a query object corresponding to all the users named Alice. The `Session` +table is then restricted by the query object, returning all the sessions performed by +Alice. The `Experiment` table is then restricted by the query object, returning all the +experiments that are part of sessions performed by Alice. + +``` matlab +query = Session & 'user = "Alice"' +Experiment & query +``` + +## Proj + +Renaming an attribute in python can be done via keyword arguments: + +```matlab +table('old_attr->new_attr') +``` + +This can be done in the context of a table definition: + +``` matlab +%{ + # Experiment Session + -> experiment.Animal + session : smallint # session number for the animal + --- + session_date : date # YYYY-MM-DD + session_start_time : float # seconds relative to session_datetime + session_end_time : float # seconds relative to session_datetime + -> User.proj(experimenter='username') + -> User.proj(supervisor='username') +%} +classdef Session < dj.Manual +end +``` + +Or as part of a query + +```matlab +Session * Session.proj('session->other') +``` + +Projection can also be used to to compute new attributes from existing ones. + +```matlab +Session.proj('session_end_time-session_start_time -> duration') & 'duration > 10' +``` + +## Aggr + +For more complicated calculations, we can use aggregation. + +```matlab +Subject.aggr(Session,'count(*)->n') % (1) +Subject.aggr(Session,'avg(session_start_time)->average_start') % (2) +``` + +1. Number of sessions per subject. +2. Average `session_start_time` for each subject + + + +## Universal set + +!!! Warning + + `dj.U` is not yet implemented in MATLAB. The feature will be added in an + upcoming release. You can track progress with + [this GitHub issue](https://github.com/datajoint/datajoint-matlab/issues/144). + +Universal sets offer the complete list of combinations of attributes. + +```matlab +dj.U('laser_wavelength', 'laser_power') & Scan % (1) +dj.U('laser_wavelength', 'laser_power').aggr(Scan, 'count(*)->n') % (2) +dj.U().aggr(Session, 'max(session)->n') % (3) +``` + +1. All combinations of wavelength and power. +2. Total number of scans for each combination. +3. Largest session number. + +`dj.U()`, as shown in the last example above, is often useful for integer IDs. +For an example of this process, see the source code for +[Element Array Electrophysiology's `insert_new_params`](https://datajoint.com/docs/elements/element-array-ephys/latest/api/element_array_ephys/ephys_acute/#element_array_ephys.ephys_acute.ClusteringParamSet.insert_new_params). diff --git a/docs/src/reproduce/make-method.md b/docs/src/reproduce/make-method.md new file mode 100644 index 00000000..57056eeb --- /dev/null +++ b/docs/src/reproduce/make-method.md @@ -0,0 +1,78 @@ +# Make Method + +Consider the following table definition from the article on +[table tiers](../table-tiers): + +```matlab +%{ Filtered image +-> test.Image +--- +filtered_image : longblob +%} + +classdef FilteredImage < dj.Computed + methods(Access=protected) + function makeTuples(self, key) + img = fetch1(test.Image & key, 'image'); + key.filtered_image = my_filter(img); + self.insert(key) + end + end +end +``` + +The `FilteredImage` table can be populated as + +``` matlab +populate(test.FilteredImage) +``` + +Note that it is not necessary to specify which data needs to be computed. DataJoint will +call `make`, one-by-one, for every key in `Image` for which `FilteredImage` has not yet +been computed. + +The `make` method receives one argument: the dict `key` containing the primary key value +of an element of `key source` to be worked on. + +## Optional Arguments + +Behavior of the `populate` method depends on the number of output arguments requested in +the function call. When no output arguments are requested, errors will halt population. +With two output arguments(`failedKeys` and `errors`), `populate` will catch any +encountered errors and return them along with the offending keys. + +## Progress + +The function `parpopulate` works identically to `populate` except that it uses a job +reservation mechanism to allow multiple processes to populate the same table in +parallel without collision. When running `parpopulate` for the first time, DataJoint +will create a job reservation table and its class `.Jobs` with the following +declaration: + +``` matlab +{% + # the job reservation table + table_name : varchar(255) # className of the table + key_hash : char(32) # key hash + --- + status : enum('reserved','error','ignore')# if tuple is missing, the job is available + key=null : blob # structure containing the key + error_message="" : varchar(1023) # error message returned if failed + error_stack=null : blob # error stack if failed + host="" : varchar(255) # system hostname + pid=0 : int unsigned # system process id + timestamp=CURRENT_TIMESTAMP : timestamp # automatic timestamp +%} +``` + +A job is considered to be available when `.Jobs` contains no matching entry. + +For each `make` call, `parpopulate` sets the job status to `reserved`. When the job is +completed, the record is removed. If the job results in error, the job record is left +in place with the status set to `error` and the error message and error stacks saved. +Consequently, jobs that ended in error during the last execution will not be attempted +again until you delete the corresponding entities from `.Jobs`. + +The primary key of the jobs table comprises the name of the class and a 32-character +hash of the job's primary key. However, the key is saved in a separate field for error +debugging purposes. diff --git a/docs/src/reproduce/table-tiers.md b/docs/src/reproduce/table-tiers.md new file mode 100644 index 00000000..aab832e3 --- /dev/null +++ b/docs/src/reproduce/table-tiers.md @@ -0,0 +1,160 @@ +# Table Tiers + +To define a DataJoint table in Matlab: + +1. Define the table via multi-line comment. +2. Define a class inheriting from the appropriate DataJoint class: + `dj.Lookup`, `dj.Manual`, `dj.Imported` or `dj.Computed`. + +## Manual Tables + +The following code defines two manual tables, `Animal` and `Session`: + +File `+experiment/Animal.m` + +``` matlab +%{ + # information about animal + animal_id : int # animal id assigned by the lab + --- + -> experiment.Species + date_of_birth=null : date # YYYY-MM-DD optional + sex='' : enum('M', 'F', '') # leave empty if unspecified +%} +classdef Animal < dj.Manual +end +``` + +File `+experiment/Session.m` + +``` matlab +%{ + # Experiment Session + -> experiment.Animal + session : smallint # session number for the animal + --- + session_date : date # YYYY-MM-DD + session_start_time : float # seconds relative to session_datetime + session_end_time : float # seconds relative to session_datetime + -> [nullable] experiment.User +%} +classdef Session < dj.Manual +end +``` + +Note that the notation to permit null entries differs for attributes versus foreign +key references. + +## Lookup Tables + +Lookup tables are commonly populated from their `contents` property. + +The table below is declared as a lookup table with its contents property +provided to generate entities. + +File `+lab/User.m` + +```matlab +%{ + # users in the lab + username : varchar(20) # user in the lab + --- + first_name : varchar(20) # user first name + last_name : varchar(20) # user last name +%} +classdef User < dj.Lookup + properties + contents = { + 'cajal' 'Santiago' 'Cajal' + 'hubel' 'David' 'Hubel' + 'wiesel' 'Torsten' 'Wiesel' + } + end +end +``` + +## Imported and Computed Tables + +Imported and Computed tables provide [`make` methods](./make-method) to determine how +they are populated, either from files or other tables. + +Imagine that there is a table `test.Image` that contains 2D grayscale images in its +`image` attribute. We can define the Computed table, `test.FilteredImage` that filters +the image in some way and saves the result in its `filtered_image` attribute. + +```matlab +%{ Filtered image +-> test.Image +--- +filtered_image : longblob +%} + +classdef FilteredImage < dj.Computed + methods(Access=protected) + function makeTuples(self, key) + img = fetch1(test.Image & key, 'image'); + key.filtered_image = my_filter(img); + self.insert(key) + end + end +end +``` + +??? Note "`makeTuples` vs. `make`" + + Currently matlab uses `makeTuples` rather than `make`. This will be + fixed in an upcoming release. You can monitor the discussion + [here](https://github.com/datajoint/datajoint-matlab/issues/141) + +## Part Tables + +The following code defines a Imported table with an associated part table. In MATLAB, +the master and part tables are declared in a separate `classdef` file. The name of the +part table must begin with the name of the master table. The part table must declare the +property `master` containing an object of the master. + +`+image/Scan.m` + +``` matlab +%{ + # Two-photon imaging scan + -> Session + scan : smallint # scan number within the session + --- + -> Lens + laser_wavelength : decimal(5,1) # um + laser_power : decimal(4,1) # mW +%} +classdef Scan < dj.Computed + methods(Access=protected) + function make(self, key) + self.insert(key) + make(image.ScanField, key) + end + end +end +``` + +`+image/ScanField.m` + +``` matlab +%{ +# Region of interest resulting from segmentation +-> image.Scan +mask : smallint +--- +ROI : longblob # Region of interest +%} + +classdef ScanField < dj.Part + properties(SetAccess=protected) + master = image.Scan + end + methods + function make(self, key) + ... + self.insert(entity) + end + end +end +``` diff --git a/docs/src/tutorials.md b/docs/src/tutorials.md new file mode 100644 index 00000000..8b29d1af --- /dev/null +++ b/docs/src/tutorials.md @@ -0,0 +1,3 @@ +# Tutorials + +Coming soon!