Update inventory; pass many tests

docs/events.rst

@@ -2,9 +2,9 @@ Events
 ======
 
 ..  toctree::
-:maxdepth: 4
+    :maxdepth: 4
 
-        event-diagram
+    event-diagram
 
 
 Rate
@@ -12,8 +12,8 @@ Rate
 Devicehub generates an rating for a device taking into consideration the
 visual, functional, and performance.
 
-.. todo:: add performance as a result of component fusion + general tests in https://
+.. todo:: add performance as a result of component fusion + general tests in `here <https://
-github.com/eReuse/Rdevicescore/blob/master/img/input_process_output.png
+   github.com/eReuse/Rdevicescore/blob/master/img/input_process_output.png>`_.
 
 A Workflow is as follows:
 
@@ -172,12 +172,10 @@ There are four events for getting rid of devices:
       been recovered under a new product.
 
 .. note:: For usability purposes, users might not directly perform
-``Dispose``, but this could automatically be done when
+   ``Dispose``, but this could automatically be done when
-          performing ``ToDispose`` + ``Receive`` to a
+   performing ``ToDispose`` + ``Receive`` to a ``RecyclingCenter``.
-          ``RecyclingCenter``.
 
 .. todo:: Ensure that ``Dispose`` is a ``Trade`` event. An Org could
-``Sell`` or ``Donate`` a device with the objective of
+    ``Sell`` or ``Donate`` a device with the objective of disposing them.
-          disposing them. Is ``Dispose`` ok, or do we want to keep
+    Is ``Dispose`` ok, or do we want to keep that extra ``Sell`` or
-          that extra ``Sell`` or ``Donate`` event? Could dispose
+    ``Donate`` event? Could dispose be a synonym of any of those?
-          be a synonym of any of those?

docs/getting.rst

@@ -1,22 +0,0 @@
-Getting
-=======
-
-Devicehub uses the same path to get devices and lots.
-
-To get the lot information ::
-
-    GET /inventory/24
-
-You can specifically filter devices::
-
-    GET /inventory?devices?
-    GET /inventory/24?type=24&type=44&status={"name": "Reserved", "updated": "2018-01-01"}
-    GET /inventory/25?price=24&price=21
-
-GET /devices/4?
-
-Returns devices that matches the filters and the lots that contain them.
-If the filters are applied to the lots, it returns the matched lots
-and the devices that contain them.
-You can join filters.
-

docs/index.rst

@@ -14,6 +14,7 @@ This is the documentation and API of the `eReuse.org DeviceHub
 
     events
     tags
+    inventory
 
 * :ref:`genindex`
 * :ref:`modindex`

docs/inventory.rst

@@ -0,0 +1,61 @@
+Inventory
+=======
+
+Devicehub uses the same path to get devices and lots.
+
+To get all devices and groups: ``GET /inventory`` or the devices of a
+specific groups: ``GET /inventory/24``.
+
+You can **filter** devices ``GET /inventory/24?filter={"type": "Computer"}``,
+and **sort** them ``GET /inventory?sort={"created": 1}``, and of course
+you can combine both in the same query. You only get the groups that
+contain the devices that pass the filters. So, if a group contains
+only one device that is filtered, you don't get that group neither.
+
+Results are **paginated**; you get up to 30 devices and up to 30
+groups in a page. Select the actual page by ``GET /inventory?page=3``.
+By default you get the page number ``1``.
+
+Query
+-----
+The query consists of 4 optional params:
+
+- **search**: Filters devices by performing a full-text search over their
+  physical properties, events, tags, and groups they are in:
+
+    - Device.type
+    - Device.serial_number
+    - Device.model
+    - Device.manufacturer
+    - Device.color
+    - Tag.id
+    - Tag.org
+    - Group.name
+
+  Search is a string.
+- **filter**: Filters devices field-by-field. Each field can be
+  filtered in different ways, see them in
+  :class:`ereuse_devicehub.resources.inventory.Filters`. Filter is
+  a JSON-encoded object whose keys are the filters. By default
+  is empty (no filter applied).
+- **sort**: Sorts the devices. You can specify multiple sort clauses
+  as it is a JSON-encoded object whose keys are fields and values
+  are truthy for *ascending* order, or falsy for *descending* order.
+  By default it is sorted by ``Device.created`` descending (newest
+  devices first).
+- **page**: A natural number that specifies the page to retrieve.
+  By default is ``1``; the first page.
+
+Result
+------
+The result is a JSON object with the following fields:
+
+- **devices**: A list of devices.
+- **groups**: A list of groups.
+- **widgets**: A dictionary of widgets.
+- **pagination**: Pagination information:
+
+  - **page**: The page you requested in the ``page`` param of the query,
+    or ``1``.
+  - **perPage**: How many devices are in every page, fixed to ``30``.
+  - **total**: How many total devices passed the filters.

ereuse_devicehub/client.py

@@ -1,10 +1,10 @@
-from typing import Any, Dict, Iterable, Tuple, Type, Union, Generator
+from inspect import isclass
+from typing import Any, Dict, Iterable, Tuple, Type, Union
 
-from boltons.typeutils import issubclass
 from flask import Response
 from werkzeug.exceptions import HTTPException
 
-from ereuse_devicehub.resources.models import Thing
+from ereuse_devicehub.resources import models, schemas
 from ereuse_utils.test import JSON
 from teal.client import Client as TealClient
 from teal.marshmallow import ValidationError
@@ -21,7 +21,7 @@ class Client(TealClient):
 
     def open(self,
              uri: str,
-             res: Union[str, Type[Thing]] = None,
+             res: Union[str, Type[Union[models.Thing, schemas.Thing]]] = None,
              status: Union[int, Type[HTTPException], Type[ValidationError]] = 200,
              query: Iterable[Tuple[str, Any]] = tuple(),
              accept=JSON,
@@ -30,14 +30,14 @@ class Client(TealClient):
              headers: dict = None,
              token: str = None,
              **kw) -> Tuple[Union[Dict[str, Any], str], Response]:
-        if issubclass(res, Thing):
+        if isclass(res) and issubclass(res, (models.Thing, schemas.Thing)):
-            res = res.__name__
+            res = res.t
         return super().open(uri, res, status, query, accept, content_type, item, headers, token,
                             **kw)
 
     def get(self,
             uri: str = '',
-            res: Union[Type[Thing], str] = None,
+            res: Union[Type[Union[models.Thing, schemas.Thing]], str] = None,
             query: Iterable[Tuple[str, Any]] = tuple(),
             status: Union[int, Type[HTTPException], Type[ValidationError]] = 200,
             item: Union[int, str] = None,
@@ -50,7 +50,7 @@ class Client(TealClient):
     def post(self,
              data: str or dict,
              uri: str = '',
-             res: Union[Type[Thing], str] = None,
+             res: Union[Type[Union[models.Thing, schemas.Thing]], str] = None,
              query: Iterable[Tuple[str, Any]] = tuple(),
              status: Union[int, Type[HTTPException], Type[ValidationError]] = 201,
              content_type: str = JSON,
@@ -67,7 +67,7 @@ class Client(TealClient):
         return self.post({'email': email, 'password': password}, '/users/login', status=200)
 
     def get_many(self,
-                 res: Union[Type[Thing], str],
+                 res: Union[Type[Union[models.Thing, schemas.Thing]], str],
                  resources: Iterable[dict],
                  key: str = None,
                  headers: dict = None,
@@ -101,7 +101,7 @@ class UserClient(Client):
 
     def open(self,
              uri: str,
-             res: Union[str, Type[Thing]] = None,
+             res: Union[str, Type[Union[models.Thing, schemas.Thing]]] = None,
              status: int or HTTPException = 200,
              query: Iterable[Tuple[str, Any]] = tuple(),
              accept=JSON,

ereuse_devicehub/resources/device/models.py

@@ -3,14 +3,14 @@ from itertools import chain
 from operator import attrgetter
 from typing import Dict, Set
 
-from sqlalchemy import BigInteger, Column, Float, ForeignKey, Integer, Sequence, SmallInteger, \
+from sqlalchemy import BigInteger, Column, Enum as DBEnum, Float, ForeignKey, Integer, Sequence, \
-    Unicode, inspect, Enum as DBEnum
+    SmallInteger, Unicode, inspect
 from sqlalchemy.ext.declarative import declared_attr
 from sqlalchemy.orm import ColumnProperty, backref, relationship
 from sqlalchemy.util import OrderedSet
 from sqlalchemy_utils import ColorType
 
-from ereuse_devicehub.resources.enums import DataStorageInterface, RamInterface, RamFormat
+from ereuse_devicehub.resources.enums import DataStorageInterface, RamFormat, RamInterface
 from ereuse_devicehub.resources.models import STR_BIG_SIZE, STR_SIZE, STR_SM_SIZE, Thing
 from ereuse_utils.naming import Naming
 from teal.db import CASCADE, POLYMORPHIC_ID, POLYMORPHIC_ON, ResourceNotFound, check_range

ereuse_devicehub/resources/device/models.pyi

@@ -3,6 +3,7 @@ from typing import Dict, List, Set
 from colour import Color
 from sqlalchemy import Column
 
+from ereuse_devicehub.resources.enums import RamInterface, RamFormat, DataStorageInterface
 from ereuse_devicehub.resources.event.models import Event, EventWithMultipleDevices, \
     EventWithOneDevice
 from ereuse_devicehub.resources.image.models import ImageList
@@ -91,10 +92,12 @@ class GraphicCard(Component):
 
 class DataStorage(Component):
     size = ...  # type: Column
+    interface = ... # type: Column
 
     def __init__(self, **kwargs) -> None:
         super().__init__(**kwargs)
         self.size = ...  # type: int
+        self.interface = ... # type: DataStorageInterface
 
 
 class HardDrive(DataStorage):
@@ -144,8 +147,12 @@ class Processor(Component):
 class RamModule(Component):
     size = ...  # type: Column
     speed = ...  # type: Column
+    interface = ... # type: Column
+    format = ... # type: Column
 
     def __init__(self, **kwargs) -> None:
         super().__init__(**kwargs)
         self.size = ...  # type: int
         self.speed = ...  # type: float
+        self.interface = ... # type: RamInterface
+        self.format = ... # type: RamFormat

ereuse_devicehub/resources/event/models.py

@@ -366,16 +366,18 @@ class StressTest(Test):
     pass
 
 
-class Benchmark(EventWithOneDevice):
+class Benchmark(JoinedTableMixin, EventWithOneDevice):
     pass
 
 
 class BenchmarkDataStorage(Benchmark):
-    readSpeed = Column(Float(decimal_return_scale=2), nullable=False)
+    id = Column(UUID(as_uuid=True), ForeignKey(Benchmark.id), primary_key=True)
-    writeSpeed = Column(Float(decimal_return_scale=2), nullable=False)
+    read_speed = Column(Float(decimal_return_scale=2), nullable=False)
+    write_speed = Column(Float(decimal_return_scale=2), nullable=False)
 
 
 class BenchmarkWithRate(Benchmark):
+    id = Column(UUID(as_uuid=True), ForeignKey(Benchmark.id), primary_key=True)
     rate = Column(SmallInteger, nullable=False)
 
 
@@ -395,7 +397,7 @@ class BenchmarkRamSysbench(BenchmarkWithRate):
 @event.listens_for(TestDataStorage.device, 'set', retval=True, propagate=True)
 @event.listens_for(Install.device, 'set', retval=True, propagate=True)
 @event.listens_for(EraseBasic.device, 'set', retval=True, propagate=True)
-def validate_device_is_data_storage(target, value, old_value, initiator):
+def validate_device_is_data_storage(target: Event, value: DataStorage, old_value, initiator):
     if not isinstance(value, DataStorage):
         raise TypeError('{} must be a DataStorage but you passed {}'.format(initiator.impl, value))
     return value

ereuse_devicehub/resources/event/models.pyi

@@ -8,7 +8,7 @@ from sqlalchemy.orm import relationship
 
 from ereuse_devicehub.resources.device.models import Component, Computer, Device
 from ereuse_devicehub.resources.enums import AppearanceRange, Bios, FunctionalityRange, \
-    RatingSoftware, SnapshotSoftware, TestHardDriveLength, SnapshotExpectedEvents
+    RatingSoftware, SnapshotExpectedEvents, SnapshotSoftware, TestHardDriveLength
 from ereuse_devicehub.resources.image.models import Image
 from ereuse_devicehub.resources.models import Thing
 from ereuse_devicehub.resources.user import User
@@ -20,7 +20,7 @@ class Event(Thing):
     name = ...  # type: Column
     date = ...  # type: Column
     type = ...  # type: Column
-    error = ... # type: Column
+    error = ...  # type: Column
     incidence = ...  # type: Column
     description = ...  # type: Column
     finalized = ...  # type: Column
@@ -90,7 +90,7 @@ class Snapshot(EventWithOneDevice):
         self.elapsed = ...  # type: timedelta
         self.device = ...  # type: Computer
         self.events = ...  # type: Set[Event]
-        self.expected_events = ... # type: List[SnapshotExpectedEvents]
+        self.expected_events = ...  # type: List[SnapshotExpectedEvents]
 
 
 class Install(EventWithOneDevice):
@@ -109,9 +109,10 @@ class SnapshotRequest(Model):
 
 
 class Rate(EventWithOneDevice):
-    rating = ... # type: Column
+    rating = ...  # type: Column
-    appearance = ... # type: Column
+    appearance = ...  # type: Column
-    functionality = ... # type: Column
+    functionality = ...  # type: Column
+
     def __init__(self, **kwargs) -> None:
         super().__init__(**kwargs)
         self.rating = ...  # type: float
@@ -216,3 +217,37 @@ class EraseBasic(EventWithOneDevice):
 class EraseSectors(EraseBasic):
     def __init__(self, **kwargs) -> None:
         super().__init__(**kwargs)
+
+
+class Benchmark(EventWithOneDevice):
+    pass
+
+
+class BenchmarkDataStorage(Benchmark):
+    read_speed = ...  # type: Column
+    write_speed = ...  # type: Column
+
+    def __init__(self, **kwargs) -> None:
+        super().__init__(**kwargs)
+        self.read_speed = ...  # type: float
+        self.write_speed = ...  # type: float
+
+
+class BenchmarkWithRate(Benchmark):
+    rate = ...  # type: Column
+
+    def __init__(self, **kwargs) -> None:
+        super().__init__(**kwargs)
+        self.rate = ...  # type: int
+
+
+class BenchmarkProcessor(BenchmarkWithRate):
+    pass
+
+
+class BenchmarkProcessorSysbench(BenchmarkProcessor):
+    pass
+
+
+class BenchmarkRamSysbench(BenchmarkWithRate):
+    pass

ereuse_devicehub/resources/inventory.py

@@ -1,16 +1,19 @@
-from flask import current_app as app, jsonify
+from flask import current_app, current_app as app, jsonify
+from flask_sqlalchemy import Pagination
 from marshmallow import Schema as MarshmallowSchema
-from marshmallow.fields import Float, Nested, Str
+from marshmallow.fields import Float, Integer, Nested, Str
+from marshmallow.validate import Range
+from sqlalchemy import Column
 
 from ereuse_devicehub.resources.device.models import Device
 from ereuse_devicehub.resources.event.models import Rate
+from ereuse_devicehub.resources.schemas import Thing
 from ereuse_devicehub.resources.tag import Tag
-from teal.marshmallow import IsType
+from teal.query import Between, FullTextSearch, ILike, Join, Or, Query, Sort, SortField
-from teal.query import Between, Equal, ILike, Or, Query
+from teal.resource import Resource, View
-from teal.resource import Resource, Schema, View
 
 
-class Inventory(Schema):
+class Inventory(Thing):
     pass
 
 
@@ -25,23 +28,56 @@ class TagQ(Query):
     org = ILike(Tag.org)
 
 
+class OfType(Str):
+    def __init__(self, column: Column, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.column = column
+
+    def _deserialize(self, value, attr, data):
+        v = super()._deserialize(value, attr, data)
+        return self.column.in_(current_app.resources[v].subresources_types)
+
+
 class Filters(Query):
-    type = Or(Equal(Device.type, Str(validate=IsType(Device.t))))
+    type = Or(OfType(Device.type))
     model = ILike(Device.model)
     manufacturer = ILike(Device.manufacturer)
     serialNumber = ILike(Device.serial_number)
-    rating = Nested(RateQ)  # todo db join
+    rating = Join(Device.id == Rate.device_id, RateQ)
-    tag = Nested(TagQ)  # todo db join
+    tag = Join(Device.id == Tag.id, TagQ)
+
+
+class Sorting(Sort):
+    created = SortField(Device.created)
 
 
 class InventoryView(View):
     class FindArgs(MarshmallowSchema):
-        where = Nested(Filters, default={})
+        search = FullTextSearch()  # todo Develop this. See more at docs/inventory.
+        filter = Nested(Filters, missing=[])
+        sort = Nested(Sorting, missing=[Device.created.desc()])
+        page = Integer(validate=Range(min=1), missing=1)
 
-    def find(self, args):
+    def find(self, args: dict):
-        devices = Device.query.filter_by()
+        """
+        Supports the inventory view of ``devicehub-client``; returns
+        all the devices, groups and widgets of this Devicehub instance.
+
+        The result can be filtered, sorted, and paginated.
+        """
+        devices = Device.query \
+            .filter(*args['filter']) \
+            .order_by(*args['sort']) \
+            .paginate(page=args['page'], per_page=30)  # type: Pagination
         inventory = {
-            'devices': app.resources[Device.t].schema.dump()
+            'devices': app.resources[Device.t].schema.dump(devices.items, many=True),
+            'groups': [],
+            'widgets': {},
+            'pagination': {
+                'page': devices.page,
+                'perPage': devices.per_page,
+                'total': devices.total,
+            }
         }
         return jsonify(inventory)
 

tests/test_device.py

@@ -3,8 +3,6 @@ from uuid import UUID
 
 import pytest
 from colour import Color
-
-from ereuse_utils.naming import Naming
 from pytest import raises
 from sqlalchemy.util import OrderedSet
 
@@ -20,6 +18,7 @@ from ereuse_devicehub.resources.device.sync import MismatchBetweenTags, Mismatch
 from ereuse_devicehub.resources.event.models import Remove, Test
 from ereuse_devicehub.resources.tag.model import Tag
 from ereuse_devicehub.resources.user import User
+from ereuse_utils.naming import Naming
 from teal.db import ResourceNotFound
 from tests.conftest import file
 
@@ -133,7 +132,7 @@ def test_add_remove():
     # c4 is not with any pc
     values = file('pc-components.db')
     pc = values['device']
-    c1, c2 = [Component(**c) for c in values['components']]
+    c1, c2 = (Component(**c) for c in values['components'])
     pc = Computer(**pc, components=OrderedSet([c1, c2]))
     db.session.add(pc)
     c3 = Component(serial_number='nc1')

tests/test_inventory.py

@@ -1,8 +1,11 @@
 import pytest
-from sqlalchemy.sql.elements import BinaryExpression
 
-from ereuse_devicehub.resources.device.models import Device
+from ereuse_devicehub.client import UserClient
-from ereuse_devicehub.resources.inventory import Filters, InventoryView
+from ereuse_devicehub.db import db
+from ereuse_devicehub.devicehub import Devicehub
+from ereuse_devicehub.resources.device.models import Desktop, Device, Laptop, Microtower, \
+    SolidStateDrive
+from ereuse_devicehub.resources.inventory import Filters, Inventory, Sorting
 from teal.utils import compiled
 
 
@@ -10,7 +13,7 @@ from teal.utils import compiled
 def test_inventory_filters():
     schema = Filters()
     q = schema.load({
-        'type': ['Microtower', 'Laptop'],
+        'type': ['Computer', 'Laptop'],
         'manufacturer': 'Dell',
         'rating': {
             'rating': [3, 6],
@@ -22,28 +25,86 @@ def test_inventory_filters():
     })
     s, params = compiled(Device, q)
     # Order between query clauses can change
-    assert '(device.type = %(type_1)s OR device.type = %(type_2)s)' in s
+    assert '(device.type IN (%(type_1)s, %(type_2)s, %(type_3)s, %(type_4)s, ' \
+           '%(type_5)s, %(type_6)s) OR device.type IN (%(type_7)s))' in s
     assert 'device.manufacturer ILIKE %(manufacturer_1)s' in s
     assert 'rate.rating BETWEEN %(rating_1)s AND %(rating_2)s' in s
     assert 'rate.appearance BETWEEN %(appearance_1)s AND %(appearance_2)s' in s
     assert '(tag.id ILIKE %(id_1)s OR tag.id ILIKE %(id_2)s)' in s
-    assert params == {
+
-        'type_1': 'Microtower',
+    # type_x can be assigned at different values
-        'rating_2': 6.0,
+    # ex: type_1 can be 'Desktop' in one execution but the next one 'Laptop'
-        'manufacturer_1': 'Dell%',
+    assert set(params.keys()) == {
-        'appearance_1': 2.0,
+        'id_1',
-        'appearance_2': 4.0,
+        'manufacturer_1',
-        'id_1': 'bcn-%',
+        'type_4',
-        'rating_1': 3.0,
+        'type_3',
-        'id_2': 'activa-02%',
+        'id_2',
-        'type_2': 'Laptop'
+        'type_1',
+        'rating_1',
+        'type_5',
+        'appearance_2',
+        'type_6',
+        'type_7',
+        'appearance_1',
+        'rating_2',
+        'type_2'
+    }
+    assert set(params.values()) == {
+        'bcn-%',
+        'Dell%',
+        'Laptop',
+        'Server',
+        'activa-02%',
+        'Computer',
+        3.0,
+        'Microtower',
+        4.0,
+        'Netbook',
+        'Laptop',
+        2.0,
+        6.0,
+        'Desktop'
     }
 
 
 @pytest.mark.usefixtures('app_context')
-def test_inventory_query():
+def test_inventory_sort():
-    schema = InventoryView.FindArgs()
+    schema = Sorting()
-    args = schema.load({
+    r = next(schema.load({'created': True}))
-        'where': {'type': ['Computer']}
+    assert str(r) == 'device.created ASC'
-    })
+
-    assert isinstance(args['where'], BinaryExpression), '``where`` must be a SQLAlchemy query'
+
+@pytest.fixture()
+def inventory_query_dummy(app: Devicehub):
+    with app.app_context():
+        db.session.add_all((  # The order matters ;-)
+            Desktop(serial_number='s1', model='ml1', manufacturer='mr1'),
+            Laptop(serial_number='s3', model='ml3', manufacturer='mr3'),
+            Microtower(serial_number='s2', model='ml2', manufacturer='mr2'),
+            SolidStateDrive(serial_number='s4', model='ml4', manufacturer='mr4')
+        ))
+        db.session.commit()
+
+
+@pytest.mark.usefixtures('inventory_query_dummy')
+def test_inventory_query_no_filters(user: UserClient):
+    i, _ = user.get(res=Inventory)
+    assert tuple(d['type'] for d in i['devices']) == (
+        'SolidStateDrive', 'Microtower', 'Laptop', 'Desktop'
+    )
+
+
+@pytest.mark.usefixtures('inventory_query_dummy')
+def test_inventory_query_filter_type(user: UserClient):
+    i, _ = user.get(res=Inventory, query=[('filter', {'type': ['Computer', 'Microtower']})])
+    assert tuple(d['type'] for d in i['devices']) == ('Microtower', 'Laptop', 'Desktop')
+
+
+@pytest.mark.usefixtures('inventory_query_dummy')
+def test_inventory_query_filter_sort(user: UserClient):
+    i, _ = user.get(res=Inventory, query=[
+        ('sort', {'created': Sorting.ASCENDING}),
+        ('filter', {'type': ['Computer']})
+    ])
+    assert tuple(d['type'] for d in i['devices']) == ('Desktop', 'Laptop', 'Microtower')