Abstract data layer

The abstract data layer is a layer between the physical storage of data and the applications that use it, allowing abstraction from physical implementation and hardware specifics, providing a single interface for working with data. It provides:

• Hardware Independence: Data is stored on different devices (servers, cloud, IoT devices) but the application interacts with it through a single interface.
• Simplification of development: Developers work with abstract entities without worrying about the details of physical storage.
• Flexibility: Changes to the physical structure of data (e.g., migration to the cloud) do not affect applications.

Unified data model

The system uses a unified data model, which is adapted for different types of equipment and allows you to work with them in a uniform way, regardless of their source or type of equipment. It includes common data structures, uniform processing rules, and semantic consistency of data across all systems and subsystems.

For example, a unified system data model includes:

• Entities: Equipment, channels, sessions, units, reference parameters.
• Relationships: Equipment has channels, channels are related to data, data is related to sessions.
• Attributes: Attributes are defined for each entity (for example, equipment.serial_number, channel_data.value).

Structure of the Unified System Data Model

The Unified System Data Model is a structure for storing and managing data related to hardware, channels, sessions, and measurements. Let’s look at each table and its purpose, as well as the relationships between them.

Table channel_data

This table stores the data received from the equipment channels within specific sessions.

Relationships:

• The unit_id foreign key references the units table.
• The archive_type_id foreign key references the referenceparameters table.
• The equipment_id foreign key references the equipment table.
• The channel_id foreign key references the channels table.
• The seance_id foreign key refers to the seances table.

Table units

This table stores the units of measurement used for the channel data.

Table referenceparameters

This table stores reference parameters such as archive types or other qualifiers.

Relationships:

• The foreign key parent_id references the same table (referenceparameters.id), allowing hierarchies to be built.

Table equipment

Table channels

This table stores information about the equipment channels.

Relationships:

• The equipment_type_id foreign key references the equipment_type table.
• The unit_id foreign key refers to the units table.

Table seances

This table stores information about the equipment’s communication sessions.

General structure and relationships

Main table - channel_data which links channel data to equipment, sessions and units.

Relationship tables:

• units - units of measurement.
• referenceparameters - archive types and other qualifiers.

Equipment Tables:

• equipment - equipment information.
• channels - equipment channels.

Sessions table - seances, which stores information about communication sessions.

Usage Example:

• Channel data (channel_data) comes from equipment (equipment) within specific sessions (seances).
• Each channel (channels) has its own unit of measurement (units).
• The type of archive (referenceparameters) determines how data should be stored or processed.

Benefits of a unified model

This data model provides flexibility and scalability to store and analyze data from different equipment.

Let’s list some of the benefits of a unified data model:

• Consistency: Data has the same structure and semantics across all systems.
• Scalability: It is easy to add new data sources or equipment types.
• Data Protection: Unified authentication, authorization, and encryption mechanisms.
• Simplify Analysis: Data can be analyzed using unified tools.

Equipment abstraction

Equipment abstraction is a key system design principle that allows you to separate the logic of data handling from the physical characteristics of the equipment. This is especially important in systems that utilize heterogeneous devices.

Equipment abstraction means that the system deals with data at a logical level that is independent of the physical devices on which the data is stored or processed. This is achieved by:

• Unification of interfaces: A single way of accessing data, regardless of equipment.
• Hiding implementation details: Physical location of data, transmission protocols, and other technical aspects are hidden from applications.
• Adapters: Conversion of data from an equipment-specific format to a unified format.

Implementing Equipment Abstraction

Let us consider the implementation of this approach in our system

Suppose we have an IoT device that sends gas consumption data. The implemented algorithms allow us to:

• Receive the data from the device through an adapter.
• Convert them into a channel_data structure.
• Store them in a database.
• Provide access to the data via API.

For each type of equipment, adapters are created that handle device-specific protocols and convert the data into a unified format.

Let’s look at an example:

For an IoT device, data can come in JSON format:

1
2
3
4
5

{
  "device_id": "sensor-123",
  "timestamp": 1739950861,
  "value": 42.5
}

The adapter converts data into a channel_data structure:

1
2
3

INSERT INTO channel_data (equipment_id, seance_id, channel_id,
 archive_type_id, value, event_time)
VALUES (1, 2, 3, 4, '42.5', 1739950861);

Data Request:

GET /api/v1/channel_data?equipment_id=1&archive_type=daily&channel_id=3

Response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[
  {
    "id": 123,
    "equipment_id": 1,
    "seance_id": 2,
    "channel_id": 3,
    "archive_type_id": 4,
    "value": 42.5,
    "event_time":  1739950861
  }
]

Advantages of Equipment Abstraction

• Hardware Independence: Applications work with data without knowing where it physically resides.
• Flexibility: Easily add new devices or modify existing devices.
• Simplification of development: Developers work with abstract entities without worrying about hardware details.
• Scalability: Data can be stored on different devices, but the system continues to operate in a uniform manner.

##Data Normalization Data normalization is the process of organizing data in a database in a way that minimizes redundancy and improves data integrity. In the context of the abstract data layer, normalization plays a key role in creating a unified and efficient data model that can be used to handle different types of hardware and data sources.

Data normalization is the process of dividing data into logical tables and establishing relationships between them in order to:

• Eliminating duplicate data.
• Simplifying data maintenance and updating.
• Ensuring data integrity.
• Improving query performance.
• The system’s unified data model is reduced to (third) normal Boyce-Codd form in terms of atomic (scalar) values. Some data model entities use composite structures to optimize the handling of rare non-standard attributes.

Benefits of normalization

• Removes redundancy: Data is stored in one place, reducing redundancy.
• Data Consistency: Maintaining data consistency is simplified.
• Flexibility: It is easy to make changes to the data structure.
• Productivity: Query performance is improved (in most cases).

Normalization in the context of the abstract data layer

Data normalization is an important step in the design of the abstract data layer.

It allows you to:

• Create a unified data model that can be used to handle different types of hardware.
• Eliminate redundancy and duplication of data.
• Ensure data integrity and consistency, regardless of its source.
• Simplify the integration of new devices and systems.