Skip to main content

Barcode Less Items Recognition (BLIR) on Self Checkout (SCO) - Integration Guide

Solution Description​

This document outlines the integration guide for integrating with Edgify agent via REST APIs. The Self Checkout UI application (“UI Application”) interacts with the Edgify Agent (“Agent”) via 2 main REST APIs: Capture & Predict and Put Sample (see below). The Edgify Agent can be installed on a Windows device, in which it will run as a service or on Linux devices in a Docker container. The agent enables SCO devices to execute real-time predictions and obtain the certainty of these predictions. The agent is capable of detecting false selections and initiating intervention procedures via the SCO screen and attendant application.

Video Examples​

You can see here an example of integrating with our software:

  • NCR Retalix SCO Integration

API Flow for Item Recognition​

The calls need to be authenticated using “Digest Authentication” *Username and Password should be in a config file and will change between customers.

POST Capture and Predict​

Endpoint​

POST http://localhost:8090/api/v2/capture-and-predict

Description:​

The Agent captures an image from the camera feed and infers a prediction.

Request Body Example:​

{
  "captureTrigger": "scale-stable",
  "family": "fruits-and-vegetables"
}

Request Body Parameters​

ParameterExplanationValues (Explanation)
captureTriggerMechanism/flow which prompted the agent to capture an imageauto-scale: (scale weight settles above x)
barcode: (Barcoded item is scanned)
button: (On-screen button, could be linked to the existing lookup/picklist button)
familyDepartment / category - informs Edgify from what item’s assortment the prediction will probably be. This value can come from the name of the menu in the SCO application. Example familyFruits
Vegetables
Bakery
OrganicVegetables
OrganicFruits
This list is not exhaustive - you can send any string value that defines the family of items you wish Edgify to predict

* family is relevant only when the “Get Prediction” API is called from a menu button, else simply pass an empty value for this parameter.

Response Example​

{
    "sampleId": "65506cf8-1aa0-4bf3-bcbf-7656ab85a158",
    "certain": false,
    "options": [
        {
            "label": "4011",
            "certainty": 0.98
        }
    ],
    "image": "/9j/4AAQSkZJRgABAQAAAQABAAD/.......",
    }

Response Body Parameters​

ParameterExplanation
"sampleId"Unique ID returned with each inference
"certain"See below
"options"Defines the start of the prediction object. This section will return one or multiple predictions
"label"PLU or barcode for a given item
"certainty"Confidence of prediction as a percentage
"image"The image captured by the agent from the camera feed, encoded in Base64

"Unknown" prediction response​

When confidence in recognition is very low, the agent will return a prediction object with “Unknown” in the prediction array. This is a normal and key part of the systems ability to learn.

“Certain” flag​

Indicates exceptionally high confidence in item recognition. When this flag returns true, it prompts the receiving application software to perform enhanced functionality, such as automatically adding the item straight into the customer's basket or printing a ticket without requiring further validation from the user. Handling of this flag is optional.

PUT samples​

Endpoint​

PUT http://localhost:8090/api/v2/samples/{sample_id}

Description​

Saves the sampleId and Image with the chosen label, after the customer has confirmed the Edgify prediction or chosen the product from a menu.​ Same as the Capture & Predict API, The call needs to be authenticated using “Digest Authentication” Each prediction has a unique sample_id, use that id in the api request In the request body.

Request Body Example​

{
  "captureTrigger": "auto-scale",
  "labelTrigger": "RecommendationStrip",
  "category": "ExoticFruits",
  "count": 0,
  "groupId": "b932de54-41ca-4f5a-99d5-b9db779f8888",
  "sampleId": "9c9847e6-591b-4f29-a988-922da9e56484",
  "image": "/9j/4AAQSkZJRgABAQAA........",
  "imgSource": "",
  "label": "4011",
  "name": "banana",
  "price": 1.2,
  "weight": 1.5
  "totalPrice": 1.8,
}

Request Body Parameters​

| Header | Explanation | Values (Explanation) | | :--------------------- | :----------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | | "sampleId" | Unique ID returned with each inference | Pass the value returned by Capture & Predict API | | "captureTrigger" | Mechanism/flow which prompted the agent to capture an image | Pass the same value as Capture & Predict API | | "labelTrigger" | Mechanism by which user selects/identifies the item they are purchasing | "EdgifyPrediction": (Edgify predictions shown on-screen for the customer to select) "ManualLookup": (Customer searches the picklist / menus to find their item - AKA the normal process on non-AI enabled scales) "HandScanner": (For SCO/Lane POS where an additional barcode scanner is employed) | | "category" optional | Retailer catalog hierarchy | Will depend on the retailer, for example: "CitrusFruits" or "Alcohol" or "PowerTools" or "ExoticVegetables" or "Pets" or "Clothing", | | etc. | | "groupId" optional | (enables future Edgify functionality) | | | "imgSource" optional | ID of the image capture device | Applicable for IP camera use only. "IP-Cam-10-44-01" or "192.168.1.101" | | "image" | The image captured by the agent from the camera feed, encoded in Base64 | | | "label" | PLU or barcode for a given item | "8547230239400" or "4011" or "4133" | | "name" | The English / Human readable item name | "DietCoke20oz" or "BananaOrganic" or "GalaApple" | | "count" | Number of items being purchased | Barcoded items count is always "1" | | | "price" | Item cost per unit weight | Applicable only for weighted items, else do not pass this header | | "weight" | The weight of item being purchased | Applicable only for weighted items, else do not pass this header | | "totalPrice" | Product of "price" x "weight" - the total price that the shopper sees on the SCO screen after the scale is stable and the item is selected | Barcoded item price should be passed here |

Response Example​

The PUT Sample call returns a matchStrength value indicating whether this sample does not match the label provided.

{
    "matchStrength": "alert-non-match"
}

Response Body Parameters​

HeaderExplanationValues (Explanation)
"matchStrength"Logic determining if the label received by Edgify in the Samples API matches the item Edgify sees"match": (Label received matches item seen - AKA no loss event suspected) "alert-non-match": (Label received does not match item seen by Edgify - Possible loss event/false selection/barcode switching) "unknown": Edgify was unable to confidently match or identify a non-match

API Flow for False Selections Intervention​

POST events - For managing Interventions​

Endpoint​

POST http://localhost:8090/api/v2/events

Description:​​

The following endpoints should be called with different values in the "name" parameter upon the shopper's selection for the false selection message. They enable the system to measure the impact of the intervention.

POST /event with "non-match-continue​"​

Endpoint​

POST http://localhost:8090/api/v2/events
{
    "name": "non-match-continue" , "sampleId": "UUID",
}

POST /event with "non-match-startover​"​

Endpoint​

POST http://localhost:8090/api/v2/events
{
    "name": "non-match-startover" , "sampleId": "UUID",
}

The following endpoints should be called upon the attendant's selection. They enable the system to measure the impact of the second level of intervention.

POST /event with "attendant-rejected"​

The attendant rejected the alert

Endpoint​

POST http://localhost:8090/api/v2/events
{
    "name": "attendant-rejected" , "sampleId": "UUID",
}

Delete Sample​

Delete Sample​- Call this endpoint in case the Store manager passes the card and removes the item from the shopping list

Endpoint​

DELETE http://localhost:8090/api/v2/samples/{sample_id}

Description​

Deletes the passed Sample from the Edgify system.

  • For specialized use cases. Please consult your Customer Success Team.

Important Logs​

Make sure you write to the SCO application log upon each call to Edgify's APIs. Include timestamp and the meta data sent and received.

Documentation​

Please provide basic documentation or a short video showing how to use / navigate the integration and staff logins or backend menus included in your application software.

Testing Integration of Self CheckOut​

After integration we do recommend jumping on a call with Edgify Sales Engineer (SE) to test the integration.

A few use cases should be tested after integration:

  • Edgify returns 1 item
  • Edgify returns multiple items
  • Edgify returns “Unknown”
  • Edgify Certain: true - (If this flow was implemented)
  • Edgify gets GT with the correct data.
  • Edgify is down (no response)
  • Toggle Silent/Active Mode

Note: For developing & testing purposes Edgify can change the configuration to provide any of the above flows every time by changing the agent parameters.

Data Collection Flow​

Edgify recommends you Implement a flow for collecting product image samples and SKUs for new Products In order to be able to add new SKUs to the system without having to wait for customers actually buying each item 500 times. You might want to add to your client software the capability to capture images of new items faster. Hence, it's important that the client software will implement a feature for capturing multiple images and ground truth rapidly.