Product Visual Search — Ultimate Guide

Introduction

HMS ML kit service searches in pre-established product image library for the same or similar products based on a product image taken by a customer, and returns the IDs of those products and related information.

Use Case

1. We will capture the product image using device camera from our developed shopping application.
2. We will show the returned products list in recycle view.

Prerequisite

1. Java JDK 1.8 or higher is recommended.
2. Android Studio is recommended.
3. Huawei android device with HMS core 4.0.0.300 or higher.
4. Before developing an app, you will need to register as a HUAWEI developer. Refer to Register a HUAWEI ID.
5. Integrating app gallery connect SDK. Refer to AppGallery Connect Service Getting Started.

Implementation

1. Enable ML kit in Manage APIs. Refer to Service Enabling.
2. Integrate following dependencies in app-level build.gradle.

// Import the product visual search SDK.
implementation 'com.huawei.hms:ml-computer-vision-cloud:2.0.1.300'

3. Add agc plugin in the top of app.gradle file.

apply plugin: 'com.huawei.agconnect'

4. Add the following permission in manifest.

• Camera permission android.permission.CAMERA: Obtains real-time images or videos from a camera.
• Internet access permission android.permission.INTERNET: Accesses cloud services on the Internet.
• Storage write permission android.permission.WRITE_EXTERNAL_STORAGE: Upgrades the algorithm version.
• Storage read permission android.permission.READ_EXTERNAL_STORAGE: Reads photos stored on a device.

5. To request camera permission in realtime.

private void requestCameraPermission() {
    final String[] permissions = new String[] {Manifest.permission.CAMERA};

    if (!ActivityCompat.shouldShowRequestPermissionRationale(this, Manifest.permission.CAMERA)) {
        ActivityCompat.requestPermissions(this, permissions, this.CAMERA_PERMISSION_CODE);
        return;
    }
}

6. Add following code in Application class

public class MyApplication extends Application {

    @Override
    public void onCreate() {
        super.onCreate();
        MLApplication.getInstance().setApiKey("API KEY");
    }
}

API key can be obtained either from AGC or integrated agcconnect-services.json.

7. To create an analyzer for product visual search.

private void initializeProductVisionSearch() {
    MLRemoteProductVisionSearchAnalyzerSetting settings = new MLRemoteProductVisionSearchAnalyzerSetting.Factory()
            // Set the maximum number of products that can be returned.
            .setLargestNumOfReturns(16)
            // Set the product set ID. (Contact mlkit@huawei.com to obtain the configuration guide.)
            //                .setProductSetId(productSetId)
            // Set the region.
            .setRegion(MLRemoteProductVisionSearchAnalyzerSetting.REGION_DR_CHINA)
            .create();
    analyzer
            = MLAnalyzerFactory.getInstance().getRemoteProductVisionSearchAnalyzer(settings);
}

8. To capture image from camera.

Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
startActivityForResult(intent, REQ_CAMERA_CODE);

9. Once image has been captured, onActivityResult() method will be executed.

@Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
    super.onActivityResult(requestCode, resultCode, data);
    Log.d(TAG, "onActivityResult");
    if(requestCode == REQ_CAMERA_CODE) {
        if (resultCode == RESULT_OK) {
            Bitmap bitmap = (Bitmap) data.getExtras().get("data");
            if (bitmap != null) {
                // Create an MLFrame object using the bitmap, which is the image data in bitmap format.
                MLFrame mlFrame = new MLFrame.Creator().setBitmap(bitmap).create();
                mlImageDetection(mlFrame);
            }

        }

    }
}

private void mlImageDetection(MLFrame mlFrame) {
    Task> task = analyzer.asyncAnalyseFrame(mlFrame);
    task.addOnSuccessListener(new OnSuccessListener>() {
        public void onSuccess(List products) {
            // Processing logic for detection success.

            displaySuccess(products);

        }})
         .addOnFailureListener(new OnFailureListener() {
        public void onFailure(Exception e) {
            // Processing logic for detection failure.
            // Recognition failure.
            try {
                MLException mlException = (MLException)e;
                // Obtain the result code. You can process the result code and customize respective messages displayed to users.
                int errorCode = mlException.getErrCode();
                // Obtain the error information. You can quickly locate the fault based on the result code.
                String errorMessage = mlException.getMessage();
            } catch (Exception error) {
                // Handle the conversion error.
            }
        }
    });
}


private void displaySuccess(List productVisionSearchList) {
    List productImageList = new ArrayList();
    String prodcutType = "";
    for (MLProductVisionSearch productVisionSearch : productVisionSearchList) {
        Log.d(TAG, "type: " + productVisionSearch.getType() );
        prodcutType =  productVisionSearch.getType();
        for (MLVisionSearchProduct product : productVisionSearch.getProductList()) {
            productImageList.addAll(product.getImageList());
            Log.d(TAG, "custom content: " + product.getCustomContent() );
        }
    }
    StringBuffer buffer = new StringBuffer();
    for (MLVisionSearchProductImage productImage : productImageList) {
        String str = "ProductID: " + productImage.getProductId() + "
        ImageID: " + productImage.getImageId() + "
        Possibility: " + productImage.getPossibility();
        buffer.append(str);
        buffer.append("
                ");
    }
    Log.d(TAG , "display success: " + buffer.toString());
    FragmentTransaction transaction = getFragmentManager().beginTransaction();
    transaction.replace(R.id.main_fragment_container, new SearchResultFragment(productImageList, prodcutType ));
    transaction.commit();
}

onSuccess() callback will give us list of MLProductVisionSearch objects, which can be used to get product id and image URL of each product. Also we can get the product type using productVisionSearch.getType(). getType() returns number which can be mapped.

At time of writing of this article product types are

10. We can achieve the product type mapping with following code.

private String getProductType(String type) {
    switch(type) {
        case "0":
            return "Others";
        case "1":
            return "Clothing";
        case "2":
            return "Shoes";
        case "3":
            return "Bags";
        case "4":
            return "Digital & Home appliances";
        case "5":
            return "Household Products";
        case "6":
            return "Toys";
        case "7":
            return "Cosmetics";
        case "8":
            return "Accessories";
        case "9":
            return "Food";
    }
    return  "Others";
    }

Images

Reference

https://developer.huawei.com/consumer/en/doc/development/HMSCore-Guides-V5/sdk-data-security-0000001050040129-V5